WIP: HYBX OMNL and deployment documentation updates
This commit is contained in:
defiQUG
231
docs/ccip-integration/operations/ADMIN_KEY_MANAGEMENT.md
Normal file
231
docs/ccip-integration/operations/ADMIN_KEY_MANAGEMENT.md
Normal file
@@ -0,0 +1,231 @@
|
||||
# Admin Key Management for CCIP Contracts
|
||||
|
||||
**Date**: 2025-01-27
|
||||
**Network**: ChainID 138 (DeFi Oracle Meta Mainnet)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document outlines best practices for managing admin keys for CCIP contracts on ChainID 138.
|
||||
|
||||
---
|
||||
|
||||
## Admin Functions
|
||||
|
||||
### CCIP Router Admin Functions
|
||||
|
||||
- `addSupportedChain(uint64 chainSelector)`
|
||||
- `removeSupportedChain(uint64 chainSelector)`
|
||||
- `addSupportedToken(uint64 chainSelector, address token)`
|
||||
- `updateFees(uint256 baseFee, uint256 dataFeePerByte)`
|
||||
- `changeAdmin(address newAdmin)`
|
||||
- `withdrawFees(uint256 amount)`
|
||||
- `withdrawNativeFees()`
|
||||
|
||||
### Bridge Admin Functions
|
||||
|
||||
- `addDestination(uint64 chainSelector, address receiverBridge)`
|
||||
- `removeDestination(uint64 chainSelector)`
|
||||
- `updateDestination(uint64 chainSelector, address receiverBridge)`
|
||||
- `updateFeeToken(address newFeeToken)`
|
||||
- `changeAdmin(address newAdmin)`
|
||||
|
||||
---
|
||||
|
||||
## Key Management Best Practices
|
||||
|
||||
### 1. Key Storage
|
||||
|
||||
**DO**:
|
||||
- Use hardware wallets (Ledger, Trezor)
|
||||
- Use secure key management systems (HSM, AWS KMS, Azure Key Vault)
|
||||
- Store keys in encrypted format
|
||||
- Use separate keys for different environments (testnet/mainnet)
|
||||
- Implement key rotation procedures
|
||||
|
||||
**DON'T**:
|
||||
- Store keys in plain text files
|
||||
- Commit keys to version control
|
||||
- Share keys via insecure channels
|
||||
- Use same key for multiple purposes
|
||||
- Store keys on shared systems
|
||||
|
||||
### 2. Access Control
|
||||
|
||||
**DO**:
|
||||
- Implement multi-sig for admin functions
|
||||
- Use role-based access control
|
||||
- Require multiple approvals for critical changes
|
||||
- Log all admin actions
|
||||
- Monitor for unauthorized access
|
||||
|
||||
**DON'T**:
|
||||
- Grant admin access to untrusted parties
|
||||
- Skip approval processes
|
||||
- Ignore access logs
|
||||
- Allow unmonitored admin access
|
||||
|
||||
### 3. Key Rotation
|
||||
|
||||
**Procedure**:
|
||||
|
||||
1. **Generate New Key**
|
||||
- Create new key pair
|
||||
- Store securely
|
||||
- Verify key works
|
||||
|
||||
2. **Transfer Admin**
|
||||
```bash
|
||||
cast send $CONTRACT \
|
||||
"changeAdmin(address)" \
|
||||
$NEW_ADMIN \
|
||||
--rpc-url $RPC_URL \
|
||||
--private-key $OLD_ADMIN_KEY
|
||||
```
|
||||
|
||||
3. **Verify Transfer**
|
||||
```bash
|
||||
cast call $CONTRACT "admin()" --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
4. **Revoke Old Key**
|
||||
- Securely delete old key
|
||||
- Update documentation
|
||||
- Notify team
|
||||
|
||||
### 4. Emergency Procedures
|
||||
|
||||
**If Key is Compromised**:
|
||||
|
||||
1. **Immediate Actions**:
|
||||
- Transfer admin to new secure key
|
||||
- Review recent transactions
|
||||
- Check for unauthorized changes
|
||||
- Notify team and stakeholders
|
||||
|
||||
2. **Investigation**:
|
||||
- Analyze access logs
|
||||
- Identify compromise source
|
||||
- Document incident
|
||||
|
||||
3. **Recovery**:
|
||||
- Restore from backup if needed
|
||||
- Revert unauthorized changes
|
||||
- Implement additional security
|
||||
|
||||
---
|
||||
|
||||
## Multi-Sig Setup
|
||||
|
||||
### Recommended: Use Multi-Sig Wallet
|
||||
|
||||
For production deployments, use a multi-sig wallet:
|
||||
|
||||
1. **Deploy Multi-Sig**
|
||||
- Use OpenZeppelin's MultiSigWallet
|
||||
- Set required confirmations (e.g., 2 of 3)
|
||||
- Add trusted signers
|
||||
|
||||
2. **Set Multi-Sig as Admin**
|
||||
```bash
|
||||
cast send $CONTRACT \
|
||||
"changeAdmin(address)" \
|
||||
$MULTISIG_ADDRESS \
|
||||
--rpc-url $RPC_URL \
|
||||
--private-key $DEPLOYER_KEY
|
||||
```
|
||||
|
||||
3. **Use Multi-Sig for Admin Functions**
|
||||
- Submit transactions through multi-sig
|
||||
- Require multiple approvals
|
||||
- Track all proposals
|
||||
|
||||
---
|
||||
|
||||
## Key Rotation Schedule
|
||||
|
||||
### Recommended Schedule
|
||||
|
||||
- **Quarterly**: Review key security
|
||||
- **Annually**: Rotate keys (if no issues)
|
||||
- **Immediately**: Rotate if compromised
|
||||
- **After Incidents**: Rotate as part of recovery
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Monitor Admin Actions
|
||||
|
||||
Track all admin function calls:
|
||||
|
||||
```bash
|
||||
# Monitor admin changes
|
||||
cast logs --address $CONTRACT \
|
||||
"AdminChanged(address,address)" \
|
||||
--rpc-url $RPC_URL
|
||||
|
||||
# Monitor configuration changes
|
||||
cast logs --address $CONTRACT \
|
||||
"DestinationAdded(uint64,address)" \
|
||||
--rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
### Alerts
|
||||
|
||||
Set up alerts for:
|
||||
- Admin changes
|
||||
- Configuration changes
|
||||
- Unusual admin activity
|
||||
- Failed admin operations
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
### Required Documentation
|
||||
|
||||
1. **Key Inventory**: List all admin keys and their purposes
|
||||
2. **Access Logs**: Maintain logs of all admin actions
|
||||
3. **Rotation Records**: Document all key rotations
|
||||
4. **Incident Reports**: Document security incidents
|
||||
|
||||
### Key Information to Document
|
||||
|
||||
- Key purpose (router admin, bridge admin, etc.)
|
||||
- Key location (hardware wallet, HSM, etc.)
|
||||
- Key holders (who has access)
|
||||
- Last rotation date
|
||||
- Next rotation date
|
||||
|
||||
---
|
||||
|
||||
## Compliance
|
||||
|
||||
### Audit Requirements
|
||||
|
||||
- Regular security audits
|
||||
- Key management reviews
|
||||
- Access control audits
|
||||
- Incident response reviews
|
||||
|
||||
### Regulatory Considerations
|
||||
|
||||
- Follow industry best practices
|
||||
- Maintain audit trails
|
||||
- Implement proper controls
|
||||
- Document procedures
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CCIP Security Guide](../security/CCIP_SECURITY.md)
|
||||
- [CCIP Runbooks](CCIP_RUNBOOKS.md)
|
||||
- [CCIP Monitoring](CCIP_MONITORING.md)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-27
|
||||
|
||||
252
docs/ccip-integration/operations/CCIP_MONITORING.md
Normal file
252
docs/ccip-integration/operations/CCIP_MONITORING.md
Normal file
@@ -0,0 +1,252 @@
|
||||
# CCIP Monitoring Guide for ChainID 138
|
||||
|
||||
**Date**: 2025-01-27
|
||||
**Network**: ChainID 138 (DeFi Oracle Meta Mainnet)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This guide provides monitoring setup and best practices for CCIP infrastructure on ChainID 138.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Components
|
||||
|
||||
### 1. CCIP Router Monitoring
|
||||
|
||||
#### Events to Monitor
|
||||
|
||||
- `MessageSent`: Track all outgoing messages
|
||||
- Parameters: `messageId`, `destinationChainSelector`, `sender`, `receiver`, `data`, `tokenAmounts`, `feeToken`, `extraArgs`
|
||||
- `MessageReceived`: Track all incoming messages
|
||||
- Parameters: `messageId`, `sourceChainSelector`, `sender`, `data`, `tokenAmounts`
|
||||
|
||||
#### Metrics to Track
|
||||
|
||||
- Message volume (sent/received per hour/day)
|
||||
- Fee collection amounts
|
||||
- Average message size
|
||||
- Success/failure rates
|
||||
- Destination chain distribution
|
||||
|
||||
#### Alerts
|
||||
|
||||
- High message failure rate (>5%)
|
||||
- Unusual fee collection patterns
|
||||
- Router contract errors
|
||||
- Unsupported chain access attempts
|
||||
|
||||
### 2. Bridge Monitoring
|
||||
|
||||
#### Events to Monitor
|
||||
|
||||
**CCIPWETH9Bridge & CCIPWETH10Bridge**:
|
||||
- `CrossChainTransferInitiated`: Track outgoing transfers
|
||||
- Parameters: `messageId`, `sender`, `destinationChainSelector`, `recipient`, `amount`, `nonce`
|
||||
- `CrossChainTransferCompleted`: Track completed transfers
|
||||
- Parameters: `messageId`, `sourceChainSelector`, `recipient`, `amount`
|
||||
- `DestinationAdded`: Track configuration changes
|
||||
- `DestinationRemoved`: Track configuration changes
|
||||
- `DestinationUpdated`: Track configuration changes
|
||||
|
||||
#### Metrics to Track
|
||||
|
||||
- Transfer volume (amount and count)
|
||||
- Average transfer size
|
||||
- Transfer success rate
|
||||
- Time to completion
|
||||
- Fee costs per transfer
|
||||
- Destination chain usage
|
||||
|
||||
#### Alerts
|
||||
|
||||
- Failed transfers
|
||||
- Stuck transfers (no completion after X hours)
|
||||
- Unusual transfer patterns
|
||||
- Configuration changes
|
||||
- Insufficient fee errors
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Setup
|
||||
|
||||
### Option 1: Event Logging Script
|
||||
|
||||
Create a script to monitor events:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Monitor CCIP events
|
||||
|
||||
RPC_URL="${RPC_URL_138:-http://localhost:8545}"
|
||||
ROUTER="${CCIP_CHAIN138_ROUTER:-}"
|
||||
BRIDGE9="${CCIPWETH9_BRIDGE_CHAIN138:-}"
|
||||
BRIDGE10="${CCIPWETH10_BRIDGE_CHAIN138:-}"
|
||||
|
||||
# Monitor router events
|
||||
cast logs --from-block latest \
|
||||
--address "$ROUTER" \
|
||||
--rpc-url "$RPC_URL" \
|
||||
"MessageSent(bytes32,uint64,address,bytes,tuple[],address,bytes)"
|
||||
|
||||
# Monitor bridge events
|
||||
cast logs --from-block latest \
|
||||
--address "$BRIDGE9" \
|
||||
--rpc-url "$RPC_URL" \
|
||||
"CrossChainTransferInitiated(bytes32,address,uint64,address,uint256,uint256)"
|
||||
```
|
||||
|
||||
### Option 2: Prometheus Integration
|
||||
|
||||
Set up Prometheus to scrape CCIP metrics:
|
||||
|
||||
```yaml
|
||||
# prometheus.yml
|
||||
scrape_configs:
|
||||
- job_name: 'ccip-router'
|
||||
static_configs:
|
||||
- targets: ['localhost:9545']
|
||||
metrics_path: '/metrics'
|
||||
```
|
||||
|
||||
### Option 3: Grafana Dashboards
|
||||
|
||||
Create dashboards for:
|
||||
- Message volume over time
|
||||
- Transfer amounts and counts
|
||||
- Fee collection
|
||||
- Success/failure rates
|
||||
- Destination chain distribution
|
||||
|
||||
---
|
||||
|
||||
## Alerting Rules
|
||||
|
||||
### Critical Alerts
|
||||
|
||||
1. **Router Down**: Router contract becomes unresponsive
|
||||
2. **Bridge Failure**: Bridge fails to process transfers
|
||||
3. **High Failure Rate**: >10% message/transfer failures
|
||||
4. **Configuration Change**: Unauthorized configuration changes
|
||||
|
||||
### Warning Alerts
|
||||
|
||||
1. **High Volume**: Unusual message/transfer volume
|
||||
2. **Fee Anomaly**: Unusual fee collection patterns
|
||||
3. **Slow Processing**: Messages taking longer than expected
|
||||
|
||||
---
|
||||
|
||||
## Logging
|
||||
|
||||
### Recommended Log Levels
|
||||
|
||||
- **INFO**: Normal operations (messages sent/received, transfers)
|
||||
- **WARN**: Recoverable errors, configuration changes
|
||||
- **ERROR**: Failed operations, contract errors
|
||||
- **DEBUG**: Detailed operation logs (for troubleshooting)
|
||||
|
||||
### Log Retention
|
||||
|
||||
- **Event Logs**: Retain for 90 days
|
||||
- **Error Logs**: Retain for 180 days
|
||||
- **Audit Logs**: Retain for 1 year
|
||||
|
||||
---
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Router Health Check
|
||||
|
||||
```bash
|
||||
# Check router is responsive
|
||||
cast call "$ROUTER" "feeToken()" --rpc-url "$RPC_URL"
|
||||
|
||||
# Check supported chains
|
||||
cast call "$ROUTER" "supportedChains(uint64)" "5009297550715157269" --rpc-url "$RPC_URL"
|
||||
```
|
||||
|
||||
### Bridge Health Check
|
||||
|
||||
```bash
|
||||
# Check bridge router connection
|
||||
cast call "$BRIDGE9" "ccipRouter()" --rpc-url "$RPC_URL"
|
||||
|
||||
# Check destinations
|
||||
cast call "$BRIDGE9" "destinations(uint64)" "5009297550715157269" --rpc-url "$RPC_URL"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
### Key Performance Indicators (KPIs)
|
||||
|
||||
1. **Message Throughput**: Messages per second
|
||||
2. **Transfer Throughput**: Transfers per hour
|
||||
3. **Average Latency**: Time from send to receive
|
||||
4. **Success Rate**: Percentage of successful operations
|
||||
5. **Fee Efficiency**: Average fee per operation
|
||||
|
||||
### Target Metrics
|
||||
|
||||
- Message success rate: >99%
|
||||
- Average latency: <5 minutes
|
||||
- Transfer success rate: >99.5%
|
||||
- System uptime: >99.9%
|
||||
|
||||
---
|
||||
|
||||
## Incident Response
|
||||
|
||||
### Escalation Procedures
|
||||
|
||||
1. **Level 1**: Automated alerts → On-call engineer
|
||||
2. **Level 2**: Critical failures → Team lead
|
||||
3. **Level 3**: System-wide issues → CTO/Management
|
||||
|
||||
### Response Playbook
|
||||
|
||||
1. **Router Failure**:
|
||||
- Check contract status
|
||||
- Verify RPC connectivity
|
||||
- Review recent transactions
|
||||
- Check for configuration changes
|
||||
|
||||
2. **Bridge Failure**:
|
||||
- Verify router connectivity
|
||||
- Check destination configuration
|
||||
- Review transfer logs
|
||||
- Verify fee payment
|
||||
|
||||
3. **High Failure Rate**:
|
||||
- Analyze failure patterns
|
||||
- Check network conditions
|
||||
- Review recent changes
|
||||
- Escalate if needed
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Tools
|
||||
|
||||
### Recommended Tools
|
||||
|
||||
- **Prometheus**: Metrics collection
|
||||
- **Grafana**: Visualization and dashboards
|
||||
- **ELK Stack**: Log aggregation
|
||||
- **PagerDuty**: Alerting and on-call
|
||||
- **Custom Scripts**: Event monitoring
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CCIP Deployment Guide](../ccip/DEPLOYMENT_GUIDE_CHAIN138.md)
|
||||
- [CCIP Review](../CCIP_CHAIN138_REVIEW.md)
|
||||
- [Operations Runbooks](CCIP_RUNBOOKS.md)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-27
|
||||
|
||||
367
docs/ccip-integration/operations/CCIP_RUNBOOKS.md
Normal file
367
docs/ccip-integration/operations/CCIP_RUNBOOKS.md
Normal file
@@ -0,0 +1,367 @@
|
||||
# CCIP Operational Runbooks for ChainID 138
|
||||
|
||||
**Date**: 2025-01-27
|
||||
**Network**: ChainID 138 (DeFi Oracle Meta Mainnet)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides operational runbooks for common CCIP operations, troubleshooting, and emergency procedures.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Deployment Runbook](#deployment-runbook)
|
||||
2. [Configuration Runbook](#configuration-runbook)
|
||||
3. [Troubleshooting Guide](#troubleshooting-guide)
|
||||
4. [Emergency Response](#emergency-response)
|
||||
|
||||
---
|
||||
|
||||
## Deployment Runbook
|
||||
|
||||
### Prerequisites Check
|
||||
|
||||
```bash
|
||||
# 1. Verify network connectivity
|
||||
cast block-number --rpc-url $RPC_URL_138
|
||||
|
||||
# 2. Verify account balance
|
||||
cast balance $DEPLOYER_ADDRESS --rpc-url $RPC_URL_138
|
||||
|
||||
# 3. Verify environment variables
|
||||
grep -E "CCIP_CHAIN138|CHAIN138_SELECTOR" .env
|
||||
```
|
||||
|
||||
### Deployment Steps
|
||||
|
||||
1. **Calculate Chain Selector**
|
||||
```bash
|
||||
./scripts/ccip/calculate-chain-selector.sh
|
||||
```
|
||||
|
||||
2. **Deploy CCIP Router**
|
||||
```bash
|
||||
forge script script/DeployCCIPRouter.s.sol \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--broadcast \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
3. **Configure Router**
|
||||
```bash
|
||||
./scripts/deployment/configure-ccip-router.sh
|
||||
```
|
||||
|
||||
4. **Deploy Bridges**
|
||||
```bash
|
||||
forge script script/DeployCCIPWETH9Bridge.s.sol \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--broadcast \
|
||||
--private-key $PRIVATE_KEY
|
||||
|
||||
forge script script/DeployCCIPWETH10Bridge.s.sol \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--broadcast \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
5. **Configure Bridges**
|
||||
```bash
|
||||
./scripts/deployment/configure-weth9-bridge.sh
|
||||
./scripts/deployment/configure-weth10-bridge.sh
|
||||
```
|
||||
|
||||
6. **Verify Deployment**
|
||||
```bash
|
||||
./scripts/deployment/verify-ccip-deployment.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Runbook
|
||||
|
||||
### Adding a New Destination Chain
|
||||
|
||||
1. **Get Chain Selector**
|
||||
- Look up selector from Chainlink docs or calculate
|
||||
|
||||
2. **Add to Router**
|
||||
```bash
|
||||
cast send $CCIP_CHAIN138_ROUTER \
|
||||
"addSupportedChain(uint64)" \
|
||||
$DESTINATION_SELECTOR \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
3. **Add Supported Tokens**
|
||||
```bash
|
||||
cast send $CCIP_CHAIN138_ROUTER \
|
||||
"addSupportedToken(uint64,address)" \
|
||||
$DESTINATION_SELECTOR \
|
||||
$TOKEN_ADDRESS \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
4. **Configure Bridge**
|
||||
```bash
|
||||
cast send $BRIDGE_ADDRESS \
|
||||
"addDestination(uint64,address)" \
|
||||
$DESTINATION_SELECTOR \
|
||||
$DESTINATION_BRIDGE_ADDRESS \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
### Updating Fee Configuration
|
||||
|
||||
```bash
|
||||
cast send $CCIP_CHAIN138_ROUTER \
|
||||
"updateFees(uint256,uint256)" \
|
||||
$NEW_BASE_FEE \
|
||||
$NEW_DATA_FEE_PER_BYTE \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
### Changing Admin
|
||||
|
||||
```bash
|
||||
# Router
|
||||
cast send $CCIP_CHAIN138_ROUTER \
|
||||
"changeAdmin(address)" \
|
||||
$NEW_ADMIN \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
|
||||
# Bridge
|
||||
cast send $BRIDGE_ADDRESS \
|
||||
"changeAdmin(address)" \
|
||||
$NEW_ADMIN \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Guide
|
||||
|
||||
### Issue: Router Not Responding
|
||||
|
||||
**Symptoms**:
|
||||
- Cannot send messages
|
||||
- Contract calls fail
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check contract code
|
||||
cast code $CCIP_CHAIN138_ROUTER --rpc-url $RPC_URL_138
|
||||
|
||||
# Check recent transactions
|
||||
cast tx $LAST_TX_HASH --rpc-url $RPC_URL_138
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Verify RPC connectivity
|
||||
2. Check contract is deployed
|
||||
3. Verify network is synced
|
||||
4. Check for contract upgrades
|
||||
|
||||
### Issue: Bridge Transfer Fails
|
||||
|
||||
**Symptoms**:
|
||||
- Transfer initiated but not completed
|
||||
- Error messages in logs
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check bridge configuration
|
||||
cast call $BRIDGE "destinations(uint64)" $DEST_SELECTOR --rpc-url $RPC_URL_138
|
||||
|
||||
# Check router connection
|
||||
cast call $BRIDGE "ccipRouter()" --rpc-url $RPC_URL_138
|
||||
|
||||
# Check message status
|
||||
cast logs --address $BRIDGE "CrossChainTransferInitiated" --rpc-url $RPC_URL_138
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Verify destination is configured
|
||||
2. Check router is accessible
|
||||
3. Verify sufficient fees
|
||||
4. Check destination chain status
|
||||
|
||||
### Issue: High Failure Rate
|
||||
|
||||
**Symptoms**:
|
||||
- Many failed messages/transfers
|
||||
- Error events in logs
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Analyze failure patterns
|
||||
cast logs --address $ROUTER "MessageSent" --rpc-url $RPC_URL_138 | grep -i error
|
||||
|
||||
# Check fee configuration
|
||||
cast call $ROUTER "baseFee()" --rpc-url $RPC_URL_138
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Review error logs
|
||||
2. Check fee configuration
|
||||
3. Verify destination chains
|
||||
4. Check network conditions
|
||||
|
||||
### Issue: Configuration Changes Not Applied
|
||||
|
||||
**Symptoms**:
|
||||
- Configuration calls succeed but changes not visible
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Verify current configuration
|
||||
cast call $CONTRACT "destinations(uint64)" $SELECTOR --rpc-url $RPC_URL_138
|
||||
|
||||
# Check transaction status
|
||||
cast tx $TX_HASH --rpc-url $RPC_URL_138
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Verify transaction was mined
|
||||
2. Check correct contract address
|
||||
3. Verify admin permissions
|
||||
4. Check for event emissions
|
||||
|
||||
---
|
||||
|
||||
## Emergency Response
|
||||
|
||||
### Emergency Pause (if implemented)
|
||||
|
||||
If contracts have pause functionality:
|
||||
|
||||
```bash
|
||||
# Pause router (if function exists)
|
||||
cast send $ROUTER "pause()" --rpc-url $RPC_URL_138 --private-key $PRIVATE_KEY
|
||||
|
||||
# Pause bridge
|
||||
cast send $BRIDGE "pause()" --rpc-url $RPC_URL_138 --private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
### Disable Destination Chain
|
||||
|
||||
```bash
|
||||
# Remove destination from bridge
|
||||
cast send $BRIDGE \
|
||||
"removeDestination(uint64)" \
|
||||
$PROBLEMATIC_SELECTOR \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
|
||||
# Remove from router
|
||||
cast send $ROUTER \
|
||||
"removeSupportedChain(uint64)" \
|
||||
$PROBLEMATIC_SELECTOR \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
### Withdraw Fees (Emergency)
|
||||
|
||||
```bash
|
||||
# Withdraw native ETH fees
|
||||
cast send $ROUTER \
|
||||
"withdrawNativeFees()" \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
|
||||
# Withdraw ERC20 fees
|
||||
cast send $ROUTER \
|
||||
"withdrawFees(uint256)" \
|
||||
$AMOUNT \
|
||||
--rpc-url $RPC_URL_138 \
|
||||
--private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
### Incident Response Checklist
|
||||
|
||||
1. **Identify Issue**
|
||||
- Check monitoring alerts
|
||||
- Review recent logs
|
||||
- Verify contract status
|
||||
|
||||
2. **Assess Impact**
|
||||
- How many users affected?
|
||||
- What operations are failing?
|
||||
- Is data at risk?
|
||||
|
||||
3. **Contain Issue**
|
||||
- Disable affected destinations
|
||||
- Pause operations if needed
|
||||
- Notify stakeholders
|
||||
|
||||
4. **Resolve Issue**
|
||||
- Apply fixes
|
||||
- Verify resolution
|
||||
- Monitor for recurrence
|
||||
|
||||
5. **Post-Incident**
|
||||
- Document incident
|
||||
- Review root cause
|
||||
- Update procedures
|
||||
|
||||
---
|
||||
|
||||
## Common Operations
|
||||
|
||||
### Check Contract Status
|
||||
|
||||
```bash
|
||||
# Router status
|
||||
cast call $ROUTER "feeToken()" --rpc-url $RPC_URL_138
|
||||
cast call $ROUTER "baseFee()" --rpc-url $RPC_URL_138
|
||||
cast call $ROUTER "supportedChains(uint64)" $SELECTOR --rpc-url $RPC_URL_138
|
||||
|
||||
# Bridge status
|
||||
cast call $BRIDGE "ccipRouter()" --rpc-url $RPC_URL_138
|
||||
cast call $BRIDGE "destinations(uint64)" $SELECTOR --rpc-url $RPC_URL_138
|
||||
cast call $BRIDGE "getDestinationChains()" --rpc-url $RPC_URL_138
|
||||
```
|
||||
|
||||
### View Recent Events
|
||||
|
||||
```bash
|
||||
# Router events
|
||||
cast logs --address $ROUTER --from-block latest --rpc-url $RPC_URL_138
|
||||
|
||||
# Bridge events
|
||||
cast logs --address $BRIDGE --from-block latest --rpc-url $RPC_URL_138
|
||||
```
|
||||
|
||||
### Calculate Fees
|
||||
|
||||
```bash
|
||||
# Estimate fee for message
|
||||
cast call $ROUTER \
|
||||
"getFee(uint64,tuple)" \
|
||||
$DEST_SELECTOR \
|
||||
$MESSAGE \
|
||||
--rpc-url $RPC_URL_138
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CCIP Deployment Guide](../ccip/DEPLOYMENT_GUIDE_CHAIN138.md)
|
||||
- [CCIP Monitoring](CCIP_MONITORING.md)
|
||||
- [CCIP Review](../CCIP_CHAIN138_REVIEW.md)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-27
|
||||
|
||||
Reference in New Issue
Block a user