Add full monorepo: virtual-banker, backend, frontend, docs, scripts, deployment
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
defiQUG
370
docs/CCIP_OPERATIONS_RUNBOOK.md
Normal file
370
docs/CCIP_OPERATIONS_RUNBOOK.md
Normal file
@@ -0,0 +1,370 @@
|
||||
# CCIP Operations Runbook
|
||||
|
||||
**Date**: 2025-01-12
|
||||
**Network**: ChainID 138
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This runbook provides step-by-step procedures for common CCIP operations and troubleshooting.
|
||||
|
||||
---
|
||||
|
||||
## Common Operations
|
||||
|
||||
### Starting CCIP Monitor Service
|
||||
|
||||
**Task**: Task 83
|
||||
|
||||
**Procedure**:
|
||||
```bash
|
||||
# Start container
|
||||
pct start 3501
|
||||
|
||||
# Start service
|
||||
pct exec 3501 -- systemctl start ccip-monitor
|
||||
|
||||
# Verify status
|
||||
pct exec 3501 -- systemctl status ccip-monitor
|
||||
|
||||
# Check health
|
||||
./scripts/check-ccip-monitor-health.sh
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
- Container running
|
||||
- Service active
|
||||
- Metrics endpoint accessible
|
||||
|
||||
---
|
||||
|
||||
### Checking Bridge Configuration
|
||||
|
||||
**Task**: Task 55
|
||||
|
||||
**Procedure**:
|
||||
```bash
|
||||
# Check all bridge destinations
|
||||
./scripts/check-bridge-config.sh
|
||||
|
||||
# Check specific chain
|
||||
./scripts/verify-destination-chain-config.sh Ethereum
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
- List of configured destinations
|
||||
- Status for each destination
|
||||
- Missing destinations highlighted
|
||||
|
||||
---
|
||||
|
||||
### Verifying Router and Sender
|
||||
|
||||
**Tasks**: Tasks 1, 2
|
||||
|
||||
**Procedure**:
|
||||
```bash
|
||||
# Verify Router
|
||||
./scripts/verify-ccip-router.sh
|
||||
|
||||
# Verify Sender
|
||||
./scripts/verify-ccip-sender.sh
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
- Contract deployment confirmed
|
||||
- Functions accessible
|
||||
- Configuration verified
|
||||
|
||||
---
|
||||
|
||||
### Configuring Bridge Destination
|
||||
|
||||
**Tasks**: Tasks 3, 50, 51
|
||||
|
||||
**Procedure**:
|
||||
```bash
|
||||
# Configure all destinations
|
||||
./scripts/configure-all-bridge-destinations.sh
|
||||
|
||||
# Configure specific destination
|
||||
./scripts/fix-bridge-errors.sh <private_key> <bridge_address>
|
||||
```
|
||||
|
||||
**Requirements**:
|
||||
- Private key for bridge owner
|
||||
- Destination bridge address
|
||||
- Sufficient gas
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
./scripts/check-bridge-config.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Verifying Token Mechanisms
|
||||
|
||||
**Tasks**: Tasks 36, 37
|
||||
|
||||
**Procedure**:
|
||||
```bash
|
||||
# Verify WETH9
|
||||
./scripts/inspect-weth9-contract.sh
|
||||
|
||||
# Verify WETH10
|
||||
./scripts/inspect-weth10-contract.sh
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
- 1:1 ratio verified
|
||||
- Contract balance = total supply
|
||||
- Functions accessible
|
||||
|
||||
---
|
||||
|
||||
### Comprehensive System Verification
|
||||
|
||||
**Task**: Task 139
|
||||
|
||||
**Procedure**:
|
||||
```bash
|
||||
# Run comprehensive verification
|
||||
./scripts/verify-complete-ccip-setup.sh
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
- All components checked
|
||||
- Status for each component
|
||||
- Summary report
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Bridge Configuration Issues
|
||||
|
||||
**Symptoms**:
|
||||
- "destination not enabled" error
|
||||
- Destinations not configured
|
||||
|
||||
**Solutions**:
|
||||
1. Check current configuration:
|
||||
```bash
|
||||
./scripts/check-bridge-config.sh
|
||||
```
|
||||
|
||||
2. Configure missing destinations:
|
||||
```bash
|
||||
./scripts/configure-all-bridge-destinations.sh
|
||||
```
|
||||
|
||||
3. Verify configuration:
|
||||
```bash
|
||||
./scripts/verify-destination-chain-config.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Fee Calculation Issues
|
||||
|
||||
**Symptoms**:
|
||||
- Fee calculation returns 0
|
||||
- Fee calculation fails
|
||||
|
||||
**Solutions**:
|
||||
1. Verify Router configuration:
|
||||
```bash
|
||||
./scripts/verify-ccip-router.sh
|
||||
```
|
||||
|
||||
2. Check fee calculation:
|
||||
```bash
|
||||
./scripts/verify-fee-calculation.sh 0.001 5009297550715157269
|
||||
```
|
||||
|
||||
3. Verify LINK token:
|
||||
```bash
|
||||
cast call 0x514910771AF9Ca656af840dff83E8264EcF986CA \
|
||||
"balanceOf(address)" \
|
||||
<bridge_address> \
|
||||
--rpc-url <rpc_url>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stuck Transactions
|
||||
|
||||
**Symptoms**:
|
||||
- Transaction pending in mempool
|
||||
- Transaction not confirming
|
||||
|
||||
**Solutions**:
|
||||
1. Check transaction status:
|
||||
```bash
|
||||
cast tx <tx_hash> --rpc-url <rpc_url>
|
||||
```
|
||||
|
||||
2. Check nonce:
|
||||
```bash
|
||||
cast nonce <address> --rpc-url <rpc_url>
|
||||
```
|
||||
|
||||
3. Replace transaction (if needed):
|
||||
```bash
|
||||
cast send <contract> <function> \
|
||||
--rpc-url <rpc_url> \
|
||||
--private-key <key> \
|
||||
--gas-price <higher_gas_price> \
|
||||
--nonce <next_nonce>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### CCIP Monitor Issues
|
||||
|
||||
**Symptoms**:
|
||||
- Monitor service not running
|
||||
- Metrics not accessible
|
||||
- Errors in logs
|
||||
|
||||
**Solutions**:
|
||||
1. Check service status:
|
||||
```bash
|
||||
./scripts/check-ccip-monitor-health.sh
|
||||
```
|
||||
|
||||
2. Check logs:
|
||||
```bash
|
||||
pct exec 3501 -- journalctl -u ccip-monitor -n 50
|
||||
```
|
||||
|
||||
3. Restart service:
|
||||
```bash
|
||||
pct exec 3501 -- systemctl restart ccip-monitor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Token Pool Issues
|
||||
|
||||
**Symptoms**:
|
||||
- Pool addresses unknown
|
||||
- Pool configuration unclear
|
||||
|
||||
**Solutions**:
|
||||
1. Find TokenAdminRegistry:
|
||||
```bash
|
||||
./scripts/verify-token-admin-registry.sh
|
||||
```
|
||||
|
||||
2. Verify pool configuration:
|
||||
```bash
|
||||
./scripts/verify-token-pool-config.sh <pool_address>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Emergency Procedures
|
||||
|
||||
### Pause Bridge Operations
|
||||
|
||||
**Procedure**:
|
||||
1. Identify bridge contracts
|
||||
2. Check for pause function
|
||||
3. Call pause function (if available)
|
||||
4. Verify pause status
|
||||
|
||||
**Note**: Not all bridge contracts may have pause functionality.
|
||||
|
||||
### Emergency Configuration Changes
|
||||
|
||||
**Procedure**:
|
||||
1. Assess situation
|
||||
2. Determine required changes
|
||||
3. Verify changes are safe
|
||||
4. Execute changes
|
||||
5. Verify changes
|
||||
6. Monitor system
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Regular Maintenance Tasks
|
||||
|
||||
1. **Daily**:
|
||||
- Check system health
|
||||
- Review monitoring alerts
|
||||
- Verify service status
|
||||
|
||||
2. **Weekly**:
|
||||
- Review configuration
|
||||
- Check rate limit usage
|
||||
- Review logs
|
||||
|
||||
3. **Monthly**:
|
||||
- Comprehensive verification
|
||||
- Security review
|
||||
- Documentation updates
|
||||
|
||||
### Backup Procedures
|
||||
|
||||
1. **Configuration Backup**:
|
||||
```bash
|
||||
# Backup bridge configuration
|
||||
./scripts/backup-ccip-config.sh
|
||||
```
|
||||
|
||||
2. **Restore Configuration**:
|
||||
```bash
|
||||
# Restore bridge configuration
|
||||
./scripts/restore-ccip-config.sh <backup_file>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Key Metrics
|
||||
|
||||
1. **System Health**:
|
||||
- Service status
|
||||
- Component health
|
||||
- Error rates
|
||||
|
||||
2. **Bridge Operations**:
|
||||
- Transaction success rate
|
||||
- Message delivery rate
|
||||
- Fee usage
|
||||
|
||||
3. **Token Operations**:
|
||||
- Token balances
|
||||
- Pool balances
|
||||
- Transfer volumes
|
||||
|
||||
### Alerting
|
||||
|
||||
1. **Critical Alerts**:
|
||||
- Service failures
|
||||
- Configuration errors
|
||||
- Security incidents
|
||||
|
||||
2. **Warning Alerts**:
|
||||
- High error rates
|
||||
- Approaching limits
|
||||
- Performance degradation
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CCIP Verification Checklist](./CCIP_VERIFICATION_CHECKLIST.md) (Task 120)
|
||||
- [CCIP Configuration Status](./CCIP_CONFIGURATION_STATUS.md)
|
||||
- [CCIP Best Practices](./CCIP_BEST_PRACTICES.md) (Task 136)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-12
|
||||
|
||||
Reference in New Issue
Block a user