smom-dbis-138/runbooks/ccip-recovery.md

CCIP Recovery Procedures

Overview

This document provides recovery procedures for CCIP failures and outages.

Recovery Scenarios

Scenario 1: Router Failure

Symptoms: Router service is down or unresponsive

Recovery Steps:

1. Check Router Status

   kubectl get pods -n besu-network -l app=ccip-router
   kubectl describe pod <router-pod> -n besu-network
   

2. Restart Router

   kubectl delete pod <router-pod> -n besu-network
   # Wait for new pod to start
   kubectl get pods -n besu-network -l app=ccip-router
   

3. Verify Recovery

   # Test router connectivity
   cast call $CCIP_ROUTER "getSupportedTokens(uint64)" $CHAIN_SELECTOR --rpc-url $RPC_URL
   

4. Resume Operations

   • Monitor message sending
   • Verify message delivery
   • Check for backlog

Scenario 2: Contract Failure

Symptoms: Sender or receiver contract is not functioning

Recovery Steps:

1. Identify Issue

   # Check contract state
   cast call $SENDER "paused()" --rpc-url $RPC_URL
   cast call $RECEIVER "processedMessages(bytes32)" $MESSAGE_ID --rpc-url $RPC_URL
   

2. Pause Operations (if needed)

   // Call pause function if available
   sender.pause();
   

3. Fix Contract

   • Update configuration
   • Fix bugs if any
   • Deploy new version if needed

4. Resume Operations

   // Unpause if paused
   sender.unpause();
   

5. Verify Recovery

   • Test message sending
   • Verify message receiving
   • Monitor for issues

Scenario 3: Message Backlog

Symptoms: Messages queued but not being processed

Recovery Steps:

1. Assess Backlog

   # Check pending messages
   cast logs --from-block latest-1000 --address $SENDER --rpc-url $RPC_URL | grep MessageSent
   

2. Identify Cause

   • Check router status
   • Verify LINK balance
   • Check target chain status

3. Clear Backlog

   • Fix underlying issue
   • Process messages in order
   • Monitor processing

4. Prevent Future Backlog

   • Increase processing capacity
   • Improve error handling
   • Add monitoring

Scenario 4: Data Corruption

Symptoms: Invalid messages or corrupted data

Recovery Steps:

1. Identify Corrupted Messages

   • Review message logs
   • Check for invalid formats
   • Identify affected messages

2. Isolate Issue

   • Pause message processing
   • Prevent further corruption
   • Assess impact

3. Recover Data

   • Resend valid messages
   • Skip corrupted messages
   • Update data if possible

4. Prevent Recurrence

   • Fix encoding/decoding
   • Add validation
   • Improve error handling

Scenario 5: Network Partition

Symptoms: Cannot communicate with target chain

Recovery Steps:

1. Verify Connectivity

   # Test target chain connectivity
   curl -X POST $TARGET_CHAIN_RPC_URL -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
   

2. Wait for Recovery

   • Monitor network status
   • Check chain status pages
   • Wait for partition to resolve

3. Resume Operations

   • Verify connectivity restored
   • Process queued messages
   • Resume normal operations

Recovery Verification

Post-Recovery Checks

1. Functionality

   • Test message sending
   • Verify message receiving
   • Check fee calculation

2. Performance

   • Check message latency
   • Verify throughput
   • Monitor error rates

3. Data Integrity

   • Verify message order
   • Check for duplicates
   • Validate message content

Prevention

Best Practices

1. Monitoring

   • Set up comprehensive monitoring
   • Configure alerts
   • Regular health checks

2. Redundancy

   • Deploy multiple router instances
   • Use backup contracts
   • Maintain backup configurations

3. Testing

   • Regular disaster recovery drills
   • Test recovery procedures
   • Update procedures based on tests

4. Documentation

   • Keep runbooks updated
   • Document lessons learned
   • Share knowledge

References

• CCIP Operations Runbook
• CCIP Incident Response
• CCIP Troubleshooting