smom-dbis-138/docs/ccip-integration/operations/CCIP_RUNBOOKS.md

CCIP Operational Runbooks for ChainID 138

Date: 2025-01-27
Network: ChainID 138 (DeFi Oracle Meta Mainnet)

---

Overview

This document provides operational runbooks for common CCIP operations, troubleshooting, and emergency procedures.

---

Table of Contents

1. Deployment Runbook
2. Configuration Runbook
3. Troubleshooting Guide
4. Emergency Response

---

Deployment Runbook

Prerequisites Check

# 1. Verify network connectivity
cast block-number --rpc-url $RPC_URL_138

# 2. Verify account balance
cast balance $DEPLOYER_ADDRESS --rpc-url $RPC_URL_138

# 3. Verify environment variables
grep -E "CCIP_CHAIN138|CHAIN138_SELECTOR" .env

Deployment Steps

1. Calculate Chain Selector

   ./scripts/ccip/calculate-chain-selector.sh
   

2. Deploy CCIP Router

   forge script script/DeployCCIPRouter.s.sol \
     --rpc-url $RPC_URL_138 \
     --broadcast \
     --private-key $PRIVATE_KEY
   

3. Configure Router

   ./scripts/deployment/configure-ccip-router.sh
   

4. Deploy Bridges

   forge script script/DeployCCIPWETH9Bridge.s.sol \
     --rpc-url $RPC_URL_138 \
     --broadcast \
     --private-key $PRIVATE_KEY
   
   forge script script/DeployCCIPWETH10Bridge.s.sol \
     --rpc-url $RPC_URL_138 \
     --broadcast \
     --private-key $PRIVATE_KEY
   

5. Configure Bridges

   ./scripts/deployment/configure-weth9-bridge.sh
   ./scripts/deployment/configure-weth10-bridge.sh
   

6. Verify Deployment

   ./scripts/deployment/verify-ccip-deployment.sh
   

---

Configuration Runbook

Adding a New Destination Chain

1. Get Chain Selector

   • Look up selector from Chainlink docs or calculate

2. Add to Router

   cast send $CCIP_CHAIN138_ROUTER \
     "addSupportedChain(uint64)" \
     $DESTINATION_SELECTOR \
     --rpc-url $RPC_URL_138 \
     --private-key $PRIVATE_KEY
   

3. Add Supported Tokens

   cast send $CCIP_CHAIN138_ROUTER \
     "addSupportedToken(uint64,address)" \
     $DESTINATION_SELECTOR \
     $TOKEN_ADDRESS \
     --rpc-url $RPC_URL_138 \
     --private-key $PRIVATE_KEY
   

4. Configure Bridge

   cast send $BRIDGE_ADDRESS \
     "addDestination(uint64,address)" \
     $DESTINATION_SELECTOR \
     $DESTINATION_BRIDGE_ADDRESS \
     --rpc-url $RPC_URL_138 \
     --private-key $PRIVATE_KEY
   

Updating Fee Configuration

cast send $CCIP_CHAIN138_ROUTER \
  "updateFees(uint256,uint256)" \
  $NEW_BASE_FEE \
  $NEW_DATA_FEE_PER_BYTE \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

Changing Admin

# Router
cast send $CCIP_CHAIN138_ROUTER \
  "changeAdmin(address)" \
  $NEW_ADMIN \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

# Bridge
cast send $BRIDGE_ADDRESS \
  "changeAdmin(address)" \
  $NEW_ADMIN \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

---

Troubleshooting Guide

Issue: Router Not Responding

Symptoms:

• Cannot send messages
• Contract calls fail

Diagnosis:

# Check contract code
cast code $CCIP_CHAIN138_ROUTER --rpc-url $RPC_URL_138

# Check recent transactions
cast tx $LAST_TX_HASH --rpc-url $RPC_URL_138

Solutions:

1. Verify RPC connectivity
2. Check contract is deployed
3. Verify network is synced
4. Check for contract upgrades

Issue: Bridge Transfer Fails

Symptoms:

• Transfer initiated but not completed
• Error messages in logs

Diagnosis:

# Check bridge configuration
cast call $BRIDGE "destinations(uint64)" $DEST_SELECTOR --rpc-url $RPC_URL_138

# Check router connection
cast call $BRIDGE "ccipRouter()" --rpc-url $RPC_URL_138

# Check message status
cast logs --address $BRIDGE "CrossChainTransferInitiated" --rpc-url $RPC_URL_138

Solutions:

1. Verify destination is configured
2. Check router is accessible
3. Verify sufficient fees
4. Check destination chain status

Issue: High Failure Rate

Symptoms:

• Many failed messages/transfers
• Error events in logs

Diagnosis:

# Analyze failure patterns
cast logs --address $ROUTER "MessageSent" --rpc-url $RPC_URL_138 | grep -i error

# Check fee configuration
cast call $ROUTER "baseFee()" --rpc-url $RPC_URL_138

Solutions:

1. Review error logs
2. Check fee configuration
3. Verify destination chains
4. Check network conditions

Issue: Configuration Changes Not Applied

Symptoms:

• Configuration calls succeed but changes not visible

Diagnosis:

# Verify current configuration
cast call $CONTRACT "destinations(uint64)" $SELECTOR --rpc-url $RPC_URL_138

# Check transaction status
cast tx $TX_HASH --rpc-url $RPC_URL_138

Solutions:

1. Verify transaction was mined
2. Check correct contract address
3. Verify admin permissions
4. Check for event emissions

---

Emergency Response

Emergency Pause (if implemented)

If contracts have pause functionality:

# Pause router (if function exists)
cast send $ROUTER "pause()" --rpc-url $RPC_URL_138 --private-key $PRIVATE_KEY

# Pause bridge
cast send $BRIDGE "pause()" --rpc-url $RPC_URL_138 --private-key $PRIVATE_KEY

Disable Destination Chain

# Remove destination from bridge
cast send $BRIDGE \
  "removeDestination(uint64)" \
  $PROBLEMATIC_SELECTOR \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

# Remove from router
cast send $ROUTER \
  "removeSupportedChain(uint64)" \
  $PROBLEMATIC_SELECTOR \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

Withdraw Fees (Emergency)

# Withdraw native ETH fees
cast send $ROUTER \
  "withdrawNativeFees()" \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

# Withdraw ERC20 fees
cast send $ROUTER \
  "withdrawFees(uint256)" \
  $AMOUNT \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

Incident Response Checklist

1. Identify Issue

   • Check monitoring alerts
   • Review recent logs
   • Verify contract status

2. Assess Impact

   • How many users affected?
   • What operations are failing?
   • Is data at risk?

3. Contain Issue

   • Disable affected destinations
   • Pause operations if needed
   • Notify stakeholders

4. Resolve Issue

   • Apply fixes
   • Verify resolution
   • Monitor for recurrence

5. Post-Incident

   • Document incident
   • Review root cause
   • Update procedures

---

Common Operations

Check Contract Status

# Router status
cast call $ROUTER "feeToken()" --rpc-url $RPC_URL_138
cast call $ROUTER "baseFee()" --rpc-url $RPC_URL_138
cast call $ROUTER "supportedChains(uint64)" $SELECTOR --rpc-url $RPC_URL_138

# Bridge status
cast call $BRIDGE "ccipRouter()" --rpc-url $RPC_URL_138
cast call $BRIDGE "destinations(uint64)" $SELECTOR --rpc-url $RPC_URL_138
cast call $BRIDGE "getDestinationChains()" --rpc-url $RPC_URL_138

View Recent Events

# Router events
cast logs --address $ROUTER --from-block latest --rpc-url $RPC_URL_138

# Bridge events
cast logs --address $BRIDGE --from-block latest --rpc-url $RPC_URL_138

Calculate Fees

# Estimate fee for message
cast call $ROUTER \
  "getFee(uint64,tuple)" \
  $DEST_SELECTOR \
  $MESSAGE \
  --rpc-url $RPC_URL_138

---

Related Documentation

• CCIP Deployment Guide
• CCIP Monitoring
• CCIP Review

---

Last Updated: 2025-01-27