smom-dbis-138/docs/ccip-integration/operations/CCIP_RUNBOOKS.md

# CCIP Operational Runbooks for ChainID 138

**Date**: 2025-01-27  
**Network**: ChainID 138 (DeFi Oracle Meta Mainnet)

---

## Overview

This document provides operational runbooks for common CCIP operations, troubleshooting, and emergency procedures.

---

## Table of Contents

1. [Deployment Runbook](#deployment-runbook)
2. [Configuration Runbook](#configuration-runbook)
3. [Troubleshooting Guide](#troubleshooting-guide)
4. [Emergency Response](#emergency-response)

---

## Deployment Runbook

### Prerequisites Check

```bash
# 1. Verify network connectivity
cast block-number --rpc-url $RPC_URL_138

# 2. Verify account balance
cast balance $DEPLOYER_ADDRESS --rpc-url $RPC_URL_138

# 3. Verify environment variables
grep -E "CCIP_CHAIN138|CHAIN138_SELECTOR" .env
```

### Deployment Steps

1. **Calculate Chain Selector**
   ```bash
   ./scripts/ccip/calculate-chain-selector.sh
   ```

2. **Deploy CCIP Router**
   ```bash
   forge script script/DeployCCIPRouter.s.sol \
     --rpc-url $RPC_URL_138 \
     --broadcast \
     --private-key $PRIVATE_KEY
   ```

3. **Configure Router**
   ```bash
   ./scripts/deployment/configure-ccip-router.sh
   ```

4. **Deploy Bridges**
   ```bash
   forge script script/DeployCCIPWETH9Bridge.s.sol \
     --rpc-url $RPC_URL_138 \
     --broadcast \
     --private-key $PRIVATE_KEY
   
   forge script script/DeployCCIPWETH10Bridge.s.sol \
     --rpc-url $RPC_URL_138 \
     --broadcast \
     --private-key $PRIVATE_KEY
   ```

5. **Configure Bridges**
   ```bash
   ./scripts/deployment/configure-weth9-bridge.sh
   ./scripts/deployment/configure-weth10-bridge.sh
   ```

6. **Verify Deployment**
   ```bash
   ./scripts/deployment/verify-ccip-deployment.sh
   ```

---

## Configuration Runbook

### Adding a New Destination Chain

1. **Get Chain Selector**
   - Look up selector from Chainlink docs or calculate

2. **Add to Router**
   ```bash
   cast send $CCIP_CHAIN138_ROUTER \
     "addSupportedChain(uint64)" \
     $DESTINATION_SELECTOR \
     --rpc-url $RPC_URL_138 \
     --private-key $PRIVATE_KEY
   ```

3. **Add Supported Tokens**
   ```bash
   cast send $CCIP_CHAIN138_ROUTER \
     "addSupportedToken(uint64,address)" \
     $DESTINATION_SELECTOR \
     $TOKEN_ADDRESS \
     --rpc-url $RPC_URL_138 \
     --private-key $PRIVATE_KEY
   ```

4. **Configure Bridge**
   ```bash
   cast send $BRIDGE_ADDRESS \
     "addDestination(uint64,address)" \
     $DESTINATION_SELECTOR \
     $DESTINATION_BRIDGE_ADDRESS \
     --rpc-url $RPC_URL_138 \
     --private-key $PRIVATE_KEY
   ```

### Updating Fee Configuration

```bash
cast send $CCIP_CHAIN138_ROUTER \
  "updateFees(uint256,uint256)" \
  $NEW_BASE_FEE \
  $NEW_DATA_FEE_PER_BYTE \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY
```

### Changing Admin

```bash
# Router
cast send $CCIP_CHAIN138_ROUTER \
  "changeAdmin(address)" \
  $NEW_ADMIN \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

# Bridge
cast send $BRIDGE_ADDRESS \
  "changeAdmin(address)" \
  $NEW_ADMIN \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY
```

---

## Troubleshooting Guide

### Issue: Router Not Responding

**Symptoms**:
- Cannot send messages
- Contract calls fail

**Diagnosis**:
```bash
# Check contract code
cast code $CCIP_CHAIN138_ROUTER --rpc-url $RPC_URL_138

# Check recent transactions
cast tx $LAST_TX_HASH --rpc-url $RPC_URL_138
```

**Solutions**:
1. Verify RPC connectivity
2. Check contract is deployed
3. Verify network is synced
4. Check for contract upgrades

### Issue: Bridge Transfer Fails

**Symptoms**:
- Transfer initiated but not completed
- Error messages in logs

**Diagnosis**:
```bash
# Check bridge configuration
cast call $BRIDGE "destinations(uint64)" $DEST_SELECTOR --rpc-url $RPC_URL_138

# Check router connection
cast call $BRIDGE "ccipRouter()" --rpc-url $RPC_URL_138

# Check message status
cast logs --address $BRIDGE "CrossChainTransferInitiated" --rpc-url $RPC_URL_138
```

**Solutions**:
1. Verify destination is configured
2. Check router is accessible
3. Verify sufficient fees
4. Check destination chain status

### Issue: High Failure Rate

**Symptoms**:
- Many failed messages/transfers
- Error events in logs

**Diagnosis**:
```bash
# Analyze failure patterns
cast logs --address $ROUTER "MessageSent" --rpc-url $RPC_URL_138 | grep -i error

# Check fee configuration
cast call $ROUTER "baseFee()" --rpc-url $RPC_URL_138
```

**Solutions**:
1. Review error logs
2. Check fee configuration
3. Verify destination chains
4. Check network conditions

### Issue: Configuration Changes Not Applied

**Symptoms**:
- Configuration calls succeed but changes not visible

**Diagnosis**:
```bash
# Verify current configuration
cast call $CONTRACT "destinations(uint64)" $SELECTOR --rpc-url $RPC_URL_138

# Check transaction status
cast tx $TX_HASH --rpc-url $RPC_URL_138
```

**Solutions**:
1. Verify transaction was mined
2. Check correct contract address
3. Verify admin permissions
4. Check for event emissions

---

## Emergency Response

### Emergency Pause (if implemented)

If contracts have pause functionality:

```bash
# Pause router (if function exists)
cast send $ROUTER "pause()" --rpc-url $RPC_URL_138 --private-key $PRIVATE_KEY

# Pause bridge
cast send $BRIDGE "pause()" --rpc-url $RPC_URL_138 --private-key $PRIVATE_KEY
```

### Disable Destination Chain

```bash
# Remove destination from bridge
cast send $BRIDGE \
  "removeDestination(uint64)" \
  $PROBLEMATIC_SELECTOR \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

# Remove from router
cast send $ROUTER \
  "removeSupportedChain(uint64)" \
  $PROBLEMATIC_SELECTOR \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY
```

### Withdraw Fees (Emergency)

```bash
# Withdraw native ETH fees
cast send $ROUTER \
  "withdrawNativeFees()" \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY

# Withdraw ERC20 fees
cast send $ROUTER \
  "withdrawFees(uint256)" \
  $AMOUNT \
  --rpc-url $RPC_URL_138 \
  --private-key $PRIVATE_KEY
```

### Incident Response Checklist

1. **Identify Issue**
   - Check monitoring alerts
   - Review recent logs
   - Verify contract status

2. **Assess Impact**
   - How many users affected?
   - What operations are failing?
   - Is data at risk?

3. **Contain Issue**
   - Disable affected destinations
   - Pause operations if needed
   - Notify stakeholders

4. **Resolve Issue**
   - Apply fixes
   - Verify resolution
   - Monitor for recurrence

5. **Post-Incident**
   - Document incident
   - Review root cause
   - Update procedures

---

## Common Operations

### Check Contract Status

```bash
# Router status
cast call $ROUTER "feeToken()" --rpc-url $RPC_URL_138
cast call $ROUTER "baseFee()" --rpc-url $RPC_URL_138
cast call $ROUTER "supportedChains(uint64)" $SELECTOR --rpc-url $RPC_URL_138

# Bridge status
cast call $BRIDGE "ccipRouter()" --rpc-url $RPC_URL_138
cast call $BRIDGE "destinations(uint64)" $SELECTOR --rpc-url $RPC_URL_138
cast call $BRIDGE "getDestinationChains()" --rpc-url $RPC_URL_138
```

### View Recent Events

```bash
# Router events
cast logs --address $ROUTER --from-block latest --rpc-url $RPC_URL_138

# Bridge events
cast logs --address $BRIDGE --from-block latest --rpc-url $RPC_URL_138
```

### Calculate Fees

```bash
# Estimate fee for message
cast call $ROUTER \
  "getFee(uint64,tuple)" \
  $DEST_SELECTOR \
  $MESSAGE \
  --rpc-url $RPC_URL_138
```

---

## Related Documentation

- [CCIP Deployment Guide](../ccip/DEPLOYMENT_GUIDE_CHAIN138.md)
- [CCIP Monitoring](CCIP_MONITORING.md)
- [CCIP Review](../CCIP_CHAIN138_REVIEW.md)

---

**Last Updated**: 2025-01-27