1fb7266469
- Introduced Aggregator.sol for Chainlink-compatible oracle functionality, including round-based updates and access control. - Added OracleWithCCIP.sol to extend Aggregator with CCIP cross-chain messaging capabilities. - Created .gitmodules to include OpenZeppelin contracts as a submodule. - Developed a comprehensive deployment guide in NEXT_STEPS_COMPLETE_GUIDE.md for Phase 2 and smart contract deployment. - Implemented Vite configuration for the orchestration portal, supporting both Vue and React frameworks. - Added server-side logic for the Multi-Cloud Orchestration Portal, including API endpoints for environment management and monitoring. - Created scripts for resource import and usage validation across non-US regions. - Added tests for CCIP error handling and integration to ensure robust functionality. - Included various new files and directories for the orchestration portal and deployment scripts.
2.2 KiB
2.2 KiB
Oracle Recovery Procedures
Recovery Scenarios
Stale Oracle Recovery
Scenario: Oracle has not updated in > 5 minutes
Steps:
- Identify stale oracle
- Check transmitter status
- Verify data sources
- Force update if needed
- Monitor recovery
Commands:
# Check oracle state
cast call $AGGREGATOR_ADDRESS "latestRoundData()" --rpc-url $RPC_URL
# Force update
cast send $AGGREGATOR_ADDRESS "updateAnswer(uint256)" $PRICE \
--rpc-url $RPC_URL \
--private-key $PRIVATE_KEY
Failed Transmitter Recovery
Scenario: Transmitter is unavailable
Steps:
- Identify failed transmitter
- Remove failed transmitter
- Add backup transmitter
- Verify updates resume
- Monitor recovery
Commands:
# Remove failed transmitter
cast send $AGGREGATOR_ADDRESS "removeTransmitter(address)" $FAILED_TRANSMITTER \
--rpc-url $RPC_URL \
--private-key $ADMIN_KEY
# Add backup transmitter
cast send $AGGREGATOR_ADDRESS "addTransmitter(address)" $BACKUP_TRANSMITTER \
--rpc-url $RPC_URL \
--private-key $ADMIN_KEY
Data Source Failure Recovery
Scenario: Data source is unavailable
Steps:
- Identify failed data source
- Switch to backup data source
- Verify updates resume
- Monitor recovery
- Document incident
CCIP Failure Recovery
Scenario: CCIP messages failing
Steps:
- Check CCIP router status
- Verify LINK balance
- Check message queue
- Retry failed messages
- Monitor recovery
Recovery Verification
Verify Oracle Updates
# Check latest round
cast call $AGGREGATOR_ADDRESS "latestRoundData()" --rpc-url $RPC_URL
# Monitor updates
watch -n 5 "cast call $AGGREGATOR_ADDRESS 'latestRoundData()' --rpc-url $RPC_URL"
Verify Transmitter Status
# Check transmitter list
cast call $AGGREGATOR_ADDRESS "getTransmitters()" --rpc-url $RPC_URL
# Check transmitter status
cast call $AGGREGATOR_ADDRESS "isTransmitter(address)" $TRANSMITTER --rpc-url $RPC_URL
Post-Recovery
- Document Incident: Document what happened
- Root Cause Analysis: Identify root cause
- Preventive Measures: Implement preventive measures
- Update Runbooks: Update runbooks if needed
- Monitor: Continue monitoring