1fb7266469
- Introduced Aggregator.sol for Chainlink-compatible oracle functionality, including round-based updates and access control. - Added OracleWithCCIP.sol to extend Aggregator with CCIP cross-chain messaging capabilities. - Created .gitmodules to include OpenZeppelin contracts as a submodule. - Developed a comprehensive deployment guide in NEXT_STEPS_COMPLETE_GUIDE.md for Phase 2 and smart contract deployment. - Implemented Vite configuration for the orchestration portal, supporting both Vue and React frameworks. - Added server-side logic for the Multi-Cloud Orchestration Portal, including API endpoints for environment management and monitoring. - Created scripts for resource import and usage validation across non-US regions. - Added tests for CCIP error handling and integration to ensure robust functionality. - Included various new files and directories for the orchestration portal and deployment scripts.
4.1 KiB
4.1 KiB
Maximum Parallel Deployment Strategy
Overview
This strategy maximizes parallel execution across ALL deployment phases, achieving the fastest possible deployment time.
Parallelism Settings
Terraform
- Parallelism: 128 (maximum)
- Default: 10
- Speedup: 12.8x faster than default
Kubernetes Configuration
- Parallelism: 24 (one per region)
- Method: xargs -P 24
- Speedup: 24x faster than sequential
Besu Deployment
- Parallelism: 48 (one per validator)
- Method: xargs -P 48
- Speedup: 48x faster than sequential
Contract Deployment
- Parallelism: 2 (Mainnet + Chain-138)
- Method: Background jobs
- Speedup: 2x faster than sequential
Monitoring
- Parallelism: 24 (one per region)
- Method: xargs -P 24
- Speedup: 24x faster than sequential
Deployment Modes
Mode 1: Maximum Parallel (Recommended)
./scripts/deployment/deploy-max-parallel.sh
- Terraform: parallelism=128
- Phases: Sequential but each phase is maximally parallel
- Time: 30-60 minutes
- Risk: Medium (may hit API limits)
Mode 2: Ultra Parallel (Fastest)
./scripts/deployment/deploy-ultra-parallel.sh
- All Phases: Run simultaneously
- Terraform: parallelism=128
- Time: 20-40 minutes
- Risk: High (may hit API limits, requires careful monitoring)
Performance Comparison
| Mode | Time | Speedup | Risk |
|---|---|---|---|
| Sequential | 5-9 hours | 1x | Low |
| Standard Parallel | 45-85 min | 6-12x | Low |
| Maximum Parallel | 30-60 min | 10-18x | Medium |
| Ultra Parallel | 20-40 min | 15-27x | High |
Azure API Rate Limits
Considerations
- Subscription Limits: May have per-subscription rate limits
- Resource Provider Limits: Each provider (Compute, Network, etc.) has limits
- Regional Limits: Each region has independent limits
Mitigation
- Retry Logic: Automatic retries with exponential backoff
- Throttling: Terraform handles throttling automatically
- Monitoring: Monitor API usage during deployment
Deployment Scripts
Maximum Parallel
deploy-max-parallel.sh- Maximum parallelism per phaseconfigure-kubernetes-max-parallel.sh- Maximum K8s parallelismdeploy-besu-max-parallel.sh- Maximum Besu parallelismverify-all-max-parallel.sh- Maximum verification parallelism
Ultra Parallel
deploy-ultra-parallel.sh- All phases simultaneously
Best Practices
1. Start with Maximum Parallel
- Use
deploy-max-parallel.shfirst - Monitor for API rate limit errors
- Adjust parallelism if needed
2. Monitor Progress
- Watch Terraform output
- Monitor Azure Portal for resource creation
- Check logs for errors
3. Handle Failures
- Some regions may fail (retry independently)
- Verify successful regions
- Retry failed regions separately
4. Gradual Rollout (Optional)
- Deploy 8 regions first (test)
- Then deploy remaining 16 regions
- Reduces risk of hitting limits
Recommended Approach
Phase 1: Test with Maximum Parallel
./scripts/deployment/deploy-max-parallel.sh
Phase 2: If Successful, Use Ultra Parallel
./scripts/deployment/deploy-ultra-parallel.sh
Phase 3: Monitor and Adjust
- Monitor API usage
- Adjust parallelism if hitting limits
- Retry failed regions
Expected Results
Maximum Parallel
- Infrastructure: 20-30 minutes
- Kubernetes: 3-5 minutes
- Besu: 5-10 minutes
- Total: 30-60 minutes
Ultra Parallel
- All Phases: 20-40 minutes
- Fastest Possible: All operations concurrent
Troubleshooting
API Rate Limits
- Symptom: "Too Many Requests" errors
- Solution: Reduce parallelism or add delays
- Command:
terraform apply -parallelism=64(half)
Partial Failures
- Symptom: Some regions succeed, others fail
- Solution: Retry failed regions independently
- Command:
terraform apply -target=module.region_deployment["region"]
Timeout Issues
- Symptom: Operations timing out
- Solution: Increase timeout or reduce parallelism
- Note: Some operations (AKS cluster creation) take 10-15 minutes