1fb7266469
- Introduced Aggregator.sol for Chainlink-compatible oracle functionality, including round-based updates and access control. - Added OracleWithCCIP.sol to extend Aggregator with CCIP cross-chain messaging capabilities. - Created .gitmodules to include OpenZeppelin contracts as a submodule. - Developed a comprehensive deployment guide in NEXT_STEPS_COMPLETE_GUIDE.md for Phase 2 and smart contract deployment. - Implemented Vite configuration for the orchestration portal, supporting both Vue and React frameworks. - Added server-side logic for the Multi-Cloud Orchestration Portal, including API endpoints for environment management and monitoring. - Created scripts for resource import and usage validation across non-US regions. - Added tests for CCIP error handling and integration to ensure robust functionality. - Included various new files and directories for the orchestration portal and deployment scripts.
153 lines
4.1 KiB
Markdown
153 lines
4.1 KiB
Markdown
# Maximum Parallel Deployment Strategy
|
|
|
|
## Overview
|
|
|
|
This strategy maximizes parallel execution across **ALL** deployment phases, achieving the fastest possible deployment time.
|
|
|
|
## Parallelism Settings
|
|
|
|
### Terraform
|
|
- **Parallelism**: 128 (maximum)
|
|
- **Default**: 10
|
|
- **Speedup**: 12.8x faster than default
|
|
|
|
### Kubernetes Configuration
|
|
- **Parallelism**: 24 (one per region)
|
|
- **Method**: xargs -P 24
|
|
- **Speedup**: 24x faster than sequential
|
|
|
|
### Besu Deployment
|
|
- **Parallelism**: 48 (one per validator)
|
|
- **Method**: xargs -P 48
|
|
- **Speedup**: 48x faster than sequential
|
|
|
|
### Contract Deployment
|
|
- **Parallelism**: 2 (Mainnet + Chain-138)
|
|
- **Method**: Background jobs
|
|
- **Speedup**: 2x faster than sequential
|
|
|
|
### Monitoring
|
|
- **Parallelism**: 24 (one per region)
|
|
- **Method**: xargs -P 24
|
|
- **Speedup**: 24x faster than sequential
|
|
|
|
## Deployment Modes
|
|
|
|
### Mode 1: Maximum Parallel (Recommended)
|
|
```bash
|
|
./scripts/deployment/deploy-max-parallel.sh
|
|
```
|
|
- **Terraform**: parallelism=128
|
|
- **Phases**: Sequential but each phase is maximally parallel
|
|
- **Time**: 30-60 minutes
|
|
- **Risk**: Medium (may hit API limits)
|
|
|
|
### Mode 2: Ultra Parallel (Fastest)
|
|
```bash
|
|
./scripts/deployment/deploy-ultra-parallel.sh
|
|
```
|
|
- **All Phases**: Run simultaneously
|
|
- **Terraform**: parallelism=128
|
|
- **Time**: 20-40 minutes
|
|
- **Risk**: High (may hit API limits, requires careful monitoring)
|
|
|
|
## Performance Comparison
|
|
|
|
| Mode | Time | Speedup | Risk |
|
|
|------|------|---------|------|
|
|
| Sequential | 5-9 hours | 1x | Low |
|
|
| Standard Parallel | 45-85 min | 6-12x | Low |
|
|
| Maximum Parallel | 30-60 min | 10-18x | Medium |
|
|
| Ultra Parallel | 20-40 min | 15-27x | High |
|
|
|
|
## Azure API Rate Limits
|
|
|
|
### Considerations
|
|
- **Subscription Limits**: May have per-subscription rate limits
|
|
- **Resource Provider Limits**: Each provider (Compute, Network, etc.) has limits
|
|
- **Regional Limits**: Each region has independent limits
|
|
|
|
### Mitigation
|
|
- **Retry Logic**: Automatic retries with exponential backoff
|
|
- **Throttling**: Terraform handles throttling automatically
|
|
- **Monitoring**: Monitor API usage during deployment
|
|
|
|
## Deployment Scripts
|
|
|
|
### Maximum Parallel
|
|
- `deploy-max-parallel.sh` - Maximum parallelism per phase
|
|
- `configure-kubernetes-max-parallel.sh` - Maximum K8s parallelism
|
|
- `deploy-besu-max-parallel.sh` - Maximum Besu parallelism
|
|
- `verify-all-max-parallel.sh` - Maximum verification parallelism
|
|
|
|
### Ultra Parallel
|
|
- `deploy-ultra-parallel.sh` - All phases simultaneously
|
|
|
|
## Best Practices
|
|
|
|
### 1. Start with Maximum Parallel
|
|
- Use `deploy-max-parallel.sh` first
|
|
- Monitor for API rate limit errors
|
|
- Adjust parallelism if needed
|
|
|
|
### 2. Monitor Progress
|
|
- Watch Terraform output
|
|
- Monitor Azure Portal for resource creation
|
|
- Check logs for errors
|
|
|
|
### 3. Handle Failures
|
|
- Some regions may fail (retry independently)
|
|
- Verify successful regions
|
|
- Retry failed regions separately
|
|
|
|
### 4. Gradual Rollout (Optional)
|
|
- Deploy 8 regions first (test)
|
|
- Then deploy remaining 16 regions
|
|
- Reduces risk of hitting limits
|
|
|
|
## Recommended Approach
|
|
|
|
### Phase 1: Test with Maximum Parallel
|
|
```bash
|
|
./scripts/deployment/deploy-max-parallel.sh
|
|
```
|
|
|
|
### Phase 2: If Successful, Use Ultra Parallel
|
|
```bash
|
|
./scripts/deployment/deploy-ultra-parallel.sh
|
|
```
|
|
|
|
### Phase 3: Monitor and Adjust
|
|
- Monitor API usage
|
|
- Adjust parallelism if hitting limits
|
|
- Retry failed regions
|
|
|
|
## Expected Results
|
|
|
|
### Maximum Parallel
|
|
- **Infrastructure**: 20-30 minutes
|
|
- **Kubernetes**: 3-5 minutes
|
|
- **Besu**: 5-10 minutes
|
|
- **Total**: 30-60 minutes
|
|
|
|
### Ultra Parallel
|
|
- **All Phases**: 20-40 minutes
|
|
- **Fastest Possible**: All operations concurrent
|
|
|
|
## Troubleshooting
|
|
|
|
### API Rate Limits
|
|
- **Symptom**: "Too Many Requests" errors
|
|
- **Solution**: Reduce parallelism or add delays
|
|
- **Command**: `terraform apply -parallelism=64` (half)
|
|
|
|
### Partial Failures
|
|
- **Symptom**: Some regions succeed, others fail
|
|
- **Solution**: Retry failed regions independently
|
|
- **Command**: `terraform apply -target=module.region_deployment["region"]`
|
|
|
|
### Timeout Issues
|
|
- **Symptom**: Operations timing out
|
|
- **Solution**: Increase timeout or reduce parallelism
|
|
- **Note**: Some operations (AKS cluster creation) take 10-15 minutes
|