smom-dbis-138/docs/deployment/DEPLOYMENT_ORDER.md

# Deployment Order - Complete Task List

This document defines the proper order for deploying the DeFi Oracle Meta Mainnet (ChainID 138) infrastructure and services.

## 📋 Deployment Phases

The deployment is organized into 8 phases, each building on the previous:

1. **Prerequisites & Setup** - Environment and tooling
2. **Foundation** - Core Azure infrastructure
3. **Networking** - Network infrastructure and security
4. **Compute** - AKS cluster and node pools
5. **Storage & Secrets** - Storage accounts and Key Vault
6. **Application** - Kubernetes workloads
7. **External Services** - DNS, SSL, and monitoring
8. **Contracts & Integration** - Smart contracts and external integrations

---

## Phase 1: Prerequisites & Setup

### 1.1 Azure Authentication & Configuration
- [ ] Install Azure CLI
- [ ] Login to Azure (`az login`)
- [ ] Verify subscription access
- [ ] Set default subscription
- [ ] Verify Azure CLI version

### 1.2 Environment Configuration
- [ ] Create `.env` file
- [ ] Set `AZURE_SUBSCRIPTION_ID`
- [ ] Set `AZURE_TENANT_ID`
- [ ] Set `AZURE_LOCATION=westeurope`
- [ ] Set `AZURE_RESOURCE_GROUP` (or use default)
- [ ] Set `CLOUDFLARE_ZONE_ID`
- [ ] Set `CLOUDFLARE_API_TOKEN`
- [ ] Verify environment variables

### 1.3 Prerequisites Verification
- [ ] Run `./scripts/azure/check-azure-prerequisites.sh`
- [ ] Verify resource providers are registered
- [ ] Check quotas for westeurope region
- [ ] Verify Terraform backend storage account exists
- [ ] Verify all required tools are installed (terraform, kubectl, helm, forge)

### 1.4 Key Generation
- [ ] Generate validator keys (`./scripts/key-management/generate-validator-keys.sh 4`)
- [ ] Generate oracle keys (`./scripts/key-management/generate-oracle-keys.sh`)
- [ ] Generate genesis file (`./scripts/generate-genesis.sh`)
- [ ] Verify keys are generated correctly

---

## Phase 2: Foundation Infrastructure

### 2.1 Terraform Initialization
- [ ] Navigate to `terraform/` directory
- [ ] Initialize Terraform (`terraform init`)
- [ ] Verify backend configuration
- [ ] Verify Terraform version (>= 1.0)

### 2.2 Terraform Configuration
- [ ] Copy `terraform.tfvars.example` to `terraform.tfvars`
- [ ] Set `environment = "prod"`
- [ ] Set `location = "westeurope"`
- [ ] Set `cluster_name` (following naming convention)
- [ ] Configure node counts and VM sizes
- [ ] Review and adjust tags

### 2.3 Resource Groups
- [ ] Create network resource group (`az-p-we-rg-net-001`)
- [ ] Create compute resource group (`az-p-we-rg-comp-001`)
- [ ] Create storage resource group (`az-p-we-rg-stor-001`)
- [ ] Create security resource group (`az-p-we-rg-sec-001`)
- [ ] Verify resource groups created

### 2.4 Terraform Planning
- [ ] Run `terraform plan`
- [ ] Review planned resources
- [ ] Verify naming convention compliance
- [ ] Check for any errors or warnings
- [ ] Save plan output for review

---

## Phase 3: Networking Infrastructure

### 3.1 Virtual Network
- [ ] Deploy virtual network (`az-p-we-vnet-main`)
- [ ] Configure address space (10.0.0.0/16)
- [ ] Verify VNet created

### 3.2 Subnets
- [ ] Create AKS subnet (`az-p-we-snet-aks`)
- [ ] Create validator subnet (`az-p-we-snet-valid`)
- [ ] Create sentry subnet (`az-p-we-snet-sent`)
- [ ] Create RPC subnet (`az-p-we-snet-rpc`)
- [ ] Create Application Gateway subnet (`az-p-we-snet-agw`)
- [ ] Configure service endpoints where needed
- [ ] Verify all subnets created

### 3.3 Network Security Groups
- [ ] Create validator NSG (`az-p-we-nsg-valid`)
- [ ] Create sentry NSG (`az-p-we-nsg-sent`)
- [ ] Create RPC NSG (`az-p-we-nsg-rpc`)
- [ ] Configure NSG rules (allow/deny)
- [ ] Associate NSGs with subnets
- [ ] Verify NSG rules

### 3.4 Public IPs and Load Balancers
- [ ] Create Application Gateway public IP (`az-p-we-pip-agw`)
- [ ] Verify public IP created
- [ ] Note public IP address for DNS configuration

---

## Phase 4: Compute Infrastructure

### 4.1 Key Vault Setup
- [ ] Create Key Vault (`az-p-we-kv-secrets-001`)
- [ ] Configure Key Vault access policies or RBAC
- [ ] Enable soft delete and purge protection
- [ ] Store validator keys in Key Vault
- [ ] Store oracle keys in Key Vault
- [ ] Verify Key Vault access

### 4.2 Log Analytics Workspace
- [ ] Create Log Analytics workspace (`az-p-we-law-main`)
- [ ] Configure retention period (90 days for prod)
- [ ] Verify workspace created

### 4.3 AKS Cluster
- [ ] Deploy AKS cluster (`az-p-we-aks-main`)
- [ ] Configure network plugin (Azure CNI)
- [ ] Configure network policy (Azure)
- [ ] Enable Azure Monitor
- [ ] Enable Azure Policy
- [ ] Configure Key Vault secrets provider
- [ ] Verify cluster is running

### 4.4 Node Pools
- [ ] Verify system node pool is created
- [ ] Create validator node pool (`az-p-we-aks-node-valid`)
- [ ] Create sentry node pool (`az-p-we-aks-node-sent`)
- [ ] Create RPC node pool (`az-p-we-aks-node-rpc`)
- [ ] Configure node labels and taints
- [ ] Verify all node pools are running

### 4.5 kubectl Configuration
- [ ] Get AKS credentials (`az aks get-credentials`)
- [ ] Verify kubectl access
- [ ] Test kubectl connection
- [ ] Verify node access

---

## Phase 5: Storage & Secrets

### 5.1 Storage Accounts
- [ ] Create backup storage account (`az-p-we-st-backup-001`)
- [ ] Create shared storage account (`az-p-we-st-shared-001`)
- [ ] Configure storage account security
- [ ] Enable versioning and soft delete
- [ ] Verify storage accounts

### 5.2 Storage Containers
- [ ] Create chaindata container
- [ ] Create config container
- [ ] Configure container access policies
- [ ] Verify containers

### 5.3 Key Vault Secrets
- [ ] Store all validator private keys
- [ ] Store oracle private key
- [ ] Store database passwords
- [ ] Store API keys
- [ ] Verify secrets are accessible from AKS

---

## Phase 6: Application Deployment

### 6.1 Kubernetes Namespace
- [ ] Create `besu-network` namespace
- [ ] Create `monitoring` namespace
- [ ] Configure namespace labels
- [ ] Verify namespaces

### 6.2 ConfigMaps and Secrets
- [ ] Create genesis config map
- [ ] Create static-nodes config map
- [ ] Create application config maps
- [ ] Create Kubernetes secrets from Key Vault
- [ ] Verify ConfigMaps and secrets

### 6.3 Validator Deployment
- [ ] Deploy validator StatefulSet
- [ ] Configure validator pods
- [ ] Verify validators are running
- [ ] Check validator logs
- [ ] Verify validators are syncing

### 6.4 Sentry Deployment
- [ ] Deploy sentry StatefulSet
- [ ] Configure sentry pods
- [ ] Verify sentries are running
- [ ] Check sentry logs
- [ ] Verify P2P connectivity

### 6.5 RPC Node Deployment
- [ ] Deploy RPC StatefulSet
- [ ] Configure RPC pods
- [ ] Verify RPC nodes are running
- [ ] Check RPC node logs
- [ ] Test RPC endpoint locally

### 6.6 Application Gateway
- [ ] Deploy Application Gateway (`az-p-we-agw-main`)
- [ ] Configure backend pools
- [ ] Configure HTTP settings
- [ ] Configure listeners
- [ ] Configure routing rules
- [ ] Configure WAF rules
- [ ] Verify Application Gateway is running

---

## Phase 7: External Services

### 7.1 DNS Configuration
- [ ] Get Application Gateway public IP
- [ ] Configure Cloudflare DNS records:
  - [ ] A record for root domain (`d-bis.org`)
  - [ ] A record for `www.d-bis.org`
  - [ ] A record for `rpc.d-bis.org`
  - [ ] A record for `rpc2.d-bis.org`
  - [ ] A record for `explorer.d-bis.org`
- [ ] Wait for DNS propagation (5-15 minutes)
- [ ] Verify DNS resolution

### 7.2 SSL/TLS Configuration
- [ ] Enable Cloudflare SSL/TLS (Full or Full Strict)
- [ ] Verify SSL certificates
- [ ] Test HTTPS access
- [ ] Configure certificate auto-renewal

### 7.3 Monitoring Setup
- [ ] Deploy Prometheus
- [ ] Deploy Grafana (optional)
- [ ] Configure alert rules
- [ ] Set up alert notifications
- [ ] Verify monitoring is collecting metrics

### 7.4 Blockscout Deployment
- [ ] Deploy PostgreSQL database for Blockscout
- [ ] Wait for database to be ready
- [ ] Deploy Blockscout application
- [ ] Run database migrations
- [ ] Configure Blockscout settings
- [ ] Verify Blockscout is accessible
- [ ] Configure CORS headers

---

## Phase 8: Contracts & Integration

### 8.1 Contract Deployment Preparation
- [ ] Set `RPC_URL` in `.env`
- [ ] Set `PRIVATE_KEY` in `.env` (deployment key)
- [ ] Verify RPC endpoint is accessible
- [ ] Test RPC connection

### 8.2 Smart Contract Deployment
- [ ] Deploy WETH contract
- [ ] Deploy Multicall contract
- [ ] Deploy Oracle Aggregator contract
- [ ] Deploy CCIP Router contract (optional)
- [ ] Verify all contracts deployed
- [ ] Save contract addresses

### 8.3 Token List Update
- [ ] Update token list with contract addresses
- [ ] Add token metadata
- [ ] Validate token list JSON
- [ ] Commit token list changes

### 8.4 Deployment Verification
- [ ] Run deployment verification script
- [ ] Test RPC endpoints (public)
- [ ] Test Blockscout explorer
- [ ] Test contract interactions
- [ ] Verify block production
- [ ] Check validator health
- [ ] Generate verification report

### 8.5 External Integration (Post-Deployment)
- [ ] Submit Ethereum-Lists PR
- [ ] Submit token list to CoinGecko
- [ ] Submit token list to Uniswap
- [ ] Verify MetaMask integration
- [ ] Test token auto-detection

---

## Quick Reference Commands

### Phase 1: Prerequisites
```bash
./scripts/deployment/azure-login.sh
./scripts/deployment/populate-env.sh
./scripts/azure/check-azure-prerequisites.sh
./scripts/key-management/generate-validator-keys.sh 4
```

### Phase 2-4: Infrastructure
```bash
cd terraform
terraform init
terraform plan
terraform apply
```

### Phase 5: Storage & Secrets
```bash
./scripts/key-management/azure-keyvault-setup.sh
```

### Phase 6: Kubernetes
```bash
az aks get-credentials --resource-group az-p-we-rg-comp-001 --name az-p-we-aks-main
kubectl apply -f k8s/base/namespace.yaml
helm install besu-validators ./helm/besu-network -f helm/besu-network/values-validators.yaml -n besu-network
```

### Phase 7: External Services
```bash
./scripts/deployment/get-app-gateway-ip.sh
./scripts/deployment/cloudflare-dns.sh --zone-id $CLOUDFLARE_ZONE_ID --api-token $CLOUDFLARE_API_TOKEN --ip <gateway-ip>
```

### Phase 8: Contracts
```bash
./scripts/deployment/deploy-weth.sh
./scripts/deployment/deploy-multicall.sh
./scripts/deployment/verify-deployment.sh
```

## Verification Checklist

After each phase, verify:

- [ ] All resources created successfully
- [ ] Naming convention followed
- [ ] Tags applied correctly
- [ ] No errors in logs
- [ ] Resources accessible
- [ ] Security configured properly

## Rollback Procedures

If deployment fails at any phase:

1. **Document the failure point**
2. **Review error logs**
3. **Fix the issue**
4. **Rollback if necessary** (terraform destroy for failed resources)
5. **Re-run from the failed phase**

## Estimated Timeline

- **Phase 1**: 30 minutes
- **Phase 2**: 1-2 hours
- **Phase 3**: 30 minutes
- **Phase 4**: 1-2 hours
- **Phase 5**: 30 minutes
- **Phase 6**: 2-3 hours
- **Phase 7**: 1 hour
- **Phase 8**: 1-2 hours

**Total**: ~8-12 hours for complete deployment

## Dependencies

```
Phase 1 (Prerequisites)
  └─> Phase 2 (Foundation)
      └─> Phase 3 (Networking)
          └─> Phase 4 (Compute)
              └─> Phase 5 (Storage)
                  └─> Phase 6 (Application)
                      └─> Phase 7 (External)
                          └─> Phase 8 (Contracts)
```

## Notes

- Each phase must be completed before moving to the next
- Some steps within a phase can be done in parallel
- Always verify each phase before proceeding
- Keep backups of configurations and keys
- Document any deviations from the standard process