smom-dbis-138/terraform/phases/phase1/ALL_NEXT_STEPS.md

# Phase 1: All Next Steps

## ✅ Completed Tasks

### Infrastructure ✅
- [x] Deploy 104 resources (VMs, networking, storage, monitoring)
- [x] Configure Nginx backend with 5 backend VM IPs
- [x] Set up Cloudflare Tunnel
- [x] Configure DNS (rpc.d-bis.org → Cloudflare Tunnel)
- [x] Enable SSL/TLS (automatic via Cloudflare)
- [x] Verify endpoint: `https://rpc.d-bis.org/health` → "healthy"

## 📋 Remaining Tasks

### Task 1: Configure Besu Nodes on Backend VMs
**Priority**: High  
**Status**: Pending  
**Requires**: VPN/Bastion access to backend VMs

**Backend VM Details:**
```
Region: centralus
  IP: 10.3.1.4
  VM: az-p-cus-vm-besu-node-0
  Command: ssh besuadmin@10.3.1.4

Region: eastus
  IP: 10.1.1.4
  VM: az-p-eus-vm-besu-node-0
  Command: ssh besuadmin@10.1.1.4

Region: eastus2
  IP: 10.4.1.4
  VM: az-p-eus2-vm-besu-node-0
  Command: ssh besuadmin@10.4.1.4

Region: westus
  IP: 10.2.1.4
  VM: az-p-wus-vm-besu-node-0
  Command: ssh besuadmin@10.2.1.4

Region: westus2
  IP: 10.5.1.4
  VM: az-p-wus2-vm-besu-node-0
  Command: ssh besuadmin@10.5.1.4
```

**Steps for Each VM:**
1. Connect via VPN/Bastion:
   ```bash
   ssh besuadmin@<backend-vm-ip>
   ```

2. Download and run setup script:
   ```bash
   wget https://raw.githubusercontent.com/your-repo/terraform/phases/phase1/scripts/setup-besu-node.sh
   chmod +x setup-besu-node.sh
   sudo ./setup-besu-node.sh besu-node 0 <region>
   ```

3. Verify Besu is running:
   ```bash
   sudo systemctl status besu.service
   sudo journalctl -u besu.service -f
   ```

4. Test RPC endpoints:
   ```bash
   curl http://localhost:8545
   curl http://localhost:8546
   curl http://localhost:9545/metrics
   ```

**Expected Result:**
- Docker container running Besu
- RPC endpoints responding
- Metrics endpoint accessible
- Service starts on boot

---

### Task 2: Implement Cross-Region Connectivity
**Priority**: High  
**Status**: Pending  
**Requires**: Network infrastructure decision

**Problem**: Nginx proxy (West Europe) needs to reach backend VMs (US regions) via private IPs.

**Options:**

#### Option A: Azure VPN/ExpressRoute (Recommended for Production)
- Set up Azure VPN Gateway or ExpressRoute
- Connect all regions to hub network
- Nginx can reach backend VMs via private IPs
- Most secure and performant

**Steps:**
1. Create VPN Gateway in hub region
2. Configure site-to-site VPN connections
3. Update route tables
4. Test connectivity: `ping 10.1.1.4` from Nginx proxy

#### Option B: Cloudflare Tunnel on Backend VMs
- Install Cloudflare Tunnel on each backend VM
- Create tunnels for each region
- Nginx connects via Cloudflare Tunnel
- Simpler but adds latency

**Steps:**
1. SSH to each backend VM (via VPN/Bastion)
2. Run: `/tmp/setup-cloudflare-tunnel-backend.sh <vm-name> <region>`
3. Configure DNS for each backend endpoint
4. Update Nginx to use Cloudflare Tunnel endpoints

#### Option C: Azure Private Link (Azure-Native)
- Use Azure Private Link for cross-region connectivity
- Most Azure-integrated solution
- Requires Private Link configuration

**Steps:**
1. Create Private Link services for backend VMs
2. Create Private Endpoints in Nginx region
3. Update Nginx configuration
4. Test connectivity

**Recommendation**: Option A (VPN/ExpressRoute) for production, Option B for quick setup.

---

### Task 3: Update Nginx Configuration for Backend Connectivity
**Priority**: Medium  
**Status**: Pending (depends on Task 2)

**After cross-region connectivity is established:**

1. Verify backend connectivity from Nginx:
   ```bash
   ssh besuadmin@20.160.58.99
   curl http://10.1.1.4:8545
   curl http://10.2.1.4:8545
   # Test all 5 backend VMs
   ```

2. Update Nginx upstream configuration if needed:
   ```bash
   sudo nano /etc/nginx/nginx.conf
   # Verify upstream blocks have correct IPs
   sudo nginx -t
   sudo systemctl reload nginx
   ```

3. Test RPC endpoints:
   ```bash
   curl -X POST https://rpc.d-bis.org/rpc \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
   ```

---

### Task 4: Configure Besu Genesis and Network
**Priority**: High  
**Status**: Pending

**Steps:**
1. Generate or obtain genesis file
2. Upload genesis file to storage account or Key Vault
3. Update Besu configuration on each VM:
   ```bash
   # On each backend VM
   sudo nano /opt/besu/config/genesis.json
   # Or download from storage
   ```

4. Configure network parameters:
   - Chain ID: 138
   - Network name: DeFi Oracle Meta Mainnet
   - Bootnodes (if applicable)

---

### Task 5: Configure Validator Keys (If Applicable)
**Priority**: Medium  
**Status**: Pending

**If backend VMs are validators:**

1. Generate validator keys (if not already done)
2. Store keys in Azure Key Vault
3. Update Besu configuration to use keys:
   ```bash
   # On validator VMs
   az keyvault secret show --vault-name <vault-name> --name validator-key-0
   # Copy to /opt/besu/keys/
   ```

---

### Task 6: Set Up Monitoring and Alerting
**Priority**: Medium  
**Status**: Pending

**Steps:**
1. Configure Log Analytics Workspace queries
2. Set up alerts for:
   - VM availability
   - Besu node health
   - Nginx proxy status
   - Cloudflare Tunnel status
3. Configure dashboards in Azure Monitor

---

### Task 7: Set Up Backup and Recovery
**Priority**: Medium  
**Status**: Pending

**Steps:**
1. Configure backup policies for VMs
2. Test backup and restore procedures
3. Document recovery procedures
4. Set up backup monitoring

---

### Task 8: Security Hardening
**Priority**: High  
**Status**: Pending

**Steps:**
1. Review and tighten NSG rules
2. Enable Azure Security Center recommendations
3. Configure Key Vault access policies
4. Review and rotate secrets
5. Enable Azure Defender (if applicable)
6. Configure network security groups for least privilege

---

### Task 9: Performance Testing
**Priority**: Medium  
**Status**: Pending

**Steps:**
1. Load test RPC endpoints
2. Test WebSocket connections
3. Verify load balancing across backend VMs
4. Monitor resource usage
5. Optimize configurations as needed

---

### Task 10: Documentation and Runbooks
**Priority**: Low  
**Status**: Pending

**Steps:**
1. Document operational procedures
2. Create runbooks for common issues
3. Document disaster recovery procedures
4. Create architecture diagrams
5. Document API endpoints and usage

---

## 🎯 Priority Order

### Immediate (High Priority)
1. **Configure Besu Nodes** (Task 1) - Required for RPC functionality
2. **Implement Cross-Region Connectivity** (Task 2) - Required for Nginx to reach backends
3. **Security Hardening** (Task 8) - Critical for production

### Short Term (Medium Priority)
4. **Update Nginx Configuration** (Task 3) - After connectivity is established
5. **Configure Besu Genesis** (Task 4) - Required for blockchain functionality
6. **Set Up Monitoring** (Task 6) - Important for operations
7. **Set Up Backup** (Task 7) - Important for data protection

### Long Term (Lower Priority)
8. **Configure Validator Keys** (Task 5) - If validators are needed
9. **Performance Testing** (Task 9) - For optimization
10. **Documentation** (Task 10) - For maintainability

---

## 📊 Current Status Summary

### ✅ Completed
- Infrastructure: 104 resources deployed
- Nginx: Configured and running
- Cloudflare Tunnel: Active and connected
- DNS: Configured and working
- SSL/TLS: Automatic via Cloudflare
- Endpoint: `https://rpc.d-bis.org/health` → "healthy"

### ⏳ In Progress
- None (all automated steps complete)

### 📋 Pending
- Besu node configuration (5 VMs)
- Cross-region connectivity
- Besu genesis configuration
- Security hardening
- Monitoring setup
- Backup configuration

---

## 🚀 Quick Start Commands

### Configure Besu on Backend VM (Example: East US)
```bash
# Via VPN/Bastion
ssh besuadmin@10.1.1.4
wget https://raw.githubusercontent.com/your-repo/terraform/phases/phase1/scripts/setup-besu-node.sh
chmod +x setup-besu-node.sh
sudo ./setup-besu-node.sh besu-node 0 eastus
sudo systemctl status besu.service
```

### Test Endpoint
```bash
curl https://rpc.d-bis.org/health
curl -X POST https://rpc.d-bis.org/rpc \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
```

### Check Services
```bash
# Nginx Proxy
ssh besuadmin@20.160.58.99
sudo systemctl status nginx
sudo systemctl status cloudflared

# Backend VM (via VPN/Bastion)
ssh besuadmin@10.1.1.4
sudo systemctl status besu.service
docker ps
```

---

**Status**: All automated steps complete. Ready for manual configuration tasks.