smom-dbis-138/docs/archive/status-reports/phase1/NEXT_STEPS_COMPLETE.md

Phase 1: Next Steps - Complete Setup Guide

✅ Deployment Verification

All Phase 1 resources have been successfully deployed:

• 6 Virtual Machines: 5 backend VMs + 1 Nginx proxy
• Networking: VNets, NSGs, and public IP for Nginx proxy
• Storage: Boot diagnostics and backup storage accounts
• Monitoring: Log Analytics Workspaces
• Backup: Recovery Services Vaults
• Security: Key Vault with Managed Identity access

📋 Next Steps Checklist

1. Verify VM Access ✅

Nginx Proxy (West Europe):

ssh besuadmin@20.160.58.99

Backend VMs (US Regions):

# Central US
ssh besuadmin@10.3.1.4

# East US
ssh besuadmin@10.1.1.4

# East US 2
ssh besuadmin@10.4.1.4

# West US
ssh besuadmin@<private-ip>  # Check terraform output

# West US 2
ssh besuadmin@<private-ip>  # Check terraform output

Note: Backend VMs use private IPs only. You'll need to:

• Connect via VPN/ExpressRoute, OR
• Use Azure Bastion, OR
• SSH from another VM in the same VNet

2. Configure Besu Nodes on Backend VMs

For each backend VM, run:

# SSH to the VM first
ssh besuadmin@<vm-private-ip>

# Run the setup script
cd /opt/besu
sudo bash /opt/besu/setup.sh  # This was created by cloud-init

# Or use the provided script
wget https://raw.githubusercontent.com/your-repo/terraform/phases/phase1/scripts/setup-besu-node.sh
chmod +x setup-besu-node.sh
./setup-besu-node.sh besu-node 0 <region-name>

Manual steps:

1. Replace /opt/besu/config/genesis.json with your actual genesis file
2. If validator, add keys to /opt/besu/keys/
3. Start Besu: sudo systemctl start besu.service
4. Verify: sudo systemctl status besu.service
5. Check logs: sudo journalctl -u besu.service -f

3. Set Up Cloudflare Tunnel on Nginx Proxy

✅ Cloudflare credentials are automatically loaded from .env file!

Option A: Cloudflare Tunnel on Nginx Proxy Only (Recommended for initial setup)

# SSH to Nginx proxy
ssh besuadmin@20.160.58.99

# Run setup script (credentials loaded automatically from .env)
wget https://raw.githubusercontent.com/your-repo/terraform/phases/phase1/scripts/setup-cloudflare-tunnel.sh
chmod +x setup-cloudflare-tunnel.sh
./setup-cloudflare-tunnel.sh rpc.yourdomain.com

Or use the automated setup script (runs locally, provides instructions):

cd terraform/phases/phase1
./scripts/automated-cloudflare-setup.sh rpc.yourdomain.com

Manual steps (if not using automated script):

1. Install cloudflared (already in cloud-init)
2. Login: sudo cloudflared tunnel login (or use API token from .env)
3. Create tunnel: sudo cloudflared tunnel create phase1-nginx-proxy
4. Configure /etc/cloudflared/config.yml with tunnel ID
5. Create DNS route: sudo cloudflared tunnel route dns phase1-nginx-proxy rpc.yourdomain.com
   • Note: DNS route creation will use Cloudflare API token from .env if available
6. Start service: sudo systemctl enable cloudflared && sudo systemctl start cloudflared

4. Configure Cross-Region Connectivity

Current Challenge: Nginx proxy (West Europe) cannot reach backend VMs (US regions) via private IPs.

Solution Options:

Option A: Azure VPN/ExpressRoute (Recommended for Production)

1. Deploy VPN Gateway in each region
2. Connect regions via VPN
3. Update NSG rules to allow Nginx proxy subnet access
4. Nginx can then reach backend VMs via private IPs

Option B: Cloudflare Tunnel on Each Backend VM

For each backend VM:

# SSH to backend VM
ssh besuadmin@<vm-private-ip>

# Run setup script
wget https://raw.githubusercontent.com/your-repo/terraform/phases/phase1/scripts/setup-cloudflare-tunnel-backend.sh
chmod +x setup-cloudflare-tunnel-backend.sh
./setup-cloudflare-tunnel-backend.sh <vm-name> <domain-prefix> <cloudflare-account-id>

Then update Nginx to use Cloudflare Tunnel endpoints instead of private IPs.

Option C: Temporary Public IPs (Testing Only)

For initial testing, you can temporarily:

1. Add public IPs to backend VMs
2. Restrict NSG rules to Nginx proxy IP only
3. Update Nginx configuration with public IPs

⚠️ WARNING: This is not recommended for production!

5. Update Nginx Backend Configuration

Once connectivity is established, update Nginx with backend IPs:

# SSH to Nginx proxy
ssh besuadmin@20.160.58.99

# Update backend IPs
wget https://raw.githubusercontent.com/your-repo/terraform/phases/phase1/scripts/update-nginx-backends.sh
chmod +x update-nginx-backends.sh
./update-nginx-backends.sh "10.1.1.4,10.2.1.4,10.3.1.4,10.4.1.4,10.5.1.4"

Or manually edit /etc/nginx/nginx.conf and update the upstream blocks.

6. Configure Cloudflare DNS and SSL

1. DNS Configuration:

   • Go to Cloudflare Dashboard → DNS
   • Add CNAME record:
     • Name: rpc (or @ for root)
     • Target: <tunnel-id>.cfargotunnel.com
     • Proxy: Enabled (orange cloud)

2. SSL/TLS Configuration:

   • Go to SSL/TLS → Overview
   • Set encryption mode to "Full" or "Full (strict)"
   • Cloudflare will automatically provision SSL certificates

3. Additional Settings:

   • Enable "Always Use HTTPS"
   • Configure Security → WAF rules as needed
   • Set up DDoS protection (automatic with Cloudflare)

7. Verify End-to-End Connectivity

1. Test Besu RPC:

   curl -X POST https://rpc.yourdomain.com/rpc \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
   

2. Test WebSocket:

   wscat -c wss://rpc.yourdomain.com/ws
   

3. Check Nginx Logs:

   sudo tail -f /var/log/nginx/access.log
   sudo tail -f /var/log/nginx/error.log
   

4. Check Cloudflare Tunnel:

   sudo cloudflared tunnel info <tunnel-id>
   sudo journalctl -u cloudflared -f
   

🔧 Troubleshooting

Backend VMs Unreachable

Issue: Nginx proxy cannot reach backend VMs.

Solutions:

1. Deploy VPN/ExpressRoute for cross-region connectivity
2. Install Cloudflare Tunnel on each backend VM
3. Temporarily use public IPs (testing only)

Cloudflare Tunnel Not Connecting

Issue: Tunnel shows as disconnected.

Solutions:

1. Check tunnel status: sudo cloudflared tunnel info <tunnel-id>
2. Verify credentials: ls -la /root/.cloudflared/
3. Check logs: sudo journalctl -u cloudflared -f
4. Re-authenticate: sudo cloudflared tunnel login

Besu Node Not Starting

Issue: Besu service fails to start.

Solutions:

1. Check logs: sudo journalctl -u besu.service -f
2. Verify Docker: sudo systemctl status docker
3. Check configuration: cat /opt/besu/config/besu-config.toml
4. Verify genesis file: cat /opt/besu/config/genesis.json

Nginx Configuration Errors

Issue: Nginx fails to reload.

Solutions:

1. Test configuration: sudo nginx -t
2. Check syntax: sudo nginx -T
3. View error log: sudo tail -f /var/log/nginx/error.log
4. Restore backup: sudo cp /etc/nginx/nginx.conf.backup.* /etc/nginx/nginx.conf

📊 Monitoring

Key Metrics to Monitor

1. Besu Nodes:

   • Block height synchronization
   • Peer connections
   • RPC request latency
   • Memory and CPU usage

2. Nginx Proxy:

   • Request rate and latency
   • Backend health status
   • Error rates
   • Connection pool usage

3. Cloudflare Tunnel:

   • Connection status
   • Data transfer
   • Error rates

Log Locations

• Besu: /opt/besu/logs/besu.log or docker logs besu-<node-type>-<index>
• Nginx: /var/log/nginx/access.log and /var/log/nginx/error.log
• Cloudflare Tunnel: journalctl -u cloudflared -f
• System: journalctl -u besu.service -f

🔐 Security Checklist

• Restrict NSG rules to specific IP ranges
• Configure Key Vault network ACLs
• Enable Cloudflare WAF rules
• Set up SSL/TLS (Full or Full strict)
• Configure firewall rules on VMs
• Enable Azure Monitor and alerts
• Set up backup verification
• Configure log retention policies

📝 Scripts Provided

All setup scripts are available in terraform/phases/phase1/scripts/:

1. setup-cloudflare-tunnel.sh - Cloudflare Tunnel on Nginx proxy
2. setup-besu-node.sh - Besu node configuration
3. update-nginx-backends.sh - Update Nginx backend IPs
4. setup-cloudflare-tunnel-backend.sh - Cloudflare Tunnel on backend VMs

🎯 Success Criteria

Phase 1 setup is complete when:

• All VMs are deployed and accessible
• Besu nodes are running on all backend VMs
• Cloudflare Tunnel is connected and routing traffic
• Nginx proxy is routing to backend VMs
• DNS is configured and resolving
• SSL/TLS is enabled and working
• RPC endpoints are accessible via domain
• Monitoring and logging are configured
• Backups are configured and tested

---

For detailed instructions, see:

• CLOUDFLARE_TUNNEL_SETUP.md - Cloudflare Tunnel configuration
• DEPLOYMENT_COMPLETE.md - Deployment summary
• Scripts in scripts/ directory