d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
master
proxmox/docs/02-architecture/NETWORK_ARCHITECTURE.md
defiQUG fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
Details
docs: Ledger Live integration, contract deploy learnings, NEXT_STEPS updates 
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands
- CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround
- CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check
- NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere
- MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates
- LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-12 15:46:57 -08:00

19 KiB
Raw Permalink Blame History

Network Architecture - Enterprise Orchestration Plan

Navigation: Home > Architecture > Network Architecture

Related: PHYSICAL_HARDWARE_INVENTORY.md | DOMAIN_STRUCTURE.md | ORCHESTRATION_DEPLOYMENT_GUIDE.md | 11-references/NETWORK_CONFIGURATION_MASTER.md | Runbooks & VLAN: 03-deployment/OPERATIONAL_RUNBOOKS.md (Phase 4, VLAN), 03-deployment/MISSING_CONTAINERS_LIST.md, 04-configuration/UDM_PRO_FIREWALL_MANUAL_CONFIGURATION.md

Last Updated: 2026-02-05
Document Version: 2.1
Status: 🟢 Active Documentation
Project: Sankofa / Phoenix / PanTel · ChainID 138 · Proxmox + Cloudflare DNS + NPMplus (edge: UDM Pro; Fastly or direct to 76.53.10.36)

Overview

This document defines the complete enterprise-grade network architecture for the Sankofa/Phoenix/PanTel Proxmox deployment, including:

  • Hardware role assignments (2× ER605, 3× ES216G, 1× ML110, 4× R630)
  • 6× /28 public IP blocks with role-based NAT pools
  • VLAN orchestration with private subnet allocations
  • Egress segmentation by role and security plane
  • Cloudflare Zero Trust integration patterns

Architecture Diagrams

Network Topology (High Level)

graph TB
    Internet[Internet]
    CF[Cloudflare Zero Trust]
    UDM[UDM Pro 76.53.10.34]
    NPM[NPMplus 192.168.11.167]
    ES1[ES216G-1 Core]
    ES2[ES216G-2 Compute]
    ML[ML110 192.168.11.10]
    R1[R630-01 192.168.11.11]
    R2[R630-02 192.168.11.12]
    Internet --> CF
    CF --> UDM
    UDM --> NPM
    NPM --> ES1
    ES1 --> ES2
    ES2 --> ML
    ES2 --> R1
    ES2 --> R2

VLAN Architecture (Selected VLANs)

graph TD
    V11[VLAN 11: MGMT-LAN<br/>192.168.11.0/24]
    V110[VLAN 110: BESU-VAL<br/>10.110.0.0/24]
    V111[VLAN 111: BESU-SEN<br/>10.111.0.0/24]
    V112[VLAN 112: BESU-RPC<br/>10.112.0.0/24]
    V132[VLAN 132: CCIP-COMMIT<br/>10.132.0.0/24]
    V133[VLAN 133: CCIP-EXEC<br/>10.133.0.0/24]
    V134[VLAN 134: CCIP-RMN<br/>10.134.0.0/24]
    V11 --> V110
    V11 --> V111
    V11 --> V112
    V11 --> V132
    V11 --> V133
    V11 --> V134

See VLAN Set (Authoritative) below for the full table.

Proxmox Cluster (Nodes)

graph LR
    ML[ml110 192.168.11.10]
    R1[r630-01 .11]
    R2[r630-02 .12]
    R3[r630-03 .13]
    R4[r630-04 .14]
    ML --- R1
    ML --- R2
    R1 --- R2
    R1 --- R3
    R2 --- R4

Core Principles

  1. No public IPs on Proxmox hosts or LXCs/VMs (default)
  2. Inbound access = Cloudflare Zero Trust + cloudflared (primary)
  3. Public IPs used for:
    • ER605 WAN addressing
    • Egress NAT pools (role-based allowlisting)
    • Break-glass emergency endpoints only
  4. Segmentation by VLAN/VRF: consensus vs services vs sovereign tenants vs ops
  5. Deterministic VMID registry + IPAM that matches

1. Physical Topology & Hardware Roles

Reference: For complete physical hardware inventory including IP addresses, credentials, and detailed specifications, see PHYSICAL_HARDWARE_INVENTORY.md.

1.1 Hardware Role Assignment

Edge / Routing

  • ER605-A (Primary Edge Router)

    • WAN1: Spectrum primary with Block #1
    • WAN2: ISP #2 (failover/alternate policy)
    • Role: Active edge router, NAT pools, routing

  • ER605-B (Standby Edge Router / Alternate WAN policy)

    • Role: Standby router OR dedicated to WAN2 policies/testing
    • Note: ER605 does not support full stateful HA. This is active/standby operational redundancy, not automatic session-preserving HA.

Switching Fabric

  • ES216G-1: Core / uplinks / trunks
  • ES216G-2: Compute rack aggregation
  • ES216G-3: Mgmt + out-of-band / staging

Compute

  • ML110 Gen9: "Bootstrap & Management" node

    • IP: 192.168.11.10
    • Role: Proxmox mgmt services, Omada controller, Git, monitoring seed

  • 4× Dell R630: Proxmox compute cluster nodes

    • Resources: 512GB RAM each, 2×600GB boot, 6×250GB SSD
    • Role: Production workloads, CCIP fleet, sovereign tenants, services

2. ISP & Public IP Plan (6× /28)

Public Block #1 (Known - Spectrum)

Property Value Status
Network 76.53.10.32/28 ✅ Configured
Gateway 76.53.10.33 ✅ Active
Usable Range 76.53.10.33–76.53.10.46 ✅ In Use
Broadcast 76.53.10.47 -
UDM Pro (edge) 76.53.10.34 (replaced ER605) ✅ Active
Available IPs 13 (76.53.10.35-46, excluding .34) ✅ Available

Public Blocks #2–#6 (Reserved - To Be Configured)

Status: Blocks #2–#6 are reserved. Document actual network/gateway/usable range when assigned by provider, or keep as placeholders until CCIP/Sankofa/Sovereign egress planning is finalized. See MASTER_PLAN.md §3.1.

Block Network Gateway Usable Range Broadcast Designated Use
#2 <PUBLIC_BLOCK_2>/28 <GW2> <USABLE2> <BCAST2> CCIP Commit egress NAT pool
#3 <PUBLIC_BLOCK_3>/28 <GW3> <USABLE3> <BCAST3> CCIP Execute egress NAT pool
#4 <PUBLIC_BLOCK_4>/28 <GW4> <USABLE4> <BCAST4> RMN egress NAT pool
#5 <PUBLIC_BLOCK_5>/28 <GW5> <USABLE5> <BCAST5> Sankofa/Phoenix/PanTel service egress
#6 <PUBLIC_BLOCK_6>/28 <GW6> <USABLE6> <BCAST6> Sovereign Cloud Band tenant egress

2.1 Public IP Usage Policy (Role-based)

Public /28 Block Designated Use Why
#1 (76.53.10.32/28) Router WAN + break-glass VIPs Primary connectivity + emergency
#2 CCIP Commit egress NAT pool Allowlistable egress for source RPCs
#3 CCIP Execute egress NAT pool Allowlistable egress for destination RPCs
#4 RMN egress NAT pool Independent security-plane egress
#5 Sankofa/Phoenix/PanTel service egress Service-plane separation
#6 Sovereign Cloud Band tenant egress Per-sovereign policy control

3. Layer-2 & VLAN Orchestration Plan

3.1 VLAN Set (Authoritative)

Migration Note: Currently on flat LAN 192.168.11.0/24. This plan migrates to VLANs while keeping compatibility.

VLAN ID VLAN Name Purpose Subnet Gateway
11 MGMT-LAN Proxmox mgmt, switches mgmt, admin endpoints 192.168.11.0/24 192.168.11.1
110 BESU-VAL Validator-only network (no member access) 10.110.0.0/24 10.110.0.1
111 BESU-SEN Sentry mesh 10.111.0.0/24 10.111.0.1
112 BESU-RPC RPC / gateway tier 10.112.0.0/24 10.112.0.1
120 BLOCKSCOUT Explorer + DB 10.120.0.0/24 10.120.0.1
121 CACTI Interop middleware 10.121.0.0/24 10.121.0.1
130 CCIP-OPS Ops/admin 10.130.0.0/24 10.130.0.1
132 CCIP-COMMIT Commit-role DON 10.132.0.0/24 10.132.0.1
133 CCIP-EXEC Execute-role DON 10.133.0.0/24 10.133.0.1
134 CCIP-RMN Risk management network 10.134.0.0/24 10.134.0.1
140 FABRIC Fabric 10.140.0.0/24 10.140.0.1
141 FIREFLY FireFly 10.141.0.0/24 10.141.0.1
150 INDY Identity 10.150.0.0/24 10.150.0.1
160 SANKOFA-SVC Sankofa/Phoenix/PanTel service layer 10.160.0.0/22 10.160.0.1
200 PHX-SOV-SMOM Sovereign tenant 10.200.0.0/20 10.200.0.1
201 PHX-SOV-ICCC Sovereign tenant 10.201.0.0/20 10.201.0.1
202 PHX-SOV-DBIS Sovereign tenant 10.202.0.0/20 10.202.0.1
203 PHX-SOV-AR Absolute Realms tenant 10.203.0.0/20 10.203.0.1

3.2 Switching Configuration (ES216G)

  • ES216G-1: Core (all VLAN trunks to ES216G-2/3 + ER605-A)
  • ES216G-2: Compute (trunks to R630s + ML110)
  • ES216G-3: Mgmt/OOB (mgmt access ports, staging, out-of-band)

All Proxmox uplinks should be 802.1Q trunk ports.

4. Routing, NAT, and Egress Segmentation (ER605)

4.1 Dual Router Roles

  • ER605-A: Active edge router (WAN1 = Spectrum primary with Block #1)
  • ER605-B: Standby router OR dedicated to WAN2 policies/testing (no inbound services)

4.2 NAT Policies (Critical)

Inbound NAT

  • Default: none
  • Break-glass only (optional):
    • Jumpbox/SSH (single port, IP allowlist, Cloudflare Access preferred)
    • Proxmox admin should remain LAN-only

Outbound NAT (Role-based Pools Using /28 Blocks)

Private Subnet Role Egress NAT Pool Public Block
10.132.0.0/24 CCIP Commit Block #2 <PUBLIC_BLOCK_2>/28 #2
10.133.0.0/24 CCIP Execute Block #3 <PUBLIC_BLOCK_3>/28 #3
10.134.0.0/24 RMN Block #4 <PUBLIC_BLOCK_4>/28 #4
10.160.0.0/22 Sankofa/Phoenix/PanTel Block #5 <PUBLIC_BLOCK_5>/28 #5
10.200.0.0/20–10.203.0.0/20 Sovereign tenants Block #6 <PUBLIC_BLOCK_6>/28 #6
192.168.11.0/24 Mgmt Block #1 (or none; tightly restricted) #1

This yields provable separation, allowlisting, and incident scoping.

5. Proxmox Cluster Orchestration

5.1 Node Layout

  • ml110 (192.168.11.10): mgmt + seed services + initial automation runner
  • r630-01..04: production compute

5.2 Proxmox Networking (per host)

  • vmbr0: VLAN-aware bridge
    • Native VLAN: 11 (MGMT)
    • Tagged VLANs: 110,111,112,120,121,130,132,133,134,140,141,150,160,200–203
  • Proxmox host IP remains on VLAN 11 only.

5.3 Storage Orchestration (R630)

Hardware:

  • 2×600GB boot (mirror recommended)
  • 6×250GB SSD

Recommended:

  • Boot drives: ZFS mirror or hardware RAID1
  • Data SSDs: ZFS pool (striped mirrors if you can pair, or RAIDZ1/2 depending on risk tolerance)
  • High-write workloads (logs/metrics/indexers) on dedicated dataset with quotas

6. Public Edge: Fastly or Direct to NPMplus

6.1 Fastly or Direct to NPMplus (Primary Public Path)

Public ingress is Fastly (Option A) or DNS direct to 76.53.10.36 (Option C). Both flow through UDM Pro port forward to NPMplus (VMID 10233 at 192.168.11.167). Cloudflare Tunnel is deprecated for public access (502 errors); Cloudflare DNS is retained for all public hostnames.

  • Flow: Internet → Cloudflare DNS → Fastly or 76.53.10.36 → UDM Pro (76.53.10.36:80/443) → NPMplus → internal services (Blockscout, RPC, DBIS, MIM4U, etc.).
  • Pre-requisite: Verify 76.53.10.36:80 and :443 are open from the internet; see 05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md. If closed (e.g. Spectrum filtering), use Option B (tunnel or VPS origin).
  • Keep Proxmox UI LAN-only; if needed, publish via Cloudflare Access or VPN with strict posture/MFA.

7. Complete VMID and Network Allocation Table

VMID Range Domain / Subdomain VLAN Name VLAN ID Private Subnet (GW .1) Public IP (Edge VIP / NAT)
EDGE UDM Pro (replaced ER605) WAN — — 76.53.10.34 (edge)
EDGE Spectrum ISP Gateway — — — 76.53.10.33 (ISP gateway)
1000–1499 Besu – Validators BESU-VAL 110 10.110.0.0/24 None (no inbound; tunnel/VPN only)
1500–2499 Besu – Sentries BESU-SEN 111 10.111.0.0/24 None (optional later via NAT pool)
2500–3499 Besu – RPC / Gateways BESU-RPC 112 10.112.0.0/24 Via NPMplus (Fastly or direct to 76.53.10.36); Alltra/HYBX via 76.53.10.38 or 76.53.10.42)
3500–4299 Besu – Archive/Snapshots/Mirrors/Telemetry BESU-INFRA 113 10.113.0.0/24 None
4300–4999 Besu – Reserved expansion BESU-RES 114 10.114.0.0/24 None
5000–5099 Blockscout – Explorer/Indexing BLOCKSCOUT 120 10.120.0.0/24 Via NPMplus (Fastly or direct to 76.53.10.36)
5200–5299 Cacti – Interop middleware CACTI 121 10.121.0.0/24 None (publish via NPMplus/Fastly if needed)
5400–5401 CCIP – Ops/Admin CCIP-OPS 130 10.130.0.0/24 None (Cloudflare Access / VPN only)
5402–5403 CCIP – Monitoring/Telemetry CCIP-MON 131 10.131.0.0/24 None (optionally publish dashboards via Cloudflare Access)
5410–5425 CCIP – Commit-role oracle nodes (16) CCIP-COMMIT 132 10.132.0.0/24 Egress NAT: Block #2
5440–5455 CCIP – Execute-role oracle nodes (16) CCIP-EXEC 133 10.133.0.0/24 Egress NAT: Block #3
5470–5476 CCIP – RMN nodes (7) CCIP-RMN 134 10.134.0.0/24 Egress NAT: Block #4
5480–5599 CCIP – Reserved expansion CCIP-RES 135 10.135.0.0/24 None
6000–6099 Fabric – Enterprise contracts FABRIC 140 10.140.0.0/24 None (publish via NPMplus/Fastly if required)
6200–6299 FireFly – Workflow/orchestration FIREFLY 141 10.141.0.0/24 76.53.10.37 (Reserved edge VIP if ever needed; primary via NPMplus)
6400–7399 Indy – Identity layer INDY 150 10.150.0.0/24 76.53.10.39 (Reserved edge VIP for DID endpoints if required; primary via NPMplus)
10235 NPMplus Alltra/HYBX MGMT-LAN 11 192.168.11.0/24 76.53.10.38 (port forward 80/81/443); 76.53.10.42 designated; see NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md)
7800–8999 Sankofa / Phoenix / PanTel – Service + Cloud + Telecom SANKOFA-SVC 160 10.160.0.0/22 Egress NAT: Block #5
10000–10999 Phoenix Sovereign Cloud Band – SMOM tenant PHX-SOV-SMOM 200 10.200.0.0/20 Egress NAT: Block #6
11000–11999 Phoenix Sovereign Cloud Band – ICCC tenant PHX-SOV-ICCC 201 10.201.0.0/20 Egress NAT: Block #6
12000–12999 Phoenix Sovereign Cloud Band – DBIS tenant PHX-SOV-DBIS 202 10.202.0.0/20 Egress NAT: Block #6
13000–13999 Phoenix Sovereign Cloud Band – Absolute Realms tenant PHX-SOV-AR 203 10.203.0.0/20 Egress NAT: Block #6

8. Network Security Model

8.1 Access Patterns

  1. No Public Access (Tunnel/VPN Only)

    • Besu Validators (VLAN 110)
    • Besu Archive/Infrastructure (VLAN 113)
    • CCIP Ops/Admin (VLAN 130)
    • CCIP Monitoring (VLAN 131)

  2. Fastly or Direct to NPMplus (Primary)

    • All public services route through NPMplus (VMID 10233) at 192.168.11.167
    • Public origin: 76.53.10.36 (UDM Pro port forwarding to NPMplus)
    • Blockscout (VLAN 120), Besu RPC (VLAN 112), FireFly (VLAN 141), Indy (VLAN 150), Sankofa/Phoenix/PanTel (VLAN 160) - Via NPMplus
    • DNS: Cloudflare. Edge: Fastly (Option A) or direct to 76.53.10.36 (Option C). Tunnel deprecated for public ingress.

  3. Role-Based Egress NAT (Allowlistable)

    • CCIP Commit (VLAN 132) → Block #2
    • CCIP Execute (VLAN 133) → Block #3
    • RMN (VLAN 134) → Block #4
    • Sankofa/Phoenix/PanTel (VLAN 160) → Block #5
    • Sovereign tenants (VLAN 200-203) → Block #6

  4. Cloudflare Access / VPN Only

    • CCIP Ops/Admin (VLAN 130)
    • CCIP Monitoring (VLAN 131) - Optional dashboard publishing

9. Implementation Notes

9.1 Gateway Configuration

  • All private subnets use .1 as the gateway address
  • Example: VLAN 110 uses 10.110.0.1 as gateway
  • VLAN 11 (MGMT) uses 192.168.11.1 (legacy compatibility)

9.2 Subnet Sizing

  • /24 subnets: Standard service VLANs (256 addresses)
  • /22 subnet: Sankofa/Phoenix/PanTel (1024 addresses)
  • /20 subnets: Phoenix Sovereign Cloud Bands (4096 addresses each)

9.3 IP Address Allocation

  • Private IPs:
    • VLAN 11: 192.168.11.0/24 (legacy mgmt)
    • All other VLANs: 10.x.0.0/24 or /20 or /22 (VLAN ID maps to second octet)
  • Public IPs: 6× /28 blocks with role-based NAT pools
  • All public access routes through NPMplus (Fastly or direct to 76.53.10.36) for security and stability

9.4 VLAN Tagging

  • All VLANs are tagged on the Proxmox bridge
  • Ensure Proxmox bridge is configured for VLAN-aware mode
  • Physical switch must support VLAN tagging (802.1Q)

10. Configuration Files

This architecture should be reflected in:

  • config/network.conf - Network configuration
  • config/proxmox.conf - VMID ranges
  • Proxmox bridge configuration (VLAN-aware mode)
  • ER605 router configuration (NAT pools, routing)
  • Fastly or direct-to-NPMplus configuration (see 05-network routing docs)
  • ES216G switch configuration (VLAN trunks)

11. References

Related Documentation

Architecture Documents

Configuration Documents

Deployment Documents

Document Status: Complete (v2.0)
Maintained By: Infrastructure Team
Review Cycle: Quarterly
Next Update: After public blocks #2-6 are assigned

Change Log

Version 2.0 (2025-01-20)

  • Added network topology Mermaid diagram
  • Added VLAN architecture Mermaid diagram
  • Added ASCII art network topology
  • Enhanced public IP block matrix with status indicators
  • Added breadcrumb navigation
  • Added status indicators

Version 1.0 (2024-12-15)

  • Initial version
  • Basic network architecture documentation
Reference in New Issue View Git Blame Copy Permalink