Sankofa/docs/DEPLOYMENT_REQUIREMENTS.md

Sankofa Phoenix - Deployment Requirements

Overview

This document outlines all requirements needed to deploy Sankofa (the ecosystem) and Sankofa Phoenix (the sovereign cloud platform). This includes infrastructure, software, network, security, and operational requirements.

---

1. Infrastructure Requirements

1.1 Edge Sites (Current Implementation)

Proxmox VE Infrastructure:

• ✅ Proxmox VE 8+ installed on physical hosts
• ✅ 2+ Proxmox nodes per site (for redundancy)
• ✅ Network bridge configured (vmbr0)
• ✅ Storage pools configured (local-lvm, ceph-fs, ceph-rbd)
• ✅ OS Images available (ubuntu-22.04-cloud.img)

Current Status:

• Site 1 (ml110-01): 192.168.11.10 - Operational ✅
• Site 2 (r630-01): 192.168.11.11 - Operational ✅

Resource Requirements (SMOM-DBIS-138):

• Total VMs: 18 (16 application + 2 infrastructure)
• Total CPU: 72 cores
• Total RAM: 140 GiB
• Total Disk: 278 GiB

1.2 Kubernetes Control Plane

Requirements:

• Kubernetes v1.24+ cluster
• 3 master nodes minimum (for HA)
• 5+ worker nodes (for production workloads)
• Container runtime: containerd or CRI-O
• CNI plugin: Calico, Flannel, or Cilium
• Storage class: Dynamic provisioning (local-path, NFS, or Ceph)

Control Plane Components:

• Crossplane: Infrastructure as Code (Proxmox provider)
• ArgoCD: GitOps deployment
• Keycloak: Identity and access management
• Prometheus/Grafana: Monitoring and observability
• Loki: Log aggregation
• Vault: Secrets management (optional)

1.3 Database Infrastructure

PostgreSQL Requirements:

• PostgreSQL 14+ (recommended: 15+)
• High availability: Primary + replicas
• Storage: NVMe SSD recommended (2TB+ per node)
• RAM: 64GB+ per node
• Backup: Automated daily backups

Database Schema:

• 26 migrations including:
  • Multi-tenancy tables
  • Billing and usage tracking
  • MFA and RBAC
  • Blockchain integration
  • Audit logging

1.4 Blockchain Infrastructure (Future)

Hyperledger Besu Validators:

• 3-5 validator nodes per core datacenter
• CPU: AMD EPYC 7763 (64 cores) or Intel Xeon Platinum 8380 (40 cores)
• RAM: 128GB DDR4 ECC
• Storage: 2x 4TB NVMe SSD (RAID 1) for blockchain state
• Network: 2x 25GbE network adapters
• HSM: Hardware Security Module for key storage

Read Replica Nodes:

• 2-3 nodes per regional datacenter
• CPU: AMD EPYC 7543 (32 cores) or Intel Xeon Gold 6338 (32 cores)
• RAM: 64GB DDR4 ECC
• Storage: 2x 2TB NVMe SSD (RAID 1)

---

2. Software Requirements

2.1 Development Tools

Required:

• Node.js 18+ (for frontend, API, portal)
• pnpm (recommended) or npm/yarn
• Go 1.21+ (for Crossplane provider)
• Docker (for local development and containerization)
• Git (version control)

Optional:

• kubectl (v1.24+) - Kubernetes CLI
• helm (v3.0+) - Kubernetes package manager
• docker-compose - Local development

2.2 Application Components

Frontend (Next.js):

• Next.js 14+
• React + TypeScript
• TailwindCSS + shadcn/ui
• TanStack Query

Backend:

• GraphQL API (Apollo Server + Fastify)
• PostgreSQL 14+
• WebSocket support
• Real-time subscriptions

Portal:

• Next.js portal application
• Keycloak OIDC integration
• Role-based dashboards

Infrastructure:

• Crossplane provider for Proxmox
• Kubernetes custom resources (ProxmoxVM)
• GitOps with ArgoCD

2.3 Monitoring and Observability

Required:

• Prometheus: Metrics collection
• Grafana: Dashboards and visualization
• Loki: Log aggregation
• Alertmanager: Alert routing

Optional:

• Jaeger: Distributed tracing
• Kiali: Service mesh visualization

---

3. Network Requirements

3.1 Edge Sites (Current)

Network Configuration:

• Network bridge: vmbr0
• IP range: 192.168.11.0/24
• Gateway: Configured
• DNS: Configured

Connectivity:

• Cloudflare Tunnel: Outbound-only secure connections
• Nginx Proxy: SSL/TLS termination and routing
• Internet: High-speed with redundancy

3.2 Cloudflare Integration

Required:

• Cloudflare account with Zero Trust
• Cloudflare Tunnel configured
• DNS records configured
• Access policies configured
• SSL/TLS certificates (managed by Cloudflare)

Tunnel Configuration:

• Tunnel credentials JSON file
• Ingress rules configured
• Health monitoring enabled

3.3 Inter-Datacenter Links (Future)

Core to Core:

• Bandwidth: 100Gbps+ per link
• Redundancy: Multiple redundant paths
• Type: Dark fiber or high-bandwidth leased lines

Core to Regional:

• Bandwidth: 10-40Gbps per link
• Redundancy: Redundant paths
• Type: Leased lines or MPLS

Regional to Edge:

• Bandwidth: 1-10Gbps per link
• Redundancy: Internet with redundancy
• Type: Internet connectivity with Cloudflare Tunnels

---

4. Security Requirements

4.1 Identity and Access Management

Keycloak:

• Keycloak 20+ deployed
• OIDC clients configured:
  • sankofa-api (backend API)
  • portal-client (portal application)
• Realms configured (multi-tenant support)
• MFA enabled (TOTP, FIDO2, SMS, Email)
• User federation configured (optional)

Access Control:

• RBAC: Role-based access control
• Tenant isolation: Multi-tenant data isolation
• API authentication: JWT tokens
• Session management: Secure session handling

4.2 Network Security

Firewalls:

• Next-generation firewalls (Palo Alto, Fortinet, Check Point)
• Access policies configured
• Intrusion detection/prevention (IDS/IPS)
• DDoS protection (Cloudflare)

Network Segmentation:

• VLANs for different tiers
• Network policies in Kubernetes
• Service mesh (optional: Istio, Linkerd)

4.3 Application Security

Security Features:

• Rate limiting: 100 req/min per IP, 1000 req/hour per user
• Security headers: CSP, HSTS, X-Frame-Options
• Input sanitization: Body sanitization middleware
• Encryption: TLS 1.2+ for all connections
• Secrets management: Kubernetes secrets or Vault

Audit Logging:

• Comprehensive audit trail for all operations
• Log retention policy configured
• Compliance logging (GDPR, SOC 2, ISO 27001)

4.4 Blockchain Security

Key Management:

• HSM: Hardware Security Module for validator keys
• Key rotation: Automated key rotation
• Multi-signature: Multi-party governance

Network Security:

• Private P2P network: Encrypted peer-to-peer connections
• Network overlay: VPN or dedicated network segment
• Consensus communication: Secure channels for validators

---

5. Environment Configuration

5.1 Environment Variables

API (.env):

DB_HOST=postgres
DB_PORT=5432
DB_NAME=sankofa
DB_USER=postgres
DB_PASSWORD=your-password
JWT_SECRET=your-jwt-secret

# Sovereign Identity (Keycloak)
KEYCLOAK_URL=http://keycloak:8080
KEYCLOAK_REALM=master
KEYCLOAK_CLIENT_ID=sankofa-api
KEYCLOAK_CLIENT_SECRET=your-keycloak-client-secret
KEYCLOAK_MULTI_REALM=true

# Multi-Tenancy
ENABLE_MULTI_TENANT=true
DEFAULT_TENANT_ID=
BLOCKCHAIN_IDENTITY_ENABLED=true

# Billing
BILLING_GRANULARITY=SECOND
BLOCKCHAIN_BILLING_ENABLED=true

# Blockchain
BLOCKCHAIN_RPC_URL=http://besu:8545
RESOURCE_PROVISIONING_CONTRACT_ADDRESS=0x...

Frontend (.env.local):

NEXT_PUBLIC_GRAPHQL_ENDPOINT=http://api:4000/graphql
NEXT_PUBLIC_GRAPHQL_WS_ENDPOINT=ws://api:4000/graphql-ws
NEXT_PUBLIC_APP_URL=http://localhost:3000
NODE_ENV=development

Portal (.env.local):

KEYCLOAK_URL=https://keycloak.sankofa.nexus
KEYCLOAK_REALM=sankofa
KEYCLOAK_CLIENT_ID=portal-client
KEYCLOAK_CLIENT_SECRET=your-secret
NEXT_PUBLIC_CROSSPLANE_API=https://crossplane.sankofa.nexus
NEXT_PUBLIC_ARGOCD_URL=https://argocd.sankofa.nexus
NEXT_PUBLIC_GRAFANA_URL=https://grafana.sankofa.nexus
NEXT_PUBLIC_LOKI_URL=https://loki.sankofa.nexus:3100

Proxmox Provider:

PROXMOX_HOST=192.168.11.10
PROXMOX_USER=root@pam
PROXMOX_PASS=your-password
# OR
PROXMOX_TOKEN=your-api-token

5.2 Kubernetes Secrets

Required Secrets:

• Database credentials
• Keycloak client secrets
• JWT secrets
• Proxmox API credentials
• Cloudflare tunnel credentials
• SSL/TLS certificates (if not using Cloudflare)

---

6. Deployment Steps

6.1 Prerequisites Checklist

• Kubernetes cluster deployed and operational
• PostgreSQL database deployed and accessible
• Keycloak deployed and configured
• Proxmox nodes accessible and configured
• Cloudflare account and tunnel configured
• Network connectivity verified
• DNS records configured
• SSL/TLS certificates configured

6.2 Database Setup

# 1. Create database
createdb sankofa

# 2. Run migrations (26 migrations)
cd api
npm run db:migrate

# 3. Verify migrations
psql -d sankofa -c "\dt"

# 4. Seed initial data (optional)
npm run db:seed

6.3 Kubernetes Deployment

# 1. Create namespaces
kubectl create namespace sankofa
kubectl create namespace crossplane-system
kubectl create namespace monitoring

# 2. Deploy Crossplane
kubectl apply -f gitops/apps/crossplane/

# 3. Deploy Proxmox Provider
kubectl apply -f crossplane-provider-proxmox/config/

# 4. Deploy ArgoCD
kubectl apply -f gitops/apps/argocd/

# 5. Deploy Keycloak
kubectl apply -f gitops/apps/keycloak/

# 6. Deploy API
kubectl apply -f gitops/apps/api/

# 7. Deploy Frontend
kubectl apply -f gitops/apps/frontend/

# 8. Deploy Portal
kubectl apply -f gitops/apps/portal/

# 9. Deploy Monitoring
kubectl apply -f gitops/apps/monitoring/

6.4 Proxmox VM Deployment

# 1. Deploy infrastructure VMs first
kubectl apply -f examples/production/nginx-proxy-vm.yaml
kubectl apply -f examples/production/cloudflare-tunnel-vm.yaml

# 2. Deploy application VMs
kubectl apply -f examples/production/smom-dbis-138/

# 3. Monitor deployment
kubectl get proxmoxvm -A -w

# 4. Check controller logs
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox --tail=50 -f

6.5 GitOps Setup (ArgoCD)

# 1. Apply ArgoCD application
kubectl apply -f gitops/apps/argocd/application.yaml

# 2. Sync application
argocd app sync sankofa-phoenix

# 3. Verify sync status
argocd app get sankofa-phoenix

6.6 Multi-Tenancy Setup

# 1. Create system tenant (via GraphQL)
curl -X POST http://api.sankofa.nexus/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <admin-token>" \
  -d '{
    "query": "mutation { createTenant(input: { name: \"system\", tier: SOVEREIGN }) { id name billingAccountId } }"
  }'

# 2. Assign admin user to system tenant
curl -X POST http://api.sankofa.nexus/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <admin-token>" \
  -d '{
    "query": "mutation { addUserToTenant(tenantId: \"<system-tenant-id>\", userId: \"<admin-user-id>\", role: TENANT_OWNER) }"
  }'

---

7. Verification and Testing

7.1 Health Checks

# API health
curl http://api.sankofa.nexus/health

# Frontend
curl http://frontend.sankofa.nexus

# Portal
curl http://portal.sankofa.nexus

# Keycloak health
curl http://keycloak.sankofa.nexus/health

# Proxmox VMs
kubectl get proxmoxvm -A

7.2 Smoke Tests

# Run smoke tests
./scripts/smoke-tests.sh

7.3 Performance Testing

# Load testing
./scripts/performance-test.sh

# k6 load test
k6 run scripts/k6-load-test.js

---

8. Operational Requirements

8.1 Monitoring and Alerting

Required:

• Prometheus metrics collection
• Grafana dashboards
• Alertmanager rules
• Notification channels (email, Slack, PagerDuty)

Key Metrics:

• API response times
• Database query performance
• VM resource utilization
• Blockchain network health
• Service availability

8.2 Backup and Disaster Recovery

Database Backups:

• Daily automated backups
• Retention policy: 30 days minimum
• Off-site backup storage
• Backup verification scripts

VM Backups:

• Proxmox backup schedules
• Snapshot management
• Disaster recovery procedures

8.3 Support and Operations

Required:

• 24/7 on-call rotation
• Incident response procedures
• Runbooks for common issues
• Escalation procedures
• Support team training

---

9. Compliance and Governance

9.1 Compliance Requirements

Data Protection:

• GDPR compliance (EU)
• Data retention policies
• Privacy policy published
• Terms of service published

Security Standards:

• SOC 2 Type II (if applicable)
• ISO 27001 (if applicable)
• Security audit procedures
• Penetration testing

9.2 Governance

Multi-Tenancy:

• Tenant isolation verified
• Resource quotas enforced
• Billing accuracy verified
• Audit logging enabled

Blockchain Governance:

• Multi-party governance nodes
• Smart contract upgrade procedures
• Network upgrade procedures

---

10. Cost Estimates

10.1 Infrastructure Costs

Edge Sites (Current):

• Proxmox hardware: $10K-$50K per site
• Network equipment: $5K-$20K per site
• Power and cooling: $1K-$5K per year per site

Kubernetes Cluster:

• Control plane: $500-$2K per month
• Worker nodes: $1K-$5K per month
• Storage: $200-$1K per month

Database:

• PostgreSQL cluster: $500-$2K per month
• Backup storage: $100-$500 per month

10.2 Cloudflare Costs

Zero Trust:

• Free tier: Up to 50 users
• Paid tier: $7 per user per month

Tunnels:

• Free: Unlimited tunnels
• Paid: Additional features

Bandwidth:

• Included in Zero Trust plan
• Additional bandwidth: $0.10-$0.50 per GB

10.3 Operational Costs

Personnel:

• DevOps engineers: $100K-$200K per year
• SRE engineers: $120K-$250K per year
• Support staff: $50K-$100K per year

Software Licenses:

• Most components are open source
• Optional commercial support: $10K-$100K per year

---

11. Quick Start Summary

Minimum Viable Deployment

For Development/Testing:

1. Single Kubernetes cluster (3 nodes minimum)
2. PostgreSQL database (single instance)
3. Keycloak (single instance)
4. 2 Proxmox nodes
5. Cloudflare account (free tier)
6. All application components deployed

For Production:

1. High-availability Kubernetes cluster (3 masters + 5 workers)
2. PostgreSQL cluster (primary + replicas)
3. Keycloak cluster (HA)
4. Multiple Proxmox sites (2+ sites)
5. Cloudflare Zero Trust (paid tier)
6. Monitoring and alerting configured
7. Backup and disaster recovery configured
8. Security hardening completed

---

12. Documentation References

• Production Deployment Ready - Current deployment status
• Launch Checklist - Pre-launch verification
• Deployment Guide - Detailed deployment instructions
• Deployment Plan - Phased rollout plan
• System Architecture - Overall architecture
• Hardware BOM - Hardware specifications
• VM Deployment Plan - VM deployment guide

---

13. Next Steps

1. Review Prerequisites: Verify all infrastructure and software requirements
2. Configure Environment: Set up environment variables and secrets
3. Deploy Database: Run migrations and seed data
4. Deploy Kubernetes: Deploy control plane components
5. Deploy Applications: Deploy API, frontend, and portal
6. Deploy VMs: Deploy Proxmox VMs via Crossplane
7. Configure Monitoring: Set up Prometheus, Grafana, and Loki
8. Verify Deployment: Run health checks and smoke tests
9. Configure Multi-Tenancy: Set up initial tenants
10. Go Live: Enable production traffic

---

Last Updated: 2025-01-XX
Status: Comprehensive deployment requirements documented