Update documentation structure and enhance .gitignore

- Added generated index files and report directories to .gitignore to prevent unnecessary tracking of transient files. - Updated README links to reflect new documentation paths for better navigation. - Improved documentation organization by ensuring all links point to the correct locations, enhancing user experience and accessibility.
2025-12-12 21:18:55 -08:00
parent 664707d912
commit fe0365757a
106 changed files with 4666 additions and 2294 deletions
--- a/docs/deployment/DEPLOYMENT_REQUIREMENTS.md
+++ b/docs/deployment/DEPLOYMENT_REQUIREMENTS.md
@@ -0,0 +1,646 @@
+# Sankofa Phoenix - Deployment Requirements
+
+## Overview
+
+This document outlines all requirements needed to deploy **Sankofa** (the ecosystem) and **Sankofa Phoenix** (the sovereign cloud platform). This includes infrastructure, software, network, security, and operational requirements.
+
+---
+
+## 1. Infrastructure Requirements
+
+### 1.1 Edge Sites (Current Implementation)
+
+**Proxmox VE Infrastructure:**
+- ✅ **Proxmox VE 8+** installed on physical hosts
+- ✅ **2+ Proxmox nodes** per site (for redundancy)
+- ✅ **Network bridge** configured (vmbr0)
+- ✅ **Storage pools** configured (local-lvm, ceph-fs, ceph-rbd)
+- ✅ **OS Images** available (ubuntu-22.04-cloud.img)
+
+**Current Status:**
+- Site 1 (ml110-01): 192.168.11.10 - Operational ✅
+- Site 2 (r630-01): 192.168.11.11 - Operational ✅
+
+**Resource Requirements (SMOM-DBIS-138):**
+- **Total VMs**: 18 (16 application + 2 infrastructure)
+- **Total CPU**: 72 cores
+- **Total RAM**: 140 GiB
+- **Total Disk**: 278 GiB
+
+### 1.2 Kubernetes Control Plane
+
+**Requirements:**
+- **Kubernetes v1.24+** cluster
+- **3 master nodes** minimum (for HA)
+- **5+ worker nodes** (for production workloads)
+- **Container runtime**: containerd or CRI-O
+- **CNI plugin**: Calico, Flannel, or Cilium
+- **Storage class**: Dynamic provisioning (local-path, NFS, or Ceph)
+
+**Control Plane Components:**
+- **Crossplane**: Infrastructure as Code (Proxmox provider)
+- **ArgoCD**: GitOps deployment
+- **Keycloak**: Identity and access management
+- **Prometheus/Grafana**: Monitoring and observability
+- **Loki**: Log aggregation
+- **Vault**: Secrets management (optional)
+
+### 1.3 Database Infrastructure
+
+**PostgreSQL Requirements:**
+- **PostgreSQL 14+** (recommended: 15+)
+- **High availability**: Primary + replicas
+- **Storage**: NVMe SSD recommended (2TB+ per node)
+- **RAM**: 64GB+ per node
+- **Backup**: Automated daily backups
+
+**Database Schema:**
+- 26 migrations including:
+  - Multi-tenancy tables
+  - Billing and usage tracking
+  - MFA and RBAC
+  - Blockchain integration
+  - Audit logging
+
+### 1.4 Blockchain Infrastructure (Future)
+
+**Hyperledger Besu Validators:**
+- **3-5 validator nodes** per core datacenter
+- **CPU**: AMD EPYC 7763 (64 cores) or Intel Xeon Platinum 8380 (40 cores)
+- **RAM**: 128GB DDR4 ECC
+- **Storage**: 2x 4TB NVMe SSD (RAID 1) for blockchain state
+- **Network**: 2x 25GbE network adapters
+- **HSM**: Hardware Security Module for key storage
+
+**Read Replica Nodes:**
+- **2-3 nodes** per regional datacenter
+- **CPU**: AMD EPYC 7543 (32 cores) or Intel Xeon Gold 6338 (32 cores)
+- **RAM**: 64GB DDR4 ECC
+- **Storage**: 2x 2TB NVMe SSD (RAID 1)
+
+---
+
+## 2. Software Requirements
+
+### 2.1 Development Tools
+
+**Required:**
+- **Node.js 18+** (for frontend, API, portal)
+- **pnpm** (recommended) or npm/yarn
+- **Go 1.21+** (for Crossplane provider)
+- **Docker** (for local development and containerization)
+- **Git** (version control)
+
+**Optional:**
+- **kubectl** (v1.24+) - Kubernetes CLI
+- **helm** (v3.0+) - Kubernetes package manager
+- **docker-compose** - Local development
+
+### 2.2 Application Components
+
+**Frontend (Next.js):**
+- Next.js 14+
+- React + TypeScript
+- TailwindCSS + shadcn/ui
+- TanStack Query
+
+**Backend:**
+- GraphQL API (Apollo Server + Fastify)
+- PostgreSQL 14+
+- WebSocket support
+- Real-time subscriptions
+
+**Portal:**
+- Next.js portal application
+- Keycloak OIDC integration
+- Role-based dashboards
+
+**Infrastructure:**
+- Crossplane provider for Proxmox
+- Kubernetes custom resources (ProxmoxVM)
+- GitOps with ArgoCD
+
+### 2.3 Monitoring and Observability
+
+**Required:**
+- **Prometheus**: Metrics collection
+- **Grafana**: Dashboards and visualization
+- **Loki**: Log aggregation
+- **Alertmanager**: Alert routing
+
+**Optional:**
+- **Jaeger**: Distributed tracing
+- **Kiali**: Service mesh visualization
+
+---
+
+## 3. Network Requirements
+
+### 3.1 Edge Sites (Current)
+
+**Network Configuration:**
+- **Network bridge**: vmbr0
+- **IP range**: 192.168.11.0/24
+- **Gateway**: Configured
+- **DNS**: Configured
+
+**Connectivity:**
+- **Cloudflare Tunnel**: Outbound-only secure connections
+- **Nginx Proxy**: SSL/TLS termination and routing
+- **Internet**: High-speed with redundancy
+
+### 3.2 Cloudflare Integration
+
+**Required:**
+- **Cloudflare account** with Zero Trust
+- **Cloudflare Tunnel** configured
+- **DNS records** configured
+- **Access policies** configured
+- **SSL/TLS certificates** (managed by Cloudflare)
+
+**Tunnel Configuration:**
+- Tunnel credentials JSON file
+- Ingress rules configured
+- Health monitoring enabled
+
+### 3.3 Inter-Datacenter Links (Future)
+
+**Core to Core:**
+- **Bandwidth**: 100Gbps+ per link
+- **Redundancy**: Multiple redundant paths
+- **Type**: Dark fiber or high-bandwidth leased lines
+
+**Core to Regional:**
+- **Bandwidth**: 10-40Gbps per link
+- **Redundancy**: Redundant paths
+- **Type**: Leased lines or MPLS
+
+**Regional to Edge:**
+- **Bandwidth**: 1-10Gbps per link
+- **Redundancy**: Internet with redundancy
+- **Type**: Internet connectivity with Cloudflare Tunnels
+
+---
+
+## 4. Security Requirements
+
+### 4.1 Identity and Access Management
+
+**Keycloak:**
+- **Keycloak 20+** deployed
+- **OIDC clients** configured:
+  - `sankofa-api` (backend API)
+  - `portal-client` (portal application)
+- **Realms** configured (multi-tenant support)
+- **MFA** enabled (TOTP, FIDO2, SMS, Email)
+- **User federation** configured (optional)
+
+**Access Control:**
+- **RBAC**: Role-based access control
+- **Tenant isolation**: Multi-tenant data isolation
+- **API authentication**: JWT tokens
+- **Session management**: Secure session handling
+
+### 4.2 Network Security
+
+**Firewalls:**
+- **Next-generation firewalls** (Palo Alto, Fortinet, Check Point)
+- **Access policies** configured
+- **Intrusion detection/prevention** (IDS/IPS)
+- **DDoS protection** (Cloudflare)
+
+**Network Segmentation:**
+- **VLANs** for different tiers
+- **Network policies** in Kubernetes
+- **Service mesh** (optional: Istio, Linkerd)
+
+### 4.3 Application Security
+
+**Security Features:**
+- **Rate limiting**: 100 req/min per IP, 1000 req/hour per user
+- **Security headers**: CSP, HSTS, X-Frame-Options
+- **Input sanitization**: Body sanitization middleware
+- **Encryption**: TLS 1.2+ for all connections
+- **Secrets management**: Kubernetes secrets or Vault
+
+**Audit Logging:**
+- **Comprehensive audit trail** for all operations
+- **Log retention** policy configured
+- **Compliance** logging (GDPR, SOC 2, ISO 27001)
+
+### 4.4 Blockchain Security
+
+**Key Management:**
+- **HSM**: Hardware Security Module for validator keys
+- **Key rotation**: Automated key rotation
+- **Multi-signature**: Multi-party governance
+
+**Network Security:**
+- **Private P2P network**: Encrypted peer-to-peer connections
+- **Network overlay**: VPN or dedicated network segment
+- **Consensus communication**: Secure channels for validators
+
+---
+
+## 5. Environment Configuration
+
+### 5.1 Environment Variables
+
+**API (.env):**
+```env
+DB_HOST=postgres
+DB_PORT=5432
+DB_NAME=sankofa
+DB_USER=postgres
+DB_PASSWORD=your-password
+JWT_SECRET=your-jwt-secret
+
+# Sovereign Identity (Keycloak)
+KEYCLOAK_URL=http://keycloak:8080
+KEYCLOAK_REALM=master
+KEYCLOAK_CLIENT_ID=sankofa-api
+KEYCLOAK_CLIENT_SECRET=your-keycloak-client-secret
+KEYCLOAK_MULTI_REALM=true
+
+# Multi-Tenancy
+ENABLE_MULTI_TENANT=true
+DEFAULT_TENANT_ID=
+BLOCKCHAIN_IDENTITY_ENABLED=true
+
+# Billing
+BILLING_GRANULARITY=SECOND
+BLOCKCHAIN_BILLING_ENABLED=true
+
+# Blockchain
+BLOCKCHAIN_RPC_URL=http://besu:8545
+RESOURCE_PROVISIONING_CONTRACT_ADDRESS=0x...
+```
+
+**Frontend (.env.local):**
+```env
+NEXT_PUBLIC_GRAPHQL_ENDPOINT=http://api:4000/graphql
+NEXT_PUBLIC_GRAPHQL_WS_ENDPOINT=ws://api:4000/graphql-ws
+NEXT_PUBLIC_APP_URL=http://localhost:3000
+NODE_ENV=development
+```
+
+**Portal (.env.local):**
+```env
+KEYCLOAK_URL=https://keycloak.sankofa.nexus
+KEYCLOAK_REALM=sankofa
+KEYCLOAK_CLIENT_ID=portal-client
+KEYCLOAK_CLIENT_SECRET=your-secret
+NEXT_PUBLIC_CROSSPLANE_API=https://crossplane.sankofa.nexus
+NEXT_PUBLIC_ARGOCD_URL=https://argocd.sankofa.nexus
+NEXT_PUBLIC_GRAFANA_URL=https://grafana.sankofa.nexus
+NEXT_PUBLIC_LOKI_URL=https://loki.sankofa.nexus:3100
+```
+
+**Proxmox Provider:**
+```env
+PROXMOX_HOST=192.168.11.10
+PROXMOX_USER=root@pam
+PROXMOX_PASS=your-password
+# OR
+PROXMOX_TOKEN=your-api-token
+```
+
+### 5.2 Kubernetes Secrets
+
+**Required Secrets:**
+- Database credentials
+- Keycloak client secrets
+- JWT secrets
+- Proxmox API credentials
+- Cloudflare tunnel credentials
+- SSL/TLS certificates (if not using Cloudflare)
+
+---
+
+## 6. Deployment Steps
+
+### 6.1 Prerequisites Checklist
+
+- [ ] Kubernetes cluster deployed and operational
+- [ ] PostgreSQL database deployed and accessible
+- [ ] Keycloak deployed and configured
+- [ ] Proxmox nodes accessible and configured
+- [ ] Cloudflare account and tunnel configured
+- [ ] Network connectivity verified
+- [ ] DNS records configured
+- [ ] SSL/TLS certificates configured
+
+### 6.2 Database Setup
+
+```bash
+# 1. Create database
+createdb sankofa
+
+# 2. Run migrations (26 migrations)
+cd api
+npm run db:migrate
+
+# 3. Verify migrations
+psql -d sankofa -c "\dt"
+
+# 4. Seed initial data (optional)
+npm run db:seed
+```
+
+### 6.3 Kubernetes Deployment
+
+```bash
+# 1. Create namespaces
+kubectl create namespace sankofa
+kubectl create namespace crossplane-system
+kubectl create namespace monitoring
+
+# 2. Deploy Crossplane
+kubectl apply -f gitops/apps/crossplane/
+
+# 3. Deploy Proxmox Provider
+kubectl apply -f crossplane-provider-proxmox/config/
+
+# 4. Deploy ArgoCD
+kubectl apply -f gitops/apps/argocd/
+
+# 5. Deploy Keycloak
+kubectl apply -f gitops/apps/keycloak/
+
+# 6. Deploy API
+kubectl apply -f gitops/apps/api/
+
+# 7. Deploy Frontend
+kubectl apply -f gitops/apps/frontend/
+
+# 8. Deploy Portal
+kubectl apply -f gitops/apps/portal/
+
+# 9. Deploy Monitoring
+kubectl apply -f gitops/apps/monitoring/
+```
+
+### 6.4 Proxmox VM Deployment
+
+```bash
+# 1. Deploy infrastructure VMs first
+kubectl apply -f examples/production/nginx-proxy-vm.yaml
+kubectl apply -f examples/production/cloudflare-tunnel-vm.yaml
+
+# 2. Deploy application VMs
+kubectl apply -f examples/production/smom-dbis-138/
+
+# 3. Monitor deployment
+kubectl get proxmoxvm -A -w
+
+# 4. Check controller logs
+kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox --tail=50 -f
+```
+
+### 6.5 GitOps Setup (ArgoCD)
+
+```bash
+# 1. Apply ArgoCD application
+kubectl apply -f gitops/apps/argocd/application.yaml
+
+# 2. Sync application
+argocd app sync sankofa-phoenix
+
+# 3. Verify sync status
+argocd app get sankofa-phoenix
+```
+
+### 6.6 Multi-Tenancy Setup
+
+```bash
+# 1. Create system tenant (via GraphQL)
+curl -X POST http://api.sankofa.nexus/graphql \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer <admin-token>" \
+  -d '{
+    "query": "mutation { createTenant(input: { name: \"system\", tier: SOVEREIGN }) { id name billingAccountId } }"
+  }'
+
+# 2. Assign admin user to system tenant
+curl -X POST http://api.sankofa.nexus/graphql \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer <admin-token>" \
+  -d '{
+    "query": "mutation { addUserToTenant(tenantId: \"<system-tenant-id>\", userId: \"<admin-user-id>\", role: TENANT_OWNER) }"
+  }'
+```
+
+---
+
+## 7. Verification and Testing
+
+### 7.1 Health Checks
+
+```bash
+# API health
+curl http://api.sankofa.nexus/health
+
+# Frontend
+curl http://frontend.sankofa.nexus
+
+# Portal
+curl http://portal.sankofa.nexus
+
+# Keycloak health
+curl http://keycloak.sankofa.nexus/health
+
+# Proxmox VMs
+kubectl get proxmoxvm -A
+```
+
+### 7.2 Smoke Tests
+
+```bash
+# Run smoke tests
+./scripts/smoke-tests.sh
+```
+
+### 7.3 Performance Testing
+
+```bash
+# Load testing
+./scripts/performance-test.sh
+
+# k6 load test
+k6 run scripts/k6-load-test.js
+```
+
+---
+
+## 8. Operational Requirements
+
+### 8.1 Monitoring and Alerting
+
+**Required:**
+- Prometheus metrics collection
+- Grafana dashboards
+- Alertmanager rules
+- Notification channels (email, Slack, PagerDuty)
+
+**Key Metrics:**
+- API response times
+- Database query performance
+- VM resource utilization
+- Blockchain network health
+- Service availability
+
+### 8.2 Backup and Disaster Recovery
+
+**Database Backups:**
+- Daily automated backups
+- Retention policy: 30 days minimum
+- Off-site backup storage
+- Backup verification scripts
+
+**VM Backups:**
+- Proxmox backup schedules
+- Snapshot management
+- Disaster recovery procedures
+
+### 8.3 Support and Operations
+
+**Required:**
+- 24/7 on-call rotation
+- Incident response procedures
+- Runbooks for common issues
+- Escalation procedures
+- Support team training
+
+---
+
+## 9. Compliance and Governance
+
+### 9.1 Compliance Requirements
+
+**Data Protection:**
+- GDPR compliance (EU)
+- Data retention policies
+- Privacy policy published
+- Terms of service published
+
+**Security Standards:**
+- SOC 2 Type II (if applicable)
+- ISO 27001 (if applicable)
+- Security audit procedures
+- Penetration testing
+
+### 9.2 Governance
+
+**Multi-Tenancy:**
+- Tenant isolation verified
+- Resource quotas enforced
+- Billing accuracy verified
+- Audit logging enabled
+
+**Blockchain Governance:**
+- Multi-party governance nodes
+- Smart contract upgrade procedures
+- Network upgrade procedures
+
+---
+
+## 10. Cost Estimates
+
+### 10.1 Infrastructure Costs
+
+**Edge Sites (Current):**
+- Proxmox hardware: $10K-$50K per site
+- Network equipment: $5K-$20K per site
+- Power and cooling: $1K-$5K per year per site
+
+**Kubernetes Cluster:**
+- Control plane: $500-$2K per month
+- Worker nodes: $1K-$5K per month
+- Storage: $200-$1K per month
+
+**Database:**
+- PostgreSQL cluster: $500-$2K per month
+- Backup storage: $100-$500 per month
+
+### 10.2 Cloudflare Costs
+
+**Zero Trust:**
+- Free tier: Up to 50 users
+- Paid tier: $7 per user per month
+
+**Tunnels:**
+- Free: Unlimited tunnels
+- Paid: Additional features
+
+**Bandwidth:**
+- Included in Zero Trust plan
+- Additional bandwidth: $0.10-$0.50 per GB
+
+### 10.3 Operational Costs
+
+**Personnel:**
+- DevOps engineers: $100K-$200K per year
+- SRE engineers: $120K-$250K per year
+- Support staff: $50K-$100K per year
+
+**Software Licenses:**
+- Most components are open source
+- Optional commercial support: $10K-$100K per year
+
+---
+
+## 11. Quick Start Summary
+
+### Minimum Viable Deployment
+
+**For Development/Testing:**
+1. Single Kubernetes cluster (3 nodes minimum)
+2. PostgreSQL database (single instance)
+3. Keycloak (single instance)
+4. 2 Proxmox nodes
+5. Cloudflare account (free tier)
+6. All application components deployed
+
+**For Production:**
+1. High-availability Kubernetes cluster (3 masters + 5 workers)
+2. PostgreSQL cluster (primary + replicas)
+3. Keycloak cluster (HA)
+4. Multiple Proxmox sites (2+ sites)
+5. Cloudflare Zero Trust (paid tier)
+6. Monitoring and alerting configured
+7. Backup and disaster recovery configured
+8. Security hardening completed
+
+---
+
+## 12. Documentation References
+
+- **[Production Deployment Ready](./PRODUCTION_DEPLOYMENT_READY.md)** - Current deployment status
+- **[Launch Checklist](./status/LAUNCH_CHECKLIST.md)** - Pre-launch verification
+- **[Deployment Guide](./DEPLOYMENT.md)** - Detailed deployment instructions
+- **[Deployment Plan](./deployment_plan.md)** - Phased rollout plan
+- **[System Architecture](./system_architecture.md)** - Overall architecture
+- **[Hardware BOM](./hardware_bom.md)** - Hardware specifications
+- **[VM Specifications](vm/VM_SPECIFICATIONS.md)** - Complete VM specifications and deployment patterns
+- **[VM Creation Procedure](vm/VM_CREATION_PROCEDURE.md)** - Step-by-step VM deployment guide
+
+---
+
+## 13. Next Steps
+
+1. **Review Prerequisites**: Verify all infrastructure and software requirements
+2. **Configure Environment**: Set up environment variables and secrets
+3. **Deploy Database**: Run migrations and seed data
+4. **Deploy Kubernetes**: Deploy control plane components
+5. **Deploy Applications**: Deploy API, frontend, and portal
+6. **Deploy VMs**: Deploy Proxmox VMs via Crossplane
+7. **Configure Monitoring**: Set up Prometheus, Grafana, and Loki
+8. **Verify Deployment**: Run health checks and smoke tests
+9. **Configure Multi-Tenancy**: Set up initial tenants
+10. **Go Live**: Enable production traffic
+
+---
+
+**Last Updated**: 2025-01-XX  
+**Status**: Comprehensive deployment requirements documented
+