Remove obsolete audit and deployment documentation files

- Deleted outdated files related to repository audit and deployment status, including AUDIT_COMPLETE.md, AUDIT_FIXES_APPLIED.md, FINAL_DEPLOYMENT_STATUS.md, and others. - Cleaned up documentation to streamline the repository and improve clarity for future maintenance. - Updated README and other relevant documentation to reflect the removal of these files.
2025-12-12 19:42:31 -08:00
parent 388ba3ba94
commit a8106e24ee
60 changed files with 841 additions and 30 deletions
--- a/docs/proxmox/reference/API_TOKENS.md
+++ b/docs/proxmox/reference/API_TOKENS.md
@@ -0,0 +1,319 @@
+# Proxmox API Token Management
+
+## Overview
+
+This document describes the management of Proxmox API tokens for the Crossplane provider. API tokens provide secure, programmatic access to Proxmox clusters without requiring password authentication.
+
+## Token Architecture
+
+### Token Structure
+
+Proxmox API tokens follow the format:
+```
+{username}@{realm}!{token-id}={token-secret}
+```
+
+Example:
+```
+root@pam!crossplane-instance-1=abc123-def456-ghi789
+```
+
+### Token Components
+
+- **Username**: User account (e.g., `root`, `service-account`)
+- **Realm**: Authentication realm (e.g., `pam`, `ldap`)
+- **Token ID**: Unique identifier for the token
+- **Token Secret**: Secret value (never log or expose)
+
+## Token Creation
+
+### Via Proxmox Web UI
+
+1. **Navigate to**: Datacenter → Permissions → API Tokens
+2. **Click**: "Add" → "API Token"
+3. **Configure**:
+   - **Token ID**: `crossplane-instance-1`
+   - **User**: `root@pam` (or dedicated service account)
+   - **Comment**: "Crossplane provider for Instance 1"
+   - **Expiration**: Set expiration date (recommended)
+   - **Privilege Separation**: Enable if using dedicated user
+4. **Save** and copy the token secret immediately
+
+### Via Proxmox CLI
+
+```bash
+# Create token via pvesh
+pvesh create /access/users/{user}/token/{token-id} \
+  --privsep 1 \
+  --expire {timestamp} \
+  --comment "Crossplane provider token"
+```
+
+### Via Proxmox API
+
+```bash
+curl -X POST "https://ml110-01.sankofa.nexus:8006/api2/json/access/users/root@pam/token/crossplane-instance-1" \
+  -H "Authorization: PVEAPIToken root@pam!existing-token=secret" \
+  -d 'privsep=1&expire=1735689600&comment=Crossplane provider'
+```
+
+## Token Permissions
+
+### Principle of Least Privilege
+
+Tokens should have **minimum required permissions**:
+
+#### Recommended Permissions
+
+**For VM Management**:
+- `VM.Allocate` - Create VMs
+- `VM.Clone` - Clone VMs
+- `VM.Config.Disk` - Manage VM disks
+- `VM.Config.Network` - Manage VM networks
+- `VM.Monitor` - Monitor VM status
+- `VM.PowerMgmt` - Start/stop VMs
+- `VM.Snapshot` - Create snapshots
+
+**For Storage Management**:
+- `Datastore.Allocate` - Allocate storage
+- `Datastore.Audit` - Audit storage
+
+**For Node Management**:
+- `Sys.Audit` - System audit
+- `Sys.Modify` - System modification (if needed)
+
+#### Full Administrator (Not Recommended)
+
+```yaml
+# ⚠️ Only for development/testing
+Permissions: Administrator
+```
+
+**Risks**:
+- Full cluster access
+- Can delete critical resources
+- Security risk if compromised
+
+### Permission Configuration
+
+```bash
+# Set token permissions via pvesh
+pvesh set /access/acl \
+  --path / \
+  --roles Administrator \
+  --users root@pam!crossplane-instance-1
+```
+
+## Token Storage
+
+### Kubernetes Secrets
+
+**Never store tokens in code or config files!**
+
+Store tokens in Kubernetes secrets:
+
+```bash
+# Create secret with token
+kubectl create secret generic proxmox-credentials \
+  --from-literal=credentials.json='{"username":"root@pam","token":"root@pam!crossplane-instance-1=abc123..."}' \
+  -n crossplane-system
+```
+
+### Secret Structure
+
+```json
+{
+  "username": "root@pam",
+  "token": "root@pam!crossplane-instance-1=abc123-def456-ghi789"
+}
+```
+
+### ProviderConfig Reference
+
+```yaml
+apiVersion: proxmox.sankofa.nexus/v1alpha1
+kind: ProviderConfig
+metadata:
+  name: proxmox-provider-config
+spec:
+  credentials:
+    source: Secret
+    secretRef:
+      name: proxmox-credentials
+      namespace: crossplane-system
+      key: credentials.json
+```
+
+## Token Rotation
+
+### Rotation Schedule
+
+- **Production**: Rotate every 90 days
+- **Staging**: Rotate every 180 days
+- **Development**: Rotate as needed
+
+### Rotation Procedure
+
+1. **Create New Token**
+   ```bash
+   # Create new token with new ID
+   pvesh create /access/users/root@pam/token/crossplane-instance-1-v2
+   ```
+
+2. **Update Kubernetes Secret**
+   ```bash
+   kubectl create secret generic proxmox-credentials \
+     --from-literal=credentials.json='{"token":"new-token"}' \
+     -n crossplane-system \
+     --dry-run=client -o yaml | kubectl apply -f -
+   ```
+
+3. **Verify Provider Works**
+   - Check provider logs
+   - Test VM operations
+   - Verify no authentication errors
+
+4. **Revoke Old Token**
+   ```bash
+   pvesh delete /access/users/root@pam/token/crossplane-instance-1
+   ```
+
+5. **Update Documentation**
+   - Update token inventory
+   - Record rotation date
+   - Update expiration dates
+
+### Automated Rotation
+
+Consider implementing automated token rotation:
+- Kubernetes CronJob
+- External secret manager (e.g., HashiCorp Vault)
+- Proxmox API integration
+
+## Token Inventory
+
+### Current Tokens
+
+| Token ID | User | Realm | Expiration | Purpose | Status |
+|----------|------|-------|------------|---------|--------|
+| crossplane-instance-1 | root | pam | TBD | Instance 1 provider | Active |
+| crossplane-instance-2 | root | pam | TBD | Instance 2 provider | Active |
+
+### Token Tracking
+
+Maintain a token inventory with:
+- Token ID
+- Associated user
+- Creation date
+- Expiration date
+- Purpose/comment
+- Last rotation date
+- Status (active/revoked)
+
+## Security Best Practices
+
+### ✅ Do
+
+- Use dedicated service accounts for tokens
+- Set token expiration dates
+- Rotate tokens regularly
+- Store tokens in Kubernetes secrets
+- Use principle of least privilege
+- Monitor token usage
+- Audit token access logs
+- Revoke unused tokens immediately
+
+### ❌ Don't
+
+- Store tokens in code or config files
+- Commit tokens to git
+- Share tokens between environments
+- Use administrator tokens unnecessarily
+- Set tokens to never expire
+- Log token secrets
+- Expose tokens in error messages
+
+## Token Monitoring
+
+### Usage Monitoring
+
+Monitor token usage via Proxmox audit logs:
+
+```bash
+# View API token usage
+pvesh get /api2/json/access/token/{token-id}
+```
+
+### Audit Logs
+
+Review Proxmox audit logs for:
+- Token creation/deletion
+- Token usage patterns
+- Failed authentication attempts
+- Unusual access patterns
+
+### Alerting
+
+Set up alerts for:
+- Token expiration (30 days before)
+- Failed authentication attempts
+- Unusual API usage patterns
+- Token rotation due dates
+
+## Troubleshooting
+
+### Error: "authentication failed"
+
+**Causes**:
+- Token expired
+- Token revoked
+- Incorrect token format
+- Token secret mismatch
+
+**Solutions**:
+1. Verify token is active: `pvesh get /access/users/{user}/token/{token-id}`
+2. Check token expiration date
+3. Verify token secret in Kubernetes secret
+4. Recreate token if needed
+
+### Error: "permission denied"
+
+**Causes**:
+- Insufficient permissions
+- Token permissions changed
+- Resource access restrictions
+
+**Solutions**:
+1. Review token permissions
+2. Check ACL rules
+3. Verify user permissions
+4. Update token permissions if needed
+
+## Compliance
+
+### SOC 2 Requirements
+
+- ✅ Token rotation procedures
+- ✅ Token expiration policies
+- ✅ Access logging and monitoring
+- ✅ Principle of least privilege
+
+### ISO 27001 Requirements
+
+- ✅ Token management procedures
+- ✅ Access control policies
+- ✅ Audit logging
+- ✅ Security incident response
+
+## Related Documentation
+
+- [Provider Configuration](./PROVIDER_CONFIG.md)
+- [Security Audit](./SECURITY_AUDIT.md)
+- [TLS Configuration](./TLS_CONFIGURATION.md)
+
+## Last Updated
+
+- **Date**: 2024-12-19
+- **Next Rotation**: TBD
+- **Review Date**: 2025-01-19
+
--- a/docs/proxmox/reference/API_TOKEN_MANAGEMENT.md
+++ b/docs/proxmox/reference/API_TOKEN_MANAGEMENT.md
@@ -0,0 +1,235 @@
+# Proxmox API Token Management Guide
+
+## Overview
+
+This guide covers best practices for managing Proxmox API tokens used by the Crossplane provider.
+
+## Token Creation
+
+### Via Proxmox Web UI
+
+1. **Navigate to API Tokens**:
+   - Log into Proxmox Web UI
+   - Go to Datacenter → Permissions → API Tokens
+   - Click "Add"
+
+2. **Configure Token**:
+   - **Token ID**: `crossplane-<site-name>` (e.g., `crossplane-us-east-1`)
+   - **User**: `root@pam` or dedicated service account
+   - **Expiration**: Set appropriate expiration (recommended: 1 year or less)
+   - **Privilege Separation**: Enable if using dedicated user
+
+3. **Set Permissions**:
+   - **Administrator**: Full access (for development)
+   - **VM-specific**: Limited to VM operations (for production)
+   - **Storage-specific**: Limited to storage operations (if needed)
+
+4. **Generate Token**:
+   - Click "Generate"
+   - **IMPORTANT**: Copy the token immediately (format: `user@realm!token-name=token-secret`)
+   - Store securely (will not be shown again)
+
+### Via Proxmox API
+
+```bash
+# Create token via API (requires existing authentication)
+curl -k -X POST \
+  -H "Authorization: PVEAuthCookie=YOUR_TICKET" \
+  -H "CSRFPreventionToken: YOUR_CSRF_TOKEN" \
+  -d "tokenid=crossplane-us-east-1&userid=root@pam&expire=31536000" \
+  https://your-proxmox:8006/api2/json/access/users/root@pam/token
+```
+
+## Token Format
+
+Proxmox API tokens use the format:
+```
+user@realm!token-name=token-secret
+```
+
+Example:
+```
+root@pam!crossplane-us-east-1=abc123def456ghi789
+```
+
+## Token Storage
+
+### Kubernetes Secret
+
+Store tokens in Kubernetes secrets:
+
+```bash
+kubectl create secret generic proxmox-credentials \
+  --from-literal=credentials.json='{"username":"root@pam","token":"root@pam!crossplane-token=abc123..."}' \
+  -n crossplane-system
+```
+
+### ProviderConfig Reference
+
+The ProviderConfig references the secret:
+
+```yaml
+spec:
+  credentials:
+    source: Secret
+    secretRef:
+      name: proxmox-credentials
+      namespace: crossplane-system
+      key: credentials.json
+```
+
+## Token Permissions
+
+### Recommended Permissions
+
+For production use, create tokens with minimal required permissions:
+
+1. **VM Operations**:
+   - `VM.Allocate`
+   - `VM.Clone`
+   - `VM.Config`
+   - `VM.Monitor`
+   - `VM.PowerMgmt`
+
+2. **Storage Operations** (if needed):
+   - `Datastore.Allocate`
+   - `Datastore.Audit`
+
+3. **Network Operations** (if needed):
+   - `SDN.Use`
+
+### Development Permissions
+
+For development/testing, Administrator role is acceptable but not recommended for production.
+
+## Token Rotation
+
+### Rotation Schedule
+
+- **Production**: Rotate every 90 days
+- **Development**: Rotate every 180 days
+- **Emergency**: Rotate immediately if compromised
+
+### Rotation Procedure
+
+1. **Create New Token**:
+   - Create new token with same permissions
+   - Test new token
+
+2. **Update Kubernetes Secret**:
+   ```bash
+   kubectl delete secret proxmox-credentials -n crossplane-system
+   kubectl create secret generic proxmox-credentials \
+     --from-literal=credentials.json='{"username":"root@pam","token":"NEW_TOKEN"}' \
+     -n crossplane-system
+   ```
+
+3. **Restart Provider**:
+   ```bash
+   kubectl delete pod -n crossplane-system -l app=crossplane-provider-proxmox
+   ```
+
+4. **Verify**:
+   ```bash
+   kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox --tail=50
+   ```
+
+5. **Revoke Old Token**:
+   - Log into Proxmox Web UI
+   - Go to API Tokens
+   - Delete old token
+
+## Token Security
+
+### Best Practices
+
+1. **Never Commit Tokens**: 
+   - Never commit tokens to git
+   - Use secrets management
+   - Rotate if accidentally exposed
+
+2. **Use Separate Tokens**:
+   - Use different tokens per site/environment
+   - Use different tokens per application
+   - Track token usage
+
+3. **Monitor Token Usage**:
+   - Review Proxmox audit logs
+   - Monitor for unusual activity
+   - Set up alerts for failures
+
+4. **Limit Token Scope**:
+   - Use principle of least privilege
+   - Grant only required permissions
+   - Review permissions regularly
+
+5. **Set Expiration**:
+   - Always set token expiration
+   - Rotate before expiration
+   - Document expiration dates
+
+## Token Troubleshooting
+
+### Authentication Failures
+
+1. **Check Token Format**:
+   - Verify format: `user@realm!token-name=token-secret`
+   - Check for typos
+   - Verify special characters are escaped
+
+2. **Verify Token Validity**:
+   - Check token expiration
+   - Verify token not revoked
+   - Check user account status
+
+3. **Test Token**:
+   ```bash
+   curl -k -H "Authorization: PVEAuthCookie=TOKEN" \
+     https://your-proxmox:8006/api2/json/version
+   ```
+
+### Permission Errors
+
+1. **Check Permissions**:
+   - Review token permissions in Proxmox
+   - Verify required permissions are granted
+   - Check user roles
+
+2. **Test Operations**:
+   - Try operation via Proxmox API
+   - Check error messages
+   - Review Proxmox logs
+
+## Token Audit
+
+### Regular Audits
+
+1. **Monthly Review**:
+   - List all active tokens
+   - Review token usage
+   - Check for unused tokens
+
+2. **Quarterly Review**:
+   - Review token permissions
+   - Verify token expiration dates
+   - Update documentation
+
+3. **Annual Review**:
+   - Complete token audit
+   - Rotate all tokens
+   - Review security practices
+
+### Audit Commands
+
+```bash
+# List tokens (via Proxmox API)
+curl -k -H "Authorization: PVEAuthCookie=TOKEN" \
+  https://your-proxmox:8006/api2/json/access/users/root@pam/token
+```
+
+## Related Documentation
+
+- [Deployment Guide](./DEPLOYMENT_GUIDE.md)
+- [Security Audit](./TASK_LIST.md#task-020)
+- [Troubleshooting Guide](../runbooks/PROXMOX_TROUBLESHOOTING.md)
+
--- a/docs/proxmox/reference/IMAGE_INVENTORY.md
+++ b/docs/proxmox/reference/IMAGE_INVENTORY.md
@@ -0,0 +1,224 @@
+# Proxmox Image Inventory and Requirements
+
+**Last Updated**: 2024-12-19
+
+## Summary
+
+All VM manifests and examples reference: **`ubuntu-22.04-cloud`**
+
+## Required Images for Remaining Tasks
+
+### Primary Image Required
+
+**Image Name**: `ubuntu-22.04-cloud`  
+**Type**: Cloud Image (qcow2 format)  
+**Purpose**: Default OS image for all test VMs and examples  
+**Required For**:
+- TASK-015: Deploy test VMs via Crossplane
+- TASK-016: End-to-end testing
+- All example VM manifests
+
+### Image References in Codebase
+
+#### VM Manifests Using `ubuntu-22.04-cloud`:
+
+1. **test-vm-instance-1.yaml** (ML110-01)
+   - Image: `ubuntu-22.04-cloud`
+   - Storage: `local-lvm`
+   - Node: `ML110-01`
+   - Site: `us-sfvalley`
+
+2. **test-vm-instance-2.yaml** (R630-01)
+   - Image: `ubuntu-22.04-cloud`
+   - Storage: `local-lvm`
+   - Node: `R630-01`
+   - Site: `us-sfvalley-2`
+
+3. **vm-example.yaml** (Example)
+   - Image: `ubuntu-22.04-cloud`
+   - Storage: `local-lvm`
+   - Node: `ML110-01`
+   - Site: `us-sfvalley`
+
+4. **README.md** (Documentation)
+   - Image: `ubuntu-22.04-cloud`
+   - Default example image
+
+5. **gitops/templates/vm/ubuntu-22.04.yaml**
+   - Image: `ubuntu-22.04-cloud`
+   - Template for VM creation
+
+6. **gitops/templates/vm/ubuntu-20.04.yaml**
+   - Image: `ubuntu-20.04-cloud`
+   - Alternative template
+
+## Image Availability Check
+
+### Current Status
+
+⚠️ **API-based image listing is limited**:
+- Storage content endpoints require additional permissions
+- Cannot verify images via API without proper access
+- Images may exist but not be visible via current API tokens
+
+### Verification Methods
+
+1. **Proxmox Web UI**:
+   - Log in to: https://ml110-01.sankofa.nexus:8006
+   - Navigate to: **Datacenter** → **Storage** → Select storage → **Content**
+   - Check for: `ubuntu-22.04-cloud` or similar
+
+2. **SSH Command**:
+   ```bash
+   ssh root@192.168.11.10
+   ls -lh /var/lib/vz/template/iso/
+   ls -lh /var/lib/vz/template/cache/
+   ```
+
+3. **Proxmox Command**:
+   ```bash
+   pveam list local
+   ```
+
+## Image Download Instructions
+
+### Method 1: Download via Proxmox Web UI
+
+1. Log in to Proxmox Web UI
+2. Go to: **Datacenter** → **Storage** → Select storage (e.g., `local`)
+3. Click **Content** tab
+4. Click **Templates** → **Download**
+5. Search for: `ubuntu-22.04-standard`
+6. Download template
+
+### Method 2: Download via Command Line (SSH)
+
+```bash
+# SSH into Proxmox node
+ssh root@192.168.11.10
+
+# List available templates
+pveam available
+
+# Download Ubuntu 22.04 template
+pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz
+
+# Verify download
+pveam list local
+```
+
+### Method 3: Download Cloud Image Manually
+
+```bash
+# Download Ubuntu 22.04 Cloud Image
+wget https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img
+
+# Upload to Proxmox storage
+# Via Web UI: Storage → Content → Upload
+# Or via API (see below)
+```
+
+### Method 4: Upload via API
+
+```bash
+source .env
+
+# Upload ISO/image file
+curl -k -H "Authorization: PVEAPIToken ${PROXMOX_TOKEN_ML110_01}" \
+  -F "filename=@ubuntu-22.04-server-cloudimg-amd64.img" \
+  "https://192.168.11.10:8006/api2/json/storage/local/upload"
+```
+
+## Image Requirements by Task
+
+### TASK-015: Deploy Test VMs via Crossplane
+
+**Required Images**:
+- ✅ `ubuntu-22.04-cloud` (or equivalent)
+- Storage: `local-lvm` (or configured storage pool)
+- Location: Both ML110-01 and R630-01 (if using multi-site)
+
+**Action**: Ensure image exists on both nodes before deployment
+
+### TASK-016: End-to-End Testing
+
+**Required Images**:
+- ✅ `ubuntu-22.04-cloud` (primary)
+- Optional: Additional OS images for testing diversity
+  - `ubuntu-20.04-cloud`
+  - `debian-12-standard`
+  - `centos-stream-9-standard`
+
+### TASK-019: Backup Procedures
+
+**Required Images**:
+- ✅ Test VM images for backup/restore testing
+- Same images as TASK-015
+
+## Image Naming Conventions
+
+### Proxmox Template Names
+
+- **Standard Templates**: `ubuntu-22.04-standard_22.04-1_amd64.tar.gz`
+- **Cloud Images**: `ubuntu-22.04-cloud` (custom name)
+- **ISO Files**: `ubuntu-22.04-server-amd64.iso`
+
+### Storage Locations
+
+- **Templates**: `/var/lib/vz/template/cache/`
+- **ISO Files**: `/var/lib/vz/template/iso/`
+- **Local Storage**: `local` (default)
+- **LVM Storage**: `local-lvm` (for VM disks)
+
+## Image Checklist
+
+### For ML110-01 (us-sfvalley)
+
+- [ ] `ubuntu-22.04-cloud` image available
+- [ ] Image accessible from storage pool: `local-lvm`
+- [ ] Image verified (can be used for VM creation)
+- [ ] (Optional) Additional test images
+
+### For R630-01 (us-sfvalley-2)
+
+- [ ] `ubuntu-22.04-cloud` image available
+- [ ] Image accessible from storage pool: `local-lvm`
+- [ ] Image verified (can be used for VM creation)
+- [ ] (Optional) Additional test images
+
+## Image Verification Script
+
+```bash
+# Check if image exists (via SSH)
+ssh root@192.168.11.10 "pveam list local | grep ubuntu-22.04"
+
+# Check storage content (via API - may require permissions)
+curl -k -H "Authorization: PVEAPIToken ${TOKEN}" \
+  "https://192.168.11.10:8006/api2/json/storage/local/content"
+```
+
+## Next Steps
+
+1. **Verify Image Availability**:
+   - Check via Web UI or SSH
+   - Use `pveam list local` command
+
+2. **Download Missing Images**:
+   - Use `pveam download` command
+   - Or download from official sources and upload
+
+3. **Update Manifests** (if needed):
+   - If image name differs, update VM manifests
+   - Ensure image name matches actual file name
+
+4. **Test Image**:
+   - Create a test VM using the image
+   - Verify VM boots correctly
+   - Verify cloud-init works (if using cloud images)
+
+## Related Documentation
+
+- [Image Requirements](./IMAGE_REQUIREMENTS.md)
+- [Task List](./TASK_LIST.md)
+- [VM Provisioning Runbook](../runbooks/PROXMOX_VM_PROVISIONING.md)
+
--- a/docs/proxmox/reference/IMAGE_REQUIREMENTS.md
+++ b/docs/proxmox/reference/IMAGE_REQUIREMENTS.md
@@ -0,0 +1,191 @@
+# Image Requirements for Remaining Steps
+
+**Last Updated**: 2024-12-19
+
+## Overview
+
+This document identifies all ISO files, disk images, and OS templates needed for the remaining deployment and testing tasks.
+
+## Current Image Status
+
+### Images Available on Proxmox Instances
+
+*(To be populated from API scan)*
+
+### Images Referenced in Configuration
+
+#### Test VM Manifests
+
+1. **test-vm-instance-1.yaml** (ML110-01):
+   - Image: *(Check manifest)*
+   - Storage: *(Check manifest)*
+   - Purpose: Test VM deployment on Instance 1
+
+2. **test-vm-instance-2.yaml** (R630-01):
+   - Image: *(Check manifest)*
+   - Storage: *(Check manifest)*
+   - Purpose: Test VM deployment on Instance 2
+
+3. **vm-example.yaml**:
+   - Image: *(Check manifest)*
+   - Storage: *(Check manifest)*
+   - Purpose: Example VM configuration
+
+## Required Images by Task
+
+### TASK-015: Deploy Test VMs via Crossplane
+
+**Required**:
+- OS template or ISO for VM creation
+- Recommended: Ubuntu 22.04 Cloud Image or similar
+- Format: qcow2, raw, or ISO
+
+**Options**:
+1. **Cloud Images** (Recommended for automation):
+   - Ubuntu 22.04 Cloud Image
+   - Debian 12 Cloud Image
+   - CentOS Stream 9 Cloud Image
+
+2. **ISO Files** (For manual installation):
+   - Ubuntu 22.04 Server ISO
+   - Debian 12 Netinst ISO
+   - CentOS Stream 9 ISO
+
+### TASK-016: End-to-End Testing
+
+**Required**:
+- Multiple OS images for testing:
+  - Linux distribution (Ubuntu/Debian)
+  - Windows Server (if needed)
+  - Specialized images for testing
+
+### TASK-019: Backup Procedures
+
+**Required**:
+- Test VM images for backup/restore testing
+- Various OS types to test backup compatibility
+
+## Image Sources
+
+### Official Proxmox Templates
+
+Proxmox provides official templates via:
+- **Proxmox VE Web UI**: Local → Templates → Download
+- **Command Line**: `pveam download <storage> <template>`
+
+### Popular Templates
+
+1. **Ubuntu**:
+   ```bash
+   pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz
+   ```
+
+2. **Debian**:
+   ```bash
+   pveam download local debian-12-standard_12.0-1_amd64.tar.gz
+   ```
+
+3. **CentOS**:
+   ```bash
+   pveam download local centos-stream-9-standard_9-1.x86_64.tar.gz
+   ```
+
+### Cloud Images
+
+Download from official sources:
+- **Ubuntu Cloud Images**: https://cloud-images.ubuntu.com/
+- **Debian Cloud Images**: https://cdimage.debian.org/cdimage/cloud/
+- **CentOS Cloud Images**: https://cloud.centos.org/
+
+### ISO Files
+
+Download from official sources:
+- **Ubuntu**: https://ubuntu.com/download/server
+- **Debian**: https://www.debian.org/CD/http-ftp/
+- **CentOS**: https://www.centos.org/download/
+
+## Image Download Scripts
+
+### Download to Proxmox Storage
+
+```bash
+# On Proxmox node
+STORAGE="local"  # or your storage pool name
+
+# Download Ubuntu template
+pveam download ${STORAGE} ubuntu-22.04-standard_22.04-1_amd64.tar.gz
+
+# Download Debian template
+pveam download ${STORAGE} debian-12-standard_12.0-1_amd64.tar.gz
+```
+
+### Upload ISO via Web UI
+
+1. Log in to Proxmox Web UI
+2. Go to: **Datacenter** → **Storage** → Select storage → **Content** → **Upload**
+3. Upload ISO file
+
+### Upload via API
+
+```bash
+# Upload ISO file
+curl -k -H "Authorization: PVEAPIToken ${TOKEN}" \
+  -F "filename=@ubuntu-22.04-server-amd64.iso" \
+  "https://${PROXMOX_IP}:8006/api2/json/storage/${STORAGE}/upload"
+```
+
+## Image Inventory Checklist
+
+### For ML110-01 (us-sfvalley)
+
+- [ ] Ubuntu 22.04 Cloud Image or Template
+- [ ] Debian 12 Cloud Image or Template
+- [ ] (Optional) CentOS Stream 9 Image
+- [ ] (Optional) ISO files for manual installation
+
+### For R630-01 (us-sfvalley-2)
+
+- [ ] Ubuntu 22.04 Cloud Image or Template
+- [ ] Debian 12 Cloud Image or Template
+- [ ] (Optional) CentOS Stream 9 Image
+- [ ] (Optional) ISO files for manual installation
+
+## Storage Requirements
+
+### Minimum Storage Needed
+
+- **Per Cloud Image**: ~500MB - 1GB
+- **Per ISO File**: ~1GB - 4GB
+- **Recommended**: 10GB+ free space for images
+
+### Storage Locations
+
+- **Local Storage**: `local` (default)
+- **NFS Storage**: If configured
+- **Ceph Storage**: If cluster storage is configured
+
+## Next Steps
+
+1. **Inventory Current Images**:
+   ```bash
+   ./scripts/list-proxmox-images.sh
+   ```
+
+2. **Download Missing Images**:
+   - Use Proxmox Web UI or `pveam` command
+   - Download to appropriate storage pools
+
+3. **Update VM Manifests**:
+   - Update image references in test VM manifests
+   - Verify image names match actual files
+
+4. **Verify Image Availability**:
+   - Check images are accessible from both nodes
+   - Test image can be used for VM creation
+
+## Related Documentation
+
+- [Task List](./TASK_LIST.md)
+- [VM Provisioning Runbook](../runbooks/PROXMOX_VM_PROVISIONING.md)
+- [Deployment Guide](./DEPLOYMENT_GUIDE.md)
+
--- a/docs/proxmox/reference/INSTANCE_INVENTORY.md
+++ b/docs/proxmox/reference/INSTANCE_INVENTORY.md
@@ -0,0 +1,49 @@
+# Proxmox Instance Inventory
+
+**Generated**: 2024-12-19  
+**Source**: Automated inventory from Proxmox API
+
+## Instance 1: ML110-01
+
+**IP**: 192.168.11.10  
+**FQDN**: ml110-01.sankofa.nexus  
+**Site**: us-sfvalley  
+**Endpoint**: https://ml110-01.sankofa.nexus:8006  
+**Proxmox Version**: 9.1.1
+
+### Node Status
+- **Status**: Online
+- **Node Name**: ML110-01
+- **API Access**: ✅ Verified
+- **Authentication**: ✅ Working
+
+### Storage Pools
+
+### Network Interfaces
+
+
+### Virtual Machines
+
+
+## Instance 2: R630-01
+
+**IP**: 192.168.11.11
+**FQDN**: r630-01.sankofa.nexus
+**Site**: us-sfvalley-2
+**Endpoint**: https://r630-01.sankofa.nexus:8006
+**Proxmox Version**: 9.1.1
+
+### Node Status
+- **Status**: Online
+- **Node Name**: R630-01
+- **API Access**: ✅ Verified
+- **Authentication**: ✅ Working
+
+### Storage Pools
+
+
+### Network Interfaces
+
+
+### Virtual Machines
+
--- a/docs/proxmox/reference/RESOURCE_INVENTORY.md
+++ b/docs/proxmox/reference/RESOURCE_INVENTORY.md
@@ -0,0 +1,159 @@
+# Proxmox Resource Inventory
+
+## Overview
+
+This document tracks the actual Proxmox resources available across all instances. This information should be gathered from the actual Proxmox clusters and updated as resources change.
+
+## Instance Mapping
+
+| Site | Hostname | IP Address | Instance | Status |
+|------|----------|------------|----------|--------|
+| us-sfvalley | ml110-01.sankofa.nexus | 192.168.11.10 | Instance 1 | Active |
+| us-sfvalley-2 | r630-01.sankofa.nexus | 192.168.11.11 | Instance 2 | Active |
+
+## Storage Pools
+
+### Instance 1 (192.168.11.10)
+
+| Name | Type | Size | Used | Available | Status |
+|------|------|------|------|----------|--------|
+| local | Directory | TBD | TBD | TBD | Active |
+| local-lvm | LVM-Thin | TBD | TBD | TBD | Active |
+
+### Instance 2 (192.168.11.11)
+
+| Name | Type | Size | Used | Available | Status |
+|------|------|------|------|----------|--------|
+| local | Directory | TBD | TBD | TBD | Active |
+| local-lvm | LVM-Thin | TBD | TBD | TBD | Active |
+
+**Note**: Run the following command to discover actual storage pools:
+```bash
+pvesh get /storage
+```
+
+## Network Bridges
+
+### Instance 1 (192.168.11.10)
+
+| Name | Type | VLAN Aware | Bridge Ports | Status |
+|------|------|------------|--------------|--------|
+| vmbr0 | Linux Bridge | No | eth0 | Active |
+
+### Instance 2 (192.168.11.11)
+
+| Name | Type | VLAN Aware | Bridge Ports | Status |
+|------|------|------------|--------------|--------|
+| vmbr0 | Linux Bridge | No | eth0 | Active |
+
+**Note**: Run the following command to discover actual network bridges:
+```bash
+pvesh get /nodes/{node}/network
+```
+
+## OS Templates / Images
+
+### Instance 1 (192.168.11.10)
+
+| Name | Type | Size | Description |
+|------|------|------|-------------|
+| ubuntu-22.04-cloud | VMA | TBD | Ubuntu 22.04 Cloud Image |
+| ubuntu-20.04-cloud | VMA | TBD | Ubuntu 20.04 Cloud Image |
+| debian-11-standard | VMA | TBD | Debian 11 Standard Template |
+
+### Instance 2 (192.168.11.11)
+
+| Name | Type | Size | Description |
+|------|------|------|-------------|
+| ubuntu-22.04-cloud | VMA | TBD | Ubuntu 22.04 Cloud Image |
+| ubuntu-20.04-cloud | VMA | TBD | Ubuntu 20.04 Cloud Image |
+| debian-11-standard | VMA | TBD | Debian 11 Standard Template |
+
+**Note**: Run the following command to discover actual templates:
+```bash
+pvesh get /nodes/{node}/storage/{storage}/content
+```
+
+## Cluster Nodes
+
+### Instance 1 Cluster
+
+| Node Name | IP Address | Status | CPU Cores | Memory | Storage |
+|-----------|------------|--------|-----------|---------|---------|
+| ML110-01 | 192.168.11.10 | Active | TBD | TBD | TBD |
+
+### Instance 2 Cluster
+
+| Node Name | IP Address | Status | CPU Cores | Memory | Storage |
+|-----------|------------|--------|-----------|---------|---------|
+| R630-01 | 192.168.11.11 | Active | TBD | TBD | TBD |
+
+**Note**: Run the following command to discover cluster nodes:
+```bash
+pvesh get /nodes
+```
+
+## Resource Discovery Script
+
+Use the following script to automatically discover and update this inventory:
+
+```bash
+#!/bin/bash
+# scripts/discover-proxmox-resources.sh
+
+INSTANCE1="https://192.168.11.10:8006"
+INSTANCE2="https://192.168.11.11:8006"
+
+echo "Discovering Proxmox Instance 1 resources..."
+# Add discovery commands here
+
+echo "Discovering Proxmox Instance 2 resources..."
+# Add discovery commands here
+```
+
+## Update Procedure
+
+1. **Connect to Proxmox Instance**
+   ```bash
+   ssh root@192.168.11.10
+   # or use pvesh CLI
+   ```
+
+2. **Gather Storage Information**
+   ```bash
+   pvesh get /storage --output-format json
+   ```
+
+3. **Gather Network Information**
+   ```bash
+   pvesh get /nodes/{node}/network --output-format json
+   ```
+
+4. **Gather Template Information**
+   ```bash
+   pvesh get /nodes/{node}/storage/{storage}/content --output-format json
+   ```
+
+5. **Gather Node Information**
+   ```bash
+   pvesh get /nodes --output-format json
+   pvesh get /nodes/{node}/status --output-format json
+   ```
+
+6. **Update This Document**
+   - Replace TBD values with actual data
+   - Add any additional resources discovered
+   - Update timestamps
+
+## Last Updated
+
+- **Date**: TBD
+- **Updated By**: TBD
+- **Method**: Manual / Script
+
+## Notes
+
+- All TBD values should be filled in after connecting to actual Proxmox instances
+- This document should be updated whenever resources change
+- Consider automating resource discovery and updates
+- Store sensitive information (IPs, credentials) securely, not in this document
--- a/docs/proxmox/reference/SITE_MAPPING.md
+++ b/docs/proxmox/reference/SITE_MAPPING.md
@@ -0,0 +1,100 @@
+# Proxmox Site Mapping
+
+This document maps physical Proxmox instances to logical sites and documents the configuration.
+
+## Physical Instances
+
+### Instance 1
+- **IP Address**: 192.168.11.10
+- **Port**: 8006
+- **Endpoint**: https://192.168.11.10:8006
+- **Status**: ✅ Verified (HTTP 401 - authentication required)
+- **Mapped Sites**: us-sfvalley (Instance 1)
+
+### Instance 2
+- **IP Address**: 192.168.11.11
+- **Port**: 8006
+- **Endpoint**: https://192.168.11.11:8006
+- **Status**: ✅ Verified (HTTP 401 - authentication required)
+- **Mapped Sites**: us-sfvalley-2 (Instance 2)
+
+## Site Configuration
+
+### us-sfvalley (US San Francisco Valley) - Instance 1
+- **Physical Instance**: Instance 1 (192.168.11.10)
+- **FQDN**: ml110-01.sankofa.nexus
+- **Endpoint**: https://ml110-01.sankofa.nexus:8006
+- **Primary Node**: ML110-01
+- **Cloudflare Tunnel**: proxmox-site-1-tunnel
+- **DNS Records Required**:
+  - `ml110-01.sankofa.nexus` → 192.168.11.10
+  - `ml110-01-api.sankofa.nexus` → 192.168.11.10
+  - `ml110-01-metrics.sankofa.nexus` → 192.168.11.10
+
+### us-sfvalley-2 (US San Francisco Valley) - Instance 2
+- **Physical Instance**: Instance 2 (192.168.11.11)
+- **FQDN**: r630-01.sankofa.nexus
+- **Endpoint**: https://r630-01.sankofa.nexus:8006
+- **Primary Node**: R630-01
+- **Cloudflare Tunnel**: proxmox-site-2-tunnel (or proxmox-site-3-tunnel)
+- **DNS Records Required**:
+  - `r630-01.sankofa.nexus` → 192.168.11.11
+  - `r630-01-api.sankofa.nexus` → 192.168.11.11
+  - `r630-01-metrics.sankofa.nexus` → 192.168.11.11
+
+## Configuration Files
+
+### Provider Config
+- **File**: `crossplane-provider-proxmox/examples/provider-config.yaml`
+- **Status**: ✅ Updated with token authentication format
+- **Sites Configured**: us-sfvalley (Instance 1), us-sfvalley-2 (Instance 2)
+
+### Cloudflare Tunnel Configs
+- **Site 1**: `cloudflare/tunnel-configs/proxmox-site-1.yaml` ✅ Updated
+- **Site 2**: `cloudflare/tunnel-configs/proxmox-site-2.yaml` ✅ Updated
+- **Site 3**: `cloudflare/tunnel-configs/proxmox-site-3.yaml` ✅ Updated
+
+## Verification Status
+
+### Connectivity
+- ✅ Instance 1: Reachable (HTTP 401)
+- ✅ Instance 2: Reachable (HTTP 401)
+
+### Authentication
+- ⏳ Instance 1: Pending (requires credentials)
+- ⏳ Instance 2: Pending (requires credentials)
+
+### DNS
+- ⏳ All hostnames: Pending configuration
+
+### Cloudflare Tunnels
+- ⏳ Tunnel credentials: Pending generation
+- ⏳ Tunnel deployment: Pending
+
+## Notes
+
+1. **Instance Sharing**: Instance 2 hosts both eu-west-1 and apac-1 sites
+   - This is acceptable for development/testing
+   - Production should have separate instances per site
+
+2. **Node Names**: 
+   - Instance 1 (192.168.11.10): ML110-01
+   - Instance 2 (192.168.11.11): R630-01
+   - Both sites on Instance 2 use the same node (R630-01)
+
+3. **DNS Configuration**: All DNS records should point to the physical IP addresses
+   - Cloudflare tunnels will handle the routing
+   - DNS is required for tunnel hostname validation
+
+4. **Tunnel Credentials**: Each site needs separate tunnel credentials
+   - Generate via Cloudflare dashboard or API
+   - Deploy to respective Proxmox nodes
+
+## Next Steps
+
+1. **Verify Node Names**: After authentication, verify actual node names
+2. **Configure DNS**: Create all required DNS records
+3. **Generate Tunnels**: Create Cloudflare tunnels for each site
+4. **Deploy Tunnels**: Install and configure cloudflared on nodes
+5. **Test Connectivity**: Verify access via Cloudflare hostnames
+