Initial commit: loc_az_hci (smom-dbis-138 excluded via .gitignore)

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-08 09:04:46 -08:00
commit c39465c2bd
386 changed files with 50649 additions and 0 deletions
--- a/docs/troubleshooting/ACCESS_PATHS_MAP.md
+++ b/docs/troubleshooting/ACCESS_PATHS_MAP.md
@@ -0,0 +1,414 @@
+# Access Paths Map - Complete Infrastructure Access Guide
+
+**Date:** 2025-11-27  
+**Purpose:** Map all access methods for troubleshooting and task completion
+
+## 🗺️ Access Paths Overview
+
+### Proxmox Hosts
+
+#### ML110 (192.168.1.206)
+
+**1. Web UI Access**
+- **URL:** https://192.168.1.206:8006
+- **Credentials:** root / [password from .env]
+- **Status:** ✅ Working
+- **Use Cases:**
+  - VM management
+  - Cluster configuration
+  - Storage management
+  - Network configuration
+  - Console access to VMs
+  - Service management
+
+**2. SSH Access**
+- **Command:** `ssh -i ~/.ssh/id_ed25519_proxmox root@192.168.1.206`
+- **Status:** ✅ Working
+- **Key File:** `~/.ssh/id_ed25519_proxmox`
+- **Use Cases:**
+  - Command-line management
+  - Script execution
+  - File transfers
+  - Service configuration
+
+**3. API Access**
+- **URL:** https://192.168.1.206:8006/api2/json
+- **Authentication:** Username/password or API tokens
+- **Status:** ✅ Working
+- **Use Cases:**
+  - Automation scripts
+  - VM operations
+  - Status queries
+  - Configuration changes
+
+**4. Console Access (Physical/KVM)**
+- **Method:** Physical access or IPMI/KVM
+- **Status:** Unknown
+- **Use Cases:**
+  - Initial setup
+  - Recovery scenarios
+  - Network troubleshooting
+
+#### R630 (192.168.1.49)
+
+**1. Web UI Access**
+- **URL:** https://192.168.1.49:8006
+- **Credentials:** root / [password from .env]
+- **Status:** ✅ Working (assumed)
+- **Use Cases:** Same as ML110
+
+**2. SSH Access**
+- **Command:** `ssh -i ~/.ssh/id_ed25519_proxmox root@192.168.1.49`
+- **Status:** ❌ Not working (authentication failing)
+- **Fix:** Enable SSH and add SSH key (see SSH_ENABLE_QUICK_GUIDE.md)
+
+**3. API Access**
+- **URL:** https://192.168.1.49:8006/api2/json
+- **Status:** ✅ Working (assumed)
+- **Use Cases:** Same as ML110
+
+**4. Console Access (Physical/KVM)**
+- **Method:** Physical access or IPMI/KVM
+- **Status:** Unknown
+
+---
+
+### Virtual Machines
+
+#### VM 100 - Cloudflare Tunnel (192.168.1.60)
+
+**1. SSH Access**
+- **Command:** `ssh -i ~/.ssh/id_ed25519_proxmox ubuntu@192.168.1.60`
+- **Status:** ❌ Not working (authentication failing)
+- **Alternative:** Use Proxmox console
+
+**2. Proxmox Console**
+- **Method:** Web UI → VM 100 → Console
+- **Status:** ✅ Available
+- **Use Cases:**
+  - Initial setup
+  - SSH key configuration
+  - Service installation
+  - Troubleshooting
+
+**3. QEMU Guest Agent**
+- **Command:** `qm guest exec 100 -- <command>`
+- **Status:** ❌ Not running (agent not installed in VM)
+- **Fix:** Install qemu-guest-agent in VM
+
+**4. Network Access**
+- **Ping:** ✅ Working
+- **Port 22:** ✅ Open
+- **Port 80/443:** ⏳ (for services)
+
+**5. Service Access (When Running)**
+- **Cloudflare Tunnel:** CLI tool
+- **Status:** Installed, needs authentication
+
+#### VM 101 - K3s Master (192.168.1.188)
+
+**1. SSH Access**
+- **Command:** `ssh -i ~/.ssh/id_ed25519_proxmox ubuntu@192.168.1.188`
+- **Status:** ❌ Not working (authentication failing)
+- **Alternative:** Use Proxmox console
+
+**2. Proxmox Console**
+- **Method:** Web UI → VM 101 → Console
+- **Status:** ✅ Available
+
+**3. QEMU Guest Agent**
+- **Command:** `qm guest exec 101 -- <command>`
+- **Status:** ❌ Not running
+
+**4. Network Access**
+- **Ping:** ✅ Working
+- **Port 22:** ✅ Open
+- **Port 6443:** ⏳ (K3s API)
+- **Port 10250:** ⏳ (Kubelet)
+
+**5. Service Access**
+- **K3s API:** `kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml`
+- **Status:** Installed, needs verification
+
+#### VM 102 - Git Server (192.168.1.121)
+
+**1. SSH Access**
+- **Command:** `ssh -i ~/.ssh/id_ed25519_proxmox ubuntu@192.168.1.121`
+- **Status:** ❌ Not working (authentication failing)
+- **Alternative:** Use Proxmox console
+
+**2. Proxmox Console**
+- **Method:** Web UI → VM 102 → Console
+- **Status:** ✅ Available
+
+**3. QEMU Guest Agent**
+- **Command:** `qm guest exec 102 -- <command>`
+- **Status:** ❌ Not running
+
+**4. Network Access**
+- **Ping:** ✅ Working
+- **Port 22:** ✅ Open
+- **Port 3000:** ⏳ (Gitea web UI)
+- **Port 2222:** ⏳ (Gitea SSH)
+
+**5. Service Access**
+- **Gitea Web UI:** http://192.168.1.121:3000
+- **Status:** Docker Compose ready, needs deployment
+
+#### VM 103 - Observability (192.168.1.82)
+
+**1. SSH Access**
+- **Command:** `ssh -i ~/.ssh/id_ed25519_proxmox ubuntu@192.168.1.82`
+- **Status:** ❌ Not working (authentication failing)
+- **Alternative:** Use Proxmox console
+
+**2. Proxmox Console**
+- **Method:** Web UI → VM 103 → Console
+- **Status:** ✅ Available
+
+**3. QEMU Guest Agent**
+- **Command:** `qm guest exec 103 -- <command>`
+- **Status:** ❌ Not running
+
+**4. Network Access**
+- **Ping:** ✅ Working
+- **Port 22:** ✅ Open
+- **Port 3000:** ⏳ (Grafana)
+- **Port 9090:** ⏳ (Prometheus)
+
+**5. Service Access**
+- **Grafana:** http://192.168.1.82:3000
+- **Prometheus:** http://192.168.1.82:9090
+- **Status:** Docker Compose ready, needs deployment
+
+---
+
+## 🔐 Authentication Methods
+
+### Proxmox Hosts
+
+**1. Root Password**
+- **Location:** `.env` file (PVE_ROOT_PASS)
+- **Use:** Web UI, API, SSH (if password auth enabled)
+- **Status:** ✅ Available
+
+**2. SSH Key**
+- **File:** `~/.ssh/id_ed25519_proxmox`
+- **Public Key:** `~/.ssh/id_ed25519_proxmox.pub`
+- **Status:** ✅ Working on ML110, ❌ Not on R630
+
+**3. API Tokens**
+- **Status:** ⏳ Not created yet
+- **Use:** Automation scripts
+- **Create:** Via Web UI or API
+
+### Virtual Machines
+
+**1. SSH Key (Cloud-init)**
+- **Status:** ⏳ Added via API but not working
+- **Fix:** Manual setup via console
+
+**2. Password Authentication**
+- **Status:** ⏳ Unknown (may be disabled)
+- **Enable:** Via console or cloud-init
+
+**3. Console Access**
+- **Status:** ✅ Available via Proxmox Web UI
+- **Use:** Initial setup, troubleshooting
+
+---
+
+## 🌐 Network Access Paths
+
+### Internal Network (192.168.1.0/24)
+
+**Gateway:** 192.168.1.254
+
+**Accessible Hosts:**
+- ✅ 192.168.1.206 (ML110 Proxmox) - SSH, Web UI, API
+- ✅ 192.168.1.49 (R630 Proxmox) - Web UI, API (SSH pending)
+- ✅ 192.168.1.60 (VM 100) - Ping, Port 22 open
+- ✅ 192.168.1.188 (VM 101) - Ping, Port 22 open
+- ✅ 192.168.1.121 (VM 102) - Ping, Port 22 open
+- ✅ 192.168.1.82 (VM 103) - Ping, Port 22 open
+
+### VLAN Networks (10.10.x.0/24)
+
+**VLAN 10 (Storage):** 10.10.10.0/24
+- Gateway: 10.10.10.1
+- **Status:** ⏳ NFS server not reachable
+
+**VLAN 20 (Compute):** 10.10.20.0/24
+- Gateway: 10.10.20.1
+- **Status:** ⏳ Configured but not in use
+
+**VLAN 30 (App Tier):** 10.10.30.0/24
+- Gateway: 10.10.30.1
+- **Status:** ⏳ Configured but not in use
+
+**VLAN 40 (Observability):** 10.10.40.0/24
+- Gateway: 10.10.40.1
+- **Status:** ⏳ Configured but not in use
+
+**VLAN 50 (Dev/Test):** 10.10.50.0/24
+- Gateway: 10.10.50.1
+- **Status:** ⏳ Configured but not in use
+
+**VLAN 60 (Management):** 10.10.60.0/24
+- Gateway: 10.10.60.1
+- **Status:** ⏳ Configured but not in use
+
+**VLAN 99 (DMZ):** 10.10.99.0/24
+- Gateway: 10.10.99.1
+- **Status:** ⏳ Configured but not in use
+
+---
+
+## 📦 Storage Access
+
+### Local Storage
+
+**ML110:**
+- **local:** Directory storage (100GB available)
+- **local-lvm:** LVM thin pool (832GB available)
+- **Access:** Via Proxmox Web UI or SSH
+
+**R630:**
+- **Status:** Unknown (not accessible via SSH)
+- **Access:** Via Web UI or API
+
+### NFS Storage
+
+**Server:** 10.10.10.1
+- **Path:** /mnt/storage
+- **Status:** ❌ Not reachable
+- **Access:** ⏳ Pending server availability
+
+---
+
+## 🔧 Troubleshooting Access Paths
+
+### When SSH to VMs Fails
+
+**Option 1: Proxmox Console**
+1. Access Proxmox Web UI
+2. Select VM → Console
+3. Login with ubuntu user
+4. Configure SSH manually
+
+**Option 2: QEMU Guest Agent**
+1. Install qemu-guest-agent in VM (via console)
+2. Use `qm guest exec` commands
+3. Execute commands remotely
+
+**Option 3: Cloud-init Reconfiguration**
+1. Update cloud-init config via API
+2. Reboot VM
+3. Cloud-init applies new configuration
+
+### When SSH to Proxmox Host Fails
+
+**Option 1: Web UI**
+- All management via Web UI
+- Console access to VMs
+- File uploads/downloads
+
+**Option 2: API**
+- Automation scripts
+- Status queries
+- Configuration changes
+
+**Option 3: Physical/Console**
+- Direct access to host
+- Recovery scenarios
+
+### When Network Access Fails
+
+**Option 1: Proxmox Console**
+- Access VM console
+- Check network configuration
+- Troubleshoot from inside VM
+
+**Option 2: QEMU Guest Agent**
+- Query network interfaces
+- Check IP configuration
+- Execute network commands
+
+**Option 3: VM Console via Web UI**
+- Direct console access
+- No network required
+
+---
+
+## 🎯 Access Path Priority Matrix
+
+### For VM Management
+
+**Priority 1:** Proxmox Web UI (always available)
+**Priority 2:** SSH to Proxmox host (working on ML110)
+**Priority 3:** Proxmox API (working)
+**Priority 4:** SSH to VMs (needs fix)
+**Priority 5:** QEMU Guest Agent (needs agent installation)
+
+### For Service Configuration
+
+**Priority 1:** SSH to VMs (needs fix)
+**Priority 2:** Proxmox Console (available)
+**Priority 3:** QEMU Guest Agent (needs agent installation)
+**Priority 4:** Service Web UIs (when services running)
+
+### For Troubleshooting
+
+**Priority 1:** Proxmox Console (direct access)
+**Priority 2:** SSH to Proxmox host (for logs)
+**Priority 3:** QEMU Guest Agent (for VM internals)
+**Priority 4:** Network tools (ping, port scans)
+
+---
+
+## 📋 Quick Reference
+
+### Working Access Methods
+
+✅ **Proxmox ML110:**
+- Web UI: https://192.168.1.206:8006
+- SSH: `ssh -i ~/.ssh/id_ed25519_proxmox root@192.168.1.206`
+- API: https://192.168.1.206:8006/api2/json
+
+✅ **All VMs:**
+- Console: Via Proxmox Web UI
+- Network: All reachable via ping
+- Port 22: All open
+
+❌ **Not Working:**
+- SSH to VMs (authentication failing)
+- SSH to R630 (authentication failing)
+- QEMU Guest Agent (not installed in VMs)
+- NFS storage (server not reachable)
+
+---
+
+## 🔄 Alternative Access Strategies
+
+### Strategy 1: Console-First Approach
+1. Use Proxmox console for all VM access
+2. Configure SSH keys manually
+3. Install QEMU Guest Agent
+4. Then use SSH for automation
+
+### Strategy 2: API-Only Approach
+1. Use Proxmox API for all operations
+2. Deploy services via cloud-init
+3. Use service APIs when available
+4. Minimal SSH dependency
+
+### Strategy 3: Hybrid Approach
+1. Use console for initial setup
+2. Use SSH once configured
+3. Use API for automation
+4. Use QEMU Guest Agent for remote execution
+
+---
+
+**Status:** All access paths mapped. Use this guide to identify alternative methods when primary access fails.
+