proxmox/docs/archive/issues/OUTSTANDING_ISSUES_RESOLUTION_GUIDE.md

# Outstanding Issues Resolution Guide

**Date:** 2026-01-03  
**Status:** Requires Physical/Console Access

---

## Summary

Two servers require physical or console access to complete setup:

1. **r630-03** (192.168.11.13) - Network connectivity issue
2. **r630-04** (192.168.11.14) - SSH password authentication failure

---

## Issue 1: r630-03 Network Connectivity

### Symptoms
- Host does not respond to ping
- "Destination Host Unreachable" error
- Cannot access via SSH or web interface

### Diagnosis
```bash
# From your machine
ping -c 3 192.168.11.13
# Result: 100% packet loss, Destination Host Unreachable
```

### Possible Causes
1. **Server powered off**
   - Check physical power status
   - Check power LED indicators
   - Verify power cable connections

2. **Network cable disconnected**
   - Check physical network cable
   - Verify cable is connected to correct switch port
   - Test with known-good cable

3. **Network switch port issue**
   - Check switch port status
   - Verify port is enabled
   - Check for port errors on switch

4. **IP configuration issue**
   - Server may have different IP
   - Network configuration may be incorrect
   - DHCP may have assigned different IP

5. **Network routing issue**
   - Check router/switch configuration
   - Verify VLAN configuration
   - Check firewall rules

### Resolution Steps

#### Step 1: Physical Inspection
1. **Check Power Status**
   - Verify server is powered on
   - Check power LED indicators
   - Listen for fan noise

2. **Check Network Cable**
   - Verify cable is connected
   - Check for physical damage
   - Try different cable if available

3. **Check Network Switch**
   - Verify switch port is active
   - Check for port errors
   - Verify VLAN configuration

#### Step 2: Access via iDRAC (Dell R630)
If iDRAC is configured, access via:
- **iDRAC IP:** Check network configuration or use default
- **Default credentials:** root/calvin (if not changed)
- **Access method:** Web browser to iDRAC IP

Once in iDRAC:
1. Check server power status
2. Access console (KVM)
3. Check network configuration
4. Verify IP address assignment

#### Step 3: Console Access
If physical access is available:
1. Connect keyboard/monitor
2. Boot server if needed
3. Check network configuration:
   ```bash
   ip addr show
   cat /etc/network/interfaces
   ```
4. Verify IP address: `192.168.11.13`
5. Test connectivity: `ping 192.168.11.1`

#### Step 4: Network Configuration Fix
If IP is incorrect or network is misconfigured:
```bash
# Edit network configuration
nano /etc/network/interfaces

# Ensure configuration matches:
# auto vmbr0
# iface vmbr0 inet static
#     address 192.168.11.13/24
#     gateway 192.168.11.1
#     bridge-ports <physical-interface>
#     bridge-vlan-aware yes

# Restart networking
systemctl restart networking
```

#### Step 5: Verification
Once accessible:
```bash
# Run verification script
./scripts/verify-r630-03-cluster-storage.sh

# Check cluster membership
ssh root@192.168.11.13 "pvecm status"

# Verify storage
ssh root@192.168.11.13 "pvesm status"
```

---

## Issue 2: r630-04 SSH Password Authentication

### Symptoms
- Host is reachable (ping works)
- SSH connection attempts fail with "Permission denied"
- All known passwords fail
- Web interface not accessible (port 8006)

### Diagnosis
```bash
# Host is reachable
ping -c 2 192.168.11.14
# Result: Success

# SSH fails
ssh root@192.168.11.14
# Result: Permission denied (password authentication)
```

### Tried Passwords
- ❌ L@kers2010
- ❌ password
- ❌ L@kers2010!
- ❌ L@kers2010@
- ❌ L@kers2010#
- ❌ All variations tested

### Resolution Steps

#### Step 1: Console Access (Required)
Physical or iDRAC console access is required to reset password.

**Option A: Physical Console**
1. Connect keyboard/monitor to r630-04
2. Boot server if needed
3. Login with current password (if known)
4. Or boot into single-user mode

**Option B: iDRAC Console**
1. Access iDRAC web interface
2. Use Remote Console (KVM)
3. Access server console remotely

#### Step 2: Reset Root Password

**Method 1: If you can login**
```bash
# Login to console
# Then run:
passwd root
# Enter new password: L@kers2010
# Confirm password
```

**Method 2: Single-User Mode (if password unknown)**
1. Boot server
2. At GRUB menu, press 'e' to edit
3. Find line starting with "linux"
4. Add `init=/bin/bash` or `single` to end of line
5. Press Ctrl+X to boot
6. Mount filesystem: `mount -o remount,rw /`
7. Reset password: `passwd root`
8. Reboot: `reboot -f`

#### Step 3: Fix Proxmox Services

Once password is reset and SSH works:
```bash
# Run the complete fix script
./scripts/fix-r630-04-complete.sh L@kers2010

# Or manually:
ssh root@192.168.11.14

# Fix /etc/hosts
echo "192.168.11.14    r630-04 r630-04.sankofa.nexus" >> /etc/hosts

# Restart Proxmox services
systemctl restart pve-cluster
systemctl restart pvestatd pvedaemon pveproxy

# Verify
systemctl status pveproxy
ss -tlnp | grep 8006
curl -k https://localhost:8006
```

#### Step 4: Join Cluster (if not already)

If r630-04 is not in cluster:
```bash
# From ml110, get join information
ssh root@192.168.11.10 "pvecm create join-info"

# On r630-04, join cluster
ssh root@192.168.11.14
pvecm add <join-info-from-ml110>
```

#### Step 5: Verify Storage

```bash
# Check storage status
ssh root@192.168.11.14 "pvesm status"

# Activate storage if needed
# (Similar to r630-01 and r630-02)
```

---

## Alternative Access Methods

### iDRAC Access (Dell R630)

Both r630-03 and r630-04 are Dell R630 servers with iDRAC.

**Default iDRAC Configuration:**
- **IP Range:** Usually 192.168.0.x or DHCP assigned
- **Default User:** root
- **Default Password:** calvin (if not changed)

**Finding iDRAC IP:**
1. Check network documentation
2. Check DHCP leases on router
3. Check Omada controller for connected devices
4. Use default IP range: 192.168.0.100-200

**Accessing iDRAC:**
1. Open web browser
2. Navigate to iDRAC IP
3. Login with credentials
4. Use Remote Console (KVM) for full access

### Network-Based Solutions

**Option 1: Check from Working Nodes**
```bash
# From ml110, r630-01, or r630-02
ssh root@192.168.11.10
ping 192.168.11.13  # r630-03
ping 192.168.11.14  # r630-04

# If reachable from cluster network, try SSH
ssh root@192.168.11.14
```

**Option 2: Check Network Switch**
- Access switch management interface
- Check port status for servers
- Verify VLAN configuration
- Check for port errors

**Option 3: Check Router/Firewall**
- Verify firewall rules
- Check routing tables
- Verify NAT configuration

---

## Verification Checklist

### r630-03 Checklist
- [ ] Server is powered on
- [ ] Network cable connected
- [ ] Network switch port active
- [ ] IP address correct (192.168.11.13)
- [ ] Ping works from cluster network
- [ ] SSH access works
- [ ] Proxmox services running
- [ ] Web interface accessible (port 8006)
- [ ] Cluster membership verified
- [ ] Storage configuration verified

### r630-04 Checklist
- [ ] Console access obtained
- [ ] Root password reset
- [ ] SSH access works
- [ ] /etc/hosts configured correctly
- [ ] Proxmox services running
- [ ] Web interface accessible (port 8006)
- [ ] Cluster membership verified
- [ ] Storage configuration verified

---

## Scripts Available

All scripts are in `/home/intlc/projects/proxmox/scripts/`:

1. **`verify-r630-03-cluster-storage.sh`**
   - Run once r630-03 is accessible
   - Verifies cluster membership and storage

2. **`fix-r630-04-complete.sh`**
   - Run once r630-04 password is reset
   - Complete fix for all r630-04 issues

3. **`verify-all-nodes-complete.sh`**
   - Run after both issues are resolved
   - Comprehensive verification of all nodes

---

## Next Steps

### Immediate Actions
1. **Physical Access Required**
   - Access r630-03 and r630-04 via console/iDRAC
   - Resolve network/password issues
   - Run verification scripts

2. **Documentation Update**
   - Update passwords in documentation
   - Update status reports
   - Document any configuration changes

### After Resolution
1. **Run Verification**
   ```bash
   ./scripts/verify-all-nodes-complete.sh
   ```

2. **Update Status Report**
   - Mark issues as resolved
   - Update cluster membership
   - Update storage status

3. **Cluster Health Check**
   - Verify all 5 nodes in cluster
   - Check quorum status
   - Verify storage on all nodes

---

## Support Resources

- **Console Access Guide:** `R630-04-CONSOLE-ACCESS-GUIDE.md`
- **Troubleshooting Guide:** `R630-04-PROXMOX-TROUBLESHOOTING.md`
- **Status Report:** `docs/PROXMOX_CLUSTER_STORAGE_STATUS_REPORT.md`
- **Hardware Inventory:** `config/physical-hardware-inventory.md`

---

**Last Updated:** 2026-01-03  
**Status:** Awaiting Physical/Console Access