Sankofa/docs/infrastructure/CEPH_SETUP_ISSUE.md

Ceph Setup Issue - Configuration Mismatch

Date: 2025-12-15
Issue: Cannot add r630-01 monitor due to configuration mismatch

Problem

When attempting to add r630-01 as a Ceph monitor, we discovered:

1. Different FSIDs:

   • ML110-01 cluster: f601f0e2-cd09-402f-9e15-4b1c9a7a7b25 (from earlier status)
   • R630-01 config: 021d20d3-446f-42cb-a219-5e01213b7b2d (from current /etc/pve/ceph.conf)

2. Network Configuration Mismatch:

   • ML110-01: 192.168.11.0/24
   • R630-01: 192.168.11.11/24 (incorrect - should be 192.168.11.0/24)

3. Monitor Connection Issues:

   • ML110-01 monitor is running but commands timeout
   • Cannot get monmap from ml110-01
   • R630-01 cannot connect to cluster

Root Cause

The /etc/pve/ceph.conf on r630-01 appears to have a different/incorrect configuration, possibly from a previous cluster setup or misconfiguration. The Proxmox cluster filesystem should sync this, but there may be a conflict.

Solution Options

Option 1: Sync Configuration from ML110-01

Since Proxmox cluster filesystem should sync /etc/pve/ceph.conf, we need to ensure:

1. ML110-01 has the correct configuration
2. Configuration syncs to r630-01
3. Then add r630-01 monitor

Option 2: Reinitialize Ceph Cluster

If the cluster on ml110-01 is not operational or has issues:

1. Stop Ceph on ml110-01
2. Reinitialize with proper configuration
3. Add r630-01 monitor
4. Add OSDs

Option 3: Use Proxmox Web UI

The Proxmox web interface may handle the configuration and monitor addition more reliably.

Current Status

• ML110-01: Monitor running but cluster commands timeout
• R630-01: Cannot connect to cluster, configuration mismatch
• Need to resolve configuration before proceeding