Sankofa/docs/infrastructure/CEPH_CURRENT_STATE.md

Ceph Cluster Current State - sfvalley-01

Date: 2025-12-15 15:18 PST
Status: ⚠️ CLUSTER NOT FULLY OPERATIONAL

Summary

The Ceph cluster on ml110-01 exists but is not responding to commands. The monitor is running but in "probing" state and cannot form quorum, which prevents:

• Adding r630-01 monitor
• Creating OSDs
• Querying cluster status

Current Configuration

Both Nodes (Synced via Proxmox cluster)

[global]
	auth_client_required = cephx
	auth_cluster_required = cephx
	auth_service_required = cephx
	cluster_network = 192.168.11.0/24
	fsid = 021d20d3-446f-42cb-a219-5e01213b7b2d
	mon_allow_pool_delete = true
	mon_host = 192.168.11.10 192.168.11.11
	ms_bind_ipv4 = true
	ms_bind_ipv6 = false
	osd_pool_default_min_size = 2
	osd_pool_default_size = 3
	public_network = 192.168.11.0/24

[client]
	keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
	keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.ml110-01]
	public_addr = 192.168.11.10

[mon.r630-01]
	public_addr = 192.168.11.11

Service Status

ML110-01

• ✅ ceph-mon@ml110-01.service: ACTIVE (running)
• ⚠️ Monitor state: "probing" (cannot form quorum)
• ❌ Ceph commands: Timeout (cluster not accessible)

R630-01

• ❌ ceph-mon@r630-01.service: FAILED (filesystem not created)
• ⚠️ Cannot create monitor without monmap from ml110-01
• ⚠️ Cannot create OSDs without cluster access

Available Disks for OSDs (R630-01)

• /dev/sdc: 232.9G (CT250MX500SSD1)
• /dev/sdd: 232.9G (CT250MX500SSD1)
• /dev/sde: 232.9G (CT250MX500SSD1)
• /dev/sdf: 232.9G (CT250MX500SSD1)
• /dev/sdg: 232.9G (CT250MX500SSD1)
• /dev/sdh: 232.9G (CT250MX500SSD1)

Total Available: ~1.4TB across 6 SSDs

Blocking Issues

1. Monitor Quorum Not Established

   • ML110-01 monitor in "probing" state
   • Cannot form quorum with single monitor
   • R630-01 monitor cannot be added without quorum

2. Cluster Commands Timeout

   • ceph -s: Timeout
   • ceph mon getmap: Timeout
   • ceph quorum_status: Timeout
   • Cannot query or manage cluster

3. Monitor Filesystem Not Created

   • R630-01 monitor directory exists but is empty
   • Needs monmap and keyring from ml110-01
   • Cannot create without cluster access

Recommended Solutions

Option 1: Use Proxmox Web UI (Recommended)

The Proxmox web interface may handle the monitor and OSD creation more reliably:

1. Access Proxmox web UI
2. Navigate to: Datacenter > Ceph > Monitors
3. Add r630-01 monitor through UI
4. Navigate to: Datacenter > Ceph > OSDs
5. Create OSDs on r630-01 disks through UI

Option 2: Reinitialize Ceph Cluster

If the cluster is not critical or can be reinitialized:

1. Stop Ceph on ml110-01
2. Remove existing Ceph data
3. Reinitialize cluster with proper configuration
4. Add r630-01 monitor
5. Add OSDs

Option 3: Fix Monitor Quorum

If there's existing data that must be preserved:

1. Investigate why ml110-01 monitor is in "probing" state
2. Check monitor logs for errors
3. Fix underlying issue preventing quorum
4. Then add r630-01 monitor

Next Steps

1. Investigate Monitor Issue: Check why ml110-01 monitor cannot form quorum
2. Use Web UI: Try adding monitor and OSDs through Proxmox web interface
3. Or Reinitialize: If acceptable, reinitialize cluster with proper setup

Notes

• Configuration is correct and synced across cluster
• Network connectivity is working (ping successful)
• Monitor ports are open (6789, 3300)
• Issue appears to be with monitor quorum formation
• May require cluster reinitialization or web UI approach