Sankofa/docs/ceph/MON_RECOVERY_SUCCESS.md

Ceph MON Recovery - Success

Date: 2025-12-13
Status: ✅ RECOVERED

Recovery Summary

Successfully recovered Ceph cluster by recreating the corrupted MON on r630-01.

What Was Done

1. Identified Root Cause

• Hardware Failure: /dev/sda on r630-01 has critical medium errors
• Symptom: Corrupted RocksDB database in /var/lib/ceph/mon/ceph-r630-01/store.db/
• Impact: MON could not start, cluster lost quorum, OSDs could not start

2. Recovery Steps Executed

1. Stopped and disabled failed MON service

   systemctl stop ceph-mon@r630-01
   systemctl disable ceph-mon@r630-01
   

2. Backed up corrupted MON directory

   • Created backup: /root/ceph-mon-r630-01-corrupted-*.tar.gz
   • Note: Backup showed file corruption (file shrank during tar)

3. Removed corrupted MON directory

   rm -rf /var/lib/ceph/mon/ceph-r630-01
   

4. Created new monmap from configuration

   monmaptool --create --fsid 5fb968ae-12ab-405f-b05f-0df29a168328 \
     --add ml110-01 192.168.11.10 \
     --add r630-01 192.168.11.11 \
     /tmp/monmap
   

5. Copied keyring from ml110-01

   • Source: /var/lib/ceph/mon/ceph-ml110-01/keyring
   • Destination: /tmp/mon-keyring on r630-01

6. Created new MON filesystem

   mkdir -p /var/lib/ceph/mon/ceph-r630-01
   chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
   ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
   

7. Fixed ownership and started MON

   chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
   systemctl enable ceph-mon@r630-01
   systemctl start ceph-mon@r630-01
   

8. Restarted OSDs

   • ceph-osd@1 on r630-01: ✅ Running
   • ceph-osd@0 on ml110-01: ✅ Running

Current Status

Services

• ✅ ceph-mon@r630-01.service: active (running)
• ✅ ceph-mon@ml110-01.service: active (running) (was already running)
• ✅ ceph-osd@1.service: active (running)
• ✅ ceph-osd@0.service: active (running)

Cluster

• Quorum: Restored (2/2 MONs)
• OSDs: 2/2 running
• Cluster Access: Restored (commands no longer timeout)

Remaining Issues

1. Hardware Failure (CRITICAL)

• Disk: /dev/sda on r630-01 still has critical medium errors
• Risk: MON database may corrupt again if disk continues to fail
• Action Required:
  • Monitor disk health: smartctl -a /dev/sda
  • Plan disk replacement
  • Consider moving MON to different disk if available

2. Ceph Health Issues (Still Present)

• OSD Count: 2 OSDs < required 3 (osd_pool_default_size = 3)
• Undersized PGs: PG 3.7f stuck undersized for 32h+
• PG Over-allocation: 288 PGs per OSD (max 250), RBD pool has 64 PGs (should be 32)

3. Password Security

• Root password was exposed in command history
• Action Required: Rotate root password immediately

Next Steps

1. Monitor cluster health: ceph -s and ceph health detail
2. Address disk failure: Check SMART status, plan replacement
3. Add third OSD: Use the 6x 250GB drives on r630-01 (once disk issue is resolved)
4. Fix PG allocation: Reduce RBD pool PGs from 64 to 32
5. Rotate passwords: Change root password on all nodes

Lessons Learned

1. Hardware monitoring is critical: Disk failures can cascade to application layer
2. MON quorum requires redundancy: With only 2 MONs, losing one breaks the cluster
3. Ownership matters: MON directory must be owned by ceph:ceph
4. Recovery is possible: Even with corrupted database, MON can be recreated from peer

Recovery Time

• Start: 19:10 PST
• MON Running: 19:24 PST
• OSDs Running: 19:25 PST
• Total: ~15 minutes