Sankofa/docs/ceph/RECOVERY_PLAN.md

Ceph MON Recovery Plan - R630-01

Current Status

✅ ML110-01 MON: Running (but in "probing" state - no quorum)
❌ R630-01 MON: Failed (corrupted RocksDB due to failing disk)
❌ Cluster Quorum: Lost (need 2/2 MONs for quorum)

Root Cause Summary

1. Hardware: /dev/sda on r630-01 has critical medium errors
2. Symptom: Ceph MON RocksDB database corrupted (001572.sst I/O error)
3. Impact: MON cannot start, cluster has no quorum, OSDs cannot start

Recovery Strategy

Since ml110-01 MON is running, we can recreate the r630-01 MON. The MON will sync cluster state from ml110-01.

Prerequisites

1. Fix or replace failing disk /dev/sda on r630-01 (or move MON to different disk)
2. Backup current state (already documented)
3. Verify ml110-01 MON is accessible from r630-01

Recovery Steps

Step 1: Stop Failed MON Service

ssh [email protected] 'systemctl stop ceph-mon@r630-01'
ssh [email protected] 'systemctl disable ceph-mon@r630-01'

Step 2: Backup Corrupted MON Directory

ssh [email protected] 'tar -czf /root/ceph-mon-r630-01-corrupted-$(date +%Y%m%d-%H%M%S).tar.gz /var/lib/ceph/mon/ceph-r630-01/'

Step 3: Remove Corrupted MON

ssh [email protected] 'pveceph mon destroy r630-01'

Alternative if pveceph fails:

ssh [email protected] 'rm -rf /var/lib/ceph/mon/ceph-r630-01'

Step 4: Recreate MON on R630-01

ssh [email protected] 'pveceph mon create r630-01'

The MON will:

• Create a new RocksDB database
• Sync cluster state from ml110-01
• Rejoin the quorum

Step 5: Verify Recovery

# Check MON status
ssh [email protected] 'systemctl status ceph-mon@r630-01'

# Check cluster quorum
ssh [email protected] 'ceph quorum_status'

# Check cluster health
ssh [email protected] 'ceph -s'

Step 6: Restart OSDs

Once quorum is restored:

ssh [email protected] 'systemctl restart ceph-osd@1'
ssh [email protected] 'systemctl restart ceph-osd@0'

Disk Issue Resolution

Option A: Move MON to Different Disk (If Available)

If r630-01 has another disk with space:

1. Create new directory on different disk: /mnt/other-disk/ceph-mon/
2. Update MON service to use new location
3. Recreate MON in new location

Option B: Replace Failing Disk

1. Replace /dev/sda with new disk
2. Restore Proxmox installation
3. Recreate MON

Option C: Fix Filesystem (If Disk is Recoverable)

1. Run fsck on filesystem
2. Check SMART status: smartctl -a /dev/sda
3. If disk is recoverable, proceed with MON recreation

Risk Assessment

• Low Risk: Recreating MON from healthy peer (ml110-01) is standard procedure
• Medium Risk: If ml110-01 MON also fails during recovery, cluster becomes unrecoverable
• High Risk: If disk continues to fail, MON will corrupt again

Post-Recovery Actions

1. Monitor disk health: Set up SMART monitoring for /dev/sda
2. Add third MON: Consider adding a third MON on a different node for redundancy
3. Backup MON data: Regularly backup /var/lib/ceph/mon/ directories
4. Monitor cluster: Watch for recurring I/O errors

Notes

• Password Security: Root password was exposed. Rotate immediately.
• Quorum: With 2 MONs, both must be healthy. Consider adding a third MON.
• OSD Recovery: OSDs will automatically recover once MON quorum is restored.