3.3 KiB
3.3 KiB
Ceph Cluster Instability - ML110-01
Date: 2025-12-13
Status: ⚠️ INVESTIGATING
Issue Summary
Symptoms
- Ceph Monitor: Constantly in "electing" state
- Slow Operations: 100+ slow operations reported
- Manager Instability: Ceph manager constantly restarting
- Connection Issues: Socket connections closing between mon0 and mon1
- Quorum Problems: Cluster appears to be having election issues
Log Patterns
- mon.ml110-01@0(electing) e2 get_health_metrics reporting 100+ slow ops
- ENGINE Bus STOPPING/STARTING (constant restarts)
- libceph: mon0/mon1 socket closed (con state OPEN)
- Session lost, hunting for new mon
Impact Analysis
VMs Affected
- Storage Type:
ceph-fs - Count: Multiple VMs configured to use ceph-fs
- Risk: VM creation may fail if Ceph is unavailable
Potential Issues
- VM Creation Failures: VMs using ceph-fs may fail to create
- Storage Unavailable: Ceph storage may be inaccessible
- Performance Degradation: Slow operations indicate cluster stress
- Data Risk: Instability could affect data integrity
Root Cause Analysis
Possible Causes
- Network Issues: Connection problems between mon0 (ML110-01) and mon1 (R630-01)
- Quorum Loss: Cluster may have lost quorum
- Resource Exhaustion: System resources may be exhausted
- Configuration Issues: Ceph configuration may be incorrect
- Time Sync: Clock synchronization issues
Investigation Steps
1. Check Ceph Cluster Health
ssh [email protected] "ceph -s"
ssh [email protected] "ceph health detail"
2. Verify Quorum Status
ssh [email protected] "ceph quorum_status"
3. Check Monitor Status
ssh [email protected] "ceph mon stat"
ssh [email protected] "ceph mon dump"
4. Check Network Connectivity
# From ML110-01 to R630-01
ssh [email protected] "ping -c 3 192.168.11.11"
ssh [email protected] "telnet 192.168.11.11 6789"
5. Check System Resources
ssh [email protected] "df -h"
ssh [email protected] "free -h"
ssh [email protected] "iostat -x 1 3"
Immediate Actions
1. Verify Impact on VM Creation
- Check if any VMs have failed due to storage issues
- Monitor provider logs for Ceph-related errors
- Verify if VM creation is blocked
2. Stabilize Ceph Cluster
- Check quorum status
- Verify network connectivity
- Restart Ceph services if needed
- Check for configuration issues
3. Alternative Storage
- Consider using
local-lvmfor critical VMs temporarily - Document which VMs require ceph-fs
- Plan migration if Ceph remains unstable
Recommendations
Short-term
- Monitor Ceph Health: Continuously monitor cluster status
- Check Network: Verify connectivity between nodes
- Review Logs: Analyze Ceph logs for root cause
- Resource Check: Verify system resources are adequate
Long-term
- Ceph Optimization: Optimize Ceph configuration
- Network Hardening: Ensure stable network between nodes
- Monitoring: Set up Ceph health monitoring
- Backup Strategy: Ensure data backup strategy is in place
Status
⚠️ INVESTIGATING - Ceph cluster instability detected, impact on VM deployment being assessed.
Last Updated: 2025-12-13
Status: ⚠️ INVESTIGATING