# Ceph OSD Creation Issues - Summary

**Date**: 2025-12-13  
**Status**: 🔍 **ISSUES IDENTIFIED - NEEDS DIAGNOSIS**

---

## Issues Identified

### 1. OSD Creation Hanging/Timeout

**Symptom**: `ceph-volume lvm create` hangs at "osd new" step

**Root Causes** (likely):
- ❌ **Bootstrap keyring missing** - Cannot authenticate with cluster
- ❌ **Ceph cluster not accessible** - RADOS timeout errors
- ❌ **Monitor connectivity issues** - Cannot reach Ceph monitors

### 2. Bootstrap Keyring Issue

**Error**: 
```
unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
RADOS timed out (error connecting to the cluster)
```

**Possible Locations**:
- `/var/lib/ceph/bootstrap-osd/ceph.keyring` (standard)
- `/etc/pve/priv/ceph.client.bootstrap-osd.keyring` (Proxmox)

**Status**: Unknown - needs verification

### 3. Ceph Cluster Connectivity

**Issue**: Commands timeout when trying to connect to cluster

**Possible Causes**:
- Monitors not running
- Network connectivity issues
- Firewall blocking ports
- Cluster configuration issues

---

## Diagnostic Steps

Run these commands on R630-01 to diagnose:

### Quick Check

```bash
# 1. Check Ceph health (does it respond?)
ceph health

# 2. Check monitors
ceph mon stat

# 3. Check bootstrap keyring
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring

# 4. Check if pveceph available
which pveceph
```

### Full Diagnostic

```bash
# Run comprehensive diagnostic
bash /tmp/diagnose-ceph-issues.sh
```

---

## Likely Solutions

### Solution 1: Use pveceph (If Proxmox-Managed)

If `pveceph` is available:

```bash
for drive in sdc sdd sde sdf sdg sdh; do
    pveceph create /dev/$drive
done
```

### Solution 2: Fix Bootstrap Keyring

If cluster is accessible:

```bash
# Get bootstrap keyring
mkdir -p /var/lib/ceph/bootstrap-osd
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

# Then create OSDs
for drive in sdc sdd sde sdf sdg sdh; do
    ceph-volume lvm create --data /dev/$drive
done
```

### Solution 3: Fix Cluster Connectivity

If cluster is not accessible:

1. Check monitor services
2. Check network connectivity
3. Check firewall rules
4. Fix connectivity issues
5. Then retry OSD creation

---

## Current State

- ✅ **6x 250GB drives**: Wiped and ready
- ❌ **OSD creation**: Hanging due to authentication/connectivity
- 🔍 **Root cause**: Needs diagnosis

---

## Next Action

**Run diagnostic script** to understand the issue:

```bash
# On R630-01
bash /tmp/diagnose-ceph-issues.sh
```

This will show:
- Ceph cluster status
- Bootstrap keyring locations
- Monitor connectivity
- Service status
- Network connectivity

Then we can apply the appropriate fix.

---

**Last Updated**: 2025-12-13