Sankofa/docs/ceph/FRESH_INSTALL_SETUP.md

Fresh Ceph Setup on Proxmox VE

Prerequisites

• Proxmox VE installed on both nodes
• Nodes are in a Proxmox cluster (or standalone)
• Network connectivity between nodes (192.168.11.0/24)
• Root access to both nodes

Node Information

• ML110-01: ml110-01.sankofa.nexus (192.168.11.10)
• R630-01: r630-01.sankofa.nexus (192.168.11.11)

Step-by-Step Setup

Step 1: Install Ceph on Both Nodes

Run on both nodes:

apt-get update
apt-get install -y ceph ceph-common ceph-base

Step 2: Initialize Ceph Cluster (ML110-01 only)

Run on ml110-01:

pveceph init --network 192.168.11.0/24 --cluster-network 192.168.11.0/24

This will:

• Create /etc/pve/ceph.conf
• Generate cluster FSID
• Set up bootstrap keyrings

Step 3: Create Monitors

Run on ml110-01:

pveceph mon create

Run on r630-01:

pveceph mon create

Step 4: Verify Quorum

Run on either node:

ceph quorum_status
ceph -s

You should see both monitors in quorum.

Step 5: Create OSDs

On ML110-01

Identify available drives:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE

Create OSD on available drive (replace /dev/sdX with actual drive):

ceph-volume lvm create --data /dev/sdX

On R630-01

Create OSDs on the 6x 250GB drives (sdc, sdd, sde, sdf, sdg, sdh):

for disk in sdc sdd sde sdf sdg sdh; do
    echo "Creating OSD on /dev/$disk..."
    ceph-volume lvm create --data /dev/$disk
    sleep 2
done

Step 6: Verify Cluster Health

Run on either node:

ceph -s
ceph osd tree
ceph health detail

Step 7: Create Storage Pools

Create RBD Pool (for VM disks)

ceph osd pool create rbd 32 32
ceph osd pool application enable rbd rbd
rbd pool init rbd

Create CephFS (for shared storage)

ceph osd pool create cephfs_data 32 32
ceph osd pool create cephfs_metadata 16 16
ceph fs new ceph-fs cephfs_metadata cephfs_data

Step 8: Add Storage to Proxmox

Add Ceph RBD Storage

In Proxmox Web UI:

1. Datacenter → Storage → Add → RBD
2. ID: ceph-rbd
3. Pool: rbd
4. Nodes: Select both nodes
5. Add

Add CephFS Storage

In Proxmox Web UI:

1. Datacenter → Storage → Add → CephFS
2. ID: ceph-fs
3. FS Name: ceph-fs
4. Nodes: Select both nodes
5. Add

Verification Checklist

• Both monitors in quorum
• At least 3 OSDs created (for replication factor 3)
• Cluster health is HEALTH_OK or HEALTH_WARN (not HEALTH_ERR)
• Storage pools created (rbd, cephfs_data, cephfs_metadata)
• CephFS filesystem created
• Storage added to Proxmox Web UI

Troubleshooting

Monitors Not Forming Quorum

1. Check monitor services:

   systemctl status ceph-mon@$(hostname -s)
   

2. Check monitor logs:

   journalctl -u ceph-mon@$(hostname -s) -n 50
   

3. Verify network connectivity:

   ping 192.168.11.10
   ping 192.168.11.11
   

4. Check firewall:

   # Allow Ceph ports
   ufw allow 6789/tcp  # Classic msgr
   ufw allow 3300/tcp  # msgr2
   

OSD Creation Fails

1. Check if drive is available:

   lsblk /dev/sdX
   

2. Wipe drive if needed:

   wipefs -a /dev/sdX
   

3. Check for existing LVM volumes:

   pvs | grep sdX
   vgs | grep ceph
   

Cluster Health Issues

1. Check detailed health:

   ceph health detail
   

2. Check OSD status:

   ceph osd tree
   ceph osd stat
   

3. Check placement groups:

   ceph pg stat
   

Next Steps

After Ceph is set up:

1. Fix PG allocation: Ensure pools have appropriate PG counts
2. Create VMs: Use Ceph storage for VM disks
3. Monitor cluster: Set up monitoring and alerts
4. Backup configuration: Backup /etc/pve/ceph.conf and keyrings

Important Notes

• Minimum OSDs: For osd_pool_default_size = 3, you need at least 3 OSDs
• PG Count: Use calculator: https://ceph.io/pgcalc/
• Network: Ensure cluster network has sufficient bandwidth
• Backup: Always backup Ceph configuration and keyrings