# Troubleshooting VM 9000 Creation - I/O Errors

## Error Summary

The VM creation failed with multiple I/O errors when reading from the source image:
- `qemu-img: error while reading at byte 130023424: Input/output error`
- Transfer stopped at ~23% (138.0 MiB of 597.2 MiB)

## Root Causes

1. **Corrupted source image file** - The uploaded image may be damaged
2. **Disk I/O issues on Proxmox host** - Storage problems on the Proxmox node
3. **File location mismatch** - File may be in wrong location or format
4. **Incomplete upload** - File transfer may have been interrupted

## Diagnostic Steps

### 1. Check File Integrity on Proxmox Host

SSH into your Proxmox host and run:

```bash
# Check if file exists and its size
ls -lh /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw
ls -lh /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img

# Check file integrity (if file is readable)
file /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw
file /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img

# Try to read file metadata
qemu-img info /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw 2>&1
qemu-img info /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img 2>&1
```

### 2. Check Disk Health

```bash
# Check disk space
df -h /var/lib/vz

# Check for disk errors
dmesg | grep -i error | tail -20
dmesg | grep -i "i/o error" | tail -20

# Check storage pool health
pvesm status
lvdisplay | grep -A 10 "pve"
```

### 3. Verify File Checksum (if original available)

If you have the original file, compare checksums:

```bash
# On your local machine (if you have the original)
sha256sum ubuntu-24.04-server-cloudimg-amd64.img

# On Proxmox host
sha256sum /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw
sha256sum /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img
```

## Quick Fix Script

**Automated fix (recommended):**
```bash
./scripts/fix-corrupted-image.sh
```

This script will:
1. Verify your local image (or download if missing)
2. Remove corrupted files on Proxmox host
3. Upload a fresh copy via SCP
4. Verify the uploaded image

## Solutions

### Solution 1: Re-upload the Image (Recommended)

1. **Delete the corrupted file** (on Proxmox host):
```bash
rm -f /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw
rm -f /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img
```

2. **Re-download the image** (on your local machine):
```bash
cd /home/intlc/projects/loc_az_hci
./scripts/download-ubuntu-cloud-image.sh 24.04
```

3. **Upload via Proxmox Web UI**:
   - Go to: **Datacenter** → **local** → **Content** → **Upload**
   - Select: `downloads/ubuntu-24.04-server-cloudimg-amd64.img`
   - Wait for upload to complete
   - Verify file appears in storage

4. **Verify upload** (on Proxmox host):
```bash
qemu-img info /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img
```

5. **Retry VM creation** using the steps in `CREATE_VM_9000_STEPS.md`

### Solution 2: Use API/CLI to Upload (Alternative)

If Web UI upload fails, use command line:

```bash
# On Proxmox host, copy file to correct location
scp ubuntu-24.04-server-cloudimg-amd64.img root@<proxmox-ip>:/var/lib/vz/template/iso/

# Or use Proxmox API (from local machine with API access)
# See scripts/create-template-via-api.sh
```

### Solution 3: Download Directly on Proxmox Host

```bash
# SSH into Proxmox host
cd /var/lib/vz/template/iso

# Download directly
wget https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img

# Verify
qemu-img info ubuntu-24.04-server-cloudimg-amd64.img
```

### Solution 4: Use Different Storage Location

If `local` storage has issues, try a different storage:

1. **Check available storage**:
```bash
pvesm status
```

2. **Upload to different storage** (e.g., `local-lvm` or NFS)

3. **Create VM using different storage** in the Disks tab

### Solution 5: Check and Fix Storage Issues

If disk I/O errors persist:

```bash
# Check LVM status
vgdisplay
lvdisplay

# Check for filesystem errors
fsck -n /dev/pve/root  # Dry run, don't fix yet

# If errors found, schedule filesystem check on next reboot
touch /forcefsck
reboot
```

## Prevention

1. **Always verify uploads**: Check file size and integrity after upload
2. **Use checksums**: Compare SHA256 checksums before and after upload
3. **Monitor disk health**: Regularly check `dmesg` for I/O errors
4. **Use reliable storage**: Prefer local-lvm or NFS over local if available

## Alternative: Create VM from ISO Instead

If cloud image continues to fail, use ISO installation method:

1. Download Ubuntu Server ISO
2. Upload ISO to Proxmox
3. Create VM with ISO attached
4. Install Ubuntu manually
5. Configure Cloud-Init
6. Convert to template

See `scripts/create-vms-from-iso.sh` for automation.

## Next Steps After Fix

Once the image is successfully uploaded and verified:

1. Follow `CREATE_VM_9000_STEPS.md` to create VM 9000
2. Configure Cloud-Init settings
3. Convert to template
4. Verify template works by cloning a test VM

## Verification Scripts

After fixing the issue, verify everything is working:

```bash
# Verify image integrity on Proxmox host
./scripts/verify-proxmox-image.sh

# Or manually check (SSH into Proxmox)
qemu-img info /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img
```

## Related Files

- `CREATE_VM_9000_STEPS.md` - Main creation steps
- `scripts/fix-corrupted-image.sh` - **Automated fix script (use this first!)**
- `scripts/verify-proxmox-image.sh` - Image verification script
- `scripts/download-ubuntu-cloud-image.sh` - Download script
- `scripts/create-proxmox-template.sh` - Template creation script
- `docs/runbooks/proxmox-operations.md` - General Proxmox operations