Files
proxmox/docs/06-besu/CRITICAL_FINDINGS_AND_FIXES.md
defiQUG fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
docs: Ledger Live integration, contract deploy learnings, NEXT_STEPS updates
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands
- CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround
- CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check
- NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere
- MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates
- LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-12 15:46:57 -08:00

3.1 KiB

Critical Findings and Fixes - Validator Issues

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation


Date: 2025-01-20
Status: ISSUES IDENTIFIED AND BEING FIXED


Critical Finding #1: Validators on Different Proxmox Host

Discovery

  • Validators 1000-1002: Found on Proxmox host r630-01 (NOT on ml110)
  • Location: Config files at /etc/pve/nodes/r630-01/lxc/1000.conf, etc.
  • Impact: Validators exist but may not be accessible from current Proxmox host

Status

  • IPs are reachable (192.168.11.100-102)
  • Network connectivity confirmed (ping successful)
  • Containers exist on different Proxmox host

Critical Finding #2: Genesis File Path Issue

Problem

Validators 1003-1004 are FAILING to start:

Unable to load genesis file. java.io.FileNotFoundException: /genesis/genesis.json (No such file or directory)

Root Cause

  • Besu service configured to look for genesis at /genesis/genesis.json
  • Actual genesis file located at /etc/besu/genesis.json
  • Path mismatch causing service failures

Solution Applied

  1. Created directory /genesis on validators 1003-1004
  2. Created symlink: /genesis/genesis.json/etc/besu/genesis.json
  3. Restarted Besu validator services
  4. Monitoring for successful startup

Fixes Applied

Fix 1: Genesis File Path

# On each validator (1003, 1004)
mkdir -p /genesis
ln -sf /etc/besu/genesis.json /genesis/genesis.json
systemctl restart besu-validator.service

Fix 2: Validator Status Check

  • Identified validators 1000-1002 on r630-01
  • Verified network connectivity
  • Checked service status

Next Steps

Immediate

  1. Fix genesis file path (completed)
  2. Restart validators 1003-1004 (completed)
  3. Verify validators start successfully
  4. Check status of validators 1000-1002 on r630-01
  5. Ensure all 5 validators are running
  6. Monitor block production resumption

Follow-up

  1. Monitor block production for 2+ minutes
  2. Check transaction confirmations
  3. Verify all validators are participating in consensus
  4. Complete remaining deployments once blocks resume

Expected Outcomes

After Genesis Fix

  • Validators 1003-1004 should start successfully
  • No more "genesis file not found" errors
  • Services should be active

With All Validators Running

  • Block production should resume
  • Consensus should reach quorum (3+ of 5 validators)
  • Transactions should start confirming
  • Network should return to normal operation

Verification

Check Validator Services

# On Proxmox host ml110
for vmid in 1003 1004; do
  pct exec $vmid -- systemctl is-active besu-validator.service
done

Check Validator Logs

# Should see successful startup, no genesis errors
journalctl -u besu-validator.service -n 20

Check Block Production

# Monitor block number
watch -n 5 "cast block-number --rpc-url http://192.168.11.211:8545"

Status: Critical fixes applied, monitoring for results
Priority: High - Block production depends on validators running correctly