Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
3.1 KiB
3.1 KiB
Critical Findings and Fixes - Validator Issues
Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation
Date: 2025-01-20
Status: ✅ ISSUES IDENTIFIED AND BEING FIXED
Critical Finding #1: Validators on Different Proxmox Host
Discovery
- Validators 1000-1002: Found on Proxmox host
r630-01(NOT onml110) - Location: Config files at
/etc/pve/nodes/r630-01/lxc/1000.conf, etc. - Impact: Validators exist but may not be accessible from current Proxmox host
Status
- IPs are reachable (192.168.11.100-102)
- Network connectivity confirmed (ping successful)
- Containers exist on different Proxmox host
Critical Finding #2: Genesis File Path Issue
Problem
Validators 1003-1004 are FAILING to start:
Unable to load genesis file. java.io.FileNotFoundException: /genesis/genesis.json (No such file or directory)
Root Cause
- Besu service configured to look for genesis at
/genesis/genesis.json - Actual genesis file located at
/etc/besu/genesis.json - Path mismatch causing service failures
Solution Applied
- Created directory
/genesison validators 1003-1004 - Created symlink:
/genesis/genesis.json→/etc/besu/genesis.json - Restarted Besu validator services
- Monitoring for successful startup
Fixes Applied
Fix 1: Genesis File Path
# On each validator (1003, 1004)
mkdir -p /genesis
ln -sf /etc/besu/genesis.json /genesis/genesis.json
systemctl restart besu-validator.service
Fix 2: Validator Status Check
- Identified validators 1000-1002 on r630-01
- Verified network connectivity
- Checked service status
Next Steps
Immediate
- ✅ Fix genesis file path (completed)
- ✅ Restart validators 1003-1004 (completed)
- ⏳ Verify validators start successfully
- ⏳ Check status of validators 1000-1002 on r630-01
- ⏳ Ensure all 5 validators are running
- ⏳ Monitor block production resumption
Follow-up
- Monitor block production for 2+ minutes
- Check transaction confirmations
- Verify all validators are participating in consensus
- Complete remaining deployments once blocks resume
Expected Outcomes
After Genesis Fix
- Validators 1003-1004 should start successfully
- No more "genesis file not found" errors
- Services should be active
With All Validators Running
- Block production should resume
- Consensus should reach quorum (3+ of 5 validators)
- Transactions should start confirming
- Network should return to normal operation
Verification
Check Validator Services
# On Proxmox host ml110
for vmid in 1003 1004; do
pct exec $vmid -- systemctl is-active besu-validator.service
done
Check Validator Logs
# Should see successful startup, no genesis errors
journalctl -u besu-validator.service -n 20
Check Block Production
# Monitor block number
watch -n 5 "cast block-number --rpc-url http://192.168.11.211:8545"
Status: Critical fixes applied, monitoring for results
Priority: High - Block production depends on validators running correctly