proxmox/scripts/verify/README.md

Verification Scripts

Scripts for ingress, NPMplus, DNS, and source-of-truth verification.

Dependencies

Required tools (install before running):

Tool	Purpose	Install
bash	Shell (4.0+)	Default on most systems
curl	API calls, HTTP	apt install curl
jq	JSON parsing	apt install jq
dig	DNS resolution	apt install dnsutils
openssl	SSL certificate inspection	apt install openssl
ssh	Remote execution	apt install openssh-client
ss	Port checking	apt install iproute2
systemctl	Service status	System (systemd)
sqlite3	Database backup	apt install sqlite3

Optional (recommended for automation): sshpass, rsync, screen, tmux, htop, shellcheck, parallel. See docs/11-references/APT_PACKAGES_CHECKLIST.md § Automation / jump host.
One-line install (Debian/Ubuntu): sudo apt install -y sshpass rsync dnsutils iproute2 screen tmux htop shellcheck parallel

Tool	Purpose
wscat or websocat	WebSocket testing (manual verification)

Scripts

• backup-npmplus.sh - Full NPMplus backup (database, API exports, certificates)
• check-contracts-on-chain-138.sh - Check that Chain 138 deployed contracts have bytecode on-chain (cast code for 31 addresses; requires cast and RPC access). Use [RPC_URL] or env RPC_URL_138; --dry-run lists addresses only (no RPC calls); SKIP_EXIT=1 to exit 0 when RPC unreachable.
• check-non-evm-network-health.sh - Read-only live check for the public Solana, Tron, and XRPL endpoints used in repo docs. Prints a concise status table and can also write reports/status/non-evm-network-health-latest.json.
• build-non-evm-lane-status.py - Builds reports/status/non-evm-lane-status-latest.{json,md} from the Solana lineup manifest plus the current on-chain inventory/completion matrix, so network health and Chain 138 explorer/source closure stay clearly separated.
• build-unified-extended-from-csv.py - Rebuilds reports/inventory/DEPLOYED_CONTRACTS_UNIFIED_EXTENDED.md from reports/inventory/deployed-contracts-by-network.csv using the pipe-row format consumed by inventory_onchain.py.
• snapshot-mainnet-cwusdc-usdc-preflight.sh - Read-only preflight snapshot for the Mainnet cWUSDC/USDC rail. Captures public-pair drift, defended DODO reserves, treasury-manager quote availability, receiver surplus, and defended-lane quote sizing into reports/status/.
• plan-mainnet-cwusdc-usdc-repeg.sh - Read-only repeg planner for the Mainnet cWUSDC/USDC rail. Consumes the latest preflight snapshot, computes defended-pool reserve-gap sizing, public-pair shortfalls, operator-wallet coverage, and emits copy-paste operator commands into reports/status/.
• build-cw-mesh-deployment-matrix.sh - Read-only merge of cross-chain-pmm-lps/config/deployment-status.json and reports/extraction/promod-uniswap-v2-live-pair-discovery-latest.json into a per-chain table (stdout markdown; optional --json-out reports/status/cw-mesh-deployment-matrix-latest.json). No RPC. Invoked from run-all-validation.sh when the discovery JSON is present.
• reconcile-env-canonical.sh - Emit recommended .env lines for Chain 138 (canonical source of truth); use to reconcile smom-dbis-138/.env with CONTRACT_ADDRESSES_REFERENCE. Usage: ./scripts/verify/reconcile-env-canonical.sh [--print]
• check-deployer-balance-blockscout-vs-rpc.sh - Compare deployer native balance from Blockscout API vs RPC (to verify index matches current chain); see EXPLORER_AND_BLOCKSCAN_REFERENCE
• check-dependencies.sh - Verify required tools (bash, curl, jq, openssl, ssh)
• check-cluster-besu-inventory.sh - Cluster-wide Besu inventory audit using pvesh /cluster/resources via a Proxmox cluster node so host placement on r630-03 / r630-04 is not missed. Prints VMID, type, node, status, name, IP, canonical-vs-extra classification, and any missing canonical VMIDs. Use --json for machine-readable output.
• check-gitea-certificate-expiry.sh - Read-only TLS expiry check for gitea.d-bis.org (or another host passed as arg). Exits 0 when outside the warning window, 1 when within WARN_DAYS (default 14), and 2 on expiry or probe failure.

Scheduling

• Install a daily cron check with bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install
• Print the cron line with bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --show
• check-pnpm-workspace-lockfile.sh - Ensures every path in pnpm-workspace.yaml has an importer in pnpm-lock.yaml (run pnpm install at root if it fails; avoids broken pnpm outdated -r)
• export-cloudflare-dns-records.sh - Export Cloudflare DNS records
• export-npmplus-config.sh - Export NPMplus proxy hosts and certificates via API
• generate-source-of-truth.sh - Combine verification outputs into canonical JSON
• run-full-verification.sh - Run full verification suite
• stitch_unified_from_read_output_chunks.py - Reassembles reports/inventory/DEPLOYED_CONTRACTS_UNIFIED_EXTENDED.md from IDE Read-tool text chunks by stripping line prefixes and omitted-line markers.
• verify-backend-vms.sh - Verify backend VMs (status, IPs, nginx configs)
• verify-end-to-end-routing.sh - E2E routing verification
• verify-udm-pro-port-forwarding.sh - UDM Pro port forwarding checks
• verify-websocket.sh - WebSocket connectivity test (requires websocat or wscat)

Task runners (no LAN vs from LAN)

• From anywhere (no LAN/creds): ../run-completable-tasks-from-anywhere.sh — runs config validation, on-chain contract check, run-all-validation.sh --skip-genesis (includes cW* mesh matrix when reports/extraction/promod-uniswap-v2-live-pair-discovery-latest.json exists), non-EVM status refresh, and reconcile-env-canonical. Use --json-out reports/status/run-completable-tasks-latest.json to capture a machine-readable run summary. On Gitea, the same run-all-validation gate runs on push (in deploy-to-phoenix before deploy) and on PRs (.gitea/workflows/validate-on-pr.yml only, no deploy).
• Validation summary JSON: run-all-validation.sh --json-out reports/status/run-all-validation-latest.json writes per-step status and timing, which is useful for comparing repeated dry-runs or CI-like local checks.
• Other wrapper summary JSON: ../run-e2e-flow-tasks-full-parallel.sh --dry-run --json-out reports/status/run-e2e-flow-tasks-latest.json, ../deployment/run-all-next-steps-chain138.sh --dry-run --json-out reports/status/run-all-next-steps-chain138-latest.json, and ../run-all-operator-tasks-from-lan.sh --dry-run --json-out reports/status/run-all-operator-tasks-latest.json write comparable machine-readable summaries for the main orchestration wrappers.
• From LAN (NPM_PASSWORD, optional PRIVATE_KEY): ../run-operator-tasks-from-lan.sh — runs W0-1 (NPMplus RPC fix), W0-3 (NPMplus backup), O-1 (Blockscout verification); use --dry-run to print commands only. See ALL_TASKS_DETAILED_STEPS.

Environment

Set variables in .env (from .env.master.example at repo root) or export before running. docs/04-configuration/VERIFICATION_GAPS_AND_TODOS.md. NPM NPM_EMAIL + NPM_PASSWORD (see that template’s NPM / NPMplus section) are required for backup-npmplus.sh API steps.