proxmox/scripts/verify/README.md

# Verification Scripts

Scripts for ingress, NPMplus, DNS, and source-of-truth verification.

## Dependencies

Required tools (install before running):

| Tool | Purpose | Install |
|------|---------|---------|
| `bash` | Shell (4.0+) | Default on most systems |
| `curl` | API calls, HTTP | `apt install curl` |
| `jq` | JSON parsing | `apt install jq` |
| `dig` | DNS resolution | `apt install dnsutils` |
| `openssl` | SSL certificate inspection | `apt install openssl` |
| `ssh` | Remote execution | `apt install openssh-client` |
| `ss` | Port checking | `apt install iproute2` |
| `systemctl` | Service status | System (systemd) |
| `sqlite3` | Database backup | `apt install sqlite3` |

Optional (recommended for automation): `sshpass`, `rsync`, `screen`, `tmux`, `htop`, `shellcheck`, `parallel`. See [docs/11-references/APT_PACKAGES_CHECKLIST.md](../../docs/11-references/APT_PACKAGES_CHECKLIST.md) § Automation / jump host.  
One-line install (Debian/Ubuntu): `sudo apt install -y sshpass rsync dnsutils iproute2 screen tmux htop shellcheck parallel`

| Tool | Purpose |
|------|---------|
| `wscat` or `websocat` | WebSocket testing (manual verification) |

## Scripts

- `backup-npmplus.sh` - Full NPMplus backup (database, API exports, certificates)
- `check-contracts-on-chain-138.sh` - Check that Chain 138 deployed contracts have bytecode on-chain (`cast code` for 31 addresses; requires `cast` and RPC access). Use `[RPC_URL]` or env `RPC_URL_138`; `--dry-run` lists addresses only (no RPC calls); `SKIP_EXIT=1` to exit 0 when RPC unreachable.
- `check-non-evm-network-health.sh` - Read-only live check for the public Solana, Tron, and XRPL endpoints used in repo docs. Prints a concise status table and can also write `reports/status/non-evm-network-health-latest.json`.
- `build-non-evm-lane-status.py` - Builds `reports/status/non-evm-lane-status-latest.{json,md}` from the Solana lineup manifest plus the current on-chain inventory/completion matrix, so network health and Chain 138 explorer/source closure stay clearly separated.
- `snapshot-mainnet-cwusdc-usdc-preflight.sh` - Read-only preflight snapshot for the Mainnet `cWUSDC/USDC` rail. Captures public-pair drift, defended DODO reserves, treasury-manager quote availability, receiver surplus, and defended-lane quote sizing into `reports/status/`.
- `plan-mainnet-cwusdc-usdc-repeg.sh` - Read-only repeg planner for the Mainnet `cWUSDC/USDC` rail. Consumes the latest preflight snapshot, computes defended-pool reserve-gap sizing, public-pair shortfalls, operator-wallet coverage, and emits copy-paste operator commands into `reports/status/`.
- `build-cw-mesh-deployment-matrix.sh` - Read-only merge of `cross-chain-pmm-lps/config/deployment-status.json` and `reports/extraction/promod-uniswap-v2-live-pair-discovery-latest.json` into a per-chain table (stdout markdown; optional `--json-out reports/status/cw-mesh-deployment-matrix-latest.json`). No RPC. Invoked from `run-all-validation.sh` when the discovery JSON is present.
- `reconcile-env-canonical.sh` - Emit recommended .env lines for Chain 138 (canonical source of truth); use to reconcile `smom-dbis-138/.env` with [CONTRACT_ADDRESSES_REFERENCE](../../docs/11-references/CONTRACT_ADDRESSES_REFERENCE.md). Usage: `./scripts/verify/reconcile-env-canonical.sh [--print]`
- `check-deployer-balance-blockscout-vs-rpc.sh` - Compare deployer native balance from Blockscout API vs RPC (to verify index matches current chain); see [EXPLORER_AND_BLOCKSCAN_REFERENCE](../../docs/11-references/EXPLORER_AND_BLOCKSCAN_REFERENCE.md)
- `check-dependencies.sh` - Verify required tools (bash, curl, jq, openssl, ssh)
- `check-cluster-besu-inventory.sh` - Cluster-wide Besu inventory audit using `pvesh /cluster/resources` via a Proxmox cluster node so host placement on `r630-03` / `r630-04` is not missed. Prints VMID, type, node, status, name, IP, canonical-vs-extra classification, and any missing canonical VMIDs. Use `--json` for machine-readable output.
- `check-pnpm-workspace-lockfile.sh` - Ensures every path in `pnpm-workspace.yaml` has an `importer` in `pnpm-lock.yaml` (run `pnpm install` at root if it fails; avoids broken `pnpm outdated -r`)
- `export-cloudflare-dns-records.sh` - Export Cloudflare DNS records
- `export-npmplus-config.sh` - Export NPMplus proxy hosts and certificates via API
- `generate-source-of-truth.sh` - Combine verification outputs into canonical JSON
- `run-full-verification.sh` - Run full verification suite
- `verify-backend-vms.sh` - Verify backend VMs (status, IPs, nginx configs)
- `verify-end-to-end-routing.sh` - E2E routing verification
- `verify-udm-pro-port-forwarding.sh` - UDM Pro port forwarding checks
- `verify-websocket.sh` - WebSocket connectivity test (requires websocat or wscat)

## Task runners (no LAN vs from LAN)

- **From anywhere (no LAN/creds):** `../run-completable-tasks-from-anywhere.sh` — runs config validation, on-chain contract check, `run-all-validation.sh --skip-genesis` (includes cW* mesh matrix when `reports/extraction/promod-uniswap-v2-live-pair-discovery-latest.json` exists), non-EVM status refresh, and reconcile-env-canonical. Use `--json-out reports/status/run-completable-tasks-latest.json` to capture a machine-readable run summary. On Gitea, the same `run-all-validation` gate runs on **push** (in `deploy-to-phoenix` before deploy) and on **PRs** (`.gitea/workflows/validate-on-pr.yml` only, no deploy).
- **Validation summary JSON:** `run-all-validation.sh --json-out reports/status/run-all-validation-latest.json` writes per-step status and timing, which is useful for comparing repeated dry-runs or CI-like local checks.
- **Other wrapper summary JSON:** `../run-e2e-flow-tasks-full-parallel.sh --dry-run --json-out reports/status/run-e2e-flow-tasks-latest.json`, `../deployment/run-all-next-steps-chain138.sh --dry-run --json-out reports/status/run-all-next-steps-chain138-latest.json`, and `../run-all-operator-tasks-from-lan.sh --dry-run --json-out reports/status/run-all-operator-tasks-latest.json` write comparable machine-readable summaries for the main orchestration wrappers.
- **From LAN (NPM_PASSWORD, optional PRIVATE_KEY):** `../run-operator-tasks-from-lan.sh` — runs W0-1 (NPMplus RPC fix), W0-3 (NPMplus backup), O-1 (Blockscout verification); use `--dry-run` to print commands only. See [ALL_TASKS_DETAILED_STEPS](../../docs/00-meta/ALL_TASKS_DETAILED_STEPS.md).

## Environment

Set variables in `.env` (from `.env.master.example` at repo root) or export before running. [docs/04-configuration/VERIFICATION_GAPS_AND_TODOS.md](../../docs/04-configuration/VERIFICATION_GAPS_AND_TODOS.md). NPM `NPM_EMAIL` + `NPM_PASSWORD` (see that template’s NPM / NPMplus section) are required for `backup-npmplus.sh` API steps.