57c717ad7e
Some checks failed
AI Code Review / claude-review (pull_request) Failing after 48s
Part of the sequenced cleanup tracked in issue #1. Scaffolding only — no rotation executed, no secret values committed. - docs/runbooks/PHOENIX_VAULT_ROTATION_RUNBOOK.md: authoritative Phoenix Vault rotation procedure (9-step: new root → rekey unseal → regenerate AppRoles → flip consumers → revoke old). Verification table + rollback path + Phase 2 handoff notes. - docs/04-configuration/VAULT_SHARD_CUSTODY_POLICY.md: decision record for the next rotation. Three options documented (named-operator / cloud-KMS auto-unseal / Transit auto-unseal); selection pending operator sign-off before rotation executes. - scripts/verify/enumerate-vault-consumers.sh: read-only grep over the tree for VAULT_ROLE_ID / VAULT_SECRET_ID / auth/approle/login references; flags which top-level consumers need a coordinated .env update at §1.6 of the runbook. - scripts/verify/verify-vault-approle-auth.sh: post-rotation sanity check — posts AppRole login + token lookup-self; returns PASS/FAIL without echoing the Role ID, Secret ID, or client token. - phoenix-deploy-api/.env.example: added VAULT_ADDR / VAULT_ROLE_ID / VAULT_SECRET_ID placeholder block with a pointer to the runbook. No values committed. - mission-control/.env.example: NEW file (previously had none); documents the launchpad NEXT_PUBLIC_* vars and the same Vault AppRole placeholder block. Server-side only — never NEXT_PUBLIC_*. Rotation execution stays with Phoenix ops; this commit only stages the runbook + env scaffolding so the eventual rotation does not require inventing infrastructure mid-incident. Co-Authored-By: Nakamoto, S <defi@defi-oracle.io>
Verification Scripts
Scripts for ingress, NPMplus, DNS, and source-of-truth verification.
Dependencies
Required tools (install before running):
| Tool | Purpose | Install |
|---|---|---|
bash |
Shell (4.0+) | Default on most systems |
curl |
API calls, HTTP | apt install curl |
jq |
JSON parsing | apt install jq |
dig |
DNS resolution | apt install dnsutils |
openssl |
SSL certificate inspection | apt install openssl |
ssh |
Remote execution | apt install openssh-client |
ss |
Port checking | apt install iproute2 |
systemctl |
Service status | System (systemd) |
sqlite3 |
Database backup | apt install sqlite3 |
Optional (recommended for automation): sshpass, rsync, screen, tmux, htop, shellcheck, parallel. See docs/11-references/APT_PACKAGES_CHECKLIST.md § Automation / jump host.
One-line install (Debian/Ubuntu): sudo apt install -y sshpass rsync dnsutils iproute2 screen tmux htop shellcheck parallel
| Tool | Purpose |
|---|---|
wscat or websocat |
WebSocket testing (manual verification) |
Scripts
backup-npmplus.sh- Full NPMplus backup (database, API exports, certificates)check-contracts-on-chain-138.sh- Check that Chain 138 deployed contracts have bytecode on-chain (cast codefor 31 addresses; requirescastand RPC access). Use[RPC_URL]or envRPC_URL_138;--dry-runlists addresses only (no RPC calls);SKIP_EXIT=1to exit 0 when RPC unreachable.reconcile-env-canonical.sh- Emit recommended .env lines for Chain 138 (canonical source of truth); use to reconcilesmom-dbis-138/.envwith CONTRACT_ADDRESSES_REFERENCE. Usage:./scripts/verify/reconcile-env-canonical.sh [--print]check-deployer-balance-blockscout-vs-rpc.sh- Compare deployer native balance from Blockscout API vs RPC (to verify index matches current chain); see EXPLORER_AND_BLOCKSCAN_REFERENCEcheck-dependencies.sh- Verify required tools (bash, curl, jq, openssl, ssh)export-cloudflare-dns-records.sh- Export Cloudflare DNS recordsexport-npmplus-config.sh- Export NPMplus proxy hosts and certificates via APIgenerate-source-of-truth.sh- Combine verification outputs into canonical JSONrun-full-verification.sh- Run full verification suiteverify-backend-vms.sh- Verify backend VMs (status, IPs, nginx configs)verify-end-to-end-routing.sh- E2E routing verificationverify-udm-pro-port-forwarding.sh- UDM Pro port forwarding checksverify-websocket.sh- WebSocket connectivity test (requires websocat or wscat)
Task runners (no LAN vs from LAN)
- From anywhere (no LAN/creds):
../run-completable-tasks-from-anywhere.sh— runs config validation, on-chain contract check, run-all-validation --skip-genesis, and reconcile-env-canonical. - From LAN (NPM_PASSWORD, optional PRIVATE_KEY):
../run-operator-tasks-from-lan.sh— runs W0-1 (NPMplus RPC fix), W0-3 (NPMplus backup), O-1 (Blockscout verification); use--dry-runto print commands only. See ALL_TASKS_DETAILED_STEPS.
Environment
Set variables in .env or export before running. See project root .env.example and docs/04-configuration/VERIFICATION_GAPS_AND_TODOS.md.