394 lines
25 KiB
Markdown
394 lines
25 KiB
Markdown
# Operator Ready Checklist — Copy-Paste Commands
|
||
|
||
**Last Updated:** 2026-05-11
|
||
**Purpose:** Single page with exact commands to complete every pending todo. Run from **repo root** on a host with **LAN** access (and `smom-dbis-138/.env` with `PRIVATE_KEY`, `NPM_PASSWORD` where noted).
|
||
|
||
**Do you have all necessary creds?** See [OPERATOR_CREDENTIALS_CHECKLIST.md](OPERATOR_CREDENTIALS_CHECKLIST.md) — per-task list of LAN, PRIVATE_KEY, NPM_PASSWORD, RPC_URL_138, SSH, LINK, gas, token balance.
|
||
|
||
**From anywhere (no LAN):** `./scripts/run-completable-tasks-from-anywhere.sh`
|
||
|
||
**Submodule working trees (CI gate):** `bash scripts/verify/submodules-clean.sh` — fails on any submodule porcelain; see [SUBMODULE_HYGIENE.md](SUBMODULE_HYGIENE.md). **2026-05-11:** **`the-order`** pinned on **`main`** (`8990270` — portal CT **10090** at **192.168.11.180**). If the gate still fails, clean **`explorer-monorepo`** and **`smom-dbis-138`** working trees (commit or stash in each submodule) before CI/push.
|
||
|
||
**Ensure this machine always has Proxmox SSH access:** `./scripts/security/ensure-proxmox-ssh-access.sh` (verifies key-based SSH to .10, .11, .12; use `--copy` to install key if missing). **NPMplus from this machine (if direct 192.168.11.167:81 unreachable):** `ssh -L 8181:192.168.11.167:81 -N root@192.168.11.11` then use `http://127.0.0.1:8181` for NPMplus API.
|
||
|
||
**If deployer needs gas on currently active public chains:** Run `./scripts/deployment/deployer-gas-auto-route.sh` (optional: `--dry-run`, `--chain 138`). See [DEPLOYER_GAS_AUTO_ROUTE_RUNBOOK.md](../03-deployment/DEPLOYER_GAS_AUTO_ROUTE_RUNBOOK.md). **Current policy:** Wemix is deferred.
|
||
|
||
**Current live execution path:** [LIVE_SESSION_CRONOS_AND_TIER1_PHASE_C.md](../03-deployment/LIVE_SESSION_CRONOS_AND_TIER1_PHASE_C.md) — close Cronos config + LINK, then activate Tier 1 Phase C on Gnosis, Polygon, and BSC. **Current priority docs:** [FULLY_OPERATIONAL_EXECUTION_CHECKLIST.md](FULLY_OPERATIONAL_EXECUTION_CHECKLIST.md), [PHASE_C_PROFIT_FIRST_PRIORITY.md](../03-deployment/PHASE_C_PROFIT_FIRST_PRIORITY.md), [PHASE_C_TIER1_EXECUTION_TASK_SHEET.md](../03-deployment/PHASE_C_TIER1_EXECUTION_TASK_SHEET.md).
|
||
|
||
**Chain 138 txpool incident standard path:** `bash scripts/fix-all-validators-and-txpool.sh` then `bash scripts/maintenance/apply-chain138-strict-future-tx-pool.sh` then `bash scripts/clear-all-transaction-pools.sh` then `bash scripts/monitoring/monitor-blockchain-health.sh`. Use this sequence when block production stalls, pending hashes keep reappearing, or future-nonce residue survives a normal txpool clear.
|
||
|
||
**Gitea HTTPS push safeguard:** `bash scripts/verify/check-gitea-certificate-expiry.sh` (optional: `WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh`). Use this when Git over HTTPS starts failing, or run it proactively before major push/deploy windows. To keep it checked automatically: `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install`.
|
||
|
||
---
|
||
|
||
## Completed / verified (2026-05-11)
|
||
|
||
| Item | Result |
|
||
|------|--------|
|
||
| DealFlow public | DNS **`dealflow`** in **`scripts/update-all-dns-to-public-ip.sh`**; NPM proxy **`dealflow.d-bis.org`** → **`https://192.168.11.94:443`** (`IP_OM_TREASURY_DEALFLOW`); Let’s Encrypt via **`request-npmplus-certificates.sh`**; backend **`CORS_ORIGIN`** on CT **10381** must include **`https://dealflow.d-bis.org`**. **`verify-dealflow-public-e2e.sh`** default URL **5/5**. Repo commit on **`master`**: DealFlow DNS + NPM updater (`update-npmplus-proxy-hosts-api.sh` optional **`forward_scheme`** for **`add_proxy_host`**). |
|
||
| NPM Wave 0 | **`bash scripts/run-wave0-from-lan.sh --skip-backup`** — **41** hosts updated, **0** failed (includes DealFlow + Studio). Use **`bash`** if **`./scripts/run-wave0-from-lan.sh`** is not executable. |
|
||
| NPMplus CT **10233** | If **`:81`** accepts TCP but returns no HTTP: **`pct reboot 10233`** on **`r630-01`**. **`POST /api/tokens`** — use **`https://192.168.11.167:81`** (plain **`http://`** can return **Payload is undefined**). |
|
||
| Studio | **`fusionai-creator-api`** on **7805** must be **Up**; **`docker update --restart unless-stopped`** on that container if it exits. |
|
||
| Completable ladder | **`run-completable-tasks-from-anywhere.sh`** exit **0** (config, **61/61** on-chain, **`run-all-validation.sh --skip-genesis`**, non-EVM, reconcile). |
|
||
|
||
---
|
||
|
||
## Completed in this session (2026-05-10)
|
||
|
||
| Item | Result |
|
||
|------|--------|
|
||
| Submodule hygiene + pushes | Committed and pushed **smom-dbis-138**, **explorer-monorepo** (rebased onto **Gitea `origin/master`** then pushed), **metamask-integration**, **cross-chain-pmm-lps**, **atomic-swap-dapp**; parent **`proxmox-cp`** **`master`** pushed (**includes** submodule pointer commits ending **`685a5d2`**). **`bash scripts/verify/submodules-clean.sh`** passed when **`the-order`** was clean. |
|
||
| No-LAN ladder | **`run-completable-tasks-from-anywhere.sh`** OK — [reports/status/run-completable-tasks-latest.json](../../reports/status/run-completable-tasks-latest.json). |
|
||
| Public E2E | **`verify-end-to-end-routing.sh --profile=public`** exit **0**; evidence [verification-evidence/e2e-verification-20260510_125658](../04-configuration/verification-evidence/e2e-verification-20260510_125658/verification_report.md); umbrella snapshot listed **`studio.sankofa.nexus`** **warn** (**502** on `/studio/`). |
|
||
| DealFlow E2E (prior) | LAN-only pass with **`DEALFLOW_SKIP_TLS_VERIFY=1`** before public hostname was fully wired. |
|
||
|
||
---
|
||
|
||
## Completed in this session (2026-03-26)
|
||
|
||
| Item | Result |
|
||
|------|--------|
|
||
| NPMplus recovery | VMID `10233` was wedged on `192.168.11.167:81` (TCP connect, no HTTP). `pct reboot 10233` on `r630-01` restored the expected `301` response on port `81`. |
|
||
| NPMplus API updater | `NPM_URL=https://192.168.11.167:81 bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh` completed with **39 hosts updated, 0 failed**. |
|
||
| Sankofa / Order / Studio routing | **Superseded 2026-03-27:** Order hostnames default to **order-haproxy** `http://192.168.11.39:80` (10210 → `.51:3000`). Through 2026-03-26 NPM pointed Order directly at portal `:3000`. `studio.sankofa.nexus` → `http://192.168.11.72:8000`. |
|
||
| Public E2E | Latest run `bash scripts/verify/verify-end-to-end-routing.sh --profile=public` exited `0` with **Failed: 0**, **DNS passed: 37**, **HTTPS passed: 22**. Sankofa, Phoenix, Studio, The Order, DBIS, Mifos, and MIM4U public endpoints passed. Evidence: `docs/04-configuration/verification-evidence/e2e-verification-20260326_115013/`. |
|
||
| Private E2E | Latest run `bash scripts/verify/verify-end-to-end-routing.sh --profile=private` exited `0` with **Failed: 0** and **DNS passed: 4**. `rpc-http-prv.d-bis.org`, `rpc-fireblocks.d-bis.org`, `rpc-ws-prv.d-bis.org`, and `ws.rpc-fireblocks.d-bis.org` all passed. Evidence: `docs/04-configuration/verification-evidence/e2e-verification-20260326_120939/`. |
|
||
| NPMplus backup | Fresh backup completed: `backups/npmplus/backup-20260326_115622.tar.gz`. API exports succeeded; direct SQLite file copy and certbot path copy were partial/warn-only, but the backup manifest and compressed bundle were created successfully. |
|
||
| Blockscout verification run | `./scripts/verify/run-contract-verification-with-proxy.sh` completed; contracts were submitted or skipped if already verified. `WETH10` returned `The address is not a smart contract`; others like `Multicall`, `Aggregator`, `Proxy`, `CCIPSender`, `CCIPWETH10Bridge`, and `CCIPWETH9Bridge` submitted successfully. |
|
||
| Private RPC redirect fix | `rpc-http-prv.d-bis.org` no longer returns HTTP `301` on JSON-RPC POST. Live NPMplus host `11` was updated to `ssl_forced=false` while preserving upstream `192.168.11.211:8545`. |
|
||
| NPM creds loading | For NPM-only runs, prefer targeted `grep` of `NPM_EMAIL` / `NPM_PASSWORD` if full `.env` export triggers `Argument list too long`. |
|
||
|
||
---
|
||
|
||
## 1. High: Cronos closure + reachable CCIP funding
|
||
|
||
**Ref:** [CONFIG_READY_CHAINS_COMPLETION_RUNBOOK](../07-ccip/CONFIG_READY_CHAINS_COMPLETION_RUNBOOK.md)
|
||
|
||
**Prereqs:** Confirm [CCIP supports](https://docs.chain.link/ccip/supported-networks) for the chains you are actively using. Current focus: **Cronos (25)**, plus reachable funded lanes. Per chain: RPC, CCIP Router, LINK, WETH9/WETH10, deployer with native gas. **Do not block the session on Wemix.**
|
||
|
||
```bash
|
||
cd smom-dbis-138
|
||
source .env
|
||
|
||
# Per chain (set RPC_URL, CCIP_ROUTER_ADDRESS, LINK_TOKEN_ADDRESS, WETH9_ADDRESS, WETH10_ADDRESS, PRIVATE_KEY)
|
||
forge script script/deploy/bridge/DeployWETHBridges.s.sol:DeployWETHBridges --rpc-url "$RPC_URL" --broadcast -vvvv
|
||
```
|
||
|
||
Then add destinations (Chain 138 ↔ each chain) and fund with LINK — use:
|
||
|
||
```bash
|
||
DRY_RUN=1 ./scripts/deployment/complete-config-ready-chains.sh # print commands
|
||
./scripts/deployment/complete-config-ready-chains.sh # run (requires bridge addresses in .env)
|
||
```
|
||
|
||
**Cronos closure:** Cronos bridges are already present on-chain. Use:
|
||
```bash
|
||
cd smom-dbis-138
|
||
DRY_RUN=1 ./scripts/deployment/complete-config-ready-chains.sh
|
||
./scripts/deployment/complete-config-ready-chains.sh
|
||
./scripts/deployment/fund-ccip-bridges-with-link.sh --dry-run # print commands
|
||
./scripts/deployment/fund-ccip-bridges-with-link.sh --link 10 --cap-to-deployer # run (deployer LINK cap per chain)
|
||
```
|
||
**Wemix:** deferred by policy. Revisit only after profitable routes fund expansion gas.
|
||
**Full live-session order:** See [LIVE_SESSION_CRONOS_AND_TIER1_PHASE_C.md](../03-deployment/LIVE_SESSION_CRONOS_AND_TIER1_PHASE_C.md).
|
||
|
||
---
|
||
|
||
## 2. Medium: LINK support on Mainnet relay
|
||
|
||
**Ref:** [RELAY_BRIDGE_ADD_LINK_SUPPORT_RUNBOOK](../07-ccip/RELAY_BRIDGE_ADD_LINK_SUPPORT_RUNBOOK.md)
|
||
|
||
**Options:** A = extend CCIPRelayBridge to accept LINK; B = deploy separate LINK receiver. After implement + deploy + fund:
|
||
|
||
```bash
|
||
# In config/token-mapping.json set relaySupported: true for LINK
|
||
# Update TOKEN_MAPPING_AND_MAINNET_ADDRESSES.md and CCIP_BRIDGE_MAINNET_CONNECTION.md
|
||
# Restart relay service on r630-01: /opt/smom-dbis-138/services/relay
|
||
```
|
||
|
||
---
|
||
|
||
## 3. LAN: Blockscout verification
|
||
|
||
```bash
|
||
source smom-dbis-138/.env 2>/dev/null
|
||
./scripts/verify/run-contract-verification-with-proxy.sh
|
||
```
|
||
|
||
Single contract retry: `./scripts/verify/run-contract-verification-with-proxy.sh --only ContractName`
|
||
|
||
---
|
||
|
||
## 4. LAN: Fix E2E 502s
|
||
|
||
```bash
|
||
./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e
|
||
# Or lighter:
|
||
./scripts/maintenance/address-all-remaining-502s.sh --run-besu-fix --e2e
|
||
```
|
||
|
||
**Runbook:** [502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md](502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md)
|
||
|
||
**Current status:** **2026-05-10** public E2E reported **`studio.sankofa.nexus`** **warn** (**502** on `/studio/`). Otherwise no broad 502 regression in that run. Use this section when Studio or other endpoints show **502**.
|
||
|
||
---
|
||
|
||
## 4b. LAN: Proxmox fleet vzdump, retention, cluster check
|
||
|
||
**Ref:** [STORAGE_GROWTH_AND_HEALTH.md](../04-configuration/STORAGE_GROWTH_AND_HEALTH.md)
|
||
|
||
- **Check corosync / nodes after any `pve-cluster` change:**
|
||
`./scripts/maintenance/verify-pve-cluster-health.sh`
|
||
Expect: **Quorate: Yes** and all expected nodes **online** in the JSON.
|
||
- **Deploy weekly vzdump file retention (all PVE nodes):**
|
||
`./scripts/maintenance/deploy-vzdump-prune-cron-to-proxmox-nodes.sh` (optional `--dry-run`). **Env:** `VZDUMP_PRUNE_KEEP=2` (default) when re-deploying.
|
||
- **R630-01 daily all-running-CT backup** lives on the node as **`/usr/local/bin/proxmox-backup.sh`**, sourced from the repo: **`scripts/maintenance/proxmox-backup-all-running-ct.sh`**. Re-install after editing: `scp` + `chmod 755` (see file header for cron and **`VZDUMP_SKIP_VMIDS`** / **`VZDUMP_COMPRESS`**).
|
||
- **Stuck `vzdump` or snap:** stop via **Proxmox UI task stop** if possible. If a **`task` worker** blocks **`/etc/pve`**, only then consider **`systemctl restart pve-cluster`** on that node, then re-run `verify-pve-cluster-health.sh`.
|
||
|
||
---
|
||
|
||
## 5. LAN: Run all operator tasks (backup + verify ± deploy ± create-vms)
|
||
|
||
```bash
|
||
./scripts/run-all-operator-tasks-from-lan.sh --dry-run # print steps
|
||
./scripts/run-all-operator-tasks-from-lan.sh # backup + Blockscout verify
|
||
./scripts/run-all-operator-tasks-from-lan.sh --deploy # + contract deploy
|
||
./scripts/run-all-operator-tasks-from-lan.sh --create-vms # + create DBIS Core + TsunamiSwap VM (5010)
|
||
./scripts/run-all-operator-tasks-from-lan.sh --deploy --create-vms
|
||
```
|
||
|
||
---
|
||
|
||
## 5e. LAN: EI matrix on-chain readiness (mainnet cWUSDC + Chain 138 cUSDC)
|
||
|
||
**Purpose:** Parallel `balanceOf` over all wallets in `config/pmm-soak-wallet-grid.json`, sharded to limit RPC load. Writes JSON + gap index files for triage (e.g. `send-cwusdc-ei-matrix-targeted.sh`).
|
||
|
||
**Prereqs:** `ETHEREUM_MAINNET_RPC` (or `RPC_URL_1`) and `RPC_URL_138` in root `.env` / `smom-dbis-138/.env` (loaded via `scripts/lib/load-project-env.sh`).
|
||
|
||
```bash
|
||
# Report-only (minima 0, exit 0): useful for refreshing artifacts
|
||
EI_MATRIX_AUDIT_MIN_MAINNET_RAW=0 EI_MATRIX_AUDIT_MIN_138_RAW=0 \
|
||
./scripts/verify/run-ei-matrix-full-readiness-audit.sh
|
||
|
||
# Operator gate: default min mainnet = 12e6 raw (12 USDC units); 138 minimum 0 unless you set EI_MATRIX_AUDIT_MIN_138_RAW
|
||
./scripts/verify/run-ei-matrix-full-readiness-audit.sh
|
||
|
||
# Custom slice / flags (forwarded to the Python audit)
|
||
./scripts/verify/run-ei-matrix-full-readiness-audit.sh -- --offset 0 --limit 500 --shard-size 200
|
||
|
||
# Ad-hoc modes (no full-grid wrapper)
|
||
./scripts/verify/audit-ei-matrix-onchain-readiness.sh --mainnet-only --min-mainnet-raw 12000000 --shard-size 400
|
||
```
|
||
|
||
**Artifacts:** `reports/status/ei-matrix-readiness-audit-latest.json`, `ei-matrix-readiness-gaps-mainnet-indices.txt`, `ei-matrix-readiness-gaps-138-indices.txt` (paths overridable via env on the full-grid script).
|
||
|
||
**Fund gaps (mainnet cWUSDC from `PRIVATE_KEY`):**
|
||
|
||
```bash
|
||
# Preferred: Multicall3 batches (one approve, then ~N/200 txs) — lowest gas vs 1 tx per wallet
|
||
./scripts/deployment/send-cwusdc-ei-matrix-multicall-batches.sh --dry-run \\
|
||
--tsv reports/status/ei-matrix-cwusdc-topup-amounts.tsv
|
||
EI_MATRIX_MC_CHUNK=200 ./scripts/deployment/send-cwusdc-ei-matrix-multicall-batches.sh --execute \\
|
||
--tsv reports/status/ei-matrix-cwusdc-topup-amounts.tsv
|
||
# Resume mid-run: --start-batch 10 (see reports/status/ei-matrix-multicall3-batch-progress.txt)
|
||
|
||
# Bulk slice (sequential transfer() per wallet)
|
||
./scripts/deployment/send-cwusdc-ei-matrix-wallets.sh --dry-run --offset 0 --limit 10 --send-raw 5000000
|
||
./scripts/deployment/continue-cwusdc-ei-matrix-wallets.sh --dry-run --send-raw 5000000
|
||
|
||
# Targeted: default indices = audit gap file
|
||
./scripts/deployment/send-cwusdc-ei-matrix-targeted.sh --dry-run --send-raw 5000000
|
||
./scripts/deployment/send-cwusdc-ei-matrix-targeted.sh --dry-run --send-raw 5000000 --indices-file reports/status/ei-matrix-readiness-gaps-mainnet-indices.txt
|
||
# Per-index amounts: TSV columns linearIndex <TAB> amountRaw
|
||
./scripts/deployment/send-cwusdc-ei-matrix-targeted.sh --dry-run --amounts-tsv reports/status/ei-matrix-cwusdc-retry-amounts-final-leg.tsv --indices-file reports/status/ei-matrix-cwusdc-retry-indices-final-leg.txt
|
||
|
||
# Audit → remediate (fixed --send-raw per gap index, or --multicall with deficit TSV)
|
||
./scripts/deployment/pipeline-ei-matrix-remediate-cwusdc-from-audit.sh --dry-run --send-raw 5000000
|
||
./scripts/deployment/pipeline-ei-matrix-remediate-cwusdc-from-audit.sh --dry-run --multicall
|
||
SKIP_EI_MATRIX_REMEDIATE_AUDIT=1 ./scripts/deployment/pipeline-ei-matrix-remediate-cwusdc-from-audit.sh --dry-run --send-raw 5000000 --indices-file path/to/subset.txt
|
||
```
|
||
|
||
**CI (optional):** `EI_MATRIX_ONCHAIN_AUDIT_CI=1 bash scripts/verify/run-all-validation.sh` — uses `EI_MATRIX_ONCHAIN_AUDIT_CI_LIMIT` (default `120`; `0` = full grid) and `EI_MATRIX_AUDIT_MIN_MAINNET_RAW_CI` / `EI_MATRIX_AUDIT_MIN_138_RAW_CI` (default `0` = report-only gate-off).
|
||
|
||
---
|
||
|
||
## 5c. LAN: TsunamiSwap VM (5010) and CCIP funding
|
||
|
||
**TsunamiSwap VM:** Create once (default r630-01, 8 vCPU, 16 GB, 160 GB at 192.168.11.91). For r630-02 use `STORAGE=thin2 ./scripts/create-tsunamiswap-vm.sh --node r630-02`. Then run post-create setup (Docker + dirs):
|
||
|
||
```bash
|
||
./scripts/create-tsunamiswap-vm.sh --dry-run # print steps
|
||
./scripts/create-tsunamiswap-vm.sh # create VMID 5010
|
||
./scripts/setup-tsunamiswap-vm-5010.sh [--dry-run] # install Docker, create /opt/tsunamiswap (from LAN)
|
||
./scripts/deploy-tsunamiswap-to-5010.sh [--dry-run] # deploy backend+UI to 5010 (first run installs Node, ~5–10 min)
|
||
```
|
||
|
||
**CCIP funding (LINK):** After deployer has LINK and native gas on each chain:
|
||
|
||
```bash
|
||
cd smom-dbis-138
|
||
./scripts/deployment/fund-ccip-bridges-with-link.sh --dry-run # print commands
|
||
./scripts/deployment/fund-ccip-bridges-with-link.sh [--link 10] [--cap-to-deployer] # run (non-fatal per chain)
|
||
```
|
||
|
||
**Ref:** [AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md](AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md), [OPERATIONAL_RUNBOOKS.md](../03-deployment/OPERATIONAL_RUNBOOKS.md) § TsunamiSwap.
|
||
|
||
---
|
||
|
||
## 5d. Sankofa Phoenix API — Enable railing proxy
|
||
|
||
**Ref:** [PHOENIX_RAILING_OPERATOR_SETUP.md](../04-configuration/PHOENIX_RAILING_OPERATOR_SETUP.md)
|
||
|
||
In the environment where **Sankofa Phoenix API** runs, set:
|
||
|
||
```bash
|
||
export PHOENIX_RAILING_URL=http://phoenix-deploy-api:4001 # or your Phoenix Deploy API URL
|
||
# Optional if railing enforces partner keys:
|
||
export PHOENIX_RAILING_API_KEY=<key>
|
||
```
|
||
|
||
Restart the API; then `/api/v1/infra/nodes`, `/api/v1/health/summary`, etc. will proxy to the railing.
|
||
|
||
---
|
||
|
||
## 5a. LAN: Token-aggregation DB and migrations (VMID 5000)
|
||
|
||
If `/health` returns "database token_aggregation does not exist":
|
||
|
||
```bash
|
||
./scripts/apply-token-aggregation-fix.sh # create DB, run migrations, restart (via Proxmox)
|
||
./scripts/apply-token-aggregation-fix.sh --dry-run # print steps only
|
||
```
|
||
|
||
If VMID 5000 has no `postgres` user, run `createdb` and migrations on the host where PostgreSQL runs, or set token-aggregation `DATABASE_URL` to `explorer_db` and run `smom-dbis-138/services/token-aggregation/scripts/run-migrations.sh` there.
|
||
|
||
---
|
||
|
||
## 5b. LAN: Chain 138 next steps (Phase 2: preflight → mirror+pool → register c* as GRU → verify)
|
||
|
||
**Ref:** [DEPLOYMENT_ORDER_OF_OPERATIONS](../03-deployment/DEPLOYMENT_ORDER_OF_OPERATIONS.md) Phase 2. Use when mirror/pool/GRU registration or verify are pending.
|
||
|
||
```bash
|
||
./scripts/deployment/run-all-next-steps-chain138.sh --dry-run # print steps only
|
||
./scripts/deployment/run-all-next-steps-chain138.sh # run all (preflight, deploy mirror+pool, register c*, verify)
|
||
./scripts/deployment/run-all-next-steps-chain138.sh --skip-mirror # pool + register + verify only (set TRANSACTION_MIRROR_ADDRESS in smom-dbis-138/.env first)
|
||
```
|
||
|
||
If TransactionMirror deploy fails with **CreateCollision:** set `TRANSACTION_MIRROR_ADDRESS=0xC7f2Cf4845C6db0e1a1e91ED41Bcd0FcC1b0E141` in `smom-dbis-138/.env` and re-run with `--skip-mirror`. See [TRANSACTION_MIRROR_CHAIN138_COLLISION_FIX](../03-deployment/TRANSACTION_MIRROR_CHAIN138_COLLISION_FIX.md).
|
||
|
||
---
|
||
|
||
## 6. Low: DODO PMM on Chain 138
|
||
|
||
**Ref:** [OPTIONAL_DEPLOYMENTS_START_HERE](../07-ccip/OPTIONAL_DEPLOYMENTS_START_HERE.md) §2B
|
||
|
||
**Prereqs:** Set in `smom-dbis-138/.env`: `DODO_VENDING_MACHINE_ADDRESS`, `COMPLIANT_USDT_ADDRESS`, `COMPLIANT_USDC_ADDRESS`.
|
||
|
||
```bash
|
||
./scripts/run-optional-deployments.sh --execute --phases 7
|
||
# Or from smom-dbis-138: ./scripts/deployment/deploy-optional-future-all.sh (Phase 7 = DODO)
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Low: Mainnet trustless stack (Lockbox138 + Mainnet)
|
||
|
||
**Ref:** [OPTIONAL_DEPLOYMENTS_START_HERE](../07-ccip/OPTIONAL_DEPLOYMENTS_START_HERE.md) §2C
|
||
|
||
**Prereqs:** `ETHEREUM_MAINNET_RPC`, Mainnet ETH for deployer.
|
||
|
||
```bash
|
||
cd smom-dbis-138
|
||
source .env
|
||
forge script script/bridge/trustless/DeployTrustlessBridge.s.sol:DeployTrustlessBridge \
|
||
--rpc-url "$ETHEREUM_MAINNET_RPC" --broadcast --via-ir --verify
|
||
# Then: Lockbox138 on 138; configure Lockbox138↔InboxETH; fund liquidity. See runbook §C.
|
||
```
|
||
|
||
---
|
||
|
||
## 8. Wave 0: sendCrossChain (real) and NPMplus backup
|
||
|
||
**sendCrossChain (real):** Requires `PRIVATE_KEY` and LINK approved in `.env`. Bridge: `0xcacfd227A040002e49e2e01626363071324f820a`.
|
||
|
||
```bash
|
||
bash scripts/bridge/run-send-cross-chain.sh 0.01 [recipient_address]
|
||
# Omit --dry-run to execute. Example: bash scripts/bridge/run-send-cross-chain.sh 0.01 0x...
|
||
```
|
||
|
||
**NPMplus backup:** Requires `NPM_PASSWORD` in `.env` and host on LAN.
|
||
|
||
```bash
|
||
bash scripts/verify/backup-npmplus.sh
|
||
# Or combined Wave 0: bash scripts/run-wave0-from-lan.sh
|
||
```
|
||
|
||
**NPMplus RPC fix (405):** From LAN: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`. Verify: `bash scripts/verify/verify-end-to-end-routing.sh`.
|
||
|
||
**Status (2026-03-26):** main NPMplus API update completed successfully with `39 hosts updated, 0 failed`; public E2E now passes for Sankofa root, Phoenix, Studio, and The Order. Re-run only when upstream targets or proxy definitions change.
|
||
|
||
**Latest backup evidence:** `backups/npmplus/backup-20260326_115622.tar.gz`
|
||
|
||
**NPMplus API unreachable (167/169):** Restart Docker inside NPMplus LXC: `./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh` (SSH to r630-01, restarts npmplus in 10233 and 10235).
|
||
|
||
**If port 81 accepts TCP but hangs at HTTP:** reboot CT `10233` with `pct reboot 10233` on `r630-01`, then retry the API updater.
|
||
|
||
**E2E from LAN (no public DNS):** If E2E fails at DNS (`Could not resolve host`), use [E2E_DNS_FROM_LAN_RUNBOOK.md](../04-configuration/E2E_DNS_FROM_LAN_RUNBOOK.md): append `config/e2e-hosts-append.txt` to `/etc/hosts`, then run `E2E_USE_SYSTEM_RESOLVER=1 ./scripts/verify/verify-end-to-end-routing.sh --profile=public`. Revert with `sudo ./scripts/verify/remove-e2e-hosts-from-etc-hosts.sh`.
|
||
|
||
**E2E profiles:** Use `--profile=public` for public endpoints (default) or `--profile=private` for private/admin RPC only. Run sequentially to avoid timestamp collision in evidence dirs. **Known E2E warnings** (502/404 and WS): [E2E_ENDPOINTS_LIST.md](../04-configuration/E2E_ENDPOINTS_LIST.md) § Known E2E warnings and Remediation. MIM4U web 502s and WS test-format warnings are **non-blocking** for contract/pool completion.
|
||
|
||
**Pre-PR validation:** Before opening Chainlist or token-list PRs, run `./scripts/run-before-pr-validations.sh` from repo root. Trust Wallet remains a separate `wallet-core` workflow: add the registry entry from `docs/04-configuration/pr-ready/trust-wallet-registry-chain138.json`, then run Trust Wallet codegen/tests inside the `wallet-core` repo.
|
||
|
||
---
|
||
|
||
## 8.5 PMM mesh (6s oracle / keeper / PMM–WETH poll)
|
||
|
||
**Ref:** `smom-dbis-138/docs/integration/ORACLE_AND_KEEPER_CHAIN138.md` (PMM mesh automation)
|
||
|
||
```bash
|
||
cd smom-dbis-138
|
||
# .env should include: PRIVATE_KEY, AGGREGATOR_ADDRESS, PRICE_FEED_KEEPER_ADDRESS (optional: KEEPER_PRIVATE_KEY if different from PRIVATE_KEY)
|
||
./scripts/reserve/set-price-feed-keeper-interval.sh 6 # once per keeper deployment if interval was 30s
|
||
./scripts/update-oracle-price.sh # verify transmitter + gas (Besu needs explicit gas limit in script)
|
||
./scripts/reserve/sync-weth-mock-price.sh # if CHAIN138_WETH_MOCK_PRICE_FEED is set (keeper WETH path)
|
||
mkdir -p logs
|
||
nohup ./scripts/reserve/pmm-mesh-6s-automation.sh >> logs/pmm-mesh-automation.log 2>&1 &
|
||
# journalctl equivalent: tail -f logs/pmm-mesh-automation.log
|
||
```
|
||
|
||
**systemd:** `config/systemd/chain138-pmm-mesh-automation.service.example` — copy, set `User` and absolute paths, `enable --now`.
|
||
|
||
---
|
||
|
||
## 9. Wemix token verification (Deferred)
|
||
|
||
This is intentionally deferred with the rest of the Wemix path. If the chain is brought back into scope later, open [scan.wemix.com/tokens](https://scan.wemix.com/tokens); confirm WETH, USDT, USDC addresses. If different, update `config/token-mapping-multichain.json` and [WEMIX_TOKEN_VERIFICATION.md](../07-ccip/WEMIX_TOKEN_VERIFICATION.md). Then:
|
||
|
||
```bash
|
||
./scripts/validation/validate-config-files.sh
|
||
```
|
||
|
||
---
|
||
|
||
## 10. DeFi Oracle Meta Mainnet (Chain 138) — phased production path (matrix-driven)
|
||
|
||
**Ref:** [dbis_chain_138_technical_master_plan.md](../../dbis_chain_138_technical_master_plan.md), [DBIS_NODE_ROLE_MATRIX.md](../02-architecture/DBIS_NODE_ROLE_MATRIX.md)
|
||
|
||
| Phase | Action |
|
||
|-------|--------|
|
||
| 1 — Reality mapping | `bash scripts/verify/run-phase1-discovery.sh` (optional: `HYPERLEDGER_PROBE=1`). Reports: `reports/phase1-discovery/`. Runbook: [PHASE1_DISCOVERY_RUNBOOK.md](../03-deployment/PHASE1_DISCOVERY_RUNBOOK.md). |
|
||
| 2 — Sovereignization roadmap | Read [DBIS_PHASE2_PROXMOX_SOVEREIGNIZATION_ROADMAP.md](../02-architecture/DBIS_PHASE2_PROXMOX_SOVEREIGNIZATION_ROADMAP.md); execute milestones (cluster expansion, Ceph, VLANs) as prioritized. |
|
||
| 3 — E2E simulation | `bash scripts/verify/run-dbis-phase3-e2e-simulation.sh` (optional: `RUN_CHAIN138_RPC_HEALTH=1`). Full flow + Indy/Fabric/CCIP manual steps: [DBIS_PHASE3_E2E_PRODUCTION_SIMULATION_RUNBOOK.md](../03-deployment/DBIS_PHASE3_E2E_PRODUCTION_SIMULATION_RUNBOOK.md). |
|
||
| Perf (Caliper) | `bash scripts/verify/print-caliper-chain138-stub.sh` — then [CALIPER_CHAIN138_PERF_HOOK.md](../03-deployment/CALIPER_CHAIN138_PERF_HOOK.md). |
|
||
|
||
**Readiness:** Resolve critical **Entity owner** / **Region** **TBD** rows in the Node Role Matrix before claiming multi-entity production governance.
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- [COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md](COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md) — full plan (required, optional, recommended)
|
||
- [TODOS_CONSOLIDATED.md](TODOS_CONSOLIDATED.md) — full task list
|
||
- [NEXT_STEPS_AND_REMAINING_TODOS.md](NEXT_STEPS_AND_REMAINING_TODOS.md) — detail and completed items
|
||
- [STEPS_FROM_PROXMOX_OR_LAN_WITH_SECRETS.md](STEPS_FROM_PROXMOX_OR_LAN_WITH_SECRETS.md) — full LAN steps
|