diff --git a/docs/04-configuration/GITEA_PHOENIX_NON_BLOCKCHAIN_VM_ROUTING_CLEANUP_TASK.md b/docs/04-configuration/GITEA_PHOENIX_NON_BLOCKCHAIN_VM_ROUTING_CLEANUP_TASK.md new file mode 100644 index 00000000..e1779773 --- /dev/null +++ b/docs/04-configuration/GITEA_PHOENIX_NON_BLOCKCHAIN_VM_ROUTING_CLEANUP_TASK.md @@ -0,0 +1,88 @@ +# Task — Gitea / Phoenix routing cleanup scoped to all non-blockchain VMIDs + +**Status:** Scoped (not started) +**Created:** 2026-05-12 +**Owner:** Operator + repo maintainers + +## Why this exists + +Follow-up work was deferred from the gov-portals / Gitea-only alignment: generated routing views, `deploy-targets.json` parity, workflow deduplication, optional `gov-portals-runtime.json`, scoped CI tokens, and strict Gitea-only remotes. That work must **not** stop at a few portal rows: it must **reconcile against every guest VMID that is not part of the Chain 138 Besu node fleet**, so automation, docs, and Phoenix CD stay aligned with what is actually running on the cluster. + +## Definitions + +### Canonical VM truth (inputs) + +| Input | Role | +|--------|------| +| `reports/status/live_inventory.json` | Ground-truth **VMID list** and placement after `bash scripts/it-ops/export-live-inventory-and-drift.sh` (collector on **r630-01** per [ALL_VMIDS_ENDPOINTS.md](./ALL_VMIDS_ENDPOINTS.md)). | +| [ALL_VMIDS_ENDPOINTS.md](./ALL_VMIDS_ENDPOINTS.md) | Human **hostname / IP / purpose**; use for FQDN and category. | +| [VMID_ALLOCATION_FINAL.md](../02-architecture/VMID_ALLOCATION_FINAL.md) | **Range semantics** (which band is Besu vs explorer vs AI vs Sankofa, etc.). | +| `phoenix-deploy-api/deploy-targets.json` | Declared **repo → target → command → healthcheck** for Pattern A. | +| [GITEA_REPO_VM_CD_CI_MATRIX.md](./GITEA_REPO_VM_CD_CI_MATRIX.md) | Curated **repo ↔ VM ↔ workflow** table (must stay consistent with generated matrix). | + +### “Non-blockchain VMID” (in scope for mandatory coverage) + +A guest **VMID counts as non-blockchain** for this task if it is **not** a **Chain 138 Besu network node** host (validator, sentry, or RPC/archive Besu process fleet). Concretely, treat the following as **excluded** from the mandatory matrix (they are “blockchain VMIDs” for closure purposes): + +- **Validators:** VMIDs **1000–1004** (see ALL_VMIDS “Blockchain Nodes - Validators”). +- **Sentries:** VMIDs **1500–1510** (extend if the live Besu map adds sentries; reconcile with [../06-besu/BESU_NODE_CONFIGURATION_MAP_20260424.md](../06-besu/BESU_NODE_CONFIGURATION_MAP_20260424.md) and `scripts/verify/check-cluster-besu-inventory.sh`). +- **RPC / archive Besu:** VMIDs listed under **“RPC Nodes”** and **Besu archive / telemetry** sections in ALL_VMIDS (e.g. **2500–2608**, **3500–3501**, **2101**, **2400–2402** — always **re-verify** against the live map before freezing an exclusion list in code). + +**Everything else** that appears as a **running** LXC/QEMU guest in `live_inventory.json` is **in scope**: infra (100–130), NPMplus (10233+), Blockscout **5000**, Cacti, CCIP DON hosts, AI/dev (**5700+**), gov portals (**7801**, **7804**, **7810**, …), sovereign DBIS (**10100+**), operator tooling CTs, etc. + +**Note:** Blockscout and CCIP are “chain-adjacent” but **not** Besu nodes — they **are in scope** (must appear in the matrix with repo/workflow or an explicit `N/A` rationale). + +### “Blockchain VMID” (out of scope for mandatory rows) + +Besu-only hosts per the exclusion list above. For those, the matrix may omit rows or carry a single **“Besu fleet — see Besu map”** pointer; they **do not** satisfy or fail the non-blockchain closure check. + +## Objectives (work packages) + +1. **Machine-readable matrix** + Add `config/gitea-phoenix/non-blockchain-vm-routing-matrix.v1.schema.json` (JSON Schema) and `config/gitea-phoenix/non-blockchain-vm-routing-matrix.v1.json` (or generated `reports/status/…` with a committed snapshot) listing, for each **in-scope VMID**: `vmid`, `hostname`, `primary_ip`, `category`, `gitea_repos[]`, `deploy_target`, `workflow_glob`, `health_url`, `notes`. + +2. **Validator / gate script** + `scripts/verify/validate-non-blockchain-vm-routing-matrix.sh` (or Python under `scripts/lib/`): + - Load latest `live_inventory.json`. + - Compute **in-scope VMIDs** = all running guests minus **exclusion list** (config-driven, e.g. `config/gitea-phoenix/besu-vmid-exclusions.v1.json`). + - Fail CI if any in-scope VMID is **missing** from the matrix file unless explicitly listed in an **`allowed_missing`** section with reason (e.g. “ephemeral lab CT”). + - Warn (or fail, policy TBD) if matrix references a VMID **not** in inventory. + +3. **`deploy-targets.json` parity** + Every **Pattern A** app that should redeploy from Gitea must have a target consistent with the matrix; health URLs must match ALL_VMIDS / E2E lists where applicable. + +4. **Workflow deduplication** + Portal templates under `config/gitea-workflow-templates/repos/` vs copies in each `Gov_Web_Portals/*` repo: reduce drift (shared snippet, documented bump process, or generator — pick in implementation). + +5. **Optional `gov-portals-runtime.json`** + Single file mapping **7804** unit names → ports → public FQDNs for operators and Phoenix (only if it reduces duplication with `deploy-targets` healthchecks). + +6. **Secrets hygiene** + Document per-repo **least-privilege** tokens (read monorepo vs deploy vs NPM API); no long-lived tokens in `.git/config` URLs (Gitea-only policy). + +7. **Docs** + Update [GITEA_REPO_VM_CD_CI_MATRIX.md](./GITEA_REPO_VM_CD_CI_MATRIX.md) from the generated matrix (or link “generated from …” to avoid double maintenance). + +## Acceptance criteria (task complete when) + +- **Closure:** Every **running** guest VMID in `live_inventory.json` that is **not** in the **Besu exclusion** config has a matrix entry **or** an approved `allowed_missing` record with owner sign-off in JSON. +- **Scripts:** `bash scripts/validation/validate-config-files.sh` (or `run-all-validation.sh`) invokes the new validator once the matrix is committed. +- **Phoenix:** `deploy-targets.json` passes `bash scripts/validation/validate-phoenix-deploy-targets.sh` and matches matrix targets for all **Phoenix-backed** non-Besu apps. +- **Docs:** ALL_VMIDS and matrix show **no contradictions** for in-scope VMIDs (IP, FQDN, VMID). +- **Gitea-only:** No canonical repo uses GitHub as `origin` for app code paths documented in the matrix (mirrors optional, clearly labeled). + +## Suggested implementation order + +1. Freeze **Besu exclusion list** in JSON (derived from ALL_VMIDS + one `check-cluster-besu-inventory.sh` run, then reviewed). +2. Scaffold matrix JSON + schema; populate from existing GITEA matrix + deploy-targets + ALL_VMIDS. +3. Run inventory export on LAN; fill gaps until validator passes. +4. Tighten CI and dedupe workflows. +5. Optional runtime JSON and token scoping. + +## References + +- [GITEA_GOV_PORTALS_LIVE_SOURCE_OF_TRUTH.md](./GITEA_GOV_PORTALS_LIVE_SOURCE_OF_TRUTH.md) +- [GOV_PORTALS_REPO_DIRECTORY_VM_FQDN_TABLE.md](./GOV_PORTALS_REPO_DIRECTORY_VM_FQDN_TABLE.md) +- [GITEA_CD_OPERATOR_CHECKLIST.md](../00-meta/GITEA_CD_OPERATOR_CHECKLIST.md) +- `phoenix-deploy-api/deploy-targets.json` +- `scripts/verify/report-gitea-cd-parity.sh` diff --git a/docs/04-configuration/GITEA_REPO_VM_CD_CI_MATRIX.md b/docs/04-configuration/GITEA_REPO_VM_CD_CI_MATRIX.md index 6be6ebaf..7187b289 100644 --- a/docs/04-configuration/GITEA_REPO_VM_CD_CI_MATRIX.md +++ b/docs/04-configuration/GITEA_REPO_VM_CD_CI_MATRIX.md @@ -4,6 +4,8 @@ Each **application repo** should carry **its own** `.gitea/workflows/*.yml` so p **Canonical integration:** [Phoenix deploy API](../../phoenix-deploy-api/server.js) + [`deploy-targets.json`](../../phoenix-deploy-api/deploy-targets.json). Gov portals two-path model: [GITEA_GOV_PORTALS_LIVE_SOURCE_OF_TRUTH.md](./GITEA_GOV_PORTALS_LIVE_SOURCE_OF_TRUTH.md). +**Planned full-cluster alignment (non-Besu VMIDs):** [GITEA_PHOENIX_NON_BLOCKCHAIN_VM_ROUTING_CLEANUP_TASK.md](./GITEA_PHOENIX_NON_BLOCKCHAIN_VM_ROUTING_CLEANUP_TASK.md) — machine-readable matrix + CI gate so every **non-blockchain** guest in `live_inventory.json` is mapped or explicitly exempt. + **Postgres / Prisma in Actions (self-hosted runners):** use **`...@postgres:5432`** (service hostname) and keep **`container.network`** empty on **`act_runner`** — [GITEA_ACT_RUNNER_SETUP.md](GITEA_ACT_RUNNER_SETUP.md) (troubleshooting **P1001**). **Operator checklist:** [docs/00-meta/GITEA_CD_OPERATOR_CHECKLIST.md](../00-meta/GITEA_CD_OPERATOR_CHECKLIST.md) diff --git a/docs/MASTER_INDEX.md b/docs/MASTER_INDEX.md index 6869162e..8bd86d04 100644 --- a/docs/MASTER_INDEX.md +++ b/docs/MASTER_INDEX.md @@ -31,6 +31,7 @@ | **Gitea TLS expiry cron** | `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install` — installs a daily warning check with `WARN_DAYS=30` | | **Gitea repo ↔ VM CI/CD matrix** | [04-configuration/GITEA_REPO_VM_CD_CI_MATRIX.md](04-configuration/GITEA_REPO_VM_CD_CI_MATRIX.md) — per-repo workflows, Phoenix deploy targets, templates under `config/gitea-workflow-templates/`; gov portals live paths: [GITEA_GOV_PORTALS_LIVE_SOURCE_OF_TRUTH.md](04-configuration/GITEA_GOV_PORTALS_LIVE_SOURCE_OF_TRUTH.md) | | **Gov portals: directory, Gitea/GitHub, VMID, FQDN** | [04-configuration/GOV_PORTALS_REPO_DIRECTORY_VM_FQDN_TABLE.md](04-configuration/GOV_PORTALS_REPO_DIRECTORY_VM_FQDN_TABLE.md) — one table for proxmox, `gov-portals-monorepo`, and DBIS/ICCC/OMNL/XOM submodules | +| **Gitea / Phoenix routing matrix (all non-Besu VMIDs)** | [04-configuration/GITEA_PHOENIX_NON_BLOCKCHAIN_VM_ROUTING_CLEANUP_TASK.md](04-configuration/GITEA_PHOENIX_NON_BLOCKCHAIN_VM_ROUTING_CLEANUP_TASK.md) — closure vs `live_inventory.json`, `deploy-targets.json`, workflows, acceptance criteria | | **Gitea CD operator checklist** | [00-meta/GITEA_CD_OPERATOR_CHECKLIST.md](00-meta/GITEA_CD_OPERATOR_CHECKLIST.md) — secrets, Phoenix host sync, `report-gitea-cd-parity.sh` | | **TsunamiSwap DEX plan** | [00-meta/AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md](00-meta/AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md) — canonical TsunamiSwap VM `5010` plan, current DEX link, and publish checklist | | **Required / optional / recommended (full plan)** | [00-meta/COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md](00-meta/COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md) |