From 60b8fb9ddc4b4689eeb057dc7f1b4f877bba3c08 Mon Sep 17 00:00:00 2001 From: defiQUG Date: Fri, 24 Apr 2026 18:25:28 -0700 Subject: [PATCH] ops: schedule gitea tls expiry monitoring --- docs/00-meta/OPERATOR_HANDOFF_2026_04_24.md | 5 +++- docs/00-meta/OPERATOR_READY_CHECKLIST.md | 2 +- docs/MASTER_INDEX.md | 1 + scripts/README.md | 1 + .../schedule-gitea-cert-check-cron.sh | 27 +++++++++++++++++++ scripts/verify/README.md | 5 ++++ 6 files changed, 39 insertions(+), 2 deletions(-) create mode 100755 scripts/maintenance/schedule-gitea-cert-check-cron.sh diff --git a/docs/00-meta/OPERATOR_HANDOFF_2026_04_24.md b/docs/00-meta/OPERATOR_HANDOFF_2026_04_24.md index e335fa7f..f9b2615e 100644 --- a/docs/00-meta/OPERATOR_HANDOFF_2026_04_24.md +++ b/docs/00-meta/OPERATOR_HANDOFF_2026_04_24.md @@ -43,13 +43,16 @@ bash scripts/monitoring/monitor-blockchain-health.sh The immediate HTTPS push blocker was an expired certificate on `gitea.d-bis.org`. The certificate was renewed and reattached through NPMplus #4, and the endpoint now verifies cleanly again. -Important: the live replacement certificate observed during the final checks expires on `2026-05-01`, so this needs near-term follow-through rather than “set and forget.” +Root cause of the short warning window: the live NPMplus certbot renewal config for `npm-7` included `required_profile = shortlived`, which forced a 7-day Let's Encrypt certificate instead of the normal 90-day issuance. + +That live config was corrected on 2026-04-24 and `gitea.d-bis.org` was reissued successfully. The current live certificate now expires on `2026-07-24`. Use this to check expiry before it becomes an outage: ```bash bash scripts/verify/check-gitea-certificate-expiry.sh WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh +bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install ``` ## Checkpoint commits diff --git a/docs/00-meta/OPERATOR_READY_CHECKLIST.md b/docs/00-meta/OPERATOR_READY_CHECKLIST.md index d359dd34..a00b18cf 100644 --- a/docs/00-meta/OPERATOR_READY_CHECKLIST.md +++ b/docs/00-meta/OPERATOR_READY_CHECKLIST.md @@ -17,7 +17,7 @@ **Chain 138 txpool incident standard path:** `bash scripts/fix-all-validators-and-txpool.sh` then `bash scripts/maintenance/apply-chain138-strict-future-tx-pool.sh` then `bash scripts/clear-all-transaction-pools.sh` then `bash scripts/monitoring/monitor-blockchain-health.sh`. Use this sequence when block production stalls, pending hashes keep reappearing, or future-nonce residue survives a normal txpool clear. -**Gitea HTTPS push safeguard:** `bash scripts/verify/check-gitea-certificate-expiry.sh` (optional: `WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh`). Use this when Git over HTTPS starts failing, or run it proactively before major push/deploy windows. +**Gitea HTTPS push safeguard:** `bash scripts/verify/check-gitea-certificate-expiry.sh` (optional: `WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh`). Use this when Git over HTTPS starts failing, or run it proactively before major push/deploy windows. To keep it checked automatically: `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install`. --- diff --git a/docs/MASTER_INDEX.md b/docs/MASTER_INDEX.md index 666f5912..a8f235cc 100644 --- a/docs/MASTER_INDEX.md +++ b/docs/MASTER_INDEX.md @@ -23,6 +23,7 @@ | **2026-04-24 operator handoff** | [00-meta/OPERATOR_HANDOFF_2026_04_24.md](00-meta/OPERATOR_HANDOFF_2026_04_24.md) — Besu hardening, duplicate RPC retirement, Gitea TLS repair, and remote reconciliation | | **Chain 138 txpool incident recovery** | `bash scripts/fix-all-validators-and-txpool.sh` → `bash scripts/maintenance/apply-chain138-strict-future-tx-pool.sh` → `bash scripts/clear-all-transaction-pools.sh` → `bash scripts/monitoring/monitor-blockchain-health.sh` | | **Gitea TLS expiry check** | `bash scripts/verify/check-gitea-certificate-expiry.sh` — warns before `gitea.d-bis.org` cert expiry blocks HTTPS pushes | +| **Gitea TLS expiry cron** | `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install` — installs a daily warning check with `WARN_DAYS=30` | | **TsunamiSwap DEX plan** | [00-meta/AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md](00-meta/AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md) — canonical TsunamiSwap VM `5010` plan, current DEX link, and publish checklist | | **Required / optional / recommended (full plan)** | [00-meta/COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md](00-meta/COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md) | | **Single task list** | [00-meta/TODOS_CONSOLIDATED.md](00-meta/TODOS_CONSOLIDATED.md) | diff --git a/scripts/README.md b/scripts/README.md index 43aaba70..6a5026e0 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -197,6 +197,7 @@ CT 2301 (besu-rpc-private-1) may fail to start with `lxc.hook.pre-start` due to - **Wave 0 from LAN:** `./scripts/run-wave0-from-lan.sh [--dry-run] [--skip-backup] [--skip-rpc-fix]` — runs NPMplus RPC fix (W0-1) and NPMplus backup (W0-3); W0-2 (sendCrossChain) run separately without `--dry-run`. - **All waves (max parallel):** `./scripts/run-all-waves-parallel.sh [--dry-run] [--skip-wave0] [--skip-wave2] [--host HOST]` — Wave 0 via SSH, Wave 1 parallel (env, cron, SSH/firewall dry-run, shellcheck, validate), Wave 2 W2-6 (create 2506/2507/2508). See `docs/00-meta/FULL_PARALLEL_EXECUTION_ORDER.md` and `FULL_PARALLEL_RUN_LOG.md`. - **NPMplus backup cron:** `./scripts/maintenance/schedule-npmplus-backup-cron.sh [--install|--show]` — add or print daily 03:00 cron for backup-npmplus.sh. +- **Gitea TLS check cron:** `./scripts/maintenance/schedule-gitea-cert-check-cron.sh [--install|--show]` — add or print daily 07:15 cron for `check-gitea-certificate-expiry.sh` (default `WARN_DAYS=30`). - **Security:** `./scripts/security/secure-env-permissions.sh [--dry-run]` or `chmod 600 .env smom-dbis-138/.env dbis_core/.env` — secure env files. **Validator keys (W1-19):** On Proxmox host as root: `./scripts/secure-validator-keys.sh [--dry-run]` (VMIDs 1000–1004). ### 12. Maintenance (135–139) diff --git a/scripts/maintenance/schedule-gitea-cert-check-cron.sh b/scripts/maintenance/schedule-gitea-cert-check-cron.sh new file mode 100755 index 00000000..97f3b8d8 --- /dev/null +++ b/scripts/maintenance/schedule-gitea-cert-check-cron.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# Schedule a daily Gitea TLS expiry check. Run from project root. +# Usage: bash scripts/maintenance/schedule-gitea-cert-check-cron.sh [--install|--show] + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +CHECK_SCRIPT="$PROJECT_ROOT/scripts/verify/check-gitea-certificate-expiry.sh" +LOG_DIR="$PROJECT_ROOT/logs" +WARN_DAYS="${WARN_DAYS:-30}" +CRON_LINE="15 7 * * * cd $PROJECT_ROOT && WARN_DAYS=$WARN_DAYS bash $CHECK_SCRIPT >> $LOG_DIR/gitea-cert-expiry.log 2>&1" + +case "${1:-}" in + --install) + mkdir -p "$LOG_DIR" + (crontab -l 2>/dev/null; echo "$CRON_LINE") | awk '!seen[$0]++' | crontab - + echo "Installed: $CRON_LINE" + ;; + --show) + echo "Crontab line: $CRON_LINE" + ;; + *) + echo "Usage: $0 [--install|--show]" + exit 0 + ;; +esac diff --git a/scripts/verify/README.md b/scripts/verify/README.md index cf62bb2c..5fe99421 100644 --- a/scripts/verify/README.md +++ b/scripts/verify/README.md @@ -40,6 +40,11 @@ One-line install (Debian/Ubuntu): `sudo apt install -y sshpass rsync dnsutils ip - `check-dependencies.sh` - Verify required tools (bash, curl, jq, openssl, ssh) - `check-cluster-besu-inventory.sh` - Cluster-wide Besu inventory audit using `pvesh /cluster/resources` via a Proxmox cluster node so host placement on `r630-03` / `r630-04` is not missed. Prints VMID, type, node, status, name, IP, canonical-vs-extra classification, and any missing canonical VMIDs. Use `--json` for machine-readable output. - `check-gitea-certificate-expiry.sh` - Read-only TLS expiry check for `gitea.d-bis.org` (or another host passed as arg). Exits `0` when outside the warning window, `1` when within `WARN_DAYS` (default `14`), and `2` on expiry or probe failure. + +## Scheduling + +- Install a daily cron check with `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install` +- Print the cron line with `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --show` - `check-pnpm-workspace-lockfile.sh` - Ensures every path in `pnpm-workspace.yaml` has an `importer` in `pnpm-lock.yaml` (run `pnpm install` at root if it fails; avoids broken `pnpm outdated -r`) - `export-cloudflare-dns-records.sh` - Export Cloudflare DNS records - `export-npmplus-config.sh` - Export NPMplus proxy hosts and certificates via API