ops: schedule gitea tls expiry monitoring
Some checks failed
Deploy to Phoenix / validate (push) Failing after 30s
Deploy to Phoenix / deploy (push) Has been skipped
Deploy to Phoenix / deploy-atomic-swap-dapp (push) Has been skipped
Deploy to Phoenix / cloudflare (push) Has been skipped

This commit is contained in:
defiQUG
2026-04-24 18:25:28 -07:00
parent 01c36a5489
commit 60b8fb9ddc
6 changed files with 39 additions and 2 deletions

View File

@@ -43,13 +43,16 @@ bash scripts/monitoring/monitor-blockchain-health.sh
The immediate HTTPS push blocker was an expired certificate on `gitea.d-bis.org`. The certificate was renewed and reattached through NPMplus #4, and the endpoint now verifies cleanly again.
Important: the live replacement certificate observed during the final checks expires on `2026-05-01`, so this needs near-term follow-through rather than “set and forget.”
Root cause of the short warning window: the live NPMplus certbot renewal config for `npm-7` included `required_profile = shortlived`, which forced a 7-day Let's Encrypt certificate instead of the normal 90-day issuance.
That live config was corrected on 2026-04-24 and `gitea.d-bis.org` was reissued successfully. The current live certificate now expires on `2026-07-24`.
Use this to check expiry before it becomes an outage:
```bash
bash scripts/verify/check-gitea-certificate-expiry.sh
WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh
bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install
```
## Checkpoint commits

View File

@@ -17,7 +17,7 @@
**Chain 138 txpool incident standard path:** `bash scripts/fix-all-validators-and-txpool.sh` then `bash scripts/maintenance/apply-chain138-strict-future-tx-pool.sh` then `bash scripts/clear-all-transaction-pools.sh` then `bash scripts/monitoring/monitor-blockchain-health.sh`. Use this sequence when block production stalls, pending hashes keep reappearing, or future-nonce residue survives a normal txpool clear.
**Gitea HTTPS push safeguard:** `bash scripts/verify/check-gitea-certificate-expiry.sh` (optional: `WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh`). Use this when Git over HTTPS starts failing, or run it proactively before major push/deploy windows.
**Gitea HTTPS push safeguard:** `bash scripts/verify/check-gitea-certificate-expiry.sh` (optional: `WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh`). Use this when Git over HTTPS starts failing, or run it proactively before major push/deploy windows. To keep it checked automatically: `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install`.
---

View File

@@ -23,6 +23,7 @@
| **2026-04-24 operator handoff** | [00-meta/OPERATOR_HANDOFF_2026_04_24.md](00-meta/OPERATOR_HANDOFF_2026_04_24.md) — Besu hardening, duplicate RPC retirement, Gitea TLS repair, and remote reconciliation |
| **Chain 138 txpool incident recovery** | `bash scripts/fix-all-validators-and-txpool.sh``bash scripts/maintenance/apply-chain138-strict-future-tx-pool.sh``bash scripts/clear-all-transaction-pools.sh``bash scripts/monitoring/monitor-blockchain-health.sh` |
| **Gitea TLS expiry check** | `bash scripts/verify/check-gitea-certificate-expiry.sh` — warns before `gitea.d-bis.org` cert expiry blocks HTTPS pushes |
| **Gitea TLS expiry cron** | `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install` — installs a daily warning check with `WARN_DAYS=30` |
| **TsunamiSwap DEX plan** | [00-meta/AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md](00-meta/AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md) — canonical TsunamiSwap VM `5010` plan, current DEX link, and publish checklist |
| **Required / optional / recommended (full plan)** | [00-meta/COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md](00-meta/COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md) |
| **Single task list** | [00-meta/TODOS_CONSOLIDATED.md](00-meta/TODOS_CONSOLIDATED.md) |

View File

@@ -197,6 +197,7 @@ CT 2301 (besu-rpc-private-1) may fail to start with `lxc.hook.pre-start` due to
- **Wave 0 from LAN:** `./scripts/run-wave0-from-lan.sh [--dry-run] [--skip-backup] [--skip-rpc-fix]` — runs NPMplus RPC fix (W0-1) and NPMplus backup (W0-3); W0-2 (sendCrossChain) run separately without `--dry-run`.
- **All waves (max parallel):** `./scripts/run-all-waves-parallel.sh [--dry-run] [--skip-wave0] [--skip-wave2] [--host HOST]` — Wave 0 via SSH, Wave 1 parallel (env, cron, SSH/firewall dry-run, shellcheck, validate), Wave 2 W2-6 (create 2506/2507/2508). See `docs/00-meta/FULL_PARALLEL_EXECUTION_ORDER.md` and `FULL_PARALLEL_RUN_LOG.md`.
- **NPMplus backup cron:** `./scripts/maintenance/schedule-npmplus-backup-cron.sh [--install|--show]` — add or print daily 03:00 cron for backup-npmplus.sh.
- **Gitea TLS check cron:** `./scripts/maintenance/schedule-gitea-cert-check-cron.sh [--install|--show]` — add or print daily 07:15 cron for `check-gitea-certificate-expiry.sh` (default `WARN_DAYS=30`).
- **Security:** `./scripts/security/secure-env-permissions.sh [--dry-run]` or `chmod 600 .env smom-dbis-138/.env dbis_core/.env` — secure env files. **Validator keys (W1-19):** On Proxmox host as root: `./scripts/secure-validator-keys.sh [--dry-run]` (VMIDs 10001004).
### 12. Maintenance (135139)

View File

@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# Schedule a daily Gitea TLS expiry check. Run from project root.
# Usage: bash scripts/maintenance/schedule-gitea-cert-check-cron.sh [--install|--show]
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
CHECK_SCRIPT="$PROJECT_ROOT/scripts/verify/check-gitea-certificate-expiry.sh"
LOG_DIR="$PROJECT_ROOT/logs"
WARN_DAYS="${WARN_DAYS:-30}"
CRON_LINE="15 7 * * * cd $PROJECT_ROOT && WARN_DAYS=$WARN_DAYS bash $CHECK_SCRIPT >> $LOG_DIR/gitea-cert-expiry.log 2>&1"
case "${1:-}" in
--install)
mkdir -p "$LOG_DIR"
(crontab -l 2>/dev/null; echo "$CRON_LINE") | awk '!seen[$0]++' | crontab -
echo "Installed: $CRON_LINE"
;;
--show)
echo "Crontab line: $CRON_LINE"
;;
*)
echo "Usage: $0 [--install|--show]"
exit 0
;;
esac

View File

@@ -40,6 +40,11 @@ One-line install (Debian/Ubuntu): `sudo apt install -y sshpass rsync dnsutils ip
- `check-dependencies.sh` - Verify required tools (bash, curl, jq, openssl, ssh)
- `check-cluster-besu-inventory.sh` - Cluster-wide Besu inventory audit using `pvesh /cluster/resources` via a Proxmox cluster node so host placement on `r630-03` / `r630-04` is not missed. Prints VMID, type, node, status, name, IP, canonical-vs-extra classification, and any missing canonical VMIDs. Use `--json` for machine-readable output.
- `check-gitea-certificate-expiry.sh` - Read-only TLS expiry check for `gitea.d-bis.org` (or another host passed as arg). Exits `0` when outside the warning window, `1` when within `WARN_DAYS` (default `14`), and `2` on expiry or probe failure.
## Scheduling
- Install a daily cron check with `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install`
- Print the cron line with `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --show`
- `check-pnpm-workspace-lockfile.sh` - Ensures every path in `pnpm-workspace.yaml` has an `importer` in `pnpm-lock.yaml` (run `pnpm install` at root if it fails; avoids broken `pnpm outdated -r`)
- `export-cloudflare-dns-records.sh` - Export Cloudflare DNS records
- `export-npmplus-config.sh` - Export NPMplus proxy hosts and certificates via API