ops: schedule gitea tls expiry monitoring
This commit is contained in:
@@ -43,13 +43,16 @@ bash scripts/monitoring/monitor-blockchain-health.sh
|
||||
|
||||
The immediate HTTPS push blocker was an expired certificate on `gitea.d-bis.org`. The certificate was renewed and reattached through NPMplus #4, and the endpoint now verifies cleanly again.
|
||||
|
||||
Important: the live replacement certificate observed during the final checks expires on `2026-05-01`, so this needs near-term follow-through rather than “set and forget.”
|
||||
Root cause of the short warning window: the live NPMplus certbot renewal config for `npm-7` included `required_profile = shortlived`, which forced a 7-day Let's Encrypt certificate instead of the normal 90-day issuance.
|
||||
|
||||
That live config was corrected on 2026-04-24 and `gitea.d-bis.org` was reissued successfully. The current live certificate now expires on `2026-07-24`.
|
||||
|
||||
Use this to check expiry before it becomes an outage:
|
||||
|
||||
```bash
|
||||
bash scripts/verify/check-gitea-certificate-expiry.sh
|
||||
WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh
|
||||
bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install
|
||||
```
|
||||
|
||||
## Checkpoint commits
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
|
||||
**Chain 138 txpool incident standard path:** `bash scripts/fix-all-validators-and-txpool.sh` then `bash scripts/maintenance/apply-chain138-strict-future-tx-pool.sh` then `bash scripts/clear-all-transaction-pools.sh` then `bash scripts/monitoring/monitor-blockchain-health.sh`. Use this sequence when block production stalls, pending hashes keep reappearing, or future-nonce residue survives a normal txpool clear.
|
||||
|
||||
**Gitea HTTPS push safeguard:** `bash scripts/verify/check-gitea-certificate-expiry.sh` (optional: `WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh`). Use this when Git over HTTPS starts failing, or run it proactively before major push/deploy windows.
|
||||
**Gitea HTTPS push safeguard:** `bash scripts/verify/check-gitea-certificate-expiry.sh` (optional: `WARN_DAYS=30 bash scripts/verify/check-gitea-certificate-expiry.sh`). Use this when Git over HTTPS starts failing, or run it proactively before major push/deploy windows. To keep it checked automatically: `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
| **2026-04-24 operator handoff** | [00-meta/OPERATOR_HANDOFF_2026_04_24.md](00-meta/OPERATOR_HANDOFF_2026_04_24.md) — Besu hardening, duplicate RPC retirement, Gitea TLS repair, and remote reconciliation |
|
||||
| **Chain 138 txpool incident recovery** | `bash scripts/fix-all-validators-and-txpool.sh` → `bash scripts/maintenance/apply-chain138-strict-future-tx-pool.sh` → `bash scripts/clear-all-transaction-pools.sh` → `bash scripts/monitoring/monitor-blockchain-health.sh` |
|
||||
| **Gitea TLS expiry check** | `bash scripts/verify/check-gitea-certificate-expiry.sh` — warns before `gitea.d-bis.org` cert expiry blocks HTTPS pushes |
|
||||
| **Gitea TLS expiry cron** | `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install` — installs a daily warning check with `WARN_DAYS=30` |
|
||||
| **TsunamiSwap DEX plan** | [00-meta/AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md](00-meta/AAVE_CHAIN138_AND_MARIONETTE_TSUNAMISWAP_PLAN.md) — canonical TsunamiSwap VM `5010` plan, current DEX link, and publish checklist |
|
||||
| **Required / optional / recommended (full plan)** | [00-meta/COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md](00-meta/COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md) |
|
||||
| **Single task list** | [00-meta/TODOS_CONSOLIDATED.md](00-meta/TODOS_CONSOLIDATED.md) |
|
||||
|
||||
@@ -197,6 +197,7 @@ CT 2301 (besu-rpc-private-1) may fail to start with `lxc.hook.pre-start` due to
|
||||
- **Wave 0 from LAN:** `./scripts/run-wave0-from-lan.sh [--dry-run] [--skip-backup] [--skip-rpc-fix]` — runs NPMplus RPC fix (W0-1) and NPMplus backup (W0-3); W0-2 (sendCrossChain) run separately without `--dry-run`.
|
||||
- **All waves (max parallel):** `./scripts/run-all-waves-parallel.sh [--dry-run] [--skip-wave0] [--skip-wave2] [--host HOST]` — Wave 0 via SSH, Wave 1 parallel (env, cron, SSH/firewall dry-run, shellcheck, validate), Wave 2 W2-6 (create 2506/2507/2508). See `docs/00-meta/FULL_PARALLEL_EXECUTION_ORDER.md` and `FULL_PARALLEL_RUN_LOG.md`.
|
||||
- **NPMplus backup cron:** `./scripts/maintenance/schedule-npmplus-backup-cron.sh [--install|--show]` — add or print daily 03:00 cron for backup-npmplus.sh.
|
||||
- **Gitea TLS check cron:** `./scripts/maintenance/schedule-gitea-cert-check-cron.sh [--install|--show]` — add or print daily 07:15 cron for `check-gitea-certificate-expiry.sh` (default `WARN_DAYS=30`).
|
||||
- **Security:** `./scripts/security/secure-env-permissions.sh [--dry-run]` or `chmod 600 .env smom-dbis-138/.env dbis_core/.env` — secure env files. **Validator keys (W1-19):** On Proxmox host as root: `./scripts/secure-validator-keys.sh [--dry-run]` (VMIDs 1000–1004).
|
||||
|
||||
### 12. Maintenance (135–139)
|
||||
|
||||
27
scripts/maintenance/schedule-gitea-cert-check-cron.sh
Executable file
27
scripts/maintenance/schedule-gitea-cert-check-cron.sh
Executable file
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
# Schedule a daily Gitea TLS expiry check. Run from project root.
|
||||
# Usage: bash scripts/maintenance/schedule-gitea-cert-check-cron.sh [--install|--show]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
CHECK_SCRIPT="$PROJECT_ROOT/scripts/verify/check-gitea-certificate-expiry.sh"
|
||||
LOG_DIR="$PROJECT_ROOT/logs"
|
||||
WARN_DAYS="${WARN_DAYS:-30}"
|
||||
CRON_LINE="15 7 * * * cd $PROJECT_ROOT && WARN_DAYS=$WARN_DAYS bash $CHECK_SCRIPT >> $LOG_DIR/gitea-cert-expiry.log 2>&1"
|
||||
|
||||
case "${1:-}" in
|
||||
--install)
|
||||
mkdir -p "$LOG_DIR"
|
||||
(crontab -l 2>/dev/null; echo "$CRON_LINE") | awk '!seen[$0]++' | crontab -
|
||||
echo "Installed: $CRON_LINE"
|
||||
;;
|
||||
--show)
|
||||
echo "Crontab line: $CRON_LINE"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 [--install|--show]"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
@@ -40,6 +40,11 @@ One-line install (Debian/Ubuntu): `sudo apt install -y sshpass rsync dnsutils ip
|
||||
- `check-dependencies.sh` - Verify required tools (bash, curl, jq, openssl, ssh)
|
||||
- `check-cluster-besu-inventory.sh` - Cluster-wide Besu inventory audit using `pvesh /cluster/resources` via a Proxmox cluster node so host placement on `r630-03` / `r630-04` is not missed. Prints VMID, type, node, status, name, IP, canonical-vs-extra classification, and any missing canonical VMIDs. Use `--json` for machine-readable output.
|
||||
- `check-gitea-certificate-expiry.sh` - Read-only TLS expiry check for `gitea.d-bis.org` (or another host passed as arg). Exits `0` when outside the warning window, `1` when within `WARN_DAYS` (default `14`), and `2` on expiry or probe failure.
|
||||
|
||||
## Scheduling
|
||||
|
||||
- Install a daily cron check with `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --install`
|
||||
- Print the cron line with `bash scripts/maintenance/schedule-gitea-cert-check-cron.sh --show`
|
||||
- `check-pnpm-workspace-lockfile.sh` - Ensures every path in `pnpm-workspace.yaml` has an `importer` in `pnpm-lock.yaml` (run `pnpm install` at root if it fails; avoids broken `pnpm outdated -r`)
|
||||
- `export-cloudflare-dns-records.sh` - Export Cloudflare DNS records
|
||||
- `export-npmplus-config.sh` - Export NPMplus proxy hosts and certificates via API
|
||||
|
||||
Reference in New Issue
Block a user