Files
CurrenciCombo/scripts/deployment/deploy-currencicombo-8604.sh
Devin AI ded7d24924
Some checks failed
CI / Frontend Lint (pull_request) Failing after 7s
CI / Frontend Type Check (pull_request) Failing after 7s
CI / Frontend Build (pull_request) Failing after 6s
CI / Frontend E2E Tests (pull_request) Failing after 7s
CI / Orchestrator Build (pull_request) Failing after 7s
CI / Orchestrator Unit Tests (pull_request) Failing after 6s
CI / Orchestrator E2E (Testcontainers) (pull_request) Has been skipped
CI / Contracts Compile (pull_request) Failing after 5s
CI / Contracts Test (pull_request) Failing after 6s
Code Quality / SonarQube Analysis (pull_request) Failing after 20s
Code Quality / Code Quality Checks (pull_request) Failing after 7s
Security Scan / Dependency Vulnerability Scan (pull_request) Failing after 4s
Security Scan / OWASP ZAP Scan (pull_request) Failing after 4s
PR AA follow-up: manual-rollback loud-failure summary + keep-min-5 backup-prune cron + root-only initial-keys handoff file
- deploy-currencicombo-8604.sh: on readiness timeout, print loud failure
  summary (journalctl tails + exact --rollback command with specific
  backup path) instead of silently exiting. Deliberately does NOT
  auto-rollback; first cutovers often fail because of env/migration
  mistakes and auto-restore hides the failure state ops needs.
- install.sh: on first run, write the three API keys + EVENT_SIGNING_SECRET
  to /root/currencicombo-first-keys.txt (0600, root:root) as a handoff
  copy. Canonical values still live in /etc/currencicombo/orchestrator.env.
  Log one pointer line (not the secrets themselves) to journald.
  Handoff file is NOT regenerated if orchestrator.env already exists.
- install-prune-cron.sh (new, opt-in): installs /etc/cron.daily/
  currencicombo-prune-backups that deletes entries older than 30 days
  from /var/lib/currencicombo/backups/ WHILE always keeping the newest
  5 regardless of age. Enforced via newest-first sort + i<KEEP_MIN skip.
- webapp-nginx.conf: drop the misleading /events/* 421 guard-rail. The
  orchestrator's SSE endpoint is /api/plans/:id/events/stream (under
  /api/), so one /api/* guard-rail covers both normal REST and SSE.
- README.md: corrected NPMplus rule table to TWO rules (/api/* with
  SSE-friendly proxy_buffering=off + 24h read_timeout + Connection ""
  + http/1.1, and /); added post-cutover smoke checks section with a
  concrete SSE streaming test that catches silent proxy_buffering=on
  misconfig; documented the /root/currencicombo-first-keys.txt handoff
  and the install-prune-cron.sh workflow; replaced stale 'not auto-pruned'
  note.

Verification:
- shellcheck --severity=warning: clean on all 3 scripts.
- bash -n: clean on install-prune-cron.sh.
- install-prune-cron.sh --dry-run: prints the pruner body with resolved
  env values as expected.
- install.sh --dry-run: walks through user/dirs/nginx-apt steps, then
  fails fast on missing psql (expected on a build box without Postgres).

Co-Authored-By: Nakamoto, S <defi@defi-oracle.io>
2026-04-22 23:30:34 +00:00

226 lines
9.7 KiB
Bash
Executable File

#!/usr/bin/env bash
# deploy-currencicombo-8604.sh — build-and-swap deploy for CurrenciCombo.
#
# Runs on a systemd host that has already had `install.sh` applied once.
# This is the script referenced by the Proxmox repo's
# `phoenix-deploy-api/deploy-targets.json` tuple
# (repo=d-bis/CurrenciCombo, branch=main, target=default).
#
# Steps (each idempotent, each can be --dry-run'd):
# 1. git clone/pull /var/lib/currencicombo/repo to the target ref.
# 2. Build orchestrator (npm ci + npm run build).
# 3. Build portal/webapp (npm ci + npm run build), baking
# VITE_ORCHESTRATOR_URL into the bundle.
# 4. Run DB migrations (npm run migrate in orchestrator/).
# 5. Stop systemd units.
# 6. rsync build output into /opt/currencicombo/{orchestrator,webapp}.
# 7. Start systemd units.
# 8. Smoke-test /ready + portal / + print EXT-* blocker summary.
#
# Rollback: `--rollback` restores the previous backup under
# /var/lib/currencicombo/backups/<timestamp>.
#
# CT 8604 is in the filename for ops-grep-ability; the script itself is
# host-agnostic. Override paths via env vars if you run it elsewhere.
set -euo pipefail
# ----- defaults (override via env) ------------------------------------
: "${CC_GIT_REMOTE:=https://gitea.d-bis.org/d-bis/CurrenciCombo.git}"
: "${CC_GIT_REF:=main}"
: "${CC_REPO_DIR:=/var/lib/currencicombo/repo}"
: "${CC_APP_HOME:=/opt/currencicombo}"
: "${CC_BACKUP_DIR:=/var/lib/currencicombo/backups}"
: "${CC_USER:=currencicombo}"
# Portal build-time env. The NPMplus ingress path-routes /api/* and
# /events/* to the orchestrator, so same-origin works.
: "${VITE_ORCHESTRATOR_URL:=https://curucombo.xn--vov0g.com}"
: "${ORCHESTRATOR_UNIT:=currencicombo-orchestrator.service}"
: "${WEBAPP_UNIT:=currencicombo-webapp.service}"
: "${CC_HEALTH_URL:=http://127.0.0.1:8080/ready}"
: "${CC_PORTAL_URL:=http://127.0.0.1:3000/}"
: "${CC_HEALTH_TIMEOUT_SECS:=60}"
# ----- flags ----------------------------------------------------------
DRY_RUN=0
SKIP_MIGRATE=0
SKIP_BUILD=0
DO_ROLLBACK=0
usage() {
cat <<'USAGE'
Usage: sudo ./deploy-currencicombo-8604.sh [flags]
Flags:
--ref=<git-ref> Override CC_GIT_REF (default: main)
--dry-run Print commands, don't run them
--skip-migrate Skip `npm run migrate` step (use for hotfix
deploys where schema hasn't changed)
--skip-build Reuse the existing build in CC_REPO_DIR/dist
(useful after `--dry-run --skip-build=no` from
the previous run)
--rollback Restore the most recent backup and restart.
Does not run git/build/migrate.
-h, --help This help
Env overrides:
CC_GIT_REMOTE, CC_GIT_REF, CC_REPO_DIR, CC_APP_HOME, CC_BACKUP_DIR,
CC_USER, VITE_ORCHESTRATOR_URL, ORCHESTRATOR_UNIT, WEBAPP_UNIT,
CC_HEALTH_URL, CC_PORTAL_URL, CC_HEALTH_TIMEOUT_SECS
USAGE
}
while [[ $# -gt 0 ]]; do
case "$1" in
--ref=*) CC_GIT_REF="${1#*=}"; shift ;;
--dry-run) DRY_RUN=1; shift ;;
--skip-migrate) SKIP_MIGRATE=1; shift ;;
--skip-build) SKIP_BUILD=1; shift ;;
--rollback) DO_ROLLBACK=1; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
esac
done
log() { printf '[deploy] %s\n' "$*" >&2; }
warn() { printf '[deploy][WARN] %s\n' "$*" >&2; }
die() { printf '[deploy][FATAL] %s\n' "$*" >&2; exit 1; }
run() { if [[ "${DRY_RUN}" -eq 1 ]]; then printf '[deploy][dry-run] %s\n' "$*" >&2; else eval "$*"; fi; }
runcc() { if [[ "${DRY_RUN}" -eq 1 ]]; then printf '[deploy][dry-run][as %s] %s\n' "${CC_USER}" "$*" >&2; else sudo -u "${CC_USER}" -H bash -lc "$*"; fi; }
[[ "$EUID" -eq 0 ]] || die "must run as root (sudo)"
# ----- rollback fast-path ---------------------------------------------
if [[ "${DO_ROLLBACK}" -eq 1 ]]; then
LATEST="$(ls -1dt "${CC_BACKUP_DIR}"/* 2>/dev/null | head -1 || true)"
[[ -n "${LATEST}" ]] || die "no backup under ${CC_BACKUP_DIR}"
log "rolling back to ${LATEST}"
run "systemctl stop '${WEBAPP_UNIT}' '${ORCHESTRATOR_UNIT}'"
run "rsync -a --delete '${LATEST}/orchestrator/' '${CC_APP_HOME}/orchestrator/'"
run "rsync -a --delete '${LATEST}/webapp/' '${CC_APP_HOME}/webapp/'"
run "systemctl start '${ORCHESTRATOR_UNIT}' '${WEBAPP_UNIT}'"
log "rollback applied. systemctl status ${ORCHESTRATOR_UNIT} to verify."
exit 0
fi
# ----- 1. git ---------------------------------------------------------
if [[ ! -d "${CC_REPO_DIR}/.git" ]]; then
log "cloning ${CC_GIT_REMOTE}${CC_REPO_DIR}"
run "install -d -o '${CC_USER}' -g '${CC_USER}' -m 0755 '${CC_REPO_DIR}'"
runcc "git clone '${CC_GIT_REMOTE}' '${CC_REPO_DIR}'"
fi
runcc "cd '${CC_REPO_DIR}' && git fetch --prune origin"
runcc "cd '${CC_REPO_DIR}' && git reset --hard 'origin/${CC_GIT_REF}'"
REF_SHA="$(sudo -u "${CC_USER}" git -C "${CC_REPO_DIR}" rev-parse --short HEAD 2>/dev/null || echo unknown)"
log "repo at ${CC_GIT_REF} = ${REF_SHA}"
# ----- 2. orchestrator build -----------------------------------------
if [[ "${SKIP_BUILD}" -eq 0 ]]; then
log "building orchestrator"
runcc "cd '${CC_REPO_DIR}/orchestrator' && npm ci --no-audit --no-fund"
runcc "cd '${CC_REPO_DIR}/orchestrator' && npm run build"
log "building portal (VITE_ORCHESTRATOR_URL=${VITE_ORCHESTRATOR_URL})"
runcc "cd '${CC_REPO_DIR}' && npm ci --include=optional --no-audit --no-fund || npm ci --include=optional --force --no-audit --no-fund"
runcc "cd '${CC_REPO_DIR}' && VITE_ORCHESTRATOR_URL='${VITE_ORCHESTRATOR_URL}' npm run build"
else
log "skipping builds (--skip-build)"
fi
# ----- 3. migrations --------------------------------------------------
if [[ "${SKIP_MIGRATE}" -eq 0 ]]; then
log "running DB migrations"
runcc "cd '${CC_REPO_DIR}/orchestrator' && npm run migrate"
else
log "skipping migrations (--skip-migrate)"
fi
# ----- 4. backup previous install ------------------------------------
TS="$(date +%Y%m%d-%H%M%S)"
BACKUP="${CC_BACKUP_DIR}/${TS}"
if [[ -d "${CC_APP_HOME}/orchestrator/dist" || -d "${CC_APP_HOME}/webapp/dist" ]]; then
log "backing up current install → ${BACKUP}"
run "install -d -o root -g root -m 0700 '${BACKUP}/orchestrator' '${BACKUP}/webapp'"
run "rsync -a '${CC_APP_HOME}/orchestrator/' '${BACKUP}/orchestrator/'"
run "rsync -a '${CC_APP_HOME}/webapp/' '${BACKUP}/webapp/'"
fi
# ----- 5. stop units --------------------------------------------------
log "stopping systemd units"
run "systemctl stop '${WEBAPP_UNIT}' || true"
run "systemctl stop '${ORCHESTRATOR_UNIT}' || true"
# ----- 6. swap in new build ------------------------------------------
log "rsyncing new build into ${CC_APP_HOME}"
# Orchestrator: dist/ + node_modules/ + package.json + package-lock.json
runcc "rsync -a --delete '${CC_REPO_DIR}/orchestrator/dist/' '${CC_APP_HOME}/orchestrator/dist/'"
runcc "rsync -a '${CC_REPO_DIR}/orchestrator/node_modules/' '${CC_APP_HOME}/orchestrator/node_modules/'"
runcc "cp '${CC_REPO_DIR}/orchestrator/package.json' '${CC_APP_HOME}/orchestrator/package.json'"
runcc "cp '${CC_REPO_DIR}/orchestrator/package-lock.json' '${CC_APP_HOME}/orchestrator/package-lock.json'"
# Webapp: dist/
runcc "rsync -a --delete '${CC_REPO_DIR}/dist/' '${CC_APP_HOME}/webapp/dist/'"
# ----- 7. start units ------------------------------------------------
log "starting systemd units"
run "systemctl start '${ORCHESTRATOR_UNIT}'"
run "systemctl start '${WEBAPP_UNIT}'"
# ----- 8. smoke -------------------------------------------------------
if [[ "${DRY_RUN}" -eq 1 ]]; then
log "dry-run: skipping smoke test"
exit 0
fi
log "waiting up to ${CC_HEALTH_TIMEOUT_SECS}s for orchestrator ${CC_HEALTH_URL}"
SECS=0
until curl -sfL --max-time 3 "${CC_HEALTH_URL}" >/dev/null 2>&1; do
SECS=$((SECS + 2))
if [[ "${SECS}" -ge "${CC_HEALTH_TIMEOUT_SECS}" ]]; then
# Loud failure summary. Deliberately does NOT auto-rollback — first
# cutovers often fail because of env/migration mistakes, and
# auto-restoring the old build hides the failure state ops needs to
# diagnose. Print the exact --rollback command with the specific
# backup path filled in, so it's one copy-paste away if desired.
{
echo
echo "================================================================"
echo "DEPLOY FAILED: orchestrator did not become ready after ${CC_HEALTH_TIMEOUT_SECS}s"
echo "================================================================"
echo
echo "## currencicombo-orchestrator (last 40 lines):"
journalctl -u "${ORCHESTRATOR_UNIT}" -n 40 --no-pager 2>&1 || echo "(journalctl unavailable)"
echo
echo "## currencicombo-webapp (last 20 lines):"
journalctl -u "${WEBAPP_UNIT}" -n 20 --no-pager 2>&1 || echo "(journalctl unavailable)"
echo
echo "## Units are in whatever state deploy left them. To restore"
echo "## the previous build (does NOT revert DB migrations):"
echo
if [[ -n "${BACKUP:-}" && -d "${BACKUP}" ]]; then
echo " sudo $0 --rollback"
echo " # (will restore ${BACKUP})"
else
echo " # No backup was taken (first deploy). Manual recovery required."
fi
echo
echo "================================================================"
} >&2
exit 1
fi
sleep 2
done
log "orchestrator ready: $(curl -sf "${CC_HEALTH_URL}")"
log "probing portal ${CC_PORTAL_URL}"
PORTAL_CODE="$(curl -s -o /dev/null -w '%{http_code}' "${CC_PORTAL_URL}" || echo ERR)"
[[ "${PORTAL_CODE}" =~ ^2 ]] || die "portal returned HTTP ${PORTAL_CODE}"
log "portal OK (HTTP ${PORTAL_CODE})"
log "EXT-* blocker summary from orchestrator boot log:"
journalctl -u "${ORCHESTRATOR_UNIT}" --no-pager -n 200 \
| grep -E 'ExternalBlockers|EXT-[A-Z0-9-]+' | tail -20 || true
log "deploy complete. ref=${CC_GIT_REF} sha=${REF_SHA} ts=${TS}"