Release Pipeline

Code reaches the live host through a pull-based release: GitHub Actions builds and publishes an artifact, and a timer on the host pulls and applies it. There is no self-hosted runner and no inbound access from GitHub to the host.

Flow

Pull request check (.github/workflows/portal-typecheck.yml)

Added by pull request 269. On every pull request (no path filter, so it can be made a required status check) it runs npm ci and npx tsc --noEmit in cyberrange/portal on Node 20. It exists because next build type-checks the portal but Jest's TypeScript transform does not: a type error passed review and broke every release build until it was hot-fixed.

CI build (.github/workflows/ampere-release.yml)

• Triggers: push to main (build only), or manual dispatch with promote=true to publish the release.
• Steps: check the portal build configs exist, run check_runtime_deps.py (fails the build if src/provisioning imports a third-party package missing from pyproject.toml, or uses FastAPI uploads without python-multipart), pack_tree.sh bundles the cyberrange/ tree, npm ci && npm run build builds the portal, then a manifest is written with the commit sha, the package-lock.json SHA-256 and a build timestamp.
• Artifact: tree.tgz, portal-build.tgz and manifest.json, kept 14 days.
• Promotion: only a manual dispatch with promote=true on main force-tags and publishes the ampere-live release.

No test suite in CI

The release workflow builds and packages and, since pull request 143, checks declared Python dependencies; the separate pull-request workflow type-checks the portal. Neither runs pytest or jest (F9), so a failing test does not stop a merge or a release (see Accuracy Notes for the suite status). A test gate in CI is a recommended improvement.

The docs site is not packaged

pack_tree.sh bundles only the cyberrange/ subdirectory. This docs-site/ folder lives at the repository root and is never included in a platform release, so adding these docs cannot affect the running platform.

Host apply (deploy/host/pull_release.sh)

Run by cyberrange-pull-release.timer (2 minutes after boot, then every 5 minutes). The step-by-step story of a release is also told in How the Web App Is Deployed.

0. PATH: since pull request 305 both the script and the unit file add /usr/sbin:/snap/bin to PATH. User units start with /usr/bin:/bin, where lxc is missing, so before this the certificate sync failed and a stale pin stayed in place (issue 304). The sync now names the PATH in its error when lxc is not found.

1. Fetch: skip if the release env or token file is missing; otherwise fetch the ampere-live release by tag from the GitHub API and download the assets.

2. Decide: release_sync.py --check-apply compares the manifest sha against ~/cyberrange/DEPLOYED_SHA. Unchanged: log pull-release skip, re-sync the Wazuh certificate pin, run the CA smoke (step 5) and exit. A failed smoke exits 1 so the unit shows failed and the next timer run retries; a check that could not run is logged as wazuh cert recheck incomplete, and only when both ran is it logged wazuh cert recheck OK (pull request 307). If the release SHA is marked bad (a previous failed apply), log pull-release STUCK: release <sha> failed to apply earlier; live site stays on <live sha> with the next step, and write ~/cyberrange-data/rollback/STUCK. The GitHub Action and release still look green in this state, so check for the STUCK file.

3. Snapshot: copy the current tree and portal build to ~/cyberrange-data/rollback/ before changing anything.

4. Integrity: verify the downloaded package-lock.json hash matches the manifest; on mismatch, fail_apply. If the lockfile changed since last time, run npm ci --omit=dev.

5. Apply: extract the tree to ~/cyberrange and the portal build to ~/cyberrange/portal. The extract keeps an existing certs/wazuh-api.crt (the host's runtime pin) instead of overwriting it with the repository's starter pin (pull request 152), then sync_wazuh_api_cert.py refreshes the pin from the live wazuh-manager when the manager's certificate fingerprint matches what localhost:55000 serves and its SAN includes DNS:localhost, backing up the old file under ~/cyberrange-data/ (pull request 155). The sync is best-effort and never fails a pull.

   Internal CA (pull request 275). Once issue_wazuh_internal_ca.py --install has written certs/wazuh-api-ca.crt, the sync no longer replaces any pin: it only checks that the live manager certificate chains to that CA. A CA smoke then runs openssl s_client against localhost:55000 with -verify_hostname localhost and that CA, and signs in as the scoring user. A chain, hostname or sign-in failure is fail_apply "Wazuh CA smoke failed", which rolls the release back. A smoke that cannot run (no lxc, no wazuh-manager guest, nothing listening on 55000, scoring credentials unset) is logged and does not roll back. Hosts without the CA file stay on the leaf pin and skip the smoke.

6. Python dependencies: if pyproject.toml changed, install its [project].dependencies into ~/cyberrange/.venv (fail_apply on error). Warn if the running API unit does not use that venv. Also warn (pull-release WARNING: ... WAZUH_PROVISION_USER) when env/.env sets WAZUH_SCORING_USER but not the provisioner pair, because a student's next lab would be refused with WAZUH_STALE_AGENT (pull request 211). Values are checked for presence only, never printed.

7. Restart and health-gate: systemctl --user restart the API, portal and bridge, then poll with backoff for up to HEALTH_TIMEOUT (default 90 s): API /health (200 and ok), portal /login (200), /api/auth/providers (JSON) and the bridge WebSocket handshake (101). If the API's NRestarts grows by 2 or more since the restart, fail fast as a crash loop.

8. fail_apply <reason>: write the reason, each unit's state and restart count, the API journal tail and the last 60 lines of /tmp/provision-api.log to stderr and rollback/last-failure.log; record the SHA as bad; restore the snapshot and restart the units, so a bad release cannot leave the host down.

9. Success: write the new DEPLOYED_SHA, record the lockfile hash, and clear bad-sha and STUCK.

Manual rollback

~/cyberrange/deploy/host/pull_release.sh --rollback

This restores the last snapshot from ~/cyberrange-data/rollback/ and restarts the services. It is the fast way back if a release is healthy enough to pass the gate but wrong in some other way.

Recovering a stuck release

cat ~/cyberrange-data/rollback/STUCK              # which release failed, which is live
cat ~/cyberrange-data/rollback/last-failure.log   # why it failed
# fix the cause, then deliberately retry that same commit:
~/cyberrange/deploy/host/pull_release.sh --retry-bad

Without --retry-bad a failed commit is skipped forever, so re-promoting the same commit does nothing. Hosts that got stuck before pull request 143 still run the old script on their first apply and need one manual recovery (install the missing Python packages into the venv, remove rollback/bad-sha, run pull_release.sh).

What a release does not carry

Release artifacts are code and the portal build only. They do not include the database, evidence images, Keycloak data, env files or golden images. Those live on the host (and inside containers) and are managed separately; see Data Stores for the backup gap.

Sources

.github/workflows/ampere-release.yml, portal-typecheck.yml, cyberrange/deploy/host/pull_release.sh, release_sync.py, sync_wazuh_api_cert.py, issue_wazuh_internal_ca.py, check_runtime_deps.py, test_pull_release.py, pack_tree.sh, cyberrange/certs/README.md, cyberrange/docs/WAZUH-ADMIN-DEGRADED-RUNBOOK.md, cyberrange/deploy/systemd/cyberrange-pull-release.service, .timer.