Appearance
Operations & Troubleshooting
Day-to-day operator commands, where the logs and config live, and fixes for the known issues. The authoritative source is manual/09-known-issues.md (issues 1 to 14) and the verification checklist in manual/08-verification.md; this page summarizes them.
Everyday commands
All platform services run as user units under llms_admin. SSH to the host as the operator first.
bash
# Service status and control (run as llms_admin)
systemctl --user status cyberrange-provision-api cyberrange-portal cyberrange-ssh-bridge
systemctl --user restart cyberrange-provision-api
journalctl --user -u cyberrange-portal -n 100 --no-pager
# Health and capacity
curl -s http://10.115.77.1:5000/health # {"status":"ok"}
curl -s http://10.115.77.1:5000/capacity # RAM, active pods, can_provision
# Labs and networks
lxc list
lxc network list
# Cloudflare tunnel (system unit)
sudo journalctl -u cloudflared -n 100 --no-pager
# Release
~/cyberrange/deploy/host/pull_release.sh # apply latest (also runs on a timer)
~/cyberrange/deploy/host/pull_release.sh --rollback # restore the last snapshot
~/cyberrange/deploy/host/pull_release.sh --retry-bad # retry a release that failed before (after fixing it)
cat ~/cyberrange-data/rollback/STUCK ~/cyberrange-data/rollback/last-failure.log
# Wazuh API certificate pin (one-shot, loud on failure)
python3 ~/cyberrange/deploy/host/sync_wazuh_api_cert.py --strict
# Wazuh one-time host setup (manual 02 and 06)
python3 ~/cyberrange/deploy/host/issue_wazuh_internal_ca.py --install # internal CA for the API
read -rs WAZUH_ADMIN_PW; export WAZUH_ADMIN_PW # current `wazuh` password, not echoed
~/cyberrange/deploy/host/harden_wazuh_api.sh # API on loopback, rotate built-in accounts (Step 2)
~/cyberrange/deploy/host/setup_wazuh_provisioner.sh # least-privilege agent clean-up user (Step 3b)
unset WAZUH_ADMIN_PW
# Lab network checks (egress ACL and manager return route)
lxc network acl show pod-<id>-egress
lxc exec wazuh-manager -- ip route # expect one 10.0.<50+id>.0/24 route per live lab
# Database migrations
cd ~/cyberrange/src/provisioning
python3 migrate.py --check # exit 0 if up to date
python3 migrate.py --applyBefore restarting services
Check active_pods in /capacity first. Restarting the API, portal or bridge while a student is mid-lab interrupts them. On the live single-lab host, wait until active_pods is 0 or coordinate with the user.
Where things are
| What | Location |
|---|---|
| Code tree | ~/cyberrange (release-synced; not a git repo) |
| Data and secrets | ~/cyberrange-data (mode 700): pod_mgmt.db, review_images/, rollback/, release token |
| API / portal / bridge config | ~/cyberrange/env/.env, portal/.env.local, bridge-src/.env.bridge (all mode 600) |
| Service logs | /tmp/provision-api.log, /tmp/portal.log, /tmp/ssh-bridge.log (lost on reboot) |
| Scoring log (in each lab container) | /var/log/cyberrange-scoring.log |
| Gateway nginx + Keycloak | inside the guacamole container; Keycloak env /etc/keycloak/admin.env |
| Wazuh manager | inside the wazuh-manager container; alerts at /var/ossec/logs/alerts/; API config /var/ossec/api/configuration/api.yaml |
| Wazuh trust and keys | ~/cyberrange/certs/wazuh-api.crt (leaf pin) or wazuh-api-ca.crt (internal CA); CA private keys in ~/cyberrange-data/wazuh-ca/ |
| nginx config (repo) | cyberrange/deploy/guacamole/nginx-ampere.conf |
Known issues and fixes
From manual/09-known-issues.md and manual/06-scoring.md. Symptoms first, so you can scan.
| Symptom | Likely cause | Fix |
|---|---|---|
New containers have no IPv4; apt "Network is unreachable" | OCI packet filter rejects FORWARD/DHCP | sudo bash ~/cyberrange/deploy/host/install_lxd_oci_iptables.sh; expect lxd_oci_iptables=PASS |
lxc commands hang | LXD/OVN state | see the manual's LXD hang steps; check systemctl status snap.lxd.daemon |
| Host under memory pressure | no swap | sudo bash ~/cyberrange/deploy/host/setup_zram_and_swap.sh |
Keycloak login fails with invalid_client | portal client secret mismatch between Keycloak and portal/.env.local | re-sync the client secret; restart the portal |
| Portal loads unstyled (no CSS) | portal build/static assets | rebuild the portal (npm run build) and restart; check nginx routing of / |
nginx listening on 0.0.0.0 | bind config | ensure nginx binds 10.115.77.12; never expose the origin publicly |
Keycloak admin console redirect_uri error | admin URL / proxy settings | use the documented admin URL; console is intended over SSH tunnel |
| Kali has no command history for scoring | history not flushed | provisioning installs enable_history_flush.sh; re-provision if an old lab lacks it |
| Wazuh agent not reporting | manager IP or agent service | check WAZUH_MANAGER_IP, restart wazuh-agent on meta/dvwa; see enrollment poll |
| Keycloak times out | container resources | check the Keycloak unit and memory in the gateway container |
| Terminal drops when idle | proxy timeout | nginx proxy_read_timeout 3600s for the WebSocket location |
| Public IP shows the default nginx page | wrong server block / tunnel origin | confirm Cloudflare Tunnel origin is 10.115.77.12:80 and the server block routes / to the portal |
| No GitHub Actions runner on the VM | by design | the host pulls the release; there is no runner |
| No Wazuh dashboard | manager-only deployment | there is no dashboard; use the portal SIEM viewer and the manager API |
| Rule 5710 missing in Scenario 3 | agent not enrolled, a legacy meta-base without the auth.log collection entry, or the manager has no route back to the lab subnet | destroy labs and rerun deploy/oci_build_goldens_arm64.sh so meta-base is rebaked and validated; if meta can ping the manager but nothing comes back, check lxc exec wazuh-manager -- ip route for a 10.0.<50+id>.0/24 via <mon-net gateway> dev eth0 route (the reaper re-adds it within 10 minutes after a manager restart). Manual 09, Issue 14 |
A student's second lab never starts; audit log shows WAZUH_STALE_AGENT | the previous lab's agents were never removed because the provisioner user is missing, so the old pod-<student>-meta name is still registered | run setup_wazuh_provisioner.sh (manual 06, Step 3b) and restart the API; the next start then deletes the leftover agents by name before it checks. SIEM clean-up skipped (WAZUH_CLEANUP_SKIPPED) in the audit log and a pull-release WARNING: ... WAZUH_PROVISION_USER line point to the same cause |
A lab rolls back with WAZUH_ENROLL_FAIL (not enrolled, an agent status, tls: or auth HTTP 401/403) | the agents did not reach active within WAZUH_AGENT_CONNECT_TIMEOUT_S (90 s), or the API's certificate or credentials are wrong | check the manager is running and the return route exists; for tls: follow the TLS-Degraded row below; for auth check WAZUH_SCORING_* in env/.env |
Scenario 1 labs never become ACTIVE; audit shows FLAGS_PLANTED_FAIL scenario=1, reason=... | Scenario 1 flag planting is fatal; most often COHORT_FLAG_SECRET is not set (it is not in the env template) | set COHORT_FLAG_SECRET in env/.env, restart the API, start the lab again |
| New code never goes live, Action and Release look green | the release commit failed to apply earlier and is skipped (pull-release STUCK in the journal, rollback/STUCK exists) | read rollback/last-failure.log, fix the cause, run pull_release.sh --retry-bad |
| Release rolls back within seconds | API crash loop after apply, often a missing Python package | last-failure.log has the traceback; check_runtime_deps.py should have caught an undeclared import in CI |
pull-release WARNING: cyberrange-provision-api does not run …/.venv/bin/python | old unit file | run deploy/systemd/install-user-units.sh, then restart the API |
| Admin → System shows Wazuh TLS-Degraded: reachable, but TLS verification failed | the pinned certs/wazuh-api.crt does not match the manager, or (with certs/wazuh-api-ca.crt present) the live certificate does not chain to the internal CA | without the CA file: python3 deploy/host/sync_wazuh_api_cert.py --strict; with it: rerun issue_wazuh_internal_ca.py --install. Runbook cyberrange/docs/WAZUH-ADMIN-DEGRADED-RUNBOOK.md. Missing WAZUH_SCORING_* credentials is a separate cause (manual 06) |
Release rolls back with Wazuh CA smoke failed, or the pull unit fails with wazuh cert recheck FAILED on unchanged release | the internal CA is installed but localhost:55000 fails the chain or hostname check, or the scoring user cannot sign in | fix the manager certificate (rerun issue_wazuh_internal_ca.py --install) or the scoring credentials; the next timer run retries. wazuh cert recheck incomplete only means a check could not run |
Journal shows lxc not found on PATH=... or wazuh CA smoke cannot run | an old cyberrange-pull-release.service whose PATH lacks /usr/sbin and /snap/bin (before pull request 305) | run deploy/systemd/install-user-units.sh once |
| Admin → Users: "User management is not configured" (503), also in Create user | setup_user_admin_client.sh never run on this host | manual 04, Step 9: run it (expect USER_ADMIN_SETUP_OK), restart the API |
| A user lost their authenticator phone | MFA is required; no self-service recovery | Admin → Users → Reset MFA; the user enrols a new device at next sign-in, password unchanged |
| A user is told the account is temporarily locked | 10 wrong passwords triggered brute-force protection | wait (1 minute, growing to 15); no Admin action is needed |
| A lab ended sooner than expected | per-scenario time limit (30 / 45 / 45 / 60 min) counted from the Start request, so provisioning time is included | expected behaviour (D17); scores are kept |
verify_* scripts fail at sign-in after MFA | password grant now meets MFA for enrolled accounts | use the cyberrange-verify accounts from setup_verify_accounts.sh (cyberrange/docs/SEC-03-verify-accounts.md) |
Diagnostic workflow
Verification checklist
manual/08-verification.md has the acceptance checks. In short: all three user services active; /health and /capacity respond; the portal login page loads through the tunnel; a test lab provisions to ACTIVE, opens a terminal, and scores a milestone in the UI (a green PASS in the portal is the real test, not a passing unit test); teardown returns the slot.
Sources
manual/09-known-issues.md, manual/08-verification.md, manual/04-core-services.md (Step 9), manual/06-scoring.md (Steps 2 and 3b), cyberrange/deploy/host/install_lxd_oci_iptables.sh, setup_zram_and_swap.sh, pull_release.sh, sync_wazuh_api_cert.py, issue_wazuh_internal_ca.py, harden_wazuh_api.sh, setup_wazuh_provisioner.sh, cyberrange/src/provisioning/pod_net.py, provision.py, wazuh_client.py, infra_health.py, cyberrange/docs/WAZUH-ADMIN-DEGRADED-RUNBOOK.md, SEC-03-verify-accounts.md, cyberrange/src/provisioning/migrate.py, cyberrange/deploy/guacamole/nginx-ampere.conf, systemd units.