Background Jobs & Integrations

Besides request handlers, the API runs background loops and several service modules that integrate with LXD, Keycloak and Wazuh.

Background loops

Both loops are started as asyncio tasks in the API's startup handler (provision_api_fastapi.py).

TTL reaper (reaper.py)

One loop with two cadences. TTL expiry runs every TTL_CHECK_INTERVAL_SECONDS (default 60), so a 30-minute lab cannot overrun by minutes; the sweep (steps 2 to 4) runs every REAP_INTERVAL_SECONDS (default 600). Keeping them in one loop means a sweep never races an in-flight TTL destroy.

1. TTL expiry: every ACTIVE lab older than its scenario's limit (ttl_minutes_for() in ttl.py: 30 for 01, 45 for 06 and 09, 60 for 11, DEFAULT_TTL_MINUTES otherwise) is switched to DESTROYING with a compare-and-swap update (only one worker can win), logged as POD_TTL_REAP, then destroyed.
2. Stuck labs: labs still PROVISIONING or DESTROYING after STUCK_POD_GRACE_MINUTES (default 30) are cleaned up or retried, so a crash cannot hold a capacity slot forever.
3. Storage drift: deletes storage_reservations rows that no longer belong to a live lab.
4. Network reconciliation: deletes pod-<id>-net networks and peers that have no live lab, but only after seeing the orphan on two consecutive sweeps, along with orphaned egress ACLs and the manager return route of an orphaned peer (pod_net.reconcile_pod_networks()). For live labs it re-installs the return route inside wazuh-manager, and for every ACTIVE lab it brings the egress ACL and the network's ACL settings back in line (ensure_pod_acl, pull request 309). The ACL refresh is skipped for the whole sweep, with an error logged, when the manager address (only WAZUH_MANAGER_IP or a successful live read count, never the default) or the uplink resolver cannot be determined.

Score poller (score_poller.py)

Runs every SCORE_POLL_INTERVAL_SECONDS (default 3). For each ACTIVE lab with a scenario, it finds milestones the student has not passed yet (SCENARIO_MILESTONE_COUNTS: scenario 1 has 4, 6 has 4, 9 has 3, 11 has 3; the flag-only fifth tasks of scenarios 1 and 6 are never polled) and runs scoring.verify_milestone() for each, tagged source='auto'. This is what makes "wait for auto-detect" in the guides true. At most once an hour it also purges scoring-diagnostic rows older than the retention period, even when diagnostics are switched off.

Growth to be aware of

Every check, including a FAIL, inserts a row into milestone_verification and an event into audit_log. Since pull request 268 (schema v15) each row records its source (auto, manual, browser, flag or review; NULL for older rows) so instructors can tell poller rows from the student's own attempts, but the row count is unchanged. With 3 or 4 unpassed milestones, one active lab can add about 1 to 1.3 rows per second to each table until the milestones pass, which can reach a few thousand rows over a 30 to 60 minute session (far less than under the old 8-hour limit). See Database.

Scoring modules

Module	What it does
scoring.py	verify_milestone(): for Scenario 2 returns an existing browser PASS immediately; otherwise runs the verifier, adds SIEM detection evidence when configured, inserts the result with its source and the lab's pod_created_at (insert_verification_row()), logs MILESTONE_VERIFIED, then records a diagnostic reason when diagnostics apply
scoring_diagnostics.py	Instructor-only failure reasons (schema v14): scoring_checks.sh prints a DIAG reason=<CODE> line before its result for scenarios 1, 9 and 11, and the DVWA proxy posts Scenario 2 near-miss codes to /internal/browser-diagnostic. Only allow-listed codes are kept, one row per student, lab, scenario and milestone (a later PASS replaces it), never artifact contents or shell output. Each scenario is switched on separately with SCORING_DIAGNOSTICS_SCENARIOS; rows expire after SCORING_DIAGNOSTICS_RETENTION_DAYS (default 14). Scoring never depends on it
ssh_verifier.py	Class SSHVerifier (name kept for compatibility): runs lxc exec pod-<student>-<role> -- bash -s <scenario> <milestone> with scoring_checks.sh on stdin; 15 s timeout (30 s for scenario 9 and above); returns PASS, FAIL, UNKNOWN or ERROR
scoring_checks.sh	The milestone checks for every scenario (behavioral history checks, artifact checks, live-state checks)
score_verifier.py	verify_siem_alert(agent, rule, since_minutes): reads the newest 64 MiB of the manager's alerts.json through the LXD file API and looks for that rule on that agent
wazuh_rule_map.py	Which milestone has a detection rule: only Scenario 3 Milestone 1 (rule 5710, on meta) among the live scenarios
hybrid_scoring.py, rubrics.py	Flag submission path: compares a submitted flag and the lab state, escalating disagreements to a staff-only review
flag_planting.py	Per-student flags: HMAC-SHA256 of the student, scenario and milestone with COHORT_FLAG_SECRET, planted into the containers at provisioning and stored in pod_milestone_flags. Scenario 1's final flag is written to /opt/cyberrange/flags/scenario1.txt on Kali, and any Scenario 1 planting failure stops the lab (prepare_scenario_flags, Scenario1FlagPlantingError)

Scoring is covered end to end in Lifecycle & Scoring.

Lab history and SIEM views

Module	What it does
lab_history.py	Records each lab's start and end in lab_sessions; when a report is submitted, captures the lab's alerts into review_alert_snapshots in a background task (errors such as an unreachable SIEM are stored on the snapshot, never raised)
alerts_reader.py	Reads live alerts.json (newest 64 MiB) and the manager's gzipped daily archives (up to 10 days, 64 MiB each), filters by the lab's agent names and time window, and keeps only six fields
alerts_endpoint.py	Student and instructor alert routes; ownership (404) and staff checks

Onboarding (onboarding_router.py)

GET and POST /users/me/orientation read and set whether the signed-in user has finished the first-login orientation, stored once per Keycloak subject in user_onboarding (schema v11, pull request 184). The POST never overwrites an earlier completion time.

Evidence screenshots (evidence_images.py)

• Accepts only PNG, JPEG and WebP, decided by decoding the bytes with Pillow, never by file name or declared type.
• Limits: 5 MB per file, 5 per review, 4000 pixels per side and 16 million pixels in total, checked from the image header before any pixel data is decoded. Pillow's decompression-bomb error is turned into a normal rejection.
• Re-encodes every image into a fresh image, which drops EXIF, GPS, color profiles, comments and trailing data. Palette images keep their palette and transparency.
• Stores files under EVIDENCE_IMAGE_DIR (outside the code tree) with random uuid names, created exclusively so nothing can be overwritten; guards against path traversal.
• Served back with X-Content-Type-Options: nosniff and a sandboxing Content Security Policy.

Keycloak integration

Module	Purpose
auth.py	Token introspection with the portal client's credentials; claims cached up to 60 s (introspect_cache.py)
keycloak_admin.py	Admin REST client using the cyberrange-user-admin service account (view, query and manage users only); also lists instructor and admin usernames (staff_usernames()) for the knowledge-gain export
infra_health.py	Health probe: an introspection call that must answer 200 or 401

Wazuh integration

Module	Purpose
wazuh_client.py	API tokens for the read-only scoring user and the provisioner user (agent read and delete); agent lookup by name and status by ID; removal of a lab's agents at teardown by stored ID and by name with purge=true; refuses anything but a single numeric agent ID other than 000. Since pull requests 197, 211 and 275: abort_if_names_remain() refuses a lab when a name survives cleanup (StaleWazuhAgent); wait_for_agents_active() waits for every agent to report active (AgentNotConnected); finish_enroll() decides which errors roll the lab back (stale names, not connected, TLS, 401/403) and which only log WAZUH_ENROLL_FAIL; without the provisioner pair, cleanup is skipped and audited as WAZUH_CLEANUP_SKIPPED
provision.py	At provisioning: resolves the manager address once for both the lab's egress ACL and the agent config, removes stale agents with the student's names, rewrites MANAGER_IP in the agent config, starts wazuh-agent on meta and dvwa (only where the image has the agent), waits for them, records agent IDs
pod_net.py	Installs and removes the manager's return route to each lab subnet (ip route replace ... via the mon-net gateway dev eth0 inside wazuh-manager, pull request 195)
score_verifier.py, alerts_reader.py	Read alerts through the LXD file API (no Wazuh credentials needed for reading)

Knowledge-gain export (knowledge_gain.py)

Computes completion rates, time from lab creation to first PASS, detection indicators and instructor rubric scores for research reporting. Student IDs are anonymized with a salted HMAC by default (the salt must be at least 16 bytes). CSV cells that start with =, +, - or @ are neutralized to prevent spreadsheet formula injection.

Since pull requests 270 and 273: time to milestone is measured from the lab session whose window contains the PASS (lab_sessions), instructor-approved PASS rows carry no time to milestone, timestamps are written as UTC ...Z, and records add the paper's scenario_number (1 to 4). Accounts an Admin marked as test or team accounts (research_exclusions, schema v16) and every instructor or admin username read from Keycloak are dropped before anonymization; the summary and the X-Excluded-Accounts / X-Staff-Exclusion CSV headers say how many were left out and whether the staff list could be read (applied, unavailable or off). No scoring data is deleted.

Sources

cyberrange/src/provisioning/reaper.py, score_poller.py, scoring.py, scoring_diagnostics.py, ssh_verifier.py, score_verifier.py, wazuh_rule_map.py, hybrid_scoring.py, rubrics.py, flag_planting.py, lab_history.py, alerts_reader.py, alerts_endpoint.py, evidence_images.py, keycloak_admin.py, wazuh_client.py, pod_net.py, provision.py, knowledge_gain.py, research_exclusions.py, onboarding_router.py, schema/v11.sql to v16.sql, cyberrange/docs/SCORING-DIAGNOSTICS.md.