Provisioning & Teardown

ExplainerHow a lab is built1:25

Provisioning sequence

Step by step

#	Step	Detail	Audit event
1	Admission	One live lab per student, MAX_PODS, lowest free slot, RAM gate, storage gate, all inside one write transaction	none (errors returned as 409/503)
2	Network	Delete any stale pod-<id>-net; resolve the manager address once (used for both the ACL and the agent config); create the pod-<id>-egress ACL and its rules, then the OVN network with the ACL attached and egress default reject; peer to mon-net (retried 3 times); install the return route for the lab subnet inside wazuh-manager. A failed ACL, uplink-DNS lookup or route raises and rolls the lab back. See Containers & Networks	NET_CREATE_OK
3	Kali	Created from kali-base with profiles default and pod-security-base, fixed IP .10, cloud-init user data writes /etc/lab.env (mode 600)	CLONE_OK
4	meta, dvwa	Created from meta-base and dvwa-base with pod-target-base, IPs .20 and .30	CLONE_OK
5	Shell setup	TARGET_META and TARGET_DVWA exported on Kali; enable_history_flush.sh on all three so commands reach history immediately (scoring reads history); passwordless sudo for student on Kali and msfadmin on meta	none
6	Guest networking	Waits until each container's eth0 is up with a route to the lab /24	GUEST_NIC_UP_OK / GUEST_NIC_UP_FAIL
7	Keys	Installs the host operator's public key into root's authorized_keys in each container (refuses keys containing newlines)	NET_APPLY_OK
8	DVWA	Brings up DVWA's database and checks the users table is seeded	DVWA_COMPOSE_OK/FAIL, DVWA_READY_OK/FAIL
9	DVWA proxy	LXD proxy device on 10.115.77.1:18300+id	LAB_PROXY_OK/FAIL
10	Wazuh	Using the manager address from step 2 (WAZUH_MANAGER_IP or its live mon-net IP): remove agents registered under the student's names with the provisioner user, then re-read with the scoring user and refuse the lab if any name is still there (pull request 197); rewrite MANAGER_IP in ossec.conf, start wazuh-agent on meta and dvwa where installed, poll for agent IDs, then wait until every agent reports active (WAZUH_AGENT_CONNECT_TIMEOUT_S, default 90 s, pull request 275); store the IDs as JSON in pods.wazuh_agent_id	WAZUH_ENROLL_OK/FAIL, WAZUH_STALE_AGENT, WAZUH_CLEANUP_SKIPPED
11	Flags	Generate per-student HMAC flags, save them in pod_milestone_flags, plant them in the containers. Scenario 1's final flag is a personal FLAG token written to /opt/cyberrange/flags/scenario1.txt on Kali (pull request 181)	FLAGS_PLANTED_OK/FAIL
12	Ready	Status ACTIVE, heartbeat set	PROVISION_SUCCESS

Step 6, steps 8 and 9, and parts of steps 10 and 11 are best effort: a failure is logged and provisioning continues (the code comment on step 6 reads "fail-open, never rollback for nic"). Since pull requests 197 and 275 the SIEM step is no longer fully best effort, because a lab whose agents cannot report would score Scenario 3 blind:

Step 10 outcome	Result
Wazuh scoring not configured (WAZUH_SCORING_* unset)	WAZUH_ENROLL_FAIL, lab still becomes ACTIVE without SIEM scoring
A same-name agent is still registered after cleanup	WAZUH_STALE_AGENT, lab rolled back. Usually the provisioner user is missing, so End Session never removed the student's previous agents
An agent never gets an ID, or is not active within the timeout	WAZUH_ENROLL_FAIL, lab rolled back
TLS error, or HTTP 401/403, from the Wazuh API	WAZUH_ENROLL_FAIL (tls: or auth HTTP ...), lab rolled back
Any other HTTP or unexpected error	WAZUH_ENROLL_FAIL, lab still becomes ACTIVE

Step 11 is fatal for Scenario 1 only (pull request 181): if its flags cannot be generated, planted or saved (for example COHORT_FLAG_SECRET is unset), the step logs FLAGS_PLANTED_FAIL with a fixed reason code (never the flag or the raw error) and the lab is rolled back. For other scenarios a flag failure is still logged and ignored. Any exception in the other steps, including a malformed host key in step 7, also triggers the rollback below. The worst-case SIEM wait is about 135 seconds (the 45-second ID poll plus the 90-second connect wait).

Source: perform_provisioning() in cyberrange/src/provisioning/provision.py.

Lab status lifecycle

DESTROYED and FAILED_ROLLBACK_COMPLETE are terminal for capacity purposes: they free the slot and let the student start a new lab (_TERMINAL in provision.py; the v2 partial unique index excludes both).

Teardown paths

Path	Trigger	Guard	Notes
Student ends session	DELETE /pods/{id}/destroy	owner only; lab ACTIVE or FAILED_ROLLBACK_COMPLETE	status set to DESTROYING, destruction in the background
Admin force-destroy	DELETE /admin/pods/{id}/force-destroy	admin only; lab ACTIVE or FAILED_ROLLBACK_COMPLETE (409 for PROVISIONING or DESTROYING)	audited as ADMIN_POD_FORCE_DESTROY with actor and previous status; a denied non-admin attempt writes nothing
TTL expiry	reaper, every 60 seconds (TTL_CHECK_INTERVAL_SECONDS), against the scenario's limit (30 / 45 / 45 / 60 min from created_at)	compare-and-swap ACTIVE to DESTROYING	logged as POD_TTL_REAP. Deploying the change reaps any running lab already past its new limit within a minute
Stuck build	reaper	PROVISIONING for longer than STUCK_POD_GRACE_MINUTES (30)	set to DESTROYING, then cleanup_after_failure ends in FAILED_ROLLBACK_COMPLETE; logged as POD_STUCK_PROVISIONING_REAP
Stuck teardown	reaper	DESTROYING for longer than 30 minutes	perform_destruction retried; logged as POD_STUCK_DESTROYING_REAP
Failed build	exception in perform_provisioning	automatic	PROVISION_FAILED, then rollback

What teardown removes

perform_destruction() and cleanup_after_failure() in provision.py:

1. Wazuh agents: removed with the least-privilege provisioner API user by stored ID and by the lab's agent names, with purge=true so the names are free for the student's next lab. If WAZUH_PROVISION_USER and WAZUH_PROVISION_PW are not set, cleanup is skipped and logged, and (on a host where Wazuh scoring is configured) audited as WAZUH_CLEANUP_SKIPPED (shown as SIEM clean-up skipped in the Admin audit log), and the student's next lab will be refused with WAZUH_STALE_AGENT (pull request 211).
2. DVWA proxy device.
3. Containers: each of kali, meta and dvwa is force-stopped and deleted (a container that is already gone is skipped).
4. Network: the return route inside wazuh-manager (a missing route or a timeout never blocks teardown), the peers, then pod-<id>-net, then the pod-<id>-egress ACL (LXD refuses to delete an ACL a network still uses).
5. Database: the storage reservation is dropped, the lab row becomes DESTROYED (or FAILED_ROLLBACK_COMPLETE after a failed build), and the lab's session in lab_sessions is closed.

If any container fails to delete, the lab stays DESTROYING and the reaper retries later. Scores are never deleted with the lab.

Safeguards against races and leaks

• Admission runs in BEGIN IMMEDIATE, and slow host probes happen before taking the lock so a hung LXD call cannot stall every writer.
• The reaper and destroy routes use conditional updates (UPDATE ... WHERE status='ACTIVE'), so only one actor tears down a lab.
• Orphaned networks are removed only after two consecutive sweeps; orphaned storage reservations are purged on every sweep. Orphaned peers also lose their manager return route, and orphaned egress ACLs are deleted.
• The sweep refreshes ACLs only for ACTIVE labs, never for labs still PROVISIONING or DESTROYING, so it cannot recreate an ACL that teardown just removed.
• A failed build always ends in a terminal status, even when LXD cleanup itself fails, so a broken lab cannot hold a slot forever.

Sources

cyberrange/src/provisioning/pods_router.py (provision_pod, destroy_pod, admin_force_destroy_pod), provision.py (perform_provisioning, plant_flags_during_provision), pod_net.py, lab_proxy.py, reaper.py, wazuh_client.py (finish_enroll, abort_if_names_remain, wait_for_agents_active, deregister_agents), flag_planting.py, lab_history.py, capacity.py, manual/06-scoring.md (Step 3b), manual/09-known-issues.md (Issue 14).