Provisioning API (FastAPI)

The backend is a single FastAPI application, "Cyber Range Pod Provisioning API (LXD Version)" version 1.2.0, in cyberrange/src/provisioning/. It owns the lab lifecycle, scoring, progress, reviews, instructor views and admin operations.

Framework and runtime

Item	Value	Source
Framework	FastAPI with Pydantic models	provision_api_fastapi.py, models.py
Server	uvicorn, started by python3 provision_api_fastapi.py	provision_api_fastapi.py (__main__)
Bind	API_BIND_HOST:API_BIND_PORT, default 10.115.77.1:5000 (private bridge address, not the public NIC)	config.py
Process manager	systemd user unit cyberrange-provision-api, Restart=always, logs appended to /tmp/provision-api.log	cyberrange/deploy/systemd/cyberrange-provision-api.service
Database	SQLite via the standard sqlite3 module	db.py
LXD access	pylxd over the local Unix socket, plus the lxc CLI for networks and peers	provision.py, pod_net.py

Directory structure

See the full module table in Components. In short:

cyberrange/src/provisioning/
├── provision_api_fastapi.py   # entry point, startup, /health, /capacity
├── pods_router.py             # labs, scoring, progress, reviews, instructor, images
├── alerts_endpoint.py         # SIEM views (student + instructor)
├── users_router.py            # admin user management (Keycloak), research exclusions
├── onboarding_router.py       # first-login orientation state
├── infra_health.py  audit_router.py
├── auth.py  introspect_cache.py
├── provision.py  pod_net.py  lab_proxy.py  guest_nic.py  dvwa_*.py
├── reaper.py  score_poller.py  scoring.py  ssh_verifier.py  scoring_checks.sh
├── score_verifier.py  wazuh_rule_map.py  wazuh_client.py  alerts_reader.py
├── scoring_diagnostics.py     # instructor-only failure reason codes
├── hybrid_scoring.py  rubrics.py  flag_planting.py  lab_history.py  evidence_images.py
├── knowledge_gain.py  research_exclusions.py
├── capacity.py  ttl.py  config.py  profiles.py  secrets_loader.py  logging_config.py
├── db.py  migrate.py  schema/v1.sql ... v16.sql
└── test_*.py                  # pytest suite (not run in CI; see Accuracy Notes)

Application startup

provision_api_fastapi.py does the following when the process starts:

1. Configures logging and loads the scoring modules dynamically (scoring_imports.load_scoring_modules()). If ssh_verifier cannot load, scoring is disabled and verification returns 503; if score_verifier or wazuh_rule_map cannot load, SIEM detection evidence is disabled.
2. Installs a RequestValidationError handler (pull request 198): on POST /admin/users and PUT /admin/users/{id}/password a 422 lists only the field location and a fixed message per error type, never the rejected input or Pydantic's own message; other routes keep FastAPI's default 422 body.
3. Mounts the routers: pods_router.router, pods_router.internal_router, users_router, infra_health, audit_router, and, with the require_app_role dependency, onboarding_router and alerts_endpoint.alerts_router.
4. On startup:
   • validate_auth_config(): refuses to start if AUTH_ENABLED is true but KEYCLOAK_CLIENT_SECRET or KEYCLOAK_INTROSPECT_URL is missing; probes introspection (non-fatal) and logs CRITICAL if Keycloak is unreachable.
   • validate_capacity_config(MAX_PODS): refuses to start with an invalid capacity setting.
   • init_db(): applies pending SQL migrations.
   • Starts two background loops: the TTL reaper and the score poller. See Background Jobs.

Routers

Router	Mounted with	Main routes
pods_router.router	APIRouter(dependencies=[Depends(auth.require_app_role)])	/pods/*, /progress/*, /reviews/*, /instructor/* (except alerts), /admin/pods/{id}/force-destroy
pods_router.internal_router	no user auth; shared secret	POST /internal/browser-score, POST /internal/browser-diagnostic (a Scenario 2 near-miss reason code from the DVWA proxy)
alerts_endpoint.alerts_router	require_app_role added in provision_api_fastapi.py	/pods/{id}/alerts, /instructor/pods/{id}/alerts, /instructor/students/{id}/labs..., /instructor/reviews/{id}/alert-snapshot
users_router	per-route _require_admin()	/admin/users..., including PUT /admin/users/{id}/research-exclusion
onboarding_router	require_app_role (any signed-in role)	GET and POST /users/me/orientation (first-login orientation, keyed by the token's sub)
infra_health	per-route require_role(["admin"])	/admin/infra-health
audit_router	per-route require_role(["admin"])	/admin/audit-log
app	none	GET /health, GET /capacity

The complete list with guards is in the API Reference.

Middleware

There is no custom HTTP middleware and no CORS middleware (the only app-level hook is the validation-error handler above). Cross-cutting concerns are implemented as FastAPI dependencies:

• verify_token (authentication, token introspection)
• require_app_role (router-level: any application role)
• require_role([...]), require_owner, require_owner_or_admin (called inside handlers)

The lack of CORS is consistent with the architecture: browsers never call the API directly; only the portal server, the bridge and host scripts do.

Error handling

Status	Meaning in this API	Example
400	Invalid input the handler checks itself	unsupported milestone, invalid status filter
401	No or malformed bearer token, inactive token, missing identity, wrong browser-score secret	verify_token()
403	Authenticated but missing the required role	require_role()
404	Not found or not yours (ownership failures deliberately look like "not found")	require_owner()
409	State conflict	second lab, lab not ACTIVE, review not in RETRY
422	Request body fails Pydantic validation; scoring and flag bodies also reject unknown fields	FastAPI default; sanitized on the admin password routes
503	Dependency unavailable or capacity full	Auth service unavailable, POD_CAP_REACHED, RAM_FULL, STORAGE_FULL, Scoring engine not available
500	Unexpected database error during provisioning	Database error: ...

Long-running work never blocks a request: provisioning and destruction run in FastAPI BackgroundTasks, and failures are recorded as audit events (for example PROVISION_FAILED) rather than raised to the client.

Configuration

Configuration comes from environment variables (the systemd unit loads ~/cyberrange/env/.env). Secrets are read through secrets_loader.get_secret(): the process environment first, then an optional SECRETS_FILE of KEY=value lines. If VAULT_ADDR and VAULT_SECRET_PATH are set, the loader fails closed because a Vault backend is not implemented.

Group	Variables	Defaults / notes
Profile and capacity	PROFILE, MAX_PODS, POD_RAM_MB, RAM_BUFFER_MB, POD_STORAGE_MB, STORAGE_LIMIT_MB, LXD_STORAGE_POOL	profiles onprem (3 labs) and oci_12gib (1 lab) in profiles.py
Lifecycle	SCENARIO_TTL_MINUTES (in code), DEFAULT_TTL_MINUTES, TTL_CHECK_INTERVAL_SECONDS, REAP_INTERVAL_SECONDS, STUCK_POD_GRACE_MINUTES, SCORE_POLL_INTERVAL_SECONDS	30 / 45 / 45 / 60 min, 60 min, 60 s, 600 s, 30 min, 3 s. POD_TTL_HOURS is no longer read (startup warns if set)
Binding and storage	API_BIND_HOST, API_BIND_PORT, DB_PATH, EVIDENCE_IMAGE_DIR	10.115.77.1, 5000, ~/cyberrange-data/pod_mgmt.db
Identity	AUTH_ENABLED, KEYCLOAK_CLIENT_ID, KEYCLOAK_CLIENT_SECRET, KEYCLOAK_INTROSPECT_URL, KEYCLOAK_USER_ADMIN_CLIENT_ID, KEYCLOAK_USER_ADMIN_CLIENT_SECRET	AUTH_ENABLED=false is a logged full bypass for bootstrap only
Wazuh	WAZUH_API_URL, WAZUH_SCORING_USER, WAZUH_SCORING_PW, WAZUH_PROVISION_USER, WAZUH_PROVISION_PW, WAZUH_CA_BUNDLE, WAZUH_CERT_PATH, WAZUH_TLS_VERIFY, WAZUH_MANAGER_IP, WAZUH_MANAGER_INSTANCE, WAZUH_AGENT_CONNECT_TIMEOUT_S	API URL defaults to https://localhost:55000 because the certificate only names localhost; manager instance wazuh-manager; connect wait 90 s
Lab network	UPLINK_DNS_IP	optional; otherwise read from vmbr1's ipv4.address for the ACL's DNS reject
Scoring diagnostics	SCORING_DIAGNOSTICS_SCENARIOS, SCORING_DIAGNOSTICS_RETENTION_DAYS, S3_DIAGNOSTICS_ENABLED	read by scoring_diagnostics.py; off unless listed (scorer ids, for example 9 or 9,1); 14 days (S3_DIAGNOSTICS_RETENTION_DAYS is the fallback); S3_DIAGNOSTICS_ENABLED=1 still turns on scenario 9
Scoring and telemetry	BROWSER_SCORE_SECRET, COHORT_FLAG_SECRET, TELEMETRY_ANONYMIZATION_SALT, SCORING_SCRIPT_PATH	COHORT_FLAG_SECRET is not in the template (see Open Questions)
Lab proxy	LAB_PROXY_PORT_BASE, LAB_PROXY_LISTEN, LAB_PUBLIC_HOST	port base 18300
Logging	LOG_FORMAT	text (default) or json

Template with placeholders only: cyberrange/env/oci-12gib.env.example.

Communication with infrastructure

Target	How	Used for
LXD	pylxd.Client (instances: create, start, stop, delete, exec, file API)	lab containers, scoring, reading Wazuh files
LXD networks	lxc network create/delete/set, lxc network acl create/delete and acl rule add/remove; peers through sudo -n /usr/local/sbin/cyberrange-lxc-peer (allows only network peer create/delete); lxc exec wazuh-manager -- ip route replace/del for the return route	per-lab OVN networks, egress ACLs, manager return routes
LXD proxy devices	lxc config device add ... proxy	DVWA access on 18300 + pod_id
Scoring	lxc exec <container> -- bash -s <scenario> <milestone> with scoring_checks.sh on stdin	milestone checks
Keycloak	HTTP: token introspection (auth.py); admin REST API with a service-account token (keycloak_admin.py)	authentication, user admin, health
Wazuh	LXD file API reads alerts.json and daily archives from the wazuh-manager container; HTTPS to localhost:55000 with the pinned certificate or internal CA for agent lookup, status and deletion	SIEM views, detection evidence, enrollment wait, agent cleanup

Logging and audit

• Application logs go to stdout (text or JSON, logging_config.py) and the unit appends them to /tmp/provision-api.log, which is cleared on reboot.
• Business events go to the audit_log table through db.log_event(): provisioning steps (NET_CREATE_OK, CLONE_OK, WAZUH_ENROLL_OK/FAIL, WAZUH_STALE_AGENT, FLAGS_PLANTED_OK/FAIL, PROVISION_SUCCESS/FAILED), SIEM housekeeping (WAZUH_CLEANUP_SKIPPED), scoring (MILESTONE_VERIFIED), teardown (POD_TTL_REAP), reviews (REVIEW_CASE_RESOLVED), resets (SCENARIO_PROGRESS_RESET) and admin actions (ADMIN_USER_* including ADMIN_USER_MFA_RESET and ADMIN_USER_RESEARCH_EXCLUDE/INCLUDE, ADMIN_POD_FORCE_DESTROY).
• The Admin audit log can filter by several event types at once (event_types, at most 64) and its OK / Failed filters also match events whose outcome is only in the name (*_OK, *_FAILED), not in result (audit_router.py).
• The portal BFF logs only an allow-listed network error code when the API is unreachable, never the exception message (portal/src/lib/apiProxy.ts).

Sources

cyberrange/src/provisioning/provision_api_fastapi.py, config.py, profiles.py, secrets_loader.py, auth.py, pods_router.py, users_router.py, onboarding_router.py, audit_router.py, pod_net.py, wazuh_client.py, scoring_diagnostics.py, models.py, db.py, logging_config.py, schema/README.md, cyberrange/env/oci-12gib.env.example, cyberrange/deploy/systemd/cyberrange-provision-api.service.