Status: source of truth for the runner software. When the runner's design changes, update this document in the same change. When a test plan's fixture surface changes, reconcile it here.
Related documents
- Specs — specs/: what MUST be true (
spec-ucentral.md,spec-cloud-discovery.md,spec-pki-2.0.md,spec-config-conformance.md). - Test plans — testplans/: the scenarios the runner executes, per spec.
- This document: how the software that runs those scenarios is built.
The runner is one program that executes the four certification test plans against an access point under test and produces a signed certification artifact per release. It:
- brings up, in-process, the cloud services the AP expects to talk to (the "software rig"),
- drives the AP into each scenario's starting state through a lab-supplied driver,
- applies the scenario stimulus,
- observes only externally visible signals,
- evaluates them against the plan (or, for config conformance, against an oracle),
- reports PASS / FAIL / ERROR and rolls results into the certification artifact.
In scope: the runner's structure, abstractions, execution model, and package layout. Out of scope: the requirements themselves (specs), the scenario catalogue (test plans), and the physical lab hardware (reached through a driver boundary, §7).
- Black-box first. Conformance is judged from outside the AP. On-device access (SSH) is an optional switch that enables extra grey-box checks (config-conformance L2), never the default and never the gate.
- The runner contains the rig. The stubs, gateway, DHCP/DNS, firmware host, and packet plane are in-process runner modules, not external systems. The only external boundary is the physical AP and its lab-specific control (§7).
- Deterministic & pinned. Every run is driven by a manifest that pins the schema commit, firmware build, AP models, cert profiles, and timeouts (§11). No implicit "latest".
- Three outcomes only. PASS / FAIL are conformance results; ERROR is a rig/oracle fault that invalidates the run until fixed (§10). No silent skips (except the SSH-gated L2 stage, which is explicitly optional).
- Oracle-driven conformance. For config-schema conformance the correct answer is computed by a reference oracle, not hand-encoded per scenario (§9).
- Lab-portable. Hardware differences are isolated behind the
DutControllerinterface so scenarios are written once and run in any lab.
┌──────────────────────────── RUNNER (src/) ─────────────────────────────┐
│ │
│ CLI / entrypoint │
│ │ │
│ Orchestration engine ── Scenario suites (ucentral · discovery · │
│ │ pki · config-conformance) │
│ │ │
│ ┌─────┴───────────────┐ ┌──────────────────┐ ┌───────────────────┐ │
│ │ Software rig │ │ Observation │ │ Oracle + corpus │ │
│ │ (in-process services)│ │ layer │ │ + coverage │ │
│ │ gateway stub (mTLS) │ │ collectors → │ │ (conformance) │ │
│ │ cds_stub, est_stub │ │ unified timeline│ └───────────────────┘ │
│ │ dhcp, dns, firmware │ │ → assertions │ │
│ │ net_control (nft) │ └──────────────────┘ ┌───────────────────┐ │
│ └──────────────────────┘ │ Reporting + │ │
│ │ cert artifact │ │
│ DutController / SSH / wired-client interfaces └───────────────────┘ │
│ │ │
└────────┼─────────────────────────────────────────────────────────────── ┘
│ ← the only external boundary
┌─────┴───────────────────────────────┐
│ Physical AP (DUT) │
│ + lab hardware: PoE, TFTP, SSH, │
│ wired-client laptop/Pi │
└──────────────────────────────────────┘
Everything except the AP and its lab-specific control driver lives inside the runner.
| Layer | Responsibility | Key modules |
|---|---|---|
| CLI / entrypoint | Parse args, load manifest, select suites/models, choose mode (black-box / SSH) | cli.py |
| Orchestration engine | Run lifecycle, model×scenario matrix, scheduling (incl. soak), isolation/reset, result aggregation | runner/engine.py, runner/scheduler.py |
| Scenario framework | Base Scenario, declarative setup/stimulus/observe/assert, discovery/registry, parametrization |
runner/scenario.py |
| Software rig (fixtures) | In-process services impersonating the cloud + controlling the packet plane | fixtures/* |
| DUT boundary | Abstract control of the physical AP; lab-specific drivers | dut/* |
| Observation | Tap every signal source, normalize into one timeline, provide the assertion API | observation/* |
| Oracle / corpus / coverage | Conformance reference verdict, config corpus, schema-node coverage | oracle/*, corpus/*, coverage/* |
| Reporting | Results model, REQ-coverage matrix, coverage report, signed cert artifact | reporting/* |
| Cross-cutting | Manifest, clock, logging/tracing, cert & key store, error taxonomy | runner/manifest.py, observation/clock.py, tools/* |
Interfaces are the contract; concrete classes are swappable. Pseudocode, not final signatures.
class Manifest: # §11 — pinned inputs for a run
schema_sha: str; schema_version: tuple
firmware_build: str; ap_models: list[Model]
cert_profiles: dict; timeouts: dict; uci_overrides: dict
ssh_switch: bool; fixture_versions: dict
class RunContext: # handed to every scenario
fixtures: FixtureManager # gateway, cds, est, dhcp, dns, firmware, netctl
dut: DutController # the AP boundary
observe: Timeline # unified observation bus
oracle: Oracle | None # conformance only
clock: Clock; manifest: Manifest
class Fixture(ABC): # every in-process rig service
def setup(self); def reset(self); def teardown(self)
def program(self, **kw): ... # scenario-specific configuration
def logs(self) -> Iterable[Event]
class DutController(ABC): # the external boundary (lab-supplied)
def power_cycle(self); def flash(self, image_id: str)
def set_uci(self, key, value); def mac(self) -> str
def ssh(self, cmd: str) -> str # only when manifest.ssh_switch
class Scenario(ABC):
id: str; plan: str; reqs: list[str]; tags: set[str]; soak_budget: float
min_schema_version: tuple | None # version applicability (see below); None = unbounded
max_schema_version: tuple | None
def setup(self, ctx: RunContext) # program rig + DUT state
def stimulus(self, ctx: RunContext) # the action under test
def observe(self, ctx: RunContext) -> Observations
def check(self, ctx, obs) -> Result # assertions → PASS/FAIL/ERROR
class Collector(ABC): # one per signal source
def events(self) -> Iterable[Event] # normalized, timestamped
class Oracle: # conformance reference (§9)
def verdict(self, config, caps) -> Verdict # {error, rejected[]}
def render(self, config, caps) -> Uci # L2 UCI, SSH mode
class Result:
scenario_id: str; outcome: PASS|FAIL|ERROR
reqs: list[str]; evidence: list[Event]; notes: strmain()
└─ load Manifest (pins) ────────────────────────────────────── §11
└─ Orchestrator.run():
build RunContext
FixtureManager.start_all() # gateway, stubs, dhcp, dns, fw, netctl
for model M in manifest.ap_models:
dut.flash(M.image); wait_online(baseline)
for scenario S in selected_suites (ordered, soak-aware):
S.setup(ctx) # program fixtures + DUT into start state
Timeline.mark(t0) # clear/checkpoint all collectors
S.stimulus(ctx) # push config / send RPC / cut traffic / cycle
obs = S.observe(ctx) # collect within the scenario window
result = S.check(ctx, obs)
record(result); S.teardown / reset for isolation
# conformance suite additionally:
run_corpus_through(configure_driver, oracle) # §9
coverage = CoverageModel.compute() # gate REQ-CC-09
└─ Reporter.emit(results, coverage) → cert artifact # §8, §12
Fixtures start once per run; DUT state and fixture programming reset per scenario for isolation. Long-soak scenarios (backoff caps, CN cooldowns, expiry windows) are grouped by the scheduler so their wall-clock overlaps where safe.
The DUT is always a real, physical AP supplied by whoever is being certified. There is no
virtual or emulated AP anywhere in this system — the runner never builds, images, or emulates the
device. The physical AP and its lab hardware are the one thing the runner cannot contain; they are
reached only through DutController (and, when enabled, ssh). This keeps scenarios portable.
- Reference drivers ship for a PoE-cycle + TFTP-flash lab and a
prompt-the-operatorbenchtop driver; both satisfy the same interface. - Wired-client harness (a laptop/Pi on the AP LAN, used to trigger wifi events in the uCentral plan) is a second driver behind the boundary — controlled, not impersonated.
- SSH driver is active only when
manifest.ssh_switchis true (config-conformance L2). On a locked production image SSH is absent and L2 scenarios are SKIPPED, not ERROR.
Each is a Fixture; the runner starts, programs, resets, and reads logs from it. Sourced from
the plans' fixture surfaces.
| Service | Impersonates / controls | Used by |
|---|---|---|
gateway |
purpose-built uCentral gateway (our own websocket + mTLS JSON-RPC server — NOT the real wlan-cloud-ucentralgw). Presents a swappable server-cert profile, requests + records the AP's client cert, logs every JSON-RPC frame with timestamps, closes sessions on demand, and injects gateway→device methods (configure, reboot, request, …) as direct calls — no NB-API/DB in the loop. Emits malformed frames on demand (bad jsonrpc, crafted errors, unknown methods) for the negative-path scenarios. |
all plans |
cds_stub |
CDS HTTPS+mTLS — per-MAC responses, swappable server cert, logs SNI/client-cert/path | discovery, pki |
est_stub |
EST HTTPS+mTLS — programmable simpleenroll/reenroll/cacerts, fault modes (stall, 5xx, bad PKCS#7), logs client cert + CSR | pki, discovery (gate) |
dhcp_server |
per-MAC options 138/224/43/42; logs DISCOVERs | discovery |
dns_resolver |
A / NXDOMAIN / CAA programming incl. openwifi.wlan.local |
discovery, pki |
firmware_host |
HTTPS serving valid vs corrupt images | ucentral, discovery |
net_control |
nftables packet plane — TCP blackhole, per-direction drop/unblock, SYN drops | discovery, pki |
Why we build the gateway instead of running real owgw. The plans observe and inject raw JSON-RPC frames and deliberately misbehaving TLS (malformed frames, cert swaps, downgrade, on-demand session close). owgw is built specifically not to do those things, carries a database/config/NB-API lifecycle we'd have to orchestrate, and would make the same component both instrument and unit-under-test. A purpose-built gateway gives the runner direct frame-level control, keeps the whole rig uniform (Python, in-process), and is deterministic and fast. The protocol surface it must implement is closed and specified — exactly the dispatch set in spec-ucentral.md REQ-UC-37 — so the built gateway carries a conformance obligation: it MUST implement that set and nothing more, and (optionally) is cross-checked against real owgw on the happy path periodically, not every run.
Gateway-side policy is out of scope for AP self-cert. PKI GW-01..03 (REQ-PKI-23/24/25) certify a real gateway's TLS policy; a built gateway can't stand in (you'd be testing the stub). Those move to a separate, optional gateway-policy suite run against real owgw — they are not part of the AP conformance the built gateway serves.
The config-conformance plan is differential: an oracle predicts the correct verdict and the runner grades the AP against it.
- Oracle — wraps the sibling
openlan-schema-validator(renderer-aware) +jsonschema, pinned to the manifest schema SHA, adjusted to the reader's real semantics (unknown keys dropped,error0/1/2 codes). Returns the predicted verdict and, in SSH mode, oracle-rendered UCI for the L2 diff. - Corpus — four generators (golden positives, schema-walk positives, mutation negatives, capability-mismatch), each config tagged with the schema node(s) it exercises.
- Coverage — a schema-node inventory built from the baseline; corpus tags are joined against it to prove 100% platform-applicable coverage (REQ-CC-09). A coverage gap fails the run.
- PASS / FAIL — conformance results for a scenario.
- ERROR — a rig or oracle fault (fixture down, timeout, oracle self-contradiction pending reconcile). ERRORs invalidate the run; they are not conformance verdicts.
- SKIPPED — reserved solely for the SSH-gated L2 stage when the switch is off. No other scenario may skip; a missing fixture is ERROR, not SKIP.
- Coverage FAIL — conformance suite only: any uncovered platform-applicable schema node fails the run even if every executed case passed.
A per-release manifest is the single source of pinned inputs, so a run is reproducible. Every
source pin derives from one anchor — the wlan-ap release tag — extracted deterministically
(see runner/pin.py), never hand-entered:
release: 4.2.6
wlan_ap_tag: v4.2.6 # the single anchor; all SHAs below are read out of it
schema: { version: [4, 2, 6], sha: 9dc64a8… } # feeds/ucentral/ucentral-schema/Makefile
client: { sha: cb48fe27… } # feeds/ucentral/ucentral-client/Makefile
cloud_discovery: { sha: <wlan-ap tag commit> } # in-tree → the wlan-ap commit itself
# owgw (wlan-cloud-ucentralgw): intentionally absent — not part of the AP release; its
# gateway-side REQs (PKI-23/24/25) are a separate, optional gateway-policy suite.
firmware: { build: <id/url> }
ap_models: [ { name: …, image: …, capabilities: … }, … ]
cert_profiles: [ prod, demo, unknown, bad-cn ]
timeouts: { validate: 120, offline: …, expiry_interval: 60, … } # incl. UCI overrides
ssh_switch: false # true → enable config-conformance L2
fixtures: { gateway: <ver>, cds_stub: <ver>, … }The manifest is what the specs' verified-against header, the runner, and the oracle/corpus/
coverage all reference, so conformance is always computed against the exact pinned commits
(REQ-CC-01, REQ-CC-10). Most releases only re-pin these SHAs; a spec REQ changes only when the
protocol/schema surface itself changes.
- Per-scenario results — outcome + evidence (the timeline events that decided it) + REQs.
- REQ-coverage matrix — every REQ mapped to the scenarios that exercised it (mirrors the matrices in the test plans).
- Schema-node coverage report — conformance only; covered/uncovered node list.
- Certification artifact — signed summary bound to the manifest (schema SHA, firmware, models, mode). Formats: machine-readable (JSON) + human summary; JUnit XML for CI.
openlan-self-certification/
├── specs/ # requirements (exist)
├── testplans/ # scenario catalogue (exist)
├── ARCHITECTURE.md # this document — source of truth
├── manifests/
│ └── release-<ver>.yaml # pinned run manifest (§11)
├── src/
│ ├── cli.py # entrypoint
│ ├── runner/ # engine, scheduler, scenario, manifest, context, result
│ ├── fixtures/ # in-process rig: gateway, cds_stub, est_stub,
│ │ # dhcp_server, dns_resolver, firmware_host, net_control
│ ├── dut/ # DutController base + poe_tftp, manual, ssh, wired_client
│ ├── observation/ # collectors, timeline, assertions, clock
│ ├── oracle/ # conformance oracle (wraps openlan-schema-validator) + render
│ ├── corpus/ # golden/, schema_walk.py, mutate.py, capability_mismatch.py
│ ├── coverage/ # schema-node inventory + coverage gate
│ ├── reporting/ # results model, formatters, cert artifact
│ └── suites/ # scenario suites per plan:
│ ├── ucentral/ cloud_discovery/ pki/ config_conformance/
├── tools/
│ ├── mint_certs.py # birth/operational cert minting (shared by plans)
│ └── tls_poison_client.py # separate/optional gateway-policy suite (PKI GW-*), vs real owgw
└── vendored/ # pinned externals
├── wlan-ucentral-schema@<sha> # baseline schema (submodule)
└── openlan-schema-validator # oracle engine (submodule)
- New scenario → add to the relevant
suites/<plan>/; reuse fixtures, observation, DUT. - New plan → new suite package + (if needed) new fixtures; engine/observation unchanged.
- New lab → implement
DutController(+ SSH / wired-client) indut/; scenarios unchanged. - New schema release → bump the manifest schema pin; oracle/corpus/coverage re-pin automatically; coverage recomputed against the new node inventory.
- New signal source → add a
Collector; assertions query the same timeline API.
| Plan | Primary fixtures | Extra components |
|---|---|---|
| uCentral protocol | gateway, firmware_host | wired-client driver; schema validator (state/health) |
| Cloud Discovery | gateway, cds_stub, est_stub, dhcp, dns, net_control | DutController (power/flash/uci/ntp), soak scheduler |
| PKI 2.0 | gateway, est_stub, cds_stub, dns (CAA) | mint_certs.py, tls_poison_client, soak scheduler |
| Config Schema Conformance | gateway | oracle + corpus + coverage; SSH driver (L2) |