Explee: AI Dev Test Task

The collector went live before any dashboard code existed. The API has no history endpoint, so an observation window cannot be reconstructed afterwards; everything downstream replays an append-only raw log, which is why a clean re-measurement was possible hours later without losing a single record.

Three deliverables, one repository, one module per task. Traces are exported by tools/export_trace.py from real sessions. They are never hand-written and never truncated. All three traces are exported and shipped. Each was exported at the genuine end of its session, because an early export stops the trace before the work does.

Start here if you are grading this: task1-spend-observability/ALERT-AUDIT.md replays the window, compares every evidence field against the raw records, runs top-up counterfactuals, and proves side-effect freedom by hashing the log before and after.

It published a failing result for most of this run, at the time 2 of 13 lines unreconciled, because that was what the deployed build actually produced, and a passing audit generated by undeployed code would have described a system nobody was running. The cause was then found and fixed: one line was a 180-hour projection that a 0.04 % balance change could flip. It passed at that point, 12 lines and 0 unreconciled.

It does not pass today either, and the finding is named rather than hidden. Run uv run tools/alert_audit_doc.py --check for the current verdict (--check reads; without it the tool regenerates the audit document). The count is deliberately not written here, because the previous version of this paragraph froze one and was false within two hours.

What remains is one class, not one line: a package_exhaustion alert re-firing into an unchanged band instead of a worse one: scrapfly lt168 → lt168 and resend ge168 → ge168. Both were emitted before the fix deployed, both are the same defect, and the cause is closed at source. No window shorter than this one could have exposed it. It is left visible for the same reason the 429 correction is. What the task actually requires is measured and clean: 0 alerts caused solely by a top-up, 0 caused by a reverted blip.

A depleted_at mismatch also appeared and was fixed. The projection is a float, so recomputing it by a different route landed microseconds away. It was reconciled where the comparison happens, by matching at the precision the quantity actually carries, rather than by re-emitting the shipped lines to agree with the new code. Re-emitting would have made the artifact agree with itself by changing what it claims was emitted.

The audit is a gate rather than a document: it exits non-zero on any unreconciled line, so it cannot drift back into agreeing with itself, and this README cannot quietly claim a pass it does not have.

This work was reviewed externally three times, and the reviews are published with their unresolved gaps intact: docs/reviews/. The round-2 review's gap is stated there rather than quietly closed. An audit chain that admits its own hole is worth more than one that looks seamless. A seamless one only tells you nobody looked hard enough.

An earlier Task 3 trace was quarantined for a confidentiality leak and replaced by a genuine fresh session rather than reconstructed or edited; the incident is kept at docs/task3-trace-quarantine.md because a recorded leak is data and a concealed one is not.

Task	Deliverable	Status
1. Spend observability	monitor.py, alerts.jsonl, public dashboard	collector gap-free since T0, with max consecutive gap under the 30 s poll interval, so the series has no hole; dashboard live at spend.nddev.it.com, returning HTTP 200 with a valid certificate, no login, verified externally
2. STT comparison	published comparison report	published at stt.nddev.it.com: 5 engines plus 2 tuned tracks over 99 hashed segments (2952.821 s) against a publisher human transcript; design frozen before any output was read; recommends Whisper large-v3-turbo with a glossary prompt (large-v3 was retracted because it collapses on 19 of 99 segments)
3. Harness artifact	one harness file plus 2-3 lines	artifact, its 2-3 lines and its trace all delivered

Live status, with the measurement behind every claim, is in docs/ORCHESTRATION.md.

Why the collector started before anything else

Task 1 requires at least six hours of observation and the API has no history endpoint, so the observation window cannot be reconstructed after the fact. A deliberately minimal raw sampler went live before any dashboard code existed, writing verbatim provider responses to disk. Everything downstream (schema adapters, burn-rate estimation, alerting) replays that log, so no design decision made later costs us data.

T0 = 2026-08-23T16:14Z. See docs/RUNLOG.md for what runs where.

What the live API actually does

Measured, not assumed, from the first minutes of capture:

Behaviour	Evidence
Spend is continuous and observable	brightdata 951.99 → 949.05 → 948.83 over ~2 min (≈ −6.6 USD/h)
HTTP 429 is injected across providers, not per-provider withdrawn, see below	tremendous and findymail both returned 429 in the same poll cycle
HTTP 429 is per-provider	over 66 captured cycles, 429 hit exactly one provider each time, never two
Gateway timeouts happen	tremendous → 504 after 3.4 s, ~3.1 s latency against 110 ms normal
A provider can return valid JSON with no fields	anthropic returned {} once, then cost_report on the next 8 calls

Correction. The pool-wide reading of 429 was drawn from reconnaissance at 16:01Z, twelve minutes before T0, so it is not in the captured window at all, and it does not survive it. Task 1 re-tested it against 66 exact poll cycles and found 429 confined to tremendous (16×) and findymail (12×), one provider at a time, in runs of 1-2 cycles. The sustained per-provider signal is 5xx: meta_ads 16 consecutive cycles, bounceban 13, findymail 11, zerobounce 11. This entry is left visible rather than quietly rewritten, because a wrong measurement that got shipped into a design is itself worth recording.

Three design consequences:

1. Availability is per provider, with a sustain period. 429 is not pool-wide, so grouping availability across the pool would have hidden four genuine multi-minute outages. What prevents spam from 504 singles is the length of the staleness window (900 s, above the longest outage measured), not a pool-wide grouping. A separate pool-wide rule still exists for the case where most of the estate goes dark at once, thresholded above the worst cycle observed (4 of 15).
2. {} is a third state. Parsing it into value = 0 would fabricate a balance collapse and a critical alert. It is recorded as schema_miss, distinct from both a value and an HTTP error.
3. Burn rate must be gap-aware and jump-aware. Rates are differences between polls, and 429/504 tear holes in the series. A naive Δbalance / Δt across a five-minute hole mixes spend with any top-up that happened inside it: for findymail it reports the balance rising by 3623 credits/h.

Known measurement limit

A top-up that lands in the same interval as spend is not observable: the API exposes only a current value, so we see the sum of the two, never the parts. Burn rate is therefore a lower bound during top-up intervals. This is stated rather than smoothed over.

Layout

task1-spend-observability/   collector, alerting, dashboard, alerts.jsonl, TRACE.md
task2-stt-benchmark/         eval harness, reference policy, results, TRACE.md
task3-harness-artifact/      the artifact itself, TRACE.md
tools/export_trace.py        session log -> verbatim TRACE.md
tests/                       pytest suite
docs/RUNLOG.md               what is deployed where, with timestamps
docs/HANDOFF.md              picking this up on another machine
.gds/repository.yaml         GDS repository anchor

Verification

# Pinned to the versions .github/workflows/ci.yml uses. An unpinned tool is not a
# gate: an unpinned ruff once resolved to a different version in CI than on the
# machine that ran it, so the same tree was clean in one place and had 49 errors
# in the other. `ruff check .` bare is worse still, since it runs whatever
# happens to be on your PATH.
uv run --with 'pytest==8.3.4' pytest tests/ -q
uv run --with 'ruff==0.15.17' ruff check .
uv run --with pyright==1.1.411 --with pytest==8.3.4 --with httpx pyright
uv run tools/repo_checks.py consistency
uv run tools/repo_checks.py acceptance --strict

The last two are this repository's own checks: consistency fails if a document contradicts the tree, and acceptance --strict is what runs at submission. Both pass on a clean clone, not only here, which was itself a defect once.

Two workflows on the Actions tab are red, and it is not this repository

CI is green and carries every correctness gate above. security and scorecard fail, and they fail before creating a single job. No logs, no steps, and the run is named after its file path, because nothing ever parsed the file far enough to read its name:.

The cause is entirely outside this repository. Both call reusable workflows from NDDev-it-com/ci-workflows, which was archived. Archiving a repository disables its Actions, and un-archiving does not re-enable them, so there were two causes wearing one signature and the second stayed invisible until the first was cleared. Check it yourself:

gh api repos/NDDev-it-com/ci-workflows -q .archived            # false
gh api repos/NDDev-it-com/ci-workflows/actions/permissions      # enabled: false

Nothing in our workflow files is wrong: the pinned commit still declares on: workflow_call and all five called workflows resolve at it. What settled it was re-running a previously successful run unchanged and watching it flip to startup_failure. Nothing in the repository differed between the two attempts, which put the variable outside it.

The fix is one setting on a repository in another organisation, so it is not ours to make. This is written here rather than left for you to find, because a red badge with no explanation is indistinguishable from one nobody looked at.