Split long CI lanes into observable parallel jobs

[Anionix]

Finding

The latest successful main CI run took 7 minutes in one verify job. Two sequential steps dominate wall time even though one step manually multiplexes several background processes into aggregated logs:

• Run tests and smoke lanes concurrently: 191 seconds
• Gate bounded robustness fixture: 131 seconds
• Synchronize frozen Python environment: 58 seconds

Baseline: https://github.com/Anionix/data-format-lab/actions/runs/29719771846

The workflow currently launches pytest and smoke profiles with shell &/wait inside one step. This hides per-lane queue/run/retry/cache timing and makes a failure harder to attribute. A duration over 60 seconds is an improvement candidate, not itself an error.

Scope

• replace shell-background multiplexing with independently observable CI jobs or native parallel execution
• preserve the bounded robustness lane's timeout semantics
• separate queue, setup, run, cache, and retry timing
• keep all existing correctness gates and failure logs
• compare before/after only on matching runner/toolchain fingerprints

Acceptance

• each pytest/smoke/robustness lane has an independent conclusion and duration
• a single lane failure does not erase sibling logs
• no test is skipped and all current gates remain blocking
• record at least 20 completed runs before deciding whether the split improves p50/p95 wall time
• feed completed run JSON into Diagnostic Triage Observer docs: publish Linux v0.1 evidence reports #65 without implicit GitHub API access in the tool

Consumer tracker: #393
Standalone Observer: Anionix/diagnostic-triage#65

[Anionix]

Five-run readback confirms this is stable cost rather than a single anomalous run:

Run	test + smoke	robustness	Python sync
29719771846	191s	131s	58s
29719765249	183s	128s	56s
29719250689	214s	141s	59s
29719069254	192s	129s	55s
29718213445	191s	134s	56s

Median: 191s test/smoke, 131s robustness, 56s sync. Both execution steps exceed the 60-second improvement threshold on every sampled run while all five runs succeeded, so duration alone is not a correctness error.

[Anionix]

New setup-phase evidence from consumer PR #430:

• normal verify: failed after 1m49 while constructing the newly pinned Diagnostic Triage Linux package
• observe-only job: failed after 2m35 at the same package build
• root cause: upstream static target/host linker bug tracked by Fix Linux Nix package mixing static target and glibc host linking diagnostic-triage#296 and fixed in PR [DFL-AUDIT-098] Six-dataset external validity #297

These durations are setup/tool-build time, not pytest or benchmark runtime. After the correctness fix, this tracker should keep the Diagnostic Triage package build as its own observable setup lane and evaluate whether a binary/Nix cache or a dedicated diagnostic shell avoids charging every unrelated CI lane. Do not fold this sample into the prior p50/p95 baseline because the toolchain closure changed.

[Anionix]

Native-parallel observation 1/20 is complete on exact SHA db51ca91caea7de2de2db575cc2fa55ed515a701.

Run: https://github.com/Anionix/data-format-lab/actions/runs/30016472145

Phase / lane	Duration	Conclusion
queue before job start	unavailable from the job payload used here	—
checkout	1s	success
Nix install	7s	success
flake evaluation	12s	success
frozen Python sync / closure setup	162s	success
Python quality gates	8s	success
pytest	177s	success
fair smoke	174s	success
claims smoke	25s	success
prompt smoke	28s	success
equivalence smoke	85s	success
native parallel-group wall time	177s	success
bounded robustness	115s	success
implementation audit	6s	success
ty observation	6s	success
total verify job	499s	success

The five child lanes now have separate conclusions, logs, and durations while sharing one job. The previous five-run median for the aggregated test/smoke step was 191s; this single native-parallel sample is 177s, but no p50/p95 conclusion is valid until 20 matching-fingerprint runs complete.

The 162s sync/setup sample is not comparable to the old 56s median because the pinned Diagnostic Triage closure changed. Keep setup and tool-run samples separate.

Implementation merged in #433. This issue remains open for 19 additional completed runs, matching runner/toolchain fingerprint checks, and Observer ingestion.

[Anionix]

Native-parallel observations 1-22 are complete with one matching workflow/toolchain fingerprint.

Cohort

• 22 successful CI / verify jobs from 2026-07-23 14:54Z through 18:51Z.
• Cohort boundaries: run 30018022506 and run 30035499889.
• Runner group: GitHub Actions; runner label: ubuntu-24.04 for all 22 jobs.
• Workflow/toolchain tree fingerprint: 4f096556f8020e6d12c5ac0c88cd41e88f6f55507b6505928bfa49b8d2550590.
• Fingerprint inputs, in path order: .github/workflows/ci.yml, flake.nix, flake.lock, pyproject.toml, uv.lock, diagnostic-triage.toml.
• The first and last cohort revisions have the same fingerprint, and no commit in the interval changes those paths.
• Queue duration is unavailable from the job payload used here and is not inferred. Cache state is also unavailable.

Results

Linear-interpolated percentiles over step durations:

Phase / lane	n	p50	p95	min-max
total verify job	22	515.5s	557.8s	438-569s
Nix install	22	8.0s	10.0s	7-11s
flake evaluation	22	12.5s	15.0s	11-17s
frozen Python sync	22	170.5s	189.5s	164-213s
blocking quality gates	22	7.5s	9.0s	6-9s
pytest	22	176.0s	201.9s	139-205s
fair smoke	22	173.5s	200.0s	134-201s
claims smoke	22	25.5s	29.0s	20-29s
prompt smoke	22	28.0s	31.0s	21-31s
equivalence smoke	22	85.5s	99.0s	66-100s
native parallel group	22	176.0s	201.0s	139-205s
bounded robustness	22	117.0s	135.9s	90-138s
implementation audit	22	7.0s	7.0s	5-7s
ty observation	22	5.0s	6.0s	4-6s

The five-run pre-split aggregate had p50 191.0s and p95 209.6s. The native group therefore improves p50 by about 7.9% and p95 by about 4.1%, while preserving independent conclusions and logs. The split is beneficial but does not make pytest/fair/equivalence or bounded robustness sub-60-second lanes.

The 170.5s Python-sync p50 is a separate setup/tool-closure cost and must not be attributed to test execution.

Run evidence

30018022506, 30019091579, 30019940296, 30020212022, 30021090569, 30021174773, 30022559222, 30022944114, 30022982139, 30023890664, 30023912789, 30025164590, 30025183747, 30026309209, 30026326757, 30028943396, 30029659736, 30030570515, 30031870146, 30032713521, 30033377912, 30035499889.

Each run is available at https://github.com/Anionix/data-format-lab/actions/runs/<run-id>.

The remaining acceptance item is feeding a real completed run JSON to the offline Diagnostic Triage Observer. A follow-up PR is being prepared with an explicit workflow_run REST fetch; the tool itself remains network-free.