Training data pipeline: disk-backed memmap rounds, background producer, batched tf.data input#117

Open

zachtheyek wants to merge 13 commits into

feature/training-data-pipeline

Commits on Jul 17, 2026

Add round_data module (memmap round datasets, producer) + ManagedProcess
zachtheyek
committed
Rewrite data generation: batched tasks writing straight into round memmaps
zachtheyek
committed
Train on disk-backed rounds: batched tf.data input + producer scheduling
zachtheyek
committed
Add round-data config fields, CLI flags, and disk-budget validation
zachtheyek
committed
Add unit tests for round data, batched generation, datasets, processes
zachtheyek
committed
Apply self-review findings: resilient drainer, stale message reference
zachtheyek
committed
Start the RoundDataProducer via spawn (fork inherited a deadlocked lock)
zachtheyek
committed
Producer logging via a spawn-context relay queue; real spawn e2e test
zachtheyek
committed
Apply review: flush in finally, clarifying notes
zachtheyek
committed
Default round-data dir under the training data path

zachtheyek
and
claude
committed
Document reviewed-low-risk caveats inline (round-data, RNG, straggler, spawn test)

zachtheyek
and
claude
committed
Update comments
zachtheyek
committed
Fix stale _single_cadence_wrapper comment references

zachtheyek
and
claude
committed