Problem
generate_train_data.py --resume uses the number of existing output/error rows to decide how many input rows to skip. That assumes output rows are written in the same order as input rows.
Concurrent generation can finish out of order. If the script writes results as futures complete, a crash can leave output rows for a non-contiguous set of input rows. On resume, the script then skips the wrong prefix, causing some samples to be regenerated and others to be skipped.
The ordered-write fix also needs bounded buffering: if an early request is slow, later completed samples should not accumulate without limit behind it.
Impact
Large data-generation jobs can produce corrupted regenerated datasets after interruption/resume. Without bounded ordered buffering, high-concurrency runs can also retain too many completed samples in memory behind one slow earlier request.
Fix
Fixed in PR #29 by writing generated rows in input order, validating JSONL paths, hardening JSONL loading for blank/empty files, and bounding outstanding ordered results so completed-but-not-yet-writable samples still apply backpressure.
Problem
generate_train_data.py --resumeuses the number of existing output/error rows to decide how many input rows to skip. That assumes output rows are written in the same order as input rows.Concurrent generation can finish out of order. If the script writes results as futures complete, a crash can leave output rows for a non-contiguous set of input rows. On resume, the script then skips the wrong prefix, causing some samples to be regenerated and others to be skipped.
The ordered-write fix also needs bounded buffering: if an early request is slow, later completed samples should not accumulate without limit behind it.
Impact
Large data-generation jobs can produce corrupted regenerated datasets after interruption/resume. Without bounded ordered buffering, high-concurrency runs can also retain too many completed samples in memory behind one slow earlier request.
Fix
Fixed in PR #29 by writing generated rows in input order, validating JSONL paths, hardening JSONL loading for blank/empty files, and bounding outstanding ordered results so completed-but-not-yet-writable samples still apply backpressure.