Skip to content

Concurrent training data generation can corrupt resume order and buffer unbounded results #35

Description

@morluto

Problem

generate_train_data.py --resume uses the number of existing output/error rows to decide how many input rows to skip. That assumes output rows are written in the same order as input rows.

Concurrent generation can finish out of order. If the script writes results as futures complete, a crash can leave output rows for a non-contiguous set of input rows. On resume, the script then skips the wrong prefix, causing some samples to be regenerated and others to be skipped.

The ordered-write fix also needs bounded buffering: if an early request is slow, later completed samples should not accumulate without limit behind it.

Impact

Large data-generation jobs can produce corrupted regenerated datasets after interruption/resume. Without bounded ordered buffering, high-concurrency runs can also retain too many completed samples in memory behind one slow earlier request.

Fix

Fixed in PR #29 by writing generated rows in input order, validating JSONL paths, hardening JSONL loading for blank/empty files, and bounding outstanding ordered results so completed-but-not-yet-writable samples still apply backpressure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions