Skip to content

comp_feat: fix flaky parallel exit-255 (atomic resolved-config write) - #69

Merged
lenzo-ka merged 4 commits into
masterfrom
kal-comp-feat-parallel-race
Aug 17, 2026
Merged

comp_feat: fix flaky parallel exit-255 (atomic resolved-config write)#69
lenzo-ka merged 4 commits into
masterfrom
kal-comp-feat-parallel-race

Conversation

@lenzo-ka

@lenzo-ka lenzo-ka commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Root cause (confirmed)

train-g2p-lda-vtln and train-parallel intermittently failed at module 000
comp_feat
with ERROR: Parallel job failed with exit code 255 and no cause in the
step log. The failing shard's error was written to a per-shard qmanager/*.err file
that CI did not archive; capturing it (commit 1) revealed the real fault:

malformed JSON string, neither array, object, number, string or atom, at character
offset 0 (before "(end of string)") at .../SphinxTrain/Resolved.pm line 199.
BEGIN failed--compilation aborted at .../make_feats.pl line 51.

Every script that loads SphinxTrain::Config calls Resolved::sync_runtime() at
BEGIN, which regenerates etc/sphinx_train.resolved.json when it looks stale.
write_file() opened the target with >, truncating it in place and bumping its
mtime. When the parallel comp_feat shards start together, one shard truncates the
file (making it look fresh to a sibling via the new mtime) while a sibling reads it
in that window — reads an empty file — and dies parsing empty JSON at BEGIN,
before writing any log. The fail-fast then aborts the stage as exit 255. Fast
machines win the race (it never reproduced on a 36-core box, nor at 2 CPUs / 12-way
oversubscription); the slower GitHub runners lose it, which is why it looked flaky and
runner-specific rather than a code bug. comp_feat is module 000, so this has nothing
to do with G2P/phonetisaurus, and nothing to do with any recent merge.

Commits

  1. ci: archive qmanager on the AN4 jobs — captures the per-shard errfile that
    was the only record of the real error. This is what pinned the root cause.
  2. WaitForScript: don't turn an already-reaped shard into a spurious failure
    waitpid() returning -1 made $? >> 8 == -1, an exit 255 with no failed
    job. Robustness, independent of the root cause.
  3. comp_feat: create output dirs under the warp folder, race-tolerantly
    make_feats.pl created warp dirs under the base feature dir and used a bare
    mkpath() that dies File exists under a parallel sibling. Correctness +
    robustness, independent of the root cause.
  4. Resolved: write the resolved config atomically (the fix) — write a
    per-process temp file and rename() it over the target (atomic on POSIX), so a
    concurrent reader always sees a complete file, never a truncated one.

Validation

All scripts pass perl -c. A full AN4 G2P+LDA+VTLN train (VTLN 0.90–1.10, NPART=2,
Queue::POSIX) with the patched runtime completes comp_feat with all 14 shard logs and
all five warp dirs fully populated (948 mfcs each) and trains through normally — no
regression, and warp features now land in the correct feat/<warp>/ directories.
Because the flake is timing-dependent on the GitHub runners, this PR's own CI run is
the real test: with the atomic write, train-g2p-lda-vtln and train-parallel
should be green.

The parallel (Queue::POSIX) training jobs redirect each background shard's
output to etc qmanager/<name>.{out,err}, but only logdir was archived, so a
shard that failed before writing its own tool log left no trace in the
uploaded artifacts -- the step log just showed "Parallel job failed with exit
code 255" with no cause. Archive an4/qmanager alongside an4/logdir so the
failing shard's actual error is captured.
…lure

The fail-fast added for parallel comp_feat read $? unconditionally after
waitfor_job(). For Queue::POSIX, waitfor_job() is waitpid(): if the child was
already reaped (or is not ours) waitpid() returns -1 and $? is stale, so
$? >> 8 became -1, i.e. a spurious "exit code 255" that aborted the whole
stage even though no job actually failed. Only derive an exit status when
waitpid() truly reaped the child, and report a signal death as 128+signo
instead of masking it to a clean zero exit.
…erantly

make_feats.pl created per-utterance output directories under the base feature
directory even for a warp pass (whose features belong in feat/<warp>/), and
used a bare mkpath() that dies "File exists" when a sibling part running in
parallel creates the same directory between mkpath's -d test and its mkdir.
Create the directory under the pass's own output folder (so warp passes are
correct) and tolerate a concurrent create instead of dying.
…d crash

sync_runtime() regenerates etc/sphinx_train.resolved.json from every script
that loads SphinxTrain::Config, so the parallel comp_feat shards can enter
write_file() concurrently. It opened the target with ">", truncating it in
place and bumping its mtime; a sibling shard then judged the file fresh, read
it while still empty, and died "malformed JSON string ... (end of string)" at
BEGIN -- surfacing as "Parallel job failed with exit code 255" with no shard
log. This is the actual cause of the flaky train-g2p-lda-vtln / train-parallel
failures. Write a per-process temp file and rename() it over the target
(atomic on POSIX) so a concurrent reader always sees a complete file.
@lenzo-ka lenzo-ka changed the title comp_feat: fix flaky parallel exit-255 and make the failure diagnosable comp_feat: fix flaky parallel exit-255 (atomic resolved-config write) Aug 17, 2026
@lenzo-ka
lenzo-ka merged commit 62563e2 into master Aug 17, 2026
8 checks passed
@lenzo-ka
lenzo-ka deleted the kal-comp-feat-parallel-race branch August 17, 2026 01:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant