comp_feat: fix flaky parallel exit-255 (atomic resolved-config write) - #69
Merged
Conversation
The parallel (Queue::POSIX) training jobs redirect each background shard's
output to etc qmanager/<name>.{out,err}, but only logdir was archived, so a
shard that failed before writing its own tool log left no trace in the
uploaded artifacts -- the step log just showed "Parallel job failed with exit
code 255" with no cause. Archive an4/qmanager alongside an4/logdir so the
failing shard's actual error is captured.
…lure The fail-fast added for parallel comp_feat read $? unconditionally after waitfor_job(). For Queue::POSIX, waitfor_job() is waitpid(): if the child was already reaped (or is not ours) waitpid() returns -1 and $? is stale, so $? >> 8 became -1, i.e. a spurious "exit code 255" that aborted the whole stage even though no job actually failed. Only derive an exit status when waitpid() truly reaped the child, and report a signal death as 128+signo instead of masking it to a clean zero exit.
…erantly make_feats.pl created per-utterance output directories under the base feature directory even for a warp pass (whose features belong in feat/<warp>/), and used a bare mkpath() that dies "File exists" when a sibling part running in parallel creates the same directory between mkpath's -d test and its mkdir. Create the directory under the pass's own output folder (so warp passes are correct) and tolerate a concurrent create instead of dying.
…d crash sync_runtime() regenerates etc/sphinx_train.resolved.json from every script that loads SphinxTrain::Config, so the parallel comp_feat shards can enter write_file() concurrently. It opened the target with ">", truncating it in place and bumping its mtime; a sibling shard then judged the file fresh, read it while still empty, and died "malformed JSON string ... (end of string)" at BEGIN -- surfacing as "Parallel job failed with exit code 255" with no shard log. This is the actual cause of the flaky train-g2p-lda-vtln / train-parallel failures. Write a per-process temp file and rename() it over the target (atomic on POSIX) so a concurrent reader always sees a complete file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (confirmed)
train-g2p-lda-vtlnandtrain-parallelintermittently failed at module 000comp_feat with
ERROR: Parallel job failed with exit code 255and no cause in thestep log. The failing shard's error was written to a per-shard
qmanager/*.errfilethat CI did not archive; capturing it (commit 1) revealed the real fault:
Every script that loads
SphinxTrain::ConfigcallsResolved::sync_runtime()atBEGIN, which regeneratesetc/sphinx_train.resolved.jsonwhen it looks stale.write_file()opened the target with>, truncating it in place and bumping itsmtime. When the parallel comp_feat shards start together, one shard truncates the
file (making it look fresh to a sibling via the new mtime) while a sibling reads it
in that window — reads an empty file — and dies parsing empty JSON at
BEGIN,before writing any log. The fail-fast then aborts the stage as
exit 255. Fastmachines win the race (it never reproduced on a 36-core box, nor at 2 CPUs / 12-way
oversubscription); the slower GitHub runners lose it, which is why it looked flaky and
runner-specific rather than a code bug. comp_feat is module 000, so this has nothing
to do with G2P/phonetisaurus, and nothing to do with any recent merge.
Commits
qmanageron the AN4 jobs — captures the per-shard errfile thatwas the only record of the real error. This is what pinned the root cause.
WaitForScript: don't turn an already-reaped shard into a spurious failure —waitpid()returning-1made$? >> 8==-1, anexit 255with no failedjob. Robustness, independent of the root cause.
make_feats.plcreated warp dirs under the base feature dir and used a baremkpath()that diesFile existsunder a parallel sibling. Correctness +robustness, independent of the root cause.
per-process temp file and
rename()it over the target (atomic on POSIX), so aconcurrent reader always sees a complete file, never a truncated one.
Validation
All scripts pass
perl -c. A full AN4 G2P+LDA+VTLN train (VTLN 0.90–1.10, NPART=2,Queue::POSIX) with the patched runtime completes comp_feat with all 14 shard logs and
all five warp dirs fully populated (948 mfcs each) and trains through normally — no
regression, and warp features now land in the correct
feat/<warp>/directories.Because the flake is timing-dependent on the GitHub runners, this PR's own CI run is
the real test: with the atomic write,
train-g2p-lda-vtlnandtrain-parallelshould be green.