chore(ci): capture zombienet node logs on ts-test failure#4062
Merged
Conversation
When the heima/paseo ts-test network fails to produce blocks, CI showed only "timed out after 30 minutes" with no logs, because: - launch-network.sh ran zombienet with `-l silent` and `> /dev/null`, dropping the orchestrator output. Now `-l info` with output written to $ZOMBIENET_DIR/zombienet.log (under the archived dir); per-node *.log files are unchanged. - The "Archive logs if test fails" step uploaded all of /tmp/parachain_dev/, but the downloaded `polkadot` relay binary has no owner read bit, so the zip failed with EACCES and uploaded nothing. Now archives only **/*.log and **/*.json (readable text). Test-infra observability only; no runtime/node change.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Two mistakes in the previous commit's launch-network.sh change: - `-l info` is not a valid zombienet log level (only table|text|silent), so zombienet errored out immediately and no network started. Use `-l text`. - `mkdir -p $ZOMBIENET_DIR` pre-created the spawn dir, which broke zombienet's spawn. Removed; the orchestrator log is written to a sibling path `$HEIMA_DIR/zombienet-$ZOMBIENET_DIR.log` instead of inside the dir. Verified locally: zombienet spawns cleanly and the sibling log captures the orchestrator (text) output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the zombienet ts-test node logs survive to CI artifacts on failure. Prompted by the
v0.9.27release stalling with the parachain stuck at block #0 and no diagnosable logs — the actual cause could only be found by reproducing locally. With these changes, a future stall is diagnosable directly from the CI artifacts.Two problems previously discarded all node logs on a failing ts-test, leaving only a bare "timed out after 30 minutes":
launch-network.shlaunched zombienet with-l silentand> /dev/null, suppressing the orchestrator (startup/scheduling) output. Now-l info, with output redirected to$ZOMBIENET_DIR/zombienet.logunder the archived dir. Per-node*.logfiles are unchanged.The archive step (
create-release-draft.yml+ci.yml) uploaded the whole/tmp/parachain_dev/, but the downloadedpolkadotrelay binary is written without an owner read bit (--wxrw--wt), so the zip failed withEACCESand uploaded nothing. Now archives only**/*.logand**/*.json(readable text), so node/relay/zombienet logs actually upload.Scope
Test-infra observability only — no runtime or node binary change. This is a companion to #4061 (the async-backing
AllowMultipleBlocksPerSlot/RelayNumberMonotonicallyIncreasesruntime fix), which is the actual block-production fix. This PR does not change block-production behavior; it just ensures that if the ts-test network fails to produce blocks again, the logs are captured.Verification
launch-network.shchange exercised locally: zombienet still spawns correctly and its orchestrator output lands inzombienet.logalongside the per-node logs, all under/tmp/parachain_dev/<dir>/which the corrected archive globs pick up.