node-api: widen the measured slice, harden the harness, record the runs to 65% - #5
Closed
Alan Wisper (wisper-dev) wants to merge 14 commits into
Closed
node-api: widen the measured slice, harden the harness, record the runs to 65%#5Alan Wisper (wisper-dev) wants to merge 14 commits into
Alan Wisper (wisper-dev) wants to merge 14 commits into
Conversation
The registry has set 900 seconds per jtreg case for a while, but the test still expected 300, so the harness suite failed before any of this work.
The manifest did not cover net, child_process, cluster, dgram, zlib, domain, readline and a dozen more groups: the slice grows from 1563 to 3007 tests. Expectations gain 163 known failures with a durable cause — the API the test exercises is absent, so the result will not move until it lands. Every test now runs with a 4 GiB heap bound. Node core includes cases whose only brake is stream backpressure: test-zlib-brotli-16GB.js decompresses 16 GB. Without a bound such a case grows until the host runs out of memory instead of failing. The ratchet is left untouched: it comes from a run, and node-api is not in the scheduled matrix, so it has to be regenerated by hand after this change.
A timed-out test was killed by pid alone, so children it left behind kept running and held the inherited stdout open; the reader then waited for an EOF that never came and one test stalled a run for hours. Tests now start in their own process group and the timeout kills the group. If a descendant escaped into a session of its own, the capture is given up two seconds later and returns what it collected, since Bun's subprocess pipes reject an explicit reader and cannot be cancelled mid-read.
The prior baseline described the 1563-test selection. Snapped group by group over the 3007-test one with four threads per run; unobserved entries are retained by the merge, so the parts add up to a full baseline. 1423 accepted failures, 6 tests dropped because they now pass. Runs report no regressions across every group.
Cluster tests are skipped: cluster is out of the measured slice for now, the implementation stays in place. The ratchet is retaken against the current runtime and drops to 1179 entries; the briefing records a verdict per failure class and the subsystem breakdown.
The suite now passes 764 of 3071 (35.8% of the runnable selection), up from 704, with no regressions. Sixty-two entries leave the accepted failure list. Only the run the ratchet was taken from is published; the intermediate runs of the same session are left out to keep the report tree small.
`--elide` usually points at the `dist/current` symlink, and the build context has to carry the real files, not a link into the host.
1075 of 2133 scored tests pass, 938 skipped. 204 tests now pass that the ratchet still lists as failing and 7 baseline entries no longer hold, so the ratchet needs a retake on the next run.
Rewrites the handoff doc around the current measurement: 1,387/2,133 = 65.0%, the failure map by subsystem with the hang subset broken out, what the static expectations skip and why, and the ranked work items that follow from it.
Alan Wisper (wisper-dev)
deleted the
node-api/coverage-and-ratchet-20260803
branch
August 13, 2026 00:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
node-api: widen the measured slice, harden the harness, record the runsBrings the
node-apisuite to an honest measurement and records three runs:50.4% → 60.1% → 65.0%. The latest is 1,387 passes out of 2,133 counting the
denominator without skips, with no regressions against the ratchet.
Harness
that outlive the test process and hold its stdout open. Killing the entry
process alone left them running and the output read never finished, so the run
stalled. The process now gets its own group and a timeout reaps the tree. A
descendant that opened its own session still escapes the group kill, so the
pipes are dropped after a two-second grace.
--elideusuallypoints at the
dist/currentsymlink, and the image build context must carrythe real files; the staged copy now dereferences instead of building around a
broken link.
the previous default.
Test selection
child_process,cluster,dgram,domainand others fromtest/parallel.-XX:MaxHeapSize=4gin the run arguments. The suitecontains cases whose only brake is stream backpressure
(
test-zlib-brotli-16GB.jsdecompresses 16 GB); unbounded, such a case eatsthe host's memory instead of failing honestly.
expectations/node-api.toml:clusterisexcluded from the measured slice by a single glob, so the decision stays
visible and reversible, and the
node:vfstests are skipped as an experimentalNode subsystem Elide does not implement.
Ratchet and reports
node-apiratchet was retaken as gaps closed; it currently holds 857known-failure entries.
summaries and impact maps; the index and the pass-rate badge are updated.
all of it generated
impact.md,results.json.gzand similar.BRIEFING-node-api.mdis rewritten around the current measurement: thefailure map by subsystem with the hang subset broken out, what the static
expectations skip and why, and the ranked work items that follow.