node-api: widen the measured slice, harden the harness, record the runs to 65%

BRIEFING-node-api.md

@@ -1,34 +1,38 @@
 # Briefing: `node-api` compat work (Node.js core test suite)
 
-_Self-contained handoff doc. Generated 2026-07-17 from the latest committed run._
+_Self-contained handoff doc. Generated 2026-08-13 from the latest committed run._
 
 ## Context
 
 This repo (`elide-dev` conformance testsuite) runs upstream conformance suites
 against pinned [Elide](https://elide.dev) builds and publishes versioned
-reports. The `node-api` suite runs a sparse slice of Node.js core's
-`test/parallel/` JS tests (assert, async_hooks, buffer, console, dns, events,
-fs, module/require, path, process, stream, timers, vm, worker groups — see
-`manifests/node-api.toml`). Tests run against a checked-in `common/` overlay
-(`node-api-overlay`), so `require('../common')` works without full Node
-internals.
+reports. The `node-api` suite runs a slice of Node.js core's `test/parallel/` JS
+tests, grouped per subsystem in `manifests/node-api.toml` (assert, async_hooks,
+buffer, child_process, cluster, dgram, dns, domain, events, fs, http, module,
+net, path, process, stream, timers, tls, vm, worker, zlib and more). Tests run
+against a checked-in `common/` overlay (`node-api-overlay`), so
+`require('../common')` works without full Node internals.
 
-This is the lowest-scoring suite (25.4%) and was, until today, hidden from the
-README summary table. It is now visible; the goal is to make the number honest
-and then move it.
+Counts are whole test *files*: one Node test file = one result.
 
-## Current state (Elide `1.4.1+20260716.b8b6531`, digest `3d3ea83ed640`)
+## Current state (Elide `1.4.2+ab01c86f3`, digest `15b14ec99d04`)
 
 | pass | fail | error | skip | total | pass rate (excl. skips) |
 |---:|---:|---:|---:|---:|---:|
-| 309 | 875 | 34 | 345 | 1,563 | **25.4%** |
+| 1,387 | 679 | 67 | 938 | 3,071 | **65.0%** |
 
-Counts are whole test *files* (one Node test file = one result). Latest reports:
+Progression of measured runs: 25.4% (sparse slice, July) → 50.4% → 60.1% →
+65.0%. The denominator excluding skips is 2,133.
 
-- `reports/1.4.1+20260716.b8b6531/3d3ea83ed640/node-api/impact.md` — 481
-  root-cause signatures, largest first (4,800 lines; read the top 100)
+Latest reports:
+
+- `reports/1.4.2+ab01c86f3/15b14ec99d04/node-api/impact.md` — failures ordered
+  by root-cause signature, largest first (read the top ~40 sections)
 - `.../node-api/impact.json` — machine-readable (`bySignature`)
-- `expectations/node-api.ratchet.toml` — known-failure baseline
+- `.../node-api/changes.md` — diff against the previous run
+- `expectations/node-api.ratchet.toml` — known-failure baseline (857 entries)
+- `expectations/node-api.toml` — static skips with a reason per test (500
+  entries)
 
 ## How to run
 
@@ -38,92 +42,108 @@ bun run setup                       # once: deps + submodules + sparse checkouts
 bun run testsuite --elide nightly --suite node-api --include 'test/parallel/test-path.js' --log
 # a target group:
 bun run testsuite --elide nightly --suite node-api --include 'test/parallel/test-buffer*.js' --log
-# full suite (~6 min):
+# full suite (~30-35 min, plus ~5 min to rebuild the harness image):
 bun run testsuite --elide nightly --suite node-api
-# after fixes land in an Elide nightly, re-baseline:
+# after fixes land, re-baseline:
 bun run testsuite --elide nightly --suite node-api --ratchet
 ```
 
-## Failure analysis (909 failing files)
-
-### Systemic root causes (fix once, unblock many)
-
-1. **`__filename` / `process.argv[1]` is the literal string `"unknown"`** —
-   **~49 files directly**, likely more indirectly. Symptoms across several
-   signatures:
-   - `The worker script or module filename ... Received "unknown"` (31+2+1+1)
-   - `ENOENT: no such file or directory, open 'unknown'` (6+3+1+1) — tests
-     that re-open `__filename`
-   Elide isn't propagating the entry-script path to `process.argv`/module
-   metadata when launched by the harness. Almost certainly ONE runtime fix.
-   **Best effort/reward ratio in the entire cross-suite triage.**
-2. **`node:child_process` unimplemented** — **~39 files direct**:
-   `spawnSync()` (14), `spawn()` (10+3), `exec()` (5), `fork()` (4),
-   `execFileSync()` (3). Also implicated in some of the 34 timeouts and in
-   cpython-core failures (subprocess-based tests) — cross-suite leverage.
-   High complexity; treat as a project, not a quick win.
-3. **`node:http` (+ `_http_common`) missing** — 64 files fail at
-   `Cannot load module: 'http'`. Very large surface; park unless http is
-   already on Elide's roadmap. Alternatively add these to `[skip]` with a
-   comment to make the pass rate reflect reachable tests.
-4. **Timeouts** — 34 files (`Node API test timed out`), e.g.
-   `test-fs-promises-watch.js`. Mostly fs.watch / event-loop-keepalive
-   related; diagnose a couple before assuming one cause.
-
-### Module-level clusters (each a coherent mini-project)
-
-- **buffer** (~80 files): assorted semantic gaps — `buffer.constants`
-  undefined, `Buffer.from` on SharedArrayBuffer/objects with `valueOf`
-  (`unsupported input type`, 3), `buf.copy` return value undefined (8),
-  inspect truncation format (20-bucket), negative-allocation / range checks
-  missing (`Missing expected exception`, part of 36+7+4).
-- **events / EventEmitter** (~25): symbol event names
-  (`Cannot convert a Symbol value to a string`, 4), `receiver is not an
-  EventEmitter` when using prototype-less receivers (4),
-  `listeners side-effects` (`Cannot convert undefined or null to object`, 3),
-  listener-count semantics.
-- **streams** (~90): `Readable.toWeb` missing (4), `Stream is not a
-  constructor` from `require('stream')` shape (3), many
-  `mustCall` count mismatches (13+10+9+6...) pointing at async scheduling /
-  destroy/finish event-ordering divergences.
-- **process** (~40): `process.chdir()` (5) and `process.getuid()` (3) throw
-  `UnsupportedOperationException`, `execve` missing (3+2),
-  `hasUncaughtExceptionCaptureCallback` missing (3),
-  `getactiveresources` tracking (2+2+...), env delete semantics.
-- **vm** (~15): internal `Symbol(elide.vm.context)` leaks into
-  `Object.getOwnPropertyNames`/`ownKeys` of contextified objects (3 —
-  trivial fix: filter the internal symbol), `defineProperty called on
-  non-object` on context args (2), property forwarding.
-- **timers** (~10): `timeout.refresh()` missing (3), timer cancellation
-  warnings (3), active-resource tracking.
-- **worker_threads** (~10 beyond the argv issue): structured-clone
-  `DOMException` on SharedArrayBuffer/MessagePort transfer (4).
+Point `--elide` at a local install dir to measure an unreleased build; the
+staged copy dereferences symlinks, so `dist/current` works.
+
+Each test runs with `-XX:MaxHeapSize=4g` (see `registry.toml`). The suite
+contains cases whose only brake is stream backpressure
+(`test-zlib-brotli-16GB.js` decompresses 16 GB); without a bound such a case
+grows until the host runs out of memory instead of failing.
+
+## What is skipped and why
+
+Of the 938 skips, `expectations/node-api.toml` carries a per-test reason:
+
+- **`cluster`** (83 tests) — the module ships and works, but its tests are out of
+  project scope for now. One glob covers the group so the decision is visible
+  and reversible.
+- **`node:vfs`** (37 tests) — an experimental Node subsystem not implemented in
+  Elide.
+- **Node-internal modules** — tests that `require('internal/...')` directly
+  (`internal/assert/myers_diff`, `internal/child_process`, `internal/async_hooks`,
+  `internal/fs/promises`, `internal/test/binding`, …). They test Node's private
+  surface, not public API behaviour.
+
+Everything else that fails sits in the ratchet, which holds no reasons because a
+run rewrites it.
+
+## Failure map (746 failing files: 679 fail + 67 error)
+
+By subsystem, largest first, with the hang subset broken out:
+
+| Subsystem | Failing | of which hangs |
+|---|---:|---:|
+| http | 88 | 14 |
+| worker_threads | 68 | 15 |
+| stream | 64 | 0 |
+| vm | 60 | 5 |
+| fs | 55 | 5 |
+| process | 50 | 0 |
+| child_process | 45 | 6 |
+| net | 36 | 2 |
+| async_hooks | 26 | 2 |
+| zlib | 21 | 2 |
+| dns | 17 | 9 |
+| module | 17 | 0 |
+| domain | 16 | 2 |
+| require | 16 | 0 |
+
+Largest root-cause signatures in the latest run:
+
+| Tests | Signature |
+|---:|---|
+| 67 | `Node API test timed out` |
+| 17 | `AssertionError: Expected values to be strictly equal` (numeric) |
+| 15 | `Mismatched <anonymous> function calls` (`mustCall` count) |
+| 14 | `AssertionError: Expected values to be strictly equal` (string) |
+| 10 | `AssertionError: Missing expected exception` |
+| 10 | `Mismatched noop function calls` |
+| 10 | `TypeError: Cannot load module` |
 
 ## Ranked work items
 
-1. **argv/`__filename` "unknown" fix** — ~49+ files, complexity **low**. Do
-   this first; re-run to see the true shape of worker/fs failures behind it.
-2. **vm internal-symbol filter** — 3 files, complexity **trivial**. Good
-   warm-up in the same session as (1).
-3. **buffer semantics sweep** — ~40–80 files, complexity **medium**, highly
-   parallelizable (each signature is independent). Run
-   `--include 'test/parallel/test-buffer*.js' --log` and burn down.
-4. **process misc (`chdir`, `getuid`, capture-callback, refresh)** — ~15
-   files, complexity **low-medium** each.
-5. **events/EventEmitter conformance** — ~25 files, complexity **medium**.
-6. **child_process `spawn`/`spawnSync`** — ~39 files + cross-suite
-   (cpython-core) leverage, complexity **high**. Schedule as its own project.
-7. **streams event-ordering** — large but diffuse; recommend AFTER (1) and
-   (3), since many stream tests also touch buffer/process gaps.
-8. **`node:http`** — 64 files; park or skip-classify for now.
+1. **Hangs — 69 files, one visible signature, several causes.** The harness
+   reports every hang identically, so the causes must be separated by area:
+   `worker_threads` (15), `http` (14), `dns` (9), `child_process` (6), `fs` (5),
+   `vm` (5), `dgram` (4). Highest density in the suite; each area is a coherent
+   mini-project.
+2. **`mustCall` count mismatches — ~32 files across two signatures.** These point
+   at event ordering and async scheduling: a callback Node fires and Elide does
+   not, or fires at a different time. Read one per subsystem before generalizing.
+3. **Error-shape assertions — ~11 files.** Deep-equal failures where the
+   comparison differs only by `code`/`message` on the thrown error. Cheap per
+   file once the expected code is known.
+4. **`vm` — 60 files.** Real contextified objects; the cluster is large and
+   coherent enough to justify runtime work rather than per-test patching.
+5. **`process` — 50 files, no hangs.** Assorted platform surface (`chdir`,
+   `getuid`, active-resource tracking, env semantics); low complexity each.
+6. **`stream` — 64 files, no hangs.** Diffuse; mostly event ordering and
+   backpressure, overlapping with item 2.
+7. **`Cannot load module` — 10 files.** Absent public modules; classify as
+   static skips only when the module is genuinely out of scope, otherwise
+   implement.
 
 ## Notes for the implementer
 
-- Fixes happen in the Elide runtime repo; this repo pins nightlies and
-  measures (`--elide nightly` picks up a fresh build).
+- Fixes happen in the Elide runtime repo; this repo pins builds and measures.
 - One file = one result here, so "N files" = N report-visible wins.
 - The `common/` overlay lives in the harness work dir
   (`.harness/work/node-api/node-api-overlay/test/common/index.js`); `mustCall`
   mismatch stack traces point there, not at a runtime bug in itself.
-- After a fix lands: full run → check `changes.md` (new passes, no
-  regressions) → `--ratchet` → commit reports + expectations together.
+- A timeout reaps the whole process group: Node's suite spawns children that
+  outlive the test and hold its stdout open. A descendant that opened its own
+  session still escapes, so the harness drops the pipes after a two-second
+  grace.
+- Zero exit status is not a pass: a skip exits zero too. Distinguish them when
+  triaging by hand.
+- Between full runs, guard against regressions with a local subset of
+  previously-passing tests (~500 files, ~10 min) rather than re-running the
+  suite.
+- After a fix lands: full run → check `changes.md` (new passes, no regressions)
+  → `--ratchet` → commit reports and expectations together.

bin/run.ts

@@ -349,7 +349,9 @@ async function buildHarnessImage(options: Options, image: string, plat: string[]
   if (isLocalInstallDir(elideRef)) {
     log("mode: local install dir");
     rmSync(resolve(ROOT, ".elide-install"), { recursive: true, force: true });
-    cpSync(elideRef, resolve(ROOT, ".elide-install"), { recursive: true });
+    // `--elide` often points at a symlink (`dist/current`); the build context must carry the real
+    // files, so the staged copy dereferences.
+    cpSync(elideRef, resolve(ROOT, ".elide-install"), { recursive: true, dereference: true });
     const digest = sha256File(resolve(ROOT, ".elide-install/bin/elide"));
     log(`building image ${image} (local install dir)...`);
     const rc = await run(["docker", "build", ...plat, "-f", "docker/harness.local.Dockerfile", "-t", image, "."]);