Groth16 prover: memory scoping + MSM acceleration + prover options#628
Open
OBrezhniev wants to merge 83 commits into
Open
Groth16 prover: memory scoping + MSM acceleration + prover options#628OBrezhniev wants to merge 83 commits into
OBrezhniev wants to merge 83 commits into
Conversation
# Conflicts: # build/snarkjs.js # build/snarkjs.min.js # src/groth16_prove.js
…y needed. Changed a few anonymous functions to named for easier profiling. Smaller chunks in joinABC to split work in most cases, and pass buffer ownership to worker threads there.
…ly when they are needed (bfj, ejs). And don't use bfj module for small json files (proofs, public signals). Comment out vm module and manual garbage collection in zkey_new.js.
…Switch between different buildABC implementations through options. Debug & logging in groth16_prove. Rebuild.
Picks up the 2-phase termination and WorkerSlot identity model from ffjavascript so all buildABC modes (js/wasm/wasm1) run reliably without intermittent "Worker terminated unexpectedly" failures or hangs.
Picks up ffjavascript console.log cleanup (engine_fft, engine_multiexp, threadman) via rebuilt bundles. No logic changes in snarkjs src. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Picks up the worker-side in-place reversePermutation (no WASM memory growth, zero-copy) via the inlined ffjavascript in the IIFE bundle. No snarkjs src changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…script Picks up the ffjavascript fix that stops pairingEq from detaching caller-owned G1.g/G2.g buffers. Restores the full prove/verify pipeline: snarkjs test suite now 49/49 passing. No snarkjs src changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ding Picks up the fastfile node:fs->fs.constants fix and the ffjavascript bn128 prebuilt-wasm loader fix (atob/arrayBuffer) so the browser builds run in a real browser again. snarkjs browser test suite (browser_tests) now passes: full setup/prove/verify in headless Chrome on both the IIFE and ESM builds. No snarkjs src or rollup config changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pt; drop dead taskmanager
- misc.getRandomBytes / sha256digest: drop `process.browser`. Prefer the Node
crypto module (createHash, randomFillSync -- no per-call size limit), fall back
to Web Crypto (getRandomValues chunked to 65536 bytes; subtle.digest on the
view, not data.buffer, so subarray byteOffset/byteLength is respected).
- askEntropy: use the browser prompt only when a real DOM window exists
(typeof window / window.prompt), not !process.browser -- "not Node" is not the
same as "browser" (Bun/Deno/edge/SES have neither).
- Delete src/taskmanager.js: unused dead code, and the only place using
`new Worker(code, {eval:true})` + require() codegen, which trips CSP/eval
scanners.
Rebuilt browser bundles. (package.json fastfile dep left as-is; separate.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…stubs Rebuilt IIFE bundles after the ffjavascript (os/crypto) and fastfile (fs/constants) "browser" field additions. No snarkjs src changes. Browser e2e (IIFE + ESM) passes; bundles have no Node-builtin leaks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up ffjavascript's vendored, statically-imported prebuilt wasm (no dynamic import of wasmcurves, no gzip decode). No snarkjs src changes. Browser e2e (IIFE + ESM) and tutorial e2e pass; node suite 49/49. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… from ffjavascript Picks up ffjavascript's lazy getWorkerSource (no Blob/btoa at import) and the base64 decoder that prefers Buffer/atob with a pure-JS SES fallback. No snarkjs src changes; browser e2e + node suite pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five base-point sections (A/B1/B2/C/H) were each read whole into a buffer and then sliced per worker chunk -- two full copies of every section, and all five held in RAM concurrently (the calc promises run in parallel). Switch them to curve.G1/G2.multiExpAffineChunked with a per-section reader (mkSectionReader) that returns each chunk directly via fdZKey.readToBuffer. No full section buffer, no main-thread slice, and only a few chunks resident at a time. Using readToBuffer directly (rather than binFileUtils.readSection per chunk) also avoids amplifying readSection's per-call console.time logging by ~40x/section. Effect scales with circuit size: - authV3 (29 MB zkey, 19 MB bases): peak RSS ~603 -> ~586 MB, time neutral. - sha256 (1.1 GB zkey, 733 MB bases): peak RSS ~3.85 -> ~3.41 GB (~12%), and a bit faster + lower variance (less GC pressure from the big transient allocs). Proof identical / verifies OK in both cases. Validated: snarkjs 49, ffjavascript 63, tutorial e2e (groth16/plonk/fflonk), authV3 + sha256 prove+verify. Bundles rebuilt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The default buildABC for circuits whose witness fits one pass was buildABCWASM1: a single worker that loads ALL coefficients (e.g. 357 MB) + witness + the three output buffers (~621 MB) and never shrinks (WASM memory can't), so that ~681 MB stayed resident for the whole prove, and the build ran single-threaded. Replace it with a streaming build: - buildABCStream: still a SINGLE full-witness pass (so each disjoint output range is computed completely -> no batchAdd/joinABC merge, unlike multi-threaded buildABC), but the domain is split into nChunks output ranges processed with bounded in-flight. Each task holds only witness + one coeff chunk + one output chunk, so a worker's high-water is ~witness+chunk instead of the whole thing. - pickStreamParams: adaptive. Each busy worker's WASM memory persists, so the floor buildABC leaves behind is ~maxInFlight x perWorker. It sizes nChunks so a worker holds ~2x witness, derives maxInFlight from a worker-memory floor budget (default 256 MB), and sets nChunks to a few per BUSY worker (not full concurrency -- that would re-copy the witness per chunk for nothing at low parallelism). Small circuits -> full parallelism; large -> bounded. Tunable via options.buildABCFloorBudget / buildABCnChunks / buildABCmaxInFlight. - Default path uses streaming whenever the witness fits a single pass (all normal circuits); multi-threaded buildABC stays as the witness-too-big fallback. buildABCWASM1 is kept for the explicit "wasm1" option. Measured (sha256, 1.1GB zkey, vs the old wasm1 default 9.11s / 3354 MB): - default 256 MB budget (n9/k2): ~8.5s / ~3.05 GB -> ~7% faster AND ~9% less peak - memory-first (<=192 MB budget, k1): ~9.3s / ~2.9 GB -> -13% peak, +2% time - raising the budget is counterproductive (more memory, eventually slower from worker oversubscription against the concurrent multiexps); the knob is useful only downward. authV3 (small) picks n45/k15 -> unchanged (~1.08s). All verify OK. Validated: snarkjs 49, tutorial e2e (groth16/plonk/fflonk), authV3 + sha256 prove+verify. Bundles rebuilt. ffjavascript untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up the binfileutils MAX_BUFFER_SIZE and ffjavascript BigBuffer PAGE_SIZE cleanup (dead `Buffer.constants` probe replaced with an explicit `1 << 30`). The IIFE and browser-ESM bundles inline those deps; main.cjs/cli.cjs import them externally and are unaffected. No behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The IFFT->applyKey->FFT pipeline drops each FFT/IFFT input immediately (buffX_T is nulled; buffXodd goes out of scope), so pass consume=true to ffjavascript's fft/ifft and skip its defensive full-input copy. The 3 FFT inputs are flat Uint8Arrays (from batchApplyKey) and are consumed; the 3 IFFT inputs are BigBuffers (from buildABC) and still flatten as before. sha256: ~100-300 MB lower peak RSS, time within noise (the FFT copies overlap the concurrent multiexps, so the win is modest). authV3 + sha256 verify OK; snarkjs 49, ffjavascript 64, tutorial e2e all pass. Bundles rebuilt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the domain fits under BigBuffer's 1 GiB page, allocate the A/B/C outputs as flat Uint8Arrays instead of BigBuffers. The downstream IFFT (consume=true) can then take them in place and skip its defensive full-input copy -- previously a BigBuffer input forced a flatten-copy. Larger domains stay paged BigBuffers (the IFFT flattens those as before). ~2.5% faster end-to-end on sha256 (8.59s -> 8.37s, quiet machine, the whole run distribution shifts down); peak RSS unchanged (the consumed copies are early in the pipeline, not at the peak). authV3 + sha256 verify OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildABCWASM1 (single worker, one full-witness pass, all coeffs at once) is exactly buildABCStream with nChunks=1 -- and the streaming default strictly dominates it (lower, bounded worker memory + tunable parallelism). Delete the ~150-line function and route the explicit "wasm1" option to buildABCStream(..., 1, 1), byte-identical. The other variants stay: "wasm" (multi-threaded, the witness-too-big-for-one-pass fallback that splits the witness across passes) and "js" (pure-JS element-at-a-time, zero bulk wasm allocation -- the universal fallback for arrays beyond wasm's 32-bit limit). Behaviour-preserving; default/wasm1/js/wasm all verify OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
r1csInfo prints header counts (wires, constraints, inputs, labels, outputs) but
called readR1cs() with defaults, which loads the entire constraints section (and
the wire-to-label map) -- reading the whole, potentially many-GB, .r1cs file just
to print a few numbers. On a 12 GB r1cs this took minutes and gigabytes of RAM.
Pass {loadConstraints: false, loadMap: false}: the counts all come from section 1
(the header). Same output, now near-instant (12 GB r1cs: ~minutes -> 0.17 s,
~79 MB RSS). r1csInfo never touches cir.constraints, and no caller uses the
returned value's constraints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
options.msmBatching selects the ffjavascript MSM batching mode and is threaded to all five multiexps (A, B1, B2, C, H): "auto" (default) batch-affine only for cache-friendly chunk sizes; "enabled" always batch (best for small/medium circuits); "disabled" plain multiexp (lowest memory; best for very large circuits). Invalid values throw.
…tion groth16 prove/fullprove accept --memlog (1s interval) or --memlog=<ms>. The API option is memoryLogging on groth16Prove. Previously the monitor ran unconditionally at 50ms whenever a logger was set and its interval was never cleared; now it is opt-in, Node-gated (process.memoryUsage feature check), cleared in a finally, and emits a final sample at completion via logger.info (so the flag works without -v).
git+https installs anonymously (git+ssh needs SSH credentials); pins updated to the heads carrying the git+https switch themselves.
This was referenced Jul 4, 2026
Member
Author
|
PR stack (landing order):
Cross-repo deps are pinned by git+https commit refs; re-pin consumers if a branch gains commits before merge. |
js-yaml bumped in-range; serialize-javascript ^7.0.5 and diff ^8.0.3 overridden (mocha pins vulnerable ranges). npm audit clean.
# Conflicts: # build/browser.esm.js # build/snarkjs.js # build/snarkjs.min.js
These options select the batch-affine MSM module and GLV/GLS endomorphism paths in ffjavascript's multiexp but had no test coverage. Verifies every combination still produces a valid proof, plus invalid-option rejection.
plonkVerify, fflonkVerify, and wtnsCheck each had a code path that called logger.error()/logger.warn() without checking logger was provided first. Since these functions accept an optional logger and are documented/tested to be called without one, a tampered PLONK/ FFLONK proof or a constraint-violating witness would throw a TypeError instead of the documented false/failure return. Added regression tests that construct a malformed PLONK proof (off-curve commitment), a public-signal-count mismatch for FFLONK, and a constraint-violating witness, confirming each now returns false/rejects cleanly instead of throwing.
Picks up the fastfile truncated-read corruption fix (transitively via binfileutils), the ThreadManager/multiexp crash fixes, and the new test coverage added across the dependency chain.
Picks up the threadman/multiexp WorkerSlot fixes and logger guards from ffjavascript's latest, inlined into the browser IIFE/ESM/min bundles. cjs/cli outputs also refreshed (small diff: two guarded logger calls).
…aulting Only "js" and "stream" are supported (buildABC="wasm"/"wasm1", the multi/single-threaded WASM variants, were retired). Any other value, including the retired ones, silently fell through to the default streaming path instead of surfacing the mistake -- unlike msmBatching/ msmGlv, which already validate. Added tests covering both valid values producing a verifying proof and rejection of wasm/wasm1/bogus values.
* fix getCutPoint lower bound * test: regression coverage for gapped coefficient rows in buildABCStream Adds a fixture circuit (circom --O0) that alternates multiplicative rows with purely-linear ones (l <== a + b). Linear constraints compile to the 0*0 = C form, and section 4 of the zkey holds records only for A-side/B-side terms -- so those rows have no coefficient records at all, leaving gaps in the sorted c-sequence: c-values: 0,0,2,2,4,4,6,6,8,8,10,10,12,12,14,14,17,18 Pre-fix, a chunk boundary landing on a missing row made getCutPoint return a cut point one coefficient too low; the dropped coefficient corrupted the QAP evaluations and the proof failed to verify. The test covers default prove options plus explicit nChunks=4/32 (a cut point at every row), and includes a tamper-check to guard against a degenerate ptau masking failures (a no-contribution tau=1 accumulator accepts even corrupted proofs). Verified: all three tests fail against the pre-fix bisection and pass with it. --------- Co-authored-by: David <9486739+teddav@users.noreply.github.com> Co-authored-by: Oleksandr Brezhniev <oleksandr.brezhniev@gmail.com>
Picks up ffjavascript's ThreadManager fixes (worker INIT failure, postMessage dispatch failure, and post-error queue stall all now reject the affected tasks instead of leaving the main thread waiting forever) via direct pin and through binfileutils/r1csfile. Bundles rebuilt; 66 tests pass.
Proving-phase audit follow-ups (same class as the ThreadManager hang fixes): - groth16_prove: the six concurrent phases (abc, piA, piB1, piB2, piC, H) were drained by sequential awaits with no partial-failure handling. One phase rejecting (e.g. a truncated zkey failing a section read) left the rest in flight with no handler attached -- an early straggler rejection fired unhandledRejection (potential process crash after caller cleanup) and both fds leaked. All six promises are now observed on creation, drained via allSettled on failure, and the fds closed on the error path. - wtns_calculate: witness calculation now runs BEFORE the output file is opened, and writes are wrapped in try/finally -- a circuit assert used to leak the output fd and leave a zero-byte wtns file. - Re-pin fastfile/ffjavascript/binfileutils/r1csfile to pick up the page-cache error-propagation fixes (failed page read rejecting all waiters, failed background flush failing fast) and the pairing sync-op wedge fix. Bundles rebuilt; 66 tests pass; straggler repro shows 0 unhandled rejections (1 pre-fix).
The previous fix closed fds only in the six-phase drain catch. Every other await between the opens and the return -- witness header read, protocol/curve/length validation throws, the witness section read, buildABC option validation -- still leaked both fds on failure. Restructured: _groth16Prove owns the open/close lifecycle with a whole-body try/catch/finally (drain in-flight phases, then close both fds via Promise.resolve() wrapping since mem-backed fds return undefined from close), and the prove body moved to groth16ProveOpened, which registers its six concurrent phase promises in the caller's inFlight list. Empirically: 5 failed proves leaked 5 Node fds on the early-throw path pre-fix, 0 post-fix. Also: wtns_check's fdR1cs.close() was missing its await. Added a failure-path regression test (early throw + mid-phase truncated-zkey throw + prover-still-works-after).
…ess" This reverts commit d6476a7.
…inally Reworks the reverted d6476a7 per review: instead of a second wrapper layer inside _groth16Prove, the fds are opened in groth16Prove -- which already has the try/finally for the memory-logging timer -- and closed in that same finally on every exit path. _groth16Prove now receives open fds and registers its six concurrent phase promises in the caller's inFlight list, which the catch drains (allSettled) before the fds close, so no straggler rejects unobserved or races a closing fd. Empirically (repro_fd_leak.mjs): 5 failed proves leaked 5 Node fds on the early-throw path before, 0 after; truncated-zkey mid-phase failure leaks 0 and leaves 0 unhandled rejections; success path unchanged. Also restores the missing await on wtns_check's fdR1cs.close() and the failure-path regression test.
Per review: the inFlight registry conflated two mechanisms -- the
multiexp/buildABC inFlight sets exist to bound concurrent disk reads,
while this one only delayed fd close until stragglers settled. That
delay isn't needed for correctness: each phase promise gets a no-op
catch at creation, so a straggler failing after the finally closes
the fds ('Reading a closing file') rejects into an observed promise
instead of crashing the process. groth16Prove keeps just try (open +
prove) / finally (close both fds).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groth16 prover: memory scoping + MSM acceleration + msmBatching option
Summary
Companion to the
ffjavascript/wasmcurvesPRs (file-linked dependencies —land all three together). Two arcs:
Memory scoping — chunked zkey bases reads (streamed to workers instead of
whole-section buffers), adaptive streaming buildABC with per-chunk witness
gathering (the witness never enters WASM, so there is no witness size limit;
replaces the retired wasm1/multi-threaded variants and the oversized-witness
fallback), fft
consumewiring in the abc pipeline, explicit 1 GiBBigBuffer/binfileutils paging, fast
r1cs info(header-only read).MSM acceleration — exposes
options.msmBatching = "auto"|"enabled"|"disabled"on
groth16Prove, threaded to all five multiexps (A, B1, B2, C, H). Defaultautopicks the batch/endomorphism path per chunk size.Results (all proofs verify; suites pass):
Also included
browser_tests/bench.mjs: instrumented in-browser proving benchmark(prove wall, main-thread heap, renderer-tree RSS scoped to the launched
Chrome, in-page verification).
only reachable via ffjavascript's unused custom-
pluginspath) stubbed outof the single-file browser builds —
snarkjs.js5.50 → 3.81 MB,snarkjs.min.js756 → 573 KB (−24%).--memlog[=ms]ongroth16 prove/fullprove(API:memoryLogging):opt-in periodic heap/RSS/external logging, Node-gated, cleared on
completion with a final sample. Debug console output removed from the
prover (timers now only via the optional logger).
mocha 11, ejs 6); eslint surfaced and fixed a latent ReferenceError in
Polynomial.expXplus assorted dead code.the manifest resolves standalone; local dev uses uncommitted
file:overrides. Land the sibling PRs first, then re-pin here before merge.
Validation
49 passing; groth16 e2e (prove+verify) on authV3 and sha256 at every step;
CLI smoke tests on the built bundles; browser runs verify in-page.
🤖 Generated with Claude Code