Phase 11: close BigInt, depth, and exception safety gaps by tazarov · Pull Request #38 · amikos-tech/pure-simdjson

tazarov · 2026-07-30T09:01:57Z

Summary

Phase 11: Upstream simdjson refresh, exact big integers, and production diagnostics

Goal: Establish the v0.2 compatibility foundation with audited simdjson v4.6.4 behavior, exact oversized-integer text, safe parser controls, and truthful diagnostics.

Status: Verified ✓

This follow-up closes the safety and validation gaps found after the initial ABI 1.2 shipment. It bounds every parser-owned recursive path, rejects malformed BigInt suffixes across all generated architecture implementations, contains C++ allocation exceptions at the Rust boundary, and records complete phase verification and Nyquist evidence.

Changes

Plan 15: Maximum-depth diagnostic safety

Defines 1024 as both the default and maximum supported parser depth.
Rejects larger values before implementation selection, allocation, or recursive replay.
Proves maximum-depth malformed input returns a typed diagnostic in a child process.
Keeps Rust, generated C, and Go documentation aligned.

Key files: parser_options.go, parser_options_test.go, src/native/simdjson_bridge.cpp, tests/rust_shim_limits.rs, tests/rust_shim_diagnostics.rs, include/pure_simdjson.h, docs/ffi-contract.md

Plan 16: BigInt token-boundary validation

Validates delimiters before every BigInt early return.
Adds fail-closed 9/9/9 generated-architecture parity guards.
Adds NUL-safe native regressions and four manifest-complete JSONTestSuite rejects.
Preserves exact kind-9 text for valid positive and negative values.

Key files: patches/simdjson-v4.6.4-positive-bigint.patch, build.rs, tests/rust_shim_bigint.rs, testdata/jsontestsuite/expectations.tsv

Plan 17: C++ exception status contract

Maps trapped runtime and allocation exceptions to status 97.
Keeps returned simdjson::MEMALLOC and explicit internal failures on status 127.
Reuses the existing hidden selector seam without widening the public ABI.

Key files: src/native/simdjson_bridge.{h,cpp}, src/runtime/mod.rs, src/lib.rs, tests/rust_shim_minimal.rs

Plan 18: Parser-aware `bad_alloc` containment

Removes allocation before the parser diagnostic guard.
Routes deterministic selector coverage through the production parser-aware catch macro.
Adds subprocess proof of status 97, output-sentinel integrity, and process survival.

Key files: src/native/simdjson_bridge.cpp, tests/rust_shim_minimal.rs

Phase closeout

Records the completed ABI 1.2 publication and public-bootstrap evidence.
Adds final review, security, verification, and validation reports.
Updates the Nyquist map for Plans 15–18 with zero missing or partial gaps.

Requirements Addressed

UP-01: Preserve the audited v4.6.4 pin and all contract/build gates.
NUM-01: Preserve valid oversized integers as exact text while rejecting malformed suffixes.
NUM-02: Regression-verify strict copied TypeBigInt / GetBigInt behavior.
DIAG-01: Regression-verify process-global kernel reporting and control.
DIAG-02: Return proven offsets or explicit unknown without process termination.
LIMIT-01: Enforce safe capacity/depth options with one maximum-depth contract.

Verification

Phase verification: passed — 104/104 must-haves.
Nyquist audit: 6/6 requirements green, 0 gaps.
make verify-contract — 91 Rust tests, deterministic header generation, 25 ABI audits, and C layout compile.
make verify-docs.
Fresh release library + go test ./... -race -count=1 -timeout=180s — all four Go packages.
Release/bootstrap contract suites — 46/46 Python tests.
Complete JSONTestSuite correctness oracle.
No-baseline PR benchmark signal with non-empty text, JSON, and Markdown outputs.
Non-strict release readiness.

Key Decisions

Depth 1024 is both the default and supported maximum across construction, replay, and materialization.
BigInt exact text is returned only after the following byte is proven to be valid JSON punctuation or whitespace.
Every generated parser copy must contain all three guarded BigInt branch shapes.
Trapped C++ exceptions return status 97; returned engine allocation failures remain status 127.
Parser-side allocation diagnostics perform no potentially throwing work before the catch-all guard.
ABI 0x00010002, public symbols, layouts, dependencies, and release behavior remain unchanged.

Shipping

Merge method: squash merge.
This PR does not publish or retag artifacts; the final v0.2 release remains Phase 16 work.

- require Go and native rejection above depth 1024 - isolate maximum-depth malformed replay in a child process

- reject depths above 1024 before native selection or allocation - share one native ceiling across replay and materialization

- document zero or 1 through 1024 as the accepted range - regenerate the public header from the Rust declarations

- cover malformed suffixes including embedded NUL bytes - preserve exact valid roots and nested values across delimiters

- validate delimiters before every BigInt early return - fail builds on guarded-branch parity drift across architectures

- cover positive and negative root suffix corruption - cover nested array and object token boundaries

- add sorted reject expectations for four token suffix fixtures - preserve every existing oracle expectation and valid BigInt control

- Extend the hidden exception seam with fixed test selectors - Cover runtime_error, bad_alloc, and unsupported selectors - Prove bad_alloc returns 127 before the mapper fix

- Align trapped bad_alloc with the normative exception status - Preserve returned MEMALLOC and internal error mappings - Keep both production catch macros on the shared mapper

- Run selector 3 in an exact-filter child process - Require exact status 97, one child test, and a success marker - Fail on child termination, wrong status, or missing evidence

- Record a fixed allocation-safe diagnostic before status mapping - Route selector 3 through the production parser-aware catch macro - Fail closed if the private helper mutates its output sentinel

Tasks completed: 1/1 - Contain parser-aware bad_alloc and prove survival through the production catch macro SUMMARY: .planning/phases/11-upstream-simdjson-refresh-bigint-and-diagnostics/11-18-SUMMARY.md

github-actions · 2026-07-30T09:05:00Z

PR Benchmark - no regressions

Threshold: >= 5.00% slower and p < 0.050.

This check is advisory; the PR is not blocked.

tazarov added 30 commits July 30, 2026 11:55

docs(11-14): complete ABI 1.2 publication plan

107c4d8

docs(11): record large-depth verification gap

52f9b86

docs(11): add large-depth safety gap plan

575d86f

docs(11): finalize gap closure plan

7999198

test(11-15): add failing maximum-depth safety contracts

71b9d65

- require Go and native rejection above depth 1024 - isolate maximum-depth malformed replay in a child process

feat(11-15): enforce maximum supported parser depth

7c06e98

- reject depths above 1024 before native selection or allocation - share one native ceiling across replay and materialization

docs(11-15): synchronize supported depth contract

979ce08

- document zero or 1 through 1024 as the accepted range - regenerate the public header from the Rust declarations

docs(11-15): complete maximum-depth safety plan

1b428ed

docs(11): add code review report

ca97e67

docs(11): record remaining verification gaps

66ad796

docs(11): add focused verification gap plans

4598492

docs(11): finalize gap-closure planning state

6baf1a8

test(11-16): add failing BigInt token-boundary regressions

c2345fc

- cover malformed suffixes including embedded NUL bytes - preserve exact valid roots and nested values across delimiters

feat(11-16): guard BigInt token boundaries

0acfdee

- validate delimiters before every BigInt early return - fail builds on guarded-branch parity drift across architectures

test(11-16): add failing malformed BigInt oracle fixtures

248fcfd

- cover positive and negative root suffix corruption - cover nested array and object token boundaries

feat(11-16): register malformed BigInt oracle rejects

237f7a1

- add sorted reject expectations for four token suffix fixtures - preserve every existing oracle expectation and valid BigInt control

docs(11-16): complete BigInt token-boundary plan

dcd15ba

test(11-17): add failing bad_alloc exception regression

f7af406

- Extend the hidden exception seam with fixed test selectors - Cover runtime_error, bad_alloc, and unsupported selectors - Prove bad_alloc returns 127 before the mapper fix

feat(11-17): map bad_alloc exceptions to status 97

3bf1f2c

- Align trapped bad_alloc with the normative exception status - Preserve returned MEMALLOC and internal error mappings - Keep both production catch macros on the shared mapper

docs(11-17): complete C++ exception status plan

98701df

docs(11): add code review report

54e7637

docs(11): record remaining verification gaps

4f09004

docs(11): plan parser-aware bad-alloc gap closure

b4573b7

docs(11): finalize parser-aware gap planning

d2d7e9a

test(11-18): add failing parser bad_alloc survival regression

a116cca

- Run selector 3 in an exact-filter child process - Require exact status 97, one child test, and a success marker - Fail on child termination, wrong status, or missing evidence

fix(11-18): contain parser bad_alloc diagnostics

68af92b

- Record a fixed allocation-safe diagnostic before status mapping - Route selector 3 through the production parser-aware catch macro - Fail closed if the private helper mutates its output sentinel

docs(11-18): complete parser bad_alloc containment plan

ec83e84

Tasks completed: 1/1 - Contain parser-aware bad_alloc and prove survival through the production catch macro SUMMARY: .planning/phases/11-upstream-simdjson-refresh-bigint-and-diagnostics/11-18-SUMMARY.md

docs(11): refresh code review after parser containment

344eba0

docs(phase-11): complete phase execution

0f64ab1

docs(phase-11): evolve PROJECT.md after phase completion

1153537

tazarov added 4 commits July 30, 2026 11:55

docs(phase-11): add security threat verification

a2efa88

docs(phase-11): update validation strategy

d8921c4

docs(phase-11): refresh rebased validation evidence

a91cca5

docs(phase-11): mark final gap plan complete

24d5a1b

docs(11): ship phase 11 — PR #38

c93e28c

tazarov merged commit d60aa9b into main Jul 30, 2026
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Phase 11: close BigInt, depth, and exception safety gaps - #38

Phase 11: close BigInt, depth, and exception safety gaps#38
tazarov merged 35 commits into
mainfrom
gsd/phase-11-gap-closure

tazarov commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

tazarov commented Jul 30, 2026

Summary

Changes

Plan 15: Maximum-depth diagnostic safety

Plan 16: BigInt token-boundary validation

Plan 17: C++ exception status contract

Plan 18: Parser-aware bad_alloc containment

Phase closeout

Requirements Addressed

Verification

Key Decisions

Shipping

Uh oh!

github-actions Bot commented Jul 30, 2026

PR Benchmark - no regressions

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Plan 18: Parser-aware `bad_alloc` containment