Phase 21: SIMD Parser Adapter - #46
Conversation
- exercise every typed scalar route through AddDocument - cover exact integers, non-coercing floats, and numeric failures
- trace the pinned pure-simdjson and bundled simdjson licenses - document opt-in activation, native loading, integrity, and fallback - explain BIGINT failure routing and atomic soft skips
- add exact typed scalar methods to the parser boundary - preserve float lexeme class and numeric error provenance - update parser documentation and the recording test sink
- keep integer and float classes on distinct canonical paths - commit an isolated deterministic stdlib golden
- add a compiling two-step opt-in example near installation - preserve stdlib-only defaults and link operator guidance - record the adapter and corrected BIGINT limitation
Tasks completed: 2/2 - Source pinned NOTICE and deployment guidance - Add README activation and CHANGELOG disclosure SUMMARY: .planning/phases/21-simd-parser-adapter/21-02-SUMMARY.md
- record typed scalar and numeric fidelity outcomes - capture verification evidence and downstream readiness
- pin pure-simdjson v0.1.4 behind the simdjson build tag - preserve typed numeric, transform, and duplicate-key semantics - release every parsed document and propagate close failures
- require exact build tags on tracked product importers - reject pure-simdjson from the default package and test graph - expose a standalone contributor isolation check
Tasks completed: 2/2 - Add lifecycle-safe parity-preserving adapter - Prove source-tag and default graph isolation SUMMARY: .planning/phases/21-simd-parser-adapter/21-03-SUMMARY.md
- Record the reported total and skipped-test count precisely
- cover cleanup-only and combined multi-cause traversal - prove lifecycle failures poison the builder exactly once
- preserve cleanup and concurrent stage causes through multi-error unwrapping - poison the builder before soft-skip or parser failure routing
- inject native-free walk and cleanup outcomes behind the simdjson tag - cover close poisoning and combined numeric error provenance
- close native documents exactly once through an injectable lifecycle seam - retain concurrent stage errors when cleanup fails
- record fatal cleanup routing and provenance-preserving lifecycle tests - document default and tagged verification evidence
- mark Phase 21 complete and ready for verification - close SIMD adapter requirements and record execution metrics
- Require identity-preserving re-panic after successful cleanup - Exercise failed cleanup, caller recovery, and an actual retry - Cover error-valued and non-error panic provenance
- Recover walk panics before closing the SIMD document - Re-panic unchanged when cleanup succeeds - Return terminal multi-cause lifecycle errors when cleanup fails
Tasks completed: 1/1 - Make panic-plus-close failure terminal and block retry dispatch SUMMARY: .planning/phases/21-simd-parser-adapter/21-05-SUMMARY.md
Route native numeric overflows through numeric failure policy, expose deterministic parser cleanup, and exercise the real SIMD path in CI.
- Add Logger() to parserSink, returning the builder's configLogger, so parser_simd.go can reach structured logging without depending on *GINBuilder directly - finishSIMDDocument now logs the walk-panic close-error via logging.Error with operation/error-type/error attrs instead of stdlib log.Printf; the recovered panic is still always re-raised - Document WithParser's Close-ownership contract for supplied CloseableParser instances - Document StageUint64's math.MaxInt64 precondition - Add Logger() to recordingSink (parser_test.go) to satisfy the parserSink interface after the new method addition
…recursion coverage - Add simdMaxNestingDepth (1024, matching pure-simdjson's native MAX_MATERIALIZE_FRAME_DEPTH) and exceedsSIMDMaxNestingDepth, a container-shape-only recursion guard - routeSIMDNumericParseFailure now bails to routed=false when the decoded document nests past simdMaxNestingDepth, as defense-in-depth around findUnstageableJSONNumber's recursion - Cover array/object recursion, deterministic first-offender ordering (sorted keys, ascending indices), and the depth-divergence guard with real-parser and direct-helper tests
…ailure crossover - Extend the native numeric-failure test with hard-numeric-hard-parser and soft-numeric-soft-parser sibling cases, completing the ParserFailureMode/NumericFailureMode crossover matrix - Add TestSIMDParserCloseErrorPropagatesWhenParserBusy: Close() returns ErrParserBusy while a document is live and succeeds once released - Add TestSIMDParserParseAfterExternalCloseReturnsUsableBuilderError: Parse after an external Close degrades to a normal per-document IngestError instead of poisoning the builder or panicking
Build and encode the same soft-skip fixture (ordinary docs interleaved with malformed JSON and a numeric overflow) through both stdlibParser and the real SIMD parser, proving byte-identical finalized indexes regardless of which parser soft-skips the bad documents.
|
@claude PTAL |
|
Claude finished @tazarov's task in 3m 58s —— View job Claude finished reviewing @tazarov's PR
|
There was a problem hiding this comment.
Pull request overview
This PR adds an opt-in, build-tagged SIMD JSON parser adapter (pure-simdjson v0.1.4) behind the existing WithParser(...) seam while keeping default builds stdlib-only. It also expands the internal parser sink contract with typed scalar staging, hardens parser lifecycle failure handling (including panic-aware cleanup + builder poisoning), and commits additional byte-parity fixtures/goldens plus deployment/attribution documentation and CI coverage.
Changes:
- Introduces a
//go:build simdjsonpure-simdjsonadapter (NewSIMDParser) with deterministic document lifecycle management and a validated stdlib fallback path for “native invalid JSON” due to numeric limits. - Extends
parserSink/GINBuilderwith typed numeric staging (StageInt64/StageUint64/StageFloat64) to preserve exact-int and float-lexeme semantics without unwanted coercion. - Adds lifecycle and parity regressions, new authored parity fixtures/goldens, documentation (deployment + NOTICE), build isolation checks, and CI jobs for default isolation + SIMD-tagged testing.
Reviewed changes
Copilot reviewed 42 out of 47 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| testdata/parity-golden/README.md | Updates parity-golden inventory to include new authored fixtures/goldens. |
| README.md | Documents opt-in SIMD activation and caller-owned parser lifecycle (Close). |
| parser.go | Adds lifecycle error type + clarifies parser numeric/lifecycle contracts; introduces CloseableParser. |
| parser_test.go | Updates test sink to satisfy expanded parserSink contract (typed numeric + logger). |
| parser_sink.go | Expands sink interface with typed numeric staging and exposes builder logger via Logger(). |
| parser_simd.go | Implements the build-tagged pure-simdjson adapter, fallback routing, typed walking, and materialization. |
| parser_simd_test.go | Adds untagged tests proving typed sink semantics and numeric failure mode behavior. |
| parser_simd_lifecycle_test.go | Adds tagged lifecycle tests for exactly-once cleanup, panic/close interactions, and builder poisoning. |
| parser_parity_test.go | Adjusts parity test structure (stdlib goldens + materializing parser parity). |
| parser_parity_simd_test.go | Adds tagged parity test asserting SIMD/stdlib byte parity under mixed soft-skip + malformed inputs. |
| parser_parity_fixtures_test.go | Promotes additional fixtures to authored suite and adds new transformer-buffered numeric fixture. |
| parser_lifecycle_test.go | Adds tests for multi-cause lifecycle errors and builder terminal-state behavior. |
| NOTICE.md | Records third-party notice chain for optional pure-simdjson and bundled upstream simdjson licenses. |
| Makefile | Adds simd-isolation-check and wires it into make test to enforce default-build dependency isolation. |
| go.mod | Pins github.com/amikos-tech/pure-simdjson v0.1.4 (opt-in via build tag). |
| go.sum | Adds checksums for pure-simdjson and transitives. |
| docs/simd-deployment.md | Provides operational guidance for activation, bootstrap/mirrors, integrity boundaries, nesting/numeric limits, and fallback. |
| CHANGELOG.md | Documents the new opt-in SIMD adapter and its behavior constraints/limits. |
| builder.go | Poisons the builder on parser panics and treats lifecycle failures as terminal regardless of soft modes. |
| .planning/STATE.md | Updates project state to reflect Phase 21 completion and Phase 22 readiness. |
| .planning/ROADMAP.md | Marks Phase 21 complete and expands Phase 21 plan breakdown. |
| .planning/REQUIREMENTS.md | Marks SIMD-04..07 complete (Phase 21). |
| .planning/quick/260729-eb3-*/260729-eb3-SUMMARY.md | Records quick-task changes covering fallback/lifecycle/parity/documentation hardening. |
| .planning/quick/260729-eb3-*/260729-eb3-PLAN.md | Captures the executed quick-task plan for the follow-up hardening work. |
| .planning/PROJECT.md | Updates milestone narrative to include Phase 21 completion. |
| .planning/phases/21-simd-parser-adapter/21-VALIDATION.md | Adds validation strategy/evidence mapping for Phase 21. |
| .planning/phases/21-simd-parser-adapter/21-UAT.md | Documents Phase 21 UAT checks and results. |
| .planning/phases/21-simd-parser-adapter/21-SECURITY.md | Captures Phase 21 security threat register + mitigations evidence. |
| .planning/phases/21-simd-parser-adapter/21-REVIEW.md | Notes scope/skip status for a prior review artifact. |
| .planning/phases/21-simd-parser-adapter/21-05-PLAN.md | Plan document for panic-aware cleanup routing and regression coverage. |
| .planning/phases/21-simd-parser-adapter/21-05-SUMMARY.md | Summary of Phase 21 plan 05 completion. |
| .planning/phases/21-simd-parser-adapter/21-04-SUMMARY.md | Summary of Phase 21 plan 04 completion (lifecycle marker + poisoning). |
| .planning/phases/21-simd-parser-adapter/21-03-SUMMARY.md | Summary of Phase 21 plan 03 completion (adapter + isolation guard). |
| .planning/phases/21-simd-parser-adapter/21-03-PLAN.md | Plan document for implementing the SIMD adapter + isolation guard. |
| .planning/phases/21-simd-parser-adapter/21-02-SUMMARY.md | Summary of Phase 21 plan 02 completion (NOTICE + deployment docs). |
| .planning/phases/21-simd-parser-adapter/21-02-PLAN.md | Plan document for docs/NOTICE/README/CHANGELOG updates. |
| .planning/phases/21-simd-parser-adapter/21-01-SUMMARY.md | Summary of Phase 21 plan 01 completion (typed sink + parity fixture). |
| .planning/phases/21-simd-parser-adapter/21-01-PLAN.md | Plan document for typed sink contract and parity golden. |
| .planning/config.json | Updates planning config (adds use_worktrees). |
| .github/workflows/ci.yml | Adds SIMD isolation verification to build job and introduces a SIMD-tagged test job (ubuntu). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Summary
Phase 21: SIMD Parser Adapter
Goal: Land an opt-in, same-package SIMD parser implementation behind the existing parser seam without changing default stdlib behavior.
Status: Verified ✓ (16/16 must-haves)
Adds an optional
pure-simdjson-backed JSON parser selectable through the existingWithParser(...)seam and gated behind a//go:build simdjsonbuild tag. Default builds stay stdlib-only with zero SIMD dependency or native shared-library requirement. The adapter preserves Phase 07 exact-integer numeric semantics (no silentfloat64coercion of overflow-sensitive values), routes numeric and transformer failures through the same policy as the stdlib path, and owns the native document lifecycle with exactly-once, panic-aware cleanup. Byte-parity with the stdlib parser is the hard contract and is asserted through committed golden fixtures.Changes
Plan 21-01: Typed sink contract, committed-state numeric tests, buildable parity fixture/golden
Five typed scalar methods on
parserSink/GINBuilder; non-coercingStageFloat64routing with numeric hard/soft provenance.Key files:
parser.go,parser_sink.go,parser_test.go,parser_parity_fixtures_test.go,testdata/parity-golden/simd-numeric-parity.binPlan 21-02: NOTICE, deployment/BIGINT guidance, README activation, CHANGELOG
Pinned MIT + bundled simdjson Apache-2.0/MIT attribution chain; operator guide for activation, native loading, fallback, and integrity boundaries.
Key files:
NOTICE.md,docs/simd-deployment.md,README.md,CHANGELOG.mdPlan 21-03: Lifecycle-safe tagged adapter, transform/duplicate-key parity, dependency isolation
Build-tagged
pure-simdjsonadapter with deterministic native document cleanup; exactjson.Numbermaterialization for transformer subtrees; source-tag + default-graph isolation guard.Key files:
parser_simd.go(new),go.mod,go.sum,MakefilePlan 21-04: Fatal lifecycle routing, builder poisoning, combined-error provenance
Private multi-cause
parserLifecycleErrorwitherrors.Is/errors.Astraversal; lifecycle failures terminate the builder before recoverable ingest routing; native-free tagged lifecycle tests.Key files:
parser.go,builder.go,parser_simd.go,parser_lifecycle_test.go,parser_simd_lifecycle_test.goPlan 21-05: Panic-aware cleanup routing and terminal-state regression
Recover a walk panic before mandatory cleanup, then re-panic unchanged on clean close or return a terminal lifecycle error on failed close; native-free caller-recovery/retry coverage.
Key files:
parser_simd.go,parser_simd_lifecycle_test.goFollow-up: SIMD parser review findings (quick task 260729-eb3)
Post-review hardening of parity and lifecycle behavior:
StageJSONNumberfallback with validatedstdlibParser.Parsestaging, so transformers, numeric policy, andencoding/jsonlast-key-wins are parser-independent by construction (one implementation, not two).Parser.Parsenow poisonsGINBuilder(tragicErr) before the panic is re-raised.Closefailure coincide,parserLifecycleErrorbecomes the panic value so both causes stay observable without a configured logger.pure-simdjsonv0.1.4 nesting boundary (1,023 containers accepted, depth 1,024 rejected) and pinned it with boundary tests; added authored byte-parity fixtures for mixed int/float, array siblings, and transformer-buffered container numerics.Key files:
builder.go,parser.go,parser_sink.go,parser_simd.go,docs/simd-deployment.md, parity fixtures + goldensRequirements Addressed
float64.any.Verification
go build ./...,go vet ./...,go test ./...clean on default build.go test -tags simdjson -race ./...andmake simd-isolation-checkpass.Deferred to Phase 22 (SIMD Validation, Benchmarks & CI):
//go:build simdjsonand proveSIMD == stdlib-goldenstructurally.-tags simdjson), platform behavior, and native-runtime availability gates.Key Decisions
ErrInvalidJSONresult below the SIMD depth limit reusesstdlibParser.Parsefor staging, eliminating a second transformer/numeric-policy implementation and making parity exact by construction.ParserFailureMode; cleanup and concurrent walk errors remain peer causes rather than being flattened into text.StageFloat64bypassesstageNativeNumericso lexeme-classified whole floats stay floats;StageInt64/StageUint64reusestageNativeNumeric.pure-simdjsonimports stay insideparser_simd.gounder the build tag; isolation is enforced through both source inspection and the default Go dependency graph.PURE_SIMDJSON_LIB_PATHintegrity is operator-owned rather than extended intogo.sum/bootstrap checksum guarantees.