Skip to content

Phase 21: SIMD Parser Adapter - #46

Merged
tazarov merged 54 commits into
mainfrom
phase/21-simd-parser-adapter
Jul 29, 2026
Merged

Phase 21: SIMD Parser Adapter#46
tazarov merged 54 commits into
mainfrom
phase/21-simd-parser-adapter

Conversation

@tazarov

@tazarov tazarov commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase 21: SIMD Parser Adapter
Goal: Land an opt-in, same-package SIMD parser implementation behind the existing parser seam without changing default stdlib behavior.
Status: Verified ✓ (16/16 must-haves)

Adds an optional pure-simdjson-backed JSON parser selectable through the existing WithParser(...) seam and gated behind a //go:build simdjson build tag. Default builds stay stdlib-only with zero SIMD dependency or native shared-library requirement. The adapter preserves Phase 07 exact-integer numeric semantics (no silent float64 coercion of overflow-sensitive values), routes numeric and transformer failures through the same policy as the stdlib path, and owns the native document lifecycle with exactly-once, panic-aware cleanup. Byte-parity with the stdlib parser is the hard contract and is asserted through committed golden fixtures.

Changes

Plan 21-01: Typed sink contract, committed-state numeric tests, buildable parity fixture/golden

Five typed scalar methods on parserSink/GINBuilder; non-coercing StageFloat64 routing with numeric hard/soft provenance.
Key files: parser.go, parser_sink.go, parser_test.go, parser_parity_fixtures_test.go, testdata/parity-golden/simd-numeric-parity.bin

Plan 21-02: NOTICE, deployment/BIGINT guidance, README activation, CHANGELOG

Pinned MIT + bundled simdjson Apache-2.0/MIT attribution chain; operator guide for activation, native loading, fallback, and integrity boundaries.
Key files: NOTICE.md, docs/simd-deployment.md, README.md, CHANGELOG.md

Plan 21-03: Lifecycle-safe tagged adapter, transform/duplicate-key parity, dependency isolation

Build-tagged pure-simdjson adapter with deterministic native document cleanup; exact json.Number materialization for transformer subtrees; source-tag + default-graph isolation guard.
Key files: parser_simd.go (new), go.mod, go.sum, Makefile

Plan 21-04: Fatal lifecycle routing, builder poisoning, combined-error provenance

Private multi-cause parserLifecycleError with errors.Is/errors.As traversal; lifecycle failures terminate the builder before recoverable ingest routing; native-free tagged lifecycle tests.
Key files: parser.go, builder.go, parser_simd.go, parser_lifecycle_test.go, parser_simd_lifecycle_test.go

Plan 21-05: Panic-aware cleanup routing and terminal-state regression

Recover a walk panic before mandatory cleanup, then re-panic unchanged on clean close or return a terminal lifecycle error on failed close; native-free caller-recovery/retry coverage.
Key files: parser_simd.go, parser_simd_lifecycle_test.go

Follow-up: SIMD parser review findings (quick task 260729-eb3)

Post-review hardening of parity and lifecycle behavior:

  • Replaced the single-leaf StageJSONNumber fallback with validated stdlibParser.Parse staging, so transformers, numeric policy, and encoding/json last-key-wins are parser-independent by construction (one implementation, not two).
  • Any panic escaping Parser.Parse now poisons GINBuilder (tragicErr) before the panic is re-raised.
  • When a walk panic and document Close failure coincide, parserLifecycleError becomes the panic value so both causes stay observable without a configured logger.
  • Documented the exact pure-simdjson v0.1.4 nesting boundary (1,023 containers accepted, depth 1,024 rejected) and pinned it with boundary tests; added authored byte-parity fixtures for mixed int/float, array siblings, and transformer-buffered container numerics.

Key files: builder.go, parser.go, parser_sink.go, parser_simd.go, docs/simd-deployment.md, parity fixtures + goldens

Requirements Addressed

  • SIMD-04 — Callers can explicitly select a same-package SIMD parser through the existing parser seam without changing the default stdlib path.
  • SIMD-05 — Default builds remain stdlib-only with no SIMD dependency or runtime shared-library requirement unless the build tag and parser selection are explicit.
  • SIMD-06 — SIMD parsing preserves Phase 07 exact-int numeric semantics and does not silently coerce overflow-sensitive values to float64.
  • SIMD-07 — The parser sink exposes typed scalar fast paths so SIMD scalar leaves do not round-trip through any.

Verification

  • Automated verification: passed (16/16 must-haves; re-verified up from 15/16 after making every failed native document close fatal on all walk outcomes, including concurrent walk panic).
  • go build ./..., go vet ./..., go test ./... clean on default build.
  • go test -tags simdjson -race ./... and make simd-isolation-check pass.

Deferred to Phase 22 (SIMD Validation, Benchmarks & CI):

  • Real native SIMD execution and encoded/query parity across authored and realistic (Phase 20) datasets — SIMD assertions here run behind //go:build simdjson and prove SIMD == stdlib-golden structurally.
  • Required tagged CI (default + -tags simdjson), platform behavior, and native-runtime availability gates.

Key Decisions

  • A well-formed native ErrInvalidJSON result below the SIMD depth limit reuses stdlibParser.Parse for staging, eliminating a second transformer/numeric-policy implementation and making parity exact by construction.
  • Failed parser cleanup is a terminal builder-integrity failure regardless of ParserFailureMode; cleanup and concurrent walk errors remain peer causes rather than being flattened into text.
  • StageFloat64 bypasses stageNativeNumeric so lexeme-classified whole floats stay floats; StageInt64/StageUint64 reuse stageNativeNumeric.
  • All pure-simdjson imports stay inside parser_simd.go under the build tag; isolation is enforced through both source inspection and the default Go dependency graph.
  • PURE_SIMDJSON_LIB_PATH integrity is operator-owned rather than extended into go.sum/bootstrap checksum guarantees.

tazarov added 30 commits July 22, 2026 20:57
- exercise every typed scalar route through AddDocument
- cover exact integers, non-coercing floats, and numeric failures
- trace the pinned pure-simdjson and bundled simdjson licenses
- document opt-in activation, native loading, integrity, and fallback
- explain BIGINT failure routing and atomic soft skips
- add exact typed scalar methods to the parser boundary
- preserve float lexeme class and numeric error provenance
- update parser documentation and the recording test sink
- keep integer and float classes on distinct canonical paths
- commit an isolated deterministic stdlib golden
- add a compiling two-step opt-in example near installation
- preserve stdlib-only defaults and link operator guidance
- record the adapter and corrected BIGINT limitation
Tasks completed: 2/2
- Source pinned NOTICE and deployment guidance
- Add README activation and CHANGELOG disclosure

SUMMARY: .planning/phases/21-simd-parser-adapter/21-02-SUMMARY.md
- record typed scalar and numeric fidelity outcomes
- capture verification evidence and downstream readiness
- pin pure-simdjson v0.1.4 behind the simdjson build tag
- preserve typed numeric, transform, and duplicate-key semantics
- release every parsed document and propagate close failures
- require exact build tags on tracked product importers
- reject pure-simdjson from the default package and test graph
- expose a standalone contributor isolation check
Tasks completed: 2/2
- Add lifecycle-safe parity-preserving adapter
- Prove source-tag and default graph isolation

SUMMARY: .planning/phases/21-simd-parser-adapter/21-03-SUMMARY.md
- Record the reported total and skipped-test count precisely
- cover cleanup-only and combined multi-cause traversal
- prove lifecycle failures poison the builder exactly once
- preserve cleanup and concurrent stage causes through multi-error unwrapping
- poison the builder before soft-skip or parser failure routing
- inject native-free walk and cleanup outcomes behind the simdjson tag
- cover close poisoning and combined numeric error provenance
- close native documents exactly once through an injectable lifecycle seam
- retain concurrent stage errors when cleanup fails
- record fatal cleanup routing and provenance-preserving lifecycle tests
- document default and tagged verification evidence
- mark Phase 21 complete and ready for verification
- close SIMD adapter requirements and record execution metrics
tazarov added 22 commits July 23, 2026 10:34
- Require identity-preserving re-panic after successful cleanup
- Exercise failed cleanup, caller recovery, and an actual retry
- Cover error-valued and non-error panic provenance
- Recover walk panics before closing the SIMD document
- Re-panic unchanged when cleanup succeeds
- Return terminal multi-cause lifecycle errors when cleanup fails
Tasks completed: 1/1
- Make panic-plus-close failure terminal and block retry dispatch

SUMMARY: .planning/phases/21-simd-parser-adapter/21-05-SUMMARY.md
Route native numeric overflows through numeric failure policy, expose deterministic parser cleanup, and exercise the real SIMD path in CI.
- Add Logger() to parserSink, returning the builder's configLogger, so
  parser_simd.go can reach structured logging without depending on
  *GINBuilder directly
- finishSIMDDocument now logs the walk-panic close-error via
  logging.Error with operation/error-type/error attrs instead of
  stdlib log.Printf; the recovered panic is still always re-raised
- Document WithParser's Close-ownership contract for supplied
  CloseableParser instances
- Document StageUint64's math.MaxInt64 precondition
- Add Logger() to recordingSink (parser_test.go) to satisfy the
  parserSink interface after the new method addition
…recursion coverage

- Add simdMaxNestingDepth (1024, matching pure-simdjson's native
  MAX_MATERIALIZE_FRAME_DEPTH) and exceedsSIMDMaxNestingDepth, a
  container-shape-only recursion guard
- routeSIMDNumericParseFailure now bails to routed=false when the
  decoded document nests past simdMaxNestingDepth, as defense-in-depth
  around findUnstageableJSONNumber's recursion
- Cover array/object recursion, deterministic first-offender ordering
  (sorted keys, ascending indices), and the depth-divergence guard with
  real-parser and direct-helper tests
…ailure crossover

- Extend the native numeric-failure test with hard-numeric-hard-parser
  and soft-numeric-soft-parser sibling cases, completing the
  ParserFailureMode/NumericFailureMode crossover matrix
- Add TestSIMDParserCloseErrorPropagatesWhenParserBusy: Close() returns
  ErrParserBusy while a document is live and succeeds once released
- Add TestSIMDParserParseAfterExternalCloseReturnsUsableBuilderError:
  Parse after an external Close degrades to a normal per-document
  IngestError instead of poisoning the builder or panicking
Build and encode the same soft-skip fixture (ordinary docs interleaved
with malformed JSON and a numeric overflow) through both stdlibParser
and the real SIMD parser, proving byte-identical finalized indexes
regardless of which parser soft-skips the bad documents.
@tazarov

tazarov commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

@claude PTAL

@tazarov
tazarov requested a review from Copilot July 29, 2026 12:18
@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Claude finished @tazarov's task in 3m 58s —— View job


Claude finished reviewing @tazarov's PR

Todo list

  • Gather context (PR description, changed files list)
  • Review diff against origin/main
  • Check CI status (default build, lint, tagged simdjson build/tests with real native library)
  • Review key files: parser.go, parser_simd.go, parser_sink.go, builder.go, Makefile/CI isolation guard, docs
  • Post review feedback

Summary

This is a well-scoped, carefully sequenced piece of work. The seam is genuinely opt-in: parser_simd.go is gated by //go:build simdjson, make simd-isolation-check verifies both that any pure-simdjson importer keeps the build tag as the literal first line and that the default go list -deps graph never pulls it in, and CI (run 30444313893, all jobs including SIMD parser) is green — the tagged job fetches the real native library and runs -race, so this isn't just structurally exercised.

Highlights from reading the code (not just the PR description):

  • builder.go:405-424 — parser panics now poison tragicErr only when they escape Parser.Parse itself (the parsing flag flips to false right after Parse returns), so a later unrelated panic in AddDocument's error-handling code isn't mislabeled as a parser panic. parserLifecycleError correctly short-circuits ParserFailureMode and terminates the builder.
  • parser_simd.go:133-162 (finishSIMDDocument) — native document Close is guaranteed exactly-once via defer; a walk panic re-panics unchanged unless Close also fails, in which case both causes are folded into a parserLifecycleError used as the panic value (so they stay observable without a configured logger) — matches the "Panic-aware cleanup" plan description.
  • parser_simd.go:80-126 (routeSIMDWellFormedFallback) — nicely avoids a second numeric/transformer implementation by re-validating with encoding/json + ensureDecoderEOF, checking only for the documented depth-1024 boundary, and then delegating straight to stdlibParser{}.Parse for staging. This is the right call — parity is structural, not maintained by hand in two places.
  • parser_simd.go:213-239 — object key iteration builds a map[string]purejson.Element before staging, which naturally reproduces encoding/json's last-key-wins duplicate-key semantics.
  • builder.go/parser_sink.goStageInt64/StageUint64/StageFloat64 cleanly route through the existing stageNativeNumeric/stageNumericObservation machinery; StageUint64 values above MaxInt64 correctly fail via the numeric policy rather than silently wrapping.
  • Docs (NOTICE.md, docs/simd-deployment.md, README.md, CHANGELOG.md) accurately describe the license chain, activation, and ownership/Close contract — I cross-checked the README example against the actual WithParser/CloseableParser signatures and it's correct.

Notes (non-blocking)

  • parser_simd.go:248-255 walks each array element twice (once for path[i], once for path[*]) by calling walkElement twice on the same purejson.Element. This assumes the native library's Element/Iter() are stateless views that can be safely re-traversed. I couldn't verify this from source (the module isn't vendored/cached in this environment), but the single-rg-array-siblings golden fixture plus the tagged CI job running against the real library both exercise exactly this path and pass, so I'm treating this as covered rather than a gap.
  • make lint locally reportedly still shows ~50 pre-existing goconst findings per the quick-task summary (.planning/quick/260729-eb3-.../260729-eb3-SUMMARY.md), but these predate this PR and the CI lint job is green, so nothing actionable here.
  • Minor: make simd-isolation-check runs go build ./... && go vet ./..., and make test depends on it before also building via gotestsum — a small amount of redundant work, not worth changing.

Environment limitation

I was not able to run go build/go test myself in this sandbox (tool calls for go build, go fetch, etc. required interactive approval that wasn't available in this automated context). I relied on static review plus the already-completed CI run (https://github.com/amikos-tech/ami-gin/actions/runs/30444313893), which shows build, lint, govulncheck, test (1.25), test (1.26), and SIMD parser all green.

Verdict

No blocking issues found. The lifecycle, panic-safety, and parity-by-construction design choices are sound, well-tested (16/16 must-haves, golden fixtures, tagged -race CI), and appropriately scoped — real benchmarking/broader validation is correctly deferred to Phase 22 rather than smuggled into this PR. LGTM.
· Branch: phase/21-simd-parser-adapter

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an opt-in, build-tagged SIMD JSON parser adapter (pure-simdjson v0.1.4) behind the existing WithParser(...) seam while keeping default builds stdlib-only. It also expands the internal parser sink contract with typed scalar staging, hardens parser lifecycle failure handling (including panic-aware cleanup + builder poisoning), and commits additional byte-parity fixtures/goldens plus deployment/attribution documentation and CI coverage.

Changes:

  • Introduces a //go:build simdjson pure-simdjson adapter (NewSIMDParser) with deterministic document lifecycle management and a validated stdlib fallback path for “native invalid JSON” due to numeric limits.
  • Extends parserSink/GINBuilder with typed numeric staging (StageInt64/StageUint64/StageFloat64) to preserve exact-int and float-lexeme semantics without unwanted coercion.
  • Adds lifecycle and parity regressions, new authored parity fixtures/goldens, documentation (deployment + NOTICE), build isolation checks, and CI jobs for default isolation + SIMD-tagged testing.

Reviewed changes

Copilot reviewed 42 out of 47 changed files in this pull request and generated no comments.

Show a summary per file
File Description
testdata/parity-golden/README.md Updates parity-golden inventory to include new authored fixtures/goldens.
README.md Documents opt-in SIMD activation and caller-owned parser lifecycle (Close).
parser.go Adds lifecycle error type + clarifies parser numeric/lifecycle contracts; introduces CloseableParser.
parser_test.go Updates test sink to satisfy expanded parserSink contract (typed numeric + logger).
parser_sink.go Expands sink interface with typed numeric staging and exposes builder logger via Logger().
parser_simd.go Implements the build-tagged pure-simdjson adapter, fallback routing, typed walking, and materialization.
parser_simd_test.go Adds untagged tests proving typed sink semantics and numeric failure mode behavior.
parser_simd_lifecycle_test.go Adds tagged lifecycle tests for exactly-once cleanup, panic/close interactions, and builder poisoning.
parser_parity_test.go Adjusts parity test structure (stdlib goldens + materializing parser parity).
parser_parity_simd_test.go Adds tagged parity test asserting SIMD/stdlib byte parity under mixed soft-skip + malformed inputs.
parser_parity_fixtures_test.go Promotes additional fixtures to authored suite and adds new transformer-buffered numeric fixture.
parser_lifecycle_test.go Adds tests for multi-cause lifecycle errors and builder terminal-state behavior.
NOTICE.md Records third-party notice chain for optional pure-simdjson and bundled upstream simdjson licenses.
Makefile Adds simd-isolation-check and wires it into make test to enforce default-build dependency isolation.
go.mod Pins github.com/amikos-tech/pure-simdjson v0.1.4 (opt-in via build tag).
go.sum Adds checksums for pure-simdjson and transitives.
docs/simd-deployment.md Provides operational guidance for activation, bootstrap/mirrors, integrity boundaries, nesting/numeric limits, and fallback.
CHANGELOG.md Documents the new opt-in SIMD adapter and its behavior constraints/limits.
builder.go Poisons the builder on parser panics and treats lifecycle failures as terminal regardless of soft modes.
.planning/STATE.md Updates project state to reflect Phase 21 completion and Phase 22 readiness.
.planning/ROADMAP.md Marks Phase 21 complete and expands Phase 21 plan breakdown.
.planning/REQUIREMENTS.md Marks SIMD-04..07 complete (Phase 21).
.planning/quick/260729-eb3-*/260729-eb3-SUMMARY.md Records quick-task changes covering fallback/lifecycle/parity/documentation hardening.
.planning/quick/260729-eb3-*/260729-eb3-PLAN.md Captures the executed quick-task plan for the follow-up hardening work.
.planning/PROJECT.md Updates milestone narrative to include Phase 21 completion.
.planning/phases/21-simd-parser-adapter/21-VALIDATION.md Adds validation strategy/evidence mapping for Phase 21.
.planning/phases/21-simd-parser-adapter/21-UAT.md Documents Phase 21 UAT checks and results.
.planning/phases/21-simd-parser-adapter/21-SECURITY.md Captures Phase 21 security threat register + mitigations evidence.
.planning/phases/21-simd-parser-adapter/21-REVIEW.md Notes scope/skip status for a prior review artifact.
.planning/phases/21-simd-parser-adapter/21-05-PLAN.md Plan document for panic-aware cleanup routing and regression coverage.
.planning/phases/21-simd-parser-adapter/21-05-SUMMARY.md Summary of Phase 21 plan 05 completion.
.planning/phases/21-simd-parser-adapter/21-04-SUMMARY.md Summary of Phase 21 plan 04 completion (lifecycle marker + poisoning).
.planning/phases/21-simd-parser-adapter/21-03-SUMMARY.md Summary of Phase 21 plan 03 completion (adapter + isolation guard).
.planning/phases/21-simd-parser-adapter/21-03-PLAN.md Plan document for implementing the SIMD adapter + isolation guard.
.planning/phases/21-simd-parser-adapter/21-02-SUMMARY.md Summary of Phase 21 plan 02 completion (NOTICE + deployment docs).
.planning/phases/21-simd-parser-adapter/21-02-PLAN.md Plan document for docs/NOTICE/README/CHANGELOG updates.
.planning/phases/21-simd-parser-adapter/21-01-SUMMARY.md Summary of Phase 21 plan 01 completion (typed sink + parity fixture).
.planning/phases/21-simd-parser-adapter/21-01-PLAN.md Plan document for typed sink contract and parity golden.
.planning/config.json Updates planning config (adds use_worktrees).
.github/workflows/ci.yml Adds SIMD isolation verification to build job and introduces a SIMD-tagged test job (ubuntu).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@tazarov
tazarov merged commit 652bcf1 into main Jul 29, 2026
7 checks passed
@tazarov
tazarov deleted the phase/21-simd-parser-adapter branch July 29, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants