Skip to content

feat: add V3 dictionaries and bounded text processing (R1–R3) - #244

Merged
skyoo2003 merged 2 commits into
mainfrom
feature/versioned-dictionaries-r1-r3
Sep 6, 2026
Merged

feat: add V3 dictionaries and bounded text processing (R1–R3)#244
skyoo2003 merged 2 commits into
mainfrom
feature/versioned-dictionaries-r1-r3

Conversation

@skyoo2003

@skyoo2003 skyoo2003 commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Pull Request

Description

Add opt-in V3 collections for million-keyword dictionaries: callers can inspect a pinned snapshot, compare or atomically update keywords with an expected version, and continue searching the previous immutable engine while a replacement builds. Existing V1/V2 APIs and the Go module version remain compatible.

  • R1: SHA-256 partitioning into 4,096 buckets, immutable chunk/manifests, constant-work Lua commit, conflict detection and operation receipts; snapshots, pagination, diff, batch writes, leases and fenced pruning; V2 copy and dictionary CLI commands.
  • R2: Reuse unchanged bucket downloads, coalesce refresh events, and support cooperative engine-build cancellation with reduced build memory.
  • R3: Add bounded Scan, MaskText, and ReplaceText APIs with original rune/byte positions, leftmost-longest rewriting, and explicit input, match, work and output limits.
  • Include migration/operation guides, API snapshots, reproducible benchmark tooling and raw R1/R2 measurements.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Test update

Checklist

  • Tests pass (make test)
  • Vet/make vet
  • Linting passes (make lint)
  • Build succeeds (make build)
  • Documentation updated if needed
  • Changelog fragment added (changes/unreleased/20260906-versioned-dictionaries.yaml)
  • Commit messages follow guidelines

Validation

  • Root and server race tests; all three modules tested and vetted; pinned golangci-lint passes for root/server.
  • Public API audit: 300/300 symbols; 25 compiled documentation examples; tidy and license checks pass.
  • New scan parity fuzzing: 819,033 executions in 30 seconds; existing Find/Add fuzz targets also pass.
  • Fault/concurrency tests cover commit-response loss, competing writes, snapshot isolation, missed notifications, refresh failure, leases and pruning fences.
  • Real Redis 8.10.1 and Valkey 9.1.2 matrices: 10k/100k/1m keywords, shared/diverse/Korean datasets, two repetitions per case for each of R1 and R2 (36 runs each), plus million-keyword safety tests.
  • At one million keywords, measured server-received bytes for a single-keyword update fell 91.9–96.2% from R1 to R2. Full environment, timing, RSS and raw results are committed under docs/content/reference/ and benchmarks/results/.

Additional Notes

Writes acknowledge the Redis commit; callers needing local search visibility must use WaitForVersion. Local engines still rebuild in full. Collections occupy one Redis Cluster slot. Operation receipts and committed-version markers are retained indefinitely.

The local Valkey default-prefetch build crashed during an exploratory run. Both successful benchmark matrices used prefetch-batch-max-size 0; default Valkey configuration is not validated by these measurements. The crash excerpt and condition are documented. Workstation measurements with two repetitions are not latency or memory guarantees. Real Cluster/Sentinel failover and production persistence-recovery matrices remain unverified.


By submitting this PR, I agree that my contributions will be licensed under the Apache License 2.0.

@github-actions github-actions Bot added documentation enhancement This issue is about improvement of project. go Pull requests that update Go code github_actions Pull requests that update Github_actions code labels Sep 6, 2026
@skyoo2003
skyoo2003 merged commit 6cd00b3 into main Sep 6, 2026
11 checks passed
@skyoo2003
skyoo2003 deleted the feature/versioned-dictionaries-r1-r3 branch September 6, 2026 02:38
skyoo2003 added a commit that referenced this pull request Sep 6, 2026
`builds[].main` named `cmd/acor/main.go`, a file rather than a package. Go
compiles a file path as `command-line-arguments` and drops its siblings in the
same package, so the release binary was built from main.go alone.

That held by accident: until #244, `cmd/acor` contained only `main.go` and
`main_test.go`. #244 added `dictionary.go`, and `main.go:224` calls
`dispatchDictionary` from it — so the v1.6.0 tag build failed with `undefined:
dispatchDictionary` after every CI gate had passed.

No gate could have caught it. `make build` and CI run `go build ./cmd/acor`,
the package; GoReleaser was compiling something else. Naming the package makes
the two builds the same build, so `make build` now gates the release path.

Verified with `goreleaser build --snapshot --single-target`: builds
`paths=cmd/acor`, and the binary's `--help` lists `dictionary`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation enhancement This issue is about improvement of project. github_actions Pull requests that update Github_actions code go Pull requests that update Go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant