[fcostaoliveira]

Follow-up to #381 (merged as 5fe1278, by @murtazayusuf). It started as #326, also by @murtazayusuf, whose commit ce46925 is the first commit on this branch.

Thank you, Murtaza. The version gate, the kividb_caps module, SEPARATOR ; on 1.1.2 and the 1.0.5 and 1.1.2 re-measurements are yours and are already on master. This PR adds what #381 left open: every TAG filter value is exact on 1.1.2, or a hard error.

The problem on master

#381 sends TAG filter values raw. Measured on kividb/kividb:v1.1.2-full (throwaway CI probe #382), 22 of 146 probe values come back wrong when sent raw:

• a\b matches ab;
• J\"org Hennig matches J"org Hennig;
• x|y matches x and y;
• a } or a leading " is a syntax error.

arxiv-filters is the shipped dataset this affects. 77 of its 10,000 queries contain \, 14 contain braces and 4 contain parentheses. Raw, these return 0 documents or fail per query. search() drops a failed query with only a log line, so recall comes out silently wrong.

What this changes (6 files against master)

Query side (src/kividb_caps.rs, library).

• tag_literal escapes every TAG value: a backslash before each ASCII punctuation character and space, which is the RediSearch rule.
• On 1.1.2 the escaped form is exact for all 32 ASCII punctuation characters plus space, at the start, middle and end of a value, bare, grouped and AND'd. That includes ( ) | @, so parentheses are not refused.
• tag_value_problem turns values with no exact encoding into hard errors:
  • ; (the separator);
  • a trailing * (a prefix wildcard even when escaped);
  • a leading $ (a query parameter even when escaped);
  • leading or trailing whitespace (trimmed).
• Two more are hard errors because they are unmeasured:
  • an empty value;
  • whitespace other than a space or a control character inside a value (tab, newline, CR, VT, FF, U+00A0, U+3000, …).
• A scan of every shipped TAG query value (h-and-m, arxiv, the three random-keyword sets, tenants-100) finds none of these shapes.
• On a server where TAG fields are unsupported (older than 1.1.2, or no readable version), every TAG value is a hard error. This also covers a string condition on a field the schema doesn't declare.

Engine (kividb.rs, filter_guard.rs).

• tag_clause and every match_any arm go through tag_literal.
• An empty value is no longer silently turned into a never-match.
• The raw-value refusal path (check_tag_value / tag_value_offender) and its tests are replaced.

Live fixtures (tests/integration_kividb.rs, tests/common/mod.rs, KiviDB 1.1.2).

• Backslash / brace / quote values and ( ) | @ values, each next to its decoy, run with --fail-on-dropped-queries.

• test_binary_kividb_tag_values_exact_counts. Each value is stored on its own number of documents. The engine runs the dataset (--fail-on-dropped-queries --keep-data, recall >= 0.95). Each value is then queried on the engine-built index with the engine's own tag_literal, bare and as (A | none) A, and must return exactly its expected document ids:

  value	docs		value	docs
  J\"org Hennig	12		Smith, John	26
  J"org Hennig	14		Smith	28
  Rasmus Bj{\o}rk	16		Francisco (Paco) Guinea	30
  Rasmus Bjork	18		Francisco	32
  a\b	20		f(x)	34
  ab	22		x|y	36
  close}brace	24		x	38

README. The KiviDB note covers the escaping rule and its 1.1.2 measurements, and the full hard-error list including the two unmeasured shapes. It also states that refusing the 11 scalar-TAG datasets below 1.1.2 narrows support: they ran correctly on the old 1.0.2 pin. The arxiv count check is labelled correctly: 2,123,402 submitter values across 2,138,591 documents, checked on 1.0.5.

Verification

• make agent-check: exit 0 (464 lib + 969 bin tests, plus the invariant suites). The head's tree is identical to the gated commit.
• Live mutation, throwaway draft tmp: #375 mutation check, TAG escaping reverted (throwaway, do not merge) #383 (closed, branch deleted): with escaping reverted, the KiviDB job went red, 3 of 26 failing. The exact-count and backslash/brace runs failed on brace syntax errors, and the paren fixture scored recall 0.630.
• Unit mutations, each restored afterwards, all red:
  • escaping reverted;
  • \ not escaped;
  • tag_literal ignoring server support;
  • each of the four hard errors allowed;
  • empty value allowed;
  • the interior-whitespace check removed;
  • that check narrowed to is_ascii_whitespace (which misses U+00A0, U+3000 and U+2028);
  • the control-character check removed;
  • a space refused too (positive control).

History and merging

• Squash-merge, please, and trim the squash text. The branch carries commit messages that later commits reversed: the 1.0.5 gate, "keep pre-1.0.5 KiviDB live-tested" and the legacy job. Keep the Co-authored-by trailer below.

• b7dda91 differs from 62ada3a. It is @murtazayusuf's 62ada3a cherry-picked with his author and message intact, but resolved against this branch's escaping work:

  • TagSemantics became tags_supported: bool;
  • the 1.0.2 raw-emission branch and its tests were dropped;
  • the round-4 refusal details were kept;
  • a prose paragraph listing the measured versions was added to TAG_MIN_VERSION's doc.

  It carries no cherry-pick trailer, so git blame on b7dda91 shows him for code he didn't write. The squash avoids that.

• origin/master (with 5fe1278) is merged in as a merge commit. The merge needed no hand edits beyond keeping this branch's side, and git diff origin/master...HEAD is this follow-up only.

Deferred

• Redis-wire TAG semantics (; in scalar values, case folding) + deferred KiviDB test/CI items from #375 #377:
  • ; in scalar TAG values across the Redis-wire engines;
  • case-insensitive TAG matching versus exact ground truth (31 h-and-m and 30 arxiv queries over-match, as on redis.rs's vector layout);
  • a 1.1.2 probe for the unmeasured whitespace and empty shapes;
  • an image digest pin / version line in the CI log;
  • the pre-release gate;
  • a live red test for search()'s skip-upload version check;
  • re-measuring two re-labelled comments;
  • the fixture's exact-version wording.
• KiviDB 1.1.2: relax the conservative multi-term / non-alphanumeric TEXT rejection #376: multi-term / non-alphanumeric TEXT is still refused.
• RESP3 FT.INFO fixture: the engine is RESP2-only.

Co-authored-by: murtazayusuf 65263321+murtazayusuf@users.noreply.github.com

🤖 Generated with Claude Code

[github-actions]

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

What I checked: the encoding path from parse_conditions → build_filter → tag_clause / build_match_any → kividb_caps::tag_literal; the tag_value_problem refusal set and the order of its checks; the gate that threads tags_supported from search()'s one INFO server read into the prefilters; the filter_guard.rs matrix arm; the three new live fixtures in tests/common/mod.rs and tests/integration_kividb.rs; and the README rewrite against the code.

Against this repo's main bug class (taxonomy item 1, "still runs, still reports a number, the number is wrong"), this PR fixes a real silent wrong-answer on master and adds none of its own:

• On master, a value that can't be encoded is sent raw: per the PR body's 1.1.2 probe, 22 of 146 probe values come back wrong, and on arxiv-filters 77 of 10,000 queries return 0 docs or a dropped query. After this PR every such value goes through tag_literal.
• tag_literal either returns the escaped literal or an Err. The Err propagates out of try_resolve_all before read_queries(), so nothing degrades to a never-match or an unfiltered run.
• The empty-value path that used to turn into never_match_tag (old tag_clause, and the continue in build_match_any) is now a hard error too. That is the right direction, since whether KiviDB indexes an empty stored TAG is unmeasured.

The test design holds up:

• In the exact-count fixture each value has its own document count (12, 14, …, 38), and the test asserts the exact ids, not just recall. Because it queries with kividb_caps::tag_literal itself, it can't fall into the PR Add kividb engine #203 failure where a test reimplements the logic it guards.
• Every escape-fixture value is also queried on its own at least once, so one mis-encoded value costs a whole query (≥0.1 recall on 10 queries), which the 0.95 floor catches.
• The pre-existing test_binary_kividb_uuid already exercises escaped hyphens live on the 1.1.2 pin, so the flipped kividb: metadata filtering broken against real KiviDB (TAG escaping + atomic tags); FT.CREATE SEPARATOR fixed #205 regression guard (uuid_hyphens_are_escaped) has live coverage, not just a unit assertion.
• The mutation list in the body (escaping reverted → 3 of 26 red live, paren fixture at recall 0.630; plus the unit mutations) is what this repo's bar asks for.

Worth fixing before merge

Nothing found.

Optional / follow-up (not blockers)

1. The arxiv corpus check ran on 1.0.5, not the CI pin. The full-corpus check (2,123,402 submitter values, every escaped query value returning its expected count) ran on 1.0.5. That 1.0.5 handles escapes the same way as 1.1.2 rests on the 146-value probe. The README is honest about this (README.md "Which shipped datasets this excludes"). Still, the one shipped dataset this PR unblocks has its end-to-end evidence on a version that is now outside the support gate. If the Redis-wire TAG semantics (; in scalar values, case folding) + deferred KiviDB test/CI items from #375 #377 re-measure pass happens anyway, a 1.1.2 re-run of that check would close the gap.
2. Empty TAG values now differ from redis.rs. The removed comment in build_match_any said empty strings were skipped "same as redis.rs". KiviDB now hard-errors on them while redis.rs presumably still skips them. That is deliberate and documented here, but it's the same kind of cross-engine question Redis-wire TAG semantics (; in scalar values, case folding) + deferred KiviDB test/CI items from #375 #377 already tracks (;, case folding). It's worth one line there so the Redis-wire engines stay comparable in one place.
3. Wording nit on tag_literal's doc. It says the !tags_supported arm "also covers a string condition on a field the schema does not declare as a TAG". That is true for strings only: the bool exact-match path (@f:{true}) is still emitted on an unsupported server, and unsupported_server_refuses_any_tag_value_and_nothing_else pins that on purpose. It's harmless, because true/false need no escaping and validate_dataset_support refuses a bool-declared dataset first. The doc could just say "string".

Squash-merging as the body asks, keeping the Co-authored-by trailer, avoids misattributing b7dda91 in blame.

Net: a careful, well-measured fix to a silent wrong-recall on the one shipped dataset it affects, with live exact-count coverage on the pinned image. Nothing here should block it; the items above are follow-up material.

[fcostaoliveira]

Reply to the automated review above (pass 1, on efbe527). Follow-ups pushed in 7f7c268.

1. A ; inside a scalar filter value under-matches on >= 1.0.5. Agreed it's real, and checked against shipped data as you asked. I streamed every shipped dataset with a TAG-typed field (h-and-m, arxiv, random-match-keyword-100, MS MARCO 100K), corpus and queries:

• ; in a query filter value: 0 everywhere (h-and-m ~10k values, arxiv 6.6k, random-keyword 16.6k; MS MARCO has no filtered vector queries).
• ; in the corpus: 12 arxiv submitter values (HTML entities such as ł) and 10 MS MARCO urls, none of them queried.

The corpus half (a stored a;b over-matching @f:{a}) can't be caught query-side, and it's identical on Redis/Valkey/Dragonfly, which all declare SEPARATOR ;. So I filed the guard as a cross-engine issue, #377 (query-side refusal + upload-side check + fixtures), rather than special-casing KiviDB's query side here. The same scan shows why the comma fix matters: h-and-m has a comma in 99 query filter values (section_name, garment_group_name), which would silently under-match on 1.0.5 with bare TAGs. Both numbers are now in the README (7f7c268).

2. --skip-upload doesn't check how an existing index was declared. Fixed (1b6138d). KiviDB 1.0.5's FT.INFO lists attributes and reports a bare TAG as SEPARATOR , (measured; 1.0.2 has no attributes). The new kividb_caps::ft_info_tag_separators + check_index_tag_separators require ; on every schema TAG field on a >= 1.0.5 server. On such a server, an index that reports no attributes is refused as unverifiable; pre-1.0.5 servers have nothing to check. search() runs it after ensure_index_exists, still before read_queries(), on both paths. On the normal path it also asserts the server honoured configure()'s declaration.

• Live test test_kividb_skip_upload_refuses_a_stale_bare_tag_index: it uploads the comma fixture with --keep-data and rebuilds the index bare over the kept keys. --skip-upload is then refused, naming author and ",", and no search result is written. As a positive control, rebuilding with SEPARATOR ; is accepted at recall >= 0.9.
• Mutations (restored after): check not called → that live test red; any separator accepted → unit + live red; no-attributes accepted on >= 1.0.5 → unit red.

3. expected_tag_semantics() defaults to separator. Kept as is, by design, as you note. The default matches the compose kividb service, and the failure message names KIVIDB_EXPECT_TAG_SEMANTICS and what the server reported.

Verification at 7f7c268: make agent-check exit 0; integration_kividb 23/23 on v1.0.5-full and 23/23 on v1.0.2-full (KIVIDB_EXPECT_TAG_SEMANTICS=atomic).

[fcostaoliveira]

Reply to the automated review (pass 2, on 7f7c268). Follow-ups pushed in 945da24.

1. TAG types written out in three places. Fixed (bc48753). is_tag_type is now the single definition, used by create_index, validate_dataset_support and tag_field_names, which scopes the FT.INFO check. tag_field_names_are_exactly_the_tag_typed_fields pins the set. Mutations: letting tag_field_names drift back to its own literal without bool → that test red; dropping bool from is_tag_type → it and unknown_version_refuses_every_tag_field_and_nothing_else red.

2. The ; note checked the wrong direction. You're right that the risk is a query equal to a fragment. I re-streamed arxiv: none of its 2,928 distinct submitter query values equals any of the 20 ;-fragments of the 12 stored values, compared case-insensitively and trimmed, as a TAG matches. MS MARCO has no filtered vector queries, and h-and-m and random-keyword store no ;. README and #377 updated (945da24).

3. Squash attribution. Thanks for flagging this. It needed more than the PR body. This repo squashes with COMMIT_MESSAGES, and ce46925 is authored as Claude <noreply@anthropic.com>, so the body's trailer would not have reached the squash commit. bc48753's message now carries Co-authored-by: murtazayusuf <65263321+murtazayusuf@users.noreply.github.com>, so the default squash text includes it, with no force-push needed. Whoever merges should still glance at the squash text.

Verification at 945da24: make agent-check exit 0; integration_kividb 23/23 on v1.0.5-full and 23/23 on v1.0.2-full.

[fcostaoliveira]

Reply to the automated review (pass 3, on 945da24). Nothing pre-merge was raised. On the optional items:

1. Comma fixture on Redis/Valkey/Dragonfly. Agreed, and it's cross-engine, so I added it to Redis-wire TAG semantics (; in scalar values, case folding) + deferred KiviDB test/CI items from #375 #377 (item 4) alongside the ; guard rather than widening this PR.
2. --skip-upload separator readback for the Redis family. Also added to Redis-wire TAG semantics (; in scalar values, case folding) + deferred KiviDB test/CI items from #375 #377 (item 5), noting the lower risk you describe.
3. Hardcoded idx:{name} in the stale-index test. Left as is. It follows the suite's existing convention (test_kividb_coexistence_skip_upload hardcodes idx:kividb-co-a the same way), and as you note a non-default KIVIDB_INDEX_NAME makes it fail loudly, not wrongly.

No code changes for this pass; head stays 945da24.

[murtazayusuf]

Thanks a lot @fcostaoliveira for carrying #326 forward, and for the thorough follow-ups.

One request before merge: could CI use KiviDB 1.1.2, our current stable release, instead of 1.0.5?

• tests/docker-compose.test.yml: quay.io/kividbio/kividb:v1.0.5-full → kividb/kividb:v1.1.2-full (Docker Hub), or ghcr.io/kividbio/kividb:v1.1.2-full. Quay no longer gets new images since 1.1.0, so there's no 1.1.2 there. Docker Hub and GHCR carry every release.
• The >= 1.0.5 gate stays right: 1.1.2 splits TAGs on SEPARATOR exactly as 1.0.5 does. Please avoid 1.1.1 if it ever comes up; it was withdrawn.
• Keeping the v1.0.2-full legacy job for the pre-1.0.5 branch makes sense. That image stays on Quay.
• If it isn't much work, it would be nice for the README's "Re-measured" section to say 1.1.2 rather than 1.0.5. If the probes come out the same, a one-line note is enough.

If it's easier, we can push that change to our fork (kividbio/vector-db-benchmark) for you to pick up. Just say which you prefer.

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)

[murtazayusuf]

@fcostaoliveira, an update to my earlier comment: we've gone further and moved the KiviDB support to 1.1.2 only. The gate is now 1.1.2 and the 1.0.2 legacy job is gone. Your commits are carried unchanged in #381, with one commit of ours on top: details there. Please merge whichever you prefer. If you'd rather keep this PR, cherry-picking 62ada3a onto it gives the same result, and we'll close #381. Thanks again for all the follow-up work.

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)

[fcostaoliveira]

Replies to the 17:14 automated pass on 96ce29d, plus the last reviewer round. The doc fixes are in 16868cb: README and tests/common/mod.rs only, no src/ change.

1. Empty match_any arm vs redis.rs: tracked in Redis-wire TAG semantics (; in scalar values, case folding) + deferred KiviDB test/CI items from #375 #377 C. We keep the hard error on KiviDB, because an empty value is unmeasured there.
2. Exact-count fixture in hybrid form: tracked in Redis-wire TAG semantics (; in scalar values, case folding) + deferred KiviDB test/CI items from #375 #377 C. Today the hybrid shape is covered by recall 1.0 in the same live runs.
3. Scan coverage: the README now says synthetic-filter-32 only emits fixed colour literals, so it can't hit a refusal.
4. The --skip-upload version-check live test: agreed. It stays deferred, because a 1.1.2-only CI can't reach that branch live. The unit tests cover both sides of TAG_MIN_VERSION.
5. "ten queries": the doc now says the queries cycle through the N_QUERIES (10) targets.

Also fixed from the reviewer round:

• README:133 no longer says the geo datasets ran on older servers. They declare no TAG field and already aborted before ROOT CAUSE: no engine distinguishes "no filter" from "filter that parsed to nothing" — silent unfiltered runs scored against filtered ground truth #219. It also no longer credits kividb: exact TAG filter values on 1.1.2 (escaping + live fixture), follow-up to #381 #375 with the pre-1.1.2 refusal, which arrived in feat(kividb): filter multi-valued labels on KiviDB, CI on the latest stable 1.1.2 #381.
• The hard-error wording now reads unambiguously: control characters are refused.

Unicode format characters (U+200B, U+FEFF, …) aren't measured on 1.1.2 and aren't refused. No shipped value contains one; this is noted for #377.

🤖 Generated with Claude Code

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)