Repository navigation
Harden config validation, move catalog data into the package, and recalibrate router thresholds per backend - #2
Merged
Conversation
…ge" -m "Settings now validates backend selectors and numeric bounds at the process boundary (fail closed) and gains explicit knobs for context size, catalog limits, and the Groq/DSPy backend. Catalog JSON moves from top-level data/ into src/meridian/data so it ships as package data. CI adds a format check and a build step; Redis ports bind to localhost only.
Introduces `--flat-demo` (and `make flat-demo`), which seeds the same knowledge into the older, single-representation `KnowledgeChunk` model and runs the identical ACL probe used by `--acl-demo`. It exists purely as a point of comparison: same access-control guarantee, same probe, but every candidate carries its full text into the search index instead of the fat/slim split's lean projection - making the cost trade-off the fat/slim section already describes concrete rather than asserted. The demo catalog's prose (knowledge_base.json / knowledge_base_fat.json) is rewritten with more grounded rationale - why full jitter avoids retry storms, why a token bucket beats a fixed window, why idempotency keys are per-operation not per-attempt - matching the sourcing note already in the README (SRE book, OWASP, CNCF/Envoy conventions). service_catalog picks up matching detail. README/README.pt-BR document the new demo and refresh the example outputs to match. Unrelated one-liner bundled in: groq.py disables dspy's warn_on_type_mismatch, which fires spuriously on every dspy.Refine retry because OfferFeedback's own internal feedback step JSON-encodes floats before passing them to a signature typed as float - a dspy-internal mismatch, not a signal about our signatures. tests/integration/test_ask_flow.py adds test_flat_chunk_acl_filter_blocks_unauthorised_source, covering the flat path with the same ACL guarantee test the fat/slim path already had.
…istic Ties were previously broken by an accident of Python's stable sort plus `self._intents` being built from `sorted(positive_texts.keys())` - alphabetical order, but never a deliberate policy, undocumented, and untested. That silently biased which intent won a tie, which is visible downstream: the disambiguation prompt's "could go a couple of ways (X, Y)" list, per-route metrics attribution, and the traced disambiguation_rule all depend on it. The sort key is now explicit: score descending, then the intent's own confidence threshold descending (a higher bar cleared signals a more specific match), then intent name ascending as the final deterministic fallback. Behaviourally unchanged for today's catalog where tied intents share thresholds, but no longer coupled to incidental iteration order. Adds two tests that construct a genuine score tie (two intents sharing a positive example) and lock in: the winner resolves by threshold, and an exact tie is always flagged ambiguous regardless of which side wins the sort (rule 3's margin check fires on any zero gap).
Running the demo with MERIDIAN_EMBEDDING_BACKEND=local (real MiniLM
embeddings) flagged nearly every non-greeting query as ambiguous, even
exact positive-example matches. Root cause, confirmed by scoring the
catalog's confusable pairs plus ~20 held-out paraphrases against MiniLM
at several penalty values: ranking correctness was never the problem
(20/21 correct regardless of negative_penalty) - the production
negative_penalty of 0.8 assumes confusable phrases score low in
similarity, but MiniLM's general-purpose space puts same-domain,
different-intent phrases around 0.5-0.6 similarity just from shared
vocabulary, so the penalty crushed even perfect positive matches to
~0.55, under every production threshold (0.70-0.78).
Adds LOCAL_AMBIGUITY / LOCAL_INTENT_THRESHOLDS (negative_penalty=0.5,
thresholds 0.30-0.38) derived from that empirical run, sitting between
the observed true-positive floor (~0.4-0.6) and true-negative ceiling
(~0.1-0.2). Wires it into RoutingPolicy.for_embedding_backend("local"),
mirroring the existing "fake" profile. Azure is left on the production
calibration - it is the production embedder, not a stand-in for it.
Verified against the real demo: every query but the routing-ambiguous
control now routes directly with 0.4-0.7 margins instead of falling
into ask_disambiguation.
tests/unit/test_routing_policy.py (new) locks in which backend maps to
which profile and asserts the local profile is strictly looser than
production - otherwise the fix fixes nothing.
main squash-merged an equivalent of this branch's own "Harden configuration validation and move catalog data under the package" commit (PR #1, 1f68ec7) under a different hash, so git had no shared history to fast-forward through. Resolved by keeping this branch's side everywhere it is a strict superset of main's (flat-demo Makefile target and README sections, the enriched knowledge-base content, the dspy.Refine warn_on_type_mismatch fix, the router tie-break tests) - verified the pre-merge diffs of the two equivalent commits were byte-identical before doing so. No conflicts touched application logic outside the two files already conflicting on wording.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
boundary (fail closed); CI gained a format check and a build step; Redis
ports bind to localhost only. Catalog JSON moved from top-level
data/into
src/meridian/dataso it ships as package data.--flat-demo, the pre-fat/slim single-representation model, as aconcrete point of comparison for the fat/slim cost trade-off the README
already argued for. Deepened the demo knowledge base's prose with grounded
rationale (backoff/jitter, token-bucket, idempotency-key, error-budget
framing), consistent with the project's stated sourcing.
accident of sort stability + alphabetical intent ordering, not a
deliberate, documented policy. Now explicit (score → own threshold →
intent name) and covered by tests that construct a real tie.
local(real MiniLM) embedding backend. Without it, the local backend - the one
the README recommends for seeing "real semantic routing behaviour" -
flagged nearly every query ambiguous, because the production
negative_penaltyassumes a score separation MiniLM's general-purposespace doesn't produce. Derived empirically against the catalog's
confusable pairs and held-out paraphrases, not guessed.
Test plan
make check(ruff format --check, ruff check, mypy, pytest) - 88 testsgreen
make demo,acl-demo,structured-demo,fatslim-demo,flat-demoagainst the in-memory backend
MERIDIAN_BACKEND=redis(Redis Stack viadocker compose) - identical output, ACL guarantee holdsMERIDIAN_EMBEDDING_BACKEND=localdemo - confirmed every query nowroutes directly with a real margin instead of asking for
disambiguation