Split out of #183 by verify round 4 (PR #186). Blocked by #183.
Why this is one change and not two
#183 assigned every catalog row its true quantization and normalised the mlx parakeet size. Both are correct. Both rotate model_id keys — 19 of the 37 rows — and measurements.jsonl is append-only, so the stored measurements keep pointing at the old spellings.
Measured on the author's store, not estimated:
|
|
| stored measurements |
383 |
carrying default as their quantization segment |
344 (89%) |
| largest single key |
whisperkit|whisper|large-v3-turbo|default — 140 |
CommandCore calls store.seed(models: ModelGrid.rows) after every successful benchmark and BenchmarkStore.seed rewrites models.jsonl wholesale, so the first run under a re-keyed catalog replaces every key while the measurements keep the old ones. The projection's per-candidate key includes the quantization, and so does Router.aggregate — so each affected candidate would appear twice in the ranking pool under one displayed name, breaking the "one record per candidate" invariant that StoreProjection states four lines above the key that breaks it.
Three findings from round 4 all reduce to this, which is why they are one issue:
- the catalog re-key (CRITICAL)
default surviving on the size axis of mega-asr and qwen3-forcedaligner — neither upstream publishes a version name, so removing it needs a decision this change can make: drop the rows, relax the spec to allow a family-only reference row, or wait for upstream
- the router excluding incomplete identities — two spec deltas carry a SHALL for it, and implementing it today drops 89% of measurements out of ranking
Scope
Tests that move here from #183
Removed from idd/183-model-identity-audit because they assert this change's target state, and listed so they come back rather than evaporate:
| Test |
File |
The same parakeet under two runtimes is one identity |
ModelGridTests |
No catalog row states the removed placeholder as its quantization |
ModelGridTests |
Only the rows whose quantization nobody recorded are unknown |
ModelGridTests |
Each runtime states its quantization in the kind that fits it |
ModelGridTests |
A row nobody recorded a quantization for is excluded, and named |
ModelGridTests |
list-models never prints the removed placeholder |
CLITests |
The same model under two runtimes shares family and size |
CLITests |
Each row says which kind of quantization fact it carries |
CLITests |
No entry spells the removed placeholder in any field |
CLITests |
A legacy record carrying the placeholder is not marked comparable |
ModelRowCodecTests |
Every catalog row ships a well-formed quantization |
IdentityValidationTests |
#183 leaves two guards behind so the deferral is visible rather than absent:
ModelGridTests.The catalog still spells the placeholder — deferred, not forgotten — asserts the count is exactly 19. It must fail when this change lands.
CatalogKeyStabilityTests.Every catalog row produces the key already on disk — asserts zero rotated keys against the committed snapshot. This change replaces it with one that asserts the new keys and the re-encoded records agree.
What #183 still delivers without this
The type work: ModelID, Quantization, identity-keyed lookup with ambiguity reported rather than settled, identity-keyed memory estimates and ranking, StoreProjection with no vendor branch, and the structured CLI/MCP surface. That is what unblocks #184 and #185; none of it touches a stored key.
🤖 Generated with Claude Code
Split out of #183 by verify round 4 (PR #186). Blocked by #183.
Why this is one change and not two
#183 assigned every catalog row its true quantization and normalised the mlx parakeet size. Both are correct. Both rotate
model_idkeys — 19 of the 37 rows — andmeasurements.jsonlis append-only, so the stored measurements keep pointing at the old spellings.Measured on the author's store, not estimated:
defaultas their quantization segmentwhisperkit|whisper|large-v3-turbo|default— 140CommandCorecallsstore.seed(models: ModelGrid.rows)after every successful benchmark andBenchmarkStore.seedrewritesmodels.jsonlwholesale, so the first run under a re-keyed catalog replaces every key while the measurements keep the old ones. The projection's per-candidate key includes the quantization, and so doesRouter.aggregate— so each affected candidate would appear twice in the ranking pool under one displayed name, breaking the "one record per candidate" invariant thatStoreProjectionstates four lines above the key that breaks it.Three findings from round 4 all reduce to this, which is why they are one issue:
defaultsurviving on the size axis ofmega-asrandqwen3-forcedaligner— neither upstream publishes a version name, so removing it needs a decision this change can make: drop the rows, relax the spec to allow a family-only reference row, or wait for upstreamScope
notApplicable/named/deferred/unknown), removing the literal from the quantization axismlx-audio parakeetsize0.6b→0.6b-v3(its pin ismlx-community/parakeet-tdt-0.6b-v3)mega-asrandqwen3-forcedalignerQuantization.isComplete's refusal of the placeholder — deferred with the re-key because refusing it while 19 rows spell it excluded every whisperkit row from benchmarking (measured:enumerateCandidatesreturned an empty list)Tests that move here from #183
Removed from
idd/183-model-identity-auditbecause they assert this change's target state, and listed so they come back rather than evaporate:The same parakeet under two runtimes is one identityModelGridTestsNo catalog row states the removed placeholder as its quantizationModelGridTestsOnly the rows whose quantization nobody recorded are unknownModelGridTestsEach runtime states its quantization in the kind that fits itModelGridTestsA row nobody recorded a quantization for is excluded, and namedModelGridTestslist-models never prints the removed placeholderCLITestsThe same model under two runtimes shares family and sizeCLITestsEach row says which kind of quantization fact it carriesCLITestsNo entry spells the removed placeholder in any fieldCLITestsA legacy record carrying the placeholder is not marked comparableModelRowCodecTestsEvery catalog row ships a well-formed quantizationIdentityValidationTests#183 leaves two guards behind so the deferral is visible rather than absent:
ModelGridTests.The catalog still spells the placeholder — deferred, not forgotten— asserts the count is exactly 19. It must fail when this change lands.CatalogKeyStabilityTests.Every catalog row produces the key already on disk— asserts zero rotated keys against the committed snapshot. This change replaces it with one that asserts the new keys and the re-encoded records agree.What #183 still delivers without this
The type work:
ModelID,Quantization, identity-keyed lookup with ambiguity reported rather than settled, identity-keyed memory estimates and ranking,StoreProjectionwith no vendor branch, and the structured CLI/MCP surface. That is what unblocks #184 and #185; none of it touches a stored key.🤖 Generated with Claude Code