Skip to content

Re-key the catalog and re-encode the 383 stored measurements, together #187

Description

@kiki830621

Split out of #183 by verify round 4 (PR #186). Blocked by #183.

Why this is one change and not two

#183 assigned every catalog row its true quantization and normalised the mlx parakeet size. Both are correct. Both rotate model_id keys — 19 of the 37 rows — and measurements.jsonl is append-only, so the stored measurements keep pointing at the old spellings.

Measured on the author's store, not estimated:

stored measurements 383
carrying default as their quantization segment 344 (89%)
largest single key whisperkit|whisper|large-v3-turbo|default — 140

CommandCore calls store.seed(models: ModelGrid.rows) after every successful benchmark and BenchmarkStore.seed rewrites models.jsonl wholesale, so the first run under a re-keyed catalog replaces every key while the measurements keep the old ones. The projection's per-candidate key includes the quantization, and so does Router.aggregate — so each affected candidate would appear twice in the ranking pool under one displayed name, breaking the "one record per candidate" invariant that StoreProjection states four lines above the key that breaks it.

Three findings from round 4 all reduce to this, which is why they are one issue:

  • the catalog re-key (CRITICAL)
  • default surviving on the size axis of mega-asr and qwen3-forcedaligner — neither upstream publishes a version name, so removing it needs a decision this change can make: drop the rows, relax the spec to allow a family-only reference row, or wait for upstream
  • the router excluding incomplete identities — two spec deltas carry a SHALL for it, and implementing it today drops 89% of measurements out of ranking

Scope

  • Re-encode the stored measurements onto the new keys, mapping what can be mapped and naming what cannot
  • Assign each catalog row its true quantization (notApplicable / named / deferred / unknown), removing the literal from the quantization axis
  • Normalise mlx-audio parakeet size 0.6b0.6b-v3 (its pin is mlx-community/parakeet-tdt-0.6b-v3)
  • Decide the size axis for mega-asr and qwen3-forcedaligner
  • Restore Quantization.isComplete's refusal of the placeholder — deferred with the re-key because refusing it while 19 rows spell it excluded every whisperkit row from benchmarking (measured: enumerateCandidates returned an empty list)
  • Implement the router exclusion + its named notes, and restore the two spec requirements moved out of 模型身分應是「模型名(提供者)」的隸屬關係,不是 provider-first 的 BackendID #183

Tests that move here from #183

Removed from idd/183-model-identity-audit because they assert this change's target state, and listed so they come back rather than evaporate:

Test File
The same parakeet under two runtimes is one identity ModelGridTests
No catalog row states the removed placeholder as its quantization ModelGridTests
Only the rows whose quantization nobody recorded are unknown ModelGridTests
Each runtime states its quantization in the kind that fits it ModelGridTests
A row nobody recorded a quantization for is excluded, and named ModelGridTests
list-models never prints the removed placeholder CLITests
The same model under two runtimes shares family and size CLITests
Each row says which kind of quantization fact it carries CLITests
No entry spells the removed placeholder in any field CLITests
A legacy record carrying the placeholder is not marked comparable ModelRowCodecTests
Every catalog row ships a well-formed quantization IdentityValidationTests

#183 leaves two guards behind so the deferral is visible rather than absent:

  • ModelGridTests.The catalog still spells the placeholder — deferred, not forgotten — asserts the count is exactly 19. It must fail when this change lands.
  • CatalogKeyStabilityTests.Every catalog row produces the key already on disk — asserts zero rotated keys against the committed snapshot. This change replaces it with one that asserts the new keys and the re-encoded records agree.

What #183 still delivers without this

The type work: ModelID, Quantization, identity-keyed lookup with ambiguity reported rather than settled, identity-keyed memory estimates and ranking, StoreProjection with no vendor branch, and the structured CLI/MCP surface. That is what unblocks #184 and #185; none of it touches a stored key.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions