Skip to content

Add backward-compatible Catalog v2 beta clients #27

Description

@hanclintoclaw-pixel

Goal

Add opt-in Catalog v2 beta support to CollectorVision without changing existing Catalog v1 behavior. Catalog v1 NPZ files and hf:// references must continue to load exactly as they do today.

Catalog producer: HanClinto/CollectorVisionCatalog.

Current compatibility gap

Python Catalog.load() currently supports local/Hugging Face NPZ catalogs containing float32 embeddings plus card_ids and optional oracle_ids. Catalog.for_game() resolves v1 Hugging Face catalogs.

The web scanner already searches packed FP16 efficiently, but its manifest expects separate embeddings, card_ids, and optional secondary-ID arrays. Catalog v2 instead publishes a release index, per-catalog manifest, FP16 matrix, aligned gzip JSONL recognition records, optional metadata, and exact-base deltas.

No CollectorVision client currently discovers, downloads, validates, caches, or applies the v2 format.

Backward-compatible API direction

  • Keep Catalog.load(), Catalog.for_game(), NPZ, and hf:// behavior unchanged.
  • Add an explicit beta entry point such as Catalog.load_v2(...) / Catalog.for_game(..., catalog_version="v2-beta"); do not silently switch defaults during beta.
  • Preserve Catalog.search() returning (score, primary_id) tuples.
  • Expose richer row descriptors, secondary IDs, face index, catalog key/version, and optional metadata through additive attributes/APIs.
  • Continue requiring users to install exactly one ONNX Runtime backend; catalog support must not add an ONNX Runtime dependency.

Python client

  • Resolve an explicit beta release index and catalog key.
  • Verify index, manifest, and asset SHA-256/size before use.
  • Stream/decompress FP16 matrices and recognition JSONL into a local atomic cache.
  • Map primary_id.value to existing card_ids; map scryfall_oracle to oracle_ids where available.
  • Validate embedding model compatibility before search.
  • Keep full metadata opt-in and lazily loaded.
  • Apply a delta only for an exact installed base version; otherwise fetch the full snapshot.
  • Atomically commit updated cache state and recover safely from interruption.
  • Test local files, HTTP prerelease assets, offline cache, checksum failure, unsupported schema, stale-base fallback, and v1 regression behavior.

Browser client

  • Adapt the web worker to v2 index/manifest discovery while retaining the existing v1 bundle loader.
  • Keep FP16 packed in memory and preserve current lookup-table search.
  • Parse recognition records into compact aligned arrays and discard transient JSON objects; avoid retaining one large JS object per row.
  • Measure decompression/parsing peak memory, especially on iOS WebKit.
  • Cache immutable assets/version state in IndexedDB.
  • Apply exact-base deltas or fall back to a full snapshot.
  • Keep optional metadata out of the recognition worker unless requested.

Beta exit criteria

  • Existing Python and browser v1 tests remain green.
  • Python and browser load the same beta catalog and return identical primary IDs for test embeddings.
  • A real beta.1 → beta.2 delta is applied and compared with the full beta.2 snapshot.
  • Download size, steady-state memory, transient parsing memory, and update size are documented.
  • Stable v2 becomes a deliberate later default change, not part of beta support.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions