Problem
The web scanner currently publishes a monolithic float16 embedding catalog and aligned ID JSON. Any catalog update makes returning browsers download the entire changed catalog again, even when the update is mostly additive.
Proposal
Publish a small catalog manifest that references ordered, content-addressed catalog shards. Each shard should keep embeddings and their aligned ID metadata together. The browser should cache shards by content hash in IndexedDB, reuse unchanged shards, and fetch only new or replaced shards.
Requirements
- The manifest declares one atomic catalog revision so IDs and embeddings cannot be mixed across revisions.
- Preserve full-catalog search behavior after all referenced shards are available.
- Choose stable shard boundaries that maximize reuse for mostly additive catalog updates.
- Measure the existing catalog update pattern before selecting a shard size and strategy.
- Retain a simple first-load path; sharding is intended to reduce update downloads, not necessarily initial bytes.
Acceptance Criteria
- Exporter emits a versioned shard manifest and deterministic content-addressed shard paths.
- Worker loads, validates, and searches a complete sharded catalog.
- IndexedDB cache reuses unchanged shards across catalog revisions.
- Tests cover append-only and replacement-shard update scenarios.
- Deployment metadata records the sharded catalog revision and content hashes.
Problem
The web scanner currently publishes a monolithic float16 embedding catalog and aligned ID JSON. Any catalog update makes returning browsers download the entire changed catalog again, even when the update is mostly additive.
Proposal
Publish a small catalog manifest that references ordered, content-addressed catalog shards. Each shard should keep embeddings and their aligned ID metadata together. The browser should cache shards by content hash in IndexedDB, reuse unchanged shards, and fetch only new or replaced shards.
Requirements
Acceptance Criteria