Every cold load downloads the whole index — a SQLite/FTS5 index would retire shard-splitting and unlock full-text search

[petfold]

Labels: enhancement

---

loadIndex() fetches the manifest and then all shards it names
(src/lib/index-api.js:95-102), unions them in memory, and searches them by
linear scan (searchRepos/searchUsers, :162-173).

Concretely, on the producer side radicle-index-service/src/snapshot.ts:5-7
sizes v1 as one repos.json plus one users.json, with both staying "in the low
megabytes" at ~7k network repos. So this isn't hypothetical bloat — it's a
few MB fetched and parsed on every cold load, growing with the network.

src/lib/index-queries.js:16-18 already names the trajectory:

the index grows toward the full network (8k+ repos), so views must not
linear-scan it per item render

The memoized Maps there fix the per-render scan. What remains is that the cost
of the first paint scales with the size of the whole network, on every cold
load, for a user who typed one search term.

docs/STATE.md follow-up #5 tracks the two consequences as separate work —
shard splitting when repos.json/users.json outgrow single files, and
full-text search ("parked deliberately", pagefind-style).

Suggestion: publish the index as SQLite, query it lazily

SQLite's 4 KB default page equals a Swarm chunk, so a B-tree lookup over a
ranged reader touches a handful of pages regardless of file size. That collapses
both parked items:

• Shard splitting stops needing a design. The B-tree is the sharding, and
  it rebalances itself. manifest.shards.* iteration goes away.
• Full-text search comes from FTS5 rather than a bespoke inverted index.

swarmlite's js/ package is a browser
reader for exactly this — wa-sqlite over a Swarm VFS:

const db = await open(`${api}/bzz/${root}/index.db`);
const rows = await db.query(
  'SELECT rid, name, description FROM repos WHERE repos MATCH ? LIMIT 50', [q]);

Measured on its own 134.5 MB demo database: a point lookup fetches 5 pages /
20 KB; an FTS5 search fetched 12 pages. Warm repeats fetch nothing.

On the publishing side, js/src/publisher.js is a pure-JS Swarm publisher, so
radicle-index-service could emit and publish the database without leaving TS.

Costs, stated honestly

• 1.14 MB of wasm (vendor/wa-sqlite/dist/wa-sqlite-async.wasm) in a dApp
  served from Swarm. It should be lazy-loaded behind the directory route. The
  claim that it's cheaper than downloading the full index is only true past some
  index size — that crossover should be measured, not assumed.
• Requires HTTP Range end to end (js/src/SwarmVFS.js:77-84 fails loudly
  without a 206 + Content-Range). I checked both handlers rather than
  guessing: desktop forwards Range and returns the upstream response
  unmodified (freedom-browser/src/main/swarm/bzz-protocol.js:76-98,
  :396-401 — range is deliberately absent from the strip-list, and the body
  is streamed), so this works there today. iOS drops all request headers
  (freedom-browser-ios/…/BzzSchemeHandler.swift:178) and answers 200 with the
  whole file; filed separately. The vite /bzz/ dev proxy passes Range through
  to Bee, so development is unaffected.
• The platform that benefits most is the one that's broken. iOS has no
  Radicle node, so per docs/STATE.md canopy there is directory-and-profiles
  only — i.e. iOS is the platform whose entire experience is the index, and
  it's where lazy reads don't work yet. Worth sequencing against the iOS fix
  rather than shipping a desktop-only win.
• Changes the offline story. The TanStack persist layer currently caches the
  whole union in IndexedDB; a page-level cache is a different shape and needs
  its own decision.
• It moves a schema decision into the index protocol (docs/index-protocol.md),
  which is a bigger commitment than the reader change. Could ship additively —
  a db entry alongside shards — so readers migrate on their own schedule.