A decentralized crawl → extract → index/graph engine that stores the index, not the pages — search/RAG and retail-radar as verticals of one engine, on the KOTVA substrate. Vuna is Swahili for to harvest / reap.
Design · Architecture · Viability · Product page
The desktop app: local + peer results with source chips, and your node's dashboard. All data shown is mock — the query/node wiring is Wave 2 (see status).
v0 preview — honest about what's real. The substrate contract and four of the six stage crates are implemented and tested (63 tests green); the desktop app builds and runs on mock data. It does not yet crawl-to-query end to end.
| Component | State |
|---|---|
vuna-core — frozen contract (types, seam traits, retail quorum reconciler) |
✅ done, tested |
vuna-crawl — polite fetch (robots.txt, per-host rate-limit, body cap) |
✅ done (24 tests) |
vuna-extract — web (chunks+links) + retail (JSON-LD/OG) extractors |
✅ done (13 tests) |
vuna-index — tantivy BM25 + per-space HNSW vectors + link graph |
✅ done (9 tests) |
vuna-frontier — distributed URL lists, dedup, DHT assignment |
✅ done (12 tests) |
vuna-query — SEARCH read path (local-first + fan-out + Min-PPR) |
🚧 Wave 2 stub |
vuna-node — daemon + kotva-core binding |
🚧 Wave 2 stub |
app/ — Tauri v2 desktop node (React) |
✅ builds, mock data |
A network of volunteer nodes crawls a distributed, subscribable list of URLs, runs pluggable extractors over each page, and contributes the result — keyword postings, vector embeddings, link/knowledge-graph edges, a snippet, and a pointer back to the live URL — into a shared index replicated K× across the network. Pages are never archived, which collapses storage from petabytes (datacenters) to ~2 GB/node (volunteers) for a billion-page index. No token, no page archive, no new crypto — it rides KOTVA identity / PUB / DHT / SEARCH.
- Embedding models — each is an independent, parallel vector space. The keyword index and graph are model-agnostic and shared. New model → new space, adopted by opt-in, never a fleet-wide flag-day. Shards keep chunk text, so re-embedding is local recompute, not a re-crawl. (This is what makes it future-proof instead of a lock-in trap.)
- Verticals — each is an extractor.
webemits chunks+links (search/RAG);retailemits price/stock observations (radar). Same crawl/frontier/distribution/query underneath. - A node serving the default space + one URL list is already a complete participant.
Your node, honestly surfaced: spaces served, lists subscribed, docs indexed, storage used, and a query-visibility disclosure (no hidden telemetry).
distributed URL lists ─▶ crawl ─▶ extract ─▶ index (keyword + vectors/space + graph) ─▶ query
(vuna-frontier) (vuna-crawl)(vuna-extract) (vuna-index) (vuna-query)
│ │
└───────────────────── KOTVA: identity · PUB · DHT · SEARCH ───────────────────────┘
(vuna-node binds kotva-core)
The index is derived, rebuildable, never authoritative (KOTVA SEARCH SRCH-2): on any
disagreement, the author's signed content wins. For the retail vertical — where the store is
a non-participant that signs nothing — consensus of K distinct, anchored observers is
the only ground truth (vuna-core::quorum, one observer can't stuff the ballot).
The workspace compiles offline: vuna-core has minimal deps and defines the frozen
contract (types + seam traits); heavy deps (tantivy, HNSW, reqwest, embedding runtimes)
live only in the crate that needs them. vuna-node is the sole kotva-core dependent,
pinned by tag. Full write-up in docs/02-architecture.md.
Storing the index (not pages), ~6 KB/page: keyword ~2 KB · embeddings (int8) ~2.5 KB · graph ~0.3 KB · metadata ~0.7 KB.
| Corpus | 1 copy | ×3 redundant / 10k nodes |
|---|---|---|
| 1 B pages | ~6 TB | ~1.8 GB / node |
| 10 B pages (Google-ish) | ~60 TB | ~18 GB / node |
Compute (embedding), not storage, is the recurring cost — and it's embarrassingly parallel.
The honest trade-offs (compute vs. disk volunteers, Sybil at small scale, freshness) are in
docs/00-viability.md — read it before believing the pitch.
# engine (workspace)
cargo test --workspace # 63 tests green
cargo build --workspace
# desktop app (mock data today)
cd app && npm install && npm run build
cargo tauri dev # or: npm run tauri dev- Wave 2 —
vuna-query(local-first fan-out + Min-PPR ranking) andvuna-node(crawl→extract→index→publish loop,kotva-corebinding) to close the end-to-end path. - Declarative adapters — the long tail of sites as
adapters/*.toml, not Rust. - Roadmap app — an opt-in browser extension contributing URLs-you-visit to the frontier (Mwmbl's proven model), off by default.
MIT OR Apache-2.0.