Ask your documents. A retrieval-grounded assistant for Uzbek-language corpora — answers with citations, not hallucinations.
Zehn lets an organisation upload its own documents — regulations, laws, policies, manuals — and get grounded, cited answers to questions asked in Uzbek. Every answer points back to the exact source passage it came from. If the corpus doesn't contain the answer, Zehn says "Ma'lumot topilmadi" instead of inventing one.
It is not a ChatGPT wrapper. The engineering is in the retrieval — and it is measured, not guessed: a built-in evaluation harness scores every retrieval strategy on a labelled Uzbek question set.
Uzbek is agglutinative: kitob → kitobda → kitoblarimizdan are all "book". Plain keyword search misses inflected forms; generic embeddings are weak on Uzbek. Zehn's answer is a hybrid retriever: PostgreSQL FTS over Uzbek-normalised text (Cyrillic→Latin transliteration, apostrophe unification, light suffix stemming) fused with pgvector similarity via Reciprocal Rank Fusion — plus an eval harness that proves each piece earns its place.
Morphology ablation — the same questions asked with different Uzbek suffix chains (ta'tillarining, tovarlarni almashtirishlarga…), section-level match, k=3, 10 queries:
| Keyword search over… | hit@3 | MRR |
|---|---|---|
| raw text (no Uzbek handling) | 0.80 | 0.750 |
| Uzbek-normalised text (Zehn) | 1.00 | 0.900 |
Retrieval strategy comparison — full labelled set (36 items, 28 answerable, 2 corpora), chunk-level (section) matching, k=3, run with the keyless fallback providers¹:
| Mode | recall@3 (adolat) | MRR (adolat) | recall@3 (navroz) | MRR (navroz) |
|---|---|---|---|---|
| hybrid (RRF) | 1.000 | 1.000 | 0.929 | 0.857 |
| vector-only | 1.000 | 1.000 | 0.929 | 0.810 |
| keyword-only (normalised) | 1.000 | 0.929 | 1.000 | 0.881 |
Both leg types saturate on a small corpus, but the pattern is the point: hybrid is never worse than vector-only and lifts MRR where they differ (+0.047 on navroz) — and the harness re-measures this claim on every retrieval change. Answer-side metrics on the same runs: faithfulness 1.00 (every claim sentence carries a citation whose passage supports it), false-refusal rate 0.00.
¹ These runs use the deterministic keyless fallbacks (
EMBED_PROVIDER=hash,LLM_PROVIDER=echo) so the whole pipeline — including this table — reproduces with zero API keys. The hash embedding is char-ngram-based (itself lexical), which narrows the hybrid-vs-vector gap; plug in a real OpenAI-compatible or local model and re-runpnpm evalto fill this table with semantic-model numbers. Answercorrectnessunderechois a token-F1 lower bound (~0.37) — the echo provider extracts, it does not generate.
upload ─▶ extract ─▶ clean ─▶ chunk ─▶ embed ─▶ store (pgvector + tsvector)
(PDF/DOCX/TXT) (uz unicode/OCR) (structure-aware) (cached+batched)
│
question ─▶ embed + normalise ─▶ hybrid retrieve ─▶ rerank ─▶ build prompt ─▶ LLM (SSE stream)
(vector ∪ FTS, (opt. LLM) │
RRF fusion) answer + citations ─▶ user
│
eval harness: recall@k · MRR · faithfulness · correctness · refusal accuracy
- Ingestion runs in a BullMQ worker — upload returns instantly, the UI shows live extract→chunk→embed progress per document.
- Chunking is structure-aware: markdown/legal headings (
12-modda.,3-bob.) and paragraphs, ~500–800 tokens with overlap, each chunk carrying{doc, section, page}for citations. - Grounded answering: the prompt forces
[n]citations and refusal when unsupported; the stream is SSE; citations map back to exact chunks and open the source passage in the UI. - Multi-tenant: orgs, users (admin/member), per-user chat history. Isolation is enforced at
the data layer — every retrieval SQL filters
org_id. Verified: an Adolat user asking a university question retrieves only Adolat chunks and gets a refusal. - Providers are pluggable (
EmbeddingProvider/LLMProvider/EmailProvider): OpenAI-compatible endpoints (OpenAI, Ollama, vLLM…) or keyless deterministic fallbacks. Embeddings are cached (never embed the same text twice) and batched. Per-org model override and monthly token cost cap included.
docker compose up --build
# → web http://localhost:3000 · api http://localhost:4000That's it — migrations run, the seed loads two real Uzbek corpora with labelled eval sets (idempotent), and everything works without any API key via the fallback providers.
Demo logins (email + OTP; in dev the code is printed to the API console and returned in the response, so the login screen shows it):
| Role | Org | |
|---|---|---|
admin@navroz.demo |
admin | Navro'z Universiteti (university regulations) |
talaba@navroz.demo |
member | Navro'z Universiteti |
admin@adolat.demo |
admin | Adolat Huquq Byurosi (labor & consumer law) |
yurist@adolat.demo |
member | Adolat Huquq Byurosi |
Try: ask "Yillik asosiy mehnat ta'tili necha kalendar kun?" as an Adolat user (cited, streamed answer), then "Bitkoin narxi qancha?" (clean refusal). Run the eval from the Sifat dashboard ("Baholashni boshlash") or the CLI and watch the hybrid-vs-vector chart fill with real numbers.
pnpm install
docker compose up -d db redis # pgvector on :5436, redis on :6382
cp .env.example .env
pnpm seed # migrate + ingest corpora + eval sets
pnpm dev # api :4000 + web :3000 (turbo)
pnpm eval -- --modes hybrid,vector --k 3 # print the metrics table
pnpm -r typecheck && pnpm -r lintThe fallbacks exist so the demo runs keylessly — real answer quality needs a real model. In .env:
# OpenAI
EMBED_PROVIDER=openai LLM_PROVIDER=openai
OPENAI_BASE_URL=https://api.openai.com/v1 OPENAI_API_KEY=sk-...
EMBED_MODEL=text-embedding-3-small EMBED_DIM=1536 LLM_MODEL=gpt-4o-mini
# or local Ollama (also keyless, but real models)
OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_API_KEY=ollama
EMBED_MODEL=nomic-embed-text EMBED_DIM=768 LLM_MODEL=llama3.1Changing EMBED_DIM changes the pgvector column: reset the DB (docker compose down -v) and
re-seed. With an OpenAI provider the eval harness automatically upgrades to LLM-as-judge for
correctness (rubric: 1.0 complete/exact · 0.7 correct but incomplete · 0.4 partially correct ·
0.0 wrong/hallucinated) and adds an LLM support-check to faithfulness.
seed/eval/*.json holds 36 labelled items {question, expected_source{doc, section}, expected_answer, answerable} — including deliberately unanswerable questions to test refusal.
Per run, per mode, the harness measures:
- recall@k / MRR — did the chunk actually containing the expected section make the top-k? (doc-level matching is trivially satisfied on a small corpus and would hide differences)
- faithfulness — share of claim sentences whose citation's passage lexically supports them (plus LLM judge when available)
- correctness — token-F1 vs expected answer (or LLM-graded with the rubric above); for unanswerable items: 1 iff refused
- refusal accuracy / false-refusal rate — the honesty metrics
Every run is stored (eval_runs/eval_results) and rendered live in the Sifat dashboard
("The Instrument") — hybrid vs vector side by side, per-question breakdown.
pnpm workspaces + Turborepo · NestJS API + BullMQ ingestion workers · Next.js 14 (App
Router) + Tailwind (design system: "Samarkand lapis" scholarly precision, UI in Uzbek) ·
PostgreSQL + pgvector (HNSW) + tsvector FTS — no separate vector DB · Redis · SSE streaming ·
email+OTP auth (pluggable delivery: console in dev, SMTP in prod).
apps/api NestJS: auth, documents, ingestion worker, retrieval, ask (SSE), eval, org settings
apps/web Next.js: landing, kirish (OTP), savol (ask+stream), suhbatlar, hujjatlar, sifat, sozlamalar
seed/ real Uzbek corpora (2 orgs × 4 docs) + labelled eval sets (2 × 18 items)
- Secrets live in
.env(gitignored);.env.exampleis the template. - JWT sessions (7d) from email+OTP (hashed codes, 10-min TTL, rate-limited, 5 attempts).
- Every corpus/conversation/eval query is
org_id-scoped at the SQL layer; documents API is admin-gated for writes. No org ever retrieves another org's chunks — this is asserted by the isolation test and by construction inretrieval.service.ts.
MIT.