Skip to content

Commit 2f36ee5

Browse files
aaltshulerclaude
andcommitted
docs(user): add task guides — hybrid search, cluster on S3, review workflow (Phase 3b)
Four new pages under docs/user/guides/, each a runnable, code-verified command sequence that composes the reference docs into a real workflow: - guides/hybrid-search.md — schema with a @embed vector + text body, load, then a query fusing bm25 and nearest with rrf. Notes that indexes are engine- maintained (no manual build step) and links embeddings.md for the provider env. - guides/cluster-on-s3.md — cluster.yaml with a storage: s3:// root, the validate→import→plan→apply flow, loading via the graph's storage URI, and config-free serving with `omnigraph-server --cluster s3://…`. - guides/review-workflow.md — load onto a branch with --from, inspect it with --branch reads / commit list, merge with --into, then delete + cleanup. - guides/index.md — the section landing page. Every command was checked against crates/omnigraph-cli/src/cli.rs (e.g. caught that `load` has no --cluster/--cluster-graph — those are storage-plane only — and used the positional storage URI instead). Wired into docs/user/index.md (new Guides section) and AGENTS.md's topic table. Verified: zero broken links; check-agents-md.sh green (61 links, 58 docs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 77dffda commit 2f36ee5

6 files changed

Lines changed: 286 additions & 0 deletions

File tree

AGENTS.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ Full diagram and concurrency model: [docs/dev/architecture.md](docs/dev/architec
101101
| Error taxonomy and result serialization | [docs/user/operations/errors.md](docs/user/operations/errors.md) |
102102
| Install (binary / Homebrew / source / channels) | [docs/user/install.md](docs/user/install.md) |
103103
| Deployment (binary / container / RustFS bootstrap / auth / build variants) | [docs/user/deployment.md](docs/user/deployment.md) |
104+
| Task guides (hybrid search, cluster on S3, review workflow) | [docs/user/guides/index.md](docs/user/guides/index.md) |
104105
| CI / release workflows | [docs/dev/ci.md](docs/dev/ci.md) |
105106
| Code ownership (CODEOWNERS source of truth, roles, regeneration) | [docs/dev/codeowners.md](docs/dev/codeowners.md) |
106107
| Branch protection policy (declarative, applied via `scripts/apply-branch-protection.sh`) | [docs/dev/branch-protection.md](docs/dev/branch-protection.md) |

docs/user/guides/cluster-on-s3.md

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Run a Cluster on S3
2+
3+
This guide takes a cluster from a local config directory to a server that boots
4+
**config-free from an object-storage bucket** — the bucket is the whole
5+
deployment artifact. For the full control-plane reference, see
6+
[operating a cluster](../clusters/index.md) and
7+
[cluster config](../clusters/config.md).
8+
9+
## 1. Declare the cluster
10+
11+
Lay out a config directory. The one S3-specific line is `storage:` — it puts the
12+
state ledger, catalog, and graph data on the bucket instead of in the folder:
13+
14+
```
15+
company-brain/
16+
├── cluster.yaml
17+
├── people.pg
18+
├── queries/
19+
│ └── people.gq
20+
└── base.policy.yaml
21+
```
22+
23+
```yaml
24+
# cluster.yaml
25+
version: 1
26+
storage: s3://my-bucket/clusters/company-brain # the deployment lives here
27+
metadata:
28+
name: company-brain
29+
graphs:
30+
knowledge:
31+
schema: people.pg
32+
queries: queries/
33+
policies:
34+
base:
35+
file: base.policy.yaml
36+
applies_to: [knowledge]
37+
```
38+
39+
Set the S3 credentials in the environment (for a non-AWS S3-compatible store such
40+
as MinIO or RustFS, also set `AWS_ENDPOINT_URL_S3`):
41+
42+
```bash
43+
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=us-east-1
44+
# export AWS_ENDPOINT_URL_S3=https://... # non-AWS S3-compatible stores
45+
```
46+
47+
## 2. Validate, plan, apply
48+
49+
`apply` is the only command that changes the world; `plan` previews it:
50+
51+
```bash
52+
omnigraph cluster validate --config company-brain # parse + typecheck
53+
omnigraph cluster import --config company-brain # create the state ledger
54+
omnigraph cluster plan --config company-brain # preview the diff
55+
omnigraph cluster apply --config company-brain # converge onto the bucket
56+
```
57+
58+
`apply` creates the graph at the derived root
59+
(`s3://my-bucket/clusters/company-brain/graphs/knowledge.omni`), applies its
60+
schema, and publishes the query and policy into the content-addressed catalog.
61+
`converged: true` means there is nothing left to do — re-running `apply` is always
62+
safe.
63+
64+
## 3. Load data
65+
66+
The control plane manages *definitions*; rows go through the normal data plane.
67+
Address the graph by its storage URI (the derived `graphs/<id>.omni` root):
68+
69+
```bash
70+
omnigraph load --data seed.jsonl --mode overwrite \
71+
s3://my-bucket/clusters/company-brain/graphs/knowledge.omni
72+
```
73+
74+
## 4. Serve config-free from the bucket
75+
76+
A serving host needs only the storage-root URI and credentials — no checkout of
77+
the config repo:
78+
79+
```bash
80+
OMNIGRAPH_SERVER_BEARER_TOKENS_JSON='{"act-reader":"s3cret"}' \
81+
omnigraph-server --cluster s3://my-bucket/clusters/company-brain --bind 0.0.0.0:8080
82+
```
83+
84+
The server boots from the **applied revision** recorded in the ledger — never from
85+
config that was merely written. Roll out a change by `apply`-ing again, then
86+
restarting replicas.
87+
88+
## 5. Maintain it
89+
90+
Storage maintenance runs out-of-band, addressed by cluster + graph name (it
91+
resolves the graph's storage URI from the served state):
92+
93+
```bash
94+
omnigraph optimize --cluster company-brain --cluster-graph knowledge
95+
omnigraph cleanup --cluster company-brain --cluster-graph knowledge --keep 10 --confirm
96+
```
97+
98+
See [maintenance](../operations/maintenance.md) for what each command does.

docs/user/guides/hybrid-search.md

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Hybrid Search End to End
2+
3+
This guide builds a small document graph and runs a **hybrid** query that fuses
4+
full-text (BM25) and vector (k-NN) rankings with Reciprocal Rank Fusion. You do
5+
not build indexes by hand — the engine maintains them; a freshly loaded row is
6+
searchable immediately.
7+
8+
See [search](../search/index.md) for the function reference and
9+
[embeddings](../search/embeddings.md) for the full provider/env matrix.
10+
11+
## 1. Schema
12+
13+
A document with a text body for full-text search and a vector for similarity.
14+
`@embed("body")` tells the engine to embed the `body` text into `embedding` at
15+
load time:
16+
17+
```
18+
node Document {
19+
title: String,
20+
body: String,
21+
embedding: Vector(768) @embed("body"),
22+
}
23+
```
24+
25+
```bash
26+
omnigraph init --schema schema.pg docs.omni
27+
```
28+
29+
## 2. Configure embeddings
30+
31+
Ingest-time embedding uses the engine's embedding client. Point it at your
32+
provider (see [embeddings](../search/embeddings.md) for every variable):
33+
34+
```bash
35+
export GEMINI_API_KEY=... # ingest-time document embeddings
36+
# For local experimentation without a provider, deterministic mock vectors:
37+
# export OMNIGRAPH_EMBEDDINGS_MOCK=1 NANOGRAPH_EMBEDDINGS_MOCK=1
38+
```
39+
40+
If you would rather supply vectors yourself, drop `@embed` and include the
41+
`embedding` array in each input record instead.
42+
43+
## 3. Load
44+
45+
```bash
46+
omnigraph load --data docs.jsonl --mode overwrite docs.omni
47+
```
48+
49+
Each row's `body` is embedded into `embedding` as it loads. The BM25 (full-text)
50+
and vector indexes are maintained by the engine — there is no separate build step.
51+
52+
## 4. Query — full-text, vector, then hybrid
53+
54+
Full-text only:
55+
56+
```gq
57+
query text_search($q: String) {
58+
match { $d: Document { } }
59+
return { $d.title, bm25($d.body, $q) as score }
60+
order { score desc }
61+
limit 10
62+
}
63+
```
64+
65+
Vector only (the query text is embedded at query time; `nearest` requires a
66+
`limit`):
67+
68+
```gq
69+
query vector_search($q: String) {
70+
match { $d: Document { } }
71+
return { $d.title, nearest($d.embedding, $q) as score }
72+
order { score desc }
73+
limit 10
74+
}
75+
```
76+
77+
Hybrid — fuse both rankings with `rrf`:
78+
79+
```gq
80+
query hybrid($q: String) {
81+
match { $d: Document { } }
82+
return {
83+
$d.title,
84+
rrf( nearest($d.embedding, $q), bm25($d.body, $q) ) as score
85+
}
86+
order { score desc }
87+
limit 10
88+
}
89+
```
90+
91+
Run it:
92+
93+
```bash
94+
omnigraph read --query queries.gq --name hybrid \
95+
--params '{"q":"trends in AI safety"}' --format table docs.omni
96+
```
97+
98+
`rrf` combines the two rankings without needing their score scales to match, so
99+
you get a single fused ordering from a lexical signal and a semantic one.

docs/user/guides/index.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Guides
2+
3+
Task-oriented walkthroughs that compose the building blocks from the reference
4+
docs into real workflows. Each one is a runnable sequence of commands.
5+
6+
- [Hybrid search end to end](hybrid-search.md) — combine full-text and vector
7+
search in one query.
8+
- [Run a cluster on S3](cluster-on-s3.md) — go from a config directory to a
9+
config-free server booting from a bucket.
10+
- [Branch-based review workflow](review-workflow.md) — stage data on a branch,
11+
review it, and merge.
12+
13+
New to OmniGraph? Start with the [quickstart](../quickstart.md) and
14+
[concepts](../concepts/index.md) first.
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Branch-Based Review Workflow
2+
3+
Branches let you stage changes off `main`, inspect them in isolation, and merge
4+
only once they look right — Git-style, atomic across the whole graph. This guide
5+
walks a typical "review an incoming batch before it hits main" flow.
6+
7+
See [branches & commits](../branching/index.md) and [merging](../branching/merge.md)
8+
for the underlying model.
9+
10+
## 1. Stage the batch on its own branch
11+
12+
Loading into a branch that does not exist is an error unless you pass `--from`,
13+
which forks it from a base first. So one command both forks the branch and loads
14+
into it:
15+
16+
```bash
17+
omnigraph load --data batch.jsonl --mode merge \
18+
--branch review/2026-04-25 --from main graph.omni
19+
```
20+
21+
(Equivalently, create the branch first with
22+
`omnigraph branch create review/2026-04-25 --from main graph.omni`, then `load`
23+
without `--from`.)
24+
25+
`main` is untouched — the batch lives only on `review/2026-04-25`.
26+
27+
## 2. Inspect the branch in isolation
28+
29+
Run any read query against the branch with `--branch`:
30+
31+
```bash
32+
omnigraph read --query checks.gq --name count_by_type \
33+
--branch review/2026-04-25 --format table graph.omni
34+
```
35+
36+
Compare it against `main` — list each branch's commits, or diff them:
37+
38+
```bash
39+
omnigraph branch list graph.omni
40+
omnigraph commit list --branch review/2026-04-25 graph.omni
41+
```
42+
43+
## 3. Merge when it looks right
44+
45+
```bash
46+
omnigraph branch merge review/2026-04-25 --into main graph.omni
47+
```
48+
49+
The merge is three-way and atomic. If both `main` and the branch changed the same
50+
data incompatibly, the merge fails with a structured list of conflicts and
51+
publishes nothing — resolve them and re-merge. See
52+
[merging](../branching/merge.md) for the conflict kinds.
53+
54+
## 4. Clean up
55+
56+
Once merged, delete the review branch:
57+
58+
```bash
59+
omnigraph branch delete review/2026-04-25 graph.omni
60+
```
61+
62+
Branch storage is reclaimed; if a transient error interrupts reclamation, the
63+
[`cleanup`](../operations/maintenance.md) command sweeps the leftovers later.

docs/user/index.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,17 @@ start with install, then follow the section that matches your task.
6565
| Understand graph layout and URI support | [concepts/storage.md](concepts/storage.md) |
6666
| Look up constants and tunables | [reference/constants.md](reference/constants.md) |
6767

68+
## Guides
69+
70+
Task-oriented walkthroughs that compose the building blocks above:
71+
72+
| Guide | Read |
73+
|---|---|
74+
| All guides | [guides/index.md](guides/index.md) |
75+
| Hybrid search end to end | [guides/hybrid-search.md](guides/hybrid-search.md) |
76+
| Run a cluster on S3 | [guides/cluster-on-s3.md](guides/cluster-on-s3.md) |
77+
| Branch-based review workflow | [guides/review-workflow.md](guides/review-workflow.md) |
78+
6879
## Releases
6980

7081
Release notes live in [releases/](../releases/). Use them for user-visible

0 commit comments

Comments
 (0)