Harvest

Ask code, not documentation - Harvesting knowledge from project source using Agents.

Chat	Explore	Document

Harvest turns versioned source code repositories into a queryable knowledge graph. Point it at a list of Git repositories, let it ingest every tagged version, then ask natural-language questions through the chat interface or the HTTP API.

┌─────────────────────┐     ┌──────────────────────┐     ┌──────────────┐
│ knowledge-harvester │────▶│      Neo4j graph     │────▶│ knowledge-   │
│  (Rust CLI / daemon)│     │  functions, classes, │     │ server       │
│  git + tree-sitter  │     │  calls, imports, …   │     │ (HTTP + SSE) │
└──────────┬──────────┘     └──────────────────────┘     └──────┬───────┘
           │  document                                          │
           ▼                                           ┌────────▼───────┐
     Diataxis docs                                     │    web-ui      │
     (markdown files)─────────────────────────────────▶│  (Vite / JS)   │
                                                       └────────────────┘
                                                                │
                                                       ┌────────▼───────┐
                                                       │ harvest-agent  │
                                                       │  (Rust daemon) │
                                                       └────────────────┘

What it does

Harvester clones each repository, walks provided git refs, and parses the source with tree-sitter. Functions, classes, imports, call edges, and class relationships (inherits, implements, embeds, uses) are written to Neo4j. Each (repo, version) pair is an atomic unit — safe to interrupt and re-run. A separate document command generates Diataxis-structured documentation for any ingested version using an LLM.
Server exposes a REST + SSE API. A query triggers an agentic loop: the LLM calls graph tools (search, source retrieval, call-graph traversal, custom Cypher) until it has enough context, then returns a structured answer with inline [repo:version:file:line] citations. The server also serves the full symbol graph for any (repo, version) pair and static Diataxis documentation pages generated by the harvester. It includes full authentication (JWT + optional Google OAuth), project/group management, conversation history, and an overview pipeline that generates AI-powered environment status dashboards.
Web UI provides a streaming chat interface, an interactive symbol graph explorer, a documentation browser, a project and agent management interface, and per-project environment overviews.
Agent daemon (harvest-agent) runs on any machine, connects to the server via SSE, and executes bash commands on behalf of the agentic loop. The LLM can use list_agents and run_command tools to inspect and control connected machines.

Repository layout

harvest/
├── knowledge-harvester/    # Rust CLI — ingests repos into Neo4j
│   ├── src/
│   │   └── documentation/  # LLM-driven Diataxis doc generation pipeline
│   └── harvester.toml      # example config
├── knowledge-server/       # Rust HTTP server — answers questions
│   ├── src/
│   │   ├── agent/          # agentic loop, graph + machine + secret tools, prompts
│   │   ├── api/            # axum routes
│   │   ├── auth/           # JWT, Google OAuth, password hashing
│   │   ├── conversations/  # user conversation history
│   │   ├── llm/            # LLM provider adapters (Anthropic, OpenAI-compat) + retry
│   │   ├── machines/       # agent daemon registry and SSE handlers
│   │   ├── overview/       # environment status pipeline
│   │   └── projects/       # project and group management
│   └── server.toml         # example config
├── agent/                  # Rust daemon — executes commands on remote machines
│   └── src/
├── web-ui/                 # Vanilla JS SPA (Vite + Vitest)
│   ├── src/
│   └── tests/
├── documentation/
│   └── developer/          # architecture, harvester, server, dev-setup docs
├── docker-compose.yml      # Neo4j (Community 5) with APOC
└── Cargo.toml              # Cargo workspace

Quick start

1 — Start Neo4j

docker compose up -d
# Neo4j browser: http://localhost:7474  (neo4j / devpassword)

2 — Ingest some repositories

Edit knowledge-harvester/harvester.toml:

[neo4j]
uri      = "bolt://localhost:7687"
user     = "neo4j"
password = "devpassword"

[storage]
clone_root = "/tmp/harvest-repos"

[[repositories]]
name = "my-repo"
url  = "https://github.com/owner/my-repo.git"
# Optional: pin specific refs instead of ingesting all tags
# refs = ["v1.0", "v2.0", "main"]

Run the harvester:

cd knowledge-harvester
RUST_LOG=info cargo run -- --config harvester.toml run

To re-process all previously ingested versions:

RUST_LOG=info cargo run -- --config harvester.toml reingest

3 — Configure and start the server

Edit knowledge-server/server.toml:

[server]
host = "127.0.0.1"
port = 8080

[neo4j]
uri      = "bolt://localhost:7687"
user     = "neo4j"
password = "devpassword"

[auth]
jwt_secret = "change-me-in-production"

# Optional: Google OAuth
# [auth.google]
# client_id     = "..."
# client_secret = "..."
# redirect_uri  = "http://localhost:8080/auth/google/callback"

# Anthropic Claude
[llm]
provider       = "anthropic"
model          = "claude-sonnet-4-6"
api_key        = "sk-ant-..."
max_iterations = 20

# — or — OpenAI-compatible (Groq, Ollama, etc.)
# [llm]
# provider = "openai-compatible"
# base_url = "https://api.groq.com/openai/v1"
# api_key  = "gsk_..."
# model    = "llama-3.3-70b-versatile"

# Optional: expose the harvest-agent binary for download
# [agents]
# binary_path = "/usr/local/bin/harvest-agent"
# public_url  = "https://harvest.example.com"

cd knowledge-server
RUST_LOG=info cargo run -- --config server.toml
# Listening on 127.0.0.1:8080

The first user to register becomes an admin.

4 — Generate documentation (optional)

To generate Diataxis-structured documentation for an ingested version, add [llm] and [documentation] sections to harvester.toml:

[llm]
provider = "anthropic"
model    = "claude-sonnet-4-6"
api_key  = "sk-ant-..."

[documentation]
docs_dir = "/tmp/harvest-docs"

Then run the document command for a specific repo:version:

RUST_LOG=info cargo run -- --config harvester.toml document my-repo:v1.2.0

Point the server at the same docs_dir (via server.toml) to serve the generated pages through the web UI.

5 — Open the chat UI

cd web-ui
npm install
npm run dev
# Open http://localhost:5173

The Vite dev server proxies all API calls to localhost:8080 automatically.

HTTP API

Authentication

All routes except /health, /auth/*, and agent endpoints require a JWT session cookie obtained via POST /auth/login or POST /auth/register. The cookie is set automatically by the browser.

# Register (first user becomes admin)
curl -c cookies.txt -s http://localhost:8080/auth/register \
  -H 'Content-Type: application/json' \
  -d '{"email":"you@example.com","name":"You","password":"secret123"}'

# Login
curl -c cookies.txt -s http://localhost:8080/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"you@example.com","password":"secret123"}'

`POST /query`

Ask a question, get a complete JSON response.

curl -b cookies.txt -s http://localhost:8080/query \
  -H 'Content-Type: application/json' \
  -d '{"query": "How does the retry logic work?"}' | jq .

{
  "answer": "The retry logic lives in `llm/retry.rs` …",
  "sources": [
    { "repo": "my-repo", "version": "v1.2.0", "file": "src/llm/retry.rs", "line": 12 }
  ],
  "tool_calls_made": 4
}

`POST /query/stream`

Same payload, streams Server-Sent Events so you can display tool calls as they happen:

Event	Payload
`tool_call`	`{type, name, input}`
`tool_result`	`{type, name, preview}`
`done`	`{type, answer, sources, tool_calls_made}`
`error`	`{type, message}`

`GET /repositories`

List all ingested repositories and their available versions.

`GET /graph/:repo/:version`

Return the full symbol graph for a (repo, version) pair as JSON — nodes (functions and classes) and edges (calls, contains, inherits, implements, embeds, uses). Results are served from an in-memory cache pre-warmed at startup. Large graphs are truncated to 1 500 nodes / 6 000 edges with a "truncated": true flag.

`GET /graph/:repo/:version/source?file=...&name=...`

Fetch the source text, signature, and line range for a single symbol.

`GET /docs/:repo/:version`

Return the documentation index (JSON) listing all generated pages organised by Diataxis section.

`GET /docs/:repo/:version/:section/:filename`

Serve a single documentation page as text/markdown.

`GET /health`

Returns {"status": "ok"}.

Projects & Conversations

GET    /groups                              — list groups the user belongs to
GET    /projects                            — list accessible projects
POST   /projects                            — create a project
GET    /projects/:pid                       — get project details
PUT    /projects/:pid                       — update project name/description
DELETE /projects/:pid                       — delete project (admin or creator)
GET    /projects/:pid/conversations         — list conversations
POST   /projects/:pid/conversations         — create a conversation
GET    /projects/:pid/conversations/:cid    — get conversation with messages
PUT    /projects/:pid/conversations/:cid    — update title and messages
DELETE /projects/:pid/conversations/:cid   — delete conversation
POST   /projects/:pid/query                 — non-streaming query with history
POST   /projects/:pid/query/stream          — streaming query (fire-and-forget; events via SSE)
GET    /projects/:pid/events                — SSE channel for real-time collaboration events

Secrets

GET    /projects/:pid/secrets               — list secret names (not values)
POST   /projects/:pid/secrets               — upsert a secret by name
DELETE /projects/:pid/secrets/:name         — delete a secret

The LLM tools list_secrets, get_secret, and save_secret give the agent controlled access to the project's secret store.

Overview

GET  /projects/:pid/overview            — get current environment status HTML + metadata
GET  /projects/:pid/overview/events     — SSE stream for in-progress generation
POST /projects/:pid/overview/regenerate — trigger a new overview generation

Agent Daemon

GET  /agents/:pid/install.sh                      — generate install script for a project
GET  /agents/binary/harvest-agent                 — download the agent binary
GET  /projects/:pid/agents                        — list connected agents
DELETE /projects/:pid/agents/:aid                  — remove an agent (deletes the LXD container too, if it has one)
POST /projects/:pid/agents/:aid/execute            — run a command on an agent
POST /projects/:pid/agents/rotate-install-token    — rotate the install token
GET  /projects/:pid/agents/flavors                 — list LXD container sizes (requires [lxd] config)
POST /projects/:pid/agents/lxd                     — provision a Harvest-managed agent on LXD (requires [lxd] config)

Agents can be added two ways: install the daemon on a machine you already have, or — if the server has an [lxd] section configured — let Harvest provision and manage a container for you on an LXD cluster. The web UI's Agents page offers both options once LXD is configured; see server.md for the full provisioning flow.

Admin

GET /admin/users                 — list all users (admin only)
PUT /admin/users/:id/role        — set a user's role (admin/regular)
PUT /admin/users/:id/groups      — set a user's group memberships
GET /admin/groups                — list all groups
POST /admin/groups               — create a group
DELETE /admin/groups/:id         — delete a group

Graph query tools

The agent has access to these Neo4j-backed tools:

Tool	Description
`list_repositories`	All repos and their ingested versions
`search_symbols`	Full-text search for functions/classes by name
`get_symbol_source`	Full source text of a specific function or class
`get_file_symbols`	All symbols defined in a file
`find_callers`	Functions that call a given function
`find_callees`	Functions called by a given function
`get_imports`	Import declarations for a file
`compare_symbol_across_versions`	Source diff for a symbol between two versions
`run_cypher`	Arbitrary read-only Cypher for custom traversals
`list_agents`	List connected agent machines in the project
`run_command`	Execute a bash command on a connected agent
`list_secrets`	List secret names stored for the project
`get_secret`	Retrieve a secret value by name
`save_secret`	Store or update a secret value

Web UI features

Chat

Streaming tool calls — each tool invocation appears as a collapsible step timeline with an AI-generated plain-English description, tool name, raw inputs, and result preview
Inline symbol graphs — answers that reference specific symbols include a mini interactive graph showing that symbol's relationships
Markdown answers — rendered with syntax-highlighted code blocks (Atom One Dark) and copy-to-clipboard buttons
Source citations — inline [repo:version:file:line] markers become amber chips that link directly to the source file; a sources panel lists them all
Attachments — attach images and PDFs to your query; the LLM receives them as vision/document input

Graph explorer

Interactive symbol graph — browse the full call and relationship graph for any (repo, version) pair, rendered with Cytoscape.js and an off-thread fcose layout
Symbol search — highlight matching nodes instantly; AI search mode finds semantically related nodes
Source panel — click any node to see its signature and full source inline

Documentation

Diataxis browser — read AI-generated documentation organised into Tutorials, How-to Guides, Explanations, and Reference sections for any ingested version

Projects

Project workspaces — group conversations by project; share a project with your team via group membership
Real-time collaboration — presence indicators, typing lock, and live event streaming via SSE so multiple users can see what is happening
Conversation history — all turns are persisted server-side with automatic compaction when histories grow large

Agents

Remote agent management — install harvest-agent on any machine with a one-liner; manage connected agents from the web UI
LLM-driven automation — the project agent can run bash commands on connected machines and store discovered credentials in the project secret store

Overview

Environment status dashboard — AI-generated HTML status card summarising the project's infrastructure, based on conversation history and real-time agent tool calls

Shell

Dark / light / auto theme — toggle in the sidebar; persists across reloads; auto follows the OS setting with no flash on reload
Responsive navigation — Vanilla Framework application shell with collapsible sidebar for mobile

Technology stack

Concern	Choice
Harvester language	Rust
Server language	Rust
HTTP framework	axum
Code parsing	tree-sitter
Graph database	Neo4j 5 Community
Neo4j Rust driver	neo4rs
LLM providers	Claude (Anthropic) · OpenAI-compatible
Streaming	Server-Sent Events (axum SSE)
Authentication	JWT (cookie) + optional Google OAuth 2.0
Web UI build	Vite
Web UI tests	Vitest (jsdom)
Graph rendering	Cytoscape.js + fcose layout
CSS framework	Canonical Vanilla Framework
Async runtime	tokio
Configuration	TOML

Running tests

# Rust unit + integration tests (no Docker needed)
cargo test

# Rust Docker-gated tests (Neo4j testcontainers)
cargo test -- --include-ignored

# Web UI tests
cd web-ui && npm test

Documentation

Detailed documentation lives under documentation/developer/:

architecture.md — system design and component overview
harvester.md — pipeline, graph schema, tree-sitter integration
server.md — API reference, LLM provider config, agentic loop
dev-setup.md — step-by-step local development setup
web-ui/README.md — web UI architecture, scripts, and test coverage

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 272 Commits
agent		agent
documentation		documentation
knowledge-harvester		knowledge-harvester
knowledge-server		knowledge-server
snap		snap
web-ui		web-ui
.dockerignore		.dockerignore
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
rust-toolchain.toml		rust-toolchain.toml

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Harvest

What it does

Repository layout

Quick start

1 — Start Neo4j

2 — Ingest some repositories

3 — Configure and start the server

4 — Generate documentation (optional)

5 — Open the chat UI

HTTP API

Authentication

POST /query

POST /query/stream

GET /repositories

GET /graph/:repo/:version

GET /graph/:repo/:version/source?file=...&name=...

GET /docs/:repo/:version

GET /docs/:repo/:version/:section/:filename

GET /health

Projects & Conversations

Secrets

Overview

Agent Daemon

Admin

Graph query tools

Web UI features

Technology stack

Running tests

Documentation

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

`POST /query`

`POST /query/stream`

`GET /repositories`

`GET /graph/:repo/:version`

`GET /graph/:repo/:version/source?file=...&name=...`

`GET /docs/:repo/:version`

`GET /docs/:repo/:version/:section/:filename`

`GET /health`

Packages