Tarkka

Tarkka is an open, agent-first research infrastructure platform for discovering, ingesting, normalizing, organizing, and serving evidence-grounded research to humans and AI agents.

License and releases

Tarkka is licensed under Apache-2.0. The license covers Tarkka software and project-authored documentation; it does not grant rights to research content acquired, processed, or referenced by Tarkka. Releases and public package publication are maintainer-controlled and will be announced through tagged GitHub releases. See docs/RELEASE_POLICY.md for the supported channel, versioning, and release-artifact policy; Tarkka is not currently published on PyPI.

The core is intentionally usable without an LLM, hosted service, or mandatory external API. External research providers, document parsers, databases, and future model providers live behind replaceable contracts.

Run the read-only HTTP API

Install the optional API host, then start the existing /v1 ASGI/OpenAPI surface on loopback:

python -m pip install 'tarkka[api]'
tarkka-api

The default listener is 127.0.0.1:8000; discover its bounded operations at http://127.0.0.1:8000/openapi.json. Passing a non-loopback --host is explicit. This initial server exposes only the documented read-only API; authentication, writes, jobs, and cumulative wallet lifecycle are not remotely exposed.

Install a release

Download the wheel and SHA256SUMS from the v0.1.0 release. After checking the wheel's SHA-256 against its entry in the manifest, install it in a virtual environment. From the download directory, with Python 3.11 or newer and a POSIX shell:

python -m venv .venv
. .venv/bin/activate
python -m pip install ./tarkka-0.1.0-py3-none-any.whl
tarkka --help

The expanded help and --replay-ready shortcut below are available in the current source checkout and will ship in the next release. With v0.1.0, use --schema-version 3 instead. The offline walkthrough includes the source fixture and checkout commands; a wheel installation does not include the examples/ directory.

Prove a research result in five minutes

Tarkka is designed to make research state inspectable and replayable rather than hide evidence behind an answer-generation interface. The repository includes a completely local walkthrough that uses a project-authored source fixture and the real CLI to demonstrate:

preserved source
  -> normalized Document
  -> Claim + exact Evidence
  -> `tarkka why` provenance
  -> schema-v3 proof bundle
  -> offline verification
  -> exact deterministic replay

The walkthrough needs no model, provider credentials, PostgreSQL server, or network access. From a checkout with Python 3.11 or newer, it can run directly through PYTHONPATH=src python -m tarkka without an installation step. See docs/QUICKSTART_PROOF_REPLAY.md.

What works today

Local research ingestion

tarkka ingest ./notes.md
tarkka inspect <document-id>
tarkka read <document-id> --section 0

# Extract deterministic sentence-level claims with exact passage evidence.
tarkka extract claims <document-id> --extractor rule
tarkka claims list <document-id>
tarkka claims show <claim-id>

# Walk a Claim back through extraction, exact Evidence, Document, and Artifact provenance.
tarkka why <claim-id>

# Create a Frozen workspace from a YAML manifest, then ingest a local source.
tarkka workspace init examples/mlb-research.yaml
tarkka workspace show <workspace-id>
tarkka workspace run <workspace-id> --source examples/proof-replay-demo.txt
tarkka challenge <claim-id>
tarkka contradictions list <workspace-id>
tarkka encyclopedia compile <workspace-id> --allow-redistribution
tarkka encyclopedia show <article-id>
tarkka encyclopedia diff <edition-a> <edition-b>
tarkka library show
tarkka library show <library-id>
tarkka library documents --library <library-id> --limit 20
tarkka library claims --library <library-id> --limit 20
tarkka library works --library <library-id> --limit 20

# Human-readable claim receipt and a stapled document brief (same evidence, no implied support).
tarkka claims receipt <claim-id>
tarkka claims receipt <claim-id> --format json
tarkka documents brief <document-id>
tarkka documents brief <document-id> --format html

Receipts and briefs need the same local JSON or PostgreSQL document backend as tarkka why. Markdown is the default human view; json is the agent view; html is the same fields as a page. Missing claims/documents, catalog errors, and oversize brief pages fail closed with exit status 2 and do not invent support. Re-running against unchanged persisted state is deterministic (receipt-v1 / brief-v1). Agents discover the same operations as research.claims.receipt and research.documents.brief. See docs/QUICKSTART_PROOF_REPLAY.md.

# Export, independently verify, and replay portable research state.
tarkka bundle create <document-id> --replay-ready --output research.tarkka
tarkka bundle verify research.tarkka
tarkka replay research.tarkka

# For JATS, LaTeX, EPUB, and semantic HTML: inspect preserved citations progressively.
tarkka citations list <document-id> --limit 20
tarkka citations show <document-id> <reference-id>
tarkka citations resolve <document-id>
# Traverse only locally persisted citation relations; this does not fetch sources.
tarkka citations traverse <work-id> --max-depth 1 --max-works 50

# Inspect preserved supplements, datasets, software, and alternate representations.
tarkka resources list <document-id> --limit 20
tarkka resources show <document-id> <resource-link-id>

# Record and inspect reviewable claim-to-evidence assessments.
tarkka verify record <claim-id> --kind supports --evidence <evidence-id> \
  --verifier human-review --verifier-version 1 --confidence 0.9
tarkka verify list <claim-id>
tarkka verify show <relation-id>

# Start agent/tool discovery with a small operation index, then load one schema.
tarkka capabilities list
tarkka capabilities show research.verify.candidates

# Expand normalized source content progressively, from a manifest to one section.
tarkka documents manifest <document-id>
tarkka documents sections <document-id> --limit 20
tarkka documents section <document-id> <section-id>
tarkka documents package <document-id> --section <section-id> --section <section-id>
tarkka documents package <document-id> --section <section-id> --save
tarkka documents saved-package <context-package-id>

# Explicitly derive and search one local lexical projection. Add --workspace
# for the durable, workspace/library-scoped job path; MCP exposes the same
# bounded read-only search operation but never creates indexes.
tarkka retrieval index <document-id> \
  --derivation-version whole-passage-v1 \
  --configuration-fingerprint whole-passage-v1 \
  --workspace <workspace-id>
tarkka retrieval search <document-id> "evidence provenance" \
  --derivation-version whole-passage-v1 \
  --configuration-fingerprint whole-passage-v1 \
  --limit 10

Tarkka stores immutable source artifacts by SHA-256, records acquisition provenance, normalizes content into Document -> Section -> Passage, and exposes compact manifests before full content. Native-structure parsers also preserve bibliography entries, inline citations, and exact contexts; the citation CLI lists compact references before expanding a single reference's source text/context. citations resolve performs exact identifier resolution and creates a native cites relation only when the citing Work is explicit or uniquely linked to the Document. They also preserve source-observed resource relationships; the resources CLI follows the same compact-list then explicit-detail pattern without fetching or identity-resolving a target. Local lexical retrieval is an explicit, versioned derived projection over canonical passages. Each hit retains its stable segment ID and exact passage-local source spans; it never replaces the Document, Evidence, or citation boundary. Search requires the exact derivation/configuration pair that was indexed and never falls back to a different projection. A workspace-scoped index request validates that the document belongs to both the workspace and its library, records a resumable durable job, and reuses only the checkpointed exact projection. The bare index command remains available for compatibility with existing local projections. Verification assessments are separately auditable and expand back to their exact source evidence.

Install the optional Docling integration for richer formats such as PDF, DOCX, PPTX, HTML, and images:

python -m pip install -e '.[docling]'
tarkka ingest ./paper.pdf

Docling is an adapter, not a core dependency.

Scholarly discovery

# Default AUTO policy (currently prefers OpenAlex for broad discovery)
tarkka discover "machine learning MLB game outcome prediction"

# One provider
tarkka discover "pitcher fatigue" --provider semantic-scholar

# Selected providers
tarkka discover "baseball forecasting" \
  --provider openalex \
  --provider crossref

# Exhaustive enabled-provider fan-out
tarkka discover "baseball forecasting" --provider all

Current scholarly adapters:

• OpenAlex
• Crossref
• Semantic Scholar

Discovery supports provider selection, bounded concurrent fan-out, retries/rate-limit handling, provider-specific continuation cursors, DOI-first deduplication, and reproducible SearchSnapshots. Providers remain independent adapters; cross-provider identity and enrichment happen in application services rather than providers calling one another.

Work and document persistence use local JSON by default and therefore need no database service. To opt into PostgreSQL, install the optional extra and select the backend explicitly; a database URL alone never changes the default. On a new database, apply the current schema migrations first:

uv sync --extra postgres
export TARKKA_DATABASE_URL=postgresql://localhost/tarkka
tarkka db upgrade
TARKKA_WORK_BACKEND=postgres \
tarkka work show <work-id>

TARKKA_DOCUMENT_BACKEND=postgres \
tarkka documents manifest <document-id>

Install the optional MCP transport to expose the same staged research services to an MCP client over stdio:

uv sync --extra mcp
tarkka-mcp

MCP uses the same transport-neutral application contracts for progressive capability discovery, bounded Document expansion, Claim lineage, and safe path-free persisted-Document replay. Start with research_capabilities, expand only the operation schema needed for the task, and then invoke the specific read operation. Remote replay accepts a stable Document handle rather than a caller-supplied server filesystem path.

For transparent, opt-in local telemetry, set a JSONL destination before starting the server. Events contain only tool name, outcome/error code, response byte count, estimated tokens, and latency; they never contain document text, request arguments, or identifiers:

export TARKKA_MCP_TELEMETRY_PATH=./var/tarkka-mcp-usage.jsonl
tarkka-mcp

Agent-first design

Tarkka uses progressive disclosure to conserve context:

capabilities
  -> manifests
  -> summaries / structured findings
  -> evidence
  -> full sections/documents
  -> raw artifacts

Agents should retrieve the smallest representation that can answer the current question, then expand only when necessary. See AGENTS.md, CLAUDE.md, and docs/CONTEXT_EFFICIENCY.md.

To connect Claude Code, Codex CLI, or Claude Desktop to a running tarkka-mcp server, see docs/MCP_CLIENT_SETUP.md.

AGENTS.md is the shared repository instruction file for Codex, Claude, and other coding agents. CLAUDE.md contains only Claude-specific context-loading guidance so architectural rules are not duplicated.

The first portable staged skill is available at:

skills/research-discovery/SKILL.md

Architecture principles

• provider-neutral and domain-neutral core
• no LLM required for core operation
• immutable content-addressed raw artifacts
• provenance as a first-class concern
• claims, evidence, and citations remain distinct
• rights/access/redistribution policy modeled separately from software licensing
• PostgreSQL is the reference production metadata store
• adapters/plugins remain replaceable
• domain-specific semantics belong in domain packs
• CLI/API/MCP/SDK should share application services

Stable design contracts

The implementation evolves through small, tested slices, but these durable contracts define the direction of that work:

• ARTIFACT_COMPOSITION.md — retain exact originals while creating addressable structural components and reproducible derivative exports.
• OCR_QUALITY_AND_DERIVATIONS.md — keep OCR/layout/vision output separate from source-native content and make conversion quality reviewable.
• SIMILARITY_CONSENSUS_AND_GRAPH.md — discover related claims and graph-shaped research relationships without silently merging facts or asserting truth.

For substantial work, GitHub issues hold the active feature specification and linked pull requests hold implementation/review/validation history. Stable decisions are promoted into the documents above and the architecture docs rather than copied into a local handoff journal.

Current roadmap position

Tarkka now has a complete auditable proof/replay vertical slice: immutable source preservation, deterministic normalization, evidence-backed Claims, provenance inspection, portable v1/v2/v3 proof bundles, offline verification, exact deterministic replay, and safe replay/lineage serving to agents.

The immediate product sequence is:

five-minute offline proof/replay adoption path
  -> workspace as the product noun
  -> durable research library
  -> claim receipts / briefs
  -> compact agent verbs + context wallet
  -> compiled encyclopedia editions

Proof/replay remains the trust kernel (issue #198). The knowledge-layer contract is docs/RESEARCH_LIBRARY_AND_ENCYCLOPEDIA.md (issue #342).

See docs/ROADMAP.md for the broader implementation sequence.

Development

python -m pip install -e '.[dev]'
ruff check .
mypy
pytest

CI validates Ruff, strict mypy, and pytest across Python 3.11–3.13. Optional Docling integration is verified separately with a CPU-only workflow.

Documentation

For users, start with:

1. docs/QUICKSTART_PROOF_REPLAY.md
2. docs/PROOF_BUNDLES.md
3. docs/MCP_CLIENT_SETUP.md
4. docs/EVALUATION_CORPUS.md and docs/EVALUATION_RESULTS.md

For contributors and coding agents, start with:

1. AGENTS.md
2. docs/PROJECT_CHARTER.md
3. docs/ARCHITECTURE.md
4. docs/ROADMAP.md

Building a third-party adapter (artifact store, repository, HTTP transport, host resolver)? See docs/CONFORMANCE.md for the published, reusable behavioral contracts Tarkka's own reference adapters run against.

Then load task-specific documents on demand. The repository intentionally avoids requiring agents or humans to read the entire documentation set before doing focused work.