Skip to content

USAGE.md

Latest commit

 

History

History
137 lines (102 loc) · 4.48 KB

USAGE.md

File metadata and controls

137 lines (102 loc) · 4.48 KB
Raw

Gather Usage

Gather turns difficult source intake into replayable, digest-backed research packets. It is designed for local CLI use, MCP hosts, and larger Project Telos workflows that need provenance before synthesis.

Install

python -m pip install gather-engine

From a source checkout:

python -m pip install -e ".[dev]"

Run

gather status --json
gather doctor --json
gather demo --json
gather --help

The same package can be exercised from source with:

python -m gather --help

Pilot

gather pilot run MANIFEST --output DIR          # capture once, write report + receipt
gather pilot refresh DIR                        # re-capture monitored sources, archive the prior view
gather pilot verify DIR                         # network-free verification of the whole root
gather pilot bundle DIR --output FILE --visibility shared
gather pilot bundle DIR --output FILE --visibility full --include-private-evidence

Exit codes: manifest refusal exits 2; a required-source failure or verification failure exits 1; success exits 0. A full bundle (which carries the private artifact root) requires --include-private-evidence. See docs/PILOT.md for the manifest boundary and the private/shared evidence split.

Web-data engine

Each command prints a receipt as JSON.

gather caps                      # what this install can do (fast / browser / stealth)
gather extract <url|file.html>   # Markdown + a per-block provenance receipt
gather markdown <url|file.html>  # structured Markdown only
gather crawl <url> --depth 2 --max-pages 50   # a witnessed, hash-chained crawl ledger

Optional capability backends (core stays zero-dependency; a missing one degrades to UNVERIFIABLE, never a fake):

pip install 'gather-engine[fast]'      # lxml, about 2x parse speed
pip install 'gather-engine[browser]'   # Playwright JS render (then: playwright install chromium)
pip install 'gather-engine[stealth]'   # curl_cffi TLS/browser impersonation

Federation

Validate a source-federation registry and compile its capture plans, all offline:

gather federation validate registry.json --json
gather federation plan registry.json --json

A registry file is a list of source rows or {"sources": [...]}; each row carries id, system, family, domain, access, adapter, url, scope, and priority. Validation is a closed contract: unknown access tokens, priorities, or extra fields are typed rejections, and the validated snapshot is sealed, so a row cannot be edited after witnessing without breaking the seal. plan adds one deterministic capture plan per source, derived from its access policy. Neither command probes a source, and a registry row is never reported as coverage or availability.

Two further audits treat a federation decision as a sealed claim surface:

gather federation policy policy.json --json
gather federation entity entities.json --json

A policy file is a list of rules or {"rules": [...]}; each rule carries rule, source_capture_ref (a content-hash capture ref), failure_class, and superseded. Each failure class maps to one typed verdict (429 to retryable_source_lead, 403 to access_escalation, 503 to retry_after). A rule with no provenance capture, an unknown failure class, or a superseded flag is a typed rejection.

An entity file is a list of candidates or {"candidates": [...]}; each candidate carries candidate_id, identifier_path (the named join key, such as ror), a confidence in [0, 1], evidence_refs, and exact_id_join. Candidates must be ordered by descending confidence, a match with no named identifier path is rejected, and a promotion to resolved requires the top candidate to be an exact-id join. Both snapshots are sealed under the federation digest, so an edited field breaks the seal.

MCP

Use gather mcp when a host needs the tool over stdio. The MCP surface should stay aligned with the CLI envelope and receipt fields.

gather mcp

Verify

python -m pytest
python examples/demo.py
python examples/pipeline.py

For public/developer delivery checks:

python -m public_surface_sweeper . --workspace --json

Boundary

Gather may collect material from live sources, but outward-facing receipts should prefer source references, content hashes, timestamps, and verdicts. Do not publish raw private payloads, secrets, credentials, or source material whose license or privacy posture does not allow redistribution.