Skip to content

INGESTION.md

Latest commit

 

History

History
58 lines (37 loc) · 2.16 KB

INGESTION.md

File metadata and controls

58 lines (37 loc) · 2.16 KB
Raw

Ingestion

The ingestion layer converts source-specific inputs into canonical documents.

Supported Inputs

  • PDF files: .pdf
  • Plain text: .txt
  • Markdown: .md, .markdown
  • HTML snapshots: .html, .htm
  • URL lists: .urls
  • Post exports: .jsonl

Post JSONL Format

Use this for X posts, threads, newsletters, scraped snippets, or exported social content:

{"source_type":"x_post","author":"@example","url":"https://x.com/example/status/1","published_at":"2026-01-01","text":"Post body"}

Each line is one source document. Extra fields are preserved as metadata.

URL Lists

Create a .urls file with one URL per line:

https://example.com/article
https://example.com/guide

The loader downloads each page, removes scripts/styles, and extracts headings, paragraphs, list items, and blockquotes.

URL ingestion depends on current network access and the current content of each webpage. For reproducible production runs, prefer saving HTML snapshots into corpus/ or archiving fetched pages alongside the source list.

Fresh Production Run

  1. Put source files in corpus/.
  2. Run:
rag rebuild corpus --yes

This removes generated data under the configured data directory, creates a clean SQLite database, ingests the current files, chunks/enriches them, and builds the vector index.

To isolate a trial rebuild:

rag --data-dir data-production-test rebuild corpus --yes

Source files are never deleted by reset or rebuild commands.

When ingesting a directory, the top-level README.md in that input directory is skipped so corpus instruction files do not pollute retrieval results. Nested README files are ingested as normal content. To index a top-level README intentionally, pass that file path directly to rag ingest.

If the top-level README contains bullets like - `filename.pdf` - Human Title, that title is used as a source title override. This is useful for curated corpora where PDF metadata titles are missing or publisher-generated. For nested corpora, prefer relative paths such as - `section-a/filename.pdf` - Human Title. Basename-only overrides are applied only when that basename is unique in the input tree.