Skip to content

plans.md

Latest commit

 

History

History
372 lines (280 loc) · 12.4 KB

plans.md

File metadata and controls

372 lines (280 loc) · 12.4 KB
Raw

Master Plan for ir_v2

Purpose

This document is the master plan for the repo. It defines the stable direction of the project:

  • how the repo is organized
  • how the CLI works
  • how YAML config drives execution
  • how implementations stay replaceable
  • what should be built first

The docs entrypoint is docs/README.md. Detailed implementation plans for specific components belong in separate docs. Parser-specific work lives in docs/sub-plans/parser-plan.md, with preprocessing details in docs/sub-plans/parser-preprocess-plan.md. Eval dataset generation lives in docs/sub-plans/eval-dataset-generation-plan.md. Open work for the library and SDK lives in docs/TODO.md, with detail plans under docs/sub-plans/.

Product Shape

The repo is a config-driven RAG pipeline with a CLI-first workflow.

Current commands:

  • rag-cli convert --source docx --target pdf --input ... --output ...
  • rag-cli chunk prepare --input data/artifacts/<run_id>/normalized/elements.json
  • rag-cli parser --config configs/baseline.yml
  • rag-cli parser --config configs/baseline.yml --output-dir data/artifacts/custom-run
  • rag-cli parser review --input data/artifacts/<run_id>/parsed/unstructured_elements.json --output-format html
  • rag-cli parser preprocess --input data/artifacts/<run_id>/parsed/unstructured_elements.json
  • rag-cli dataset peek --dataset all --root data/datasets --pretty
  • rag-cli eval create-dataset --input data/artifacts/<run_id>/chunked/chunks.json --config configs/eval-dataset.yml
  • rag-cli indexing --config configs/baseline.yml

The config decides whether a run is:

  • parser_only
  • full

convert prepares source files for parsing when page-preserving PDF conversion is needed. parser produces raw parser artifacts in the Unstructured output format. chunk prepare builds structured retrieval units from normalized parser elements. indexing consumes chunk artifacts, optionally enriches chunk retrieval text, and builds search indexes.

Engineering Defaults

  • Python 3.11+
  • uv for dependency and environment management
  • typer for CLI construction
  • pydantic for typed models and validation
  • pydantic-settings for settings management
  • langchain as the integration layer where it adds value
  • ruff for linting and formatting
  • YAML config as the runtime source of truth
  • .env for local secrets

Architectural Rules

1. Config-first

  • Runtime config lives in top-level configs/
  • Code loads config through typed models and validation
  • CLI commands do not hardcode provider-specific behavior

2. Provider-based subsystem design

Each subsystem follows one pattern:

  • base.py defines the contract
  • factory.py resolves provider name from config
  • <provider>.py contains one concrete implementation

This is the main extension mechanism for the repo.

3. Thin pipelines

Pipelines orchestrate work. They should:

  • load validated config
  • build subsystem implementations through factories
  • call the stages in order
  • write manifests, artifacts, and telemetry

Pipelines should not contain parser-specific or index-specific vendor logic.

4. Persist artifacts between stages

Stage outputs must be saved so that:

  • parser can run independently
  • indexing can run later from parser artifacts
  • runs are reproducible and debuggable

5. Normalize external outputs early

Parser output is provider-standardized at the artifact layer. For now, the parser standard is raw Unstructured output.

Repo Structure

ir_v2/
├── configs/
│   ├── baseline.yml
│   └── parser-only.yml
├── data/
│   ├── raw/
│   └── artifacts/
│       └── <run_id>/
├── docs/
│   ├── plans.md
│   ├── parser-plan.md
│   ├── terms.md
│   └── survey/
├── src/
│   ├── cli.py
│   ├── config/
│   │   ├── loader.py
│   │   ├── models.py
│   │   └── validator.py
│   ├── core/
│   │   ├── common/
│   │   ├── convert/
│   │   ├── parser/
│   │   ├── chunker/
│   │   ├── enrichment/
│   │   ├── dataset_generation/
│   │   ├── embedding/
│   │   ├── vector_store/
│   │   ├── sparse_index/
│   │   └── artifact_store/
│   ├── pipelines/
│   │   ├── parser.py
│   │   ├── chunk_prepare.py
│   │   ├── eval_create_dataset.py
│   │   ├── indexing.py
│   │   └── helpers.py
│   └── models/
│       ├── manifest.py
│       └── result.py
└── tests/
    ├── unit/
    ├── integration/
    └── fixtures/

Subsystem Layout

Every subsystem under xrag/core/ should use the same layout:

xrag/core/<subsystem>/
├── base.py
├── factory.py
└── <provider>.py

Examples:

  • xrag/core/parser/base.py
  • xrag/core/parser/factory.py
  • xrag/core/parser/unstructured.py
  • xrag/core/vector_store/qdrant.py

This layout is preferred over splitting interfaces and adapters into separate top-level packages.

CLI Contract

rag-cli parser --config <path>

  • loads config
  • validates the parser section
  • runs parser pipeline
  • writes raw parser artifacts, manifest, and telemetry

rag-cli parser --config <path> --output-dir <dir>

  • overrides the default artifact root for that run
  • writes all parser outputs directly under the provided directory

rag-cli parser review --input <path> --output-format html

  • reads saved raw parser artifacts from parsed/unstructured_elements.json
  • reconstructs Unstructured elements from serialized JSON
  • falls back to non-paged HTML when elements do not include page_number
  • writes a rendered review file next to the JSON input unless --output is provided

rag-cli parser preprocess --input <path> [--config <yaml>] [--output-dir <dir>]

  • reads saved raw parser artifacts from parsed/unstructured_elements.json
  • writes normalized parser artifacts under normalized/
  • writes normalized/elements.json and normalized/review.html
  • keeps raw parser output unchanged for traceability
  • emits structured warnings for extraction gaps such as missing page metadata
  • supports YAML step toggles, with merge_table_columns disabled by default

rag-cli chunk prepare --input <path> [--output-dir <dir>]

  • reads normalized parser elements from normalized/elements.json
  • writes chunk artifacts under chunked/
  • supports pluggable chunker providers via core/chunker
  • section_table provider: section-aware text/table chunking
  • section_token provider (formerly ragflow): token-budgeted parent-child chunking ported from RAGFlow's naive chunker
  • title_hierarchy provider: multi-level title-tree chunker ported from RAGFlow's HierarchyTitleChunker
  • preserves table structure and nearby text context for retrieval

rag-cli convert --source docx --target pdf --input <path> --output <path>

  • converts one source file into a target format
  • current supported pair is docx -> pdf
  • intended to support page-preserving parsing inputs

rag-cli indexing --config <path>

  • loads config
  • validates indexing sections
  • loads chunk artifacts from chunked/chunks.json
  • optionally runs enrichment.pipeline before embedding
  • runs indexing pipeline
  • writes embedding and index metadata plus telemetry

uv run python scripts/enrichment/preview.py --input <chunks.json> --config <path>

  • loads the same enrichment config used by indexing
  • runs only the enrichment pipeline over chunks.json
  • writes inspectable JSON under enriched/
  • avoids embedding and vector-store side effects when validating enrichment fields

rag-cli dataset peek --dataset <name> --root <path>

  • inspects downloaded local benchmark datasets
  • prints a compact JSON summary plus sample rows or files
  • currently supports financebench, t2-ragbench, finder, and all
  • intended for quick EDA before writing normalization or eval code

rag-cli eval create-dataset --input <path> --config <yaml> [--output-dir <dir>]

  • reads chunk artifacts from chunked/chunks.json
  • generates evaluation QA items using a provider-based subsystem
  • supports template, ragas, and hybrid providers
  • writes eval/testset.jsonl and optional review artifacts
  • uses standalone config, not AppConfig
  • details in docs/sub-plans/eval-dataset-generation-plan.md

indexing is planned next and not implemented yet.

Config Contract

Current parser-only baseline shape:

pipeline:
  name: baseline
  mode: parser_only
  artifact_dir: data/artifacts

sources:
  - path: "data/ir-sample-a.docx"
    doc_id: avatar_comments_doc

parser:
  provider: unstructured
  options:
    mode: on_demand_job
    template_id: hi_res_and_enrichment
    poll_interval_seconds: 5
    save_raw_output: true

artifact_store:
  provider: local_fs
  options: {}

enrichment:
  pipeline: []
  options: {}

Rules:

  • parser_only mode may omit the indexing section
  • full mode must include parser and indexing sections
  • provider names map directly to factory-supported implementations

Shared Models

The repo should own these shared model categories:

  • Manifest Metadata describing run outputs, providers, inputs, and statuses
  • Result Structured pipeline outcomes for CLI and tests

These models should remain provider-agnostic.

Phase Plan

Phase 1: Repo foundation

  • create pyproject.toml
  • set Python 3.11+
  • set up uv
  • set up ruff
  • add package structure under src/
  • add config loader and validator
  • add CLI skeleton for parser and indexing

Phase 2: Parser pipeline baseline

  • implemented parser subsystem with provider factory
  • implemented artifact store
  • implemented pipelines/parser.py
  • persist raw parser artifacts, manifest, and telemetry

Details for this phase live in docs/parser-plan.md.

Phase 3: Indexing baseline — DONE

  • implemented enrichment subsystem (xrag/core/enrichment/) for RAGFlow-aligned table_context, auto_keywords, and auto_questions
  • implemented embedding subsystem (xrag/core/embedding/) with OpenAI provider
  • implemented vector store subsystem (xrag/core/vector_store/) with Chroma provider
  • implemented retrieval subsystem (xrag/core/retrieval/) with simple vector similarity
  • implemented generation subsystem (xrag/core/generation/) with OpenAI provider
  • implemented evaluation subsystem (xrag/core/evaluation/) with RAGAS provider
  • implemented pipelines/indexing.py, pipelines/rag.py, pipelines/eval_run.py
  • added CLI commands: indexing, query, eval run
  • created configs/rag-baseline.yml (AMD 2022 10-K + FinanceBench)

Phase 4: Eval dataset generation

  • implement dataset_generation subsystem with provider factory
  • Phase 4a: template provider — deterministic table QA, rule-based validation
  • Phase 4b: ragas provider — synthetic generation over pre-chunked inputs
  • Phase 4c: hybrid provider — combined generation with LLM validation
  • implement pipelines/eval_create_dataset.py
  • add eval create-dataset CLI command

Details for this phase live in docs/sub-plans/eval-dataset-generation-plan.md.

Phase 5: Hardening

  • improve config validation
  • improve telemetry
  • add integration tests
  • add compatibility checks for artifact versions

Testing Strategy

  • unit tests for config validation
  • unit tests for each subsystem factory
  • unit tests for provider implementations against base contracts
  • integration tests for rag-cli parser --config ...
  • fast unit tests are implemented and should remain network-free
  • integration tests for rag-cli indexing --config ...
  • regression tests for artifact compatibility between phases
  • CI should enforce lint, package validation, CLI smoke checks, and installed-wheel contract coverage

Decisions Already Made

  • CLI name is rag-cli
  • use parser, not ingest, as the subsystem and command name
  • keep implementation grouped under xrag/core/
  • use provider-based extension with base.py and factory.py
  • base xrag install includes hosted Unstructured API support and lightweight local parsing; heavy local PDF/OCR parsing lives behind an extra
  • use Python 3.11+, uv, typer, pydantic, pydantic-settings, langchain, and ruff
  • use separate docs for major component plans

Out of Scope for the Master Plan

These belong in component-specific docs, not here:

  • Unstructured API request details
  • provider-specific polling logic
  • exact parser normalization rules
  • query/generation architecture
  • evaluation benchmark details