PRD: Deepen Document ingest into an ingest session module

[henryle97]

Problem Statement

As an xrag maintainer or SDK integrator, I currently have to understand too much of the ingest implementation to reason about one Document ingest. Source materialization, artifact directory ownership, locking, AppConfig synthesis, stage execution, timeout handling, cleanup policy, event emission, and process-local Document registration are split across multiple modules with partially overlapping responsibilities.

This makes the ingest module shallow at its public seam. The same ingest behavior is spread across the Xrag client surface and the pipeline orchestration surface, so changes to one Document ingest often require edits in several places. The result is weak locality, harder failure analysis, and tests that must patch across multiple seams instead of exercising one stable ingest interface.

Solution

Introduce a deep ingest session module that owns the lifecycle of one Document ingest from source materialization through final Document registration. From the user's perspective, ingest should continue to feel like one operation, but the implementation should concentrate the behavior behind a single seam.

The ingest session module should accept a fully described ingest request, resolve the source into a local Document input, manage artifacts and locking, run the configured ingest stages in canonical order, emit ingest events, apply cleanup policy, and return a structured result that the SDK and CLI can both consume. Existing public entry points should become thin adapters over that module rather than co-owning ingest policy.

User Stories

1. As an SDK integrator, I want one stable ingest interface, so that I can ingest a Document without learning internal pipeline choreography.
2. As a backend engineer, I want per-request artifact ownership handled in one place, so that failed ingests are easier to inspect and successful ingests clean up predictably.
3. As a maintainer, I want source materialization logic concentrated behind one seam, so that Path, bytes, and file-like inputs behave consistently.
4. As a maintainer, I want lock acquisition and release to belong to the ingest session lifecycle, so that concurrent ingests are governed by one policy.
5. As a maintainer, I want stage execution order to be owned by one module, so that parser, preprocess, chunk, and index rules do not drift across callers.
6. As a maintainer, I want the ingest session to own timeout behavior, so that long-running ingests fail consistently regardless of caller.
7. As a maintainer, I want the ingest session to own cleanup policy for artifact directories, so that ephemeral and explicit artifact paths follow one contract.
8. As a CLI user, I want progress events to come from one authoritative ingest lifecycle, so that logs reflect the real ingest state.
9. As an SDK user, I want the returned Document metadata to reflect ingest outcomes consistently, so that artifact paths, chunk counts, and failures are surfaced uniformly.
10. As a maintainer, I want IngestRequest handling to stay aligned with ADR-0001, so that optional Settings remain above the boundary and mandatory Settings remain below it.
11. As a maintainer, I want the ingest session to preserve canonical doc_id handling, so that Document identity stays stable across parser, chunk, index, and retrieval artifacts.
12. As a maintainer, I want stage-skipping and resume-from-artifacts behavior to stay available through one seam, so that callers do not need bespoke resume logic.
13. As a maintainer, I want Document registration to happen after a successful ingest session outcome, so that partial failures do not leave ambiguous process-local state.
14. As a test author, I want to test ingest behavior through one interface, so that tests can assert observable outcomes instead of patching multiple internal modules.
15. As a maintainer, I want failure information to be assembled in one place, so that errors, warnings, and per-stage outcomes are easier to interpret.
16. As a future wire-API implementer, I want a deep ingest session module underneath the SDK, so that a persistent backend can reuse the same ingest lifecycle without copying policy.
17. As a maintainer, I want the Xrag client to adapt to the ingest session rather than re-implement it, so that the client surface stays small and easier to evolve.
18. As a maintainer, I want artifact retention policy to be explicit and testable, so that changing cleanup behavior does not require a full ingest integration test every time.
19. As a maintainer, I want source validation and naming rules concentrated in one place, so that malformed ingest inputs fail early and consistently.
20. As a maintainer, I want the ingest session to expose a small, durable result contract, so that downstream modules can depend on outcomes instead of internal control flow.

Implementation Decisions

• Build a dedicated ingest session module as the primary module for one Document ingest lifecycle.
• Keep existing public entry points for ingest, but reduce them to adapters that translate user-facing input into an ingest session request and translate the result back into SDK or CLI output.
• Keep the seam deep: the ingest session interface should describe what to ingest, where artifacts live, which stages run, how events are emitted, and what cleanup policy applies, without exposing stage-by-stage implementation details to callers.
• Preserve the canonical ingest stage order of parser, preprocess, chunk, and index.
• Preserve resume behavior for skipped stages by keeping artifact-based stage handoff as the contract for partial runs.
• Preserve the existing Document and ingest result concepts; the refactor is about concentrating behavior, not changing user-facing ingest semantics by default.
• Respect ADR-0001: Settings remain optional only at the public orchestration boundary and mandatory below that boundary.
• Keep Document identity centered on the canonical doc_id language from CONTEXT.md; the ingest session should not introduce a competing identity term.
• Extract source materialization into its own deep module or submodule owned by the ingest session. It should normalize Path, raw bytes, and file-like inputs into one local source contract.
• Extract artifact ownership and retention policy into a dedicated module or submodule owned by the ingest session. It should decide ephemeral versus caller-owned paths, lock scope, and cleanup semantics.
• Extract ingest event emission behind a seam so CLI logging, future tracing, and backend progress reporting can consume the same lifecycle events.
• Keep AppConfig synthesis close to the ingest session or move it behind a dedicated builder owned by the session, so the client no longer co-owns ingest pipeline assembly.
• Ensure Document registration happens only after a successful ingest session outcome, while failure paths preserve enough context for retry and inspection.
• Treat the process-local Document registry as an adapter concern around a successful ingest session result, not as the source of truth for ingest lifecycle state.
• Avoid introducing new hypothetical seams unless there are already at least two adapters or a clear variation point. The refactor should deepen real modules, not add pass-through abstractions.

Testing Decisions

• Good tests should exercise external behavior at the module interface, not implementation details or temporary helper decomposition.
• The ingest session module should receive the heaviest testing because it becomes the main test surface for one Document ingest lifecycle.
• Source materialization behavior should be tested through observable input normalization outcomes for Path, raw bytes, and file-like sources.
• Artifact ownership and retention policy should be tested through observable directory creation, lock handling, cleanup, and artifact preservation behavior.
• Event emission should be tested through emitted ingest events and stage transitions, not by asserting which internal helper produced them.
• Failure-path tests should verify user-visible contracts such as timeout errors, cleanup semantics, stage error recording, and retry-safe outcomes.
• Existing prior art already covers adjacent behavior and should be reused as style guidance: the current unit coverage around ingest pipeline orchestration, ingest locking, client document operations, keep-artifacts behavior, and Settings propagation demonstrates the kinds of observable assertions this refactor should preserve.
• Regression tests should prove that SDK-level ingest behavior remains stable after the refactor, especially artifact retention, chunk count reporting, external ID conflict handling, and retriever invalidation after successful writes.

Out of Scope

• Changing retrieval, reranking, or generation behavior.
• Changing Chunk V2 or Pipeline Chunk Artifact schema.
• Introducing persistent Document storage or replacing the process-local registry with a database-backed source of truth.
• Adding new ingest stages beyond parser, preprocess, chunk, and index.
• Reworking parser, chunker, or indexing algorithms for retrieval-quality improvements.
• Re-litigating ADR-0001 or changing the asymmetric Settings boundary.
• Building a wire API or multi-process job system in this PRD.
• Redefining doc_id, Document, Chunk, or Retrieval Text terminology.

Further Notes

• This PRD is the architecture deepening follow-up for candidate 4 from the architecture review: the goal is higher locality and leverage in the ingest module, not new end-user functionality.
• The deletion test should guide the refactor. If the ingest session module were removed, the lifecycle complexity should reappear across multiple callers. That is the signal that the module is earning its keep.
• The preferred outcome is a smaller, clearer Xrag client surface and a single authoritative ingest lifecycle that future SDK, CLI, and backend adapters can all reuse.