ingest: emit stage-level progress events on documents.ingest

[henryle97]

Context

Today documents.ingest(...) returns one terminal Document (status ready or failed). There is no public hook to observe what stage the pipeline is in (parsing → cleaning → splitting → embedding → indexing).

For comparison, Dify's dataset API exposes 8 states (waiting / parsing / cleaning / splitting / indexing / completed / error / paused) with per-stage timestamps (parsing_completed_at, cleaning_completed_at, splitting_completed_at, ...) and segment counters (completed_segments / total_segments). This lets a UI render a real progress bar instead of a spinner.

Why this matters for production

• Long ingests (large PDFs, slow embedding providers) look frozen to the operator with no signal.
• Consumers (e.g. llm-agent) currently store only pending → ready | failed in their durable registry because that is all xrag exposes — see docs/rag/architecture.md §3 in the consumer repo.
• Without per-stage timestamps it is impossible to know whether a given ingest is slow at parse, embed, or upsert. That is the first question on every prod incident.

Proposed shape

Two non-mutually-exclusive options:

Option A — callback (additive, no async-generator return-type change)

async def on_event(ev: IngestEvent) -> None: ...

await client.for_tenant(t).documents.ingest(
    path,
    collection=\"docs\",
    on_event=on_event,
)

where

class IngestEvent(BaseModel):
    document_id: str
    stage: Literal[\"parse\", \"clean\", \"split\", \"embed\", \"index\"]
    phase: Literal[\"started\", \"progress\", \"completed\", \"failed\"]
    completed_units: int | None = None   # e.g. chunks embedded
    total_units: int | None = None
    elapsed_ms: int
    detail: dict[str, Any] | None = None

Option B — async iterator variant for callers who prefer pull semantics:

async for ev in client.for_tenant(t).documents.ingest_iter(path, collection=\"docs\"):
    ...

Either works; A is the smaller change.

Acceptance

• Each stage emits started and completed (and optionally progress updates for embed/index where unit counts are known).
• Errors emit failed before the final XragError is raised.
• Existing callers without `on_event` see no behaviour change.
• Documented in the resource docstring with one runnable example.
• Unit tests cover happy path + parser-stage failure + cancellation interleaving (Feat/xrag v0.2 #2).

[henryle97]

Closer than the title suggests — most of this already exists.

The event union (StageStarted / StageCompleted / ProgressTick / SourceStarted / SourceCompleted / SourceFailed) is in xrag/models/ingest.py:90, and IngestRequest.on_event is wired through xrag/pipelines/ingest.py:311-321. The remaining gap is purely SDK plumbing: DocumentsResource.ingest() (xrag/client/resources/documents.py:145-157) doesn't accept on_event and doesn't thread it into the IngestRequest(...) it builds at line 222.

Proposing this becomes a small PR (~10 LOC + a test) that adds the parameter and forwards it. Keeping the existing tuple-of-events at the pipeline level is fine; we just need the surface on the public method. Will pick this up first since the cost is trivial.