feat(binary): any-binary normalization stage with distroless-friendly Office conversion

[ancongui]

Summary

Phase 0 of the bbox refinement initiative. The IDP service now accepts any binary callers can throw at it — DOCX, XLSX, PPTX, RTF, ODT, HTML, HEIC/HEIF/AVIF, multi-frame TIFF, SVG, ZIP/7z/TAR bundles, EML/MSG email with attachments — and turns each one into LLM-renderable bytes before extraction. Multilingual by construction: Office docs preserve their original script through Gotenberg/LibreOffice; archives + email pass content through unchanged; the multimodal LLM handles the rest.

Today the loader silently passes unknown formats through to the provider, which fails. This change makes the contract real.

Architecture

• core/services/binary/ — new package, one adapter per format class:
  • PdfGuard — encrypted/corrupt PDF detection
  • ImageNormalizer — Pillow + pillow-heif (HEIC/HEIF/AVIF), multi-frame TIFF → multi-page PDF, SVG via cairosvg, BMP → PNG
  • OfficeConverter protocol with two adapters:
    • GotenbergConverter (HTTP) — default, distroless-friendly: API + worker containers carry no soffice, just POST bytes to a Gotenberg sidecar
    • LibreOfficeConverter (subprocess) — fallback for slim/dev images that bundle LibreOffice
  • ArchiveUnpacker — ZIP + 7z + TAR + GZIP with zip-bomb guards
  • EmailUnpacker — EML (stdlib) + MSG (extract-msg) → attachments + inline body
  • BinaryNormalizer — orchestrates them all, recurses into archives/emails up to a configurable depth, fans out into the existing documents[] shape
• Pluggable office adapter wired through pyfly DI: pick via FLYDESK_IDP_OFFICE_CONVERTER=gotenberg|libreoffice. Adding a new adapter (OnlyOfficeConverter, CollaboraConverter, ...) is one new bean + one new enum value.
• Typed errors (EncryptedPdfError, OfficeConversionError, ArchiveExtractionError, ImageConversionError, UnsupportedBinaryError) all map to RFC 7807 422 problem-details via ExceptionAdvice with stable code strings.
• Multi-doc bundles (a ZIP, an email with attachments) fan out into multiple per-attachment slots — no API surface change. derived_from carries the unpack chain for traceability.

Plumbing

• PipelineOrchestrator._step_load calls the normalizer per inbound file before building _FileSlot rows.
• docker-compose.yml grows a gotenberg service the api + worker depend on. Flip FLYDESK_IDP_OFFICE_CONVERTER=libreoffice to switch back to the in-container subprocess.
• Dockerfile runtime stage adds libheif1 + libcairo2 + libpango + libgdk-pixbuf so HEIC/SVG conversion works without bloating the image with LibreOffice.
• New IDPSettings knobs: office_converter, gotenberg_url, gotenberg_timeout_s, binary_normalize_enabled (kill switch), binary_max_recursion_depth, binary_max_expanded_files, binary_libreoffice_path, binary_libreoffice_timeout_s.

Why distroless

Office conversion is the only place subprocess + filesystem writes are forced — soffice won't take stdin/stdout for Office formats and needs a writable user-profile dir. Pure-Python alternatives (python-docx, openpyxl) lose layout fidelity, which would break the Phase 1 bbox refiner downstream. The Gotenberg sidecar is the canonical pattern: bytes-in, PDF-out over HTTP, with soffice + headless Chromium isolated in its own container. The API + worker containers stay distroless-eligible.

Test plan

• pytest tests/unit → 147 passed, 1 skipped (the SVG test skips on macOS dev machines without libcairo; runs in the Dockerised image where libcairo2 is installed)
• ruff check . clean
• pyright src/flydesk_idp 0 errors
• 53 new unit tests across sniffer, pdf guard, image, archive, email, normalizer (incl. ZIP/EML fan-out + recursion limits), gotenberg (HTTP mocked via respx)
• Integration smoke against docker compose up with real Gotenberg (covered by Phase 0 follow-up)

Phase plan

This is Phase 0 of the three-phase bbox-refinement initiative:

1. Phase 0 (this PR) — Binary normalization. Independent of bbox; also fixes today's silent failure on DOCX/EML/ZIP/HEIC.
2. Phase 1 — Sync grounded bboxes (PyMuPDF + PaddleOCR + script-aware matcher).
3. Phase 2 — Async out-of-band refinement with PARTIAL_SUCCEEDED job state + new EDA destination.

Multilingual + any-binary are hard requirements across all three phases.