Document Gotenberg-powered file conversion for non-core formats (#2218)

Adds a "Convertible Formats (via Gotenberg)" section to
docs/upload_methods/supported_formats.md explaining the ~126
LibreOffice-backed extensions the optional pre-parse converter can turn
into PDF, and a screenshotted "File Converters (Gotenberg)" walkthrough
in docs/pipelines/pipeline_configuration.md covering enabling/disabling
via the Admin UI or DEFAULT_FILE_CONVERTER. Also documents the
already-running gotenberg compose service in choose-and-configure-docker-stack.md,
adds GOTENBERG_* examples to the sample .django env files, and updates
README's Supported Formats section.

Adds a data-testid to the File Converter row in FiletypeDefaults.tsx and
three docScreenshot captures in the existing file-converter on/off
Playwright component tests to back the walkthrough.

Co-authored-by: Claude <noreply@anthropic.com>

Author: JSv4

README.md

@@ -287,7 +287,8 @@ Browse the full documentation at [jsv4.github.io/OpenContracts](https://jsv4.git
 | [PDF Data Format](docs/architecture/PDF-data-layer.md)                      | How text maps to PDF coordinates     |
 | [LLM Framework](docs/architecture/llms/README.md)                           | PydanticAI integration and agents    |
 | [Vector Stores](docs/extract_and_retrieval/vector_stores.md)                | Semantic search architecture         |
-| [Pipeline Overview](docs/pipelines/pipeline_overview.md)                    | Parser and embedder system           |
+| [Pipeline Overview](docs/pipelines/pipeline_overview.md)                    | Parser, embedder, and file-converter system |
+| [Supported File Formats](docs/upload_methods/supported_formats.md)          | Core parsers plus Gotenberg-convertible formats |
 | [Custom Extractors](docs/walkthrough/advanced/write-your-own-extractors.md) | Build your own data extraction tasks |
 | [v3.0.0.b3 Release Notes](docs/releases/v3.0.0.b3.md)                       | Latest features and migration guide  |
 
@@ -335,7 +336,13 @@ OpenContracts collects anonymous usage data to guide development priorities: ins
 - DOCX (Word documents, via the [Docxodus](https://github.com/JSv4/Docxodus) microservice — character-offset annotations aligned with WASM rendering)
 - Plain text (`.txt`, split into sentence annotations via spaCy)
 
-See [Supported File Formats](docs/upload_methods/supported_formats.md) for parser details and the `supportedMimeTypes` GraphQL query that exposes the live list to the frontend.
+Plus **126 additional formats** — legacy Microsoft Office, OpenDocument, Apple
+iWork, WordPerfect, images, HTML, and more — convertible to PDF by an optional
+[Gotenberg](https://github.com/gotenberg/gotenberg)-powered file converter
+(off by default; enable it in Admin → Pipeline Configuration, no
+docker-compose changes needed).
+
+See [Supported File Formats](docs/upload_methods/supported_formats.md) for parser and conversion details, and the `supportedMimeTypes` / `convertibleExtensions` GraphQL queries that expose the live lists to the frontend.
 
 ---
 

changelog.d/gotenberg-docs-overhaul.changed.md

@@ -0,0 +1,23 @@
+- **Documented Gotenberg-powered file conversion for non-core formats.**
+  `docs/upload_methods/supported_formats.md` gained a "Convertible Formats
+  (via Gotenberg)" section explaining the ~126 legacy Office/OpenDocument/
+  iWork/image/HTML extensions the optional pre-parse converter can turn into
+  PDF, how it fits into the ingest pipeline, and its off-by-default posture.
+  `docs/pipelines/pipeline_configuration.md` gained a "File Converters
+  (Gotenberg)" section with a screenshotted step-by-step walkthrough for
+  enabling/disabling conversion via the Admin UI or the `DEFAULT_FILE_CONVERTER`
+  env var, plus a note that the `gotenberg` compose service already ships in
+  `local.yml`/`production.yml` and needs no compose changes to use.
+  `docs/configuration/choose-and-configure-docker-stack.md` gained an
+  "Optional Services" section covering `gotenberg` (and pointers for
+  `warp-ingest`/`privacy_filter`). Sample `.django` env files
+  (`docs/sample_env_files/backend/{local,production}/.django`) now document
+  `GOTENBERG_SERVICE_URL`, `GOTENBERG_CONVERTER_TIMEOUT`, and
+  `DEFAULT_FILE_CONVERTER`. `README.md`'s "Supported Formats" section and
+  documentation table now mention the conversion capability.
+- **New `data-testid="file-converter-row"`** on the File Converter row in
+  `frontend/src/components/admin/system_settings/FiletypeDefaults.tsx`, used
+  by three new `docScreenshot` captures in
+  `frontend/tests/system-settings-flows.ct.tsx`'s "file converter on/off"
+  suite (disabled row, converter picker modal, enabled row) that back the new
+  docs walkthrough.

docs/assets/images/screenshots/auto/admin--pipeline-settings--file-converter-disabled.png

@@ -170,6 +170,47 @@ docker compose -f production.yml up
 
 You should now be able to access the OpenContracts frontend by visiting your configured domain (served through Traefik on port 80/443).
 
+## Optional Services
+
+Both `local.yml` and `production.yml` ship a handful of services that start
+alongside the core stack but sit idle, doing no work and costing negligible
+resources, until you opt into the feature they power. None require a
+compose-file edit to "turn on" — you don't need to add or remove services;
+enabling the feature is a Django/`PipelineSettings` configuration change,
+and the container is already there to receive requests once you do.
+
+| Service | Powers | Enable via |
+|---|---|---|
+| `gotenberg` | Pre-parse file-to-PDF conversion for non-core formats (`.doc`, `.odt`, `.pptx`, images, ...) | `PipelineSettings.default_file_converter` (Admin UI or `DEFAULT_FILE_CONVERTER` env var) |
+| `warp-ingest` | Alternative deterministic PDF parser | `warp-ingest` compose profile + `PDF_PARSER` |
+| `privacy_filter` | PII redaction pass | `PRIVACY_FILTER_API_KEY` env var |
+
+### Gotenberg (file conversion for non-core formats)
+
+The `gotenberg` service (`gotenberg/gotenberg:8`) is defined in both
+`local.yml` and `production.yml` with no compose profile gate, so it starts
+automatically with `docker compose -f local.yml up` / `docker compose -f
+production.yml up` — there's nothing to add here. It has no published host
+port; `django` and `celeryworker` reach it internally at
+`http://gotenberg:3000` on the docker bridge network, and both declare it as
+an optional dependency (`required: false`) so the stack still starts if the
+container is ever removed from your override.
+
+By default no file converter is selected, so uploads outside PDF/TXT/DOCX
+are rejected and the `gotenberg` container never receives a request. To
+accept the ~120 additional formats it can convert (legacy Office,
+OpenDocument, iWork, images, and more), configure it as the default file
+converter — see the step-by-step walkthrough with screenshots in
+[File Converters (Gotenberg)](../pipelines/pipeline_configuration.md#file-converters-gotenberg),
+and the capability/security overview in
+[Supported File Formats](../upload_methods/supported_formats.md#convertible-formats-via-gotenberg).
+
+If you want to remove the service entirely (e.g. a minimal-footprint
+deployment that will never need conversion), delete or comment out the
+`gotenberg` block in your compose override — just make sure
+`default_file_converter` stays unset, or ingest will fail for any upload
+routed through it.
+
 ## ENV File Configurations
 
 OpenContracts is configured via .env files. For a local deployment, these should go in `.envs/.local`. For production,

docs/assets/images/screenshots/auto/admin--pipeline-settings--file-converter-enabled.png

@@ -172,6 +172,111 @@ Sensitive configuration (API keys, credentials) is stored encrypted in the datab
 > 1. Export secrets via Django shell: `PipelineSettings.get_instance().get_secrets()`
 > 2. After rotation, re-import: `instance.set_secrets(exported_secrets); instance.save()`
 
+## File Converters (Gotenberg)
+
+File converters are an **optional pre-parse step** that turns an upload with
+no native parser (`.doc`, `.rtf`, `.odt`, `.pptx`, `.xlsx`, `.png`, and ~120
+other extensions — see [Supported File Formats](../upload_methods/supported_formats.md#convertible-formats-via-gotenberg))
+into a PDF before the normal parser/thumbnailer/embedder stages run. This is
+a single install-wide setting (`PipelineSettings.default_file_converter`),
+**not** file-type-scoped like the Parser/Thumbnailer columns above — there is
+one converter selection for the whole install.
+
+**Disabled by default.** A fresh install accepts only the three core formats
+(PDF, TXT, DOCX) until you configure a converter. OpenContracts ships one
+implementation out of the box, `GotenbergFileConverter`, which delegates to a
+[Gotenberg](https://github.com/gotenberg/gotenberg) service's LibreOffice
+route. See [File Converters](pipeline_overview.md#file-converters) in the
+Pipeline Architecture doc for the full extension-eligibility and security
+model (stored-MIME-type hardening, conversion-service egress/SSRF posture).
+
+### The `gotenberg` service already runs in your stack
+
+`local.yml` and `production.yml` both define a `gotenberg` service
+(`gotenberg/gotenberg:8`) with no compose profile gate, so it starts
+automatically with the rest of the stack (`docker compose -f local.yml up`,
+etc.) alongside `django` and `celeryworker` — you don't need to add or start
+anything at the compose level. It has no published host port (it's reachable
+only on the docker bridge at `http://gotenberg:3000`, avoiding a collision
+with the frontend dev server's own port 3000) and `django`/`celeryworker`
+declare it as an optional dependency (`required: false`), so the stack still
+starts normally if the container is ever removed. Until a converter is
+selected in `PipelineSettings`, the container simply sits idle — no requests
+are ever sent to it.
+
+**Enabling conversion is therefore purely a `PipelineSettings` change**, made
+either through the admin UI at runtime or via an environment variable at
+first boot.
+
+### Enabling via the Admin UI (runtime, no restart)
+
+1. Log in as a superuser and navigate to **Admin → Pipeline Configuration**.
+   The **File Converter** row lives in the Filetype Defaults panel, below
+   **Default Embedder**. A fresh install shows it disabled:
+
+   ![File Converter row — disabled](../assets/images/screenshots/auto/admin--pipeline-settings--file-converter-disabled.png)
+
+2. Click **Edit** on the File Converter row to open the picker. Choose the
+   **Gotenberg PDF Converter** card (or type a custom converter class path
+   directly into the input, if you've registered your own `BaseFileConverter`
+   subclass):
+
+   ![File Converter picker — Gotenberg selected](../assets/images/screenshots/auto/admin--pipeline-settings--file-converter-picker.png)
+
+3. Click **Save**. The row now shows the configured converter's class path,
+   and every subsequent upload whose extension is in Gotenberg's supported
+   set is converted to PDF before parsing:
+
+   ![File Converter row — enabled](../assets/images/screenshots/auto/admin--pipeline-settings--file-converter-enabled.png)
+
+Changes take effect immediately for new uploads; documents already ingested
+are not reprocessed.
+
+### Disabling via the Admin UI
+
+Repeat the same flow and pick **None (conversion disabled)** in the picker,
+then **Save**. This writes an empty string to `default_file_converter`,
+which the backend treats as "conversion off" — uploads outside the three
+core formats are rejected again, exactly like a fresh install.
+
+### Enabling/disabling via environment variable
+
+For first-boot / infrastructure-as-code setups, set `DEFAULT_FILE_CONVERTER`
+in your `.django` env file before running migrations — it seeds
+`PipelineSettings.default_file_converter` the same way `PDF_PARSER` seeds the
+preferred parser (see [First-Time Setup](#first-time-setup-fresh-install)
+above):
+
+```bash
+# .envs/.local/.django or .envs/.production/.django
+
+# Enable Gotenberg-powered conversion for non-core formats:
+DEFAULT_FILE_CONVERTER=opencontractserver.pipeline.file_converters.gotenberg_converter.GotenbergFileConverter
+
+# Leave unset (or empty) to keep conversion disabled — the default.
+# DEFAULT_FILE_CONVERTER=
+```
+
+This only takes effect on first migration of a fresh `PipelineSettings`
+singleton, or after `migrate_pipeline_settings --sync-preferences`, per the
+[Configuration Priority](#configuration-priority) rules above — an existing
+install should use the Admin UI instead, since the database is the runtime
+source of truth.
+
+Two related settings tune the Gotenberg connection itself (also configurable
+as `GotenbergFileConverter` component settings in the Admin UI's Component
+Library):
+
+| Env var | Default | Purpose |
+|---|---|---|
+| `GOTENBERG_SERVICE_URL` | `http://gotenberg:3000` | Base URL of the Gotenberg service |
+| `GOTENBERG_CONVERTER_TIMEOUT` | `300` | Conversion request timeout, in seconds |
+
+To narrow the converter to a subset of extensions (e.g. only spreadsheets),
+set its `convert_extensions` component setting to a comma-separated list —
+see the **Component Library** panel or `GotenbergFileConverter.Settings` in
+[`gotenberg_converter.py`](../../opencontractserver/pipeline/file_converters/gotenberg_converter.py).
+
 ## Management Command Reference
 
 ### `migrate_pipeline_settings`
@@ -290,6 +395,7 @@ separately via their own delete mutations if desired.
 ## See Also
 
 - [Pipeline Architecture Overview](pipeline_overview.md)
+- [Supported File Formats](../upload_methods/supported_formats.md)
 - [Docling Parser](docling_parser.md)
 - [LlamaParse Parser](llamaparse_parser.md)
 - [Multimodal Embedder](multimodal_embedder.md)

docs/assets/images/screenshots/auto/admin--pipeline-settings--file-converter-picker.png

@@ -63,3 +63,14 @@ WARP_INGEST_PARSER_SERVICE_URL=http://warp-ingest:5001/api/parse
 # Must match the WARP_API_KEY set on the warp-ingest container. Warp-Ingest always
 # requires a key (its own default is "abc123"), so a blank value yields a 401.
 WARP_INGEST_API_KEY=
+
+# Gotenberg file converter (optional pre-parse conversion of non-core formats
+# -- .doc, .odt, .pptx, images, etc. -- to PDF). The `gotenberg` compose
+# service already runs by default; it sits idle until a converter is
+# selected. See docs/pipelines/pipeline_configuration.md#file-converters-gotenberg.
+# ------------------------------------------------------------------------------
+# Uncomment to enable conversion for non-core formats on first boot (or set
+# it later at runtime via the Admin -> Pipeline Configuration UI):
+# DEFAULT_FILE_CONVERTER=opencontractserver.pipeline.file_converters.gotenberg_converter.GotenbergFileConverter
+GOTENBERG_SERVICE_URL=http://gotenberg:3000
+GOTENBERG_CONVERTER_TIMEOUT=300

docs/configuration/choose-and-configure-docker-stack.md

@@ -81,3 +81,14 @@ WARP_INGEST_PARSER_SERVICE_URL=http://warp-ingest:5001/api/parse
 # Must match the WARP_API_KEY set on the warp-ingest container. Warp-Ingest always
 # requires a key (its own default is "abc123"), so a blank value yields a 401.
 WARP_INGEST_API_KEY=
+
+# Gotenberg file converter (optional pre-parse conversion of non-core formats
+# -- .doc, .odt, .pptx, images, etc. -- to PDF). The `gotenberg` compose
+# service already runs by default; it sits idle until a converter is
+# selected. See docs/pipelines/pipeline_configuration.md#file-converters-gotenberg.
+# ------------------------------------------------------------------------------
+# Uncomment to enable conversion for non-core formats on first boot (or set
+# it later at runtime via the Admin -> Pipeline Configuration UI):
+# DEFAULT_FILE_CONVERTER=opencontractserver.pipeline.file_converters.gotenberg_converter.GotenbergFileConverter
+GOTENBERG_SERVICE_URL=http://gotenberg:3000
+GOTENBERG_CONVERTER_TIMEOUT=300

docs/pipelines/pipeline_configuration.md

@@ -9,7 +9,7 @@ effects that certain annotation types trigger on import.
 
 | Page | Description |
 |------|-------------|
-| [Supported File Formats](supported_formats.md) | File types accepted for upload and which parsers handle them |
+| [Supported File Formats](supported_formats.md) | File types accepted for upload, which parsers handle them, and the optional Gotenberg conversion step for everything else |
 | [Single Document Upload](single_upload.md) | Uploading individual documents through the UI or API |
 | [Bulk ZIP Import](bulk_zip_import.md) | Importing many documents at once with folder structure, metadata, and relationships |
 | [Corpus Export/Import](corpus_export_import.md) | Exporting and re-importing full corpuses with annotations, labels, and configuration |