fix: documents open as their own text; ship the readers that make them readable (#1279, #1278)

[accesswatch]

Closes #1279. Closes #1278.

Two user reports on the same surface turned out to share a root cause in packaging, so this branch carries the fixes, the packaging change they imply, and the two features that were sitting uncommitted alongside them.

The reports

#1279 — "when I load a word document, it says # DOCX Extract at the top." Two defects behind it:

1. Every fallback extract prepended a synthetic banner. Word, ODT, PowerPoint, and spreadsheet extracts opened with # <FORMAT> Extract, and PDFs opened with a four-line # PDF Extract / Engine: / Quality score: / low-confidence block — text that is not in the user's file, read out first by a screen reader every time, and (for PDFs) fed to the model on the AI document-QA path. That provenance already lives in source_metadata; it is now reported by build_intake_summary (the spoken/status line on open) and build_intake_report (Document Intake Report), which gained the low-confidence/OCR advice the PDF header used to carry. This extends EDS-21's earlier removal of # RTF Extract to every remaining reader. # Notebook and # SQLite Database: stay — those texts are wholly synthesized reports of a non-text source, not a document's own words.
2. The reporter was on the weakest reader QUILL has. MarkItDown is an optional download, so a packaged install always fell to _format_docx's hand-rolled word/document.xml walk: one flat line per paragraph, headings/lists/tables discarded. python-docx is a base dependency, so Word now renders through it (new quill/io/docx_text.py: heading levels, list styles and direct w:numPr numbering, tables as GFM, in document order), with the XML walk kept as the last resort. Read chain: MarkItDown when present → python-docx → raw XML.

#1278 — JSONDecodeError: Unexpected UTF-8 BOM opening a .json. read_structured_document's text branch used a bare path.read_text(), so a BOM survived into the parser. It now reads through quill.io.text.read_text_document, giving .json/.xml/.toml/.yaml/.ipynb the contract plain text has had since #648/#649: BOM stripped and remembered as utf-8-sig (a save re-adds it byte-for-byte), original line ending kept (CRLF was previously flattened here), and the #867 cp1252/latin-1 fallback. On 0.9.0 this crashed the open; after #1228's guard it silently degraded the file to unformatted raw text.

Packaging: measured, not assumed

Bundling the whole pdf-ocr extra looked like the obvious fix. Measuring it first changed the answer: MarkItDown hard-requires magika, which hard-requires onnxruntime and numpy, and its xlsx/xls converters require pandas — about 150 MB installed, not the "~30 MB" the component dialog claimed.

So the lean half ships and the heavy half stays optional:

• New office-text extra — pdfplumber, pypdf, openpyxl (~16 MB) — bundled through DEFAULT_BUNDLED_DEPENDENCY_GROUPS on Windows and through setup_macos.py's includes + macos-release.yml's install specs on macOS. Without pdfplumber/pypdf there is no PDF text reader at all; without openpyxl an .xlsx cannot open.
• macOS was worse than Windows: every reader is imported function-locally, so py2app's tracer bundled none of them — the .app had no docx, no PDF reader, and no openpyxl. No packages entry is needed; build_macos.sh already lifts zipped packages carrying native binaries out of the zip before signing.
• MarkItDown remains a one-click download, relabeled with its true size and described as the enhancement it is. A test asserts it is not in the bundled groups, so a future re-add is a deliberate decision rather than a silent 150 MB installer.
• A read that lands on a reduced reader stamps extraction_pack_missing, which the intake summary and report turn into a one-line pointer at the download — suppressed when python-docx already read the file well, so it never nags about 150 MB for a document that came through fine.

Also on this branch (previously uncommitted)

• Live365 link normalization. A Live365 station page or web-player link is HTML, not a stream, so pasting one into Add Custom Station saved a station that could never play. normalize_live365 rewrites any station/player link — or a bare a##### id — to the canonical stream URL and the dialog says it did. Pure string transform: no network call, no scraping, no use of Live365's auth-gated API, nothing to gate in Safe Mode.
• Quill Converter as a standalone product (Audio Studio: universal audio converter — mixed file/folder queue, basic + advanced modes, rich DSP, background/tray operation #1255) landed separately on main (PR build(converter): standalone Quill Converter build entry + tile icon (#1255) #1275) while this branch was in flight; this branch keeps only its documentation (CHANGELOG, PRD family-app list).

Docs

All four surfaces, per the incremental-docs rule: CHANGELOG 1.0.0, release notes 1.0.0, the user guide (Document Intake, Adding your own stations, the PDF-reader pointer), the PRD (a new "Reader chain and the document-text contract" section, the corrected docx_read_engine chain and Tier-1 table row, the Live365 contract, Quill Converter in the family app list), and tutorial 09. HTML/EPUB regenerated with the same pandoc invocations CI uses; scripts/check_docs_artifacts.py passes.

Verification

• pytest tests/unit tests/accessibility -n 4 --dist=loadfile (the PR CI invocation) — see the CI run on this PR
• Gates: GATE-11 (structured.py 776 → 787 with the reasoning recorded; optional_components.py and main_frame_speech_downloads.py were at their ceilings and the wording was condensed to fit rather than rebaselined), banned patterns, error codes, dialog inventory — all green
• ruff check / ruff format --check clean; mypy quill\core quill\io clean
• New tests: real .docx files built with python-docx and read back; BOM'd JSON/XML; CRLF preservation; no-banner assertions per format; the extraction_pack_missing flag both ways; bundled-vs-not-bundled packaging invariants for Windows and macOS

Two known local-only failures are unrelated and do not appear in CI: test_committed_installer_iss_is_in_sync_with_generator (re-derives identity from the gitignored build/version.toml) and the about_info/build_info channel tests (read the gitignored generated quill/_build_info.py).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Fixes two user-reported document-open regressions by enforcing Quill’s “opened document text must be the document’s own text” contract, improving .docx extraction quality in packaged installs, and bundling the minimum reader dependencies required for PDF and spreadsheet opens—plus a small radio UX fix for Live365 links.

Changes:

• Removed synthetic “extract banners” from multiple readers and moved provenance/quality messaging into intake summary/report.
• Made structured text reads BOM/line-ending aware by routing through read_text_document, and added a python-docx-based .docx text reader.
• Adjusted packaging to bundle a new lean office-text extra (and updated tests/docs), and normalized Live365 links in the Add Custom Station dialog.

Reviewed changes

Copilot reviewed 28 out of 34 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/test_packaging_dependencies.py	Adds assertions for macOS workflow extras and py2app includes for document readers.
tests/unit/scripts/test_build_windows_distribution.py	Verifies Windows bundled dependency groups include office-text and exclude MarkItDown.
tests/unit/io/test_structured.py	Adds regression tests for BOM handling, CRLF preservation, no-banner extracts, and docx fallback chain/metadata.
tests/unit/io/test_pdf.py	Updates PDF formatting test to assert no banner is prepended.
tests/unit/io/test_docx_text.py	New tests for python-docx-based .docx text extraction behavior and ordering.
tests/unit/core/test_live365.py	New tests for Live365 URL/id normalization.
tests/unit/core/test_intake.py	Tests new intake summary/report messaging for low-confidence extraction and missing extraction pack.
scripts/setup_macos.py	Adds py2app includes for docx/pdf/spreadsheet reader modules.
scripts/build_windows_distribution.py	Bundles office-text by default and updates Offline Edition install commentary.
quill/ui/radio/add_station_dialog.py	Normalizes Live365 URLs/ids to playable stream URLs during station creation.
quill/ui/main_frame_speech_downloads.py	Updates optional-component download prompt to reflect MarkItDown’s true size and role.
quill/tools/module_size_budgets.json	Rebases the structured.py budget entry and adds a new rebaseline note.
quill/io/structured.py	Routes structured reads through BOM/line-ending-aware text loader; improves .docx fallback via python-docx; removes extract banners from multiple formats; stamps extraction-pack hint metadata.
quill/io/pdf.py	Removes banner/header from formatted PDF text; leaves provenance in metadata/intake surfaces.
quill/io/docx_text.py	New python-docx .docx text renderer plus raw XML last-resort extract (no banner).
quill/core/radio/live365.py	New pure string-based Live365 station/player link normalization utility.
quill/core/pdf_ocr_install.py	Adds PACK_ASSISTED_SUFFIXES and extraction_pack_missing() helper to support intake hints.
quill/core/optional_components.py	Rewords/renames the optional PDF/Office extraction component to reflect MarkItDown’s role/size.
quill/core/intake.py	Moves low-confidence and “missing extraction pack” messaging into intake summary/report.
pyproject.toml	Introduces office-text optional dependency group and updates comments around pdf-ocr.
docs/user guide/userguide.md	Documents “no synthetic headers” intake contract and updates PDF guidance wording.
docs/tutorials/09-internet-radio.md	Documents Live365 paste/normalization behavior in Add Custom Station.
docs/tutorials/09-internet-radio.html	Regenerated tutorial HTML reflecting Live365 documentation changes.
docs/release notes/release1.0.0.md	Updates release notes to include reader-chain fixes and Live365 behavior.
docs/release notes/release1.0.0.html	Regenerated release-notes HTML reflecting the updated markdown.
docs/Product Requirement Documents and Specifications/QUILL-PRD.md	Updates PRD sections for reader chain/document-text contract and Live365 normalization.
CHANGELOG.md	Adds changelog entries for Live365 normalization and document intake/banner/BOM fixes.
.github/workflows/macos-release.yml	Installs office-text in macOS build/test jobs.

Suppressed comments (1)

quill/io/structured.py:128

• extraction_pack_missing is currently set for any suffix in PACK_ASSISTED_SUFFIXES whenever MarkItDown is not installed. This will also mark reads that already used the bundled baseline readers (e.g., PDFs extracted by pdfplumber/pypdf and spreadsheets extracted by openpyxl), causing the intake summary/report to suggest downloading the 150MB pack even when the shipped readers handled the file as intended.

    # #1279: a fallback read still succeeds, but MarkItDown reads these formats
    # better -- flag it so the intake summary points at the one-click download.
    # Not for a Word file python-docx already read well: that path is good enough
    # that nagging about a 150 MB download would be noise, not help.
    if (