fix: documents open as their own text; ship the readers that make them readable (#1279, #1278) - #1280
fix: documents open as their own text; ship the readers that make them readable (#1279, #1278)#1280accesswatch wants to merge 5 commits into
Conversation
Two reports on the same surface. #1279: "when I load a word document it says # DOCX Extract at the top". #1278: opening a UTF-8-with-BOM .json crashed with JSONDecodeError. No synthetic text in an opened document. The fallback Word, ODT, PPTX and spreadsheet extracts drop their "# <FORMAT> Extract" banners, and format_pdf_document drops the "# PDF Extract / Engine / Quality score / low-confidence" header. All of that provenance already lives in source_metadata; build_intake_summary (the spoken line on open) and build_intake_report (Document Intake Report) now also carry the low-confidence/OCR advice the PDF header used to. Extends EDS-21's earlier removal of "# RTF Extract" to every remaining reader. The AI document-QA path reads format_pdf_document too, so it stops feeding the banner to the model. "# Notebook" and "# SQLite Database:" stay -- those texts are wholly synthesized reports of a non-text source, not a document's own words. Word now reads through python-docx (a base dependency) instead of a hand-rolled word/document.xml walk: headings, list styles and direct w:numPr numbering, and tables as GFM, in document order. New module quill/io/docx_text.py, with the XML walk kept as the last resort. The read chain is MarkItDown when present -> python-docx -> raw XML. BOM-aware structured reads: read_structured_document's text branch goes through quill.io.text.read_text_document, so .json/.xml/.toml/.yaml/.ipynb inherit the contract plain text has had since #648/#649 -- BOM stripped and remembered as utf-8-sig for a byte-for-byte save, original line ending kept, and the #867 cp1252/latin-1 fallback. Before this a BOM crashed the open on 0.9.0 and, after #1228's guard, silently degraded the file to raw text. A read that lands on a reduced reader stamps extraction_pack_missing, which the intake summary/report turn into a one-line pointer at the optional download -- suppressed when python-docx already read the file well. GATE-11: structured.py 776 -> 787 (Word extraction moved out to the new module), rebaselined with the reasoning in module_size_budgets.json. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#1279's root cause was packaging, not just the banner: a packaged install never had MarkItDown (an optional download), so every Word open fell to the weakest reader QUILL has, and a fresh install had no PDF text reader and no .xlsx reader at all. Bundling the whole pdf-ocr extra was measured, not assumed. MarkItDown hard- requires magika, which hard-requires onnxruntime and numpy, and its xlsx/xls converters require pandas: about 150 MB installed, not the "~30 MB" the component dialog claimed. So the lean half ships and the heavy half stays optional: - New `office-text` extra (pdfplumber, pypdf, openpyxl, ~16 MB), bundled via DEFAULT_BUNDLED_DEPENDENCY_GROUPS on Windows and via setup_macos.py's includes + macos-release.yml's install specs on macOS. Word reads through python-docx, already a base dependency. - macOS needed the includes because every reader is imported function-locally and py2app's tracer misses them -- the .app had no docx, no PDF reader and no openpyxl. No `packages` entry is needed: build_macos.sh already lifts zipped packages carrying native binaries out of the zip before signing. - MarkItDown stays a one-click download, now labeled with its real size and described as the enhancement it is rather than something you need. A test asserts it is NOT bundled, so a future re-add is a deliberate decision. - The offline edition no longer installs the pdf-ocr packages separately; every build now gets office-text from the base bundle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The link a listener has for a Live365 station is almost never the stream: it is the station page (live365.com/station/<slug>-a25891) or the web player (player.live365.com/a25891), both HTML. Pasted into Add Custom Station those saved a station that could never play, with nothing to explain why. normalize_live365 rewrites any Live365 station/player link -- or a bare a##### id -- to the canonical https://streaming.live365.com/<id> stream, and the dialog reports that it did. Deliberately a pure string transform: the station id is already present in every such link, so there is no network call, no scraping, and no use of Live365's auth-gated directory API -- nothing to gate in Safe Mode and no new egress site. A bare station slug with no id would need that API and is left untouched, as is any non-Live365 URL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Universal Audio Converter becomes a full family product alongside Quill Radio and Quill Cast, not only a window inside QUILL: - standalone/converter/ carries the wrapper: the launcher package (anchoring QUILL_APP_ROOT so the bundled ffmpeg next to the exe is found, and portable mode keeps data on the stick), the PyInstaller spec, project metadata, and the plum tile icon plus the script that regenerates it. - build_portable.py gains a `converter` product: ffmpeg staged, no speech engines and no mpv, because the whole job is FFmpeg conversion. - QuillConverter.exe joins storage_mode's portable-evidence allowlist so a portable bundle routes data to its own data/ folder like every sibling. Also fixes the project's own test helper: build_portable defines a @DataClass, and dataclasses resolves string annotations through sys.modules[cls.__module__], so loading it via importlib without registering the module raised AttributeError before any assertion ran. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers everything in this branch across all four surfaces: - CHANGELOG 1.0.0: the two reader fixes (#1279/#1278), the Word/python-docx and bundled-reader improvements, Live365 link support, and Quill Converter as its own product. - Release notes 1.0.0: reading-improvement entries, a Live365 paragraph in the radio section, and two Fixes bullets. - User guide: "an imported document opens with its own text and nothing else" under Document Intake, Live365 handling under Adding your own stations, and a note that the PDF-reader pointer only applies to pip or source installs. - PRD: a new "Reader chain and the document-text contract (#1279 / #1278)" section, the corrected docx_read_engine chain and Tier-1 table row, the Live365 normalization contract in the radio section, and Quill Converter in the QuillVille family app list. - Tutorial 09 (internet radio): Live365 links in Add Custom Station. HTML/EPUB artifacts regenerated with the same pandoc invocations CI uses; scripts/check_docs_artifacts.py passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Fixes document-import regressions where extracted content was polluted with synthetic “Extract” banners and where BOM-prefixed structured files could crash or degrade to raw text. It also improves packaged-reader availability (Windows/macOS), adds a stronger .docx fallback reader via python-docx, and includes two previously-uncommitted features: Live365 URL normalization and the standalone Quill Converter wrapper.
Changes:
- Remove synthetic extract banners from fallback readers and move provenance/quality warnings into intake summary/report metadata.
- Make structured-text opens BOM-aware (via the shared text-reader contract) and preserve original line endings/encoding behavior.
- Bundle a lean
office-textreader extra in packaged builds; add Live365 normalization; add standalone Converter wrapper + tests.
Reviewed changes
Copilot reviewed 40 out of 47 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_packaging_dependencies.py | Extends packaging/workflow assertions for new office-text bundling and macOS includes. |
| tests/unit/scripts/test_standalone_converter.py | Adds unit tests ensuring the Converter product is wired into portable build + wrapper assets exist. |
| tests/unit/scripts/test_build_windows_distribution.py | Adds assertions that Windows bundling includes office-text readers and excludes MarkItDown pack. |
| tests/unit/io/test_structured.py | Adds tests for BOM parsing/CRLF preservation and banner-free Office/PDF fallback behavior. |
| tests/unit/io/test_pdf.py | Updates PDF formatting expectations to ensure no synthetic banner is prepended. |
| tests/unit/io/test_docx_text.py | Adds coverage for real .docx extraction through python-docx (headings/lists/tables/order, no banner). |
| tests/unit/core/test_storage_mode.py | Updates portable-evidence allowlist tests for QuillConverter.exe. |
| tests/unit/core/test_live365.py | Adds test coverage for pure string-based Live365 URL normalization. |
| tests/unit/core/test_intake.py | Verifies intake summary/report reflect low-confidence extraction and missing-pack hints (not prepended to document text). |
| standalone/studio/scripts/build_portable.py | Adds converter as a build product with FFmpeg staged and minimal dependencies. |
| standalone/converter/run-quill-converter.bat | Adds a dev launcher batch file for the standalone converter wrapper. |
| standalone/converter/README.md | Documents running/building the standalone Converter wrapper. |
| standalone/converter/quill-converter.spec | Adds PyInstaller spec for Converter onedir build with targeted excludes. |
| standalone/converter/quill_converter/main.py | Adds module entrypoint for the converter wrapper package. |
| standalone/converter/quill_converter/init.py | Sets QUILL_APP_ROOT/portable env in frozen builds; hands off to quill.apps.converter. |
| standalone/converter/pyproject.toml | Declares the wrapper package metadata and dependency on quill[ui]. |
| standalone/converter/launcher.py | Adds PyInstaller entrypoint delegating to wrapper main(). |
| standalone/converter/assets/make_quill_converter_icon.py | Adds regenerable icon generator producing a multi-resolution .ico. |
| scripts/setup_macos.py | Ensures py2app includes document reader modules that are imported lazily at runtime. |
| scripts/build_windows_distribution.py | Adds office-text to default bundled dependency groups; updates offline-edition notes. |
| quill/ui/radio/add_station_dialog.py | Normalizes Live365 links/ids to canonical stream URLs in the Add Custom Station dialog. |
| quill/ui/main_frame_speech_downloads.py | Updates optional-component download prompt text/size claims for MarkItDown pack. |
| quill/tools/module_size_budgets.json | Updates budgets/rebaseline note for structured.py growth and extraction changes. |
| quill/io/structured.py | Switches .docx fallback to python-docx renderer + XML last resort; structured text reads go through read_text_document; removes banners from Office fallback extractors; stamps missing-pack metadata. |
| quill/io/pdf.py | Removes synthetic PDF banner and returns extracted text only; delegates provenance to metadata/intake. |
| quill/io/docx_text.py | New wx-free .docx text renderer using python-docx, plus XML-based last-resort extractor. |
| quill/core/storage_mode.py | Adds QuillConverter.exe to portable-evidence allowlist. |
| quill/core/radio/live365.py | New pure string transformer for Live365 station/player URLs and ids. |
| quill/core/pdf_ocr_install.py | Adds PACK_ASSISTED_SUFFIXES and extraction_pack_missing() predicate. |
| quill/core/optional_components.py | Rewords MarkItDown pack as optional enhancement; corrects size disclosure. |
| quill/core/intake.py | Adds low-confidence and missing-pack hints to intake summary/report (instead of document text). |
| pyproject.toml | Adds office-text extra and updates pdf-ocr/reader versions. |
| docs/user guide/userguide.md | Documents the “no synthetic text in opened documents” intake contract and related guidance. |
| docs/tutorials/09-internet-radio.md | Documents Live365 link normalization in Add Custom Station. |
| docs/tutorials/09-internet-radio.html | Regenerated HTML tutorial artifact reflecting Live365 doc changes. |
| docs/release notes/release1.0.0.md | Adds release-note entries for improved readers, banner removal, BOM handling, and Live365 fix. |
| docs/release notes/release1.0.0.html | Regenerated HTML release notes reflecting doc changes. |
| docs/Product Requirement Documents and Specifications/QUILL-PRD.md | Updates PRD sections describing reader chains, Live365 normalization, and Converter wrapper. |
| CHANGELOG.md | Adds changelog entries for Converter standalone product, Live365 normalization, banner removal, and BOM fix. |
| .github/workflows/macos-release.yml | Ensures macOS build/test jobs install office-text extra so readers can be bundled. |
| # #1279: a fallback read still succeeds, but MarkItDown reads these formats | ||
| # better -- flag it so the intake summary points at the one-click download. | ||
| # Not for a Word file python-docx already read well: that path is good enough | ||
| # that nagging about a 150 MB download would be noise, not help. | ||
| if ( | ||
| suffix in PACK_ASSISTED_SUFFIXES | ||
| and metadata.get("engine") != "python-docx" | ||
| and extraction_pack_missing() | ||
| ): | ||
| metadata["extraction_pack_missing"] = True |
| # The whole app lives in the quill package (quill.apps.converter); this | ||
| # project is the thin product wrapper: entry point, icon, installer. | ||
| dependencies = ["quill[ui] @ git+https://github.com/Community-Access/quill.git@main"] |
| # PDF/Office text extraction is no longer listed here: since #1279 the pdf-ocr | ||
| # extra is in DEFAULT_BUNDLED_DEPENDENCY_GROUPS, so *every* build installs it | ||
| # above, not just the Offline Edition. |
| @@ -1,5 +1,5 @@ | |||
| { | |||
| "_rebaseline_2026_07_30_issue_and_feature_sweep": "Open-issue sweep + feature requests (see xc.md). Bug fixes: io/structured.py 759->776 (+17: _format_structured_text dispatch that falls back to raw text on a malformed JSON/TOML/XML/CSV/notebook instead of crashing the open flow, #1228); ui/ai_setup_wizard.py 954->1015 (+61: _alive() guard + guarded async callbacks so a closed AI-setup wizard's freed StaticText is never touched, #1230, and a ListModels fallback in _probe_provider so a valid Gemini key is accepted even when the default probe model 404s, #1231 -- mostly explanatory comments); ui/main_frame.py 19606->19619 (+13: Notifications dialog now copies only on explicit Ctrl+C / double-click / button instead of on every EVT_LISTBOX selection change, #1247). Feature #1246 (Ctrl+T new document tab -- reuses new_file, no core-path change): wiring only -- core/keymap.py 1015->1018 (window.new_document_tab default chord), main_frame_commands.py 2513->2522 (register + id-map), main_frame_menu.py 3328->3333 (id + Window-menu item). Feature #1249/#1253 (Radio export-favorites-to-playlist + output-device quick picker): apps/radio.py 2055->2069 -- menu Append/Bind wiring only; all logic lives in NEW under-cap modules (core/radio/playlist_export.py, ui/radio/playlist_export_ui.py, ui/radio/output_device_ui.py). Menu/command/keymap registrations and the notifications dialog cannot be extracted. One-time acknowledgment of the grown state, not license for further growth.", | |||
| "_rebaseline_2026_08_01_extract_banners_and_bom": "User-reported bug fixes #1279/#1278. io/structured.py 776->787 (+11): (a) #1278 -- the structured text branch now reads through quill.io.text.read_text_document so a UTF-8 BOM is stripped (and remembered as utf-8-sig for a round-trip save), the original line ending survives, and the cp1252/latin-1 fallback applies; that needs its own Document construction because the encoding/line ending no longer match the binary-format tail. (b) #1279 -- the fallback office extracts drop their synthetic banners, and a degraded read stamps extraction_pack_missing so the intake summary can point at Help > Download Optional Components (suffix set + predicate live in core/pdf_ocr_install.py). Word extraction moved OUT to the NEW module io/docx_text.py (python-docx rendering plus the old raw-XML walk as last resort), which is why the net growth is smaller than the change. One-time acknowledgment of the grown state, not license for further growth.", "_rebaseline_2026_07_30_issue_and_feature_sweep": "Open-issue sweep + feature requests (see xc.md). Bug fixes: io/structured.py 759->776 (+17: _format_structured_text dispatch that falls back to raw text on a malformed JSON/TOML/XML/CSV/notebook instead of crashing the open flow, #1228); ui/ai_setup_wizard.py 954->1015 (+61: _alive() guard + guarded async callbacks so a closed AI-setup wizard's freed StaticText is never touched, #1230, and a ListModels fallback in _probe_provider so a valid Gemini key is accepted even when the default probe model 404s, #1231 -- mostly explanatory comments); ui/main_frame.py 19606->19619 (+13: Notifications dialog now copies only on explicit Ctrl+C / double-click / button instead of on every EVT_LISTBOX selection change, #1247). Feature #1246 (Ctrl+T new document tab -- reuses new_file, no core-path change): wiring only -- core/keymap.py 1015->1018 (window.new_document_tab default chord), main_frame_commands.py 2513->2522 (register + id-map), main_frame_menu.py 3328->3333 (id + Window-menu item). Feature #1249/#1253 (Radio export-favorites-to-playlist + output-device quick picker): apps/radio.py 2055->2069 -- menu Append/Bind wiring only; all logic lives in NEW under-cap modules (core/radio/playlist_export.py, ui/radio/playlist_export_ui.py, ui/radio/output_device_ui.py). Menu/command/keymap registrations and the notifications dialog cannot be extracted. One-time acknowledgment of the grown state, not license for further growth.", | |||
| # Optional Components (quill/core/pdf_ocr_install.py). Shipped builds bundle the | ||
| # lean `office-text` extra below instead of this one -- see the note there. | ||
| pdf-ocr = [ | ||
| "markitdown[docx,pptx,xlsx,xls,pdf]>=0.1.6", | ||
| "pdfplumber>=0.11.10", | ||
| "pypdf>=6.14.2", | ||
| ] |
|
Superseded by #1281. While this was in flight, the standalone Quill Converter landed on main independently (#1275), so this branch's converter commit duplicated it. #1281 is the same work rebased onto current main with that commit dropped -- the two reader fixes (#1279, #1278), the packaging change, Live365 link normalization, and the docs -- and keeps only the converter's documentation. |
Closes #1279. Closes #1278.
Two user reports on the same surface turned out to share a root cause in packaging, so this branch carries the fixes, the packaging change they imply, and the two features that were sitting uncommitted alongside them.
The reports
#1279 — "when I load a word document, it says
# DOCX Extractat the top." Two defects behind it:# <FORMAT> Extract, and PDFs opened with a four-line# PDF Extract/Engine:/Quality score:/ low-confidence block — text that is not in the user's file, read out first by a screen reader every time, and (for PDFs) fed to the model on the AI document-QA path. That provenance already lives insource_metadata; it is now reported bybuild_intake_summary(the spoken/status line on open) andbuild_intake_report(Document Intake Report), which gained the low-confidence/OCR advice the PDF header used to carry. This extends EDS-21's earlier removal of# RTF Extractto every remaining reader.# Notebookand# SQLite Database:stay — those texts are wholly synthesized reports of a non-text source, not a document's own words._format_docx's hand-rolledword/document.xmlwalk: one flat line per paragraph, headings/lists/tables discarded.python-docxis a base dependency, so Word now renders through it (newquill/io/docx_text.py: heading levels, list styles and directw:numPrnumbering, tables as GFM, in document order), with the XML walk kept as the last resort. Read chain: MarkItDown when present → python-docx → raw XML.#1278 —
JSONDecodeError: Unexpected UTF-8 BOMopening a.json.read_structured_document's text branch used a barepath.read_text(), so a BOM survived into the parser. It now reads throughquill.io.text.read_text_document, giving.json/.xml/.toml/.yaml/.ipynbthe contract plain text has had since #648/#649: BOM stripped and remembered asutf-8-sig(a save re-adds it byte-for-byte), original line ending kept (CRLF was previously flattened here), and the #867 cp1252/latin-1 fallback. On 0.9.0 this crashed the open; after #1228's guard it silently degraded the file to unformatted raw text.Packaging: measured, not assumed
Bundling the whole
pdf-ocrextra looked like the obvious fix. Measuring it first changed the answer: MarkItDown hard-requiresmagika, which hard-requiresonnxruntimeandnumpy, and its xlsx/xls converters requirepandas— about 150 MB installed, not the "~30 MB" the component dialog claimed.So the lean half ships and the heavy half stays optional:
office-textextra —pdfplumber,pypdf,openpyxl(~16 MB) — bundled throughDEFAULT_BUNDLED_DEPENDENCY_GROUPSon Windows and throughsetup_macos.py'sincludes+macos-release.yml's install specs on macOS. Without pdfplumber/pypdf there is no PDF text reader at all; without openpyxl an.xlsxcannot open..apphad nodocx, no PDF reader, and noopenpyxl. Nopackagesentry is needed;build_macos.shalready lifts zipped packages carrying native binaries out of the zip before signing.extraction_pack_missing, which the intake summary and report turn into a one-line pointer at the download — suppressed when python-docx already read the file well, so it never nags about 150 MB for a document that came through fine.Also on this branch (previously uncommitted)
normalize_live365rewrites any station/player link — or a barea#####id — to the canonical stream URL and the dialog says it did. Pure string transform: no network call, no scraping, no use of Live365's auth-gated API, nothing to gate in Safe Mode.standalone/converter/gains the missing launcher package and the tile icon (plus the script that regenerates it),build_portable.pygains aconverterproduct (ffmpeg staged; no speech engines, no mpv), andQuillConverter.exejoinsstorage_mode's portable-evidence allowlist. Also fixes the project's own test helper:build_portabledefines a@dataclass, and dataclasses resolves string annotations throughsys.modules[cls.__module__], so loading it via importlib without registering the module raisedAttributeErrorbefore any assertion ran.Docs
All four surfaces, per the incremental-docs rule: CHANGELOG 1.0.0, release notes 1.0.0, the user guide (Document Intake, Adding your own stations, the PDF-reader pointer), the PRD (a new "Reader chain and the document-text contract" section, the corrected
docx_read_enginechain and Tier-1 table row, the Live365 contract, Quill Converter in the family app list), and tutorial 09. HTML/EPUB regenerated with the same pandoc invocations CI uses;scripts/check_docs_artifacts.pypasses.Verification
pytest tests/unit tests/accessibility -n 4 --dist=loadfile(the PR CI invocation) — see the CI run on this PRoptional_components.pyandmain_frame_speech_downloads.pywere at their ceilings and the wording was condensed to fit rather than rebaselined), banned patterns, error codes, dialog inventory — all greenruff check/ruff format --checkclean;mypy quill\core quill\ioclean.docxfiles built with python-docx and read back; BOM'd JSON/XML; CRLF preservation; no-banner assertions per format; theextraction_pack_missingflag both ways; bundled-vs-not-bundled packaging invariants for Windows and macOSTwo known local-only failures are unrelated and do not appear in CI:
test_committed_installer_iss_is_in_sync_with_generator(re-derives identity from the gitignoredbuild/version.toml) and theabout_info/build_infochannel tests (read the gitignored generatedquill/_build_info.py).🤖 Generated with Claude Code