Skip to content

fix: documents open as their own text; ship the readers that make them readable (#1279, #1278) - #1280

Closed
accesswatch wants to merge 5 commits into
mainfrom
fix/document-readers-and-office-packaging
Closed

fix: documents open as their own text; ship the readers that make them readable (#1279, #1278)#1280
accesswatch wants to merge 5 commits into
mainfrom
fix/document-readers-and-office-packaging

Conversation

@accesswatch

Copy link
Copy Markdown
Contributor

Closes #1279. Closes #1278.

Two user reports on the same surface turned out to share a root cause in packaging, so this branch carries the fixes, the packaging change they imply, and the two features that were sitting uncommitted alongside them.

The reports

#1279 — "when I load a word document, it says # DOCX Extract at the top." Two defects behind it:

  1. Every fallback extract prepended a synthetic banner. Word, ODT, PowerPoint, and spreadsheet extracts opened with # <FORMAT> Extract, and PDFs opened with a four-line # PDF Extract / Engine: / Quality score: / low-confidence block — text that is not in the user's file, read out first by a screen reader every time, and (for PDFs) fed to the model on the AI document-QA path. That provenance already lives in source_metadata; it is now reported by build_intake_summary (the spoken/status line on open) and build_intake_report (Document Intake Report), which gained the low-confidence/OCR advice the PDF header used to carry. This extends EDS-21's earlier removal of # RTF Extract to every remaining reader. # Notebook and # SQLite Database: stay — those texts are wholly synthesized reports of a non-text source, not a document's own words.
  2. The reporter was on the weakest reader QUILL has. MarkItDown is an optional download, so a packaged install always fell to _format_docx's hand-rolled word/document.xml walk: one flat line per paragraph, headings/lists/tables discarded. python-docx is a base dependency, so Word now renders through it (new quill/io/docx_text.py: heading levels, list styles and direct w:numPr numbering, tables as GFM, in document order), with the XML walk kept as the last resort. Read chain: MarkItDown when present → python-docx → raw XML.

#1278JSONDecodeError: Unexpected UTF-8 BOM opening a .json. read_structured_document's text branch used a bare path.read_text(), so a BOM survived into the parser. It now reads through quill.io.text.read_text_document, giving .json/.xml/.toml/.yaml/.ipynb the contract plain text has had since #648/#649: BOM stripped and remembered as utf-8-sig (a save re-adds it byte-for-byte), original line ending kept (CRLF was previously flattened here), and the #867 cp1252/latin-1 fallback. On 0.9.0 this crashed the open; after #1228's guard it silently degraded the file to unformatted raw text.

Packaging: measured, not assumed

Bundling the whole pdf-ocr extra looked like the obvious fix. Measuring it first changed the answer: MarkItDown hard-requires magika, which hard-requires onnxruntime and numpy, and its xlsx/xls converters require pandas — about 150 MB installed, not the "~30 MB" the component dialog claimed.

So the lean half ships and the heavy half stays optional:

  • New office-text extra — pdfplumber, pypdf, openpyxl (~16 MB) — bundled through DEFAULT_BUNDLED_DEPENDENCY_GROUPS on Windows and through setup_macos.py's includes + macos-release.yml's install specs on macOS. Without pdfplumber/pypdf there is no PDF text reader at all; without openpyxl an .xlsx cannot open.
  • macOS was worse than Windows: every reader is imported function-locally, so py2app's tracer bundled none of them — the .app had no docx, no PDF reader, and no openpyxl. No packages entry is needed; build_macos.sh already lifts zipped packages carrying native binaries out of the zip before signing.
  • MarkItDown remains a one-click download, relabeled with its true size and described as the enhancement it is. A test asserts it is not in the bundled groups, so a future re-add is a deliberate decision rather than a silent 150 MB installer.
  • A read that lands on a reduced reader stamps extraction_pack_missing, which the intake summary and report turn into a one-line pointer at the download — suppressed when python-docx already read the file well, so it never nags about 150 MB for a document that came through fine.

Also on this branch (previously uncommitted)

  • Live365 link normalization. A Live365 station page or web-player link is HTML, not a stream, so pasting one into Add Custom Station saved a station that could never play. normalize_live365 rewrites any station/player link — or a bare a##### id — to the canonical stream URL and the dialog says it did. Pure string transform: no network call, no scraping, no use of Live365's auth-gated API, nothing to gate in Safe Mode.
  • Quill Converter as a standalone product (Audio Studio: universal audio converter — mixed file/folder queue, basic + advanced modes, rich DSP, background/tray operation #1255). standalone/converter/ gains the missing launcher package and the tile icon (plus the script that regenerates it), build_portable.py gains a converter product (ffmpeg staged; no speech engines, no mpv), and QuillConverter.exe joins storage_mode's portable-evidence allowlist. Also fixes the project's own test helper: build_portable defines a @dataclass, and dataclasses resolves string annotations through sys.modules[cls.__module__], so loading it via importlib without registering the module raised AttributeError before any assertion ran.

Docs

All four surfaces, per the incremental-docs rule: CHANGELOG 1.0.0, release notes 1.0.0, the user guide (Document Intake, Adding your own stations, the PDF-reader pointer), the PRD (a new "Reader chain and the document-text contract" section, the corrected docx_read_engine chain and Tier-1 table row, the Live365 contract, Quill Converter in the family app list), and tutorial 09. HTML/EPUB regenerated with the same pandoc invocations CI uses; scripts/check_docs_artifacts.py passes.

Verification

  • pytest tests/unit tests/accessibility -n 4 --dist=loadfile (the PR CI invocation) — see the CI run on this PR
  • Gates: GATE-11 (structured.py 776 → 787 with the reasoning recorded; optional_components.py and main_frame_speech_downloads.py were at their ceilings and the wording was condensed to fit rather than rebaselined), banned patterns, error codes, dialog inventory — all green
  • ruff check / ruff format --check clean; mypy quill\core quill\io clean
  • New tests: real .docx files built with python-docx and read back; BOM'd JSON/XML; CRLF preservation; no-banner assertions per format; the extraction_pack_missing flag both ways; bundled-vs-not-bundled packaging invariants for Windows and macOS

Two known local-only failures are unrelated and do not appear in CI: test_committed_installer_iss_is_in_sync_with_generator (re-derives identity from the gitignored build/version.toml) and the about_info/build_info channel tests (read the gitignored generated quill/_build_info.py).

🤖 Generated with Claude Code

Jeff Bishop and others added 5 commits August 1, 2026 13:41
Two reports on the same surface. #1279: "when I load a word document it
says # DOCX Extract at the top". #1278: opening a UTF-8-with-BOM .json
crashed with JSONDecodeError.

No synthetic text in an opened document. The fallback Word, ODT, PPTX and
spreadsheet extracts drop their "# <FORMAT> Extract" banners, and
format_pdf_document drops the "# PDF Extract / Engine / Quality score /
low-confidence" header. All of that provenance already lives in
source_metadata; build_intake_summary (the spoken line on open) and
build_intake_report (Document Intake Report) now also carry the
low-confidence/OCR advice the PDF header used to. Extends EDS-21's earlier
removal of "# RTF Extract" to every remaining reader. The AI document-QA
path reads format_pdf_document too, so it stops feeding the banner to the
model. "# Notebook" and "# SQLite Database:" stay -- those texts are wholly
synthesized reports of a non-text source, not a document's own words.

Word now reads through python-docx (a base dependency) instead of a
hand-rolled word/document.xml walk: headings, list styles and direct
w:numPr numbering, and tables as GFM, in document order. New module
quill/io/docx_text.py, with the XML walk kept as the last resort. The read
chain is MarkItDown when present -> python-docx -> raw XML.

BOM-aware structured reads: read_structured_document's text branch goes
through quill.io.text.read_text_document, so .json/.xml/.toml/.yaml/.ipynb
inherit the contract plain text has had since #648/#649 -- BOM stripped and
remembered as utf-8-sig for a byte-for-byte save, original line ending kept,
and the #867 cp1252/latin-1 fallback. Before this a BOM crashed the open on
0.9.0 and, after #1228's guard, silently degraded the file to raw text.

A read that lands on a reduced reader stamps extraction_pack_missing, which
the intake summary/report turn into a one-line pointer at the optional
download -- suppressed when python-docx already read the file well.

GATE-11: structured.py 776 -> 787 (Word extraction moved out to the new
module), rebaselined with the reasoning in module_size_budgets.json.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#1279's root cause was packaging, not just the banner: a packaged install
never had MarkItDown (an optional download), so every Word open fell to the
weakest reader QUILL has, and a fresh install had no PDF text reader and no
.xlsx reader at all.

Bundling the whole pdf-ocr extra was measured, not assumed. MarkItDown hard-
requires magika, which hard-requires onnxruntime and numpy, and its xlsx/xls
converters require pandas: about 150 MB installed, not the "~30 MB" the
component dialog claimed. So the lean half ships and the heavy half stays
optional:

- New `office-text` extra (pdfplumber, pypdf, openpyxl, ~16 MB), bundled via
  DEFAULT_BUNDLED_DEPENDENCY_GROUPS on Windows and via setup_macos.py's
  includes + macos-release.yml's install specs on macOS. Word reads through
  python-docx, already a base dependency.
- macOS needed the includes because every reader is imported function-locally
  and py2app's tracer misses them -- the .app had no docx, no PDF reader and
  no openpyxl. No `packages` entry is needed: build_macos.sh already lifts
  zipped packages carrying native binaries out of the zip before signing.
- MarkItDown stays a one-click download, now labeled with its real size and
  described as the enhancement it is rather than something you need. A test
  asserts it is NOT bundled, so a future re-add is a deliberate decision.
- The offline edition no longer installs the pdf-ocr packages separately;
  every build now gets office-text from the base bundle.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The link a listener has for a Live365 station is almost never the stream:
it is the station page (live365.com/station/<slug>-a25891) or the web player
(player.live365.com/a25891), both HTML. Pasted into Add Custom Station those
saved a station that could never play, with nothing to explain why.

normalize_live365 rewrites any Live365 station/player link -- or a bare
a##### id -- to the canonical https://streaming.live365.com/<id> stream, and
the dialog reports that it did. Deliberately a pure string transform: the
station id is already present in every such link, so there is no network
call, no scraping, and no use of Live365's auth-gated directory API --
nothing to gate in Safe Mode and no new egress site. A bare station slug
with no id would need that API and is left untouched, as is any non-Live365
URL.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Universal Audio Converter becomes a full family product alongside Quill
Radio and Quill Cast, not only a window inside QUILL:

- standalone/converter/ carries the wrapper: the launcher package (anchoring
  QUILL_APP_ROOT so the bundled ffmpeg next to the exe is found, and portable
  mode keeps data on the stick), the PyInstaller spec, project metadata, and
  the plum tile icon plus the script that regenerates it.
- build_portable.py gains a `converter` product: ffmpeg staged, no speech
  engines and no mpv, because the whole job is FFmpeg conversion.
- QuillConverter.exe joins storage_mode's portable-evidence allowlist so a
  portable bundle routes data to its own data/ folder like every sibling.

Also fixes the project's own test helper: build_portable defines a
@DataClass, and dataclasses resolves string annotations through
sys.modules[cls.__module__], so loading it via importlib without registering
the module raised AttributeError before any assertion ran.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers everything in this branch across all four surfaces:

- CHANGELOG 1.0.0: the two reader fixes (#1279/#1278), the Word/python-docx
  and bundled-reader improvements, Live365 link support, and Quill Converter
  as its own product.
- Release notes 1.0.0: reading-improvement entries, a Live365 paragraph in
  the radio section, and two Fixes bullets.
- User guide: "an imported document opens with its own text and nothing
  else" under Document Intake, Live365 handling under Adding your own
  stations, and a note that the PDF-reader pointer only applies to pip or
  source installs.
- PRD: a new "Reader chain and the document-text contract (#1279 / #1278)"
  section, the corrected docx_read_engine chain and Tier-1 table row, the
  Live365 normalization contract in the radio section, and Quill Converter
  in the QuillVille family app list.
- Tutorial 09 (internet radio): Live365 links in Add Custom Station.

HTML/EPUB artifacts regenerated with the same pandoc invocations CI uses;
scripts/check_docs_artifacts.py passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@accesswatch
accesswatch requested a review from kellylford as a code owner August 1, 2026 20:46
Copilot AI review requested due to automatic review settings August 1, 2026 20:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes document-import regressions where extracted content was polluted with synthetic “Extract” banners and where BOM-prefixed structured files could crash or degrade to raw text. It also improves packaged-reader availability (Windows/macOS), adds a stronger .docx fallback reader via python-docx, and includes two previously-uncommitted features: Live365 URL normalization and the standalone Quill Converter wrapper.

Changes:

  • Remove synthetic extract banners from fallback readers and move provenance/quality warnings into intake summary/report metadata.
  • Make structured-text opens BOM-aware (via the shared text-reader contract) and preserve original line endings/encoding behavior.
  • Bundle a lean office-text reader extra in packaged builds; add Live365 normalization; add standalone Converter wrapper + tests.

Reviewed changes

Copilot reviewed 40 out of 47 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tests/unit/test_packaging_dependencies.py Extends packaging/workflow assertions for new office-text bundling and macOS includes.
tests/unit/scripts/test_standalone_converter.py Adds unit tests ensuring the Converter product is wired into portable build + wrapper assets exist.
tests/unit/scripts/test_build_windows_distribution.py Adds assertions that Windows bundling includes office-text readers and excludes MarkItDown pack.
tests/unit/io/test_structured.py Adds tests for BOM parsing/CRLF preservation and banner-free Office/PDF fallback behavior.
tests/unit/io/test_pdf.py Updates PDF formatting expectations to ensure no synthetic banner is prepended.
tests/unit/io/test_docx_text.py Adds coverage for real .docx extraction through python-docx (headings/lists/tables/order, no banner).
tests/unit/core/test_storage_mode.py Updates portable-evidence allowlist tests for QuillConverter.exe.
tests/unit/core/test_live365.py Adds test coverage for pure string-based Live365 URL normalization.
tests/unit/core/test_intake.py Verifies intake summary/report reflect low-confidence extraction and missing-pack hints (not prepended to document text).
standalone/studio/scripts/build_portable.py Adds converter as a build product with FFmpeg staged and minimal dependencies.
standalone/converter/run-quill-converter.bat Adds a dev launcher batch file for the standalone converter wrapper.
standalone/converter/README.md Documents running/building the standalone Converter wrapper.
standalone/converter/quill-converter.spec Adds PyInstaller spec for Converter onedir build with targeted excludes.
standalone/converter/quill_converter/main.py Adds module entrypoint for the converter wrapper package.
standalone/converter/quill_converter/init.py Sets QUILL_APP_ROOT/portable env in frozen builds; hands off to quill.apps.converter.
standalone/converter/pyproject.toml Declares the wrapper package metadata and dependency on quill[ui].
standalone/converter/launcher.py Adds PyInstaller entrypoint delegating to wrapper main().
standalone/converter/assets/make_quill_converter_icon.py Adds regenerable icon generator producing a multi-resolution .ico.
scripts/setup_macos.py Ensures py2app includes document reader modules that are imported lazily at runtime.
scripts/build_windows_distribution.py Adds office-text to default bundled dependency groups; updates offline-edition notes.
quill/ui/radio/add_station_dialog.py Normalizes Live365 links/ids to canonical stream URLs in the Add Custom Station dialog.
quill/ui/main_frame_speech_downloads.py Updates optional-component download prompt text/size claims for MarkItDown pack.
quill/tools/module_size_budgets.json Updates budgets/rebaseline note for structured.py growth and extraction changes.
quill/io/structured.py Switches .docx fallback to python-docx renderer + XML last resort; structured text reads go through read_text_document; removes banners from Office fallback extractors; stamps missing-pack metadata.
quill/io/pdf.py Removes synthetic PDF banner and returns extracted text only; delegates provenance to metadata/intake.
quill/io/docx_text.py New wx-free .docx text renderer using python-docx, plus XML-based last-resort extractor.
quill/core/storage_mode.py Adds QuillConverter.exe to portable-evidence allowlist.
quill/core/radio/live365.py New pure string transformer for Live365 station/player URLs and ids.
quill/core/pdf_ocr_install.py Adds PACK_ASSISTED_SUFFIXES and extraction_pack_missing() predicate.
quill/core/optional_components.py Rewords MarkItDown pack as optional enhancement; corrects size disclosure.
quill/core/intake.py Adds low-confidence and missing-pack hints to intake summary/report (instead of document text).
pyproject.toml Adds office-text extra and updates pdf-ocr/reader versions.
docs/user guide/userguide.md Documents the “no synthetic text in opened documents” intake contract and related guidance.
docs/tutorials/09-internet-radio.md Documents Live365 link normalization in Add Custom Station.
docs/tutorials/09-internet-radio.html Regenerated HTML tutorial artifact reflecting Live365 doc changes.
docs/release notes/release1.0.0.md Adds release-note entries for improved readers, banner removal, BOM handling, and Live365 fix.
docs/release notes/release1.0.0.html Regenerated HTML release notes reflecting doc changes.
docs/Product Requirement Documents and Specifications/QUILL-PRD.md Updates PRD sections describing reader chains, Live365 normalization, and Converter wrapper.
CHANGELOG.md Adds changelog entries for Converter standalone product, Live365 normalization, banner removal, and BOM fix.
.github/workflows/macos-release.yml Ensures macOS build/test jobs install office-text extra so readers can be bundled.

Comment thread quill/io/structured.py
Comment on lines +124 to +133
# #1279: a fallback read still succeeds, but MarkItDown reads these formats
# better -- flag it so the intake summary points at the one-click download.
# Not for a Word file python-docx already read well: that path is good enough
# that nagging about a 150 MB download would be noise, not help.
if (
suffix in PACK_ASSISTED_SUFFIXES
and metadata.get("engine") != "python-docx"
and extraction_pack_missing()
):
metadata["extraction_pack_missing"] = True
Comment on lines +21 to +23
# The whole app lives in the quill package (quill.apps.converter); this
# project is the thin product wrapper: entry point, icon, installer.
dependencies = ["quill[ui] @ git+https://github.com/Community-Access/quill.git@main"]
Comment on lines +1650 to +1652
# PDF/Office text extraction is no longer listed here: since #1279 the pdf-ocr
# extra is in DEFAULT_BUNDLED_DEPENDENCY_GROUPS, so *every* build installs it
# above, not just the Offline Edition.
@@ -1,5 +1,5 @@
{
"_rebaseline_2026_07_30_issue_and_feature_sweep": "Open-issue sweep + feature requests (see xc.md). Bug fixes: io/structured.py 759->776 (+17: _format_structured_text dispatch that falls back to raw text on a malformed JSON/TOML/XML/CSV/notebook instead of crashing the open flow, #1228); ui/ai_setup_wizard.py 954->1015 (+61: _alive() guard + guarded async callbacks so a closed AI-setup wizard's freed StaticText is never touched, #1230, and a ListModels fallback in _probe_provider so a valid Gemini key is accepted even when the default probe model 404s, #1231 -- mostly explanatory comments); ui/main_frame.py 19606->19619 (+13: Notifications dialog now copies only on explicit Ctrl+C / double-click / button instead of on every EVT_LISTBOX selection change, #1247). Feature #1246 (Ctrl+T new document tab -- reuses new_file, no core-path change): wiring only -- core/keymap.py 1015->1018 (window.new_document_tab default chord), main_frame_commands.py 2513->2522 (register + id-map), main_frame_menu.py 3328->3333 (id + Window-menu item). Feature #1249/#1253 (Radio export-favorites-to-playlist + output-device quick picker): apps/radio.py 2055->2069 -- menu Append/Bind wiring only; all logic lives in NEW under-cap modules (core/radio/playlist_export.py, ui/radio/playlist_export_ui.py, ui/radio/output_device_ui.py). Menu/command/keymap registrations and the notifications dialog cannot be extracted. One-time acknowledgment of the grown state, not license for further growth.",
"_rebaseline_2026_08_01_extract_banners_and_bom": "User-reported bug fixes #1279/#1278. io/structured.py 776->787 (+11): (a) #1278 -- the structured text branch now reads through quill.io.text.read_text_document so a UTF-8 BOM is stripped (and remembered as utf-8-sig for a round-trip save), the original line ending survives, and the cp1252/latin-1 fallback applies; that needs its own Document construction because the encoding/line ending no longer match the binary-format tail. (b) #1279 -- the fallback office extracts drop their synthetic banners, and a degraded read stamps extraction_pack_missing so the intake summary can point at Help > Download Optional Components (suffix set + predicate live in core/pdf_ocr_install.py). Word extraction moved OUT to the NEW module io/docx_text.py (python-docx rendering plus the old raw-XML walk as last resort), which is why the net growth is smaller than the change. One-time acknowledgment of the grown state, not license for further growth.", "_rebaseline_2026_07_30_issue_and_feature_sweep": "Open-issue sweep + feature requests (see xc.md). Bug fixes: io/structured.py 759->776 (+17: _format_structured_text dispatch that falls back to raw text on a malformed JSON/TOML/XML/CSV/notebook instead of crashing the open flow, #1228); ui/ai_setup_wizard.py 954->1015 (+61: _alive() guard + guarded async callbacks so a closed AI-setup wizard's freed StaticText is never touched, #1230, and a ListModels fallback in _probe_provider so a valid Gemini key is accepted even when the default probe model 404s, #1231 -- mostly explanatory comments); ui/main_frame.py 19606->19619 (+13: Notifications dialog now copies only on explicit Ctrl+C / double-click / button instead of on every EVT_LISTBOX selection change, #1247). Feature #1246 (Ctrl+T new document tab -- reuses new_file, no core-path change): wiring only -- core/keymap.py 1015->1018 (window.new_document_tab default chord), main_frame_commands.py 2513->2522 (register + id-map), main_frame_menu.py 3328->3333 (id + Window-menu item). Feature #1249/#1253 (Radio export-favorites-to-playlist + output-device quick picker): apps/radio.py 2055->2069 -- menu Append/Bind wiring only; all logic lives in NEW under-cap modules (core/radio/playlist_export.py, ui/radio/playlist_export_ui.py, ui/radio/output_device_ui.py). Menu/command/keymap registrations and the notifications dialog cannot be extracted. One-time acknowledgment of the grown state, not license for further growth.",
Comment thread pyproject.toml
Comment on lines +199 to 205
# Optional Components (quill/core/pdf_ocr_install.py). Shipped builds bundle the
# lean `office-text` extra below instead of this one -- see the note there.
pdf-ocr = [
"markitdown[docx,pptx,xlsx,xls,pdf]>=0.1.6",
"pdfplumber>=0.11.10",
"pypdf>=6.14.2",
]
@accesswatch

Copy link
Copy Markdown
Contributor Author

Superseded by #1281. While this was in flight, the standalone Quill Converter landed on main independently (#1275), so this branch's converter commit duplicated it. #1281 is the same work rebased onto current main with that commit dropped -- the two reader fixes (#1279, #1278), the packaging change, Live365 link normalization, and the docs -- and keeps only the converter's documentation.

@accesswatch accesswatch closed this Aug 1, 2026
@accesswatch
accesswatch deleted the fix/document-readers-and-office-packaging branch August 6, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants