New code_pdf module: progress update

[Salomon-Mazatlan]

New code_pdf module: progress update

The module is almost ready to submit as a proposal. What remains is integrating @kaixxx's updates to the code margins and the new CodeTreeController. In the meantime, here's an overview of the progress.

Updates:

• The original module was rewritten from scratch. The previous version processed the PDF multiple times on opening (layout analysis with pdfminer, text redaction with PyMuPDF, and redrawing of text boxes as Qt objects), which caused slowness. The new version uses PyMuPDF (fitz) exclusively, with native rendering.
• In theory it is about 10x faster, even with large files.
• Pages are rasterized only once per zoom level and stored in an LRU cache of QPixmaps, keeping memory usage bounded for large PDFs.
• Continuous scroll with all pages visible.
• Integrated coding margin.
• Allows coding both text and images on files within the same module.
• Supports coding in multi-column documents.
• Codings applied to PDF files in other modules are also displayed on the document within code_pdf.
• The AI text analysis option was integrated, so AI-assisted text analysis can be applied directly from the module.
• PDF highlight export: generates a copy of the original PDF with the coded segments embedded as native highlight annotations (in each code's color, with the code name as annotation comment), so the coded document can be opened and reviewed in any standard PDF reader.
• ODT report: exports a coding report in OpenDocument Text format (.odt), listing the coded segments with their codes (text and images).
• A refactoring method was added for older projects: the fulltext is re-extracted and existing codings are matched against it. Any codings that cannot be remapped are recorded in Journals as lost codes, so the user can review them.
• The module follows the patterns of code_text.py and view_image.py (coding margin, resize handles) to stay consistent with the rest of the application.

Point for discussion:

Coding in the PDF module is anchored to character offsets in the fulltext field of the database, which must match exactly the text extracted from the PDF with PyMuPDF. If the user edits that text, the offsets become desynchronized and the codings end up shifted or invalid. So I'd like to ask which option you prefer:

1. Block editing of the text of PDF files. To compensate for this restriction, I added an option to generate a .txt file with a copy of the PDF text, which can be edited and coded independently.
2. Or keep editing enabled and only show a warning before editing.

---

Here are some screenshots:

[qwertygc]

Amazing, thanks!

[Salomon-Mazatlan]

Amazing, thanks!

Gracias!

Also, if you have any suggestions, they're very welcome.

[kaixxx]

Again: Congrats, very nice improvements!

Editing: I would keep the extracted text strictly read only, no editing, as this is only going to mess up things.

Matching existing codings and new text:
In the AI chat, I use a special pattern matching algorithm to find quotes delivered by the AI in the original source. This algorithm comes from biology and is normally used for DNA-pattern matching. It will find a source string in a large text even if it does not match 100%, e.g., because an \n is omitted or something like that. You can adjust the level of mismatch acceptable. I only use this method as a fallback if the exact string match fails. Maybe you can take some inspiration from it. Note that in the AI module, quite large gaps are tolerated, because the AI sometimes leaves out irrelevant text in the middle of a quote. In your case, I would tolerate only a very small mismatch. But it might be a way to make the transition from the old to the new text extraction method more robust.
The function is here: https://github.com/kaixxx/QualCoder/blob/a30dc9fb46ebc620d46ffa13768bbb3e272864df/src/qualcoder/ai_llm.py#L687-L733 (my fork, but it is in the main branch as well).

[ccbogel]

Looks great. Cant waitto see it pulled into QualCoder.

Editing: I would keep the extracted text strictly read only, no editing, as this is only going to mess up things.
Colin - Yes agree on this.

[Salomon-Mazatlan]

New: PDFs with highlights can now be imported and their highlighted segments coded automatically, one code per highlight colour under a 'PDF Highlights' category, using the closest QualCoder palette colours.