Investigate PDF encryption and evaluate a change to the used library

[SORMAS-JanBoehme]

Problem Description

We have received the following feedback from Luxembourg:

When sending documents by email, SORMAS converts them into PDF and secures them using the national health ID. During this process, the layout of the document changes significantly, which makes the final prescription difficult to use.

As a result, I have to spend considerable time adjusting and testing the original document to ensure that the final secured PDF resembles the intended layout.

I also tested uploading a PDF directly into the “Documents” section and then sending it. In that case, the layout is preserved, but this approach is not practical for daily use due to the time required.

Proposed Change

According to a first rough check, this may be related to the used library (Apache PDFBox) for creating and encrypting the PDF files.

Using another library like iText may resolve the issue.

Added Value/Benefit

Making sure that documents look the same one converted into PDF is very relevant to avoid manual corrections and trust of the user in SORMAS.

Acceptance Criteria

• PDF files should look exactly the same as the original document after generation and encrpytion.

Implementation Details

No response

Mockups

No response

Additional Information

No response

[Pa-Touche]

@SORMAS-JanBoehme can you please provide reproduction steps ?

[SORMAS-JanBoehme]

@Pa-Touche

The theoretical steps are:

1. create an email template through the admin UI
2. open a case
3. Click on "Send Email"

4. send an email to the person with an attached document

This request came in from Xavier. It can't really be tested fully on the Foundation's test system as we lack the official email and document templates he is using. The best way is probably to ask him to show it to you on the Lux test system with the actual templates used.

[Pa-Touche]

After discussion with @XavierCol

After discussion with @XavierCol

Most import type of documents

• Prescription
• Survey invitation

Idea which is not final / validated:

Generate document as PDFs: always ?
Keep template as *.docx but avoids having 2 versions, and problem that it does not look the same when printed/send through email.

Reproduction steps

Survey invitation:

• Go to case directory and Go to a case
• Surveys -> New Survey
• Send document
• Select email address and survey
• "Send Document"

On contact page:

• Go to contact directory and open contact
• Emails -> Send Email (if already created)

If no document:

• Documents -> New Document

To send crypted documents, the contact needs to have:

• National health id: 1960090900163 (valid one, not mine)
• email address in contact tab

Technical analysis, steps:

• Template is docx
• Docx is parsed using: Apache POI (https://poi.apache.org/)
• Generated to PDF: using XDocReport (https://github.com/opensagres/xdocreport)
• PDF is encrypted after using Apache PDFBox (https://pdfbox.apache.org/)

From my understanding PDFBox only mutates the meta-data and adds the password, the PDF should not be mutated, this needs to be tested / confirmed.
Meaning: the problem could also be at the usage of: XDocReport

[Pa-Touche]

Entry points in the code:
de.symeda.sormas.backend.externalemail.AttachmentService: PDF encryption (Apache PdfBox) and Docx to PDF conversion

DocXConverter: performs the actual conversion using the tools mentionned above (Apache POI for docx parsing and XDocReport for generation from XWPFDocument)

[Pa-Touche]

When excluding options due to restrictions (Microsoft Cloud, proprietary libraries), those are 2 feasible approaches:

"Typical option"

docx4j + FOP: Docx4J is already used in SORMAS for another purpose, but it could also be reused for PDF conversions when adding Apache FOP (https://xmlgraphics.apache.org/fop/).
See: export-FO in: https://www.docx4java.org/blog/2020/09/office-pptxxlsxdocx-to-pdf-to-in-docx4j-8-2-3/

Criteria	Result
Fidelity	Suitable for simple documents (text, tables, supported image types, header/footers)
Suitability	simple docx

Advantages

• Set-up should be quick
• Similar approach as the current one

Limitations

• This could come with similar limitations that the current set-up with a non-"pixel-perfect" rendering

"Atypical approach" leveraging Libre office installation on server

https://jodconverter.github.io/jodconverter/latest/getting-started/java-library/

Advantages

• Could be tested on any user's PC having libre office installed to make sure the "template" works as expected.
• High document <-> PDF fidelity
• If fidelity is better: use ODF format

Limitations

• Complex server set-up / requires some code changes to adapt to different approach
• Might still have some rendering issues of docx

[Pa-Touche]

@SORMAS-JanBoehme what I suggested to @XavCol and he agrees is to start with the docx4j + FOP approach, to see if it provides better result, if not then we can investigate if we want to use the second approach which is more complex. Or opt for something else.
The first approach should be implemented in a quick manner.

doc4j is already used, Apache FOP must be added.

<dependency>
	<groupId>org.apache.xmlgraphics</groupId>
	<artifactId>fop-events</artifactId>
	<version>2.8</version>
	<scope>provided</scope>
</dependency>

Is already used but the "full" library is required.

@SORMAS-JanBoehme okay for you ?

[SORMAS-JanBoehme]

@Pa-Touche

Yes, I agree to investigate this option first as it seems like it would require the least changes

[Pa-Touche]

Told to @XavCol already the encryption of the PDF does not mutate the PDF layout.

So it's really exclusively linked to docx -> pdf conversion and the used library for that purpose

[Pa-Touche]

As it can be seen in the PR: #14268

The new library (pair) does not provide much better results, and for some results even results that are worse than the current set-up.

As expected it does not provide "pixel-perfect" conversion.

Here are a few options that were identified:

• Feasible:

  • Go for Libre Office installation approach
  • Call another service using REST that provides high fidelity docx -> PDF conversion (internal to organization)
  • Try any other format conversion:
    • LaTeX: complex to write, probably provides pixel perfect results
    • HTML/CSS / with TemplateEngine: Apache velocity ? (used within Events ?). HTML to PDF conversion is not exact science either.
    • Markdown/Pandoc: does not allow exact positioning, does not seem appropriate.

• Feasible with workaround:

  • Using a paid license of a library (would require to be present in the adapter, and the call delegated from sormas-core), per example:
    • Aspose.Words.
    • Spire.Doc for Java

• Not feasible:

  • Microsoft Graph API (Microsoft Cloud)
  • Documents4J: delegates to local Office installation

[Pa-Touche]

@SORMAS-JanBoehme suggested to try to edit the PDF instead of doing a conversion.

I'll see what options are possible to do so

[SORMAS-JanBoehme]

More context in case someone reads this later on:
Patrick will be investigating how much effort it would be / if it would be possible to replicate the same replacement mechanism of placeholders that is being used for word files.

If that works, we can allow users to upload PDF files as document templates directly, which would remove the need for runtime conversion and possible layout issues.

[Pa-Touche]

Modifying a PDF results in following type of artifacts:

Should have displayed following arguments:

"John",
"Doe",
"5",
"Rue de Paris",
"Luxembourg"

Where text does overlap or does not properly align