feat: setup pdf parser service using docling by ayushpahwa · Pull Request #15 · My-Health-Log/core

ayushpahwa · 2025-12-17T07:11:12Z

Summary

Set up the pdf-parse Python service using FastAPI + Docling for extracting structured data from health report PDFs
Built a full extraction pipeline: PDF → Docling → typed Pydantic models → page-grouped JSON output
Created typed models for all output: section headers, groups (KV and list via discriminated union), tables with row-level bounding boxes
Added generic ApiResponse[T] wrapper with unified exception handling across all endpoints
Migrated from raw dict parsing to Docling's typed objects for type safety
Set up CI workflow, linting (ruff), type checking (pyright), and dependency management (uv)

Key files

pdf-parse/app/services/docling.py — extraction provider with parse_section_headers, parse_groups, parse_tables, normalise_extraction
pdf-parse/app/schemas/extract.py — Pydantic models: Page, ParsedTable, TableRow, ParseSectionHeader, ParsedGroups, KVGroup | ListGroup
pdf-parse/app/schemas/api_response.py — generic ApiResponse[T] response wrapper
pdf-parse/app/exception_handlers.py — unified exception handler for HTTPException, validation errors, and unhandled exceptions
pdf-parse/app/schemas/docling.py — type aliases bridging Docling union types

Known Docling artifacts (deferred to LLM normalization step)

Method text merges into test names (e.g., "Haemoglobin (HB) Method: Photometry")
Cross-page row contamination on first row of continuation pages
Colon prefix on group values (: Ayush Pahwa)
Patient info duplicated across pages
OCR spacing artifacts (DEPARTME N T)

Test plan

Upload 3-page haematology PDF → verify all 29 test values extracted correctly
Upload non-PDF file → verify 400 error wrapped in ApiResponse format
Upload file > 5MB → verify 413 error wrapped in ApiResponse format
Verify null fields excluded from JSON response
Verify section headers, groups, and tables are page-grouped correctly

🤖 Generated with Claude Code

Copilot

Pull request overview

This PR adds a Python-based PDF parsing microservice to the monorepo using Docling and FastAPI. The service enables local extraction of health metrics from PDF reports without sending sensitive data to cloud services, addressing privacy requirements for the health data management application.

Key Changes:

New PDF parsing service with provider pattern for swappable extraction backends
FastAPI endpoints for health checks and PDF extraction
Integration with pnpm workspace via postinstall hooks and custom scripts
Documentation updates for setup and development workflows

Reviewed changes

Copilot reviewed 14 out of 20 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
src/pdf-parse/pyproject.toml	Python project configuration with dependencies for Docling, FastAPI, and PyTorch
src/pdf-parse/app/main.py	FastAPI application entry point with router configuration
src/pdf-parse/app/routers/health.py	Health check endpoint returning service status
src/pdf-parse/app/routers/extract.py	PDF extraction endpoint with file upload handling
src/pdf-parse/app/services/provider.py	Abstract provider interface for extraction backends
src/pdf-parse/app/services/docling.py	Docling-based implementation of extraction provider
src/pdf-parse/app/schemas/extract.py	Pydantic models for API response schemas
src/pdf-parse/.python-version	Python version requirement (3.13)
src/pdf-parse/README.md	Service-specific documentation with setup and usage instructions
src/package.json	Added postinstall hook for Python deps and PDF service scripts
src/AGENTS.md	Updated architecture documentation to include PDF service
docs/setup.md	Added Python/uv prerequisites and PDF service endpoints
.gitignore	Added Python-specific ignore patterns

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-17T07:23:54Z

+app = FastAPI(
+    title="PDF Service",
+    description="PDF extraction service for My Health Log",
+    version="0.1.0",
+)


Missing CORS configuration. Since this is a microservice that will be called from a web application (the client app), it needs CORS headers configured to allow requests from the frontend. Without proper CORS middleware, the browser will block cross-origin requests to this API. Consider adding FastAPI's CORSMiddleware with appropriate allowed origins.

Copilot · 2025-12-17T07:23:57Z

+    if not file.filename.endswith(".pdf"):
+        raise HTTPException(400, "Only PDF files supported")
+


The file validation only checks the extension, which is insufficient for security. An attacker could upload a malicious file with a .pdf extension. Consider adding MIME type validation and/or file content validation to ensure the uploaded file is actually a PDF. For example, verify the file starts with the PDF magic bytes "%PDF".

Suggested change

if not file.filename.endswith(".pdf"):

raise HTTPException(400, "Only PDF files supported")

# Basic filename check for user convenience

if not file.filename.lower().endswith(".pdf"):

raise HTTPException(400, "Only PDF files supported")

# Validate reported MIME type

if file.content_type != "application/pdf":

raise HTTPException(400, "Invalid content type; only PDF files supported")

# Validate file signature (magic bytes) to ensure it is actually a PDF

header = await file.read(4)

if header != b"%PDF":

raise HTTPException(400, "Uploaded file is not a valid PDF")

# Reset file pointer so downstream code reads from the beginning

await file.seek(0)

Docling itself validates the PDF structure during conversion and will raise on invalid files. Magic byte validation is not needed here.

Copilot · 2025-12-17T07:23:58Z

+    async def extract(self, file: UploadFile) -> dict:
+        content = await file.read()
+        stream = DocumentStream(name=file.filename, stream=io.BytesIO(content))
+        result = self.converter.convert(stream)


The extract method is defined as async but calls synchronous blocking operations (file.read() and converter.convert()). The DocumentConverter.convert() call likely performs CPU-intensive ML operations that will block the event loop, degrading performance for concurrent requests. Consider using run_in_executor to run the blocking conversion in a thread pool, or verify if Docling provides async APIs.

@ayushpahwa yo check this

Acknowledged. This is a single-user local service for now. Will move to run_in_executor if concurrency becomes a concern.

Created #16 to track this. Added a TODO in code linking to the issue.

Fixed and closed #16. Also added pyright and tsc type checking to CI pipeline. Test coverage tracked as #17.

DPirate · 2025-12-18T14:05:20Z

@ayushpahwa , I think if we are enabling AI reviews then rhe author should resolve the ai comments before human review. It will increase clarity about what is already being addressed and also cleanup the review view of github.

Copilot

Pull request overview

Copilot reviewed 15 out of 21 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-23T06:18:33Z

+    async def extract(self, file: UploadFile) -> dict:
+        if file.size is None or file.filename is None:
+            raise HTTPException(404, "Please use a valid file")
+        print(file.size)


The debug print statement should be removed or replaced with proper logging. This will clutter the console output in production and doesn't follow the logging pattern established elsewhere in the application (using the logger from app.state).

Suggested change

print(file.size)

Fixed — replaced all print statements with logger.error() using the uvicorn logger.

Copilot · 2025-12-23T06:18:33Z

+    # TODO: replace return type from dict to ExtractionResponse once mapping is fixed
+    async def extract(self, file: UploadFile) -> dict:
+        if file.size is None or file.filename is None:
+            raise HTTPException(404, "Please use a valid file")


Error message clarity issue: The message "Please use a valid file" is vague and doesn't explain what makes a file valid. This will be unhelpful to API consumers trying to debug upload failures. Consider a more descriptive message like "File upload is invalid: missing file size or filename".

Suggested change

raise HTTPException(404, "Please use a valid file")

raise HTTPException(404, "File upload is invalid: missing file size or filename.")

This is now routed through the unified exception handler which returns a structured ApiResponse with the error details.

Copilot · 2025-12-23T06:18:33Z

+| `pnpm dev` | Start server, client, and PDF service in parallel |
 | `pnpm dev:server` | Start server only |
 | `pnpm dev:client` | Start client only |
+| `pnpm dev:pdf-parse` | Start PDF parsing service (dev mode) |


The documentation references script name "pnpm dev:pdf-parse" but earlier in the same document (line 9) it shows "pnpm dev" running "server + client + PDF service in parallel". However, looking at package.json, the "dev" script doesn't actually include the PDF service automatically. The documentation should clarify that users need to run "pnpm dev:pdf-parse" separately, or the package.json dev script should be updated to include it.

The dev script on line 9 runs all services in parallel including pdf-parse. The separate dev:pdf-parse script is for running the PDF service standalone. This is documented correctly.

Copilot · 2025-12-23T06:18:33Z

+    # TODO: replace return type from dict to ExtractionResponse once mapping is fixed
+    async def extract(self, file: UploadFile) -> dict:
+        if file.size is None or file.filename is None:
+            raise HTTPException(404, "Please use a valid file")


HTTP status code 404 (Not Found) is semantically incorrect for invalid file uploads. A 404 status code indicates that the requested resource doesn't exist on the server. For file validation failures, use 400 (Bad Request) instead to indicate that the client sent an invalid request.

Suggested change

raise HTTPException(404, "Please use a valid file")

raise HTTPException(400, "Please use a valid file")

Fixed — changed to HTTPException(400).

Copilot · 2025-12-23T06:18:34Z

+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=settings.cors_origins,
+    allow_credentials=True,


Missing HTTP methods validation in CORS configuration. The CORSMiddleware is configured without specifying allowed HTTP methods (allow_methods) or headers (allow_headers). This could cause CORS preflight requests to fail when the frontend tries to POST files to /extract. Add allow_methods=[""] or a specific list of methods, and allow_headers=[""] or specific headers needed for file uploads.

Suggested change

allow_credentials=True,

allow_credentials=True,

allow_methods=["*"],

allow_headers=["*"],

Fixed — added allow_methods=[""] and allow_headers=[""].

Copilot · 2025-12-23T06:18:34Z

  "scripts": {
-    "dev": "pnpm --parallel --filter '*' dev",
+    "postinstall": "[ \"$CI\" = 'true' ] || (cd pdf-parse && uv sync)",
+    "dev": "pnpm --parallel --filter '*' dev & pnpm dev:pdf-parse",


The "&" operator for running parallel commands is not cross-platform compatible. It works on Unix-like systems (Linux/macOS) but will fail on Windows. Since the repository appears to be using pnpm's built-in parallel execution, use "pnpm --parallel" instead, or use a cross-platform tool like "concurrently" or "npm-run-all".

Copilot · 2025-12-23T06:18:34Z

 pnpm dev                  # Run server + client in parallel
 pnpm dev:server           # Server only
 pnpm dev:client           # Client only
+pnpm pdf:dev              # PDF parsing service (dev mode)


The documentation references script name "pnpm pdf:dev" which doesn't exist in package.json. The actual script name is "pnpm dev:pdf-parse". This inconsistency will confuse users following the documentation. Update this to match the actual script name in package.json.

Suggested change

pnpm pdf:dev # PDF parsing service (dev mode)

pnpm dev:pdf-parse # PDF parsing service (dev mode)

Copilot · 2025-12-23T06:18:35Z

 pnpm dev:server           # Server only
 pnpm dev:client           # Client only
+pnpm pdf:dev              # PDF parsing service (dev mode)
+pnpm pdf:start            # PDF parsing service (production)


The documentation references script name "pnpm pdf:start" which doesn't exist in package.json. The actual script name is "pnpm start:pdf-parse". This inconsistency will confuse users following the documentation. Update this to match the actual script name in package.json.

Suggested change

pnpm pdf:start # PDF parsing service (production)

pnpm start:pdf-parse # PDF parsing service (production)

Copilot · 2025-12-23T06:18:35Z

+    # TODO: replace return type from dict to ExtractionResponse once mapping is fixed
+    # async def extract(self, file: UploadFile) -> ExtractionResponse:
+    async def extract(self, file: UploadFile) -> dict:
+        pass


The abstract method uses "pass" but doesn't raise NotImplementedError. While Python's ABC will prevent instantiation of classes that don't implement abstract methods, it's a best practice to include "raise NotImplementedError" in the abstract method body for clarity and to catch potential issues if the method is called via super().

Suggested change

pass

raise NotImplementedError("Subclasses of ExtractionProvider must implement the 'extract' method.")

Copilot · 2025-12-23T06:18:35Z

+class Element(BaseModel):
+    text: str
+    bbox: BoundingBox
+    type: str  # "text", "table", "heading"


The type field in Element uses a string literal type with specific expected values ("text", "table", "heading") indicated in the comment, but the actual type annotation is just 'str'. This allows any string value, which could lead to inconsistent data. Consider using a Literal type or Enum to enforce the allowed values: type: Literal["text", "table", "heading"].

Copilot

Pull request overview

Copilot reviewed 21 out of 27 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-11T07:50:07Z

+                        top=bbox.t,
+                        bottom=bbox.b,
+                        right=bbox.r,
+                        coord_origin=CoordOriginEnum(value=bbox.coord_origin.value),


CoordOriginEnum is being constructed incorrectly: CoordOriginEnum(value=...) will raise TypeError because enums don’t accept a value= keyword. This will cause section header parsing to fail and be silently swallowed by the surrounding try/except. Use CoordOriginEnum(bbox.coord_origin.value) (or map the docling origin to your enum) instead.

Suggested change

coord_origin=CoordOriginEnum(value=bbox.coord_origin.value),

coord_origin=CoordOriginEnum(bbox.coord_origin.value),

This works correctly — Python StrEnum accepts value= as a keyword argument. We've verified the output multiple times with correct coord_origin values. Dismissing.

Copilot · 2026-04-11T07:50:07Z

+pnpm pdf:dev              # PDF parsing service (dev mode)
+pnpm pdf:start            # PDF parsing service (production)


The commands pnpm pdf:dev and pnpm pdf:start referenced here don’t exist in src/package.json (the scripts are dev:pdf-parse / start:pdf-parse). This will mislead contributors following the setup instructions. Update the docs to the actual script names (or rename the scripts to match the docs).

Suggested change

pnpm pdf:dev # PDF parsing service (dev mode)

pnpm pdf:start # PDF parsing service (production)

pnpm dev:pdf-parse # PDF parsing service (dev mode)

pnpm start:pdf-parse # PDF parsing service (production)

Fixed — updated AGENTS.md to use the correct script names (dev:pdf-parse, start:pdf-parse).

DPirate

lgtm

ayushpahwa requested review from DPirate and Copilot December 17, 2025 07:11

ayushpahwa self-assigned this Dec 17, 2025

ayushpahwa marked this pull request as ready for review December 17, 2025 07:11

Copilot started reviewing on behalf of ayushpahwa December 17, 2025 07:11 View session

ayushpahwa changed the title ~~Feat/pdf parser setup~~ feat: setup pdf parser service using docling Dec 17, 2025

Copilot AI reviewed Dec 17, 2025

View reviewed changes

ayushpahwa requested a review from Copilot December 23, 2025 06:13

Copilot started reviewing on behalf of ayushpahwa December 23, 2025 06:13 View session

Copilot AI reviewed Dec 23, 2025

View reviewed changes

ayushpahwa added 19 commits March 21, 2026 21:33

create: start a new uv project

58617e3

create: add schemas for extraction results

a267169

create: add route for checking server health

b5af912

create: service to handle docling parsing

f75f03f

create: add abstract class for providers

79aaff7

create: register route to extract pdf data

5e770ac

update: conform to server's return type

e4bb619

update: add to monorepo scripts

476890a

create: init readme for the pdf parsing service

65be171

chore: update agents instructions file

1c6c337

update: add pdf service setup instructions to setup.md

759cd30

chore: add deps for ci runs

0ad988d

chore: add uv caching for faster ci runs

ee69fca

chore: add venv to prettier ignore

0c45b36

chore: fix prettier issue

2f8093b

chore: fix prettier issues

19fcab5

chore: remove unnecessary downloads for linting

59c7786

chore: add lint rules with fixes

e7e00bc

chore: added pyright to handle code completion

9ac3b2d

ayushpahwa added 13 commits April 6, 2026 10:03

create: new file to bridge unions from docling to code

fbe5645

update: use typed values for section header extraction

9456ffd

update: fix child ref fetch

57b5f99

chore: delete redundant model

d99a50e

create: add a generic model to wrap all api responses

cdf3303

update: wrapping endpoint output to api response model

be5d37f

create: add exception handlers to ensure api response structure

8bbef8e

udpate: add functions to route exceptions through handler

05274e2

chore: remove redundant todo

647715e

update: ensure uv is installed before running postinstall

7c5dcd2

update: allow cors methods

537c96d

update: replace print with logger statements

944ed75

update: use correct http code for bad requests

db97ef0

ayushpahwa mentioned this pull request Apr 11, 2026

Move Docling converter.convert() to run_in_executor for async safety #16

Closed

chore: added comment for blocking convert issue

8209e47

ayushpahwa requested a review from Copilot April 11, 2026 07:47

Copilot started reviewing on behalf of ayushpahwa April 11, 2026 07:47 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

ayushpahwa added 3 commits April 11, 2026 15:00

update: moved blocking function to separate thread

e0ec1b8

update: let errors propogate outside of the service

1921c2e

update: use update loop method

0511f5f

ayushpahwa mentioned this pull request Apr 11, 2026

Add test coverage for pdf-parse service #17

Open

ayushpahwa added 5 commits April 11, 2026 22:59

chore: remove commented import

e5329a6

update: add type check steps to ci

9c06b1a

chore: fix agents instructions file

f226f0c

update: fix tsc type check folders

33556d7

chore: comment out server check in ci

61700c9

DPirate approved these changes Apr 14, 2026

View reviewed changes

ayushpahwa merged commit c58a74c into trunk Apr 15, 2026
1 check passed

ayushpahwa deleted the feat/pdf-parser-setup branch April 15, 2026 04:20

		if not file.filename.endswith(".pdf"):
		raise HTTPException(400, "Only PDF files supported")

-    if not file.filename.endswith(".pdf"):
-        raise HTTPException(400, "Only PDF files supported")
+    # Basic filename check for user convenience
+    if not file.filename.lower().endswith(".pdf"):
+        raise HTTPException(400, "Only PDF files supported")
+    # Validate reported MIME type
+    if file.content_type != "application/pdf":
+        raise HTTPException(400, "Invalid content type; only PDF files supported")
+    # Validate file signature (magic bytes) to ensure it is actually a PDF
+    header = await file.read(4)
+    if header != b"%PDF":
+        raise HTTPException(400, "Uploaded file is not a valid PDF")
+    # Reset file pointer so downstream code reads from the beginning
+    await file.seek(0)

	raise HTTPException(404, "Please use a valid file")
	raise HTTPException(404, "File upload is invalid: missing file size or filename.")

	raise HTTPException(404, "Please use a valid file")
	raise HTTPException(400, "Please use a valid file")

	pnpm pdf:dev # PDF parsing service (dev mode)
	pnpm dev:pdf-parse # PDF parsing service (dev mode)

	pnpm pdf:start # PDF parsing service (production)
	pnpm start:pdf-parse # PDF parsing service (production)

	pass
	raise NotImplementedError("Subclasses of ExtractionProvider must implement the 'extract' method.")

	coord_origin=CoordOriginEnum(value=bbox.coord_origin.value),
	coord_origin=CoordOriginEnum(bbox.coord_origin.value),

Uh oh!

Conversation

ayushpahwa commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key files

Known Docling artifacts (deferred to LLM normalization step)

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DPirate commented Dec 18, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

ayushpahwa commented Dec 17, 2025 •

edited

Loading