Skip to content

Stream PDF from archives #117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lfoppiano wants to merge 2 commits into
base: master
Choose a base branch
from
+663 −105
Open

Stream PDF from archives #117

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
88040f2
Stream PDFs from zip/tar archives without full decompression
lfoppiano Jul 23, 2026
74b0046
Support glob patterns for --input and use pathlib for directory trave…
lfoppiano Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ concurrent processing capabilities for PDF documents, reference strings, and pat
- **Sentence Segmentation**: Layout-aware sentence segmentation capabilities
- **JSON Output**: Convert TEI XML output to structured JSON format with CORD-19-like structure
- **Markdown Output**: Convert TEI XML output to clean Markdown format with structured sections
- **Archive Streaming**: Process files directly from `.zip`/`.tar`/`.tar.gz` archives without fully decompressing them

## 📋 Prerequisites

Expand Down Expand Up @@ -166,8 +167,26 @@ grobid_client --server https://grobid.example.com --input ~/citations.txt proces

# Force reprocessing with sentence segmentation and JSON output
grobid_client --input ~/docs --force --segment_sentences --json processFulltextDocument

# Process PDFs directly from a zip or tar.gz archive (streamed, not fully decompressed)
grobid_client --input ~/papers.zip --output ~/results processFulltextDocument
grobid_client --input ~/papers.tar.gz --output ~/results processFulltextDocument

# --input also accepts glob patterns (quote them so the shell does not expand them)
grobid_client --input "~/papers/*.zip" --output ~/results processFulltextDocument # many archives
grobid_client --input "~/data/**/*.pdf" --output ~/results processFulltextDocument # PDFs in subdirectories
```

> [!NOTE]
> `--input` accepts a directory, a single file, an **archive**, or a **glob pattern**:
> - **Archives** (`.zip`, `.tar`, `.tar.gz`/`.tgz`, `.tar.bz2`/`.tbz2`) are streamed: eligible entries are extracted in
> chunks of `batch_size` to a temporary directory, sent to GROBID, written to `--output`, and deleted before the next
> chunk. The archive is never fully decompressed, so disk usage stays bounded. If `--output` is omitted, results go to a
> directory named after the archive (e.g. `papers.zip` → `papers/`).
> - **Glob patterns** (`paper.zip`, `paper*.zip`, `**/paper*.zip`, `**/*.pdf`, …) are expanded with `**` recursion; each
> match is handled by type (archive → streamed, directory → recursed, file → processed). Quote the pattern so your shell
> passes it through to the client unexpanded.

### Python Library

#### Basic Usage
Expand Down
Loading
Loading