feat(web): add reliable SSE downloads and cancellation - #33
Open
jung233 wants to merge 25 commits into
Open
Conversation
These functions were referenced in sources/__init__.py but never defined in pdf_utils.py, causing ImportError on every download attempt. - is_suspicious_pdf: checks if PDF is < 50KB (likely preview/cover) - suspicious_pdf: returns failure result and cleans up the file Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add fastapi and jinja2 as optional [web] extra in pyproject.toml - Guard web.py imports with friendly error message when deps missing - Update Dockerfile to install [web,instsci] instead of [tor,instsci] Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When a source succeeds, immediately call pool.shutdown(wait=False) and clean up temp files before returning. Previously the cleanup was in a finally block that ran after return, allowing background threads to continue running for up to a minute after success. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add /api/download/stream endpoint with Server-Sent Events - Add progress callback support to download function - Add /api/download/file endpoint for fetching completed downloads - Add /api/downloads/active endpoint for monitoring - Update frontend to use SSE for real-time status updates - Show download phase (free_sources, institutional) in UI
- Replace batch event delivery with asyncio.Queue for real-time streaming - Use loop.call_soon_threadsafe for thread-safe event publishing - Add keepalive comments to prevent connection timeout - Progress events now stream in real-time during download
The previous approach embedded base64-encoded PDF data (~4.3MB) directly
in SSE success events, which caused JSON parse failures and missing
file_data in the client. Replace with a secure one-time token system:
- Add _create_download_token/_consume_download_token with 5min TTL
- Add GET /api/download/file/{token} endpoint (token-consumed on use)
- Remove vulnerable GET /api/download/file?path= endpoint (arbitrary file read)
- SSE success event now sends small download_token instead of huge base64
- Frontend triggers browser download via token URL
- Add progress callbacks for cache hits in sources/__init__.py
Tested: Nature DOI, Wiley DOI, arXiv - all produce valid PDFs via token.
Token is single-use (404 on second attempt).
jung233
force-pushed
the
codex/web-download-streaming
branch
from
July 17, 2026 08:38
38d99ee to
8e8c8bf
Compare
jung233
marked this pull request as ready for review
July 17, 2026 09:31
There was a problem hiding this comment.
Pull request overview
This PR enhances the web UX and download pipeline by adding SSE-based progress streaming, cooperative cancellation on client disconnect, and more deterministic resource/file handling across multiple download sources and browser-driven workflows.
Changes:
- Added SSE download streaming with progress events, one-time file tokens, and client-disconnect cancellation in the FastAPI web UI.
- Propagated cancellation support and improved cleanup/atomic file publishing across multiple sources (Sci-Hub/LibGen/WebVPN/EZProxy/CARSI) and the core race orchestration.
- Hardened browser lifecycle management and batch workflows (per-thread context ownership, safer cleanup, profile handling), plus packaging/runtime updates (templates in package data, PyMuPDF dependency, Docker runtime libs/tini).
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/scansci_pdf/web.py | Adds SSE download stream, one-time file tokens, and client disconnect cancellation; exposes active download status. |
| src/scansci_pdf/templates/index.html | Switches UI download flow to SSE stream consumption and token-based file download; adds phase display/cancel plumbing. |
| src/scansci_pdf/sources/scihub.py | Adds cancellation propagation, atomic writes, bounded HTML reads, and safer browser/resource cleanup during domain races. |
| src/scansci_pdf/sources/libgen.py | Adds cancellation propagation and response cleanup to LibGen fetch/download flow. |
| src/scansci_pdf/sources/instsci.py | Adds cancellation-aware rate limiting/waits, atomic writes, and more deterministic browser/session cleanup. |
| src/scansci_pdf/sources/ezproxy.py | Adds cancellation propagation, atomic writes for captured PDFs, and improved browser/session cleanup. |
| src/scansci_pdf/sources/carsi.py | Adds cancellation hooks, safer response capture/write paths, and safer session close semantics. |
| src/scansci_pdf/sources/carsi_source.py | Threads cancellation through CARSI source entrypoint and ensures client cleanup. |
| src/scansci_pdf/sources/init.py | Refactors race orchestration: cancellation support, browser-slot handling, atomic publishing, per-download locks, and stale race-dir cleanup. |
| src/scansci_pdf/server.py | Improves Elsevier API verification routing and ensures CARSI client cleanup. |
| src/scansci_pdf/publisher_batch.py | Reworks parallelism to avoid cross-thread Playwright context sharing; improves worker profile handling and cleanup. |
| src/scansci_pdf/pdf_utils.py | Adds cancellation-safe atomic PDF writes/publish helpers and session-bound response closing helper; tweaks suspicious-PDF heuristic. |
| src/scansci_pdf/main.py | Aligns CLI commands with new config paths/modules and improves resource cleanup in fetch flows. |
| src/scansci_pdf/fetcher.py | Adds safer response closing patterns and more robust browser response-capture cleanup. |
| src/scansci_pdf/extractors/pdf_extractor.py | Makes PyMuPDF optional at import-time and improves doc closing safety/error messages. |
| src/scansci_pdf/deps.py | Adds PyMuPDF as a core dependency description. |
| src/scansci_pdf/cloakbrowser_compat.py | Adds launch wrapper to stop Playwright driver when CloakBrowser launch fails mid-flight. |
| src/scansci_pdf/cli.py | Ensures fetcher and broker contexts are closed reliably; adds helper for broker record fetch with context invalidation. |
| src/scansci_pdf/cache.py | Uses collision-resistant temp filenames for cache writes. |
| src/scansci_pdf/browser_login.py | Tightens ownership/thread-safety for persistent browser resources; adds safer launch/cleanup behavior. |
| src/scansci_pdf/browser_cookies.py | Uses driver-cleanup launch wrapper and more deterministic close confirmation. |
| src/scansci_pdf/auth.py | Improves browser lifecycle handling with leased slots and close confirmation; makes session close safer. |
| pyproject.toml | Adds PyMuPDF dependency and includes templates in package data. |
| Dockerfile | Installs web/cloakbrowser extras, runtime libs, tini entrypoint, and memory env tuning. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
162
to
+164
| @app.get("/", response_class=HTMLResponse) | ||
| async def index(request: Request): | ||
| return templates.TemplateResponse("index.html", {"request": request}) | ||
| return templates.TemplateResponse(request, "index.html") |
Comment on lines
+64
to
+67
| def _cleanup_download_tokens() -> None: | ||
| now = time.time() | ||
| expired = [token for token, info in _download_tokens.items() if now > info["expires"]] | ||
| for token in expired: |
Comment on lines
+323
to
+326
| def enqueue() -> None: | ||
| if callback_open.is_set() and not event_queue.full(): | ||
| event_queue.put_nowait(event) | ||
|
|
Comment on lines
+433
to
446
| @app.get("/api/downloads/active") | ||
| async def api_active_downloads(): | ||
| """List currently active downloads.""" | ||
| return JSONResponse({ | ||
| "active": [ | ||
| { | ||
| "task_id": tid, | ||
| "identifier": info["identifier"], | ||
| "status": info["status"], | ||
| "elapsed": asyncio.get_event_loop().time() - info["started_at"], | ||
| } | ||
| for tid, info in _active_downloads.items() | ||
| ] | ||
| }) |
Comment on lines
88
to
92
| # Count PDF page objects: look for "/Type /Page" not followed by "s" | ||
| import re | ||
| pages = len(re.findall(rb"/Type\s*/Page\b", content)) | ||
| if pages <= 1: | ||
| if pages == 1: | ||
| return True |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dependency
This is a stacked PR on #32. The head includes #32 so it is directly runnable. After #32 merges, GitHub will reduce this PR to its 12 Web and download-pipeline commits.
Verification