Chunked (resumable) uploads to get past the 100MB proxy cap (#1840)

* Add chunked (resumable) uploads to get past the 100MB proxy cap

Upstream proxies (Cloudflare) cap a single proxied request body at 100MB,
which blocked large document and zip imports through the REST endpoints.
This adds a generic chunked-upload mechanism: the client slices a file
into sub-100MB parts, uploads each independently, and the server
reassembles them before handing the whole file to the existing import
services (so there is one import code path per kind, chunked or not).

Backend:
- New models ChunkedUploadSession + ChunkedUploadPart. Parts persist
  through Django storage (not local disk) so any web/worker process can
  reassemble a session. Initial migration 0001_initial.
- REST endpoints: POST /api/imports/chunked/start/,
  PUT|POST /api/imports/chunked/<id>/parts/<index>/,
  POST /api/imports/chunked/<id>/complete/, and GET .../<id>/ for resume.
  complete returns the same response shape as the matching single-request
  endpoint. Part PUTs use a looser throttle scope (document_import_chunks).
- Service layer start_chunked_upload / store_chunk /
  complete_chunked_upload with streaming reassembly (peak memory bounded
  to one 8MB block), per-user IDOR isolation, arithmetic + integrity
  validation, and fast-fail permission gating at start. All four import
  kinds supported (single document + the three zip flows).
- Hourly purge_stale_chunked_uploads GC; new CHUNKED_UPLOAD_* settings.

Frontend:
- importHttp transparently routes files above CHUNK_THRESHOLD_BYTES (50MB)
  through the chunked protocol; call sites and response handling unchanged.
- Raised the artificial 100MB dropzone cap (2GB single document; bulk-zip
  dropzone uses the 500MB cap via new FileDropZone props).

Tests: backend round-trip/validation/IDOR/GC coverage
(test_document_imports_chunked.py) and a frontend chunked-transport suite.
CHANGELOG updated.

* Make chunked-upload temp-file cleanup non-silent

Address code-quality "empty except" findings on the best-effort temp-file
unlink in the chunked-upload assembler/completer. Extract a shared
_safe_unlink helper that swallows OSError (cleanup failure is non-fatal —
the file lingers in the OS temp dir and the original exception must not be
masked) but records it at debug level instead of an unexplained pass.

* Fix linter findings in chunked upload (pyupgrade + mypy)

- pyupgrade: drop the now-redundant quotes on the _chunk_part_path
  forward-ref annotation (safe under `from __future__ import annotations`).
- mypy: restore corpus narrowing after the _resolve_corpus_for_edit
  refactor by branching on `corpus is None` (the helper returns
  Optional[Corpus], so the old `corpus_error is not None` check left
  `corpus` typed Optional at the later `.import_content`/`.id` uses).
- mypy: widen the zip service `zip_source` params from
  `UploadedFile | bytes` to `File | bytes` so the chunked completer can
  hand them a plain django File (UploadedFile remains a valid subtype).

* Consolidate File import with the .base import line (consistency)

* Address review: fix chunked-upload concurrency races, public normalise_optional, COMPLETED-session retention

- store_chunk: lock the session row (select_for_update) around the check-then-create so concurrent same-index uploads can't race the uniq_chunk_part_per_session constraint into a 500.
- complete_chunked_upload: claim the session via an atomic PENDING->ASSEMBLING compare-and-swap UPDATE; a 0-row result (double-complete) is refused with 409 instead of assembling/importing twice.
- Document the DOCUMENT-kind memory behaviour: it buffers the whole file (bounded by MAX_DOCUMENT_IMPORT_SIZE_BYTES) because import_content takes bytes; the per-block streaming guarantee holds only for the ZIP kinds.
- Rename _normalise_optional -> normalise_optional (public) so views no longer import a private helper across module boundaries; update its test.
- purge_stale_chunked_uploads: also purge COMPLETED sessions older than CHUNKED_UPLOAD_COMPLETED_RETENTION_DAYS (default 30; 0 disables) so the audit-trail rows don't grow unbounded. New setting + test.
- Add clarifying comment to test_non_zip_bytes_rejected_for_zip_kind.

* Address PR #1840 review: chunked-upload resilience and GC race

Resolve the Medium and Low items from the latest review:

- Frontend uploads parts with bounded concurrency (default 4) instead of
  strictly sequentially, with per-part exponential-backoff retry on
  transient/5xx/network errors (4xx fail fast), and an optional onProgress
  callback on every public import helper.
- purge_stale_chunked_uploads no longer purges ASSEMBLING sessions inside the
  normal stale window (could delete parts under a live complete reassembly);
  they are reclaimed after a longer grace window (CHUNKED_UPLOAD_ASSEMBLING_
  GRACE_HOURS, default 6h) so crashed mid-assembly workers are still cleaned up.
- store_chunk reports session info from the freshly-locked row.
- ChunkedUploadCompleteView declares only [JSONParser] (no request body).
- Celery purge task exposes completed_retention_days.
- Service cap helpers use direct settings access instead of dead getattr
  fallbacks; cross-reference comment between frontend chunk size and backend
  part cap.
- Test imports normalise_optional from services (its real home).
- Single-document whole-file-in-RAM reassembly tracked as follow-up (#1843).

New tests: ASSEMBLING grace-window GC; frontend concurrency cap, per-part
retry, 4xx fail-fast, and progress reporting.

* Address review: chunked-upload GC batching, configurable knobs, admin, nit

- purge_stale_chunked_uploads streams each queryset via .iterator(chunk_size=100)
  instead of materializing every stale session into memory (bounds peak memory
  when the GC runs against a large backlog).
- Move CHUNKED_UPLOAD_ASSEMBLING_GRACE_HOURS and CHUNK_ASSEMBLY_BLOCK_SIZE to
  env-backed settings in config/settings/base.py for operator tunability,
  consistent with the other CHUNKED_UPLOAD_* knobs; services.py reads them from
  settings (module symbols preserved for the existing test import + comments).
- Register read-only ChunkedUploadSession / ChunkedUploadPart admins so
  operators can inspect stale/FAILED sessions without dropping into the shell.
- importHttp.ts: drop redundant '?? undefined' — withOptional/appendIfDefined
  already guard null.

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: JSv4

CHANGELOG.md

@@ -7,6 +7,71 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **Chunked (resumable) uploads for large files** — work around the 100 MB
+  per-request body ceiling that upstream proxies (Cloudflare) impose on the
+  document-import REST endpoints. The client slices a file into sub-100 MB
+  parts, uploads each independently, and the server reassembles them before
+  handing the whole file to the **existing** import services (so there is one
+  import code path per kind, chunked or not).
+  - **Backend models** (`opencontractserver/document_imports/models.py`):
+    `ChunkedUploadSession` + `ChunkedUploadPart`. Parts persist through Django
+    storage (not local disk) so any web/worker process can reassemble a
+    session. Initial migration `0001_initial`.
+  - **REST endpoints** (`opencontractserver/document_imports/views.py`,
+    `urls.py`): `POST /api/imports/chunked/start/`,
+    `PUT|POST /api/imports/chunked/<id>/parts/<index>/`,
+    `POST /api/imports/chunked/<id>/complete/`, and
+    `GET /api/imports/chunked/<id>/` (progress, for resuming). `complete`
+    returns the **same** response shape as the matching single-request
+    endpoint. Part PUTs use a looser throttle scope (`document_import_chunks`,
+    5000/hour) than the whole-file `document_imports` scope (120/hour).
+  - **Service layer** (`opencontractserver/document_imports/services.py`):
+    `start_chunked_upload` / `store_chunk` / `complete_chunked_upload` plus a
+    streaming reassembler (peak memory bounded to one 8 MB block regardless of
+    file size) and `purge_stale_chunked_uploads`. All four import kinds are
+    supported (single document + the three zip flows). Per-user session
+    isolation (IDOR-safe 404s), arithmetic + integrity validation, and
+    fast-fail permission gating at `start`.
+  - **Periodic cleanup** (`opencontractserver/document_imports/tasks.py`,
+    `config/settings/base.py` `CELERY_BEAT_SCHEDULE`): hourly
+    `purge_stale_chunked_uploads` GCs abandoned sessions and their stored parts
+    after `CHUNKED_UPLOAD_STALE_HOURS` (default 24h).
+  - **New settings** (`config/settings/base.py`):
+    `CHUNKED_UPLOAD_PART_MAX_BYTES` (default 90 MB, must stay below the proxy
+    cap), `CHUNKED_UPLOAD_MAX_PARTS`, `CHUNKED_UPLOAD_STALE_HOURS`, and the
+    `document_import_chunks` throttle rate.
+  - **Frontend** (`frontend/src/utils/importHttp.ts`): the four public import
+    helpers transparently route files above `UPLOAD.CHUNK_THRESHOLD_BYTES`
+    (50 MB) through the chunked protocol — call sites and response handling are
+    unchanged. The artificial 100 MB dropzone cap was raised
+    (`constants.ts`: `MAX_FILE_SIZE_BYTES` → 2 GB for single documents;
+    bulk-zip dropzone now uses the 500 MB `MAX_IMPORT_ZIP_BYTES` cap via new
+    `FileDropZone` `maxSizeBytes`/`maxSizeDisplay` props).
+  - **Resilient frontend transport** (`frontend/src/utils/importHttp.ts`,
+    `constants.ts`): parts now upload with bounded concurrency
+    (`UPLOAD.CHUNK_CONCURRENCY`, default 4) instead of strictly sequentially,
+    each part retries with exponential backoff on transient/5xx/network
+    failures (`CHUNK_MAX_ATTEMPTS`, `CHUNK_RETRY_BASE_DELAY_MS`) while 4xx
+    client errors fail fast, and an optional `onProgress(fraction)` callback
+    on every public import helper drives a progress bar for large uploads.
+  - **GC race hardening** (`opencontractserver/document_imports/services.py`):
+    `purge_stale_chunked_uploads` no longer purges `ASSEMBLING` sessions inside
+    the normal stale window (which could delete parts out from under a live
+    `complete` reassembly); they are reclaimed only after a longer grace window
+    (`CHUNKED_UPLOAD_ASSEMBLING_GRACE_HOURS`, default 6h) so a crashed
+    mid-assembly worker is still cleaned up. The Celery task now also accepts
+    `completed_retention_days` so an operator can override retention at enqueue
+    time. The single-document reassembly's whole-file-in-RAM tradeoff is
+    tracked as a streaming follow-up (issue #1843).
+  - **Tests**: `opencontractserver/tests/test_document_imports_chunked.py`
+    (round-trip byte-exact reassembly, validation, IDOR isolation, integrity
+    checks, zip kinds, stale-session GC, ASSEMBLING grace window) and the
+    `importHttp chunked transport` suite in
+    `frontend/src/utils/__tests__/importHttp.test.ts` (concurrency cap,
+    per-part retry, 4xx fail-fast, progress reporting).
+
 ### Security
 
 - **SmartLabelListMutation IDOR fix** (`config/graphql/smart_label_mutations.py`)

config/settings/base.py

@@ -792,6 +792,11 @@
         "schedule": MEMORY_CURATION_CHECK_INTERVAL_SECONDS,
         "options": {"queue": "celery"},
     },
+    "chunked-uploads-purge-stale": {
+        "task": "opencontractserver.document_imports.tasks.purge_stale_chunked_uploads",
+        "schedule": 3600.0,  # hourly
+        "options": {"queue": "celery"},
+    },
 }
 
 # Worker Upload Processing
@@ -820,6 +825,43 @@
     )
 )
 
+# Chunked (resumable) upload limits
+# ------------------------------------------------------------------------------
+# Back the /api/imports/chunked/* endpoints, which slice a large file into
+# sub-ceiling parts to get past the 100 MB per-request body cap on upstream
+# proxies (Cloudflare). The assembled-file total is still bounded by
+# MAX_DOCUMENT_IMPORT_SIZE_BYTES above.
+#
+# Maximum size (bytes) of a single uploaded part. MUST stay below the smallest
+# upstream proxy body limit (Cloudflare: 100 MB). Default: 90 MB.
+CHUNKED_UPLOAD_PART_MAX_BYTES = int(
+    env("CHUNKED_UPLOAD_PART_MAX_BYTES", default=str(90 * 1024 * 1024))
+)
+# Hard cap on the number of parts in one session (bounds metadata / abuse).
+CHUNKED_UPLOAD_MAX_PARTS = int(env("CHUNKED_UPLOAD_MAX_PARTS", default="100000"))
+# Hours of inactivity before an unfinished session and its stored parts are
+# eligible for garbage collection by ``purge_stale_chunked_uploads``.
+CHUNKED_UPLOAD_STALE_HOURS = int(env("CHUNKED_UPLOAD_STALE_HOURS", default="24"))
+# Days a COMPLETED session row is retained as an audit trail before
+# ``purge_stale_chunked_uploads`` removes it (its parts were already deleted on
+# completion, so this only reclaims small metadata rows). Prevents the table
+# from growing unboundedly. Set to 0 to keep COMPLETED rows forever.
+CHUNKED_UPLOAD_COMPLETED_RETENTION_DAYS = int(
+    env("CHUNKED_UPLOAD_COMPLETED_RETENTION_DAYS", default="30")
+)
+# Grace window (hours) before an ``ASSEMBLING`` session is treated as a crashed
+# worker and made eligible for GC. Deliberately far larger than any real
+# reassembly so the staleness GC can never delete parts out from under a live
+# assembly. See ``document_imports.services.purge_stale_chunked_uploads``.
+CHUNKED_UPLOAD_ASSEMBLING_GRACE_HOURS = int(
+    env("CHUNKED_UPLOAD_ASSEMBLING_GRACE_HOURS", default="6")
+)
+# Block size (bytes) used when streaming stored parts into the reassembled temp
+# file. Bounds peak assembly memory to O(block), independent of file size.
+CHUNK_ASSEMBLY_BLOCK_SIZE = int(
+    env("CHUNK_ASSEMBLY_BLOCK_SIZE", default=str(8 * 1024 * 1024))
+)
+
 # Maximum metadata JSON size (in bytes) accepted by the worker upload endpoint.
 # Default: 500 MB. Set to 0 to disable the limit.
 MAX_WORKER_METADATA_SIZE_BYTES = int(
@@ -895,6 +937,11 @@
         "annotation_images": "200/hour",  # Image retrieval endpoint, authenticated (higher bandwidth)
         "annotation_images_anon": "200/hour",  # Image retrieval endpoint, anonymous
         "document_imports": "120/hour",  # Multipart document import endpoints
+        # Chunked-upload part PUTs: one large file fans out into many part
+        # requests, so this scope is far looser than ``document_imports``
+        # (which is sized for whole-file imports). ``start``/``complete`` stay
+        # on the strict ``document_imports`` scope.
+        "document_import_chunks": "5000/hour",
     },
 }
 

frontend/src/assets/configurations/constants.ts

@@ -137,17 +137,60 @@ export const DEBOUNCE = {
 
 // Upload constraints
 export const UPLOAD = {
-  /** Maximum file size in bytes (100MB) */
-  MAX_FILE_SIZE_BYTES: 100 * 1024 * 1024,
-  /** Maximum file size display string */
-  MAX_FILE_SIZE_DISPLAY: "100MB",
+  /**
+   * Maximum single-document size in bytes (2GB). Files larger than
+   * ``CHUNK_THRESHOLD_BYTES`` are uploaded in chunks (see ``importHttp``),
+   * so this is no longer bounded by the 100MB upstream proxy (Cloudflare)
+   * request cap. The backend ``MAX_DOCUMENT_IMPORT_SIZE_BYTES`` is the
+   * authoritative ceiling and returns 413 above it.
+   */
+  MAX_FILE_SIZE_BYTES: 2 * 1024 * 1024 * 1024,
+  /** Maximum single-document size display string */
+  MAX_FILE_SIZE_DISPLAY: "2GB",
+  /**
+   * Slice size (50MB) for chunked uploads. Must stay below the smallest
+   * upstream proxy body limit (Cloudflare caps proxied requests at 100MB);
+   * 50MB leaves ~2x headroom for multipart framing. Mirrors the backend
+   * ``CHUNKED_UPLOAD_PART_MAX_BYTES`` guard (90MB).
+   *
+   * NOTE: this MUST stay below the backend ``CHUNKED_UPLOAD_PART_MAX_BYTES``
+   * (``config/settings/base.py``); a part larger than that backend cap is
+   * rejected with a 413 at ``store_chunk``. Do not widen this past the
+   * backend value without bumping it too.
+   */
+  CHUNK_SIZE_BYTES: 50 * 1024 * 1024,
+  /**
+   * Files larger than this are uploaded via the chunked endpoints instead
+   * of a single request. Equal to one chunk, so anything that would risk
+   * the proxy cap is chunked while small files keep the single-shot path.
+   */
+  CHUNK_THRESHOLD_BYTES: 50 * 1024 * 1024,
+  /**
+   * How many parts to upload concurrently. The backend serialises writes
+   * per ``(session, index)`` and supports idempotent re-upload, so parallel
+   * part PUTs are safe; 4 keeps a fat pipe busy without overwhelming the
+   * browser's per-host connection pool.
+   */
+  CHUNK_CONCURRENCY: 4,
+  /**
+   * Maximum attempts per part before the whole upload aborts. The backend
+   * persists parts and accepts idempotent re-upload, so a transient network
+   * blip on one part of a long upload can be retried instead of discarding
+   * the whole transfer.
+   */
+  CHUNK_MAX_ATTEMPTS: 3,
+  /**
+   * Base delay (ms) for exponential backoff between part retries:
+   * attempt N waits ``CHUNK_RETRY_BASE_DELAY_MS * 2**(N-1)``.
+   */
+  CHUNK_RETRY_BASE_DELAY_MS: 500,
   /** Progress percentage shown while bulk upload is in flight (before completion) */
   BULK_PROGRESS_INITIAL: 50,
   /** Maximum number of corpuses to show in the inline selector preview */
   CORPUS_PREVIEW_LIMIT: 5,
-  /** Maximum corpus-import ZIP size in bytes (500MB) */
+  /** Maximum corpus-import / bulk ZIP size in bytes (500MB) */
   MAX_IMPORT_ZIP_BYTES: 500 * 1024 * 1024,
-  /** Maximum corpus-import ZIP size display string */
+  /** Maximum corpus-import / bulk ZIP size display string */
   MAX_IMPORT_ZIP_DISPLAY: "500MB",
 } as const;
 

frontend/src/components/widgets/modals/UploadModal/UploadModal.tsx

@@ -457,6 +457,8 @@ export const UploadModal: React.FC<UploadModalProps> = ({
                 selectedFile={zipFile}
                 disabled={isUploading}
                 acceptedFileTypes={acceptedFileTypes}
+                maxSizeBytes={UPLOAD.MAX_IMPORT_ZIP_BYTES}
+                maxSizeDisplay={UPLOAD.MAX_IMPORT_ZIP_DISPLAY}
                 onFilesSelected={handleFilesSelected}
                 onFileRejected={handleFileRejected}
               />

frontend/src/components/widgets/modals/UploadModal/components/FileDropZone.tsx

@@ -23,6 +23,15 @@ interface FileDropZoneProps {
   hasFiles?: boolean;
   /** Accepted file types for single mode (from backend). Falls back to PDF-only. */
   acceptedFileTypes?: AcceptedFileType[];
+  /**
+   * Maximum accepted file size in bytes. Defaults to the single-document
+   * cap; callers pass the ZIP cap for bulk mode. Files above the cap are
+   * still chunked on upload, but the dropzone bounds what a user can pick
+   * so the backend's per-flow ceiling isn't exceeded.
+   */
+  maxSizeBytes?: number;
+  /** Human-readable form of ``maxSizeBytes`` (e.g. "2GB"). */
+  maxSizeDisplay?: string;
   onFilesSelected: (files: File[]) => void;
   onFileRejected?: (rejections: FileRejection[]) => void;
 }
@@ -39,6 +48,8 @@ export const FileDropZone: React.FC<FileDropZoneProps> = ({
   selectedFile,
   hasFiles = false,
   acceptedFileTypes,
+  maxSizeBytes = UPLOAD.MAX_FILE_SIZE_BYTES,
+  maxSizeDisplay = UPLOAD.MAX_FILE_SIZE_DISPLAY,
   onFilesSelected,
   onFileRejected,
 }) => {
@@ -93,7 +104,7 @@ export const FileDropZone: React.FC<FileDropZoneProps> = ({
     accept: acceptConfig,
     multiple: mode === "single",
     disabled: disabled || (mode === "single" && hasFiles),
-    maxSize: UPLOAD.MAX_FILE_SIZE_BYTES,
+    maxSize: maxSizeBytes,
   });
 
   const handleBrowseClick = useCallback(
@@ -115,13 +126,13 @@ export const FileDropZone: React.FC<FileDropZoneProps> = ({
         const oversizedFiles: FileRejection[] = [];
 
         for (const file of files) {
-          if (file.size > UPLOAD.MAX_FILE_SIZE_BYTES) {
+          if (file.size > maxSizeBytes) {
             oversizedFiles.push({
               file,
               errors: [
                 {
                   code: "file-too-large",
-                  message: `File exceeds ${UPLOAD.MAX_FILE_SIZE_DISPLAY} limit`,
+                  message: `File exceeds ${maxSizeDisplay} limit`,
                 },
               ],
             });
@@ -223,8 +234,8 @@ export const FileDropZone: React.FC<FileDropZoneProps> = ({
         </div>
         <div className="secondary-text">
           {mode === "bulk"
-            ? `ZIP should contain documents (max ${UPLOAD.MAX_FILE_SIZE_DISPLAY})`
-            : `Supported: ${acceptedLabels} · Max ${UPLOAD.MAX_FILE_SIZE_DISPLAY} per file`}
+            ? `ZIP should contain documents (max ${maxSizeDisplay})`
+            : `Supported: ${acceptedLabels} · Max ${maxSizeDisplay} per file`}
         </div>
       </DropZoneText>
       <Button