aws-cdr-gateway

A serverless AWS pipeline that performs Content Disarmament and Reconstruction (CDR) on files uploaded to S3. Files are structurally disarmed — active/executable content is stripped — and routed to a sanitised or quarantine bucket. The pipeline is designed to fail closed: anything it cannot prove safe is quarantined, never labelled sanitised.

Status: production-ready code, Deployed and smoke tested on AWS. Follow, the deployment steps below require live AWS credentials and either the SAM CLI or OpenTofu/Terraform.

---

How it works

Detailed diagrams of the full dual-layer (CDR + malware scan + aggregator) design:

EventBridge fires on S3 ObjectCreated (PutObject / CompleteMultipartUpload) and invokes the CDR Lambda, which strips active content and routes each file to the sanitised or quarantine bucket. This repo contains the CDR Lambda and its infrastructure — the malware-scan Lambda, result aggregator, and presigned-upload backend are out of scope (see Known gaps).

CDR by format

Format	Approach	What is removed
Office (all 17 OOXML/ZIP variants)	ZIP-level surgery — drop/scrub parts, never re-serialise through an Office library	VBA, ActiveX, embedded OLE/package objects, external links, query tables, connections, web add-ins, macro content types, dangerous .rels, field codes (DDE/AUTOOPEN/WEBSERVICE/…), altChunk imports; external hyperlink targets neutralised in place
xlsb (with worksheet binaries)	Format conversion via pyxlsb → openpyxl	Everything — only plain cached cell values survive; no BIFF12 records pass through
PDF	pikepdf surgery + full re-serialise	JavaScript, /OpenAction, /AA, embedded files, all annotation actions, /FileAttachment specs, multimedia annotations, outline (bookmark) actions, AcroForm field + root actions
Images (jpg/png/gif/bmp/tiff/webp)	Re-encode through Pillow	All EXIF / ICC / XMP metadata; GIF comment blocks; per-frame metadata on multi-frame TIFFs
Legacy OLE (doc/xls/ppt)	Quarantine, no CDR	— (format too opaque to safely reconstruct)
Unknown extensions	Fail closed — quarantine	Never reaches the sanitised bucket

Security guarantees baked in

• Decompression-bomb defence — every ZIP entry is read through a chunked byte counter (_read_zip_entry_safe) that never trusts the attacker-controlled central-directory file_size. A 90 KB entry that inflates to gigabytes is killed at the per-entry limit.
• ZIP structural hard-rejects — bad magic, non-standard compression, duplicate entries, local/central method mismatch, or a missing [Content_Types].xml → quarantine, no CDR.
• ReDoS-hardened — all neutralisation regexes are bounded; a crafted text node cannot hang the Lambda past its timeout.
• Fault isolation — SNS publish, source delete, and metric emission can never turn a successful CDR into an EventBridge retry of an already-sanitised file. adds a magic-byte gate; security guards are kept at behavioural parity.

The CDR code has been through five review passes (Codex → Gemini → a multi-agent adversarial audit → two remediation rounds); the Terraform/build/CI was separately audited and hardened. See docs/progress.md.

---

Repository layout

Path	Purpose
src/lambda_function.py	General CDR Lambda — all Office/PDF/image formats; exposes the pure cdr_dispatch decision core
src/app.py	Local CDR service — FastAPI wrapper around cdr_dispatch; disarm files over HTTP with no AWS account
src/requirements-local.txt	Extra deps for the local service (fastapi, uvicorn) — not part of the Lambda layer
src/requirements-dev.txt	Test-only deps (pytest, python-docx) — in neither the Lambda layer nor the container image
src/test_cdr_local.py	Tests for the local service + cdr_dispatch core (own file, per the per-module rule)
docs/local-cdr.md	Local CDR service guide — run, configure, embed, proxy, API contract, security model
docs/local_cdr_architecture.svg	"One core, two front-ends" diagram (cloud handler + local app.py over cdr_dispatch)
Dockerfile	Container image for the local CDR service (slim, non-root, healthcheck)
docker-compose.yml	Runnable sidecar example (hardened: read-only, cap-drop, healthcheck)
docs/deploy-container.md	Container / Compose / Kubernetes-sidecar deploy guide + hardening checklist
src/template.yaml	AWS SAM infrastructure (buckets, IAM, DLQ, alarms, EventBridge)
terraform/	Terraform port of the SAM template (parallel deploy path)
scripts/build.sh	Builds the Lambda zip with Linux wheels (for the Terraform path)
src/requirements.txt	Pinned dependencies for the tests, the container image and local dev
scripts/lambda-requirements.txt	What the deployed Lambda is actually built from — the same versions, hash-pinned and installed --require-hashes. Regenerate with scripts/regen_lambda_requirements.py; CI fails if it drifts from src/requirements.txt
docs/00–04	Production-readiness manuals (setup, smoke tests, IAM review, runbook)
docs/deployment-runbook.md	End-to-end staging deploy guide
docs/05-alarm-demo-walkthrough.md	Subscribe to alarm notifications + fire the ZIP-anomaly alarm (live demo)
docs/benchmark.py	Standalone load benchmark with tuning recommendations
docs/fixtures/	Threat fixtures + generator covering every CDR path
docs/comparison-docbleach.md	CDR coverage compared against DocBleach (per-format, with audit rationale)
docs/cdr-gap-analysis-stevens.md	Gap analysis vs Didier Stevens' maldoc toolkit (oledump/pdfid/zipdump/emldump) — what's caught, where, and the JBIG2/JPX hardening

---

Development

The Lambda runtime is Python 3.12 (per template.yaml); the local dev venv may be newer. Dependencies are pinned in two files that must stay in step — src/requirements.txt (tests, container image, local dev) and scripts/lambda-requirements.txt (hash-pinned, what the deployed Lambda is built from). Bumping a dependency means changing both; see Bumping a dependency.

# Create a venv (none is committed to the repo)
python3.12 -m venv .venv && source .venv/bin/activate

# Install dependencies (Lambda + local-service + test-only deps)
pip install -r src/requirements.txt -r src/requirements-local.txt -r src/requirements-dev.txt

# Run the full test suite (454 tests: 404 CDR Lambda + 50 local variant)
cd src && pytest test_cdr.py test_cdr_local.py -v

# Run one class or test
cd src && pytest test_cdr.py::TestOfficeCDR -v
cd src && pytest test_cdr.py::TestOfficeCDR::test_vba_macro_removed -v

# Lint (byte-compile)
cd src && python -m py_compile lambda_function.py app.py

Tests construct malicious fixtures entirely in memory; S3/SNS are mocked. No live AWS credentials are needed to run them. src/requirements-local.txt (FastAPI/uvicorn) is only needed for the local CDR service and its tests — it is deliberately excluded from the deployed Lambda package.

Run pytest bare — do not export SANITISED_BUCKET/QUARANTINE_BUCKET around it. The test defaults are set with os.environ.setdefault, which yields to anything already in the environment, so exported bucket names override them and fail two tests that assert on the literal names — an environmental failure that reads exactly like a code regression.

Bumping a dependency

src/requirements.txt is what the tests and the container image install; scripts/lambda-requirements.txt is what the deployed Lambda is built from. A bump applied to one file alone ships a different wheel than the one the tests exercised — pikepdf and Pillow drifted two releases behind exactly this way, and the Pillow gap moved bundled native code (liblcms2, libpng16) that CI never ran. Dependabot watches both directories, but each PR bumps a single file, so the two still have to be reconciled.

# 1. edit the version in src/requirements.txt, then regenerate the hash-pinned file
python scripts/regen_lambda_requirements.py

# 2. verify they agree (this is what CI runs)
python scripts/check_lambda_requirements.py

The regenerator resolves wheels for the Lambda target (manylinux_2_28_x86_64, CPython 3.12) and rewrites the pins with their sha256 hashes. boto3 is exempt — the Lambda runtime provides it, so it is deliberately absent from the shipped set.

---

Local CDR service (no AWS account)

Drop-in file disarming for any app — one POST, no AWS, no state. Run it as a container sidecar next to your service: every upload goes through CDR before you trust it, and you get back a clean file (or a fail-closed rejection). Same disarm engine as the AWS pipeline, on plain HTTP, in any language.

• 🛡️ Same engine as the cloud — the local service and the Lambda share one pure, I/O-free core (cdr_dispatch), so a file disarmed locally is disarmed by identical logic. No second implementation to drift or fall behind.
• 🔌 Language-agnostic — it's an HTTP endpoint. POST bytes, read bytes. Python, Go, Node, Java, a shell script — all integrate the same way.
• ☁️ Zero AWS — no account, no credentials, no S3/SNS. Runs on a laptop, in CI, on-prem, or air-gapped.
• 📦 Container-ready — a small, non-root, read-only-capable image with a built-in healthcheck. docker run and you have a disarming sidecar.
• 🔒 Fail-closed by design — anything it can't disarm (RTF, legacy OLE, unknown extensions, malformed archives) is rejected, never passed through as "clean".

The Lambda's CDR routing — size guard, fail-closed unknown-extension gate, RTF/legacy rejection, ZIP structural validation, and the per-format disarm of Office/PDF/images — is factored into a pure, I/O-free function cdr_dispatch(data, ext) in lambda_function.py. The cloud handler and the local app.py both call it, so the two cannot drift on a security decision; there is no second CDR implementation.

Full reference: docs/local-cdr.md — run, configure, embed, deploy-behind-a-proxy, the API contract, the security model, and the threat/hardening matrix.

app.py is a thin FastAPI wrapper: it never touches S3, SNS, or any AWS service. Any local app can disarm a file by POSTing it.

cd src
pip install -r requirements.txt -r requirements-local.txt
uvicorn app:app --host 127.0.0.1 --port 8000      # or: python app.py

# Disarm a macro-enabled doc → get a clean .docx back (note the extension remap)
curl -sS -o clean.docx -D - -F file=@dirty.docm http://127.0.0.1:8000/sanitise
#   200 OK
#   x-cdr-status: sanitised
#   x-cdr-sanitised-ext: docx
#   x-cdr-removals: 1
#   x-cdr-report: {"format":"docm","removed":["word/vbaProject.bin"],...}

# Fail-closed: RTF and unknown extensions are rejected, never "sanitised"
curl -sS -F file=@evil.rtf http://127.0.0.1:8000/sanitise
#   422 {"status":"unsupported-format","reason":"format rejected by design: rtf",...}

Endpoint	Behaviour
POST /sanitise (multipart file)	200 + clean bytes (X-CDR-* headers carry status/report) on success; 413 JSON when the upload exceeds CDR_MAX_FILE_BYTES; 422 JSON for rejected/unsupported input; 500 JSON for an unparseable file
GET /healthz	Liveness + the formats this build will attempt to disarm

This is a single-process service for trusted local/internal use (a sidecar, a desktop integration, a batch tool) — it has no built-in auth or rate limiting. Put it behind your own controls before exposing it beyond localhost. The HTTP layer is nonetheless hardened: the body is size-bounded before it is fully buffered (early Content-Length reject plus an authoritative counted read, so a multi-GB upload can't OOM the process); the Content-Disposition filename is RFC 6266/5987 encoded (no header-injection via a crafted filename); response headers carry only sanitised, length-capped values; and internal errors return a generic message (the real exception is logged server-side only).

Tests for the local variant live in src/test_cdr_local.py (its own file, per the per-module test rule) and prove both that cdr_dispatch does no I/O and that the endpoint returns the right status for every routing branch:

cd src && pytest test_cdr_local.py -v

Run as a container / sidecar

A small, non-root, healthcheck-equipped image makes the service a drop-in disarming sidecar:

# Pull the published multi-arch image (amd64 + arm64)
docker run --rm -p 8000:8000 ghcr.io/douglasmun/aws-cdr-gateway:latest

# …or build from source
docker build -t cdr-gateway:local .
docker run --rm -p 8000:8000 cdr-gateway:local
curl -sS -o clean.docx -F file=@dirty.docm http://localhost:8000/sanitise

# or wire it next to your app (with hardening) via Compose:
docker compose up --build

Your app calls http://cdr:8000/sanitise and uses the disarmed response — see the commented app: service in docker-compose.yml. Full container, Compose, and Kubernetes-sidecar guidance (plus the hardening checklist) is in docs/deploy-container.md.

Git hooks

main has no server-side branch protection (a GitHub free-private-repo limitation — see Known Gap #6). A local pre-push hook stands in for it: it blocks force-pushes and deletions of main in your clone. Install it once after cloning:

./scripts/install-hooks.sh

The hook is tracked in scripts/hooks/; .git/hooks/ is not version-controlled, so each clone must install it. Bypass for a one-off with git push --no-verify.

---

Deployment

Two equivalent IaC paths provision the same stack — pick one. Both require live AWS credentials. See docs/deployment-runbook.md for the full guide and docs/00-production-readiness-index.md for the sign-off checklist.

Option A — AWS SAM (needs the SAM CLI):

cd src
sam build
sam deploy --guided          # choose bucket names, quarantine bucket, SNS subscribers

Option B — OpenTofu / Terraform (build the package, then apply; see terraform/README.md):

./scripts/build.sh           # Linux wheels + handler → build/lambda.zip
cd terraform
cp terraform.tfvars.example terraform.tfvars   # set bucket names + region
tofu init && tofu apply      # or: terraform init && terraform apply

Smoke test + load benchmark against the deployed stack:

python docs/benchmark.py --bucket <source-bucket> --files docs/fixtures/ \
  --log-group /aws/lambda/cdr-lambda

Key configuration (SAM parameters / Terraform variables / env vars)

Variable	Required	Default	Purpose
SANITISED_BUCKET	yes	—	destination for clean files
QUARANTINE_BUCKET	no	—	destination for rejected/errored/unsupported files
RESULT_TOPIC_ARN	no	—	SNS topic for CDR result metadata
CDR_MAX_FILE_BYTES	no	100 MB	pre-download size limit
CDR_MAX_ENTRY_BYTES	no	200 MB	per-ZIP-entry decompression limit

The Lambda runs at 1024 MB / 300 s timeout with ReservedConcurrentExecutions: 20; an SQS DLQ and CloudWatch alarms (errors, p99 duration, throttles, DLQ depth, passthrough, ZIP anomalies) are provisioned by the template.

---

License

MIT © 2026 Douglas Mun