feat: five-bucket app improvement pass (correctness, ATT&CK, HTTP analysis, API, CI)

.dockerignore

@@ -16,3 +16,4 @@ docker-compose.yml
 *.db
 *.pcap
 *.pcapng
+!pcaps/demo.pcap

.github/dependabot.yml

@@ -0,0 +1,8 @@
+version: 2
+updates:
+  - package-ecosystem: pip
+    directory: "/"
+    schedule: {interval: weekly}
+  - package-ecosystem: github-actions
+    directory: "/"
+    schedule: {interval: weekly}

.github/workflows/ci.yml

@@ -51,10 +51,67 @@ jobs:
         run: pip install -r requirements.txt
 
       - name: Run tests with coverage
-        run: PYTHONPATH=. pytest tests/ -v --cov=app
+        run: PYTHONPATH=. pytest tests/ -v --cov=app --cov-report=term-missing --cov-fail-under=58
 
       - name: Ruff lint
         run: ruff check .
 
       - name: Ruff format check
         run: ruff format --check .
+
+  audit:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    # Advisory only: a discovered CVE surfaces here without blocking merges.
+    # The required check for branch protection stays `test`.
+    continue-on-error: true
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v5
+        with:
+          persist-credentials: false
+
+      - name: Set up Python 3.11
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.11"
+          cache: pip
+
+      - name: Install pip-audit
+        run: pip install pip-audit
+
+      - name: Audit dependencies
+        run: pip-audit -r requirements.txt
+
+  docker:
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    # Advisory only: the Dockerfile is the canonical deploy artifact (two
+    # Zeek OBS-repo installs, WeasyPrint/kaleido system libs, multi-stage
+    # builder->runtime->test) but the `test` job above never builds it, so a
+    # broken apt pin or requirements resolution failure would only surface
+    # when someone runs `make docker-build`. This job builds the `test`
+    # target and runs its suite, which also has real zeek installed (the
+    # `test` job above skips zeek-dependent e2e tests). It is deliberately
+    # NOT the required check — that stays `test`.
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v5
+        with:
+          persist-credentials: false
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build test-stage image (mirrors make docker-verify)
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          target: test
+          tags: pcap-hunter:test
+          load: true
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Run in-image test suite (ruff format + lint + pytest, with real zeek)
+        run: docker run --rm pcap-hunter:test

.gitignore

@@ -39,6 +39,7 @@ htmlcov/
 data/
 *.pcap
 !tests/fixtures/*.pcap
+!pcaps/demo.pcap
 *.log
 .DS_Store
 

CHANGELOG.md

@@ -19,7 +19,7 @@ First stable release. Production-ready installer, hardened pipeline, polished UX
 
 ### Changed
 - **PDF cover page redesign** — logo + tagline above the title, with classification banner and metadata block.
-- **Kaleido upgraded to 1.x** — 0.x reaches end-of-life September 2025; 1.x is the active branch.
+- **Kaleido pinned to `0.2.1`** — kaleido 1.x refuses to install alongside the pinned plotly 5.x ("not compatible"), so 0.2.1 is the only working pairing; it bundles its own headless Chromium and needs no system browser. See `requirements.txt` for the full rationale, and revisit together with a future `plotly>=6.1` upgrade.
 - **Testing discipline overhauled** — production-shape test data (real `CorrelationSignal` dataclasses, real DataFrames, nested dicts the pipeline actually produces) instead of simplified inputs. Documented in `CLAUDE.md` with bug-pattern history. New integration tests for every PDF section and chart.
 - **Version bumped to 1.0.0** with consolidated release notes.
 

CLAUDE.md

@@ -198,6 +198,7 @@ GitHub Actions (`.github/workflows/ci.yml`):
 - Default PyShark limit: 200,000 packets
 - OSINT top IPs default: 50
 - `MAX_FLOW_SAMPLES`: 5,000 per-flow packet timestamps/lengths (true totals kept in `count`/`first_ts`/`last_ts`)
+- `ZEEK_TABLE_MAX_ROWS`: 50,000 per-log row cap on in-memory Zeek tables (DNS analysis + UI preview read these capped frames; JA3 reads the full uncapped log via `zeek_log_paths`); truncation appends `WARNING_ZEEK_TRUNCATED` to `PipelineResult.warnings`
 - `RUN_DIR_RETENTION_SECONDS`: 7 days — per-run `data/zeek|carved/<run_id>/` dirs pruned on the next run
 - Subprocess timeouts: `ZEEK_TIMEOUT_SECONDS` 600, `PCAP_COUNT_TIMEOUT_SECONDS` 120, `CARVE_TIMEOUT_SECONDS` 300, `TLS_EXTRACT_TIMEOUT_SECONDS` 300, `LLM_PROBE_TIMEOUT_SECONDS` 15
 

CONTRIBUTING.md

@@ -0,0 +1,142 @@
+# Contributing to PCAP Hunter
+
+Thanks for your interest in contributing. This is a concise guide to how the
+repo actually works day-to-day — see `CLAUDE.md` for the full architecture
+and conventions reference.
+
+## Getting set up
+
+```bash
+git clone <repo-url>
+cd pcap-hunter
+make install          # delegates to scripts/install.py (cross-platform)
+```
+
+`make install` installs both system binaries (tshark, Zeek, etc.) and Python
+dependencies, then verifies them. Useful variants:
+
+```bash
+make install-system   # system binaries only
+make install-python   # Python packages only
+make check-deps       # or `make doctor` — verify everything is present
+```
+
+Run the app locally with:
+
+```bash
+make run              # streamlit run app/main.py (checks deps first)
+```
+
+## Before you commit: `make verify`
+
+**`make verify` is the pre-commit gate and it must pass before every
+commit.** It runs, in order:
+
+1. `ruff format --check .` — formatting check
+2. `ruff check .` — lint
+3. `PYTHONPATH=. pytest tests/ -q` — full test suite
+
+```bash
+make verify
+```
+
+CI (`.github/workflows/ci.yml`) runs the same three checks on every push and
+PR to `main`, plus an advisory `pip-audit` dependency-scan job and a Docker
+build/test job. If `make verify` passes locally, it will pass in CI.
+
+You can also run the pieces individually:
+
+```bash
+make test             # PYTHONPATH=. pytest tests/ -v --cov=app --cov-report=term-missing --cov-fail-under=58
+make test-pdf         # focused PDF/chart smoke tests — run after touching pdf_generator.py, chart_images.py, or the charts module
+make lint             # ruff check .
+make format           # ruff format .
+```
+
+Always run tests with `PYTHONPATH=.` (or via `make test`/`make verify`,
+which set it for you) — the codebase uses absolute imports
+(`from app.pipeline.beacon import rank_beaconing`) and needs it on the path.
+
+### Host-Python caveat
+
+macOS machines frequently have multiple coexisting Python installs (Framework
+Python, Homebrew Python, pyenv, etc.). `make test`/`make verify` use whichever
+interpreter `streamlit` is installed under (see the `PYTHON` detection at the
+top of the `Makefile`), but if your environment has `pytest` and other
+dependencies split across interpreters, `make verify` on the host can fail
+in ways that don't reflect a real problem. If you hit interpreter confusion:
+
+- confirm `pip show pytest` and `pip show streamlit` agree on the same
+  interpreter, or
+- fall back to the canonical, environment-independent path below.
+
+### Docker: the canonical build-and-verify path
+
+Any verification that depends on a clean install (dependency changes,
+install-path changes, or anything you want to be **certain** works outside
+your local environment) should go through Docker rather than the host:
+
+```bash
+make docker-verify    # builds the `test` image and runs format+lint+tests inside it
+make docker-up         # build + run the UI at http://localhost:8501
+make docker-down
+```
+
+This mirrors what CI's `docker` job does and is the same environment the app
+ships in, so it's the most trustworthy signal for anything build-shaped.
+
+## Code conventions
+
+- **Style**: Ruff, line length 120, double quotes, 4-space indent (see the
+  `select` list and per-file ignores in `pyproject.toml` for the exact rules)
+- **Imports**: absolute only (`from app.pipeline.beacon import ...`),
+  stdlib → third-party → local, ordering enforced by ruff's `I` rule
+- **Naming**: `snake_case.py` modules, `PascalCase` classes, `snake_case`
+  functions, `UPPER_SNAKE_CASE` constants, leading underscore for private
+  helpers
+- **Data modeling**: prefer `dataclass` for structured data, `Enum` for
+  fixed categories
+- **Errors/logging**: custom exceptions inherit from `Exception`; use the
+  `logging` module, never `print()`
+- **Type hints**: used extensively, with `from __future__ import annotations`
+  for forward compatibility
+- **Docstrings**: Google-style (`Args`/`Returns`) on public functions
+
+## Tests
+
+- One test file per major module: `tests/test_<module>.py`
+- Test classes `Test<Feature>`, test functions `test_<scenario>`
+- Cover both the happy path and edge cases (empty, `None`, malformed input)
+- **Use production-shape test data.** If a function consumes
+  `list[CorrelationSignal]` dataclasses, pass real dataclass instances in
+  tests, not dicts with similar-looking keys — simplified test inputs have
+  previously let real bugs ship. See `tests/test_pdf_integration.py` for the
+  expected shapes.
+- No shared `conftest.py` fixtures — tests are independent
+- New PDF sections need a corresponding assertion in
+  `tests/test_pdf_integration.py::test_html_contains_every_expected_section`;
+  new PDF charts need a kaleido smoke test in `tests/test_chart_rendering.py`
+
+## Commit messages
+
+Conventional-commits style, lowercase description after the prefix:
+
+```
+feat: add single-ioc exact-match lookup endpoint
+fix: escape crlf in cef output to prevent syslog log injection
+docs: refresh readme and user manuals
+chore: cover all runtime deps in dependency check
+```
+
+Common prefixes: `feat:`, `fix:`, `docs:`, `style:`, `chore:`.
+
+## Submitting changes
+
+1. Make your change, keeping it focused.
+2. Run `make verify` (and `make docker-verify` if the change touches
+   dependencies, install paths, or anything build-shaped).
+3. Commit using the conventions above.
+4. Open a pull request against `main` describing what changed and why.
+
+CI must pass (tests + coverage floor, lint, format check) before a PR can be
+merged.

Dockerfile

@@ -61,6 +61,10 @@ RUN pip install --no-cache-dir /wheels/*
 # The previous flattened COPY (app/ -> /app/) broke `uvicorn app.api.app`.
 COPY app/ ./app/
 
+# Synthetic first-run demo capture (see pcaps/_make_demo.py) — powers the
+# "Load demo capture" button so a fresh install has something to analyze.
+COPY pcaps/demo.pcap ./pcaps/demo.pcap
+
 # Non-root + data dirs
 RUN useradd -m runner && mkdir -p /data /app/data && chown -R runner:runner /app /data
 USER runner

Makefile

@@ -29,7 +29,7 @@ check-deps doctor:
 # -------------------------------------------------------------------------
 
 test:
-	PYTHONPATH=. pytest tests/ -v --cov=app
+	PYTHONPATH=. pytest tests/ -v --cov=app --cov-report=term-missing --cov-fail-under=58
 
 # Focused smoke test: PDF generation with charts + correlations + beacon DF.
 # Run this after any change to pdf_generator.py, chart_images.py, main.py

SECURITY.md

@@ -0,0 +1,67 @@
+# Security Policy
+
+## Supported versions
+
+PCAP Hunter is developed on a single rolling `main` branch — there is no
+long-term-support branch. Security fixes are applied to `main` and released
+as the next version; only the latest released version is supported.
+
+| Version | Supported |
+|---------|-----------|
+| latest (`main`) | yes |
+| older releases  | no  |
+
+## Reporting a vulnerability
+
+Please do not open a public GitHub issue for security vulnerabilities.
+
+Instead, report privately using one of:
+
+- Open a private GitHub Security Advisory for this repository
+  (repo → **Security** tab → **Advisories** → **Report a vulnerability**), or
+- Email the maintainer directly (see the repository's commit history /
+  GitHub profile for a current contact) with a description of the issue,
+  affected version/commit, and reproduction steps.
+
+Please include enough detail to reproduce the issue (PCAP sample or steps,
+affected module/endpoint, expected vs. actual behavior). We'll acknowledge
+reports and follow up with next steps as the issue is triaged.
+
+## Scope and hardening expectations
+
+A few things are worth calling out explicitly because of what this tool does:
+
+- **PCAP Hunter parses hostile, untrusted input by design.** PCAP files come
+  from real (potentially adversary-controlled) network traffic, and the
+  pipeline runs several native tools (Zeek, tshark, PyShark) and file-format
+  parsers (TLS/X.509, YARA) against that input. Treat any parser crash,
+  memory-safety issue, or resource-exhaustion bug in these paths as a
+  security-relevant finding, not just a bug.
+- **The app makes outbound network calls.** OSINT enrichment (VirusTotal,
+  AbuseIPDB, GreyNoise, OTX, Shodan, WHOIS, reverse DNS) and LLM calls (LM
+  Studio, OpenAI, Anthropic) send indicators and/or sanitized context to
+  third-party or self-hosted endpoints. If you operate in an environment
+  where that egress is sensitive, review `app/security/opsec.py` and the
+  Config tab before enabling providers, and consider network-level egress
+  controls.
+- **The Streamlit UI has no built-in authentication.** It is designed to run
+  as a local, single-analyst tool. Do not expose it directly to an untrusted
+  network or the public internet. If remote/shared access is required, put
+  it behind a reverse proxy that terminates TLS and enforces authentication
+  (e.g., an OAuth2 proxy, SSO gateway, or VPN-only access) rather than
+  relying on Streamlit itself for access control. The same applies to the
+  optional integrations API (`app/api/`) — run it behind a proxy/firewall
+  and use its API-key auth; do not expose it unauthenticated to the internet.
+- **Config secrets are encrypted at rest but machine-bound.** API keys saved
+  via the Config tab are encrypted with a PBKDF2 key derived from local
+  machine identifiers (see `ConfigManager` in `CLAUDE.md`). Don't commit
+  `~/.pcap_hunter_config.json` or `.env` files, and don't share them across
+  machines expecting the encryption to travel with them.
+
+## Dependency scanning
+
+Dependencies are scanned on every push/PR via an advisory `pip-audit` CI job
+(see `.github/workflows/ci.yml`) and kept up to date via Dependabot
+(`.github/dependabot.yml`, weekly for both `pip` and GitHub Actions). Findings
+there don't block merges automatically but are reviewed as part of normal
+maintenance.