Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,37 @@

All notable changes to this project will be documented in this file.

## [2.1.0] - 2026-07-17

Version 2.1 adds explicit control over how much analysis evidence is sent to an LLM, while improving report grounding, background-processing parity, and performance on larger investigations.

### Why 2.1.0

This is a minor-version release because it adds backward-compatible, user-facing LLM controls and meaningful runtime improvements without intentionally removing API routes, changing authentication contracts, or requiring a destructive data migration. Existing installations adopt a safe 32K-token default with a 50% input budget; saved cases and configuration continue to load, and SQLite schema updates remain automatic.

### Added

- **Adjustable model context window** in Config → LLM Integration, ranging from 10K to 1M tokens and persisted across sessions.
- **Conservative 50% input budget** that scales retained flows, IOCs, OSINT records, protocol evidence, and other report context with the selected window while reserving the other half for output and provider/tokenizer variance.
- **No context window limit** option that disables the slider and sends all available sanitized analysis context in one request, including through LM Studio's OpenAI-compatible endpoint.
- **Shared context-budget utilities and tests** covering normalization, multilingual token estimates, proportional evidence limits, prompt fitting, output caps, and unlimited mode.

### Changed

- **LLM reports are more evidence-grounded** with clearer coverage statements, observed-versus-inferred distinctions, deterministic ATT&CK constraints, calibrated uncertainty, and richer IOC/risk evidence.
- **Background and API-triggered reports now receive foreground-equivalent context**, including correlations, flow anomalies, JA3 evidence, reverse DNS, final ATT&CK mapping, capture metrics, completed stages, and warnings.
- **Case storage scales better** through WAL mode, busy timeouts, bounded list queries, batch IOC reads/writes, deterministic compressed JSON, and a stale-job lookup index.
- **Large dashboards remain responsive** by capping browser-side profile samples, time-stratifying flow markers, and aggregating long capture timelines without dropping volume totals.
- **Geographic selectors use cached indexes** instead of repeatedly scanning the complete city dataset on every Streamlit rerun.
- **Docker builds use a bind-mounted wheelhouse**, keeping build artifacts out of the runtime image, and Streamlit's supported floor is now 1.50 for the current width/iframe APIs.

### Fixed

- Database-backed API keys are authenticated once per request and their resolved names are reused for accurate audit logging.
- Persisted false-valued settings, including the unlimited-context toggle, now survive explicit config reloads.
- Foreground and background report paths now apply the selected context policy consistently.
- Case list summaries include tags and analysis counts without loading every saved analysis.

## [2.0.0] - 2026-07-14

Version 2 turns PCAP Hunter into a more evidence-aware investigation workbench and strengthens both interactive and headless analysis paths.
Expand Down
21 changes: 7 additions & 14 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,18 +1,10 @@
# syntax=docker/dockerfile:1.7

# ---------- Builder ----------
FROM python:3.11-bookworm AS builder
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
curl ca-certificates gnupg \
tshark wireshark-common \
gcc g++ make libpcap0.8 libpcap0.8-dev \
&& rm -rf /var/lib/apt/lists/*

# Add Zeek repo for Debian 12 (bookworm) and install Zeek (headers not needed here)
RUN echo "deb [signed-by=/usr/share/keyrings/zeek.gpg] https://download.opensuse.org/repositories/security:/zeek/Debian_12/ /" \
> /etc/apt/sources.list.d/zeek.list \
&& curl -fsSL https://download.opensuse.org/repositories/security:/zeek/Debian_12/Release.key \
| gpg --dearmor -o /usr/share/keyrings/zeek.gpg \
&& apt-get update && apt-get install -y --no-install-recommends zeek \
gcc g++ make libpcap0.8-dev \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /w
Expand Down Expand Up @@ -52,9 +44,10 @@ RUN echo "deb [signed-by=/usr/share/keyrings/zeek.gpg] https://download.opensuse
ENV PATH="/opt/zeek/bin:${PATH}"

WORKDIR /app
COPY --from=builder /wheels /wheels
COPY requirements.txt .
RUN pip install --no-cache-dir /wheels/*
# Mount, rather than copy, the wheelhouse so build artifacts do not remain in
# the runtime image after installation.
RUN --mount=type=bind,from=builder,source=/wheels,target=/wheels \
pip install --no-cache-dir /wheels/*

# Repo-shaped layout: the package lives at /app/app so absolute imports
# (from app.pipeline import ...) resolve identically to a local checkout.
Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# PCAP Hunter

[![CI](https://github.com/ninedter/pcap-hunter/actions/workflows/ci.yml/badge.svg)](https://github.com/ninedter/pcap-hunter/actions/workflows/ci.yml)
[![Release: v2.0.0](https://img.shields.io/badge/release-v2.0.0-7c3aed.svg)](https://github.com/ninedter/pcap-hunter/releases/tag/v2.0.0)
[![Release: v2.1.0](https://img.shields.io/badge/release-v2.1.0-7c3aed.svg)](https://github.com/ninedter/pcap-hunter/releases/tag/v2.1.0)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

Expand All @@ -15,8 +15,12 @@ By combining industry-standard network analysis tools (**Zeek**, **Tshark**, **P

---

## What's new in version 2
## What's new in version 2.1

- **Adjustable LLM context** — choose a 10K–1M-token model window; PCAP Hunter uses at most 50% for input so output and tokenizer variance do not force context compression.
- **Optional unlimited context** — disable the window cap and send every available sanitized evidence item in one request; the slider is disabled while this mode is active.
- **Richer, better-grounded reports** — foreground and background generation now share correlations, flow anomalies, JA3, final ATT&CK mapping, capture metrics, stage status, and warnings.
- **Faster large investigations** — bounded case queries, batched IOC persistence, cached geographic indexes, and capped browser chart samples reduce database and dashboard overhead.
- **Dedicated MITRE ATT&CK workspace** — evidence-backed technique hypotheses, ATT&CK v19.1 metadata, analyst dispositions, capture coverage, visibility gaps, and Navigator export.
- **Capture-quality telemetry** — packet/flow scale, parse ratio, time window, sampling limits, completed stages, and warnings now travel with UI and API results and persist with cases.
- **Durable UI analysis** — Streamlit now submits PCAP work to a process-backed queue, autosaves full evidence to SQLite, and restores recent jobs after a page stop or browser reload.
Expand All @@ -29,7 +33,7 @@ By combining industry-standard network analysis tools (**Zeek**, **Tshark**, **P

## Table of Contents

- [What's new in version 2](#whats-new-in-version-2)
- [What's new in version 2.1](#whats-new-in-version-21)
- [Visual Tour](#visual-tour)
- [Key Features](#key-features)
- [Integrations API](#integrations-api)
Expand Down Expand Up @@ -141,6 +145,7 @@ sparkline. Environment-variable keys are shown as read-only bootstrap entries.
### 10. Config — centralized settings

An **LLM Integration** section with three providers (LM Studio, OpenAI, Anthropic),
an adjustable 10K–1M-token context window, an optional unlimited-context mode,
a **YARA Rules** section with a configurable rules directory, OSINT provider keys
with a **Test Providers** live-check button, home location for the world map,
binary paths, and pipeline thresholds — all in one place with per-section clear
Expand All @@ -153,7 +158,9 @@ buttons. API keys are PBKDF2-encrypted at rest.
Pick the backend that fits your environment: **LM Studio** for local, air-gapped
analysis (chunked per-section generation), or **OpenAI** / **Anthropic** for
single-shot full-context cloud reports. Each provider keeps its own credentials
and model picker.
and model picker. The selected context window controls the evidence budget for
every provider; unlimited mode sends all sanitized evidence in a single request
and may be rejected if it exceeds the model's physical limit.

![LLM provider selection](docs/images/09-llm-providers.png)

Expand All @@ -166,6 +173,7 @@ and model picker.
- **LM Studio** (local) — privacy-first, air-gapped friendly; reports are generated section-by-section to fit small context windows.
- **OpenAI** (cloud) — single-shot report with the entire evidence corpus in one full-context call.
- **Anthropic** (cloud) — Claude via the official `anthropic` SDK (`claude-opus-4-8`, `claude-sonnet-4-6`, `claude-haiku-4-5`), single-shot with streaming.
- **Configurable Context Budget** — select a 10K–1M-token model window with a conservative 50% input ceiling, or explicitly enable unlimited mode to send all sanitized evidence at once.
- **Evidence-Grounded Reporting** — SOC-ready reports with severity-calibrated assessments, false-positive awareness, confidence qualifiers, a Risk Matrix rendered as a real Markdown table, and an IOC Summary table.
- **LLM-Optional Evidence View** — parsed packet, flow, IOC, correlation, stage, and warning evidence remains visible when generation is skipped or the provider is unavailable.
- **Multi-Language Reports** — 9 languages with region-specific terminology: English, Traditional Chinese (Taiwan), Simplified Chinese, Japanese, Korean, Italian, Spanish, French, German.
Expand Down
2 changes: 1 addition & 1 deletion app/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "2.0.0"
__version__ = "2.1.0"
18 changes: 4 additions & 14 deletions app/api/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,10 +109,8 @@ def _title_for_status(status: int) -> str:
}.get(status, "Error")


def _identify_key(request: Request, settings) -> str:
"""Derive key name for audit logging (NOT used for auth decisions)."""
import hashlib

def _identify_env_key(request: Request, settings) -> str:
"""Identify environment-backed keys without duplicating database authentication."""
auth = request.headers.get("Authorization", "")
if not auth.startswith("Bearer "):
return "-"
Expand All @@ -121,15 +119,6 @@ def _identify_key(request: Request, settings) -> str:
return "env:main"
if settings.feed_key and secrets.compare_digest(presented, settings.feed_key):
return "env:feed"
# Try DB key lookup for audit log
try:
key_repo = get_key_repo()
key_hash = hashlib.sha256(presented.encode("utf-8")).hexdigest()
api_key = key_repo.get_key_by_hash(key_hash)
if api_key:
return api_key.name
except Exception:
pass
return "-"


Expand Down Expand Up @@ -208,9 +197,10 @@ async def request_id_middleware(request: Request, call_next):
if not rid: # sanitisation stripped everything
rid = uuid.uuid4().hex
request.state.request_id = rid # stash for exception handler
key_name = _identify_key(request, settings)
fallback_key_name = _identify_env_key(request, settings)
start = time.monotonic()
response = await call_next(request)
key_name = getattr(request.state, "key_name", fallback_key_name)
response.headers["X-Request-ID"] = rid
duration_ms = int((time.monotonic() - start) * 1000)
logger.info(
Expand Down
35 changes: 0 additions & 35 deletions app/api/auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,44 +2,9 @@

from __future__ import annotations

import secrets
from enum import Enum

from app.api.settings import APISettings


class Scope(str, Enum):
FULL = "full"
FEED = "feed"


def _const_eq(a: str, b: str) -> bool:
return secrets.compare_digest(a.encode("utf-8"), b.encode("utf-8"))


def check_bearer(authorization: str | None, settings: APISettings, required: Scope) -> Scope:
"""Validate the Authorization header and return the granted scope.

Raises:
ValueError: missing/malformed header or wrong key (-> 401)
PermissionError: valid key, insufficient scope (-> 403)
"""
if not authorization or not authorization.startswith("Bearer "):
raise ValueError("missing_or_malformed_auth")

presented = authorization.removeprefix("Bearer ").strip()
if not presented:
raise ValueError("missing_or_malformed_auth")

granted: Scope | None = None
if settings.main_key and _const_eq(presented, settings.main_key):
granted = Scope.FULL
elif settings.feed_key and _const_eq(presented, settings.feed_key):
granted = Scope.FEED

if granted is None:
raise ValueError("invalid_key")

if required == Scope.FULL and granted != Scope.FULL:
raise PermissionError("insufficient_scope")
return granted
11 changes: 7 additions & 4 deletions app/api/deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import os
from functools import lru_cache

from fastapi import Header, HTTPException
from fastapi import Header, HTTPException, Request

from app.api.auth import Scope
from app.api.key_auth import RateLimitError, authenticate
Expand Down Expand Up @@ -57,7 +57,7 @@ def get_usage_tracker() -> UsageTracker:
return UsageTracker()


def _do_auth(authorization: str | None, required: Scope) -> Scope:
def _do_auth(request: Request, authorization: str | None, required: Scope) -> Scope:
"""Shared auth logic for both scope levels."""
settings = get_settings()
key_repo = get_key_repo()
Expand All @@ -72,6 +72,7 @@ def _do_auth(authorization: str | None, required: Scope) -> Scope:
usage_tracker=usage_tracker,
required=required,
)
request.state.key_name = result.key_name
return result.scope
except ValueError as exc:
# RFC 6750 §3: Bearer-auth APIs must advertise the scheme on 401.
Expand All @@ -87,12 +88,14 @@ def _do_auth(authorization: str | None, required: Scope) -> Scope:


def require_full_scope(
request: Request,
authorization: str | None = Header(default=None),
) -> Scope:
return _do_auth(authorization, Scope.FULL)
return _do_auth(request, authorization, Scope.FULL)


def require_feed_scope(
request: Request,
authorization: str | None = Header(default=None),
) -> Scope:
return _do_auth(authorization, Scope.FEED)
return _do_auth(request, authorization, Scope.FEED)
Loading