Skip to content

DESIGN.md

Latest commit

 

History

History
232 lines (167 loc) · 8.81 KB

DESIGN.md

File metadata and controls

232 lines (167 loc) · 8.81 KB
Raw

PCAPGraph — Design Spec

A BloodHound-style network visualizer for packet captures. Drop in a .pcap / .pcapng file and see the conversations between hosts as an interactive node-link graph — built to make incident response fast, and general network visibility easy.

Sibling project to RDPGraph (which does the same for Windows RDP event logs). Same stack, same drag-drop-and-read-the-graph philosophy.

Status: Local-only for now. Designed so it can be published to GitHub later (same path RDPGraph took), but nothing is pushed yet.

1. Purpose

Two audiences, one tool:

  • Incident response / threat hunting (signals ON by default) — quickly spot beaconing, C2, exfil, scanning, and lateral movement when packet captures are what you have to work with.
  • General network visibility — "what is actually talking to what on my network," who the top talkers are, what's going out to the internet.

IR-oriented highlighting is on by default but dismissible, so the same graph serves both uses.

2. Stack

Mirrors RDPGraph for consistency:

  • Backend: Flask (Python 3.8+), drag-drop upload → parse → JSON graph.
  • Frontend: vis-network (vis.js) node-link graph + side panel + filters.
  • Parsing: streaming, never load the whole file into RAM (see §6).
  • Run: Python venv → pip install -r requirements.txt → python app.py → http://127.0.0.1:5000. README will document venv setup for Linux/macOS and Windows, same as RDPGraph.

3. Core model — don't draw packets, draw relationships

We never render individual packets. Packets are aggregated into flows / conversations as we stream-parse:

  • Node = a host (IP). Internal vs. external classified by RFC1918 / loopback / link-local / multicast vs. routable public.
  • Edge = a conversation between two hosts (aggregated over the 5-tuple, or looser src↔dst depending on view).
  • Edge weight = bytes and/or packet count.
  • Node size = total traffic / number of conversations touching it.

This alone collapses millions of packets into a readable graph.

4. The "hairball" problem — aggregation rules

The whole game is deciding what to collapse. One metric drives everything:

For each external IP, count the distinct internal hosts that contacted it.

4a. Fan-out (1 internal host → many external IPs)

Classic clutter. Collapse external destinations into super-nodes so one host's 300 outbound destinations don't fill the screen. Grouping options:

  • ASN / organization (best for IR — "200 IPs all on one cheap VPS provider")
  • Subnet (/24)
  • Service / port
  • Geo / country (available, noisier)

Super-nodes are collapsible: show 🌐 Internet (312 hosts) by default, click to expand into ASN sub-groups or individual members.

4b. Fan-in / convergence (many internal hosts → 1 external IP) ← key IR view

The opposite treatment. A shared external destination is the signal, so we promote it instead of burying it:

  • External IP touched by 1 internal host → bury in the cloud (4a).
  • External IP touched by many internal hosts → break out into its own prominent hub node.

Render as a hub-and-spoke / star: shared IP in the center, contacting internal hosts as spokes around it.

   [Host .12]──┐
   [Host .15]──┤
   [Host .23]──┼──▶ ( 203.0.113.7 )   ⚠ shared by 5 hosts
   [Host .31]──┤        C2?  AS-cheapVPS
   [Host .42]──┘

Treatment: size by # distinct internal hosts, alert color (red/orange), label ⚠ shared by N hosts, force layout pulls spokes inward into a visible star.

4c. Subnet angle

  • Subnet tinting — color/badge internal nodes by subnet, so "the whole 10.1.5.0/24 VLAN is beaconing out" is obvious at a glance.
  • Subnet-level aggregated edge — optional toggle to collapse the internal side too: one thick edge [10.1.5.0/24 — 5 hosts] ══▶ (C2), weight = host count. Two altitudes: per-subnet = "where's the outbreak", per-host = "who do I quarantine" (same dual-altitude idea as RDPGraph).

4d. Top-N with honest labeling

Default to top-N conversations by volume, but say so — "showing top 50 of 412 flows" — never silently truncate. Slider to reveal more.

5. Detection signals (IR mode, on by default)

Surface what matters; dim the rest:

  • Shared external destinations (§4b) — the headline C2 indicator.
  • Beaconing — regular inter-packet timing to a destination (low jitter, periodic). Turns a "shared destination" into a near-certain C2 call.
  • High fan-out — a host touching > X destinations (scanning / spray).
  • Exfil shape — large outbound:inbound byte ratio.
  • Scanning — many connection attempts with RST / no completed handshake.
  • Rare ports / unusual protocols.

Also surfaced as a ranked "Shared external destinations" table outside the graph for fast triage:

External IP # internal hosts Subnets involved ASN Bytes Beaconing?
203.0.113.7 5 10.1.5.0/24 AS-cheapVPS … ✅

Sort by "# internal hosts" → real C2 floats to the top. Click a row → zoom/ highlight that hub in the graph.

6. Performance & upload limits

Streaming parse, incremental aggregation — we hold a flow table (keyed by 5-tuple) and increment counters, never a list of packets. Memory scales with unique conversations, not file size. Use Scapy PcapReader or tshark; never rdpcap() the whole file.

Limit Default Env var Why
Max upload size 500 MB MAX_UPLOAD_MB Flask MAX_CONTENT_LENGTH rejects oversized uploads cleanly
Accepted types .pcap, .pcapng — Reject everything else up front
Max packets processed 5,000,000 MAX_PACKETS Safety valve; report "processed X of Y", never silent truncation
Parse time budget 120 s PARSE_TIMEOUT Don't let one upload hang the worker

For bigger captures the answer is pre-slicing, documented in the README:

editcap -c 1000000 big.pcapng chunk.pcapng                        # split by packet count
tshark  -r big.pcapng -Y "ip.addr==10.1.5.0/24" -w focused.pcap   # filter by subnet

7. IP enrichment — click a public IP → see its origin

Click a public IP node → side panel/modal with country flag, geo, owner (ASN/org), abuse contact, and reputation.

⚠️ OPSEC-first design

Querying a third-party about a suspicious IP leaks which IPs you're investigating, and many IR environments are air-gapped. So: offline-first, online-optional.

Tier 1 — offline (default, always on):

  • MaxMind GeoLite2 local .mmdb (Country, City, ASN) — fully offline lookups, no rate limits, no data leakage, no keys at query time. (DB-IP lite is an alternative if avoiding the MaxMind account.)

Tier 2 — online (opt-in toggle, OFF by default):

  • ip-api.com — free, no key (~45 req/min): geo + ASN + ISP.
  • RDAP — free, no key, authoritative ownership + abuse contact.
  • AbuseIPDB — free tier (~1k/day, one key): reputation score. High value for IR.
  • (Later: VirusTotal / OTX / GreyNoise — free tiers, need keys.)

Rules:

  • Enrich public IPs only — skip RFC1918 / loopback / multicast.
  • Cache every result (in-memory or small SQLite/JSON) per session.
  • Pre-color from offline data — tint by country, flag known bulletproof/VPS ASNs — so signal appears before you even click.

8. UI summary

  • Drag-drop .pcap / .pcapng onto the page.
  • Force-directed graph: internal hosts detailed & central, external grouped to the rim; C2 hubs highlighted red.
  • Filters: by host, subnet, port/protocol, direction, volume threshold; toggles for IR-highlighting, per-host vs. per-subnet, online enrichment.
  • Click node → side-panel details (conversations, ports, bytes; enrichment for public IPs).
  • "Shared external destinations" triage table.

9. Proposed file layout

pcap_visualiser/
  app.py              # Flask server + upload handling + limits
  pcap_parser.py      # streaming parse → flows (Scapy PcapReader / tshark)
  graph_builder.py    # flows → nodes/edges, aggregation rules (§4)
  enrichment.py       # GeoLite2 offline + opt-in online lookups, cache (§7)
  detect.py           # IR signals: shared-dest, beaconing, fan-out, exfil (§5)
  templates/index.html
  static/             # graph.js, css
  requirements.txt    # Flask, scapy (or pyshark), geoip2, requests, Werkzeug
  README.md
  .gitignore          # venv/, uploads/, *.pcap, *.pcapng, *.mmdb, __pycache__

10. Open / later

  • Name confirmed: PCAPGraph.
  • Local-only now; GitHub publish later (mirror RDPGraph's release process).
  • Decide Scapy vs. pyshark/tshark as the default parser during build (tshark is faster on big files but adds a Wireshark dependency).