vlnr — Multi-agent exploit pipeline for the Python supply chain

vlnr discovers high-risk PyPI packages, scans them for vulnerabilities with deterministic AST taint analysis and external SAST tools, classifies findings with a rule-based noise filter, and generates proof-of-concept exploits in Docker containers via a bounded retry loop. The multi-agent architecture adds optional LLM-based strategic planning, maker-checker verification, parallel package scanning, memory management, structured observability logging, and human-in-the-loop exit points — all gated behind CLI flags.

For authorized security research only. PoC execution runs in transient Docker containers. Never run against systems you do not own.

---

Quick Start

Requirements

• Python 3.14+
• uv package manager
• Docker (for sandbox PoC validation)
• An OpenAI-compatible LLM API (only needed for LLM-assisted triage, discovery scoring, and PoC generation/refinement)

Install

git clone https://github.com/nandrzej/vlnr
cd vlnr
uv sync

Configure

# .env
CUSTOM_OPENAI_API_KEY=<your-key>
GITHUB_TOKEN=<optional — higher-rate GitHub API access>

Model routing is configured in llm_config.yaml:

default:
  base_url: "https://api.vlnr.ai/v1"
  model: "qwen-3.5-4b"
  temperature: 0.0
  reasoning_effort: "low"

tier_1:   # PoC generation, deep reasoning
  model: "openai/qwen-3.5-397b-instruct"
  temperature: 0.1
  reasoning_effort: "high"

tier_2:   # Triage, refinement
  model: "openai/gemma-4-31b-it"
  temperature: 0.0

tier_3:   # Metadata classification, intent scoring
  model: "openai/qwen-3.5-4b-instruct"
  temperature: 0.0

Run

# Full pipeline: discover → scan → agent
vlnr run --out-dir results/ --osv-dump osv.zip --packages requests

# Scan only (static analysis, no agent)
vlnr run --out-dir results/ --osv-dump osv.zip --packages requests --skip-agent

# With LLM triage
vlnr run --out-dir results/ --osv-dump osv.zip --packages requests --llm-triage

# Parallel scanning (4 concurrent packages)
vlnr run --out-dir results/ --osv-dump osv.zip --packages requests --parallel 4

# Full multi-agent mode: planner + verifier + parallel + observability
vlnr run --out-dir results/ --osv-dump osv.zip --packages requests \
    --llm-triage --planner --verify --parallel 4 --obs-log trace.jsonl

# Apply operator review decisions
vlnr run --out-dir results/ --osv-dump osv.zip --packages requests \
    --resolve-reviews my_decisions.json

# Run stages individually
vlnr discover --packages requests,flask --out-dir results/ --osv-dump osv.zip
vlnr scan results/candidates.json --out-dir findings/ --llm-triage
vlnr agent --package requests --state-path results/agent_session.json

Migrating from legacy entry points

Old	New
poc-find-candidates main	vlnr discover
poc-find-candidates agent	vlnr agent
poc-scan-vulnerabilities	vlnr scan
(none)	vlnr run (full pipeline)

The legacy scripts still work but emit a DeprecationWarning.

---

How It Works

Discover (vlnr discover)

Loads OSV/EPSS data, fetches PyPI package lists, and scores candidates by centrality (reverse-dependency count), downloads, GitHub stars, and recency. An optional LLM intent score (Tier 3) can be blended 50/50 with the static score — gated behind VLNR_LLM_DISCOVERY=1. Output: candidates.json.

Scan (vlnr scan)

Per candidate package:

1. Clone — shallow git clone of the source repository
2. Entry points — discover from pyproject.toml, setup.cfg, setup.py
3. Metadata scan — inspect .dist-info/METADATA for suspicious patterns
4. External SAST — bandit, ruff, semgrep (best-effort; skipped if tools are missing)
5. AST taint analysis — intra-procedural dataflow from sources (sys.argv, os.environ, input) to sinks (subprocess.run, eval, pickle.loads, open, etc.)
6. External hit fallback — unmatched tool hits get their own slices
7. Conjunctive escalation — co-occurring signals (base64+exec, network+subprocess, metadata+exec) escalate severity
8. Slice construction — per-sink code snippets with surrounding context
9. Static scoring — per-slice risk score (base 0.5, Command Injection 0.8, Deserialization 0.6, with adjustments for static class and sources)
10. LLM triage (optional, Tier 2) — batch evaluation of slices with risk_score_static > 0.3 or tool hits (≤10 per call)
11. NoiseFilterAgent — deterministic 6-rule classifier assigns every slice a SliceDecision: POC_CANDIDATE, STATIC_ONLY, IGNORE, or HUMAN_REVIEW
12. Output — findings JSON, slices JSONL, VEX for false positives

Agent (vlnr agent)

A deterministic scheduler drives a while iterations < max_iterations and budget_remaining > 0 loop with optional multi-agent features:

1. Memory management — hard cap on history entries (default 100), summarize older entries beyond the recent threshold (50), prune completed slices.
2. Planner (optional, --planner) — LLM-based strategic planning (Tier 3). Generates ordered PlanItems. Falls back to deterministic if the LLM fails. Replans every 5 iterations (Tier 2). Additive only — never replaces the deterministic scheduler.
3. Verifier (optional, --verify) — Maker-checker for POC_CANDIDATE decisions using a second LLM (Tier 3). Disagreement escalates to HUMAN_REVIEW. LLM errors pass through.
4. HITL surfacing — HUMAN_REVIEW slices are surfaced as structured HumanReviewPrompt objects in human_review_prompts.json. Operator decisions can be applied via --resolve-reviews.
5. deterministic_next_action(state) — pure function, no LLM (fallback):
   • Priority 1: slices with decision == POC_CANDIDATE and no PoC data → run_poc_loop
   • Priority 2: unscanned packages in candidate_pool → scan_package
   • Priority 3: nothing left → stop
6. _do_dispatch — routes to process_package() (parallel via ThreadPoolExecutor when --parallel N>1) or PocAgent.run_loop(), with structured JSONL observability logging.
7. Update state and persist to agent_session.json (resumable)

Budget deduction: $0.06 before each run_poc_loop. No per-LLM-call deduction in the scheduler.

PoC Loop (PocAgent)

Bounded retry loop (default 2 attempts):

• Attempt 0: generate_poc (Tier 1 LLM) → validate in Docker
• Attempt 1+: refine_poc (Tier 2 LLM) with prior exploit code and failure logs → re-validate
• Runtime_Reachable → success, break
• ContainerIsolationError → stop immediately (Docker unavailable)
• Exhaustion → verification_steps = "Failed_after_N_attempts"

---

LLM Tiers

All model routing in llm_config.yaml. One model per tier. API key from CUSTOM_OPENAI_API_KEY env var or api_key in config. Fallback when config is missing: http://127.0.0.1:1234/v1 with qwen3.5-2b-mlx.

Tier	Enum	Role	Example Model
tier_1	LLMTier.TIER_1	PoC generation, whole-repo reasoning	openai/qwen-3.5-397b-instruct
tier_2	LLMTier.TIER_2	Contextual triage, PoC refinement, replanning	openai/gemma-4-31b-it
tier_3	LLMTier.TIER_3	Metadata classification, intent scoring, strategic planning, maker-checker verification	openai/qwen-3.5-4b-instruct

---

Agents

vlnr uses five specialized agents, each with a formal identity (AgentId, AgentIdentity, LifecycleState). All LLM-based agents are gated behind CLI flags and fall back safely on failure.

AgentLoop — Orchestrator (AgentId.ORCHESTRATOR)

The central coordinator. Runs a while loop of determine → dispatch → update → manage memory → surface reviews. Scheduling is deterministic by default — a pure function (deterministic_next_action) picks the next action without any LLM call. When --planner is enabled, the loop consults the PlannerAgent before falling back to deterministic scheduling.

Key responsibilities: action dispatch, state merge, memory management, human-review surfacing, observability event timing, budget tracking, AgentBus wiring.

NoiseFilterAgent — Classifier (AgentId.CLASSIFIER)

Deterministic 6-rule post-triage classifier. Takes a Slice and returns a SliceDecision. Never calls an LLM on its own — the only LLM path is the mid-range refinement which is invoked by the existing triage pipeline. Pure function: no side effects, no state access.

Decisions: IGNORE (false positive), STATIC_ONLY (low risk), POC_CANDIDATE (worth exploiting), HUMAN_REVIEW (operator needed).

PocAgent — PoC Executor (AgentId.POC_EXECUTOR)

Bounded retry loop for exploit generation and validation. Generates PoC code via Tier 1 LLM, validates in a Docker container, and refines via Tier 2 LLM on failure. Default 2 attempts. Stops immediately on container errors.

Key behaviors: Tier 1 → validate → Tier 2 refine → re-validate. Container unavailable → stop. Exhaustion → last attempt recorded.

VerifierAgent — Maker-Checker (AgentId.VERIFIER) — --verify

Cross-checks NoiseFilterAgent decisions before expensive PoC execution. Only inspects POC_CANDIDATE slices — all other decisions pass through unchecked. Uses Tier 3 (cheapest model). Disagreement escalates the slice to HUMAN_REVIEW. LLM failures result in pass-through (never blocks the pipeline). Off by default.

PlannerAgent — Strategic Planner (AgentId.PLANNER) — --planner

LLM-based strategic planning for the agent loop. Generates an ordered list of PlanItems (which packages to scan, which slices to run PoC on) using Tier 3. Replans every 5 iterations using Tier 2 when the plan is stale or exhausted. Additive only — never replaces the deterministic scheduler. If the LLM fails or returns an empty plan, the loop falls back to deterministic_next_action. Off by default.

Inter-Agent Communication

Agents communicate over an in-process AgentBus with typed Pydantic messages (TASK, RESULT, ERROR). The bus is wired for future async operation but currently wraps the existing direct dispatch path. Excluded from state serialization.

CLI Flags

Agent / Multi-Agent Flags

Flag	Default	Description
--planner	off	Enable LLM-based strategic planner (Tier 3 plan, Tier 2 replan)
--verify	off	Enable maker-checker verification of POC_CANDIDATE decisions
--parallel N	1	Max parallel package scans (1 = sequential)
--max-history N	100	Cap on agent history entries
--summarize-threshold N	50	Recent entries to retain before summarization
--[no-]prune-completed	prune	Enable/disable pruning of completed slices from active state
--obs-log PATH	agent_trace.jsonl	Path for structured observability JSONL log
--resolve-reviews PATH	none	Apply operator review decisions from a JSON file
--max-workers N	4	Max concurrent package scans for vlnr scan
--max-fetches N	2	Max concurrent source fetches

Examples

# Quiet run with just the deterministic scheduler
vlnr run --out-dir out/ --osv-dump osv.zip --packages requests

# Multi-agent: parallel scanning + planner + verifier + observability
vlnr run --out-dir out/ --osv-dump osv.zip --packages requests \
    --llm-triage --parallel 4 --planner --verify --obs-log trace.jsonl

# With memory tuning for long runs
vlnr run --out-dir out/ --osv-dump osv.zip --packages requests \
    --llm-triage --max-history 200 --summarize-threshold 75 --parallel 8

# Resume with operator review decisions
vlnr run --out-dir out/ --osv-dump osv.zip --packages requests \
    --llm-triage --resolve-reviews my_decisions.json

Output Formats

File	Description
candidates.json	Ranked candidates with scores, OSV IDs, category
all-findings-index.json	Index of all scanned packages with stats
<pkg>-findings.json	Per-package structured findings
<pkg>-slices.jsonl	Per-package taint dataflow slices
<pkg>-<slice_id>-vex.json	OpenVEX v0.2.0 records (false positives only)
agent_session.json	Resumable agent state
agent_trace.jsonl	Structured observability log (configurable via --obs-log)
human_review_prompts.json	Structured prompts for HUMAN_REVIEW slices

---

CWE Coverage

CWE	Sink	Detection
CWE-78	subprocess.run(shell=True)	AST taint + bypass scan
CWE-22	open(file_path)	Path traversal via os.path.join
CWE-502	pickle.loads()	__reduce__ payload construction
CWE-94	eval(), exec()	Builtins reflection via getattr
CWE-918	urllib, httpx, aiohttp	Semgrep SSRF rules (vlnr/rules/ssrf.yaml)

---

Limitations

• Intra-procedural AST — taint analysis does not track dataflow across function boundaries. Each FunctionDef is analyzed independently.
• Hardcoded budget estimates — $0.06 deducted before each PoC loop. LLM calls within the loop are not tracked. No reconciliation against actual API billing.
• Docker required — PoC validation runs in Docker containers. ContainerIsolationError stops the PoC loop if Docker is unavailable.
• VEX only for false positives — OpenVEX documents are generated exclusively for slices marked as false positive.
• AST parse timeout — a 5-second threading.Timer guards source parsing. Thread-safe and works on all platforms (replaced the old Unix-only SIGALRM).
• External tools best-effort — bandit, ruff, and semgrep are attempted via subprocess.run. Missing tools are skipped with a warning.
• Planner and verifier are LLM-dependent — both require an LLM client. Planner falls back to deterministic scheduling on failure. Verifier passes through POC_CANDIDATE decisions on failure.

---

License

MIT — see LICENSE.