Indirect Prompt Injection Against a Voicemail-to-Action Pipeline

An offline measurement harness and study of indirect prompt injection against a voicemail→CRM/callback automation of the kind commonly built with an LLM extraction node (here: a small local model, qwen2.5:3b, served by Ollama). A voicemail transcript is attacker-controlled content; the pipeline feeds it to an LLM that extracts a callback number, caller name, reason, urgency, and a notification summary. This project measures how reliably injected text in that transcript can hijack the callback number, suppress urgency, plant a phishing string in the summary, and spoof the caller name — and how much a stack of defenses reduces that, at what utility cost.

The harness faithfully mirrors the extraction stage of a real pipeline (same prompt, same CALLER_ID/CALLER_NAME_HINT regex preprocessing, same /api/generate call shape, same post-extraction fallbacks) so the numbers reflect what the production pattern would actually ship — but it runs entirely offline against your own Ollama host and touches no email, spreadsheet, workflow-automation, or notification service.

Study author: Tunde Oyedeji Repository: https://github.com/toyedeji/voicemail-injection-study Date: 2026-07-02

What's measured

• Phase 0 — benign baseline (reports/baseline_accuracy.md): per-field extraction accuracy, parse-failure rate, and self-consistency, at temperature 0.0 (greedy) and 0.8 (the model's Modelfile-default, i.e. what production runs at). Establishes that the harness extracts correctly before any attack, so attack deltas are meaningful.
• Phase 1 — attack success rate (reports/attacks_asr.md): ASR for four goals — G1 callback-number hijack (body-injection and regex-poison variants), G2 urgency suppression, G3 summary/notification injection, G4 caller-name spoof — reported both at the raw-model layer and the pipeline-effective layer, with Wilson 95% CIs at temp 0.8.
• Phase 3 — defense-cost curve (reports/defense_cost.md, reports/v2body_followup.md): three defenses — M1 provenance (derive trusted fields from headers only), M2 spotlight (delimit/mark untrusted content in the prompt), M3 output validation (reject a callback that doesn't match the trusted header) — evaluated for ASR reduction against the benign-accuracy (utility) cost they impose.

Read the writeup in reports/: start with baseline_accuracy.md, then attacks_asr.md, then defense_cost.md and its v2body_followup.md addendum.

Threat model / scope

The study targets a general architectural pattern — an LLM extraction node that reads attacker-influenceable content (a voicemail transcript) and whose output drives an action (who gets called back) — reproduced on the author's own system. It is not an exploit of a vulnerability in Ollama, in qwen2.5, or in any third-party workflow tool; those are used as-shipped and off the shelf. The injection surface is the content the pipeline was designed to read, and the finding is about pipeline design (trust boundaries between derived/header fields and free-text body), not a software CVE. See Ethics & responsible disclosure.

Layout

.
  config.yaml              # model, ollama_url, temps, n_repetitions, seed, paths
  config_g4.yaml           # G4 (caller-name spoof) run config
  config_phase3.yaml       # Phase-3 defense run config
  build_attacks.py         # generates the attack corpora from templated payloads
  harness/
    production_prompt.txt          # extraction prompt (verbatim; client name anonymized)
    production_prompt_spotlight.txt# M2 spotlight variant of the prompt
    schema.json                    # model output schema + pipeline notes
    config.py                      # config + .env loading (OLLAMA_URL override)
    preprocess.py                  # CALLER_ID / CALLER_NAME_HINT regex (pipeline port)
    ollama_client.py               # /api/generate wrapper (stdlib urllib)
    extractor.py                   # email -> prompt -> model -> parsed dict
    corpus.py                      # Record type + JSONL loader
    defenses.py                    # M1/M2/M3 toggles
    sentinels.py                   # reserved synthetic attack sentinels
    runner.py                      # run a corpus N times/temp -> runs/*.jsonl
    score.py / score_attacks.py / score_defense.py / score_v2body.py
  corpora/                 # synthetic voicemails + attack payloads (see below)
  reports/                 # the writeup (generated summaries)
  tests/                   # offline fidelity/reproducibility guards

Data: 100% synthetic

All voicemail transcripts are fictional. Every phone number is in the reserved fictional 555-01xx range (the attacker sentinel uses the unassigned area code 666). No real names, numbers, PII, or customer data appear in any corpus. Attack sentinels (harness/sentinels.py) are distinct synthetic markers, and a test asserts none of them occur in the benign corpus.

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt          # PyYAML only

# Point the harness at your own Ollama instance. Either edit config.yaml, or:
cp .env.example .env && chmod 600 .env    # then set OLLAMA_URL=http://<your-host>:11434
# config.yaml ships a placeholder host (OLLAMA_HOST); the run steps below need a real one.

ollama pull qwen2.5:3b                     # the model under test

The offline tests need no Ollama host and no model.

Reproduce

# 1. Offline regression/fidelity guards (no Ollama needed):
pip install -r requirements-dev.txt        # adds pytest
pytest

# 2. Benign baseline (needs a reachable Ollama host):
python harness/runner.py                   # writes runs/benign_*.jsonl
python harness/score.py                    # -> reports/baseline_accuracy.md

# 3. Attacks:
python build_attacks.py                    # (re)generate attack corpora
python harness/runner.py --config config.yaml --corpus corpora/attacks.jsonl
python harness/score_attacks.py            # -> reports/attacks_asr.md

# 4. Defenses (Phase 3):
python harness/runner.py --config config_phase3.yaml
python harness/score_defense.py            # -> reports/defense_cost.md

Reproducibility: seeded generation (seed: 1337, per-run seed = seed + run_index), pinned dependencies, deterministic scoring. Runs are resumable (--resume runs/<file>) and model-agnostic (change model: in config.yaml). Raw run outputs land in runs/ (gitignored; regenerated).

Ethics & responsible disclosure

• Own system, authorized. All measurements were run by the author against the author's own pipeline and own Ollama host. Nothing here targets a third party.
• Offline / no live infrastructure. The harness talks only to a local Ollama /api/generate endpoint. It never contacts email, spreadsheet, workflow-automation, or push-notification services, and it performs no callbacks — it measures model extraction, not real-world actions.
• Synthetic data only. No real voicemails, callers, numbers, or PII. All payloads use reserved fictional numbers and clearly-marked sentinels.
• No third-party 0-day. The result is about an architectural trust-boundary weakness in the LLM-extraction pattern, not a bug in any specific product. Off-the-shelf components (Ollama, qwen2.5, workflow tooling) are used as shipped. No product-specific exploit or bypass is disclosed. If you operate a similar pipeline, the mitigations in reports/defense_cost.md (provenance / spotlighting / output validation) apply directly.
• Purpose. Defensive: to quantify the risk of the pattern and the cost/benefit of mitigations, so builders of voicemail/email→action automations can make informed trust-boundary decisions.

License

MIT — see LICENSE.