Add trace-based behavioral tests with Monocle Test Tools

[imohammedansari]

Summary

Adds a behavioral test suite that asserts against DeerFlow's Monocle execution traces: which agent ran, which tools it called, what it was asked and what it returned, and its token and duration cost. Additive only, under backend/tests/monocle/, with no app-code changes.

Why

DeerFlow already has 350+ backend tests, but they exercise the agent with the LLM and tools controlled, not by checking what a real run actually did. This adds that missing layer. It asserts on the emitted trace, so if a later change alters how the agent routes, which tools it calls, or how many tokens it burns, a test catches it.

How it works

It uses Monocle Test Tools in two layers. One offline test loads a recorded trace from file with with_trace_source("file", trace_path=...) and shows the full fluent vocabulary (called_agent, called_tool, contains_input / contains_any_output, does_not_call_tool, under_token_limit, under_duration). It needs no keys and is deterministic, so it is the worked example for writing assertions. Two live tests drive the agent end to end through DeerFlowClient and assert on the trace the real run emits; these are the behavioral guards, and they skip without OPENAI_API_KEY or the app. The README documents the intended workflow: capture a run you are happy with as a golden trace, turn it into assertions, then enforce those same assertions on the live agent.

Changes (all under backend/tests/monocle/)

• test_deerflow.py: 1 offline example (full assertion vocabulary) plus 2 live tests (web-research and sandbox paths).
• _helpers.py: paths and run_deerflow() (model resolved from config.yaml).
• conftest.py: fixtures only (the run_agent fixture, live path only).
• traces/: 1 recorded 0.8.8 trace (web_research_ev_battery.json).
• requirements.txt: standalone install pinning monocle_test_tools==0.8.8, kept out of the backend lock because it hard-depends on the ML eval stack (torch, transformers); importorskip skips the suite when it is absent.
• README.md.

---

PS: if Monocle looks useful, a ⭐ helps the project (https://github.com/monocle2ai/monocle). The companion PR that adds the instrumentation these tests run against is #4024.

[CLAassistant]

All committers have signed the CLA.

[willem-bd]

Reviewing the trace-based behavioral test suite (companion to #4024). Additive only, under tests/monocle/. I verified the load-bearing pieces: monocle_test_tools auto-registers its pytest plugin (pytest11 entry point) so the monocle_trace_asserter fixture is available; DeerFlowClient.chat is sync with a matching signature; and test_workflow unpacks *test_case.test_input -> run_deerflow(message), so the live-test call chain is correctly wired. The README's claim that declarative eval is a silent no-op is accurate (_evaluate_span has no call sites). The committed traces contain no secrets (scanned). See inline comments for the issues.

Headline: as shipped, 4 of 5 tests assert against frozen JSON fixtures and so give no regression protection for DeerFlow itself; the only real-behavior test is skipped by default and not wired into CI.

What's good:

• Additive, scoped to tests/monocle/, no app-code changes.
• Adds .monocle/ to .gitignore (fixes a finding from #4024).
• Offline tests need no keys, deterministic, fast.
• Honest README (accurately documents the eval no-op and the q1 caveat).
• Correctly wired machinery (verified).

Recommendation: request changes. Priority: (1) justify/reframe the offline tests' value; (2) resolve the monocle_apptrace version skew with #4024; (3) move helpers out of conftest.py and scope Monocle setup + .env load to the live test; (4) decide and document whether this runs in CI.

Also (not code-anchored): the PR is labeled area:docs ("Documentation and Markdown only"), but it adds Python test code, a requirements.txt, and trace data -- the label is inaccurate and may misroute review.

[willem-bd]

This suite isn't wired into any test target: the root Makefile has no test goal, and backend tests run from backend/ (uv run pytest tests/), which collects backend/tests/ -- not repo-root tests/monocle/. So these tests only run via the manual pip install -r tests/monocle/requirements.txt && pytest path, and the one test with real behavior value (test_web_research_live) is gated on OPENAI_API_KEY + app importability and won't run in CI unless explicitly wired. If CI coverage is intended, add a target; if not, state that it's a developer-only manual tool.

Also: the docstring path monocle-test/ doesn't match the actual directory tests/monocle/ (same mismatch in the README and the inline comment on line 21).

[willem-bd]

The four offline tests load static JSON committed under traces/ and assert called_tool / under_token_limit / under_duration against data that never changes. They pass forever regardless of how DeerFlow's routing, tool selection, or token usage evolves -- they only verify the recorded trace files haven't changed and that the asserter API works. So they give no regression protection for DeerFlow itself (the PR summary's "if a later change alters how the agent routes ... a test catches it" holds only for the live test, which is skipped by default).

Consider either reframing these as trace-schema/asserter smoke tests, or regenerating traces in-CI against current code so they reflect actual behavior.

[willem-bd]

Importing from conftest.py as a regular module is discouraged and breaks under pytest's importlib import mode (which newer pytest steers toward). It works today only because prepend mode puts the test dir on sys.path. Move TRACES and run_deerflow into a normal helper module (e.g. _helpers.py) and keep conftest.py for fixtures only.

[willem-bd]

Trace filenames embed a hash + timestamp and are hardcoded here, so re-recording a trace (per the README's "add your own test" flow) changes the filename and breaks these imports. The four committed blobs are also ~2,940 lines -- 93% of this PR's additions -- which makes the diff noisy. If committing traces is the intended replay mechanism, consider stable names and/or a slimmed representation.

[willem-bd]

The README labels this "vector_db_comparison (subagent path)," but the docstring below admits "there is no distinct task/subagent span to assert on ... We assert only what this trace contains." The test asserts web_search + vector-DB keywords -- it doesn't exercise the subagent path at all. Rename it or drop the "subagent path" framing so the coverage isn't overstated.

[willem-bd]

under_duration(180) on a live run that does LLM calls + DuckDuckGo web_search/web_fetch is inherently flaky on slow CI/network. Budget/duration assertions on network-dependent runs should be lenient or omitted, otherwise this test will intermittently fail when it does run.

[willem-bd]

setup_monocle_telemetry(...) and load_dotenv(...) run at conftest import (collection time) for every test in the suite. The four offline tests need neither -- they parse JSON from disk and don't run the agent or touch keys -- yet collecting the suite installs the global OTel TracerProvider, monkey-patches the OpenAI client, and loads every secret from .env into the process env. Scope both to the live path (e.g. inside run_deerflow or a live-only session fixture) so offline tests stay side-effect-free.

[willem-bd]

Two issues with run_deerflow:

• model_name="gpt-4o" is hardcoded, bypassing DeerFlow's model-resolution logic (the repo has test_lead_agent_model_resolution.py for a reason). Prefer reading the configured model.
• config_path=REPO_ROOT / "config.yaml" -- config.yaml is gitignored. If it's missing but deerflow is importable and OPENAI_API_KEY is set, the live test errors instead of skipping. Add a skip guard for config absence (the skipif/importorskip gates don't cover it).

[willem-bd]

monocle_test_tools==0.8.8 transitively pins monocle_apptrace to 0.8.8, but PR #4024 adds monocle_apptrace to backend/pyproject.toml unpinned. If both merge, the app resolves the latest monocle_apptrace while the test asserter expects 0.8.8 -- a version skew between the runtime tracer and the asserter's trace format. The two PRs need to be version-aligned (both pin 0.8.8). Also, the repo manages deps via pyproject.toml/uv workspaces, not requirements.txt; consider declaring test deps in the backend dev group instead of a standalone file.

[imohammedansari]

@willem-bd Thanks for the thorough review 🙏 I pushed a follow-up that restructures the suite around your headline point.

Behavioural coverage is now the live tests. They drive the agent end to end through DeerFlowClient and assert on the trace the real run emits, so a change to routing, tool selection, or token cost fails a test. I kept one offline test as a worked example of the assertion API and dropped the other three frozen-fixture tests, which also removed about 2,200 lines of trace blobs, and the single remaining trace has a stable name. The README now documents the intended workflow: capture a run you are happy with as a golden trace, turn it into assertions, then point those same assertions at the live agent so every later run has to reproduce that behaviour.

The rest is in too. The suite moved under backend/tests/monocle/ so backend pytest collects it, helpers moved into _helpers.py with conftest holding only the run_agent fixture, Monocle setup is owned by the validator and the .env load is scoped to the live path, the model is resolved from config.yaml instead of being hardcoded, the live tests skip when the key, app, or config.yaml is missing, and I dropped the wall-clock duration bound on live runs.

On the dependency, I kept it in a standalone requirements.txt rather than the backend dev group because monocle_test_tools hard-depends on the ML eval stack (torch, transformers, sentence-transformers), which is roughly 48 packages and about 950 lines in uv.lock. Isolating it keeps the app's locked deps clean, and importorskip skips the suite when it is not installed so a plain backend venv still collects without error. Happy to move it into the workspace if you would rather carry that weight.

Ready for another look 🚀

[willem-bd]

Reviewed against the PR head (baf98de4) and the current codebase. Verdict: approve with minor fixes - test-only, additive, skips cleanly in CI, and the recorded trace satisfies every offline assertion (web_search=1, web_fetch=5 >= min_count=2, image_search=0, 43,194 tokens < 100k, ~16s < 60s). Companion instrumentation PR #4024 is merged; all referenced symbols exist (DeerFlowClient(config_path=...), chat(message, thread_id=...), and the web_search / web_fetch / write_file / image_search tools).

Two actionable inline comments below; the rest are non-blocking meta:

• area:docs label looks wrong - this adds 4 Python files + a JSON fixture + a README (test code), not docs-only. Likely a bot mislabel; should be the test area label.
• size/XL is inflated by the 918-line auto-generated trace JSON; hand-authored code is ~174 lines. Noting for review calibration only.

[willem-bd]

These pytest paths are missing the backend/ prefix. The actual location is backend/tests/monocle/ (which is what the README uses), so pytest tests/monocle/ run from the repo root fails - no such directory. Copy-paste trap.

Suggested:

pytest backend/tests/monocle/            # offline example (no keys)
pytest backend/tests/monocle/ -k live    # add the live behavioural tests

[willem-bd]

This says the suite "skips cleanly" when the dep is absent, but it's worth stating explicitly that, because monocle_test_tools is deliberately kept out of the backend deps, none of these tests run in CI - make test collects-and-skips the whole module (including the deterministic offline example). The PR summary frames them as guards that "catch" routing/tool/token regressions, which implies CI enforcement that doesn't currently exist; the assertions only run on a manual pip install -r (plus OPENAI_API_KEY + the app for the live ones).

A one-liner here, e.g. "This suite is opt-in and not run by CI; run it manually after install," would make the coverage gap visible to maintainers.

[willem-bd]

Heads-up: this committed trace embeds the full data.input of the first LLM span - the entire DeerFlow system prompt (<skill_system> with all skill descriptions, <subagent_system>, <clarification_system>, <critical_reminders>, plus sandbox paths). DeerFlow is open-source so the prompt already lives in-repo - not a secret leak - but the fixture bakes in a point-in-time snapshot of a fast-moving prompt. Fine for the stated "trace format / asserter wiring" guard (and the README is upfront about it); just flagging the size/drift for awareness, no action needed.

[imohammedansari]

@willem-bd @WillemJiang made the changes you suggested. Ready for another look 🙏

[willem-bd]

Overall review: this is a clean, additive, well-isolated addition. Zero app-code changes; monocle_test_tools==0.8.8 is kept out of backend deps so CI stays light, the whole module skips cleanly via importorskip, and the live path is correctly gated behind OPENAI_API_KEY + app + config.yaml. I verified DeerFlowClient(config_path=...) and .chat(message, thread_id=...) match the real signatures, and the offline assertions line up with the committed trace (5 web_fetch >= min_count=2; 1 web_search; no image_search). README is unusually honest — including why local content-evals are omitted.

The inline comments are refinements, not blockers. The main one is a conscious call on the committed trace (trim or document it); the rest are docs tightening + one confirmation that -k live has been run green end-to-end.

Triage: Approve with a small follow-up on the trace artifact.

[willem-bd]

Worth a conscious decision on the committed trace. This loads web_research_ev_battery.json (~130 KB). Since it's a real recorded run, it embeds:

• the full DeerFlow 2.0 system prompt verbatim (already in-repo, so duplicated here),
• fetched third-party article content/URLs (e.g. electrek.co, insideevs.com, cars.com) plus the agent's final briefing output.

I scanned for credentials (sk-…, AKIA…, ghp_…, xox…) and found none — good. But the offline example only checks structural things (agent/tool/token/duration); it doesn't need the article bodies. Two options:

1. Trim the trace to spans + metadata + short I/O snippets — same assertion coverage at a fraction of the size (and it's permanent in git history once merged).
2. Or document in the README that this is a full real-run recording so future contributors know what they're committing.

Either is fine — just don't want it slipping in as an accident.

[willem-bd]

These offline assertions bind tightly to this exact trace: the "LangGraph" agent span name, the literal contains_input("solid-state EV batteries"), and the tool names. Any rename of the agent span, a tool, or the system-prompt wrapping will break this even when behavior is unchanged.

That's acceptable for an offline wiring test (the README says as much), but since the suite isn't in CI the breakage surfaces as a surprise to whoever next installs the requirements. A one-line README note — "offline assertions are pinned to this trace + instrumentation 0.8.8; re-record when prompt/tools/model change" — would save that future debugging.

[willem-bd]

The recorded trace has 5 web_fetch spans but this asserts only >= 2. If the intent is "at least a couple of fetches," a short comment saying so (and why it's not the exact count) would help — otherwise a reader may "tighten" it to the exact 5, which would flake the moment this shape is pointed at a live run (fetch count genuinely varies). The live tests correctly avoid asserting web_fetch counts, which is the right call.

[willem-bd]

run_agent here is run_deerflow(message: str). This only works if test_workflow unpacks the test_input tuple as positional args (run_agent(*test_input)); if it instead passes test_input as a kwarg, the live test raises TypeError rather than skipping.

It's monocle's documented pattern so presumably correct — but the PR description doesn't say the live path was actually executed end-to-end (only that it "skips by default"). Can you confirm -k live went green against a real app + OPENAI_API_KEY once? That's the main thing I can't verify from the diff alone.

[willem-bd]

The monocle_trace_asserter fixture is referenced with no import, so this depends on monocle_test_tools registering a pytest plugin (entry point) that auto-provides it. If it doesn't auto-register, collection errors with "unknown fixture."

Standard for the library, but worth a line in the README noting the plugin auto-enables (or whether a pytest_plugins line / config is needed) so a first-time installer isn't surprised.

[willem-bd]

Minor: monocle_test_tools==0.8.8 is pinned (good — the file trace source needs >=0.8), but python-dotenv is unpinned. A loose pin like python-dotenv>=1.0 would match the style and avoid an open-ended install. Not blocking.

[WillemJiang]

@imohammedansari Could you take a look at the latest review comments?

[imohammedansari]

@willem-bd @WillemJiang made the changes you suggested. Ready for another look 🙏

[fancyboi999]

Thanks @imohammedansari. The latest follow-up addresses the prior documentation comments, but I found two execution-gating issues in the live suite that should be fixed before this is ready.

Reviewed base SHA: c9b6131f8fc4beb186632556ea3d589488edc90f
Reviewed head SHA: bdac1add8dd2c7c1f79373c977d12aa1a7ef1a9f

[P2] Make the live tests explicitly opt-in

• Location: backend/tests/monocle/test_deerflow.py:72-95 and backend/tests/monocle/README.md:67-84
• Problem: The command documented as the offline, no-network path collects both live tests as well. After a developer installs the standalone requirements, a normal configured checkout has the key and config.yaml that make run_agent proceed, so pytest backend/tests/monocle/ (and later make test) can unexpectedly spend model tokens, perform web requests, and write to a sandbox.
• Evidence: Collection against this head selects all three tests for the unfiltered command: test_assertion_api_example, test_web_research_live, and test_sandbox_write_file_live. -k live narrows the second command, but omitting -k does not exclude those tests; run_agent also loads the repository .env, making the supposedly offline command most likely to go live in a real DeerFlow development checkout. Monocle 0.8.8's pytest plugin initializes the instrumentation for those selected tests.
• Suggested fix: Put the live tests behind an explicit opt-in marker/CLI option or environment flag that defaults off, and make the documented offline command select only the recorded-trace test. The default backend test run must remain incapable of making external/model calls even after the optional dependency is installed.
• Test: With a usable config and credentials plus a spy runner, assert that the default suite never invokes the runner, while the explicit live opt-in invokes both workflows.

[P2] Do not use an OpenAI-only credential gate for a config-resolved model

• Location: backend/tests/monocle/conftest.py:34-36
• Problem: run_deerflow() intentionally lets config.yaml select DeerFlow's model, but the fixture refuses to run unless OPENAI_API_KEY exists. Valid Anthropic, Gemini, Volcengine, DeepSeek, and other provider configurations are therefore skipped, while an unrelated OpenAI key lets the fixture proceed even when the selected provider's credentials are missing.
• Evidence: The repository's model config supports provider-specific keys such as ANTHROPIC_API_KEY, GEMINI_API_KEY, and VOLCENGINE_API_KEY. Executing this fixture with ANTHROPIC_API_KEY set and OPENAI_API_KEY absent exits with OPENAI_API_KEY not set before consulting the configured model. DuckDuckGo web search does not make OpenAI credentials a requirement for these workflows.
• Suggested fix: After adding the explicit live opt-in above, let DeerFlow's resolved model/config validate its own credentials, or validate the selected model's configured provider rather than one hard-coded environment variable. Update the README to describe configured-model credentials instead of OpenAI specifically.
• Test: Cover at least one OpenAI and one non-OpenAI configured model and assert that the fixture does not skip either solely because the other provider's key is absent.

[imohammedansari]

@fancyboi999 @WillemJiang Both fixed. The live tests are now explicit opt-in via MONOCLE_LIVE_TESTS=1, checked before anything reads .env, so the plain pytest backend/tests/monocle/ run cannot make model calls, web requests, or sandbox writes no matter what credentials the checkout has; test_live_gate_defaults_off pins the gate. And the run_agent fixture no longer requires OPENAI_API_KEY since config.yaml resolves the model, which can be any provider; credentials are validated by the configured model itself. Verified both modes locally: the default run is 2 passed 2 skipped with no network on a fully configured checkout, and with the flag on all 4 pass against real end-to-end runs 🚀