diff --git a/README.md b/README.md index 13238872152..c02f476bb8a 100644 --- a/README.md +++ b/README.md @@ -630,6 +630,8 @@ Gateway-generated follow-up suggestions now normalize both plain-string model ou Interrupted first-turn runs still persist a fallback conversation title, so stopping a streaming response does not leave the thread as "Untitled" after refresh. +In the Web UI, completed assistant turns can be branched into a new main conversation. The new thread starts from that turn's checkpoint. Because workspace files are not checkpointed, the branch only receives a best-effort copy of the current workspace when you branch from the latest turn; branching from an older turn keeps just the restored message history so the branch never inherits files that were created in a later part of the conversation. + ``` # Paths inside the sandbox container /mnt/skills/public diff --git a/README_zh.md b/README_zh.md index e3824eb2389..4103136f6c7 100644 --- a/README_zh.md +++ b/README_zh.md @@ -484,6 +484,8 @@ Tools 也是同样的思路。DeerFlow 自带一组核心工具：网页搜索 Gateway 生成后续建议时，现在会先把普通字符串输出和 block/list 风格的富文本内容统一归一化，再去解析 JSON 数组响应，因此不同 provider 的内容包装方式不会再悄悄把建议吞掉。 +Web UI 支持从已完成的 assistant 回复分叉出一个新的主对话。新 thread 会从该回复对应的 checkpoint 开始，并尽力复制当前 thread 的工作区文件。 + ```text # sandbox 容器内的路径 /mnt/skills/public diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 7e1b1929c59..a6116df4b53 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -227,28 +227,29 @@ Lead-agent middlewares are assembled in strict order across three functions: the 7. **LLMErrorHandlingMiddleware** - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later stages run 8. **GuardrailMiddleware** - *(optional, if `guardrails.enabled`)* Pre-tool-call authorization via pluggable `GuardrailProvider`; returns an error ToolMessage on deny. Providers: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) 9. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution -10. **ReadBeforeWriteMiddleware** - *(optional, if `read_before_write.enabled`, default on)* Version gate on file writes (issue #3857): `read_file` stamps a content hash onto its ToolMessage; `write_file` (append/overwrite-existing) and `str_replace` are blocked unless the newest mark for that path matches the file's current hash. Marks live on messages, so summarization dropping the read result invalidates the gate automatically; writes never refresh marks, forcing a re-read between consecutive edits. Gate check + tool execution are serialized per (thread, path) so same-turn parallel writes cannot reuse one stale mark; on sandboxes whose `read_file` reports failures as `"Error: ..."` strings instead of raising (AIO/E2B), uninspectable targets fail open (creation proceeds, no mark stamped) -11. **ToolErrorHandlingMiddleware** - Receives `AppConfig`, converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string. +10. **ReadBeforeWriteMiddleware** - *(optional, if `read_before_write.enabled`, default on)* Outermost write gate (issue #3857): `read_file` stamps a content hash onto its ToolMessage; `write_file` (append/overwrite-existing) and `str_replace` are blocked unless the newest mark for that path matches the file's current hash. Sits outside ToolProgressMiddleware and ToolErrorHandlingMiddleware so a blocked write returns immediately without consuming a ToolProgress slot. Blocked results call `normalize_tool_result` directly to stamp `deerflow_tool_meta` (`recoverable_by_model=True`) before returning, keeping the result well-formed for any outer consumer. Marks live on messages, so summarization dropping the read result invalidates the gate automatically; writes never refresh marks, forcing a re-read between consecutive edits. Gate check + tool execution are serialized per (thread, path) so same-turn parallel writes cannot reuse one stale mark; on sandboxes whose `read_file` reports failures as `"Error: ..."` strings instead of raising (AIO/E2B), uninspectable targets fail open (creation proceeds, no mark stamped) +11. **ToolProgressMiddleware** - *(optional, if `tool_progress.enabled`)* State-machine-based stagnation guard (RFC #3177). Outer wrapper around ToolErrorHandlingMiddleware so its `wrap_tool_call` receives results already stamped with `deerflow_tool_meta`. Tracks per-(thread, tool) consecutive "no-new-info" calls across three error categories: (a) `recoverable_by_model=True` (no_results, not_found, permission, Jaccard-duplicate success): ACTIVE → WARNED (terminal — hint re-injected on each subsequent problem); (b) `recoverable_by_model=False, action≠stop` (rate_limited, transient): ACTIVE → WARNED → BLOCKED after `warn_escalation_count` more problems; (c) `recoverable_by_model=False, action=stop` (auth, config, internal): immediately BLOCKED on first occurrence. **Division of labor with LoopDetectionMiddleware (item 25):** ToolProgressMiddleware is a result-quality guard — fires after tool execution and blocks specific tools that stop producing new information; LoopDetectionMiddleware is a call-pattern guard — fires after the model responds and hard-stops the whole turn when the model repeatedly issues identical tool_calls. Both can inject HumanMessage hints in the same model call without conflict; neither reads the other's internal state. +12. **ToolErrorHandlingMiddleware** - Receives `AppConfig`, converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting, stamps every result with `deerflow_tool_meta` (status / error_type / recoverable_by_model / recommended_next_action / source) via `tool_result_meta.normalize_tool_result`, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string. **Lead-only middlewares** (`build_middlewares`, appended after the base): -12. **DynamicContextMiddleware** - Injects the current date (and optionally memory) as a `` into the first HumanMessage, keeping the base system prompt fully static for prefix-cache reuse -13. **SkillActivationMiddleware** - Detects strict `/skill-name task` syntax on the latest real user message, resolves only enabled and runtime-allowed skills, injects the `SKILL.md` body as hidden current-turn context, and records a `middleware:skill_activation` audit event -14. **DurableContextMiddleware** - Captures `task` delegations into `ThreadState.delegations` (including in-progress dispatches and terminal result summaries) and loaded skill-file references (name/path/description, parsed in-memory - not the body) into `ThreadState.skill_context` before summarization can compact the paired tool-call/result messages, then projects durable context into each model request. Static authority rules are injected as a `SystemMessage`; untrusted field values (`summary_text`, delegation results, skill descriptions) are injected separately as a hidden `HumanMessage` data block so compressed history, delegated work, and which skills are active stay visible without being stored as `messages` or promoted to system-role instructions. -15. **SummarizationMiddleware** - *(optional, if enabled)* Context reduction when approaching token limits -16. **TodoListMiddleware** - *(optional, if `is_plan_mode`)* Task tracking with the `write_todos` tool -17. **TokenUsageMiddleware** - *(optional, if `token_usage.enabled`)* Records token usage metrics; subagent usage is merged back into the dispatching AIMessage by message position -18. **TitleMiddleware** - Auto-generates the thread title after the first complete exchange and normalizes structured message content before prompting the title model. If a first-turn run is interrupted before this middleware can write a title, `runtime/runs/worker.py` keeps the run in a finalizing state, persists a local fallback title from the latest checkpoint or original run input, and then syncs it to `threads_meta.display_name`. Replacement runs admitted by `multitask_strategy="interrupt"` / `"rollback"` wait for older same-thread finalization before entering the graph; the interrupted run only skips the fallback title write once a later run has started and may have advanced the checkpoint. -19. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses) -20. **ViewImageMiddleware** - *(optional, if the model supports vision)* Injects base64 image data before the LLM call -21. **DeferredToolFilterMiddleware** - *(optional, if `tool_search.enabled`)* Hides deferred (MCP) tool schemas from the bound model until `tool_search` promotes them (reads per-thread promotions from `ThreadState.promoted`, hash-scoped) -22. **SystemMessageCoalescingMiddleware** - Merges every SystemMessage into a single leading SystemMessage per request; provider-agnostic fix for strict backends (vLLM/SGLang/Qwen/Anthropic) that reject non-leading system messages. Touches the per-request payload only (checkpoint state unchanged); on midnight crossings only the latest `dynamic_context_reminder` SystemMessage survives -23. **SubagentLimitMiddleware** - *(optional, if `subagent_enabled`)* Truncates excess `task` tool calls to enforce the `MAX_CONCURRENT_SUBAGENTS` limit -24. **LoopDetectionMiddleware** - *(optional, if `loop_detection.enabled`)* Detects repeated tool-call loops; hard-stop clears both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer -25. **TokenBudgetMiddleware** - *(optional, if `token_budget.enabled`)* Enforces per-run token limits -26. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before the safety/clarification tail -27. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after custom middlewares so LangChain's reverse-order `after_model` dispatch runs it first -28. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last) +13. **DynamicContextMiddleware** - Injects the current date (and optionally memory) as a `` into the first HumanMessage, keeping the base system prompt fully static for prefix-cache reuse +14. **SkillActivationMiddleware** - Detects strict `/skill-name task` syntax on the latest real user message, resolves only enabled and runtime-allowed skills, injects the `SKILL.md` body as hidden current-turn context, and records a `middleware:skill_activation` audit event +15. **DurableContextMiddleware** - Captures `task` delegations into `ThreadState.delegations` (including in-progress dispatches and terminal result summaries) and loaded skill-file references (name/path/description, parsed in-memory - not the body) into `ThreadState.skill_context` before summarization can compact the paired tool-call/result messages, then projects durable context into each model request. Static authority rules are injected as a `SystemMessage`; untrusted field values (`summary_text`, delegation results, skill descriptions) are injected separately as a hidden `HumanMessage` data block so compressed history, delegated work, and which skills are active stay visible without being stored as `messages` or promoted to system-role instructions. +16. **SummarizationMiddleware** - *(optional, if enabled)* Context reduction when approaching token limits +17. **TodoListMiddleware** - *(optional, if `is_plan_mode`)* Task tracking with the `write_todos` tool +18. **TokenUsageMiddleware** - *(optional, if `token_usage.enabled`)* Records token usage metrics; subagent usage is merged back into the dispatching AIMessage by message position +19. **TitleMiddleware** - Auto-generates the thread title after the first complete exchange and normalizes structured message content before prompting the title model. If a first-turn run is interrupted before this middleware can write a title, `runtime/runs/worker.py` keeps the run in a finalizing state, persists a local fallback title from the latest checkpoint or original run input, and then syncs it to `threads_meta.display_name`. Replacement runs admitted by `multitask_strategy="interrupt"` / `"rollback"` wait for older same-thread finalization before entering the graph; the interrupted run only skips the fallback title write once a later run has started and may have advanced the checkpoint. +20. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses) +21. **ViewImageMiddleware** - *(optional, if the model supports vision)* Injects base64 image data before the LLM call +22. **DeferredToolFilterMiddleware** - *(optional, if `tool_search.enabled`)* Hides deferred (MCP) tool schemas from the bound model until `tool_search` promotes them (reads per-thread promotions from `ThreadState.promoted`, hash-scoped) +23. **SystemMessageCoalescingMiddleware** - Merges every SystemMessage into a single leading SystemMessage per request; provider-agnostic fix for strict backends (vLLM/SGLang/Qwen/Anthropic) that reject non-leading system messages. Touches the per-request payload only (checkpoint state unchanged); on midnight crossings only the latest `dynamic_context_reminder` SystemMessage survives +24. **SubagentLimitMiddleware** - *(optional, if `subagent_enabled`)* Truncates excess `task` tool calls to enforce the `MAX_CONCURRENT_SUBAGENTS` limit +25. **LoopDetectionMiddleware** - *(optional, if `loop_detection.enabled`)* Detects repeated tool-call loops; hard-stop clears both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer +26. **TokenBudgetMiddleware** - *(optional, if `token_budget.enabled`)* Enforces per-run token limits +27. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before the safety/clarification tail +28. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after custom middlewares so LangChain's reverse-order `after_model` dispatch runs it first +29. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, writes a readable `ToolMessage.content` fallback plus structured `ToolMessage.artifact.human_input` request payload, and interrupts via `Command(goto=END)` (must be last) ### Configuration System @@ -283,6 +284,11 @@ Config values starting with `$` are resolved as environment variables (e.g., `$O MCP servers and skills are configured together in `extensions_config.json` in project root: +Docker development mounts the project directory at `/app/project` and points +`DEER_FLOW_CONFIG_PATH` / `DEER_FLOW_EXTENSIONS_CONFIG_PATH` into that directory. +Keep mutable config files behind a directory bind mount: single-file bind mounts +can become stale or inaccessible when a host editor replaces a file on save. + Configuration priority: 1. Explicit `config_path` argument 2. `DEER_FLOW_EXTENSIONS_CONFIG_PATH` environment variable @@ -306,7 +312,7 @@ CORS is same-origin by default when requests enter through nginx on port 2026. S | **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) | | **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data | | **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete | -| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; unexpected failures are logged server-side and return a generic 500 detail | +| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - create a new main-thread branch from a completed assistant turn checkpoint. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline; `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; unexpected failures are logged server-side and return a generic 500 detail | | **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types | | **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`...`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing | | **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `POST /regenerate/prepare` - prepare clean input + checkpoint metadata for regenerating the latest assistant answer; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /{rid}/workspace-changes` - workspace/output file change summary and optional diffs; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens | @@ -343,6 +349,12 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti **Implementations**: - `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. - `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths. +- `BoxliteProvider` (`packages/harness/deerflow/community/boxlite/`) - BoxLite micro-VM isolation. The `boxlite` runtime is optional (`deerflow-harness[boxlite]`) and lazy-imported only when this provider is selected. The provider owns one private asyncio event loop on a daemon thread because BoxLite handles are loop-affine; sync `Sandbox` calls marshal onto that loop with `run_coroutine_threadsafe`. + Boxes are named deterministically from `user_id:thread_id`, released into an in-process warm pool after each agent turn, and reclaimed only by the same user/thread. Warm-pool health checks use a short explicit timeout and forward that timeout through both BoxLite `exec(timeout=...)` and the private-loop `.result(timeout)` bridge so a hung VM cannot pin the per-thread acquire lock indefinitely. + `sandbox.replicas` caps active + warm VMs per gateway process; if capacity is exhausted, only warm-pool VMs are evicted. `sandbox.idle_timeout` stops idle warm VMs after the configured seconds. `reset()` is intentionally a lightweight registry clear for `reset_sandbox_provider()` and does not close boxes, stop the idle reaper, or close the private loop; full teardown remains `shutdown()`. + + +**Shared warm-pool lifecycle:** community sandbox providers that keep released sandboxes alive for fast reuse share `deerflow.community.warm_pool_lifecycle.WarmPoolLifecycleMixin`. The mixin owns the common `DEFAULT_IDLE_TIMEOUT=600`, `IDLE_CHECK_INTERVAL=60`, `DEFAULT_REPLICAS=3`, idle-checker loop, warm-pool expiry, oldest-warm eviction, replica counting, and soft-cap logging. Providers remain responsible for their own active registries, creation/discovery, health checks, and destroy hook (`_destroy_warm_entry`): AIO destroys `SandboxInfo` through its backend; Boxlite closes loop-affine `BoxliteBox` handles. AIO keeps active-idle cleanup outside the mixin and delegates only warm-pool expiry to the shared helper. **Virtual Path System**: - Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills` @@ -364,6 +376,7 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti **Concurrency**: `MAX_CONCURRENT_SUBAGENTS = 3` enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`); default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box) **Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result **Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out` +**Turn-budget cap (#3875 Phase 2)**: `recursion_limit` on the subagent `run_config` equals `max_turns`, so exhausting the turn budget raises `GraphRecursionError` from `agent.astream`. `executor.py::_aexecute` catches it specifically (before the generic `except Exception`) and sets `SubagentStatus.MAX_TURNS_REACHED` — distinct from `FAILED` — with the **partial result recovered from the last streamed chunk** via `_extract_final_result` (which delegates to the shared `utils/messages.py::message_content_to_text`, returning a `"No response generated"` sentinel when no text survived). Previously the exception fell through to the generic handler and was misclassified as `FAILED`, so the lead could not tell "broken subagent" from "out of budget" and the work already streamed into `final_state` was discarded. `task_tool.py` returns it through the shared `_task_result_command(status="max_turns_reached", result=partial, error=cap)`, which `format_subagent_result_message` renders as `Task reached max turns. Partial result: ` and `make_subagent_additional_kwargs` stamps on `additional_kwargs` — `max_turns_reached` is the one status that carries **both** `subagent_result_brief`/`subagent_result_sha256` (the recovered partial work, like `completed`) and `subagent_error` (the cap notice). The polling loop emits a `task_failed` event so the card transitions out of running; the structured `subagent_status` is the precise reason. The cross-language status contract (`contracts/subagent_status_contract.json` + `subagents/status_contract.py` + `frontend/.../subtask-result.ts`) collapses `max_turns_reached` to the frontend's `failed` pill while the cap detail and recovered work survive on `error`/`result_brief`; the durable delegation ledger prefers the partial `result_brief` and renders model-facing guidance to reuse it, retry with a tighter scope, or raise the per-agent `max_turns`. **Step capture & persistence (#3779)**: `executor.py` captures both assistant turns (`AIMessage`) **and** tool outputs (`ToolMessage`) via `subagents/step_events.py::capture_new_step_messages`, which walks the *newly-appended tail* of each `stream_mode="values"` chunk (not just `messages[-1]`) so a multi-tool-call turn — where LangGraph's `ToolNode` appends several `ToolMessage`s in one super-step — keeps every tool output instead of dropping all but the last. `runtime/runs/worker.py::_SubagentEventBuffer` additionally persists these `task_*` custom events to the `RunEventStore` as `subagent.start`/`subagent.step`/`subagent.end` (`category="subagent"`, `task_id` in `metadata`). It **batches** writes via `put_batch` (flushing on a terminal `subagent.end`, at `FLUSH_THRESHOLD` events, and in the worker's `finally`) rather than one `put()` per step, since `put()` is a documented low-frequency path (per-thread advisory lock per call) and a deep subagent (`max_turns=150`) emits hundreds of steps on the hot stream loop. `build_subagent_step` caps both the per-step `text` and each tool call's serialized `args` at `SUBAGENT_STEP_MAX_CHARS` (flagged `truncated` / `args_truncated`) so a large `write_file`/`bash` payload can't produce an unbounded row. The dedicated category keeps them out of `list_messages` (the thread feed) while `list_events` returns them for the frontend's fetch-on-expand backfill. `list_events` accepts `task_id` (filters on `metadata["task_id"]` — SQL-side in `DbRunEventStore` via `event_metadata["task_id"].as_string()`, in-memory in the JSONL/memory stores) plus an `after_seq` forward cursor, so the card pages through one subagent's steps without the run-wide `limit` truncating the tail (no schema migration: the filter rides the existing run-scoped index). `step_events.py` is a pure, unit-tested layer (`build_subagent_step` / `subagent_run_event`). **Deferred MCP tools** (if `tool_search.enabled`): `SubagentExecutor._build_initial_state` assembles deferral after policy filtering via the shared `assemble_deferred_tools` (fail-closed), appends the `tool_search` tool, injects the `` section into the subagent's `SystemMessage`, and threads the setup to `_create_agent`, which attaches `DeferredToolFilterMiddleware` through `build_subagent_runtime_middlewares(deferred_setup=...)`. Subagents thus withhold full MCP schemas until promotion, same as the lead agent; each task run gets a fresh `ThreadState` so promotion is isolated per run **Checkpointer isolation**: Subagent graphs are compiled with `checkpointer=False` to avoid inheriting the parent run's checkpointer, since subagents are one-shot and never resume. @@ -375,7 +388,7 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti 2. **MCP tools** - From enabled MCP servers (lazy initialized, cached with mtime invalidation) 3. **Built-in tools**: - `present_files` - Make output files visible to user (only `/mnt/user-data/outputs`) - - `ask_clarification` - Request clarification (intercepted by ClarificationMiddleware → interrupts) + - `ask_clarification` - Request clarification (intercepted by ClarificationMiddleware, which preserves text fallback and adds `artifact.human_input` for Web UI Human Input Cards) - `view_image` - Read image as base64 (added only if model supports vision) - `setup_agent` - Bootstrap-only: persist a brand-new custom agent's `SOUL.md` and `config.yaml`. Bound only when `is_bootstrap=True`. - `update_agent` - Custom-agent-only: persist self-updates to the current agent's `SOUL.md` / `config.yaml` from inside a normal chat (partial update + atomic write). Bound when `agent_name` is set and `is_bootstrap=False`. @@ -547,7 +560,8 @@ The cached value is reused for both the blocking (`runs.wait`) and streaming (`_ 2. Queue debounces (30s default), batches updates, deduplicates per-thread 3. Background thread invokes LLM to extract context updates and facts, using the stored `user_id` (not the contextvar, which is unavailable on timer threads) 4. Applies updates atomically (temp file + rename) with cache invalidation, skipping duplicate fact content before append -5. Next interaction injects top 15 facts + context into `` tags in system prompt +5. **Staleness pass** (same LLM invocation as step 3, no extra API call): when `staleness_review_enabled` is `true` and at least `staleness_min_candidates` aged facts exist, `_select_stale_candidates` selects facts older than `staleness_age_days` that are not in `staleness_protected_categories` (default: `correction`), surfaces them in the prompt, and the LLM judges each as KEEP or REMOVE. `_apply_updates` enforces the guardrail unconditionally at apply time: it intersects the LLM-returned removal set with `_select_stale_candidates` output before applying the per-cycle cap (`staleness_max_removals_per_cycle`), so protected and non-aged facts can never be deleted regardless of model behavior or the feature flag setting. +6. Next interaction injects top 15 facts + context into `` tags in system prompt **Token counting** (`packages/harness/deerflow/agents/memory/prompt.py`): - `_count_tokens` budgets the injection. In default `tiktoken` mode, the encoding is loaded lazily and cached. @@ -565,6 +579,11 @@ Focused regression coverage for the updater lives in `backend/tests/test_memory_ - `max_facts` / `fact_confidence_threshold` - Fact storage limits (100 / 0.7) - `max_injection_tokens` - Token limit for prompt injection (2000) - `token_counting` - Token counting strategy for the injection budget: `tiktoken` (default, accurate but may download BPE data from a public endpoint on first use — can block for a long time in network-restricted environments, see issues #3402/#3429) or `char` (network-free CJK-aware char estimate, never touches tiktoken) +- `staleness_review_enabled` - Enable proactive staleness pruning of aged facts (default: `true`; only triggers when aged candidates exist) +- `staleness_age_days` - Age in days before a fact becomes a staleness candidate (default: 180; range: 1–3650) +- `staleness_min_candidates` - Minimum aged candidates required to trigger a review cycle (default: 3; range: 1–50) +- `staleness_max_removals_per_cycle` - Maximum facts removed in a single cycle; lowest-confidence entries are kept when the LLM requests more (default: 5; range: 1–20) +- `staleness_protected_categories` - Fact categories that are never pruned by staleness review (default: `["correction"]`) ### Reflection System (`packages/harness/deerflow/reflection/`) @@ -682,7 +701,7 @@ Returns `{}` when Langfuse is not in the enabled providers — LangSmith-only de - `title` - Auto-title generation (enabled, max_words, max_chars, model_name; null model_name uses fast local fallback, explicit model_name uses the prompt_template LLM path) - `summarization` - Context summarization (enabled, trigger conditions, keep policy) - `subagents.enabled` - Master switch for subagent delegation -- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens) +- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens, staleness_review_enabled, staleness_age_days, staleness_min_candidates, staleness_max_removals_per_cycle, staleness_protected_categories) **`extensions_config.json`**: - `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description) diff --git a/backend/app/gateway/deps.py b/backend/app/gateway/deps.py index c3d935696f3..93197f2ffde 100644 --- a/backend/app/gateway/deps.py +++ b/backend/app/gateway/deps.py @@ -19,6 +19,7 @@ import asyncio import logging +import os from collections.abc import AsyncGenerator, Callable from contextlib import AsyncExitStack, asynccontextmanager from typing import TYPE_CHECKING, TypeVar, cast @@ -45,6 +46,27 @@ _RUN_DRAIN_TIMEOUT_SECONDS = 5.0 +def _enforce_postgres_for_multi_worker(config: AppConfig) -> None: + """Refuse to start when GATEWAY_WORKERS > 1 and the DB backend is not Postgres. + + SQLite write-locks cannot support concurrent multi-process access. + This gate runs once at startup before any persistence engine is + initialised so the error message is clear and the process exits + immediately. + """ + try: + workers = int(os.environ.get("GATEWAY_WORKERS", "1")) + except (TypeError, ValueError): + workers = 1 + + if workers <= 1: + return + + backend = getattr(config.database, "backend", None) + if backend != "postgres": + raise SystemExit(f"GATEWAY_WORKERS={workers} requires database.backend='postgres', but database.backend is '{backend}'. SQLite cannot support concurrent multi-process access. Set GATEWAY_WORKERS=1 or switch to Postgres.") + + async def _drain_inflight_runs(run_manager: RunManager) -> None: """Drain in-flight runs before the checkpointer is torn down (issue #3373). @@ -209,6 +231,12 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen from deerflow.runtime.checkpointer.async_provider import make_checkpointer from deerflow.runtime.events.store import make_run_event_store + # ------------------------------------------------------------------ + # Multi-worker safety gate: reject SQLite when GATEWAY_WORKERS > 1. + # SQLite write-locks cannot support concurrent multi-process access. + # ------------------------------------------------------------------ + _enforce_postgres_for_multi_worker(startup_config) + async with AsyncExitStack() as stack: config = startup_config diff --git a/backend/app/gateway/routers/memory.py b/backend/app/gateway/routers/memory.py index 6d1c356c837..75f41934bec 100644 --- a/backend/app/gateway/routers/memory.py +++ b/backend/app/gateway/routers/memory.py @@ -131,6 +131,11 @@ class MemoryConfigResponse(BaseModel): ..., description="Token ceiling for guaranteed-category facts (displaces regular lines in the common case; additive only when guaranteed alone overflows max_injection_tokens)", ) + staleness_review_enabled: bool = Field(..., description="Whether staleness review is enabled for aged facts") + staleness_age_days: int = Field(..., description="Facts older than this many days are candidates for staleness review") + staleness_min_candidates: int = Field(..., description="Minimum stale facts required to trigger a review cycle") + staleness_max_removals_per_cycle: int = Field(..., description="Maximum number of facts staleness review can remove per cycle") + staleness_protected_categories: list[str] = Field(..., description="Fact categories exempt from staleness review") class MemoryStatusResponse(BaseModel): @@ -360,6 +365,11 @@ async def get_memory_config_endpoint() -> MemoryConfigResponse: token_counting=config.token_counting, guaranteed_categories=config.guaranteed_categories, guaranteed_token_budget=config.guaranteed_token_budget, + staleness_review_enabled=config.staleness_review_enabled, + staleness_age_days=config.staleness_age_days, + staleness_min_candidates=config.staleness_min_candidates, + staleness_max_removals_per_cycle=config.staleness_max_removals_per_cycle, + staleness_protected_categories=config.staleness_protected_categories, ) @@ -391,6 +401,11 @@ async def get_memory_status(http_request: Request) -> MemoryStatusResponse: token_counting=config.token_counting, guaranteed_categories=config.guaranteed_categories, guaranteed_token_budget=config.guaranteed_token_budget, + staleness_review_enabled=config.staleness_review_enabled, + staleness_age_days=config.staleness_age_days, + staleness_min_candidates=config.staleness_min_candidates, + staleness_max_removals_per_cycle=config.staleness_max_removals_per_cycle, + staleness_protected_categories=config.staleness_protected_categories, ), data=MemoryResponse(**memory_data), ) diff --git a/backend/app/gateway/routers/threads.py b/backend/app/gateway/routers/threads.py index b8ca4c0dd24..7d0173c76d7 100644 --- a/backend/app/gateway/routers/threads.py +++ b/backend/app/gateway/routers/threads.py @@ -12,8 +12,11 @@ from __future__ import annotations +import copy import logging +import shutil import uuid +from pathlib import Path from typing import Any from fastapi import APIRouter, HTTPException, Request @@ -28,6 +31,7 @@ from deerflow.runtime import serialize_channel_values_for_api from deerflow.runtime.goal import DEFAULT_MAX_GOAL_CONTINUATIONS, build_goal_state, ensure_thread_checkpoint, goal_thread_lock, read_thread_goal, write_thread_goal from deerflow.runtime.user_context import get_effective_user_id +from deerflow.utils.file_io import run_file_io from deerflow.utils.time import coerce_iso, now_iso logger = logging.getLogger(__name__) @@ -41,6 +45,9 @@ # row-level invariant is still ``threads_meta.user_id`` populated from # the auth contextvar; this list closes the metadata-blob echo gap. _SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id"}) +_SIDECAR_METADATA_KEY = "deerflow_sidecar" +_BRANCH_METADATA_KEY = "deerflow_branch" +_BRANCH_HISTORY_SCAN_LIMIT = 200 def _strip_reserved_metadata(metadata: dict[str, Any] | None) -> dict[str, Any]: @@ -50,6 +57,155 @@ def _strip_reserved_metadata(metadata: dict[str, Any] | None) -> dict[str, Any]: return {k: v for k, v in metadata.items() if k not in _SERVER_RESERVED_METADATA_KEYS} +def _message_id(message: Any) -> str | None: + if isinstance(message, dict): + raw = message.get("id") + else: + raw = getattr(message, "id", None) + return raw if isinstance(raw, str) and raw else None + + +def _message_type(message: Any) -> str | None: + if isinstance(message, dict): + raw = message.get("type") + else: + raw = getattr(message, "type", None) + return raw if isinstance(raw, str) and raw else None + + +def _message_additional_kwargs(message: Any) -> dict[str, Any]: + if isinstance(message, dict): + raw = message.get("additional_kwargs") + else: + raw = getattr(message, "additional_kwargs", None) + return raw if isinstance(raw, dict) else {} + + +def _is_branch_visible_message(message: Any) -> bool: + if _message_additional_kwargs(message).get("hide_from_ui") is True: + return False + return _message_type(message) in {"human", "ai"} + + +def _is_branch_assistant_message(message: Any) -> bool: + return _message_type(message) == "ai" + + +def _checkpoint_messages(checkpoint_tuple: Any) -> list[Any]: + checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {} + channel_values = checkpoint.get("channel_values", {}) or {} + messages = channel_values.get("messages") or [] + return list(messages) if isinstance(messages, list) else [] + + +def _checkpoint_id(checkpoint_tuple: Any) -> str | None: + config = getattr(checkpoint_tuple, "config", {}) or {} + raw = config.get("configurable", {}).get("checkpoint_id") + return raw if isinstance(raw, str) and raw else None + + +def _matches_branch_target(messages: list[Any], target_message_ids: set[str]) -> bool: + if not target_message_ids: + return False + + index_by_id = {_message_id(message): index for index, message in enumerate(messages) if _message_id(message)} + if not target_message_ids.issubset(index_by_id.keys()): + return False + if any(not _is_branch_assistant_message(messages[index_by_id[message_id]]) for message_id in target_message_ids): + return False + + target_end_index = max(index_by_id[message_id] for message_id in target_message_ids) + return not any(_is_branch_visible_message(message) for message in messages[target_end_index + 1 :]) + + +async def _find_branch_checkpoint(checkpointer: Any, thread_id: str, target_message_ids: set[str]) -> Any: + config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}} + try: + async for checkpoint_tuple in checkpointer.alist(config, limit=_BRANCH_HISTORY_SCAN_LIMIT): + if _matches_branch_target(_checkpoint_messages(checkpoint_tuple), target_message_ids): + return checkpoint_tuple + except Exception: + logger.exception("Failed to scan branch checkpoint history for thread %s", sanitize_log_param(thread_id)) + raise HTTPException(status_code=500, detail="Failed to find branch checkpoint") + raise HTTPException(status_code=409, detail="This turn can no longer be branched from.") + + +async def _branch_targets_latest_turn(checkpointer: Any, thread_id: str, target_message_ids: set[str]) -> bool: + """Return True when the target turn is the final visible turn in the current state. + + ``alist`` yields newest-first; we take the newest checkpoint that actually holds + messages (thread creation writes an empty checkpoint that must be skipped) and + reuse ``_matches_branch_target`` to check the target turn is its tail. Used to + decide whether cloning the (uncheckpointed) workspace onto a branch is safe: only + a branch from the latest turn shares the current workspace timeline. On any lookup + failure we fail closed (treat as historical) so a branch from an older turn never + inherits a later timeline's workspace files. + """ + config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}} + try: + async for checkpoint_tuple in checkpointer.alist(config, limit=_BRANCH_HISTORY_SCAN_LIMIT): + messages = _checkpoint_messages(checkpoint_tuple) + if not messages: + continue + return _matches_branch_target(messages, target_message_ids) + except Exception: + logger.warning( + "Failed to resolve latest turn for thread %s; treating branch as historical", + sanitize_log_param(thread_id), + exc_info=True, + ) + return False + + +def _ignore_branch_user_data(directory: str, names: list[str]) -> set[str]: + ignored: set[str] = set() + base = Path(directory) + for name in names: + path = base / name + if name.startswith(".upload-") and name.endswith(".part"): + ignored.add(name) + elif path.is_symlink(): + ignored.add(name) + return ignored + + +def _copy_branch_user_data_sync(paths: Paths, source_thread_id: str, target_thread_id: str, *, user_id: str) -> str: + source = paths.sandbox_user_data_dir(source_thread_id, user_id=user_id) + target = paths.sandbox_user_data_dir(target_thread_id, user_id=user_id) + if not source.exists(): + return "not_found" + + shutil.copytree(source, target, ignore=_ignore_branch_user_data, dirs_exist_ok=True) + return "current_thread_best_effort" + + +async def _copy_branch_user_data(source_thread_id: str, target_thread_id: str) -> str: + paths = get_paths() + user_id = get_effective_user_id() + try: + return await run_file_io(_copy_branch_user_data_sync, paths, source_thread_id, target_thread_id, user_id=user_id) + except Exception: + logger.warning( + "Failed to copy user-data for branch %s -> %s", + sanitize_log_param(source_thread_id), + sanitize_log_param(target_thread_id), + exc_info=True, + ) + return "failed" + + +def _default_branch_display_name(source_title: Any, *, source_is_branch: bool = False) -> str | None: + if not isinstance(source_title, str): + return None + + display_name = source_title.strip() + if source_is_branch: + while display_name.lower().startswith("branch:"): + display_name = display_name[len("branch:") :].strip() + + return display_name or None + + # --------------------------------------------------------------------------- # Response / request models # --------------------------------------------------------------------------- @@ -181,6 +337,24 @@ class ThreadHistoryRequest(BaseModel): before: str | None = Field(default=None, description="Cursor for pagination") +class ThreadBranchRequest(BaseModel): + """Request body for creating a branch from a completed assistant turn.""" + + message_id: str = Field(..., min_length=1, description="Target assistant message ID to branch from") + message_ids: list[str] = Field(default_factory=list, description="All assistant message IDs in the target turn") + title: str | None = Field(default=None, max_length=256, description="Optional title for the branched thread") + + +class ThreadBranchResponse(BaseModel): + """Response model for a thread branch.""" + + thread_id: str + parent_thread_id: str + parent_checkpoint_id: str + branched_from_message_id: str + workspace_clone_mode: str + + # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- @@ -367,6 +541,98 @@ async def create_thread(body: ThreadCreateRequest, request: Request) -> ThreadRe ) +@router.post("/{thread_id}/branches", response_model=ThreadBranchResponse) +@require_permission("threads", "write", owner_check=True, require_existing=True) +async def branch_thread(thread_id: str, body: ThreadBranchRequest, request: Request) -> ThreadBranchResponse: + """Create a new main-thread branch from a completed assistant turn.""" + from app.gateway.deps import get_thread_store + + checkpointer = get_checkpointer(request) + thread_store = get_thread_store(request) + + source_record = await thread_store.get(thread_id) + if source_record is None: + raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found") + + source_metadata = source_record.get("metadata") or {} + if source_metadata.get(_SIDECAR_METADATA_KEY) is True: + raise HTTPException(status_code=409, detail="Branching is only available in the main conversation.") + + target_message_ids = {body.message_id, *body.message_ids} + checkpoint_tuple = await _find_branch_checkpoint(checkpointer, thread_id, target_message_ids) + parent_checkpoint_id = _checkpoint_id(checkpoint_tuple) + if not parent_checkpoint_id: + raise HTTPException(status_code=409, detail="This turn can no longer be branched from.") + + # Workspace files are not checkpointed, so they only reflect the *current* thread + # state. Cloning them onto a branch from an older turn would leak files created + # after that turn (message history rolls back, workspace would not). Restrict the + # best-effort clone to branches taken from the latest turn so history and workspace + # stay consistent. + branch_from_latest_turn = await _branch_targets_latest_turn(checkpointer, thread_id, target_message_ids) + + new_thread_id = str(uuid.uuid4()) + now = now_iso() + branch_metadata = { + _BRANCH_METADATA_KEY: True, + "branch_parent_thread_id": thread_id, + "branch_parent_checkpoint_id": parent_checkpoint_id, + "branch_parent_message_id": body.message_id, + "branch_created_at": now, + } + + display_name = body.title or _default_branch_display_name( + source_record.get("display_name"), + source_is_branch=source_metadata.get(_BRANCH_METADATA_KEY) is True, + ) + thread_owner_user_id = get_trusted_internal_owner_user_id(request) + thread_owner_kwargs = {"user_id": thread_owner_user_id} if thread_owner_user_id else {} + + checkpoint = copy.deepcopy(getattr(checkpoint_tuple, "checkpoint", {}) or {}) + metadata = copy.deepcopy(getattr(checkpoint_tuple, "metadata", {}) or {}) + checkpoint["id"] = str(uuid6()) + metadata.update( + { + "source": "branch", + "updated_at": now, + "created_at": now, + **branch_metadata, + } + ) + + write_config = {"configurable": {"thread_id": new_thread_id, "checkpoint_ns": ""}} + new_versions = dict(checkpoint.get("channel_versions", {}) or {}) + try: + await checkpointer.aput(write_config, checkpoint, metadata, new_versions) + except Exception: + logger.exception("Failed to write branch checkpoint for thread %s", sanitize_log_param(new_thread_id)) + raise HTTPException(status_code=500, detail="Failed to create branch") from None + + try: + await thread_store.create( + new_thread_id, + assistant_id=source_record.get("assistant_id"), + display_name=display_name, + metadata=branch_metadata, + **thread_owner_kwargs, + ) + except Exception: + logger.exception("Failed to write branch thread_meta for %s", sanitize_log_param(new_thread_id)) + raise HTTPException(status_code=500, detail="Failed to create branch") from None + + if branch_from_latest_turn: + workspace_clone_mode = await _copy_branch_user_data(thread_id, new_thread_id) + else: + workspace_clone_mode = "skipped_historical_turn" + return ThreadBranchResponse( + thread_id=new_thread_id, + parent_thread_id=thread_id, + parent_checkpoint_id=parent_checkpoint_id, + branched_from_message_id=body.message_id, + workspace_clone_mode=workspace_clone_mode, + ) + + @router.post("/search", response_model=list[ThreadResponse]) async def search_threads(body: ThreadSearchRequest, request: Request) -> list[ThreadResponse]: """Search and list threads. diff --git a/backend/docs/CONFIGURATION.md b/backend/docs/CONFIGURATION.md index 5c4c875b335..f2f93e8688f 100644 --- a/backend/docs/CONFIGURATION.md +++ b/backend/docs/CONFIGURATION.md @@ -342,6 +342,32 @@ sandbox: use: deerflow.community.aio_sandbox:AioSandboxProvider # Docker-based sandbox ``` +**BoxLite micro-VM Sandbox** (runs sandbox code in daemonless OCI micro-VMs): +```yaml +sandbox: + use: deerflow.community.boxlite:BoxliteProvider + image: python:3.12-slim + memory_mib: 1024 # optional per-box memory cap + cpus: 2 # optional per-box vCPUs + replicas: 3 # max active + warm VMs per gateway process + idle_timeout: 600 # warm VM idle seconds before stop; 0 disables idle reaping + environment: + PYTHONUNBUFFERED: "1" +``` + +Install the optional runtime before selecting this provider: + +```bash +pip install "deerflow-harness[boxlite]" +``` + +BoxLite boxes are named from the effective `(user_id, thread_id)` scope and are +released into an in-process warm pool after each turn. The same user/thread can +reclaim its warm VM on the next acquire; different threads cannot share a VM. +`replicas` caps active plus warm VMs. When the cap is reached only warm VMs are +evicted; active VMs continue and the provider may temporarily exceed the cap if +all boxes are active. + **Docker Execution with Kubernetes** (runs sandbox code in Kubernetes pods via provisioner service): This mode runs each sandbox in an isolated Kubernetes Pod on your **host machine's cluster**. Requires Docker Desktop K8s, OrbStack, or similar local K8s setup. diff --git a/backend/packages/harness/deerflow/agents/human_input.py b/backend/packages/harness/deerflow/agents/human_input.py new file mode 100644 index 00000000000..bea935b78a9 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/human_input.py @@ -0,0 +1,76 @@ +"""Structured human-input message metadata helpers.""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Literal, TypedDict + +HUMAN_INPUT_RESPONSE_KEY = "human_input_response" + + +class HumanInputTextResponse(TypedDict): + version: Literal[1] + kind: Literal["human_input_response"] + source: str + request_id: str + response_kind: Literal["text"] + value: str + + +class HumanInputOptionResponse(TypedDict): + version: Literal[1] + kind: Literal["human_input_response"] + source: str + request_id: str + response_kind: Literal["option"] + option_id: str + value: str + + +HumanInputResponse = HumanInputTextResponse | HumanInputOptionResponse + + +def _non_empty_string(value: object) -> str | None: + return value if isinstance(value, str) and value.strip() else None + + +def read_human_input_response(additional_kwargs: Mapping[str, object] | None) -> HumanInputResponse | None: + """Read a valid human-input response payload from message metadata.""" + if not additional_kwargs: + return None + raw = additional_kwargs.get(HUMAN_INPUT_RESPONSE_KEY) + if not isinstance(raw, Mapping): + return None + if raw.get("version") != 1 or raw.get("kind") != "human_input_response": + return None + + source = _non_empty_string(raw.get("source")) + request_id = _non_empty_string(raw.get("request_id")) + value = _non_empty_string(raw.get("value")) + if source is None or request_id is None or value is None: + return None + + response_kind = raw.get("response_kind") + if response_kind == "text": + return { + "version": 1, + "kind": "human_input_response", + "source": source, + "request_id": request_id, + "response_kind": "text", + "value": value, + } + if response_kind == "option": + option_id = _non_empty_string(raw.get("option_id")) + if option_id is None: + return None + return { + "version": 1, + "kind": "human_input_response", + "source": source, + "request_id": request_id, + "response_kind": "option", + "option_id": option_id, + "value": value, + } + return None diff --git a/backend/packages/harness/deerflow/agents/memory/message_processing.py b/backend/packages/harness/deerflow/agents/memory/message_processing.py index 5cdaa8d4e29..81feaedfd56 100644 --- a/backend/packages/harness/deerflow/agents/memory/message_processing.py +++ b/backend/packages/harness/deerflow/agents/memory/message_processing.py @@ -6,6 +6,8 @@ from copy import copy from typing import Any +from deerflow.agents.human_input import read_human_input_response + _UPLOAD_BLOCK_RE = re.compile(r"[\s\S]*?\n*", re.IGNORECASE) _CORRECTION_PATTERNS = ( re.compile(r"\bthat(?:'s| is) (?:wrong|incorrect)\b", re.IGNORECASE), @@ -66,7 +68,8 @@ def filter_messages_for_memory(messages: list[Any]) -> list[Any]: # hide_from_ui and must never reach the memory-updating LLM — otherwise # framework-internal text pollutes long-term memory (and the p0 __memory # payload could trigger a self-amplification loop). - if getattr(msg, "additional_kwargs", {}).get("hide_from_ui"): + additional_kwargs = getattr(msg, "additional_kwargs", {}) or {} + if additional_kwargs.get("hide_from_ui") and read_human_input_response(additional_kwargs) is None: continue content_str = extract_message_text(msg) if "" in content_str: diff --git a/backend/packages/harness/deerflow/agents/memory/prompt.py b/backend/packages/harness/deerflow/agents/memory/prompt.py index ac562023633..9fc8ae2bc96 100644 --- a/backend/packages/harness/deerflow/agents/memory/prompt.py +++ b/backend/packages/harness/deerflow/agents/memory/prompt.py @@ -115,7 +115,8 @@ "newFacts": [ {{ "content": "...", "category": "preference|knowledge|context|behavior|goal|correction", "confidence": 0.0-1.0 }} ], - "factsToRemove": ["fact_id_1", "fact_id_2"] + "factsToRemove": ["fact_id_1", "fact_id_2"], + "staleFactsToRemove": [{{ "id": "fact_id", "reason": "brief explanation" }}] }} Important Rules: @@ -135,9 +136,40 @@ session-specific and ephemeral — they will not be accessible in future sessions. Recording upload events causes confusion in subsequent conversations. +{staleness_review_section} + Return ONLY valid JSON, no explanation or markdown.""" +# Prompt section injected into MEMORY_UPDATE_PROMPT when staleness review triggers. +# Surfaces aged facts explicitly so the LLM can semantically judge each one, +# rather than relying on passive contradiction from the current conversation. +STALENESS_REVIEW_PROMPT = """## Staleness Review + +The following facts were created more than {age_days} days ago and may no longer +accurately reflect the user's current situation. Review each one against the full +conversation context and your understanding of the user. + + +{stale_facts} + + +For each fact, decide KEEP or REMOVE: +- KEEP: Still likely valid — even if not mentioned in this conversation. + Stable attributes (native language, core expertise, personality traits) often + remain true indefinitely. +- REMOVE: Outdated, contradicted by recent context, or no longer relevant. + Examples: tech-stack migrations, job changes, relocated offices, abandoned projects. + +Add REMOVE decisions to "staleFactsToRemove" in your output JSON. +Each entry must be {{"id": "fact_id", "reason": "brief explanation"}}. +The reason should cite what signal in the conversation (or absence thereof) +supports the removal. + +Be conservative — when in doubt, KEEP. Removing a valid fact is worse than +keeping a slightly stale one, because the next review cycle will re-evaluate it.""" + + # Prompt template for extracting facts from a single message FACT_EXTRACTION_PROMPT = """Extract factual information about the user from this message. diff --git a/backend/packages/harness/deerflow/agents/memory/updater.py b/backend/packages/harness/deerflow/agents/memory/updater.py index 1391c71909f..c53cc3c1c70 100644 --- a/backend/packages/harness/deerflow/agents/memory/updater.py +++ b/backend/packages/harness/deerflow/agents/memory/updater.py @@ -11,10 +11,12 @@ import re import uuid from contextlib import nullcontext +from datetime import UTC, datetime, timedelta from typing import Any from deerflow.agents.memory.prompt import ( MEMORY_UPDATE_PROMPT, + STALENESS_REVIEW_PROMPT, format_conversation_for_update, ) from deerflow.agents.memory.storage import ( @@ -301,11 +303,30 @@ def _normalize_memory_update_data(update_data: dict[str, Any]) -> dict[str, Any] 0, ) + # ── Normalize staleness review removals ── + stale_removals_raw = update_data.get("staleFactsToRemove") + normalized_stale_removals: list[dict[str, str]] = [] + if isinstance(stale_removals_raw, list): + for entry in stale_removals_raw: + if not isinstance(entry, dict): + continue + fact_id = entry.get("id") + if not isinstance(fact_id, str) or not fact_id: + continue + reason = entry.get("reason", "") + normalized_stale_removals.append( + { + "id": fact_id, + "reason": reason if isinstance(reason, str) else "", + } + ) + return { "user": user if isinstance(user, dict) else {}, "history": history if isinstance(history, dict) else {}, "newFacts": normalized_new_facts, "factsToRemove": normalized_facts_to_remove, + "staleFactsToRemove": normalized_stale_removals, } @@ -376,6 +397,73 @@ def _fact_content_key(content: Any) -> str | None: return stripped.casefold() +# ── Staleness review helpers ────────────────────────────────────────────── + + +def _parse_fact_datetime(raw: str) -> datetime | None: + """Parse an ISO-8601 datetime string from a fact's createdAt field. + + Returns ``None`` on any parse failure so callers can safely skip malformed facts. + """ + if not raw: + return None + try: + result = datetime.fromisoformat(raw) + # Naive datetimes (no tzinfo) would cause TypeError when compared + # with the timezone-aware cutoff. Assume UTC for safety. + if result.tzinfo is None: + result = result.replace(tzinfo=UTC) + return result + except (ValueError, TypeError): + return None + + +def _select_stale_candidates( + current_memory: dict[str, Any], + config: Any, +) -> list[dict[str, Any]]: + """Return facts that are older than ``staleness_age_days`` and not protected. + + Protected categories (default: ``correction``) are excluded because they + represent explicit user feedback that should not be auto-pruned by age. + """ + cutoff = datetime.now(UTC) - timedelta(days=config.staleness_age_days) + protected = frozenset(config.staleness_protected_categories) + candidates: list[dict[str, Any]] = [] + for fact in current_memory.get("facts", []): + if not isinstance(fact, dict): + continue + category = fact.get("category", "") + if isinstance(category, str) and category in protected: + continue + created_at = _parse_fact_datetime(fact.get("createdAt", "")) + if created_at is not None and created_at < cutoff: + candidates.append(fact) + return candidates + + +def _build_staleness_section( + stale_candidates: list[dict[str, Any]], + age_days: int, +) -> str: + """Format the staleness review prompt section from candidate facts.""" + if not stale_candidates: + return "" + lines: list[str] = [] + for fact in stale_candidates: + fid = fact.get("id", "?") + cat = str(fact.get("category", "context")).strip() or "context" + conf = fact.get("confidence", 0.0) + created_raw = fact.get("createdAt", "") + created_short = created_raw[:10] if isinstance(created_raw, str) and len(created_raw) >= 10 else created_raw + content = str(fact.get("content", "")) + lines.append(f'- [{fid} | {cat} | {conf:.2f} | {created_short}] "{content}"') + return STALENESS_REVIEW_PROMPT.format( + stale_facts="\n".join(lines), + age_days=age_days, + ) + + class MemoryUpdater: """Updates memory using LLM based on conversation context.""" @@ -443,10 +531,22 @@ def _prepare_update_prompt( correction_detected=correction_detected, reinforcement_detected=reinforcement_detected, ) + + # ── Build staleness review section ── + staleness_section = "" + if config.staleness_review_enabled: + stale_candidates = _select_stale_candidates(current_memory, config) + if len(stale_candidates) >= config.staleness_min_candidates: + staleness_section = _build_staleness_section( + stale_candidates, + config.staleness_age_days, + ) + prompt = MEMORY_UPDATE_PROMPT.format( current_memory=json.dumps(current_memory, indent=2, ensure_ascii=False), conversation=conversation_text, correction_hint=correction_hint, + staleness_review_section=staleness_section, ) return current_memory, prompt @@ -664,11 +764,49 @@ def _apply_updates( "updatedAt": now, } - # Remove facts + # Remove facts (contradiction-based) facts_to_remove = set(update_data.get("factsToRemove", [])) if facts_to_remove: current_memory["facts"] = [f for f in current_memory.get("facts", []) if f.get("id") not in facts_to_remove] + # ── Staleness review removals ── + stale_removals = update_data.get("staleFactsToRemove", []) + if isinstance(stale_removals, list) and stale_removals: + stale_ids_to_remove = {entry["id"] for entry in stale_removals if isinstance(entry, dict) and "id" in entry} + + # Deterministic guardrail: intersect with actual staleness + # candidates so an LLM slip that emits a protected-category or + # non-aged fact id is silently rejected. Runs unconditionally + # so the apply-layer protection is independent of model behavior + # AND of the staleness_review_enabled flag. + candidate_ids = {f["id"] for f in _select_stale_candidates(current_memory, config)} + stale_ids_to_remove &= candidate_ids + + if not stale_ids_to_remove: + # After intersection with candidate set, nothing to remove. + stale_removals = [] + else: + # Safety cap: limit max staleness removals per cycle. + # When the LLM returns more than the cap, keep only the + # lowest-confidence entries up to the limit so the most + # questionable facts are removed first. + max_stale = config.staleness_max_removals_per_cycle + if len(stale_ids_to_remove) > max_stale: + stale_facts = [f for f in current_memory.get("facts", []) if f.get("id") in stale_ids_to_remove] + stale_facts.sort(key=lambda f: f.get("confidence", 0)) + stale_ids_to_remove = {f["id"] for f in stale_facts[:max_stale]} + + current_memory["facts"] = [f for f in current_memory.get("facts", []) if f.get("id") not in stale_ids_to_remove] + + # Log removals for observability + for entry in stale_removals: + if isinstance(entry, dict) and entry.get("id") in stale_ids_to_remove: + logger.info( + "Staleness review removed fact %s: %s", + entry["id"], + entry.get("reason", "no reason provided"), + ) + # Add new facts existing_fact_keys = {fact_key for fact_key in (_fact_content_key(fact.get("content")) for fact in current_memory.get("facts", [])) if fact_key is not None} new_facts = update_data.get("newFacts", []) diff --git a/backend/packages/harness/deerflow/agents/middlewares/clarification_middleware.py b/backend/packages/harness/deerflow/agents/middlewares/clarification_middleware.py index 03fe86c7890..3adf0d9c417 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/clarification_middleware.py +++ b/backend/packages/harness/deerflow/agents/middlewares/clarification_middleware.py @@ -4,7 +4,7 @@ import logging from collections.abc import Callable from hashlib import sha256 -from typing import override +from typing import Any, override from langchain.agents import AgentState from langchain.agents.middleware import AgentMiddleware @@ -44,6 +44,60 @@ def _stable_message_id(self, tool_call_id: str, formatted_message: str) -> str: digest = sha256(formatted_message.encode("utf-8")).hexdigest()[:16] return f"clarification:{digest}" + def _normalize_options(self, raw_options: Any) -> list[str]: + """Normalize tool-provided options into displayable string values.""" + options = raw_options + + # Some models (e.g. Qwen3-Max) serialize array parameters as JSON strings + # instead of native arrays. Deserialize and normalize so `options` + # is always a list for the rendering logic below. + if isinstance(options, str): + try: + options = json.loads(options) + except (json.JSONDecodeError, TypeError): + options = [options] + + if options is None: + return [] + if not isinstance(options, list): + options = [options] + + return [str(option) for option in options] + + def _build_human_input_payload(self, args: dict[str, Any], *, tool_call_id: str, request_id: str) -> dict[str, Any]: + """Build the structured UI payload while keeping ToolMessage.content as fallback.""" + options = self._normalize_options(args.get("options", [])) + clarification_type = str(args.get("clarification_type", "missing_info")) + + payload: dict[str, Any] = { + "version": 1, + "kind": "human_input_request", + "source": "ask_clarification", + "request_id": request_id, + "clarification_type": clarification_type, + "question": str(args.get("question") or ""), + "input_mode": "choice_with_other" if options else "free_text", + } + + if tool_call_id: + payload["tool_call_id"] = tool_call_id + + if "context" in args: + context = args.get("context") + payload["context"] = None if context is None else str(context) + + if options: + payload["options"] = [ + { + "id": f"option-{index}", + "label": option, + "value": option, + } + for index, option in enumerate(options, 1) + ] + + return payload + def _is_chinese(self, text: str) -> bool: """Check if text contains Chinese characters. @@ -67,21 +121,7 @@ def _format_clarification_message(self, args: dict) -> str: question = args.get("question", "") clarification_type = args.get("clarification_type", "missing_info") context = args.get("context") - options = args.get("options", []) - - # Some models (e.g. Qwen3-Max) serialize array parameters as JSON strings - # instead of native arrays. Deserialize and normalize so `options` - # is always a list for the rendering logic below. - if isinstance(options, str): - try: - options = json.loads(options) - except (json.JSONDecodeError, TypeError): - options = [options] - - if options is None: - options = [] - elif not isinstance(options, list): - options = [options] + options = self._normalize_options(args.get("options", [])) # Type-specific icons type_icons = { @@ -174,13 +214,17 @@ def _handle_clarification(self, request: ToolCallRequest) -> Command: # Get the tool call ID tool_call_id = request.tool_call.get("id", "") + request_id = self._stable_message_id(tool_call_id, formatted_message) + human_input_payload = self._build_human_input_payload(args, tool_call_id=tool_call_id, request_id=request_id) + # Create a ToolMessage with the formatted question # This will be added to the message history tool_message = ToolMessage( - id=self._stable_message_id(tool_call_id, formatted_message), + id=request_id, content=formatted_message, tool_call_id=tool_call_id, name="ask_clarification", + artifact={"human_input": human_input_payload}, ) # Return a Command that: diff --git a/backend/packages/harness/deerflow/agents/middlewares/delegation_ledger.py b/backend/packages/harness/deerflow/agents/middlewares/delegation_ledger.py index 8122f66bd7f..20725ccbcdc 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/delegation_ledger.py +++ b/backend/packages/harness/deerflow/agents/middlewares/delegation_ledger.py @@ -63,6 +63,8 @@ def _status_guidance(status: str) -> str: return "timed-out attempt; may retry with a changed plan" if status == "polling_timed_out": return "polling timed-out attempt; may retry with a changed plan" + if status == "max_turns_reached": + return "hit the turn budget with a partial result; reuse the partial result, retry with a tighter scope, or raise the per-agent max_turns" return "prior attempt; inspect status before retrying" diff --git a/backend/packages/harness/deerflow/agents/middlewares/input_sanitization_middleware.py b/backend/packages/harness/deerflow/agents/middlewares/input_sanitization_middleware.py index 480ae065fe3..55a27994974 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/input_sanitization_middleware.py +++ b/backend/packages/harness/deerflow/agents/middlewares/input_sanitization_middleware.py @@ -31,6 +31,7 @@ from langchain_core.messages import HumanMessage from langgraph.errors import GraphBubbleUp +from deerflow.agents.human_input import read_human_input_response from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY, message_content_to_text logger = logging.getLogger(__name__) @@ -90,15 +91,15 @@ def _escape_tag_match(match: re.Match) -> str: def _is_genuine_user_message(message: object) -> bool: """Return True for real user messages, excluding system-injected HumanMessages. - System-injected context is marked via ``hide_from_ui`` — the same convention - used by DynamicContextMiddleware and TodoMiddleware. + ``hide_from_ui`` is also used by hidden UI replies from HumanInputCard, so + only skip hidden HumanMessages that do not carry a valid user response. """ if not isinstance(message, HumanMessage): return False - if message.additional_kwargs.get("hide_from_ui"): - return False if message.name == _SUMMARY_MESSAGE_NAME: return False + if message.additional_kwargs.get("hide_from_ui") and read_human_input_response(message.additional_kwargs) is None: + return False return True diff --git a/backend/packages/harness/deerflow/agents/middlewares/read_before_write_middleware.py b/backend/packages/harness/deerflow/agents/middlewares/read_before_write_middleware.py index ea75acdcd6c..1198d75773f 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/read_before_write_middleware.py +++ b/backend/packages/harness/deerflow/agents/middlewares/read_before_write_middleware.py @@ -39,6 +39,7 @@ from langgraph.prebuilt.tool_node import ToolCallRequest from langgraph.types import Command +from deerflow.agents.middlewares.tool_result_meta import normalize_tool_result from deerflow.sandbox.tools import read_current_file_content logger = logging.getLogger(__name__) @@ -107,7 +108,9 @@ def wrap_tool_call( with self._lock_for(request, path): blocked = self._check_write_gate(request) if blocked is not None: - return blocked + # Stamp deerflow_tool_meta so ToolProgressMiddleware can classify + # the blocked write even though it bypasses ToolErrorHandlingMiddleware. + return normalize_tool_result(blocked) return handler(request) if name in _READ_TOOLS: path = self._requested_path(request) @@ -138,7 +141,7 @@ async def awrap_tool_call( try: blocked = await asyncio.to_thread(self._check_write_gate, request) if blocked is not None: - return blocked + return normalize_tool_result(blocked) return await handler(request) finally: lock.release() diff --git a/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py b/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py index d6adced3ed1..ee7bba16b82 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py +++ b/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py @@ -16,6 +16,10 @@ _tool_call_path, build_skill_entry_metadata_from_read, ) +from deerflow.agents.middlewares.tool_result_meta import ( + normalize_tool_result, + stamp_exception_meta, +) from deerflow.config.app_config import AppConfig from deerflow.config.summarization_config import DEFAULT_SKILL_FILE_READ_TOOL_NAMES from deerflow.constants import DEFAULT_SKILLS_CONTAINER_PATH @@ -79,7 +83,8 @@ def _build_error_message(self, request: ToolCallRequest, exc: Exception) -> Tool # failures raised before task_tool can build its own Command still # carry the same structured metadata. structured_error = f"{exc.__class__.__name__}: {detail}" - return _stamp_task_exception_status(message, tool_name=tool_name, error=structured_error) + message = _stamp_task_exception_status(message, tool_name=tool_name, error=structured_error) + return stamp_exception_meta(message, structured_error) def _stamp_skill_read_metadata( self, @@ -127,7 +132,7 @@ def wrap_tool_call( except Exception as exc: logger.exception("Tool execution failed (sync): name=%s id=%s", request.tool_call.get("name"), request.tool_call.get("id")) return self._build_error_message(request, exc) - return self._maybe_stamp(result, request) + return normalize_tool_result(self._maybe_stamp(result, request)) @override async def awrap_tool_call( @@ -143,7 +148,7 @@ async def awrap_tool_call( except Exception as exc: logger.exception("Tool execution failed (async): name=%s id=%s", request.tool_call.get("name"), request.tool_call.get("id")) return self._build_error_message(request, exc) - return self._maybe_stamp(result, request) + return normalize_tool_result(self._maybe_stamp(result, request)) def _build_runtime_middlewares( @@ -213,14 +218,42 @@ def _build_runtime_middlewares( tail.append(SandboxAuditMiddleware()) + # ReadBeforeWriteMiddleware is the outermost write gate: it blocks writes to files + # the model hasn't read in their current version. It must sit outside ToolProgress + # and ToolErrorHandling so that a blocked write returns immediately without consuming + # a ToolProgress slot. The middleware stamps deerflow_tool_meta on the blocked + # ToolMessage itself so downstream callers receive a well-formed result. if app_config.read_before_write.enabled: from deerflow.agents.middlewares.read_before_write_middleware import ReadBeforeWriteMiddleware tail.append(ReadBeforeWriteMiddleware()) + # ToolProgressMiddleware must be outer (lower index) so its wrap_tool_call handler + # chain includes ToolErrorHandlingMiddleware (inner), which stamps deerflow_tool_meta + # on every result before ToolProgressMiddleware reads it in _update_state_from_result. + # Framework rule: first in list = outermost (types.py: "compose with first in list as outermost layer"). + tool_progress_config = app_config.tool_progress + _ToolProgressMiddleware = None + if tool_progress_config.enabled: + from deerflow.agents.middlewares.tool_progress_middleware import ToolProgressMiddleware as _ToolProgressMiddleware + + tail.append(_ToolProgressMiddleware.from_config(tool_progress_config)) + tail.append(ToolErrorHandlingMiddleware(app_config=app_config)) - return [*outer_wrappers, *thread_hooks, *tail] + middlewares = [*outer_wrappers, *thread_hooks, *tail] + + # Guard: ToolProgressMiddleware (outer) must appear before ToolErrorHandlingMiddleware (inner) + # so that its wrap_tool_call chain encloses the stamping step. Fail loudly at build time + # rather than silently no-oping at runtime if a future insertion reverses the order. + # Uses isinstance (not type().__name__) so subclasses and renames are covered. + if _ToolProgressMiddleware is not None: + _progress_idx = next((i for i, m in enumerate(middlewares) if isinstance(m, _ToolProgressMiddleware)), None) + _error_idx = next((i for i, m in enumerate(middlewares) if isinstance(m, ToolErrorHandlingMiddleware)), None) + if _progress_idx is not None and _error_idx is not None and _progress_idx > _error_idx: + raise RuntimeError(f"ToolProgressMiddleware must be outer (index {_progress_idx}) of ToolErrorHandlingMiddleware (index {_error_idx}) — check middleware append order") + + return middlewares def build_lead_runtime_middlewares(*, app_config: AppConfig, lazy_init: bool = True) -> list[AgentMiddleware]: diff --git a/backend/packages/harness/deerflow/agents/middlewares/tool_progress_middleware.py b/backend/packages/harness/deerflow/agents/middlewares/tool_progress_middleware.py new file mode 100644 index 00000000000..ce63e1a73e2 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/middlewares/tool_progress_middleware.py @@ -0,0 +1,578 @@ +"""Middleware for task-level tool call progress tracking with a state machine. + +Implements RFC #3177: structured tool result signals drive a per-(thread, tool) +state machine that detects stagnation and repetition, injects hints early +(WARNED), and hard-blocks the tool when it has stopped producing value (BLOCKED). + +Architecture: + ToolProgressMiddleware (outer) + └── handler → ToolErrorHandlingMiddleware (inner) → actual tool + ↓ + ToolProgressMiddleware reads deerflow_tool_meta from the normalized result + +State machine transitions per (thread_id, tool_name): + ACTIVE → WARNED (at stagnation_threshold problems) + Any problem-free call resets consecutive_problems=0 and reverts to ACTIVE. + + Whether WARNED can escalate to BLOCKED depends on recoverable_by_model: + - recoverable_by_model=True (no_results, not_found, permission, Jaccard-duplicate success): + WARNED is terminal. The model received a hint and is expected to change strategy; + blocking would prevent a legitimate retry with different parameters. + - recoverable_by_model=False, action≠stop (transient, rate_limited): + WARNED → BLOCKED after warn_escalation_count more problems. The model cannot fix + these by retrying the same tool, so hard-blocking conserves API calls. + - recoverable_by_model=False, action=stop (auth, config, internal): + Immediately BLOCKED on the first occurrence — no retry can help. + +Division of labor with LoopDetectionMiddleware (middleware position 23): + ToolProgressMiddleware (position 10) is a result-quality guard — it fires + after a tool executes, inspects what came back, and blocks *specific tools* + that have stopped producing new information. + + LoopDetectionMiddleware is a call-pattern guard — it fires after the model + responds (before tools execute), inspects the tool_calls signature in the + AIMessage, and forces the *whole turn* to stop when the model keeps issuing + the same calls regardless of results. + + They are complementary, not competing: + - ToolProgressMiddleware is fine-grained (per-tool BLOCK, other tools normal). + - LoopDetectionMiddleware is coarse-grained (strips all tool_calls, ends turn). + - Both can inject HumanMessage hints in the same model call without conflict; + the model sees both sets of hints and can reason about them. + - If LoopDetectionMiddleware hard-stops (strips tool_calls), no wrap_tool_call + is issued so ToolProgressMiddleware never fires — there is no double-stop. + - If ToolProgressMiddleware BLOCKs a tool (returns an error ToolMessage), + the model still makes a tool call that LoopDetectionMiddleware tracks; both + continue to operate on their own independent state. +""" + +from __future__ import annotations + +import logging +import re +import threading +from collections import OrderedDict, defaultdict +from collections.abc import Awaitable, Callable, Sequence +from dataclasses import dataclass, field, replace +from typing import TYPE_CHECKING, Literal, override + +from langchain.agents import AgentState +from langchain.agents.middleware import AgentMiddleware +from langchain.agents.middleware.types import ModelCallResult, ModelRequest, ModelResponse +from langchain_core.messages import HumanMessage, ToolMessage +from langgraph.prebuilt.tool_node import ToolCallRequest +from langgraph.runtime import Runtime +from langgraph.types import Command + +from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY, ToolResultMeta + +if TYPE_CHECKING: + from deerflow.config.tool_progress_config import ToolProgressConfig + +logger = logging.getLogger(__name__) + +_MAX_PENDING_PER_RUN = 3 +# Jaccard word-set computation is capped to avoid O(n) regex work on very large tool results. +_MAX_CONTENT_FOR_WORDSET = 8192 + + +# --------------------------------------------------------------------------- +# State data structures + + +@dataclass(slots=True) +class ToolPhaseState: + """Per (thread_id, tool_name) tracking state.""" + + phase: Literal["active", "warned", "blocked"] = "active" + consecutive_problems: int = 0 + block_reason: str | None = None + # Immutable tuple so that dataclasses.replace() calls that omit recent_word_sets + # (problem paths) cannot accidentally share a mutable list between the old and new + # state objects and cause silent cross-state corruption via .append(). + recent_word_sets: tuple[frozenset[str], ...] = field(default_factory=tuple) + + +# --------------------------------------------------------------------------- +# Content helpers + + +def word_set(content: str) -> frozenset[str]: + """Extract lowercase words of length >= 3 for Jaccard similarity. + + Content is capped at _MAX_CONTENT_FOR_WORDSET chars to bound memory and CPU cost on + large tool results (e.g. web pages). Tail content beyond the cap is omitted from the + set, which is acceptable because duplicate-detection is a heuristic, not a guarantee. + """ + return frozenset(re.findall(r"\b\w{3,}\b", content[:_MAX_CONTENT_FOR_WORDSET].lower())) + + +def is_near_duplicate( + current: frozenset[str], + recent: Sequence[frozenset[str]], + threshold: float, + min_words: int, +) -> bool: + """Return True if current is similar to any of the last 3 recent word sets.""" + if len(current) < min_words: + return False + for prev in recent[-3:]: + if len(prev) < min_words: + continue + union = len(current | prev) + if union == 0: + continue + if len(current & prev) / union >= threshold: + return True + return False + + +def _message_content_str(msg: ToolMessage) -> str: + return msg.content if isinstance(msg.content, str) else "" + + +def _parse_tool_meta(meta_dict: object) -> ToolResultMeta | None: + """Safely deserialize a ToolResultMeta from a raw dict; returns None on schema mismatch.""" + if not isinstance(meta_dict, dict): + return None + try: + return ToolResultMeta(**meta_dict) + except TypeError: + logger.warning("Unexpected tool meta schema, skipping progress tracking: %s", meta_dict) + return None + + +# --------------------------------------------------------------------------- +# Hint / block reason formatting + + +def _format_hint(meta: ToolResultMeta) -> str: + action_map = { + "rewrite_query": "Try rephrasing your search query with different keywords or approach.", + "try_alternative": "Consider using a different tool or strategy.", + "summarize": "Consider summarizing your current findings and moving forward.", + "stop": "Do not retry this operation — it is not recoverable.", + # Near-duplicate success results: recommended_next_action is "continue" by default, + # but the model should still change strategy to avoid re-fetching the same content. + "continue": "Try rephrasing your query or using a different search term.", + } + base = { + "no_results": "[PROGRESS HINT] Your search returned no results.", + "not_found": "[PROGRESS HINT] The resource was not found repeatedly.", + "rate_limited": "[PROGRESS HINT] The tool is being rate-limited.", + "transient": "[PROGRESS HINT] The tool encountered repeated transient failures.", + "partial_success": "[PROGRESS HINT] The tool has returned incomplete results multiple times.", + # Jaccard near-duplicate success: the tool is returning the same content repeatedly. + "success": "[PROGRESS HINT] The tool is returning duplicate results.", + }.get( + meta.error_type or meta.status, + "[PROGRESS HINT] The tool is not producing new information.", + ) + suffix = action_map.get(meta.recommended_next_action, "") + return f"{base} {suffix}".strip() + + +def _block_reason(meta: ToolResultMeta) -> str: + return { + "no_results": "Repeated no-results — rewrite your query or try a different tool.", + "not_found": "Repeated not-found — rewrite your query or try a different resource.", + "rate_limited": "Repeated rate-limiting — summarize current findings and proceed.", + "transient": "Repeated transient failures — try a different approach.", + "auth": "Authentication failure — this tool cannot be used.", + "config": "Tool is not configured — this tool cannot be used.", + "internal": "Repeated internal errors — this tool is unavailable.", + }.get( + meta.error_type or "", + "Tool has not produced new information after multiple attempts — summarize and move on.", + ) + + +# --------------------------------------------------------------------------- +# Middleware + + +class ToolProgressMiddleware(AgentMiddleware[AgentState]): + """State-machine-based tool stagnation guard (RFC #3177).""" + + def __init__( + self, + *, + stagnation_threshold: int = 3, + warn_escalation_count: int = 2, + inject_assessment: bool = True, + jaccard_threshold: float = 0.8, + min_words: int = 10, + exempt_tools: set[str] | None = None, + max_tracked_threads: int = 100, + ) -> None: + self._stagnation_threshold = stagnation_threshold + self._warn_escalation = warn_escalation_count + self._inject_assessment = inject_assessment + self._jaccard_threshold = jaccard_threshold + self._min_words = min_words + self._exempt_tools: set[str] = exempt_tools if exempt_tools is not None else {"ask_clarification", "write_todos", "present_files", "task"} + self._max_tracked_threads = max_tracked_threads + + # threading.Lock (not asyncio.Lock): critical sections are short in-memory dict + # ops with no I/O, so event-loop stall risk is negligible. asyncio.Lock would + # not protect the sync wrap_tool_call path used by subagent executor thread + # pools — two separate locks would be required instead. This matches the + # existing LoopDetectionMiddleware pattern; see module docstring for details. + self._lock = threading.Lock() + # LRU-evicting store: thread_id → {tool_name → ToolPhaseState} + self._phase_states: OrderedDict[str, dict[str, ToolPhaseState]] = OrderedDict() + # Pending hint queue: (thread_id, run_id) → [hint texts] + self._pending: dict[tuple[str, str], list[str]] = defaultdict(list) + + @classmethod + def from_config(cls, config: ToolProgressConfig) -> ToolProgressMiddleware: + return cls( + stagnation_threshold=config.stagnation_threshold, + warn_escalation_count=config.warn_escalation_count, + inject_assessment=config.inject_assessment, + jaccard_threshold=config.jaccard_similarity_threshold, + min_words=config.min_word_count_for_similarity, + exempt_tools=set(config.exempt_tools), + max_tracked_threads=config.max_tracked_threads, + ) + + # ------------------------------------------------------------------ + # Runtime helpers + + @staticmethod + def _thread_id(runtime: Runtime) -> str: + tid = runtime.context.get("thread_id") if runtime.context else None + return str(tid) if tid else "default" + + @staticmethod + def _run_id(runtime: Runtime) -> str: + rid = runtime.context.get("run_id") if runtime.context else None + return str(rid) if rid else "default" + + def _pending_key(self, runtime: Runtime) -> tuple[str, str]: + return self._thread_id(runtime), self._run_id(runtime) + + # ------------------------------------------------------------------ + # State store (caller holds lock) + + def _get_state(self, thread_id: str, tool_name: str) -> ToolPhaseState: + if thread_id not in self._phase_states: + self._phase_states[thread_id] = {} + while len(self._phase_states) > self._max_tracked_threads: + evicted_thread, _ = self._phase_states.popitem(last=False) + # Evict pending hints for the evicted thread to prevent unbounded growth. + for key in [k for k in self._pending if k[0] == evicted_thread]: + del self._pending[key] + self._phase_states.move_to_end(thread_id) + return self._phase_states[thread_id].get(tool_name, ToolPhaseState()) + + def _set_state(self, thread_id: str, tool_name: str, state: ToolPhaseState) -> None: + self._phase_states[thread_id][tool_name] = state + + def _get_block_reason(self, runtime: Runtime, tool_name: str) -> str | None: + thread_id = self._thread_id(runtime) + with self._lock: + thread_tools = self._phase_states.get(thread_id) + if thread_tools is None: + return None + # Read-only check: do NOT call move_to_end here. Bumping recency on the read path + # would keep blocked threads permanently warm in the LRU, preventing healthy active + # threads from occupying those slots. Recency is updated only on _get_state writes. + tool_state = thread_tools.get(tool_name) + return tool_state.block_reason if tool_state is not None and tool_state.phase == "blocked" else None + + def _make_blocked_message(self, request: ToolCallRequest, tool_name: str, block_reason: str) -> ToolMessage: + return ToolMessage( + content=f"[TOOL_BLOCKED] {block_reason}", + tool_call_id=str(request.tool_call.get("id", "")), + name=tool_name, + status="error", + additional_kwargs={ + TOOL_META_KEY: { + "status": "error", + "error_type": "blocked_by_progress_guard", + "recoverable_by_model": True, + "recommended_next_action": "summarize", + "source": "progress_middleware", + } + }, + ) + + def _update_state_from_result( + self, + result: ToolMessage | Command, + tool_name: str, + runtime: Runtime, + ) -> ToolMessage | Command: + """Update the state machine from a tool result; queue hints if warranted.""" + if not isinstance(result, ToolMessage): + return result + meta = _parse_tool_meta((result.additional_kwargs or {}).get(TOOL_META_KEY)) + if meta is None: + if tool_name not in self._exempt_tools: + logger.warning( + "tool_progress: deerflow_tool_meta missing for non-exempt tool %s — verify ToolProgressMiddleware is outer of ToolErrorHandlingMiddleware", + tool_name, + ) + return result + content = _message_content_str(result) + thread_id = self._thread_id(runtime) + with self._lock: + state = self._get_state(thread_id, tool_name) + new_state, hint = self._assess_and_transition(state, meta, content) + self._set_state(thread_id, tool_name, new_state) + if new_state.phase != state.phase: + if new_state.phase == "blocked": + logger.warning( + "tool_progress: %s/%s -> BLOCKED: %s", + thread_id, + tool_name, + new_state.block_reason, + ) + elif new_state.phase == "warned": + logger.info( + "tool_progress: %s/%s -> WARNED (consecutive_problems=%d)", + thread_id, + tool_name, + new_state.consecutive_problems, + ) + elif new_state.phase == "active": + logger.info( + "tool_progress: %s/%s -> ACTIVE (reset after good result)", + thread_id, + tool_name, + ) + if hint and self._inject_assessment: + self._queue_assessment(runtime, hint) + return result + + # ------------------------------------------------------------------ + # State machine + + def _assess_and_transition( + self, + state: ToolPhaseState, + meta: ToolResultMeta, + content: str, + ) -> tuple[ToolPhaseState, str | None]: + """Return (new_state, hint_text_or_None). + + The outer wrap_tool_call gate intercepts already-blocked states before + the handler is called, so this function is normally reached only for + active/warned states. If a blocked state arrives (e.g., concurrent + transition), the function returns it unchanged — no counter inflation, + no phase regression. + """ + # Guard: blocked is a terminal state; nothing should change it here. + # (In normal flow this branch is unreachable because wrap_tool_call + # intercepts blocked tools before calling the handler. The check exists + # to make concurrent-race semantics well-defined and prevent a + # recoverable-error result from silently demoting the phase back to warned.) + if state.phase == "blocked": + return state, None + + # Count this call as a problem before branching so all exit paths leave + # consecutive_problems in a consistent state (never 0 when the tool has failed). + new_count = state.consecutive_problems + 1 + + # Immediately block on unrecoverable stop signals (auth, config, internal). + if not meta.recoverable_by_model and meta.recommended_next_action == "stop": + return replace( + state, + phase="blocked", + consecutive_problems=new_count, + block_reason=_block_reason(meta), + ), None + + # Compute word_set only for success results: error/partial_success are problems by + # definition and never reach the Jaccard check, so the O(n) regex is wasted on them. + ws = word_set(content) if meta.status == "success" else frozenset() + is_problem = meta.status in ("error", "partial_success") or (meta.status == "success" and is_near_duplicate(ws, state.recent_word_sets, self._jaccard_threshold, self._min_words)) + + if not is_problem: + # Good result: reset consecutive count, return to active. + new_recent = (*state.recent_word_sets, ws)[-3:] + return replace(state, consecutive_problems=0, phase="active", recent_word_sets=new_recent), None + + hint: str | None = None + + if new_count >= self._stagnation_threshold + self._warn_escalation: + if meta.recoverable_by_model: + # Model can fix this by changing strategy — keep warned, re-inject hint. + # BLOCKED would prevent a legitimate retry with different parameters. + hint = _format_hint(meta) + new_state = replace(state, consecutive_problems=new_count, phase="warned") + else: + # Model cannot fix this by retrying — block the tool. + reason = _block_reason(meta) + new_state = replace(state, consecutive_problems=new_count, phase="blocked", block_reason=reason) + elif new_count >= self._stagnation_threshold: + hint = _format_hint(meta) + new_state = replace(state, consecutive_problems=new_count, phase="warned") + else: + new_state = replace(state, consecutive_problems=new_count) + + return new_state, hint + + # ------------------------------------------------------------------ + # Pending queue helpers + + def _queue_assessment(self, runtime: Runtime, text: str) -> None: + key = self._pending_key(runtime) + thread_id = key[0] + with self._lock: + # Guard against creating a phantom _pending entry for a thread that was just + # evicted from _phase_states by the LRU. Such entries can never be cleaned up + # by the eviction loop (which only walks _phase_states) and accumulate silently. + if thread_id not in self._phase_states: + return + queue = self._pending[key] + if len(queue) < _MAX_PENDING_PER_RUN: + queue.append(text) + + def _drain_pending(self, runtime: Runtime) -> list[str]: + key = self._pending_key(runtime) + with self._lock: + return self._pending.pop(key, []) + + def _clear_stale_pending(self, runtime: Runtime) -> None: + thread_id, current_run = self._pending_key(runtime) + with self._lock: + for key in list(self._pending): + if key[0] == thread_id and key[1] != current_run: + del self._pending[key] + + def _reset_run_states(self, runtime: Runtime) -> None: + """Reset all per-run tool state for the thread at the start of a new agent run. + + Every tool's consecutive_problems counter and recent_word_sets Jaccard window are + cleared unconditionally so state from a previous run never bleeds into the next: + - BLOCKED/WARNED tools are reset to ACTIVE (they re-block immediately if the root + cause persists, and the model has no memory of the prior-run hint). + - ACTIVE tools with non-zero consecutive_problems or non-empty recent_word_sets from + the previous run are also cleared so a single first-call problem in the new run + cannot falsely trip WARNED against stale context from a run the model no longer sees. + + **Cross-run scoping vs LoopDetectionMiddleware**: this per-run reset is an intentional + policy choice, not an oversight. Errors like ``rate_limited`` and ``transient`` are + time-bound: their root cause may resolve between user turns, so carrying a stale + counter forward risks a false-positive BLOCKED on calls that would now succeed. + LoopDetectionMiddleware takes the opposite stance — it retains ``_history`` across + runs (only clearing other-run *pending* warnings at ``before_agent``), because + call-pattern loops are time-invariant: a model that keeps issuing the same tool_calls + regardless of results does so regardless of when the run started. The two middlewares + therefore guard different failure modes (result quality vs. call pattern) and their + cross-run scoping policies intentionally differ as a consequence. + """ + thread_id = self._thread_id(runtime) + with self._lock: + thread_tools = self._phase_states.get(thread_id) + if thread_tools is None: + return + for tool_name, tool_state in list(thread_tools.items()): + thread_tools[tool_name] = replace( + tool_state, + phase="active", + consecutive_problems=0, + block_reason=None, + recent_word_sets=(), + ) + + # ------------------------------------------------------------------ + # wrap_tool_call + + @override + def wrap_tool_call( + self, + request: ToolCallRequest, + handler: Callable[[ToolCallRequest], ToolMessage | Command], + ) -> ToolMessage | Command: + tool_name = str(request.tool_call.get("name", "")) + if not tool_name or tool_name in self._exempt_tools: + return handler(request) + runtime = getattr(request, "runtime", None) + if runtime is None: + return handler(request) + block_reason = self._get_block_reason(runtime, tool_name) + if block_reason: + logger.info( + "tool_progress: %s/%s call intercepted (blocked): %s", + self._thread_id(runtime), + tool_name, + block_reason, + ) + return self._make_blocked_message(request, tool_name, block_reason) + return self._update_state_from_result(handler(request), tool_name, runtime) + + @override + async def awrap_tool_call( + self, + request: ToolCallRequest, + handler: Callable[[ToolCallRequest], Awaitable[ToolMessage | Command]], + ) -> ToolMessage | Command: + tool_name = str(request.tool_call.get("name", "")) + if not tool_name or tool_name in self._exempt_tools: + return await handler(request) + runtime = getattr(request, "runtime", None) + if runtime is None: + return await handler(request) + block_reason = self._get_block_reason(runtime, tool_name) + if block_reason: + logger.info( + "tool_progress: %s/%s call intercepted (blocked): %s", + self._thread_id(runtime), + tool_name, + block_reason, + ) + return self._make_blocked_message(request, tool_name, block_reason) + return self._update_state_from_result(await handler(request), tool_name, runtime) + + # ------------------------------------------------------------------ + # wrap_model_call: drain pending hints and inject before model sees messages + + def _augment_request(self, request: ModelRequest) -> ModelRequest: + hints = self._drain_pending(request.runtime) + if not hints: + return request + deduped = list(dict.fromkeys(hints)) + logger.debug( + "tool_progress: injecting %d hint(s) for %s/%s", + len(deduped), + *self._pending_key(request.runtime), + ) + new_messages = [ + *request.messages, + HumanMessage(content="\n\n".join(deduped), name="progress_hint"), + ] + return request.override(messages=new_messages) + + @override + def wrap_model_call( + self, + request: ModelRequest, + handler: Callable[[ModelRequest], ModelResponse], + ) -> ModelCallResult: + return handler(self._augment_request(request)) + + @override + async def awrap_model_call( + self, + request: ModelRequest, + handler: Callable[[ModelRequest], Awaitable[ModelResponse]], + ) -> ModelCallResult: + return await handler(self._augment_request(request)) + + # ------------------------------------------------------------------ + # before_agent: clean up stale pending hints from previous runs + + @override + def before_agent(self, state: AgentState, runtime: Runtime) -> dict | None: + self._clear_stale_pending(runtime) + self._reset_run_states(runtime) + return None + + @override + async def abefore_agent(self, state: AgentState, runtime: Runtime) -> dict | None: + self._clear_stale_pending(runtime) + self._reset_run_states(runtime) + return None diff --git a/backend/packages/harness/deerflow/agents/middlewares/tool_result_meta.py b/backend/packages/harness/deerflow/agents/middlewares/tool_result_meta.py new file mode 100644 index 00000000000..dd31cb5e4f6 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/middlewares/tool_result_meta.py @@ -0,0 +1,212 @@ +"""Unified tool result semantics for structured signal production. + +Every tool result that passes through ToolErrorHandlingMiddleware gets a +``deerflow_tool_meta`` entry in additional_kwargs. Downstream consumers +(ToolProgressMiddleware, etc.) read this key instead of parsing text. +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass +from typing import Literal + +from langchain_core.messages import ToolMessage +from langgraph.types import Command + +TOOL_META_KEY = "deerflow_tool_meta" + +_ERROR_PREFIX = "Error:" +_PARTIAL_MARKERS = ( + "partial results", + "limited results", + "truncated", + "results may be incomplete", + # Tools that return status="success" with a no-results body (instead of status="error") + # must still be caught by stagnation detection so the model is prompted to try a different query. + "no results found", + "no content found", + "no images found", +) + + +@dataclass(frozen=True, slots=True) +class ToolResultMeta: + status: Literal["success", "error", "partial_success"] + error_type: str | None + recoverable_by_model: bool + recommended_next_action: Literal["continue", "rewrite_query", "try_alternative", "summarize", "stop"] + source: Literal["exception", "tool_return", "content_analysis", "progress_middleware"] + + +_ERROR_RULES: list[tuple[list[str], dict[str, object]]] = [ + ( + ["401", "403", "unauthorized", "authentication", "invalid api key"], + {"error_type": "auth", "recoverable_by_model": False, "recommended_next_action": "stop"}, + ), + ( + ["rate limit", "rate limited", "rate_limit"], + {"error_type": "rate_limited", "recoverable_by_model": False, "recommended_next_action": "summarize"}, + ), + ( + ["timeout", "timed out", "connection", "network error", "temporarily unavailable"], + {"error_type": "transient", "recoverable_by_model": False, "recommended_next_action": "try_alternative"}, + ), + ( + ["not configured", "not installed", "missing required", "disabled", "no api key"], + {"error_type": "config", "recoverable_by_model": False, "recommended_next_action": "stop"}, + ), + ( + ["permission denied", "access denied", "path traversal", "forbidden"], + {"error_type": "permission", "recoverable_by_model": True, "recommended_next_action": "try_alternative"}, + ), + ( + ["no results found", "no content found", "no images found", "no results"], + {"error_type": "no_results", "recoverable_by_model": True, "recommended_next_action": "rewrite_query"}, + ), + ( + ["not found", "no such file", "does not exist", "404"], + {"error_type": "not_found", "recoverable_by_model": True, "recommended_next_action": "rewrite_query"}, + ), + ( + ["unexpected error", "internal error", "500"], + {"error_type": "internal", "recoverable_by_model": False, "recommended_next_action": "stop"}, + ), +] + +_UNKNOWN_ERROR: dict[str, object] = { + "error_type": "unknown", + "recoverable_by_model": True, + "recommended_next_action": "try_alternative", +} + +# Pre-compiled at module load from _ERROR_RULES. Anchoring bare numeric codes (401, 403, 404, +# 500) to word boundaries prevents substring hits on unrelated numbers like "took 500ms". +# Computed here (after _ERROR_RULES) so the set is authoritative and thread-safe — no lazy +# writes on the hot classification path. +_NUMERIC_KW_RE: dict[str, re.Pattern[str]] = {kw: re.compile(rf"\b{kw}\b") for rule_keywords, _ in _ERROR_RULES for kw in rule_keywords if kw.isdigit()} + +_SEMANTIC_ZERO_ERROR_STRINGS: frozenset[str] = frozenset({"none", "null", "false", "no", "ok", "success", "n/a", ""}) + + +def _extract_json_error_text(content: str) -> str | None: + """Return the error string from a JSON-wrapped error like {"error": "...", "query": "..."}. + + Returns None when the ``error`` field is falsy (JSON null / 0 / false / empty + string) or is a sentinel string that conventionally means "no error" (e.g. + ``"none"``, ``"null"``, ``"false"``). This prevents tools that return + ``{"error": "none", "results": [...]}`` on success from being misclassified + as errors. + """ + try: + data = json.loads(content) + except (json.JSONDecodeError, ValueError): + return None + error = data.get("error") if isinstance(data, dict) else None + if not error: + return None + if isinstance(error, str) and error.lower().strip() in _SEMANTIC_ZERO_ERROR_STRINGS: + return None + # Serialize non-string values to JSON so _classify_error_text sees a predictable + # format (e.g. {"error": 404} → "404", {"error": [...]} → "[...]") instead of + # Python repr which can spuriously match keyword rules like "missing required". + return error if isinstance(error, str) else json.dumps(error) + + +def _match_keyword(kw: str, lower: str) -> bool: + """Match a keyword against lowercased text, using word boundaries for numeric codes.""" + if kw.isdigit(): + return bool(_NUMERIC_KW_RE[kw].search(lower)) + return kw in lower + + +def _classify_error_text(text: str) -> dict[str, object]: + lower = text.lower() + for keywords, attrs in _ERROR_RULES: + if any(_match_keyword(kw, lower) for kw in keywords): + return {**attrs} + return {**_UNKNOWN_ERROR} + + +def _make_meta(*, status: str, source: str, error_type: str | None = None, recoverable_by_model: bool = True, recommended_next_action: str = "continue") -> dict[str, object]: + return { + "status": status, + "error_type": error_type, + "recoverable_by_model": recoverable_by_model, + "recommended_next_action": recommended_next_action, + "source": source, + } + + +def stamp_exception_meta(msg: ToolMessage, exc_info: str) -> ToolMessage: + """Stamp deerflow_tool_meta with source='exception' onto an exception-derived ToolMessage. + + Unlike normalize_tool_message (which preserves existing stamps), this function always + overwrites any pre-existing TOOL_META_KEY entry. Exception-derived classification is + more authoritative than a tool's own return-time stamp. + """ + attrs = _classify_error_text(exc_info) + updated_kwargs = dict(msg.additional_kwargs or {}) + updated_kwargs[TOOL_META_KEY] = _make_meta(status="error", source="exception", **attrs) + msg.additional_kwargs = updated_kwargs + return msg + + +def normalize_tool_message(msg: ToolMessage) -> ToolMessage: + """Attach deerflow_tool_meta to a ToolMessage if not already present.""" + existing = (msg.additional_kwargs or {}).get(TOOL_META_KEY) + if existing is not None: + return msg + + content = msg.content if isinstance(msg.content, str) else "" + # Pre-compute once; reused by the partial-success marker check below to avoid calling + # content.lower() once per _PARTIAL_MARKERS entry inside the generator. + content_lower = content.lower() + + # Non-standard error: tool returned status="error" without the "Error:" prefix convention. + # (Actual exceptions from ToolErrorHandlingMiddleware are pre-stamped by stamp_exception_meta + # and exit early above — they never reach this branch.) + # Try JSON extraction first so classification uses only the "error" field value, not + # keywords that appear incidentally in other JSON fields (e.g. "query"). + if msg.status == "error" and not content.startswith(_ERROR_PREFIX): + json_error = _extract_json_error_text(content) + if json_error is not None: + attrs = _classify_error_text(json_error) + else: + # Determine whether content is a JSON object that simply has no 'error' key. + # If so, do NOT classify from the raw JSON string — incidental field values + # (e.g. {"user_id": 401}) would spuriously match keyword rules and hard-block + # the tool. Classify raw text only when the content is not valid JSON. + try: + is_json_dict = isinstance(json.loads(content), dict) + except (json.JSONDecodeError, ValueError): + is_json_dict = False + attrs = {**_UNKNOWN_ERROR} if is_json_dict else _classify_error_text(content) + meta = _make_meta(status="error", source="tool_return", **attrs) + elif content.startswith(_ERROR_PREFIX): + attrs = _classify_error_text(content[len(_ERROR_PREFIX) :]) + meta = _make_meta(status="error", source="tool_return", **attrs) + elif (json_error := _extract_json_error_text(content)) is not None: + attrs = _classify_error_text(json_error) + meta = _make_meta(status="error", source="tool_return", **attrs) + elif any(m in content_lower for m in _PARTIAL_MARKERS): + meta = _make_meta( + status="partial_success", + source="content_analysis", + recommended_next_action="rewrite_query", + ) + else: + meta = _make_meta(status="success", source="content_analysis") + + updated_kwargs = dict(msg.additional_kwargs or {}) + updated_kwargs[TOOL_META_KEY] = meta + msg.additional_kwargs = updated_kwargs + return msg + + +def normalize_tool_result(result: ToolMessage | Command) -> ToolMessage | Command: + """Normalize a tool result, handling Command wrappers transparently.""" + if isinstance(result, ToolMessage): + return normalize_tool_message(result) + return result diff --git a/backend/packages/harness/deerflow/client.py b/backend/packages/harness/deerflow/client.py index 70d7041738f..7fd13a67e2b 100644 --- a/backend/packages/harness/deerflow/client.py +++ b/backend/packages/harness/deerflow/client.py @@ -1319,6 +1319,11 @@ def get_memory_config(self) -> dict: "token_counting": config.token_counting, "guaranteed_categories": config.guaranteed_categories, "guaranteed_token_budget": config.guaranteed_token_budget, + "staleness_review_enabled": config.staleness_review_enabled, + "staleness_age_days": config.staleness_age_days, + "staleness_min_candidates": config.staleness_min_candidates, + "staleness_max_removals_per_cycle": config.staleness_max_removals_per_cycle, + "staleness_protected_categories": config.staleness_protected_categories, } def get_memory_status(self) -> dict: diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py b/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py index b0c7c635b2a..8874cd47e31 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py @@ -27,6 +27,14 @@ fcntl = None # type: ignore[assignment] import msvcrt +from deerflow.community.warm_pool_lifecycle import ( + DEFAULT_IDLE_TIMEOUT, + DEFAULT_REPLICAS, + WarmPoolLifecycleMixin, +) +from deerflow.community.warm_pool_lifecycle import ( + IDLE_CHECK_INTERVAL as _SHARED_IDLE_CHECK_INTERVAL, +) from deerflow.config import get_app_config from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths from deerflow.runtime.user_context import get_effective_user_id @@ -45,9 +53,7 @@ DEFAULT_IMAGE = "enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest" DEFAULT_PORT = 8080 DEFAULT_CONTAINER_PREFIX = "deer-flow-sandbox" -DEFAULT_IDLE_TIMEOUT = 600 # 10 minutes in seconds -DEFAULT_REPLICAS = 3 # Maximum concurrent sandbox containers -IDLE_CHECK_INTERVAL = 60 # Check every 60 seconds +IDLE_CHECK_INTERVAL = _SHARED_IDLE_CHECK_INTERVAL THREAD_LOCK_EXECUTOR_WORKERS = min(32, (os.cpu_count() or 1) + 4) _THREAD_LOCK_EXECUTOR = ThreadPoolExecutor(max_workers=THREAD_LOCK_EXECUTOR_WORKERS, thread_name_prefix="sandbox-lock-wait") atexit.register(_THREAD_LOCK_EXECUTOR.shutdown, wait=False, cancel_futures=True) @@ -105,7 +111,7 @@ def _release_cancelled_lock_acquire(lock: threading.Lock, task: asyncio.Future[b lock.release() -class AioSandboxProvider(SandboxProvider): +class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): """Sandbox provider that manages containers running the AIO sandbox. Architecture: @@ -352,28 +358,13 @@ def _get_skills_mount() -> tuple[str, str, bool] | None: # ── Idle timeout management ────────────────────────────────────────── - def _start_idle_checker(self) -> None: - """Start the background thread that checks for idle sandboxes.""" - self._idle_checker_thread = threading.Thread( - target=self._idle_checker_loop, - name="sandbox-idle-checker", - daemon=True, - ) - self._idle_checker_thread.start() - logger.info(f"Started idle checker thread (timeout: {self._config.get('idle_timeout', DEFAULT_IDLE_TIMEOUT)}s)") - - def _idle_checker_loop(self) -> None: - idle_timeout = self._config.get("idle_timeout", DEFAULT_IDLE_TIMEOUT) - while not self._idle_checker_stop.wait(timeout=IDLE_CHECK_INTERVAL): - try: - self._cleanup_idle_sandboxes(idle_timeout) - except Exception as e: - logger.error(f"Error in idle checker loop: {e}") + def _cleanup_idle_resources(self, idle_timeout: float) -> None: + """Clean AIO resources idle longer than ``idle_timeout`` seconds.""" + self._cleanup_idle_sandboxes(idle_timeout) def _cleanup_idle_sandboxes(self, idle_timeout: float) -> None: current_time = time.time() active_to_destroy = [] - warm_to_destroy: list[tuple[str, SandboxInfo]] = [] with self._lock: # Active sandboxes: tracked via _last_activity @@ -383,14 +374,6 @@ def _cleanup_idle_sandboxes(self, idle_timeout: float) -> None: active_to_destroy.append(sandbox_id) logger.info(f"Sandbox {sandbox_id} idle for {idle_duration:.1f}s, marking for destroy") - # Warm pool: tracked via release_timestamp stored in _warm_pool - for sandbox_id, (info, release_ts) in list(self._warm_pool.items()): - warm_duration = current_time - release_ts - if warm_duration > idle_timeout: - warm_to_destroy.append((sandbox_id, info)) - del self._warm_pool[sandbox_id] - logger.info(f"Warm-pool sandbox {sandbox_id} idle for {warm_duration:.1f}s, marking for destroy") - # Destroy active sandboxes (re-verify still idle before acting) for sandbox_id in active_to_destroy: try: @@ -412,13 +395,7 @@ def _cleanup_idle_sandboxes(self, idle_timeout: float) -> None: except Exception as e: logger.error(f"Failed to destroy idle sandbox {sandbox_id}: {e}") - # Destroy warm-pool sandboxes (already removed from _warm_pool under lock above) - for sandbox_id, info in warm_to_destroy: - try: - self._backend.destroy(info) - logger.info(f"Destroyed idle warm-pool sandbox {sandbox_id}") - except Exception as e: - logger.error(f"Failed to destroy idle warm-pool sandbox {sandbox_id}: {e}") + self._reap_expired_warm(idle_timeout) # ── Signal handling ────────────────────────────────────────────────── @@ -650,23 +627,29 @@ def _drop_unhealthy_sandbox(self, sandbox_id: str, reason: str, *, expected_info logger.warning(f"Dropped unhealthy sandbox {sandbox_id}: {reason}") - def _replica_count(self) -> tuple[int, int]: - """Return configured replicas and currently tracked sandbox count.""" - replicas = self._config.get("replicas", DEFAULT_REPLICAS) - with self._lock: - total = len(self._sandboxes) + len(self._warm_pool) - return replicas, total + def _active_count_locked(self) -> int: + """Return active AIO sandbox count while ``_lock`` is held.""" + return len(self._sandboxes) - def _log_replicas_soft_cap(self, replicas: int, sandbox_id: str, evicted: str | None) -> None: - """Log the result of enforcing the warm-pool replica budget.""" - if evicted: - logger.info(f"Evicted warm-pool sandbox {evicted} to stay within replicas={replicas}") + def _destroy_warm_entry(self, sandbox_id: str, entry: SandboxInfo, *, reason: str) -> None: + """Destroy a warm-pool sandbox using AIO-specific backend logging.""" + try: + self._backend.destroy(entry) + except Exception as e: + if reason == "idle_timeout": + logger.error(f"Failed to destroy idle warm-pool sandbox {sandbox_id}: {e}") + elif reason == "replica_enforcement": + logger.error(f"Failed to destroy warm-pool sandbox {sandbox_id}: {e}") + else: + logger.error(f"Failed to destroy warm-pool sandbox {sandbox_id} for {reason}: {e}") return - # All slots are occupied by active sandboxes — proceed anyway and log. - # The replicas limit is a soft cap; we never forcibly stop a container - # that is actively serving a thread. - logger.warning(f"All {replicas} replica slots are in active use; creating sandbox {sandbox_id} beyond the soft limit") + if reason == "idle_timeout": + logger.info(f"Destroyed idle warm-pool sandbox {sandbox_id}") + elif reason == "replica_enforcement": + logger.info(f"Destroyed warm-pool sandbox {sandbox_id}") + else: + logger.info(f"Destroyed warm-pool sandbox {sandbox_id} for {reason}") # ── Core: acquire / get / release / shutdown ───────────────────────── @@ -822,26 +805,6 @@ async def _discover_or_create_with_lock_async(self, thread_id: str, sandbox_id: await asyncio.to_thread(_unlock_file, lock_file) await asyncio.to_thread(lock_file.close) - def _evict_oldest_warm(self) -> str | None: - """Destroy the oldest container in the warm pool to free capacity. - - Returns: - The evicted sandbox_id, or None if warm pool is empty. - """ - with self._lock: - if not self._warm_pool: - return None - oldest_id = min(self._warm_pool, key=lambda sid: self._warm_pool[sid][1]) - info, _ = self._warm_pool.pop(oldest_id) - - try: - self._backend.destroy(info) - logger.info(f"Destroyed warm-pool sandbox {oldest_id}") - except Exception as e: - logger.error(f"Failed to destroy warm-pool sandbox {oldest_id}: {e}") - return None - return oldest_id - def _create_sandbox(self, thread_id: str | None, sandbox_id: str, *, user_id: str | None = None) -> str: """Create a new sandbox via the backend. @@ -989,11 +952,7 @@ def shutdown(self) -> None: warm_items = list(self._warm_pool.items()) self._warm_pool.clear() - # Stop idle checker - self._idle_checker_stop.set() - if self._idle_checker_thread is not None and self._idle_checker_thread.is_alive(): - self._idle_checker_thread.join(timeout=5) - logger.info("Stopped idle checker thread") + self._stop_idle_checker() logger.info(f"Shutting down {len(sandbox_ids)} active + {len(warm_items)} warm-pool sandbox(es)") diff --git a/backend/packages/harness/deerflow/community/boxlite/README.md b/backend/packages/harness/deerflow/community/boxlite/README.md index c0c1e745dda..bedecba9006 100644 --- a/backend/packages/harness/deerflow/community/boxlite/README.md +++ b/backend/packages/harness/deerflow/community/boxlite/README.md @@ -16,14 +16,24 @@ sandbox: image: python:3.12-slim # any OCI image, run unchanged (default: python:3.12-slim) memory_mib: 1024 # per-box memory cap (optional) cpus: 2 # per-box vCPUs (optional) + replicas: 3 # active + warm VM cap per gateway process (default: 3) + idle_timeout: 600 # warm VM idle seconds before stop; 0 disables reaping environment: # injected into every command PYTHONUNBUFFERED: "1" ``` +Install the optional runtime before selecting this provider: + ```bash -pip install boxlite # an optional `[boxlite]` extra + uv.lock update will follow once the approach lands +pip install "deerflow-harness[boxlite]" ``` +The `boxlite` package is an optional DeerFlow harness extra, not part of the +default install. It is also limited to the host platforms and architectures +where BoxLite publishes wheels and can boot micro-VMs. Unsupported development +hosts, such as Windows, should use another sandbox provider or run DeerFlow from +a supported Linux/macOS environment. + **Host requirement:** BoxLite boots micro-VMs, so a Linux host needs KVM — i.e. nested virtualization when DeerFlow runs inside a cloud VM. macOS uses Hypervisor.framework. This is the main deployment constraint to weigh vs. the @@ -34,10 +44,9 @@ container-based providers. DeerFlow's `Sandbox` contract is synchronous; BoxLite's SDK is async-native and its box handles are event-loop-affine. The provider owns **one** private asyncio loop on a daemon thread and marshals every coroutine onto it via -`run_coroutine_threadsafe`. This keeps all operations on the loop the box was -started on and is safe under DeerFlow's `asyncio.to_thread` worker pool — without -using BoxLite's greenlet sync facade, which refuses to run inside an async -context and is thread-affine. +`run_coroutine_threadsafe`. BoxLite boxes are named deterministically from +`user_id:thread_id`, released into an in-process warm pool after each agent turn, +and reclaimed by the same thread on the next acquire. | File | Role | | --- | --- | @@ -57,8 +66,9 @@ inside the box and reuse `deerflow.sandbox.search`, mirroring `e2b_sandbox`: The provider creates `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/skills` on box start so those virtual paths resolve natively. -**Out of scope for this pass** (follow-ups): warm pooling, idle reaping, mount -syncing, and remote/provisioner modes. +Warm-pool capacity is governed by `sandbox.replicas` across active + warm VMs. +`sandbox.idle_timeout` controls how long released warm VMs stay running; `0` +disables idle reaping. Active boxes are never evicted to satisfy the cap. ## Status diff --git a/backend/packages/harness/deerflow/community/boxlite/__init__.py b/backend/packages/harness/deerflow/community/boxlite/__init__.py index 125a8eadcd4..20adefb019a 100644 --- a/backend/packages/harness/deerflow/community/boxlite/__init__.py +++ b/backend/packages/harness/deerflow/community/boxlite/__init__.py @@ -17,13 +17,14 @@ image: python:3.12-slim # any OCI image; runs unchanged memory_mib: 1024 # per-box memory cap (optional) cpus: 2 # per-box vCPUs (optional) + replicas: 3 # active + warm VM cap per gateway process + idle_timeout: 600 # warm VM idle seconds before stop; 0 disables environment: # injected into every command PYTHONUNBUFFERED: "1" -Install the runtime (an optional ``[boxlite]`` extra + lockfile update will -follow once the approach lands):: +Install the optional runtime before selecting this provider:: - pip install boxlite + pip install "deerflow-harness[boxlite]" Host requirement: BoxLite boots micro-VMs, so a Linux host needs KVM (nested virtualization when DeerFlow itself runs inside a cloud VM); macOS uses diff --git a/backend/packages/harness/deerflow/community/boxlite/box.py b/backend/packages/harness/deerflow/community/boxlite/box.py index d670e8e1f48..b923c29b22b 100644 --- a/backend/packages/harness/deerflow/community/boxlite/box.py +++ b/backend/packages/harness/deerflow/community/boxlite/box.py @@ -29,7 +29,7 @@ from deerflow.sandbox.search import GrepMatch, path_matches, should_ignore_path, truncate_line if TYPE_CHECKING: - from collections.abc import Awaitable, Callable + from collections.abc import Callable from boxlite import SimpleBox @@ -61,7 +61,7 @@ def __init__( self, id: str, box: SimpleBox, - run: Callable[[Awaitable[T]], T], + run: Callable[..., T], *, default_env: dict[str, str] | None = None, ) -> None: @@ -124,6 +124,11 @@ def execute_command( DeerFlow passes a bash command *string*; BoxLite's ``exec`` takes argv, so it runs through ``sh -lc``. Per-call ``env`` is layered over the static config environment and scoped to this command only. + + *timeout* bounds both layers: BoxLite's SDK ``exec(timeout=...)`` handles + command timeout inside the VM, and the event-loop bridge receives the + same value so ``run_coroutine_threadsafe(...).result(timeout)`` cannot + block the caller forever if the SDK future itself never resolves. """ _validate_extra_env(env) # POSIX env-var key rule; raises ValueError on a bad key merged_env = {**self._default_env, **(env or {})} or None @@ -132,7 +137,7 @@ def execute_command( return "Error: sandbox has been closed" box = self._box try: - result = self._run(box.exec("sh", "-lc", command, env=merged_env, timeout=timeout)) + result = self._run(box.exec("sh", "-lc", command, env=merged_env, timeout=timeout), timeout=timeout) except Exception as e: logger.error("Failed to execute command in BoxLite box %s: %s", self.id, e) return f"Error: {e}" diff --git a/backend/packages/harness/deerflow/community/boxlite/provider.py b/backend/packages/harness/deerflow/community/boxlite/provider.py index e1e61f94193..33c62a2597b 100644 --- a/backend/packages/harness/deerflow/community/boxlite/provider.py +++ b/backend/packages/harness/deerflow/community/boxlite/provider.py @@ -8,15 +8,18 @@ may appear under ``sandbox:`` in ``config.yaml`` even though they are not declared on the model — see this package's ``__init__`` docstring for the full set. The provider creates one micro-VM per ``(user, thread)`` and reuses it within the -process; warm pooling, idle reaping and remote modes are out of scope for now. +process. """ from __future__ import annotations import asyncio import atexit +import hashlib import logging import threading +import time +import uuid from collections.abc import Awaitable from typing import TYPE_CHECKING, Any, TypeVar @@ -26,6 +29,7 @@ from deerflow.sandbox.sandbox import Sandbox from deerflow.sandbox.sandbox_provider import SandboxProvider +from ..warm_pool_lifecycle import WarmPoolLifecycleMixin from .box import BoxliteBox if TYPE_CHECKING: @@ -36,6 +40,7 @@ T = TypeVar("T") DEFAULT_IMAGE = "python:3.12-slim" +_BOX_NAME_PREFIX = "deer-flow-boxlite-" # DeerFlow's virtual prefixes, materialised on the box rootfs at start so the # Sandbox file APIs (which address /mnt/user-data/...) resolve natively. _VIRTUAL_DIRS = ( @@ -55,10 +60,19 @@ def _import_simplebox() -> type[SimpleBox]: try: from boxlite import SimpleBox except ImportError as e: # pragma: no cover - depends on the optional dependency - raise ImportError("BoxliteProvider requires the 'boxlite' package. Install it with: pip install boxlite.") from e + raise ImportError("BoxliteProvider requires the optional 'boxlite' dependency. Install it with: pip install 'deerflow-harness[boxlite]' or pip install boxlite.") from e return SimpleBox +def _import_sync_boxlite_runtime(): + """Import BoxLite's sync runtime lazily for startup reconciliation.""" + try: + from boxlite import SyncBoxlite + except ImportError as e: # pragma: no cover - depends on the optional dependency + raise ImportError("BoxliteProvider requires the optional 'boxlite' dependency. Install it with: pip install 'deerflow-harness[boxlite]' or pip install boxlite.") from e + return SyncBoxlite + + class _EventLoopThread: """A private asyncio event loop running on a dedicated daemon thread. @@ -71,34 +85,106 @@ class _EventLoopThread: """ def __init__(self) -> None: - self._loop = asyncio.new_event_loop() - self._thread = threading.Thread(target=self._loop.run_forever, name="boxlite-loop", daemon=True) + self._loop: asyncio.AbstractEventLoop | None = None + self._ready = threading.Event() + self._thread = threading.Thread(target=self._run_forever, name="boxlite-loop", daemon=True) self._thread.start() + self._ready.wait(timeout=5) + + def _run_forever(self) -> None: + self._loop = asyncio.new_event_loop() + asyncio.set_event_loop(self._loop) + self._loop.call_soon(self._ready.set) + self._loop.run_forever() def run(self, coro: Awaitable[T], *, timeout: float | None = None) -> T: + if self._loop is None: + raise RuntimeError("BoxLite event loop is not ready") return asyncio.run_coroutine_threadsafe(coro, self._loop).result(timeout) def close(self) -> None: + if self._loop is None: + return self._loop.call_soon_threadsafe(self._loop.stop) + wake = getattr(self._loop, "_write_to_self", None) + if wake is not None: + wake() self._thread.join(timeout=5) if not self._loop.is_running(): self._loop.close() -class BoxliteProvider(SandboxProvider): +class _SyncBoxAdapter: + """Adapt a sync BoxLite ``Box`` handle to the async ``SimpleBox`` methods we use.""" + + def __init__(self, runtime: Any, box: Any) -> None: + self._runtime = runtime + self._box = box + + async def exec( + self, + cmd: str, + *args: str, + env: dict[str, str] | None = None, + user: str | None = None, + timeout: float | None = None, + cwd: str | None = None, + ) -> Any: + return self._box.exec( + cmd, + *args, + env=env, + user=user, + timeout=timeout, + cwd=cwd, + ) + + async def stop(self) -> None: + try: + self._box.stop() + finally: + self._runtime.stop() + + +def _run_sync_adapter[T](coro: Awaitable[T], *, timeout: float | None = None) -> T: + """Run sync-adapter coroutines without using the BoxLite async loop.""" + if timeout is None: + return asyncio.run(coro) + return asyncio.run(asyncio.wait_for(coro, timeout=timeout)) + + +class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider): """Run each DeerFlow sandbox as a BoxLite micro-VM.""" uses_thread_data_mounts = False needs_upload_permission_adjustment = True + _idle_checker_thread_name = "boxlite-idle-reaper" + + @staticmethod + def _sandbox_id(thread_id: str, user_id: str) -> str: + """Deterministic sandbox ID from user/thread scope. + + Includes user_id so a box created for one user's bucket cannot be + reclaimed by another user's thread with the same thread_id. + """ + return hashlib.sha256(f"{user_id}:{thread_id}".encode()).hexdigest()[:8] + + # ── Provider ──────────────────────────────────────────────────────── def __init__(self) -> None: self._lock = threading.Lock() self._boxes: dict[str, BoxliteBox] = {} self._thread_boxes: dict[tuple[str, str], str] = {} + self._warm_pool: dict[str, tuple[BoxliteBox, float]] = {} + self._acquire_locks: dict[str, threading.Lock] = {} + self._idle_checker_stop = threading.Event() + self._idle_checker_thread: threading.Thread | None = None self._shutdown_called = False self._config = self._load_config() self._loop = _EventLoopThread() atexit.register(self.shutdown) + self._reconcile_orphans() + self._start_idle_checker() def _load_config(self) -> dict[str, Any]: sandbox_config = get_app_config().sandbox @@ -108,39 +194,172 @@ def _opt(name: str, default: Any = None) -> Any: # $VARS in config.yaml are already resolved by AppConfig.resolve_env_variables # (which raises on a missing var), so the environment dict is used as-is. + replicas = _opt("replicas") + idle_timeout = _opt("idle_timeout") return { "image": _opt("image") or DEFAULT_IMAGE, "memory_mib": _opt("memory_mib"), "cpus": _opt("cpus"), "environment": dict(_opt("environment") or {}), + "replicas": replicas if replicas is not None else self.DEFAULT_REPLICAS, + "idle_timeout": idle_timeout if idle_timeout is not None else self.DEFAULT_IDLE_TIMEOUT, } @staticmethod def _thread_key(thread_id: str, user_id: str | None) -> tuple[str, str]: return (user_id or "", thread_id) + @staticmethod + def _box_name(sandbox_id: str) -> str: + return f"{_BOX_NAME_PREFIX}{sandbox_id}" + + @staticmethod + def _sandbox_id_from_box_name(name: str | None) -> str | None: + if not name or not name.startswith(_BOX_NAME_PREFIX): + return None + sandbox_id = name[len(_BOX_NAME_PREFIX) :] + return sandbox_id or None + + def _lock_for_sandbox(self, sandbox_id: str) -> threading.Lock: + """Return the per-sandbox acquire lock for a deterministic sandbox id.""" + with self._lock: + lock = self._acquire_locks.get(sandbox_id) + if lock is None: + lock = threading.Lock() + self._acquire_locks[sandbox_id] = lock + return lock + + def _start_idle_checker(self) -> None: + """Start idle cleanup when enabled; idle_timeout=0 keeps it disabled.""" + if self._config["idle_timeout"] <= 0: + return + super()._start_idle_checker() + + def _active_count_locked(self) -> int: + """Return active BoxLite box count while ``_lock`` is held.""" + return len(self._boxes) + + def _destroy_warm_entry(self, sandbox_id: str, entry: BoxliteBox, *, reason: str) -> None: + """Close a removed warm-pool entry and log with context.""" + try: + entry.close() + if reason == "idle_timeout": + logger.info("Idle reaper destroyed expired warm-pool box %s", sandbox_id) + elif reason == "replica_enforcement": + logger.info("Replica enforcement evicted oldest warm-pool box %s", sandbox_id) + else: + logger.info("Destroyed warm-pool box %s (reason=%s)", sandbox_id, reason) + except Exception as e: + if reason == "idle_timeout": + logger.warning("Error closing expired BoxLite box %s: %s", sandbox_id, e) + elif reason == "replica_enforcement": + logger.warning("Error closing evicted BoxLite box %s: %s", sandbox_id, e) + else: + logger.warning("Error closing BoxLite box %s (reason=%s): %s", sandbox_id, reason, e) + + def _reconcile_orphans(self) -> None: + """Adopt DeerFlow-owned BoxLite boxes left by a previous provider/process. + + BoxLite boxes are discovered by a DeerFlow-specific name prefix. Adopted + boxes enter the warm pool so the normal idle reaper can reclaim them. + """ + try: + adopted = self._adopt_existing_boxes() + except ImportError: + logger.debug("BoxLite is not installed; skipping startup reconciliation") + return + except Exception as e: + logger.warning("Failed to reconcile existing BoxLite boxes: %s", e) + return + + if adopted: + logger.info("Startup reconciliation adopted %s BoxLite box(es)", adopted) + + def _adopt_existing_boxes(self) -> int: + runtime_cls = _import_sync_boxlite_runtime() + now = time.time() + adopted = 0 + + list_runtime = runtime_cls.default().start() + try: + infos = list_runtime.list_info() + finally: + list_runtime.stop() + + for info in infos: + name = getattr(info, "name", None) + sandbox_id = self._sandbox_id_from_box_name(name) + if sandbox_id is None: + continue + with self._lock: + if sandbox_id in self._boxes or sandbox_id in self._warm_pool: + continue + + box_runtime = runtime_cls.default().start() + try: + box = box_runtime.get(name) + except Exception as e: + box_runtime.stop() + logger.warning("Failed to retrieve existing BoxLite box %s: %s", name, e) + continue + if box is None: + box_runtime.stop() + continue + + wrapped = BoxliteBox(sandbox_id, _SyncBoxAdapter(box_runtime, box), _run_sync_adapter, default_env=self._config["environment"]) + with self._lock: + if sandbox_id in self._boxes or sandbox_id in self._warm_pool: + box_runtime.stop() + continue + self._warm_pool[sandbox_id] = (wrapped, now) + adopted += 1 + logger.info("Adopted existing BoxLite box %s (%s) into warm pool", sandbox_id, name) + + return adopted + + # ── Acquire / release ──────────────────────────────────────────────── + def acquire(self, thread_id: str | None = None, *, user_id: str | None = None) -> str: - if thread_id is not None: - key = self._thread_key(thread_id, user_id) + if thread_id is None: + sandbox_id = str(uuid.uuid4())[:8] + box = self._create_box(sandbox_id) + with self._lock: + self._boxes[box.id] = box + return box.id + + key = self._thread_key(thread_id, user_id) + sandbox_id = self._sandbox_id(thread_id, user_id) + acquire_lock = self._lock_for_sandbox(sandbox_id) + with acquire_lock: with self._lock: existing = self._thread_boxes.get(key) if existing is not None and existing in self._boxes: return existing - box = self._create_box() + reclaimed = self._reclaim_warm_pool(sandbox_id) + if reclaimed is not None: + with self._lock: + self._thread_boxes[key] = reclaimed + return reclaimed - with self._lock: - self._boxes[box.id] = box - if thread_id is not None: - self._thread_boxes[self._thread_key(thread_id, user_id)] = box.id - return box.id - - def _create_box(self) -> BoxliteBox: + box = self._create_box(sandbox_id) + with self._lock: + self._boxes[box.id] = box + self._thread_boxes[key] = box.id + return box.id + + def _create_box(self, sandbox_id: str) -> BoxliteBox: + # Enforce replica limit: evict oldest warm-pool box if active + warm boxes are at capacity. + replicas, total = self._replica_count() + if total >= replicas: + evicted = self._evict_oldest_warm() + self._log_replicas_soft_cap(replicas, sandbox_id, evicted) simplebox_cls = _import_simplebox() mkdir_cmd = "mkdir -p " + " ".join(_VIRTUAL_DIRS) async def _make() -> SimpleBox: box = simplebox_cls( + name=self._box_name(sandbox_id), image=self._config["image"], memory_mib=self._config["memory_mib"], cpus=self._config["cpus"], @@ -151,36 +370,109 @@ async def _make() -> SimpleBox: return box box = self._loop.run(_make()) - logger.info("Created BoxLite box %s (image=%s)", box.id, self._config["image"]) - return BoxliteBox(box.id, box, self._loop.run, default_env=self._config["environment"]) + logger.info("Created BoxLite box %s (name=%s, image=%s)", sandbox_id, self._box_name(sandbox_id), self._config["image"]) + return BoxliteBox(sandbox_id, box, self._loop.run, default_env=self._config["environment"]) def get(self, sandbox_id: str) -> Sandbox | None: with self._lock: return self._boxes.get(sandbox_id) def release(self, sandbox_id: str) -> None: + """Release a sandbox into the warm pool — VM stays running. + + The box is moved from _boxes to _warm_pool; _thread_boxes entries are + cleared so the thread no longer holds an active reference. The VM is + NOT stopped unless shutdown has already begun. + """ + close_box: BoxliteBox | None = None with self._lock: box = self._boxes.pop(sandbox_id, None) for key in [k for k, sid in self._thread_boxes.items() if sid == sandbox_id]: self._thread_boxes.pop(key, None) - if box is not None: + if box is None: + return + if self._shutdown_called: + close_box = box + else: + self._warm_pool[sandbox_id] = (box, time.time()) + + if close_box is not None: + close_box.close() + logger.info("Closed released sandbox %s because shutdown is in progress", sandbox_id) + else: + logger.info("Released sandbox %s to warm pool (VM still running)", sandbox_id) + + def _reclaim_warm_pool(self, sandbox_id: str) -> str | None: + """Try to reclaim a warm-pool box by sandbox_id. + + Returns sandbox_id on success, None if not found or dead. + """ + with self._lock: + if sandbox_id not in self._warm_pool: + return None + box, _ = self._warm_pool[sandbox_id] + + # Health check: run a simple command to verify the VM is alive + try: + result = box.execute_command("echo ok", timeout=5) + if "ok" not in result: + logger.warning("Warm pool box %s health check failed: %s", sandbox_id, result) + with self._lock: + self._warm_pool.pop(sandbox_id, None) + box.close() + return None + except Exception as e: + logger.warning("Warm pool box %s health check error: %s", sandbox_id, e) + with self._lock: + self._warm_pool.pop(sandbox_id, None) box.close() + return None + + # Promote from warm pool to active + with self._lock: + warm_entry = self._warm_pool.pop(sandbox_id, None) + if warm_entry is None: + return None # Raced with another thread + box, _ = warm_entry + self._boxes[sandbox_id] = box + + logger.info("Reclaimed warm-pool box %s", sandbox_id) + return sandbox_id def reset(self) -> None: + """Release tracked BoxLite VMs to this instance's warm-pool cleanup. + + ``reset_sandbox_provider()`` drops the provider singleton and calls this + lightweight hook so config changes take effect on the next provider + construction. Teardown belongs to ``shutdown()``; reset intentionally + leaves running VMs alive, but keeps them visible to this instance's idle + reaper and atexit shutdown instead of orphaning them. + """ with self._lock: + now = time.time() + for sandbox_id, box in self._boxes.items(): + self._warm_pool.setdefault(sandbox_id, (box, now)) self._boxes.clear() self._thread_boxes.clear() + self._acquire_locks.clear() def shutdown(self) -> None: with self._lock: if self._shutdown_called: return self._shutdown_called = True + + self._stop_idle_checker() + + with self._lock: active = list(self._boxes.values()) + warm = [box for box, _ in self._warm_pool.values()] self._boxes.clear() + self._warm_pool.clear() self._thread_boxes.clear() + self._acquire_locks.clear() - for box in active: + for box in active + warm: try: box.close() except Exception as e: # pragma: no cover - defensive diff --git a/backend/packages/harness/deerflow/community/warm_pool_lifecycle.py b/backend/packages/harness/deerflow/community/warm_pool_lifecycle.py new file mode 100644 index 00000000000..ffb5c066e50 --- /dev/null +++ b/backend/packages/harness/deerflow/community/warm_pool_lifecycle.py @@ -0,0 +1,127 @@ +"""Shared warm-pool lifecycle helpers for community sandbox providers.""" + +from __future__ import annotations + +import logging +import threading +import time +from typing import Any + +logger = logging.getLogger(__name__) + +DEFAULT_IDLE_TIMEOUT = 600 +DEFAULT_REPLICAS = 3 +IDLE_CHECK_INTERVAL = 60 + + +class WarmPoolLifecycleMixin[WarmEntryT]: + """Mixin for provider warm-pool expiry and replica lifecycle mechanics.""" + + DEFAULT_IDLE_TIMEOUT = DEFAULT_IDLE_TIMEOUT + DEFAULT_REPLICAS = DEFAULT_REPLICAS + IDLE_CHECK_INTERVAL = IDLE_CHECK_INTERVAL + _idle_checker_thread_name = "warm-pool-idle-checker" + + _lock: threading.Lock + _warm_pool: dict[str, tuple[WarmEntryT, float]] + _config: dict[str, Any] + _idle_checker_stop: threading.Event + _idle_checker_thread: threading.Thread | None + + def _active_count_locked(self) -> int: + """Return active entry count while ``_lock`` is held.""" + raise NotImplementedError + + def _destroy_warm_entry(self, sandbox_id: str, entry: WarmEntryT, *, reason: str) -> None: + """Destroy a warm-pool entry after it has been removed from the pool.""" + raise NotImplementedError + + def _replica_count(self) -> tuple[int, int]: + """Return configured replicas and current active + warm entry count.""" + replicas = int(self._config.get("replicas", DEFAULT_REPLICAS)) + with self._lock: + total = self._active_count_locked() + len(self._warm_pool) + return replicas, total + + def _log_replicas_soft_cap(self, replicas: int, sandbox_id: str, evicted: str | None) -> None: + """Log the result of enforcing the warm-pool replica soft cap.""" + if evicted is not None: + logger.info("Evicted warm-pool sandbox %s to stay within replicas=%s", evicted, replicas) + return + + logger.warning( + "All %s replica slots are in active use; creating sandbox %s beyond the soft limit", + replicas, + sandbox_id, + ) + + def _evict_oldest_warm(self) -> str | None: + """Remove and destroy the oldest warm entry by timestamp.""" + with self._lock: + if not self._warm_pool: + return None + sandbox_id, (entry, _) = min(self._warm_pool.items(), key=lambda item: item[1][1]) + self._warm_pool.pop(sandbox_id) + + self._destroy_warm_entry(sandbox_id, entry, reason="replica_enforcement") + return sandbox_id + + def _reap_expired_warm(self, idle_timeout: float | None = None) -> None: + """Remove and destroy warm entries older than ``idle_timeout`` seconds.""" + timeout = float(self._config.get("idle_timeout", DEFAULT_IDLE_TIMEOUT) if idle_timeout is None else idle_timeout) + if timeout <= 0: + return + + now = time.time() + expired: list[tuple[str, WarmEntryT]] = [] + with self._lock: + for sandbox_id, (entry, timestamp) in self._warm_pool.items(): + if now - timestamp > timeout: + expired.append((sandbox_id, entry)) + for sandbox_id, _ in expired: + self._warm_pool.pop(sandbox_id, None) + + for sandbox_id, entry in expired: + self._destroy_warm_entry(sandbox_id, entry, reason="idle_timeout") + + def _start_idle_checker(self) -> None: + """Start the daemon thread that periodically cleans idle warm entries.""" + if self._idle_checker_thread is not None and self._idle_checker_thread.is_alive(): + return + + self._idle_checker_stop.clear() + self._idle_checker_thread = threading.Thread( + target=self._idle_checker_loop, + name=self._idle_checker_thread_name, + daemon=True, + ) + self._idle_checker_thread.start() + logger.info("Started warm-pool idle checker thread (timeout: %ss)", self._config.get("idle_timeout", DEFAULT_IDLE_TIMEOUT)) + + def _stop_idle_checker(self) -> None: + """Stop the idle checker thread and wait for it to exit when running.""" + self._idle_checker_stop.set() + thread = self._idle_checker_thread + if thread is not None and thread.is_alive() and thread is not threading.current_thread(): + thread.join(timeout=5) + + def _idle_checker_loop(self) -> None: + """Run periodic idle cleanup until the stop event is set.""" + idle_timeout = float(self._config.get("idle_timeout", DEFAULT_IDLE_TIMEOUT)) + while not self._idle_checker_stop.wait(self.IDLE_CHECK_INTERVAL): + try: + self._cleanup_idle_resources(idle_timeout) + except Exception: + logger.exception("Error in warm-pool idle checker loop") + + def _cleanup_idle_resources(self, idle_timeout: float) -> None: + """Clean resources idle longer than ``idle_timeout`` seconds.""" + self._reap_expired_warm(idle_timeout) + + +__all__ = [ + "DEFAULT_IDLE_TIMEOUT", + "DEFAULT_REPLICAS", + "IDLE_CHECK_INTERVAL", + "WarmPoolLifecycleMixin", +] diff --git a/backend/packages/harness/deerflow/config/app_config.py b/backend/packages/harness/deerflow/config/app_config.py index 217d1df2f15..548a01d300f 100644 --- a/backend/packages/harness/deerflow/config/app_config.py +++ b/backend/packages/harness/deerflow/config/app_config.py @@ -39,6 +39,7 @@ from deerflow.config.token_usage_config import TokenUsageConfig from deerflow.config.tool_config import ToolConfig, ToolGroupConfig from deerflow.config.tool_output_config import ToolOutputConfig +from deerflow.config.tool_progress_config import ToolProgressConfig from deerflow.config.tool_search_config import ToolSearchConfig, load_tool_search_config_from_dict load_dotenv() @@ -174,6 +175,7 @@ class AppConfig(BaseModel): ), ) loop_detection: LoopDetectionConfig = Field(default_factory=LoopDetectionConfig, description="Loop detection middleware configuration") + tool_progress: ToolProgressConfig = Field(default_factory=ToolProgressConfig, description="Tool progress state machine middleware configuration") read_before_write: ReadBeforeWriteConfig = Field(default_factory=ReadBeforeWriteConfig, description="Read-before-write file gate middleware configuration") safety_finish_reason: SafetyFinishReasonConfig = Field(default_factory=SafetyFinishReasonConfig, description="Provider safety-filter finish_reason interception middleware configuration") auth: AuthAppConfig = Field(default_factory=AuthAppConfig, description="Authentication configuration (local + OIDC SSO)") diff --git a/backend/packages/harness/deerflow/config/memory_config.py b/backend/packages/harness/deerflow/config/memory_config.py index f14e777e9d8..867f1e6e19e 100644 --- a/backend/packages/harness/deerflow/config/memory_config.py +++ b/backend/packages/harness/deerflow/config/memory_config.py @@ -98,6 +98,40 @@ class MemoryConfig(BaseModel): "safety-truncation ceiling is raised accordingly." ), ) + # ── Staleness review ──────────────────────────────────────────────── + staleness_review_enabled: bool = Field( + default=True, + description=( + "Enable staleness review for aged facts. When enabled, facts older " + "than ``staleness_age_days`` are surfaced in the memory-update prompt " + "so the LLM can semantically judge whether each is still valid or " + "should be removed. This solves the 'silent staleness' problem where " + "outdated facts persist because no future conversation explicitly " + "contradicts them." + ), + ) + staleness_age_days: int = Field( + default=90, + ge=30, + le=365, + description=("Facts older than this many days become candidates for staleness review. 90 days (~one quarter) balances between catching genuine changes (job switches, tech-stack migrations) and avoiding noise on stable facts."), + ) + staleness_min_candidates: int = Field( + default=3, + ge=1, + le=50, + description=("Minimum number of stale facts required to trigger a review cycle. Below this threshold the prompt overhead is not justified."), + ) + staleness_max_removals_per_cycle: int = Field( + default=10, + ge=1, + le=50, + description=("Maximum number of facts the staleness review can remove in a single update cycle. Prevents the LLM from over-pruning when reviewing a large backlog of aged facts."), + ) + staleness_protected_categories: list[str] = Field( + default_factory=lambda: ["correction"], + description=("Fact categories exempt from staleness review. Correction facts represent explicit user feedback and should not be auto-pruned based on age alone."), + ) # Global configuration instance diff --git a/backend/packages/harness/deerflow/config/sandbox_config.py b/backend/packages/harness/deerflow/config/sandbox_config.py index d8434ccf899..d2f0eb38296 100644 --- a/backend/packages/harness/deerflow/config/sandbox_config.py +++ b/backend/packages/harness/deerflow/config/sandbox_config.py @@ -30,14 +30,16 @@ class SandboxConfig(BaseModel): allow_host_bash: Enable host-side bash execution for LocalSandboxProvider. Dangerous and intended only for fully trusted local workflows. + AioSandboxProvider and BoxliteProvider shared options: + image: Sandbox image to use (Docker/AIO image or BoxLite OCI image) + replicas: Maximum active + warm sandboxes/VMs per gateway process (default: 3). When the limit is reached, warm/least-recently-used sandboxes are evicted to make room; active sandboxes are not forcibly stopped. + idle_timeout: Idle timeout in seconds before released warm sandboxes/VMs are stopped (default: 600 = 10 minutes). Set to 0 to disable. + environment: Environment variables to inject into the sandbox (values starting with $ are resolved from host env) + AioSandboxProvider specific options: - image: Docker image to use (default: enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest) port: Base port for sandbox containers (default: 8080) - replicas: Maximum number of concurrent sandbox containers (default: 3). When the limit is reached the least-recently-used sandbox is evicted to make room. container_prefix: Prefix for container names (default: deer-flow-sandbox) - idle_timeout: Idle timeout in seconds before sandbox is released (default: 600 = 10 minutes). Set to 0 to disable. mounts: List of volume mounts to share directories with the container - environment: Environment variables to inject into the container (values starting with $ are resolved from host env) """ use: str = Field( @@ -50,7 +52,7 @@ class SandboxConfig(BaseModel): ) image: str | None = Field( default=None, - description="Docker image to use for the sandbox container", + description="Sandbox image to use (Docker/AIO image or BoxLite OCI image)", ) port: int | None = Field( default=None, @@ -58,7 +60,7 @@ class SandboxConfig(BaseModel): ) replicas: int | None = Field( default=None, - description="Maximum number of concurrent sandbox containers (default: 3). When the limit is reached the least-recently-used sandbox is evicted to make room.", + description="Maximum active + warm sandboxes/VMs per gateway process (default: 3). Warm/least-recently-used entries are evicted to make room; active sandboxes are not forcibly stopped.", ) container_prefix: str | None = Field( default=None, @@ -66,7 +68,7 @@ class SandboxConfig(BaseModel): ) idle_timeout: int | None = Field( default=None, - description="Idle timeout in seconds before sandbox is released (default: 600 = 10 minutes). Set to 0 to disable.", + description="Idle timeout in seconds before released warm sandboxes/VMs are stopped (default: 600 = 10 minutes). Set to 0 to disable.", ) mounts: list[VolumeMountConfig] = Field( default_factory=list, diff --git a/backend/packages/harness/deerflow/config/tool_progress_config.py b/backend/packages/harness/deerflow/config/tool_progress_config.py new file mode 100644 index 00000000000..47d1f909443 --- /dev/null +++ b/backend/packages/harness/deerflow/config/tool_progress_config.py @@ -0,0 +1,45 @@ +"""Configuration for tool progress tracking middleware.""" + +from pydantic import BaseModel, Field + + +class ToolProgressConfig(BaseModel): + """Configuration for task-level tool call progress tracking.""" + + enabled: bool = Field( + default=False, + description="Whether to enable tool progress tracking middleware", + ) + stagnation_threshold: int = Field( + default=3, + ge=1, + description="Number of consecutive problem calls before injecting a warning hint", + ) + warn_escalation_count: int = Field( + default=2, + ge=1, + description="Additional problem occurrences after WARNED before escalating to BLOCKED", + ) + inject_assessment: bool = Field( + default=True, + description="Whether to inject progress assessment hints into model requests", + ) + jaccard_similarity_threshold: float = Field( + default=0.8, + ge=0.0, + le=1.0, + description="Word-set Jaccard similarity threshold for near-duplicate result detection", + ) + min_word_count_for_similarity: int = Field( + default=10, + description="Minimum unique word count to apply Jaccard check; shorter content skips near-duplicate detection entirely", + ) + exempt_tools: set[str] = Field( + default_factory=lambda: {"ask_clarification", "write_todos", "present_files", "task"}, + description="Tool names excluded from progress tracking", + ) + max_tracked_threads: int = Field( + default=100, + ge=1, + description="Maximum number of thread histories to keep in memory (LRU eviction)", + ) diff --git a/backend/packages/harness/deerflow/subagents/executor.py b/backend/packages/harness/deerflow/subagents/executor.py index a1f3dbd3a05..e4a63f6db07 100644 --- a/backend/packages/harness/deerflow/subagents/executor.py +++ b/backend/packages/harness/deerflow/subagents/executor.py @@ -19,6 +19,7 @@ from langchain.tools import BaseTool from langchain_core.messages import AIMessage, HumanMessage, SystemMessage from langchain_core.runnables import RunnableConfig +from langgraph.errors import GraphRecursionError from deerflow.agents.thread_state import SandboxState, ThreadDataState, ThreadState from deerflow.config import get_app_config @@ -31,6 +32,7 @@ from deerflow.subagents.token_collector import SubagentTokenCollector from deerflow.trace_context import DEERFLOW_TRACE_METADATA_KEY from deerflow.tracing import build_tracing_callbacks, inject_langfuse_metadata +from deerflow.utils.messages import message_content_to_text if TYPE_CHECKING: # Imported lazily at runtime inside _build_initial_state: importing @@ -57,6 +59,7 @@ class SubagentStatus(Enum): FAILED = "failed" CANCELLED = "cancelled" TIMED_OUT = "timed_out" + MAX_TURNS_REACHED = "max_turns_reached" @property def is_terminal(self) -> bool: @@ -65,6 +68,7 @@ def is_terminal(self) -> bool: type(self).FAILED, type(self).CANCELLED, type(self).TIMED_OUT, + type(self).MAX_TURNS_REACHED, } @@ -137,6 +141,49 @@ def try_set_terminal( return True +def _extract_final_result(final_state: Any, *, trace_id: str, name: str) -> str: + """Extract a human-readable result string from the streamed subagent state. + + Finds the last ``AIMessage`` in the conversation and stringifies its + content via the shared :func:`message_content_to_text` helper; falls back + to the last message of any type when no AIMessage is present. Returns a + sentinel string (``"No response generated"``) when there is nothing to + extract — including when the shared helper yields an empty string — so + callers never confuse a missing result with a legitimately empty one. + + Used on both the normal-completion path and the max-turns path + (#3875 Phase 2): when ``recursion_limit`` aborts the run mid-flight, + ``final_state`` holds the last chunk streamed before the limit fired, so + this recovers the partial work instead of dropping it. + """ + if final_state is None: + logger.warning(f"[trace={trace_id}] Subagent {name} no final state") + return "No response generated" + + messages = final_state.get("messages", []) + logger.info(f"[trace={trace_id}] Subagent {name} final messages count: {len(messages)}") + + last_ai_message = None + for msg in reversed(messages): + if isinstance(msg, AIMessage): + last_ai_message = msg + break + + if last_ai_message is not None: + text = message_content_to_text(last_ai_message.content) + return text if text else "No response generated" + + if messages: + last_message = messages[-1] + logger.warning(f"[trace={trace_id}] Subagent {name} no AIMessage found, using last message: {type(last_message)}") + raw_content = last_message.content if hasattr(last_message, "content") else str(last_message) + text = message_content_to_text(raw_content) + return text if text else "No response generated" + + logger.warning(f"[trace={trace_id}] Subagent {name} no messages in final state") + return "No response generated" + + # Global storage for background task results _background_tasks: dict[str, SubagentResult] = {} _background_tasks_lock = threading.Lock() @@ -660,87 +707,34 @@ async def _aexecute(self, task: str, result_holder: SubagentResult | None = None logger.info(f"[trace={self.trace_id}] Subagent {self.config.name} completed async execution") token_usage_records = collector.snapshot_records() - final_result: str | None = None - - if final_state is None: - logger.warning(f"[trace={self.trace_id}] Subagent {self.config.name} no final state") - final_result = "No response generated" - else: - # Extract the final message - find the last AIMessage - messages = final_state.get("messages", []) - logger.info(f"[trace={self.trace_id}] Subagent {self.config.name} final messages count: {len(messages)}") - - # Find the last AIMessage in the conversation - last_ai_message = None - for msg in reversed(messages): - if isinstance(msg, AIMessage): - last_ai_message = msg - break - - if last_ai_message is not None: - content = last_ai_message.content - # Handle both str and list content types for the final result - if isinstance(content, str): - final_result = content - elif isinstance(content, list): - # Extract text from list of content blocks for final result only. - # Concatenate raw string chunks directly, but preserve separation - # between full text blocks for readability. - text_parts = [] - pending_str_parts = [] - for block in content: - if isinstance(block, str): - pending_str_parts.append(block) - elif isinstance(block, dict): - if pending_str_parts: - text_parts.append("".join(pending_str_parts)) - pending_str_parts.clear() - text_val = block.get("text") - if isinstance(text_val, str): - text_parts.append(text_val) - if pending_str_parts: - text_parts.append("".join(pending_str_parts)) - final_result = "\n".join(text_parts) if text_parts else "No text content in response" - else: - final_result = str(content) - elif messages: - # Fallback: use the last message if no AIMessage found - last_message = messages[-1] - logger.warning(f"[trace={self.trace_id}] Subagent {self.config.name} no AIMessage found, using last message: {type(last_message)}") - raw_content = last_message.content if hasattr(last_message, "content") else str(last_message) - if isinstance(raw_content, str): - final_result = raw_content - elif isinstance(raw_content, list): - parts = [] - pending_str_parts = [] - for block in raw_content: - if isinstance(block, str): - pending_str_parts.append(block) - elif isinstance(block, dict): - if pending_str_parts: - parts.append("".join(pending_str_parts)) - pending_str_parts.clear() - text_val = block.get("text") - if isinstance(text_val, str): - parts.append(text_val) - if pending_str_parts: - parts.append("".join(pending_str_parts)) - final_result = "\n".join(parts) if parts else "No text content in response" - else: - final_result = str(raw_content) - else: - logger.warning(f"[trace={self.trace_id}] Subagent {self.config.name} no messages in final state") - final_result = "No response generated" - - if final_result is None: - final_result = "No response generated" - + final_result = _extract_final_result(final_state, trace_id=self.trace_id, name=self.config.name) result.try_set_terminal( SubagentStatus.COMPLETED, result=final_result, token_usage_records=token_usage_records, ) + except GraphRecursionError: + # ``recursion_limit`` on run_config == ``self.config.max_turns`` + # (set above). Hitting it means the subagent exhausted its turn + # budget before producing a final answer — previously this fell + # through to the generic ``except Exception`` and was + # misclassified as FAILED, so the lead agent could not tell + # "broken subagent" from "out of budget" and the partial work + # already streamed into ``final_state`` was discarded (#3875). + # ``final_state`` holds the last chunk yielded before the limit + # fired, so recover whatever the subagent had produced and surface + # a distinct terminal status the lead can act on. + max_turns = self.config.max_turns + logger.warning(f"[trace={self.trace_id}] Subagent {self.config.name} reached max_turns={max_turns} (GraphRecursionError); recovering partial result") + partial = _extract_final_result(final_state, trace_id=self.trace_id, name=self.config.name) + result.try_set_terminal( + SubagentStatus.MAX_TURNS_REACHED, + result=partial, + error=f"Reached max_turns={max_turns}", + token_usage_records=collector.snapshot_records() if collector is not None else None, + ) + except Exception as e: logger.exception(f"[trace={self.trace_id}] Subagent {self.config.name} async execution failed") result.try_set_terminal( diff --git a/backend/packages/harness/deerflow/subagents/status_contract.py b/backend/packages/harness/deerflow/subagents/status_contract.py index 6651dd29680..cb55d0d6b05 100644 --- a/backend/packages/harness/deerflow/subagents/status_contract.py +++ b/backend/packages/harness/deerflow/subagents/status_contract.py @@ -38,6 +38,7 @@ "cancelled", "timed_out", "polling_timed_out", + "max_turns_reached", ] #: Enumeration of every value ``subagent_status`` may take. Mirrors the @@ -49,8 +50,18 @@ "cancelled", "timed_out", "polling_timed_out", + "max_turns_reached", ) +#: Statuses that carry a recoverable result in ``subagent_result_brief`` / +#: ``subagent_result_sha256``. ``completed`` is the obvious case; +#: ``max_turns_reached`` (#3875 Phase 2) is included because a turn-capped +#: subagent may have produced useful partial work before hitting the budget, +#: and that work should survive on the wire (and in the delegation ledger) +#: the same way a completed result does — not be discarded with the cap +#: notice alone. Other non-completed statuses carry only ``subagent_error``. +_RESULT_BEARING_STATUSES: frozenset[SubagentStatusValue] = frozenset({"completed", "max_turns_reached"}) + class StructuredSubagentResult(TypedDict): status: SubagentStatusValue @@ -93,9 +104,11 @@ def make_subagent_additional_kwargs( if status not in SUBAGENT_STATUS_VALUES: raise ValueError(f"invalid subagent status {status!r}; expected one of {SUBAGENT_STATUS_VALUES}") payload: dict[str, str] = {SUBAGENT_STATUS_KEY: status} - if status == "completed" and isinstance(result, str) and result.strip(): + if status in _RESULT_BEARING_STATUSES and isinstance(result, str) and result.strip(): payload[SUBAGENT_RESULT_BRIEF_KEY] = _bound_metadata_text(result) payload[SUBAGENT_RESULT_SHA256_KEY] = hashlib.sha256(result.encode("utf-8")).hexdigest() + # ``max_turns_reached`` is result-bearing AND carries the cap notice as + # ``subagent_error``; only ``completed`` (a clean success) suppresses it. if status != "completed" and isinstance(error, str) and error.strip(): payload[SUBAGENT_ERROR_KEY] = _bound_metadata_text(error) return payload @@ -130,6 +143,15 @@ def format_subagent_result_message( detail = error_text or "Task polling timed out." return detail, detail + if status == "max_turns_reached": + # Turn-budget cap (#3875 Phase 2): the cap reason travels on + # ``error`` (metadata), and the model-visible text leads with the + # partial result the executor recovered so the lead can reuse the + # work instead of seeing a bare failure. + detail = error_text or "Turn budget reached." + partial = result_text.strip() if result_text.strip() else "No partial result was produced before the turn budget was reached." + return f"Task reached max turns. {detail} Partial result: {partial}", detail + detail = error_text or "Task failed." if detail == "Task failed.": return detail, detail @@ -149,7 +171,7 @@ def read_subagent_result_metadata( raw_result = additional_kwargs.get(SUBAGENT_RESULT_BRIEF_KEY) raw_hash = additional_kwargs.get(SUBAGENT_RESULT_SHA256_KEY) raw_error = additional_kwargs.get(SUBAGENT_ERROR_KEY) - if status == "completed" and isinstance(raw_result, str) and raw_result.strip(): + if status in _RESULT_BEARING_STATUSES and isinstance(raw_result, str) and raw_result.strip(): payload["result_brief"] = _bound_metadata_text(raw_result) if isinstance(raw_hash, str) and _SHA256_HEX_RE.fullmatch(raw_hash): payload["result_sha256"] = raw_hash diff --git a/backend/packages/harness/deerflow/tools/builtins/task_tool.py b/backend/packages/harness/deerflow/tools/builtins/task_tool.py index 7b94f269dd9..4b9ec3f5c51 100644 --- a/backend/packages/harness/deerflow/tools/builtins/task_tool.py +++ b/backend/packages/harness/deerflow/tools/builtins/task_tool.py @@ -61,7 +61,7 @@ def pop_cached_subagent_usage(tool_call_id: str) -> dict | None: def _is_subagent_terminal(result: Any) -> bool: """Return whether a background subagent result is safe to clean up.""" - return result.status in {SubagentStatus.COMPLETED, SubagentStatus.FAILED, SubagentStatus.CANCELLED, SubagentStatus.TIMED_OUT} or getattr(result, "completed_at", None) is not None + return result.status in {SubagentStatus.COMPLETED, SubagentStatus.FAILED, SubagentStatus.CANCELLED, SubagentStatus.TIMED_OUT, SubagentStatus.MAX_TURNS_REACHED} or getattr(result, "completed_at", None) is not None async def _await_subagent_terminal(task_id: str, max_polls: int) -> Any | None: @@ -483,6 +483,27 @@ async def task_tool( status="timed_out", error=result.error, ) + elif result.status == SubagentStatus.MAX_TURNS_REACHED: + # Turn-budget cap (#3875 Phase 2): the subagent hit + # ``recursion_limit`` (= ``max_turns``) before producing a + # final answer. ``_task_result_command`` formats a distinct + # ``Task reached max turns`` message that carries the partial + # result the executor recovered, and stamps ``result_brief`` + + # the cap notice on ``subagent_error`` so the delegation ledger + # and frontend card keep both. The polling loop emits + # ``task_failed`` so any live listener transitions the card + # out of running; the structured status is the precise reason. + _cache_subagent_usage(tool_call_id, usage, enabled=cache_token_usage) + _report_subagent_usage(runtime, result) + writer({"type": "task_failed", "task_id": task_id, "error": f"Reached max_turns={config.max_turns}", "usage": usage}) + logger.warning(f"[trace={trace_id}] Task {task_id} reached max_turns={config.max_turns}; returning partial result") + cleanup_background_task(task_id) + return _task_result_command( + tool_call_id=tool_call_id, + status="max_turns_reached", + result=result.result, + error=f"Reached max_turns={config.max_turns}", + ) # Still running, wait before next poll await asyncio.sleep(5) diff --git a/backend/packages/harness/pyproject.toml b/backend/packages/harness/pyproject.toml index 8b6d6e6e681..746d5badfdd 100644 --- a/backend/packages/harness/pyproject.toml +++ b/backend/packages/harness/pyproject.toml @@ -64,6 +64,7 @@ postgres = [ # always installs this extra because Docker defaults to the redis bridge. redis = ["redis>=5.0.0"] pymupdf = ["pymupdf4llm>=0.0.17"] +boxlite = ["boxlite>=0.9.7"] [build-system] requires = ["hatchling"] diff --git a/backend/tests/test_aio_sandbox_provider.py b/backend/tests/test_aio_sandbox_provider.py index 3c0aea357a1..e0349f0a098 100644 --- a/backend/tests/test_aio_sandbox_provider.py +++ b/backend/tests/test_aio_sandbox_provider.py @@ -626,3 +626,66 @@ def test_destroy_swallows_close_errors_and_still_destroys_backend(tmp_path, capl assert "Error closing sandbox sandbox-dest-err during destroy" in caplog.text provider._backend.destroy.assert_called_once() + + +def test_cleanup_idle_sandboxes_keeps_active_cleanup_and_delegates_warm_expiry(tmp_path): + """AIO active-idle cleanup must remain local while warm expiry uses the shared lifecycle.""" + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + provider = _make_provider(tmp_path) + provider._lock = aio_mod.threading.Lock() + provider._sandboxes = {"active-old": MagicMock()} + provider._sandbox_infos = { + "active-old": aio_mod.SandboxInfo(sandbox_id="active-old", sandbox_url="http://active-old"), + } + provider._thread_sandboxes = {("default", "thread-old"): "active-old"} + provider._last_activity = {"active-old": 0.0} + provider._warm_pool = { + "warm-old": ( + aio_mod.SandboxInfo(sandbox_id="warm-old", sandbox_url="http://warm-old"), + 0.0, + ) + } + + calls = [] + provider.destroy = MagicMock(side_effect=lambda _sandbox_id: calls.append("active")) + provider._reap_expired_warm = MagicMock(side_effect=lambda _idle_timeout: calls.append("warm")) + + provider._cleanup_idle_sandboxes(1.0) + + provider.destroy.assert_called_once_with("active-old") + provider._reap_expired_warm.assert_called_once_with(1.0) + assert calls == ["active", "warm"] + + +def test_create_sandbox_evicts_oldest_warm_replica_via_shared_lifecycle(tmp_path, monkeypatch): + """Replica enforcement must destroy the oldest warm SandboxInfo before creating another.""" + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + provider = _make_provider(tmp_path) + provider._lock = aio_mod.threading.Lock() + provider._config = {"replicas": 2} + provider._sandboxes = {} + provider._sandbox_infos = {} + provider._thread_sandboxes = {} + provider._last_activity = {} + + oldest_info = aio_mod.SandboxInfo(sandbox_id="warm-oldest", sandbox_url="http://warm-oldest") + newest_info = aio_mod.SandboxInfo(sandbox_id="warm-newest", sandbox_url="http://warm-newest") + created_info = aio_mod.SandboxInfo(sandbox_id="created", sandbox_url="http://created") + provider._warm_pool = { + "warm-newest": (newest_info, 20.0), + "warm-oldest": (oldest_info, 10.0), + } + provider._backend = SimpleNamespace( + create=MagicMock(return_value=created_info), + destroy=MagicMock(), + ) + monkeypatch.setattr(aio_mod.AioSandboxProvider, "_get_extra_mounts", lambda _self, _thread_id, *, user_id=None: []) + monkeypatch.setattr(aio_mod, "wait_for_sandbox_ready", lambda _url, *, timeout=60: True) + + sandbox_id = provider._create_sandbox(None, "created", user_id="default") + + assert sandbox_id == "created" + provider._backend.destroy.assert_called_once_with(oldest_info) + assert "warm-oldest" not in provider._warm_pool + assert provider._warm_pool == {"warm-newest": (newest_info, 20.0)} + assert provider._sandbox_infos["created"] is created_info diff --git a/backend/tests/test_boxlite_provider.py b/backend/tests/test_boxlite_provider.py index 4c92ff1c807..98c1abe08c8 100644 --- a/backend/tests/test_boxlite_provider.py +++ b/backend/tests/test_boxlite_provider.py @@ -1,12 +1,15 @@ """Unit tests for the BoxLite community provider. - These run in CI without BoxLite installed: they cover the lazy-import error path, -provider lifecycle, and the path-safety guards — none of which need a live box. +provider lifecycle, the path-safety guards, and the warm pool lifecycle — none of +which need a live box. """ from __future__ import annotations +import asyncio import sys +import threading +import time import types import pytest @@ -14,15 +17,80 @@ from deerflow.community.boxlite.box import BoxliteBox from deerflow.community.boxlite.provider import BoxliteProvider, _import_simplebox +# ── Fake BoxLite SDK ────────────────────────────────────────────────── + + +class _FakeBox: + """A fake SimpleBox that records lifecycle calls without starting real VMs.""" + + def __init__(self, *, image=None, name=None, memory_mib=None, cpus=None, **kwargs): + self.id = name or "auto-gen-id" + self.name = name + self._image = image + self._started = False + self._stopped = False + self._exec_history: list[tuple] = [] + + async def start(self): + self._started = True + + async def exec(self, *argv, env=None, timeout=None): + self._exec_history.append((argv, env, timeout)) + _FakeResult = type("_FakeResult", (), {"stdout": "", "stderr": "", "exit_code": 0}) + # Health check: box.execute_command("echo ok") → exec("sh", "-lc", "echo ok") + if len(argv) >= 3 and argv[0] == "sh" and argv[1] == "-lc" and argv[2] == "echo ok": + return type("_FakeResult", (), {"stdout": "ok\n", "stderr": "", "exit_code": 0})() + return _FakeResult() + + async def stop(self): + self._stopped = True + + +def _fake_run(coro, *, timeout=None): + """Sync runner that executes coroutines on a temporary event loop (no daemon thread).""" + return asyncio.run(coro) + + +# ── Config stub ─────────────────────────────────────────────────────── + + +def _stub_config(sandbox_attrs=None): + """Stub get_app_config to return a config with given sandbox attrs.""" + attrs = sandbox_attrs or {} + stub = types.SimpleNamespace(sandbox=types.SimpleNamespace(**attrs)) + return stub + def _no_boxlite(monkeypatch: pytest.MonkeyPatch) -> None: """Make ``import boxlite`` raise, regardless of whether it is installed.""" monkeypatch.setitem(sys.modules, "boxlite", None) +@pytest.fixture(autouse=True) +def _no_existing_boxlite_boxes(monkeypatch: pytest.MonkeyPatch) -> None: + """Keep provider startup reconciliation isolated from the real SDK state.""" + + class _EmptyRuntime: + def start(self): + return self + + def stop(self): + pass + + def list_info(self): + return [] + + class _EmptyBoxlite: + @staticmethod + def default(): + return _EmptyRuntime() + + monkeypatch.setattr("deerflow.community.boxlite.provider._import_sync_boxlite_runtime", lambda: _EmptyBoxlite) + + def test_import_simplebox_missing_raises_actionable(monkeypatch: pytest.MonkeyPatch) -> None: _no_boxlite(monkeypatch) - with pytest.raises(ImportError, match=r"pip install boxlite"): + with pytest.raises(ImportError, match=r"deerflow-harness\[boxlite\]"): _import_simplebox() @@ -34,7 +102,7 @@ def test_acquire_without_boxlite_raises_and_shuts_down_cleanly(monkeypatch: pyte provider = BoxliteProvider() try: - with pytest.raises(ImportError, match=r"pip install boxlite"): + with pytest.raises(ImportError, match=r"deerflow-harness\[boxlite\]"): provider.acquire("thread-1", user_id="u") finally: provider.shutdown() # must not raise even though no box was ever created @@ -70,3 +138,573 @@ def _fail_run(_coro: object) -> None: box = BoxliteBox("box-id", box=object(), run=_fail_run) with pytest.raises(ValueError, match=r"POSIX"): box.execute_command("echo hi", env={"BAD KEY": "x"}) + + +def test_execute_command_forwards_timeout_to_sdk_and_loop_runner() -> None: + """Command timeout must bound both BoxLite exec and the loop bridge future.""" + run_timeouts: list[float | None] = [] + + def _recording_run(coro, *, timeout=None): + run_timeouts.append(timeout) + return asyncio.run(coro) + + fake = _FakeBox(name="box-id") + box = BoxliteBox("box-id", box=fake, run=_recording_run) + + output = box.execute_command("echo ok", timeout=5) + + assert "ok" in output + assert fake._exec_history[-1] == (("sh", "-lc", "echo ok"), None, 5) + assert run_timeouts == [5] + + +def test_sandbox_id_deterministic(monkeypatch): + """_sandbox_id produces the same id for the same inputs.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + provider = BoxliteProvider() + id1 = provider._sandbox_id("thread-1", "user-a") + id2 = provider._sandbox_id("thread-1", "user-a") + assert id1 == id2 + assert len(id1) == 8 + + +def test_sandbox_id_different_users(monkeypatch): + """Different users produce different ids for the same thread.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + provider = BoxliteProvider() + id_a = provider._sandbox_id("thread-1", "user-a") + id_b = provider._sandbox_id("thread-1", "user-b") + assert id_a != id_b + + +def test_sandbox_id_different_threads(monkeypatch): + """Different threads produce different ids for the same user.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + provider = BoxliteProvider() + id_a = provider._sandbox_id("thread-1", "user-a") + id_b = provider._sandbox_id("thread-2", "user-a") + assert id_a != id_b + + +def test_idle_timeout_zero_is_preserved_and_disables_reaper(monkeypatch): + """idle_timeout=0 is a valid config value and disables the reaper thread.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config({"idle_timeout": 0}), + ) + + provider = BoxliteProvider() + + assert provider._config["idle_timeout"] == 0 + assert provider._idle_checker_thread is None + provider.shutdown() + + +def test_create_box_passes_prefixed_sandbox_id_as_name(monkeypatch): + """_create_box gives BoxLite a DeerFlow-owned name prefix.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + # Inject fake SimpleBox and fake loop runner + created_boxes = [] + + class _RecordingBox(_FakeBox): + def __init__(self, **kwargs): + super().__init__(**kwargs) + created_boxes.append(kwargs) + + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _RecordingBox, + ) + + provider = BoxliteProvider() + # Replace _loop.run with our sync runner + provider._loop.run = _fake_run + + box = provider._create_box("test-sandbox-id") + assert len(created_boxes) == 1 + assert created_boxes[0]["name"] == "deer-flow-boxlite-test-sandbox-id" + assert box.id == "test-sandbox-id" + + +def test_startup_reconciliation_adopts_prefixed_existing_boxes(monkeypatch): + """Existing DeerFlow-named BoxLite boxes are adopted into the warm pool.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + stopped: list[str] = [] + + class _NativeBox: + def stop(self): + stopped.append("adopted") + + class _Runtime: + def start(self): + return self + + def stop(self): + pass + + def list_info(self): + return [ + types.SimpleNamespace(name="deer-flow-boxlite-adopted"), + types.SimpleNamespace(name="unrelated-box"), + types.SimpleNamespace(name=None), + ] + + def get(self, name): + if name == "deer-flow-boxlite-adopted": + return _NativeBox() + raise AssertionError(f"unexpected box lookup: {name}") + + class _Boxlite: + @staticmethod + def default(): + return _Runtime() + + monkeypatch.setattr("deerflow.community.boxlite.provider._import_sync_boxlite_runtime", lambda: _Boxlite) + + provider = BoxliteProvider() + + assert list(provider._warm_pool) == ["adopted"] + adopted_box = provider._warm_pool["adopted"][0] + assert adopted_box.id == "adopted" + + provider.shutdown() + assert stopped == ["adopted"] + + +def test_release_parks_in_warm_pool(monkeypatch): + """After release, box is in warm pool, not destroyed.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + # Acquire a box + sid = provider.acquire("thread-1", user_id="u1") + + # Verify box is active + assert sid in provider._boxes + assert sid not in provider._warm_pool + + # Release + provider.release(sid) + + # Verify box is in warm pool, not active + assert sid not in provider._boxes + assert sid in provider._warm_pool + box, ts = provider._warm_pool[sid] + assert isinstance(box, BoxliteBox) + assert not box._box._stopped # VM not destroyed + + +def test_acquire_reclaims_from_warm_pool(monkeypatch): + """acquire reclaims a warm pool box for the same thread.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + # First acquire → create + sid1 = provider.acquire("thread-1", user_id="u1") + provider.release(sid1) + + # Second acquire → should reclaim from warm pool + sid2 = provider.acquire("thread-1", user_id="u1") + assert sid1 == sid2 # Same deterministic ID + assert sid2 in provider._boxes + assert sid2 not in provider._warm_pool + + +def test_acquire_different_threads_dont_reclaim_each_other(monkeypatch): + """Thread A's box can't be reclaimed by thread B.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + sid_a = provider.acquire("thread-a", user_id="u1") + provider.release(sid_a) + + # Thread B acquires — should NOT get thread A's box + sid_b = provider.acquire("thread-b", user_id="u1") + assert sid_b != sid_a # Different deterministic ID + assert sid_a in provider._warm_pool # A's box still in warm pool + assert sid_b in provider._boxes # B's box is new + + +def test_warm_pool_reclaim_failed_health_check_creates_new(monkeypatch): + """Dead warm pool box is evicted and a new one created.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + sid1 = provider.acquire("thread-1", user_id="u1") + provider.release(sid1) + assert sid1 in provider._warm_pool + + # Corrupt the warm pool box: close it so health check fails + box, _ = provider._warm_pool[sid1] + box.close() # Stop VM, marks _closed=True + + # Re-acquire — health check should fail on the dead box + # A new box is created with the same deterministic ID + sid2 = provider.acquire("thread-1", user_id="u1") + assert sid2 == sid1 # Same deterministic ID + assert sid2 in provider._boxes + + +def test_concurrent_same_thread_acquire_creates_one_box(monkeypatch): + """Concurrent acquires for one thread serialize before creating a named box.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + original_create_box = provider._create_box + create_started = threading.Event() + created: list[str] = [] + + def slow_create_box(sandbox_id: str) -> BoxliteBox: + create_started.set() + time.sleep(0.1) + created.append(sandbox_id) + return original_create_box(sandbox_id) + + provider._create_box = slow_create_box # type: ignore[method-assign] + results: list[str] = [] + + def acquire() -> None: + results.append(provider.acquire("thread-1", user_id="u1")) + + first = threading.Thread(target=acquire) + second = threading.Thread(target=acquire) + first.start() + assert create_started.wait(timeout=2) + second.start() + first.join(timeout=2) + second.join(timeout=2) + + assert len(results) == 2 + assert results[0] == results[1] + assert len(created) == 1 + assert results[0] in provider._boxes + provider.shutdown() + + +def test_release_during_shutdown_closes_instead_of_reparking(monkeypatch): + """release() must not park a VM after shutdown has begun.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + sid = provider.acquire("thread-1", user_id="u1") + box = provider._boxes[sid] + with provider._lock: + provider._shutdown_called = True + + provider.release(sid) + + assert sid not in provider._boxes + assert sid not in provider._warm_pool + assert box._closed + provider._loop.close() + + +def test_reset_parks_running_resources_for_later_cleanup(monkeypatch): + """reset() stops thread reuse but leaves VMs tracked for cleanup.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + sid_active = provider.acquire("thread-active", user_id="u1") + sid_warm = provider.acquire("thread-warm", user_id="u1") + provider.release(sid_warm) + active_box = provider._boxes[sid_active] + warm_box = provider._warm_pool[sid_warm][0] + checker_thread = provider._idle_checker_thread + loop_thread = provider._loop._thread + + provider.reset() + + assert provider._boxes == {} + assert provider._warm_pool[sid_active][0] is active_box + assert provider._warm_pool[sid_warm][0] is warm_box + assert provider._thread_boxes == {} + assert provider._acquire_locks == {} + assert not active_box._closed + assert not warm_box._closed + assert not provider._shutdown_called + assert not provider._idle_checker_stop.is_set() + assert checker_thread is not None + assert checker_thread.is_alive() + assert loop_thread.is_alive() + + provider.shutdown() + assert active_box._closed + assert warm_box._closed + assert provider._idle_checker_stop.is_set() + assert not checker_thread.is_alive() + + +def test_reset_parked_resources_are_reaped_after_idle_timeout(monkeypatch): + """VMs parked by reset remain visible to warm-pool idle cleanup.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + sid_active = provider.acquire("thread-active", user_id="u1") + sid_warm = provider.acquire("thread-warm", user_id="u1") + provider.release(sid_warm) + active_box = provider._boxes[sid_active] + warm_box = provider._warm_pool[sid_warm][0] + + provider.reset() + + provider._warm_pool[sid_active] = (active_box, time.time() - 9999) + provider._warm_pool[sid_warm] = (warm_box, time.time() - 9999) + provider._reap_expired_warm(idle_timeout=1) + + assert provider._warm_pool == {} + assert active_box._closed + assert warm_box._closed + provider.shutdown() + + +# ── Task 6: Idle reaper ─────────────────────────────────────────────── + + +def test_idle_reaper_destroys_expired_warm_boxes(monkeypatch): + """Idle reaper daemon destroys warm pool boxes that exceed the idle timeout.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + # Use a very short check interval so the reaper runs quickly + monkeypatch.setattr(BoxliteProvider, "IDLE_CHECK_INTERVAL", 0.1) + provider = BoxliteProvider() + provider._loop.run = _fake_run + + # Acquire and release a box into the warm pool + sid = provider.acquire("thread-1", user_id="u1") + provider.release(sid) + + assert sid in provider._warm_pool + + # Backdate the warm-pool timestamp so it appears long-expired + warm_box = provider._warm_pool[sid][0] + provider._warm_pool[sid] = (warm_box, time.time() - 9999) + + # Wait long enough for the reaper to detect and destroy it + time.sleep(0.3) + + # Box should be gone from warm pool and closed + assert sid not in provider._warm_pool + assert warm_box._closed + + provider.shutdown() + + +# ── Task 7: Replica enforcement ─────────────────────────────────────── + + +def test_replica_enforcement_evicts_oldest_warm(monkeypatch): + """When warm pool exceeds replica limit, the oldest box is evicted.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config({"replicas": 2}), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + # Fill warm pool with 2 boxes from different threads + sid_a = provider.acquire("thread-a", user_id="u1") + provider.release(sid_a) + + sid_b = provider.acquire("thread-b", user_id="u1") + provider.release(sid_b) + + assert len(provider._warm_pool) == 2 + assert sid_a in provider._warm_pool + assert sid_b in provider._warm_pool + + # Make sid_a definitely older by backdating its timestamp + box_a = provider._warm_pool[sid_a][0] + provider._warm_pool[sid_a] = (box_a, time.time() - 100) + # Refresh sid_b's timestamp so it's newer + box_b = provider._warm_pool[sid_b][0] + provider._warm_pool[sid_b] = (box_b, time.time()) + + # Acquiring a third thread triggers replica enforcement: + # warm pool count (2) >= replicas (2) → evict oldest (sid_a) + sid_c = provider.acquire("thread-c", user_id="u1") + + # Oldest (sid_a) should be evicted (gone from warm pool, closed) + assert sid_a not in provider._warm_pool + assert box_a._closed + # Newer (sid_b) should remain in warm pool + assert sid_b in provider._warm_pool + # New box (sid_c) should be active + assert sid_c in provider._boxes + assert sid_c not in provider._warm_pool + + provider.shutdown() + + +def test_replica_enforcement_counts_active_and_warm(monkeypatch): + """replicas caps active + warm boxes, not warm boxes alone.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config({"replicas": 2}), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + sid_active = provider.acquire("thread-active", user_id="u1") + sid_warm = provider.acquire("thread-warm", user_id="u1") + provider.release(sid_warm) + warm_box = provider._warm_pool[sid_warm][0] + + sid_new = provider.acquire("thread-new", user_id="u1") + + assert sid_active in provider._boxes + assert sid_new in provider._boxes + assert sid_warm not in provider._warm_pool + assert warm_box._closed + provider.shutdown() + + +# ── Task 8: Shutdown and reset including warm pool ──────────────────── + + +def test_shutdown_stops_idle_reaper_and_destroys_all_boxes(monkeypatch): + """shutdown stops the idle reaper thread and destroys all active + warm boxes.""" + monkeypatch.setattr( + "deerflow.community.boxlite.provider.get_app_config", + lambda: _stub_config(), + ) + monkeypatch.setattr( + "deerflow.community.boxlite.provider._import_simplebox", + lambda: _FakeBox, + ) + + provider = BoxliteProvider() + provider._loop.run = _fake_run + + # Create one active box (thread-1) and one warm pool box (thread-2 released) + sid_active = provider.acquire("thread-1", user_id="u1") + sid_warm = provider.acquire("thread-2", user_id="u1") + provider.release(sid_warm) + + assert sid_active in provider._boxes + assert sid_warm in provider._warm_pool + + # Get box references before shutdown + box_active = provider._boxes[sid_active] + box_warm = provider._warm_pool[sid_warm][0] + + # Remember the idle checker thread + checker_thread = provider._idle_checker_thread + + provider.shutdown() + + # Idle checker should be stopped + assert provider._idle_checker_stop.is_set() + assert checker_thread is not None + assert not checker_thread.is_alive() + + # All boxes (active + warm) should be closed + assert box_active._closed + assert box_warm._closed + + # All collections should be empty + assert len(provider._boxes) == 0 + assert len(provider._warm_pool) == 0 + assert len(provider._thread_boxes) == 0 diff --git a/backend/tests/test_clarification_middleware.py b/backend/tests/test_clarification_middleware.py index e729ac4ef97..040081e122b 100644 --- a/backend/tests/test_clarification_middleware.py +++ b/backend/tests/test_clarification_middleware.py @@ -122,6 +122,99 @@ def test_json_string_with_mixed_types(self, middleware): assert "4. None" in result +class TestHumanInputPayload: + """Tests for structured human input request payloads.""" + + def test_payload_with_native_options(self, middleware): + payload = middleware._build_human_input_payload( + { + "question": "Which environment should I deploy to?", + "clarification_type": "approach_choice", + "context": "Need the target environment for config.", + "options": ["development", "staging", "production"], + }, + tool_call_id="call-abc", + request_id="clarification:call-abc", + ) + + assert payload == { + "version": 1, + "kind": "human_input_request", + "source": "ask_clarification", + "request_id": "clarification:call-abc", + "tool_call_id": "call-abc", + "clarification_type": "approach_choice", + "question": "Which environment should I deploy to?", + "context": "Need the target environment for config.", + "input_mode": "choice_with_other", + "options": [ + {"id": "option-1", "label": "development", "value": "development"}, + {"id": "option-2", "label": "staging", "value": "staging"}, + {"id": "option-3", "label": "production", "value": "production"}, + ], + } + + def test_payload_with_json_string_options(self, middleware): + payload = middleware._build_human_input_payload( + { + "question": "Pick one", + "clarification_type": "approach_choice", + "options": json.dumps(["Option A", 2, True, None]), + }, + tool_call_id="call-abc", + request_id="clarification:call-abc", + ) + + assert payload["input_mode"] == "choice_with_other" + assert payload["options"] == [ + {"id": "option-1", "label": "Option A", "value": "Option A"}, + {"id": "option-2", "label": "2", "value": "2"}, + {"id": "option-3", "label": "True", "value": "True"}, + {"id": "option-4", "label": "None", "value": "None"}, + ] + + def test_payload_with_plain_string_option(self, middleware): + payload = middleware._build_human_input_payload( + { + "question": "Pick one", + "clarification_type": "approach_choice", + "options": "just one option", + }, + tool_call_id="call-abc", + request_id="clarification:call-abc", + ) + + assert payload["input_mode"] == "choice_with_other" + assert payload["options"] == [{"id": "option-1", "label": "just one option", "value": "just one option"}] + + def test_payload_without_options_is_free_text(self, middleware): + payload = middleware._build_human_input_payload( + { + "question": "Tell me more", + "clarification_type": "missing_info", + "options": None, + }, + tool_call_id="call-abc", + request_id="clarification:call-abc", + ) + + assert payload["input_mode"] == "free_text" + assert "options" not in payload + + def test_payload_missing_options_is_free_text(self, middleware): + payload = middleware._build_human_input_payload( + { + "question": "Tell me more", + "clarification_type": "missing_info", + }, + tool_call_id="call-abc", + request_id="clarification:call-abc", + ) + + assert payload["input_mode"] == "free_text" + assert "options" not in payload + + class TestClarificationCommandIdempotency: """Clarification tool-call retries should not duplicate messages in state.""" @@ -147,12 +240,41 @@ def test_repeated_tool_call_uses_stable_message_id(self, middleware): assert first_message.id == "clarification:call-clarify-1" assert second_message.id == first_message.id assert second_message.tool_call_id == first_message.tool_call_id + assert first_message.artifact["human_input"]["request_id"] == "clarification:call-clarify-1" + assert first_message.artifact["human_input"]["tool_call_id"] == "call-clarify-1" + assert first_message.artifact["human_input"]["clarification_type"] == "approach_choice" + assert first_message.artifact["human_input"]["input_mode"] == "choice_with_other" merged = add_messages(add_messages([], [first_message]), [second_message]) assert len(merged) == 1 assert merged[0].id == "clarification:call-clarify-1" assert merged[0].content == first_message.content + assert merged[0].artifact == first_message.artifact + + def test_tool_message_model_dump_preserves_human_input_artifact(self, middleware): + request = SimpleNamespace( + tool_call={ + "name": "ask_clarification", + "id": "call-clarify-1", + "args": { + "question": "Which environment should I use?", + "clarification_type": "approach_choice", + "options": ["dev", "prod"], + }, + } + ) + + result = middleware.wrap_tool_call(request, lambda _req: pytest.fail("handler should not be called")) + message = result.update["messages"][0] + dumped = message.model_dump() + + assert dumped["artifact"]["human_input"]["request_id"] == "clarification:call-clarify-1" + assert dumped["artifact"]["human_input"]["options"] == [ + {"id": "option-1", "label": "dev", "value": "dev"}, + {"id": "option-2", "label": "prod", "value": "prod"}, + ] + assert "Which environment should I use?" in dumped["content"] class TestClarificationDisabled: @@ -178,6 +300,7 @@ def test_disabled_returns_toolmessage_not_command(self, middleware): assert isinstance(result, ToolMessage) assert result.tool_call_id == "call-clarify-1" + assert result.artifact is None def test_disabled_message_tells_agent_to_proceed(self, middleware): request = self._request(runtime_context={"disable_clarification": True}) diff --git a/backend/tests/test_client.py b/backend/tests/test_client.py index 0be07e78bde..823baaaabc0 100644 --- a/backend/tests/test_client.py +++ b/backend/tests/test_client.py @@ -2617,6 +2617,11 @@ def test_get_memory_config(self, client): mem_cfg.injection_enabled = True mem_cfg.max_injection_tokens = 2000 mem_cfg.token_counting = "tiktoken" + mem_cfg.staleness_review_enabled = True + mem_cfg.staleness_age_days = 90 + mem_cfg.staleness_min_candidates = 3 + mem_cfg.staleness_max_removals_per_cycle = 10 + mem_cfg.staleness_protected_categories = ["correction"] with patch("deerflow.config.memory_config.get_memory_config", return_value=mem_cfg): result = client.get_memory_config() @@ -2636,6 +2641,11 @@ def test_get_memory_status(self, client): mem_cfg.injection_enabled = True mem_cfg.max_injection_tokens = 2000 mem_cfg.token_counting = "tiktoken" + mem_cfg.staleness_review_enabled = True + mem_cfg.staleness_age_days = 90 + mem_cfg.staleness_min_candidates = 3 + mem_cfg.staleness_max_removals_per_cycle = 10 + mem_cfg.staleness_protected_categories = ["correction"] memory_data = { "version": "1.0", diff --git a/backend/tests/test_delegation_ledger.py b/backend/tests/test_delegation_ledger.py index 28eed63de39..6441c23aa8b 100644 --- a/backend/tests/test_delegation_ledger.py +++ b/backend/tests/test_delegation_ledger.py @@ -181,6 +181,34 @@ def test_structured_error_metadata_wins_over_misleading_content(self): assert out[0]["status"] == "failed" assert out[0]["result_brief"] == "structured boom" + def test_max_turns_reached_task_carries_partial_result_in_brief(self): + """#3875 Phase 2: a turn-capped delegation is result-bearing like + ``completed``, so the recovered partial result lands in + ``result_brief`` (preferred over the cap notice on ``error``) — the + lead's durable context shows the work produced before the budget ran + out, not just the cap reason.""" + msgs = [ + _ai_task_call("call_capped", "deep research"), + ToolMessage( + content="Task reached max turns. Reached max_turns=150 Partial result: investigated 3 of 5 sources", + tool_call_id="call_capped", + id="tm_capped", + additional_kwargs={ + "subagent_status": "max_turns_reached", + "subagent_result_brief": "investigated 3 of 5 sources", + "subagent_result_sha256": "a" * 64, + "subagent_error": "Reached max_turns=150", + }, + ), + ] + + out = extract_delegations(msgs) + + assert out[0]["status"] == "max_turns_reached" + # result_brief wins over error, so the partial work is what the lead sees. + assert "investigated 3 of 5 sources" in out[0]["result_brief"] + assert out[0]["result_sha256"] == "a" * 64 + def test_terminal_looking_content_without_structured_metadata_keeps_dispatch_in_progress(self): msgs = [ _ai_task_call("call_2", "bad task"), diff --git a/backend/tests/test_gateway_runtime_cleanup.py b/backend/tests/test_gateway_runtime_cleanup.py index 7574124c0c1..0c17368f4ed 100644 --- a/backend/tests/test_gateway_runtime_cleanup.py +++ b/backend/tests/test_gateway_runtime_cleanup.py @@ -43,6 +43,16 @@ def test_service_launchers_always_use_gateway_runtime(): assert "LANGGRAPH_REWRITE" not in content, path +def test_docker_dev_mounts_mutable_configs_through_project_directory(): + compose = _read("docker/docker-compose-dev.yaml") + + assert re.search(r"^\s*-\s*\.\./:/app/project(?:\:\S+)?\s*$", compose, re.M) + assert not re.search(r"^\s*-\s*[^\n#]*config\.yaml\s*:\s*[^\n#]*$", compose, re.M) + assert not re.search(r"^\s*-\s*[^\n#]*extensions_config\.json\s*:\s*[^\n#]*$", compose, re.M) + assert "DEER_FLOW_CONFIG_PATH=/app/project/config.yaml" in compose + assert "DEER_FLOW_EXTENSIONS_CONFIG_PATH=/app/project/extensions_config.json" in compose + + def test_local_dev_gateway_reload_excludes_runtime_state_with_absolute_dirs(): serve_sh = _read("scripts/serve.sh") diff --git a/backend/tests/test_gateway_services.py b/backend/tests/test_gateway_services.py index 385f1967a33..10ffc3db25a 100644 --- a/backend/tests/test_gateway_services.py +++ b/backend/tests/test_gateway_services.py @@ -132,6 +132,37 @@ def test_normalize_input_preserves_additional_kwargs_and_id(): assert msg.additional_kwargs == {"files": files, "custom": "keep-me"} +def test_normalize_input_preserves_human_input_response_metadata(): + from langchain_core.messages import HumanMessage + + from app.gateway.services import normalize_input + + response = { + "version": 1, + "kind": "human_input_response", + "source": "ask_clarification", + "request_id": "clarification:call-abc", + "response_kind": "option", + "option_id": "option-2", + "value": "staging", + } + result = normalize_input( + { + "messages": [ + { + "type": "human", + "content": [{"type": "text", "text": "For your clarification, my answer is: staging"}], + "additional_kwargs": {"human_input_response": response}, + } + ] + } + ) + + msg = result["messages"][0] + assert isinstance(msg, HumanMessage) + assert msg.additional_kwargs["human_input_response"] == response + + def test_normalize_input_passes_through_basemessage_instances(): from langchain_core.messages import HumanMessage diff --git a/backend/tests/test_harness_packaging.py b/backend/tests/test_harness_packaging.py new file mode 100644 index 00000000000..1e80ae5ae89 --- /dev/null +++ b/backend/tests/test_harness_packaging.py @@ -0,0 +1,16 @@ +from __future__ import annotations + +import tomllib +from pathlib import Path + + +def test_boxlite_is_optional_harness_dependency() -> None: + """BoxLite should not make core harness installs platform-dependent.""" + pyproject_path = Path(__file__).resolve().parents[1] / "packages" / "harness" / "pyproject.toml" + pyproject = tomllib.loads(pyproject_path.read_text(encoding="utf-8")) + + core_dependencies = pyproject["project"]["dependencies"] + optional_dependencies = pyproject["project"]["optional-dependencies"] + + assert not any(dep.startswith("boxlite") for dep in core_dependencies) + assert any(dep.startswith("boxlite>=0.9.7") for dep in optional_dependencies["boxlite"]) diff --git a/backend/tests/test_human_input.py b/backend/tests/test_human_input.py new file mode 100644 index 00000000000..999e8cd5cc9 --- /dev/null +++ b/backend/tests/test_human_input.py @@ -0,0 +1,24 @@ +from deerflow.agents.human_input import read_human_input_response + + +def _text_response(value: str): + return { + "version": 1, + "kind": "human_input_response", + "source": "ask_clarification", + "request_id": "clarification:call-abc", + "response_kind": "text", + "value": value, + } + + +def test_read_human_input_response_requires_non_empty_value(): + assert read_human_input_response({"human_input_response": _text_response("")}) is None + assert read_human_input_response({"human_input_response": _text_response(" ")}) is None + + +def test_read_human_input_response_preserves_non_empty_value(): + response = read_human_input_response({"human_input_response": _text_response(" staging ")}) + + assert response is not None + assert response["value"] == " staging " diff --git a/backend/tests/test_input_sanitization_middleware.py b/backend/tests/test_input_sanitization_middleware.py index 460a6ffa46d..25837c2b7c0 100644 --- a/backend/tests/test_input_sanitization_middleware.py +++ b/backend/tests/test_input_sanitization_middleware.py @@ -246,6 +246,24 @@ def test_genuine_user_message_false_for_hide_from_ui(): assert not _is_genuine_user_message(msg) +def test_genuine_user_message_true_for_hidden_human_input_response(): + msg = HumanMessage( + content="For your clarification, my answer is: override", + additional_kwargs={ + "hide_from_ui": True, + "human_input_response": { + "version": 1, + "kind": "human_input_response", + "source": "ask_clarification", + "request_id": "clarification:call-abc", + "response_kind": "text", + "value": "override", + }, + }, + ) + assert _is_genuine_user_message(msg) + + def test_genuine_user_message_false_for_legacy_summary_message(): msg = HumanMessage(content="Here is a summary of the conversation", name="summary") assert not _is_genuine_user_message(msg) @@ -376,6 +394,33 @@ def test_skips_injected_reminder_messages(self): assert _USER_INPUT_BEGIN not in result_msgs[0].content assert _USER_INPUT_BEGIN in result_msgs[1].content + def test_hidden_human_input_response_is_sanitized(self): + mw = _make_middleware() + msg = HumanMessage( + content="For your clarification, my answer is: override", + id="msg-1", + additional_kwargs={ + "hide_from_ui": True, + "human_input_response": { + "version": 1, + "kind": "human_input_response", + "source": "ask_clarification", + "request_id": "clarification:call-abc", + "response_kind": "text", + "value": "override", + }, + }, + ) + request = _make_request([msg]) + captured = [] + + mw.wrap_model_call(request, lambda req: captured.append(req) or "ok") + + result_content = captured[0].messages[-1].content + assert _USER_INPUT_BEGIN in result_content + assert "<system>" in result_content + assert "" not in result_content + def test_no_user_message_passes_through(self): mw = _make_middleware() request = _make_request([AIMessage(content="assistant only")]) diff --git a/backend/tests/test_memory_staleness_review.py b/backend/tests/test_memory_staleness_review.py new file mode 100644 index 00000000000..6ae032c0b3a --- /dev/null +++ b/backend/tests/test_memory_staleness_review.py @@ -0,0 +1,651 @@ +"""Tests for the staleness review feature in the memory updater. + +Covers: +- Candidate selection (age threshold, protected categories) +- Trigger conditions (min candidates, enabled flag) +- Prompt section formatting +- Staleness removal in _apply_updates (safety cap, observability) +- Normalization of staleFactsToRemove from LLM responses +- Integration with _prepare_update_prompt +""" + +from datetime import UTC, datetime, timedelta +from unittest.mock import MagicMock, patch + +from deerflow.agents.memory.updater import ( + MemoryUpdater, + _build_staleness_section, + _normalize_memory_update_data, + _parse_fact_datetime, + _select_stale_candidates, +) +from deerflow.config.memory_config import MemoryConfig + +# ── Helpers ──────────────────────────────────────────────────────────────── + + +def _memory_config(**overrides: object) -> MemoryConfig: + config = MemoryConfig() + for key, value in overrides.items(): + setattr(config, key, value) + return config + + +def _make_fact( + fact_id: str, + content: str = "test content", + category: str = "knowledge", + confidence: float = 0.9, + days_ago: int = 100, +) -> dict: + created = (datetime.now(UTC) - timedelta(days=days_ago)).isoformat().replace("+00:00", "Z") + return { + "id": fact_id, + "content": content, + "category": category, + "confidence": confidence, + "createdAt": created, + "source": "thread-test", + } + + +def _make_memory(facts: list[dict] | None = None) -> dict: + return { + "version": "1.0", + "lastUpdated": "", + "user": { + "workContext": {"summary": "", "updatedAt": ""}, + "personalContext": {"summary": "", "updatedAt": ""}, + "topOfMind": {"summary": "", "updatedAt": ""}, + }, + "history": { + "recentMonths": {"summary": "", "updatedAt": ""}, + "earlierContext": {"summary": "", "updatedAt": ""}, + "longTermBackground": {"summary": "", "updatedAt": ""}, + }, + "facts": facts or [], + } + + +# ── _parse_fact_datetime ────────────────────────────────────────────────── + + +class TestParseFactDatetime: + def test_z_suffix(self): + result = _parse_fact_datetime("2025-06-01T12:00:00Z") + assert result is not None + assert result.year == 2025 + assert result.month == 6 + + def test_offset_format(self): + result = _parse_fact_datetime("2025-06-01T12:00:00+00:00") + assert result is not None + assert result.year == 2025 + + def test_empty_string(self): + assert _parse_fact_datetime("") is None + + def test_invalid_format(self): + assert _parse_fact_datetime("not-a-date") is None + + def test_naive_datetime_gets_utc(self): + """Naive datetime (no tzinfo) should be treated as UTC, not cause TypeError.""" + result = _parse_fact_datetime("2025-06-01T12:00:00") + assert result is not None + assert result.tzinfo is not None + assert result.utcoffset().total_seconds() == 0 + + +# ── _select_stale_candidates ────────────────────────────────────────────── + + +class TestSelectStaleCandidates: + def test_old_facts_selected(self): + memory = _make_memory( + [ + _make_fact("fact_old", days_ago=100), + _make_fact("fact_new", days_ago=10), + ] + ) + config = _memory_config(staleness_age_days=90) + candidates = _select_stale_candidates(memory, config) + assert len(candidates) == 1 + assert candidates[0]["id"] == "fact_old" + + def test_protected_category_excluded(self): + memory = _make_memory( + [ + _make_fact("fact_correction", category="correction", days_ago=200), + _make_fact("fact_knowledge", category="knowledge", days_ago=200), + ] + ) + config = _memory_config(staleness_age_days=90, staleness_protected_categories=["correction"]) + candidates = _select_stale_candidates(memory, config) + assert len(candidates) == 1 + assert candidates[0]["id"] == "fact_knowledge" + + def test_custom_protected_categories(self): + memory = _make_memory( + [ + _make_fact("fact_goal", category="goal", days_ago=200), + ] + ) + config = _memory_config(staleness_age_days=90, staleness_protected_categories=["goal"]) + candidates = _select_stale_candidates(memory, config) + assert len(candidates) == 0 + + def test_no_facts(self): + memory = _make_memory([]) + config = _memory_config(staleness_age_days=90) + assert _select_stale_candidates(memory, config) == [] + + def test_all_recent(self): + memory = _make_memory( + [ + _make_fact("fact_a", days_ago=10), + _make_fact("fact_b", days_ago=30), + ] + ) + config = _memory_config(staleness_age_days=90) + assert _select_stale_candidates(memory, config) == [] + + +# ── Trigger conditions via _select_stale_candidates + config ───────────── + + +class TestStalenessTriggerConditions: + """The old _should_run_staleness_review was removed; trigger logic is now + inlined in _prepare_update_prompt. We verify the gating conditions here + through _select_stale_candidates + config flags directly.""" + + def test_disabled_means_no_section(self): + memory = _make_memory([_make_fact(f"f{i}", days_ago=100) for i in range(5)]) + config = _memory_config(staleness_review_enabled=False, staleness_age_days=90, staleness_min_candidates=3) + candidates = _select_stale_candidates(memory, config) + # Even though candidates exist, the caller checks enabled flag first + assert config.staleness_review_enabled is False + assert len(candidates) >= config.staleness_min_candidates + + def test_below_min_candidates(self): + memory = _make_memory([_make_fact("fact_only", days_ago=100)]) + config = _memory_config(staleness_review_enabled=True, staleness_age_days=90, staleness_min_candidates=3) + candidates = _select_stale_candidates(memory, config) + assert len(candidates) < config.staleness_min_candidates + + def test_at_min_candidates(self): + memory = _make_memory([_make_fact(f"fact_{i}", days_ago=100) for i in range(3)]) + config = _memory_config(staleness_review_enabled=True, staleness_age_days=90, staleness_min_candidates=3) + candidates = _select_stale_candidates(memory, config) + assert len(candidates) >= config.staleness_min_candidates + + def test_above_min_candidates(self): + memory = _make_memory([_make_fact(f"fact_{i}", days_ago=100) for i in range(10)]) + config = _memory_config(staleness_review_enabled=True, staleness_age_days=90, staleness_min_candidates=3) + candidates = _select_stale_candidates(memory, config) + assert len(candidates) >= config.staleness_min_candidates + + +# ── _build_staleness_section ────────────────────────────────────────────── + + +class TestBuildStalenessSection: + def test_empty_candidates(self): + assert _build_staleness_section([], 90) == "" + + def test_includes_fact_details(self): + candidates = [ + _make_fact("fact_vue", "User uses Vue.js", "knowledge", 0.95, days_ago=120), + ] + section = _build_staleness_section(candidates, 90) + assert "fact_vue" in section + assert "User uses Vue.js" in section + assert "0.95" in section + assert "90 days" in section + + def test_multiple_facts(self): + candidates = [ + _make_fact("fact_a", "Fact A", "knowledge", 0.9, days_ago=100), + _make_fact("fact_b", "Fact B", "preference", 0.8, days_ago=150), + ] + section = _build_staleness_section(candidates, 90) + assert "fact_a" in section + assert "fact_b" in section + assert "" in section + + +# ── _apply_updates with staleness removals ───────────────────────────────── + + +class TestApplyUpdatesStaleness: + def test_stale_facts_removed(self): + updater = MemoryUpdater() + current_memory = _make_memory( + [ + _make_fact("fact_keep", "User knows Python", days_ago=100), + _make_fact("fact_stale", "User uses Vue.js", days_ago=120), + ] + ) + update_data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [ + {"id": "fact_stale", "reason": "User switched to React"}, + ], + } + + with patch( + "deerflow.agents.memory.updater.get_memory_config", + return_value=_memory_config(max_facts=100, staleness_max_removals_per_cycle=10), + ): + result = updater._apply_updates(current_memory, update_data) + + assert len(result["facts"]) == 1 + assert result["facts"][0]["id"] == "fact_keep" + + def test_safety_cap_limits_removals(self): + updater = MemoryUpdater() + # 5 stale facts, but cap is 2 → only 2 lowest-confidence should be removed + current_memory = _make_memory( + [ + _make_fact("fact_high", confidence=0.95, days_ago=100), + _make_fact("fact_mid", confidence=0.80, days_ago=100), + _make_fact("fact_low1", confidence=0.70, days_ago=100), + _make_fact("fact_low2", confidence=0.65, days_ago=100), + _make_fact("fact_low3", confidence=0.60, days_ago=100), + ] + ) + update_data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [ + {"id": "fact_high", "reason": "outdated"}, + {"id": "fact_mid", "reason": "outdated"}, + {"id": "fact_low1", "reason": "outdated"}, + {"id": "fact_low2", "reason": "outdated"}, + {"id": "fact_low3", "reason": "outdated"}, + ], + } + + with patch( + "deerflow.agents.memory.updater.get_memory_config", + return_value=_memory_config(max_facts=100, staleness_max_removals_per_cycle=2), + ): + result = updater._apply_updates(current_memory, update_data) + + # 5 - 2 = 3 facts remain; the 2 lowest-confidence removed + assert len(result["facts"]) == 3 + remaining_ids = {f["id"] for f in result["facts"]} + assert "fact_high" in remaining_ids + assert "fact_mid" in remaining_ids + assert "fact_low1" in remaining_ids + + def test_empty_stale_removals_no_effect(self): + updater = MemoryUpdater() + current_memory = _make_memory( + [ + _make_fact("fact_a", days_ago=100), + ] + ) + update_data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [], + } + + with patch( + "deerflow.agents.memory.updater.get_memory_config", + return_value=_memory_config(max_facts=100), + ): + result = updater._apply_updates(current_memory, update_data) + + assert len(result["facts"]) == 1 + + def test_missing_stale_removals_key_no_effect(self): + """When LLM doesn't return staleFactsToRemove, existing behavior is preserved.""" + updater = MemoryUpdater() + current_memory = _make_memory([_make_fact("fact_a")]) + update_data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + # no staleFactsToRemove key + } + + with patch( + "deerflow.agents.memory.updater.get_memory_config", + return_value=_memory_config(max_facts=100), + ): + result = updater._apply_updates(current_memory, update_data) + + assert len(result["facts"]) == 1 + + def test_contradiction_and_staleness_removals_combined(self): + """Both factsToRemove and staleFactsToRemove work together.""" + updater = MemoryUpdater() + current_memory = _make_memory( + [ + _make_fact("fact_keep", days_ago=10), + _make_fact("fact_contradicted", days_ago=10), + _make_fact("fact_stale", days_ago=200), + ] + ) + update_data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": ["fact_contradicted"], + "staleFactsToRemove": [{"id": "fact_stale", "reason": "old"}], + } + + with patch( + "deerflow.agents.memory.updater.get_memory_config", + return_value=_memory_config(max_facts=100, staleness_max_removals_per_cycle=10), + ): + result = updater._apply_updates(current_memory, update_data) + + assert len(result["facts"]) == 1 + assert result["facts"][0]["id"] == "fact_keep" + + def test_protected_category_fact_refused_at_apply(self): + """Regression: LLM hallucinating a correction-category fact id in + staleFactsToRemove must be silently rejected at the apply layer, + even though it appears in the serialized prompt JSON.""" + updater = MemoryUpdater() + current_memory = _make_memory( + [ + _make_fact("fact_stale", category="knowledge", days_ago=200), + _make_fact("fact_correction", category="correction", days_ago=200), + ] + ) + update_data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [ + {"id": "fact_stale", "reason": "outdated"}, + {"id": "fact_correction", "reason": "LLM slip"}, + ], + } + + with patch( + "deerflow.agents.memory.updater.get_memory_config", + return_value=_memory_config( + max_facts=100, + staleness_review_enabled=True, + staleness_age_days=90, + staleness_min_candidates=1, + staleness_max_removals_per_cycle=10, + staleness_protected_categories=["correction"], + ), + ): + result = updater._apply_updates(current_memory, update_data) + + # fact_stale removed, fact_correction kept (protected) + assert len(result["facts"]) == 1 + assert result["facts"][0]["id"] == "fact_correction" + + def test_non_aged_fact_refused_at_apply(self): + """Regression: LLM returning a fresh (non-aged) fact id in + staleFactsToRemove must be silently rejected.""" + updater = MemoryUpdater() + current_memory = _make_memory( + [ + _make_fact("fact_stale", days_ago=200), + _make_fact("fact_fresh", days_ago=10), + ] + ) + update_data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [ + {"id": "fact_stale", "reason": "outdated"}, + {"id": "fact_fresh", "reason": "LLM hallucination"}, + ], + } + + with patch( + "deerflow.agents.memory.updater.get_memory_config", + return_value=_memory_config( + max_facts=100, + staleness_review_enabled=True, + staleness_age_days=90, + staleness_min_candidates=1, + staleness_max_removals_per_cycle=10, + staleness_protected_categories=["correction"], + ), + ): + result = updater._apply_updates(current_memory, update_data) + + # fact_stale removed, fact_fresh kept (not in candidate set) + assert len(result["facts"]) == 1 + assert result["facts"][0]["id"] == "fact_fresh" + + def test_guardrail_runs_when_staleness_review_disabled(self): + """Regression: guardrail must reject invalid ids even when + staleness_review_enabled=False, so the protection is independent + of the feature flag and model behavior.""" + updater = MemoryUpdater() + current_memory = _make_memory( + [ + _make_fact("fact_stale", days_ago=200), + _make_fact("fact_fresh", days_ago=5), + ] + ) + update_data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [ + {"id": "fact_stale", "reason": "LLM hallucination"}, + {"id": "fact_fresh", "reason": "LLM hallucination"}, + ], + } + + with patch( + "deerflow.agents.memory.updater.get_memory_config", + return_value=_memory_config( + max_facts=100, + staleness_review_enabled=False, + staleness_age_days=90, + staleness_min_candidates=3, + staleness_max_removals_per_cycle=10, + staleness_protected_categories=["correction"], + ), + ): + result = updater._apply_updates(current_memory, update_data) + + # Guardrail runs regardless of feature flag: + # fact_stale is a valid candidate (200 days old) → removed + # fact_fresh is not a candidate (5 days old) → kept + assert len(result["facts"]) == 1 + assert result["facts"][0]["id"] == "fact_fresh" + + +# ── _normalize_memory_update_data with staleFactsToRemove ───────────────── + + +class TestNormalizeStaleFactsToRemove: + def test_valid_entries(self): + data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [ + {"id": "fact_a", "reason": "User moved offices"}, + {"id": "fact_b", "reason": "Tech stack changed"}, + ], + } + result = _normalize_memory_update_data(data) + assert len(result["staleFactsToRemove"]) == 2 + assert result["staleFactsToRemove"][0]["id"] == "fact_a" + assert result["staleFactsToRemove"][1]["reason"] == "Tech stack changed" + + def test_missing_key(self): + data = {"user": {}, "history": {}, "newFacts": [], "factsToRemove": []} + result = _normalize_memory_update_data(data) + assert result["staleFactsToRemove"] == [] + + def test_non_list_ignored(self): + data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": "not a list", + } + result = _normalize_memory_update_data(data) + assert result["staleFactsToRemove"] == [] + + def test_non_dict_entries_skipped(self): + data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": ["just a string", 42, {"id": "fact_ok", "reason": "valid"}], + } + result = _normalize_memory_update_data(data) + assert len(result["staleFactsToRemove"]) == 1 + assert result["staleFactsToRemove"][0]["id"] == "fact_ok" + + def test_empty_id_skipped(self): + data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [{"id": "", "reason": "no id"}], + } + result = _normalize_memory_update_data(data) + assert result["staleFactsToRemove"] == [] + + def test_non_string_reason_defaulted(self): + data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [{"id": "fact_a", "reason": 123}], + } + result = _normalize_memory_update_data(data) + assert result["staleFactsToRemove"][0]["reason"] == "" + + def test_missing_reason_defaulted(self): + data = { + "user": {}, + "history": {}, + "newFacts": [], + "factsToRemove": [], + "staleFactsToRemove": [{"id": "fact_a"}], + } + result = _normalize_memory_update_data(data) + assert result["staleFactsToRemove"][0]["reason"] == "" + + +# ── Integration: _prepare_update_prompt ──────────────────────────────────── + + +class TestPrepareUpdatePromptStaleness: + def test_staleness_section_included_when_triggered(self): + updater = MemoryUpdater() + old_facts = [_make_fact(f"fact_{i}", days_ago=100) for i in range(5)] + memory = _make_memory(old_facts) + + msg = MagicMock() + msg.type = "human" + msg.content = "Hello, I'm using React now" + + config = _memory_config( + enabled=True, + staleness_review_enabled=True, + staleness_age_days=90, + staleness_min_candidates=3, + ) + + with ( + patch("deerflow.agents.memory.updater.get_memory_config", return_value=config), + patch("deerflow.agents.memory.updater.get_memory_data", return_value=memory), + ): + result = updater._prepare_update_prompt( + messages=[msg], + agent_name=None, + correction_detected=False, + reinforcement_detected=False, + ) + + assert result is not None + _, prompt = result + assert "Staleness Review" in prompt + assert "" in prompt + + def test_staleness_section_omitted_when_not_triggered(self): + updater = MemoryUpdater() + memory = _make_memory([]) # no facts at all + + msg = MagicMock() + msg.type = "human" + msg.content = "Hello" + + config = _memory_config( + enabled=True, + staleness_review_enabled=True, + staleness_age_days=90, + staleness_min_candidates=3, + ) + + with ( + patch("deerflow.agents.memory.updater.get_memory_config", return_value=config), + patch("deerflow.agents.memory.updater.get_memory_data", return_value=memory), + ): + result = updater._prepare_update_prompt( + messages=[msg], + agent_name=None, + correction_detected=False, + reinforcement_detected=False, + ) + + assert result is not None + _, prompt = result + assert "Staleness Review" not in prompt + assert "" not in prompt + + def test_staleness_section_omitted_when_disabled(self): + updater = MemoryUpdater() + old_facts = [_make_fact(f"fact_{i}", days_ago=200) for i in range(10)] + memory = _make_memory(old_facts) + + msg = MagicMock() + msg.type = "human" + msg.content = "Hello" + + config = _memory_config( + enabled=True, + staleness_review_enabled=False, + ) + + with ( + patch("deerflow.agents.memory.updater.get_memory_config", return_value=config), + patch("deerflow.agents.memory.updater.get_memory_data", return_value=memory), + ): + result = updater._prepare_update_prompt( + messages=[msg], + agent_name=None, + correction_detected=False, + reinforcement_detected=False, + ) + + assert result is not None + _, prompt = result + assert "Staleness Review" not in prompt diff --git a/backend/tests/test_memory_upload_filtering.py b/backend/tests/test_memory_upload_filtering.py index 803d40fde1e..463d1fe2a19 100644 --- a/backend/tests/test_memory_upload_filtering.py +++ b/backend/tests/test_memory_upload_filtering.py @@ -177,6 +177,59 @@ def test_p0_memory_payload_is_excluded(self): assert "Help me with Python." in human_contents[0] assert not any("" in c for c in human_contents) + def test_hide_from_ui_human_input_response_is_preserved(self): + """Hidden card replies are user-authored answers, not framework context.""" + hidden_response = HumanMessage( + content="For your clarification, my answer is: staging", + additional_kwargs={ + "hide_from_ui": True, + "human_input_response": { + "version": 1, + "kind": "human_input_response", + "source": "ask_clarification", + "request_id": "clarification:call-abc", + "response_kind": "option", + "option_id": "option-2", + "value": "staging", + }, + }, + ) + msgs = [ + _human("Deploy the app."), + _ai("Which environment?"), + hidden_response, + _ai("Deploying to staging."), + ] + + result = filter_messages_for_memory(msgs) + + human_contents = [m.content for m in result if m.type == "human"] + assert "Deploy the app." in human_contents + assert "For your clarification, my answer is: staging" in human_contents + + def test_hide_from_ui_malformed_human_input_response_is_excluded(self): + hidden_response = HumanMessage( + content="For your clarification, my answer is: staging", + additional_kwargs={ + "hide_from_ui": True, + "human_input_response": { + "version": 1, + "kind": "human_input_response", + "source": "ask_clarification", + "request_id": "clarification:call-abc", + "response_kind": "option", + "value": "staging", + }, + }, + ) + msgs = [_human("Deploy the app."), _ai("Which environment?"), hidden_response] + + result = filter_messages_for_memory(msgs) + + human_contents = [m.content for m in result if m.type == "human"] + assert "Deploy the app." in human_contents + assert "For your clarification, my answer is: staging" not in human_contents + def test_hide_from_ui_false_is_preserved(self): """Messages without hide_from_ui (or with it set to False) are kept.""" visible_msg = HumanMessage(content="Visible message", additional_kwargs={"hide_from_ui": False}) diff --git a/backend/tests/test_multi_worker_postgres_gate.py b/backend/tests/test_multi_worker_postgres_gate.py new file mode 100644 index 00000000000..f87e5e7b9d3 --- /dev/null +++ b/backend/tests/test_multi_worker_postgres_gate.py @@ -0,0 +1,142 @@ +"""Tests for the multi-worker Postgres startup gate. + +Pins the contract documented in ``docs/multi_worker.md`` work item 1 +(issue #3948): when ``GATEWAY_WORKERS > 1`` and the configured +database backend is not Postgres, the Gateway must refuse to start. +The gate runs inside :func:`langgraph_runtime` *before* any +persistence engine is initialised so operators see a clear error +instead of intermittent SQLite ``database is locked`` failures in +production. +""" + +from __future__ import annotations + +from contextlib import asynccontextmanager +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest +from fastapi import FastAPI + +from app.gateway.deps import _enforce_postgres_for_multi_worker, langgraph_runtime +from deerflow.config.database_config import DatabaseConfig + + +def _config_with_backend(backend: str) -> SimpleNamespace: + return SimpleNamespace(database=DatabaseConfig(backend=backend)) + + +# --------------------------------------------------------------------------- +# Unit tests of the gate function itself +# --------------------------------------------------------------------------- + + +def test_gate_noop_when_gateway_workers_unset(monkeypatch): + """With GATEWAY_WORKERS unset, every backend must be accepted.""" + monkeypatch.delenv("GATEWAY_WORKERS", raising=False) + for backend in ("sqlite", "memory", "postgres"): + _enforce_postgres_for_multi_worker(_config_with_backend(backend)) + + +def test_gate_noop_for_single_worker(monkeypatch): + """GATEWAY_WORKERS=1 preserves the historical single-worker behavior.""" + monkeypatch.setenv("GATEWAY_WORKERS", "1") + for backend in ("sqlite", "memory", "postgres"): + _enforce_postgres_for_multi_worker(_config_with_backend(backend)) + + +def test_gate_allows_multi_worker_with_postgres(monkeypatch): + monkeypatch.setenv("GATEWAY_WORKERS", "2") + _enforce_postgres_for_multi_worker(_config_with_backend("postgres")) + + +def test_gate_rejects_multi_worker_with_sqlite(monkeypatch): + monkeypatch.setenv("GATEWAY_WORKERS", "2") + with pytest.raises(SystemExit) as exc_info: + _enforce_postgres_for_multi_worker(_config_with_backend("sqlite")) + msg = str(exc_info.value) + assert "GATEWAY_WORKERS=2" in msg + assert "postgres" in msg.lower() + assert "sqlite" in msg.lower() + + +def test_gate_rejects_multi_worker_with_memory(monkeypatch): + """The gate is not sqlite-specific: memory is also unsafe across processes.""" + monkeypatch.setenv("GATEWAY_WORKERS", "2") + with pytest.raises(SystemExit): + _enforce_postgres_for_multi_worker(_config_with_backend("memory")) + + +def test_gate_rejects_high_worker_counts(monkeypatch): + """The threshold is >1, not ==2; prod-scale counts must also be gated.""" + monkeypatch.setenv("GATEWAY_WORKERS", "4") + with pytest.raises(SystemExit) as exc_info: + _enforce_postgres_for_multi_worker(_config_with_backend("sqlite")) + assert "GATEWAY_WORKERS=4" in str(exc_info.value) + + +def test_gate_treats_invalid_env_as_single_worker(monkeypatch): + """Non-integer GATEWAY_WORKERS values must not crash startup. + + Uvicorn itself rejects these later; the gate should not preempt + that with its own crash. Falling back to 1 keeps the gate inert. + """ + for invalid in ("", "auto", "1.5", "abc", "0x4"): + monkeypatch.setenv("GATEWAY_WORKERS", invalid) + _enforce_postgres_for_multi_worker(_config_with_backend("sqlite")) + + +def test_gate_treats_zero_and_negatives_as_single_worker(monkeypatch): + """GATEWAY_WORKERS <= 1 (including 0 and negatives) skips the gate.""" + for value in ("0", "-1", "-999"): + monkeypatch.setenv("GATEWAY_WORKERS", value) + _enforce_postgres_for_multi_worker(_config_with_backend("sqlite")) + + +def test_gate_error_message_lists_both_remediations(monkeypatch): + """Operators must see both fix options without reading docs.""" + monkeypatch.setenv("GATEWAY_WORKERS", "2") + with pytest.raises(SystemExit) as exc_info: + _enforce_postgres_for_multi_worker(_config_with_backend("sqlite")) + msg = str(exc_info.value) + assert "GATEWAY_WORKERS=1" in msg, "must mention the rollback knob" + assert "Postgres" in msg, "must mention the alternative backend" + + +# --------------------------------------------------------------------------- +# Integration: the gate is wired into langgraph_runtime before init_engine +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_langgraph_runtime_invokes_gate_before_persistence_setup(monkeypatch): + """When the gate trips, no persistence / stream-bridge setup may run. + + Guards against regressions that reorder the gate behind + ``init_engine_from_config`` (or any other expensive startup step). + """ + monkeypatch.setenv("GATEWAY_WORKERS", "2") + + init_engine_from_config = AsyncMock(name="init_engine_from_config") + + @asynccontextmanager + async def _noop_stream_bridge(_config): + yield MagicMock() + + with ( + patch( + "deerflow.persistence.engine.init_engine_from_config", + init_engine_from_config, + ), + patch("deerflow.runtime.make_stream_bridge", side_effect=_noop_stream_bridge) as make_stream_bridge, + patch("deerflow.runtime.make_store", side_effect=_noop_stream_bridge) as make_store, + ): + app = FastAPI() + startup_config = _config_with_backend("sqlite") + with pytest.raises(SystemExit): + async with langgraph_runtime(app, startup_config): + pass + + init_engine_from_config.assert_not_called() + make_stream_bridge.assert_not_called() + make_store.assert_not_called() diff --git a/backend/tests/test_read_before_write_middleware.py b/backend/tests/test_read_before_write_middleware.py index c07a0b8d8d1..c502e0ea94c 100644 --- a/backend/tests/test_read_before_write_middleware.py +++ b/backend/tests/test_read_before_write_middleware.py @@ -213,6 +213,16 @@ def test_normalized_path_matching(self): handler.assert_called_once() assert result.status != "error" + def test_blocked_write_has_deerflow_tool_meta(self): + from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY + + mw = _middleware({self.PATH: "v1"}) + request = _make_request("write_file", {"description": "d", "path": self.PATH, "content": "v2"}) + result = mw.wrap_tool_call(request, MagicMock()) + meta = (result.additional_kwargs or {}).get(TOOL_META_KEY) + assert meta is not None, "blocked write must carry deerflow_tool_meta" + assert meta["recoverable_by_model"] is True + class TestAsyncPaths: PATH = "/mnt/user-data/outputs/report.md" @@ -229,6 +239,22 @@ async def handler(_request): result = asyncio.run(mw.awrap_tool_call(request, handler)) assert result.status == "error" + def test_async_blocked_write_has_deerflow_tool_meta(self): + import asyncio + + from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY + + mw = _middleware({self.PATH: "v1"}) + request = _make_request("write_file", {"description": "d", "path": self.PATH, "content": "v2"}) + + async def handler(_request): + raise AssertionError("handler must not run when blocked") + + result = asyncio.run(mw.awrap_tool_call(request, handler)) + meta = (result.additional_kwargs or {}).get(TOOL_META_KEY) + assert meta is not None, "async blocked write must carry deerflow_tool_meta" + assert meta["recoverable_by_model"] is True + def test_async_read_stamps_mark(self): import asyncio diff --git a/backend/tests/test_subagent_executor.py b/backend/tests/test_subagent_executor.py index b6da865bbba..e4fb8eb7e69 100644 --- a/backend/tests/test_subagent_executor.py +++ b/backend/tests/test_subagent_executor.py @@ -889,6 +889,80 @@ async def test_aexecute_handles_agent_exception(self, classes, base_config, mock assert "Agent error" in result.error assert result.completed_at is not None + @pytest.mark.anyio + async def test_aexecute_recursion_error_classified_as_max_turns_reached(self, classes, base_config, mock_agent, msg): + """#3875 Phase 2: ``GraphRecursionError`` (``recursion_limit`` == + ``max_turns``) must surface as ``MAX_TURNS_REACHED`` with the partial + work recovered from the last streamed chunk — not as a generic FAILED + that hides the budget cap and discards the partial result. + + Before this fix the exception fell through to the generic + ``except Exception`` and the subagent was reported as broken, so the + lead could not tell "out of budget" from "broken subagent" and the + work already streamed into ``final_state`` was lost. + """ + from langgraph.errors import GraphRecursionError + + SubagentExecutor = classes["SubagentExecutor"] + SubagentStatus = classes["SubagentStatus"] + + partial_ai = msg.ai("Found 3 of 5 sources; still working", "msg-1") + partial_state = {"messages": [msg.human("Task"), partial_ai]} + + async def mock_astream(*args, **kwargs): + yield partial_state + raise GraphRecursionError("Recursion limit of 10 reached") + + mock_agent.astream = mock_astream + + executor = SubagentExecutor( + config=base_config, + tools=[], + thread_id="test-thread", + ) + + with patch.object(executor, "_create_agent", return_value=mock_agent): + result = await executor._aexecute("Task") + + assert result.status == SubagentStatus.MAX_TURNS_REACHED + # The partial work from the last streamed chunk is preserved, not dropped. + assert result.result == "Found 3 of 5 sources; still working" + # The cap is surfaced so the lead can tell "out of budget" from "broken". + assert result.error is not None + assert str(base_config.max_turns) in result.error + assert result.completed_at is not None + + @pytest.mark.anyio + async def test_aexecute_recursion_error_before_first_chunk_uses_sentinel(self, classes, base_config, mock_agent): + """If ``GraphRecursionError`` fires before any chunk is yielded there is + no partial state to recover; the result must still be + ``MAX_TURNS_REACHED`` (with the ``No response generated`` sentinel) + rather than FAILED, so the budget-cap signal survives even when no + work was streamed.""" + from langgraph.errors import GraphRecursionError + + SubagentExecutor = classes["SubagentExecutor"] + SubagentStatus = classes["SubagentStatus"] + + async def mock_astream(*args, **kwargs): + raise GraphRecursionError("Recursion limit reached before first step") + yield # pragma: no cover - make this an async generator + + mock_agent.astream = mock_astream + + executor = SubagentExecutor( + config=base_config, + tools=[], + thread_id="test-thread", + ) + + with patch.object(executor, "_create_agent", return_value=mock_agent): + result = await executor._aexecute("Task") + + assert result.status == SubagentStatus.MAX_TURNS_REACHED + assert result.result == "No response generated" + assert result.completed_at is not None + @pytest.mark.anyio async def test_aexecute_no_final_state(self, classes, base_config, mock_agent): """Test handling when no final state is returned.""" @@ -1614,6 +1688,31 @@ def test_cleanup_removes_terminal_timed_out_task(self, executor_module, classes) assert task_id not in executor_module._background_tasks + def test_cleanup_removes_terminal_max_turns_reached_task(self, executor_module, classes): + """Test that cleanup removes a MAX_TURNS_REACHED task (#3875 Phase 2). + + ``is_terminal`` includes MAX_TURNS_REACHED so the task_tool polling + loop's cleanup path treats a budget-capped subagent as done and + removes it from the background registry, matching COMPLETED / FAILED / + TIMED_OUT.""" + SubagentResult = classes["SubagentResult"] + SubagentStatus = classes["SubagentStatus"] + + task_id = "test-max-turns-task" + result = SubagentResult( + task_id=task_id, + trace_id="test-trace", + status=SubagentStatus.MAX_TURNS_REACHED, + result="partial work recovered", + error="Reached max_turns=10", + completed_at=datetime.now(), + ) + executor_module._background_tasks[task_id] = result + + executor_module.cleanup_background_task(task_id) + + assert task_id not in executor_module._background_tasks + def test_cleanup_skips_running_task(self, executor_module, classes): """Test that cleanup does NOT remove a RUNNING task. diff --git a/backend/tests/test_subagent_status_contract.py b/backend/tests/test_subagent_status_contract.py index 886d30a3839..a3790064f7e 100644 --- a/backend/tests/test_subagent_status_contract.py +++ b/backend/tests/test_subagent_status_contract.py @@ -13,6 +13,7 @@ SUBAGENT_STATUS_KEY, SUBAGENT_STATUS_VALUES, _bound_metadata_text, + format_subagent_result_message, make_subagent_additional_kwargs, read_subagent_result_metadata, ) @@ -61,6 +62,32 @@ def test_make_subagent_additional_kwargs_bounds_large_result_metadata(): assert len(kwargs[SUBAGENT_RESULT_SHA256_KEY]) == 64 +def test_make_subagent_additional_kwargs_max_turns_reached_carries_result_and_error(): + """#3875 Phase 2: a turn-capped run is result-bearing — the partial work + the executor recovered must travel on ``subagent_result_brief`` / ``sha256`` + (so the delegation ledger and card keep it) AND the cap notice must travel + on ``subagent_error``. This is the one status that carries both.""" + kwargs = make_subagent_additional_kwargs("max_turns_reached", result="investigated 3 of 5 sources", error="Reached max_turns=150") + assert kwargs[SUBAGENT_STATUS_KEY] == "max_turns_reached" + assert kwargs[SUBAGENT_RESULT_BRIEF_KEY] == "investigated 3 of 5 sources" + assert len(kwargs[SUBAGENT_RESULT_SHA256_KEY]) == 64 + assert kwargs[SUBAGENT_ERROR_KEY] == "Reached max_turns=150" + + +def test_format_subagent_result_message_max_turns_reached_leads_with_partial_result(): + """The model-visible text leads with the recovered partial result and + names the cap; the metadata error carries the cap reason only.""" + content, metadata_error = format_subagent_result_message("max_turns_reached", result="investigated 3 of 5 sources", error="Reached max_turns=150") + assert content.startswith("Task reached max turns") + assert "investigated 3 of 5 sources" in content + assert metadata_error == "Reached max_turns=150" + + +def test_format_subagent_result_message_max_turns_reached_uses_sentinel_when_no_partial(): + content, _metadata_error = format_subagent_result_message("max_turns_reached", result=None, error="Reached max_turns=150") + assert "No partial result was produced" in content + + def test_bound_metadata_text_respects_small_caps(): text = "A" * 100 @@ -100,6 +127,26 @@ def test_read_subagent_result_metadata_returns_bounded_payload(): } +def test_read_subagent_result_metadata_max_turns_reached_reads_result_brief_and_error(): + """A turn-capped result carries both result metadata and the cap error; + the reader must surface both so the delegation ledger can prefer the + partial result and still expose the cap reason.""" + parsed = read_subagent_result_metadata( + { + SUBAGENT_STATUS_KEY: "max_turns_reached", + SUBAGENT_RESULT_BRIEF_KEY: "investigated 3 of 5 sources", + SUBAGENT_RESULT_SHA256_KEY: "a" * 64, + SUBAGENT_ERROR_KEY: "Reached max_turns=150", + } + ) + assert parsed == { + "status": "max_turns_reached", + "result_brief": "investigated 3 of 5 sources", + "result_sha256": "a" * 64, + "error": "Reached max_turns=150", + } + + def test_read_subagent_result_metadata_rejects_unknown_status(): assert read_subagent_result_metadata({SUBAGENT_STATUS_KEY: "future"}) is None diff --git a/backend/tests/test_task_tool_core_logic.py b/backend/tests/test_task_tool_core_logic.py index 831119347da..3ff2c205d0c 100644 --- a/backend/tests/test_task_tool_core_logic.py +++ b/backend/tests/test_task_tool_core_logic.py @@ -32,6 +32,7 @@ class FakeSubagentStatus(Enum): FAILED = "failed" CANCELLED = "cancelled" TIMED_OUT = "timed_out" + MAX_TURNS_REACHED = "max_turns_reached" def _make_runtime(*, app_config=None) -> SimpleNamespace: @@ -158,6 +159,24 @@ def test_task_result_command_derives_content_from_status_payload(): assert timed_out_without_detail.additional_kwargs[SUBAGENT_STATUS_KEY] == "timed_out" assert timed_out_without_detail.additional_kwargs[SUBAGENT_ERROR_KEY] == "Task timed out." + # #3875 Phase 2: a turn-capped run is the one status that carries BOTH a + # recovered partial result (result_brief + sha256) and a cap notice + # (error), and the model-visible content leads with the partial work. + max_turns = _task_tool_message( + task_tool_module._task_result_command( + tool_call_id="tc-max-turns", + status="max_turns_reached", + result="investigated 3 of 5 sources", + error="Reached max_turns=150", + ) + ) + assert max_turns.content.startswith("Task reached max turns") + assert "investigated 3 of 5 sources" in max_turns.content + assert max_turns.additional_kwargs[SUBAGENT_STATUS_KEY] == "max_turns_reached" + assert max_turns.additional_kwargs[SUBAGENT_RESULT_BRIEF_KEY] == "investigated 3 of 5 sources" + assert len(max_turns.additional_kwargs[SUBAGENT_RESULT_SHA256_KEY]) == 64 + assert max_turns.additional_kwargs[SUBAGENT_ERROR_KEY] == "Reached max_turns=150" + async def _no_sleep(_: float) -> None: return None @@ -713,6 +732,45 @@ def test_task_tool_returns_timed_out_message(monkeypatch): assert events[-1]["error"] == "timeout" +def test_task_tool_returns_max_turns_reached_message(monkeypatch): + """#3875 Phase 2: a MAX_TURNS_REACHED subagent surfaces a distinct + ``Task reached max turns`` message that carries the recovered partial + result, and stamps ``result_brief`` + the cap notice on ``error`` — the + one status that carries both. The polling loop emits ``task_failed`` so + the card transitions out of running; the structured status is the reason.""" + config = _make_subagent_config() + events = [] + + monkeypatch.setattr(task_tool_module, "SubagentStatus", FakeSubagentStatus) + monkeypatch.setattr(task_tool_module, "get_subagent_config", lambda _: config) + monkeypatch.setattr( + task_tool_module, + "get_background_task_result", + lambda _: _make_result(FakeSubagentStatus.MAX_TURNS_REACHED, result="investigated 3 of 5 sources", error="Reached max_turns=50"), + ) + monkeypatch.setattr(task_tool_module, "get_stream_writer", lambda: events.append) + monkeypatch.setattr(task_tool_module.asyncio, "sleep", _no_sleep) + monkeypatch.setattr("deerflow.tools.get_available_tools", lambda **kwargs: []) + + output = _run_task_tool( + runtime=_make_runtime(), + description="执行任务", + prompt="do capped work", + subagent_type="general-purpose", + tool_call_id="tc-max-turns", + ) + + message = _task_tool_message(output) + assert message.content.startswith("Task reached max turns") + assert "investigated 3 of 5 sources" in message.content + assert str(config.max_turns) in message.content + assert message.additional_kwargs[SUBAGENT_STATUS_KEY] == "max_turns_reached" + assert message.additional_kwargs[SUBAGENT_RESULT_BRIEF_KEY] == "investigated 3 of 5 sources" + assert len(message.additional_kwargs[SUBAGENT_RESULT_SHA256_KEY]) == 64 + assert message.additional_kwargs[SUBAGENT_ERROR_KEY] == f"Reached max_turns={config.max_turns}" + assert events[-1]["type"] == "task_failed" + + def test_task_tool_polling_safety_timeout(monkeypatch): config = _make_subagent_config() # Keep max_poll_count small for test speed: (1 + 60) // 5 = 12 diff --git a/backend/tests/test_threads_router.py b/backend/tests/test_threads_router.py index b035c9bf7cf..d4e4d37a99a 100644 --- a/backend/tests/test_threads_router.py +++ b/backend/tests/test_threads_router.py @@ -1,3 +1,4 @@ +import asyncio import re from types import SimpleNamespace from unittest.mock import patch @@ -6,6 +7,8 @@ from _router_auth_helpers import make_authed_test_app from fastapi import FastAPI, HTTPException from fastapi.testclient import TestClient +from langchain_core.messages import AIMessage, HumanMessage +from langgraph.checkpoint.base import empty_checkpoint from langgraph.checkpoint.memory import InMemorySaver from langgraph.store.memory import InMemoryStore @@ -65,6 +68,32 @@ def _build_thread_app() -> tuple[FastAPI, InMemoryStore, InMemorySaver]: return app, store, checkpointer +async def _write_checkpoint( + checkpointer: InMemorySaver, + thread_id: str, + checkpoint_id: str, + messages: list[object], + *, + step: int, +) -> dict: + checkpoint = empty_checkpoint() + checkpoint["id"] = checkpoint_id + checkpoint["channel_values"] = {"messages": messages} + checkpoint["channel_versions"] = {"messages": step} + return await checkpointer.aput( + {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}, + checkpoint, + { + "step": step, + "source": "loop", + "writes": {"test": {"messages": messages}}, + "parents": {}, + "created_at": f"2026-07-05T00:00:0{step}+00:00", + }, + {"messages": step}, + ) + + def test_delete_thread_data_removes_thread_directory(tmp_path): paths = Paths(tmp_path) thread_dir = paths.thread_dir("thread-cleanup") @@ -539,6 +568,216 @@ async def _seed() -> None: assert _ISO_TIMESTAMP_RE.match(entry["created_at"]), entry +# ── branch threads from completed assistant turns ───────────────────────────── + + +def test_branch_thread_from_older_assistant_turn_creates_truncated_thread() -> None: + app, store, checkpointer = _build_thread_app() + source_thread_id = "source-thread" + + human_1 = HumanMessage(id="human-1", content="First question") + ai_1 = AIMessage(id="ai-1", content="First answer") + human_2 = HumanMessage(id="human-2", content="Second question") + ai_2 = AIMessage(id="ai-2", content="Second answer") + human_3 = HumanMessage(id="human-3", content="Third question") + ai_3 = AIMessage(id="ai-3", content="Third answer") + + async def _seed() -> None: + await _write_checkpoint(checkpointer, source_thread_id, "0001", [human_1, ai_1], step=1) + await _write_checkpoint(checkpointer, source_thread_id, "0002", [human_1, ai_1, human_2, ai_2], step=2) + await _write_checkpoint(checkpointer, source_thread_id, "0003", [human_1, ai_1, human_2, ai_2, human_3, ai_3], step=3) + + asyncio.run(_seed()) + + with TestClient(app) as client: + created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}, "assistant_id": "agent"}) + assert created.status_code == 200, created.text + asyncio.run( + store.aput( + THREADS_NS, + source_thread_id, + { + "thread_id": source_thread_id, + "assistant_id": "agent", + "user_id": None, + "status": "idle", + "created_at": "2026-07-05T00:00:00Z", + "updated_at": "2026-07-05T00:00:00Z", + "display_name": "Original chat", + "metadata": {}, + }, + ) + ) + + response = client.post( + f"/api/threads/{source_thread_id}/branches", + json={"message_id": "ai-2", "message_ids": ["ai-2"]}, + ) + assert response.status_code == 200, response.text + body = response.json() + new_thread_id = body["thread_id"] + state_response = client.get(f"/api/threads/{new_thread_id}/state") + search_response = client.post("/api/threads/search", json={"limit": 10}) + + assert body["parent_thread_id"] == source_thread_id + assert body["parent_checkpoint_id"] == "0002" + assert body["branched_from_message_id"] == "ai-2" + assert body["workspace_clone_mode"] == "skipped_historical_turn" + + assert state_response.status_code == 200, state_response.text + messages = state_response.json()["values"]["messages"] + assert [message["id"] for message in messages] == ["human-1", "ai-1", "human-2", "ai-2"] + assert "Third answer" not in [message.get("content") for message in messages] + assert search_response.status_code == 200, search_response.text + branch_entry = next(item for item in search_response.json() if item["thread_id"] == new_thread_id) + assert branch_entry["values"]["title"] == "Original chat" + + +def test_branch_display_name_strips_legacy_branch_prefix_only_for_branch_sources() -> None: + assert threads._default_branch_display_name("Original chat") == "Original chat" + assert threads._default_branch_display_name("Branch: Original chat") == "Branch: Original chat" + assert threads._default_branch_display_name("Branch: Branch: Original chat", source_is_branch=True) == "Original chat" + + +def test_branch_thread_rejects_sidecar_threads() -> None: + app, _store, _checkpointer = _build_thread_app() + + with TestClient(app) as client: + created = client.post( + "/api/threads", + json={"thread_id": "sidecar-thread", "metadata": {"deerflow_sidecar": True}}, + ) + assert created.status_code == 200, created.text + + response = client.post( + "/api/threads/sidecar-thread/branches", + json={"message_id": "ai-1", "message_ids": ["ai-1"]}, + ) + + assert response.status_code == 409 + assert "main conversation" in response.json()["detail"] + + +def test_branch_thread_rejects_non_assistant_targets() -> None: + app, _store, checkpointer = _build_thread_app() + source_thread_id = "source-human-target" + human = HumanMessage(id="human-1", content="Question") + ai = AIMessage(id="ai-1", content="Answer") + + async def _seed() -> None: + await _write_checkpoint(checkpointer, source_thread_id, "0001", [human, ai], step=1) + + asyncio.run(_seed()) + + with TestClient(app) as client: + created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}}) + assert created.status_code == 200, created.text + + response = client.post( + f"/api/threads/{source_thread_id}/branches", + json={"message_id": "human-1", "message_ids": ["human-1"]}, + ) + + assert response.status_code == 409 + assert "can no longer be branched" in response.json()["detail"] + + +def test_branch_thread_best_effort_copies_current_workspace(tmp_path) -> None: + paths = Paths(tmp_path) + app, _store, checkpointer = _build_thread_app() + source_thread_id = "source-with-files" + user_id = "branch-user" + + source_user_data = paths.sandbox_user_data_dir(source_thread_id, user_id=user_id) + source_outputs = paths.sandbox_outputs_dir(source_thread_id, user_id=user_id) + source_uploads = paths.sandbox_uploads_dir(source_thread_id, user_id=user_id) + source_outputs.mkdir(parents=True, exist_ok=True) + source_uploads.mkdir(parents=True, exist_ok=True) + (source_outputs / "result.txt").write_text("answer", encoding="utf-8") + (source_uploads / ".upload-stale.part").write_text("partial", encoding="utf-8") + + human = HumanMessage(id="human-file", content="Make a file") + ai = AIMessage(id="ai-file", content="Done") + + async def _seed() -> None: + await _write_checkpoint(checkpointer, source_thread_id, "0001", [human, ai], step=1) + + asyncio.run(_seed()) + + with ( + patch("app.gateway.routers.threads.get_paths", return_value=paths), + patch("app.gateway.routers.threads.get_effective_user_id", return_value=user_id), + TestClient(app) as client, + ): + created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}}) + assert created.status_code == 200, created.text + + response = client.post( + f"/api/threads/{source_thread_id}/branches", + json={"message_id": "ai-file", "message_ids": ["ai-file"]}, + ) + + assert response.status_code == 200, response.text + body = response.json() + assert body["workspace_clone_mode"] == "current_thread_best_effort" + + target_user_data = paths.sandbox_user_data_dir(body["thread_id"], user_id=user_id) + assert target_user_data.exists() + assert (target_user_data / "outputs" / "result.txt").read_text(encoding="utf-8") == "answer" + assert not (target_user_data / "uploads" / ".upload-stale.part").exists() + assert source_user_data.exists() + + +def test_branch_thread_from_historical_turn_skips_workspace_clone(tmp_path) -> None: + """Branching from a non-latest turn must not clone the current workspace. + + Workspace files are not checkpointed, so cloning them onto a branch rooted at + an older turn would leak files created after that turn (regression for the + historical-turn workspace-leak review on PR #3950). + """ + paths = Paths(tmp_path) + app, _store, checkpointer = _build_thread_app() + source_thread_id = "source-historical" + user_id = "branch-user" + + source_outputs = paths.sandbox_outputs_dir(source_thread_id, user_id=user_id) + source_outputs.mkdir(parents=True, exist_ok=True) + # ``future.txt`` only exists in the current (latest) workspace timeline. + (source_outputs / "future.txt").write_text("future", encoding="utf-8") + + human_1 = HumanMessage(id="human-1", content="First question") + ai_1 = AIMessage(id="ai-1", content="First answer") + human_2 = HumanMessage(id="human-2", content="Second question") + ai_2 = AIMessage(id="ai-2", content="Second answer") + + async def _seed() -> None: + await _write_checkpoint(checkpointer, source_thread_id, "0001", [human_1, ai_1], step=1) + await _write_checkpoint(checkpointer, source_thread_id, "0002", [human_1, ai_1, human_2, ai_2], step=2) + + asyncio.run(_seed()) + + with ( + patch("app.gateway.routers.threads.get_paths", return_value=paths), + patch("app.gateway.routers.threads.get_effective_user_id", return_value=user_id), + TestClient(app) as client, + ): + created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}}) + assert created.status_code == 200, created.text + + response = client.post( + f"/api/threads/{source_thread_id}/branches", + json={"message_id": "ai-1", "message_ids": ["ai-1"]}, + ) + + assert response.status_code == 200, response.text + body = response.json() + assert body["parent_checkpoint_id"] == "0001" + assert body["workspace_clone_mode"] == "skipped_historical_turn" + + target_user_data = paths.sandbox_user_data_dir(body["thread_id"], user_id=user_id) + assert not target_user_data.exists() + + # ── Metadata filter validation at API boundary ──────────────────────────────── diff --git a/backend/tests/test_tool_error_handling_middleware.py b/backend/tests/test_tool_error_handling_middleware.py index 065f0e3f1c7..3936502bfba 100644 --- a/backend/tests/test_tool_error_handling_middleware.py +++ b/backend/tests/test_tool_error_handling_middleware.py @@ -10,6 +10,7 @@ build_lead_runtime_middlewares, build_subagent_runtime_middlewares, ) +from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY from deerflow.agents.middlewares.view_image_middleware import ViewImageMiddleware from deerflow.config import summarization_config from deerflow.config.app_config import AppConfig, CircuitBreakerConfig @@ -156,6 +157,72 @@ def __init__(self, *, app_config): assert isinstance(middlewares[-1], SafetyFinishReasonMiddleware) +def test_tool_progress_middleware_is_outer_relative_to_error_handling(monkeypatch: pytest.MonkeyPatch): + # ToolProgressMiddleware must have a lower index than ToolErrorHandlingMiddleware + # so that the framework's "first in list = outermost" rule makes it outer. + # Only then can it read deerflow_tool_meta stamped by ToolErrorHandlingMiddleware. + from deerflow.agents.middlewares.tool_progress_middleware import ToolProgressMiddleware + from deerflow.config.tool_progress_config import ToolProgressConfig + + app_config = AppConfig( + models=[ + ModelConfig( + name="test-model", + display_name="test-model", + description=None, + use="langchain_openai:ChatOpenAI", + model="test-model", + ) + ], + sandbox=SandboxConfig(use="test"), + guardrails=GuardrailsConfig(enabled=False), + circuit_breaker=CircuitBreakerConfig(failure_threshold=7, recovery_timeout_sec=11), + tool_progress=ToolProgressConfig(enabled=True), + ) + + _stub_runtime_middleware_imports(monkeypatch) + + middlewares = build_subagent_runtime_middlewares(app_config=app_config, lazy_init=False) + + progress_idx = next(i for i, m in enumerate(middlewares) if isinstance(m, ToolProgressMiddleware)) + error_idx = next(i for i, m in enumerate(middlewares) if isinstance(m, ToolErrorHandlingMiddleware)) + assert progress_idx < error_idx, f"ToolProgressMiddleware (index {progress_idx}) must be outer (lower index) than ToolErrorHandlingMiddleware (index {error_idx}); order: {[type(m).__name__ for m in middlewares]}" + + +def test_middleware_ordering_guard_raises_when_progress_is_inner(monkeypatch: pytest.MonkeyPatch): + """_build_runtime_middlewares must raise RuntimeError when ToolProgressMiddleware ends up + at a higher index than ToolErrorHandlingMiddleware. + + We trigger the wrong-order condition by patching SandboxAuditMiddleware to be an actual + ToolErrorHandlingMiddleware instance, which appears BEFORE ToolProgressMiddleware in the + list. The guard's isinstance() check finds it first, making error_idx < progress_idx. + """ + from deerflow.agents.middlewares.tool_error_handling_middleware import ( + ToolErrorHandlingMiddleware, + build_lead_runtime_middlewares, + ) + from deerflow.config.tool_progress_config import ToolProgressConfig + + _stub_runtime_middleware_imports(monkeypatch) + # Override the SandboxAuditMiddleware stub with a real ToolErrorHandlingMiddleware so it + # becomes the FIRST ToolErrorHandlingMiddleware in the list, appearing before + # ToolProgressMiddleware and triggering the ordering guard. + monkeypatch.setitem( + sys.modules, + "deerflow.agents.middlewares.sandbox_audit_middleware", + _module( + "deerflow.agents.middlewares.sandbox_audit_middleware", + SandboxAuditMiddleware=ToolErrorHandlingMiddleware, + ), + ) + + app_config = _make_app_config() + app_config = app_config.model_copy(update={"tool_progress": ToolProgressConfig(enabled=True)}) + + with pytest.raises(RuntimeError, match="ToolProgressMiddleware must be outer"): + build_lead_runtime_middlewares(app_config=app_config, lazy_init=False) + + def test_lead_runtime_middlewares_thread_app_config_to_tool_error_handling(monkeypatch: pytest.MonkeyPatch): monkeypatch.setitem( sys.modules, @@ -330,6 +397,23 @@ def _boom(_req): assert "network down" in result.text +def test_wrap_tool_call_stamps_tool_meta_on_exception(): + middleware = ToolErrorHandlingMiddleware() + req = _request(name="web_search", tool_call_id="tc-42") + + def _boom(_req): + raise ConnectionError("connection refused") + + result = middleware.wrap_tool_call(req, _boom) + + assert isinstance(result, ToolMessage) + assert TOOL_META_KEY in result.additional_kwargs + meta = result.additional_kwargs[TOOL_META_KEY] + assert meta["status"] == "error" + assert meta["source"] == "exception" + assert meta["error_type"] == "transient" + + def test_task_exception_wrapper_uses_subagent_result_formatter(): middleware = ToolErrorHandlingMiddleware() req = _request(name="task", tool_call_id="tc-task") diff --git a/backend/tests/test_tool_progress_middleware.py b/backend/tests/test_tool_progress_middleware.py new file mode 100644 index 00000000000..c9addcfe8a1 --- /dev/null +++ b/backend/tests/test_tool_progress_middleware.py @@ -0,0 +1,1479 @@ +"""Tests for ToolProgressMiddleware state machine (RFC #3177).""" + +from __future__ import annotations + +import logging +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest +from langchain_core.messages import HumanMessage, ToolMessage +from langgraph.types import Command + +from deerflow.agents.middlewares.tool_progress_middleware import ( + ToolProgressMiddleware, + is_near_duplicate, + word_set, +) +from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY + +# --------------------------------------------------------------------------- +# Helpers + + +def _make_runtime(thread_id: str = "t1", run_id: str = "r1") -> MagicMock: + rt = MagicMock() + rt.context = {"thread_id": thread_id, "run_id": run_id} + return rt + + +def _make_tool_request(tool_name: str = "web_search", *, runtime: MagicMock | None = None) -> SimpleNamespace: + rt = runtime or _make_runtime() + return SimpleNamespace( + tool_call={"name": tool_name, "id": f"tc-{tool_name}"}, + runtime=rt, + ) + + +def _meta_kwargs( + *, + status: str = "success", + error_type: str | None = None, + recoverable_by_model: bool = True, + recommended_next_action: str = "continue", + source: str = "content_analysis", +) -> dict[str, object]: + return { + TOOL_META_KEY: { + "status": status, + "error_type": error_type, + "recoverable_by_model": recoverable_by_model, + "recommended_next_action": recommended_next_action, + "source": source, + } + } + + +def _make_tool_message( + content: str = "A" * 200, + *, + tool_name: str = "web_search", + meta_kwargs: dict[str, object] | None = None, +) -> ToolMessage: + return ToolMessage( + content=content, + tool_call_id=f"tc-{tool_name}", + name=tool_name, + status="success", + additional_kwargs=meta_kwargs or _meta_kwargs(), + ) + + +def _make_non_recoverable_error_message( + content: str = "Error: rate limited", + *, + tool_name: str = "web_search", + error_type: str = "rate_limited", + recommended_next_action: str = "summarize", +) -> ToolMessage: + """Non-recoverable stagnation error (recoverable_by_model=False, non-stop). + Unlike auth/config, these go through the stagnation counter, but should + still reach BLOCKED because the model cannot fix them by retrying. + """ + return ToolMessage( + content=content, + tool_call_id=f"tc-{tool_name}", + name=tool_name, + status="error", + additional_kwargs=_meta_kwargs( + status="error", + error_type=error_type, + recoverable_by_model=False, + recommended_next_action=recommended_next_action, + ), + ) + + +def _make_error_message( + content: str = "Error: no results found", + *, + tool_name: str = "web_search", + error_type: str = "no_results", + recoverable_by_model: bool = True, + recommended_next_action: str = "rewrite_query", +) -> ToolMessage: + return ToolMessage( + content=content, + tool_call_id=f"tc-{tool_name}", + name=tool_name, + status="error", + additional_kwargs=_meta_kwargs( + status="error", + error_type=error_type, + recoverable_by_model=recoverable_by_model, + recommended_next_action=recommended_next_action, + ), + ) + + +def _make_model_request(messages: list, runtime: MagicMock) -> MagicMock: + req = MagicMock() + req.messages = list(messages) + req.runtime = runtime + + def _override(**kw) -> MagicMock: + updated = MagicMock() + updated.messages = kw.get("messages", req.messages) + updated.runtime = runtime + updated.override = req.override + return updated + + req.override = _override + return req + + +def _make_mw(**kwargs) -> ToolProgressMiddleware: + defaults = { + "stagnation_threshold": 3, + "warn_escalation_count": 2, + "inject_assessment": True, + "jaccard_threshold": 0.8, + "min_words": 5, + } + defaults.update(kwargs) + return ToolProgressMiddleware(**defaults) + + +# --------------------------------------------------------------------------- +# Unit tests: word_set and is_near_duplicate + + +def test_word_set_extracts_words_ge_3(): + ws = word_set("go quick brown fox") + assert "go" not in ws + assert "quick" in ws + assert "brown" in ws + assert "fox" in ws + + +def test_is_near_duplicate_above_threshold(): + ws1 = frozenset("quick brown fox jumps over lazy dog".split()) + ws2 = frozenset("quick brown fox jumps over lazy dog".split()) + assert is_near_duplicate(ws2, [ws1], threshold=0.8, min_words=5) + + +def test_is_near_duplicate_near_threshold(): + # ws1 has 8 words; ws2 shares 7 of them and adds 1 new word. + # intersection=7, union=9 → Jaccard = 7/9 ≈ 0.778 < 0.8 → NOT duplicate. + # ws3 shares all 8 original words and adds 1 new word. + # intersection=8, union=9 → Jaccard = 8/9 ≈ 0.889 >= 0.8 → IS duplicate. + base = frozenset("alpha bravo charlie delta echo foxtrot golf hotel".split()) + nearly_below = frozenset("alpha bravo charlie delta echo foxtrot golf india".split()) # 7/9 ≈ 0.778 + nearly_above = frozenset("alpha bravo charlie delta echo foxtrot golf hotel india".split()) # 8/9 ≈ 0.889 + assert not is_near_duplicate(nearly_below, [base], threshold=0.8, min_words=5) + assert is_near_duplicate(nearly_above, [base], threshold=0.8, min_words=5) + + +def test_is_near_duplicate_below_threshold(): + ws1 = frozenset("apple banana cherry delta echo".split()) + ws2 = frozenset("xray yankee zulu alpha bravo".split()) + assert not is_near_duplicate(ws2, [ws1], threshold=0.8, min_words=5) + + +def test_is_near_duplicate_too_short_skips_check(): + ws1 = frozenset("apple".split()) + ws2 = frozenset("apple".split()) + # min_words=5 but len==1, so not a duplicate + assert not is_near_duplicate(ws2, [ws1], threshold=0.8, min_words=5) + + +# --------------------------------------------------------------------------- +# Scenario 1: Normal call → no hint, phase stays active + + +def test_normal_call_no_hint_phase_active(): + mw = _make_mw() + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + msg = _make_tool_message("A" * 300) + + def handler(_r): + return msg + + result = mw.wrap_tool_call(req, handler) + + assert result is msg + assert mw._phase_states["t1"]["web_search"].phase == "active" + assert mw._phase_states["t1"]["web_search"].consecutive_problems == 0 + + +# --------------------------------------------------------------------------- +# Scenario 2: consecutive no_results → hint injected, phase=warned + + +def test_repeated_no_results_reaches_warned(): + mw = _make_mw(stagnation_threshold=2) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + + def handler(_r): + return error_msg + + # stagnation_threshold=2, so the second problem call tips into warned + mw.wrap_tool_call(req, handler) + mw.wrap_tool_call(req, handler) + + state = mw._phase_states["t1"]["web_search"] + assert state.phase == "warned" + assert state.consecutive_problems == 2 + + # Hint should be queued + hints = mw._drain_pending(rt) + assert len(hints) == 1 + assert "PROGRESS HINT" in hints[0] + + +# --------------------------------------------------------------------------- +# Scenario 3: Non-recoverable errors escalate warned → blocked + + +def test_warned_to_blocked_after_escalation(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=2) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + # Non-recoverable error (rate_limited): model cannot fix this by retrying, + # so stagnation should escalate to BLOCKED. + error_msg = _make_non_recoverable_error_message() + + def handler(_r): + return error_msg + + for _ in range(4): + mw.wrap_tool_call(req, handler) + + state = mw._phase_states["t1"]["web_search"] + assert state.phase == "blocked" + assert state.block_reason is not None + + +# --------------------------------------------------------------------------- +# Scenario 4: Blocked tool is front-gate intercepted (handler NOT called) + + +def test_blocked_tool_is_intercepted_without_calling_handler(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=1) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + # Non-recoverable error: stagnation escalates to BLOCKED, handler is never called. + error_msg = _make_non_recoverable_error_message() + call_count = [0] + + def handler(r): + call_count[0] += 1 + return error_msg + + # 2 calls → warned + 1 more = blocked + for _ in range(3): + mw.wrap_tool_call(req, handler) + + assert mw._phase_states["t1"]["web_search"].phase == "blocked" + call_count_before = call_count[0] + + # Next call should be intercepted + result = mw.wrap_tool_call(req, handler) + + assert call_count[0] == call_count_before + assert isinstance(result, ToolMessage) + assert "[TOOL_BLOCKED]" in result.content + + +# --------------------------------------------------------------------------- +# Scenario 4b: Recoverable errors never escalate to BLOCKED — WARNED is terminal + + +def test_recoverable_errors_stay_warned_indefinitely(): + # stagnation_threshold=2, warn_escalation_count=1 → would block at call 3 for + # non-recoverable errors, but recoverable errors must stay in WARNED forever. + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=1) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() # recoverable_by_model=True + + def handler(_r): + return error_msg + + # 10 calls — well past the threshold+escalation + for _ in range(10): + mw.wrap_tool_call(req, handler) + + state = mw._phase_states["t1"]["web_search"] + assert state.phase == "warned", "recoverable errors must never escalate to BLOCKED" + assert state.consecutive_problems == 10 + + +def test_recoverable_error_re_injects_hint_past_escalation(): + # After crossing threshold+escalation for a recoverable error, each additional + # problem call should still queue a hint. + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=1, inject_assessment=True) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + + def handler(_r): + return error_msg + + # Reach warned (call 2) and past escalation (call 3+) + for _ in range(4): + mw.wrap_tool_call(req, handler) + + # All hints from call 2 onward should have been queued (capped at _MAX_PENDING_PER_RUN=3). + # >= 2 proves that at least one hint was queued *inside* the escalation zone (calls 3+), + # not just the initial WARNED hint at call 2. + hints = mw._drain_pending(rt) + assert len(hints) >= 2 + assert all("PROGRESS HINT" in h for h in hints) + + +# --------------------------------------------------------------------------- +# Scenario 5: Auth error → immediately blocked (no warned phase) + + +def test_auth_error_immediately_blocked(): + mw = _make_mw(stagnation_threshold=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + auth_msg = _make_error_message( + error_type="auth", + recoverable_by_model=False, + recommended_next_action="stop", + ) + + def handler(_r): + return auth_msg + + mw.wrap_tool_call(req, handler) + + state = mw._phase_states["t1"]["web_search"] + assert state.phase == "blocked" + assert "auth" in state.block_reason.lower() or "Authentication" in state.block_reason + # consecutive_problems must be 1 (not 0) even on immediate-block paths so diagnostic + # logs and future consumers see a consistent non-zero count after a failed call. + assert state.consecutive_problems == 1 + + +# --------------------------------------------------------------------------- +# Scenario 6: Valid result after problems resets to active + + +def test_valid_result_after_problems_resets_to_active(): + mw = _make_mw(stagnation_threshold=3, warn_escalation_count=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + good_msg = _make_tool_message("A" * 300) + + def handler_error(_r): + return error_msg + + def handler_good(_r): + return good_msg + + mw.wrap_tool_call(req, handler_error) + mw.wrap_tool_call(req, handler_error) + mw.wrap_tool_call(req, handler_error) + + state = mw._phase_states["t1"]["web_search"] + assert state.phase == "warned" + + # Good result resets + mw.wrap_tool_call(req, handler_good) + + state = mw._phase_states["t1"]["web_search"] + assert state.phase == "active" + assert state.consecutive_problems == 0 + + +# --------------------------------------------------------------------------- +# Scenario 7: Two different tools have independent states + + +def test_two_tools_have_independent_states(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=1) + rt = _make_runtime() + req_search = _make_tool_request("web_search", runtime=rt) + req_read = _make_tool_request("read_file", runtime=rt) + + # Non-recoverable errors so web_search escalates to BLOCKED. + error_search = _make_non_recoverable_error_message(tool_name="web_search") + error_read = _make_error_message(tool_name="read_file") + + # Drive web_search to BLOCKED (2 → warned, 1 more → blocked) + for _ in range(3): + mw.wrap_tool_call(req_search, lambda r: error_search) + + assert mw._phase_states["t1"]["web_search"].phase == "blocked" + + # read_file should still be active — independent state per tool name + mw.wrap_tool_call(req_read, lambda r: error_read) + assert mw._phase_states["t1"]["read_file"].phase == "active" + + +# --------------------------------------------------------------------------- +# Scenario 8: Jaccard near-duplicate result counts as problem + + +def test_jaccard_near_duplicate_counts_as_problem(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5, jaccard_threshold=0.8, min_words=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + + # First call: good unique content (establishes baseline) + words = "apple banana cherry delta echo foxtrot golf hotel india juliet" + msg1 = _make_tool_message(words) + mw.wrap_tool_call(req, lambda r: msg1) + + # Second call: exact same content (Jaccard = 1.0) → near-duplicate → problem count goes up + msg2 = _make_tool_message(words) + mw.wrap_tool_call(req, lambda r: msg2) + + state = mw._phase_states["t1"]["web_search"] + assert state.consecutive_problems >= 1 + + +# --------------------------------------------------------------------------- +# Scenario 9: Different Jaccard content does NOT count as problem + + +def test_jaccard_different_content_not_a_problem(): + mw = _make_mw(stagnation_threshold=3, warn_escalation_count=5, jaccard_threshold=0.8, min_words=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + + words1 = "apple banana cherry delta echo foxtrot golf hotel india juliet" + words2 = "xray yankee zulu alpha bravo charlie sierra tango uniform victor" + msg1 = _make_tool_message(words1) + msg2 = _make_tool_message(words2) + + mw.wrap_tool_call(req, lambda r: msg1) + mw.wrap_tool_call(req, lambda r: msg2) + + state = mw._phase_states["t1"]["web_search"] + assert state.consecutive_problems == 0 + assert state.phase == "active" + + +# --------------------------------------------------------------------------- +# Scenario 9b: production default min_words=10 skips Jaccard for short content + + +def test_jaccard_skipped_when_content_below_production_min_words(): + """Production default min_words=10 must skip Jaccard for content with 6-9 unique words. + + _make_mw() uses min_words=5 to make most tests easier to set up. This test + uses the production default (min_words=10) to verify that short but repeated + content does NOT count as a near-duplicate stagnation problem. + """ + mw = ToolProgressMiddleware( + stagnation_threshold=3, + warn_escalation_count=2, + jaccard_threshold=0.8, + min_words=10, # production default + ) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + + # 7 unique words — above min_words=5 but below production min_words=10. + # With min_words=10 the Jaccard check is skipped → never a problem → phase stays active. + words = "apple banana cherry delta echo foxtrot golf" + msg = _make_tool_message(words) + + for _ in range(5): + mw.wrap_tool_call(req, lambda r: msg) + + state = mw._phase_states["t1"]["web_search"] + assert state.phase == "active", "7-word repeated content must not trigger stagnation with production min_words=10" + assert state.consecutive_problems == 0 + + +# --------------------------------------------------------------------------- +# Scenario 10: exempt_tools are not tracked + + +def test_exempt_tools_not_tracked(): + mw = _make_mw(stagnation_threshold=1, warn_escalation_count=1) + rt = _make_runtime() + req = _make_tool_request("ask_clarification", runtime=rt) + error_msg = _make_error_message(tool_name="ask_clarification") + + def handler(_r): + return error_msg + + for _ in range(5): + mw.wrap_tool_call(req, handler) + + assert "ask_clarification" not in mw._phase_states.get("t1", {}) + + +# --------------------------------------------------------------------------- +# Scenario 11: before_agent clears stale pending hints from previous runs + + +def test_before_agent_clears_stale_pending(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt_run1 = _make_runtime(thread_id="t1", run_id="old-run") + rt_run2 = _make_runtime(thread_id="t1", run_id="new-run") + req = _make_tool_request(runtime=rt_run1) + error_msg = _make_error_message() + + # Produce a hint for old-run + mw.wrap_tool_call(req, lambda r: error_msg) + mw.wrap_tool_call(req, lambda r: error_msg) + + mw._drain_pending(rt_run1) + # Re-queue manually to simulate pending state + mw._queue_assessment(rt_run1, "old hint") + + # before_agent with new-run should clear the old-run's pending hints + state_mock = MagicMock() + mw.before_agent(state_mock, rt_run2) + + # Old pending should be gone + leftovers = mw._pending.get(("t1", "old-run"), []) + assert leftovers == [] + + +@pytest.mark.anyio +async def test_abefore_agent_clears_stale_pending(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt_run1 = _make_runtime(thread_id="t1", run_id="old-run") + rt_run2 = _make_runtime(thread_id="t1", run_id="new-run") + req = _make_tool_request(runtime=rt_run1) + error_msg = _make_error_message() + + mw.wrap_tool_call(req, lambda r: error_msg) + mw.wrap_tool_call(req, lambda r: error_msg) + mw._drain_pending(rt_run1) + mw._queue_assessment(rt_run1, "old hint") + + state_mock = MagicMock() + await mw.abefore_agent(state_mock, rt_run2) + + leftovers = mw._pending.get(("t1", "old-run"), []) + assert leftovers == [] + + +def test_before_agent_preserves_current_run_hints(): + # _clear_stale_pending deletes keys where thread_id matches but run_id differs. + # Hints for the *current* run must not be evicted. + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt = _make_runtime(thread_id="t1", run_id="current-run") + # _queue_assessment guards against phantom entries by checking _phase_states; seed the + # thread so the direct call below isn't silently dropped by the L1 guard. + mw._phase_states["t1"] = {} + mw._queue_assessment(rt, "current hint") + + state_mock = MagicMock() + mw.before_agent(state_mock, rt) + + preserved = mw._pending.get(("t1", "current-run"), []) + assert preserved == ["current hint"] + + +def test_before_agent_resets_blocked_states_for_new_run(): + """BLOCKED and WARNED tool states must both be cleared at the start of a new run. + + A tool BLOCKED in run R1 must not silently remain blocked in R2. + A tool WARNED in R1 must not carry its consecutive_problems count into R2 + (the model has not seen the warning context, so it would be hard-blocked + without ever receiving a hint in the current session). + recent_word_sets must also be cleared so stale Jaccard windows don't cause + false near-duplicate detections on the first success call of the new run. + """ + mw = _make_mw(stagnation_threshold=1, warn_escalation_count=1) + rt_run1 = _make_runtime(thread_id="t1", run_id="run-1") + rt_run2 = _make_runtime(thread_id="t1", run_id="run-2") + req = _make_tool_request(runtime=rt_run1) + + # Drive the tool to BLOCKED via auth error (immediate block, no WARN stage) + auth_msg = ToolMessage( + content="Error: invalid api key", + tool_call_id="tc-web_search", + name="web_search", + status="error", + additional_kwargs=_meta_kwargs( + status="error", + error_type="auth", + recoverable_by_model=False, + recommended_next_action="stop", + ), + ) + mw.wrap_tool_call(req, lambda _r: auth_msg) + assert mw._phase_states["t1"]["web_search"].phase == "blocked" + + # Simulate start of run 2 + state_mock = MagicMock() + mw.before_agent(state_mock, rt_run2) + + # _reset_run_states always replaces the entry in-place; it is never None. + tool_state = mw._phase_states.get("t1", {}).get("web_search") + assert tool_state is not None + assert tool_state.phase == "active" + assert tool_state.consecutive_problems == 0 + assert tool_state.block_reason is None + assert tool_state.recent_word_sets == () + + +def test_before_agent_resets_warned_states_for_new_run(): + """WARNED tool state must also be cleared by before_agent. + + A tool with phase='warned' and accumulated consecutive_problems at end of run R1 + must not carry that count into R2; the model has no warning context and would + be hard-blocked after just a few calls without receiving a hint. + """ + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt_run1 = _make_runtime(thread_id="t1", run_id="run-1") + rt_run2 = _make_runtime(thread_id="t1", run_id="run-2") + req = _make_tool_request(runtime=rt_run1) + error_msg = _make_error_message() + + # Drive to WARNED (stagnation_threshold=2 means 2 problems → warned) + mw.wrap_tool_call(req, lambda _r: error_msg) + mw.wrap_tool_call(req, lambda _r: error_msg) + assert mw._phase_states["t1"]["web_search"].phase == "warned" + assert mw._phase_states["t1"]["web_search"].consecutive_problems == 2 + + state_mock = MagicMock() + mw.before_agent(state_mock, rt_run2) + + tool_state = mw._phase_states.get("t1", {}).get("web_search") + assert tool_state is not None + assert tool_state.phase == "active" + assert tool_state.consecutive_problems == 0 + assert tool_state.recent_word_sets == () + + +def test_before_agent_resets_active_state_consecutive_problems_and_word_sets(): + """ACTIVE tools with sub-threshold problems must also be cleaned at run boundaries. + + An ACTIVE tool (phase never left 'active') can exit a run with non-zero + consecutive_problems and non-empty recent_word_sets. If _reset_run_states only + touched BLOCKED/WARNED tools, the counter from R1 would bleed into R2: a single + problem on R2's first call could then trip WARNED against stale R1 context that + the model has never seen. + """ + # stagnation_threshold=3 so two errors keep the tool ACTIVE. + mw = _make_mw(stagnation_threshold=3, warn_escalation_count=5) + rt_run1 = _make_runtime(thread_id="t1", run_id="run-1") + rt_run2 = _make_runtime(thread_id="t1", run_id="run-2") + req = _make_tool_request(runtime=rt_run1) + + # Two successes → recent_word_sets grows. + success_a = _make_tool_message("alpha beta gamma delta epsilon zeta eta theta iota kappa") + success_b = _make_tool_message("lambda mu nu xi omicron pi rho sigma tau upsilon phi chi") + mw.wrap_tool_call(req, lambda _r: success_a) + mw.wrap_tool_call(req, lambda _r: success_b) + + # One recoverable error → consecutive_problems=1, phase stays ACTIVE. + error_msg = _make_error_message() + mw.wrap_tool_call(req, lambda _r: error_msg) + + state_r1 = mw._phase_states.get("t1", {}).get("web_search") + assert state_r1 is not None + assert state_r1.phase == "active" + assert state_r1.consecutive_problems == 1 + assert len(state_r1.recent_word_sets) > 0 + + # Start of run 2: all per-run state must be cleared. + state_mock = MagicMock() + mw.before_agent(state_mock, rt_run2) + + state_r2 = mw._phase_states.get("t1", {}).get("web_search") + assert state_r2 is not None + assert state_r2.phase == "active" + assert state_r2.consecutive_problems == 0 + assert state_r2.recent_word_sets == () + + +# --------------------------------------------------------------------------- +# Scenario 12: LRU eviction when max_tracked_threads exceeded + + +def test_get_block_reason_does_not_create_phantom_entries(): + # _get_block_reason is called on every wrap_tool_call before the handler. + # It must not insert an empty entry for new threads (which could prematurely + # evict another thread's WARNED state via LRU). + mw = _make_mw(max_tracked_threads=2, stagnation_threshold=2) + rt_a = _make_runtime(thread_id="thread-a") + rt_b = _make_runtime(thread_id="thread-b") + rt_c = _make_runtime(thread_id="thread-c") + + req_a = _make_tool_request(runtime=rt_a) + error_msg = _make_error_message() + + # Drive thread-a to WARNED state (needs 2 error calls with threshold=2). + mw.wrap_tool_call(req_a, lambda r: error_msg) + mw.wrap_tool_call(req_a, lambda r: error_msg) + assert mw._phase_states["thread-a"]["web_search"].phase == "warned" + + # Drive thread-b so it has a real entry too. + req_b = _make_tool_request(runtime=rt_b) + good_msg = _make_tool_message("A" * 300) + mw.wrap_tool_call(req_b, lambda r: good_msg) + assert "thread-b" in mw._phase_states + + # Now thread-c makes its very first call. max_tracked_threads=2, so adding + # thread-c must evict one of {thread-a, thread-b} — but the eviction must + # only happen in _update_state_from_result (the write path), not in + # _get_block_reason (the read path that runs first). + # After wrap_tool_call completes, the two survivors should be thread-b and + # thread-c (thread-a is oldest because thread-b was accessed most recently). + req_c = _make_tool_request(tool_name="read_file", runtime=rt_c) + mw.wrap_tool_call(req_c, lambda r: good_msg) + + # thread-c must now have a real entry (not an empty phantom). + assert "thread-c" in mw._phase_states + assert mw._phase_states["thread-c"].get("read_file") is not None + + # No more than max_tracked_threads entries should exist. + assert len(mw._phase_states) <= 2 + + +def test_lru_eviction_of_oldest_thread(): + mw = _make_mw(max_tracked_threads=2) + error_msg = _make_error_message() + + for i in range(3): + rt = _make_runtime(thread_id=f"thread-{i}") + req = _make_tool_request(runtime=rt) + mw.wrap_tool_call(req, lambda r: error_msg) + + assert len(mw._phase_states) == 2 + # thread-0 should have been evicted (oldest); thread-1 and thread-2 remain + assert "thread-0" not in mw._phase_states + assert "thread-1" in mw._phase_states + assert "thread-2" in mw._phase_states + + +def test_pending_evicted_with_phase_states_on_lru_overflow(): + """M1 regression: _pending keys for evicted threads must be cleaned up. + + When _phase_states evicts a thread via LRU, any pending hint entries + for that thread must also be removed so _pending cannot grow unboundedly. + """ + mw = _make_mw(max_tracked_threads=2, stagnation_threshold=2) + error_msg = _make_error_message() + + # Thread-0: produce a hint (reach WARNED) so it has a pending entry. + rt0 = _make_runtime(thread_id="thread-0", run_id="run-0") + req0 = _make_tool_request(runtime=rt0) + mw.wrap_tool_call(req0, lambda r: error_msg) + mw.wrap_tool_call(req0, lambda r: error_msg) + # Verify thread-0 has a pending hint. + assert len(mw._pending.get(("thread-0", "run-0"), [])) >= 1 + + # Thread-1: occupy the second slot. + rt1 = _make_runtime(thread_id="thread-1", run_id="run-1") + req1 = _make_tool_request(runtime=rt1) + good_msg = _make_tool_message("A" * 300) + mw.wrap_tool_call(req1, lambda r: good_msg) + + # Thread-2: adding this forces LRU eviction of thread-0. + rt2 = _make_runtime(thread_id="thread-2", run_id="run-2") + req2 = _make_tool_request(runtime=rt2) + mw.wrap_tool_call(req2, lambda r: good_msg) + + # thread-0 must be evicted from phase_states. + assert "thread-0" not in mw._phase_states + + # The pending entry for thread-0 must also be gone (no memory leak). + assert ("thread-0", "run-0") not in mw._pending, "_pending entry for evicted thread-0 should have been cleaned up" + + +# --------------------------------------------------------------------------- +# Hint injection via wrap_model_call + + +def test_hint_injected_into_model_call(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + + # Trigger hint + mw.wrap_tool_call(req, lambda r: error_msg) + mw.wrap_tool_call(req, lambda r: error_msg) + + model_req = _make_model_request([], rt) + captured_messages = [] + + def model_handler(r): + captured_messages.extend(r.messages) + return MagicMock() + + mw.wrap_model_call(model_req, model_handler) + + assert any(isinstance(m, HumanMessage) for m in captured_messages) + hint_msgs = [m for m in captured_messages if isinstance(m, HumanMessage)] + assert any("PROGRESS HINT" in m.content for m in hint_msgs) + + +def test_partial_success_hint_is_specific_not_generic(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + partial_msg = ToolMessage( + content="Here are some partial results from the search.", + tool_call_id="tc-web_search", + name="web_search", + status="success", + additional_kwargs=_meta_kwargs( + status="partial_success", + recommended_next_action="rewrite_query", + ), + ) + + def handler(_r): + return partial_msg + + mw.wrap_tool_call(req, handler) + mw.wrap_tool_call(req, handler) + + hints = mw._drain_pending(rt) + assert len(hints) == 1 + assert "incomplete results" in hints[0].lower() + assert "not producing new information" not in hints[0] + + +def test_jaccard_near_dup_hint_is_specific_and_actionable(): + """Near-duplicate success hint must be specific (not generic fallback) and include action guidance. + + Before the fix, status='success'/error_type=None fell through to the generic fallback + '[PROGRESS HINT] The tool is not producing new information.' with no action suffix + (recommended_next_action='continue' was absent from action_map). The fix adds a + 'success' key to the base dict and a 'continue' key to the action_map. + """ + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5, jaccard_threshold=0.8, min_words=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + + # First call: good unique content to seed recent_word_sets. + words = "apple banana cherry delta echo foxtrot golf hotel india juliet" + good_msg = _make_tool_message(words) + mw.wrap_tool_call(req, lambda r: good_msg) + + # Second and third calls: exact same content → near-duplicate → stagnation_threshold=2 → WARNED. + dup_msg = _make_tool_message(words) + + def handler(_r): + return dup_msg + + mw.wrap_tool_call(req, handler) + mw.wrap_tool_call(req, handler) + + hints = mw._drain_pending(rt) + assert len(hints) == 1 + hint = hints[0] + # Must contain a specific near-dup message, not the generic fallback. + assert "duplicate" in hint.lower(), f"expected 'duplicate' in hint, got: {hint!r}" + # Must include an actionable suggestion (from action_map["continue"]). + assert "rephras" in hint.lower() or "different" in hint.lower(), f"expected action guidance in hint, got: {hint!r}" + + +def test_no_hint_when_inject_assessment_disabled(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5, inject_assessment=False) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + + mw.wrap_tool_call(req, lambda r: error_msg) + mw.wrap_tool_call(req, lambda r: error_msg) + + hints = mw._drain_pending(rt) + assert hints == [] + + +def test_augment_request_deduplicates_identical_hints(): + """L2: _augment_request must deduplicate identical hint strings via dict.fromkeys. + + If the same hint text appears multiple times in the queue (e.g. two successive + no_results errors produce identical hint strings), only one copy should be + injected into the model message. + """ + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5, inject_assessment=True) + rt = _make_runtime() + + # _queue_assessment guards against phantom entries; seed the thread so the direct + # calls below aren't dropped by the L1 guard. + mw._phase_states["t1"] = {} + # Manually queue two identical hints to simulate duplicates. + mw._queue_assessment(rt, "[PROGRESS HINT] same hint text") + mw._queue_assessment(rt, "[PROGRESS HINT] same hint text") + + model_req = _make_model_request([], rt) + captured: list = [] + + def model_handler(r): + captured.extend(r.messages) + return MagicMock() + + mw.wrap_model_call(model_req, model_handler) + + hint_msgs = [m for m in captured if isinstance(m, HumanMessage)] + assert len(hint_msgs) == 1 + # The single injected message must contain the hint exactly once. + assert hint_msgs[0].content.count("[PROGRESS HINT] same hint text") == 1 + + +# --------------------------------------------------------------------------- +# L1: _assess_and_transition called with already-blocked state is idempotent + + +def test_assess_and_transition_blocked_state_immediate_stop_is_idempotent(): + """L1: _assess_and_transition must handle an already-blocked state without error. + + The docstring states the immediate-block branch re-applies idempotently. + This test verifies that re-entering with a blocked state + stop-action meta + stays blocked and does not corrupt the block_reason. + """ + from deerflow.agents.middlewares.tool_progress_middleware import ToolPhaseState + + mw = _make_mw() + blocked_state = ToolPhaseState( + phase="blocked", + consecutive_problems=5, + block_reason="Authentication failure — this tool cannot be used.", + ) + auth_meta_kwargs = _meta_kwargs( + status="error", + error_type="auth", + recoverable_by_model=False, + recommended_next_action="stop", + )[TOOL_META_KEY] + from deerflow.agents.middlewares.tool_result_meta import ToolResultMeta + + auth_meta = ToolResultMeta(**auth_meta_kwargs) + + new_state, hint = mw._assess_and_transition(blocked_state, auth_meta, "") + + assert new_state.phase == "blocked" + assert new_state.block_reason is not None + assert hint is None # no hint on immediate block path + + +def test_assess_and_transition_blocked_state_non_stop_increments_count(): + """L1: A blocked state receiving a non-stop problem increments counter, stays blocked. + + Simulates a concurrent race where two threads both process results for the + same tool: the second thread's _assess_and_transition receives a stale + 'blocked' snapshot. The result must remain blocked. + """ + from deerflow.agents.middlewares.tool_progress_middleware import ToolPhaseState + + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=1) + blocked_state = ToolPhaseState( + phase="blocked", + consecutive_problems=3, + block_reason="Repeated rate-limiting — summarize current findings and proceed.", + ) + rate_meta_kwargs = _meta_kwargs( + status="error", + error_type="rate_limited", + recoverable_by_model=False, + recommended_next_action="summarize", + )[TOOL_META_KEY] + from deerflow.agents.middlewares.tool_result_meta import ToolResultMeta + + rate_meta = ToolResultMeta(**rate_meta_kwargs) + + new_state, _hint = mw._assess_and_transition(blocked_state, rate_meta, "") + + # Must stay blocked (not regress to warned or active). + assert new_state.phase == "blocked" + # Counter must NOT be incremented: blocked is terminal, state returned unchanged. + assert new_state.consecutive_problems == 3 + + +def test_assess_and_transition_blocked_recoverable_does_not_regress_to_warned(): + """L1: A blocked state with recoverable errors must not silently regress to warned. + + Before the fix, _assess_and_transition had no guard for already-blocked states. + A recoverable error arriving on a blocked state (concurrent race) would take + the `warned` branch because recoverable_by_model=True, demoting the phase from + blocked back to warned. This test locks the fixed behavior. + """ + from deerflow.agents.middlewares.tool_progress_middleware import ToolPhaseState + + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=1) + blocked_state = ToolPhaseState( + phase="blocked", + consecutive_problems=5, + block_reason="Repeated no-results — rewrite your query or try a different tool.", + ) + # Recoverable no_results error (would normally only WARN, never block on its own) + no_results_meta_kwargs = _meta_kwargs( + status="error", + error_type="no_results", + recoverable_by_model=True, + recommended_next_action="rewrite_query", + )[TOOL_META_KEY] + from deerflow.agents.middlewares.tool_result_meta import ToolResultMeta + + no_results_meta = ToolResultMeta(**no_results_meta_kwargs) + + new_state, hint = mw._assess_and_transition(blocked_state, no_results_meta, "") + + assert new_state.phase == "blocked", "blocked must not regress to warned even when the new error is recoverable" + assert hint is None + assert new_state is blocked_state # exact same object returned (no copy) + + +# --------------------------------------------------------------------------- +# Tool without runtime attribute is passed through + + +def test_no_runtime_passthrough(): + mw = _make_mw() + req = SimpleNamespace(tool_call={"name": "web_search", "id": "tc-1"}) + # No runtime attribute + msg = _make_tool_message() + + def handler(_r): + return msg + + result = mw.wrap_tool_call(req, handler) + assert result is msg + + +# --------------------------------------------------------------------------- +# Command results are passed through unchanged + + +def test_command_result_passthrough(): + mw = _make_mw() + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + cmd = Command(goto="some_node") + + def handler(_r): + return cmd + + result = mw.wrap_tool_call(req, handler) + assert result is cmd + + +# --------------------------------------------------------------------------- +# from_config round-trip + + +def test_from_config(): + from deerflow.config.tool_progress_config import ToolProgressConfig + + cfg = ToolProgressConfig( + enabled=True, + stagnation_threshold=4, + warn_escalation_count=3, + jaccard_similarity_threshold=0.7, + min_word_count_for_similarity=8, + ) + mw = ToolProgressMiddleware.from_config(cfg) + assert mw._stagnation_threshold == 4 + assert mw._warn_escalation == 3 + assert mw._jaccard_threshold == pytest.approx(0.7) + assert mw._min_words == 8 + + +def test_from_config_empty_exempt_tools_clears_exemptions(): + """Empty exempt_tools in config must produce an empty set, not the default fallback. + + H1 regression: `exempt_tools or {default}` would silently ignore an empty set + because set() is falsy in Python. The fix uses `is not None` so an explicit + empty set from config actually disables all exemptions. + """ + from deerflow.config.tool_progress_config import ToolProgressConfig + + cfg = ToolProgressConfig(enabled=True, exempt_tools=set()) + mw = ToolProgressMiddleware.from_config(cfg) + assert mw._exempt_tools == set(), "empty exempt_tools in config must clear all exemptions, not fall back to defaults" + + +def test_exempt_tools_none_uses_defaults(): + """None exempt_tools in __init__ must use the built-in default set.""" + mw = ToolProgressMiddleware(exempt_tools=None) + assert "ask_clarification" in mw._exempt_tools + assert "write_todos" in mw._exempt_tools + assert "present_files" in mw._exempt_tools + + +def test_from_config_default_exempt_tools_round_trip(): + """Default exempt_tools from config must match the __init__ default.""" + from deerflow.config.tool_progress_config import ToolProgressConfig + + cfg = ToolProgressConfig(enabled=True) + mw = ToolProgressMiddleware.from_config(cfg) + assert mw._exempt_tools == {"ask_clarification", "write_todos", "present_files", "task"} + + +# --------------------------------------------------------------------------- +# Defensive meta parsing: malformed dicts must not crash the middleware + + +def test_wrap_tool_call_malformed_meta_passthrough(): + """Malformed deerflow_tool_meta dict must not crash the middleware.""" + mw = _make_mw() + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + bad_msg = ToolMessage( + content="some content", + tool_call_id="tc-web_search", + name="web_search", + status="success", + additional_kwargs={TOOL_META_KEY: {"unexpected_field": True}}, + ) + + def handler(_r): + return bad_msg + + result = mw.wrap_tool_call(req, handler) + + assert result is bad_msg + assert mw._phase_states.get("t1", {}).get("web_search") is None + + +def test_missing_meta_on_non_exempt_tool_emits_warning(caplog): + """When deerflow_tool_meta is completely absent for a non-exempt tool, + the middleware must emit a warning pointing to the likely ordering misconfiguration. + """ + import logging + + mw = _make_mw() + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + no_meta_msg = ToolMessage( + content="some content", + tool_call_id="tc-web_search", + name="web_search", + status="success", + additional_kwargs={}, # no TOOL_META_KEY at all + ) + + with caplog.at_level(logging.WARNING, logger="deerflow.agents.middlewares.tool_progress_middleware"): + mw.wrap_tool_call(req, lambda _r: no_meta_msg) + + assert any("deerflow_tool_meta missing" in r.message for r in caplog.records), "Expected a warning about missing meta for non-exempt tool" + + +# --------------------------------------------------------------------------- +# Async path: awrap_tool_call mirrors sync path + + +@pytest.mark.anyio +async def test_awrap_tool_call_normal_passthrough(): + mw = _make_mw() + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + msg = _make_tool_message("A" * 300) + + result = await mw.awrap_tool_call(req, AsyncMock(return_value=msg)) + + assert result is msg + assert mw._phase_states["t1"]["web_search"].phase == "active" + + +@pytest.mark.anyio +async def test_awrap_tool_call_blocked_intercepted_without_calling_handler(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=1) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + # Non-recoverable error: stagnation escalates to BLOCKED. + error_msg = _make_non_recoverable_error_message() + call_count = [0] + + async def handler(r): + call_count[0] += 1 + return error_msg + + # 3 calls: 2 → warned, 1 more → blocked + for _ in range(3): + await mw.awrap_tool_call(req, handler) + + assert mw._phase_states["t1"]["web_search"].phase == "blocked" + before = call_count[0] + + result = await mw.awrap_tool_call(req, handler) + + assert call_count[0] == before + assert isinstance(result, ToolMessage) + assert "[TOOL_BLOCKED]" in result.content + + +@pytest.mark.anyio +async def test_awrap_tool_call_auth_error_immediately_blocked(): + mw = _make_mw(stagnation_threshold=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + auth_msg = _make_error_message( + error_type="auth", + recoverable_by_model=False, + recommended_next_action="stop", + ) + + await mw.awrap_tool_call(req, AsyncMock(return_value=auth_msg)) + + state = mw._phase_states["t1"]["web_search"] + assert state.phase == "blocked" + assert state.block_reason is not None + + +@pytest.mark.anyio +async def test_awrap_tool_call_no_runtime_passthrough(): + mw = _make_mw() + req = SimpleNamespace(tool_call={"name": "web_search", "id": "tc-1"}) + msg = _make_tool_message() + + result = await mw.awrap_tool_call(req, AsyncMock(return_value=msg)) + + assert result is msg + assert "t1" not in mw._phase_states + + +@pytest.mark.anyio +async def test_awrap_tool_call_command_result_passthrough(): + mw = _make_mw() + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + cmd = Command(goto="some_node") + + result = await mw.awrap_tool_call(req, AsyncMock(return_value=cmd)) + + assert result is cmd + + +@pytest.mark.anyio +async def test_awrap_tool_call_malformed_meta_passthrough(): + """Malformed deerflow_tool_meta dict must not crash the middleware.""" + mw = _make_mw() + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + bad_msg = ToolMessage( + content="some content", + tool_call_id="tc-web_search", + name="web_search", + status="success", + additional_kwargs={TOOL_META_KEY: {"unexpected_field": True}}, + ) + + result = await mw.awrap_tool_call(req, AsyncMock(return_value=bad_msg)) + + assert result is bad_msg + # No state was tracked — malformed meta is silently skipped + assert mw._phase_states.get("t1", {}).get("web_search") is None + + +@pytest.mark.anyio +async def test_awrap_model_call_drains_and_injects_hints(): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + + # Trigger hint via sync path (state machine is shared) + mw.wrap_tool_call(req, lambda r: error_msg) + mw.wrap_tool_call(req, lambda r: error_msg) + + model_req = _make_model_request([], rt) + captured: list = [] + + async def model_handler(r): + captured.extend(r.messages) + return MagicMock() + + await mw.awrap_model_call(model_req, model_handler) + + hint_msgs = [m for m in captured if isinstance(m, HumanMessage)] + assert any("PROGRESS HINT" in m.content for m in hint_msgs) + + +# --------------------------------------------------------------------------- +# Logging behavior + +_MW_LOGGER = "deerflow.agents.middlewares.tool_progress_middleware" + + +def test_log_active_to_warned_emits_info(caplog): + mw = _make_mw(stagnation_threshold=2) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + + with caplog.at_level(logging.INFO, logger=_MW_LOGGER): + mw.wrap_tool_call(req, lambda _r: error_msg) + mw.wrap_tool_call(req, lambda _r: error_msg) + + info_records = [r for r in caplog.records if r.levelname == "INFO" and "WARNED" in r.message] + assert len(info_records) == 1 + assert "web_search" in info_records[0].message + + +def test_log_immediate_block_emits_warning(caplog): + mw = _make_mw(stagnation_threshold=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + auth_msg = _make_error_message( + error_type="auth", + recoverable_by_model=False, + recommended_next_action="stop", + ) + + with caplog.at_level(logging.WARNING, logger=_MW_LOGGER): + mw.wrap_tool_call(req, lambda _r: auth_msg) + + warning_records = [r for r in caplog.records if r.levelname == "WARNING" and "BLOCKED" in r.message] + assert len(warning_records) == 1 + assert "web_search" in warning_records[0].message + + +def test_log_escalation_block_emits_warning(caplog): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=2) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_non_recoverable_error_message() + + with caplog.at_level(logging.WARNING, logger=_MW_LOGGER): + for _ in range(4): + mw.wrap_tool_call(req, lambda _r: error_msg) + + warning_records = [r for r in caplog.records if r.levelname == "WARNING" and "BLOCKED" in r.message] + assert len(warning_records) == 1 + + +def test_log_blocked_call_intercepted_emits_info(caplog): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=1) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_non_recoverable_error_message() + + for _ in range(3): + mw.wrap_tool_call(req, lambda _r: error_msg) + + with caplog.at_level(logging.INFO, logger=_MW_LOGGER): + mw.wrap_tool_call(req, lambda _r: error_msg) + + intercepted = [r for r in caplog.records if "intercepted" in r.message and "web_search" in r.message] + assert len(intercepted) == 1 + + +def test_log_warned_to_active_reset_emits_info(caplog): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + good_msg = _make_tool_message("A" * 300) + + # Drive to WARNED + mw.wrap_tool_call(req, lambda _r: error_msg) + mw.wrap_tool_call(req, lambda _r: error_msg) + + with caplog.at_level(logging.INFO, logger=_MW_LOGGER): + mw.wrap_tool_call(req, lambda _r: good_msg) + + reset_records = [r for r in caplog.records if r.levelname == "INFO" and "ACTIVE" in r.message] + assert len(reset_records) == 1 + assert "web_search" in reset_records[0].message + + +def test_log_hint_injection_emits_debug(caplog): + mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5) + rt = _make_runtime() + req = _make_tool_request(runtime=rt) + error_msg = _make_error_message() + + mw.wrap_tool_call(req, lambda _r: error_msg) + mw.wrap_tool_call(req, lambda _r: error_msg) + + model_req = _make_model_request([], rt) + with caplog.at_level(logging.DEBUG, logger=_MW_LOGGER): + mw.wrap_model_call(model_req, lambda _r: MagicMock()) + + debug_records = [r for r in caplog.records if r.levelname == "DEBUG" and "injecting" in r.message] + assert len(debug_records) == 1 + assert "injecting 1 hint" in debug_records[0].message + + +# --------------------------------------------------------------------------- +# Coexistence: ToolProgressMiddleware + LoopDetectionMiddleware + + +def test_tool_progress_and_loop_detection_coexist_without_interfering(): + """ToolProgressMiddleware and LoopDetectionMiddleware operate on separate signals + and must not interfere when both are active simultaneously. + + ToolProgressMiddleware (position 8): result-quality guard, fires after tool execution, + tracks per-(thread, tool) stagnation, BLOCKs specific tools. + LoopDetectionMiddleware (position 19): call-pattern guard, fires after model response, + tracks repeated tool_call signatures, hard-stops the whole turn. + + Both can inject HumanMessage hints in the same model call; neither reads or writes + the other's internal state. + """ + from langchain_core.messages import AIMessage + + from deerflow.agents.middlewares.loop_detection_middleware import LoopDetectionMiddleware + + tp_mw = _make_mw(stagnation_threshold=2, warn_escalation_count=5, inject_assessment=True) + ld_mw = LoopDetectionMiddleware(warn_threshold=3, hard_limit=10) + + tp_rt = _make_runtime(thread_id="t1", run_id="r1") + # LoopDetection uses its own runtime/thread context + ld_rt = _make_runtime(thread_id="ld-thread", run_id="ld-run") + req = _make_tool_request(runtime=tp_rt) + + # --- Drive ToolProgress to WARNED via repeated error results (result-quality signal) --- + error_msg = _make_error_message() # recoverable error, stagnation_threshold=2 + tp_mw.wrap_tool_call(req, lambda _: error_msg) + tp_mw.wrap_tool_call(req, lambda _: error_msg) + + assert tp_mw._phase_states["t1"]["web_search"].phase == "warned" + tp_hints = list(tp_mw._pending.get(("t1", "r1"), [])) + assert len(tp_hints) == 1, "ToolProgress must queue exactly one hint at stagnation" + + # --- Drive LoopDetection to WARNED via repeated AIMessage tool_calls (call-pattern signal) --- + repeated_call = [{"name": "web_search", "args": {"query": "q"}, "id": "tc-1"}] + ld_state = {"messages": [AIMessage(content="", tool_calls=repeated_call)]} + for _ in range(3): # warn_threshold=3 + ld_mw._apply(ld_state, ld_rt) + + ld_warnings_live = ld_mw._pending_warnings.get(("ld-thread", "ld-run"), []) + assert len(ld_warnings_live) >= 1, "LoopDetection must queue at least one warning" + # Snapshot a copy so the final cross-contamination check compares a frozen + # baseline to the live state — a same-object comparison would always be True. + ld_warnings_snapshot = list(ld_warnings_live) + + # --- Verify no cross-contamination between the two middlewares --- + # ToolProgress internal state is not visible to LoopDetection + assert not hasattr(ld_mw, "_phase_states"), "LoopDetection must not have _phase_states" + # LoopDetection internal state is not visible to ToolProgress + assert not hasattr(tp_mw, "_history"), "ToolProgress must not have _history" + # LoopDetection does not track ToolProgress's thread id + assert "t1" not in ld_mw._history, "LoopDetection must not have entries for ToolProgress's thread" + # ToolProgress does not have loop detection warnings + assert not any("LOOP" in h for h in tp_hints), "ToolProgress hints must not contain loop-detection text" + + # --- ToolProgress hint injection is independent of LoopDetection --- + model_req = _make_model_request([], tp_rt) + captured: list = [] + + def capture_handler(r): + captured.extend(r.messages) + return MagicMock() + + tp_mw.wrap_model_call(model_req, capture_handler) + injected = [m for m in captured if isinstance(m, HumanMessage)] + assert len(injected) == 1, "ToolProgress must inject exactly one hint message" + assert "PROGRESS HINT" in injected[0].content + + # After ToolProgress drains, its queue is empty; LoopDetection warnings unchanged. + # Compare live state against the snapshot taken before the model call — a same-object + # comparison would be trivially True and would not detect accidental modifications. + assert tp_mw._pending.get(("t1", "r1"), []) == [] + assert ld_mw._pending_warnings.get(("ld-thread", "ld-run"), []) == ld_warnings_snapshot diff --git a/backend/tests/test_tool_result_meta.py b/backend/tests/test_tool_result_meta.py new file mode 100644 index 00000000000..d5be2a15d54 --- /dev/null +++ b/backend/tests/test_tool_result_meta.py @@ -0,0 +1,474 @@ +"""Tests for tool_result_meta normalization logic.""" + +from __future__ import annotations + +import json + +import pytest +from langchain_core.messages import ToolMessage +from langgraph.types import Command + +from deerflow.agents.middlewares.tool_result_meta import ( + TOOL_META_KEY, + ToolResultMeta, + normalize_tool_message, + normalize_tool_result, + stamp_exception_meta, +) + + +def _make_msg(content: str, *, status: str = "success", kwargs: dict[str, object] | None = None) -> ToolMessage: + return ToolMessage( + content=content, + tool_call_id="tc-1", + name="test_tool", + status=status, + additional_kwargs=kwargs or {}, + ) + + +def _meta(msg: ToolMessage) -> dict[str, object]: + return msg.additional_kwargs[TOOL_META_KEY] + + +# --------------------------------------------------------------------------- +# Already-stamped messages are not overwritten + + +def test_existing_meta_is_preserved(): + existing = {"status": "success", "source": "custom"} + msg = _make_msg("hello", kwargs={TOOL_META_KEY: existing}) + result = normalize_tool_message(msg) + assert result.additional_kwargs[TOOL_META_KEY] is existing + + +# --------------------------------------------------------------------------- +# Error prefix (tool_return path) + + +@pytest.mark.parametrize( + "snippet,expected_type", + [ + ("Error: 401 unauthorized", "auth"), + ("Error: permission denied for path", "permission"), + ("Error: 429 rate limit exceeded", "rate_limited"), + ("Error: connection timeout", "transient"), + ("Error: tool not configured", "config"), + ("Error: no results found for query", "no_results"), + ("Error: file not found", "not_found"), + ("Error: internal error 500", "internal"), + ("Error: something completely unexpected happened", "unknown"), + ], +) +def test_error_prefix_classification(snippet: str, expected_type: str): + msg = _make_msg(snippet, status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == expected_type + assert m["source"] == "tool_return" + + +def test_auth_error_is_unrecoverable_and_stop(): + msg = _make_msg("Error: invalid api key", status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["recoverable_by_model"] is False + assert m["recommended_next_action"] == "stop" + + +def test_no_api_key_is_config_not_auth(): + # Distinguish "missing key" (config) from "wrong key" (auth): + # - auth rule keyword: "invalid api key" (key provided but rejected) + # - config rule keyword: "no api key" (key not set at all) + # The two phrases do not overlap, so rule order does not affect this particular + # case. This test documents the semantic distinction — a missing API key is a + # configuration issue, not an authentication failure. + msg = _make_msg("Error: no api key configured", status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["error_type"] == "config", "missing API key is a config issue, not auth" + assert m["recommended_next_action"] == "stop" + assert m["recoverable_by_model"] is False + + +def test_rate_limited_error_suggests_summarize(): + msg = _make_msg("Error: rate limited", status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["recommended_next_action"] == "summarize" + assert m["recoverable_by_model"] is False + + +def test_no_results_suggests_rewrite_query(): + msg = _make_msg("Error: no results found", status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["recoverable_by_model"] is True + assert m["recommended_next_action"] == "rewrite_query" + + +# --------------------------------------------------------------------------- +# Non-standard error path (status="error", no "Error:" prefix) + + +def test_nonstd_error_status_classifies_from_content(): + # Tools that return status="error" without the "Error:" prefix are tool_return, not exception. + # Actual exceptions are pre-stamped by stamp_exception_meta and exit normalize_tool_message early. + msg = _make_msg("ConnectionError: connection refused", status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["source"] == "tool_return" + assert m["error_type"] == "transient" + + +def test_nonstd_error_status_timeout_content(): + msg = _make_msg("timeout occurred", status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["source"] == "tool_return" + assert m["error_type"] == "transient" + + +def test_nonstd_error_status_json_classifies_from_error_field(): + # When status="error" and content is JSON, classification must use only the "error" + # field value — not keywords that appear in other fields like "query". + content = '{"error": "api limit exceeded", "query": "connection test timeout"}' + msg = _make_msg(content, status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["source"] == "tool_return" + # "connection" and "timeout" in query must not trigger transient; the error field + # "api limit exceeded" doesn't match any rule → unknown. + assert m["error_type"] == "unknown" + + +def test_nonstd_error_status_json_no_error_key_is_unknown(): + # JSON with no 'error' key must NOT be classified from other field values. + # Previously, {"message": "connection refused"} would be passed to _classify_error_text + # and match the transient rule via "connection"; now the full JSON is treated as unknown. + content = '{"message": "connection refused"}' + msg = _make_msg(content, status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == "unknown", "JSON dict with no 'error' key must resolve to unknown" + + +def test_nonstd_error_status_json_no_error_key_with_dangerous_field_is_unknown(): + # {"user_id": 401} previously triggered auth stop; must now be unknown. + content = '{"user_id": 401, "action": "login"}' + msg = _make_msg(content, status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == "unknown" + assert m["recommended_next_action"] != "stop", "spurious 401 in non-error field must not trigger stop" + + +def test_nonstd_error_status_non_json_content_still_classified(): + # Plain text (not JSON) with status="error" must still be classified from content. + content = "connection refused: remote host unreachable" + msg = _make_msg(content, status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == "transient" + + +def test_json_error_field_dict_is_serialized_not_repr(): + # FastAPI-style: {"error": [{"loc": ["body"], "msg": "missing required field"}]} + # str() would produce Python repr containing 'missing required' → config → stop. + # json.dumps produces a clean JSON string that should not spuriously match. + import json as _json + + error_val = [{"loc": ["body"], "msg": "missing required field"}] + content = _json.dumps({"error": error_val}) + msg = _make_msg(content, status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + # The JSON-serialized error list contains "missing required field" which IS in the config rule. + # This is correct classification (the validation error IS a config-class problem). + # The key requirement is that we're classifying from the error field value, not repr noise. + assert m["error_type"] == "config" + + +def test_no_results_success_response_is_partial_success(): + # Tools that return status="success" with "no results found" content must be treated as + # partial_success so ToolProgressMiddleware can detect stagnation. + for phrase in ("no results found", "No Content Found here", "no images found for query"): + msg = _make_msg(phrase, status="success") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "partial_success", f"expected partial_success for: {phrase!r}" + assert m["recommended_next_action"] == "rewrite_query" + + +# --------------------------------------------------------------------------- +# Partial success detection + + +def test_partial_markers_detected(): + for marker in ("partial results available", "limited results returned", "truncated output", "results may be incomplete"): + msg = _make_msg(f"Here are some {marker} from the search.", status="success") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "partial_success", f"expected partial_success for: {marker}" + assert m["recommended_next_action"] == "rewrite_query" + + +def test_short_terse_success_is_not_partial(): + # "Ok." is a valid, complete success response from mutation tools like write_file/str_replace. + # partial_success is now gated only on _PARTIAL_MARKERS, not content length. + msg = _make_msg("Ok.", status="success") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "success" + assert m["source"] == "content_analysis" + + +def test_empty_content_is_not_partial(): + # Empty content has no partial markers, so it falls through to success. + msg = _make_msg("", status="success") + result = normalize_tool_message(msg) + m = _meta(result) + # Empty content falls through to success (no partial markers) + assert m["status"] == "success" + + +# --------------------------------------------------------------------------- +# Success path + + +def test_substantial_content_is_success(): + content = "A" * 200 + msg = _make_msg(content, status="success") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "success" + assert m["source"] == "content_analysis" + assert m["recommended_next_action"] == "continue" + assert m["error_type"] is None + + +# --------------------------------------------------------------------------- +# ToolResultMeta dataclass round-trip + + +def test_tool_result_meta_from_dict(): + msg = _make_msg("A" * 200) + result = normalize_tool_message(msg) + meta_dict = _meta(result) + meta = ToolResultMeta(**meta_dict) + assert meta.status == "success" + assert meta.error_type is None + assert meta.recommended_next_action == "continue" + + +# --------------------------------------------------------------------------- +# stamp_exception_meta + + +def test_stamp_exception_meta_classifies_from_exc_info_not_content(): + # Content says "no results" but exc_info says "connection refused" — + # stamp_exception_meta must use exc_info, producing transient, not no_results. + msg = _make_msg("Error: no results found", status="error") + result = stamp_exception_meta(msg, "ConnectionError: connection refused") + m = _meta(result) + assert m["source"] == "exception" + assert m["error_type"] == "transient" + + +def test_stamp_exception_meta_overwrites_existing_meta(): + pre_existing = {TOOL_META_KEY: {"source": "tool_return", "error_type": "unknown"}} + msg = _make_msg("Error: no results found", status="error", kwargs=pre_existing) + result = stamp_exception_meta(msg, "PermissionError: access denied") + m = _meta(result) + assert m["source"] == "exception" + assert m["error_type"] == "permission" + + +def test_stamp_exception_meta_preserves_other_additional_kwargs(): + msg = _make_msg("irrelevant", status="error", kwargs={"subagent_status": "running"}) + result = stamp_exception_meta(msg, "TimeoutError: timed out") + assert result.additional_kwargs["subagent_status"] == "running" + assert TOOL_META_KEY in result.additional_kwargs + + +# --------------------------------------------------------------------------- +# normalize_tool_result handles Command wrappers + + +def test_normalize_tool_result_passthrough_command(): + cmd = Command(goto="next_node") + result = normalize_tool_result(cmd) + assert result is cmd + + +def test_normalize_tool_result_stamps_tool_message(): + msg = _make_msg("A" * 200) + result = normalize_tool_result(msg) + assert isinstance(result, ToolMessage) + assert TOOL_META_KEY in result.additional_kwargs + + +# --------------------------------------------------------------------------- +# JSON-wrapped error detection + + +def test_normalize_json_error_config_classified_as_error(): + content = '{"error": "BRAVE_SEARCH_API_KEY is not configured", "query": "test"}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == "config" + assert m["source"] == "tool_return" + + +def test_normalize_json_error_no_results_classified_correctly(): + content = '{"error": "No results found", "query": "test"}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == "no_results" + assert m["recoverable_by_model"] is True + + +def test_normalize_json_null_error_not_treated_as_error(): + content = '{"error": null, "query": "test"}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] != "error" + + +def test_normalize_json_no_error_key_not_treated_as_error(): + content = '{"results": [{"title": "page one", "url": "https://example.com/one", "content": "summary one"}], "total": 1}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "success" + + +def test_normalize_malformed_json_not_treated_as_error(): + content = '{"error": "broken json' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] != "error" + + +def test_normalize_json_error_with_leading_whitespace(): + content = ' {"error": "No results found", "query": "test"}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == "no_results" + + +def test_normalize_json_numeric_error_classified_correctly(): + content = '{"error": 404, "query": "test"}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == "not_found" + + +def test_normalize_json_zero_error_not_treated_as_error(): + content = '{"error": 0, "query": "test"}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] != "error" + + +def test_normalize_json_false_error_not_treated_as_error(): + content = '{"error": false, "query": "test"}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] != "error" + + +def test_normalize_json_boolean_true_error_classified_as_unknown(): + """Boolean True in the error field means 'an error occurred' and must be classified. + + str(True) = "True" which matches no keyword rule, so the result is error/unknown. + This is intentional: a boolean True error is a real error with no further detail. + """ + content = '{"error": true, "query": "test"}' + msg = _make_msg(content) + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == "unknown" # str(True)="True" matches no keyword rule + assert m["recoverable_by_model"] is True + assert m["recommended_next_action"] == "try_alternative" + + +# --------------------------------------------------------------------------- +# M2 regression: semantic-zero error strings must NOT be treated as errors + + +@pytest.mark.parametrize( + "error_value", + ["none", "None", "NONE", "null", "Null", "false", "False", "no", "ok", "success", "n/a", ""], +) +def test_normalize_json_semantic_zero_error_string_not_treated_as_error(error_value: str): + """M2 regression: error field containing a conventional 'no-error' string must not trigger misclassification. + + Tools sometimes return {"error": "none", "results": [...]} on success. + The string "none" is truthy in Python, so without this guard the message + would have been classified as error (unknown), inflating stagnation counters. + + Note: the empty-string case ("") is handled by the falsy guard (`if not error: return None`) + in _extract_json_error_text rather than by _SEMANTIC_ZERO_ERROR_STRINGS. Both paths produce + the same outcome (no misclassification), but the mechanism differs from the other cases here. + """ + content = json.dumps({"error": error_value, "results": ["item1", "item2", "item3"]}) + msg = _make_msg(content, status="success") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] != "error", f'error="{error_value}" should not be treated as an error; got status={m["status"]!r}' + + +# --------------------------------------------------------------------------- +# Numeric keyword word-boundary matching (_match_keyword) + + +@pytest.mark.parametrize( + "content, expected_error_type", + [ + # Positive: numeric code at a word boundary → correct classification + ("Error: HTTP 500 Internal Server Error", "internal"), + ("Error: returned status 500", "internal"), + ("Error: 401 Unauthorized", "auth"), + ("Error: 404 Not Found", "not_found"), + # Negative: numeric code embedded inside a longer token → must resolve to "unknown". + # Use exact "unknown" assertions so any future rule additions that accidentally + # absorb these strings are caught (a broad exclusion list would miss new matches). + ("Error: took 500ms to respond", "unknown"), + ("Error: query returned 4010 rows", "unknown"), + ("Error: batch 401A failed", "unknown"), + ("Error: response contained 5000 items", "unknown"), + ], +) +def test_numeric_keyword_word_boundary(content: str, expected_error_type: str): + """Numeric HTTP codes must match only at word boundaries to avoid false positives. + + '500ms', '4010', '401A', '5000' must not trigger internal/auth/not_found rules. + Negative cases assert exactly 'unknown' so future rule additions that accidentally + absorb these strings are caught — a broad exclusion-list assertion would not be. + """ + msg = _make_msg(content, status="error") + result = normalize_tool_message(msg) + m = _meta(result) + assert m["status"] == "error" + assert m["error_type"] == expected_error_type, f"{content!r} → expected {expected_error_type!r}, got {m['error_type']!r}" diff --git a/backend/tests/test_warm_pool_lifecycle.py b/backend/tests/test_warm_pool_lifecycle.py new file mode 100644 index 00000000000..ae917b3c12f --- /dev/null +++ b/backend/tests/test_warm_pool_lifecycle.py @@ -0,0 +1,95 @@ +"""Unit tests for shared warm-pool lifecycle mechanics.""" + +from __future__ import annotations + +import threading +import time +from typing import Any + +from deerflow.community.warm_pool_lifecycle import DEFAULT_IDLE_TIMEOUT, DEFAULT_REPLICAS, WarmPoolLifecycleMixin + + +class _Provider(WarmPoolLifecycleMixin[str]): + _idle_checker_thread_name = "test-warm-pool-reaper" + + def __init__(self, *, replicas: int = DEFAULT_REPLICAS, idle_timeout: float = DEFAULT_IDLE_TIMEOUT, active_count: int = 0) -> None: + self._lock = threading.Lock() + self._warm_pool: dict[str, tuple[str, float]] = {} + self._config: dict[str, Any] = {"replicas": replicas, "idle_timeout": idle_timeout} + self._idle_checker_stop = threading.Event() + self._idle_checker_thread: threading.Thread | None = None + self.active_count = active_count + self.destroyed: list[tuple[str, str, str]] = [] + + def _active_count_locked(self) -> int: + return self.active_count + + def _destroy_warm_entry(self, sandbox_id: str, entry: str, *, reason: str) -> None: + self.destroyed.append((sandbox_id, entry, reason)) + + +def test_replica_count_includes_active_and_warm_entries() -> None: + provider = _Provider(replicas=2, active_count=1) + provider._warm_pool["warm-1"] = ("entry-1", time.time()) + + assert provider._replica_count() == (2, 2) + + +def test_evict_oldest_warm_removes_and_destroys_oldest_entry() -> None: + provider = _Provider() + provider._warm_pool["new"] = ("entry-new", 200.0) + provider._warm_pool["old"] = ("entry-old", 100.0) + + evicted = provider._evict_oldest_warm() + + assert evicted == "old" + assert "old" not in provider._warm_pool + assert "new" in provider._warm_pool + assert provider.destroyed == [("old", "entry-old", "replica_enforcement")] + + +def test_evict_oldest_warm_returns_none_when_pool_empty() -> None: + provider = _Provider() + + assert provider._evict_oldest_warm() is None + assert provider.destroyed == [] + + +def test_reap_expired_warm_destroys_only_expired_entries() -> None: + provider = _Provider() + now = time.time() + provider._warm_pool["expired"] = ("entry-expired", now - 100) + provider._warm_pool["fresh"] = ("entry-fresh", now) + + provider._reap_expired_warm(idle_timeout=10) + + assert "expired" not in provider._warm_pool + assert "fresh" in provider._warm_pool + assert provider.destroyed == [("expired", "entry-expired", "idle_timeout")] + + +def test_reap_expired_warm_noops_when_timeout_disabled() -> None: + provider = _Provider(idle_timeout=0) + provider._warm_pool["expired"] = ("entry-expired", time.time() - 100) + + provider._reap_expired_warm(idle_timeout=0) + + assert "expired" in provider._warm_pool + assert provider.destroyed == [] + + +def test_start_idle_checker_uses_monkeypatchable_interval(monkeypatch) -> None: + provider = _Provider(idle_timeout=0.01) + monkeypatch.setattr(_Provider, "IDLE_CHECK_INTERVAL", 0.01) + provider._warm_pool["expired"] = ("entry-expired", time.time() - 10) + + provider._start_idle_checker() + deadline = time.time() + 1 + while "expired" in provider._warm_pool and time.time() < deadline: + time.sleep(0.01) + provider._stop_idle_checker() + + assert "expired" not in provider._warm_pool + assert provider.destroyed == [("expired", "entry-expired", "idle_timeout")] + assert provider._idle_checker_thread is not None + assert not provider._idle_checker_thread.is_alive() diff --git a/backend/uv.lock b/backend/uv.lock index 0d9df677d8c..6c6bdf15369 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -485,6 +485,17 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/95/c1/84fc6811122f54b20de2e5afb312ee07a3a47a328755587d1e505475239b/blockbuster-1.5.26-py3-none-any.whl", hash = "sha256:f8e53fb2dd4b6c6ec2f04907ddbd063ca7cd1ef587d24448ef4e50e81e3a79bb", size = 13226, upload-time = "2025-12-05T10:43:48.778Z" }, ] +[[package]] +name = "boxlite" +version = "0.9.7" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/77/34fe448e7a1411b95c33df673305d64bacdfb031a5faae9faec82a32eaf8/boxlite-0.9.7-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:715ba342a80432279115ad41edf42b92993134402a7f4c73f23eb1327aa7ec29", size = 35518202, upload-time = "2026-07-01T12:55:41.757Z" }, + { url = "https://files.pythonhosted.org/packages/02/2f/8ebf72c81afd07b3779ce206e85b32cf7b7baed98b811e966508ab9509cc/boxlite-0.9.7-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:01a2dd8b871213beb037a0c332bd94d9568aea7db0c1d39385ae789b73610055", size = 37616271, upload-time = "2026-07-01T12:55:45.01Z" }, + { url = "https://files.pythonhosted.org/packages/5f/32/6d9d8e9dca8f6e387dfe874a6872b49529b7795b75e814b2a75015a66036/boxlite-0.9.7-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:2281d4613af8f56e05856883aac2cbaf0028d50895a67299cd4932585661f13a", size = 33781989, upload-time = "2026-07-01T12:55:47.944Z" }, + { url = "https://files.pythonhosted.org/packages/e0/79/068b8a7de1fc9724d89ca7086754f0e6c6c80f8c74f9d95267006c49fe33/boxlite-0.9.7-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:b8e68f2efe493112445f3150adac74de592974ba9c05025440e0ddf54870bc53", size = 35519183, upload-time = "2026-07-01T12:55:50.896Z" }, +] + [[package]] name = "bracex" version = "2.7" @@ -885,6 +896,9 @@ dependencies = [ ] [package.optional-dependencies] +boxlite = [ + { name = "boxlite" }, +] ollama = [ { name = "langchain-ollama" }, ] @@ -911,6 +925,7 @@ requires-dist = [ { name = "aiosqlite", specifier = ">=0.19" }, { name = "alembic", specifier = ">=1.13" }, { name = "asyncpg", marker = "extra == 'postgres'", specifier = ">=0.29" }, + { name = "boxlite", marker = "extra == 'boxlite'", specifier = ">=0.9.7" }, { name = "croniter", specifier = ">=6.0.0" }, { name = "cryptography", specifier = ">=48.0.1" }, { name = "ddgs", specifier = ">=9.10.0" }, @@ -950,7 +965,7 @@ requires-dist = [ { name = "textual", marker = "extra == 'tui'", specifier = ">=0.80" }, { name = "tiktoken", specifier = ">=0.8.0" }, ] -provides-extras = ["redis", "tui", "groundroute", "ollama", "postgres", "pymupdf"] +provides-extras = ["tui", "groundroute", "ollama", "postgres", "redis", "pymupdf", "boxlite"] [[package]] name = "defusedxml" diff --git a/config.example.yaml b/config.example.yaml index 733359be52d..f48a33da61f 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -15,7 +15,7 @@ # ============================================================================ # Bump this number when the config schema changes. # Run `make config-upgrade` to merge new fields into your local config.yaml. -config_version: 18 +config_version: 19 # ============================================================================ # Logging @@ -934,6 +934,31 @@ loop_detection: # warn: 150 # hard_limit: 300 +# ============================================================================ +# Tool Progress State Machine Configuration (RFC #3177) +# ============================================================================ +# Detects tool stagnation and repetition at the (thread, tool) level. +# Tracks consecutive "no-new-info" calls (error, partial_success, near-duplicate success). +# Three transition paths (determined by deerflow_tool_meta.recoverable_by_model): +# recoverable=true (no_results, not_found, permission): ACTIVE → WARNED (terminal; hint re-injected each call) +# recoverable=false (rate_limited, transient): ACTIVE → WARNED → BLOCKED after warn_escalation_count more +# recoverable=false + action=stop (auth, config): ACTIVE → BLOCKED immediately +# Requires ToolErrorHandlingMiddleware to be active (always on). + +# tool_progress: +# enabled: false +# stagnation_threshold: 3 # Consecutive problems before WARNED +# warn_escalation_count: 2 # More problems after WARNED before BLOCKED +# inject_assessment: true +# jaccard_similarity_threshold: 0.8 # Word-set similarity threshold for near-duplicate detection +# min_word_count_for_similarity: 10 # Min unique words to apply Jaccard check +# max_tracked_threads: 100 +# exempt_tools: +# - ask_clarification +# - write_todos +# - present_files +# - task + # ============================================================================ # Read-Before-Write File Gate (issue #3857) # ============================================================================ @@ -1086,7 +1111,32 @@ sandbox: # # API_KEY: $MY_API_KEY # Reads from host's MY_API_KEY env var # # DATABASE_URL: $DATABASE_URL # Reads from host's DATABASE_URL env var -# Option 3: Provisioner-managed AIO Sandbox (docker-compose-dev) +# Option 3: BoxLite micro-VM Sandbox +# Runs each sandbox as a BoxLite micro-VM. Released boxes stay in an in-process +# warm pool and can be reclaimed by the same user/thread without a cold start. +# Requires the boxlite runtime and host virtualization support (KVM on Linux, +# Hypervisor.framework on macOS). +# sandbox: +# use: deerflow.community.boxlite:BoxliteProvider +# image: python:3.12-slim +# +# # Optional: Per-box memory and CPU limits. +# # memory_mib: 1024 +# # cpus: 2 +# +# # Optional: Maximum active + warm BoxLite VMs per gateway process (default: 3). +# # Active boxes are never evicted; only warm-pool boxes are stopped to make room. +# # replicas: 3 +# +# # Optional: Seconds before an idle warm-pool VM is stopped (default: 600). +# # Set to 0 to keep warm VMs until shutdown or replica eviction. +# # idle_timeout: 600 +# +# # Optional: Environment variables to inject into every command. +# # environment: +# # PYTHONUNBUFFERED: "1" +# +# Option 4: Provisioner-managed AIO Sandbox (docker-compose-dev) # Each sandbox_id gets a dedicated Pod in k3s, managed by the provisioner. # Recommended for production or advanced users who want better isolation and scalability.: # sandbox: @@ -1317,6 +1367,26 @@ memory: guaranteed_categories: - correction guaranteed_token_budget: 500 + # Staleness review: periodically prune aged facts that may no longer reflect + # the user's current situation. When triggered, the LLM reviews facts older + # than ``staleness_age_days`` during the normal memory-update call (same LLM + # invocation — no extra API call) and decides KEEP or REMOVE for each. + # staleness_review_enabled - master switch (default: true) + # staleness_age_days - facts older than this are candidates (default: 90) + # staleness_min_candidates - minimum stale facts required to trigger a review + # cycle; avoids wasteful LLM calls when there are + # very few candidates (default: 3) + # staleness_max_removals_per_cycle - safety cap on removals per cycle; when + # exceeded, the lowest-confidence entries + # are kept (default: 10) + # staleness_protected_categories - fact categories exempt from review + # (default: ["correction"]) + staleness_review_enabled: true + staleness_age_days: 90 + staleness_min_candidates: 3 + staleness_max_removals_per_cycle: 10 + staleness_protected_categories: + - correction # ============================================================================ # Custom Agent Management API diff --git a/contracts/subagent_status_contract.json b/contracts/subagent_status_contract.json index 6d998759d07..cd901c76eb0 100644 --- a/contracts/subagent_status_contract.json +++ b/contracts/subagent_status_contract.json @@ -1,5 +1,5 @@ { "version": 1, "description": "Cross-language contract fixture for the structured subagent status field. Task result text is display content only and is not part of this wire contract.", - "valid_status_values": ["completed", "failed", "cancelled", "timed_out", "polling_timed_out"] + "valid_status_values": ["completed", "failed", "cancelled", "timed_out", "polling_timed_out", "max_turns_reached"] } diff --git a/docker/docker-compose-dev.yaml b/docker/docker-compose-dev.yaml index 490c8353247..19ffc2e4c06 100644 --- a/docker/docker-compose-dev.yaml +++ b/docker/docker-compose-dev.yaml @@ -155,8 +155,10 @@ services: # Preserve the .venv built during Docker image build — mounting the full backend/ # directory above would otherwise shadow it with the (empty) host directory. - gateway-venv:/app/backend/.venv - - ../config.yaml:/app/config.yaml - - ../extensions_config.json:/app/extensions_config.json + # Mount the project directory instead of its mutable config files individually. + # Host editors commonly replace files on save; a directory bind keeps those + # replacements visible inside Docker Desktop/WSL containers. + - ../:/app/project - ../skills:/app/skills - ../logs:/app/logs # Use a Docker-managed uv cache volume instead of a host bind mount. @@ -178,6 +180,8 @@ services: - CI=true - DEER_FLOW_PROJECT_ROOT=/app - DEER_FLOW_HOME=/app/backend/.deer-flow + - DEER_FLOW_CONFIG_PATH=/app/project/config.yaml + - DEER_FLOW_EXTENSIONS_CONFIG_PATH=/app/project/extensions_config.json - DEER_FLOW_STREAM_BRIDGE_REDIS_URL=${DEER_FLOW_STREAM_BRIDGE_REDIS_URL:-redis://redis:6379/0} - DEER_FLOW_CHANNELS_LANGGRAPH_URL=${DEER_FLOW_CHANNELS_LANGGRAPH_URL:-http://gateway:8001/api} - DEER_FLOW_CHANNELS_GATEWAY_URL=${DEER_FLOW_CHANNELS_GATEWAY_URL:-http://gateway:8001} diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index b46cb27ebc4..85041212aba 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -71,6 +71,8 @@ The frontend is a stateful chat application. Users create **threads** (conversat `/goal` is a built-in composer command, not a skill activation. `src/components/workspace/input-box.tsx` intercepts `/goal`, `/goal clear`, and `/goal ` before normal chat submission, calling Gateway `GET/PUT/DELETE /api/threads/{thread_id}/goal`. Setting `/goal ` also submits the condition text as the next user task so the agent starts running immediately; status and clear do not start a run. Goal requests are tied to the current `threadId` with an `AbortController`, so switching threads or unmounting the composer aborts in-flight goal requests and stale responses cannot update the new thread's goal state. The chat pages render `GoalStatus` above the composer from `AgentThreadState.goal`, with local optimistic state until the next stream `values` update arrives. +Human input requests are a structured message protocol layered on normal chat history. The backend writes request payloads to `ToolMessage.artifact.human_input`, `src/core/messages/human-input.ts` owns the runtime validators/types, and `src/components/workspace/messages/human-input-card.tsx` renders the reusable card. `MessageList` owns answered/latest/pending state for visible cards, but derives answered responses from raw `thread.messages` because replies are hidden; pending cards clear when the hidden reply appears, when dispatch is dropped, or when a new `thread.error` reports an async stream failure. Page-level submit callbacks must send a normal human message and put `hide_from_ui: true` plus the response payload in the fourth `sendMessage(..., options)` argument as `options.additionalKwargs`; the third argument remains run context such as `{ agent_name }`. Composer entry points should disable normal bottom input while `hasOpenHumanInputRequest(...)` is true so users answer through the card and preserve response metadata. + ### Key Patterns - **Server Components by default**, `"use client"` only for interactive components @@ -82,7 +84,9 @@ The frontend is a stateful chat application. Users create **threads** (conversat ### Interaction Ownership - `src/app/workspace/chats/[thread_id]/page.tsx` owns composer busy-state wiring. +- `src/app/workspace/chats/[thread_id]/page.tsx` owns branch-from-turn submission and navigation; sidecar `MessageList` instances do not receive the branch action. - `src/app/workspace/chats/[thread_id]/page.tsx` and `src/app/workspace/agents/[agent_name]/chats/[thread_id]/page.tsx` own active-goal display state for their composer overlays. +- `src/components/workspace/messages/message-list.tsx` owns human-input card answered/latest/pending gating; entry pages only translate a submitted card response into `sendMessage` calls. - `src/core/threads/hooks.ts` owns pre-submit upload state and thread submission. ## Code Style diff --git a/frontend/src/app/page.tsx b/frontend/src/app/page.tsx index e5d3187b71c..71f8de179bd 100644 --- a/frontend/src/app/page.tsx +++ b/frontend/src/app/page.tsx @@ -9,7 +9,7 @@ import { WhatsNewSection } from "@/components/landing/sections/whats-new-section export default function LandingPage() { return ( -
+
diff --git a/frontend/src/app/workspace/agents/[agent_name]/chats/[thread_id]/page.tsx b/frontend/src/app/workspace/agents/[agent_name]/chats/[thread_id]/page.tsx index d725bf3e49c..50573ba2793 100644 --- a/frontend/src/app/workspace/agents/[agent_name]/chats/[thread_id]/page.tsx +++ b/frontend/src/app/workspace/agents/[agent_name]/chats/[thread_id]/page.tsx @@ -2,7 +2,7 @@ import { BotIcon, PlusSquare } from "lucide-react"; import { useParams, useRouter } from "next/navigation"; -import { useCallback, useEffect, useState } from "react"; +import { useCallback, useEffect, useMemo, useState } from "react"; import type { PromptInputMessage } from "@/components/ai-elements/prompt-input"; import { Button } from "@/components/ui/button"; @@ -32,6 +32,13 @@ import { Tooltip } from "@/components/workspace/tooltip"; import { useActiveGoal } from "@/components/workspace/use-active-goal"; import { useAgent } from "@/core/agents"; import { useI18n } from "@/core/i18n/hooks"; +import { + buildHumanInputResponseText, + hasOpenHumanInputRequest, + type HumanInputRequest, + type HumanInputResponse, +} from "@/core/messages/human-input"; +import { isHiddenFromUIMessage } from "@/core/messages/utils"; import { useModels } from "@/core/models/hooks"; import { useNotification } from "@/core/notification/hooks"; import { useLocalSettings, useThreadSettings } from "@/core/settings"; @@ -168,6 +175,31 @@ export default function AgentChatPage() { [sendMessage, threadId, agent_name], ); + const handleSubmitHumanInput = useCallback( + async (request: HumanInputRequest, response: HumanInputResponse) => { + let sent = false; + await sendMessage( + threadId, + { + text: buildHumanInputResponseText(request, response), + files: [], + }, + { agent_name }, + { + additionalKwargs: { + hide_from_ui: true, + human_input_response: response, + }, + onSent: () => { + sent = true; + }, + }, + ); + return sent; + }, + [agent_name, sendMessage, threadId], + ); + const handleStop = useCallback(async () => { await thread.stop(); }, [thread]); @@ -180,6 +212,14 @@ export default function AgentChatPage() { threadId, thread.values.goal, ); + const hasOpenHumanInputCard = useMemo( + () => + hasOpenHumanInputRequest( + thread.messages, + (message) => !isHiddenFromUIMessage(message), + ), + [thread.messages], + ); return ( @@ -253,6 +293,11 @@ export default function AgentChatPage() { loadMoreHistory={loadMoreHistory} isHistoryLoading={isHistoryLoading} tokenUsageInlineMode={tokenUsageInlineMode} + onSubmitHumanInput={ + isMock || env.NEXT_PUBLIC_STATIC_WEBSITE_ONLY === "true" + ? undefined + : handleSubmitHumanInput + } />
@@ -322,7 +367,9 @@ export default function AgentChatPage() { } disabled={ env.NEXT_PUBLIC_STATIC_WEBSITE_ONLY === "true" || - isUploading + isUploading || + hasOpenHumanInputCard || + (!isNewThread && isHistoryLoading) } onContextChange={(context) => setSettings("context", context) diff --git a/frontend/src/app/workspace/agents/new/page.tsx b/frontend/src/app/workspace/agents/new/page.tsx index e7553d9d40b..67ec4931523 100644 --- a/frontend/src/app/workspace/agents/new/page.tsx +++ b/frontend/src/app/workspace/agents/new/page.tsx @@ -38,6 +38,13 @@ import { getAgent, } from "@/core/agents/api"; import { useI18n } from "@/core/i18n/hooks"; +import { + buildHumanInputResponseText, + hasOpenHumanInputRequest, + type HumanInputRequest, + type HumanInputResponse, +} from "@/core/messages/human-input"; +import { isHiddenFromUIMessage } from "@/core/messages/utils"; import { useThreadStream } from "@/core/threads/hooks"; import { uuid } from "@/core/utils/uuid"; import { isIMEComposing } from "@/lib/ime"; @@ -110,6 +117,14 @@ export default function NewAgentPage() { }); }, }); + const hasOpenHumanInputCard = useMemo( + () => + hasOpenHumanInputRequest( + thread.messages, + (message) => !isHiddenFromUIMessage(message), + ), + [thread.messages], + ); useEffect(() => { if (typeof window === "undefined" || step !== "chat") { @@ -208,14 +223,43 @@ export default function NewAgentPage() { const handleChatSubmit = useCallback( async (text: string) => { const trimmed = text.trim(); - if (!trimmed || thread.isLoading) return; + if (!trimmed || thread.isLoading || hasOpenHumanInputCard) return; await sendMessage( threadId, { text: trimmed, files: [] }, { agent_name: agentName }, ); }, - [agentName, sendMessage, thread.isLoading, threadId], + [agentName, hasOpenHumanInputCard, sendMessage, thread.isLoading, threadId], + ); + + const handleSubmitHumanInput = useCallback( + async (request: HumanInputRequest, response: HumanInputResponse) => { + if (!agentName) { + return false; + } + + let sent = false; + await sendMessage( + threadId, + { + text: buildHumanInputResponseText(request, response), + files: [], + }, + { agent_name: agentName }, + { + additionalKwargs: { + hide_from_ui: true, + human_input_response: response, + }, + onSent: () => { + sent = true; + }, + }, + ); + return sent; + }, + [agentName, sendMessage, threadId], ); const handleSaveAgent = useCallback(async () => { @@ -365,6 +409,9 @@ export default function NewAgentPage() { className={cn("size-full", showSaveHint ? "pt-4" : "pt-10")} threadId={threadId} thread={thread} + onSubmitHumanInput={ + agentName ? handleSubmitHumanInput : undefined + } />
@@ -394,15 +441,18 @@ export default function NewAgentPage() { ) : ( void handleChatSubmit(text)} > - + )} diff --git a/frontend/src/app/workspace/chats/[thread_id]/page.tsx b/frontend/src/app/workspace/chats/[thread_id]/page.tsx index 909d9c57af8..0281062bb34 100644 --- a/frontend/src/app/workspace/chats/[thread_id]/page.tsx +++ b/frontend/src/app/workspace/chats/[thread_id]/page.tsx @@ -1,7 +1,8 @@ "use client"; import { useRouter } from "next/navigation"; -import { useCallback, useEffect, useRef, useState } from "react"; +import { useCallback, useEffect, useMemo, useRef, useState } from "react"; +import { toast } from "sonner"; import { type PromptInputMessage } from "@/components/ai-elements/prompt-input"; import { SidebarTrigger } from "@/components/ui/sidebar"; @@ -33,10 +34,18 @@ import { TokenUsageIndicator } from "@/components/workspace/token-usage-indicato import { useActiveGoal } from "@/components/workspace/use-active-goal"; import { Welcome } from "@/components/workspace/welcome"; import { useI18n } from "@/core/i18n/hooks"; +import { + buildHumanInputResponseText, + hasOpenHumanInputRequest, + type HumanInputRequest, + type HumanInputResponse, +} from "@/core/messages/human-input"; +import { isHiddenFromUIMessage } from "@/core/messages/utils"; import { useModels } from "@/core/models/hooks"; import { useNotification } from "@/core/notification/hooks"; import { useLocalSettings, useThreadSettings } from "@/core/settings"; import { + useBranchThread, useThreadMetadata, useThreadStream, useThreadTokenUsage, @@ -68,6 +77,7 @@ export default function ChatPage() { enabled: !isNewThread && !isMock, isMock, }); + const branchThread = useBranchThread(); const backendTokenUsage = threadTokenUsageToTokenUsage(threadTokenUsage.data); const mountedRef = useRef(false); useSpecificChatMode(); @@ -167,6 +177,30 @@ export default function ChatPage() { }, [sendMessage, threadId], ); + const handleSubmitHumanInput = useCallback( + async (request: HumanInputRequest, response: HumanInputResponse) => { + let sent = false; + await sendMessage( + threadId, + { + text: buildHumanInputResponseText(request, response), + files: [], + }, + undefined, + { + additionalKwargs: { + hide_from_ui: true, + human_input_response: response, + }, + onSent: () => { + sent = true; + }, + }, + ); + return sent; + }, + [sendMessage, threadId], + ); const handleStop = useCallback(async () => { await thread.stop(); }, [thread]); @@ -175,6 +209,32 @@ export default function ChatPage() { regenerateMessage(threadId, messageId, supersededMessageIds), [regenerateMessage, threadId], ); + const handleBranchTurn = useCallback( + async (messageId: string, messageIds: string[]) => { + if ( + isNewThread || + isMock || + env.NEXT_PUBLIC_STATIC_WEBSITE_ONLY === "true" + ) { + return; + } + + try { + const response = await branchThread.mutateAsync({ + threadId, + messageId, + messageIds, + }); + toast.success(t.conversation.branchCreated); + router.push(`/workspace/chats/${response.thread_id}`); + } catch (error) { + toast.error( + error instanceof Error ? error.message : t.conversation.branchFailed, + ); + } + }, + [branchThread, isMock, isNewThread, router, t, threadId], + ); const tokenUsageInlineMode = tokenUsageEnabled ? localSettings.tokenUsage.inlineMode @@ -184,6 +244,14 @@ export default function ChatPage() { threadId, thread.values.goal, ); + const hasOpenHumanInputCard = useMemo( + () => + hasOpenHumanInputRequest( + thread.messages, + (message) => !isHiddenFromUIMessage(message), + ), + [thread.messages], + ); return ( @@ -246,6 +314,20 @@ export default function ChatPage() { !thread.isLoading } onRegenerateMessage={handleRegenerate} + onSubmitHumanInput={ + isMock || env.NEXT_PUBLIC_STATIC_WEBSITE_ONLY === "true" + ? undefined + : handleSubmitHumanInput + } + canBranch={ + !isNewThread && + !isMock && + env.NEXT_PUBLIC_STATIC_WEBSITE_ONLY !== "true" && + !isUploading && + !thread.isLoading && + !branchThread.isPending + } + onBranchTurn={handleBranchTurn} />
setSettings("context", context) diff --git a/frontend/src/components/landing/header.tsx b/frontend/src/components/landing/header.tsx index de7540a0270..6adb8b15dd5 100644 --- a/frontend/src/components/landing/header.tsx +++ b/frontend/src/components/landing/header.tsx @@ -8,6 +8,8 @@ import { getI18n } from "@/core/i18n/server"; import { env } from "@/env"; import { cn } from "@/lib/utils"; +import { MobileNav } from "./mobile-nav"; + export type HeaderProps = { className?: string; homeURL?: string; @@ -21,20 +23,21 @@ export async function Header({ className, homeURL, locale }: HeaderProps) { return (
-
+ -
+
); diff --git a/frontend/src/components/landing/hero.tsx b/frontend/src/components/landing/hero.tsx index 503c5b88c55..34aa91762e8 100644 --- a/frontend/src/components/landing/hero.tsx +++ b/frontend/src/components/landing/hero.tsx @@ -1,15 +1,33 @@ "use client"; import { ChevronRightIcon } from "lucide-react"; +import { AnimatePresence, motion } from "motion/react"; import Link from "next/link"; +import { useEffect, useState } from "react"; +import { AuroraText } from "@/components/ui/aurora-text"; import { Button } from "@/components/ui/button"; import { FlickeringGrid } from "@/components/ui/flickering-grid"; import Galaxy from "@/components/ui/galaxy"; -import { WordRotate } from "@/components/ui/word-rotate"; import { env } from "@/env"; import { cn } from "@/lib/utils"; +const HERO_WORDS = [ + "Deep Research", + "Collect Data", + "Analyze Data", + "Generate Webpages", + "Vibe Coding", + "Generate Slides", + "Generate Images", + "Generate Podcasts", + "Generate Videos", + "Generate Songs", + "Organize Emails", + "Do Anything", + "Learn Anything", +]; + export function Hero({ className }: { className?: string }) { return (
-
-

- {" "} -
with DeerFlow
+
+

+ DeerFlow

+
+ + SuperAgent +
{env.NEXT_PUBLIC_STATIC_WEBSITE_ONLY && ( )} -

+

An open-source SuperAgent harness that researches, codes, and creates. - With -
- the help of sandboxes, memories, tools, skills and subagents, it - handles -
- different levels of tasks that could take minutes to hours. + With the help of sandboxes, memories, tools, skills and subagents, it + handles different levels of tasks that could take minutes to hours.

- @@ -89,6 +90,47 @@ export function Hero({ className }: { className?: string }) { ); } +function HeroWordRotate({ + words, + duration = 2200, +}: { + words: string[]; + duration?: number; +}) { + const [index, setIndex] = useState(0); + + useEffect(() => { + const interval = setInterval(() => { + setIndex((prevIndex) => (prevIndex + 1) % words.length); + }, duration); + + return () => clearInterval(interval); + }, [words, duration]); + + return ( +
+ + + + {words[index]} + + + +
+ ); +} + function BytePlusIcon(props: React.SVGProps) { return ( + + + + + + DeerFlow + + + + + ); +} diff --git a/frontend/src/components/landing/section.tsx b/frontend/src/components/landing/section.tsx index 97146697469..01b6fff7d7e 100644 --- a/frontend/src/components/landing/section.tsx +++ b/frontend/src/components/landing/section.tsx @@ -12,18 +12,20 @@ export function Section({ children: React.ReactNode; }) { return ( -
-
-
+
+
+
{title}
{subtitle && ( -
+
{subtitle}
)}
-
{children}
+
{children}
); } diff --git a/frontend/src/components/landing/sections/case-study-section.tsx b/frontend/src/components/landing/sections/case-study-section.tsx index aeb9163e897..a08e6a2db2d 100644 --- a/frontend/src/components/landing/sections/case-study-section.tsx +++ b/frontend/src/components/landing/sections/case-study-section.tsx @@ -51,7 +51,7 @@ export function CaseStudySection({ className }: { className?: string }) { title="Case Studies" subtitle="See how DeerFlow is used in the wild" > -
+
{caseStudies.map((caseStudy) => ( { - const interval = setInterval(() => { - setIndex((prevIndex) => (prevIndex + 1) % words.length); - }, duration); - - // Clean up interval on unmount - return () => clearInterval(interval); - }, [words, duration]); - - return ( -
- - - - {words[index]} - - - -
- ); -} diff --git a/frontend/src/components/workspace/artifacts/artifact-trigger.tsx b/frontend/src/components/workspace/artifacts/artifact-trigger.tsx index 40d353559a0..3ce35e7f243 100644 --- a/frontend/src/components/workspace/artifacts/artifact-trigger.tsx +++ b/frontend/src/components/workspace/artifacts/artifact-trigger.tsx @@ -17,17 +17,19 @@ export const ArtifactTrigger = () => { return null; } return ( - + ); diff --git a/frontend/src/components/workspace/chats/chat-box.tsx b/frontend/src/components/workspace/chats/chat-box.tsx index 6fa8c97bd49..e712d32f239 100644 --- a/frontend/src/components/workspace/chats/chat-box.tsx +++ b/frontend/src/components/workspace/chats/chat-box.tsx @@ -4,7 +4,15 @@ import { useEffect, useMemo, useRef, useState } from "react"; import { ConversationEmptyState } from "@/components/ai-elements/conversation"; import { Button } from "@/components/ui/button"; +import { + Sheet, + SheetContent, + SheetDescription, + SheetHeader, + SheetTitle, +} from "@/components/ui/sheet"; import { env } from "@/env"; +import { useIsMobile } from "@/hooks/use-mobile"; import { cn } from "@/lib/utils"; import { @@ -24,6 +32,7 @@ const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({ threadId, }) => { const { thread } = useThread(); + const isMobile = useIsMobile(); const pathname = usePathname(); const threadIdRef = useRef(threadId); @@ -127,6 +136,102 @@ const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({ } }, [artifactsOpen, setArtifactsOpen, sidecarOpen]); + const rightPanelContent = useMemo(() => { + if (renderedRightPanel === "sidecar") { + return ; + } + if (renderedRightPanel === "artifacts" && selectedArtifact) { + return ( + + ); + } + if (renderedRightPanel === "artifacts") { + return ( +
+
+ +
+ {artifacts.length === 0 ? ( + } + title="No artifact selected" + description="Select an artifact to view its details" + /> + ) : ( +
+
+

Artifacts

+
+
+ +
+
+ )} +
+ ); + } + return null; + }, [ + renderedRightPanel, + selectedArtifact, + threadId, + artifacts, + setArtifactsOpen, + ]); + + if (isMobile) { + return ( + <> +
{children}
+ { + if (open) { + return; + } + if (sidecarOpen) { + sidecar?.close(); + } + if (artifactsOpen) { + setArtifactsOpen(false); + } + }} + > + + + + {renderedRightPanel === "sidecar" ? "Sidecar" : "Artifacts"} + + + Browse the side panel for this conversation. + + +
{rightPanelContent}
+ + + + ); + } + return (
= ({ rightPanelOpen ? "opacity-100" : "opacity-0", )} > - {renderedRightPanel === "sidecar" ? ( - - ) : renderedRightPanel === "artifacts" && selectedArtifact ? ( - - ) : renderedRightPanel === "artifacts" ? ( -
-
- -
- {artifacts.length === 0 ? ( - } - title="No artifact selected" - description="Select an artifact to view its details" - /> - ) : ( -
-
-

Artifacts

-
-
- -
-
- )} -
- ) : null} + {rightPanelContent}
diff --git a/frontend/src/components/workspace/export-trigger.tsx b/frontend/src/components/workspace/export-trigger.tsx index b75d4e45135..a42784a541c 100644 --- a/frontend/src/components/workspace/export-trigger.tsx +++ b/frontend/src/components/workspace/export-trigger.tsx @@ -58,11 +58,12 @@ export function ExportTrigger({ threadId }: { threadId: string }) { diff --git a/frontend/src/components/workspace/input-box-helpers.ts b/frontend/src/components/workspace/input-box-helpers.ts index f89610eb6bd..d771654aa9d 100644 --- a/frontend/src/components/workspace/input-box-helpers.ts +++ b/frontend/src/components/workspace/input-box-helpers.ts @@ -1,10 +1,11 @@ import type { Skill } from "@/core/skills"; +export { + SUGGESTION_TEMPLATE_PLACEHOLDER_PATTERN, + findSuggestionTemplatePlaceholder, +} from "@/core/suggestions/placeholders"; export const MAX_SKILL_SUGGESTIONS = 6; -export const SUGGESTION_TEMPLATE_PLACEHOLDER_PATTERN = - /\[(?:主题|来源|topic|source)\]/i; - export type SlashSuggestion = { name: string; description: string; @@ -100,18 +101,6 @@ export function isAbortError(error: unknown): boolean { ); } -export function findSuggestionTemplatePlaceholder(text: string) { - const match = SUGGESTION_TEMPLATE_PLACEHOLDER_PATTERN.exec(text); - if (!match) { - return null; - } - - return { - start: match.index, - end: match.index + match[0].length, - }; -} - export function getLeadingSlashSkillQuery(value: string): string | null { if (!value.startsWith("/")) { return null; diff --git a/frontend/src/components/workspace/input-box.tsx b/frontend/src/components/workspace/input-box.tsx index fcd4cb016fc..b6af8472b15 100644 --- a/frontend/src/components/workspace/input-box.tsx +++ b/frontend/src/components/workspace/input-box.tsx @@ -23,7 +23,6 @@ import { useState, type ComponentProps, type KeyboardEvent, - type RefObject, } from "react"; import { toast } from "sonner"; @@ -614,7 +613,7 @@ export function InputBox({ } const placeholder = findSuggestionTemplatePlaceholder(message.text); if (placeholder) { - toast.error(t.inputBox.suggestionPlaceholderRequired); + toast.warning(t.inputBox.suggestionPlaceholderRequired); requestAnimationFrame(() => { const textarea = textareaRef.current; if (!textarea) { @@ -1069,6 +1068,18 @@ export function InputBox({ threadId, ]); + const onSelectPlaceholder = useCallback((newText: string) => { + const placeholder = findSuggestionTemplatePlaceholder(newText); + if (placeholder) { + requestAnimationFrame(() => { + const textarea = textareaRef.current; + if (!textarea) return; + textarea.focus(); + textarea.setSelectionRange(placeholder.start, placeholder.end); + }); + } + }, []); + return (
- {!isWelcomeMode && ( -
- )} + {!isWelcomeMode && ( +
+ )} {isWelcomeMode && searchParams.get("mode") !== "skill" && !showSkillSuggestions && (
- +
)} @@ -1598,9 +1609,9 @@ export function InputBox({ } function SuggestionList({ - textareaRef, + onSelectPlaceholder, }: { - textareaRef: RefObject; + onSelectPlaceholder: (newText: string) => void; }) { const { t } = useI18n(); const { textInput } = usePromptInputController(); @@ -1608,16 +1619,9 @@ function SuggestionList({ (prompt: string | undefined) => { if (!prompt) return; textInput.setInput(prompt); - requestAnimationFrame(() => { - const textarea = textareaRef.current; - const placeholder = findSuggestionTemplatePlaceholder(prompt); - if (textarea && placeholder) { - textarea.focus(); - textarea.setSelectionRange(placeholder.start, placeholder.end); - } - }); + onSelectPlaceholder(prompt); }, - [textareaRef, textInput], + [textInput, onSelectPlaceholder], ); return ( diff --git a/frontend/src/components/workspace/messages/human-input-card.tsx b/frontend/src/components/workspace/messages/human-input-card.tsx new file mode 100644 index 00000000000..e4d2687523b --- /dev/null +++ b/frontend/src/components/workspace/messages/human-input-card.tsx @@ -0,0 +1,244 @@ +"use client"; + +import { + CheckCircle2Icon, + Loader2Icon, + MessageCircleQuestionMarkIcon, +} from "lucide-react"; +import { useId, useState, type KeyboardEvent } from "react"; + +import { Badge } from "@/components/ui/badge"; +import { Button } from "@/components/ui/button"; +import { Textarea } from "@/components/ui/textarea"; +import { useI18n } from "@/core/i18n/hooks"; +import { + createHumanInputOptionResponse, + createHumanInputTextResponse, + type HumanInputOption, + type HumanInputRequest, + type HumanInputResponse, +} from "@/core/messages/human-input"; +import { isIMEComposing } from "@/lib/ime"; +import { cn } from "@/lib/utils"; + +import { MarkdownContent } from "./markdown-content"; + +export type HumanInputSubmitResult = boolean | void; + +export function shouldSubmitHumanInputTextOnKeyDown( + event: KeyboardEvent, + isComposing = false, +) { + return ( + event.key === "Enter" && + !event.shiftKey && + !isIMEComposing(event, isComposing) + ); +} + +export function HumanInputCard({ + request, + disabled = false, + pending = false, + answeredResponse = null, + onSubmit, +}: { + request: HumanInputRequest; + disabled?: boolean; + pending?: boolean; + answeredResponse?: HumanInputResponse | null; + onSubmit?: ( + response: HumanInputResponse, + ) => HumanInputSubmitResult | Promise; +}) { + const { t } = useI18n(); + const [text, setText] = useState(""); + const [error, setError] = useState(""); + const [isComposing, setIsComposing] = useState(false); + const titleId = useId(); + const textInputId = useId(); + const allowText = + request.input_mode === "free_text" || + request.input_mode === "choice_with_other"; + const options = request.options ?? []; + const readOnly = !onSubmit; + const isDisabled = + disabled || pending || Boolean(answeredResponse) || readOnly; + const statusLabel = answeredResponse + ? t.humanInput.answered + : pending + ? t.humanInput.pending + : readOnly + ? t.humanInput.readOnly + : null; + + const submitResponse = async (response: HumanInputResponse) => { + if (isDisabled || !onSubmit) { + return; + } + setError(""); + const result = await onSubmit(response); + if (result !== false && response.response_kind === "text") { + setText(""); + } + }; + + const handleOptionClick = (option: HumanInputOption) => { + void submitResponse(createHumanInputOptionResponse(request, option)); + }; + + const handleTextSubmit = (event: { preventDefault(): void }) => { + event.preventDefault(); + const value = text.trim(); + if (!value) { + setError(t.humanInput.emptyError); + return; + } + void submitResponse(createHumanInputTextResponse(request, value)); + }; + + const handleTextKeyDown = (event: KeyboardEvent) => { + if (shouldSubmitHumanInputTextOnKeyDown(event, isComposing)) { + event.preventDefault(); + const value = text.trim(); + if (!value) { + setError(t.humanInput.emptyError); + return; + } + void submitResponse(createHumanInputTextResponse(request, value)); + } + }; + + return ( +
+
+
+ +
+
+
+
+

+ {request.title ?? t.toolCalls.needYourHelp} +

+ {request.context ? ( +
+ +
+ ) : null} +
+ {statusLabel ? ( + + {pending ? ( + + ) : null} + {answeredResponse ? ( + + ) : null} + {statusLabel} + + ) : null} +
+ +
+ +
+ + {options.length > 0 ? ( +
+ {options.map((option) => ( + + ))} +
+ ) : null} + + {allowText ? ( +
+ +