feat(trace): add agent observability with Monocle

.gitignore

@@ -66,3 +66,6 @@ config.yaml.bak
 /frontend/playwright-report/
 .gstack/
 .worktrees
+
+# Monocle agent-observability trace output (local file exporter)
+.monocle/

README.md

@@ -58,7 +58,8 @@ DeerFlow has newly integrated the intelligent search and crawling toolset indepe
       - [IM Channels](#im-channels)
       - [LangSmith Tracing](#langsmith-tracing)
       - [Langfuse Tracing](#langfuse-tracing)
-      - [Using Both Providers](#using-both-providers)
+      - [Monocle Tracing](#monocle-tracing)
+      - [Using Multiple Providers](#using-multiple-providers)
   - [From Deep Research to Super Agent Harness](#from-deep-research-to-super-agent-harness)
   - [Core Features](#core-features)
     - [Skills \& Tools](#skills--tools)
@@ -593,11 +594,25 @@ If you are using a self-hosted Langfuse instance, set `LANGFUSE_BASE_URL` to you
 
 These are injected into `RunnableConfig.metadata` at the graph invocation root for both the gateway path (`runtime/runs/worker.py::run_agent`) and the embedded path (`client.py::DeerFlowClient.stream`), so any LangChain-compatible callback can read them. Set `DEER_FLOW_ENV` (or `ENVIRONMENT`) to tag traces by deployment environment.
 
-#### Using Both Providers
+#### Monocle Tracing
 
-If both LangSmith and Langfuse are enabled, DeerFlow attaches both tracing callbacks and reports the same model activity to both systems.
+DeerFlow also supports [Monocle](https://github.com/monocle2ai/monocle), an OpenTelemetry-based tracer for agentic applications. It records each run end-to-end: LLM calls, agent steps, and tool and MCP invocations, with their inputs, outputs, timings, and token counts.
 
-If a provider is explicitly enabled but missing required credentials, or if its callback fails to initialize, DeerFlow fails fast when tracing is initialized during model creation and the error message names the provider that caused the failure.
+Add the following to your `.env` file:
+
+```bash
+MONOCLE_TRACING=true
+MONOCLE_EXPORTERS=file          # file, console, okahu, s3, blob, gcs (default: file)
+OKAHU_API_KEY=okh_xxxxxxxx      # required only for the `okahu` exporter
+```
+
+Each run writes one trace file to `.monocle/`; open it in the [Monocle VS Code extension](https://marketplace.visualstudio.com/items?itemName=OkahuAI.monocle-apptrace) to inspect the span timeline and token counts. Connect to [Okahu](https://www.okahu.ai), an agent-observability platform, to analyze traces across runs and run trace-based and agentic evaluations (via the `okahu` exporter).
+
+Traces capture span inputs and outputs verbatim — prompts, tool arguments, and model responses — plus token usage and timings. The `file` exporter keeps them on local disk and never rotates or cleans them up, so prune `.monocle/` periodically; the remote exporters (`okahu`, `s3`, `blob`, `gcs`) send that same data off-box, so enable only destinations you trust. Monocle is initialized once at Gateway startup: a configuration error (unknown exporter, missing `OKAHU_API_KEY`) is logged there and tracing stays off until the Gateway restarts.
+
+#### Using Multiple Providers
+
+LangSmith and Langfuse attach as LangChain callbacks, so you can enable both and DeerFlow reports each run to both. If an enabled provider is missing required credentials or fails to initialize, DeerFlow fails fast and names it. Monocle uses a global OpenTelemetry provider rather than a callback; Langfuse shares that provider, so all three can run together. Because both span processors sit on the same shared provider, Monocle's exporters also see Langfuse's spans when both are enabled.
 
 For Docker deployments, tracing is disabled by default. Set `LANGSMITH_TRACING=true` and `LANGSMITH_API_KEY` in your `.env` to enable it.
 

backend/AGENTS.md

@@ -711,6 +711,12 @@ LangSmith and Langfuse are both supported. The wiring lives in two layers:
 
 Returns `{}` when Langfuse is not in the enabled providers — LangSmith-only deployments are unaffected. Set `DEER_FLOW_ENV` (or `ENVIRONMENT`) to tag traces by deployment environment. Tests live in `tests/test_tracing_factory.py`, `tests/test_tracing_metadata.py`, `tests/test_worker_langfuse_metadata.py`, `tests/test_client_langfuse_metadata.py`, and `tests/test_subagent_executor.py::TestSubagentTracingWiring`.
 
+**Monocle telemetry** is a third provider, structurally unlike LangSmith/Langfuse. It is **not** a LangChain callback: `tracing/monocle.py::setup_monocle_tracing_if_enabled()` calls `monocle_apptrace.setup_monocle_telemetry()` once, which installs a **process-global OTel `TracerProvider`**, patches span serialization, and auto-instruments the openai/langchain/langgraph clients. Because that is a one-time, process-global side effect (not a per-run callback), it is initialized from the **Gateway lifespan** (`app/gateway/app.py`) — never from `build_tracing_callbacks()` — and it is **off by default**. The setup call was deliberately moved out of `agents/__init__.py`, so `import deerflow.agents` must never start tracing (pinned by `tests/test_monocle_tracing.py::test_no_import_time_setup`). The Gateway lifespan is the **sole call site** (pinned by `test_gateway_lifespan_initializes_monocle`), so unlike LangSmith/Langfuse — which attach at the graph roots and cover every path — the embedded `DeerFlowClient` and the TUI are not instrumented; embedded users who want Monocle traces call `setup_monocle_tracing_if_enabled()` themselves before running the agent.
+
+Unlike the Langfuse metadata above, DeerFlow injects **no** per-run fields into Monocle traces — the only attribute it sets is `workflow_name="deer-flow"`; every span attribute (`span.type`, `entity.*`, token usage, span inputs/outputs, `scope.agentic.session`) is produced by Monocle's own metamodel and auto-instrumentation, so there is no DeerFlow trace-attribute layer to maintain here.
+
+Config is env-driven like the others — `MonocleTracingConfig`, built in `get_tracing_config()` and gated by `is_monocle_tracing_enabled()`. `MONOCLE_TRACING` enables it; `MONOCLE_EXPORTERS` selects exporters (default `file` → trace JSON in `.monocle/`; also `console`, `okahu`, `s3`, `blob`, `gcs`, where `okahu` requires `OKAHU_API_KEY`). `setup_monocle_tracing_if_enabled()` stays a thin wrapper on purpose: `monocle_apptrace` already guards duplicate setup (`instrumentor.py::check_duplicate_setup`) and never force-overrides an existing global provider, so the wrapper only gates on config. Coexistence with Langfuse (v4, also OTel-based) is **verified**: whichever library initializes second reuses the existing global `TracerProvider` and attaches its own span processor, so neither side loses spans (pinned by `test_coexists_with_langfuse`). Both processors see all spans, so Monocle's exporters also capture Langfuse's spans when both are enabled. (LangSmith is a plain callback and coexists trivially.) Tests: `tests/test_monocle_tracing.py`.
+
 ### Config Schema
 
 **`config.yaml`** key sections:

backend/app/gateway/app.py

@@ -37,6 +37,7 @@
 from app.gateway.trace_middleware import TraceMiddleware, resolve_trace_enabled
 from deerflow.config import app_config as deerflow_app_config
 from deerflow.logging_config import DEFAULT_LOG_DATE_FORMAT, DEFAULT_LOG_FORMAT, configure_logging
+from deerflow.tracing.monocle import setup_monocle_tracing_if_enabled
 from deerflow.uploads.manager import cleanup_stale_upload_staging_files
 
 AppConfig = deerflow_app_config.AppConfig
@@ -189,6 +190,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
     config = get_gateway_config()
     logger.info(f"Starting API Gateway on {config.host}:{config.port}")
 
+    # Agent observability (Monocle). Off by default; enabled with
+    # MONOCLE_TRACING. Initialized here at startup — not at import time — so a
+    # plain `import deerflow.agents` never installs a process-global tracer.
+    # Unlike LangSmith/Langfuse, whose validation failures abort the agent run,
+    # a bad Monocle config only logs: the Gateway keeps serving without tracing.
+    try:
+        setup_monocle_tracing_if_enabled()
+    except Exception:  # observability must never break startup
+        logger.exception("Monocle tracing setup failed; continuing without it")
+
     # Pre-warm tiktoken encoding cache so the first memory-injection request
     # never blocks on the BPE data download (which hits an OpenAI/Azure URL
     # that may be unreachable in restricted networks — see issue #3402).

backend/packages/harness/deerflow/config/__init__.py

@@ -9,6 +9,7 @@
     get_enabled_tracing_providers,
     get_explicitly_enabled_tracing_providers,
     get_tracing_config,
+    is_monocle_tracing_enabled,
     is_tracing_enabled,
     validate_enabled_tracing_providers,
 )
@@ -27,6 +28,7 @@
     "get_tracing_config",
     "get_explicitly_enabled_tracing_providers",
     "get_enabled_tracing_providers",
+    "is_monocle_tracing_enabled",
     "is_tracing_enabled",
     "validate_enabled_tracing_providers",
 ]

backend/packages/harness/deerflow/config/tracing_config.py

@@ -47,11 +47,47 @@ def validate(self) -> None:
             raise ValueError(f"Langfuse tracing is enabled but required settings are missing: {', '.join(missing)}")
 
 
+# Manual mirror of monocle_apptrace's supported exporters, kept local so a typo
+# fails at startup with a clear message instead of an opaque upstream error.
+# Update this tuple when a monocle_apptrace bump adds or renames an exporter.
+_MONOCLE_EXPORTERS = ("file", "console", "okahu", "s3", "blob", "gcs")
+
+
+class MonocleTracingConfig(BaseModel):
+    """Configuration for Monocle telemetry."""
+
+    enabled: bool = Field(...)
+    exporters: str = Field(...)
+    okahu_api_key: str | None = Field(...)
+
+    @property
+    def is_enabled(self) -> bool:
+        # Unlike the siblings' is_configured, no credential check here: that is
+        # exporter-dependent and lives in validate(), run at Gateway startup.
+        return self.enabled
+
+    @property
+    def exporter_list(self) -> list[str]:
+        """The configured exporters, parsed once so validation and setup agree."""
+        return [e.strip() for e in self.exporters.split(",") if e.strip()]
+
+    def validate(self) -> None:
+        if not self.enabled:
+            return
+        selected = self.exporter_list
+        unknown = [e for e in selected if e not in _MONOCLE_EXPORTERS]
+        if unknown:
+            raise ValueError(f"MONOCLE_EXPORTERS has unknown exporter(s): {', '.join(unknown)}. Allowed: {', '.join(_MONOCLE_EXPORTERS)}.")
+        if "okahu" in selected and not self.okahu_api_key:
+            raise ValueError("Monocle 'okahu' exporter is selected but OKAHU_API_KEY is not set.")
+
+
 class TracingConfig(BaseModel):
     """Tracing configuration for supported providers."""
 
     langsmith: LangSmithTracingConfig = Field(...)
     langfuse: LangfuseTracingConfig = Field(...)
+    monocle: MonocleTracingConfig = Field(...)
 
     @property
     def is_configured(self) -> bool:
@@ -125,6 +161,11 @@ def get_tracing_config() -> TracingConfig:
                 secret_key=_first_env_value("LANGFUSE_SECRET_KEY"),
                 host=_first_env_value("LANGFUSE_BASE_URL") or "https://cloud.langfuse.com",
             ),
+            monocle=MonocleTracingConfig(
+                enabled=_env_flag_preferred("MONOCLE_TRACING"),
+                exporters=_first_env_value("MONOCLE_EXPORTERS") or "file",
+                okahu_api_key=_first_env_value("OKAHU_API_KEY"),
+            ),
         )
         return _tracing_config
 
@@ -149,6 +190,16 @@ def is_tracing_enabled() -> bool:
     return get_tracing_config().is_configured
 
 
+def is_monocle_tracing_enabled() -> bool:
+    """Whether Monocle OTel observability is enabled (via ``MONOCLE_TRACING``).
+
+    Kept separate from :func:`get_enabled_tracing_providers` because Monocle is a
+    process-global instrumentor activated at startup, not a per-run LangChain
+    callback.
+    """
+    return get_tracing_config().monocle.is_enabled
+
+
 def reset_tracing_config() -> None:
     """Discard the cached :class:`TracingConfig` so the next call rebuilds it.
 

backend/packages/harness/deerflow/tracing/__init__.py

@@ -1,8 +1,10 @@
 from .factory import build_tracing_callbacks
 from .metadata import build_langfuse_trace_metadata, inject_langfuse_metadata
+from .monocle import setup_monocle_tracing_if_enabled
 
 __all__ = [
     "build_langfuse_trace_metadata",
     "build_tracing_callbacks",
     "inject_langfuse_metadata",
+    "setup_monocle_tracing_if_enabled",
 ]

backend/packages/harness/deerflow/tracing/factory.py

@@ -1,12 +1,17 @@
 from __future__ import annotations
 
+import logging
 from typing import Any
 
 from deerflow.config import (
     get_enabled_tracing_providers,
     get_tracing_config,
+    is_monocle_tracing_enabled,
     validate_enabled_tracing_providers,
 )
+from deerflow.tracing.monocle import is_monocle_setup_completed
+
+logger = logging.getLogger(__name__)
 
 
 def _create_langsmith_tracer(config) -> Any:
@@ -32,6 +37,12 @@ def _create_langfuse_handler(config) -> Any:
 def build_tracing_callbacks() -> list[Any]:
     """Build callbacks for all explicitly enabled tracing providers."""
     validate_enabled_tracing_providers()
+    # Monocle is not a callback provider; this per-run path is just where an
+    # embedded process that skipped Gateway-lifespan setup can be told about it.
+    if is_monocle_tracing_enabled() and not is_monocle_setup_completed():
+        logger.debug(
+            "MONOCLE_TRACING is set but Monocle is not initialized in this process — only the Gateway lifespan runs setup automatically; embedded/TUI callers must call deerflow.tracing.setup_monocle_tracing_if_enabled() themselves."
+        )
     enabled_providers = get_enabled_tracing_providers()
     if not enabled_providers:
         return []

backend/packages/harness/deerflow/tracing/monocle.py

@@ -0,0 +1,69 @@
+"""Monocle telemetry: initialized once from the Gateway lifespan when ``MONOCLE_TRACING`` is set."""
+
+from __future__ import annotations
+
+import logging
+
+from deerflow.config import (
+    get_enabled_tracing_providers,
+    get_tracing_config,
+    is_monocle_tracing_enabled,
+)
+
+logger = logging.getLogger(__name__)
+
+# Read by build_tracing_callbacks() to hint embedded/TUI processes that
+# enabled MONOCLE_TRACING but never ran the Gateway-lifespan setup.
+_setup_completed = False
+
+
+def is_monocle_setup_completed() -> bool:
+    """Whether :func:`setup_monocle_tracing_if_enabled` ran in this process."""
+    return _setup_completed
+
+
+def setup_monocle_tracing_if_enabled() -> bool:
+    """Initialize Monocle telemetry when ``MONOCLE_TRACING`` is enabled; a no-op otherwise.
+
+    ``monocle_apptrace.setup_monocle_telemetry()`` is idempotent, so this stays a thin,
+    config-gated wrapper. Returns ``True`` when enabled.
+    """
+    if not is_monocle_tracing_enabled():
+        return False
+
+    monocle = get_tracing_config().monocle
+    # Fail fast on an unknown MONOCLE_EXPORTERS value or a missing OKAHU_API_KEY,
+    # with a clear message, before instrumenting. Validated here (not in the
+    # per-run callback path) so a config typo never breaks agent runs.
+    monocle.validate()
+
+    # Coexistence with Langfuse (v4, also OTel-based) is verified: whichever
+    # library initializes second reuses the existing global TracerProvider and
+    # attaches its own span processor, so neither side loses spans (see
+    # test_coexists_with_langfuse). Both processors see all spans, so Monocle's
+    # exporters also capture Langfuse's spans when both are enabled.
+    exporters = monocle.exporters
+
+    # `console` stays on local stdout, so only the remote exporters are flagged.
+    off_box = [e for e in monocle.exporter_list if e not in ("file", "console")]
+    if off_box:
+        # Monocle's exporters see every span on the shared global provider, so a
+        # co-enabled OTel provider's spans leave the box too.
+        langfuse_note = " Langfuse is also enabled and shares the global provider, so its spans are exported there as well." if "langfuse" in get_enabled_tracing_providers() else ""
+        logger.warning(
+            "Monocle is exporting trace data (prompts, tool inputs/outputs, completions) beyond the local .monocle/ file via: %s. Make sure that destination is trusted.%s",
+            ", ".join(off_box),
+            langfuse_note,
+        )
+
+    try:
+        from monocle_apptrace import setup_monocle_telemetry
+    except ImportError as exc:
+        raise RuntimeError("MONOCLE_TRACING is enabled but monocle_apptrace is not installed. Install the 'monocle' extra: `uv sync --extra monocle` in backend/, or `pip install 'deerflow-harness[monocle]'`.") from exc
+
+    # monocle_exporters_list takes the comma-separated string as-is (monocle_apptrace's API).
+    setup_monocle_telemetry(workflow_name="deer-flow", monocle_exporters_list=exporters)
+    global _setup_completed
+    _setup_completed = True
+    logger.info("Monocle telemetry enabled (exporters=%s)", exporters)
+    return True

backend/packages/harness/pyproject.toml

@@ -65,6 +65,9 @@ postgres = [
 redis = ["redis>=5.0.0"]
 pymupdf = ["pymupdf4llm>=0.0.17"]
 boxlite = ["boxlite>=0.9.7"]
+# Agent observability (Monocle). Optional so a default install stays free of the
+# OpenTelemetry stack; only pulled in when MONOCLE_TRACING is used.
+monocle = ["monocle_apptrace>=0.8.8"]
 
 [build-system]
 requires = ["hatchling"]

backend/pyproject.toml

@@ -28,6 +28,7 @@ dependencies = [
 postgres = ["deerflow-harness[postgres]"]
 redis = ["deerflow-harness[redis]"]
 discord = ["discord.py>=2.7.0"]
+monocle = ["deerflow-harness[monocle]"]
 
 [dependency-groups]
 dev = [
@@ -37,6 +38,9 @@ dev = [
     "pytest>=9.0.3",
     "pytest-asyncio>=1.3.0",
     "ruff>=0.14.11",
+    # Monocle tracer (also the deerflow-harness[monocle] extra); kept in the dev
+    # group so the tracing tests can import it without forcing it onto installs.
+    "monocle_apptrace>=0.8.8",
     # redis is an optional runtime extra (deerflow-harness[redis]); pin it in the
     # dev group so the stream-bridge tests can always import/exercise the redis
     # bridge without forcing it onto production installs.