Repository navigation
Conversation
The plugin depends on opentelemetry-api only, following Datasette core: no provider, no exporter, no sampler. datasette_agent/telemetry.py owns the tracer and meter under the plugin's own `datasette_agent` scope and the instrument helpers; datasette_agent/telemetry_registry.py is the single source of truth for every signal (empty for now, plus the bucket sets model calls, token counts and human waits need - core's SQLite buckets stop at 10 s). tests/conftest.py imports Datasette's plugin telemetry kit fixtures and front-loads the subprocess sdk-isolation test, the way core and the sibling plugins do. tests/test_telemetry_registry.py holds the static half of the conformance suite. The kit (datasette.telemetry_registry / telemetry / telemetry_testing) is public plugin API on the asg017/otel-phase1-6-plugin-kit branch (PR #2899) and not in a release yet, so pyproject carries a dev-only [tool.uv.sources] pin and CI a matching temporary pip step, both marked for removal once the datasette>= floor covers it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011e2hp8Ena11KG4YxXNoykH
…spans
One `invoke_agent datasette-agent` span per turn (chat, resume and cli
modes), a `chat {model}` CLIENT span per model response covering
exactly the streaming window, and a `datasette_agent.system_prompt`
span around the per-database permission checks and table listings that
run on every turn. The three follow the OpenTelemetry GenAI semantic
conventions (gen_ai.operation.name, gen_ai.provider.name, gen_ai.usage.*
including the 1.40 cache-token spellings) so trace backends render
them as GenAI spans.
Metrics: gen_ai.client.token.usage and gen_ai.client.operation.duration
(the semconv pair), time to first token, turn duration by mode and
outcome, chain steps per turn, and an UpDownCounter of turns in flight.
Outcome rules: a suspension on ask_user() or browser_task() is the
designed way a turn ends, not an error; only `error` and `chain_limit`
(llm's bare ValueError, matched on its message prefix) set ERROR
status, and a cancellation is neither. The turn span opens before the
notification drain and user-message insert so every db.query of the
turn nests under it, and run_agent's SSE error path reports to it
instead of escaping it.
Nothing user-supplied is recorded: no prompt or output text, no tool
arguments, no error messages. The conformance test plants sentinels in
the user message and the SQL and asserts they reach no signal in this
scope (the SQL reaches core's db.query span by core's own design).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011e2hp8Ena11KG4YxXNoykH
…ion counter
One `execute_tool {tool}` span per tool invocation, following the GenAI
execute_tool convention, with core's db.query spans (and the nested
internal SERVER span execute_write_sql produces) nesting underneath.
The non-context branch of make_llm_tools now routes through
execute_agent_tool too, so there is one code path and every tool -
this plugin's own and any registered by another plugin - gets a span.
AgentTool gains a plugin_name field that get_agent_tools fills from the
hookimpl, so an operator can see which plugin a slow tool came from.
Outcome rules: tools in this repo return errors as data, so the
returned payload is classified - permission_denied and not_found for
the two messages our own tools emit, error for any other top-level
"error" key, suspended for an llm.PauseChain - and only a raised
exception sets span status ERROR. The message is never recorded.
Metrics: datasette_agent.tool.duration by tool, plugin and outcome
(call count and error rate derive from it), and a
tool.output.truncated counter recorded at the three after_call sites
that run prepare_tool_output_for_model, via a small helper that
detects the marker it writes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011e2hp8Ena11KG4YxXNoykH
…on spans A background run is one `invoke_agent` turn (mode background or explorer) started as a root span in its own trace with a link back to whatever span was current when the task was created - the API request, or the spawning turn's tool span - via core's linked_root_span_kwargs(). asyncio.create_task copies the ambient context, so without this every span of a 30-minute run would be filed as a child of a request that ended in milliseconds. Each loop pass gets a `datasette_agent.background.iteration` span, and the non-streaming model calls get chat spans (streaming=False, no first-token event). The run's outcome is completed, error, max_iterations or cancelled; the cancel endpoint's CancelledError reaches the span as outcome=cancelled with status UNSET, while max_iterations is an ERROR. A background.iterations histogram by mode and outcome records how close runs come to the cap; in-flight runs are the mode=background / explorer series of turns.active. run_background_agent is split into the span-owning entry point, the run body and one function per iteration; api.start_background_agent and explorer.start_explorer pass the mode through. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011e2hp8Ena11KG4YxXNoykH
…rn back ask_user() and browser_task() end a turn by persisting a row and raising an llm.PauseChain; the wait that follows can outlive the process, so it is not a span. Instead the execute_tool span that suspended carries the suspension kind, the question type and the question or task id, a datasette_agent.suspensions counter is incremented where the pending row is inserted (not where an already-pending call re-raises on resume), and a datasette_agent.suspension.wait histogram is recorded at the four resolution sites - answer, complete, cancel, expire - after their guarded UPDATE succeeds, so a lost race is not double-counted. The resumed turn records what it continues from (resumed_from) and links back to the suspending span: agent_questions and agent_browser_tasks gain trace_id / span_id columns (W3C hex, NULL when nothing is recording) written at insert time and read by the answer, complete and cancel views, which hand the row to resume_agent. The re-executed tool call that consumes the stored answer or result is marked tool.replayed=true. ensure_tables grows a small column migration table covering the existing html column and the two new ones. The tests' echo-driven tool plugins move into conftest.py under their own names so every telemetry test file can use them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011e2hp8Ena11KG4YxXNoykH
An Observability section in the README explains the API-only stance, how an operator turns telemetry on (opentelemetry-instrument or the datasette-otel-viewer plugin), the trace shape of a chat turn, the nested SERVER span execute_write_sql produces, the linked roots for background runs and resumed turns, and the privacy rules. Below it, scripts/telemetry-doc.py renders every span, attribute and metric from datasette_agent/telemetry_registry.py into a marked block, so the reference cannot drift from the code: a test regenerates and compares it, and CI runs the script's --check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011e2hp8Ena11KG4YxXNoykH
Datasette 1.0a41 ships the plugin telemetry kit, so drop the git pin to the kit branch. The datasette>=1.0a37 floor stays: telemetry_compat re-exports the kit when present and falls back to stand-in registry classes and linked_root_span_kwargs otherwise, so the agent's own spans and metrics are emitted either way. Kit-dependent tests importorskip, and CI gains a datasette 1.0a37 leg to exercise the fallback. TODO: raise the floor to the first stable 1.x with the kit and delete the shim. Also accept 1.0a41's "private, no-store" Cache-Control on authenticated stream responses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Author
|
Superseded by a smaller approach: llm and datasette-llm now emit the model-call and tool-call spans themselves (simonw/llm#1712, datasette/datasette-llm#8, with a purpose metric in datasette/datasette-llm#9), and datasette-agent only adds a turn span on top: #43 (turn span), #44 (background agents as linked roots). The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Claude-generated PR description
Instruments datasette-agent with OpenTelemetry, through
opentelemetry-apionly. The plugin never installs a provider or exporter. With no SDK present, every span and instrument is a no-op.What's traced
invoke_agent datasette-agentwraps each turn, with achat {model}span per model response (kindCLIENT) and adatasette_agent.system_promptspan. The span shapes follow the OpenTelemetry GenAI semantic conventions (gen_ai.*).execute_toolspan for each call, plus a duration histogram and a counter for truncated tool output.All signals are declared in
datasette_agent/telemetry_registry.py. The README's signal reference is generated from that file byscripts/telemetry-doc.py, and CI runs--checkso the reference can't go stale.Datasette compatibility
Datasette 1.0a41 ships the plugin telemetry kit (
datasette.telemetry_registry,datasette.telemetry,datasette.telemetry_testing) and instruments core itself. This PR keeps thedatasette>=1.0a37floor:datasette_agent/telemetry_compat.pyre-exports the kit when it is installed, and falls back to stand-in registry classes andlinked_root_span_kwargswhen it is not.db.queryspans are absent.importorskipit.tests/test_telemetry_compat.pyforces the fallback path, so it is exercised on 1.0a41 as well.TODO: raise the floor to the first stable 1.x release that includes the kit, then delete the shim.
Also in this PR
With 1.0a41, core sends
Cache-Control: private, no-storeon authenticated stream responses instead of the stream'sno-cache.test_stream_endpoint_headersnow accepts either value. The test would fail onmaintoo oncemainpicks up 1.0a41.Testing
scripts/telemetry-doc.py --checkpasses on both versions.🤖 Generated with Claude Code