feat(sdk,cli): 'rocketride otel' — OpenTelemetry bridge over the documented ingester protocol (zero engine changes)#1612
Conversation
…mented ingester protocol (rocketride-org#1609) Maps the engine's monitor event stream to standard OTel telemetry with zero engine/server changes — the client-side refinement invited by tracking issue rocketride-org#616 after the engine-invasive rocketride-org#498 was archived. - mapper.py: stateful FlowSpanMapper reconstructs span trees from apaevt_flow — task root span, pipe span per pipe id, component child spans paired by component identity (per the observability doc's warning against stack-position pairing); trace.error -> span status ERROR + exception event; apaevt_sse -> span events; unclosed spans at end/shutdown flagged rocketride.span.unclosed=true; per-pipe state is bounded. MetricsMapper maps apaevt_status_update snapshots to OTel instruments (object counts, rates, cpu/gpu memory + peaks). - Semantic conventions per the July 2026 semantic-conventions-genai snapshot: gen_ai.provider.name (not the deprecated gen_ai.system), gen_ai.operation.name from the well-known list only, span names '{operation} {model}' style, INTERNAL span kind for in-process component executions. - Privacy by default: lane payloads / prompt / result content never reach attributes or events unless include_content is set (8KB truncation cap when it is); enforced by a sentinel-scan test across all exported attributes and events. - setup.py: TracerProvider/MeterProvider + OTLP http/protobuf exporters; ALL opentelemetry imports are lazy so the base package imports without the extra. 53 unit tests (InMemorySpanExporter/InMemoryMetricReader; importorskip so the base CI matrix stays green), including a full replay of the captured real-engine event fixture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ocketride-org#1609) rocketride otel [--endpoint URL] [--protocol http|grpc] [--service-name] [--headers k=v,...] [--include-content] [--no-metrics] + the standard connection args. Subscribes via rrext_monitor with token '*' (the observability doc's recommended ingestion scope, types TASK|SUMMARY| FLOW|SSE), dispatches to the mappers, reconnects with capped backoff and resubscribes (subscriptions are per-connection and not durable), and flushes exporters on SIGINT/SIGTERM. Endpoint resolution honors the standard OTLP env contract: explicit --endpoint > OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (verbatim) > OTEL_EXPORTER_OTLP_ENDPOINT (base URL + /v1/traces|/v1/metrics) > default. Packaging: opentelemetry-sdk + OTLP http exporter live behind an optional extra — the base install gains zero dependencies, and running the subcommand without the extra exits 2 with the exact "pip install 'rocketride[otel]'" hint before any connection attempt. 25 tests: config precedence matrix, dispatch/reconnect/resubscribe with a fake client, shutdown flush ordering, missing-extra exit path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ecipes Setup guide covering the span model (task -> pipe -> component tree), the privacy default and --include-content, the FLOW-requires-trace-level callout, reconnect semantics, and copy-paste recipes with exact OTLP endpoints/auth for Jaeger all-in-one, Langfuse (/api/public/otel, Basic auth), and Datadog. Notes that Jaeger does not ingest OTLP metrics — pair it with --no-metrics or point metrics at a collector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
docs
🤖 Internal: Discord sync markerAuto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete. |
nihalnihalani
left a comment
nihalnihalani
left a comment
There was a problem hiding this comment.
Self-review from the author: inline notes on the decisions most worth a reviewer's attention. Full evidence chain (78 unit tests, bare-venv dependency audit, and the live engine → bridge → Jaeger e2e with the span tree verified through Jaeger's API) is in the PR description.
| pipes: Dict[int, _PipeState] = field(default_factory=dict) | ||
|
|
||
|
|
||
| class FlowSpanMapper: |
There was a problem hiding this comment.
The pairing state machine is the heart of the bridge and follows the observability doc's explicit warning: enter/leave are paired by component identity, never by stack position. Per (project_id, source, pipe-id) it tracks the open root and component spans; edge cases are all tested — interleaved pipes (no cross-contamination of parentage), nested same-name components, leave without enter, end arriving before all leaves (spans closed with rocketride.span.unclosed=true), and restart mid-run. Per-pipe state is bounded so a long-running ingestion service can't grow without limit. One honest limitation, stated in the docstrings rather than hidden: span timestamps are event-arrival times — the wire protocol carries no timestamps.
| ATTR_SSE_DATA = 'rocketride.sse.data' | ||
| ATTR_ERROR_TYPE = 'error.type' | ||
|
|
||
| GEN_AI_OPERATION_NAME = 'gen_ai.operation.name' |
There was a problem hiding this comment.
Semantic conventions were researched against the July 2026 semantic-conventions-genai repo rather than trained memory, and it mattered: gen_ai.system was deprecated for gen_ai.provider.name in semconv v1.36, the old gen_ai.prompt/gen_ai.completion content attributes were removed outright, and span naming settled on {operation} {model}. The mapper emits only gen_ai.provider.name / gen_ai.operation.name (well-known values only) / gen_ai.tool.name, with INTERNAL span kind for in-process component executions per the agent-spans guidance. Our internal devil's-advocate pass diffed every emitted attribute against the researched registry.
Content capture follows the spec's "SHOULD NOT capture by default" rule: without --include-content, no lane payload/prompt/result text reaches any attribute or event — enforced by a sentinel-scan test over the full export, and confirmed live in the Jaeger e2e where the exported trace provably lacks the message text.
| Attributes: | ||
| endpoint: OTLP endpoint base URL, or None to defer to the OTLP | ||
| exporters' own environment/default semantics (including the | ||
| signal-specific ``OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`` / |
There was a problem hiding this comment.
Endpoint resolution implements the OTLP SDK env contract precisely, because the two env vars have different semantics that are easy to get wrong: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT/_METRICS_ENDPOINT are used verbatim (they must already carry the signal path), while OTEL_EXPORTER_OTLP_ENDPOINT is a base URL to which /v1/traces / /v1/metrics are appended. Explicit --endpoint wins over both. The precedence matrix has its own test class. This came out of our internal review round — the first cut only honored the base-URL variable.
| "python-dotenv>=1.0.0", | ||
| ] | ||
|
|
||
| otel = [ |
There was a problem hiding this comment.
Dependency discipline is the make-or-break packaging decision here: everything OpenTelemetry lives behind the [otel] extra, all imports are lazy, and the base install gains zero dependencies. Verified in a bare venv audit: every other subcommand's --help works without the extra, the otel test modules skip cleanly via importorskip (so the existing CI matrix stays green), and rocketride otel exits 2 with the exact pip install 'rocketride[otel]' hint before any connection attempt. The gRPC exporter is deliberately not part of the extra — --protocol grpc names the additional package in its error (http/protobuf is the default and covers Jaeger/Langfuse/Datadog).
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughChangesThe Python client adds OpenTelemetry configuration and exports
CLI command and user-facing setup
Provider construction and monitor runtime
Span and metric reconstruction
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant RocketRideClient
participant run_bridge
participant FlowSpanMapper
participant MetricsMapper
participant OTLPProviders
RocketRideClient->>run_bridge: deliver TASK, SUMMARY, FLOW, and SSE events
run_bridge->>FlowSpanMapper: route task, flow, and SSE events
run_bridge->>MetricsMapper: route status snapshots
FlowSpanMapper->>OTLPProviders: export reconstructed spans
MetricsMapper->>OTLPProviders: export metric deltas and gauges
run_bridge->>OTLPProviders: flush providers during shutdown
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/README-otel-bridge.md`:
- Around line 216-218: Update the Errors documentation to state that the
described mapping applies to a component leave event handled by _flow_leave,
rather than any flow event. Keep the documented span status, exception event,
and error.type behavior unchanged.
- Around line 136-143: Replace inline credential construction and --headers
arguments with the documented OTEL_EXPORTER_OTLP_HEADERS environment-variable
configuration in docs/README-otel-bridge.md lines 136-143 and 156-160, and
packages/docs/content-static/cli.mdx lines 288-290. Preserve the existing
Langfuse and LangSmith header values while ensuring credentials are not exposed
in shell history or process arguments.
In `@docs/README-python-client.md`:
- Line 558: Update the get_all_projects CLI example in the documentation to use
the supported store command registered by RocketRideCLI.setup_parser(),
replacing rrext_store with store while preserving the example’s project-listing
intent.
In `@packages/client-python/src/rocketride/cli/commands/otel.py`:
- Around line 177-184: Update the informational message in the trace_level
handling block of the otel command so the valid none value explicitly states
that tracing remains disabled instead of suggesting it will emit FLOW events.
Preserve the existing guidance for other trace levels, using a special case for
none if needed.
In `@packages/client-python/src/rocketride/otelbridge/bridge.py`:
- Around line 75-77: Update the _log function to call the rocketlib.debug
diagnostic utility instead of print, while preserving the existing otel-bridge
message prefix and required stderr output behavior.
In `@packages/client-python/src/rocketride/otelbridge/config.py`:
- Around line 168-169: Update the header selection around headers_arg and
parse_headers so environment-derived generic OTLP headers are not passed as an
explicit exporter header set. Only parse and assign headers when the CLI
supplies args.headers; otherwise leave headers unset so signal-specific OTLP
header variables retain precedence.
In `@packages/client-python/src/rocketride/otelbridge/mapper.py`:
- Around line 533-540: Update the trace.error handling in the mapper method
containing this block so raw error content is never exported when
include_content is false. Use a generic error message for exception.message and
span status by default; when content capture is enabled, serialize and sanitize
the error, truncate it using the bridge’s existing limits, and use that safe
text for both outputs.
- Around line 296-318: The _get_run method currently creates a duplicate state
when a canonical run_id follows a fallback-correlated run. Update _resolve_key
or _get_run to detect the project_id|source alias, promote that existing
_RunState to the canonical run_id, update _runs and _aliases consistently, and
preserve its parentage and span data. Add a regression test covering a flow
without __id followed by a task event with run_id.
In `@packages/client-python/src/rocketride/otelbridge/setup.py`:
- Around line 182-185: Update the shutdown function so meter_provider.shutdown()
always executes even when tracer_provider.shutdown() raises: wrap the tracer
shutdown in a try/finally and invoke the meter shutdown in the finally block.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 85c1d8a9-12c1-434f-9984-3e707b181635
📒 Files selected for processing (18)
docs/README-clients.mddocs/README-otel-bridge.mddocs/README-python-client.mdpackages/client-python/pyproject.tomlpackages/client-python/src/rocketride/cli/commands/__init__.pypackages/client-python/src/rocketride/cli/commands/otel.pypackages/client-python/src/rocketride/cli/main.pypackages/client-python/src/rocketride/otelbridge/__init__.pypackages/client-python/src/rocketride/otelbridge/bridge.pypackages/client-python/src/rocketride/otelbridge/config.pypackages/client-python/src/rocketride/otelbridge/mapper.pypackages/client-python/src/rocketride/otelbridge/setup.pypackages/client-python/tests/fixtures/otel_bridge_events.jsonpackages/client-python/tests/test_otel_bridge.pypackages/client-python/tests/test_otel_cli.pypackages/client-python/tests/test_otel_mapper.pypackages/client-python/tests/test_otel_setup.pypackages/docs/content-static/cli.mdx
| rocketride list # List all active tasks | ||
| rocketride events ALL --token <token> # Stream task events | ||
| rocketride otel # Export pipeline traces + metrics over OpenTelemetry | ||
| rocketride rrext_store get_all_projects # List stored projects |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Replace the unsupported rrext_store command.
RocketRideCLI.setup_parser() only registers store; rocketride rrext_store get_all_projects will be rejected as an unknown command. Replace it with a supported rocketride store ... example or remove it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/README-python-client.md` at line 558, Update the get_all_projects CLI
example in the documentation to use the supported store command registered by
RocketRideCLI.setup_parser(), replacing rrext_store with store while preserving
the example’s project-listing intent.
There was a problem hiding this comment.
Verified and you're right that the example is wrong (main.py registers store, and get_all_projects doesn't exist) — but git diff shows that line is pre-existing content this PR never touched (our diff only added the adjacent otel line). Keeping this PR's scope tight; flagged it for a separate docs fix.
There was a problem hiding this comment.
@nihalnihalani, agreed—thanks for verifying. The documentation issue is valid but unrelated to this PR’s otel addition, so deferring it to a focused docs fix is appropriate.
Would you like me to open a follow-up GitHub issue for the invalid rrext_store get_all_projects example?
…rter import CI exposed a real environment our matrix missed: opentelemetry-api/sdk present as another package's transitive dependency (tool_daytona pulls them in) while the OTLP HTTP exporter is absent. Two fixes: - setup.py: the http trace/metric exporter imports are now guarded like the grpc ones — a partial install raises OtelNotInstalledError with the pip install 'rocketride[otel]' hint (CLI exit 2) instead of a raw ModuleNotFoundError. Verified live in a reproduced partial venv. - test_otel_setup.py: importorskip targets the most specific module the tests exercise (opentelemetry.exporter.otlp.proto.http), so the module skips cleanly in CI's partial environment; plus a regression test that simulates the partial install via a None sys.modules entry and asserts the friendly error. Partial venv: 63 passed + 1 module skip; full extra: 79 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…otion, header precedence, shutdown ordering CodeRabbit review round on rocketride-org#1612, each finding verified before acting: - mapper: trace.error no longer bypasses the privacy gate — by default only the error signal exports (status ERROR, bare exception event, generic description); --include-content restores verbatim text, now subject to the same 8KB truncation strings previously skipped. The sentinel privacy test plants the sentinel in trace.error. - mapper: a run first seen via flow events (before the task 'running' snapshot) is now promoted from its project|source fallback key to the canonical run id when it arrives — previously a second run state was created (broken parentage) and the orphaned fallback state leaked permanently. Confirmed with a failing repro; guard test ensures two distinct canonical runs sharing (project, source) never merge. - config/setup: only --headers become explicit exporter headers; env headers are left to the SDK so OTEL_EXPORTER_OTLP_TRACES_HEADERS / _METRICS_HEADERS signal precedence works (it was silently defeated). Precedence matrix tests added. - setup: meter provider shutdown wrapped in finally so metrics flush even when tracer shutdown raises. - CLI: --trace-level none message now says flow tracing stays disabled; docs recipes pass collector auth via OTEL_EXPORTER_OTLP_HEADERS env instead of argv (shell-history hygiene); error-mapping doc wording matches the implementation (component leave events). Full extra: 90 passed. Partial install (api/sdk without exporter): 69 passed + clean skip. ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/client-python/src/rocketride/otelbridge/mapper.py`:
- Around line 36-39: Update snapshot reconciliation so when canonical run_id
state already exists and the fallback (project_id, source) state is later
encountered, the fallback state is merged into or closed/dropped rather than
retained. Preserve the rule that distinct canonical IDs are never merged, and
add a regression test covering SSE creation, ID-less fallback creation, then
canonical reconciliation without duplicate spans or a stranded fallback run.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: debe47c1-821f-4d1f-83b9-2d52043a4180
📒 Files selected for processing (10)
docs/README-otel-bridge.mdpackages/client-python/src/rocketride/cli/commands/otel.pypackages/client-python/src/rocketride/otelbridge/config.pypackages/client-python/src/rocketride/otelbridge/mapper.pypackages/client-python/src/rocketride/otelbridge/setup.pypackages/client-python/tests/test_otel_bridge.pypackages/client-python/tests/test_otel_cli.pypackages/client-python/tests/test_otel_mapper.pypackages/client-python/tests/test_otel_setup.pypackages/docs/content-static/cli.mdx
…cedes it Follow-on to the run-promotion fix: an SSE event creates the canonical run state WITHOUT registering the (project, source) alias (SSE bodies carry no project/source per the protocol), so an id-less flow for the same run could still open a parallel fallback-keyed state — and because the canonical state already existed, the promotion branch was skipped, stranding the fallback state (kept alive forever by snapshot reconciliation) with duplicate spans. _get_run now reconciles whenever an event carries both the canonical id and project/source: no canonical state → promote (unchanged); canonical state exists → _absorb_fallback_run migrates non-colliding pipes (open spans re-stamped with the canonical run id), closes colliding pipes as rocketride.span.unclosed=true, and drops the fallback key. Only fallback-format keys are touched, so distinct canonical runs never merge (guard test unchanged and green). Both new regression tests failed before the fix. Full extra: 92 passed; partial install: 71 passed + expected skip; ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
#Hire-Me-RocketRide
Contribution Type
Feature —
rocketride otel: an OpenTelemetry bridge that exports pipeline traces and metrics over OTLP by consuming the engine's documented monitor protocol. Zero engine/server changes.Closes #1609 · Refs #616 (the refined successor to the concept archived from #498)
Summary
RocketRide's traces are closed — the observability doc says it itself: the stream "is not OpenTelemetry, Prometheus, Sentry, or webhooks", and invites users to "write your own ingester." This PR ships that ingester as a first-party CLI, so pipeline runs land in the dashboards ops teams already run (Jaeger/Tempo, Datadog, Langfuse, LangSmith, Phoenix):
pip install 'rocketride[otel]' rocketride otel --endpoint http://collector:4318Why this architecture: #498 proposed OTel by instrumenting the server (middleware in
packages/ai) and was archived in #616 as "promising … not ready for merge." This is the invited refinement with the invasiveness removed entirely — a client-side bridge overrrext_monitor(token: "*", which the observability doc calls "the recommended scope for an ingestion service"). The engine is untouched; native instrumentation can still come later without conflict.apaevt_flow→ a span tree per run — task root span → pipe span → component child spans, paired by component identity exactly as the protocol doc warns (never stack position).trace.error→ span statusERROR+ exception event;apaevt_sse("thinking", "tool_call") → span events; spans left open atend/shutdown are closed and flaggedrocketride.span.unclosed=true.semantic-conventions-genairepo, not memory:gen_ai.provider.name(the deprecatedgen_ai.systemis never emitted),gen_ai.operation.namefrom the well-known list only,{operation} {model}span naming,INTERNALspan kind for in-process component executions. Langfuse/Datadog render the spans natively.--include-contentis set (8 KB truncation when it is) — matching the spec's "SHOULD NOT capture by default" guidance. Enforced by a sentinel-scan test across every exported attribute and event, and proven live (see below).--no-metricsto disable):apaevt_status_updatesnapshots → OTel instruments (object counts, rates, CPU %, CPU/GPU memory + peaks) on the same endpoint.--endpoint>OTEL_EXPORTER_OTLP_TRACES_ENDPOINTverbatim >OTEL_EXPORTER_OTLP_ENDPOINTbase + signal path), capped-backoff reconnect with resubscribe (subscriptions are per-connection, per the doc), SIGINT/SIGTERM → close spans, flush, exit 0.pip install 'rocketride[otel]'— the base install gains zero dependencies, every other subcommand works without it, androcketride otelwithout the extra exits 2 with the exact install hint before any connection attempt.Validation
Unit (no server): 78/78 with the extra installed; 31 passed + 3 clean
importorskipskips in a bare venv (so the existing CI matrix stays green with no new base deps). Coverage: pairing state machine (interleaved pipes, nested same-name components, leave-without-enter, end-before-leaves, restart), parentage, error mapping, SSE events, privacy sentinel scan, metrics deltas, endpoint/env precedence matrix, reconnect + resubscribe, shutdown flush order, missing-extra exit path. Includes a full replay of a captured real-engine event fixture.ruffclean on all 12 files;gitleaksclean.Live end-to-end (the money demo) — real engine image +
jaegertracing/all-in-one:1.76.0in Docker: an echo pipeline started via the Python SDK withpipelineTraceLevel='summary', one question driven through it, bridge running--endpoint http://127.0.0.1:4318. After SIGINT, Jaeger's HTTP API returned exactly 1 trace with 6 spans in the contracted hierarchy (task root → pipe → component children),rocketride.*attributes present — and the message content provably absent from the exported trace (privacy default). A bare-venv audit separately proved the CLI degrades exactly as documented.Devil's-advocate catches (internal review, fixed before this PR)
pk_…/tk_…auth keys) — scrubbed to digit-free dummies; gitleaks re-run clean. Exactly the class of leak this fixture pipeline needed to be checked for.--protocol grpcwithout the gRPC exporter now exits 2 (was 1) with a precise install hint; signal-specificOTEL_EXPORTER_OTLP_TRACES_ENDPOINT/_METRICS_ENDPOINTenv vars are honored per the SDK spec; mapper per-pipe state is bounded.Notes for reviewers
pipelineTraceLevel(documented + called out in the guide);TASK/SUMMARYtelemetry flows for every run regardless./v1/metrics); the guide says to pair it with--no-metricsor point metrics at a collector. Trace export is unaffected.🤖 Generated with Claude Code
Summary by CodeRabbit
rocketride otelCLI command to export live pipeline traces and metrics via OTLP, with HTTP/gRPC protocol support, service naming,--no-metrics, and optional payload capture (--include-content).oteloptional dependency extra for the Python client.