Skip to content

feat(runtime): add OpenTelemetry observability - #168

Draft
joeyorlando wants to merge 13 commits into
mainfrom
joeyorlando/openappa-observability
Draft

feat(runtime): add OpenTelemetry observability#168
joeyorlando wants to merge 13 commits into
mainfrom
joeyorlando/openappa-observability

Conversation

@joeyorlando

@joeyorlando joeyorlando commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • export privacy-bounded hook traces, structured logs, decision/latency metrics, and a runtime heartbeat over standard OTLP
  • add a source-controlled OpenAPPA Grafana dashboard with policy/runtime health signals and same-dashboard trace-to-log drilldown
  • add an optional Archestra Agent run filter so OpenAPPA traces and correlated logs can be isolated to one orchestrated coding session
  • lay out decision traces and correlated logs in a left-hand stack while giving the selected trace waterfall the full combined height on the right
  • optionally capture exact tool-call JSON on spans as gen_ai.tool.call.arguments via the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true switch
  • keep tool arguments out of logs and metrics; replace payloads larger than 32 KiB with byte-size and SHA-256 span metadata
  • document local LGTM setup and provide Tempo/Loki/Prometheus starter datasource provisioning
  • add an opt-in PR-label workflow for immutable development artifacts

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test --workspace --locked
  • ran appa runtime against a local grafana/otel-lgtm stack and verified OTLP traces, logs, counters, latency histograms, and uptime
  • sent a real tool call through the local runtime and verified Tempo held the exact JSON argument attribute while Loki had trace-correlated decision logs with neither the argument attribute nor a unique payload marker
  • imported the dashboard in Grafana and verified the canonical Appa lockup, policy/runtime signal cards, in-dashboard trace selection, full-height span waterfall, trace-correlated Loki logs, and per-Agent-run filtering
  • deployed the immutable PR development artifact in the staging Lobster Claude runner, executed a fresh uniquely marked Bash tool call, and verified the exact command JSON on the matching Tempo span while its trace-correlated Loki log omitted the marker

@joeyorlando joeyorlando added the build-development-artifacts Build immutable development artifacts for this pull request label Sep 3, 2026
@joeyorlando joeyorlando added build-development-artifacts Build immutable development artifacts for this pull request and removed build-development-artifacts Build immutable development artifacts for this pull request labels Sep 3, 2026
@joeyorlando joeyorlando added build-development-artifacts Build immutable development artifacts for this pull request and removed build-development-artifacts Build immutable development artifacts for this pull request labels Sep 3, 2026
@joeyorlando joeyorlando added build-development-artifacts Build immutable development artifacts for this pull request and removed build-development-artifacts Build immutable development artifacts for this pull request labels Sep 3, 2026
@joeyorlando joeyorlando added build-development-artifacts Build immutable development artifacts for this pull request and removed build-development-artifacts Build immutable development artifacts for this pull request labels Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build-development-artifacts Build immutable development artifacts for this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants