feat(observability): name AI traces in PostHog with a $ai_trace envelope - #10223
Merged
Conversation
Contributor
|
Important 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏳ LoopOver is waiting…LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 5785282 | Commit Preview URL Branch Preview URL |
Jul 31 2026, 01:49 PM |
Bundle ReportChanges will increase total bundle size by 277 bytes (0.0%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
JSONbored
force-pushed
the
feat/ai-trace-envelope
branch
from
July 31, 2026 13:20
1c72c29 to
60e2cf9
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10223 +/- ##
==========================================
- Coverage 92.25% 91.38% -0.87%
==========================================
Files 938 938
Lines 114659 114688 +29
Branches 27680 27689 +9
==========================================
- Hits 105774 104809 -965
- Misses 7580 8769 +1189
+ Partials 1305 1110 -195
Flags with carried forward coverage won't be shown. Click here to find out more.
|
PostHog's Traces view takes a trace's NAME from a trace-level $ai_trace event, not from any property on the generations underneath it -- confirmed upstream in PostHog/posthog#33179, where an $ai_span_name set on a generation does not populate that column. This project emitted only generations, so every trace read as an anonymous id with no way to tell a gate review from an embedding batch. Emit the envelope from withReviewPipelineSpan, which already wraps a whole review and carries the repo, PR and the ambient OTel trace id the generations group by. It runs INSIDE the OTel span, because that id only exists once the span is open. Two guards, both load-bearing: - Outermost only. withReviewPipelineSpan is called from several sites that can nest, and PostHog expects one $ai_trace per trace. A depth counter means inner calls contribute nothing and the surviving name is the whole operation rather than whichever leaf finished last. - Only when generations exist. Not every pipeline span wraps an AI call -- the gate does not. Emitting unconditionally would manufacture trace rows with nothing under them, which reads worse than an unnamed trace. A pass-through when PostHog is off or no ambient trace exists, so it never changes what the wrapped work does or what it throws. Closes #10221
JSONbored
force-pushed
the
feat/ai-trace-envelope
branch
from
July 31, 2026 13:46
60e2cf9 to
5785282
Compare
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Trace name column is empty for every row in PostHog's Traces view, because this project emits no
$ai_traceevent at all.PostHog derives the trace row itself from the generations underneath — which is why latency and cost populate — but the name comes from a trace-level event. Confirmed upstream in PostHog/posthog#33179: an
$ai_span_nameset on a generation does not populate that column. So every trace read as an anonymous id, with no way to tell a gate review from an embedding batch from a summary at a glance.The envelope is emitted from
withReviewPipelineSpan, which already wraps a whole review and carries the repo, the PR number, and the ambient OTel trace id that #10187 made the generations group by. It runs inside the OTel span, because that trace id only exists once the span is open — outside it there is nothing yet to name.The two guards, both load-bearing
withReviewPipelineSpanis called from several sites insrc/queue/processors.tsand they can nest, while PostHog expects one$ai_traceper trace id. A depth counter means inner calls contribute nothing, and the name that survives is the whole operation rather than whichever leaf happened to finish last.capturePostHogAiGenerationmarks its trace as having real work; the envelope is skipped otherwise.It is a transparent pass-through when PostHog is off or there is no ambient trace, so it never changes what the wrapped work does or what it throws — the callback's return value and exception propagate untouched.
Closes #10221
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateAlso run and green, because this change touches their inputs:
npm run dead-exports:check— green.POSTHOG_AI_TRACE_EVENTis referenced from the tests and the docs drift guard, followingPOSTHOG_MONITOR_HEARTBEAT_EVENT's existing pattern.prettier --checkon the changed MDX — clean.If any required check was skipped, explain why:
selfhost-posthog+selfhost-otel(the suite that already driveswithReviewPipelineSpan): 100% of the changed lines AND branches, verified line-by-line against lcov.Tests added
Nine cases, covering every branch rather than the happy path: pass-through with PostHog off; pass-through with no ambient trace; the envelope claiming the same trace id its generations carry; a pipeline span with no AI calls getting no envelope; nested spans emitting exactly one, named by the outermost; a throwing callback marked errored and rethrown; a non-
Errorthrown value and a bounded 500-char message; the repo group omitted when there is no repo; and a second span reusing the same trace id starting from a clean slate rather than inheriting the first's generation count.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.The envelope carries only the span name, timing, an error flag and the already-processed operational context — the same metadata-only posture as
$ai_generation. It reusesoperationalPropertiesandrepoGroup, so under the shared central key the repo is HMAC-anonymized and fail-closed exactly as every other event on this path. No prompt, completion or trace state is attached.UI Evidence
Not applicable — no visible UI or frontend change. The docs change is a
FeatureRowentry on the existing self-hosting operations page, pinned by a drift test.Notes
The bookkeeping map is keyed by trace id and deleted when depth returns to zero, in a
finally, so a throwing span cannot leak an entry. A test asserts a later span over the same trace id starts clean rather than inheriting a stale generation count.captureAiTraceEnvelopetakes the client that was active when the span opened rather than re-reading the module binding in thefinally— which also means it needs no second, unreachable off-switch check of its own.