fix(observability): group AI generations by their real trace, and attribute spend per repo - #10187
Merged
Merged
Conversation
…he MCP event contract Umami is fully decommissioned and PostHog is the only analytics sink. Web analytics (#8293, #8299). posthog-js replaces the hand-rolled Umami beacon in routes/__root.tsx, behind the same first-party /stats proxy the Umami collect endpoint used. lib/analytics.ts is the single seam; the SDK is loaded via a dynamic import so an unconfigured build pays zero bytes, matching browser-sentry.ts. Env is read at call time, not module scope, so the token gate is stubbable. Umami parity was audited against the removed beacon's actual payload: posthog-js supplies url/referrer/screen/language automatically, page_title is added explicitly (it has no $title equivalent), geo still comes from the proxy's x-forwarded-for, and respect_dnt replaces the beacon's own doNotTrack check. persistence is "localStorage", not "memory" — memory resets identity every reload and inflates unique visitors, the regression JSONbored/metagraphed#8210 had to correct. cookieless_mode was rejected on its documented behavior: it strips the IP before GeoIP runs, which would drop the country data #8299 requires. The proxy keeps its allowlist, one level looser (path prefixes, not exact paths) so a posthog-js upgrade cannot silently break capture. Its original justification was Umami-specific — that host served its admin API from the collect origin — which does not carry over, but bounding a public unauthenticated route still does. Session-replay snapshots get their own 2 MiB ceiling so enabling #8295 cannot silently 413. One behavior deliberately changes: the beacon fired once per page load and never on SPA navigation, so pageview counts run legitimately higher after cutover. MCP telemetry (#10175). $mcp_tool_call was emitting LoopOver's own snake_case property names under PostHog's reserved event name, so their built-in MCP dashboards ingested every event and rendered every breakdown empty. It now carries the canonical $mcp_* properties, with LoopOver's surface/transport/ category alongside them (PostHog's custom-server docs sanction extra properties, and those three have no canonical equivalent). usage_event is untouched — it stays the minimal, payload-free LoopOver event in its own vocabulary, and the allowlist meta-test now checks the two vocabularies separately. Adds the two missing canonical events: $mcp_initialize, sourced from the handshake's own clientInfo rather than the x-loopover-mcp-* headers that only our published client sets, and $mcp_tools_list, reporting what this server registered. Both carry $session_id from Mcp-Session-Id, which is what makes a funnel across handshake, discovery, and tool calls possible at all. The closed LoopOver error-code set is projected onto PostHog's closed $mcp_error_type set so their error breakdown populates without losing the precise code. Reshaping the wire format is free right now and stops being free immediately: the project holds zero $mcp_tool_call events, because POSTHOG_API_KEY was never set on the Worker. Prettier (#10176). A root prettier run found no config, fell back to the 80-column default, and rewrapped backend source that nothing in CI formats — repeatedly, inside unrelated PRs. Adds a default-deny root .prettierignore that re-admits only the three governed workspaces, restates the generated-file and content/docs exemptions that a root run cannot inherit, and pins ui-kit to its current effective width so it is stated rather than inherited from a vendor default. Verified by running prettier --write . from the root and confirming a no-op. No existing code is reformatted. Also formats two ui-kit test files that were already failing their own format:check on main, which would otherwise have turned this PR red. Closes #8293 Closes #8299 Closes #10175 Closes #10176
…ribute spend per repo Two faults made PostHog's AI observability structurally unable to answer the questions that matter most for a product where AI is the product. Every generation was its own trace. capturePostHogAiGeneration set $ai_trace_id to a fresh randomUUID() per call, producing 13,428 AI events across 13,428 distinct trace ids on the live project. A review fans out across RAG embeddings, both dual-review legs, N retries per model, and any self-consistency runs — and each landed as an unrelated single-event trace, so what a review actually did, and where its time and money went, was not recoverable. The fix was already in the file. operationalProperties has called currentOtelTraceIds() since #8296 and attached the real ids as plain trace_id/span_id properties; the line below then minted a UUID and PostHog grouped by that instead. withReviewPipelineSpan already wraps a whole review, so the ambient trace id is exactly the grouping key wanted — every provider attempt inside one review now nests under one PostHog trace. $ai_span_id and a readable $ai_span_name come from the same source. randomUUID() stays as the fallback rather than being removed: AI_EMBED, AI_VISION and AI_ADVISORY run outside any review span, so an orphan trace is a legitimate outcome there — but such an event must not claim a span id it does not have, so $ai_span_id is omitted rather than faked. AI spend was unattributable. Every AI event carried distinct_id "loopover-selfhost" and no $groups at all, so cost per repository was answerable only by SQL against ai_usage_events, never in PostHog. A repo group is now stamped on every event. It deliberately reads the already-processed repo value off operationalProperties rather than the raw context: under the shared LOOPOVER_CENTRAL_POSTHOG_KEY that value is HMAC-anonymized, and is dropped entirely when the anon secret has not been injected, so the group inherits that fail-closed behavior for free and a raw private repo name cannot reach the group index by this path. Both branches of both changes are covered: ambient span vs none, and repo present vs dropped. Content capture ($ai_input/$ai_output_choices) is deliberately untouched — the metadata-only policy is documented in self-hosting-operations.mdx and changing it is a privacy decision, not a wiring one. Closes #10185
Contributor
|
Important 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏳ LoopOver is waiting…LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Bundle ReportChanges will decrease total bundle size by 214 bytes (-0.0%) ⬇️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
This was referenced Jul 31, 2026
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
This was referenced Jul 31, 2026
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10185
Two faults made PostHog's AI observability structurally unable to answer the questions that matter most for a product where AI is the product. Both are fixed here; both are covered on each branch.
1. Every generation was its own trace
capturePostHogAiGenerationset$ai_trace_idto a freshrandomUUID()per call. Live project state: 13,428 AI events across 13,428 distinct trace IDs. A review fans out across RAG embeddings, both dual-review legs, N retries per model, and any self-consistency runs — each landing as an unrelated single-event trace, so what a review actually did, and where its time and money went, was unrecoverable.The fix was already in the file.
operationalPropertieshas calledcurrentOtelTraceIds()since #8296 and attached the real ids as plaintrace_id/span_idproperties — then the line below minted a UUID and PostHog grouped by that instead.withReviewPipelineSpanalready wraps a whole review, so the ambient trace id is exactly the grouping key wanted.randomUUID()stays as the fallback, not removed:AI_EMBED/AI_VISION/AI_ADVISORYrun outside any review span, so an orphan trace is a legitimate outcome there — but such an event must not claim a span id it doesn't have, so$ai_span_idis omitted rather than faked.2. AI spend was unattributable
Every AI event carried
distinct_id: "loopover-selfhost"and no$groups, so cost per repository was answerable only by SQL againstai_usage_events— never in PostHog. For scale:claude-sonnet-5is $489.87 across 2,282 calls in ~5 days.A
repogroup is now stamped on every AI event. It deliberately reads the already-processed repo value offoperationalPropertiesrather than the raw context: under the sharedLOOPOVER_CENTRAL_POSTHOG_KEYthat value is HMAC-anonymized, and is dropped entirely when the anon secret hasn't been injected — so the group inherits that fail-closed behavior for free, and a raw private repo name cannot reach the group index by this path.Out of scope
Content capture (
$ai_input/$ai_output_choices) is deliberately untouched — the metadata-only policy is documented inself-hosting-operations.mdx, and changing it is a privacy decision, not a wiring one.Known-adjacent bug, filed not fixed: #10186 — on the failure path
$ai_modelreports the@cf/placeholderresolveModeldiscards, so every failed call is labelled with a model that never ran. It needs an optionalresolvedModel()onSelfHostAithreaded through all four provider factories, which is a wider change than this PR.Verification
npm run typecheckcleantest/unit/selfhost-posthog.test.ts99 passed (94 pre-existing + 5 new, covering ambient-span vs none and repo-present vs dropped)