feat(observability): opt-in capture of AI prompts and completions - #10220
Merged
Conversation
PostHog's LLM Analytics conversation view, sentiment, evaluations and LLM-judge surfaces all read $ai_input/$ai_output_choices, which were deliberately never populated -- so every trace in the project showed a dash for input, output and sentiment, and the whole content-dependent half of the product was unavailable. Populate them behind LOOPOVER_POSTHOG_AI_CONTENT, default OFF. Upgrading an ORB must never start shipping private PR diffs and model completions to a vendor as a side effect of a version bump, so an operator who has not opted in is byte-identical to before. The gate lives in the capture function rather than at each call site: ai.ts always passes the content and posthog.ts alone decides, so a future call site cannot forget it. Captured content passes through the same before_send scrub as every other field, not a second weaker path. Each message is capped at 10,000 chars (LOOPOVER_POSTHOG_AI_CONTENT_MAX_CHARS) and marked when truncated -- a review prompt can exceed 300k chars and would otherwise blow past PostHog's payload limits and be dropped wholesale; a non-numeric, zero or negative override falls back to the default rather than disabling the cap. An image block is dropped rather than base64-encoded into the event, reusing the same projection the subscription CLIs already apply when building their stdin prompt. Embedding requests report their raw texts under a synthetic user role so one property shape covers both request kinds. The failure path captures the prompt and no completion -- a claude_stalled_no_output or a context-length rejection is only diagnosable against the input that produced it. The self-hosting doc promised metadata-only unconditionally and would have become false; it now documents the opt-in, the truncation, and what enabling it actually sends. The absence-assertion test now covers BOTH states rather than being deleted. Closes #10218
Contributor
|
Important 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏳ LoopOver is waiting…LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 0eb9721 | Commit Preview URL Branch Preview URL |
Jul 31 2026, 12:54 PM |
Bundle ReportChanges will increase total bundle size by 5.27kB (0.07%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10220 +/- ##
==========================================
- Coverage 92.22% 91.35% -0.87%
==========================================
Files 937 937
Lines 114334 114355 +21
Branches 27645 27651 +6
==========================================
- Hits 105440 104467 -973
- Misses 7588 8777 +1189
+ Partials 1306 1111 -195
Flags with carried forward coverage won't be shown. Click here to find out more.
|
![loopover-orb[bot]](https://avatars.githubusercontent.com/in/4139483?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PostHog's LLM Analytics Traces view shows a dash for Input message, Output message and Sentiment on every row in this project, because
$ai_inputand$ai_output_choiceswere deliberately never populated. That default was defensible, but it also made the entire content-dependent half of the product unreachable: the conversation view, sentiment, evaluations, LLM-as-judge and online evals all read the captured prompt and completion.This populates them behind
LOOPOVER_POSTHOG_AI_CONTENT, default OFF. An operator who has not opted in gets byte-identical behaviour to before — upgrading an ORB must never start shipping private PR diffs and model completions to a vendor as a side effect of a version bump.Design decisions worth reviewing
ai.tsalways passes the content andposthog.tsalone decides whether to emit it. A future call site therefore cannot forget the check — the failure mode of a per-call-site gate is exactly the silent leak this feature must not have.before_send→scrubPostHogEvent→scrubRecordapplies the same credential/JWT/query-string scrubbing it applies to every other field. A test asserts a leaked PAT in a prompt comes out[redacted].…[truncated], so a clipped prompt is never mistaken for the whole one. A review prompt carries up to 120k chars of diff plus a 240k-char context budget — uncapped, these events would exceed PostHog's payload limits and be dropped wholesale, which is a worse outcome than truncation. A non-numeric, zero, negative or fractional override falls back to the default rather than disabling the cap.contentText, the same projection the subscription CLIs already apply when building their stdin prompt — so what is captured is what the provider was actually given, and no base64 blob lands in a telemetry event.userrole; it has no completion, so$ai_output_choicesis omitted rather than emitted empty.claude_stalled_no_outputor a context-length rejection is only diagnosable against the input that produced it.Closes #10218
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateAlso run and green, because this change touches their inputs specifically:
npm run selfhost:env-reference— regenerated and committed; two newenv.*reads undersrc/selfhost/**land inapps/loopover-ui/src/lib/selfhost-env-reference.ts.selfhost:env-reference:checkpasses.npm run dead-exports:check— green. The two resolver helpers are module-local on purpose: exporting them with no consumer outside the file is precisely what putdead-exports:checkred on a cleanmainin fix(ci): unbreak main — drop the unused MCP_SERVER_NAME export #10195, and they are covered throughinitPostHoginstead.prettier --checkon the changed MDX and the generated reference — clean.If any required check was skipped, explain why:
Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.This is the change on this page that most deserves scrutiny, so it is stated plainly rather than buried: with the flag on, prompts contain the PR diff and the assembled repo context, and on a private repository that is private source code leaving the operator's infrastructure. The redaction pass removes credential shapes, not source code — the docs say so in those words rather than implying the scrub makes it safe. The default is off, the flag is a deliberate operator action, and nothing about the emitted event changes until it is set.
UI Evidence
Not applicable — no visible UI or frontend change. The docs change is prose plus a
FeatureRowentry on the existing self-hosting operations page, whose accuracy is pinned by a drift test rather than a screenshot.Notes
Tests cover both states throughout rather than only the new one: content present and absent, each field omitted independently, at-cap and over-cap, the flag off with content supplied, images dropped, embeddings, and the failure path. The existing "never carries prompt/response content" assertion was flipped to cover both states rather than deleted — it now passes content in and asserts it does not appear, which is a strictly stronger check than the original.
Token usage is a separate root cause and is not addressed here — see #10211.