ai(observability): review quality never reaches PostHog — no $ai_metric is ever emitted

[JSONbored]

Part of epic #8286 (Phase 3 — AI observability).

Problem

Rich review-quality signal reaches SQL, Grafana and the maintainer recap, but never PostHog. So cost, model and latency can be read there, but not whether the review was any good — the two halves of the question live in different systems and cannot be joined.

PostHog has a purpose-built event for this: $ai_metric, which attaches a named measurement to a trace id. With the trace grouping from #10187 and the trace envelope from #10221, a quality metric emitted against the review's own $ai_trace_id joins directly to the generations, spend and model that produced it.

Property contract

Taken from the SDK rather than guessed — @posthog/core's captureTraceMetric:

captureTraceMetric(traceId, metricName, metricValue) {
    this.capture('$ai_metric', {
        $ai_metric_name: metricName,
        $ai_metric_value: String(metricValue),
        $ai_trace_id: String(traceId)
    });
}

Note the value is stringified by the SDK. A hand-built event should match, so an SDK-built and a hand-built metric are indistinguishable downstream.

Sources worth emitting

• reviewer_vote — each reviewer's stance (src/queue/ai-review-orchestration.ts)
• inter-run agreement from scoreJudgmentAgreement (src/review/judgment-agreement.ts)
• self-consistency runs (src/review/self-consistency.ts)
• routing shadow / provider track records
• gate + rule precision (src/services/gate-precision.ts, src/review/public-rule-precision.ts)
• eval score records (src/review/eval-score-records.ts)

Deliverables

• A capture helper emitting $ai_metric to the SDK's exact contract, defaulting the trace id to the ambient OTel trace so metrics land on the review that produced them.
• No event when there is no ambient trace — an orphan quality score joins to nothing and only inflates counts.
• reviewer_vote and judgment agreement wired first; the remaining four sources follow.
• An uncorroborated review reports no agreement rather than the UNCORROBORATED_AGREEMENT floor — that constant is a deliberate placeholder, not a measurement, and publishing it would drop a fabricated 0.5 into an average alongside real scores.

Not in scope

$ai_feedback and $ai_evaluation are deliberately excluded for now:

• $ai_feedback is PostHog's user-feedback event ($ai_feedback_text), and its documented path is the Surveys integration keyed on $ai_trace_id. There is no human feedback signal at these call sites to put in it — inventing one from a machine vote would misrepresent what the event means.
• $ai_evaluation has no property contract in the SDK or the docs beyond a one-line taxonomy description, so any shape chosen now would be a guess. Worth revisiting once PostHog documents it, or once online evaluations are enabled (which content capture, ai(observability): PostHog's conversation, sentiment and evaluation views are empty — $ai_input/$ai_output_choices are never populated #10218, unblocks).

[linear-code]

JSO-7137