perf(telemetry): unblock per-tool quality analysis (GDPR annotations + diagnostic fields + InvocationGuard)#1650
Open
wenytang-ms wants to merge 3 commits into
Open
perf(telemetry): unblock per-tool quality analysis (GDPR annotations + diagnostic fields + InvocationGuard)#1650wenytang-ms wants to merge 3 commits into
wenytang-ms wants to merge 3 commits into
Conversation
added 2 commits
June 9, 2026 14:29
…ionGuard infra Telemetry contract changes to unblock per-tool quality analysis (driven by the 9-query deep-dive in debugagent/06-深度诊断-PR定向-2026-06-09.md): 1. GDPR annotation blocks (the root cause of PR #1644 fields being filtered) - Added /* __GDPR__ ... */ blocks above the three sanitizedSend call sites (recordToolInvocation, recordChatActivation, recordLaunchInternal) so the ddtelfiltered cluster keeps new Properties keys instead of stripping them. 2. Five new enums to replace stringly-typed fields: - OperatingSystem: win | mac | linux | other - ClassNameDetectionStrategy: mavenStandard | gradleStandard | vscodeSrc | workspaceRoot - ClassNameDetectionFailure: sourceDirMissing | fileNotFound | parseError | noPackageDeclaration - SentinelOutcome: recorded | silentReturn | cancelled | exception 3. ToolInvocationRecord extended with platform/version context: - os, javaMajorVersion, projectSystem, retryCount, previousOutcome These split the per-tool funnel by OS / Java major / project system so we can confirm the Linux 4.3 percent vs Windows 27.7 percent start-rate divergence. 4. LaunchInternalEvent union extended with four new sentinel variants and elapsedMs/thresholdMs: - debugSession.sentinel: emitted on EVERY invoke so even infinite hangs leave a trail - debugSession.silentReturn: invoke returned with no outcome event (the 32.8 percent gap) - debugSession.cancelled: user cancelled mid-flight - debugSession.exception: handler threw an exception, classified - classNameDetection.failed: replaces collapsed boolean with strategy + failureReason - debugSessionStarted/Timeout now carry elapsedMs + thresholdMs for histograms 5. InvocationGuard infrastructure (beginDebugSessionInvocation()): - Returns markOutcomeRecorded / markException / close methods - Emits sentinel at begin; close auto-emits silentReturn if no outcome recorded - Lets us measure the 32.8 percent silent-loss bucket with closed-loop attribution 6. SessionInvocationTracker (nextAttempt / completeAttempt): - Per-window, per-tool in-memory retry counter - Stores ONLY enum (previousOutcome) and integer (count) — no user data - Quantifies the LM auto-retry-pays-off pattern (17.7 percent at 1 -> 64.9 percent at 6-10) Compiles clean (tsc --noEmit) and passes tslint.
…meouts
Consumes the new telemetry contracts from the previous commit and threads them
through all 10 LMT invoke handlers. Also tunes the two session-wait thresholds
to cover the Started P95 latency observed in 30d telemetry.
Timeout tuning (driven by 30d Started latency: P50=12s / P90=67s / P95=100s):
SESSION_WAIT_TIMEOUT: 45000 -> 120000 (eventBased path, waitForSession=true)
SMART_POLLING_MAX_WAIT: 15000 -> 90000 (default polling path)
Previous values clipped ~10 percent of legitimate slow starts as timeouts and
biased the two strategies against each other. Aligning at 120s/90s also lets
us cleanly compare cross-strategy success rates.
Per-tool context fields (added to all 10 recordToolInvocation call sites):
os / javaMajorVersion: from getOs() / getJavaMajorVersion()
getJavaMajorVersion probes the redhat.java extension API (cached),
falls back to JAVA_VERSION env var, then 'unknown'.
classifyJavaMajorVersion handles legacy '1.8.0_xxx' -> '8' and modern '21.0.1' -> '21'.
retryCount / previousOutcome: from nextAttempt() / completeAttempt()
Per-window, per-tool counter. Used to test the LM auto-retry-pays-off pattern.
debug_java_application handler:
Wraps the entire invoke in a beginDebugSessionInvocation() guard so even
infinite hangs leave a sentinel + silentReturn trail. This is the closed-loop
half of the 32.8 percent silent-loss bucket detected by D2 in the deep-dive.
classNameDetection rewrite:
findFullyQualifiedClassName() now returns { className, failureReason, strategy }
instead of string|null. The failure branch emits the new classNameDetection.failed
event with structured strategy + failureReason so we can distinguish
sourceDirMissing / fileNotFound / parseError / noPackageDeclaration — the
previous boolean detected:false collapsed all four causes.
Timeout / Started events:
debugSessionStarted.eventBased + debugSessionTimeout.eventBased +
debugSessionTimeout.smartPolling now carry elapsedMs + thresholdMs so we can
histogram the Started/Timeout latency distribution and verify the threshold
bump moves the right mass.
Compiles clean (tsc --noEmit) and passes tslint.
wenytang-ms
requested review from
chagong,
jdneo and
testforstephen
as code owners
wenytang-ms
changed the title
This was referenced Jun 9, 2026
There was a problem hiding this comment.
Pull request overview
This PR extends the Language Model Tool (LMT) telemetry contract and wiring to unblock per-tool quality analysis in downstream pipelines that enforce /* __GDPR__ */ allowlists. It also adds richer diagnostic context (OS/Java version/retry attribution) and introduces a closed-loop “InvocationGuard” concept intended to attribute silent debug_java_application outcomes.
Changes:
- Add GDPR annotation blocks and extend LMT telemetry schemas with new diagnostic fields and enums in
src/lmToolTelemetry.ts. - Wire new diagnostic fields + per-session retry attribution through all LMT invoke handlers; tune launch timeouts in
src/languageModelTool.ts. - Add new structured class-name detection result/failure reasoning and additional launch internal events.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| src/lmToolTelemetry.ts | Adds GDPR annotations, new enums/diagnostic fields, launch internal event variants, and InvocationGuard/retry-tracker utilities. |
| src/languageModelTool.ts | Threads new telemetry fields into tool invocations, tunes timeouts, and refactors class-name detection to return structured results. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address review feedback on PR #1650: 1. InvocationGuard was effectively disabled — guard.markOutcomeRecorded() was called unconditionally in finally, so guard.close() could never emit debugSession.silentReturn / cancelled / exception. Moved the mark calls inline next to the four session-terminal recordLaunchInternal sites (debugSessionStarted.eventBased, debugSessionTimeout.eventBased, debugSessionDetected, debugSessionTimeout.smartPolling). Threaded guard parameter into debugJavaApplication. The catch path now lets close() emit debugSession.exception via markException as designed. 2. targetInfo formatting at the call site stringified the new structured ClassNameDetectionResult as [object Object]. Renamed local to 'detection' and explicitly read detection.className. The if-truthy branch was also always taken, suppressing the no-package warning. 3. The 'found file but no package' branch returned { className: simpleClassName } which made callers treat it as a successful detection, hiding the failure event. Now returns { className: null, failureReason: 'noPackageDeclaration' } so classNameDetection.failed fires. Behavior is preserved because the call site already falls back to input.target on null. 4. ClassNameDetectionResult is now a discriminated union (success has no failureReason; failure requires it). Eliminates the 'unused on success' sentinel and lets TS narrow at use sites. Required strict-null narrowing (className !== null) at the two call sites. tsc clean; tslint clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A 30-day deep-dive against
ddtelfilteredconfirmed that none of the diagnostic fields added in #1644 are reaching the cluster:outcome(success/failure/timeout/cancelled)errorcategory(timeout / noActiveSession / …)errortypefailurereasonerrorcode ≠ "0"Root cause:
ddtelfilteredstrips anyPropertieskey that is not declared inside a/* __GDPR__ ... */annotation block. #1644 added the fields to the TS layer but never added the annotations, so the schema-filter dropped them at ingest.This also leaves several known quality issues invisible:
debug_java_application.invokerows with no matching outcome event within 60sSESSION_WAIT_TIMEOUTwas 45s, so ~10% of legitimate slow starts were being recorded as timeoutsWhat
src/lmToolTelemetry.ts(commit 1 — telemetry contract)sanitizedSendcall sites (recordToolInvocation,recordChatActivation,recordLaunchInternal) — fixes the root cause of fields being filtered.OperatingSystem,ClassNameDetectionStrategy,ClassNameDetectionFailure,SentinelOutcome.ToolInvocationRecordextended withos/javaMajorVersion/projectSystem/retryCount/previousOutcome.LaunchInternalEventunion extended with 4 new sentinel variants:debugSession.sentinel— emitted on every invoke so even infinite hangs leave a traildebugSession.silentReturn— invoke returned with no outcome (the 32.8% gap)debugSession.cancelled/debugSession.exception— closed-loop attributionclassNameDetection.failed— replaces the collapsed booleandebugSessionStarted/Timeoutnow carryelapsedMs+thresholdMsfor histogramsInvocationGuard(beginDebugSessionInvocation()) —markOutcomeRecorded/markException/closefor closed-loop sentinel tracking.SessionInvocationTracker(nextAttempt()/completeAttempt()) — per-window, per-tool in-memory retry counter. Stores only enum + integer, no user data.src/languageModelTool.ts(commit 2 — wiring)SESSION_WAIT_TIMEOUT:45000 → 120000SMART_POLLING_MAX_WAIT:15000 → 90000os/javaMajorVersion/retryCount/previousOutcomeintorecordToolInvocation, and callcompleteAttemptafter.debug_java_applicationis additionally wrapped inbeginDebugSessionInvocation()so the 32.8% silent-loss bucket gets closed-loop attribution.findFullyQualifiedClassName()rewritten to return{ className, failureReason, strategy }. Failure branch emitsclassNameDetection.failedwith the structured reason.elapsedMs+thresholdMs.Privacy
All new fields are either:
os,failureReason,strategy,previousOutcome, …), orretryCount,elapsedMs,thresholdMs, …), orNo free-form strings, no user paths, no class names, no expression text. Each field has a
classification+purpose+ownerdeclared inside the matching__GDPR__block.getJavaMajorVersion()reads only the major number (e.g."21","8") from theredhat.javaextension API orJAVA_VERSIONenv var — never the full vendor string.Validation plan (after ship)
debugagent/lmt-quality-queries.mdhas the 10 queries that become runnable once data flows:debugSession.silentReturnlandsTest
npx tsc --noEmit— cleannpm run tslint— cleannpm test) require VS Code download; not run in this CI passRelated
debugagent/06-深度诊断-PR定向-2026-06-09.md,debugagent/05-PR1643-效果验证-2026-06-09.mdCo-authored-by: Copilot 223556219+Copilot@users.noreply.github.com