Skip to content

fix: exclude unmeasurable LangSmith runs from outliers - #554

Open
Gautam Sharma (GautamSharma99) wants to merge 1 commit into
langchain-ai:mainfrom
GautamSharma99:fix/langsmith-latency-outliers
Open

fix: exclude unmeasurable LangSmith runs from outliers#554
Gautam Sharma (GautamSharma99) wants to merge 1 commit into
langchain-ai:mainfrom
GautamSharma99:fix/langsmith-latency-outliers

Conversation

@GautamSharma99

Copy link
Copy Markdown
Contributor

Summary

Fixes #544.

LangSmith sampling used latencyMs(run) ?? -1 while calculating and filling the latency-outlier slice. When there were fewer measurable runs than the outlier budget, incomplete roots were labeled as slow despite having no latency evidence.

Changes

  • Build the outlier candidate pool from non-error roots with a valid, non-negative measured latency only.
  • Calculate the proportional quarter cap from that measurable pool, and select only those candidates.
  • Continue using the original non-error order to backfill remaining capacity as baseline, so incomplete runs remain available as context without unsupported outlier labels.
  • Add regressions for all-missing latency and mixed measurable/unmeasurable pools.
  • Add a patch changeset.

Behavior

A run with no usable start/end timestamps can now be selected only as baseline (or omitted by the overall cap), never as a latency outlier. Every root in the outlier bucket has a defined measured latency.

Validation

  • pnpm run lint:check
  • pnpm run typecheck
  • pnpm exec prettier --check (changed files)
  • pnpm test — 67 files, 793 tests passed

@changeset-bot

changeset-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: d5a1d3c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
openwiki Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: LangSmith runs without measurable latency are labeled as latency outliers

1 participant