Persist agent eval metrics in Braintrust - #334
Merged
Merged
Conversation
Expose contained prompt paths and neutral final answers in the normalized report so downstream persistence can use the validated artifact boundary.
Validate imported suite evidence and produce deterministic allowlisted rows with prompt identity, usage metrics, and tool telemetry without invoking Braintrust.
Add the pinned SDK boundary and persist deterministic top-level eval spans without fabricating quality scores or tracing agent execution.
Provide strict local and CI arguments, credential-free validation, safe result artifacts, and shared Bun and bt eval entrypoints.
Export normalized suite rows after concise reporting, append the experiment link, and surface scenario, report, or persistence failures in the final job status.
Record the validated export contract, local and CI usage, PoC readback evidence, and the internal Braintrust inspection skill.
Keep the exporter contracts unchanged while making fixtures and workflow expression assertions type-safe and lint-clean.
Align the documented CI timing with the observed run and clarify that the accepted artifacts were exported locally through bt eval.
Reject empty suite inputs, preserve asserted metrics identity, and document current CI secret visibility and observed run timing.
Keep implementation, operations, and internal skill guidance aligned with per-suite workload-cell validation.
jlitola
added
enhancement
Use Braintrust-native token, duration, tool, and cost fields while preserving GitHits-specific telemetry. Carry recorded span timestamps and document the pending native-root verification path.
Document the verified 23-row native-root experiment and distinguish accurate root metrics from the unresolved structural tool-span comparison path.
jlitola
added
agent-eval
jlitola
added
agent-eval
Define stable channel-aware experiment names and explicit latest-main base resolution for PR, local, and recurring main comparisons.
Add stable channel-aware experiment identities, explicit main baseline discovery, and safe base experiment readback while preserving the existing row and tool-span contract.
Route pull-request and main event context through environment-bound exporter arguments, guard non-main manual dispatches, and report experiment/base linkage in the workflow summary.
Document stable experiment identity, explicit main-baseline linkage, safe local operations, and the remaining live readback proof. Keep historical experiments and current quality-score boundaries explicit.
Make base readback mandatory through the SDK and publisher boundaries, preserve actual null results, and version the safe CLI result after adding base linkage.
Build the shell-array expansion assertion without embedding a template-like string, preserving the quoted argument contract while satisfying the linter.
Read the pinned SDK Experiment.name after publishing so reused local experiment names report the server-created identity. Keep validate-only names deterministic and document same-SHA main baseline eligibility.
jlitola
marked this pull request as ready for review
September 1, 2026 06:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
bun test— 3,745 passed, 0 failedbun run typecheckbun run format:checkbun run lintbun run buildactionlint .github/workflows/agent-evals.ymlbun run plugins:checkPending operational validation
BRAINTRUST_API_KEYis not currently visible in the repository or selected organization Actions-secret inventory. Theagent-evallabel is intentionally not applied yet: a labeled run would spend the eval budget and then fail persistence. After the secret is available, apply the label and verify one 23-row CI experiment/readback.