Skip to content

AGENTS.md

Latest commit

 

History

History
198 lines (140 loc) · 11.5 KB

AGENTS.md

File metadata and controls

198 lines (140 loc) · 11.5 KB
Raw

AGENTS.md

this file is the agent-facing layer and covers the day-to-day dev loop. you usually won't need more, but if you do: README.md has end-user usage (calling the sdk) and CONTRIBUTING.md has human setup (jdk, ide).

What this is

the braintrust java sdk: tools for writing evals and tracing ai apps, built on opentelemetry. we publish three artifacts (all under the dev.braintrust group):

  • braintrust-sdk: programmatic sdk code to write evals and instrument popular ai frameworks
  • braintrust-java-agent: autoinstrumentation (no-code instrumentation of popular ai frameworks)
  • braintrust-otel-extension: more niche. applies to autoinstrumentation users who are also running the opentelemetry java agent

Architecture

  • braintrust-sdk: core sdk code. contains evals and instrumentation
  • braintrust-sdk/instrumentation/*: instrumentation modules, one per supported library version (e.g. openai_2_15_0)
  • braintrust-api: autogenerated client against the braintrust openapi yaml
  • braintrust-java-agent: autoinstrumentation. attaches as a -javaagent and rewrites bytecode to apply the instrumentation modules automatically
  • braintrust-otel-extension: fat jar packaging the sdk as an extension of the upstream otel java agent
  • test-harness: shared testing framework and utils
  • btx: spec runner that executes shared sdk specs (see below)

the agent and the manual wrappers (BraintrustOpenAI.wrapOpenAI etc.) share the same instrumentation code under braintrust-sdk/instrumentation/*: the agent discovers modules via ServiceLoader and applies them with bytecode transforms, while manual users call the wrapper directly. either way, spans are exported to braintrust over otel.

See also: spec repo

there's a cross-sdk spec repo that describes general sdk behavior shared across languages. we keep a shallow checkout of it to run some automated spec tests (see the btx section).

repo: https://github.com/braintrustdata/braintrust-spec

the ref we check out is pinned in gradle.properties as braintrustSpecRef

the fetchSpec gradle task downloads the pinned ref into btx/build/spec/ (it runs automatically before btx:test). the spec files themselves live under btx/build/spec/test/llm_span/ — read those locally if you want to see what behavior is being asserted. to run against a local checkout instead, set BTX_SPEC_ROOT=/path/to/braintrust-spec/test/llm_span.

Build, test, lint

  • formatter: ./gradlew spotlessApply
  • build: ./gradlew compileJava compileTestJava
  • test: ./gradlew test (scope to a module while iterating, e.g. ./gradlew :braintrust-sdk:test)
  • full ci gate: ./gradlew check

use check as a one-time final sanity check before opening a PR, not as part of the normal dev loop. it runs every muzzle check (slow, and downloads a lot of artifacts). while iterating, prefer the scoped compileTestJava / test commands above.

Testing

tests are mostly written with junit5. there are a few exceptions in cases where we have to manage the test env carefully (e.g. autoinstrumentation global java logging test requires a custom env that junit can't guarantee).

Test harness

TestHarness.java provides a consistent environment for unit testing. It allows us to:

  • set up a fresh otel state
  • inspect exported otel traces
  • fully simulate the real braintrust backend
  • fully simulate various AI vendors (openai, anthropic, etc)

External calls are simulated with the VCR.java test utility (see section below).

VCR (recorded HTTP)

VCR allows us to:

  • make real external calls: VCR_MODE=off ./gradlew test
  • make real external calls and record the results to "cassette" files: VCR_MODE=record ./gradlew test
  • replay previously recorded calls instead of making real external requests: VCR_MODE=replay ./gradlew test

The default VCR mode is replay. this allows us to fully test the sdk with zero external calls

in replay mode, tests must pass quickly (~30 seconds)

a full re-record of cassettes should be done every once in a while (1-2 months) to ensure the sdk still works with external AI vendors.

to re-record all cassettes: ./scripts/re-record-cassettes.sh. this script will take a long time to run. it deliberately turns off concurrency to reduce the chances of being rate-limited by llm providers. If a full re-recording is botched, it may require waiting over an hour before trying again because some tests exercise token caches and need that amount of time before the cache is in a state where the test can be retried (note: off mode should always pass because it uses nonces for cache tests).

BTX spec test runner

The btx project is a testing framework that executes cross-language sdk specs: https://github.com/braintrustdata/braintrust-spec/tree/main/test

these tests verify that llm spans:

  • are reported to braintrust
  • set the proper span names and attributes

like other tests, btx also supports vcr.

VCR_MODE=off ./gradlew :btx:test <-- makes real calls to AI vendors, sends spans to real braintrust, fetches them out of the backend for assertions

Normal record/replay also applies (and replay is also the default).

When to add a spec test vs a unit test

if a behavior is specific to the SDK, use a junit test. If the behavior under test is a cross-language llm semconv, update the spec instead.

for example, if we had a springai instrumentation module:

the junit test may assert that an llm span is created, but not assert on its specific name or attributes:

  ... other assertions
  String json = span.getAttributes().get(AttributeKey.stringKey("braintrust.metrics"));
  assertNotNull(json, "braintrust.metrics should be set");
  ... other assertions

and the completions.yaml spec test would assert on specific metrics attribute names:

name: completions
type: llm_span_test
provider: openai
endpoint: /v1/chat/completions
requests:
  - model: gpt-4o-mini
    temperature: 0.0
    messages:
      - role: system
        content: you are a helpful assistant
      - role: user
        content: What is the capital of France?
expected_brainstore_spans:
  - metrics:
      tokens: !fn is_non_negative_number
      prompt_tokens: !fn is_non_negative_number
      completion_tokens: !fn is_non_negative_number
      prompt_cached_tokens: !fn is_non_negative_number
  ... other assertions

Adding instrumentation

The general steps of adding or updating instrumentation:

  • add a new subproject module braintrust-sdk/instrumentation/*
    • when possible, favor copy-pasting a similar module to start with (e.g. prior version)
  • check in with a human:
    • should we be making a new module or updating an older one to be forwards compatible?
    • is our instrumentation api/strategy correct? are we hooking the right place, etc
    • feel free to make suggestions, but check in rather than just making a call (this is an important decision and it's more of an art than a science)
  • get the hooks working
  • update module's build.gradle:
    • compile against the minimum required version
    • add a muzzle directive: versions = "[${myModuleVersion},)" <-- this will test up to the latest release on maven
    • find the max supported version (or leave unbound if we support everything). verify with human that this is acceptable
  • add a basic unit test that creates basic span(s)
  • run with VCR_MODE=off and set up a basic unit test that asserts:
    • basic spans are created with the proper parenting
    • attributes are tagged. don't be overly specific about this though (assertNotNull(someAttribute) is usually better than assertEquals(12, someAttribute)). ask human if you're unsure
  • record cassettes for the specific test. e.g. VCR_MODE=record ./gradlew :braintrust-sdk:instrumentation:mymodule:test --tests 'dev.braintrust.instrumentation.mymodule.v1_0_0.BraintrustMyModuleTest'
  • finally, verify the module's checks pass: VCR_MODE=replay ./gradlew :braintrust-sdk:instrumentation:mymodule:check

Dev workflow

  • when testing, use VCR_MODE=off. get all tests to pass. Then finally, record cassettes for the specific test being changed
# when developing
VCR_MODE=off ./gradlew :braintrust-sdk:test --tests 'dev.braintrust.devserver.DevserverTest.testExperimentEval'
# when test is solid:
VCR_MODE=record ./gradlew :braintrust-sdk:test --tests 'dev.braintrust.devserver.DevserverTest.testExperimentEval'
  • when running btx, use the spec filter to target what is specifically under development: VCR_MODE=off ./gradlew :btx:test -Pbtx.spec.filter=openai/prompt_cach --rerun
  • don't reformat the whole repo, but do run ./gradlew spotlessApply on files you changed before committing. the pre-commit hook and ./gradlew check both run spotlessCheck, which fails on unformatted code.

Gotchas

  • don't hand-edit cassettes. they're content-hashed and guarded against committed secrets. a failing VCR test means the recorded interaction changed — re-record it (see the VCR section), don't patch the json.
  • braintrust-api is generated code. don't edit sources under it by hand; it's regenerated from the braintrust openapi spec pinned as braintrustOpenApiRef in gradle.properties.
  • there are no version constants to bump. the sdk version is derived from git tags at build time (generateVersion() in build.gradle) and written into braintrust.properties. "bump the version" is not a source change.

Releasing

Note: Releases require a human and should not be automatically done by an agent.

Releases are driven end-to-end from a single GitHub Actions workflow.

To cut a release:

  1. Make sure everything you want included is merged to main and CI is green.
  2. Go to Actions → Release → Run workflow.
  3. Enter:
    • version: the release version as vX.Y.Z (semver, no -SNAPSHOT).
    • sha: the full 40-character commit SHA on main you want to release. Copy it from the commit page on GitHub using "Copy full SHA". A branch name is intentionally not accepted — pinning to a SHA prevents commits that land on main during the approval gate from sneaking into the release.
  4. The job runs in the protected release GitHub Environment and will pause for required-reviewer approval before doing anything. Approve from the workflow run page (or the repo's Deployments tab).
  5. Once approved, the Release workflow will, in one job:
    • Validate the version and the SHA, and verify the SHA is reachable from origin/main.
    • Check out the pinned SHA and run ./gradlew check.
    • Create and push the annotated tag vX.Y.Z pointing at the SHA (using the default GITHUB_TOKEN — no separate bot identity is needed since the publish steps are in the same workflow).
    • Check out the tag, re-run ./gradlew check, and build release artifacts.
    • Create the GitHub Release with the SDK, agent, and OTel extension jars attached.
    • Publish to Maven Central via Sonatype, signed with the project GPG key.
    • Poll Maven Central until the new version is visible (this can take many hours).

The Sonatype and GPG signing secrets (SONATYPE_USERNAME, SONATYPE_PASSWORD, GPG_SIGNING_KEY, GPG_SIGNING_PASSWORD) are scoped to the release environment.

The SDK version is computed from git tags at build time (see generateVersion() in build.gradle) and embedded into braintrust.properties, so there are no version constants to bump in source.

Re-publishing a failed release

If the workflow fails partway through, re-run Release with the same version. The workflow detects that the tag already exists, skips tag creation, and resumes from the build/publish steps against the existing tag. GitHub Release asset uploads use --clobber so partial uploads from a prior run are replaced.