Skip to content

chore(AIC-3500): fail CI on an undeclared breaking change to the public API - #196

Merged
jeffdupont merged 11 commits into
mainfrom
jdupont/AIC-3500/api-breaking
Oct 22, 2026
Merged

jeffdupont merged 11 commits into
mainfrom
jdupont/AIC-3500/api-breaking

Conversation

@jeffdupont

@jeffdupont jeffdupont commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #195 (the API reports).

Why

The API reports make every public change visible, but they don't stop a breaking change from shipping in a minor or patch release. After 1.0 a breaking change has to be deliberate: declared, and released as a major version.

What

scripts/api_breaking.py uses griffe to compare the names in each public module's __all__ with another git ref. The new API compatibility workflow runs it on every PR, and re-runs it when the title or labels change.

  • On a PR (pr): compares with the PR's base. A breaking change fails unless one of these is true:
    • the PR title declares it with ! (feat!: ..., fix(client)!: ...), which makes release-please cut a major version (a minor one before 1.0.0);
    • an owner applied the api-compatible label because they judge the flagged change compatible.
  • On the release PR (release): for each package whose version changes in .release-please-manifest.json, compares the code with that package's last release tag (<component>-<version>). It fails on a breaking change unless the new version is a major one (a minor one before 1.0.0), or an owner applied api-compatible to the release PR. A package with no release tag yet is reported and skipped.
  • What counts as breaking:
    • from griffe: removed names, removed or moved parameters, new required parameters, changed defaults, changed attribute values;
    • added by the script: names dropped from __all__, and changed return types, which griffe doesn't flag.
  • Experimental modules are skipped, because an experimental feature may change in a minor release.

Tested

  • tests/test_api_breaking.py (23 tests) uses a throwaway git repo with a tagged release. It covers:

    • each breaking kind, and additive changes that pass;
    • experimental modules being skipped;
    • the title and label rules;
    • release versions before and after 1.0, including 0.9.0 → 1.0.0;
    • a package with no release tag;
    • api-compatible passing an undeclared breaking release, and another label not passing it.

    The fixture isolates git from the developer's global config.

  • Full suite: uv run pytest gives 2579 passed, 14 skipped. ruff check and ruff format --check are clean, and actionlint passes on the workflow.

  • Against real history:

    • pr against main: no breaking changes, so it passes.
    • pr against launchdarkly-ai-server-0.3.0: 23 breaking changes. That's 14 names that moved to experimental, plus capture_content's default changing from False to None in every provider factory, create_handler and ProviderHandler. It exits 1 with a plain title, and 0 with ! or the label.

The open release PR #166

Running release against #166 flags 6 provider packages:

  • claude-agents, langchain-agents, openai-agents: 0.2.5 → 0.2.6
  • claude-messages, langchain-messages, openai-messages: 0.2.4 → 0.2.5

Each changed capture_content's default from False to None (#158) since its last release, but is getting only a patch bump. client (0.3.0 → 0.4.0) passes.

If None means "use the default from configure()", callers see no difference. So the api-compatible label now also passes the release check (06050f1): an owner can judge a release PR compatible, the same way as any other PR. Verified on #166: it exits 1 without the label and 0 with it. The label now exists in this repo.

🤖 Generated with Claude Code


Note

Overview
Adds CI enforcement so undeclared public API breaks cannot ship in patch/minor releases.

A new API compatibility GitHub Actions workflow runs on every PR (including title/label edits). For normal PRs it runs scripts.api_breaking pr, comparing stable __all__ exports against the merge base via griffe (plus extra checks for __all__ drops and return-type changes). CI fails unless the title uses conventional-commit ! or an owner applies api-compatible. On release-please branches it runs release, comparing each bumped package to its last release tag and requiring a major (or pre-1.0 minor) bump when breaking changes slipped through.

api_report.resolve is refactored to take a ModulesCollection so the breaking checker can reuse the same resolution logic. tests/test_api_breaking.py covers detection rules, title/label gates, semver alignment, experimental skips, and removed packages.

Reviewed by Cursor Bugbot for commit a61cf9e. Bugbot is set up for automated code reviews on this repo. Configure here.

@jeffdupont
jeffdupont marked this pull request as ready for review October 20, 2026 22:24
@jeffdupont
jeffdupont requested a review from a team October 20, 2026 22:24

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 06050f1. Configure here.

Comment thread scripts/api_breaking.py
jeffdupont and others added 3 commits October 20, 2026 15:33
Each public module (every package root, and each module of the core
package's experimental subpackage) now has a committed report at
packages/<package>/api-report/<module>.txt, generated by
scripts/api_report.py with griffe. A report lists each name in __all__ with
its full signature, and a class's public members indented below it.

tests/test_api_report.py fails while any report is out of date, so `make
test` and CI catch a public name or signature change whose report wasn't
regenerated. It also checks the generator against a fixture package.
`make api-report` writes the reports, `make api-report-check` checks them,
and .github/CODEOWNERS gives them to team-ai-agents. griffelib is now an
explicit dev dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to the public API

scripts/api_breaking.py compares the names in each public module's __all__
with another git ref, using griffe:

- `pr`: compares with the PR's base. A breaking change passes only if the
  PR title declares it with `!` (so release-please cuts a major version, or
  a minor one before 1.0.0), or if an owner applied the api-compatible label.
- `release`: on the release-please PR, compares each package whose version
  changes with its last release tag, and fails on a breaking change unless
  the new version is a major one (a minor one before 1.0.0).

Experimental modules are skipped. griffe reports removed names, parameter
changes, changed defaults and changed attribute values; the script adds
names dropped from __all__ and changed return types, which griffe does not
flag. The API compatibility workflow runs it on every PR, and re-runs it when
the title or labels change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… too

An owner can now judge a flagged change compatible on the release PR, as on
any other PR. Without it, a compatible change flagged on the release PR (the
capture_content default on #166) could only pass by bumping the version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeffdupont
jeffdupont force-pushed the jdupont/AIC-3500/api-report branch from 13e38fb to 741f773 Compare October 20, 2026 22:33
@jeffdupont
jeffdupont force-pushed the jdupont/AIC-3500/api-breaking branch from 06050f1 to b7acfb0 Compare October 20, 2026 22:33
Base automatically changed from jdupont/AIC-3500/api-report to main October 22, 2026 22:56
# Conflicts:
#	scripts/api_report.py
@jeffdupont
jeffdupont merged commit 5ba5e05 into main Oct 22, 2026
8 checks passed
@jeffdupont
jeffdupont deleted the jdupont/AIC-3500/api-breaking branch October 22, 2026 23:00
jeffdupont added a commit that referenced this pull request Oct 23, 2026
Main's **Tests** job has failed since #202 merged (run 39959179499):
`tests/test_api_report.py::test_committed_reports_are_current` reports
that `launchdarkly_ai_server.experimental.evaluations.txt` is out of
date.

#202 removed `EvaluationsModule.api`. Its last commit was pushed before
the API report check (#195, #196) merged, so its own CI never ran the
check. This PR is the output of `make api-report`: one line removed
(`api: LDApiClient [property]`).

Checked locally: `make api-report-check` reports "15 API reports are
current", and `tests/test_api_report.py` passes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Regenerates the committed **evaluations** API report so it matches the
current experimental public surface after `EvaluationsModule.api` was
removed in #202.
> 
> The only report change is dropping the `api: LDApiClient` property
line from `EvaluationsModule`; this unblocks
`test_committed_reports_are_current` / `make api-report-check`.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
728bc50. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants