[tonytrinh3]

Summary

Plain version: We publish packages to npm. A customer installs them and runs them on AWS Lambda.
This change adds a test that does the same thing right after each release.
The test installs the published packages, runs them in a real AWS Lambda, and then deletes that Lambda.
If the test fails, the packages are broken for customers, and we know in minutes.

What it means for the code (junior eng): A canary is a small test against production artifacts.
This canary closes AIC-3370.
The suite from PR1 packs the workspace with npm pack, so it tests the tarball that we plan to ship.
This canary tests the artifact that npm actually serves, in the runtime that customers use.

Closes AIC-3417.
AIC-3416 is merged.
The base branch is main now. I merged main into this branch and resolved the conflicts.

• The AWS account and the OIDC role are not confirmed. I found no approved AWS account for this repository. I found no GitHub OIDC role for this repository. The id-token permission here serves npm trusted publishing only. The job reads vars.AWS_LAMBDA_CANARY_ROLE_ARN. GitHub skips the job while that variable is empty. I added no credentials. The README lists the trust policy and the permissions.
• integration-tests/lambda-commonjs/ holds the fixture. It uses Serverless 3, Webpack 5, and webpack-node-externals 3. The output is commonjs2. The runtime is nodejs22.x. The fixture sets no externals allowlist, so Node resolves the LaunchDarkly packages at runtime. That path is the AIC-3370 failure path.
• canary.mjs has three commands. wait polls npm. build installs the exact versions and packages the app. assert checks the payload from the Lambda. The assert step requires the exact type of each export.
• release-please.yml gets a release-canary job. A new canary_versions dispatch input runs the canary on demand. The cleanup step runs under if: always(). The step fails the job when it cannot delete the stack or the log group.
• The canary loads packages only. It uses no SDK key. It evaluates no flag. It calls no model.
• The canary pins @launchdarkly/ai-server too. @launchdarkly/ai-node depends on the core package with *, so npm would otherwise pick any visible core version. A core-only release now also runs the canary.

Which AIC-3370 asks this PR answers

Plain version: This PR answers ask 3. It runs the packages that npm serves inside a real Lambda after each release.
It does not answer the other four asks. Other PRs in the stack answer those.

What it means for the code (junior eng): This PR completes ask 3.
It installs exact published npm versions. It packages a CommonJS Lambda with Webpack 5 and webpack-node-externals 3.
It deploys that Lambda on Node 22. It invokes the Lambda after each release.

Ask 3 has two halves. #76 covers the packed-tarball half. This PR covers the published-artifact half.

The real AWS path is gated on vars.AWS_LAMBDA_CANARY_ROLE_ARN.
No approved AWS account exists yet. No GitHub OIDC role exists yet.
So the deploy path, the invoke path, and the cleanup path are unverified against real AWS.

This PR does not answer asks 1, 2, 4, or 5. #77 answers ask 1. #76 answers asks 2 and 4. #78 answers ask 5.

Implementation details

Why this test exists

The suite in integration-tests/commonjs-webpack packs the workspace with npm pack.
That suite cannot catch a publish-time regression.
Examples are a wrong files list, a stale dist folder, or a bad exports map in the published tag.
That suite also does not run on Lambda.

The difference is real today. I invoked the generated bundle against the published 0.2.0 artifacts.

$ node -e "require('./handler.js').handler()..."
ERR_PACKAGE_PATH_NOT_EXPORTED No "exports" main defined in node_modules/@launchdarkly/ai-node/package.json

That error is the AIC-3370 bug. The canary passes after we publish the dual output from PR2.

The driver

canary.mjs holds all of the logic. It takes one exact version for each package.
You pass --ai-node=0.3.0 and the other flags.
You can also set LD_AI_CANARY_VERSIONS='ai-node=0.3.0,openai-messages=0.3.0,ai-otel=0.2.0'.

Command	Behavior
wait	Calls npm view <pkg>@<version> version up to 30 times, 10 seconds apart. A new publish is not readable at once.
build	Copies fixture/ into .canary/app. Runs npm install --save-exact. Checks the tree with npm ls. Writes versions.json. Runs serverless package when you pass --package.
assert	Reads the invoke payload and checks it.

The driver checks exactness two times.
It first matches the input against /^\d+\.\d+\.\d+(?:-[0-9A-Za-z-.]+)?$/.
It then compares the installed tree from npm ls --depth=0 --json against the request.

The build writes versions.json into the app, and the handler returns that file.
So the assert step checks the versions that the Lambda loaded.
It does not trust the versions that CI asked for.

The handler returns this shape. The assert step compares each capability against EXPECTED_CAPABILITIES.

{ "ok": true, "runtime": "v22.x", "versions": { }, "capabilities": { "config": "function", "initClient": "function", "shutdown": "function", "openaiMessages": "function", "aiOtel": "object" } }

The workflow job

release-canary:
  needs: [release-please, release-client, release-ai-node, release-ai-otel, release-openai-messages]
  if: always() && no failed or cancelled need && vars.AWS_LAMBDA_CANARY_ROLE_ARN != ''
      && (dispatch needs canary_versions, push needs a released canary package)
  permissions: { contents: read, id-token: write }
  env: { CANARY_STAGE: c${{ github.run_id }}-${{ github.run_attempt }} }

A push reads the versions from .release-please-manifest.json.
A manual run reads them from the canary_versions input.
Each run and attempt gets its own stage, so the function name never collides.

The existing publish jobs do not change.
manual-publish now also requires an empty canary_versions.
So a canary-only manual run publishes nothing.

The cleanup step fails loudly. It ignores only a log group that AWS already removed.

serverless remove --stage "$CANARY_STAGE" || status=1
aws logs delete-log-group --log-group-name "$log_group"   # ResourceNotFoundException is the only pass
exit "$status"

Decisions

• The config schema of Serverless 3 does not know nodejs22.x. The package step prints a warning. configValidationMode: warn keeps it a warning. CloudFormation accepts the runtime. The canary reads process.version and fails on any runtime that is not Node 22.
• aws-actions/configure-aws-credentials is pinned to a SHA for v4.3.1. This matches the pinning style of this repository.
• .canary/ is now in .gitignore and in the Biome ignore list. Serverless writes generated CloudFormation JSON there, and code:check fails on those files.

Testing

• yarn code:fix, yarn code:check, yarn lint:pkg, yarn build, yarn typecheck, and yarn test all pass.
• canary.test.ts adds 8 tests to yarn test:integration. The tests use no network and no AWS account. They cover the happy path, a version that is not exact, a version mismatch, a wrong runtime, an export that did not resolve, an export with the wrong type, a missing export, and an error payload from Lambda.
• The local dry run packaged the app against the published artifacts. The archive holds the three external LaunchDarkly packages.

  node canary.mjs build --ai-node=0.2.0 --openai-messages=0.2.0 --ai-otel=0.1.1 --package --stage=local
  

• I did not verify the deploy path, the invoke path, or the cleanup path against AWS. There is no role to assume yet.

Reviewer guide

1. Read integration-tests/lambda-commonjs/README.md for the AWS trust policy, the permissions, and the owner.
2. Read the release-canary job in .github/workflows/release-please.yml. Check the if condition, the OIDC step, and the cleanup step.
3. Read canary.mjs for the version checks and the assert contract.
4. Read fixture/. Confirm that you want no externals allowlist in this test.

Link to Devin session: https://app.devin.ai/sessions/bc2f6d69a1b54d9a8ee7b3f1cdee2f8c
Open in Devin Desktop: https://app.devin.ai/desktop/session/bc2f6d69a1b54d9a8ee7b3f1cdee2f8c?variant=devin
Requested by: @tonytrinh3

---

Note

Overview
Adds a published-artifact release canary that runs after relevant npm publishes (or on demand) to catch CommonJS Lambda load failures like AIC-3370.

The new integration-tests/lambda-commonjs harness uses canary.mjs to poll npm for exact semver versions, stage a Serverless 3 + Webpack 5 + webpack-node-externals app with no externals allowlist, and validate an invoke payload (Node 22 runtime, export types, installed versions). canary.test.ts covers assert/version guards offline; yarn test:integration now includes it. Generated .canary/ output is gitignored and excluded from Biome.

release-please.yml gains a release-canary job (GitHub OIDC → ephemeral ld-ai-canary-* deploy/invoke/cleanup) gated on AWS_LAMBDA_CANARY_ROLE_ARN and AWS_LAMBDA_CANARY_PERMISSIONS_BOUNDARY, plus optional canary_versions on workflow_dispatch for canary-only runs. Manual publish is skipped when canary_versions is set.

^{Reviewed by Cursor Bugbot for commit 11acd8d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

@cursor review

[devin-ai-integration]

Devin Review found 3 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[jeffdupont]

This canary does what it says, and I checked that by making it fire locally. Building the fixture against the pre-fix 8 Sep releases (ai-server 0.2.0, ai-node 0.2.0, openai-messages 0.2.0, ai-otel 0.1.1) and invoking the packaged handler.js throws ERR_PACKAGE_PATH_NOT_EXPORTED No "exports" main defined, which is the AIC-3370 error. Webpack turns the handler's import() calls into require(), so the broken path is the one being tested. Against the current releases (0.4.0 / 0.3.0 / 0.4.0 / 0.2.0), all six capabilities load with the right types, and assert rejects only the runtime because I ran Node 24 locally. build --package also works with no AWS credentials. #76, #77 and #78 are all merged, the base is main, and CI is green. I did not run the AWS deploy, invoke or remove steps, because there is no role yet.

Since the AWS role doesn't exist yet, this README is what whoever creates it will follow, so it's worth getting right now.

1. Must fix: the README asks for a role that can escalate its own privileges.

• The trust policy allows sub repo:launchdarkly/js-ai-sdk:*, so any branch can assume the role. Anyone with write access can push a branch that edits this workflow and then dispatch it.
• The permissions include iam:CreateRole, PutRolePolicy, AttachRolePolicy and PassRole, plus lambda:CreateFunction and InvokeFunction. Even scoped to ld-ai-canary-*, that's enough to create a role, give it any policy, and run code as it.

Fix:

• Limit sub to repo:launchdarkly/js-ai-sdk:ref:refs/heads/main. The release push runs from main, and a canary dispatch can be started from main too.
• Drop AttachRolePolicy and DetachRolePolicy. The generated stack only uses inline Policies on IamRoleLambdaExecution; I checked the packaged template.
• Require a permissions boundary on CreateRole and PutRolePolicy (an iam:PermissionsBoundary condition), and set provider.iam.role.permissionsBoundary in serverless.yml.

2. Should fix: AWS credentials are set before npm install runs. Configure AWS credentials comes before Build the canary package, and the build runs npm install for the Serverless 3 / Webpack toolchain. Every install script in that tree runs with the session credentials in its environment. The build doesn't need AWS: I ran build --package with the credential and config files set to /dev/null, and it packaged fine. Moving the credentials step to just before Deploy fixes it. It also means the cleanup step's cd .canary/app no longer fails when the credentials step is what failed.

3. Should fix: the README's debug command points at a path that doesn't exist. After serverless package, .webpack/ is gone (serverless-webpack deletes it), so require('./.webpack/service/handler.js') fails with MODULE_NOT_FOUND. Unzipping .serverless/ld-ai-canary.zip and requiring handler.js from there works; that's how I reproduced the bug. Alternatively, set keepOutputDirectory: true under custom.webpack.

[devin-ai-integration]

Plain version: Thanks Jeff. I fixed all five points in bd31003.

What it means for the code (junior eng):

• The README limits the OIDC sub to repo:launchdarkly/js-ai-sdk:ref:refs/heads/main.
• The README drops iam:AttachRolePolicy and iam:DetachRolePolicy.
• The README requires an iam:PermissionsBoundary condition on iam:CreateRole and iam:PutRolePolicy. serverless.yml sets the same boundary on the execution role through provider.iam.role.permissionsBoundary. The workflow passes it as vars.AWS_LAMBDA_CANARY_PERMISSIONS_BOUNDARY.
• The Configure AWS credentials step now runs after the build and right before the deploy.
• The README debug command now unzips .serverless/ld-ai-canary.zip and loads handler.js from there.

I verified the packaging locally against the current releases. The template drops PermissionsBoundary with no variable. It holds the ARN with the variable set.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bd31003. Configure here.}

[jeffdupont]

Thanks for the quick turnaround. I re-checked all four points against 41e0fdf, and they're fixed:

• The OIDC sub is limited to ref:refs/heads/main. The job has no environment:, so that matches both the release push and a dispatch from main.
• AttachRolePolicy and DetachRolePolicy are gone, and the boundary is required in both places.
• The AWS credentials now start after the build, so npm install never sees them.
• The debug command unzips the package first, and .unzipped/ sits under the ignored .canary/.

The cleanup gate on steps.aws-credentials.outcome also fixes Bugbot's point. CI is green.

One thing before approval: the README never names the repo variable for the boundary (inline on README.md:117). The workflow reads vars.AWS_LAMBDA_CANARY_PERMISSIONS_BOUNDARY, but the README only names the fixture's env var LD_AI_CANARY_PERMISSIONS_BOUNDARY, and the variable table lists only the role ARN and region. Someone who follows the README sets the role ARN but not the boundary. The job then runs, because it's gated on the role ARN only. The template has no PermissionsBoundary, so the role's own condition denies CreateRole, and the first real canary fails at Deploy with an IAM AccessDenied. That fails safe, but it's a confusing first run for whoever sets this up.

There's one optional cleanup inline too. I didn't run the deploy, invoke or remove steps, because the role still doesn't exist.

[jeffdupont]

11acd8d fixes both remaining points: the README names AWS_LAMBDA_CANARY_PERMISSIONS_BOUNDARY, and the job is gated on it. The dead folder check is gone too. CI is green. Approving.

The job stays skipped until someone creates the AWS role (main-only sub, permissions boundary) and sets both repository variables, so merging this is safe now.

[tonytrinh3]

thank you for the thorough review Jeff!