ampx pipeline-deploy: reproducible ENOENT on cdk.out/manifest.json in Amplify Hosting CI, same schema succeeds via ampx sandbox locally

[willburnette]

Environment

@aws-amplify/backend: 1.14.1
@aws-amplify/backend-cli: 1.8.3
aws-cdk: ^2.1000.1
aws-cdk-lib: ^2.179.0
Node (CI, confirmed via `node -v` immediately before the failing command): v22.23.1
Node (local, where the same schema deploys successfully): v22.23.1 (identical)

Hosting: AWS Amplify Hosting (Amplify Gen 2, Next.js SSR / WEB_COMPUTE), building via a custom amplify.yml backend phase running npx ampx pipeline-deploy --branch $AWS_BRANCH --app-id $AWS_APP_ID.

Description

ampx pipeline-deploy fails 100% reproducibly (6/6 attempts, across 6 separate fresh CodeBuild containers/jobs, over ~2 hours) with:

[BackendBuildError] Unable to deploy due to CDK Assembly Error
 ∟ Caused by: [AssemblyError] Assembly builder failed
 ∟ Caused by: [Error] ENOENT: no such file or directory, open '.../.amplify/artifacts/cdk.out/manifest.json'

With --debug, the failure traces into @aws-cdk/toolkit-lib itself:

BackendBuildError: Unable to deploy due to CDK Assembly Error
    at CdkErrorMapper.getCDKError (.../node_modules/@aws-amplify/backend-deployer/lib/cdk_error_mapper.js:110:20)
    at CdkErrorMapper.getAmplifyError (.../node_modules/@aws-amplify/backend-deployer/lib/cdk_error_mapper.js:28:35)
    at CDKDeployer.deploy (.../node_modules/@aws-amplify/backend-deployer/lib/cdk_deployer.js:108:39)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.handler (.../node_modules/@aws-amplify/backend-cli/lib/commands/pipeline-deploy/pipeline_deploy_command.js:46:9)
AssemblyError: Assembly builder failed
    at AssemblyError.withCause (.../node_modules/@aws-cdk/toolkit-lib/lib/toolkit/toolkit-error.js:91:16)
    at Object.produce (.../node_modules/@aws-cdk/toolkit-lib/lib/api/cloud-assembly/source-builder.js:142:61)
    at async ContextAwareCloudAssemblySource.produce (.../node_modules/@aws-cdk/toolkit-lib/lib/api/cloud-assembly/private/context-aware-source.js:49:33)
    at async assemblyFromSource (.../node_modules/@aws-cdk/toolkit-lib/lib/toolkit/private/index.js:21:85)
    at async synthAndMeasure (.../node_modules/@aws-cdk/toolkit-lib/lib/toolkit/toolkit.js:1290:21)
    at async Toolkit.synth (.../node_modules/@aws-cdk/toolkit-lib/lib/toolkit/toolkit.js:245:26)
    at async CDKDeployer.deploy (.../node_modules/@aws-amplify/backend-deployer/lib/cdk_deployer.js:51:29)

The debug output reports synthesis as apparently having succeeded immediately before this: "Synthesis time: 2.34s" with the ENOENT embedded in the same timing/error payload, and Bundling asset .../AmplifyBranchLinker/CustomResourceLambda/Code/Stage... completing (Done in 84ms) just before.

Crucially, the exact same schema (two models: one allow.guest().to(['create']), one with a custom .identifier(), plus CDK L1 overrides via AmplifyDynamoDbTableWrapper for PITR/TTL/deletion protection) deploys successfully via npx ampx sandbox --once on my local machine (same Node version, same installed package versions per package-lock.json, same schema file, same backend.ts) — both on a first clean attempt and reliably on retry when it hit the identical transient-looking ENOENT once locally.

This specifically reproduces with pipeline-deploy (which bundles an additional AmplifyBranchLinker/CustomResourceLambda asset that sandbox does not need) — this may be relevant to the root cause given it's the one asset visibly bundled differently between the two code paths in the logs.

Steps ruled out

• tsx version bug (a real, previously-reported issue where tsx@4.20.1+ causes a similar ENOENT): already on the fixed 4.19.4.
• @aws-amplify/backend-cli outdated: already on 1.8.3, the latest published stable at the time of testing.
• Node version mismatch between CI and local: confirmed and ruled out. CI's build image defaults to Node v22.18.0; explicitly pinned the backend build phase to v22.23.1 (matching local exactly, confirmed via node -v in the build log immediately before the failing command) — failure is byte-for-byte identical.
• Plain retry / retry-with-rm -rf .amplify-and-delay: both fail identically on every attempt; this is not a one-off transient race resolved by a simple retry (unlike the single time I saw this locally).
• Package version drift: @aws-amplify/backend resolves to exactly 1.14.1 per the lockfile in both environments — no drift to account for.

Reproduction

1. amplify/data/resource.ts defines two a.model()s, one with allow.guest().to(['create']), one with allow.guest().to(['create','read','update']) and a custom .identifier(['bucketKey']).
2. amplify/backend.ts calls defineBackend({ auth, data }), then applies L1 overrides via backend.data.resources.cfnResources.amplifyDynamoDbTables['<Model>'] (.pointInTimeRecoveryEnabled, .deletionProtectionEnabled, .timeToLiveAttribute).
3. amplify.yml backend phase: npx ampx pipeline-deploy --branch $AWS_BRANCH --app-id $AWS_APP_ID.
4. Push to a branch with an Amplify Hosting PR preview / branch deploy configured (enablePullRequestPreview: true).
5. Build fails with the ENOENT above, every time.
6. npx ampx sandbox --once with the identical amplify/ directory succeeds locally.

Happy to share the full amplify/ directory and complete --debug build logs (6 full logs across attempts) privately if useful — omitted verbose logs here for length but have them archived.