P0: CloudFormation 500-resource-per-stack ceiling breached — main at 492/500, #681 at 501, blocks any PR adding a route

[scottschreckengaust]

Summary

main is 492 of 500 resources for --context compute_type=ecs. CloudFormation's per-stack resource ceiling is a hard 500, so the stack has 8 resources of headroom. PR #681 adds 9 and has already crossed it — its build (agentcore) check fails with 28 test failures, all of them this:

«TooManyResourcesInStack» Number of resources in stack 'TestAgentStackEcs':
501 is greater than allowed maximum of 500

This is not a test artifact. CloudFormation enforces the limit at changeset creation, so a real compute_type=ecs deployment of #681's branch would be rejected the same way. Any PR that adds a Lambda plus API Gateway wiring — roughly 7–9 resources — now fails, regardless of whether the change is correct.

Measured

Synthesized on a branch whose diff touches zero cdk/ files, so these are main's numbers (main @ 521bf647):

Context	Resources	Headroom vs 500
default (agentcore)	480	20
compute_type=ecs	492	8
compute_type=ecs + PR #681	501	−1 (over)

main's own post-merge build is green, confirming main is still (barely) under the ceiling. The breach is only visible on PRs that add resources.

Heaviest contributors in the 501-resource template:

Type	Count
AWS::Lambda::Permission	70
AWS::ApiGateway::Method	67
AWS::IAM::Role	58
AWS::IAM::Policy	56
AWS::Lambda::Function	50
AWS::ApiGateway::Resource	35
AWS::DynamoDB::Table	21

The API-Gateway-plus-Lambda pattern dominates: each new authenticated route costs a Function, a Role, a Policy, one or two Resources, a Method, and a Permission.

Why this needs its own issue

#735 tracks the 1 MB template-size ceiling (98.4% as of 2026-08-06) and mentions 486/500 resources in passing, proposing a stack split that "also helps the 500-resource ceiling". #830 (same 1 MB wall) was closed NOT_PLANNED.

Those are a different limit with a different trigger. Template bytes grow with metadata and policy verbosity; resource count grows with construct count. A change can be cheap in bytes and expensive in count, or the reverse. The 1 MB issue is a warning; this one is an active blocker with a PR already failing CI on it.

Note #735 currently carries no labels and no priority.

Impact

• Blocks PRs. feat(cli): bgagent linear remove-workspace + DELETE route + fail-closed resolver (#306) #681 (issue feat(cli): bgagent linear remove-workspace command #306) cannot go green as written. Every subsequent feature adding a route hits the same wall.
• Blocks compute_type=ecs deployments of any such branch — CloudFormation rejects the changeset before creating resources.
• Failure mode is misleading. The error names the stack, not the change that tipped it, so authors reasonably conclude their own diff is broken. It took a per-variant synth count to attribute it here.
• Silent until breach. The synth-time annotation is INFO-level, so 492/500 does not fail anything. The first signal is a red build on an unrelated PR.

Proposed remediation

1. Structural (the real fix): split the stack. Move a cohesive group into a nested stack — the integration constructs (Linear/Jira/GitHub webhook + link + removal Lambdas and their API resources) are the natural seam, and are exactly what is growing. This raises both ceilings at once, so it also resolves fix(cdk): agent stack template at 98.4% of the CloudFormation 1 MB limit #735.
2. Make the ceiling fail loudly, not informationally. Add a synth-time guard that fails the build above a budget (e.g. 470) so the signal arrives on the PR that consumes headroom, not on the next one. fix(cdk): agent stack template at 98.4% of the CloudFormation 1 MB limit #735 proposes the same for bytes; one guard should cover both, and it should run per compute_type variant, since the default variant (480) is 12 under the ECS variant and would not have caught this.
3. Cheap reclamation, if feat(cli): bgagent linear remove-workspace + DELETE route + fail-closed resolver (#306) #681 needs to land first. Consolidate per-Lambda inline policies into shared managed policies (IAM::Policy = 56) and review whether every route needs a distinct ApiGateway::Resource. This buys headroom without restructuring, but it is a delaying action, not a fix.

Acceptance criteria

• compute_type=ecs synthesizes with documented headroom below 500, and the number is asserted by a test
• A synth-time guard fails the build when any compute_type variant exceeds the budget, naming the variant and the count
• feat(cli): bgagent linear remove-workspace + DELETE route + fail-closed resolver (#306) #681 / feat(cli): bgagent linear remove-workspace command #306 unblocked without weakening its authorization model
• fix(cdk): agent stack template at 98.4% of the CloudFormation 1 MB limit #735 either resolved by the same split or explicitly re-scoped to bytes only

Reproduce

cd cdk
npx cdk synth -q -c compute_type=ecs 2>&1 | grep 'Number of resources'
# INFO Number of resources: 492 is approaching allowed maximum of 500

Failing run on #681: https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/actions/runs/33706036413

Related: #735 (1 MB template limit), #830 (closed, same wall), #306 / #681 (blocked by this).

[scottschreckengaust]

Remediation planning has moved to #852, which supersedes the "Proposed remediation" section above. #851 stays the incident record (measurement, attribution, failing run).

#852 corrects one thing stated here: the provisional fix above led with a nested-stack split. A synth-level census found a cheaper and better-founded first step — LambdaIntegration defaults allowTestInvoke: true, which emits a second Lambda::Permission per method for the console-only test-invoke-stage. 30 of this stack's 492 resources are those. Setting it false across the 34 call sites takes the stack to 462 (headroom 38) and is a least-privilege improvement on its own merits, independent of the resource count.

The structural split is still needed; it is no longer the first move.