feat(aws-cloudformation): production-hardened ECS Fargate templates (dedicated webhook + HA tiers) - #169
Conversation
…hook service Add n8n-w-multimain-queuemode-webhooks.yaml, a variant of the multi-main queue-mode ECS Fargate template that breaks production webhook handling out into its own service. - new webhook target group, listener rule (routes /webhook/*, /webhook-waiting/*, /form/* and /form-waiting/* to the webhook service; UI, REST API and /webhook-test/* stay on the mains) - new webhook task definition (runs `n8n webhook`) plus its service and CPU/memory autoscaling, mirroring the worker tier - set N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true on the mains so production webhooks are served only by the webhook service - N8nWebhookServiceUrl output Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- pin RDS EngineVersion to 16.9 (bare "16" is deprecated for new instances, W3691) - drop redundant DependsOn entries already implied by Ref/GetAtt (W3005) - suppress W1011 on PostgresDB: MasterUserPassword is a user-supplied stack parameter by design cfn-lint now passes with no errors or warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…late Brings the tier-agnostic hardening from the webhooks template into the base n8n-w-multimain-queuemode.yaml, without the dedicated webhook tier (which remains the reason to use the -webhooks variant). - pin EngineVersion "16.9" (was deprecated "16"); resolve cfn-lint warnings (W1011 suppression on PostgresDB; remove redundant DependsOn) -> exit 0 - DB connection tuning (keep-alive + connection/idle timeouts) on main + worker - graceful shutdown: N8N_GRACEFUL_SHUTDOWN_TIMEOUT=90 + container StopTimeout=120 - ALB connection draining + slow start on the main target group - readiness-gated ALB health check (/healthz/readiness, gates on DB+Redis) - Redis HA as an opt-in knob (RedisReplicasPerNodeGroup, default 0) - DBInstanceClass sizing guidance; note pointing to the -webhooks template - pin the n8nio/n8n image to 2.23.4 (was :latest) for reproducible deploys Does NOT set N8N_DISABLE_PRODUCTION_MAIN_PROCESS: the base has no webhook tier, so the main process must keep serving production webhooks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(aws-cloudformation): ECS Fargate template with dedicated webhook service
…-hardening # Conflicts: # aws-cloudformation/ecs-fargate/n8n-w-multimain-queuemode.yaml
feat(aws-cloudformation): backport reliability hardening to base multimain-queuemode template
… main The full ECS Fargate work was committed to the ecs-fargate-0.1.1 release tag only and never brought forward to main. This ports the missing deliverables onto main: - Add n8n-w-multimain-queuemode-webhooks-ha.yaml (HA/scale tier template). - Update n8n-w-multimain-queuemode-webhooks.yaml to the complete release version (queue-depth worker autoscaling, request-rate webhook autoscaling, DB failover tuning, graceful shutdown + ALB draining, Redis HA, readiness-gated health checks, execution-data pruning, opt-in Python task runner). - Add ecs-fargate/RELEASE.md. The base template (n8n-w-multimain-queuemode.yaml) is left as-is: main's version already supersedes the tag's (ECS scaling params + the n8n-io#164 S3 fix). The tag was cut from upstream 1.9.0, before fix n8n-io#164, so its -webhooks and -ha templates carried the collision-prone explicit S3 BucketName. That fix is reapplied here so both templates auto-generate a globally-unique bucket name, matching the base template. cfn-lint exit 0 on all three templates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….1.1 feat(aws-cloudformation): forward-port ecs-fargate 0.1.1 templates to main
Remove the release-note file and fold its still-useful content into the README so there is a single source of truth for the ecs-fargate templates. - Add a "Which template to use" tier table (base / -webhooks / -ha). - Fix the stale Queue-Depth Scaling section: the -webhooks and -ha templates now implement queue-depth worker autoscaling directly (in-VPC Lambda + WorkerBacklogPerTask target tracking); the base template stays CPU/memory only. - Note the dedicated webhook tier and the Enterprise-license requirement. - Warm up the intro. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the dig at Kubernetes/control-plane cost; frame ECS Fargate as a serverless-container option alongside the EKS module instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
…tely Address cubic review: the intro referenced an unnamed, unlinked "EKS module". Point readers at the actual paths instead: the in-repo Helm chart (charts/n8n) and the terraform-aws-n8n EKS module, and link that module from the tier-parity table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
n8n 2.x does not load DB_POSTGRESDB_SSL_CA as a file path, so the RDS CA bundle downloaded by the rds-cert-fetcher init container was never trusted. With DB_POSTGRESDB_SSL_REJECT_UNAUTHORIZED left at its secure default the tasks crash on DB init with "self-signed certificate in certificate chain"; the templates masked that by disabling verification (REJECT_UNAUTHORIZED false), which connects to Postgres without validating the server cert. Point NODE_EXTRA_CA_CERTS at the same bundle instead (node trusts it natively, and it also covers the ElastiCache Redis TLS chain), and set DB_POSTGRESDB_SSL_REJECT_UNAUTHORIZED to true so the RDS certificate is actually verified. Applies to the main, worker, and webhook task definitions across all three templates. The rds-cert-fetcher init container, /certs volume, and DB_POSTGRESDB_SSL_ENABLED are unchanged. Validated against live RDS: tasks reach "Migrations in progress", stacks reach CREATE_COMPLETE, /healthz serves 200 over HTTPS, and workflows execute and autoscale under load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(aws-cloudformation): verify RDS TLS via NODE_EXTRA_CA_CERTS
Follow-up to the RDS TLS fix, covering the other five cubic findings across
all three ecs-fargate templates.
- Stack-scope Secrets Manager names (${AWS::StackName}-app-user/*) and every
name-based ARN reference, so tier variants can coexist in one account. [P1]
- Worker-scaler Lambda: drop the CERT_NONE / check_hostname bypass so the
Redis TLS peer certificate is verified against the default trust store. [P1]
- Remove the weak default DB and Redis passwords so a value must be supplied,
and update the now-stale parameter descriptions. [P2]
- Add DependsOn: VPCGatewayAttachment to the NAT gateways (create-order race). [P2]
- HA tier: add a third NAT gateway + public subnet + per-AZ route table so
PrivateSubnet3 no longer routes egress through AZ1's NAT (cross-AZ SPOF). [P2]
Validated locally: cfn-lint exit 0 and aws cloudformation validate-template
pass on all three templates. Runtime TLS behavior (Redis) to be confirmed on
a live deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(aws-cloudformation): address remaining cubic review findings (secret scoping, Redis TLS, NAT HA, weak defaults)
gml3ff
left a comment
gml3ff
left a comment
There was a problem hiding this comment.
We've discussed this already and it sounds like you've been able to successfully deploy this already, I'll merge and provide some additional updates shortly after.
krider2010
left a comment
krider2010
left a comment
There was a problem hiding this comment.
Really nice PR — the tier-ladder framing is clear, and I like how much of the hardening is opt-in (Redis HA, the Python runner, the readiness gating). A couple of things that caught my eye too: the SG rules split out to sidestep the circular refs, and the per-AZ NAT on the HA tier.
One thing I'd want fixed before merge, plus a handful of smaller inline notes.
The blocker: queue-depth autoscaling reads the wrong Redis key, so it silently never scales. The Lambda does LLEN bull:jobs:wait, but n8n's Bull queue is default — our own KEDA examples and this PR's README both use bull:default:wait. Worth an end-to-end check after fixing, since the metric would've read a flat 0 during load testing and CPU/mem scaling might have masked it.
Rest are inline and lower stakes — a webhook health-check path I'd double-check, an S3 endpoint that misses the third AZ on the HA tier, a hardcoded worker count, the Aurora engine version, and an upgrade-path note for the base template. None of those block the concept, just polish. Details inline.
| s = ctx.wrap_socket(raw, server_hostname=host) | ||
| try: | ||
| _resp(s, "AUTH", token) | ||
| return int(_resp(s, "LLEN", "bull:jobs:wait")) |
There was a problem hiding this comment.
bull:jobs:wait won't exist — the queue is named default, so this wants bull:default:wait. Compare charts/n8n/examples/keda-autoscaling.yaml and the README in this same PR, both of which use default. As-is LLEN returns 0 every minute and the backlog metric is always 0, so the target-tracking policy never fires and you're silently back on CPU/mem-only scaling. Worth fixing the comment just above too.
There was a problem hiding this comment.
Good catch, fixed in 7796403: now bull:default:wait (matching charts/n8n/examples/keda-autoscaling.yaml), and updated the comment just above too.
There was a problem hiding this comment.
Correction to my earlier reply here: I switched to bull:default:wait following this repo's charts/n8n/examples/keda-autoscaling.yaml, but on verifying against n8n source that example is actually wrong. n8n names the queue jobs (QUEUE_NAME = 'jobs' in packages/cli/src/scaling/constants.ts, same at the pinned n8n@2.23.4) with default prefix bull (@n8n/config docstring example 'bull:jobs:23'), so the correct key is bull:jobs:wait, which was the PR's original value. Reverted in 9d61987, so this template is now correct. Heads up that the same bull:default:wait bug lives in charts/n8n/examples/keda-autoscaling.yaml:35, so that KEDA scaler would never fire either, worth a separate fix outside this PR.
| s = ctx.wrap_socket(raw, server_hostname=host) | ||
| try: | ||
| _resp(s, "AUTH", token) | ||
| return int(_resp(s, "LLEN", "bull:jobs:wait")) |
There was a problem hiding this comment.
bull:jobs:wait won't exist — the queue is named default, so this wants bull:default:wait. Compare charts/n8n/examples/keda-autoscaling.yaml and the README in this same PR, both of which use default. As-is LLEN returns 0 every minute and the backlog metric is always 0, so the target-tracking policy never fires and you're silently back on CPU/mem-only scaling. Worth fixing the comment just above too.
There was a problem hiding this comment.
Good catch, fixed in 7796403: now bull:default:wait (matching charts/n8n/examples/keda-autoscaling.yaml), and updated the comment just above too.
There was a problem hiding this comment.
Correction to my earlier reply here: I switched to bull:default:wait following this repo's charts/n8n/examples/keda-autoscaling.yaml, but on verifying against n8n source that example is actually wrong. n8n names the queue jobs (QUEUE_NAME = 'jobs' in packages/cli/src/scaling/constants.ts, same at the pinned n8n@2.23.4) with default prefix bull (@n8n/config docstring example 'bull:jobs:23'), so the correct key is bull:jobs:wait, which was the PR's original value. Reverted in 9d61987, so this template is now correct. Heads up that the same bull:default:wait bug lives in charts/n8n/examples/keda-autoscaling.yaml:35, so that KEDA scaler would never fire either, worth a separate fix outside this PR.
| HealthyThresholdCount: 3 | ||
| UnhealthyThresholdCount: 5 | ||
| # readiness (DB + Redis) gates traffic, fixes the new-task 503s during rolling deploys | ||
| HealthCheckPath: /healthz/readiness |
There was a problem hiding this comment.
Small inconsistency worth confirming: the target group health-checks /healthz/readiness, but the comment just above says the webhook process serves /healthz (not /), and the container's own health check curls /healthz. If the webhook process doesn't actually serve /healthz/readiness with a 200, every webhook target stays unhealthy and production /webhook/* + /form/* all 503. Can you confirm the webhook process serves readiness? If not, /healthz is the safe bet here.
There was a problem hiding this comment.
Switched to /healthz in 7796403 to match the container health check, the safe consistent choice you suggested. Also fixed the comment above.
| HealthyThresholdCount: 3 | ||
| UnhealthyThresholdCount: 5 | ||
| # readiness (DB + Redis) gates traffic, fixes the new-task 503s during rolling deploys | ||
| HealthCheckPath: /healthz/readiness |
There was a problem hiding this comment.
Small inconsistency worth confirming: the target group health-checks /healthz/readiness, but the comment just above says the webhook process serves /healthz (not /), and the container's own health check curls /healthz. If the webhook process doesn't actually serve /healthz/readiness with a 200, every webhook target stays unhealthy and production /webhook/* + /form/* all 503. Can you confirm the webhook process serves readiness? If not, /healthz is the safe bet here.
There was a problem hiding this comment.
Switched to /healthz in 7796403 to match the container health check, the safe consistent choice you suggested. Also fixed the comment above.
| ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3" | ||
| RouteTableIds: | ||
| - !Ref PrivateRouteTableAz1 | ||
| - !Ref PrivateRouteTableAz2 |
There was a problem hiding this comment.
On the HA tier this misses the third AZ — you added PrivateRouteTableAz3 for PrivateSubnet3, but the S3 endpoint only lists Az1/Az2. Workers run in all three subnets and are the ones doing S3 binary IO, so anything scheduled into AZ3 goes out via the NAT gateway instead of the free endpoint (NAT data-processing cost, and you lose the endpoint isolation). Adding !Ref PrivateRouteTableAz3 sorts it. (The -webhooks template maps subnet3 to Az1's table which is already on the endpoint, so it's only this one.)
There was a problem hiding this comment.
Fixed in 7796403: added !Ref PrivateRouteTableAz3 to the S3 gateway endpoint so AZ3 workers use the endpoint instead of the NAT gateway.
| Properties: | ||
| ServiceName: !Sub "${AWS::StackName}-svc-worker" | ||
| Cluster: !GetAtt ECSCluster.Arn | ||
| DesiredCount: 4 |
There was a problem hiding this comment.
WorkerMinTasks is a parameter but the initial DesiredCount is hardcoded to 4, so if someone raises the floor the service still launches 4 on create and only ratchets up once autoscaling kicks in — under-provisioned right when the raised floor was meant to help. The webhook service uses !Ref WebhookMinTasks a bit further down; worth matching that with !Ref WorkerMinTasks here (same in the -webhooks template).
There was a problem hiding this comment.
Fixed in 7796403: the worker service now uses DesiredCount: !Ref WorkerMinTasks (in both the -webhooks and -ha templates), matching the webhook service.
| - W1011 | ||
| Properties: | ||
| Engine: aurora-postgresql | ||
| EngineVersion: "16.9" |
There was a problem hiding this comment.
Worth double-checking 16.9 is actually an offered aurora-postgresql version — Aurora curates its minor versions separately from community RDS Postgres, so a version that's valid on the RDS-instance templates isn't guaranteed here. If it's not offered in the region, cluster creation fails with InvalidParameterCombination. Quick aws rds describe-db-engine-versions --engine aurora-postgresql confirms it.
There was a problem hiding this comment.
Confirmed via aws rds describe-db-engine-versions --engine aurora-postgresql: 16.9 is an offered aurora-postgresql version, so no change needed here. Thanks for the flag.
| Properties: | ||
| Engine: postgres | ||
| EngineVersion: 16 | ||
| EngineVersion: "16.9" |
There was a problem hiding this comment.
Pinning these is the right move for new stacks, just flagging the upgrade path for anyone already running the old base template: bumping EngineVersion from 16 to 16.9 forces an engine modify/reboot on the single RDS instance, and if AutoMinorVersionUpgrade already moved them past 16.9 the update rolls back. Same story with latest -> 2.23.4 on the image (lines 1006/1205) — a stack that's pulled a newer n8n and migrated the DB would crash-loop on the downgrade. Probably just needs a line in the README's upgrade notes rather than a code change.
There was a problem hiding this comment.
Agreed, kept the pins (right for new stacks) and documented the upgrade path in the README in 7796403: added an "Upgrading existing stacks" section covering the RDS 16->16.9 engine modify/reboot (and the rollback if AutoMinorVersionUpgrade already moved past it) and the n8n image-downgrade crash-loop. No code change to the pins.
- Correct Bull backlog key to bull:default:wait (queue is named default) so the worker queue-depth autoscaler actually reads a non-zero LLEN. - Point webhook target-group health check at /healthz to match the container health check and avoid all-webhook 503s on startup. - Add PrivateRouteTableAz3 to the HA S3 gateway endpoint so AZ3 workers use the free endpoint instead of the NAT gateway. - Use !Ref WorkerMinTasks for the worker service DesiredCount so a raised floor provisions on create instead of ratcheting up via autoscaling. - Document the engine/image upgrade path in the README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
…pgrade notes The queue-depth scaler Lambda must read the real Bull key. n8n names the queue `jobs` (QUEUE_NAME = 'jobs' in scaling/constants.ts) with default prefix `bull` (config docstring example: 'bull:jobs:23'), so the key is `bull:jobs:wait`. The earlier switch to `bull:default:wait` (following the repo's keda-autoscaling.yaml example, which is itself wrong) would make LLEN always return 0 and disable queue-depth autoscaling. Reverted in both webhook templates (code + comment). Also clarify in the README that EngineVersion and the n8n image tag are hardcoded properties in the templates, not stack parameters, so an upgrade means editing the template value rather than overriding a parameter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
krider2010
left a comment
krider2010
left a comment
There was a problem hiding this comment.
Re-reviewed after the latest commits — all good from me. The webhook health-check path, the AZ3 S3 route, the worker DesiredCount, and the README upgrade notes are all sorted, and I confirmed Aurora 16.9 is a valid engine version so no change needed there.
4 participants
Summary
This adds a production-hardened ECS Fargate deployment path for n8n in queue mode, as a tier ladder of CloudFormation templates. It builds on the existing
n8n-w-multimain-queuemode.yamlexample: that template gets tier-agnostic reliability hardening, and two new opt-in variants add a dedicated webhook tier and a high-availability tier.All templates are
cfn-lintclean (exit 0) and were validated against load and resilience testing on live ECS Fargate deploys.Which template
n8n-w-multimain-queuemode.yaml(modified)n8n-w-multimain-queuemode-webhooks.yaml(new)n8n-w-multimain-queuemode-webhooks-ha.yaml(new)Reliability hardening (all templates)
STATEMENT_TIMEOUTleft at the n8n default to avoid failing long migrations.N8N_GRACEFUL_SHUTDOWN_TIMEOUT=90, containerStopTimeout=120, target-groupderegistration_delay.timeout_seconds=30so in-flight executions and HTTP requests finish on scale-in and rolling deploys./healthz/readiness(gates on DB + Redis) instead of a shallow check, plusslow_start.duration_seconds=30. Closes transient 503s when new tasks took traffic before their connections were ready.RedisReplicasPerNodeGroup):>=1adds a replica and enables Multi-AZ + automatic failover;0keeps a single node (no behaviour change on update).EngineVersion: "16.9"(was deprecated16), resolve cfn-lint warnings, and keep the auto-generated (globally-unique) S3 binary bucket name.DBInstanceClassparameter description.Dedicated webhook + HA templates add
/webhook/*,/webhook-waiting/*,/form/*,/form-waiting/*to an8n webhookservice (UI / REST //webhook-test/*stay on the mains).N8N_DISABLE_PRODUCTION_MAIN_PROCESS=trueon mains. NewN8nWebhookServiceUrloutput.WorkerBacklogPerTaskmetric from the Bull backlog key; a target-tracking policy holds backlog-per-worker at target (ECS analog of the EKS KEDA queue scaler). CPU/memory policies stay as a safety net.ALBRequestCountPerTarget(tuned viaWebhookRequestsPerTask), with a sensibleWebhookMinTasksfloor to absorb cold bursts during scale-out lag.ExecutionsDataMaxAge/ExecutionsDataMaxCount/ConcurrencyProductionLimit.EnablePythonTaskRunner, defaultfalse): external runner mode +n8nio/runnerssidecar + a root init container that patches the import allowlist fromPythonStdlibAllow/PythonExternalAllow(default empty = no imports). No sidecar / no cost when off. Validated end-to-end on a live deploy.Base template: deliberately excluded
N8N_DISABLE_PRODUCTION_MAIN_PROCESSis not set (no webhook tier, so mains must keep serving production webhooks).Testing
cfn-lintexit 0 on all three templates.