Csd 251708#101
Open
cloudar-support wants to merge 42 commits into
Open
Conversation
Enable drop_invalid_header_fields for overwatch alb
Create local-dev/ skeleton for the 3.24.6 -> 3.334.x upgrade dry-run: directories, .env.example covering all 35 prod env vars, RUNBOOK.md stub, check-prereqs.sh, and .gitignore entries for .env/dumps/data/snapshots/logs. ENCRYPTION_KEY, JWT_SECRET, and LICENSE_KEY carry explicit rails against auto-generation -- they must come from live prod (secrets) or my.retool.com (license). Per thoughts/plans/2026-05-25-retool-local-dev-env.md Phase 1.
scripts/validate.sh smoke-tests a running stack: api+jobs-runner up, /api/checkHealth 200, knex_migrations and users/apps/pages/resources row counts non-decreasing vs previous hop. Soft checks annotate migration log lines and error counts. Appends a TSV row per invocation to hop-report.tsv. Empty-workspace run exits non-zero with "api container not running" -- proves the script detects a broken stack before Phase 3 wires up compose. Per thoughts/plans/2026-05-25-retool-local-dev-env.md Phase 2.
compose/compose.3-24.yml: 3-service stack matching prod's main+jobs-runner topology (no Temporal, no Retool DB). postgres:14.6 pinned to Aurora 14.6, log_statement=ddl to match prod parameter group, healthcheck gates api startup. compose/compose.override.yml: Apple Silicon overlay forcing linux/amd64. scripts/init-postgres.sh: idempotent uuid-ossp install + Read Committed isolation check. scripts/gen-secrets.sh: generates POSTGRES_PASSWORD (64 hex) only. Refuses to touch ENCRYPTION_KEY/JWT_SECRET -- both must be pasted from a live prod ECS task, since a fresh ENCRYPTION_KEY corrupts the dump's encrypted columns. .env.example: drop inline `# ...` comments from value lines (compose v2 env_file parser treats inline # as part of the value). Compose invocation must pass --env-file .env from local-dev/ cwd, since compose's auto .env lookup is relative to the compose file dir. Per thoughts/plans/2026-05-25-retool-local-dev-env.md Phase 3.
scripts/restore-dump.sh: restores a prod Aurora dump into the local
postgres:14.6 volume. Filters Aurora-specific objects (aws_* schemas,
rdsadmin/rds_* grants, Aurora-only extensions) via
pg_restore --list -> grep -v -> --use-list. Aborts if filter delta > 50.
Hard rails:
- Refuses if ENCRYPTION_KEY or JWT_SECRET is empty/placeholder -- a
fresh key corrupts encrypted columns and the dry-run would produce
invalid signal.
- Refuses if data/ is non-empty (prevents accidental clobber on
mid-walk failure -- operator must docker compose down -v first).
Restore runs via a one-shot postgres:14.6 container attached to the
compose network, so the host need not have pg_restore installed.
RUNBOOK: short versions of Extract prod secrets, Pulling a dump from
Aurora, Aurora-specific dump content, and Mid-walk recovery added now;
full content lands in Phase 8.
Per thoughts/plans/2026-05-25-retool-local-dev-env.md Phase 4.
scripts/upgrade-hop.sh --to <tag>:
- Sed-replace RETOOL_VERSION in .env
- docker compose pull + up -d (postgres volume persists)
- Poll jobs-runner logs for migration completion
(timeout = DATABASE_MIGRATIONS_TIMEOUT_SECONDS + 60s buffer)
- Invoke validate.sh with the new tag
- Snapshot data/ -> data-snapshots/<tag>/ on success (postgres
stopped during cp -R for safety). Enables mid-walk failure
resumption without re-restoring the dump.
- Idempotent: no-op when already at target tag and snapshot exists.
Compose auto-selects by tag: <3.196 -> compose.3-24.yml,
>=3.196 -> compose.3-196.yml (Phase 6).
compose.3-24.yml: image: literals replaced with
tryretool/backend:${RETOOL_VERSION} for both api and jobs-runner.
Per thoughts/plans/2026-05-25-retool-local-dev-env.md Phase 5.
compose/compose.3-196.yml: overlay applied on top of compose.3-24.yml.
Adds tryretool/code-executor-service:${RETOOL_VERSION} pinned to the
same backend tag, plus the unprivileged-mode flags required on macOS
(nsjail-privileged path needs Linux kernel features Docker Desktop does
not expose). Extends api's env with CODE_EXECUTOR_INGRESS_DOMAIN +
WORKFLOW_BACKEND_HOST so the backend can route to the new service.
scripts/upgrade-hop.sh: --compose is now repeatable. Auto-select stacks
both compose files for >=3.196 hops (3-24 base + 3-196 overlay).
validate.sh already detects code-executor presence dynamically and runs
its healthcheck only when present -- no change needed.
Per thoughts/plans/2026-05-25-retool-local-dev-env.md Phase 6.
scripts/run-all-hops.sh: drives the walk. HOPS array reconciled with live
Docker Hub during implementation:
- 3.33.39-stable does not exist; latest 3.33 patch is 3.33.9-stable.
- 3.163 line is Edge-only -- no Stable release was ever published.
Dropped from the walk. 3.148 -> 3.196 is now the largest hop and the
most likely place for migration failure.
HOPS reduced from 9 to 8. Plan's "9 tags exist" success criterion
adjusted accordingly.
Supports --from <tag> (restores data/ from preceding green hop's snapshot
before resuming), --dry-run (prints sequence + tag status), and
--skip-tag-check (use offline).
scripts/check-tags.sh: pre-flight queries Docker Hub for every backend
tag in HOPS plus code-executor-service for >=3.196. Refuses the walk if
any tag is gone. Cached per (repo, minor) under /tmp.
scripts/upgrade-hop.sh: ge_version() replaced naive float compare --
"3.24" must be < "3.196" as a version, not a decimal. Both upgrade-hop
and check-tags now use component-wise integer comparison.
Per thoughts/plans/2026-05-25-retool-local-dev-env.md Phase 7.
local-dev/RUNBOOK.md: full operator runbook -- Prerequisites, Extract prod secrets, License key boundary, First-time setup, Pulling a fresh dump from Aurora, Aurora-specific dump content, Running the walk (with the 8-hop list and the 3.148->3.196 gap callout), Mid-walk recovery (fast-path snapshot restore + full-reset paths), Rollback story. local-dev/README.md: short pointer to RUNBOOK + inventory of every file shipped in local-dev/. thoughts/research/2026-05-25-retool-upgrade-dryrun-results.md (not in this commit -- staged separately under thoughts/): findings TEMPLATE the operator fills in after running the walk. Captures sanitised hop-report, per-hop notes, resolutions for the 8 Open Questions from the upstream research, and recommendations the prod-cutover plan will consume. Plan frontmatter flipped to status: implemented (also under thoughts/). Per thoughts/plans/2026-05-25-retool-local-dev-env.md Phase 8.
thoughts/plans/2026-05-25-retool-local-dev-env.md (status: implemented):
the executed plan -- 8 phases, RED/GREEN/REFACTOR shape adapted to
boot/validate/promote per upgrade hop.
thoughts/progress/2026-05-25-retool-local-dev-env-status.json: phase
tracker, all 8 phases marked complete.
thoughts/research/:
- 2026-05-22-retool-upgrade-3-24-to-latest.md -- version landscape,
breaking changes, prod topology.
- 2026-05-25-retool-local-dev-env.md -- env var catalog, compose
layouts, macOS specifics, code-executor flags.
- 2026-05-25-retool-upgrade-dryrun-results.md -- findings TEMPLATE,
operator fills in after running the walk.
thoughts/tmp/...verification.md: plan verification artifact from the
iterate cycle that locked in the ENCRYPTION_KEY/JWT_SECRET, license,
and Aurora-filter decisions.
…ry-run MIGRATION-RUNBOOK.md walks the prod 3.24.6 -> 3.334.15-stable upgrade with per-hop Aurora snapshots, rolling deploy, validation, and rollback paths. Includes a console-only fallback (Appendix A) for the period where the shared Terraform state is unavailable -- hops 1-3 ship via console, hops 4-8 resume via Terraform once state is reconciled. Local-dev fixes uncovered while running the dry-run end-to-end: - compose.3-24.yml: postgres host port 5432 -> 55432 to coexist with other local Postgres instances (platform-api-db). Container-internal port stays 5432 so api + jobs-runner are unaffected. - restore-dump.sh: filter pattern catches Aurora's "SCHEMA - public rdsadmin" entry which vanilla postgres rejects with "schema public already exists". - upgrade-hop.sh: poll both api and jobs-runner logs for migration completion; jobs-runner does not log the "up to date" line in newer versions. Also accept "Database migrations are up to date" alongside "migration complete" for no-op baseline hops. - validate.sh: health check retries up to 60s for the HTTP listener to catch up with the migration-complete log. Migration count now sums SequelizeMeta + knex_migrations so the monotonic invariant holds across the upgrade walk (Retool migrated trackers between versions). - RUNBOOK.md: documents the 55432 host port.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Csd 251708
add terraform apply workflows
Csd 251708
Staging login fails with redirect_uri_mismatch because staging and production shared one Google OAuth client whose redirect URIs we can no longer manage. Point the staging workspace at its own OAuth client. - Add `staging_client_id` variable for the new staging OAuth app - Select the client ID by workspace: production keeps `client_id`, staging uses `staging_client_id` The matching client secret is provisioned out-of-band in SSM at /overwatch/staging/GOOGLE_OAUTH2_SSO_CLIENT_SECRET (read-only data source; not managed by this repo). Production is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #5 removed module.apideck_acm_certificate_east, but that cert (c3f1967e…, *.stagingapideck.com in us-east-1) is still in use, so DeleteCertificate fails with ResourceInUseException and hangs every apply for ~15min before erroring. Re-add the block so Terraform keeps managing it instead of trying to delete it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix: separate Google SSO OAuth client for staging
Add prod Retool upgrade runbook + fix local-dev walk issues
# Conflicts: # modules/aws_ecs_fargate/certificate.tf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.