Skip to content

cmd: add tester migrate and make startup migrations optional for a PRE_DEPLOY job (MARSOHS-1643) - #13

Merged
ElanHasson merged 1 commit into
pgx-v5from
predeploy-migrate
Sep 21, 2026
Merged

ElanHasson merged 1 commit into
pgx-v5from
predeploy-migrate

Conversation

@ElanHasson

@ElanHasson ElanHasson commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #15 (pgx v5 / tern v2 upgrade) — merge that first; this PR's diff against main includes it until then.

Story: MARSOHS-1643 (under epic MARSOHS-1585).

Motivation

tester serve runs the tern migrations at startup, i.e. inside the web container's health-check window on App Platform:

  • a slow migration (e.g. an index build on the 15 GB tests table) makes the deploy look unhealthy and roll back mid-migration;
  • multiple web replicas serialize on tern's advisory lock;
  • the worker component rolls independently, so it can start against a not-yet-migrated schema.

A PRE_DEPLOY job runs once, to completion, before any component of the new deployment starts.

What changed

  • tester migrate (cmd/tester/migrate.go): connects with --pg-dsn / MIGRATE_PG_DSN, falling back to SERVE_PG_DSN so the job can reuse the web service's env verbatim; applies pending migrations; exits 0 (non-zero on failure). It does not read config.json — the DSN is the only thing it needs (in this repo the DSN is a serve flag/env, not a config-file field).
  • Migration logging: Migrator.OnStart is wired in (*db.PG).migrator, so both migrate and serve log every migration they apply (db: applying migration 5 "..." (up)).
  • serve --migrate-on-start flag / SERVE_MIGRATE_ON_START env (bool, default true, so current behaviour is unchanged; same pattern as --pg-dsn / SERVE_PG_DSN). When false, serve skips Init and calls the new (*db.PG).CheckSchema, which compares tern's current version to len(pgMigrations) and fails with database schema is at version N, this binary expects M: run tester migrate`` if behind — so a misordered deploy fails fast rather than serving a stale schema. A newer schema is tolerated so the image can be rolled back. It is a flag rather than a config.json field so the `web` component can be switched over with an app-spec edit, no e2e image rebuild. `serve` uses the concrete `*db.PG`, so the `db.DB` interface and `db_mock.go` are unchanged.
  • README: documents tester migrate, --migrate-on-start, an App Platform PRE_DEPLOY snippet, and the rollout order.
  • Tests: TestPG_CheckSchema (current → rolled back one migration → re-migrated → ahead) against real Postgres; TestServe_migrateOnStart (default true, env overrides default, flag overrides env).

Rollout order

  1. Deploy an image containing this change with --migrate-on-start at its default (true). No behaviour change.
  2. Add the PRE_DEPLOY job to the e2e app spec. Both the job and the server now migrate; the server's is a no-op.
  3. On the web component, set SERVE_MIGRATE_ON_START=false in envs (or run_command: tester serve --migrate-on-start=false ...). No image rebuild. The server now only verifies the schema.

App spec snippet (goes in digitalocean/e2e, separate manual step, not in this PR)

jobs:
  - name: migrate
    kind: PRE_DEPLOY
    image:                    # same image reference as the web/worker components
      registry_type: DOCR
      registry: do-e2e-canaries
      repository: <e2e image repository>
      tag: <same tag as web/worker>
    run_command: tester migrate
    envs:
      - key: SERVE_PG_DSN     # same DB env as web; MIGRATE_PG_DSN also works
        scope: RUN_TIME
        value: ${db.DATABASE_URL}

The binary is /bin/tester in the image (Dockerfile), invoked the same way serve/run are today.

Verification

gofmt -l ., go vet ./..., go generate ./... && git diff --exit-code db/db_mock.go, PG_DSN=... go test -race ./... all pass locally against postgres:16. Also exercised the built binary: serve --migrate-on-start=false and SERVE_MIGRATE_ON_START=false serve against an empty DB both exit 1 with the version message; migrate (via SERVE_PG_DSN) applies 5 migrations and exits 0; serve with the env set then starts; default serve on a fresh DB applies the 5 migrations itself.

Comment thread cmd/tester/migrate.go Outdated
"log"

"github.com/digitalocean/tester/db"
"github.com/jackc/pgx/v4/pgxpool"

@semgrep-code-digitalocean semgrep-code-digitalocean Bot Sep 21, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use github.com/jackc/pgx/v5 or newer instead of other postgres driver libraries

🥳 Fixed in commit f251d49 🥳

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by upgrading the whole repo to github.com/jackc/pgx/v5 + github.com/jackc/tern/v2 in #15, which this PR is now stacked on. cmd/tester/migrate.go imports github.com/jackc/pgx/v5/pgxpool as of f251d49; pgx/v4, jackc/pgconn, tern v1 and lib/pq are gone from go.mod.

@ElanHasson
ElanHasson marked this pull request as ready for review September 21, 2026 19:34
@ElanHasson ElanHasson changed the title cmd: add tester migrate and make startup migrations optional for a PRE_DEPLOY job (MARSOHS-1479) cmd: add tester migrate and make startup migrations optional for a PRE_DEPLOY job (MARSOHS-1643) Sep 21, 2026
@ElanHasson
ElanHasson marked this pull request as draft September 21, 2026 19:38
ElanHasson added a commit that referenced this pull request Sep 21, 2026
Semgrep flags any new file importing github.com/jackc/pgx/v4 ("Use
github.com/jackc/pgx/v5 or newer instead of other postgres driver
libraries"), which blocks #13 (cmd/tester/migrate.go). Move the whole repo
to pgx v5 and tern v2 rather than suppressing the finding.

API changes:

- github.com/jackc/pgx/v4 -> /v5, pgxpool and pgconn likewise
  (github.com/jackc/pgconn is now github.com/jackc/pgx/v5/pgconn).
- github.com/jackc/tern/migrate -> github.com/jackc/tern/v2/migrate.
  NewMigrator, AppendMigration, Migrate, MigrateTo, GetCurrentVersion and
  OnStart are unchanged in shape; v2 just takes a v5 *pgx.Conn.
- pgxpool.Connect is gone; pgxpool.New connects lazily, so serve (and the
  test harness) now Ping after constructing the pool to keep a bad DSN
  failing at startup.
- github.com/lib/pq dropped: pq.Array was only used for runs.args
  (varchar[]), which v5 encodes and scans as []string natively.
- pgx v5 encodes a nil slice/pointer for a json/jsonb parameter as SQL NULL
  where v4 sent JSON null. tests.result, tests.logs and runs.meta are jsonb
  NOT NULL and the runner produces nil Logs for a test with no output, so
  those columns are now pre-marshalled with encoding/json (db.jsonb) to
  keep the v4 wire behaviour. TestPG_AddTest_NoLogs pins this; without the
  helper TestPG_ListRunSummariesInRange fails with the NOT NULL violation.

Indirect jackc modules (pgtype, pgproto3, pgio, chunkreader, puddle v1) drop
out of go.mod. .github/dependabot.yml's github.com/jackc/* group already
covers the v5/v2 paths.

Co-authored-by: Cursor <cursoragent@cursor.com>
…PRE_DEPLOY job (MARSOHS-1643)

`tester serve` runs the tern migrations at startup, which means they run
inside the web container's health-check window on App Platform. A slow
migration (e.g. an index build on the 15 GB tests table) makes the deploy
look unhealthy and roll back mid-migration; multiple web replicas serialize
on the advisory lock; and the worker component rolls independently, so it
can start against a not-yet-migrated schema. A PRE_DEPLOY job runs once, to
completion, before any component of the new deployment starts.

- `tester migrate`: connects with --pg-dsn / MIGRATE_PG_DSN (falling back to
  SERVE_PG_DSN so the job can reuse the web env), applies pending
  migrations, exits 0, non-zero on failure.
- Every migration tern applies is now logged (Migrator.OnStart), for both
  `migrate` and `serve`.
- New `serve --migrate-on-start` flag / SERVE_MIGRATE_ON_START env (default
  true, preserving current behaviour). When false, `serve` skips migrating
  and instead calls the new `(*db.PG).CheckSchema`, exiting non-zero with
  the current and expected versions if the schema is behind, so a misordered
  deploy fails fast rather than serving a stale schema. A newer schema is
  tolerated so the image can be rolled back. `serve` uses the concrete
  *db.PG, so the db.DB interface and mock are unchanged. A flag rather than
  a config.json field so the web component can be switched over with an
  app-spec env edit, without rebuilding the e2e image.
- README: document `tester migrate`, `--migrate-on-start`, an App Platform
  PRE_DEPLOY job snippet and the rollout order.
- Tests: TestPG_CheckSchema (current / behind / ahead) and
  TestServe_migrateOnStart (default / env / flag precedence).

The app-spec change in digitalocean/e2e is a separate, manual step.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ElanHasson
ElanHasson changed the base branch from main to pgx-v5 September 21, 2026 19:55
@ElanHasson
ElanHasson marked this pull request as ready for review September 21, 2026 20:01
@ElanHasson
ElanHasson merged commit 06b3c67 into pgx-v5 Sep 21, 2026
3 of 5 checks passed
@ElanHasson
ElanHasson deleted the predeploy-migrate branch September 21, 2026 20:01
ElanHasson added a commit that referenced this pull request Sep 21, 2026
* deps: upgrade to pgx v5 and tern v2 (MARSOHS-1643)

Semgrep flags any new file importing github.com/jackc/pgx/v4 ("Use
github.com/jackc/pgx/v5 or newer instead of other postgres driver
libraries"), which blocks #13 (cmd/tester/migrate.go). Move the whole repo
to pgx v5 and tern v2 rather than suppressing the finding.

API changes:

- github.com/jackc/pgx/v4 -> /v5, pgxpool and pgconn likewise
  (github.com/jackc/pgconn is now github.com/jackc/pgx/v5/pgconn).
- github.com/jackc/tern/migrate -> github.com/jackc/tern/v2/migrate.
  NewMigrator, AppendMigration, Migrate, MigrateTo, GetCurrentVersion and
  OnStart are unchanged in shape; v2 just takes a v5 *pgx.Conn.
- pgxpool.Connect is gone; pgxpool.New connects lazily, so serve (and the
  test harness) now Ping after constructing the pool to keep a bad DSN
  failing at startup.
- github.com/lib/pq dropped: pq.Array was only used for runs.args
  (varchar[]), which v5 encodes and scans as []string natively.
- pgx v5 encodes a nil slice/pointer for a json/jsonb parameter as SQL NULL
  where v4 sent JSON null. tests.result, tests.logs and runs.meta are jsonb
  NOT NULL and the runner produces nil Logs for a test with no output, so
  those columns are now pre-marshalled with encoding/json (db.jsonb) to
  keep the v4 wire behaviour. TestPG_AddTest_NoLogs pins this; without the
  helper TestPG_ListRunSummariesInRange fails with the NOT NULL violation.

Indirect jackc modules (pgtype, pgproto3, pgio, chunkreader, puddle v1) drop
out of go.mod. .github/dependabot.yml's github.com/jackc/* group already
covers the v5/v2 paths.

Co-authored-by: Cursor <cursoragent@cursor.com>

* cmd: add `tester migrate` and make startup migrations optional for a PRE_DEPLOY job (MARSOHS-1643) (#13)

`tester serve` runs the tern migrations at startup, which means they run
inside the web container's health-check window on App Platform. A slow
migration (e.g. an index build on the 15 GB tests table) makes the deploy
look unhealthy and roll back mid-migration; multiple web replicas serialize
on the advisory lock; and the worker component rolls independently, so it
can start against a not-yet-migrated schema. A PRE_DEPLOY job runs once, to
completion, before any component of the new deployment starts.

- `tester migrate`: connects with --pg-dsn / MIGRATE_PG_DSN (falling back to
  SERVE_PG_DSN so the job can reuse the web env), applies pending
  migrations, exits 0, non-zero on failure.
- Every migration tern applies is now logged (Migrator.OnStart), for both
  `migrate` and `serve`.
- New `serve --migrate-on-start` flag / SERVE_MIGRATE_ON_START env (default
  true, preserving current behaviour). When false, `serve` skips migrating
  and instead calls the new `(*db.PG).CheckSchema`, exiting non-zero with
  the current and expected versions if the schema is behind, so a misordered
  deploy fails fast rather than serving a stale schema. A newer schema is
  tolerated so the image can be rolled back. `serve` uses the concrete
  *db.PG, so the db.DB interface and mock are unchanged. A flag rather than
  a config.json field so the web component can be switched over with an
  app-spec env edit, without rebuilding the e2e image.
- README: document `tester migrate`, `--migrate-on-start`, an App Platform
  PRE_DEPLOY job snippet and the rollout order.
- Tests: TestPG_CheckSchema (current / behind / ahead) and
  TestServe_migrateOnStart (default / env / flag precedence).

The app-spec change in digitalocean/e2e is a separate, manual step.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant