Skip to content

Bound the PDF cleanup outbox and refresh runtime dependencies - #10

Open
Mavengence wants to merge 1 commit into
mainfrom
fix/cleanup-backoff-and-deps
Open

Bound the PDF cleanup outbox and refresh runtime dependencies#10
Mavengence wants to merge 1 commit into
mainfrom
fix/cleanup-backoff-and-deps

Conversation

@Mavengence

Copy link
Copy Markdown
Contributor

Durable-cleanup reliability

Retry backoff. record_pdf_artifact_cleanup_failure incremented attempt_count but never advanced not_before, while the worker selects on not_before <= now(). A row whose object delete failed permanently was therefore re-claimed every cycle — a hot loop of failing network calls with no signal that it needed attention. Failures now back off exponentially (1, 2, 4, 8, 16, 32 minutes) capped at one hour.

The row is deliberately never dropped: it exists because a user deleted a document or erased an account, so the object must eventually leave storage. Discarding it would orphan the object permanently and silently.

Read-path drain. GET /api/cvs and GET /api/cover-letters drained up to ten queue rows inline, one object round-trip each. Now bounded to three objects, at most once per minute per process, with the timestamp stamped before the attempt so a failing backend is not retried on every subsequent read.

Container

gunicorn 26 adds a management socket defaulting to $HOME/.gunicorn/gunicorn.ctl. The read-only rootfs already refuses it — but loudly (an ERROR on every boot) and only incidentally. --no-control-socket makes the refusal intentional.

Dependencies

package change
gunicorn 23.0.0 → 26.0.0
boto3 / botocore 1.43.40 → 1.43.65
supabase, python-dotenv declared floors raised; lock already satisfied

Locks recompiled with --upgrade-package so nothing unrelated moved.

Verification

tools/db/migration_probes.sql gains four executed probes for the backoff — first failure defers the row, delay grows with attempts, stays capped, row survives. Reverting the migration fails them with first failure did not defer the row: 00:00:00.

Because gunicorn crosses three majors on the WSGI server, it was verified by booting the real image rather than by test suite alone: clean startup log, 200 on /healthz, / and /app, no control socket present. That check is what surfaced the control-socket issue above.

Durable-cleanup reliability
- Failed R2 deletes now back off exponentially (1, 2, 4, 8, 16, 32 minutes,
  capped at one hour) instead of being re-claimed on every worker cycle. The
  queue row is never dropped: it exists because a document or account was
  erased, so the object must eventually leave storage, and discarding the row
  would orphan it silently.
- The in-request drain on library reads is bounded to three objects and at
  most once per minute per process, so object-store latency no longer scales
  with reads of the CV and cover-letter lists.

Container
- Refuse the gunicorn 26 control socket explicitly. It defaults to
  $HOME/.gunicorn, which the read-only rootfs already rejects — loudly, on
  every boot, and only incidentally. --no-control-socket makes the absence
  intentional.

Dependencies
- gunicorn 23.0.0 -> 26.0.0, boto3/botocore 1.43.40 -> 1.43.65, and declared
  floors raised for supabase and python-dotenv. Locks recompiled per package
  so nothing unrelated moved.

Verification
- tools/db/migration_probes.sql gains four executed probes for the backoff:
  the first failure defers the row, the delay grows, it stays capped, and the
  row survives.
- The gunicorn bump was verified by booting the real image: clean startup log
  and 200 responses on /healthz, / and /app.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant