Bound the PDF cleanup outbox and refresh runtime dependencies - #10
Open
Mavengence wants to merge 1 commit into
Open
Bound the PDF cleanup outbox and refresh runtime dependencies#10Mavengence wants to merge 1 commit into
Mavengence wants to merge 1 commit into
Conversation
Durable-cleanup reliability - Failed R2 deletes now back off exponentially (1, 2, 4, 8, 16, 32 minutes, capped at one hour) instead of being re-claimed on every worker cycle. The queue row is never dropped: it exists because a document or account was erased, so the object must eventually leave storage, and discarding the row would orphan it silently. - The in-request drain on library reads is bounded to three objects and at most once per minute per process, so object-store latency no longer scales with reads of the CV and cover-letter lists. Container - Refuse the gunicorn 26 control socket explicitly. It defaults to $HOME/.gunicorn, which the read-only rootfs already rejects — loudly, on every boot, and only incidentally. --no-control-socket makes the absence intentional. Dependencies - gunicorn 23.0.0 -> 26.0.0, boto3/botocore 1.43.40 -> 1.43.65, and declared floors raised for supabase and python-dotenv. Locks recompiled per package so nothing unrelated moved. Verification - tools/db/migration_probes.sql gains four executed probes for the backoff: the first failure defers the row, the delay grows, it stays capped, and the row survives. - The gunicorn bump was verified by booting the real image: clean startup log and 200 responses on /healthz, / and /app.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Durable-cleanup reliability
Retry backoff.
record_pdf_artifact_cleanup_failureincrementedattempt_countbut never advancednot_before, while the worker selects onnot_before <= now(). A row whose object delete failed permanently was therefore re-claimed every cycle — a hot loop of failing network calls with no signal that it needed attention. Failures now back off exponentially (1, 2, 4, 8, 16, 32 minutes) capped at one hour.The row is deliberately never dropped: it exists because a user deleted a document or erased an account, so the object must eventually leave storage. Discarding it would orphan the object permanently and silently.
Read-path drain.
GET /api/cvsandGET /api/cover-lettersdrained up to ten queue rows inline, one object round-trip each. Now bounded to three objects, at most once per minute per process, with the timestamp stamped before the attempt so a failing backend is not retried on every subsequent read.Container
gunicorn 26 adds a management socket defaulting to
$HOME/.gunicorn/gunicorn.ctl. The read-only rootfs already refuses it — but loudly (anERRORon every boot) and only incidentally.--no-control-socketmakes the refusal intentional.Dependencies
Locks recompiled with
--upgrade-packageso nothing unrelated moved.Verification
tools/db/migration_probes.sqlgains four executed probes for the backoff — first failure defers the row, delay grows with attempts, stays capped, row survives. Reverting the migration fails them withfirst failure did not defer the row: 00:00:00.Because gunicorn crosses three majors on the WSGI server, it was verified by booting the real image rather than by test suite alone: clean startup log, 200 on
/healthz,/and/app, no control socket present. That check is what surfaced the control-socket issue above.