fix(pm): hold a bulk send's attachments for the length of the cron run - #6417
Open
e107help[bot] wants to merge 1 commit into
Open
e107help[bot] wants to merge 1 commit into
e107help[bot] wants to merge 1 commit into
Conversation
The cron task took a queued bulk send off the generic table before private_message::add() inserted the messages it stood for. For the length of that run the recipients it had yet to reach held their attachment by neither a private_msg row nor a queue row, so a delete landing in the window read the file as unused and unlinked it, and the download those recipients were about to be handed was broken before they got it. The row is claimed now rather than removed: its gen_type moves to pm_bulk_running for the run and the row is dropped once add() returns. The claim is conditional on the row still being queued, so a run starting alongside another cannot take the same chunk, and the survivor check reads both values, so a claimed row still speaks for what it owes. The survivor check reads the queue before it reads the messages, which closes the two-statement version of the same window: a run drops its claimed row only once the messages are in, so a delete that reads the queue first can miss neither. A run that dies part way through leaves its row claimed, which is the point of claiming rather than deleting afterwards: deleting after add() would turn a send lost to a crash into a send delivered twice. The cost is that the chunk stays undelivered and the attachments that row names stay held, so nothing will unlink them until the row is cleared. Ageing a claim out would deliver part of that chunk a second time, which is the same question again and not one this change answers. The messenger the run adds through comes from a protected factory so a test can watch the queue from inside the send, the queue read is protected so a test can finish a run between the survivor check's two reads, and the attachment fixtures both pm unit tests need move into a base they share. pm_cron also declares the property it has always written, which PHP 8.2 deprecates creating on the fly. Refs #6159
e107help
Bot
force-pushed
the
e107help/6159-bulk-queue
branch
from
September 13, 2026 21:12
0366d0f to
87e9ac1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A private message send over
pm_max_sendgoes out in chunks: the residue is sent now, and the rest of the recipients are serialised into thegenerictable for the cron task to deliver later. Those rows are the only thing saying an attachment is still owed to the members whose messages do not exist yet.The cron task deleted that row before it inserted anything (
e_cron.phpL141-L147). For the length of the run the recipients it had yet to reach held their attachment by neither aprivate_msgrow nor a queue row, so a delete landing in that window read the file as unused and unlinked it, and the download every one of those members was about to be handed was broken before they got it.@Deltik settled the shape of the fix on 2026-09-12: mark the queued row in progress for the run and remove it after, rather than deleting it first, and rather than simply moving the delete after the send, because that turns a send lost to a mid-run fatal into a send delivered twice.
Refs #6159
What Changed
gen_typemoves frompm_bulktopm_bulk_runningfor the length of the run, and the row is deleted onceadd()returns.pm_class.phpL284-L294 is the query that widens).private_msgnow, which closes the two-statement version of the same window: a run drops its claimed row only once the messages are in the table, so a delete that reads the queue first can miss neither (pm_class.phpL243-L268 is the check as it stood).pm_crondeclares the property it has always written. Creating it on the fly is deprecated from PHP 8.2, and the new tests are the first thing in the unit suite to reach that line.plugin.xmltakes the date of this commit. The version stays at 3.1, which was already bumped after the last tag.What the survivor check costs on a large site, a full scan of
private_msgper attachment and a read of every queued chunk, is #6420. It predates this PR and its fix is a schema change, so it is filed rather than folded in.How It Was Tested
New unit coverage in
e107_tests/tests/unit/plugins/pm/pm_e_cronTest.php, on the double that puts both attachment trees somewhere disposable: a delete landing inside a bulk run keeps the attachment the queue still owes and the queued row goes once the messages are in; a run that dies part way through keeps its claim, and the next run does not send that chunk again; two runs cannot claim the same send; agenericrow of another type is not a send to claim and is left as it stands. That last one is what reds if the claim stops reading the row's type.The read order has its own case in
private_messageAttachmentDeletionTest.php: a bulk run finishes between the survivor check's two reads, and the attachment survives. Reverting the whole file cannot express that mutation, because the seam the test needs is itself part of the change, so it was proven by swapping the two reads by hand: that case fails on its own and the other five stay green.Run on PHP 8.5 with MariaDB 10.11: the cron cases are green with the fix, red without it, green again with it restored. The whole unit suite, PHP lint and the downgrade check pass. The acceptance suite did not run locally; CI is its first run and the first run of the full matrix.
Backwards Compatibility
Rendered HTML is untouched, no signature changes, and nothing outside the plugin reads or writes
gen_type='pm_bulk', so a row moving topm_bulk_runningfor the length of a run hides it from nothing that was looking. A queue row written by the current code and processed by the new code is read the same way, and the reverse holds too, so an upgrade mid-queue delivers what is waiting.One consequence is worth stating plainly, because it is the cost of the decision rather than an oversight. A run that dies part way through a chunk leaves its row claimed. The recipients in that chunk stay undelivered, as they did before, and now the attachments that row names stay held as well, so nothing will unlink them until the row is cleared. Ageing a claim out would deliver part of that chunk a second time, which is the same delivery question again, so this PR does not answer it. On a site where that happens, the row is visible in
genericasgen_type='pm_bulk_running'.AI Model
Claude Opus 5 (claude-opus-5), as e107help.
Checklist
masterat aa36836 before changing anything