load-backup-archives: restore collections asynchronously and verify completion#71
Open
OmarB97 wants to merge 9 commits into
Open
load-backup-archives: restore collections asynchronously and verify completion#71OmarB97 wants to merge 9 commits into
OmarB97 wants to merge 9 commits into
Conversation
…ompletion The synchronous Collections API RESTORE call hard-times-out after 180 seconds server-side, which large collections (release-group, url, recording) routinely exceed on modest hardware. The status check then accepted the error response anyway, because '=~ 0' is a substring match and "500" contains "0"; the extracted backup files were deleted from under the still-running background restore, leaving those collections silently empty (artist and release survived only by finishing under 180s). Submit the restore with async=<id>, poll REQUESTSTATUS until a terminal state, fail loudly on failed/notfound, and only then clean up the request status and the extracted files.
Contributor
|
@OmarB97: First of all, thanks a lot for this much needed fix. I will review it more in-depth later on this week. |
Printing to stderr seems to be more readable like that. Reference: https://www.shellcheck.net/wiki/SC2005
Beyond the three restore states (`completed`, `failed`, and `notfound`) explicitly handled already, two other states (`running` and `submitted`) are silently handled too. However, it doesn’t handle unexpected restore states due to either future changes in Solr or transient network issues. This patch makes handling all known restore status explicitly, to allow handling unexpected restore states too. It will also allow reporting progress in a separate amending commit. Note that when the output of `REQUESTSTATUS` doesn’t contain any `.status.state` key, then `jq` will return the `null` string. References: https://github.com/apache/solr/blob/releases/solr/9.7.0/solr/solrj/src/java/org/apache/solr/client/solrj/response/RequestStatusState.java#L28-L41 https://solr.apache.org/guide/solr/latest/configuration-guide/collections-api.html#requeststatus https://jqlang.org/manual/#object-identifier-index
While the synchronous requests were timing out too soon due to a hard server-side time limit of 180s, it is still safer to have a time limit for the asynchronous requests, preferably client-side and customizable. This patch limits the number of polling to 360. That number can be changed through the environment variable `MUSICBRAINZ_SEARCH_MAX_POLL`. With 10s between each polling, it will stop after one hour at least.
Clarify that this check is a string comparison with the right-hand side.
This comment is in git commit message already. It is about previous bugged code only.
To better avoid conflicts with stalled requests from previous attempts, this patch adds the current timestamp to the asynchronous request ID. It also prefixes each number with a descriptive abbreviation.
A delete status failure is most likely not blocking, for example if the asynchronous request ID already expired. Still, it is worth warning about it to help with debugging in case of serious issues. Reference: https://solr.apache.org/guide/solr/9_7/configuration-guide/collections-api.html#deletestatus
Restoring large collections such as recordings can take several minutes. To help with monitoring the progress during that time, this patch logs a progress message about every minute, actually every 6 polling attempts.
yvanzo
approved these changes
Jul 1, 2026
yvanzo
left a comment
yvanzo
left a comment
Contributor
There was a problem hiding this comment.
Sorry that it took a couple more weeks.
I amended your work with a few changes, including:
- Handling of unexpected restore status states
- Customizable client-side timeout to abort the whole process when the server isn’t restoring a collection in the expected time, set to 1h for now. (It does take 6m on our test server for the
recordingcollection.)
I tested these changes by reloading backup archives in a local mirror.
Now just requesting a peer-review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
load-backup-archivescan report success while leaving large collections silently empty. Three compounding issues:RESTOREcall hard-times-out server-side after 180 seconds (restore the collection time out:180s, SolrException, HTTP 500). Large collections —release-group,url, and especiallyrecording— routinely exceed that on modest hardware;artist/releaseonly survive when they finish under the limit.[[ ! "$(... | jq .responseHeader.status)" =~ 0 ]]is a substring match, and"500"contains"0", so the timeout error response passes the check.Observed live on a musicbrainz-docker deployment (2026-06): after a full
load-backup-archivesrun that logged success for all 14 collections,release-groupandurlhadnumDocs=0whileartist(2.89M) andrelease(5.55M) were fine. Re-running those two restores with&async=+REQUESTSTATUSpolling loaded them correctly (4.33M and 19.6M docs).Fix
Submit the restore with
async=<request-id>, pollREQUESTSTATUSevery 10s until a terminal state, fail loudly onfailed/notfound, and only aftercompletedclean up the request status and the extracted backup files. The submit-time status check is now an exact comparison.Happy to adjust style/poll cadence to project preferences.