Frequent "TypeError: Failed to fetch"

[nimuxoha]

Normally I open magicisrc from Harmony after adding new releases but very often when submitting ISRCs or just loading the page I get this error:
When fetching MusicBrainz release: TypeError: Failed to fetch

It's very frustrating having to repeatedly reload the page in hopes there won't be an error. I suspect it's related to the MB API rate-limit and me going too fast. Might there be a another solution?

[kepstin]

I doubt that rate limiting is the issue. I've tried on purpose to hit rate limits in magicisrc and i simply haven't been able to reload the page fast enough to cause any issues. Also, the requests that magicisrc makes to musicbrainz can be cached by the browser, which reduces issues.

If you're still able to reproduce this problem, I would appreciate a screenshot of the browser web inspector "Network" tab with the request to musicbrainz.org/ws/2/... selected so i can see what's going wrong with the request. That should show the response status code and headers.

[nimuxoha]

It happens mostly when I add many releases and open magicisrc for each one. I'll try to reproduce it with network logs at the weekend if possible.

[kepstin]

The musicbrainz api normally allows 1 request per second, but you can exceed that a bit temporarily as long as you stay with ~1/s on average. I suppose it might be possible to click extremely quickly and open multiple copies of magicisrc within a second, but it would be difficult to hit the rate limit otherwise unless you're unfortunate enough to have your ip address in a range which has been flagged for abuse and has a lower rate limit applied. (if this has happened to you, you might be able to contact musicbrainz to get this cleared).

I'd be interested in the network logs to determine what the exact cause of the error you're hitting is. If it is something recoverable like hitting the rate limit, i can look into adding code to automatically retry after waiting for the rate limit to reset - or i could at least display a more descriptive error message.

[nimuxoha]

Got something. It's a 503 Service Unavailable for the MB ws2 requests that failed.

I opened exactly 12 magicisrc tabs from Harmony. No login on magicisrc yet.
4 of them failed in total. The ones that failed indicated in response headers:

x-mb-rate-limiter: lua
x-ratelimit-zone: per-ip

The requests that succeeded looked like this:

x-cache-status: MISS
x-ratelimit-limit: 1200
x-ratelimit-remaining: 557
x-ratelimit-reset: 1744396474
x-runtime: 0.048336

zas explained these a few years ago:
https://community.metabrainz.org/t/what-do-the-x-ratelimit-headers-mean-and-how-should-they-be-used/259737/2

(if this has happened to you, you might be able to contact musicbrainz to get this cleared).

I'm doubtful that's the case for me as I'm quite an active editor and rarely hit the rate-limit on the MB website unless I literally open hundreds of tabs in quick succession. So this problem seems limited to the API. :(

[kepstin]

This is unfortunate :/ Good to know the specific status code is 503 at least.

Just to confirm - do the failed requests also the x-ratelimit-reset header, or only the succeeded ones? If the failed ones have that, I might be able to be a bit smarter about the retry timing when i hit an error.

Note that the numbers in x-ratelimit-* headers on the working request are actually for the global ratelimit (i.e. musicbrainz.org only allows a total of 1200 requests every 2 seconds for anyone regardless of ip). But the failed request you showed is from the "per-ip" zone, indicating that you've exceeded the "average 1 request per second" per-ip limit, and that's not something I've ever managed to do when testing :/

With the current implementation of magicisrc, I strongly recommend that you only open a single magicisrc tab at a time, and it might be helpful to wait for a second after the isrc submission completes before you open another magicisrc tab.

I definitely will look into adding some retry logic / rate limiting into magicisrc. This probably means that I'll have to show a "loading" screen for longer in some cases. I'm not sure whether or not I'll be able to coordinate this across multiple magicisrc tabs. It looks like there are some modern browser apis - e.g. the Web Locks API - which make this possible to do nowadays.

[nimuxoha]

Just to confirm - do the failed requests also the x-ratelimit-reset header, or only the succeeded ones? If the failed ones have that, I might be able to be a bit smarter about the retry timing when i hit an error.

No but there is a retry-after header with a number as value. So far I've seen ones with 0, 1 or 2

[kepstin]

I've done some initial work on error recovery and rate limiting which is now live on https://magicisrc-beta.kepstin.ca/ The approach I used is based on what MusicBrainz Picard does, which has automatic adaptation of the delay between requests, so hopefully it won't break if you're using MagicISRC at the same time as some other tool using the MusicBrainz API like Picard or beets. I've also improved the error display in case the request still fails even after being retried.

I was able to use Web Locks API to synchronize the rate limiter across multiple tabs, so multiple MagicISRC tabs will share a single rate limit.

There's still a bit more work that needs to be done - I still need to run the final ISRC submission through the rate limiter, and I need to verify that the code restructuring didn't break any templates.

[nimuxoha]

Awesome! So quick. I'll be using the beta from now on and see how it well it works.

Edit:
Can't even login on the beta :(

(index):171 New location: https://magicisrc-beta.kepstin.ca/
(index):177 poppedState null
(index):1100 Uncaught ReferenceError: doLogin is not defined
    at HTMLButtonElement.onclick ((index):1100:2)
onclick @ (index):1100

[kepstin]

Well, missed that in my testing - I happened to be already logged in :) I know what happened there, that's one of the template breakages that I was worried might happen. I'll go through and double-check all the variable references in templates soon.

[kepstin]

I believe I have corrected the remaining errors regarding template rendering; please let me know if you run into any other issues on the magicisrc-beta server.

[nimuxoha]

Still not much luck. Login works but upon submitting it shows the animation endlessly and logs an error:

Uncaught TypeError: Cannot read properties of undefined (reading '0') 
at buildISRCXML (?isrc1=CH6541778173&....1edb073dc5ab:651:40) 
at postISRCs (?isrc1=CH6541778173&...1edb073dc5ab:615:13) 
at doSubmitISRCs (?isrc1=CH6541778173&...-1edb073dc5ab:602:3) 
at HTMLButtonElement.onclick (?isrc1=CH6541778173&...b5-1edb073dc5ab:1:1)

[kepstin]

I'm still working on this… I hope to be able to finish and properly test some further updates this weekend.

[kepstin]

I've updated the https://magicisrc-beta.kepstin.ca/ site with a new version that has been more thoroughly tested - you should no longer be seeing submission errors. It now applies the rate limiting and retries on both loading releases and submitting ISRCs. Please test this out and let me know if it's working for you.

[nimuxoha]

I'm afraid it's still not fully working. Although it does claim "The ISRCs have been successfully submitted." the console shows an error and indeed no edit was created:

TypeError: Cannot read properties of undefined (reading 'recording')
at buildISRCMetadata (?isrc1=UKR6V2073434& .. 93f1a35aba3e:656:54)
at doSubmitISRCs (?isrc1=UKR6V2073434&93f1a35aba3e:624:22)
at HTMLButtonElement.onclick (?mbid=94d7117c-da6a -... cf-93f1a35aba3e:1:1)