Handle pagination

[aereaux]

Hello,

It currently looks like this library doesn't handle API pagination. For example, if I do a lookupCollectionContents for a collection that has more than 25 elements then I only get the first 25 elements. Is this something that can be added, or is there something I am doing wrong here?

Thanks!

[kellnerd]

Hi, the client is still very incomplete. Basically it only has typed functions for lookup requests.
Pagination will be added together with the browse and search requests which need them.

The lookupCollectionContents function is kind of an outlier, the API endpoint is a version of the collection lookup with an extra browse query. While it would be possible to specify pagination parameters (and even include parameters) here, the API docs don't recommend this:

To get the description and the summarized contents of a collection, perform a lookup request with the collection MBID and the appropriate entity subquery:

 GET /ws/2/collection/f4784850-3844-11e0-9e42-0800200c9a66/releases

To get the contents of a collection, perform a browse request on the appropriate entity endpoint, using the collection MBID as a parameter.

So I will probably even deprecate that method once typed browse requests are supported. After all it would return the redundant basic collection properties for every page of contents.

When I last looked at adding functions for those, there were a couple of annoying inconsistencies in the API format... A few of those have been fixed on the MBS side, so hopefully it is no longer necessary to have separate type definitions for browse requests and the client can reuse the lookup type definitions now.

[aereaux]

Thanks for the response! Maybe I am asking the wrong question. What I am trying to do with this library is get the contents of a release collection, with the medium formats for each release. I used to do this with mb.get("release", {collection: mbid, inc: "media+release-groups", limit: 10000})), but recently the API has started limiting this request to 100 in the limit parameter. Is there a better way to accomplish this with this library? I believe that I'm now required to use pagination for large collections.

[kellnerd]

recently the API has started limiting this request to 100 in the limit parameter. Is there a better way to accomplish this with this library? I believe that I'm now required to use pagination for large collections.

Yes, that's correct. Your get call is the same thing as the planned browse function.
It currently requires manual pagination, but I will probably add a pagination helper to the library once I get to those functions where it is useful.

[aereaux]

Thanks for the info. I implemented some naive pagination myself, but now I'm running into network ratelimiting errors. I believe this library should handle rate limiting, right? Is there something I need to do to activate that? I create an array of promises and then wait on them all at once, and it looks like all the network requests get sent simultaneously.

[kellnerd]

What kind of errors are you running into? HTTP errors or "Connection reset" errors?
The first should be handled by the library, even when you are awaiting an array of promises (unless I've messed up something), the second is not yet handled: #6

[aereaux]

I am getting this:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://musicbrainz.org/ws/2/release?collection=c3759587-4182-4a0d-83f9-37a884b75b41&inc=media%2Brelease-groups&offset=100&limit=100. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 503. 
Uncaught (in promise) TypeError: NetworkError when attempting to fetch resource. 

It looks like the rate-limit errors don't provide CORS headers, because this is what the actual HTTP response is:

{"error": "Your requests are exceeding the allowable rate limit. Please see http://wiki.musicbrainz.org/XMLWebService for more information."}

[kellnerd]

Interesting, I haven't seen that issue since I haven't yet used the library in a browser for more than a single request. For my request-heavy server-side apps CORS isn't an issue of course.

But glad to see that it can be fixed on the server side: https://tickets.metabrainz.org/browse/MBH-593

[aereaux]

It looks like this fix has been applied, but the requests are still getting sent all at once, and I am still getting rate limit errors. Do I have to handle rate limiting myself when using this library?

Uncaught (in promise) ApiError: Your requests are exceeding the allowable rate limit. Please see http://wiki.musicbrainz.org/XMLWebService for more information.