Is there a rate limit? Getting transient empty responses with many requests.

[Aariq]

When performing many requests (~2000) to the /awards/ endpoints in succession I get many of them failed ( ~1000) with nothing returned from the server (no headers, no data). But then if I re-run those failed requests one by one manually, they work fine. I feel like I must be hitting some kind of rate limit, but I don't see one documented anywhere. I'm currently throttling to 100 requests / 30 seconds with a maximum of 5 requests in parallel, but still having this issue.

An example failed request that worked on the second try: https://api.usaspending.gov/api/v2/awards/ASST_NON_2331010_4900/

[aguest-kc]

There is a global rate limit that applies to most endpoints of 1,000 request in 300 seconds (some endpoints are further rate limited). For the ~1,000 request that returned nothing (no headers, no data) do you know what status code you received?

[Aariq]

I'm using httr2 and I just get nothing

Error in `req_perform_parallel()`:
! Failed to perform HTTP request.
Caused by error:
! Server returned nothing (no headers, no data) [api.usaspending.gov]:
Empty reply from server

If I try to ask for the status code I just get an error. I'm not exactly sure what would cause this, but perhaps it isn't waiting long enough to get a response or has something to do with parallel requests. Currently the max number of concurrent requests is 10.

A few give 404 errors, but those seem to be correct, e.g. https://api.usaspending.gov/api/v2/awards/ASST_NON_2502740_4900/

[Aariq]

Even if I use sequential requests, I still get some of these empty reply errors

[nburt]

I think I'm having the same issue. I'm trying to pull in historical bulk data across 15 agencies for contracts and assistance. I'm using the https://api.usaspending.gov/api/v2/bulk_download/list_monthly_files/ endpoint to get the filename and then the https://files.usaspending.gov/award_data_archive endpoint to download it. I'm going back to 2010 so by my estimate I'd be making 960 total api calls. Are these endpoints some of the ones which are further rate limited more than the 1000 requests per 300 seconds?

I'm using the Python requests library and getting lots of connection related errors, the first two don't have a status code, the 3rd is a 500 with an HTML response.

Connection broken: ConnectionResetError(104, 'Connection reset by peer')

('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Failed to download file FY2025_070_Assistance_Full_20251206.zip: 500 - <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"...

I'm running each combination of agency / year / award type as a separate job and running a max of 3 in parallel. I have exponential backoff on failure every 5, 10, 20, 40 minutes.

Once I can get the full import working I'll be switching to delta files which will reduce the total number of jobs. For now should I be running everything sequentially or is there another strategy I should be using to avoid either rate limiting or overwhelming the server?