I've been trying to use datagovindia to fetch data from a resource (60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7) but hit two issues:
Issue 1: KeyError when API times out
When the data.gov.in API returns a 504 or similar error, get_data() crashes:
Traceback (most recent call last):
File "test.py", line 9, in <module>
data = client.get_data("60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7")
File ".../datagovindia/__init__.py", line 509, in get_data
limit = self.get_resource_info(resource_id)["total"]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'total'
The problem is on line 509 where it tries to access ["total"] but get_resource_info() returns {} when the API fails (via the retry logic in get_api_info()).
This makes the library unusable whenever the API is temporarily unstable.
Issue 2: Pickle error in Python 3.13
Even when I specify a limit to avoid the first issue, I get a different error in Python 3.13:
multiprocessing.pool.MaybeEncodingError: Error sending result: '...'.
Reason: 'TypeError("cannot pickle '_thread.RLock' object")'
This happens because the @retry decorator on make_request_with_retry() creates unpicklable state (an RLock), and then pool.map() tries to serialize it when sending results back from worker processes.
Test case
from datagovindia import DataGovIndia
client = DataGovIndia()
# Either of these will crash:
data = client.get_data("60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7") # KeyError
data = client.get_data("60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7", limit=10) # Pickle error (Python 3.13)
Possible fixes
For issue 1: Check if resource_info is empty or doesn't contain "total" before trying to access it.
For issue 2: Create a non-decorated wrapper function for make_request_with_retry() that gets passed to the multiprocessing pool, or disable multiprocessing by default.
Happy to submit a PR if you're interested.
I've been trying to use datagovindia to fetch data from a resource (
60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7) but hit two issues:Issue 1: KeyError when API times out
When the data.gov.in API returns a 504 or similar error,
get_data()crashes:The problem is on line 509 where it tries to access
["total"]butget_resource_info()returns{}when the API fails (via the retry logic inget_api_info()).This makes the library unusable whenever the API is temporarily unstable.
Issue 2: Pickle error in Python 3.13
Even when I specify a limit to avoid the first issue, I get a different error in Python 3.13:
This happens because the
@retrydecorator onmake_request_with_retry()creates unpicklable state (an RLock), and thenpool.map()tries to serialize it when sending results back from worker processes.Test case
Possible fixes
For issue 1: Check if
resource_infois empty or doesn't contain "total" before trying to access it.For issue 2: Create a non-decorated wrapper function for
make_request_with_retry()that gets passed to the multiprocessing pool, or disable multiprocessing by default.Happy to submit a PR if you're interested.