Skip to content

Bug: get_data() crashes on API errors and Python 3.13 #9

Description

@parthashirolkar

I've been trying to use datagovindia to fetch data from a resource (60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7) but hit two issues:

Issue 1: KeyError when API times out

When the data.gov.in API returns a 504 or similar error, get_data() crashes:

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    data = client.get_data("60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7")
  File ".../datagovindia/__init__.py", line 509, in get_data
    limit = self.get_resource_info(resource_id)["total"]
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'total'

The problem is on line 509 where it tries to access ["total"] but get_resource_info() returns {} when the API fails (via the retry logic in get_api_info()).

This makes the library unusable whenever the API is temporarily unstable.

Issue 2: Pickle error in Python 3.13

Even when I specify a limit to avoid the first issue, I get a different error in Python 3.13:

multiprocessing.pool.MaybeEncodingError: Error sending result: '...'. 
Reason: 'TypeError("cannot pickle '_thread.RLock' object")'

This happens because the @retry decorator on make_request_with_retry() creates unpicklable state (an RLock), and then pool.map() tries to serialize it when sending results back from worker processes.

Test case

from datagovindia import DataGovIndia

client = DataGovIndia()

# Either of these will crash:
data = client.get_data("60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7")  # KeyError
data = client.get_data("60a68cec-7d1a-4e0e-a7eb-73ee1c7f29b7", limit=10)  # Pickle error (Python 3.13)

Possible fixes

For issue 1: Check if resource_info is empty or doesn't contain "total" before trying to access it.

For issue 2: Create a non-decorated wrapper function for make_request_with_retry() that gets passed to the multiprocessing pool, or disable multiprocessing by default.

Happy to submit a PR if you're interested.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions