TypeError: __init__() missing 1 required positional argument: 'history'

[dhulke]

I have a web crawling app that creates a buffer of about 500 documents and bulk saves them to couch. Whenever I'm getting too many page errors, all of a sudden, for some unknown reason, this error pops up. I don't understand why page load errors would interfere with bulk save since I only buffer documents that have been correctly loaded. Any idea why this is happening?

Traceback (most recent call last):
File "fetch.py", line 28, in
fetcher.fetch()
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 31, in fetch
loop.run_until_complete(self.run())
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", line 467, in run_until_complete
return future.result()
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 43, in run
await asyncio.gather(*all_jobs)
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 55, in cnpj_job
self.flush()
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 63, in flush
self.cnpjcouchdb.save_bulk(list(self.couch_buffer))
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\cnpjcouchdb.py", line 25, in save_bulk
return self.db.save_bulk(docs)
File "C:\Users\danilo\Projects\CNPJReceita\venv\lib\site-packages\pycouchdb\client.py", line 403, in save_bulk
_docs = copy.deepcopy(docs)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 274, in _reconstruct
y = func(*args)
TypeError: init() missing 1 required positional argument: 'history'

[histrio]

Hi, @dhulke !
Can you show what kind of object you trying to save as bulk? It seems to be some issues in object reconstruction.
We need to make it reproducible.
Thanks!

[dhulke]

I haven't been able to find a consistent way to reproduce this error. Some times it happens very frequently and other times it doesn't happen at all for over a week. All I have are simple json objects I'm retrieving through aiohttp response.json(). Some of the text has weird characters, but that isn't consistently causing this error. I tried looking for the reduce functions which are getting called at y = func(*args) but I haven't found a single one that requires a history attribute.
Is there any other way we could try and find the cause for this without being able to consistently reproduce it?

[dhulke]

Or if we at least knew what function expects the history attribute we could probably infer what the error is