After all SSE batch-processing retries fail (502/503 from https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2), the recovery flow proceeds with first_offset_in_batch = None instead of a numeric offset. The f-string at src/qlever/commands/update_wikidata.py:1021 accepts None silently and writes a file literally named update.None.1.sparql, which is then POSTed to the SPARQL endpoint. Later, the cleanup pass at :1080-1083 globs update.*.sparql, parses the middle component as the offset, and sorts via int(x) — which raises ValueError: invalid literal for int() with base 10: 'None' and aborts the command.
Different from #275 / commit 6b6d430 (which made the SSE iteration tolerate disconnects); this is a downstream bug in the post-retry recovery path.
Log:
SSE stream connection lost (fetch https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2 failed with wrong response status: 502), will reconnect ...
Consuming stream from date: None
SSE stream connection for batch processing failed (attempt 1/10): ... 502. Retrying in 5s ...
SSE stream connection for batch processing failed (attempt 2/10): ... 502. Retrying in 10s ...
SSE stream connection for batch processing failed (attempt 3/10): ... 503. Retrying in 30s ...
SSE stream connection for batch processing failed (attempt 4/10): ... 502. Retrying in 1min ...
Encountered message with date 2026-06-01T16:54:58Z, which is within 1 second of the current time, finishing the current batch
Assembled batch #7632, #messages: 1, date range: 2026-06-01T16:54:58Z - 2026-06-01T16:54:58Z [assembly time: 15,728ms, min delta to NOW: 0.6s]
curl -s -X POST "http://tajo:7001?access-token=wikidata_AhnTHyQTrTPD" -H 'Content-Type: application/sparql-update' --data-binary @update.None.1.sparql
An unexpected error occurred: invalid literal for int() with base 10: 'None'
Traceback (most recent call last):
File "/local/data-ssd/qlever/qlever-control/src/qlever/qlever_main.py", line 38, in main
command_successful = command_object.execute(args)
File "/local/data-ssd/qlever/qlever-control/src/qlever/commands/update_wikidata.py", line 1081, in execute
sorted_offsets = sorted(
File "/local/data-ssd/qlever/qlever-control/src/qlever/commands/update_wikidata.py", line 1082, in <lambda>
update_files.keys(), key=lambda x: int(x)
ValueError: invalid literal for int() with base 10: 'None'
Note the line Consuming stream from date: None just before the failure — that is the recovery path with no resume point, which is where first_offset_in_batch ends up unset.
Suggested fix
Two surfaces, both worth addressing:
- Don't let
first_offset_in_batch reach the filename construction as None. Assert/raise (or take a different code path) in the recovery branch that leaves the offset unknown, so the bad filename is never written in the first place.
- Defensively, the cleanup glob at
:1080-1083 should skip filenames whose middle component isn't a valid integer, rather than crash the whole command on one stale file. Even after (1) is fixed, an existing update.None.*.sparql from a prior run would re-trigger the crash on the next invocation.
After all SSE batch-processing retries fail (502/503 from
https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2), the recovery flow proceeds withfirst_offset_in_batch = Noneinstead of a numeric offset. The f-string atsrc/qlever/commands/update_wikidata.py:1021acceptsNonesilently and writes a file literally namedupdate.None.1.sparql, which is then POSTed to the SPARQL endpoint. Later, the cleanup pass at:1080-1083globsupdate.*.sparql, parses the middle component as the offset, and sorts viaint(x)— which raisesValueError: invalid literal for int() with base 10: 'None'and aborts the command.Different from #275 / commit 6b6d430 (which made the SSE iteration tolerate disconnects); this is a downstream bug in the post-retry recovery path.
Log:
Note the line
Consuming stream from date: Nonejust before the failure — that is the recovery path with no resume point, which is wherefirst_offset_in_batchends up unset.Suggested fix
Two surfaces, both worth addressing:
first_offset_in_batchreach the filename construction asNone. Assert/raise (or take a different code path) in the recovery branch that leaves the offset unknown, so the bad filename is never written in the first place.:1080-1083should skip filenames whose middle component isn't a valid integer, rather than crash the whole command on one stale file. Even after (1) is fixed, an existingupdate.None.*.sparqlfrom a prior run would re-trigger the crash on the next invocation.