Skip to content

update-wikidata crashes with int('None') after SSE batch retries fail #296

Description

@hannahbast

After all SSE batch-processing retries fail (502/503 from https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2), the recovery flow proceeds with first_offset_in_batch = None instead of a numeric offset. The f-string at src/qlever/commands/update_wikidata.py:1021 accepts None silently and writes a file literally named update.None.1.sparql, which is then POSTed to the SPARQL endpoint. Later, the cleanup pass at :1080-1083 globs update.*.sparql, parses the middle component as the offset, and sorts via int(x) — which raises ValueError: invalid literal for int() with base 10: 'None' and aborts the command.

Different from #275 / commit 6b6d430 (which made the SSE iteration tolerate disconnects); this is a downstream bug in the post-retry recovery path.

Log:

SSE stream connection lost (fetch https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2 failed with wrong response status: 502), will reconnect ...
Consuming stream from date: None
SSE stream connection for batch processing failed (attempt 1/10): ... 502. Retrying in 5s ...
SSE stream connection for batch processing failed (attempt 2/10): ... 502. Retrying in 10s ...
SSE stream connection for batch processing failed (attempt 3/10): ... 503. Retrying in 30s ...
SSE stream connection for batch processing failed (attempt 4/10): ... 502. Retrying in 1min ...
Encountered message with date 2026-06-01T16:54:58Z, which is within 1 second of the current time, finishing the current batch
Assembled batch #7632, #messages:  1, date range: 2026-06-01T16:54:58Z - 2026-06-01T16:54:58Z  [assembly time: 15,728ms, min delta to NOW: 0.6s]
curl -s -X POST "http://tajo:7001?access-token=wikidata_AhnTHyQTrTPD" -H 'Content-Type: application/sparql-update' --data-binary @update.None.1.sparql
An unexpected error occurred: invalid literal for int() with base 10: 'None'

Traceback (most recent call last):
  File "/local/data-ssd/qlever/qlever-control/src/qlever/qlever_main.py", line 38, in main
    command_successful = command_object.execute(args)
  File "/local/data-ssd/qlever/qlever-control/src/qlever/commands/update_wikidata.py", line 1081, in execute
    sorted_offsets = sorted(
  File "/local/data-ssd/qlever/qlever-control/src/qlever/commands/update_wikidata.py", line 1082, in <lambda>
    update_files.keys(), key=lambda x: int(x)
ValueError: invalid literal for int() with base 10: 'None'

Note the line Consuming stream from date: None just before the failure — that is the recovery path with no resume point, which is where first_offset_in_batch ends up unset.

Suggested fix

Two surfaces, both worth addressing:

  1. Don't let first_offset_in_batch reach the filename construction as None. Assert/raise (or take a different code path) in the recovery branch that leaves the offset unknown, so the bad filename is never written in the first place.
  2. Defensively, the cleanup glob at :1080-1083 should skip filenames whose middle component isn't a valid integer, rather than crash the whole command on one stale file. Even after (1) is fixed, an existing update.None.*.sparql from a prior run would re-trigger the crash on the next invocation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions