Add qlever upgrade-index command - #335
Merged
Merged
Conversation
So far, upgrading an index to the new format introduced on 2026-09-01 required running the `qlever-upgrade-index` binary by hand. For a setup that runs QLever with Docker or Podman, that binary is inside the image and not on the user's machine, so the command from the error message did not work there. There is now `qlever upgrade-index`, which runs the binary natively or in a container, depending on the `SYSTEM` from the Qleverfile, just like `qlever index` does. The output is also written to a log file. The command only converts from the previous index format to the one introduced on 2026-09-01, which is also stated in its help text.
There was a problem hiding this comment.
Pull request overview
Adds a new qlever upgrade-index CLI command to upgrade an existing index to the format introduced on 2026-09-01, supporting both native execution and containerized execution (Docker/Podman), and capturing output to a log file.
Changes:
- Introduce
UpgradeIndexCommand(qlever upgrade-index) with optional--upgrade-index-binaryoverride. - Implement container wrapping via
ContainerizewhenSYSTEMis a container engine. - Add unit tests covering native/container/show/failure paths.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| src/qlever/commands/upgrade_index.py | Implements the new upgrade-index command, command construction, optional containerization, and binary existence checks. |
| test/qlever/commands/test_upgrade_index_execute.py | Adds unit tests for the new command’s execution logic in multiple scenarios. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+49
to
+56
| upgrade_index_binary = args.upgrade_index_binary | ||
| if upgrade_index_binary is None: | ||
| directory, slash, _ = args.index_binary.rpartition("/") | ||
| upgrade_index_binary = ( | ||
| f"{directory}/qlever-upgrade-index" | ||
| if slash | ||
| else "qlever-upgrade-index" | ||
| ) |
Comment on lines
+81
to
+84
| if not binary_exists( | ||
| upgrade_index_binary, "upgrade-index-binary", args | ||
| ): | ||
| return False |
hannahbast
changed the title
qlever upgrade-index commandqlever upgrade-index command
hannahbast
added a commit
to ad-freiburg/qlever
that referenced
this pull request
Sep 1, 2026
… the resulting new index format (#3159) The vocabulary of an index is fixed once the index is built. For new words created later, there are so-called local vocabularies. An entry in such a vocabulary is an `Id` that holds a `LocalVocabIndex`: a pointer into memory, which is expensive to compare and cannot be persisted to disk. This change adds the new datatype `SecondaryVocabIndex` for words in a secondary vocabulary. The words in that vocabulary are kept in sorted order and can be persisted to disk. It behaves much like the original vocabulary, but can grow dynamically. It will be the vocabulary of choice for SPARQL updates, materialized views, and other precomputed results. This is the first change in the series that implements this vocabulary and the features that need it. For now, nothing but the unit tests creates such a vocabulary. The new datatype is inserted directly after `LocalVocabIndex` in the `Datatype` enum. With this placement, comparing `Id`s stays cheap, and the new datatype can later join the contiguous block of the string datatypes. All operations that depend on the internal order of `Id`s (`JOIN`, `GROUP BY`, `DISTINCT`, exporting) already work correctly with it. The semantically correct comparison of its words (for `ORDER BY` and `FILTER`) is deliberately left to a follow-up. That follow-up comes before the secondary vocabulary is actually used. **THIS IS AN INDEX BREAKING CHANGE:** Inserting a datatype in the middle of the enum renumbers the datatypes after it. The index format version is raised, and an old binary refuses a new index (and vice versa) with a clear error message. **Existing indexes must be upgraded, but do not have to be rebuilt from scratch.** The new binary `qlever-upgrade-index <basename>` (and the associated new `qlever upgrade-index`, see qlever-dev/qlever-control#335) upgrades an index in place. It rewrites the index `Id` by `Id` into a temporary subdirectory (without re-sorting anything, because the insertion preserves the relative order of all existing datatypes), checks that the result can be loaded and has the right number of triples, and only then moves the original index to the subdirectory `index-in-old-format.<date of its build>` and puts the upgraded index in its place. The upgrade is more than 10 times faster than a rebuild from scratch. The upgrade is not byte-deterministic because the blocks of the permutations are compressed in parallel. Co-authored-by: Hannah Bast <bast@informatik.uni-freiburg.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Complements ad-freiburg/qlever#3159, which changes the index format and introduces the
qlever-upgrade-indexbinary to convert an index from the previous format to the new format.All the new command does is call the binary. It works for all settings of
SYSTEM(native, docker, podman). In particular, it is useful whenqlever-*binaries are not installed on the user's system but only in the Docker image.