Skip to content

Monitor and plot RSS and CPU usage during index builds of non-qlever engines - #327

Draft
tanmay-9 wants to merge 11 commits into
qlever-dev:mainfrom
tanmay-9:share-resource-usage-plot-and-monitor
Draft

Monitor and plot RSS and CPU usage during index builds of non-qlever engines#327
tanmay-9 wants to merge 11 commits into
qlever-dev:mainfrom
tanmay-9:share-resource-usage-plot-and-monitor

Conversation

@tanmay-9

@tanmay-9 tanmay-9 commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Since #305 , the QLever binaries sample their own RSS and CPU usage and write the resource-usage log themselves, so the python-side sampler was removed. The other engines do not write such a log, and need a universal python-side sampler.

With this change, an index build of any engine records RSS and CPU usage over time and renders the same kind of plot as qlever index does: memory and CPU on two axes, one shaded band per build phase, and a subtitle with the settings the build ran with. The bands and the subtitle are the only engine-specific parts and are provided per engine; everything else is shared. For an engine that builds its index in several runs, the runs land in one continuous log, and a run that fails does not leave its samples behind.

The log and the plot files of a non-QLever engine are prefixed <dataset>.<engine>, so that it is clear which engine a file belongs to. QLever keeps the plain <dataset> prefix, since its own binaries write that log.

NOTE: The log written on the client side has only the columns that the index build and its plot need, not the full format of the log that the QLever binaries write. The plot also still needs numpy and matplotlib (pip install qlever[plot]).

@tanmay-9 tanmay-9 changed the title Share resource usage plot and monitor Monitor and plot RSS and CPU usage during index builds of non-qlever engines Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant