test(odb): establish dynamic store correctness and performance baselines by Byron · Pull Request #2853 · GitoxideLabs/gitoxide

Sebastian Thiel (Byron) · 2026-07-30T06:54:39Z

Tasks

refackiew
ensure there is no performance regression (around parallelism)
- investigate!

Performance

Multi-threaded it's a tad slower wallclock wise, 1-3% in a parallel lookup test, but that's not going to be a problem IRL where object-decoding dominates.

❯ hyperfine -i './target/release/gix -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup' 'gix -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup'
Benchmark 1: ./target/release/gix -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup
  Range (min … max):    4.077 s …  4.545 s    10 runs

  Warning: Ignoring non-zero exit code.

Benchmark 2: gix -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup
  Time (mean ± σ):      4.207 s ±  0.201 s    [User: 57.023 s, System: 1.243 s]
  Range (min … max):    3.698 s …  4.393 s    10 runs

Summary
  gix -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup ran
    1.03 ± 0.06 times faster than ./target/release/gix -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup

Single-threaded, there seems to be a 5% regression, but I think that's Ok for now. It's unclear where it's coming from as well.

❯ hyperfine -i './target/release/gix -t1 -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup' 'gix -t1 -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup'
Benchmark 1: ./target/release/gix -t1 -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup
  Time (mean ± σ):     21.588 s ±  2.943 s    [User: 20.323 s, System: 0.386 s]
  Range (min … max):   19.598 s … 29.076 s    10 runs

Benchmark 2: gix -t1 -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup
  Time (mean ± σ):     20.593 s ±  1.234 s    [User: 20.216 s, System: 0.368 s]
  Range (min … max):   19.025 s … 22.069 s    10 runs

Summary
  gix -t1 -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup ran
    1.05 ± 0.16 times faster than ./target/release/gix -t1 -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux --trace odb stats --extra-header-lookup

Scaling report by Codex

Dynamic ODB thread-scaling experiment

Measured on 2026-08-13 with commit 9d8ae4941ae4a420c87829ee08403c79b2acf686
(gix v0.56.0-242-gea600f8c04) on a machine exposing 16 logical CPUs.

The repository was the bare Linux kernel repository at
~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux, with a 2.3 GiB
object database, two pack files, and about 10.9 million objects.

Method

The current release binary was built first. A 16-thread warm-up populated the
filesystem cache, followed by one separate process for each measured thread
count:

/usr/bin/time -p ./target/release/gix \
  -t <threads> \
  -r ~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux \
  --trace odb stats --extra-header-lookup >/dev/null

--extra-header-lookup performs the normal statistics pass, reopens the
repository to obtain a cold ODB, and looks up all known object headers again.
Speedup and efficiency use the end-to-end wall time of the one-thread run as
their baseline.

Results

Each result is one separate warmed run over 10.9M objects on a 16-logical-CPU machine.

Threads	Main first pass	PR first pass	Main cold lookup	PR cold lookup	Main wall / speedup	PR wall / speedup	PR vs main	Result
1	10.21 s	10.06 s	10.29 s	10.37 s	20.54 s / 1.00×	20.46 s / 1.00×	0.4% faster	Both pass
2	5.39 s	5.11 s	5.17 s	4.91 s	10.58 s / 1.94×	10.04 s / 2.04×	5.1% faster	Both pass
4	2.61 s	2.68 s	2.83 s	2.77 s	5.47 s / 3.75×	5.48 s / 3.73×	0.2% slower	Both pass
8	1.36 s	1.33 s	1.64 s	1.62 s	3.03 s / 6.78×	2.97 s / 6.89×	2.0% faster	Both pass
16	0.80 s	0.81 s	3.09 s	2.81 s	3.92 s / 5.24×	3.65 s / 5.61×	6.9% faster¹	Main passes; PR misses 11 objects
32	0.82 s	0.88 s	3.40 s	3.11 s	4.25 s / 4.83×	4.02 s / 5.09×	5.4% faster¹	Main passes; PR misses 27 objects

¹ Not a valid performance improvement because the PR run failed.

Interpretation

Both implementations scale nearly identically through eight threads. The PR ranges from effectively unchanged to 5% faster, which is too small to treat as significant from single samples. Eight threads is optimal for both:

Main: 6.78× speedup, 85% efficiency.
PR: 6.89× speedup, 86% efficiency.

Beyond eight threads, both suffer a substantial contention or scheduling cliff in the cold second pass. The first pass continues scaling to 16 threads, while cold lookup slows from roughly 1.6 seconds to 3 seconds. User CPU also rises sharply:

Main: 23.36 seconds at 8 threads to 56.44 seconds at 16.
PR: 22.96 seconds at 8 threads to 50.29 seconds at 16.

The critical difference is correctness. Main completed successfully at every thread count. The PR failed at both 16 and 32 threads, reporting IDs produced by the first pass as missing from the freshly opened ODB. A separate 16-thread PR warm-up also failed, with 14 missing objects instead of 11, demonstrating nondeterminism.

Therefore:

There is no meaningful performance regression or improvement through eight threads.
Both versions have poor scaling beyond eight threads for this workload.
The PR introduces or exposes a high-contention correctness regression that main does not exhibit.
Diagnosing that failure takes priority over collecting repeated performance samples; the 16- and 32-thread PR timings are not trustworthy benchmark results.

Purpose

This PR establishes the correctness and performance baseline for upcoming work on the multi-handle, multi-threaded dynamic object database. It intentionally changes test infrastructure rather than production ODB behavior: the aim is to make filesystem transitions, refresh behavior, slot pressure, and handle interactions reproducible before changing the implementation.

The infrastructure is intended to support this sequence of follow-up work:

Remove stray yields.
Remove loading races, propagate the winning load error correctly, and avoid repeatedly attempting the same failing file.
Allow the slot map to grow dynamically, which can also remove the need to scan the packs directory when opening a repository.
Avoid refresh races and unnecessary refreshes after misses while retaining configurable correctness bounds: strict/current behavior, a tolerant freshness window, and Refresh::Never.

What this adds

Shared ODB scenarios

A gix-testtools fixture script builds a deterministic catalog of valid SHA-1 and SHA-256 packs and indices. A utility module under gix-odb/tests/ copies and mutates that catalog for tests, benchmarks, and fuzzing without introducing another support crate.

The shared vocabulary can:

publish or remove pack, index, reverse-index, bitmap, promisor, mtimes, and keep components independently;
create and replace multi-pack indices;
add and remove alternates;
corrupt and restore indices;
model complete valid states as well as temporary states produced by Git-style maintenance.

This keeps all three kinds of coverage grounded in the same well-known repositories and object IDs.

Deterministic correctness coverage

The integration tests exercise multiple live handles in deliberately interleaved, single-threaded sequences while:

packs are published and removed component by component;
MIDX files are rewritten and referenced packs disappear;
alternates change;
malformed indices are restored;
fixed slots are over-committed.

The existing threaded packed-object smoke test also now always uses at least two workers and verifies that every enumerated object is actually found.

Benchmarks

Criterion benchmarks provide baselines for:

opening with a pack-directory scan versus a fixed slot count;
cold and warm hits through standalone indices and a MIDX;
repeated misses with default refresh behavior versus Refresh::Never;
bulk prefix disambiguation with both refresh modes;
concurrent hits and misses at one worker, available parallelism, and twice available parallelism.

These should make regressions and improvements in refresh suppression, prefix lookup, startup behavior, and slot over-commitment visible.

Deterministic fuzz VM

A single-threaded bytecode VM interleaves operations across several handles with disk mutations. It covers lookup, header access, iteration, prefix queries, refresh modes, pack-component transitions, MIDX transitions, alternates, malformed indices, and slot pressure.

The seed corpus includes standalone packs, MIDX use, alternates, malformed recovery, slot pressure, pack replacement, MIDX expiration, and interrupted maintenance. Maintenance states can expose markers or new components before old packs are removed, remove packs while a MIDX still references them, and abandon or reverse transitions.

The VM checks returned object hashes, header/data agreement, fresh-store equivalence at valid checkpoints, and existing handle metrics. Store statistics can be used or extended further where they help characterize later correctness or performance changes.

Verification

cargo test -p gix-odb --all-features --lib --test odb
SHA-256 runs of the new dynamic scenarios
cargo clippy -p gix-odb --all-features --tests --benches -- -D warnings
Criterion benchmark smoke run
nightly fuzz build, fixed-seed replay, and a short generated-input run
shellcheck for both fixture scripts

Add a gix-testtools-generated SHA-1/SHA-256 ODB catalogue and a shared mutation helper for publishing and removing pack components, rewriting MIDX files, changing alternates, and constructing maintenance-like intermediate states. Exercise deterministic single-threaded interactions across multiple handles, malformed-index recovery, changing packs and alternates, MIDX transitions, and slot over-commitment. This establishes the ordinary regression vocabulary used by the subsequent implementation changes, benchmarks, and fuzz model.

Use the shared deterministic ODB fixture to measure startup scanning, cold and warm packed lookups, repeated misses, prefix disambiguation, pack publication, fixed-slot pressure, concurrent access, and CPU over-commitment before changing the dynamic ODB implementation. Compare the original strict and never-refresh policies, and use Store metrics outside timed paths to verify that each workload reaches its intended refresh behavior. These common benchmark groups remain available after the implementation changes for direct Criterion comparisons.

Make disk reconciliation atomic when slots are exhausted or reused, retry changed indices after a cached load failure, and remember unchanged pack-load failures without repeatedly reopening the same file. Record an attempted lazy initialization for never-refresh handles so misses do not rescan forever, and recover unexpectedly vacant recorded slots through normal reassignment instead of assuming they are unreachable. Existing handles retain their last usable state throughout failed or partial maintenance transitions.

The fixed-size slot mode is a hard upper bound: once all slots are occupied, new disk state yields InsufficientSlots. Calling it Given obscures that behavior, especially now that a separate mode will be able to grow. Rename the public variant and update all workspace callers and documentation. This commit contains only the breaking vocabulary change so downstream adaptation is easy to identify.

Opening a store should not require listing every pack merely to size a fixed slot map, and later maintenance should not fail only because the repository outgrew its opening-time estimate. Add a Growable slot mode whose initial allocation performs no disk scan. Grow by publishing copy-on-write snapshots of stable slot pointers under the existing structural lock, so readers remain lock-free and all prior slot IDs stay valid. AsNeededByDiskState also gains growth after its initial estimate, while Limit retains the former hard-cap behavior. Ordinary tests cover growth past the initial allocation with shared and stable handles.

Publish the slot index and its backing resource nodes through one catalog so handles cannot combine refresh generations or lazily load a pack through a reassigned slot. Snapshots and stable locations retain the exact index and pack resources they observed, without Store-local PackId encoding or artificial slot/member bit limits. Recover republished indices and packs only when their on-disk state actually changed, preserve loaded MIDX packs across redundant loads, and replace rewritten MIDX resources at their logical slot even at a configured limit. Add test-only synchronization hooks and make contending index loads, pack loads, and refreshes share their success or error outcome. This prevents duplicate I/O and keeps failed maintenance transitions from partially replacing usable state.

Add RefreshMode::AfterDuration as a shared gix-odb freshness window. Repeated misses avoid rescanning packs and alternates until the last successful post-initialization refresh reaches the configured age, while failed refreshes, known missing packs, and explicit stale notification remain immediately retryable. Configure gix object handles with a one-second default and expose gitoxide.objects.refreshAfter in milliseconds. Fetch invalidates freshness after publishing a pack, configuration reloads update active handles, and zero restores strict refresh-on-every-miss behavior. Use the test-support clock for deterministic deadline and contention coverage, keep feature-unified workspace builds and Repository size guards valid with the added configuration, and make the minimal CI dependency assertion portable to images without ripgrep.

Add one deterministic single-threaded opcode VM for multiple handles, component-wise pack publication and removal, MIDX replacement, alternates, malformed resources, loose-object lifecycle, slot pressure, stable locations, iteration, prefix operations, and refresh policies. Give every opcode an explicit reproducible outcome and validate queries, shared-store checkpoints, retained locations, refresh counts, and fixture convergence against the generated disk model. Deterministic time covers strict, duration-based, and never-refresh behavior without wall-clock races. Keep readable corpus seeds for interrupted maintenance, load recovery, stale snapshots, cloned handles, MIDX transitions, growable stores, and minimized regressions.

Extend the pre-change dynamic-store baseline with scan-free growable opening, first-growth cost, fresh and expired duration policies, and post-publication discovery through an expired freshness window. Retain the common strict and never-refresh benchmark groups unchanged so Criterion results remain directly comparable across the implementation boundary, while the additional cases measure behavior that did not exist in the baseline API.

Retain object IDs during the serial first pass so extra-header-lookup provides a one-thread baseline instead of rejecting the invocation. Process the cold second pass in 1,000-ID chunks, matching the first pass. This replaces per-object sequentially-consistent work claims and progress updates with one claim and update per chunk while preserving independent ODB handles per worker.

An index loader previously claimed the next slot before incrementing the active-load count. A contender exhausting the slots in that window could conclude that nothing was loading and return before the index was published. Register first, retain the guard for a successful claimant, and drop a failed claimant's own registration before waiting. Extend the existing test hook with the active count and coordinate two cold handles to enforce this ordering without timing assumptions.

A handle could verify its snapshot before blocking behind another index loader, then let load_next_index establish its change baseline only after that loader published. This made the loader report no change even though the caller's snapshot was stale. Recheck the caller's marker after an exhausted load attempt and collect current state before applying miss-refresh policy. Exercise the lock handoff deterministically with two indices and two handles.

Allow the dynamic ODB scenario suite to fan each shared object or miss assertion out across synchronized handle clones. This reuses existing state-change coverage while exercising lazy index, pack, and refresh transitions under shared-store contention. Keep serial behavior as the default and select the worker count through GIX_ODB_TEST_THREADS so CI or local stress runs can control resource use.

A stale handle could enter refresh while another handle was publishing a new catalog. Refresh coalescing compared shared catalog snapshots and could return no update even though the caller's own snapshot predated the published state. Recheck the caller marker whenever a coalesced refresh returns no snapshot. Add a deterministic two-handle regression that holds a waiter between observing the old completion count and snapshotting the catalog, then lets the competing refresh publish before the waiter proceeds.

Cloning workers immediately before an assertion gives them snapshots of the current shared Store, leaving only the original handle genuinely stale. Keep reusable cohorts created before disk changes so every synchronized worker carries its own pre-mutation snapshot through later lookups. Apply persistent cohorts to pack publication and removal, multi-index rewrites, alternate changes, and dynamic slot growth. The same tests remain serial by default and use GIX_ODB_TEST_THREADS to select the cohort size.

Use deterministic Store synchronization points to remove a pack while handles contend for its slot, rewrite a multi-pack index while a refresh is paused, and resolve an old stable location while a grown catalog is being published. These schedules model maintenance transitions without relying on scheduler luck. Bound synchronization waits and assert that pack loads and refreshes complete once, publish a coherent catalog, and preserve stable locations.

Drive find, header, exact-prefix, iteration, miss, and stable-location operations from distinct pre-mutation handles after publishing a new pack. This exercises shared refresh and loading through different public paths instead of making every worker repeat one lookup. Use a bounded result channel for liveness and Store metrics to require one coalesced refresh for the known disk mutation. Iteration asserts its documented current-catalog snapshot semantics while lookup operations discover the new pack.

Provide a dedicated Just recipe that exercises the dynamic ODB scenario suite with eight persistent or synchronized handles. Keep the test harness itself single-threaded so each scenario controls its filesystem mutations and Store interleavings while applying real intra-scenario contention. Have CI call the reproducible recipe and use the workflow-native five-minute step timeout so retry loops and deadlocks fail without adding a platform-specific command to the Justfile.

A lazy-initialization waiter could observe the old refresh completion count and then snapshot the catalog published by the active initializer. Refresh coalescing consequently returned no update because its two shared catalog pointers matched, even though the caller still held the uninitialized handle snapshot. Reconcile a missing consolidation result against the caller marker for initialization as well as later refreshes. Add a deterministic two-handle regression that holds the waiter between reading completion and sampling the newly published catalog.

Sebastian Thiel (Byron) changed the title ~~improve odb correctness~~ test(odb): establish dynamic store correctness and performance baselines Jul 30, 2026

Sebastian Thiel (Byron) force-pushed the odb-parallelism branch 8 times, most recently from d106dfd to af4b335 Compare July 31, 2026 11:15

Sebastian Thiel (Byron) linked an issue Aug 5, 2026 that may be closed by this pull request

gix-odb: Handle::contains issues a sched_yield on every miss, including under RefreshMode::Never #2878

Open

Sebastian Thiel (Byron) force-pushed the odb-parallelism branch from af4b335 to 924d9de Compare August 12, 2026 06:35

Codex (codex) added 6 commits August 12, 2026 08:37

Sebastian Thiel (Byron) mentioned this pull request Aug 12, 2026

various improvements #2905

Draft

3 tasks

Sebastian Thiel (Byron) force-pushed the odb-parallelism branch from 924d9de to 960ebd3 Compare August 13, 2026 10:45

Codex (codex) added 11 commits August 13, 2026 14:05

Codex (codex) added 2 commits August 13, 2026 14:06

Sebastian Thiel (Byron) force-pushed the odb-parallelism branch from 2c1a46c to 01062cc Compare August 13, 2026 12:07

Navigation Menu

Uh oh!

Uh oh!

test(odb): establish dynamic store correctness and performance baselines - #2853

test(odb): establish dynamic store correctness and performance baselines#2853
Sebastian Thiel (Byron) wants to merge 19 commits into
mainfrom
odb-parallelism

Sebastian Thiel (Byron) commented Jul 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Sebastian Thiel (Byron) commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Performance

Dynamic ODB thread-scaling experiment

Method

Results

Interpretation

Purpose

What this adds

Shared ODB scenarios

Deterministic correctness coverage

Benchmarks

Deterministic fuzz VM

Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sebastian Thiel (Byron) commented Jul 30, 2026 •

edited

Loading