Skip to content

compare-results: warn when the two sides come from different builds (needs a build block in DumpHeader) #371

Open

Description

@fcostaoliveira

Follow-up from the review of #369 (prebuilt release binaries, #366).

Context

After #369, every search and upload result file records the build that produced it under params.build:

  • crate_version, git_sha, git_tree, git_sha_source
  • target, profile, features, rustc
  • the openssl and hdf5 versions actually loaded at run time

Two runs of the same commit can use different client stacks:

  • Release archive: vendored OpenSSL 3.x and bundled HDF5 1.10.7.
  • Docker image or cargo build: the system's OpenSSL 3.0 and HDF5 1.10.8.

TLS cost is on the timed path for the managed and cloud engines. A comparison across the two can therefore show a client-side difference as an engine difference.

Ask

compare-results should warn, like it already does for differing index_layout (#344), when the two sides were produced by different builds. "Different" should cover at least:

  • a different git_sha, or a dirty tree;
  • different features or target;
  • a different openssl or hdf5.

Blocker

compare-results reads per-query dumps, and their DumpHeader (src/results_dump.rs) has no build block. So the first step is to:

  1. Add an optional build field to DumpHeader, holding the same value as params.build (vector_db_benchmark::build_info::provenance()).
  2. Write it from experiment.rs.
  3. Keep it optional when reading, so older dumps still load.

Then add the comparison and the warning, with tests on both sides of each field.

Deliberately not in #369, to keep that PR to release packaging.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions