feat: Accept sparse repodata in solve

[soapy1]

Description

This change introduces the ability to pass sparse repodata to the rattler.solve function. This change moves towards unifying the py-rattler solve apis (solve and solve_with_sparse_repodata).

How Has This Been Tested?

• unit tests
• conda-rattler-solve-example: this is intended to be used in conda-rattler-solver, for example https://github.com/conda/conda-rattler-solver/compare/main...soapy1:add-installed-records-to-solve-pool-2?expand=1
• small example: consider the sample script that exercises using sparse repodata and a channel with the rattler.solve funciton

import asyncio
from rattler import solve, SparseRepoData, Channel

async def main() -> None:
    # Setup sparse repodata object to represent a slice of repodata
    # originating from a local channel that has been sharded
    chn = Channel("http://127.0.0.1:40093/noarch")
    repo = SparseRepoData(chn, "noarch", "./sparse_repodata.json")

    solved_records = await solve(
        # Sources (channels or custom RepoDataSource) to use for solving
        sources=[repo, "conda-forge"],
        # The specs to solve for
        specs=["tzdata", "foo ==1"],
    )
    print("solved required dependencies:")
    for rec in solved_records:
        print(f"  * {rec.name.normalized} v{rec.version}")


if __name__ == "__main__":
    asyncio.run(main())

with the sample sparse repodata file called `./sparse_repodata.json`

``` { "repodata_version": 2, "info": { "subdir": "noarch", "repodata_revisions": { "v3": {} }, "base_url": "http://127.0.0.1:40093/noarch" }, "packages": {}, "packages.conda": { "foo-1-0_a.conda": { "name": "foo", "version": "1", "build": "0_a", "build_number": 0, "depends": [ "bar" ], "constrains": [ "splat<3" ], "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" }, "bar-1-0_a.conda": { "name": "bar", "version": "1", "build": "0_a", "build_number": 0, "depends": [], "constrains": [ "splat<3" ], "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" } }, "v3": { "tar.bz2": {}, "conda": {}, "whl": {} } }```

produces the result:

$ python solve.py
solved required dependencies:
  * foo v1
  * bar v1
  * tzdata v2026c

AI Disclosure

• This PR contains AI-generated content.
  • I have tested any AI-generated content in my PR.
  • I take responsibility for any AI-generated content in my PR.
    Tools: Claude

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added sufficient tests to cover my changes.

[soapy1]

I'm not sure about this package format selection here. In rattler.solve_with_sparse_repodata we have a package_format_selection field that the user can set.
Should this pr be updated so that rattler.solve also supports this parameter?

[baszalmstra]

I think that would be worth a seperate PR but yeah adding the package_format_selection to the gateway query (and thus the rattler.solve function) makes a lot of sense to me.

[baszalmstra]

Something along the lines of:

gateway
    .query(sources, platforms, specs)
    .package_format_selection(PackageFormatSelection::PreferCondaWithWhl)
    .execute()
    .await?;

[danyeaw]

I agree that a follow up PR would be good. Should we change the type to PreferConda for now and then users can still pass in solve_with_sparse_repodata(..., PREFER_CONDA_WITH_WHL)?

[baszalmstra]

This already looks really good, but I'd be curious to see if there are performance gaps between the two implementations. Did you benchmark any of this?

[soapy1]

I'd be curious to see if there are performance gaps between the two implementations. Did you benchmark any of this?

I did some quick benchmarks in the conda-rattler-solver context. Results are here conda/conda-rattler-solver#101 (comment). There are some caveat's with these measurements (eg. hyperfine is just measuring how long it takes to run a command, the measurement includes much more than just the solve time, results will vary between machines, etc.). These results suggest that using this pr is <1s faster for a large package like vaex.

I wrote some quick pytest benchmarks. They show the opposite result, that rattler.solve_with_sparse_repodata is marginally faster for small repodata and small packages.

This difference doesn't inspire much confidence. I'd be interested to run pytest benchmark style test on some more realistic size solve.

[soapy1]

Ok, I've added one more test to my little benchmarks branch. It addiitonally compares using rattler.solve with a sparse repodata source implemented in python. As expected, this is quite a bit slower than using rattler.SparseRepoData directly. So, I think this change provides an improvement to users trying to do solves with sparse repodata. And there is still opportunity to make rattler.solve more performant (but maybe that is better addressed in a more perfomance oriented pr).

@baszalmstra does this investigation resolve your questions around performance gaps?

[baszalmstra]

@soapy1 Looking at your benchmarks I think a 1.5x slowdown is still significant. I think we should be able to improve this significantly. When using channels directly internally the repodata gateway also just use a sparse repodata. Given that you already provide it a sparse repodata should therefor be very fast.

[github-actions]

cargo-semver-checks detected API breaking changes compared with the pull request's base revision.

Details

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_variant_added.ron

Failed in:
  variant Source:SparseRepoData in /home/runner/work/rattler/rattler/crates/rattler_repodata_gateway/src/gateway/source.rs:52
    Building rattler_repodata_gateway v0.31.0 (current)
       Built [  57.931s] (current)
     Parsing rattler_repodata_gateway v0.31.0 (current)
      Parsed [   0.062s] (current)
    Building rattler_repodata_gateway v0.31.0 (baseline)
       Built [  58.032s] (baseline)
     Parsing rattler_repodata_gateway v0.31.0 (baseline)
      Parsed [   0.059s] (baseline)
    Checking rattler_repodata_gateway v0.31.0 -> v0.31.0 (assume minor change)
     Checked [   0.212s] 196 checks: 195 pass, 1 fail, 0 warn, 57 skip

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 120.517s] rattler_repodata_gateway

[soapy1]

looks like this is a breaking change that can be avoided by adding #[non_exhaustive]. I think we probably don't want to add that, because every case should always proabably be handled. But not sure if that's the rust way?

[soapy1]

@baszalmstra, I've updated this pr to add sparse repodata as source for the gateway. I've also updated the benchmark notes https://github.com/soapy1/rattler/pull/. The benchmarks are still pretty jittery, like each run has pretty different results. So, looking to make a few improvements there too.