Consider installed packages in solve

[soapy1]

Description

This change allows packages installed in the environment to be considered in the solve by including them in them in the RattlerIndexHelper. We do not include virtual packages in the set of installed packages.

With this change, removing a channel from an environment and running an update no longer produces an Unsatisfiable error.

fixes #88

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[soapy1]

This is not used anywhere.

[danyeaw]

Hey @soapy1, thanks, this looks great! Just a minor / non-blocking comment about network calls used by one of the new tests.

[danyeaw]

Hey @soapy1, your current implementation is consistent with the other tests above which is great, but do you think it is possible to use the TmpChannelFixture and the mamba_repo here to avoid network calls? This would speed up the test a ton and I don't think it would give up much for test coverage.

[jaimergp]

I need to take a deeper look but I don't think we need this. The rattler APIs are different than libmamba's. Please don't merge until I review next week.

[soapy1]

@jaimergp thanks for reviewing this! I'm definitely not super confident that this is the right approach. Sharing some notes:

I was looking at some alternative ways at passing information related to what is already installed in the conda prefix to the rattler solver.

• currently, c-r-s uses rattler.solve_with_sparse_repodata. It allows callers to specify the set of available packages with the sparse_repodata parameter
• py-rattler also has a solve function which exposes a sources parameter to allow a caller to specify sources to query for a packages. It can take a few types, including Channel's and RepoDataSource's.

digging more into py-rattler rattler.solve:

• the RepoDataSource is pretty compelling. Like, could represent the already installed packages as a repodata source and pass it in alongside some channel sources.
• this would mean giving up using solve_with_sparse_repodata. Right now, c-r-s uses the sparse_repodata option to make sharded repodata and loading packages from the package cache work.
• I think moving away from solve_with_spare_repodata towards solve can work with c-r-s. rattler.solve also exposes the ability to set the gateway used for getting repodata. There are gateways that support sharded repodata
• this will also avoid the current c-r-s sharded implementation that write a bunch of temporary files for the repodata that is recomposed from shards.

[baszalmstra]

@soapy1 I actually tried this, but the overhead of marshaling the data from msgpack.zst -> Python -> Rust is significant. I explicitly introduced RepoDataSources for this usecase but I have not been able to make this faster than the current approach with the temporary files.

One thing we could improve, though, is to allow the rattler.solve function to additionally use an SparseRepodata object directly as an additional argument. We can then unify all these different methods under a single API.

For what it's worth, at least on the Rust side, SparseRepodata can also work with in-memory data, which would at least forgo temporary files on disk.

Even better would be to let py-rattler handle the orchestration of determining which shards should be downloaded and the parsing of the shards. But I can see how that would not be ideal at this point.

[soapy1]

Even better would be to let py-rattler handle the orchestration of determining which shards should be downloaded and the parsing of the shards. But I can see how that would not be ideal at this point.

ah, I see. I was under the impression that gateway already had this functionality.

One thing we could improve, though, is to allow the rattler.solve function to additionally use an SparseRepodata object directly as an additional argument. We can then unify all these different methods under a single API.

I really like this idea, trying it out!

[soapy1]

Trying out using the RepoDataSource's api in this branch https://github.com/conda/conda-rattler-solver/compare/main...soapy1:add-installed-records-to-solve-pool-2?expand=1. Depends on changes in rattler from https://github.com/conda/rattler/compare/main...soapy1:rattler:solve-sparse-repodata?expand=1.

This is just a rough demo, I think there is a more elegant way to structure this change in c-r-s. I did some initial benchmarking and it looks like this is not introducing new performance issues.

edit: below are some results from a quick test (ref: #42 (comment))

this pr

$ hyperfine -w 2 "python -m conda create -d jupyterlab -c conda-forge --override-channels --solver=rattler"
Benchmark 1: python -m conda create -d jupyterlab -c conda-forge --override-channels --solver=rattler
  Time (mean ± σ):      2.052 s ±  0.051 s    [User: 1.796 s, System: 0.173 s]
  Range (min … max):    1.999 s …  2.150 s    10 runs

$ hyperfine -w 2 "python -m conda create -d vaex -c conda-forge --override-channels --solver=rattler"      

Benchmark 1: python -m conda create -d vaex -c conda-forge --override-channels --solver=rattler
  Time (mean ± σ):     10.126 s ±  0.225 s    [User: 9.391 s, System: 0.653 s]
  Range (min … max):    9.776 s … 10.595 s    10 runs

using the alternate demo

ref: conda/rattler#2627 and https://github.com/conda/conda-rattler-solver/compare/main...soapy1:add-installed-records-to-solve-pool-2?expand=1

$ hyperfine -w 2 "python -m conda create -d jupyterlab -c conda-forge --override-channels --solver=rattler"
Benchmark 1: python -m conda create -d jupyterlab -c conda-forge --override-channels --solver=rattler
  Time (mean ± σ):      1.925 s ±  0.025 s    [User: 2.005 s, System: 0.233 s]
  Range (min … max):    1.890 s …  1.958 s    10 runs

$ hyperfine -w 2 "python -m conda create -d vaex -c conda-forge --override-channels --solver=rattler"
Benchmark 1: python -m conda create -d vaex -c conda-forge --override-channels --solver=rattler
  Time (mean ± σ):      8.735 s ±  0.129 s    [User: 9.270 s, System: 0.727 s]
  Range (min … max):    8.453 s …  8.925 s    10 runs

on main

 $ hyperfine -w 2 "python -m conda create -d jupyterlab -c conda-forge --override-channels --solver=rattler"

Benchmark 1: python -m conda create -d jupyterlab -c conda-forge --override-channels --solver=rattler
  Time (mean ± σ):      2.077 s ±  0.051 s    [User: 1.797 s, System: 0.178 s]
  Range (min … max):    2.007 s …  2.151 s    10 runs

$ hyperfine -w 2 "python -m conda create -d vaex -c conda-forge --override-channels --solver=rattler"

Benchmark 1: python -m conda create -d vaex -c conda-forge --override-channels --solver=rattler
  Time (mean ± σ):      9.930 s ±  0.173 s    [User: 9.203 s, System: 0.629 s]
  Range (min … max):    9.764 s … 10.352 s    10 runs

[ryanskeith]

Overall looks good to me. I have a small suggestion for optimization.

[ryanskeith]

Just a small optimization. It looks like that platform, size and timestamp should always be included in the dump so it could be omitted here.

[jezdez]

Thanks for working through this, Sophia. The issue is real, and the default reproduction now succeeds. I found two supported solver configurations where installed records still disappear as usable candidates. Both come from representing prefix records as ordinary channel repodata.

With strict channel priority, a package in the active channel excludes a compatible installed dependency from its removed origin channel. With CONDA_USE_ONLY_TAR_BZ2=true, an installed .conda record is filtered out even though keeping it requires no download.

I think the fix should keep the first solve unchanged, then add only an installed record named by a No candidates were found failure to locked_packages on the retry. Rattler inserts locked records directly into the candidate pool. This avoids channel priority and package format filtering while preserving normal update --all behavior for packages that do have current candidates. The missing-installed condition should remain separate from generic conflicts so we do not lock every relaxed conflict.

I left direct suggestions for the concrete channel and subdir bug in the current implementation, plus comments for the solver semantics and deterministic coverage. Those suggestions fix the metadata bug but not the central fallback semantics. The repoquery relaxation is unrelated and should move separately. I would keep this draft until the fallback is represented outside ordinary repodata.

[jezdez]

Passing every installed record into the ordinary repodata pool does not give these records the fallback semantics we need. I reproduced two failures on this head:

• With strict channel priority, an active channel containing foo=1 excludes installed foo=2 from its removed origin channel, even when installed bar requires foo>=2.
• With CONDA_USE_ONLY_TAR_BZ2=true, installed .conda records are filtered out of this pool even though they require no download.

Could we keep the initial solve unchanged, carry a separate missing_installed: set[str] across retries, add a name to it only in the existing No candidates were found branch when the spec matches the installed record, and add only those records to locked_packages in _collect_specs_main? Rattler inserts locked records directly into the candidate pool, so that path succeeds under both settings without locking every package during a normal update --all solve. Keeping this separate from out_state.conflicts also preserves the existing no-history transitive-drop behavior.

[jezdez]

This depends on live defaults and conda-canary, then only checks the return code and stderr. Channel contents can change and make the test pass without exercising the fallback. Could we use local repodata and assert that the installed record and its URL remain unchanged? A deterministic version can create test-package from tests/data/mamba_repo, switch to a local empty channel, and assert the no-op result. Please also cover the two cases that currently fail on this head: strict priority with an incompatible same-name candidate in the active channel, and CONDA_USE_ONLY_TAR_BZ2=true with an installed .conda record.

[jezdez]

This network-sensitive repoquery relaxation is unrelated to making installed records available during a solve. Could we move it to a separate PR so this change keeps a focused failure surface?