Packages missing from environments installed via conda-lock due to corrupt mamba/micromamba cache

[maresb]

Problem

Users can end up with incomplete environments where some packages are silently missing after installing from a conda-lock lockfile. The lockfile generates without errors, but the resulting environment is broken because some expected packages were never installed.

This affects users whose mamba/micromamba package cache contains corrupt metadata — a situation created by mamba/micromamba versions 2.1.1 through 2.3.2 (released May–October 2025). The corrupt cache entries persist after upgrading mamba, so users continue to be affected even on current versions.

Causal chain

The issue involves a complex chain of events:

1. User installs from an explicit lockfile (e.g., conda-lock install, or micromamba install -f <lockfile>) using mamba/micromamba 2.1.1–2.3.2. This writes corrupt repodata_record.json files to the package cache where metadata fields including depends are zeroed out. (mamba-org/mamba#4052)
2. User generates a new lockfile with conda-lock lock or conda-lock lock --update. The solver uses channel repodata (which is correct) to determine which packages are needed.
3. The solver sees the user's package cache. conda-lock attempts to isolate the cache by setting CONDA_PKGS_DIRS to a temporary directory, but pkgs_dirs is a sequence parameter in conda — values from .condarc are merged rather than replaced (conda#7769). If the user or system has pkgs_dirs configured in .condarc, those directories leak into the solver's view. The solver finds cached packages there and returns LINK-only actions (no FETCH) for them. LINK-only actions contain reduced metadata — notably missing depends, url, and hash fields that conda-lock needs for the lockfile. The solver's plan is correct and includes all needed packages.
4. conda-lock reconstructs the missing metadata from cache. To fill in the fields missing from LINK-only actions, _reconstruct_fetch_actions() reads repodata_record.json from the package cache. When those records are corrupt, the empty depends: [] is propagated into the lockfile as dependencies: {}. The corruption only affects metadata — the solver's package selection is still correct.
5. Transitive dependencies are dropped. apply_categories() walks the dependency graph starting from user-requested (direct) dependencies to label packages with categories like "main". When a package has dependencies: {}, the walk terminates and its transitive dependencies never receive a category. Packages with no category (categories == set()) silently vanish during V1 serialization — to_v1() produces zero entries for them. When the lockfile is rendered to explicit format, render_lockfile_for_platform() filters on p.categories & categories_to_install, and uncategorized packages are excluded. In testing with conda-lock's own environment, 59 out of 179 expected packages were missing.
6. The installed environment is incomplete. The user gets no error during lockfile generation or rendering. The problem only manifests at install time or at runtime when imports fail.

Who is affected

• Users who used mamba/micromamba 2.1.1–2.3.2 to install from explicit lockfiles at any point
• Users whose .condarc (or system-level config) has pkgs_dirs configured — this defeats conda-lock's CONDA_PKGS_DIRS isolation, allowing the solver to see the corrupt cache
• The corrupt cache entries persist after upgrading — the cache is not self-healing
• Both conda-lock lock (fresh solve) and conda-lock lock --update are affected, since both call _reconstruct_fetch_actions() which reads from cache whenever the solver returns LINK-only actions

A partial upstream fix in mamba 2.3.3 (mamba-org/mamba#4071) prevents new corrupt entries from being created (it recovers depends/constrains from info/index.json). However, it does not repair existing corrupt entries in the cache. A comprehensive upstream fix including cache healing is proposed in mamba-org/mamba#4110.

Detection

In repodata_record.json

The corrupt cache entries have a clear signature: timestamp == 0 AND license == "". This combination does not occur in legitimate records and is the same signature used for cache healing in the upstream fix (mamba-org/mamba#4110).

The corresponding info/index.json in the same package directory is never corrupted by this bug (it is written at build time, not at install time), and can serve as a fallback source.

In an existing lockfile

• Missing sha256: _reconstruct_fetch_actions() passes through whatever fields the cached repodata_record.json contains, including sha256 if present. The repodata_record.json will lack sha256 when it was generated by an explicit lockfile install whose URLs contain only md5 hashes — which is the case for explicit lockfiles rendered by conda-lock. A repodata_record.json generated by a spec-based solve (where mamba has the full channel repodata.json including sha256) will have sha256. So missing sha256 in a lockfile entry specifically identifies an entry whose cache was populated via explicit lockfile install — the same install path where the corruption bug manifests. However, missing sha256 alone does not prove corruption: micromamba 2.1.0 (pre-bug) also produces entries without sha256 but with correct dependencies. Missing sha256 combined with empty dependencies on a non-leaf package is the strong corruption signal.
• Python packages with dependencies: {}: A package whose build string contains py or pyh typically depends on python. Empty dependencies on such a package is a strong corruption indicator.

How conda-lock reads from cache

conda-lock reads from the cache through a single code path:

_reconstruct_fetch_actions()     [conda_solver.py:224]
  -> _get_repodata_record()      [conda_solver.py:165]
       -> reads info/repodata_record.json from pkgs_dirs

Two call sites reach this:

• solve_specs_for_arch() (conda_solver.py, line 342) — regular conda-lock lock
• update_specs_for_arch() (conda_solver.py, line 499) — conda-lock lock --update

Currently, _get_repodata_record() reads the JSON and returns it with no validation. The corrupt data flows into LockedDependency objects with dependencies: {}.

Why CONDA_PKGS_DIRS isolation is incomplete

conda-lock sets CONDA_PKGS_DIRS to a temporary directory in conda_env_override() (invoke_conda.py:274). The intent is to prevent the solver from finding packages in the user's cache, forcing FETCH actions for all packages (which carry correct metadata from channel repodata).

However, pkgs_dirs is a sequence parameter in conda — setting it via environment variable does not replace values from .condarc, it merges them (conda#7769). If the user or system has pkgs_dirs configured in any .condarc file, those directories are appended to the list alongside conda-lock's temporary directory. The solver then finds cached packages in the user's directories and returns LINK-only actions for them, triggering the _reconstruct_fetch_actions() path.

Note that the corrupt cache is shared between solvers: mamba/micromamba writes the corrupt repodata_record.json files, but conda-standalone reads from the same pkgs_dirs and is equally affected.

conda-lock cannot simply ignore .condarc entirely because it depends on settings from it that cannot easily be replicated via environment variables:

• channel_alias — needed to resolve short channel names to URLs
• proxy_servers — needed for network access behind corporate proxies (a map type, awkward to set via env var)
• ssl_verify / client_cert / client_cert_key — needed for authenticated or proxied channels
• custom_channels / custom_multichannels / default_channels — needed for channel name resolution

Channel selection is already isolated via --override-channels, but channel resolution (what URL does a channel name map to?) still requires the user's config.

micromamba supports --no-rc and --rc-file flags that could be used for selective config loading, but conda and conda-standalone have no equivalent (conda#6902, conda#14345).

Proposed mitigations

1. Detect and reject corrupt repodata_record.json entries

In _get_repodata_record(), check for the corruption signature (timestamp == 0 AND license == ""). When detected, fall back to info/index.json from the same package directory and log a warning.

index.json is always present and never corrupted by this bug. For the vast majority of packages, index.json and repodata_record.json agree on depends. They diverge only when channel repodata hotfixes have modified the dependency list, in which case using index.json is strictly better than using the corrupt zeroed values.

It might also be possible to read from mamba's channel repodata cache (mamba caches the original repodata.json files alongside the .solv files it generates from them), which would preserve repodata hotfixes. However, the cached files have mangled/hashed names and this approach has no public API, so it would be fragile.

2. Pre-write orphan detection

After apply_categories() runs (at solve time, before the lockfile is written), check whether any solved package ended up with categories == set() (no category assigned). The solver included these packages because they are transitively needed, so having no category means the dependency walk from direct dependencies couldn't reach them — a signal that some package in the chain has corrupt (empty) dependencies.

Currently, these orphaned packages silently vanish during V1 serialization (to_v1() produces zero entries for packages with empty categories). This check would turn a silent data loss into a visible error or warning.

Note: this check detects the symptom (unreachable packages) rather than the cause (which specific package has the empty deps). It works because the solver's output is a complete dependency closure, so every package should be reachable from a direct dependency via the dependency graph. However, it relies on the lockfile's own dependencies dicts for the walk, so it cannot pinpoint which entries are corrupt — only that the graph is broken somewhere.

3. Detect and warn on leaked pkgs_dirs

conda-lock already queries the effective pkgs_dirs via _get_pkgs_dirs(). After the query, check whether the returned list contains directories beyond conda-lock's own temporary directory. If so, log a warning indicating that the user's .condarc pkgs_dirs are leaking into the solve, and that cached packages from those directories may be used.

For stronger isolation, micromamba supports --no-rc / --rc-file flags that could selectively exclude pkgs_dirs while preserving other config. conda and conda-standalone have no equivalent (conda#6902).

4. Warn on cache-sourced entries

Log a warning whenever _reconstruct_fetch_actions() reads from cache instead of using channel data. This makes the degraded metadata path visible to users.

Upstream context

• mamba-org/mamba#4052 — original upstream bug (affects 2.1.1–2.3.2)
• mamba-org/mamba#4071 — partial fix in 2.3.3 (recovers depends/constrains from info/index.json)
• mamba-org/mamba#4095 — remaining issues post-fix (still open)
• mamba-org/mamba#4110 — comprehensive upstream fix including cache healing (draft PR)
• conda/conda#7769 — CONDA_PKGS_DIRS merges with .condarc pkgs_dirs rather than replacing (by design)
• conda/conda#6902 — feature request for CONDARC override / CONDA_IGNORE_CONFIG

Reproduction

See conda/conda-lock#862 for a complete reproduction pipeline using Docker and multiple micromamba versions, including:

• Captured repodata_record.json archives from micromamba 2.1.0 (good), 2.1.1 (corrupt), 2.3.3 (partial fix)
• A clobber script that synthetically reproduces the corruption and verifies it matches upstream output
• Generated lockfiles from each cache variant for comparison
• Detailed analysis documents

Related

• conda/conda-lock#433 — Reconstruct RepodataRecord from lockfile (context on which fields the lockfile model can represent)
• conda/conda-lock#862 — reproduction pipeline (draft PR on this repo)