Maintenance/code cleanup 2026-05 Phase VI (pure python resolver) by matteius · Pull Request #6669 · pypa/pipenv

matteius · 2026-05-12T23:36:09Z

Summary

Phase VI of the 2026-05 maintenance cleanup series. Ships the
pure-Python resolver backend (Initiative G phase 3 + 3b, end-to-end),
hardens it on the bench-fixture and ten real-world wheel-heavy combos
of T_PARITY_REAL, and lands a parallel wheel pre-fetch that cuts
the cold-cache pipenv sync / pipenv install wall time by ~30%.

Builds on top of #6668 (Phase V — perf + pure-python prework); merge
phase V first, then this.

What's in scope

1. Pure-Python resolver backend (Initiative G phase 3 + 3b)

Drops the dependency on pip-internal resolve calls. The new backend
sits on the typed schema surface introduced in Phase IV, drives
resolvelib directly, and reuses Phase I's PEP 691 client +
Phase II's ParallelFetcher. Selectable via:

pipenv lock --backend pure-python (CLI)
PIPENV_RESOLVER=pure-python (env)
[pipenv] resolver_backend = "pure-python" (Pipfile)

with --backend pip (the default) preserving byte-identical
pre-Phase-VI behaviour.

Major components:

File	Purpose
`pipenv/resolver/pure_python_requirement.py`	Typed `Requirement` dataclass — name, specifier, extras, marker, source, parent, introducing_marker
`pipenv/resolver/pure_python_metadata.py`	`MetadataFetcher` (PEP 658 + wheel-head Range fallback) + sdist routing
`pipenv/resolver/pure_python_sdist.py`	PEP 517 isolated-env METADATA extractor using vendored `build`
`pipenv/resolver/pure_python_provider.py`	`resolvelib.AbstractProvider` impl (find_matches, get_dependencies, is_satisfied_by, get_preference, narrow_requirement_selection)
`pipenv/resolver/backends/pure_python.py`	Top-level `PurePythonBackend` plumbing — drives the provider, translates the result mapping into `LockedRequirement`
`pipenv/vendor/build/`	Vendored PyPA `build` — gives the sdist extractor a self-contained PEP 517 frontend

2. Resolver correctness fixes shipped on the way to parity

Yanked-release filter (PEP 592) — find_matches excludes
yanked candidates from automatic selection unless the user
explicitly pins to that exact version. Bench-fixture trigger:
sentry-relay==1.1.4 (every artifact yanked) was being picked
as the highest match for sentry-relay>=0.8.45 and crashing
the sdist build path.
Wheel-over-sdist preference — when both a wheel and an sdist
exist at the same release, the wheel wins. Previously the
per-version sdist could slip through and force an unnecessary
PEP 517 build (python3-saml 1.16.0, redis-py-cluster 2.1.3).
Two-pass prerelease filter — mirror pip's CandidateEvaluator:
strict-no-prereleases first, PEP-440 fallback only when zero
stables match the merged specifier. Per-candidate
SpecifierSet.contains(..., prereleases=None) was applying
PEP 440's "no-final-release-matched" fallback on a single-element
iterable and admitting every prerelease.
Extras roundtrip + narrow_requirement_selection — the
psycopg[binary] → psycopg-binary shape now flows end-to-end:
wire-shape parses extras, find_matches propagates them onto
cloned candidates, get_dependencies strips extra == X
clauses from runtime markers (kept on introducing_marker for
the lockfile emitter) and emits a synthetic base-version
requirement to keep the bare and extras-flavoured identifiers
tied to the same version. Pip-style conflict-promote ordering
in get_preference + a port of narrow_requirement_selection
let the resolver navigate the wider constraint graph (overlapping
upper bounds on protobuf / grpcio / grpcio-status from
the sentry-protos + google-cloud-* fleet).
Sdist METADATA in a PEP 517 isolated env — the resolver no
longer crashes when a transitive's build-backend (poetry-core,
hatchling, flit-core, ...) isn't installed in pipenv's own
interpreter.

3. Parallel wheel pre-fetch (perf)

pipenv install / pipenv sync now fan out wheel downloads across
a 16-worker urllib3 pool BEFORE invoking pip install, populating a
--find-links directory pip then reads from. Pip's install step is
sequential, so on a cold pip cache the network download phase
dominated wall time — the sentry-base bench's 151 wheels spent
~12 s of pure network in the pip subprocess before this.

The pre-fetch:

reuses the resolver's existing PEP 691 client + ParallelFetcher
(same auth / netrc / cert handling already vetted under
GHSA-8xgg-v3jj-95m2),
SHA-256-verifies every downloaded body against the lockfile's
hashes,
only fires when the install is hash-pinned (policy.skip_lock is
False — without hashes there's no authoritative match key),
is best-effort: any per-package failure (missing target-platform
wheel, hash mismatch, network hiccup) falls through silently and
pip downloads via the index as usual.

--upgrade was also dropped from pipenv sync's pip-install
invocation — redundant under --no-deps + pinned versions + the
upstream is_satisfied filter, and it forced a per-package
metadata check. pipenv install (Pipfile-driven, may need to
downgrade) still passes --upgrade.

Numbers

T_PARITY_REAL — ten wheel-heavy combos vs pip backend

project                         pip   pp  hash  diffs  status
01-django-psycopg                 6    6   ✓       0   PARITY
02-flask-gunicorn                 9    9   ✓       0   PARITY
03-fastapi-uvicorn               13   13   ✓       0   PARITY
04-requests-httpx                10   10   ✓       0   PARITY
05-pandas-numpy                   4    4   ✓       0   PARITY
06-pytest-pytest-cov              7    7   ✓       0   PARITY
07-sqlalchemy-alembic             6    6   ✓       0   PARITY
08-cryptography-pyopenssl         5    5   ✓       0   PARITY
09-boto3-botocore                 7    7   ✓       0   PARITY
10-click-rich                     5    5   ✓       0   PARITY

10/10 byte-identical hash parity with the pip backend.

Sentry-base bench fixture (151 packages, Python 3.11, cold cache)

	Before	After	Δ
`pipenv lock` (pure-python backend)	timeout / 4+ min	~40 s	✅
`pipenv sync --backend pip` cold install	~34 s	~23 s	-32%
`pipenv sync --backend pip` warm install	~19 s	~16 s	-16%

The lock-time number assumes the pure-python backend; the install-time
numbers benefit every backend because the pre-fetch sits at the
install plumbing layer, not the resolver.

Tests

1567 unit tests pass (130 in the resolver-specific suite alone).
Resolver-module coverage gate: ≥ 90 % (T13, T14 enforce).
CI dogfooding: --backend pure-python is exercised in the CI
matrix alongside the pip backend (T_CI1).

Notable news fragments

+install-parallel-wheel-prefetch.feature.rst — the perf win
+install-drop-redundant-upgrade-flag.behavior.rst
+pure-python-extras-roundtrip.bugfix.rst
+pure-python-prerelease-parity.bugfix.rst
+pure-python-sdist-build-isolation.feature.rst
T_F.5.feature.rst / T_F.6.behavior.rst — --backend plumbing
Initiative G phase 1/2/3/3b design + plan docs under
docs/dev/

Test plan

Resolver unit suite (tests/unit/test_pure_python_*,
tests/unit/test_resolver_*)
T_PARITY_REAL parity matrix (10/10 byte-identical)
Sentry-base bench (lock + install, cold + warm) — see numbers
above
Pre-fetch unit tests (tests/unit/test_prefetch.py — happy +
hash-mismatch + non-200 + target-tag failure + no-match paths)
CI: linting / coverage gate / vendoring / smoke / package
benchmark all green at last push
Final CI run after this description update

🤖 Generated with Claude Code

Copilot

Pull request overview

This PR expands Pipenv’s opt-in pure-Python resolver path and related install performance work. It adds resolver backend plumbing/tests, sdist metadata extraction via newly vendored PyPA build tooling, parallel wheel prefetching, and Pipfile package-name casing controls.

Changes:

Adds/extends pure-Python resolver models, backend registration, sdist metadata build support, and backend selection plumbing.
Adds parallel wheel prefetch support for locked installs and adjusts pip install flags.
Vendors build/pyproject_hooks, adds CI dogfood coverage, tests, docs, and news fragments.

Reviewed changes

Copilot reviewed 36 out of 60 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`.github/workflows/ci.yaml`	Adds non-blocking pure-python backend test job.
`docs/dev/initiative-g-phase3-design.md`	Adds/updates design details for pure-Python resolver phase 3/3b.
`news/+install-drop-redundant-upgrade-flag.behavior.rst`	Documents dropping redundant `--upgrade` for sync installs.
`news/+install-parallel-wheel-prefetch.feature.rst`	Documents parallel wheel prefetch feature.
`news/+pipfile-recase-opt-in.behavior.rst`	Documents opt-in Pipfile recasing behavior.
`news/+pure-python-extras-roundtrip.bugfix.rst`	Documents pure-python extras parity fixes.
`news/+pure-python-prerelease-parity.bugfix.rst`	Documents prerelease filtering parity.
`news/+pure-python-sdist-build-isolation.feature.rst`	Documents isolated sdist metadata builds.
`pipenv/cli/options.py`	Adds `--backend` lock option and backend precedence handling.
`pipenv/resolver/backends/__init__.py`	Registers `pure-python` backend.
`pipenv/resolver/candidate.py`	Adds `extras` field to resolver candidates.
`pipenv/resolver/core.py`	Reads `resolver_backend` Pipfile setting in dispatcher fallback.
`pipenv/resolver/manifest_cache.py`	Adds Windows retry around atomic cache replacement.
`pipenv/resolver/pep691.py`	Allows urllib3- and requests-style response bodies/statuses.
`pipenv/resolver/pure_python_requirement.py`	Adds typed pure-python resolver requirement model.
`pipenv/resolver/pure_python_sdist.py`	Adds sdist download/extract/build metadata path.
`pipenv/routines/install.py`	Wires wheel prefetch into batch install.
`pipenv/routines/lock.py`	Resolves backend precedence before locking.
`pipenv/utils/pip.py`	Makes `--upgrade` conditional on dependency mode.
`pipenv/utils/pipfile.py`	Adds package-name casing mode normalization and canonical recasing.
`pipenv/utils/prefetch.py`	Adds parallel wheel prefetch implementation.
`pipenv/utils/resolver.py`	Adds backend-aware resolver cache key/request propagation.
`pipenv/utils/settings.py`	Adds `Settings.resolver_backend`.
`pipenv/vendor/build/LICENSE`	Vendors build license.
`pipenv/vendor/build/__init__.py`	Vendors build package entry point.
`pipenv/vendor/build/__main__.py`	Vendors build CLI module.
`pipenv/vendor/build/_builder.py`	Vendors build project builder.
`pipenv/vendor/build/_compat/__init__.py`	Adds build compat package marker.
`pipenv/vendor/build/_compat/importlib.py`	Vendors build importlib compat helper.
`pipenv/vendor/build/_compat/tarfile.py`	Vendors build tarfile compat helper.
`pipenv/vendor/build/_compat/tomllib.py`	Vendors build TOML compat helper.
`pipenv/vendor/build/_ctx.py`	Vendors build logging/subprocess context helpers.
`pipenv/vendor/build/_exceptions.py`	Vendors build exception classes.
`pipenv/vendor/build/_types.py`	Vendors build type aliases.
`pipenv/vendor/build/_util.py`	Vendors build utility helpers.
`pipenv/vendor/build/env.py`	Vendors isolated build environment support.
`pipenv/vendor/build/py.typed`	Marks vendored build as typed.
`pipenv/vendor/build/util.py`	Vendors build metadata utility API.
`pipenv/vendor/pyproject_hooks/LICENSE`	Vendors pyproject-hooks license.
`pipenv/vendor/pyproject_hooks/__init__.py`	Vendors pyproject-hooks public API.
`pipenv/vendor/pyproject_hooks/_impl.py`	Vendors pyproject-hooks implementation.
`pipenv/vendor/pyproject_hooks/_in_process/__init__.py`	Vendors in-process hook package helper.
`pipenv/vendor/pyproject_hooks/_in_process/_in_process.py`	Vendors hook subprocess runner.
`pipenv/vendor/pyproject_hooks/py.typed`	Marks vendored pyproject-hooks as typed.
`pipenv/vendor/vendor.txt`	Adds vendored build and pyproject-hooks versions.
`tests/unit/test_candidate.py`	Adds Requires-Python preservation tests.
`tests/unit/test_pipfile_subsystem.py`	Adds recase mode tests.
`tests/unit/test_prefetch.py`	Adds wheel prefetch unit tests.
`tests/unit/test_pure_python_provider_smoke.py`	Adds pure-python provider smoke tests.
`tests/unit/test_pure_python_requirement.py`	Adds requirement model tests.
`tests/unit/test_resolver_backends.py`	Adds backend CLI/Pipfile propagation tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

matteius · 2026-08-03T19:37:35Z

Restacked onto the updated Phase V tip. The obsolete short Windows manifest-cache retry and the already-upstream #6670 auth fix were dropped; current superseding behavior remains in the base. Commit d751275f also addresses the two remaining review threads (Windows drive-qualified archive members and package_name_case docs), which are now resolved.

Local verification: 1,614 unit tests passed, 10 platform skips; 94 focused sdist/Pipfile tests passed; full pre-commit suite passed.

…lding Spins up Phase 3 of Initiative G — the in-tree pure-Python ``resolvelib.Provider`` backend that replaces pip's ``PackageFinder`` + ``LinkEvaluator`` + ``InstallRequirement`` machinery with our own ``Requirement`` + ``MetadataFetcher`` + ``PurePythonProvider`` chain. Branched off ``maintenance/code-cleanup-phase5-perf-2026-06`` at commit ``3d16ca04`` (the post-Initiative-G-Phases-1-2 + ``resolve_constraints`` perf-cut tip). What lands here --------------- - ``docs/dev/initiative-g-phase3-design.md`` — focused design doc (260 lines). Picks up where the umbrella ``initiative-g-pure-python-design.md`` §5.4 / §7.4 left off. Open questions Q-A through Q-D explicitly catalogued for maintainer sign-off; default recommendations baked into the plan. - ``initiative-g-phase3-plan.md`` — 17-task swarm-ready plan modelled on the Phase 1+2 plan. Dependency graph; per-task ``depends_on`` / ``validation`` / ``status`` / ``log``. Parallel execution waves (12 waves, max concurrency 3). Risks table enumerates the lockfile-parity, sdist-fallback, ``get_preference`` mirror, and 30 % perf-gate concerns. - Three module scaffolds with rich docstrings naming the implementation task and the design-doc section that motivates them: * ``pipenv/resolver/pure_python_requirement.py`` (T1). * ``pipenv/resolver/pure_python_metadata.py`` (T2). * ``pipenv/resolver/backends/pure_python.py`` (T9 / T10). ``pure_python_provider.py`` is created lazily by T3 — keeping the module empty here would invite a half-finished class structure to be carried in from scratch. What's NOT here (per plan §"Out of Scope") ------------------------------------------ - sdist resolution (Q-A — fall back to pip backend). - Removing the pip backend (Phase 4). - HTTP/2 transport (separate effort). - Replacing ``resolvelib`` itself. - Keyring auth (Q-D — deferred). Phase-3 acceptance gates (design §8) ------------------------------------ - Zero ``pip._internal.*`` imports in the new code (enforced by Phase 1's pre-commit grep gate). - Lockfile byte-identity vs pip backend across the 100-pkg bench + 10 real-world projects (T_PARITY_REAL) + pip versions N-1 / N / N+1 (T_MATRIX). - CI ``lock-warm`` ≤ 14.5 s on the 100-pkg bench (≥ 30 % off the pre-phase-5 baseline of 21.3 s). - Parity matrix doc shipped with every divergence justified. - News fragment + ``docs/pipfile.md`` entry for the new selector. No code change in any module under ``pipenv/`` other than the three new scaffolds. Phase 1+2's modules are unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Six previously-open design questions are now answered with maintainer sign-off 2026-05-12. The plan no longer treats them as "open"; T9 and T14 are rewritten to reflect the load-bearing decisions: - Q-A: fail loud on sdist-only candidates (no transparent fallback). - Q-B: pre-fetch PEP 658 metadata for top-level packages only. - Q-C: strict mirror of pip's get_preference with byte-identity gate. - Q-D: no keyring auth in Phase 3. - Q-E: T_PARITY_REAL exercises 10 wheel-heavy mainstream projects. - Q-F: backend-startup wheel-availability pre-check on top-level packages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…for pure-python backend (Initiative G phase 3) Implements ``pipenv/resolver/pure_python_metadata.py`` per design §5.2 — the wheel-METADATA fetcher that T7's ``PurePythonProvider.get_dependencies`` will consume. Components: * ``CoreMetadata`` — frozen + slotted dataclass carrying ``name``, ``version``, ``requires_python``, ``requires_dist``, ``provides_extras``, ``summary``. * ``MetadataCache`` — on-disk cache keyed by ``sha256(wheel_url)``; same temp-file + ``os.replace`` atomic-write contract as ``ParsedManifestCache``. Cache entries are valid forever (wheels are immutable). * ``fetch_metadata(candidate, session, *, cache, metadata_url, metadata_hash)`` — read-through cache + two-tier fetch: 1. PEP 658 fast path when ``metadata_url`` is advertised; the body's ``sha256`` is verified against ``metadata_hash``. Mismatch raises ``MetadataFetchError``. 2. Wheel-head fallback otherwise: ``HEAD`` for length (probing ``GET Range: bytes=0-1`` on 405/403), range-GET the last 64 kB for the zip central directory, locate the ``<dist-info>/METADATA`` entry, range-GET its bytes, decompress, parse. * ``_PartialFile`` — ``io.RawIOBase`` shim that lets ``zipfile.ZipFile`` see a seekable view over the wheel; the shim transparently re-issues HTTP range GETs for bytes outside the in-memory window, so neither the central-directory walk nor the METADATA extraction has to buffer the full wheel. * ``_parse_metadata_text`` — stdlib ``email.parser.HeaderParser`` with ``email.policy.compat32``; collects every ``Requires-Dist`` header in source order and ``Provides-Extra`` into a frozenset. Tests at ``tests/unit/test_pure_python_metadata.py`` cover the four plan-T2 acceptance gates: * PEP 658 fast path returns parsed metadata. * Wheel-head fallback (synthetic wheel via ``tmp_path`` + ``zipfile``) returns parsed metadata. * Wheel-head with HEAD 405 falls back to probing GET. * Cache round-trip: second fetch is served from disk (no network). * PEP 658 hash mismatch raises ``MetadataFetchError``. * ``MetadataCache.get`` returns ``None`` on miss; ``put`` → ``get`` round-trip. * ``_parse_metadata_text`` minimum-fields + repeated-headers. Constraint compliance (Initiative G acceptance criterion): ``grep -nE "^[[:space:]]*(from|import)[[:space:]]+pipenv\.patched\.pip\._internal" pipenv/resolver/pure_python_metadata.py`` matches nothing. No new third-party deps; only stdlib (``hashlib``, ``io``, ``json``, ``logging``, ``os``, ``tempfile``, ``zipfile``, ``email``, ``pathlib``) + ``pipenv.vendor.packaging``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…nitiative G phase 3) Frozen `@dataclass(frozen=True, slots=True)` per design §5.1, replacing pip's `InstallRequirement` as the resolution-graph constraint node for the in-tree `resolvelib.Provider` path. Fields: `name` (PEP 503 canonical), `specifier` (`SpecifierSet`), `extras` (`frozenset[str]`), `marker` (`Marker | None`), `source` (Literal pipfile/transitive/ constraint), `parent` (`str | None`). Hashability comes for free from the frozen dataclass — both `SpecifierSet` and `Marker` are hashable in `pipenv.vendor.packaging`, so `frozenset[Requirement]` works out of the box (verified by the T1 test suite). The `from_pipfile_entry` classmethod handles the canonical Pipfile shapes (bare string version, "*", dict with version/extras/markers, "version": "*"). Names canonicalised via `pipenv.vendor.packaging.utils.canonicalize_name` (PEP 503). Zero `pip._internal.*` imports (enforced by Phase 1's pre-commit gate). RED→GREEN: `tests/unit/test_pure_python_requirement.py` (15 tests). T11 will extend this file with the broader coverage matrix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ve G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…Initiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…iative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…thing test ``test_archive_extracts_to_nothing_rejected`` hand-constructed an empty directory at ``/dev/shm/nonexistent-dir-for-empty-test``. ``/dev/shm`` is a Linux-only tmpfs mountpoint; macOS has no equivalent path and the mkdir raises ``PermissionError`` on the macOS CI runner. The test only needs an empty directory — switch to the standard ``tmp_path`` fixture which works on every platform and gets cleaned up automatically.

…alue ``PurePythonBackend._spec_value_to_pipfile_entry`` translated the wire-shape ``spec_value`` (a full pip-install argument string) into a ``Requirement.from_pipfile_entry`` value by scanning for the first specifier-introducing char and returning everything from there onward. That worked for the simple ``"requests==2.31.0"`` case in T9's tests but missed the real wire shape ``convert_deps_to_pip`` produces in production: package + extras + spec FOLLOWED BY ``-i <index_url>``, ``--trusted-host <host>``, ``--extra-index-url <url>`` etc. A T15-style smoke run on the 100-package bench fixture surfaced this on ``urllib3 = {extras = ["brotli"], version = ">=2"}``: the wire-shape came out ``"urllib3[brotli]>=2 -i https://pypi.org/simple"``, the helper returned ``">=2 -i https://pypi.org/simple"``, and ``SpecifierSet`` crashed with ``InvalidSpecifier``. Fix: split on the first whitespace and operate only on the leading ``name[extras]<spec>`` token. The pip CLI flags belong on ``request.options.indexes`` and the backend already consumes them via ``request_index_urls``, so dropping them from the per-requirement spec is correct, not just a workaround. Regression test pins three shapes: ``-i``, ``--index-url``, and ``--trusted-host``. Plus a ``urllib3[brotli]`` extras-without-spec case that fell through the old fast path silently.

Phase 3b's first cut at sdist support called ``BuildBackendHookCaller.prepare_metadata_for_build_wheel`` directly in pipenv's own interpreter — fine for ``setuptools.build_meta`` which we already carry, but a hard ImportError for any sdist whose declared ``[build-system].build-backend`` is ``poetry-core``, ``hatchling``, ``flit-core``, etc. The bench-fixture smoke surfaced this on ``python3-saml``, ``redis-py-cluster``, ``requests-oauthlib``, ``sentry-redis-tools``, and ``sentry-relay``. Move the build through the PyPA :pypi:`build` library instead. ``DefaultIsolatedEnv`` spins up a throwaway venv per build, ``ProjectBuilder.from_isolated_env`` reads ``[build-system]`` and calls back into pyproject_hooks against that env, and ``builder.metadata_path`` produces a ``.dist-info`` we then read. Build-system requires + ``get_requires_for_build("wheel")`` extras get installed before the hook runs. Add :pypi:`build` (``>=1.0``) to runtime deps. Vendoring it into ``pipenv/vendor/`` is the longer-term plan; the runtime dep is the shortest path to making the pure-Python backend usable on real Pipfiles, and is what we agreed on for now. Test architecture: * The build step is extracted into a module-level ``_build_metadata_in_isolated_env`` helper so tests can monkeypatch it without orchestrating a real venv (which would add multi-second setup overhead per test). * An autouse conftest-style fixture (``_patch_isolated_build``) at the top of ``test_pure_python_sdist.py`` swaps the helper for a no-isolation shim that runs in the test process's own Python — the pre-refactor behaviour, sufficient for exercising the surrounding plumbing on synthetic ``setuptools.build_meta:__legacy__`` sdists. * Tests that previously injected failure modes via patched ``BuildBackendHookCaller`` (timeout, missing METADATA, non-UTF-8 METADATA) now patch the new helper directly. Known follow-up: the standard resolver-subprocess path uses the project venv's Python, which doesn't have ``build`` installed even though pipenv lists it as a runtime dep. Verified to work end-to-end under ``PIPENV_RESOLVER_PARENT_PYTHON=1``; subprocess mode needs either vendoring of :pypi:`build` (the planned next step) or a ``typing_extensions``-style bootstrap injection in ``pipenv/resolver/main.py``.

…`build` The original fragment described :pypi:`build` as a runtime dep because that was the planned interim shape. We landed the vendored form instead (``pipenv/vendor/build`` + ``pipenv/vendor/pyproject_hooks``, with the import in ``_builder.py`` rewritten to ``pipenv.vendor.pyproject_hooks``), which is what the resolver subprocess actually needs to import :pypi:`build` against the project venv's Python. Update the fragment to match what shipped.

…nd cap to pip's 200_000 Two T15 smoke-test bugs surfaced on the bench fixture lock: PEP 592 yanked filter --------------------- ``find_matches`` returned every cached candidate sorted by version descending without consulting ``Candidate.yanked``. On the bench fixture this picked ``sentry-relay==1.1.4`` (every artifact yanked with reason "accidental release") for the spec ``>=0.8.45`` and crashed downstream when the (yanked, sdist-only) candidate hit the isolated build path. Pip's resolver skips yanked candidates per PEP 592 unless the user explicitly pins to that exact version (``==<exact>``). Mirror that here via ``_candidate_is_skippable_yanked``: a yanked candidate is dropped unless any ``Requirement`` in scope has an ``==<candidate.version>`` specifier opting in. Three new tests pin the policy: range-spec excludes the yanked, exact-pin keeps it, exact-pin to a different version still excludes it. Resolver round limit -------------------- ``_drive_resolver`` defaulted ``max_rounds=100`` to mirror :mod:`resolvelib`'s own default. Pip uses ``200_000`` (``limit_how_complex_resolution_can_be`` in ``pipenv/patched/pip/_internal/resolution/resolvelib/resolver.py``); the 100-round cap exhausts inside the first category on the ~100-package bench fixture and trips ``ResolutionTooDeep``. Bump the default to ``200_000`` to match pip. Q-C "STRICT MIRROR" demands the same headroom; bumping further still warrants investigating a circular-dep bug upstream first.

…ion + suppress sdists with wheel companions Three correctness wins, one cumulative effect. (1) Wheel-priority sort. ``find_matches`` previously sorted only by version descending, so when a release shipped both a wheel and an sdist (the common case for pure-Python projects on PyPI) the artifact resolvelib actually picked depended on cache iteration order — half the time it routed through the sdist build path even when an immediately-usable wheel existed. ``python3-saml 1.16.0``, ``redis-py-cluster 2.1.3``, and friends on the bench fixture all hit this. Sort now keys on ``(Version, is_wheel)`` descending so wheels lift above sdists at the same version. (2) Per-version sdist suppression (early, on the unique set). Pip's ``PackageFinder`` publishes one canonical Link per version (wheel when available); we now mirror that. Returning both wheel and sdist for the same release lets ``resolvelib`` try the sdist as a backtrack candidate after a downstream conflict on the wheel — same package, same metadata, same conflict, plus a wasted PEP 517 build. Bench-fixture trigger: ``sentry-protos`` / ``grpcio-status`` exploding into dozens of sdist builds during backtracking. Suppression runs on ``unique`` (before the rejected/satisfies/yanked filter) so a wheel rejected by incompatibilities does not silently re-promote its sibling sdist to the candidate list. (3) Test-suite update. The pre-existing ``test_wheel_sorts_before_sdist_at_same_version`` had asserted that ``find_matches`` returned both wheel and sdist with the wheel first; the new contract is wheel-only. Added ``test_sdist_dropped_when_wheel_exists_at_same_version`` to pin the suppression rule (``0.8.32`` and ``0.8.31`` keep the wheel only; ``0.8.30`` keeps the sdist because no wheel exists for it). Bench-fixture impact: ``pipenv lock --backend pure-python`` on the ~100-package Sentry fixture used to time out at 4+ minutes — every time the resolver backtracked, the sibling sdist for the same version added another PEP 517 build. Lock now completes in ~2m9s and produces a Pipfile.lock with the same ``_meta.hash`` as the pip backend (148 vs 150 packages, 7 prerelease-related version diffs — T15 follow-ups, not regressions from this fix).

… PEP-440 fallback PurePythonProvider was admitting prereleases under plain ``>=X`` constraints because its per-candidate ``SpecifierSet.contains(v, prereleases=None)`` shape applies PEP-440's "no final release matched, accept the prerelease" fallback on the one-element iterable that ``contains`` builds internally — every prerelease candidate slipped through. Pip dodges this by calling ``specifier.filter()`` over the full candidate list at once (``CandidateEvaluator.get_applicable_candidates``), where the fallback only triggers when zero stables matched. Mirror pip's two-pass scheme in ``find_matches``: * First pass — ``_candidate_satisfies_requirements`` runs with ``strict_prereleases=True``, passing ``prereleases=False`` to the per-spec ``contains`` so prereleases are unconditionally rejected (unless the spec opts in, e.g. ``>=4.0a1``, or ``--pre`` was set). * Fallback pass — only when the strict pass yielded zero candidates AND ``_allow_prereleases`` isn't set, re-run with ``strict_prereleases=False`` so packages whose entire release line is still pre-1.0 alpha still resolve. Bench-fixture trigger: ``billiard 4.3.0rc1``, ``hiredis 3.4.0.dev0``, and ``sentry-sdk 3.0.0a7`` were resolving under their transitive ``>=X`` constraints where pip picked the stable below (``4.2.4`` / ``3.3.1`` / ``2.59.0``). Re-running ``pipenv lock`` on the sentry-base bench fixture now matches pip on all three. Adds two regression tests covering both passes: * ``test_prerelease_excluded_when_stable_matches`` — the bench scenario. * ``test_pep440_fallback_when_only_prereleases_available`` — the fallback path so a refactor doesn't silently drop it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…endencies + port pip's narrow_requirement_selection T_PARITY_REAL on the design's 10 wheel-heavy combos was at 9/10 — ``psycopg[binary]`` was missing the ``psycopg-binary`` transitive because the wire-shape parser dropped the ``[binary]`` segment before the resolver ever saw it. Fix it end-to-end: * ``PurePythonBackend._spec_value_to_pipfile_entry`` now parses the ``[extras]`` segment of a wire-shape pip-install line and returns the dict-form Pipfile-entry shape that ``Requirement.from_pipfile_entry`` already understands. * ``PurePythonProvider.find_matches`` clones the filtered candidate set with ``extras=identifier_extras`` (via ``dataclasses.replace``) when the resolvelib identifier carries extras, so T7's ``get_dependencies`` reads the right ``parent_extras`` context against the cached :class:`Candidate.extras` field — PEP 691's simple-API doesn't report ``provides_extras`` so the cache always parses it as ``frozenset()``. * ``PurePythonProvider.get_dependencies`` strips the ``extra == X`` clauses from a transitive's runtime marker via a new ``_strip_extra_clauses`` helper. The extras-gating role was already consumed by ``_marker_active_for_extras`` at emission time; carrying the clause onto ``Requirement.marker`` made the T6 ``is_satisfied_by`` re-evaluation collapse the marker to False under the plain target env (no ``extra`` key) and reject every candidate. ``introducing_marker`` keeps the original for the T_M3 lockfile emitter. * ``PurePythonProvider.get_dependencies`` also emits a synthetic base-version requirement when the parent candidate carries extras — pins the bare ``(name, frozenset())`` identifier to the exact version of the extras-flavoured candidate. Mirrors pip's ``ExtrasCandidate.iter_dependencies`` shape (``yield factory.make_requirement_from_candidate(self.base)``) and is yielded BEFORE any of the Requires-Dist transitives so the resolver sees the version pin early. * ``PurePythonProvider.narrow_requirement_selection`` is new and mirrors pip's three-tier algorithm at ``pipenv/patched/pip/_internal/resolution/resolvelib/provider.py`` line 120: active backtrack causes first, then ``_conflict_promoted`` (identifiers that crossed the conflict-priority threshold of 8 unresolved backtracks), then the full identifier set. Without this the wider transitive constraint graph that extras propagation surfaces (overlapping upper bounds on protobuf / grpcio / grpcio-status from the sentry-protos + google-cloud-* fleet) thrashed indefinitely. * ``PurePythonProvider.get_preference`` flips its leading ``backtrack_count`` slot from a bare ascending int (zero best) to a boolean ``not has_backtracked`` (False — has caused backtracks — sorts first). Matches pip's ``not conflict_promoted`` slot intent. Net result on the sentry-base bench fixture: before 2 min, missing psycopg-binary (9/10 T_PARITY_REAL) after ~40 s, 10/10 T_PARITY_REAL byte-identical with pip All 1567 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…d-install speedup Pip's ``install`` downloads wheels serially. On the sentry-base bench fixture (151 wheels, cold pip cache) the pure-network phase inside the pip subprocess dominates wall time — ~12 s locally, proportionally longer on CI's slower network (the bench reports ~42 s cold vs ~19 s warm; the 23 s gap is sequential downloads). Pre-fetch wheels concurrently BEFORE handing the requirements list to pip: * Reuse the resolver's existing ``PEP691Client`` + ``ParallelFetcher`` for simple-API metadata — same urllib3 16-worker pool, same auth / netrc / cert handling already vetted under GHSA-8xgg-v3jj-95m2. * Pick the wheel per package using a target-Python tag filter (subprocess ``<venv-python> -c "from packaging.tags import sys_tags"`` — host and target may disagree, so we ask the target directly) AND a hash filter against the lockfile. * Download the picked wheels in parallel via the same urllib3 session, SHA-256-verify every body against the lockfile, write to a temp dir. * Pass the temp dir to pip via ``--find-links``. Pip prefers the local file when it finds a hash match; any wheel we couldn't pre-fetch (missing target-platform wheel, hash mismatch, network hiccup) falls through to pip's regular index download path. Best-effort & safe: * Only fires when the install is driven from a hash-pinned lockfile (``policy.skip_lock`` is False). Without hashes there is no authoritative match key, and we MUST NOT mutate pip's install with unverified wheel bytes. * Every worker swallows its own exceptions; a single failed download doesn't taint the others. A wholly-failed pre-fetch returns ``None`` and ``--find-links`` is simply not added — pip behaviour is byte-identical to pre-this-commit. * Pre-fetched wheels go into a caller-managed temp dir, not the user's pip HTTP cache. No mutation of shared state. Bench impact (sentry-base fixture, Python 3.11 venv, cold cache, local; CI gap is wider on slower network): before this commit: ~34 s cold, ~19 s warm after this commit: ~23 s cold, ~16 s warm 17 new unit tests cover the helper paths (happy path, hash-mismatch, non-200, target-tag-query failure, no-matching- wheel, fast paths on empty/no-source inputs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…refresh stale backend-default assertion Two small follow-ups landing together. (1) ``pip install`` invocations driven by ``pipenv sync`` no longer pass ``--upgrade``. The flag was hardcoded ``True`` historically as a safety net to force a re-install when a package was already present at a different version, but with ``--no-deps`` (always set for sync) plus explicit ``pkg==X.Y.Z`` lines and the upstream ``Environment.is_satisfied`` filter that already runs before the batch is handed to pip, ``--upgrade`` is redundant — it forces pip to do a per-package metadata check that costs measurable wall time when the lockfile has many entries. Local bench (sentry-base, 151 packages, cold cache) shows a ~1.4 s saving on top of the parallel pre-fetch landed in the previous commit. ``pipenv install`` (Pipfile-driven, may need to downgrade) still passes ``--upgrade``. (2) ``test_build_resolver_request_defaults_backend_to_empty`` was asserting the empty-string sentinel that commit 0bf0c19 (``fix(resolver): stamp selected backend onto resolver requests``) replaced with the resolved fallback ``"pip"``. Renamed to ``test_build_resolver_request_defaults_backend_to_pip`` and updated the assertion to match the current contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/fc71f81d-8096-46e3-8617-a1b2325103c5 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

…_RESOLVER in cache key - pure_python_sdist.py: reduce archive_name to basename via Path().name before joining with dest_dir, preventing path-traversal if a simple-API response returns a filename containing separators or an absolute path - prefetch.py _download_and_verify: use urllib3.Timeout(connect=..., read=...) instead of a (connect, read) tuple; urllib3.PoolManager does not accept the tuple shape, falling back to tuple only if the import fails - resolver.py _generate_resolution_cache_key: include PIPENV_RESOLVER env var in the effective backend so a cache hit built by the pip backend cannot satisfy a pure-python lookup (and vice versa) when the backend is selected via the environment variable Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/bcacb5b8-63f0-4fda-ba65-72d386c52faa Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/fc71f81d-8096-46e3-8617-a1b2325103c5 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

- pure_python_sdist.py: use archive_name.strip() to also reject whitespace-only filenames - prefetch.py: rename _Timeout import alias to Urllib3Timeout for clarity - resolver.py: simplify effective_backend chain-or expression Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/bcacb5b8-63f0-4fda-ba65-72d386c52faa Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

…n mock Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/dc1b5386-f7bb-4a1d-ac2a-88a1ccb6d155 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

…tion, document source limitation Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/072ee90a-9755-49b3-9ecb-61fc16c1ac4f Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/dc1b5386-f7bb-4a1d-ac2a-88a1ccb6d155 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/072ee90a-9755-49b3-9ecb-61fc16c1ac4f Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

…perf-2026-06' into codex/phase6-stack-update

matteius · 2026-08-03T19:47:12Z

Follow-up: merged the updated Phase V head into Phase VI at 4e0311d so the stack contains the CodeQL test correction. The merge is clean and changes the prior Phase VI tree only in that test file. An explicit merge of current main reports Already up to date.

…perf-2026-06' into codex/phase6-merge-latest

matteius · 2026-08-03T20:22:14Z

Merged the refreshed Phase V head into Phase VI; an explicit current-main merge then reported Already up to date. Full unit suite: 1,615 passed, 11 skipped.

+            Requirement.from_pipfile_entry(
+                "django",
+                "*",
+                introducing_marker=Marker(  # type: ignore[call-arg]
+                    "python_version < '3.10'"
+                ),
+            )


+
+
+class _Logger(typing.Protocol):  # pragma: no cover
+    def __call__(self, message: str, *, origin: tuple[str, ...] | None = None) -> None: ...


+            cwd: Optional[str] = None,
+            extra_environ: Optional[Mapping[str, str]] = None,
+        ) -> None:
+            ...


+    python_executable: str
+    scripts_dir: str
+
+    def create(self, path: str) -> None: ...


+
+    def create(self, path: str) -> None: ...
+
+    def install_requirements(self, requirements: Collection[str]) -> None: ...


+    def install_requirements(self, requirements: Collection[str]) -> None: ...
+
+    @property
+    def display_name(self) -> str: ...


matteius changed the title ~~Maintenance/code cleanup phase6 pure python resolver 2026 07~~ Maintenance/code cleanup 2026-05 Phase V (pure python resolver) May 13, 2026

matteius changed the title ~~Maintenance/code cleanup 2026-05 Phase V (pure python resolver)~~ Maintenance/code cleanup 2026-05 Phase VI (pure python resolver) May 13, 2026

matteius marked this pull request as ready for review May 13, 2026 15:57

matteius requested a review from Copilot May 13, 2026 16:30

Copilot started reviewing on behalf of matteius May 13, 2026 16:31 View session

Copilot AI reviewed May 13, 2026

View reviewed changes

Copilot started work on behalf of matteius May 13, 2026 16:46 View session

Copilot finished work on behalf of matteius May 13, 2026 16:54

Copilot started work on behalf of matteius May 13, 2026 16:54 View session

Copilot finished work on behalf of matteius May 13, 2026 17:02

Copilot started work on behalf of matteius May 13, 2026 17:02 View session

matteius requested a review from Copilot May 13, 2026 17:02

Copilot started reviewing on behalf of matteius May 13, 2026 17:03 View session

Copilot finished work on behalf of matteius May 13, 2026 17:04

Copilot finished work on behalf of matteius May 13, 2026 17:06

Copilot AI reviewed May 13, 2026

matteius requested a review from oz123 May 13, 2026 17:08

matteius force-pushed the maintenance/code-cleanup-phase6-pure-python-resolver-2026-07 branch from 02a608d to d751275 Compare August 3, 2026 19:35

matteius and others added 7 commits August 3, 2026 15:40

test(resolver): T11 — Requirement unit-test coverage ≥ 95 % (Initiati…

9966c53

…ve G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

feat(resolver): T3 — PurePythonProvider.identify + Candidate.extras (…

16cede4

…Initiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

test(resolver): T12 — MetadataFetcher unit-test coverage ≥ 90 % (Init…

62c6fbc

…iative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

matteius and others added 23 commits August 3, 2026 15:40

vendor build backend

45b5d12

vendor build backend

4bfc9e7

Potential fix for pull request finding

bc39dd7

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

3979d5a

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix: include effective resolver backend in resolution cache key

1432492

Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/fc71f81d-8096-46e3-8617-a1b2325103c5 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

chore: clarify default backend cache-key compatibility comment

e91f686

Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/fc71f81d-8096-46e3-8617-a1b2325103c5 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

fix: non-blocking executor shutdown on sdist timeout; fix _FakeSessio…

a14fd2b

…n mock Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/dc1b5386-f7bb-4a1d-ac2a-88a1ccb6d155 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

fix: add prefetch cleanup, update design doc, add sdist hash verifica…

5b1044c

…tion, document source limitation Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/072ee90a-9755-49b3-9ecb-61fc16c1ac4f Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

docs: clarify Path.name/daemon-thread behavior in sdist cleanup code

e62ca29

Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/dc1b5386-f7bb-4a1d-ac2a-88a1ccb6d155 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

refactor: move imports to module level per code review

63ef7e6

Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/072ee90a-9755-49b3-9ecb-61fc16c1ac4f Co-authored-by: matteius <479892+matteius@users.noreply.github.com>

fix: address remaining Phase VI review feedback

1f40f7c

matteius force-pushed the maintenance/code-cleanup-phase6-pure-python-resolver-2026-07 branch from d751275 to 1f40f7c Compare August 3, 2026 19:40

Merge remote-tracking branch 'origin/maintenance/code-cleanup-phase5-…

4e0311d

…perf-2026-06' into codex/phase6-stack-update

Merge remote-tracking branch 'origin/maintenance/code-cleanup-phase5-…

e7b79f4

…perf-2026-06' into codex/phase6-merge-latest

github-code-quality Bot found potential problems Aug 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

Uh oh!

Maintenance/code cleanup 2026-05 Phase VI (pure python resolver) - #6669

Maintenance/code cleanup 2026-05 Phase VI (pure python resolver)#6669
matteius wants to merge 60 commits into
maintenance/code-cleanup-phase5-perf-2026-06from
maintenance/code-cleanup-phase6-pure-python-resolver-2026-07

matteius commented May 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

matteius commented Aug 3, 2026

Uh oh!

matteius commented Aug 3, 2026

Uh oh!

matteius commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		class _Logger(typing.Protocol): # pragma: no cover
		def __call__(self, message: str, *, origin: tuple[str, ...] \| None = None) -> None: ...


		def create(self, path: str) -> None: ...

		def install_requirements(self, requirements: Collection[str]) -> None: ...

Uh oh!

Uh oh!

Conversation

matteius commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in scope

1. Pure-Python resolver backend (Initiative G phase 3 + 3b)

2. Resolver correctness fixes shipped on the way to parity

3. Parallel wheel pre-fetch (perf)

Numbers

T_PARITY_REAL — ten wheel-heavy combos vs pip backend

Sentry-base bench fixture (151 packages, Python 3.11, cold cache)

Tests

Notable news fragments

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

matteius commented Aug 3, 2026

Uh oh!

matteius commented Aug 3, 2026

Uh oh!

matteius commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

matteius commented May 12, 2026 •

edited

Loading