Maintenance/code cleanup 2026-05 Phase V (performance and pure python resolver prework)

[matteius]

Summary

Phase V of the 2026-05 modernization track. Two intertwined workstreams:

1. Performance cuts on the existing pip-driven resolver path — chasing the
   warm-relock and CLI-startup ceilings identified by the May 2026 benchmark
   suite (benchmarks/benchmark.py). Final measured win is modest: ~7-11 %
   across the CI bench (lock-warm, lock-cold, install-warm,
   add-package, import) from lazy-import work; the architectural ceiling
   is pip's per-package sequential index revalidation and Link.from_json
   loop, which pipenv can't move from outside pip.

2. Initiative G — pure-Python simple-API resolver groundwork — a new
   in-tree implementation (PEP 691 JSON + PEP 503 HTML client, parsed-manifest
   cache, parallel fetcher) under pipenv/resolver/ that imports zero
   pip._internal.* symbols. Phases 1 + 2 of a four-phase plan ship here;
   Phase 3 (the full pure-python backend replacing pip's PackageFinder)
   and Phase 4 (promote to default) are explicitly future work.

Companion design + plan docs in docs/dev/initiative-g-pure-python-design.md
and initiative-g-phase1-2-plan.md. No user-visible API change, no new
runtime dependencies, no lockfile-format change. The opt-in Phase-2 setting
defaults false.

---

What's in this PR

Phase-5 perf cuts (lazy imports + empty category skip)

• perf(lock): skip resolver subprocess for empty Pipfile categories (49c7a249) —
  do_lock always iterated ["default", "develop", …] even when a section
  was empty, paying ~5-6 s of subprocess + Resolver/Session setup to produce
  an empty lockfile section. Skip the call when packages is an empty
  mapping. Stale entries still cleared because lockfile.pop(category)
  already ran.

• perf(startup): defer pip-internal network imports in fileutils + internet (7335a1c6)
  and perf(startup): defer pip-internal InstallCommand + unpack + Downloader imports (74a466b1) —
  the parent CLI and resolver subprocess were both pulling
  pip._internal.network.download, pip._internal.commands.install.InstallCommand,
  pip._internal.operations.prepare (~78 ms each, cumulative) at module
  load. Moved these to first-use inside the functions that need them.
  Tests follow the import-target rename (the prior fix for the same
  symptom on integration tests is at c85cd3b6).

• perf(resolver): eliminate redundant find_best_candidate walk in resolve_constraints (cf53eb17) —
  Resolver.resolve_constraints called pip's PackageFinder.find_best_candidate(name, specifier) once
  per resolved package solely to read candidate.link.requires_python. The resolved tree already
  carries that link (pip's resolvelib stores the chosen candidate on every InstallRequirement it
  returns). Read the marker directly off result.link. Measured on the 100-package bench:
  in-process 31.4 s → 23.4 s (−25.5 %); subprocess warm ~22.6 s → ~17.9 s (−21 %, ~4.7 s saved).
  Lockfile hash byte-identical for standard pipenv users.

  Subtle interaction with tasks/vendoring/patches/patched/pip_finder_ignore_compatability.patch
  (commit 3d16ca04 documents this explicitly): users who flip
  finder._ignore_compatibility = True in the resolve pipeline (cross-platform locking workflows,
  the patched-pip flag) will see cross-compat packages now CARRY their advertised requires-python
  markers in the lockfile. Pre-fix, the strict find_best_candidate returned None for those
  candidates and the marker was silently dropped. This is arguably a correctness fix but IS a
  behavioural change for any consumer that relied on those markers being absent. A regression-
  pinning test (test_resolve_constraints_marker_for_ignore_compatibility_link) and a docstring
  on resolve_constraints document the trade-off.

• perf(lock): feed prior Pipfile.lock pins as pip constraints on warm relock (20092de9)
  was reverted (a2157da4) after maintainer review caught a semantic
  regression: the change silently froze wildcard versions across
  pipenv lock runs, breaking the historical contract that pipenv lock
  picks up newer matching versions. The revert is preserved in history
  for traceability.

Measured CI delta vs the pre-phase-5 baseline (median across the bench
suite, single CI run — not multi-run statistical):

stat	before	after	delta
lock-warm	21.295 s	19.250 s	−9.6 %
lock-cold	25.389 s	22.695 s	−10.6 %
install-warm	19.951 s	18.648 s	−6.5 %
install-cold	43.191 s	41.796 s	−3.2 %
add-package	30.740 s	27.405 s	−10.8 %
import (full)	73.263 s	69.889 s	−4.6 %

These are real-but-modest gains. The architectural ceiling is documented
in detail in docs/dev/initiative-g-pure-python-design.md §2.2.

Initiative G phases 1 + 2 — pure-Python resolver groundwork

A new pipenv/resolver/ surface (zero pip._internal.* imports, enforced
by a pre-commit grep gate scoped to pipenv/resolver/):

• pipenv/resolver/candidate.py — Candidate dataclass + Hash
  namedtuple + Candidate.from_filename helper. Frozen, slotted, pure
  data; wheel tags derived once at parse time via pipenv.vendor.packaging.tags.
• pipenv/resolver/pep691_types.py — SimplePageResponse and
  FetchError typed result envelopes.
• pipenv/resolver/pep691.py — _parse_pep691_json (PEP 691 JSON),
  _parse_pep503_html (PEP 503 HTML fallback), and PEP691Client class.
  Threads per-request verify / cert into the underlying session
  (FU3, landed in this PR). Deliberately does not send
  Cache-Control: max-age=0 (deliberate divergence from pip).
• pipenv/resolver/manifest_cache.py — ParsedManifestCache with
  TTL, atomic write, schema versioning, and (per FU1, landed here)
  peek_etag for stale-cache If-None-Match short-circuits.
• pipenv/resolver/fetcher.py — ParallelFetcher with capped 16-worker
  ThreadPoolExecutor. Sends If-None-Match for stale cache entries
  (FU1); option-a TTL refresh on 304 Not Modified.
• pipenv/resolver/auth.py — netrc / URL-embedded basic-auth /
  PIP_CLIENT_CERT helpers. Keyring deferred to Phase 3.

Phase-2 integration (opt-in, off by default):

• New setting [pipenv] prefetch_index_manifests + env-var override
  PIPENV_PREFETCH_INDEX_MANIFESTS=1. When enabled,
  do_lock calls _prefetch_index_manifests_if_enabled to fan out
  per-source parallel pre-fetches (FU2, per-verify_ssl policy)
  through pip's own PipSession — so pip's SafeFileCache is warmed
  as a side effect (no on-disk format reverse engineering).

User-facing doc for the new setting in docs/pipfile.md.

Why opt-in / why no measured Phase-2 perf claim

T21 (CI bench measurement for the prefetch path) was explicitly
deferred during execution review — no appetite for a multi-run
statistically-guarded CI bench step at this time. The Phase-2 perf
hypothesis (parallel cold-cache pre-fetch helps slow networks) is sound
based on the phase-5 I/O analysis but is theoretical, not measured
against the current CI baseline. The design doc §11a (Phase 2a — sign-off note)
is explicit about this; the user-doc points readers at the phase-5
branch history rather than quoting a percentage.

Phase-3 follow-ups landed early

Three items originally scoped as Phase-3 work were resolved during the
plan execution after their respective Wave agents flagged them:

• FU1 (91c1e4e9) — ParsedManifestCache.peek_etag() + fetcher
  integration. Closes a dead-code path: T19's status="not-modified"
  TTL-refresh branch was unreachable before this commit because the
  fetcher never sent If-None-Match. Now stale-but-present cache entries
  short-circuit to a 304 instead of re-downloading.
• FU2 (4a0ff8a1) — per-source verify_ssl fan-out in
  _prefetch_index_manifests_if_enabled. Original T19 cut routed every
  target through the majority-verify session; mixed-policy projects
  (self-signed private index alongside public PyPI) silently fell
  through to pip's cold fetch. Now one ParallelFetcher per unique
  verify_ssl value. Single-policy projects (common case) unchanged.
• FU3 (0047a2e3) — PEP691Client.fetch threads verify / cert
  into the per-request session.request(...) call. Pre-fix these
  kwargs were stored on self but never reached the request — so
  FU2's per-source verify routing was effectively a no-op at the
  request layer until FU3 landed.

Bug fixes caught during execution

• c85cd3b6 — fixes 4 integration tests in test_import_requirements.py
  that monkey-patched pipenv.utils.dependencies.unpack_url (no longer
  a module attribute after the phase-5 lazy-import work). Patch target
  moved to the canonical source (pipenv.utils.unpack.unpack_url).
• c76ecb42 — T20's integration test surfaced a cache-path
  mismatch: T17's _clear_parsed_manifest_cache wiped
  <PIPENV_CACHE_DIR>/manifests-v1/ while T19's prefetch wrote to
  <PIPENV_CACHE_DIR>/pipenv-manifests/manifests-v1/. Aligned both on
  the namespaced path.

---

Test plan

• Full unit suite green locally: 1263 tests passing (+341 since
  the design-doc commit a6832ce3; +307 from Initiative G alone).
• Targeted resolver-module coverage (T17's CI gate, narrow scope):
  99.68 % across the six new modules (603 statements, 2 missed
  — the Windows-only _netrc branch in auth.py, reaches 100 % on
  Windows CI).
• Phase-1 acceptance (T10): byte-equivalent parity vs pip's
  Link.from_json / Link.from_element across the T2 fixture
  suite (six, django, cryptography, tablib, yanked-pkg, missing-hash
  × JSON + HTML, including a 3496-file cryptography snapshot).
  Zero semantic divergences; representation diffs documented in
  tests/unit/test_pep691_parity_known_diffs.md.
• Phase-2 integration (T20): 5 scenarios passing, 1 skipped
  (self-signed-cert fixture not available — gap documented in
  docstring for a future Phase-3 follow-up).
• FU1+FU2+FU3 each carry their own dedicated tests; coverage
  maintained at 100 % on candidate.py, fetcher.py,
  manifest_cache.py, pep691.py, pep691_types.py.
• CI: Lint, build, unit, resolver-module-coverage gate, benchmark.

Test fixtures added

tests/unit/fixtures/pep691/*.json and tests/unit/fixtures/pep503/*.html
contain real-PyPI snapshots captured 2026-05-12 plus two hand-crafted
synthetics (yanked-pkg, missing-hash) exercising edge cases.
Provenance and re-baseline procedure documented in
tests/unit/fixtures/README.md.

---

Companion docs

• docs/dev/initiative-g-pure-python-design.md — Initiative G design
  (motivation, architecture, four-phase plan, dependency strategy, open
  questions for sign-off). Status updated to reflect Phase 1 + 2 shipped.
• initiative-g-phase1-2-plan.md — the 22-task dependency-aware swarm
  plan that executed Phases 1 + 2 (21 tasks done, T21 deferred, 3
  Phase-3 follow-ups landed early). Each task's status: / log: /
  files edited/created: fields populated post-execution.
• docs/pipfile.md — user-facing entry for [pipenv] prefetch_index_manifests
  alongside the existing cool-down-period block.

News fragments:

• news/initiative-g-phase1-pep691-client.feature.rst — Phase 1 ship.
• news/initiative-g-phase2-prefetch-bridge.feature.rst — Phase 2 ship.

---

Migration notes

None. Per-commit details:

• All Initiative G modules are new code under pipenv/resolver/. No
  existing imports break; the modules are reachable via
  from pipenv.resolver import PEP691Client, ParsedManifestCache, ParallelFetcher, Candidate, Hash, FetchError, SimplePageResponse, CachedManifest if any caller wants them. Phase 1 ships them
  standalone; Phase 2's do_lock hook is gated on the opt-in setting.
• Lazy-import changes (fileutils.py, internet.py, dependencies.py,
  project.py, environment.py, utils/resolver.py) keep all
  public attribute names addressable. Tests that monkey-patched
  module-level symbols that became function-scope imports were updated
  to point at the canonical source — the unpack_url fix in
  c85cd3b6 covers the four cases CI flagged. No other tests in the
  repo follow the same anti-pattern (verified by sweep).

What's NOT in this PR

• T21 (CI bench measurement for the prefetch path) — explicitly
  deferred per maintainer scope call. The Phase-2 perf claim is
  theoretical, not measured. Design-doc §11a documents this.
• Phase 3 — the full pure_python backend that replaces pip's
  PackageFinder.find_all_candidates via a resolvelib.Provider
  implementation. Three follow-up items (FU1, FU2, FU3) that were
  scoped as Phase-3 work landed here, but the main Phase-3 deliverable
  (the backend itself) is a separate future PR.
• Self-signed-cert integration test fixture — T20's skipped
  scenario; needs work in tests/pytest-pypi/ that's out of scope here.
• CI bench workflow changes — the existing benchmark job is
  unchanged. Adding a PIPENV_PREFETCH_INDEX_MANIFESTS=1 matrix entry
  would require T21-style statistical guards; deferred.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR continues the May 2026 modernization track by (1) reducing startup/lock overhead on the existing pip-driven resolver path and (2) landing major groundwork for a future pure-Python Simple API resolver (PEP 691/503), along with scaffolding for pluggable resolver backends.

Changes:

• Performance-oriented refactors (notably lazy imports) and a Windows virtualenv detection fix.
• Adds a new pipenv/resolver/ pure-Python Simple API client surface (types, candidate model, auth, cache, parallel fetch) plus targeted coverage gating.
• Introduces resolver-backend scaffolding (--resolver, PIPENV_RESOLVER, [pipenv] resolver) and an opt-in manifest prefetch setting (prefetch_index_manifests).

Reviewed changes

Copilot reviewed 105 out of 110 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/test_utils.py	Update unit tests for Pipfile/Lockfile subsystem API moves + new venv detection tests
tests/unit/test_update.py	Update update-routine unit tests to new project.pipfile.* APIs
tests/unit/test_unpack.py	New tests for pipenv.utils.unpack relocation + behavior pins
tests/unit/test_settings.py	Add tests for prefetch_index_manifests setting + env override; update cache invalidation paths
tests/unit/test_resolver_regressions.py	Update regression tests for resolve_constraints marker extraction behavior
tests/unit/test_resolver_parent_dispatch.py	Update resolver parent-dispatch stubs to new Pipfile subsystem fields
tests/unit/test_resolver_diagnostics.py	Add regression tests preventing resolver-log propagation to root
tests/unit/test_resolver_backends.py	New unit tests for resolver-backend registry/dispatch scaffolding
tests/unit/test_pylock.py	Update unit tests to new project.lockfile.* subsystem APIs
tests/unit/test_project_caching.py	Update Pipfile caching tests to Pipfile subsystem implementation
tests/unit/test_prefetch_fan_out.py	New unit tests for per-source verify_ssl prefetch fan-out
tests/unit/test_pep691_parity_known_diffs.md	New documentation of known parity normalizations/divergences
tests/unit/test_pep691_parity_fixtures.py	New parity tests vs pip’s Link parsing for frozen fixtures
tests/unit/test_locking_no_mutation.py	Update test to new Pipfile subsystem write API
tests/unit/test_lockfile.py	New unit tests for extracted Lockfile subsystem
tests/unit/test_lock_sync_uninstall_context_routing.py	Update context-routing tests for Pipfile/Lockfile subsystem APIs
tests/unit/test_do_update_context_routing.py	Update update context-routing tests for lockfile.any_exists
tests/unit/test_do_install_context_routing.py	Update install context-routing stub for pipfile.exists
tests/unit/test_dependencies.py	Update deps tests for new project.pipfile.project_directory usage
tests/unit/test_dependencies_bridges.py	Update “requirementslib removed” assertion to module nonexistence
tests/unit/test_core.py	Update core tests to project.pipfile.* and NON_CATEGORY_SECTIONS import move
tests/unit/fixtures/README.md	New fixture provenance documentation (PEP 691/503 snapshots)
tests/unit/fixtures/pep691/yanked-pkg.json	New synthetic PEP 691 fixture for yanked variants
tests/unit/fixtures/pep691/missing-hash.json	New synthetic PEP 691 fixture for empty-hash edge case
tests/unit/fixtures/pep503/yanked-pkg.html	New synthetic PEP 503 fixture for yanked variants
tests/integration/test_run.py	Update integration test to project.pipfile.build_script
tests/integration/test_resolver_protocol.py	Normalize new deadline_seconds + redact diagnostics in protocol canary
tests/integration/test_pylock.py	Update integration tests to project.lockfile.* APIs
tests/integration/test_pipenv.py	Update integration test to project.pipfile.proper_names
tests/integration/test_lockfile.py	Update integration tests to project.lockfile.load
tests/integration/test_install_twists.py	Comment update to reflect new pipfile writer location
tests/integration/test_install_markers.py	Update integration tests to new lockfile/hash/pipfile hash accessors
tests/integration/test_import_requirements.py	Patch target update for moved unpack_url
tests/integration/fixtures/resolver_protocol/response.json	Update protocol golden with redacted diagnostics field
tests/integration/fixtures/resolver_protocol/request.json	Update protocol golden with redacted deadline_seconds
pyproject.toml	Configure coverage to target pipenv/resolver for a dedicated gate
pipenv/utils/virtualenv.py	Update to Pipfile subsystem fields (location, project_directory, name, required python)
pipenv/utils/venv_locator.py	Update to use project.pipfile.* fields
pipenv/utils/unpack.py	Add module docstring for relocated pip-fork unpack helpers
pipenv/utils/toml.py	Use project.pipfile.get_package_categories()
pipenv/utils/sources.py	Update to project.pipfile.* + project.lockfile.* APIs
pipenv/utils/shell.py	Fix Windows Path.glob crash by guarding bindir existence
pipenv/utils/settings.py	Add env-var override plumbing + new resolver accessor + migrate to project.pipfile.parsed
pipenv/utils/pylock.py	Add TODO for resolver backend metadata in pylock conversion
pipenv/utils/project.py	Update required-python warning logic to Pipfile subsystem
pipenv/utils/locking.py	Switch to PlettePipfile loader naming after subsystem extraction
pipenv/utils/internet.py	Lazy-import pip network stack; adjust session cache_dir defaulting
pipenv/utils/fileutils.py	Lazy-import pip network utilities; avoid heavy imports at module load
pipenv/utils/environment.py	Use project.pipfile.project_directory for .env lookup
pipenv/utils/dependencies.py	Lazy-import pip internals in hot modules; migrate to new Pipfile APIs
pipenv/routines/update.py	Update update routine to new Pipfile/Lockfile APIs
pipenv/routines/uninstall.py	Update uninstall routine to new Pipfile/Lockfile APIs
pipenv/routines/sync.py	Update sync routine to project.lockfile.any_exists
pipenv/routines/shell.py	Update shell/run routines to Pipfile subsystem project directory + scripts
pipenv/routines/scan.py	Update scan to use Pipfile/Lockfile subsystem fields
pipenv/routines/requirements.py	Update requirements generation to new Pipfile/Lockfile APIs
pipenv/routines/outdated.py	Update outdated routine to new Pipfile APIs
pipenv/routines/install.py	Update install routines to new Pipfile/Lockfile APIs; propagate --resolver + --clear
pipenv/routines/context.py	Add ExecutionOptions.resolver plumbing from CLI
pipenv/routines/clean.py	Update clean routine to new lockfile hash/package-names accessors
pipenv/routines/check.py	Update check routine to use Pipfile/Lockfile subsystem fields
pipenv/routines/audit.py	Update audit routine to use project.lockfile.* APIs
pipenv/resolver/schema.py	Add additive ResolverOptions.backend + suppress empty backend on wire
pipenv/resolver/pep691_types.py	New typed envelopes for Simple API fetch results/errors
pipenv/resolver/main.py	Fix resolved-default-deps structure + avoid PackageFinder second-pass marker lookup
pipenv/resolver/core.py	Add resolver-log propagation fix + implement backend dispatcher scaffolding
pipenv/resolver/candidate.py	New pure-Python Candidate/Hash data model
pipenv/resolver/backends/pip.py	New pip backend adapter wrapping existing resolve flow
pipenv/resolver/backends/base.py	New Backend protocol + shared registry
pipenv/resolver/backends/init.py	New backend registry + lookup helpers
pipenv/resolver/auth.py	New pure-Python auth helpers (netrc, URL creds, client cert)
pipenv/resolver/init.py	Re-export new Initiative G resolver surface
pipenv/help.py	Update diagnostics printing to new Pipfile/Lockfile subsystem locations
pipenv/environments.py	Add env vars for prefetch + resolver backend selection
pipenv/environment.py	Lazy-import InstallCommand to reduce startup cost
pipenv/cli/options.py	Add --resolver flag plumbing into CLI state
pipenv/cli/command.py	Thread resolver selection into contexts; update to Pipfile subsystem APIs
news/T_F.5.feature.rst	News fragment for resolver backend scaffolding
news/initiative-g-phase2-prefetch-bridge.feature.rst	News fragment for opt-in manifest prefetch
news/initiative-g-phase1-pep691-client.feature.rst	News fragment for PEP 691/503 client + cache surface
docs/pipfile.md	Document [pipenv] prefetch_index_manifests setting + env var
docs/dev/modernization-plan.md	Update task status/logs for Initiative D/E/F/G work
docs/dev/initiative-f-backends-design.md	Add maintainer sign-off decisions and updated scope
.pre-commit-config.yaml	Add guard preventing pip internal imports in pipenv/resolver/
.github/workflows/ci.yaml	Add resolver-module coverage gate job

Comments suppressed due to low confidence (1)

pipenv/resolver/backends/base.py:76

• REGISTRY is annotated as dict[str, Backend], but the registry actually stores backend classes (e.g. REGISTRY["pip"] = PipBackend) as well as instances in tests. This type mismatch will confuse type checkers/IDEs and contradicts the module docstring (“dict from name to backend class”). Update the type to something like dict[str, type[Backend] | Backend] (or introduce a dedicated BackendEntry type alias) so the registry’s intended contents are accurately represented.

# Single shared registry.  The ``backends/__init__.py`` populates this
# on import with the in-tree backends.  Keeping it here (rather than on
# ``__init__``) lets test code patch via ``mock.patch.dict`` against a
# single canonical reference no matter which module the patch targets.
REGISTRY: dict[str, Backend] = {}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[matteius]

Restacked onto the updated Phase IV tip and corrected this PR's base from the obsolete maintenance/code-cleanup-phase4-resolver-followups-2026-05 branch to maintenance/code-cleanup-phase4-deferred-2026-05. Range-diff confirms all 46 Phase V commits remain patch-equivalent.

Local verification: 1,283 unit tests passed, 10 platform skips; full pre-commit suite passed.

[matteius]

Follow-up: CodeQL flagged the integration test hostname substring assertions. Commit 75fe0a3 now parses each URL and compares exact hostnames; Ruff and diff checks pass. An explicit merge of current main reports Already up to date.

[matteius]

Merged the refreshed Phase IV head into Phase V; an explicit current-main merge then reported Already up to date. Full unit suite: 1,284 passed, 11 skipped.