Replace JSON API usage

[cooperlees]

Motivation

bandersnatch currently fetches full per-package metadata from PyPI's legacy JSON API
(GET /pypi/<project>/json) for every package, every sync (Master.get_package_metadata,
src/bandersnatch/master.py:204-213). Package-listing/changelog discovery already moved to
the PEP 691 Simple API in 7.0/8.0, and 8.0.0 added PEP 658/714 core-metadata mirroring — but
the per-package metadata fetch itself is still the old, Warehouse-specific JSON API,
unconditionally, for every package.

This has come up before without resolution (#1124, #287, #548), and in #1124 I wrote:

We could also evaluate pulling the simple API directly from PyPI too ... unless we feel us
calculating it saves a lot of network calls / is overall cheaper ... would save a lot of
code / PEP implementation ...

There's also precedent for PyPI wanting mirrors off the heavy per-project JSON blob: a 2022
backwards-incompatible change (discuss.python.org/t/17154) dropped releases from the
versioned JSON endpoint specifically because computing it for high-release-count projects
was taking ~2.5s and destabilizing web workers. The unversioned endpoint bandersnatch uses
still returns the full blob today, but it's the same class of expensive endpoint, and it's not
hard to imagine PyPI wanting to trim it further or deprecate it outright eventually.

Goal of this issue: figure out exactly what, if anything, only the JSON API can give us, and
use that to scope what a PEP (or PEPs) would need to add before bandersnatch could drop the
JSON API and run entirely on PEP-defined interfaces (Simple API + core-metadata).

Proposal: mirror PyPI's Simple API output directly, instead of regenerating it

Every field bandersnatch's own sync/download logic hardcodes on today has a 1:1 counterpart
already in the upstream PEP 503/691/700/658/714 Simple API (see table below). In other words,
src/bandersnatch/simple.py's generate_html_simple_page/generate_json_simple_page
(~150 lines: digest-name selection, HTML tag generation, JSON shape assembly) is bandersnatch
recomputing something PyPI has already computed correctly, from a fatter JSON blob than it
needs.

If we're going to stop depending on the JSON API for this data, the simplest shape isn't "swap
the data source but keep re-deriving our own Simple pages from it" — it's to fetch and store
PyPI's own GET /simple/<project>/ HTML/JSON responses verbatim (only doing local
URL-rewriting for root_uri/relative paths), the same way json = true already stores the
raw JSON API response verbatim today (Package.save_json_metadata,
src/bandersnatch/mirror.py:615-639). That lets us delete the reimplementation rather than
port it — one less place bandersnatch's own code can drift from what PyPI actually serves.

#1440 ("package json digest dict mapped to simple json hashes dict causes pip >23 to fail") is
a concrete example of the cost of reimplementing: a bug in bandersnatch's own
digests→hashes translation broke pip installs against bandersnatch mirrors. Mirroring PyPI's
already-correct Simple API output verbatim sidesteps that whole class of bug. This feels like
the right default philosophy for a mirroring tool: mirror what the index already generated,
don't recompute it.

What bandersnatch's core sync loop needs — already 1:1 in the Simple API today

No gap. Every field bandersnatch's download/currency/digest logic reads
(src/bandersnatch/mirror.py, src/bandersnatch/simple.py) is already in PEP
503/691/700/658/714:

Field bandersnatch uses	Simple API source
filename	PEP 503/691 files[].filename
url	PEP 503/691 files[].url
size	PEP 700 files[].size
upload_time_iso_8601	PEP 700 files[].upload-time
requires_python (per file)	PEP 503/691 data-requires-python / files[].requires-python
digests/hashes (sha256)	PEP 503 URL fragment / PEP 691 files[].hashes
yanked/yanked_reason	PEP 592 data-yanked / files[].yanked
core-metadata digest	PEP 714 data-core-metadata / files[].core-metadata
last_serial	meta._last-serial / X-PyPI-Last-Serial header
project name, versions	PEP 503 listing / PEP 700 files[].versions

What the optional filter plugins can access today — the real gap

docs/filtering_configuration.md states that "all metadata provided by json is available" to
the regex_project_metadata / regex_release_file_metadata / metadata_filter.py plugins,
which do unbounded dotted-path matching over the raw JSON response (info.*, release_file.*
— e.g. classifiers, requires_dist, license/license_expression, provides_extra,
author/maintainer, project_urls, packagetype, and anything else PyPI happens to put
under those keys).

These plugins are what actually depend on the unlimited JSON blob, and would need to be
ported to whatever replaces the JSON API. Most of that data is derivable per-file from a PEP
658/714 core-metadata fetch (they're plain Core Metadata headers), with two caveats worth
flagging rather than solving here: core-metadata sidecars are wheel-only today (sdists never
get one), and fetching them means one extra HTTP request per release file instead of today's
single request per package. Whether that's an acceptable trade for these plugins, or something
worth asking PyPI to close via a PEP (e.g. sdist coverage, or embedding a filterable-field
subset directly in the Simple API JSON), is an open question for this thread.

Fields genuinely only in the JSON API, with no Simple API or core-metadata equivalent

• downloads (project- and file-level) — hardcoded -1/-1/-1 server-side in Warehouse.
  Dead/fake data, not a real gap.
• bugtrack_url — hardcoded None always server-side. Dead field.
• docs_url — legacy PyPI-hosted-docs feature retired in 2018, almost always None.
• package_url/project_url/release_url — mechanically reconstructable from the
  (normalized) project name + version. Not real metadata.
• ownership (roles/organization) — no Simple API or core-metadata equivalent, and not
  something a mirror actually needs.
• vulnerabilities (OSV-sourced advisory data) — no public read endpoint besides the JSON
  API today. This is the one legitimate, actionable gap here if a mirror wants to surface
  advisory data to its users.
• Extra digest algorithms md5, blake2b_256 — Simple API only exposes sha256.

Aside

Worth noting the gap runs both ways: PEP 740 provenance/attestations and PEP 792
project-status are Simple-API-only fields with zero JSON API equivalent. bandersnatch mirrors
neither today, independent of this discussion — not proposing we tackle that here, just
flagging it exists.

Prior art

• Discuss: Support Core Metadata Mirroring #1124 — Discuss: Support Core Metadata Mirroring
• JSON duplication and PEP 566 #287 — JSON duplication and PEP 566
• Proposal: A general API for PyPI mirroring tools #548 — Proposal: A general API for PyPI mirroring tools
• package json digest dict mapped to simple json hashes dict causes pip >23 to fail #1440 — package json digest dict mapped to simple json hashes dict causes pip >23 to fail
• Backwards Incompatible Change to PyPI JSON API

Open question for this thread

Given the above, do we want to:

1. Move to mirroring PyPI's Simple API output directly (deleting our own generation code) and
   accept the N+1/wheel-only cost for the filter plugins that need full metadata, or
2. Push for a PEP to close the sdist-coverage / per-file-request-amplification gap first, so
   the filter plugins can be ported cleanly, or
3. Leave the JSON API as a "for now" dependency and revisit once PyPI's own APIs evolve
   further.