components: add pip repo for Python distributions

[yeetypete]

Add a component repo for Python distributions installed by pip, uv, or any other installer following the "Recording installed projects" specification (PEP 376).

chunkah is pretty nice for Python-heavy images. Any ML/CUDA venvs can get to be 10s of gigs in size 😅. The classic workaround is to manually split the installation into separate RUN commands so large dependencies land in their own layers (see astral-sh/uv#16113 and astral-sh/uv#7712). With this, each distribution becomes its own component and layers can be reused across updates regardless of how the image was built.

How it works:

• Files listed in RECORD are strongly claimed, along with their parent directories up to site-packages. Directories shared between distributions end up in each component's layer, as with the other repos.
• Names are normalized per PEP 503, so Foo_Pkg-1.2.3.dist-info becomes pip/foo-pkg.
• Bytecode compiled after installation (not in RECORD, e.g. by uv's --compile-bytecode) is claimed for the component owning the corresponding .py.
• Distributions vendored inside another one (e.g. setuptools/_vendor, see Stop vendoring packages pypa/setuptools#2825) belong to the outer distribution.
• We set the PipRepo priority below distro package managers so Python packages installed by the distro package manager stay with the distro package.
• PyPI packages don't have any changelog data. We assume a fixed 30-day update interval for stability.

Assisted-by: AI
Assistance-level: 4

[yeetypete]

@jlebon if you have a chance to take a look that would be great! I think this could be a big win for people doing ML stuff with lage Python wheels + using containerization :)

[jlebon]

Thanks for opening this! I'm mostly AFK these days which is why turnaround is slow. Please bear with me until I'm fully back online. :)

Alongside the AI disclosure, one thing that would be helpful (which I'd like to eventually reflect in the guidelines) is basically alongside Assisted-by: AI, to have something like Assistance-level: which would specify e.g. some level on a spectrum (a recent proposed strawman I saw go by was https://sph.sh/en/posts/ai-assistance-spectrum-professional-software-engineering/ but I'm sure there are others). All this to ask: how much time/effort was spent on reviewing the AI output?

Commit bac2943 should be folded into commit 067d612. Commit 89824ea should be split per-file and folded into the respective earlier commits which add those files. (Also fine if you just fold the e2e test into 067d612.)

[jlebon]

One question before reviewing this more deeply: did you check if there were any pure Rust libraries (with minimal deps) which can parse the dist-info format into nice data structures already? For ALPM the answer was no (at least at the time), which is why we went ahead and implemented a basic parser.

[yeetypete]

Yeah I took a look at a collection of uv crates including uv-install-wheel that could be useful. They provide RECORD parsing, wheel / sdist file name parsing, and METADATA structs. The implementation here closely follows the uv one.

Unfortunately uv advertises all published internal crates as unstable so I'm not sure it would be a great idea to use them here: https://github.com/astral-sh/uv/blob/main/docs/reference/policies/versioning.md#crate-versioning. These crates also pull in a lot of other unrelated uv and third party crates that we don't really need.

[yeetypete]

Alongside the AI disclosure, one thing that would be helpful (which I'd like to eventually reflect in the guidelines) is basically alongside Assisted-by: AI, to have something like Assistance-level

Thats a neat post, I updated the PR description to include this. As is expected I fully self-reviewed the code before submitting the PR :)