From e4599bf2c7e5bb6bc1ec2b2758b322347c8a7b0d Mon Sep 17 00:00:00 2001 From: Bas Zalmstra Date: Thu, 16 Jan 2025 13:53:46 +0100 Subject: [PATCH 1/6] run_exports in sharded repodata --- cep-0016-2.md | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 cep-0016-2.md diff --git a/cep-0016-2.md b/cep-0016-2.md new file mode 100644 index 00000000..2ee73bed --- /dev/null +++ b/cep-0016-2.md @@ -0,0 +1,67 @@ + + + + + + + + +
Title Run-exports in sharded Repodata.
Status Draft
Author(s) Bas Zalmstra <bas@prefix.dev>
Created Jan 16, 2025
Updated Jan 16, 2025
Discussion
Implementation
+ +# Run-exports in sharded Repodata + +We propose to add run-export information to the sharded repodata shards. + +## Motivation + +When building conda packages the build infrastructure needs to extract run-export information from conda packages in the host- and build environments. +Run-export information is stored in a package and can be extracted by downloading the package and extracting the `run_exports.json` file. +Even with the possibility to stream parts of `.conda` files this is a relatively resource-intensive operation. + +[CEP-12](https://github.com/conda/ceps/blob/main/cep-0012.md) formalized a `run_exports.json` file that is stored next `repodata.json` file. +However, not all channels on the default server (conda.anaconda.org) provide this information which means falling back to downloading and extracting this information from the packages. It is possible to extract the data by only sparsly reading the file but the overhead is still relatively large. + +Having two separate files also poses some problems as extra mechanisms have to be introduced in the build infrastructure to manage and sync both files on the build machines. + +## Specification + +CEP-12 mentions the following reasons for splitting the information into two files: + +> * It would require extending the repodata schema, currently not formally standardized. +> * It would increase the size of the already heavy repodata.json files. +> * (Typed) repodata parsers would need to be updated to handle the new field. + +We propose that these reasons no longer hold with [sharded repodata](https://github.com/conda/ceps/blob/main/cep-0016.md). + +**It would require extending the repodata schema, currently not formally standardized.** + +We propose to add a `run_export` field to each record that mimics the specification from CEP-12. + +If the `run_export` field is not present in the record it means no `run_export` information is stored with the record, and a fallback mechanism should be used to acquire the run-export information. + +Since *adding* a field will not break existing parsers we feel this is safe and does not require a schema change. + +**It would increase the size of the already heavy repodata.json files.** + +The size of the shards would grow if run-exports are added. + +Let's take a look at the current sizes of run_exports.json and repodata.json files. + +| channel + subdir | size of repo_data.json | size of run_exports.json | repodata.json.zst | run_exports.json.zst | +|------------------|------------------------|--------------------------|--------------------------|-----------------------------| +| conda-forge + linux-64 | 254 MB | 34.7 MB (11%) | 38.4 MB | 2.2MB (5%) | +| conda-forge + noarch | 107 MB | 13.6 MB (11%) | 16.7 MB | 0.9 MB (5%) | +| conda-forge + osx-arm64 | 99.8 MB | 11.8 MB (11%) | 12.6 MB | 0.8 MB (6%) | +| conda-forge + win-64 | 185 MB | 22.1 MB (11%) | 24.7 MB | 1.4 MB (5%) | + +Since the repodata shards are also compressed we can conclude that in practice adding run exports information would increase the size of the repodata shards by roughly 5-6%. + +With the introduction of sharded repodata in [CEP-16](https://github.com/conda/ceps/blob/main/cep-0016.md) the issues with size (and scale) have been effectively mitigated. Adding 5-6% to the total size of the shards will not pose a risk since all advantages of sharded repodata mentioned in the CEP still hold. + +**(Typed) repodata parsers would need to be updated to handle the new field.** + +Unless parsers are very strict about unknown fields (which was not a requirement for sharded repodata shards) this will not pose an issue. Since the absence of the field means that the state of the run-exports is unknown adding the field does not require a schema change. + +## Patching + +With the run-exports part of the repodata we propose to also allow repodata patching these fields. The original run-exports can still be extracted from the packages. We do not see a reason how the run-export information is different from other patchable information stored in the repodata (like the dependencies). From 5b491b16eb714fc922eb0f712e3020b244275c9f Mon Sep 17 00:00:00 2001 From: Bas Zalmstra Date: Thu, 6 Feb 2025 14:34:32 +0100 Subject: [PATCH 2/6] run export patching --- cep-0016-2.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/cep-0016-2.md b/cep-0016-2.md index 2ee73bed..9eae95fc 100644 --- a/cep-0016-2.md +++ b/cep-0016-2.md @@ -30,7 +30,6 @@ CEP-12 mentions the following reasons for splitting the information into two fil > * It would require extending the repodata schema, currently not formally standardized. > * It would increase the size of the already heavy repodata.json files. > * (Typed) repodata parsers would need to be updated to handle the new field. - We propose that these reasons no longer hold with [sharded repodata](https://github.com/conda/ceps/blob/main/cep-0016.md). **It would require extending the repodata schema, currently not formally standardized.** @@ -65,3 +64,24 @@ Unless parsers are very strict about unknown fields (which was not a requirement ## Patching With the run-exports part of the repodata we propose to also allow repodata patching these fields. The original run-exports can still be extracted from the packages. We do not see a reason how the run-export information is different from other patchable information stored in the repodata (like the dependencies). + +To facilitate these patches a new file is added to the repodata-patches package called `run_exports_patch_instructions.json` so that `run_exports` can be modified per package: + +```json +{ + "packages": { + "_libarchive_static_for_cph-3.3.3-h0921cf1_1.tar.bz2": { + "run_exports": { + "weak": ["libarchive >=3,<4"], + } + }, + ... + } +} +``` + +For backwards compatibility reasons we propose *not* to add the patches to the existing `patch_instructions.json` file because older implementations do not have to ability to filter certain instructions. Without that ability the `run_exports` patches would end up in the `repodata.json` file which is undesirable. + +For a channel to provide run-export patches, it MUST contain a `run_exports.json` file. +For build tools to support patched run exports it MUST always query the sharded repodata, or use a `run_exports.json` file as the source of truth for the run exports. +If neither sharded repodata or a `run_exports.json` file is present build tools can assume no run export patches exist. From 97a7de54a28f02229ac60dd535ba45d9cf8cddc9 Mon Sep 17 00:00:00 2001 From: Bas Zalmstra Date: Thu, 6 Feb 2025 17:34:53 +0100 Subject: [PATCH 3/6] Update cep-0016-2.md Co-authored-by: Matthew R. Becker --- cep-0016-2.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cep-0016-2.md b/cep-0016-2.md index 9eae95fc..deba2cd7 100644 --- a/cep-0016-2.md +++ b/cep-0016-2.md @@ -80,6 +80,8 @@ To facilitate these patches a new file is added to the repodata-patches package } ``` +Implementations MUST use this file to patch any sharded repodata and patch the `run_exports.json` if that file exists. + For backwards compatibility reasons we propose *not* to add the patches to the existing `patch_instructions.json` file because older implementations do not have to ability to filter certain instructions. Without that ability the `run_exports` patches would end up in the `repodata.json` file which is undesirable. For a channel to provide run-export patches, it MUST contain a `run_exports.json` file. From 023121445e1eeb58973b493a333f5df22081d643 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 26 Feb 2025 18:01:56 +0100 Subject: [PATCH 4/6] Update cep-0016-2.md Use `patch_instructions_version: 2` --- cep-0016-2.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/cep-0016-2.md b/cep-0016-2.md index deba2cd7..a2d8309b 100644 --- a/cep-0016-2.md +++ b/cep-0016-2.md @@ -65,12 +65,13 @@ Unless parsers are very strict about unknown fields (which was not a requirement With the run-exports part of the repodata we propose to also allow repodata patching these fields. The original run-exports can still be extracted from the packages. We do not see a reason how the run-export information is different from other patchable information stored in the repodata (like the dependencies). -To facilitate these patches a new file is added to the repodata-patches package called `run_exports_patch_instructions.json` so that `run_exports` can be modified per package: +To facilitate these patches, the `patch_instructions_version` in `patch_instructions.json` files is incremented to `2`. A `patch_instructions_version: 2` file CAN contain a `run_exports: ...` field in the patch instructions that MUST be used to patch generated `run_exports.json` files. ```json { "packages": { "_libarchive_static_for_cph-3.3.3-h0921cf1_1.tar.bz2": { + ..., "run_exports": { "weak": ["libarchive >=3,<4"], } @@ -80,10 +81,6 @@ To facilitate these patches a new file is added to the repodata-patches package } ``` -Implementations MUST use this file to patch any sharded repodata and patch the `run_exports.json` if that file exists. - -For backwards compatibility reasons we propose *not* to add the patches to the existing `patch_instructions.json` file because older implementations do not have to ability to filter certain instructions. Without that ability the `run_exports` patches would end up in the `repodata.json` file which is undesirable. - -For a channel to provide run-export patches, it MUST contain a `run_exports.json` file. +For a channel to provide run-export patches, it MUST contain a `run_exports.json` file. For build tools to support patched run exports it MUST always query the sharded repodata, or use a `run_exports.json` file as the source of truth for the run exports. If neither sharded repodata or a `run_exports.json` file is present build tools can assume no run export patches exist. From d1de8d9195494728fc8e8e152e8276bbd50d4e20 Mon Sep 17 00:00:00 2001 From: Bas Zalmstra Date: Thu, 20 Mar 2025 15:50:26 +0100 Subject: [PATCH 5/6] finalize cep --- cep-0016-2.md => cep-0021.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) rename cep-0016-2.md => cep-0021.md (95%) diff --git a/cep-0016-2.md b/cep-0021.md similarity index 95% rename from cep-0016-2.md rename to cep-0021.md index a2d8309b..95acf790 100644 --- a/cep-0016-2.md +++ b/cep-0021.md @@ -1,11 +1,11 @@ - + - - - + + +
Title Run-exports in sharded Repodata.
Status Draft
Status Approved
Author(s) Bas Zalmstra <bas@prefix.dev>
Created Jan 16, 2025
Updated Jan 16, 2025
Discussion
Implementation
Updated March 20, 2025
Discussion https://github.com/conda/ceps/pull/108
Implementation NA
# Run-exports in sharded Repodata From f6462174a19ffc54821afc770a5f6ceab915730f Mon Sep 17 00:00:00 2001 From: Bas Zalmstra Date: Thu, 20 Mar 2025 16:10:13 +0100 Subject: [PATCH 6/6] small style fix --- cep-0021.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-0021.md b/cep-0021.md index 95acf790..a8af040a 100644 --- a/cep-0021.md +++ b/cep-0021.md @@ -65,7 +65,7 @@ Unless parsers are very strict about unknown fields (which was not a requirement With the run-exports part of the repodata we propose to also allow repodata patching these fields. The original run-exports can still be extracted from the packages. We do not see a reason how the run-export information is different from other patchable information stored in the repodata (like the dependencies). -To facilitate these patches, the `patch_instructions_version` in `patch_instructions.json` files is incremented to `2`. A `patch_instructions_version: 2` file CAN contain a `run_exports: ...` field in the patch instructions that MUST be used to patch generated `run_exports.json` files. +To facilitate these patches, the `patch_instructions_version` in `patch_instructions.json` files is incremented to `2`. A `patch_instructions_version: 2` file MAY contain a `run_exports: ...` field in the patch instructions that MUST be used to patch generated `run_exports.json` files. ```json {