Tracking issue to enable `rattler-build` in conda-forge

[wolfv]

I would like work towards enabling rattler-build on conda-forge and the new recipe format in Q1 2024.

This is a tracking issue for the remaining tasks:

• Stabilize recipe format and "rendered" recipe format. For this we mainly use the CEP process:
  • add cep for definition of new keys & values in build format conda/ceps#56
  • A new test for file existence inside a package (for the new recipe format) conda/ceps#57
• Write a CFEP (conda-forge enhancement proposal) outlining how the transition works and get it voted on
• Make conda-smithy aware of the new recipe format
• Make the bots aware of the new recipe format

Other notes and important items

• The bot depends on the jinja2 parsing and making changes to it in order to generate certain bot PRs, especially the ones related generating pinning migrations.
• We should not migrate any of our core services feedstocks until things are very stable.

[0xbe7a]

• Adapt conda-forge-ci-setup to output folder structure of rattler-build (see "Validating outputs" here)

[beckermr]

I want to comment that we should not have to adapt conda-forge-ci-setup to anything. Besides calling rattler-build itself, everything else should be made compatible including where packages are output. Otherwise, people are going to run into all sorts of weird issues when adopting it.

[0xbe7a]

When building a package, conda-forge-ci-setup expects a (possibly) empty folder for the build platform and noarch iirc. I would argue that conda-forge-ci-setup should simply discover all output architectures. But that only requires a change of < 10 lines.

[beckermr]

Discovering all outputs is IMHO a security issue.

[0xbe7a]

Can you explain why?

[beckermr]

conda-build currently tells us exactly what the recipe builds, so we should only attempt to upload that. Anything else we find is not supposed to be there and we should reject it even if we think a later stage in the upload process would take care of it.

[jaimergp]

Make conda-smithy aware of the new recipe format
Make the bots aware of the new recipe format

@xhochy, @beckermr, @Hind-M and I discussed this in the last infra monthly. Some of the items that came up (please correct me if I am misremembering):

• The current bot infra (including regro/cf-scripts and conda-smithy) assume meta.yaml-like recipe contents in many places. Adjusting the code to be compatible with the upcoming recipe.yaml format might be tricky without a significant refactor. At that point, other benefits might be within reach, so we might be able to fix / improve certain extra aspects of the infra at a relatively small cost.
• New recipe.yaml revisions might be added in the future. It might be worth considering having an abstraction layer with versioned recipe formats, including a "backend" for meta.yaml. This could be one way to tackle the problem.
• While recipe rendering and the "time-to-build.sh" metrics will be significantly better with rattler-build, we will still need to pay the set-up times for a base conda installation because most of the infra uses that (e.g. conda-forge-ci-setup). One possible way to accelerate this part is to revisit @mariusvniekerk's idea of using lockfiles to provision the base environments.

[pavelzw]

What is contained/needed in the "base conda installation"?
rattler-build is a single statically linked binary that can also upload to anaconda.org. If only building and uploading is needed, we could also just download this single binary...

[xhochy]

There is a lot of logic in conda-forge-ci-setup. This logic is mostly independent of the used build tool.

[nichmor]

conda-forge/conda-forge-ci-setup-feedstock#316 was merged and started support rattler-build for upload

[tdejager]

Current status and issues, we are working on them now, this should be seen in additon to @wolfv list above (this is mostly updating the bots part):

Done:

• Allow to up update teams in `recipe.yaml: add rattler-build recipe functionality for update-team conda-forge-webservices#630
• Add recipe.yaml support in staged-recipes: Add support for recipe.yaml files using rattler-build staged-recipes#26981
• Trigger feedstock generation for `recipe.yaml: feat: runs feedstock generation when recipe.yaml exists admin-requests#1030

Doing:

• Add support for recipe.yaml in: feat: add support for rattler-build recipe.yaml admin-migrations#109
• Add linting of recipe.yaml in conda-simity: feat: add rattler build linting conda-smithy#1979
• Fill graph based on recipe-yaml in cf-scripts: feat: Extend feedstock functions to allow for recipe_yaml conda-forge-bot#2845 and Build graph from recipe.yaml conda-forge-bot#2851
• Fix staged-recipe build CI: Adds rattler-build recipe compatibility (and adds jolt-physics) staged-recipes#27008
• Add linting of recipe.yaml in conda-forge-webservices: feat: handle recipe.yaml when linting conda-forge-webservices#631

Not Started:

• Update version in recipe.yaml for cf-scripts
• Add parse_recipe_yaml in cf-scripts`
• "Add user command should update recipe.yaml" in conda-forg-webservices

[h-vetinari]

conda-forge-ci-config currently pins rattler-build-conda-compat >=0.0.2,<1.0.0a, which will presumably have to be fixed now that other parts like smithy are starting to require >1.