RFC: Drop ppc64le

[minrk]

Headline: if you are a ppc user of conda-forge, please make yourself known with a comment below, describe some of the packages you use and how/where!

I propose that conda-forge stop supporting ppc64le.

In general, ppc seems to have dropped off significantly in HPC, to the point where it is unclear if conda-forge has more than 0 active ppc users.

ppc64le support has been very costly, both in maintainer time and CI costs, while also being the least widely used architecture across conda-forge.
As an increasingly rare architecture, it has often had problems seen on no other platforms, and cost a lot of time to debug.

It is hard to prove that users don't exist, but so far, every known ppc user who has been pinged about ppc support issues has said something to the effect of "I don't have use ppc anymore." To my knowledge, not one current ppc user has spoken up in various conversations about ppc support troubles over the last couple of years.

Supporting evidence

• Summit has now been retired. It has been replaced by the amd64 machine Frontier. I'm not aware of any active ppc64le systems in US national HPC centers.

  ref

• From a UK HPC colleague:

  From my colleagues at the Cambridge HPC system:

  PowerPC is pretty uncommon these days. The only national (well, non-local) HPC system in the UK I’m aware of that still has it is BEDE.

  ref

• As an ex-user of HAL & ex-VAP in Statistics @ UIUC, I can attest that HAL was decommissioned on October 1st, 2025. The main reason was the system age and better systems came online for UIUC's campus, c.f. DeltaAI

  ref

• CUDA dropped ppc support in 12.4

• Conda-Forge download statistics are consistent with ~100% of ppc downloads being from conda-forge's own CI (inconclusive)

Options

1. stop adding ppc64le builds, except by request (least disruptive), keep current low bar for manual removal
2. actively remove ppc64le builds, and keep only those specifically requested (saves more maintainer and CI time, but needs to keep all infrastructure)
3. fully prevent ppc64le builds, removing support from conda-smithy, conda-forge-pinning, etc. (most drastic because I'm not sure maintainers can keep ppc builds if we do this).

I think we should do option 3 2.

Tasks

• announce RFC, call for ppc users to speak up (I don't think any have in other conversations, yet)
• announce decision
• remove ppc from aarch migrator to stop getting new builds: remove ppc from arch migrator conda-forge-bot#5862
• linter hint to suggest ppc removal: add hint to remove linux_ppc64le conda-smithy#2504
• stop building native ppc compiler packages? (not sure about this one)

Originally suggested, but not necessary

• remove ppc from conda-forge-pinning
• remove ppc from docker-images
• remove ppc from conda-smithy
• removal migrator (only if remove on rerender is not going to be enough)

References:

• related discussion: when to drop (or add) ppc support #2510
• previous rfc, dropping pypy: RFC: Drop PyPy #2255

[leofang]

I second that we remove ppc support asap.

[danielnachun]

Thanks for initiating this. Two other motivations for this discussion are worth mentioning:

• We now have native ARM runners for both macOS and Linux, and will hopefully have them for Windows soon, meaning that cross-compilation will be great reduced compared to before.
• While there was a period where other manufacturers were making POWER systems, that is no longer the case after POWER9, so it seems the only people purchasing new PPC64LE hardware are legacy enterprise customers who are vendor locked to IBM. This is tiny market share and a significant fraction of those users are running AIX, not Linux.

Although I think we need greatly reduce PPC64LE support, I'm not sure all of these steps have to be done right away. The most important one is to drop ppc64le from the Linux arch migrator. It does not make sense to couple it to adding support for aarch64 on Linux, whose relevance has grown significantly both due to the growth of native Linux aarch64 systems as well as the ability to run Linux VMs on Apple Silicon without needing to emulate the architecture.

As for some of the other steps:

• conda-forge-pinning: right now it doesn't seem like ppc64le creates too much complexity there, but others may disagree. Importantly, individual feedstocks can override these values (though there are some consequences to doing this). As long as it doesn't contribute any substantial maintenance burden it might be good to keep it around, but I don't feel very strongly about this.
• Docker images: I think it's not a bad a idea to build these for PPC64LE (I think we also build for S390?) provided that:
  • the upstream distro (currently Almalinux) continues to support PPC64LE
  • the specific Docker image for PPC64LE does not require extra maintenance burden as compared to other architectures
• conda-smithy: similar to the Docker images, if this doesn't require extra maintenance burden, it might be better to keep it around in case we do need to add it to certain feedstocks
• compiler packages: this is the change that we should be most cautious about. The cross compilers (target PPC64LE from x86_64 and aarch64) are genuinely useful even outside of conda-forge if some wants to quickly set up a cross-compilation toolkit. The native compilers (PPC64LE on PPC64LE) could still be built through cross-compilation for the sake of speed, but are probably only useful when emulating, which as we know is slow. If we are building cross compilers from PPC64LE to other architectures, we could definitely consider dropping those as that seems extremely niche.
• removal migrator: As noted above if it's isn't possible to achieve this by rerendering alone, this would be needed. If we decide not to remove PPC64LE from conda-smithy, I think having some way to still keep PPC64LE as an option would be good. But I would very much like to have it be dropped automatically from the feedstocks I maintain, as I have no personal interest in supporting it unless a user specific requests it for a feedstock.

[jaimergp]

The most important one is to drop ppc64le from the Linux arch migrator.

I think this is the most obvious next step. (edit: see conda-forge/conda-forge-bot#5866)

If we decide not to remove PPC64LE from conda-smithy, I think having some way to still keep PPC64LE as an option would be good.

We can include a hint or message during rerender to announce the intent to drop ppc64le, pointing to this issue. I don't know if we have a mechanism to communicate to the user via rerenders, but I think it'd be useful. Otherwise, the linter may suffice.

[minrk]

I agree we should only remove it where there is a cost to maintenance. If the compiler toolchains don't create a cost (e.g. we'll keep the ppc cross target, but may lose future builds of native ppc compilers if we stop ppc builds). I guess the same goes for conda-smithy, since ppc is 'just a platform' like any other.

If we do stop ppc native builds of compilers, that will also mean that ppc builds will stop working for any feedstock that doesn't cross compile.

I'd be fully on board with a recipe hint suggesting that removal of ppc is permitted, and stopping the migrator (already open at conda-forge/conda-forge-bot#5862), if that's enough. Suggested hint: conda-forge/conda-smithy#2504

IIUC, ppc is built if linux_ppc64le shows up in either providers or build_platform specifies linux_ppc64le: linux_64 for cross?

So that leaves really the only todo list as:

• announcements
• stop migrator adding more ppc builds: remove ppc from arch migrator conda-forge-bot#5862
• hint: add hint to remove linux_ppc64le conda-smithy#2504

This is the gentlest removal, I think.

[isuruf]

If the compiler toolchains don't create a cost (e.g. we'll keep the ppc cross target, but may lose future builds of native ppc compilers if we stop ppc builds). I guess the same goes for conda-smithy, since ppc is 'just a platform' like any other.

It's not a burden to keep the toolchain alive. We'll keep the ppc cross and native compilers in ctng-compilers-feedstock. All of them are built on x86_64 anyway.

[clnperez]

Hi conda-forge team. Thank you for opening this and giving folks a chance to comment. We (IBM Power and Friends) certainly can never know exactly how everyone using our platforms operates, so public RFCs such as this are much appreciated.

We certainly don’t want to burden the community and can offer assistance to projects in a few ways. After some internal discussion with a small group, we’ve organized a few thoughts:

It's not a burden to keep the toolchain alive. We'll keep the ppc cross and native compilers in ctng-compilers-feedstock. All of them are built on x86_64 anyway.

Great news!

I agree we should only remove it where there is a cost to maintenance.

We do have P10 systems at OSUOSL (and P11 systems soon). OSUOSL have provided Power VM’s for CI/CD for both Anaconda and conda-forge since 2016 and 2019, respectively. OSUOSL provides great admin support, when needed. They would be glad to spin up Power 10/ppc64le (P10) and soon P11 VMs to support conda-forge going forward and eliminate CI costs.

AlmaLinux has a dedicated Power machine at OSU to focus on ongoing Power support, along with Alpine, Rocky, Debian, FreeBSD, etc..

We now have native ARM runners for both macOS and Linux, and will hopefully have them for Windows soon, meaning that cross-compilation will be great reduced compared to before.

If this is one of the concerns for Power (cross-compilation being a burden), we also have recently introduced Power runners for GH Actions, but also could have OSUOSL help you integrate VMs as runners if that would be useful to you.

While there was a period where other manufacturers were making POWER systems, that is no longer the case after POWER9, so it seems the only people purchasing new PPC64LE hardware are legacy enterprise customers who are vendor locked to IBM. This is tiny market share and a significant fraction of those users are running AIX, not Linux.

This is a great opportunity for us to do a little bragging. Yes it’s true that we have many “legacy” enterprise customers, but most are loyal, not locked in. 😀 We recognize that as a whole, Power does have a smaller footprint than other hardware manufacturers in the world as a whole -- but at the same time, we’re seeing adoption by our existing customers of Linux on Power as well as expanding their existing workloads on AIX and i. We have a strong OpenShift on Power (and z) customer base, we’ve been certified as a Hyperscaler for SAP RISE (link https://www.ibm.com/products/rise-with-sap) through Power Virtual Server (https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-getting-started) just to name a few of our non AIX & i offerings.

It appears most IBM customers on Power have moved to pip based ppc64le/pyeco for their packaging needs. But given that we can’t know for sure what everyone does with their systems, whatever we can reasonably ask you to keep, we’d like to.

We appreciate the gentle-ness of your move as needed.

[mtarsel]

You can submit a request on our actionspz repo to use our github actions service for power and z. Here's the FAQ page for more info: https://github.com/IBM/actionspz/blob/main/docs/FAQ.md

Additionally, here is more info about OSU OSL: https://osuosl.org/services/powerdev/

[minrk]

Thanks, that's useful information! I imagine ppc workers would be appreciated with our move support for github-actions if we want to keep supporting ppc, especially with native runners.

Though we still haven't heard from any users of conda-forge on PPC. @mtarsel / @clnperez do you have any user contacts that are known to be using conda-forge on ppc?

[danielnachun]

Thanks, that's useful information! I imagine ppc workers would be appreciated with our move support for github-actions if we want to keep supporting ppc, especially with native runners.

Though we still haven't heard from any users of conda-forge on PPC. @mtarsel / @clnperez do you have any user contacts that are known to be using conda-forge on ppc?

I agree 100% here on both points. Native runners would make the maintenance burden very minimal for us. But it's important that we have real users actually using our packages routinely. We have to balance the thoroughness of our tests in CI with maintainability and compute usage, but real users actually put our packages through the paces. So hearing from end-users will help us know that our packages are working as intended in practical use cases.

[jackpots28]

Hi - I'm reaching out on behalf of a large enterprise organization to respond directly to the call for ppc64le users to make themselves known. We are active, current users of ppc64le, and dropping support would have serious consequences for our workflows.

Who we are

We are a private enterprise running a fleet of ~20 ppc64le nodes spanning both POWER10, and POWER11 hardware. POWER11 is IBM's current-generation platform, actively sold and supported - this is not legacy infrastructure we are winding down. It is an ongoing, strategic investment.

How we use conda-forge on ppc64le

Our ppc64le systems are used for AI/ML training and inference workloads, and conda-forge is a foundational part of how we manage our software environment. We rely heavily on packages from both the Python scientific stack (numpy, scipy, pandas, and related libraries) and ML/AI frameworks. Losing ppc64le builds from conda-forge would leave us with a combination of painful outcomes:

• Building and maintaining packages from source ourselves, which is a significant ongoing engineering burden
• Losing access to packages that have no viable ppc64le alternative, stalling workloads entirely
• Facing pressure to migrate to a different architecture, which would require substantial capital investment and operational disruption

None of these are trivial. For an organization running active AI/ML workloads at this scale, conda-forge's ppc64le support is not a convenience - it is infrastructure we depend on.

We have an extensive list of AI/ML integral packages we source from conda-forge since Anaconda burned bridges with us.

On the question of CI cost and maintainer burden

We understand that the core concern driving this RFC is the real and legitimate cost of maintaining ppc64le CI infrastructure, and the maintainer time spent debugging platform-specific issues. We take that seriously, and we don't want to simply ask the community to absorb costs on our behalf.

To that end: we have deep relationships with IBM and other business partners, and we are willing to advocate directly for IBM to donate or sponsor CI build capacity for ppc64le on conda-forge. We believe this is a realistic path given IBM's continued investment in the POWER platform and their interest in keeping the ecosystem healthy. We would be glad to facilitate introductions or support that conversation however we can.

Our position on the proposed options

Of the three options outlined in the RFC, we would strongly prefer Option 1 (stop adding new ppc64le builds except by request, keep the current low bar for manual removal) as the least disruptive path - and ideally we'd work toward a funded CI solution that makes even that trade-off unnecessary. Option 3 (removing support from conda-smithy entirely) would be deeply damaging and very difficult to reverse if enterprise ppc64le use turns out to be more widespread than current signal suggests.

We recognize the burden the community has carried, and we're not here just to object - we want to be part of a solution. Please don't make this decision without hearing from enterprise users first. We're here, we're active, and we're willing to help.

[minrk]

@jackpots28 thanks for sharing, that's super helpful!