load list of modules in one go as opposed to running separate module load commands for each of them

[boegel]

This should help quite a bit with resolving the performance regression reported in #5052, since all dependencies are being loaded in one go rather than one by one, at least for the "parent" installation.

We're still doing a separate module load for the toolchain, but I won't look into that yet since changing that is lot more involved (there are separate private methods called from Toolchain._load_modules: _load_toolchain_module and _load_dependencies_modules).

As I write this (rubber ducking 🦆 ) , I realize that I'm not sure how much impact this will have for extensions, since we only load the "fake" (temporary) module file for each extension (so a single module load) via EasyBlock.fake_module_environment...

[branfosj]

Using an easyconfig that is the first 49 extensions from R-bundle-CRAN/2025.11-foss-2025:

• EB 4.9.4: 14:14
• develop: 30:34
• With this PR: 28:24

And with an easyconfig that is the first 12 extensions from R-bundle-CRAN/2025.11-foss-2025:

• EB 4.9.4: 5:46
• develop: 11:44
• With this PR: 10:49

Each of these are timings for a single build. So this PR might improve the situation a little, but it does not get us anywhere close to EB4.

[boegel]

I just dug into this a bit more, partially because we're planning to get rid of our SSD-based shared scratch filesystem soonish, which is making me switch to our regular GPFS scratch filesystem for my EasyBuild test environment.

This results in a significant slowdown of test builds, to the point where an installation of for example Python/3.11.3-GCCcore-12.3.0 takes ~61min now, compared to the ~13min it took before (on SSD scratch)...

Looking into this a bit, a significant amount of the time goes to loading modules for (build) dependencies, which can be slow because EasyBuild internally sets $LMOD_IGNORE_CACHE (because often the modules for the dependencies are not in the Lmod cache yet, or may never be, like for my own test environment).

For Python/3.11.3-GCCcore-12.3.0, a grep in the log shows 39 module load commands being run:

• The first 12 are the initial setup of the build environment. With the change proposed in this PR, those would reduced to 2 module load commands: one for the toolchain + one for all dependencies combined (and with a bit more work in Toolchain.prepare, it could be a single one).
• For every extension (8, in this case), there are 3 module load commands run when setting up the build environment for the extension: one for the "fake" Python/... module, plus two more because there are 2 build dependencies in this case (which don't get loaded as a dependency for the fake module).
  • This can be reduced to a single module load command per extension relatively easily, with some work in both EasyBlock.install_extensions_sequential and EasyBlock.install_extensions_parallel.
  • With a bit more work, it can be recuded to a single module load command that is used to figure out the build environment for extensions, caching it, and then restoring that before starting the installation of an extension.
• At the end, there are 3 more module load commands run for the Python/... module itself. That must be part of the sanity check, and can probably be reduced to a single one as well with a bit of effort (at least in most cases).

When environment modules & the installation for the toolchain & dependencies are on slow filesystems, it's definitely worth reducing the module load commands being run...
In our case, on our non-SSD scratch filesystem, some of those module load commands take ~10 seconds, some even take ~30 sec. Very unpredictable of course, depending on the load on the filesystem, etc.
Assuming 20 seconds each, these module load commands alone account for ~12min of the total installation time, roughly 1/5th of the total installation time, which is obviously absurd.

Next to the module load commands, there's also others:

• 11 module use --priority commands (each taking 10-30 sec)
  • see also Avoid using a priority in prepend_module_path (Lmod) to avoid costly module calls #3636
• 11 module unload Python commands (each taking 10-30 sec)
• 9 module show GCCcore/12.3.0 commands (each taking 10-30 sec)
• 11 module --terse list commands (but these are instant)

A big chunk of these can probably also be avoided with caching the build environment and restoring it...

I also see this as a proxy for accessing other files, i.e. the actual binaries/libraries/header files etc. that are provided by the dependencies. That can't be fixed easily, of course. We could check how many stats Lmod does for a single module load...

Also, we're still running Lmod 8.7, significant performance improvements have been implemented in Lmod 9.x, which would also help.

Long story short: there's clearly a lot of room for improvement here...