Questions Regarding DRBD/ZFS Kernel Requirements and TrinityX SLURM Setup

[jw-26-sudo]

Hi Antoine,

Based on the DRBD kmod installation error on Alma/Rocky 9.5 mentioned in the ClusterVision troubleshooting documentation, does this indicate that using the latest kernel version is necessary for DRBD support?

Additionally, would the same recommendation apply to ZFS as well? In other words, is it advisable or required to keep the system on the latest kernel version when using ZFS?

I would also like to check if it is convenient to clarify a few TrinityX with SLURM questions below, or if the relevant update/upgrade details will be shared later when TrinityX version 16 becomes available.

After running the following command from the head node to set up SLURM after the TrinityX installation:

ansible-playbook controller.yml --tags slurm

would we then need to review and edit the following files manually?

• /etc/slurm/slurm-nodes.conf
• /etc/slurm/slurm-partitions.conf

Should I also consider creating or updating an inventory file under the ~/trinityX/site directory?

In addition, when running:

scontrol show nodes

from the head node, it currently returns:

"No nodes in the system"

Does this mean the compute nodes must first be provisioned before they can appear in SLURM?

I also understand that SLURM is currently not supported in the Ubuntu OS compute image creation and deployment workflow, and that the related updates/upgrades are expected in TrinityX version 16.

Thank you for your assistance.

Regards,
Jing Wei

[aphmschonewille]

Hi Jing,

It's not necessary to run the latest kernel for drbd. In the troubleshooting page, the work around was because the used kernel was the latest. Having this said, with all recently discovered kernel bugs, I'd recommend to use the latest kernel. For EL9 latest/streaming we haven't seen any issues with drbd. The same applies to zfs.

For slurm indeed, ubuntu is not included yet. That will come in 16. The slurm config is by default generated by luna, but it'll do this only for EL distros at the moment. Ensure the whole playbook was run before doing just the --tags=slurm. If this was the case, try to call the regen hook by cloning a node in luna and removing it again. Then run scontrol reconf. If all config is correct, you would see nodes with sinfo. No other workflow should be required.

-Antoine

[jw-26-sudo]

Hi Antoine,

Just to clarify regarding SLURM integration:

Am I correct that I do not need to recreate or redeploy the OS image specifically for SLURM, unless the existing image contains missing or incorrect SLURM-related configuration/packages?

In addition, regarding the kernel recommendation, may I confirm that the latest EL9 kernel is currently considered stable and known to work well with both DRBD and ZFS in your experience?

Thank you for your clarification.

Regards,
Jing Wei

[jw-26-sudo]

Hi Antoine,

Are you able to advise the following:

slurm_reconfigure error: Unable to contact slurm controller (connect failure), after running scontrol reconf.

Should I consider:

1. Enabling firewall port numbers required for SLURM?
2. Recreating and redeploying osimage?
3. Reprovisioning compute node(s)?

Thank you.

Regards,
Jing Wei

[aphmschonewille]

Hi Jing,

I assume you see this error on the controller?
What are the top 10 lines in slurm.conf saying? Does this match the config of the controller?
Can the nodes ping the controller?
Was there a controller hostname change?
What does: systemctl status slurmctld say?

Regarding kernel, drbd and zfs. Indeed, current version/latest is fully working for the trio. The troubleshoot section was written some time ago and only applies when you would hit the described error.

-Antoine

[jw-26-sudo]

Hi Antoine,

After running ansible-playbook controller.yml --tags slurm, the first 10 lines of /etc/slurm/slurm.conf are auto-generated as shown:

1 #
2 # Ansible managed
3 #
4
5 ClusterName=cluster
6 SlurmctldHost=<my_controller1_hostname> ##remain unchanged
7 SlurmctldAddr=<my_controller1_internal_ip_addr>
8
9 Include /etc/slurm/slurm-nodes.conf
10 Include /etc/slurm/slurm-partitions.conf

Do I need to check and edit ClusterName based on trix_domain from all.yml file?

I recreated and redeployed new osimage with SLURM, then reprovisioned compute node(s) using that new osimage. I cannot ping the controller's internal IP from my compute node(s). And from my compute node(s), the systemctl status slurmd says:

From my controller, the systemctl status slurmctld says:

000000 12:31:37 [root@odsmaster site]# systemctl status slurmctld
× slurmctld.service - Slurm controller daemon
Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/slurmctld.service.d
└─trinity.conf
Active: failed (Result: exit-code) since Tue 2026-06-02 11:07:53 +08; 1h 37min ago
Duration: 90ms
Main PID: 365264 (code=exited, status=1/FAILURE)
CPU: 72ms

Jun 02 11:07:53 odsmaster systemd[1]: slurmctld.service: Scheduled restart job, restart counter is at 5.
Jun 02 11:07:53 odsmaster systemd[1]: Stopped Slurm controller daemon.
Jun 02 11:07:53 odsmaster systemd[1]: slurmctld.service: Start request repeated too quickly.
Jun 02 11:07:53 odsmaster systemd[1]: slurmctld.service: Failed with result 'exit-code'.
Jun 02 11:07:53 odsmaster systemd[1]: Failed to start Slurm controller daemon.

In addition, when we recreate new osimage, what are the things to take note of, other than it will pull the latest osimage kernel?

Thank you.

Regards,
Jing Wei

[jw-26-sudo]

Hi Antoine,

Just to update, I redid everything and ran scontrol reconf. But the following issues persist:

From controller node:

1. the systemctl status slurmctld says the same error.
2. the auto-generated /etc/slurm/slurm.conf is showing the same first 10 lines.
3. the auto-generated /etc/slurm/slurm-nodes.conf is showing NodeName=DEFAULT Boards=1 SocketsPerBoard=1 CoresPerSocket=1 ThreadsPerCore=1 RealMemory=100 State=UNKNOWN. The rest of the lines are being commented off. Do I need to edit this file?
4. the auto-generated /etc/slurm/slurm-partition is showing PartitionName=defq Nodes=ALL Shared=YES MaxTime=INFINITE State=UP Default=YES AllowAccounts=ALL. The rest of the lines are being commented off. Do I need to edit this file?

From compute node(s):

1. I can now ping the controller's internal IP.
2. The systemctl status slurmd says:

As per earlier question, when we recreate new osimage, what are the things to take note of, other than it will pull the latest osimage kernel?

Appreciate the soonest advice.

Thank you.

Regards,
Jing Wei

[jw-26-sudo]

Hi Antoine,

I hope you are doing well.

I would like to check whether there are any anticipated delays to the release of TrinityX Version 16. I completely understand if additional time is required, particularly for the implementation and validation of the following features:

1. Diskfull and RAID1 support for Ubuntu OS compute images.
2. Deployment of ARM-based images from an x86-based controller.
3. TrinityX cluster deployment with SLURM integration.

Speaking of updating luna for point 1, do you mean editing these 2 files from the controller(s):

1. vi /etc/yum.repos.d/luna2.repo

[luna2]
baseurl = https://updates.clustervision.com/trinityx/15.3/luna2/rocky/9/x86_64
gpgcheck = 0
gpgkey = file://
name = luna2

2. vi /etc/yum.repos.d/luna2-noarch.repo

[luna2-noarch]
baseurl = https://updates.clustervision.com/trinityx/15.3/luna2/noarch/rpms
gpgcheck = 0
gpgkey = file://
name = luna2-noarch

Note that the above file contents for both luna repo files serve as examples only.

In addition to rephrase one of my previous questions, will the latest kernel always be pulled when we create new RHEL/Rocky OS compute images?

I would appreciate any updates you can share regarding the expected release timeline.

Thank you for your time and assistance.

Regards,
Jing Wei

[aphmschonewille]

Hi Jing-Wei

Not sure how you ended up here. It feels like a weird dns or host or resolution problem. You said you redid everything, but was it really from scratch?
What you could try, as i know for sure a working setup is the result out-of-box, overwriting the slurm config: ansible-playbook controller.yml --tags=resolv,slurm -e override=yes
Then clone a dummy node to be thrown away afterwards. Then set the top line in slurm-nodes.conf to match any node, then scontrol reconf. It should render a working slurm config

There shouldn't be a need to edit repositories manually. With update (or did i mention upgrade) is an integral part of trix. See the upgrade section of the docu https://docs.clustervision.com/install/upgrading/

The plan is to release 16 coming thursday. It does contain the diskfull scripts for ubuntu and our slurm builds to support ubuntu as well. We are still working towards the mixed architecture part. That functionality is expected for 16.1.

Hopefully i didn't forget anything, if i did, reach out to us.

-Antoine

[jw-26-sudo]

Hi Antoine,

Thank you for the update. I would appreciate your comments on the following points:

1. Last year, my team and I used TrinityX 15.1 for our first project. When running ansible-playbook controller.yml, we encountered several errors related to SLURM. To proceed, we commented out the SLURM-related configurations in logrotate, controller.yml, and all.yml. After rerunning ansible-playbook controller.yml, the errors were resolved.

At the time, we were unsure whether our conclusion was correct—that either SLURM was not installed by default or that there were compatibility issues with SLURM in TrinityX 15.1. We applied similar changes to the compute-*.yml files whenever we needed to create or recreate OS compute images without SLURM-related errors.

2. For future releases, if we again choose to disable SLURM-related configurations in logrotate, controller.yml, and all.yml and successfully run ansible-playbook controller.yml, would it be possible to enable SLURM later by making the necessary configuration changes and then running:

ansible-playbook controller.yml --tags=resolv,slurm -e override=yes

Likewise, after making the required amendments, would it be appropriate to recreate OS compute images and run:

ansible-playbook compute-*.yml --tags=slurm -e override=yes

3. When creating or recreating RHEL/Rocky OS compute images, will the latest available kernel always be included by default?

4. Could you advise on the level of effort required to update Luna for Diskfull and RAID1 support for Ubuntu OS compute images? Alternatively, are these enhancements expected to be incorporated as part of the TrinityX 16.1 release?

5. Are there any planned updates in TrinityX 16.1 regarding the deployment of ARM-based images from an x86-based controller?

6. In /etc/slurm/slurm.conf, should the ClusterName parameter match the trix_domain value defined in all.yml?

7. Do I need to review, modify, save, and distribute the following files from the controller node(s) to all compute nodes?

• slurm-nodes.conf
• slurm.conf
• slurm-partition.conf

8. In the future release, may I ask whether TrinityX supports deployment on Ubuntu-based head/controller nodes? If so, are there any known limitations or special considerations?

I completely understand if additional time is required to address these questions. I am looking forward to the release of TrinityX 16.1 and appreciate any guidance you can provide.

Thank you.

Regards,
Jing Wei

[aphmschonewille]

Quite a few questions...

Release 16 is scheduled this thursday and will include improved diskfull and raid scripts to include ubuntu. A mixed architecture x86/arm is likely for 16.1

Slurm should have worked out of the box since release 15. Why it didn't work for you is not clear to me. You could have also 'commented out' the scheduler, slurm in all.yml. same effect but slightly cleaner.

Yes, using future releases would allow enabling slurm (again) for the controller and images.

Once a kernel is installed on the controller or in images, these will not automatically updated. They stay as is and up to you to upgrade if required. For new created images, the os distri release based kernel is used, which can be upgraded and selected as kernel for luna osimage(s), then packed and booted.

In general, i believe is indeed the clustername equal to trix_domain. However i don't really think of this as the playbook renders, normally, a working config.

The config is automatically distributed to the nodes as the directory /etc/slurm points to a location on trix_shared, serving all nodes.

After the release it's very easy to include just support for ubuntu disk scripts: ansible-playbook controller.yml --tags=luna, but please wait till after the 16 release. After this step, ensure script is set for node/group following the online documentation and it should work out of the box.

It's not clear at all whether there will be playbooks for controllers supporting ubuntu. Since ubuntu is supported as compute image, the demand is very little.

-Antoine

[jw-26-sudo]

Hi Antoine,

I refer to the pointers of the upcoming TrinityX version 16 release.

Regarding the "Deprecation of OpenHPC stack", wasn't this required for OS compute images creation, followed by compute nodes deployment?

Appreciate your advice please.

Thank you.

Regards,
Jing Wei

[aphmschonewille]

Hi Jing Wei

We utilized openhpc for slurm packages and libraries. Since we noticed that most administrators prefer easybuild over openhpc and openhpc not providing slurm packages for e.g. ubuntu, it was a logical step to introduce trinityx-slurm and deprecate openhpc. We already used our in-house developed provisioning engine luna and so no dependencies are left.

-Antoine

[jw-26-sudo]

Hi Antoine,

Appreciate the advice for the following settings if we want to set up TrinityX version 16.X cluster (be it HA or non-HA) with SLURM in future:

1. Enable required ports for full SLURM functionality.
2. Enable trinityx_slurm_repositories.
3. enable_ood: true
4. enable_openhpc: false (Under what circumstances do we need OpenHPC?)
5. enable_trinityx_slurm: true
6. enable_slurm_pam: true
7. enable_easybuild: true
8. enable_authentication: true

In addition, can I confirm if there are outstanding SLURM issues to be resolved, such as slurmd and slurmdbd? How about slurmctld? Are these expected to be released in TrinityX 16.1?

Thank you.

Regards,
Jing Wei