Gui Installer issues and Phusion Passenger

[julianvreug]

Hi,

I will be rolling TrinityX out on the Cluster I look after soon, previous install worked perfectly. Has anyone had issues installing latest build in a VM?
TrinityX 16 I found GUI Installer does not start installtion correctly and not a smooth install, can be different errors each time, I cant also export yaml. Directly after it installed GIT I got access denied errors. It seems prepare.sh was not excutable.

Git is already installed.
/usr/bin/git
Git is available at: /usr/bin/git.
TrinityX Directory created at: /trinity/install/trinityx_20260722_103655

TrinityX cloned at: /trinity/install/trinityx_20260722_103655
Checkout to Latest Version:
Created Host File /trinity/install/trinityx_20260722_103655/site/hostsAdded Host control in ansible host file.
Last login: Wed Jul 22 10:23:26 2026 from 192.168.1.189
cd /trinity/install/trinityx_20260722_103655 && WITH_ZFS=no USE_CURRENT_KERNEL=yes GITLAB_CI=yes ./prepare.sh
[root@control ~]# cd /trinity/install/trinityx_20260722_103655 && WITH_ZFS=no USE_CURRENT_KERNEL=yes GITLAB_CI=yes ./prepare.sh
-bash: ./prepare.sh: Permission denied
[root@control trinityx_20260722_103655]#

Manually configuring and deploying works but found I came accross many dependencies I had to manually install.
Had to run
cd /trinity/install/trinityx_20260722_103655

chmod +x *.sh

WITH_ZFS=no USE_CURRENT_KERNEL=yes GITLAB_CI=yes ./prepare.sh

cd /trinity/install/trinityx_20260722_103655
./INSTALL.sh

Failed:
TASK [trinity/slurm : Start slurmdbd services] *************************************************************************************************************
failed: [controller1] (item=slurmdbd) => {"ansible_loop_var": "item", "changed": false, "item": "slurmdbd", "msg": "Unable to start service slurmdbd: A dependency job for slurmdbd.service failed. See 'journalctl -xe' for details.\n"}

PLAY RECAP *************************************************************************************************************************************************
controller1 : ok=409 changed=205 unreachable=0 failed=1 skipped=365 rescued=0 ignored=6

Ran:
systemctl status munge --no-pager
systemctl status mariadb --no-pager
journalctl -u slurmdbd --no-pager | tail -n 20

journalctl -u slurmdbd --no-pager | tail -n 20
● munge.service - MUNGE authentication service
Loaded: loaded (/usr/lib/systemd/system/munge.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/munge.service.d
└─trinity.conf
Active: activating (auto-restart) (Result: exit-code) since Wed 2026-07-22 11:01:38 AEST; 991ms ago
Invocation: 89aff65b383a45b4af26aa62b16868d3
Docs: man:munged(8)
Process: 83823 ExecStart=/usr/sbin/munged --key-file /trinity/shared/etc/munge/munge.key (code=exited, status=1/FAILURE)
Mem peak: 4M
CPU: 13ms
● mariadb.service - MariaDB 10.11 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
Active: active (running) since Wed 2026-07-22 10:57:10 AEST; 4min 28s ago
Invocation: 498c435a388f4956baefb2f0ac0bf595
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Main PID: 80686 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 10 (limit: 203198)
Memory: 1.1G (peak: 1.1G)
CPU: 537ms
CGroup: /system.slice/mariadb.service
└─80686 /usr/libexec/mariadbd --basedir=/usr

Jul 22 10:57:10 control systemd[1]: Starting mariadb.service - MariaDB 10.11 database server...
Jul 22 10:57:10 control mariadb-prepare-db-dir[80645]: Database MariaDB is probably initialized in /trinity/local/var/lib/mysql already, nothing is done.
Jul 22 10:57:10 control mariadb-prepare-db-dir[80645]: If this is not the case, make sure the /trinity/local/var/lib/mysql is empty before runnin…re-db-dir.
Jul 22 10:57:10 control (mariadbd)[80686]: mariadb.service: Referenced but unset environment variable evaluates to an empty string: MYSQLD_OPTS,…NEW_CLUSTER
Jul 22 10:57:10 control systemd[1]: Started mariadb.service - MariaDB 10.11 database server.
Hint: Some lines were ellipsized, use -l to show in full.
Jul 22 10:59:43 control systemd[1]: Dependency failed for slurmdbd.service - Slurm DBD accounting daemon.
Jul 22 10:59:43 control systemd[1]: slurmdbd.service: Job slurmdbd.service/start failed with result 'dependency'.

Ran
sudo chmod 755 /trinity
sudo systemctl restart munge
sudo systemctl status munge --no-pager

Now cant get past

RUNNING HANDLER [trinity/ood : Run update_ood] *************************************************************************************************************
fatal: [controller1]: FAILED! => {"changed": true, "cmd": ["/opt/ood/ood-portal-generator/sbin/update_ood_portal", "-f", "&&", "/opt/ood/nginx_stage/sbin/nginx_stage", "nginx_clean", "-f"], "delta": "0:00:00.141693", "end": "2026-07-22 12:02:12.790405", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2026-07-22 12:02:12.648712", "stderr": "StandardError:oidc_crypto_passphrase must be set when using OIDC.\n/opt/ood/ood-portal-generator/lib/ood_portal_generator/view.rb:132:in fetch_oidc_crypto_passphrase'\n/opt/ood/ood-portal-generator/lib/ood_portal_generator/view.rb:222:in update_oidc_attributes'\n/opt/ood/ood-portal-generator/lib/ood_portal_generator/dex.rb:50:in initialize'\n/opt/ood/ood-portal-generator/lib/ood_portal_generator/application.rb:182:in new'\n/opt/ood/ood-portal-generator/lib/ood_portal_generator/application.rb:182:in generate'\n/opt/ood/ood-portal-generator/lib/ood_portal_generator/application.rb:201:in update_ood_portal'\n/opt/ood/ood-portal-generator/lib/ood_portal_generator/application.rb:305:in start'\n-e:1:in

'\nRun 'update_ood_portal --help' to see a full list of available options.", "stderr_lines": ["StandardError:oidc_crypto_passphrase must be set when using OIDC.", "/opt/ood/ood-portal-generator/lib/ood_portal_generator/view.rb:132:in fetch_oidc_crypto_passphrase'", "/opt/ood/ood-portal-generator/lib/ood_portal_generator/view.rb:222:in update_oidc_attributes'", "/opt/ood/ood-portal-generator/lib/ood_portal_generator/dex.rb:50:in initialize'", "/opt/ood/ood-portal-generator/lib/ood_portal_generator/application.rb:182:in new'", "/opt/ood/ood-portal-generator/lib/ood_portal_generator/application.rb:182:in generate'", "/opt/ood/ood-portal-generator/lib/ood_portal_generator/application.rb:201:in update_ood_portal'", "/opt/ood/ood-portal-generator/lib/ood_portal_generator/application.rb:305:in start'", "-e:1:in '", "Run 'update_ood_portal --help' to see a full list of available options."], "stdout": "", "stdout_lines": []}

PLAY RECAP *************************************************************************************************************************************************
controller1 : ok=557 changed=72 unreachable=0 failed=1 skipped=509 rescued=0 ignored=4

I did get Luna2 and OOD working partially but clicking anything like Cluster or Shell I get

[Technical details for the administrator of this website]
Error ID:
a356b804
Details:
Web application could not be started by the Phusion Passenger(R) application server.

Please read the Passenger log file (search for the Error ID) to find the details of the error.

I have done a lot of debugging over the last 2 days. Just wondering if anyone else was able to do a succesfull install within a Virtual Machine.
Sorry about this being AI generated but I put most of the notes I can and got it to generate the output so its easier to read.

Environment: Rocky 10.2 Linux (VM deployment), TrinityX OOD 3.0.x
Summary: The current installation pipeline for Open OnDemand on Rocky Linux requires significant manual intervention. The installer fails at multiple stages due to missing system dependencies, malformed YAML generation, missing configuration files, and unhandled Python exceptions in the application code.

Phase 1: Pre-Installation & Installer Failures
The initial installation script fails to validate or bootstrap its own environment, resulting in early crashes.

Missing Dependencies: The installer assumes certain packages and dependencies are already present on a base EL system. It fails abruptly rather than using dnf to install missing prerequisite packages (e.g., required development tools, Python/Node/Ruby dependencies). These had to be identified from the error output and installed manually before the script would progress.

Malformed YAML Generation: The installer generates broken or incomplete YAML configuration files (such as ood_portal.yml or cluster configs). This requires manually stopping the deployment, debugging the YAML syntax/missing keys, and rewriting the files before the installer can successfully parse them and continue.

Phase 2: Missing API Configuration
Once the installer finally completes, the application crashes immediately upon the first web request because it fails to generate the required [API] block.

File: /trinity/local/ondemand/3.0/config/luna.ini

Error: Exception: You need to fix following errors: API section is not found in /trinity/local/ondemand/3.0/config/luna.ini.

The Fix: The installer script needs to be updated to automatically append the [API] header and the required local URLs during the setup phase.

Phase 3: Python TypeError Crashing the Web App
After manually adding the [API] section, the application throws a Passenger internal error due to a hardcoded type error in the INI parser. When reading VERIFY_CERTIFICATE, it processes it as a Python boolean but attempts to call .lower() on it.

File: /trinity/local/ondemand/trinityx-ood-*/slurm/base/ini.py (line 55)

Error: AttributeError: 'bool' object has no attribute 'lower'

The Fix: Cast the variable to a string before calling .lower():

Python
config["VERIFY_CERTIFICATE"] = True if str(security).lower() in ['y', 'yes', 'true'] else False
Phase 4: Missing SELinux Policies (Passenger / V8 Crashes)
The installer does not configure the necessary SELinux exceptions for modern EL systems. This results in silent background crashes when the web server attempts to spawn user sessions or compile JavaScript.

Symptom 1: Passenger throws Check failed: 12 == (*__errno_location ()). from v8::internal::MemoryAllocator because SELinux denies httpd_t from allocating executable memory.

Symptom 2: Spawning PUNs (Per-User Nginx) fails with PAM account management error: Permission denied because Apache cannot access the audit socket.

The Fix: The installation script must explicitly install the OOD SELinux package and set the required memory and PAM booleans:

Bash
dnf install -y ondemand-selinux
setsebool -P httpd_execmem 1
setsebool -P httpd_mod_auth_pam 1
setsebool -P ondemand_manage_user_home_dir 1

Actual error:

[localadmin@control ~]$ latest_error=$(ls -t /tmp/passenger-error-.html 2>/dev/null | head -n 1)
awk '/

/,/</pre>/' "$latest_error"

"problem_description_html" : "
The Phusion Passenger(R) application server tried to start the web application through a Passenger-internal helper tool called the "wrapper". But Passenger was unable to execute that helper tool because it encountered an internal error.
The stdout/stderr output of the subprocess so far is:
\n\n#\n# Fatal error in , line 0\n# Check failed: 12 == (__errno_location ()).\n#\n#\n#\n#FailureMessage Object: 0x7ffdc813f7d0\n----- Native stack trace -----\n\n 1: 0x15349b24c589  [/lib64/libnode.so.127]\n 2: 0x15349b8bdb6f V8_Fatal(char const*, ...) [/lib64/libnode.so.127]\n 3: 0x15349b8c53a5  [/lib64/libnode.so.127]\n 4: 0x15349d42947e v8::internal::MemoryAllocator::SetPermissionsOnExecutableMemoryChunk(v8::internal::VirtualMemory*, unsigned long, unsigned long, unsigned long) [/lib64/libnode.so.127]\n 5: 0x15349d429808 v8::internal::MemoryAllocator::AllocateAlignedMemory(unsigned long, unsigned long, unsigned long, v8::internal::AllocationSpace, v8::internal::Executability, void*, v8::internal::VirtualMemory*) [/lib64/libnode.so.127]\n 6: 0x15349d4298e6 v8::internal::MemoryAllocator::AllocateUninitializedChunkAt(v8::internal::BaseSpace*, unsigned long, v8::internal::Executability, unsigned long, v8::internal::PageSize) [/lib64/libnode.so.127]\n 7: 0x15349d429d69 v8::internal::MemoryAllocator::AllocatePage(v8::internal::MemoryAllocator::AllocationMode, v8::internal::Space*, v8::internal::Executability) [/lib64/libnode.so.127]\n 8: 0x15349d447977 v8::internal::PagedSpaceBase::TryExpand(v8::internal::LocalHeap*, v8::internal::AllocationOrigin) [/lib64/libnode.so.127]\n 9: 0x15349d3f28e3 v8::internal::PagedSpaceAllocatorPolicy::TryExpandAndAllocate(unsigned long, v8::internal::AllocationOrigin) [/lib64/libnode.so.127]\n10: 0x15349d3f2b4e v8::internal::PagedSpaceAllocatorPolicy::RefillLab(int, v8::internal::AllocationOrigin) [/lib64/libnode.so.127]\n11: 0x15349d3f11cd v8::internal::MainAllocator::EnsureAllocation(int, v8::internal::AllocationAlignment, v8::internal::AllocationOrigin) [/lib64/libnode.so.127]\n12: 0x15349d3f1442 v8::internal::MainAllocator::AllocateRawSlowUnaligned(int, v8::internal::AllocationOrigin) [/lib64/libnode.so.127]\n13: 0x15349d394b45 v8::internal::Factory::CodeBuilder::AllocateUninitializedInstructionStream(bool) [/lib64/libnode.so.127]\n14: 0x15349d3ab0ef v8::internal::Factory::CodeBuilder::BuildInternal(bool) [/lib64/libnode.so.127]\n15: 0x15349da2b643 v8::internal::baseline::BaselineCompiler::Build(v8::internal::LocalIsolate*) [/lib64/libnode.so.127]\n16: 0x15349d1d8846 v8::internal::GenerateBaselineCode(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>) [/lib64/libnode.so.127]\n17: 0x15349d243b2b v8::internal::Compiler::CompileSharedWithBaseline(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [/lib64/libnode.so.127]\n18: 0x15349d2478a7 v8::internal::Compiler::CompileBaseline(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [/lib64/libnode.so.127]\n19: 0x15349da29952 v8::internal::baseline::BaselineBatchCompiler::CompileBatch(v8::internal::Handle<v8::internal::JSFunction>) [/lib64/libnode.so.127]\n20: 0x15349d3372cb v8::internal::TieringManager::OnInterruptTick(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::CodeKind) [/lib64/libnode.so.127]\n21: 0x15349d8aa0bd  [/lib64/libnode.so.127]\n22: 0x15349d8ac055 v8::internal::Runtime_BytecodeBudgetInterruptWithStackCheck_Ignition(int, unsigned long*, v8::internal::Isolate*) [/lib64/libnode.so.127]\n23: 0x15343cfecdf6 \n
",

[localadmin@control ~]$ sudo setsebool -P httpd_execmem 1

[localadmin@control ~]$ sudo getsebool httpd_execmem

httpd_execmem --> on

Building OS Image
[localadmin@control ~]$ cd /trinity/install/trinityx_20260721_110826/site && ansible-playbook compute-ubuntu.yml

[WARNING]: log file at /var/log/trinity.log is not writeable and we cannot create it, aborting
[WARNING]: Invalid characters were found in group names but not replaced, use

-vvvv to see details

ERROR! couldn't resolve module/action 'mysql_db'. This often indicates a misspelling, missing collection, or incorrect module path.
The error appears to be in '/trinity/install/trinityx_20260721_110826/site/roles/trinity/slurm/tasks/main.yml': line 476, column 7, but may

be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: Create DB for accounting 
  ^ here 

[localadmin@control site]$

[aphmschonewille]

Thanks for this — really useful report, and your diagnosis was right pretty much everywhere. I've
been through all of it against the code.

First thing: you installed tag 16, not 16u1. The installer grabs whatever tag is newest when
it clones, and 16u1 went out on 22 July — your install directory is stamped before that, so you missed it by about nine hours.
That matters because the OIDC error is fixed in 16u1. In 16 the Dex block of
ood_portal.yml.j2 has no oidc_crypto_passphrase line at all, so the generated portal config
declares Dex and never sets a passphrase — it would fail every time, no matter how often you re-ran
it. Re-run the installer or check out 16u1 and that one's gone. You can check what you're on with
git -C /trinity/install/trinityx_* describe --tags.
We're also going to make the installer print and record the tag it checked out if that's technically possible, because right now
nothing tells you which one you got.
The rest, briefly:

• prepare.sh not executable — the repo is fine (it's 755 in git). Our GUI installer rewrites
  the file to strip the ZFS block, which drops it to 644, and the chmod +x afterwards is fired
  off without waiting. We're removing the rewrite.
• Broken YAML — confirmed, and it's the worst one on your list. The installer sends all.yml
  through an unquoted heredoc, so the remote shell expands it: a password like P@ss$word1 arrives
  as P@ss, and it stays valid YAML so nothing complains. That's very likely your "different errors
  each time". Moving it to SFTP.
• munge / slurmdbd — your chmod 755 /trinity was exactly right. We pin the mode on the munge
  directories but not on /trinity itself, so it keeps whatever it was first created with. Fixing
  that, plus we investigate if we can do a check whether munge can read its key before we try to start slurmdbd.
• SELinux — worth knowing this is deliberate for now: TrinityX only supports permissive SELinux,
  and nothing installs ondemand-selinux or sets any booleans. Your machine was enforcing, hence
  the V8 and PUN failures. Your setsebool commands are the
  right workaround. Supporting enforcing
  properly is a separate piece of work we want to do.
• luna.ini missing [API] — confirmed. The config is only written when the app payload is
  downloaded, so after any partial run it gets skipped for good. Something to look into.
• VERIFY_CERTIFICATE — one correction: it's not a bool
  from the file, it's the missing-option
  sentinel, so the crash means the option wasn't there (knock-on from the previous point). And the
  same line is in three apps — slurm, users and infiniband —
  so if you're patching by hand, patch
  all three.
• mysql_db — run the image build as root. ansible-galaxy puts the collection in root's home,
  where your localadmin session can't see it. We'll install it system-wide instead.

All in all quite a few things, some recent, some long time issues being discovered now.
These issues will be raised and worked on.

-Antoine