Skip to content

Change installation path to the system managed path#61

Open
seantronsen wants to merge 7 commits into
OpenCHAMI:mainfrom
seantronsen:main
Open

Change installation path to the system managed path#61
seantronsen wants to merge 7 commits into
OpenCHAMI:mainfrom
seantronsen:main

Conversation

@seantronsen

@seantronsen seantronsen commented Jun 23, 2026

Copy link
Copy Markdown

Pull Request Template

Thank you for your contribution! Please ensure the following before submitting:

Checklist

  • My code follows the style guidelines of this project
  • I have added/updated comments where needed
  • I have added tests that prove my fix is effective or my feature works
  • I have run make test (or equivalent) locally and all tests pass
  • DCO Sign-off: All commits are signed off (git commit -s) with my real name and email
  • REUSE Compliance:
    • Each new/modified source file has SPDX copyright and license headers
    • Any non-commentable files include a <filename>.license sidecar
    • All referenced licenses are present in the LICENSES/ directory

Description

Moves plain Systemd units and Podman quadlets to the correct system-managed paths:

  • /etc/containers/systemd -> /usr/share/containers/systemd
  • /etc/systemd/system -> /usr/lib/systemd/system

Effect

This frees up the administrator-managed paths for drop-ins and local overrides. On the quadlet side, this makes simple changes much easier (e.g., testing a different image tag). More generally, the full set of Systemd override mechanisms is available.

Functionally, this PR is little more than updating the RPM to install files into the proper locations for system-managed artifacts.

Caveats

If this is accepted, we'll need a quick pass over the openchami.org tutorials to update any directory listings that still reference the old paths.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

For more info, see Contributing Guidelines.

@seantronsen

Copy link
Copy Markdown
Author

It's also possible to update scripts/{openchami-certificate-update,bootstrap_openchami.sh} to instead leverage the local admin path (e.g., /etc/containers/systemd).

For now, I kept it simple and just hit those files with the same sed subst.

If it's desirable, I can fix up those scripts too.

Signed-off-by: Sean Tronsen <sean.tronsen@gmail.com>
@seantronsen

seantronsen commented Jun 24, 2026

Copy link
Copy Markdown
Author

For reference/convenience, here is an example script for making overrides. I'm currently using this to replicate bug behaviors w.r.t. specific tagged versions of our microservices:

[rocky@head ~]$ cat override-quadlet-images.bash
#!/usr/bin/env bash

function log-error() {
        if ! [ ${#} -eq 1 ]; then
                log-error "log-error must receive only one argument, but received $#"
        fi
        PREFIX="error: "
        MESSAGE="${PREFIX} ${1}"
        echo "${MESSAGE}"
        logger --id=$$ "${MESSAGE}"
        exit 1
}

if ! [ "${UID}" -eq 0 ]; then
        log-error "superuser privileges are required to execute this script!"
fi

echo "creating container image overrides for OpenCHAMI quadlet files:"

DEST="/etc/containers/systemd"
FNAME_OVERRIDE="10-override.conf"
while IFS= read -r filename; do
        TARGET="${DEST}/$(basename ${filename}).d/${FNAME_OVERRIDE}"
        mkdir -v -p "$(dirname ${TARGET})"
        cp -v "${filename}" "${TARGET}"
        sed -i '/\[Container\]\|Image/!d' "${TARGET}"
done < <(rpm -ql openchami | grep -i '\.container')

echo "finished."

Warning

It's imperative that you use the <service-name.container>.d/<priority>-<semantic-name>.conf style for Systemd overrides. If you instead naively create /etc/containers/systemd/<service-name>.container, this ends up wiping out the previous config specified in the system managed path.

If you do follow with this advice, then the config is appended to the corresponding sections upon the next daemon-reload, thus resulting in the desired override behavior.

Example

[rocky@head ~]$ cat /etc/containers/systemd/coresmd-coredns.container.d/10-override.conf
[Container]
Image=ghcr.io/openchami/coresmd:v0.6.3
[rocky@head ~]$ sudo systemctl daemon-reload
[rocky@head ~]$ sudo systemctl cat coresmd-coredns.service
# /run/systemd/generator/coresmd-coredns.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
#
[Unit]
Wants=network-online.target
After=network-online.target
Description=The CoreSMD CoreDNS container
Wants=haproxy.service
After=haproxy.service
PartOf=openchami.target
SourcePath=/usr/share/containers/systemd/coresmd-coredns.container
RequiresMountsFor=%t/containers
RequiresMountsFor=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
RequiresMountsFor=/etc/openchami/configs/Corefile

[X-Container]
ContainerName=coresmd-coredns

HostName=coresmd-coredns
Image=ghcr.io/openchami/coresmd:v0.4.3

Exec=/coredns

# Capabilities
AddCapability=NET_ADMIN
AddCapability=NET_RAW

# Volumes
Volume=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/root_ca/root_ca.crt:ro,Z
Volume=/etc/openchami/configs/Corefile:/Corefile:ro,Z

# Networks for the Container to use
Network=host

# Unsupported by generator options
# Proxy settings
PodmanArgs=--http-proxy=false
Image=ghcr.io/openchami/coresmd:v0.6.3

[Service]
Restart=always
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -v -f -i coresmd-coredns
ExecStopPost=-/usr/bin/podman rm -v -f -i coresmd-coredns
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name coresmd-coredns --replace --rm --cgroups=split --hostname coresmd-coredns --network host --sdnotify=conmon -d --cap-add net_admin --cap-add net_raw -v /etc/pki/ca-trust/extracted/pem/tls-ca-bund>

Notice how the new Image= configuration is now specified at the bottom of the [X-Container] section.

@synackd

synackd commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

It's also possible to update scripts/{openchami-certificate-update,bootstrap_openchami.sh} to instead leverage the local admin path (e.g., /etc/containers/systemd).

For now, I kept it simple and just hit those files with the same sed subst.

If it's desirable, I can fix up those scripts too.

This presents a new challenge in that we need to figure out how to apply the changes that the scripts apply in an idiomatic way that doesn't interfere with the user's expectations. Taking into account the following:

if you instead naively create /etc/containers/systemd/<service-name>.container, this ends up wiping out the previous config specified in the system managed path.

I can think of a couple of solutions that we should decide on, or come up with a different solution:

OPTION 1: sed the system service into /etc/containers/systemd/

This is easy, because each change is applied with sed 's/source_stuff/dest_stuff/g' /usr/share/containers/systemd/<svc>.container > /etc/containers/systemd/<svc>.container.

However, this completely overrides the system container settings as mentioned in the quote above, which may lead to unexpected behavior when making overrides. On the flipside, this would create files the user can see have been overridden, and can make the changes in them instead of creating overrides in the /etc/containers/systemd/<svc>.container.d/ directory.

OPTION 2: Put changes into override file with known ordinal, e.g. 10-script-override.conf

This is also simple because the changes are written directly to the override file without the need for sed.

However, 10-script-override.conf (or whatever the name becomes) becomes reserved and must be documented.

@synackd

synackd commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

This change seems more idiomatic of Systemd anyway, so I support it. If I were to choose, I think I would probably lean to OPTION 2 since it is more DRY.

@seantronsen

seantronsen commented Jun 30, 2026

Copy link
Copy Markdown
Author

This change seems more idiomatic of Systemd anyway, so I support it. If I were to choose, I think I would probably lean to OPTION 2 since it is more DRY.

I do like option 2, but a few questions first:

  1. Can we get rid of scripts/bootstrap_openchami.sh? See addendum for evidence on how it isn't used
  2. In general, do we actually need a script that modifies systemd unit files post install? Based on the contents of openchami-certificate-update, do we even need one that creates override files? See addendum with further explanation below.

Addendum

Evidence for lack of use for scripts/bootstrap_openchami.sh

sean@matterhorn:release$ grep -Rni bootstrap_openchami.sh
openchami.spec:45:cp scripts/bootstrap_openchami.sh       %{buildroot}/usr/libexec/openchami/
openchami.spec:51:chmod +x %{buildroot}/usr/libexec/openchami/bootstrap_openchami.sh
openchami.spec:53:chmod +x %{buildroot}/usr/libexec/openchami/bootstrap_openchami.sh
openchami.spec:68:/usr/libexec/openchami/bootstrap_openchami.sh
openchami.spec:85:/usr/libexec/openchami/bootstrap_openchami.sh

Note

scripts/bootstrap_openchami.sh is installed, yet never used.

Systemd has built-in features which make openchami-certificate-update unnecessary

Inside of the openchami-certificate-update, there is one primary function which is show below.

update_dns() {
    local system_fqdn=$1
    local short_name="${system_fqdn%%.*}" 
    local dns_name="${system_fqdn#*.}"
    local primary_ip=$(hostname -I | awk '{print $1}')
    
    # Update names in environment and acme containers
    sed -i "s/^SYSTEM_NAME=.*/SYSTEM_NAME=${short_name}/" /etc/openchami/configs/openchami.env
    sed -i "s/^SYSTEM_DOMAIN=.*/SYSTEM_DOMAIN=${dns_name}/" /etc/openchami/configs/openchami.env
    sed -i "s/^SYSTEM_URL=.*/SYSTEM_URL=${system_fqdn}/" /etc/openchami/configs/openchami.env
    sed -i "s|^URLS_SELF_ISSUER=.*|URLS_SELF_ISSUER=https://${system_fqdn}|" /etc/openchami/configs/openchami.env
    sed -i "s|^URLS_SELF_PUBLIC=.*|URLS_SELF_PUBLIC=https://${system_fqdn}|" /etc/openchami/configs/openchami.env
    sed -i "s|^URLS_LOGIN=.*|URLS_LOGIN=https://${system_fqdn}/login|" /etc/openchami/configs/openchami.env
    sed -i "s|^URLS_CONSENT=.*|URLS_CONSENT=https://${system_fqdn}/consent|" /etc/openchami/configs/openchami.env
    sed -i "s|^URLS_LOGOUT=.*|URLS_LOGOUT=https://${system_fqdn}/logout|" /etc/openchami/configs/openchami.env
    sed -i "s|-d .* \\\\|-d ${system_fqdn} \\\\|" /usr/share/containers/systemd/acme-deploy.container
    sed -i "s/^ContainerName=.*/ContainerName=${system_fqdn}/" /usr/share/containers/systemd/acme-register.container
    sed -i "s/^HostName=.*/HostName=${system_fqdn}/" /usr/share/containers/systemd/acme-register.container
    sed -i "s|-d .* \\\\|-d ${system_fqdn} \\\\|" /usr/share/containers/systemd/acme-register.container
    sed -i "s|--add-host='.*|--add-host='${system_fqdn}:${primary_ip}'|" /usr/share/containers/systemd/opaal.container

    # Reload systemD after .container changes
    systemctl daemon-reload

    echo "Changed FQDN to ${1}"
    echo 'Either restart all of the OpenCHAMI services:'
    echo
    echo '  sudo systemctl restart openchami.target'
    echo
    echo 'or run the following to just regenerate/redeploy the certificates:'
    echo
    echo '  sudo systemctl restart acme-deploy'
    echo
}

This function has one responsibility, which is to update fqdn/hostname/dns information from the defaults written in the quadlet files from the release RPM. Even if we don't want to outright discard the config script, most of these configuration values can be automatically set via systemd unit specifiers. See below for a dummy example:

[rocky@demo systemd]$ hostnamectl
 Static hostname: demo.openchami.cluster
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: 2ffdb717e29c4ad6aaed8d7e6f92dbd7
         Boot ID: 182038bbc3af4927a2f3a8729d26efb4
  Virtualization: kvm
Operating System: Rocky Linux 9.8 (Blue Onyx)
     CPE OS Name: cpe:/o:rocky:rocky:9::baseos
          Kernel: Linux 5.14.0-687.15.1.el9_8.x86_64
    Architecture: x86-64
 Hardware Vendor: Red Hat
  Hardware Model: KVM
Firmware Version: edk2-20241117-8.el9
[rocky@demo systemd]$ cat spec-test.container
[Unit]
Description=Quadlet all specifier test

[Container]
Image=docker.io/library/alpine:latest
ContainerName=specifier-test

Exec=sh -c 'echo "a=%a"; echo "H=%H"; echo "i=%i"; echo "I=%I"; echo "i=%i"; echo "l=%l"; echo "i=%i"'
[rocky@demo systemd]$ sudo journalctl -f -u spec-test.service
[sudo] password for rocky:
Jun 30 12:24:16 demo.openchami.cluster specifier-test[3416]: H=demo.openchami.cluster
Jun 30 12:24:16 demo.openchami.cluster specifier-test[3416]: i=
Jun 30 12:24:16 demo.openchami.cluster specifier-test[3416]: I=
Jun 30 12:24:16 demo.openchami.cluster specifier-test[3416]: i=
Jun 30 12:24:16 demo.openchami.cluster specifier-test[3416]: l=demo
Jun 30 12:24:16 demo.openchami.cluster specifier-test[3416]: i=
Jun 30 12:24:16 demo.openchami.cluster spec-test[3380]: 81b3d7e5e771b1ed19e7798a5768d7f50d68d9ad23a87963658ad778c4c813e6
Jun 30 12:24:16 demo.openchami.cluster podman[3420]: 2026-06-30 12:24:16.83957026 -0400 EDT m=+0.026994547 container died 81b3d7e5e771b1ed19e7798a5768d7f50d68d9ad23a87963658ad778c4c813e6 (image=docker.io/library/alpine:latest, name=specifier-test, PODMAN_SYSTEMD_UNIT=spec-test.service)
Jun 30 12:24:16 demo.openchami.cluster podman[3420]: 2026-06-30 12:24:16.936829175 -0400 EDT m=+0.124253122 container remove 81b3d7e5e771b1ed19e7798a5768d7f50d68d9ad23a87963658ad778c4c813e6 (image=docker.io/library/alpine:latest, name=specifier-test, PODMAN_SYSTEMD_UNIT=spec-test.service)
Jun 30 12:24:17 demo.openchami.cluster systemd[1]: spec-test.service: Deactivated successfully.

Warning

If we opt for the systemd automation proposed above:

  • cluster DNS is hard locked to the systemd fqdn assuming a proper one is set, so we lose customization if that's truly needed.
  • it will also require changing some of the upstream services, or at least their corresponding unit files to define those env vars themselves. HOWEVER, this can largely be achieved automatically with /usr/share/containers/systemd/container.d/<ordinal>-openchami-env-vars.conf, which would then set those env vars for all of the quadlet services. This also lends well to the override scheme this PR is going for as these values can be overriden in /etc/containers/systemd/containers.d/<ordinal>-<some-name>.conf.

@seantrons

Copy link
Copy Markdown

See above message, a potential option four would be to heavily slim that script down such that it only creates the global override.conf file mentioned at the bottom (and placing it in the admin path). This fixes the two potential issues with the idea:

  • needing customization for a particular installation target (e.g, using the hostname from the primary namespace isn't desirable)
  • needing the primary IP + dns name, which isn't covered by systemd specifiers.

@seantronsen

Copy link
Copy Markdown
Author

See above message, a potential option four would be to heavily slim that script down such that it only creates the global override.conf file mentioned at the bottom (and placing it in the admin path). This fixes the two potential issues with the idea:

  • needing customization for a particular installation target (e.g, using the hostname from the primary namespace isn't desirable)
  • needing the primary IP + dns name, which isn't covered by systemd specifiers.

We could also keep this automated by having a systemd oneshot unit run a script to populate the values in the override.conf automatically. Then, the moment someone updates the system hostname with hostnamectl hostname <name>, a quick restart oenchami.target handles everything.

This seems like the path of least friction, so I'll start there even though the systemd specifiers seems cooler/lazier in the less code is better code sense.

@synackd

synackd commented Jun 30, 2026

Copy link
Copy Markdown
Contributor
  1. Can we get rid of scripts/bootstrap_openchami.sh? See addendum for evidence on how it isn't used

It's used here as a postinstall script to do things like set up the Podman secrets. The logic there was a bit complex to include entirely within the spec file, which is why it's a separate script.

2. In general, do we actually need a script that modifies systemd unit files post install? Based on the contents of openchami-certificate-update, do we even need one that creates override files? See addendum with further explanation below.

I think the %H systemd hostname specifier will work if we assume that the FQDN of the host should be the default name for certs, which could be overridden with systemd overrides. However, since the hostname appears in more than one place (hostname of acme-register, certificate name with acme-deploy/-register), it may be better to either:

  • script the change (which is the purpose of openchami-cert-update) by generating the overrides, or
  • keep the openchami.env populated with the SYSTEM_URL

As an aside, I think removing demo.openchami.cluster as the default container (host)name for acme-register would be preferred since that would prevent us from having to modify/override that value in the service, simplifying the customization process.

In sum, keeping the SYSTEM_URL in openchami.env would allow easy customization of the FQDN, while using overrides might add undesirable complexity. Open to discuss.

@seantronsen

Copy link
Copy Markdown
Author
  1. Can we get rid of scripts/bootstrap_openchami.sh? See addendum for evidence on how it isn't used

It's used here as a postinstall script to do things like set up the Podman secrets. The logic there was a bit complex to include entirely within the spec file, which is why it's a separate script.

Ah... that's even in the grep output I posted. Sorry about that, I was reading too fast.

Signed-off-by: Sean Tronsen <sean.tronsen@gmail.com>
Signed-off-by: Sean Tronsen <sean.tronsen@gmail.com>
Signed-off-by: Sean Tronsen <sean.tronsen@gmail.com>
Signed-off-by: Sean Tronsen <sean.tronsen@gmail.com>
Signed-off-by: Sean Tronsen <sean.tronsen@gmail.com>
@seantronsen

seantronsen commented Jun 30, 2026

Copy link
Copy Markdown
Author
  1. In general, do we actually need a script that modifies systemd unit files post install? Based on the contents of openchami-certificate-update, do we even need one that creates override files? See addendum with further explanation below.

I think the %H systemd hostname specifier will work if we assume that the FQDN of the host should be the default name for certs, which could be overridden with systemd overrides. However, since the hostname appears in more than one place (hostname of acme-register, certificate name with acme-deploy/-register), it may be better to either:

  • script the change (which is the purpose of openchami-cert-update) by generating the overrides, or
  • keep the openchami.env populated with the SYSTEM_URL

As an aside, I think removing demo.openchami.cluster as the default container (host)name for acme-register would be preferred since that would prevent us from having to modify/override that value in the service, simplifying the customization process.

In sum, keeping the SYSTEM_URL in openchami.env would allow easy customization of the FQDN, while using overrides might add undesirable complexity. Open to discuss.

@synackd In the recent commits, I've tried some things out and made a few assumptions which deviate a bit from your recommendation a bit, but:

  • remains simple (assuming people only use the update script)
    • the system hostname, if configured with an fqdn, it can now serve as the default
    • overrides are marked with openchami managed essentially, and placed in the admin path. helper doc comments have been inserted to guide users to the correct files too.
  • I removed unused variables... possible red flag, though I did grep around all our repositories to see if any were actually used.
    • I'm unsure about the variables in the opaal.yaml file as those were never handled by the sed update functions to begin with. I've pinged @davidallendj to determine whether they are necessary expansions.

@seantronsen

Copy link
Copy Markdown
Author

Hold off on testing. Something is broken.

@seantronsen seantronsen force-pushed the main branch 2 times, most recently from 0166a09 to ba33bc2 Compare June 30, 2026 21:43
Signed-off-by: Sean Tronsen <sean.tronsen@gmail.com>
@seantrons

Copy link
Copy Markdown

@synackd it's ready again. Ignore the WIP commit for now. Figured I'd save commit cleanup for once this is farther along in the review process (assuming squash doesn't occur).

I was able to bypass the issue before with a distro path level containers.d/10-openchami.conf file.

Note

I still like this solution, but it requires an additional set of changes to be more elegant. The containers.d file I mentioned above applies the environment variables system wide to any quadlets. Obviously this poses an issue. However, systemd allows us to narrow the scope if we change our naming scheme for the quadlet files. See this link and quote:

"Moreover, for unit names containing dashes ("-"), the set of directories generated by repeatedly truncating the unit name after all dashes is searched too. Specifically, for a unit name foo-bar-baz.service not only the regular drop-in directory foo-bar-baz.service.d/ is searched but also both foo-bar-.service.d/ and foo-.service.d/. This is useful for defining common drop-ins for a set of related units, whose names begin with a common prefix. This scheme is particularly useful for mount, automount and slice units, whose systematic naming structure is built around dashes as component separators. Note that equally named drop-in files further down the prefix hierarchy override those further up, i.e. foo-bar-.service.d/10-override.conf overrides foo-.service.d/10-override.conf."

https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html?__goaway_challenge=meta-refresh&__goaway_id=1b2e43a18112a5608ad5d5274d105a59&__goaway_referer=https%3A%2F%2Fwww.google.com%2F

However, I figured it'd be best to ask before going off and making major naming changes.

@seantrons

seantrons commented Jun 30, 2026

Copy link
Copy Markdown

openchami-certificate-update should probably have a discard subcommand now, which when executed removes the quadlet override conf files. I'll try to add this tomorrow morning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants