Skip to content

Unify coreos build - #85

Open
madhu-pillai wants to merge 1 commit into
trusted-execution-clusters:mainfrom
madhu-pillai:unify
Open

madhu-pillai wants to merge 1 commit into
trusted-execution-clusters:mainfrom
madhu-pillai:unify

Conversation

@madhu-pillai

Copy link
Copy Markdown

hi,
Removed the /usr , Containerfile and modified justfile to pull the images default from fedora-coreos-tec-config

Thanks
Madhu

Comment thread coreos/justfile Outdated
#kbc_image := "quay.io/trusted-execution-clusters/trustee-attester:fedora-b13fd8a"
#clevis_pin_trustee_image := "quay.io/trusted-execution-clusters/clevis-pin-trustee:fedora-75015a5"
#ignition_image := "quay.io/trusted-execution-clusters/ignition:fedora-af7bcce09"
tec_config := env("TEC_CONFIG", "../../fedora-coreos-tec-config")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where are you fetching those configuration? I think it require the remote github repository

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reconstructed

Comment thread coreos/justfile Outdated
Comment on lines +31 to +33
#kbc_image := "quay.io/trusted-execution-clusters/trustee-attester:fedora-b13fd8a"
#clevis_pin_trustee_image := "quay.io/trusted-execution-clusters/clevis-pin-trustee:fedora-75015a5"
#ignition_image := "quay.io/trusted-execution-clusters/ignition:fedora-af7bcce09"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think those variable are still necessary, we have no defaults in https://github.com/trusted-execution-clusters/fedora-coreos-tec-config/blob/main/Containerfile#L6-L9

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reconstructed

Comment thread coreos/justfile Outdated
Comment on lines +46 to +47
##@echo "kbc_image {{kbc_image}}"
#@echo "clevis_pin_trustee_image {{clevis_pin_trustee_image}}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why removing those?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reconstructed.

Comment thread coreos/justfile Outdated
Comment on lines +69 to +79
podman build --no-cache \
--build-arg BASE={{base}} \
--build-arg KBC_IMG="${KBC_IMG}" \
--build-arg CLEVIS_PIN_IMG="${CLEVIS_PIN_IMG}" \
--build-arg IGNITION_IMG="${IGNITION_IMG}" \
--build-arg NAME={{os_name}} \
--build-arg ID={{os}} \
--build-arg VERSION="${VERSION}" \
--build-arg STREAM="${STREAM}" \
--build-arg DESCRIPTION="{{os_name}} for trusted execution clusters" \
-t {{image}} -f {{tec_config}}/Containerfile {{tec_config}}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to use cosa init and cose build instead of building directly with podman

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@alicefr

alicefr commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Could you please sign the commit and add a short description in the commit body?

@madhu-pillai
madhu-pillai force-pushed the unify branch 2 times, most recently from d93d607 to 6f9b598 Compare September 1, 2026 17:11
@madhu-pillai

Copy link
Copy Markdown
Author

@alicefr ,
I have done the testing following way.


1. sudo podman pod start compose-pod                                                  #Create a httpd trustee pod.
2. export IGNITION_IMG="quay.io/trusted-execution-clusters/ignition:fedora-af7bcce09" # working ignition at present
3. export REGISTRY_AUTH_FILE=~/.config/containers/pull-secret.json                    # building rhcos
4. cd investigations/coreos
5. just build && just build-qemu
6. images_fcos=/var/lib/libvirt/images/rhcos-10.1.20260901-dev0-qemu.x86_64.qcow2   # The rhel actually took 9.8 (not rhel 10)
7. export TRUSTEE_ADDR=192.168.122.1
8. bash -xv scripts/install_vm.sh -k /home/azureuser/coreos.key.pub -b configs/ak.bu -i ${images_fcos} -n rhcos10 

Able to boot the coreos image using attestation (testing).

[  OK  ] Finished Commit a transient machine-id on disk.

Red Hat Enterprise Linux CoreOS 9.8.20260826-0 (Plow)
Kernel 5.14.0-687.42.1.el9_8.x86_64 on an x86_64

SSH host key: SHA256:Dey3ORTWJP3xqM7p6VMHNUPk4DxvTU6QPqAkhZ5ayDY (ECDSA)
SSH host key: SHA256:+IQ/IxKCszVzoQs2uwXRUbf5Br745Vc9/zgzFj/X1F8 (ED25519)
SSH host key: SHA256:KSIs9Lr/VJAoWURZ4VVAlv04VnHiqQMqX8CtKdDgCxw (RSA)
enp1s0: 192.168.122.163 fe80::88c0:a942:aaba:2395
Ignition: ran on 2026/09/01 05:29:59 UTC (this boot)
Ignition: user-provided config was applied
Ignition: warning at $.attestation: unused key attestation
localhost login: core (automatic login)

Red Hat Enterprise Linux CoreOS 9.8.20260826-0
  Part of OpenShift 5.0, RHCOS is a Kubernetes-native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/5.0/architecture/architecture-rhcos.html

---

############################################################################
WARNING: The root filesystem is too small. It is strongly recommended to
allocate at least 8 GiB of space to allow for upgrades. From June 2021, this
condition will trigger a failure in some cases. For more information, see:
https://docs.fedoraproject.org/en-US/fedora-coreos/storage/

You may delete this warning using:
sudo rm /etc/motd.d/60-coreos-rootfs-size.motd
############################################################################

[core@localhost ~]$ rpm-ostree status
State: idle
Deployments:
● ostree-image-signed:oci-archive:/rhcos-10.1.20260901-dev0-ostree.x86_64.ociarchive
                   Digest: sha256:579a726e7f7c8c24ce865b0fd4b7f732507cb63515269246161468703d9dcbca
                  Version: 10.1.20260901-dev0 (2026-09-01T05:07:51Z)
[core@localhost ~]$  cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="9.8.20260826-0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.8"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 9.8.20260826-0 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.8"
OSTREE_VERSION='9.8.20260826-0'
IMAGE_VERSION='9.8.20260826-0'
VARIANT=CoreOS
VARIANT_ID=coreos
OPENSHIFT_VERSION="5.0"
[core@localhost ~]$   ostree admin status
* rhcos 2aa13dd7b0ce41d26e53b8baabc7419413355394d86fa12d48c667d296fd5465.0
    origin: <unknown origin type>




[core@localhost ~]$   journalctl -b -1 | grep -i "clevis\|luks\|cryptsetup"
Sep 01 05:28:44 localhost.localdomain systemd[1]: Starting RHCOS Check For Legacy LUKS Configuration...
Sep 01 05:28:44 localhost.localdomain systemd[1]: Finished RHCOS Check For Legacy LUKS Configuration.
Sep 01 05:28:59 localhost.localdomain ignition[910]: disks: createLuks: op(1): [started]  waiting for devices [/dev/disk/by-partlabel/root]
Sep 01 05:28:59 localhost.localdomain ignition[910]: disks: createLuks: op(1): [finished] waiting for devices [/dev/disk/by-partlabel/root]
Sep 01 05:28:59 localhost.localdomain ignition[910]: disks: createLuks: created device alias for "/dev/disk/by-partlabel/root": "/run/ignition/dev_aliases/dev/disk/by-partlabel/root" -> "/dev/vda4"
Sep 01 05:28:59 localhost.localdomain ignition[910]: disks: createLuks: op(2): [started]  wiping filesystem signatures from "/run/ignition/dev_aliases/dev/disk/by-partlabel/root"
Sep 01 05:28:59 localhost.localdomain ignition[910]: disks: createLuks: op(2): executing: "wipefs" "-a" "/run/ignition/dev_aliases/dev/disk/by-partlabel/root"
Sep 01 05:28:59 localhost.localdomain ignition[910]: disks: createLuks: op(2): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/disk/by-partlabel/root"
Sep 01 05:28:59 localhost.localdomain ignition[910]: disks: createLuks: op(3): [started]  creating "root"
Sep 01 05:28:59 localhost.localdomain ignition[910]: disks: createLuks: op(3): executing: "cryptsetup" "luksFormat" "--type" "luks2" "--key-file" "/tmp/ignition-luks-1833367759" "--label" "luks-root" "--uuid" "8ec9cda3-6b77-45d7-bb56-a95cd9e83234" "/run/ignition/dev_aliases/dev/disk/by-partlabel/root"
Sep 01 05:29:05 localhost.localdomain ignition[910]: disks: createLuks: op(3): [finished] creating "root"
Sep 01 05:29:05 localhost.localdomain ignition[910]: disks: createLuks: op(4): [started]  opening luks device root
Sep 01 05:29:05 localhost.localdomain ignition[910]: disks: createLuks: op(4): executing: "cryptsetup" "luksOpen" "/run/ignition/dev_aliases/dev/disk/by-partlabel/root" "root" "--key-file" "/tmp/ignition-luks-1833367759" "--persistent"
Sep 01 05:29:07 localhost.localdomain ignition[910]: disks: createLuks: op(4): [finished] opening luks device root
Sep 01 05:29:07 localhost.localdomain ignition[910]: disks: createLuks: op(5): [started]  Clevis bind
Sep 01 05:29:07 localhost.localdomain ignition[910]: disks: createLuks: op(5): executing: "clevis" "luks" "bind" "-f" "-k" "/tmp/ignition-luks-1833367759" "-d" "/run/ignition/dev_aliases/dev/disk/by-partlabel/root" "trustee" "{\n  \"servers\":[{\"url\":\"http://192.168.122.1:8080\",\"cert\":\"\"}],\n  \"path\":\"default/machine/root\",\n  \"attestation_key\": {\n    \"registration\": {\n       \"url\": \"http://192.168.122.1:5001\",\n       \"uuid\": \"machine\",\n       \"cert\": \"\"\n    }\n  }\n}\n"
Sep 01 05:29:14 localhost.localdomain ignition[910]: disks: createLuks: op(5): [finished] Clevis bind
Sep 01 05:29:14 localhost.localdomain ignition[910]: disks: createLuks: op(6): [started]  removing key file for root
Sep 01 05:29:14 localhost.localdomain ignition[910]: disks: createLuks: op(6): executing: "cryptsetup" "luksRemoveKey" "/run/ignition/dev_aliases/dev/disk/by-partlabel/root" "/tmp/ignition-luks-1833367759"
Sep 01 05:29:16 localhost.localdomain ignition[910]: disks: createLuks: op(6): [finished] removing key file for root
Sep 01 05:29:16 localhost.localdomain ignition[910]: disks: createLuks: op(7): [started]  waiting for triggered uevent
Sep 01 05:29:16 localhost.localdomain ignition[910]: disks: createLuks: op(7): executing: "udevadm" "trigger" "--settle" "/dev/vda4"
Sep 01 05:29:16 localhost.localdomain ignition[910]: disks: createLuks: op(7): [finished] waiting for triggered uevent
Sep 01 05:29:18 localhost.localdomain ignition-ostree-growfs[2377]: Decrypt with header: ClevisHeader { pin: "trustee", servers: [Server { url: "http://192.168.122.1:8080", cert: "" }], path: "default/machine/root", initdata: None, num_retries: None }
Sep 01 05:29:18 localhost.localdomain ignition-ostree-growfs[2377]: Attempting to fetch LUKS key (attempt 1/10)
Sep 01 05:29:18 localhost.localdomain ignition-ostree-growfs[2377]: Successfully fetched LUKS key from URL: http://192.168.122.1:8080
Sep 01 05:29:59 localhost.localdomain coreos-boot-edit[2537]: Injected kernel arguments into BLS: rd.luks.name=8ec9cda3-6b77-45d7-bb56-a95cd9e83234=root rd.neednet=1 rd.luks.options=_netdev root=UUID=910678ff-f77e-4a7d-8d53-86f2ac47a823 rw
Sep 01 05:30:00 localhost.localdomain systemd[1]: rhcos-fail-boot-for-legacy-luks-config.service: Deactivated successfully.
Sep 01 05:30:00 localhost.localdomain systemd[1]: Stopped RHCOS Check For Legacy LUKS Configuration.
Sep 01 05:30:01 localhost.localdomain systemd[1]: clevis-luks-askpass.path: Deactivated successfully.
Sep 01 05:30:01 localhost.localdomain systemd[1]: Stopped Forward Password Requests to Clevis Directory Watch.
Sep 01 05:30:03 localhost.localdomain systemd[1]: clevis-luks-askpass.path: Deactivated successfully.
Sep 01 05:30:03 localhost.localdomain systemd[1]: Stopped Forward Password Requests to Clevis Directory Watch.
Sep 01 05:30:03 localhost.localdomain systemd[1]: systemd 252-67.el9_8.4 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Sep 01 05:30:03 localhost.localdomain systemd[1]: Created slice Cryptsetup Units Slice.
Sep 01 05:30:03 localhost.localdomain systemd[1]: Started Forward Password Requests to Clevis Directory Watch.
Sep 01 05:30:04 localhost.localdomain systemd-cryptsetup[2783]: Volume root already active.

@Jakob-Naucke Jakob-Naucke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @madhu-pillai, could you commit under an author email associated with the same GH account and take a look at the CI failure?

I also still want to test the KubeVirt image

Comment thread coreos/justfile Outdated
cd cache
cosa init --force {{config}}
cosa import oci-archive:/srv/{{archive}}
# 1. Init with upstream config (provides osbuild, platforms.yaml, BUILDER_IMG)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove numeral steps, requires shifting every time we change them

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do that

@Jakob-Naucke Jakob-Naucke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does, but I'd probably want to bump the components in the fedora config repo before we merge this

@madhu-pillai
madhu-pillai force-pushed the unify branch 2 times, most recently from dce4ea0 to 5515bd2 Compare September 7, 2026 06:33
Replace the two-phase podman build + cosa import workflow with a
unified cosa build approach that works across fcos, scos, and rhcos.

Previous approach:
  1. podman build --build-arg BASE=<img> -f Containerfile
  2. skopeo copy → oci-archive
  3. cosa init --force <upstream-config>
  4. cosa import oci-archive
  5. cosa osbuild qemu

New approach:
  1. cosa init --force <base_config>   (upstream osbuild infra)
  2. git clone <tec_config> + overlay  (TEC files on top)
  3. cosa build                        (buildah multi-stage)
  4. cosa osbuild qemu

Key changes:
- Split config variable into base_config (upstream osbuild infra)
  and tec_config (TEC Containerfile, build-args, dracut modules)
- Remove podman build, oci-archive, init targets — replaced by
  unified build target with overlay logic
- Fix cosa function: replace sudo podman run -u 0 with
  podman run --userns=keep-id:uid=1000,gid=1000 matching upstream
- Add BUILDER_IMG save/restore across TEC overlay (TEC build-args.conf
  doesn't carry BUILDER_IMG needed for osbuild buildroot python3)
- Add base_img override: SCOS shares RHCOS TEC config which has
  restricted Red Hat base — override with public OKD scos-content
- Make component images (KBC_IMG, CLEVIS_PIN_IMG, IGNITION_IMG)
  and OS_VERSION overridable via environment variables
- Add usage header and step-numbered comments in build target
- Add podman rm -f cosa before targets to handle stale containers
- Add changes in CI .github/workflows/build-fcos-images.yml

Signed-off-by: Madhu Pillai <mapillai@redhat.com>
@madhu-pillai

Copy link
Copy Markdown
Author

@Jakob-Naucke ,

thanks @madhu-pillai, could you commit under an author email associated with the same GH account and take a look at the CI failure?

I also still want to test the KubeVirt image

Looks like my redhat github account mapillai@redhat.com has locked out, i do not have recovery codes and 2FA, all my contribution done so far from my present id. Let me fix the commit id

Comment thread coreos/justfile
Comment on lines +126 to +129
# Apply env overrides if set
[ -n "{{kbc_image}}" ] && sed -i "s|^KBC_IMG=.*|KBC_IMG={{kbc_image}}|" src/config/build-args.conf
[ -n "{{clevis_pin_trustee_image}}" ] && sed -i "s|^CLEVIS_PIN_IMG=.*|CLEVIS_PIN_IMG={{clevis_pin_trustee_image}}|" src/config/build-args.conf
[ -n "{{ignition_image}}" ] && sed -i "s|^IGNITION_IMG=.*|IGNITION_IMG={{ignition_image}}|" src/config/build-args.conf

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that before we were overwriting the variables, but maybe we shouldn't. If somebody wants to try a local build they need to modify the fedora/rhcos config directly. This scripts has become too complex imo

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do not requires overrides then i'll remove it.

Comment thread coreos/justfile
Comment on lines +112 to +119
# Override the os version
[ -n "{{os_version}}" ] && sed -i "s|^VERSION=.*|VERSION={{os_version}}|" src/config/build-args.conf

cp /tmp/tec-overlay/manifest.yaml src/config/
cp /tmp/tec-overlay/versionary src/config/
cp -rL /tmp/tec-overlay/usr src/config/
rm -rf /tmp/tec-overlay
echo "!/usr/" >> src/config/.containerignore

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary? Won't this be already correctly set based on the configuration we take?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

112 and 113, will remove.
what i am trying to achieve here was using cosa instead of manually running the podman cosa init <upstream config> then copy the necessary tec config to /src/config then build the image. So any changes user requires can directly apply from tec config.

However, I tried with only the cosa init <tec config> the build works, but when we create image just build-qemu it fails in error says following. Looks like it does not have the builder-img.

I did not try the cosa

Building FCOS buildroot container
[1/2] STEP 1/3: FROM overridden AS builder
Error: creating build container: short-name resolution enforced but cannot prompt without a TTY
failed to execute cmd-buildextend-qemu: exit status 125
+ rc=125
+ set +x

Comment thread coreos/justfile
Comment on lines +109 to +110
cp "$BUILD_ARGS" src/config/build-args.conf
[ -n "{{base_img}}" ] && sed -i "s|^BASE=.*|BASE={{base_img}}|" src/config/build-args.conf

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I commented below, the script is too complex. I think we should set the default in the configuration and rely on those. If the users want to build a different version, they can always fork and change the configuration in their repository

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for clarification: we do want to keep the option to override guest components images, right? that I see myself using a lot and forking and switching branches in l. 100 is really a lot more clumsy

Comment thread coreos/justfile

# Clone TEC config and overlay (Containerfile, build-args, manifest, versionary, usr/)
rm -rf /tmp/tec-overlay
git clone --depth=1 --recurse-submodules {{tec_config}} /tmp/tec-overlay

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I barely dare ask this, but regardless of env support, do we need to support branches here?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better to have branch support too. Its easier for testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants