Build fails immediately on Aarch64 / DGX Spark

[7schroet]

Actual behavior
The first instruction in a Dockerfile always fails with error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory.

Expected behavior
The expected behaviour is not crashing. I also tested this with buildah, and that works fine. Sidenote: buildah produces these warnings

time="2026-09-07T14:48:29Z" level=warning msg="pkg/chroot: error unmounting \"/var/tmp/buildah184258130/mnt/rootfs\" (retried 0 times): operation not permitted"
time="2026-09-07T14:48:29Z" level=warning msg="pkg/bind: error detaching \"/var/tmp/buildah184258130/mnt/rootfs\": operation not permitted"
time="2026-09-07T14:48:29Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah184258130/mnt/rootfs\": device or resource busy"
time="2026-09-07T14:48:29Z" level=warning msg="pkg/bind: error detaching \"/var/tmp/buildah184258130/mnt\": operation not permitted"
time="2026-09-07T14:48:29Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah184258130/mnt\": device or resource busy"

but I assume that is not relevant, as it looks similar to kaniko's output before #754.

To Reproduce
Dockerfile:

FROM ubuntu:24.04

RUN echo "hi"

Ran via GitLab CI:

build-container-kaniko:
  stage: manual
  image:
    name: docker.io/martizih/kaniko:v1.28.4-debug
    entrypoint: ['']
  variables:
    IMAGE_NAME: "$CI_REGISTRY_IMAGE/kaniko:$CI_COMMIT_SHA"
  script:
    - /kaniko/executor --context . --dockerfile Dockerfile --destination $IMAGE_NAME --verbosity=debug
  rules:
    - when: manual

Additional Information
Using martizih/kaniko@sha256:45cd9555c2379c22523c613f1a1a1cb4419264aaf9e296a69bd4291f5f71d491, which is the ARM image.

Important to note: this runs on a DGX Spark, so the Nvidia runtime is involved. I don't know if this is specific to this system, or if it fails on other ARM systems as well.

The complete logfile from GitLab CI:

Running with gitlab-runner 19.1.1 (24b9b726)
  on spark-builder sUWXE8HHQ, system ID: s_83ac57b0bd91
Preparing the "docker" executor 00:03
Using Docker executor with image docker.io/martizih/kaniko:v1.28.4-debug ...
Using effective pull policy of [always] for container docker.io/martizih/kaniko:v1.28.4-debug
Pulling docker image docker.io/martizih/kaniko:v1.28.4-debug ...
Using docker image sha256:d8082f5ea52a391235e412a3d85d5c1d312f971d075d45019e7fa45b5694c3b1 for docker.io/martizih/kaniko:v1.28.4-debug with digest martizih/kaniko@sha256:45cd9555c2379c22523c613f1a1a1cb4419264aaf9e296a69bd4291f5f71d491 ...
Preparing environment 00:00
Using effective pull policy of [always] for container sha256:74c906d45be542ce4e2dd21a7bbc55229fbf82881c8b86f454c626a0f4c600e5
Running on runner-suwxe8hhq-project-141876-concurrent-0 via spark1...
Getting source from Git repository 00:01
Gitaly correlation ID: 01M1Y5FZPSG91VA53C1D8YRECS
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/k202183/icon-spark-image/.git/
Created fresh repository.
Checking out febffacc as detached HEAD (ref is repro)...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:03
Using effective pull policy of [always] for container docker.io/martizih/kaniko:v1.28.4-debug
Using docker image sha256:d8082f5ea52a391235e412a3d85d5c1d312f971d075d45019e7fa45b5694c3b1 for docker.io/martizih/kaniko:v1.28.4-debug with digest martizih/kaniko@sha256:45cd9555c2379c22523c613f1a1a1cb4419264aaf9e296a69bd4291f5f71d491 ...
$ /kaniko/executor --context . --dockerfile Dockerfile --destination $IMAGE_NAME
INFO[0000] active feature flags: FF_KANIKO_BUILDKIT_ARG_ENV_PRECEDENCE, FF_KANIKO_CLEAN_KANIKO_DIR, FF_KANIKO_DEPRECATE_INTER_STAGE_RESTORE, FF_KANIKO_NO_PROPAGATE_ANNOTATIONS, FF_KANIKO_OCI_WARMER, FF_KANIKO_PRESERVE_HARDLINKS, FF_KANIKO_RELATIVE_LINK_TARGETS, FF_KANIKO_PRESERVE_MOUNTED_PATHS, FF_KANIKO_RUN_MOUNT_BIND, FF_KANIKO_SECUREJOIN_EXTRACTION, FF_KANIKO_VOLUME_SKIP_MKDIR, FF_KANIKO_WARMER_CACHE_LOCK 
INFO[0000] credential providers by priority: [env, google, ecr, acr, gitlab] 
INFO[0000] Retrieving image manifest ubuntu:24.04       
INFO[0000] Retrieving image ubuntu:24.04 from registry index.docker.io 
INFO[0000] credential providers by priority: [env, google, ecr, acr, gitlab] 
INFO[0001] Retrieving image manifest ubuntu:24.04       
INFO[0001] Returning cached image manifest              
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest ubuntu:24.04       
INFO[0001] Returning cached image manifest              
INFO[0001] Building stage 'ubuntu:24.04' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd RUN echo "hi" requires it. 
INFO[0002] Initializing snapshotter ...                 
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] RUN echo "hi"                                
INFO[0002] Cmd: /bin/sh                                 
INFO[0002] Args: [-c echo "hi"]                         
INFO[0002] Running: [/bin/sh -c echo "hi"]              
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1

Triage Notes for the Maintainers

Description	Yes/No
Please check if this is a new feature you are proposing	No
Please check if the build works in docker but not in kaniko	Yes
Please check if this error is seen when you use --cache flag	not relevant
Please check if your dockerfile is a multistage dockerfile	not relevant

[mzihlmann]

Hi @7schroet thank you for the report. I will have a look shortly

[mzihlmann]

It doesn't fix the underlying issue but if you don't need nvidia runtime for that container you can disable injection

NVIDIA_VISIBLE_DEVICES: void

the issue is real and novel, it just wasn't discoverable prior to #754 because we already errored out beforehand.