FF_KANIKO_NATIVE_COPY deletes the kaniko dir when the executor is relocated with KANIKO_DIR

[mzihlmann]

Actual behavior

With FF_KANIKO_NATIVE_COPY=1 and the executor relocated via KANIKO_DIR (or --kaniko-dir), the relocation destroys the kaniko dir instead of moving it. The source dir is removed and the target dir ends up empty.

The build keeps going, because the executor is already running from memory, and then dies as soon as anything reads a file out of the kaniko dir. With FF_KANIKO_RUN_VIA_TINI=1 that is the first RUN:

INFO[0001] Running: [/kaniko2/tini -s -- /bin/sh -c echo hello]
error building image: error building stage: failed to execute command: starting command: fork/exec /kaniko2/tini: no such file or directory

-v debug shows the whole dir being dropped during relocation:

DEBU[0000] Skipping copy for ignored path: /kaniko2
DEBU[0000] Skipping copy for ignored path: /kaniko2/.docker
DEBU[0000] Skipping copy for ignored path: /kaniko2/docker-credential-acr-env
DEBU[0000] Skipping copy for ignored path: /kaniko2/docker-credential-ecr-login
DEBU[0000] Skipping copy for ignored path: /kaniko2/docker-credential-gcr
DEBU[0000] Skipping copy for ignored path: /kaniko2/executor
DEBU[0000] Skipping copy for ignored path: /kaniko2/ssl
DEBU[0000] Skipping copy for ignored path: /kaniko2/ssl/certs
DEBU[0000] Skipping copy for ignored path: /kaniko2/ssl/certs/ca-certificates.crt
DEBU[0000] Skipping copy for ignored path: /kaniko2/tini
DEBU[0000] Skipping copy for ignored path: /kaniko2/warmer
INFO[0000] updating env: SSL_CERT_DIR=/kaniko2/ssl/certs
INFO[0000] updating env: DOCKER_CONFIG=/kaniko2/.docker/

State after the failed build:

--- /kaniko2:
Dockerfile
--- /kaniko:
ls: /kaniko: No such file or directory

Without FF_KANIKO_RUN_VIA_TINI the same build reports success, so on that path the dir is destroyed silently and only the next consumer of a file in it (tini, the CA bundle, a credential helper) notices.

Expected behavior

Relocating the executor with KANIKO_DIR moves the whole kaniko dir to the new location, with or without FF_KANIKO_NATIVE_COPY.

To Reproduce

mkdir ctx && printf 'FROM alpine:3.21\nRUN echo hello\n' > ctx/Dockerfile

docker run --rm \
  -e KANIKO_DIR=/kaniko2 \
  -e FF_KANIKO_NATIVE_COPY=1 \
  -e FF_KANIKO_RUN_VIA_TINI=1 \
  -v "$PWD/ctx:/workspace:ro" \
  ghcr.io/osscontainertools/kaniko:v1.28.4-debug \
  --context /workspace --dockerfile Dockerfile --no-push

Fails with fork/exec /kaniko2/tini: no such file or directory.

To see the emptied dir directly:

docker run --rm \
  -e KANIKO_DIR=/kaniko2 -e FF_KANIKO_NATIVE_COPY=1 -e FF_KANIKO_RUN_VIA_TINI=1 \
  -v "$PWD/ctx:/workspace:ro" \
  --entrypoint /busybox/sh ghcr.io/osscontainertools/kaniko:v1.28.4-debug \
  -c '/kaniko/executor --context /workspace --dockerfile Dockerfile --no-push >/dev/null 2>&1; \
      echo "exit=$?"; echo "--- /kaniko2:"; /busybox/ls -A /kaniko2; \
      echo "--- /kaniko:"; /busybox/ls -A /kaniko 2>&1'

Dropping FF_KANIKO_NATIVE_COPY=1 makes it pass. Running the same command against ghcr.io/osscontainertools/kaniko:v1.28.3-debug passes as well, the flag does not exist there.

Analysis

MoveDir tries os.Rename first and falls back to copy-then-delete on EXDEV. On overlayfs, renaming a directory that still lives in a lower layer returns EXDEV, so for the relocation of /kaniko the fallback is effectively always taken.

pkg/util/fs_util.go:

func MoveDir(src, dest string) error {
	err := os.Rename(src, dest)
	if err == nil {
		return nil
	}

	if errors.Is(err, syscall.EXDEV) {
		// Cross-device move: copy + delete
		if config.FF.NativeCopy {
			err = CopyTree(src, dest, FileContext{})
		} else {
			...otiai10Cpy.Copy(src, dest, opts)
		}
		if err != nil {
			return err
		}

		err = os.RemoveAll(src)

CopyTree calls copyDirInner with skipIgnoreList=false:

func CopyTree(src, dest string, context FileContext) error {
	files, err := RelativeFiles("", src)
	if err != nil {
		return err
	}
	_, err = copyDirInner(files, src, dest, context, DoNotChangeUID, DoNotChangeGID, mode.Set{}, true, false)
	return err
}

and copyDirInner then tests the destination against the ignore list:

destPath := filepath.Join(dest, file)
if !skipIgnoreList && CheckIgnoreList(destPath) {
	logrus.Debugf("Skipping copy for ignored path: %s", destPath)
	continue
}

The first entry of the ignore list is config.KanikoDir, which during relocation is the destination. So every file is skipped, the copy is a no-op, and os.RemoveAll(src) then deletes the originals. The otiai10Cpy.Copy branch never consulted the ignore list, which is why the non-native path is unaffected.

Suggested fix

MoveDir relocates kaniko's own dir and should not consult the ignore list at all. Give CopyTree a skipIgnoreList parameter and pass true from MoveDir. The other caller, the RUN --mount=type=cache path in pkg/commands/run.go, copies into the build filesystem and should keep false.

Additional Information

• Dockerfile: FROM alpine:3.21 + RUN echo hello, see above. Any Dockerfile with a RUN reproduces it.
• Build Context: none beyond the Dockerfile.
• Kaniko Image: ghcr.io/osscontainertools/kaniko:v1.28.4-debug@sha256:716ab47e2f730a52edec6ab75a2866668012eaaa9c2d09ffec319c4a349448e3
• Introduced by d8ae987 (1743: preserve hardlinks during COPY --from - part 2 #626), which added the FF_KANIKO_NATIVE_COPY branch to MoveDir. First shipped in v1.28.4. The flag is off by default, so only opt-in users combining it with KANIKO_DIR relocation are affected.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this is a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile