Modifying existing file next to a RUN --mount path resets mode&ownership of all parent directories

[vit-zikmund]

Issue Description

Hi, this is a pretty strange bug, but here we are. When using a RUN --mount=type=cache while touching a previously existing directory (or a file within) next to the cache mount, the whole chain of parent directories (from the cache mount parent up to /) get botched mode&ownership flags (seems like a reset).

This was very hard to pinpoint, touching a file in the same directory hides the problem, operating on/in the cache mount doesn't trigger it.

Steps to reproduce the issue

Here's a rather minimalist but verbose Dockerfile to reproduce the issue:

FROM docker.io/library/ubuntu:latest
ARG WORKDIR=/home/testuser
ARG UID=1234

RUN set -eux ;\
    mkdir -p "$WORKDIR" ;\
    useradd -s /usr/sbin/nologin -d "$WORKDIR" -M -g 0 -u "$UID" testuser ;\
    chown -R "$UID:0" "$WORKDIR"
USER $UID
WORKDIR $WORKDIR

RUN \
    # let's create a directory owned by the user, notice the . directory (/home/testuser) is owned by it too
    mkdir -p somedir ;\
    ls -al ;\
    :

# let's mount a cache somewhere inside the workdir
RUN \
    --mount=type=cache,target=.cache \
    # alone it's not enough to cause something, but as soon an existing directory is touched
    # the whole parent tree of the workdir ownership/mode gets messed up:
    # thiss triggers it
    #touch somedir/newfile ;\
    # this too
    touch somedir ;\
    # this doesn't, though, in fact it partially prevents the problem
    #touch newfile ;\
    # still at this moment there's nothing wrong
    ls -al ;\
    :

#until now
RUN \
    ls -dla /;\
    ls -dla /home ;\
    ls -al ;\
    :

Build it with:

• podman build -t workdir-test:local --no-cache .
• or (for reference) docker build -t workdir-test:local --no-cache --progress=plain .

Describe the results you received

Run the container, checking the flags:

• podman

  $ podman run --rm -it -u0 workdir-test:local sh -c 'ls -dal /; ls -dal /home; ls -al /home/testuser;'
  dr-xr-xr-x. 1 root root 12 Dec 15 23:36 /               <<< 0555 instead of 0755, but that's most likely unrelated
  drwxr-xr-x. 1 root root 16 Dec 15 21:58 /home
  total 0
  drwxr-xr-x. 1 root     root 14 Dec 15 21:58 .           <<< uid(root) instead of uid(testuser), this is it
  drwxr-xr-x. 1 root     root 16 Dec 15 21:58 ..
  drwxr-xr-x. 1 testuser root  0 Dec 15 21:58 somedir
  

• docker

  $ docker run --rm -it -u0 workdir-test:local sh -c 'ls -dal /; ls -dal /home; ls -al /home/testuser;'
  drwxr-xr-x. 1 root root 0 Dec 15 23:36 /
  drwxr-xr-x. 1 root root 16 Dec 15 21:35 /home
  total 12
  drwxr-xr-x. 1 testuser root 14 Dec 15 21:35 .        <<< uid(testuser), no change
  drwxr-xr-x. 1 root     root 16 Dec 15 21:35 ..
  drwxr-xr-x. 1 testuser root  0 Dec 15 21:35 somedir
  

wagoodman/dive also produces quite some weird output:

• podman (dive <(podman save localhost/workdir-test:local) --source=docker-archive, 3rd layer)

  ┃ ● Current Layer Contents ┣━━━━━━━━━━━━━━━━━━━━━━━━━━
  Permission     UID:GID       Size  Filetree           
  ----------         0:0     4.8 kB  ├── home           
  ----------         0:0        0 B  │   ├── testuser   
  drwxr-xr-x      1234:0        0 B  │   │   └── somedir
  

• docker (dive <(docker save workdir-test:local) --source=docker-archive)

  ┃ ● Current Layer Contents ┣━━━━━━━━━━━━━━━━━━━━━━━━━━
  Permission     UID:GID       Size  Filetree           
  drwxr-xr-x         0:0     4.8 kB  ├── home           
  drwxr-xr-x      1234:0        0 B  │   ├── testuser   
  drwxr-xr-x      1234:0        0 B  │   │   └── somedir
  

Describe the results you expected

I'd definitely expect such an innocent command - run under a local user - wouldn't reset the owner/permissions of the parent directory.

podman version output

Client:        Podman Engine
Version:       5.8.1
API Version:   5.8.1
Go Version:    go1.25.7 X:nodwarf5
Git Commit:    c6077f645788743258a1a749f8005b4fb3cbe533
Built:         Wed Mar 11 01:00:00 2026
Build Origin:  Fedora Project
OS/Arch:       linux/amd64

podman info output

host:
  arch: amd64
  buildahVersion: 1.43.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.2.1-2.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.2.1, commit: '
  cpuUtilization:
    idlePercent: 83.15
    systemPercent: 3.74
    userPercent: 13.11
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  emulatedArchitectures:
  - linux/arm
  - linux/arm64
  - linux/arm64be
  - linux/loong64
  - linux/mips
  - linux/mips64
  - linux/ppc
  - linux/ppc64
  - linux/ppc64le
  - linux/riscv32
  - linux/riscv64
  - linux/s390x
  eventLogger: journald
  freeLocks: 2047
  hostname: feathora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.19.8-100.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2072834048
  memTotal: 33280327680
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    defaultNetwork: podman
    dns:
      package: aardvark-dns-1.17.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.17.0
    package: netavark-1.17.2-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.17.2
  ociRuntime:
    name: crun
    package: crun-1.26-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.26
      commit: 3241e671f92c33b0c003cd7de319e4f32add6231
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20260120.g386b5f5-1.fc42.x86_64
    version: |
      pasta 0^20260120.g386b5f5-1.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 114688
  swapTotal: 8589930496
  uptime: 224h 29m 4.00s (Approximately 9.33 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/<myself>/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/<myself>/.local/share/containers/storage
  graphRootAllocated: 427133239296
  graphRootUsed: 404489203712
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 126
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/<myself>/.local/share/containers/storage/volumes
version:
  APIVersion: 5.8.1
  BuildOrigin: Fedora Project
  Built: 1773187200
  BuiltTime: Wed Mar 11 01:00:00 2026
  GitCommit: c6077f645788743258a1a749f8005b4fb3cbe533
  GoVersion: go1.25.7 X:nodwarf5
  Os: linux
  OsArch: linux/amd64
  Version: 5.8.1

Provide your storage.conf

# This file is is the configuration file for all tools
# that use the containers/storage library. The storage.conf file
# overrides all other storage.conf files. Container engines using the
# container/storage library do not inherit fields from other storage.conf
# files.
#
#  Note: The storage.conf file overrides other storage.conf files based on this precedence:
#      /usr/containers/storage.conf
#      /etc/containers/storage.conf
#      $HOME/.config/containers/storage.conf
#      $XDG_CONFIG_HOME/containers/storage.conf (If XDG_CONFIG_HOME is set)
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver, Must be set for proper operation.
driver = "overlay"

# Temporary storage location
runroot = "/run/containers/storage"

# Primary Read/Write location of container storage
# When changing the graphroot location on an SELINUX system, you must
# ensure  the labeling matches the default locations labels with the
# following commands:
# semanage fcontext -a -e /var/lib/containers/storage /NEWSTORAGEPATH
# restorecon -R -v /NEWSTORAGEPATH
graphroot = "/var/lib/containers/storage"


# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"

# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file.  These ranges will be partitioned
# to containers configured to create automatically a user namespace.  Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = "false"

# Inodes is used to set a maximum inodes of the container image.
# inodes = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

# Set to skip a PRIVATE bind mount on the storage home directory.
# skip_mount_home = "false"

# Size is used to set a maximum size of the container image.
# size = ""

# ForceMask specifies the permissions mask that is used for new files and
# directories.
#
# The values "shared" and "private" are accepted.
# Octal permission masks are also accepted.
#
#  "": No value specified.
#     All files/directories, get set with the permissions identified within the
#     image.
#  "private": it is equivalent to 0700.
#     All files/directories get set with 0700 permissions.  The owner has rwx
#     access to the files. No other users on the system can access the files.
#     This setting could be used with networked based homedirs.
#  "shared": it is equivalent to 0755.
#     The owner has rwx access to the files and everyone else can read, access
#     and execute them. This setting is useful for sharing containers storage
#     with other users.  For instance have a storage owned by root but shared
#     to rootless users as an additional store.
#     NOTE:  All files within the image are made readable and executable by any
#     user on the system. Even /etc/shadow within your image is now readable by
#     any user.
#
#   OCTAL: Users can experiment with other OCTAL Permissions.
#
#  Note: The force_mask Flag is an experimental feature, it could change in the
#  future.  When "force_mask" is set the original permission mask is stored in
#  the "user.containers.override_stat" xattr and the "mount_program" option must
#  be specified. Mount programs like "/usr/bin/fuse-overlayfs" present the
#  extended attribute permissions to processes within containers rather then the
#  "force_mask"  permissions.
#
# force_mask = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# metadata_size is used to set the `pvcreate --metadatasize` options when
# creating thin devices. Default is 128k
# metadata_size = ""

# Size is used to set a maximum size of the container image.
# size = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Installation Source

Distribution package (DNF, apt, yay)

Additional environment details

Additional environment details

Additional information

I've originally submitted this issue as podman-container-tools/podman#27777, but no maintainer made it to the issue so far :( Trying my luck here, also it seems that is indeed a problem with podman build rather than podman itself...

[awillinger-work]

Noticed this issue as well, but with bind mounts. In my case, a WAR file is mounted into the container and extracted into a directory, after which all permissions go haywire, just as shown above (owned by 0:0 instead of 1000:0).

As a workaround, I switch to the root user and set the permissions to 777 recursively, before switching back to the actual run user. Even using chown on the parent directory does not fix it, the owner still remains root. Interestingly, after setting the permissions to 777 and launching the container, the intended owner (UID 1000) is re-set on the hierarchy.

Even though a workaround exists, imo this is a critical bug, as it makes using bind mounts together with rootless containers useless when you can't properly launch them afterwards. In my case, I am using them to start Wildfly, which is then unable to create its data directory and refuses to start up.

Using a semi-recent (2 week old) version of the sources built myself, as the distribution package in Debian 13 is providing a too old package which still contains the bug which got fixed in bc05858.

[vit-zikmund]

I spent some time poking around and found a couple interesting pieces to this puzzle.

As I wrote before in the Dockerfile comments, I thought I was able to workaround the bug - at least locally in the lowest directory - by touching a file there. This led to a state where that directory's permissions didn't get reset, only the upper ones took the hit. That was good enough for me, so I was appending the following line to RUN commands everywhere I used mounts:

rm -f .podman-bug-workaround; touch .podman-bug-workaround

I don't know if the rm was necessary, but that's irrelevant. It worked... until once it suddenly didn't. And then it worked again. That's where I found out the behavior is non-deterministic. So there's likely some sort of a race condition 🤔

I thought this has something to do with the [local] layer caching, but I ruled that out by running with --no-cache. It seems it's the committed layer that eventually gets spoiled, so when using caching, the spoiled layer is cached and - unless invalidated - reused in the next build.

Talking about layers, I've also accidentally stumbled over another workaround - not using them! I read that podman build is actually just a proxy to buildah build, so I tried that out of curiosity and bam! The build was quicker and the bug never manifested 🤯 What I found soon is that some of the arguments to podman|buildah build don't have the same defaults. Here it's the --layers parameter, which is true for podman, but false for buildah. So by default, buildah runs through the whole Dockerfile, producing just one layer at the very end (more if there are named stages). It's the same as with podman build --layers=false.
This, however, prevents the use of any [layer] cache 🫤, and any image inspection is pretty lame after that (the final layer command only takes the last directive). But it sure can fit well in some less-demanding use cases.

My last workaround I'm about to use now is to provide a conditionally corrective layer after each RUN --mount where this could be relevant:

RUN \[ $(stat -c '%u' "$WORKDIR") = "$UID" ] || { printf "Fixing WORKDIR permissions!\n" >&2; chown "$UID" "$WORKDIR"; }

It works, but boy, is that ugly 🫩

[awillinger-work]

@vit-zikmund Noticed the layer thing as well. In my case, I build the base image using "regular" Docker on a local machine (as it rarely changes), the overlay/deployment image (which extends from the base) is built automatically by Jenkins using Buildah. Unlike regular Docker and as you have already found out, it only creates a single layer for the entire Dockerfile, which made inspecting the final image a bit tricky.

In my case, the bug really seems to manifest from Buildah itself, as the permissions are entirely fine in the base image built by Docker. It also only started occuring once I upgraded from Debian 12 to 13 which itself has a broken version of Buildah, so I am currently using a self-built version from this repository.

Fwiw, I am using the "bud" (build using Dockerfile) not the regular "build" command.

[vit-zikmund]

@awillinger-work I confirm your observations (at the end of the previous podman issue), this indeed is a regression introduced somewhere in podman 5 (and the relevan buildah). But podman 4's buildah has the problem with layer caching & RUN mounts you already mentioned, so we still have to fix this to be able to use RUN mounts together with layer caching (a little CI holy grail).

Hats off for you building the stuff from scratch, I've resorted to podman-static on our Ubuntu 24.04 CI nodes (that also ship "broken" podman v4 / buildah).

Ultimately I'd also like to re-add here the observation that the same problem happens (at least) also with a writable bind mount:

RUN --mount=type=bind,from=base,source=/tmp,target=.cache,rw

which I originally tried as a workaround to the cache mount.

FWIW buildah build is just a modern alias for buildah bud (that was originally a shorthand for - now obsolete - build-using-dockerfile 😉)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[awillinger-work]

Commenting here to remove the stale tag.

Any input from the devs?