Running docker inside a sysbox-container - inner docker db (containerd snapshotter) gets ruined in case the sysbox-container deleted and re-created

[yogch]

I'm running Docker + Sysbox on Ubuntu - running a sysbox-container with an inner Docker engine.

Ubuntu 22.04.1 LTS
Kernel: 5.15.0-43-generic

• Outer Docker + Sysbox:

# docker --version
Docker version 29.6.1, build 8900f1d

# docker info | grep Storage -A1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1

# sysbox-mgr --version
sysbox-mgr
        edition:        Community Edition (CE)
        version:        0.7.0
        commit:         bff3721f86e737cfa129dfe9fe2e7933692aba26
        built at:       Tue Mar  3 21:52:05 UTC 2026
        built by:       Cesar Talledo

• Inner Docker:

# docker --version
Docker version 29.5.2, build 79eb04c

# docker info | grep Storage -A1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1

I run the sysbox-container with 2 volumes, so, in case the sysbox-container needs to be deleted and re-created, the inner's docker db will be kept:

docker run --runtime=sysbox-runc ... -v docker_volume:/var/lib/docker -v containerd_volume:/var/lib/containerd ...

After the sysbox-container starts, I can run the following inside fine:

# docker run -it --name alpine_sleep alpine sleep 1
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
55afa1ecc21d: Pull complete
f5124fb579e2: Download complete
56dceff11b33: Download complete
Digest: sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b
Status: Downloaded newer image for alpine:latest
# docker start alpine_sleep
alpine_sleep

Then, I'm deleting and re-creating the sysbox-container and run the commands again.
Note - the new sysbox-container uses the same volumes as before.

# docker start alpine_sleep
Error response from daemon: RWLayer of container deedb7851b87c5e01816fd9b724bd7eb99b93f2ba864a27194f3cb7007a56c2a is unexpectedly nil
failed to start containers: alpine_sleep

# docker image ls
                                                                                                                                                                       i Info →   U  In Use
IMAGE           ID             DISK USAGE   CONTENT SIZE   EXTRA
alpine:latest   28bd5fe8b56d       3.93MB         3.93MB    U

# docker rm alpine_sleep
alpine_sleep
# docker run -it --name alpine_sleep alpine sleep 1
docker: Error response from daemon: NotFound: failed to create snapshot: missing parent "moby/3/sha256:34884abbe92863fce933ed7c39c0e045631af0ed86d5cc0dfbdf9fdca426ce3c" bucket: not found

Run 'docker run --help' for more information

# docker image rm alpine:latest
Untagged: alpine:latest
Deleted: sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b

# docker run -it --name alpine_sleep alpine sleep 1
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
56dceff11b33: Download complete
f5124fb579e2: Download complete
docker: unable to prepare extraction snapshot: AlreadyExists: target snapshot "sha256:34884abbe92863fce933ed7c39c0e045631af0ed86d5cc0dfbdf9fdca426ce3c": already exists

Run 'docker run --help' for more information

As can be seen, some db is corrupted. The container alpine_sleep still exists, but it cannot be started.
Also, the saved image alpine:latest cannot be used.

This flow works fine without --runtime=sysbox-runc (and with --privileged, or other caps instead).

I've tried to --disable-shiftfs, to use file binds instead of volumes. It did not work.
When removing both mounts from the sysbox-container's docker run command: -v ${var_lib_docker_volume}:/var/lib/docker -v ${containerd_volume}:/var/lib/containerd everything works fine. But, I do not have any docker db and need to start everything from scratch.

• Is there a way to make the new sysbox-container use the old sysbox-container's volumes without an issue?
• Are there other files written to the fs which aren't in /var/lib/docker and /var/lib/containerd that I need to put in a volume and share between containers as well?

This seems like a legitimate flow - using persistent volumes. I don't understand why it wouldn't work.

Thank you.

[yogch]

Looks like it relates to Storage Driver: overlayfs (The new containerd snapshotter)

Inside the sysbox container:

• Current state:

# docker info | grep -i storage
 Storage Driver: overlayfs

• I forced overlay2

# vi /etc/docker/daemon.json # Added: "storage-driver": "overlay2"
# systemctl restart docker

• Now I get overlay2:

# docker info | grep -i storage
 Storage Driver: overlay2

And everything works as expected - sysbox container's docker's db does not get corrupted after reboots/deletion+creation.