Lepton (Android compat tool): titles start but immediately exit

[loki666]

Summary

Valve's Lepton compat tool (Steam app 3029110) is selected correctly by the Steam client on
our image, but cannot launch anything: Lepton drives its Android container entirely through
podman, and our rootfs ships no container runtime at all.

Measured on an AYANEO Pocket ACE (SM8550), dev build 9bde3fe / BUILD_ID=20260822T094920Z,
kernel 7.2.0.

Symptom

Launching an Android title (AOG.apk, app 4545850) starts the tool and dies ~5s later:

[18:13:26] AppID 4545850 adding PID 2345 as a tracked process
  ... reaper SteamLaunch AppId=4545850 -- '.../Lepton'/lepton waitforexitandrun -- '.../AOG.apk'
[18:13:31] AppID 4545850 no longer tracking PID 2349, exit code -1
[18:13:31] AppID 4545850 no longer tracking PID 2348, exit code -1
[18:13:31] Remove 4545850 from running list

steamapps/compatdata/4545850/ contains only baked/data_overlay — it never got as far as
building a container.

Root cause

Lepton calls podman for every container operation, resolved from PATH, and does not vendor a
copy (find over the whole installed tree returns no podman, crun, or steamvr binary):

• liblepton/liblepton.sh — podman run / start / wait / exec / stop / rm / ps /
  events / inspect
• liblepton/liblepton.sh:356 — env -i PATH="..." $(which podman) exec ... lepton-${LEPTON_CONTEXT}
• liblepton/mounting.sh — builds PODMAN_CMDLINE[@], mounts via podman_mount_entry

Dependency state on the device

dependency	state
podman	MISSING
crun / runc	MISSING
pasta / passt	MISSING
fuse-overlayfs	MISSING
newuidmap / newgidmap	present
/etc/subuid, /etc/subgid for deck	present (100000:65536)
overlay, fuse in /proc/filesystems	present
/proc/sys/user/max_user_namespaces	30436

pasta (not slirp4netns) is the one it wants — liblepton/networking.sh builds
-I eth0, --ipv4-only, --no-ndp, --no-dhcp, --no-dhcpv6, which are passt/pasta flags.

A repo-wide grep for podman|pasta|passt|crun|fuse-overlayfs matches nothing: we have never
shipped a container runtime, so this is a gap, not a regression.

The rootless prerequisites are otherwise already in place (subuid/subgid, userns, overlay, fuse),
so this may be purely a matter of adding packages.

What already works — do not chase it

The Steam-side integration is fine. compat_log.txt shows the tool registered and selected:

Registering tool lepton, AppID 3029110
Skip mapping AppID 4545850 to tool "native" with priority 75:
  mapping to tool "lepton" with priority 250 already exists.
Posting tool registration callback 3029110 lepton
Loaded manifest for tool 3029110.

This is not the account-gating situation seen with the FEX compat tool (3127680) — selection
works, the runtime is simply absent. Note 3127680 is registered but not installed; 3029110 is
installed.

Also ruled out: binder is irrelevant here. Lepton has zero binder/ashmem references and its
README states it "runs as a non-root user without requiring any rootful setup or helpers" — it
avoids binder by design. Our CONFIG_ANDROID_BINDER_IPC is not set does not affect this.

Secondary defect: Lepton is silent on our image

Worth fixing alongside, because it is why this left no diagnostic trail:

liblepton/utils.sh:85:   STEAMVR_LOGS_DIR="$(steamvr logpath || true)"
liblepton/utils.sh:103:  print "${1:-/${STEAMVR_LOGS_DIR}/lepton-${CONTEXT}.log}"

We ship no steamvr binary, and || true swallows that, so STEAMVR_LOGS_DIR is empty and the
log path collapses to /lepton-.log — on a root mounted ro,relatime (btrfs). log_to_file's
tee then fails into its own || true and all output is discarded.

The same empty variable also disables:

• logcat_debug_dir() (liblepton/performance_debugging.sh:3)
• the EARLY_SHUTDOWN automatic logcat dump (lepton:205), which fires on any exit under 30
  seconds and would have caught this run

Pointing STEAMVR_LOGS_DIR at a writable path before launch should restore diagnostics.

Scope decision needed before this is worked

Adding podman + crun + pasta + fuse-overlayfs is a non-trivial addition to an image we have been
actively trimming, in exchange for Android-game support that is not currently a committed feature.
That is a product call, not a bug fix — hence P3. Re-label if Android titles become in-scope.

Closing criteria (hw-gate)

Cannot be closed on a build-time check. Closing requires an Android title actually launching and
rendering through Lepton on real hardware, plus a Lepton log written to a real path.

[loki666]

Consolidated state

Single rolling comment; supersedes the earlier ~20-comment thread. Everything below is measured on hardware unless marked otherwise.

Lepton (Steam app 3029110) is a Waydroid fork — its image ships system/etc/init/init.waydroid.rc. Source reads are against the installed tree at steamapps/common/Lepton/{lepton,liblepton/*.sh}.

Status (2026-08-24): blocker 1 (adb) FIXED. Blocker 2 (no Vulkan/GLES driver) FIXED — the guest now renders on our Turnip. SurfaceFlinger boots, APKs install, the app process starts. Remaining: the app starts and then exits, reason unmeasured.

---

1. What the image ships

podman + crun + passt + fuse-overlayfs (crun→criu, containers-common→netavark are the only providers in the snapshot); /usr/share/guestos/android (must EXIST — mounting.sh pushdes it under set -e; now populated — see §6); a /usr/bin/steamvr shim answering logpath/path/configpath; which (liblepton.sh:356); and adb.

Kernel: CONFIG_ANDROID_BINDER_IPC=y, CONFIG_ANDROID_BINDERFS=y, CONFIG_NETFILTER_NETLINK_LOG=y (to select the promptless CONFIG_NETFILTER_NETLINK), CONFIG_XFRM_USER=y.

2. netd — fixed

netd died on netlink socket for family 12: Protocol not supported (21 restarts/boot) → never registered → system_server blocked in NetdService.get → watchdog kill → zygote. Family 12 is NETLINK_NETFILTER.

	before	after
init.svc.netd	dies	running
restarts/boot	21	1
watchdog kills system_server	yes	0
guest eth0	none	10.0.4.42

CONFIG_DUMMY rejected deliberately: a built-in dummy0 lands in the initial netns, the guest is in pasta's.

3. Blocker 1 (adb) — FIXED

Cause. Lepton prints Boot complete! while the guest is still booting, then fires one unchecked adb connect (liblepton.sh:445) and installs against a device that never registered.

Boot complete! is not evidence of boot: liblepton.sh:225-250 runs a waiter for /data/lepton-onboot alongside a second waiter that podman waits and then pkills the first when the container exits, and prints the message unconditionally after either returns. sys.boot_completed was empty in every getprop sample of every run that printed it.

Measured: container up T+0 · host LISTEN 5555 T+2 · init.svc.adbd=running T+4 · teardown T+5. The port was open and unused.

Fix — shipped, HW-validated. A wrapper at the path liblepton.sh:17 hardcodes, which is the dest of our own packages/adb pin. liblepton.sh cannot be patched (Steam app, overwritten on update); that path being ours is the whole leverage. The pin moves to platform-tools-upstream and fs-overlay supplies platform-tools/adb — the split matters, because fs-overlay lands during rootfs assembly after prebuilts extract into the base, so a same-path overlay would silently replace the real binary. Upstream is relocatable (verified by running it from a copied tree). Only connect is retried, bounded (default 45s); every other verb is exec'd through untouched.

Result on a fresh SD card, no tmpfs: Performing Streamed Install / Success, after five launches that never got past device 'localhost:5555' not found.

4. SD I/O — a real aggravator

/home is /dev/mmcblk0p8 (ext4 on SD); rootless podman drives the :O overlay mounts through fuse-overlayfs, so every copy-up is a userspace round-trip on slow flash.

	SD (4 runs)	tmpfs upperdir
guest listener on 5555	never	guest_tcp6=1
host LISTEN 5555	never	YES
init.svc.adbd	empty	running

Mount at compatdata/<appid>, not .../baked — Lepton rms baked itself on early exit and a mount point cannot be removed.

5. Blocker: SQLite on the baked /data — a deterministic two-state cycle

Once installs succeeded this unmasked immediately:

FATAL EXCEPTION IN SYSTEM PROCESS
Caused by: SQLiteException: not an error (code 0 SQLITE_OK): Could not open the database in read/write mode.
  at LineageSettingsProvider.establishDbTracking(LineageSettingsProvider.java:767)
  at ActivityManagerService.installSystemProviders(:8015) <- SystemServer.startOtherServices(:1135)
E Zygote: Zygote failed to write to system_server FD: Connection refused

It is not intermittent. It is a cycle:

1. install run → Success, 684M baked, container lives >30s so not "early", baked data survives
2. next run takes the warm path → SQLite kills system_server in ~15s → container dies early → Clearing baked app data due to early exit wipes the bake
3. next run must reinstall → succeeds → repeat

EARLY_SHUTDOWN keys off the container living <30s, not the app — which is why the fast crash is what resets the cycle. This exactly reproduces the previously-reported "alternation".

Why the container is always SIGKILLed (Exited 137): it is deliberate, not a timeout. kill_container() runs podman stop -t 0 — zero grace period. Android is never allowed to shut down, so SQLite never checkpoints and a dirty -wal/-shm is left in the upperdir on every run — confirmed on disk: data_overlay/system/locksettings.db{,-wal,-shm}.

Leading hypothesis: a cleared bake reads the DB from the clean lower layer with no WAL to recover, and works; the next run must recover a dirty WAL through fuse-overlayfs, which requires writing -shm (SQLite maps it shared — the classic weak spot on FUSE). Ordinary SQLite crash recovery would cope, so the filesystem is the suspect, not the DB.

Secondary observation: copy-up changes the mode — lower -rwxr-xr-x 1000:1000, upper -rw-rw---- deck:deck. The 0755 in the lower is probably wrong (Steam depots don't carry unix modes, which is why Lepton ships sysbake.xattrs + setfattr --restore). Host-side ownership means little under rootless mapping; check from inside the guest.

Refuted: user.containers.override_stat — zero occurrences in sysbake.xattrs.

6. Blocker 2 — FIXED. The guest has a driver, and zink was the last obstacle

packages/mesa-android builds the Android/bionic driver from the same source pin and patch list
as packages/mesa (host) and packages/mesa-x86 (FEX guest), cross-compiled with the NDK, staged
into the slot. Landed on feat/lepton-podman.

Measured on a Pocket S2 (Adreno 750):

D libEGL : loaded /vendor/lib64/egl/libEGL_mesa.so   (+ GLESv1_CM, GLESv2)
D vndksupport: Loading vulkan.freedreno.so from current namespace
I MESA   : Using gralloc0 CrOS API
I RenderEngine: vendor: freedreno | renderer: FD750 | OpenGL ES 3.2 Mesa 26.2.1
I SurfaceFlinger: Boot is finished (2207 ms)
I DisplayManagerService: Display device added: "Built-in Screen", 2560 x 1440, 60fps
-> adb connects (27 attempts, 26s) -> Performing Streamed Install -> Success
-> Android boots to launcher -> am_proc_start: com.DefaultCompany.ArmiesOfGielinor

The earlier vulkan.waydroid.so not found is gone: with the driver present, ro.hardware.vulkan=freedreno resolves to the file we ship.

ZINK WAS THE LAST OBSTACLE, and it is a Lepton default we have to override.
properties.sh sets mesa.loader.driver.override=zink on the mesa path. Our build cannot create a
DRI2 screen under that override, and the failure is fatal to the whole guest rather than only to
rendering:

W MESA-EGL: DRI2: failed to create screen / Failed to open any DRM device
W libEGL  : eglInitialize(...) failed (EGL_NOT_INITIALIZED)
F RenderEngine: eglQueryStringImplementationANDROID(EGL_VERSION) failed
-> SurfaceFlinger never registers -> ServiceManager blocks -> libc++abi: terminating (pid 1)
-> container dead ~4s in -> adbd never opens its listener -> install fails "adb: device offline"

That last chain is why this looked like an adb regression for hours. Fix: the slot now also ships
vendor/etc/init/novadeck-gfx.rc, which setprops the override to msm. Lepton is a Steam app and
cannot be patched; the override is not a ro. property, so a later setprop wins.

Failed to open any DRM device IS A MISLEADING SUMMARY. Per platform_android.c, it is what
droid_open_device() prints after every candidate failed; a genuine open failure logs
droid_open_device() Failed to open DRM device /dev/dri/renderD128, which never appeared. The node
opened fine every time — dri2_create_screen() was failing. Read the source before chasing device
access.

What the slot ships: libEGL_mesa.so, libGLESv1_CM_mesa.so, libGLESv2_mesa.so,
vulkan.freedreno.so, libgallium_dri.so, libgbm_mesa.so, dri_gbm.so, plus the .rc above.
It does not owe gralloc (Lepton ships gralloc.minigbm_msm.so and the mapper), and — correcting
an earlier revision of this comment — it does not owe restore_vulkan_layers.sh either: Lepton's
own rootfs ships and runs it, which is where the benign find: '/vendor/enabled_vulkan_layers/'
comes from.

Build notes worth keeping: target API 30 (measured: guest is SDK 30 / Android 11 — not 33);
gallium drivers freedreno,zink; libdrm linked statically, because Android has no platform
libdrm and shipping our own libdrm.so would shadow the guest's own /vendor/lib64/libdrm.so
that its gralloc links against.

7. Not the cause — do not re-investigate

• Graphics as the cause of the earlier container deaths. SwiftShader loaded, eglInitialize/eglCreateContext/eglMakeCurrent all EGL_SUCCESS, surfaceflinger ran. Waiting for service 'package_native'/'statscompanion' is normal early boot.
• --no-splice and the whole -t <range> matrix. dev and Steam pasta argv are character-identical apart from the netns UUID. -t auto comes from Lepton (networking.sh:85-87).
• Property ordering. setup_podman_network precedes setup_props; service.adb.tcp.port measures 5555 on the Steam path.
• ro.lepton.devmode. /etc/steamos-devkit-enabled absent → false on both paths; gates only debuginfod/perfetto.
• sysbake asymmetry. is_sysbake is CI-only; dev and steamlaunch both mount the same pre-baked /data lower.
• adbd not starting / auth / IPv6-only listener / port offset.
• Device-node permissions (2026-08-24). /dev/dri/renderD128, card0 and /dev/kgsl-3d0 are all present in the guest and bind-mounted nosuid but not nodev; they show as overflowuid because host root:render/video do not map through the rootless userns. renderD128 is already 0666, and chmod 0666 /dev/dri/card0 on the host changed nothing. (An open test returning rc=0 for both was run as userns root — CAP_DAC_OVERRIDE — and proves nothing; the chmod is the test that counts.)
• sysfs enumeration (2026-08-24). /sys is mounted, /sys/dev/char/226:{0,128} resolve to the real platform paths, /sys/class/drm/renderD128/device/ lists identically to the host. drmGetDevices2() has what it needs.
• A gralloc-vs-DRM device filter. droid_filter_device() only runs when the property drm.gpu.vendor_name is set, and compares against drmGetVersion()->name. Gralloc is not involved — refuted by source, not by measurement.
• Missing __driDriverGetExtensions_* exports. Zero in our megadriver, but mesa 26 replaced that lookup with a directly-exported DRI API, and libEGL comes from the same build. Not the bug.

podman_forward_ports() forwards nothing — it computes 5555 + OFFSET and exports LEPTON_ADB_PORT. pasta -t auto mirroring a guest listener is the only forwarding mechanism.

8. Method — how to see anything at all on the Steam path

• podman exec <ctr> logcat -b all IS THE WAY IN, and needs no adb. "The Android side is dark" was only ever true of adb. podman logs is empty (Android writes to logd, not container stdout) — that is not evidence of a silent guest. Poll podman inspect -f {{.State.Running}} every 0.1s and attach a streaming logcat redirected to a host file; logcat -d snapshot loops with sleeps lose the race when the container lives ~4s.
• Lepton's own dump_logcat cannot help an early death. It uses podman_attach /bin/logcat (not adb) but is gated if is_context_live and runs at teardown, so a <30s death prints "Dumping logcat to …" and writes nothing.
• LEPTON_FORCE_SOFTWARE must be exactly true — [[ … == "true" ]]. =1 is a no-op and will silently void a control run. By contrast LEPTON_DUMP_LOGCAT is -n (any value) and LEPTON_DEBUG is != "false". There is no LEPTON_LOG.
• Force-software changes MOUNTS, not just props (mounting.sh:429,550): it mounts no GPU nodes at all and disables minigbm + qti_display. So "SwiftShader works" never exonerates the mesa path.
• LEPTON_DUMP_LOGCAT=1 in the Steam launch options (LEPTON_FORCE_SOFTWARE=true LEPTON_DUMP_LOGCAT=1 %command%). Lepton exports it only on the install-failure path (liblepton.sh:467), so a successful install that then exits dumps nothing and cleanup_container removes the directory. Streamed logcat fails at container start (logd not up) and is never retried.
• Clear the baked dir first or the run takes the warm path and dies in SQLite before reaching the app.
• Use lepton start dev for guest-side work; never race a Steam launch — the adb server is shared per-user, so a probe's adb kill-server destroys the launch it is measuring.
• su - deck -c has no user session: export XDG_RUNTIME_DIR=/run/user/1000 and DBUS_SESSION_BUS_ADDRESS, or lepton dies at liblepton.sh:122 before calling podman.
• Read /proc/PID/net/tcp6, not just tcp — adbd's listener is the IPv6 0A row. An empty PID collapses /proc//net/* to the host table and silently lies.
• Keep podman out of a hot sampling loop (~2s per exec); never 2>/dev/null a probe's own command; a negative needs a positive control.

[loki666]

2026-08-24 — shipped msm fix HW-validated; "app starts and exits" is a Lepton TIMEOUT, not a crash

Card 20260824T062609Z / a775a13, Pocket S2, seven container runs of Age of Gods (Steam appid
4545850 — note its package id is com.DefaultCompany.ArmiesOfGielinor, a stale Unity default, so
this is the SAME title as the earlier trail, not a second sample). Unity 6000.3.1f1, IL2CPP, arm64.

1. The baked msm override works, unattended — CLOSED

First card carrying novadeck-gfx.rc from fs-overlay rather than a hand-placed copy:

RenderEngine GLES Backend
vendor    : freedreno
renderer  : FD750
version   : OpenGL ES 3.2 Mesa 26.2.1
shader cache generated - 48 shaders in 97.6ms

zink: zero occurrences, every run. Graphics are done.

2. "App starts and immediately exits" is lepton.active_app_launch_timeout=60

liblepton/properties.sh:130. Measured container-start → app-start: 52s (run 1) and 57s
(run 5), with teardown within ~3s of app start both times. No tombstone, no ANR, no fatal signal,
no libc++abi — because the app never crashed. Lepton ended it, then reported a clean Exited!.

Why the app is that late: adb needs ~26s to connect (the known adbd listener race), and install
plus boot consume the rest of the 60s budget.

The property is NOT ro., so the slot .rc can raise it — same lever as the mesa override.
That is the next thing to try.

3. Cold/warm alternation, and the warm failure is NOT the databases

	Cold (empty upper)	Warm (populated upper)
Runs	4	3
Result	boots, renders, app starts	system_server dies in ~1s

Warm always dies on SQLiteException: not an error (code 0 SQLITE_OK): Could not open the database in read/write mode — run 2 on accounts at USER_UNLOCKED, runs 4/6 on LineageSettingsProvider
in startOtherServices.

Tested and refuted: archived and deleted all 21 *.db* files from baked/data_overlay, then
launched warm. Identical fatal, same database, same phase. The file contents are innocent —
they also pass integrity_check and open read/write fine from the host. The fault is in the :O
fuse-overlayfs /data mount or its userns uid mapping, not in any data.

Two copy-up artifacts SQLite would never produce, noted but not the cause of the observed fatals:
locksettings.db-wal/-shm present as char-0,0 overlay whiteouts, and the media module's
WAL files carrying mode -rwxr-xr-x.

4. Instrumentation notes

• Lepton already writes a full host-side logcat to ~/.local/share/lepton/logs/lepton-steamlaunch-<appid>.log
  (appended per launch, ANSI-coloured). The old note that lepton-logcats/ is never populated is
  stale. It is not reliable, though — one run produced logcat: Logcat read failure and no body.
• Lepton deletes compatdata/<appid>/baked/ wholesale on an early exit, taking data_overlay
  with it. A guest-side capture under /data is destroyed before it can be read. external/
  (= /data/media/0) and internal/ survive.
• No in-guest capture can record the final moments. logcat -f strands its last stdio block; a
  line-durable shell loop cannot keep pace with boot spew (5043 lines covering 2s of a 90s run).
  The container is SIGKILLed, so the deciding lines are still in logcat's pipe either way.

Capture landed in 50cb488, dev-cards-only.