eCAL SHM subscriber can try to open stale/missing memfile after publisher process restart churn

[thomaskolBG]

Problem Description

With local SHM-only pub/sub, a restarting publisher process can leave subscribers trying to open an eCAL SHM memory file that no longer exists:

shm_open failed to OPEN memory file (memfile::os::AllocFile): /ecal_<id> errno: No such file or directory

In our production system, this appears as subscribers/probes no longer receiving data from an otherwise-running publisher after the worker process re-started, until the affected endpoint/process is recreated. The attached minimal reproducer uses a fixed 3 MiB payload and preallocates publisher SHM to 6 MiB, so this does not require runtime SHM resize or unusually large camera frames.

How to reproduce

Attachment:
ecal_shm_stale_memfile_ticket_repro_3m_20260503T175834Z.tar.gz

Build:

  tar -xzf ecal_shm_stale_memfile_ticket_repro_3m_20260503T175834Z.tar.gz
  cd ecal_shm_stale_memfile_ticket_repro_3m_20260503T175834Z

  cmake -S source/upstream_minimal -B /tmp/ecal_shm_resize_repro_build
  cmake --build /tmp/ecal_shm_resize_repro_build -j

Run:

  python3 source/upstream_minimal/run_ecal_shm_resize_repro.py \
    --endpoint-bin /tmp/ecal_shm_resize_repro_build/ecal_shm_resize_endpoint \
    --output-dir /tmp/ecal_shm_resize_repro_$(date -u +%Y%m%dT%H%M%SZ) \
    --duration-seconds 180 \
    --topics 4 \
    --publisher-run-seconds 0.35 \
    --publisher-gap-seconds 0.0 \
    --probe-interval-seconds 0.3 \
    --probe-run-seconds 0.8 \
    --payload-sizes 3145728 \
    --publish-period-ms 5 \
    --callback-sleep-ms 5 \
    --shm-memfile-min-size-bytes 6291456 \
    --stop-on-failure

Observed locally:

  shm_open failed to OPEN memory file (memfile::os::AllocFile): /ecal_6255555b errno: No such file or directory

This reproduced in about 37 seconds after 392 publisher restarts and 112 probe subscriber starts.

The repro uses rapid publisher restarts to make the race deterministic enough for a small attachment. We do not claim this restart cadence is a normal production workload. However, the same failure signature was observed in a real long-running system with much slower restarts: publisher/worker restarts on the order of minutes, logger restarts on the order of minutes, and a long-lived subscriber stopped receiving from an otherwise-running publisher until the affected process was recreated. The concern is not that eCAL must guarantee lossless delivery across every rapid restart, but that after this race the subscriber can be left trying to observe a stale/missing SHM memfile and the system does not automatically recover.

How did you get eCAL?

Ubuntu PPA (apt-get)

Environment

eCAL package: 6.1.1-1ppa1~jammy
eCAL runtime: v6.1.1 (2026-03-31 15:25:50 +0200)
OS: Ubuntu 22.04.5 LTS
Python: 3.10.12
CMake: 3.22.1
g++: 11.4.0
Transport: local SHM-only pub/sub
Publisher SHM config in repro:
memfile_min_size_bytes=6291456
memfile_buffer_count=1
memfile_reserve_percent=0

eCAL System Information

[eCAL][Config] Yaml configuration loaded from "/etc/ecal/ecal.yaml".
------------------------- SYSTEM ---------------------------------
Version                  : v6.1.1 (2026-03-31 15:25:50 +0200)
Platform                 : linux

------------------------- CONFIGURATION --------------------------
Default INI              : /etc/ecal/ecal.yaml

------------------------- NETWORK --------------------------------
Host name                : p2-dev-1
SHM transport domain     : p2-dev-1
Network mode             : local
Network ttl              : 1
Network sndbuf           : 5 MByte
Network rcvbuf           : 5 MByte
Multicast cfg version    : v2
Multicast group          : 239.0.0.1
Multicast mask           : 255.255.255.240
Multicast ports          : 14002 - 14012
Multicast join all IFs   : off

------------------------- TIME -----------------------------------
Synchronization realtime : ecaltime-localtime
Synchronization replay   :
State                    :  synchronized
Master / Slave           :  Master
Status (Code)            : "everything is fine." (0)

------------------------- SUBSCRIPTION LAYER DEFAULTS ------------
Layer Mode UDP MC        : on
Drop out-of-order msgs   : on