Summary
AducIotAgent crashes with SIGSEGV on every failed startup: whenever
HealthCheck() fails, the agent dies in the shutdown path instead of exiting with
a non-zero code. Seen on aarch64, 1.4.0; the same code is on develop.
Log right before the crash:
[E] Invalid connection info. [HealthCheck:630]
[E] Agent health check failed. [main:1161]
→ SIGSEGV, si_code=1 (SEGV_MAPERR), fault address 0xd0
Because the unit restarts the agent after a failure, this repeats until the
systemd start limit stops the service — on our devices 10 cores in 60 s while the
identity service could not hand out connection info.
Root cause
uninit_api_svc() (src/agent/api/src/apisvc.c) joins g_api_svc_thread
unconditionally:
atomic_store(&g_api_svc_thread_running, false);
int res = pthread_join(g_api_svc_thread, (void**)&threadRet);
g_api_svc_thread is pthread_t g_api_svc_thread = { 0 }; and is only ever set by
init_api_svc(). On the health-check failure path init_api_svc() never runs:
main() does goto done before StartupAgent(), and done: calls
ShutdownAgent(), which calls uninit_api_svc(). So pthread_join() gets a NULL
thread descriptor.
glibc's join implementation dereferences the descriptor to validate it. On aarch64
that faults at NULL + 0xd0; x86-64 glibc happens to check the handle for NULL and
returns ESRCH, which is why the crash only shows on arm.
Evidence from a device core (two cores, identical):
- faulting instruction
ldr w0, [x0, #208] with x0 = 0, inside the join
implementation in libc
- return address points right after
bl pthread_join@plt in uninit_api_svc()
The analogous timer thread teardown already guards against this
(_stop_thread() in src/utils/timer_utils/src/timer.c only joins when
threadCreated); the API service thread does not.
Reproduces when
Any shutdown that was not preceded by a successful StartupAgent(), i.e. every
HealthCheck() failure (identity service unavailable, unprovisioned device, bad
connection info). Deterministic on arm.
Fix
Take the running flag with atomic_exchange() in uninit_api_svc() and skip the
teardown when it was already false; this also makes a repeat uninit a no-op. PR
attached.
Summary
AducIotAgentcrashes withSIGSEGVon every failed startup: wheneverHealthCheck()fails, the agent dies in the shutdown path instead of exiting witha non-zero code. Seen on aarch64, 1.4.0; the same code is on
develop.Log right before the crash:
Because the unit restarts the agent after a failure, this repeats until the
systemd start limit stops the service — on our devices 10 cores in 60 s while the
identity service could not hand out connection info.
Root cause
uninit_api_svc()(src/agent/api/src/apisvc.c) joinsg_api_svc_threadunconditionally:
g_api_svc_threadispthread_t g_api_svc_thread = { 0 };and is only ever set byinit_api_svc(). On the health-check failure pathinit_api_svc()never runs:main()doesgoto donebeforeStartupAgent(), anddone:callsShutdownAgent(), which callsuninit_api_svc(). Sopthread_join()gets a NULLthread descriptor.
glibc's join implementation dereferences the descriptor to validate it. On aarch64
that faults at
NULL + 0xd0; x86-64 glibc happens to check the handle for NULL andreturns
ESRCH, which is why the crash only shows on arm.Evidence from a device core (two cores, identical):
ldr w0, [x0, #208]withx0 = 0, inside the joinimplementation in libc
bl pthread_join@pltinuninit_api_svc()The analogous timer thread teardown already guards against this
(
_stop_thread()insrc/utils/timer_utils/src/timer.conly joins whenthreadCreated); the API service thread does not.Reproduces when
Any shutdown that was not preceded by a successful
StartupAgent(), i.e. everyHealthCheck()failure (identity service unavailable, unprovisioned device, badconnection info). Deterministic on arm.
Fix
Take the running flag with
atomic_exchange()inuninit_api_svc()and skip theteardown when it was already false; this also makes a repeat uninit a no-op. PR
attached.