Skip to content

Fix SIGSEGV on failed startup: do not join the api service thread when it was never started - #912

Open
Buell (JanZachmann) wants to merge 1 commit into
Azure:developfrom
JanZachmann:upstream-fix/apisvc-uninit-null-join
Open

Fix SIGSEGV on failed startup: do not join the api service thread when it was never started#912
Buell (JanZachmann) wants to merge 1 commit into
Azure:developfrom
JanZachmann:upstream-fix/apisvc-uninit-null-join

Conversation

@JanZachmann

Copy link
Copy Markdown

Fixes #911

What

uninit_api_svc() joined g_api_svc_thread unconditionally. The handle is
zero-initialized and only set by init_api_svc(), so a shutdown that never started
the service passed a NULL thread descriptor to pthread_join(). On aarch64 glibc
that dereferences it and the agent dies with SIGSEGV (fault address 0xd0, inside
the join implementation); x86 glibc happens to return ESRCH, so the crash only
shows on arm.

The agent hits this on every failed startup: HealthCheck() fails, main() goes to
done, ShutdownAgent() calls uninit_api_svc() although StartupAgent() never
ran. The unit restarts the agent after the failure, so the crash repeats until the
start limit stops the service.

Fix

Take the running flag with atomic_exchange() and skip the teardown when it was
already false. A second uninit_api_svc() is a no-op too, matching the guard the
timer thread already has in _stop_thread() (timer_utils/src/timer.c).

Tests

  • New sections in apisvc_unit_tests.cpp: "uninit without init" and "uninit twice".
    They interpose pthread_join() (real handles forwarded to libc via
    dlsym(RTLD_NEXT, ...)) and assert that no join is attempted on the zero handle.
    A plain return-value check would not catch the bug on x86, where the faulty join
    just returns ESRCH.
  • apisvc_unit_tests: 26/26 assertions pass with the fix; 2 assertions fail without
    it.
  • Full ctest run on develop: the failing-test set is identical before and after
    this change (80 pre-existing failures in my container setup, none in apisvc).

Root-caused from a symbolized device core; details in #911.

uninit_api_svc() joined g_api_svc_thread unconditionally. The handle is
zero-initialized and only set by init_api_svc(), so a shutdown that never
started the service passed a NULL thread descriptor to pthread_join(). On
aarch64 glibc that dereferences it and the process dies with SIGSEGV
(fault address 0xd0, inside the join implementation). x86 glibc happens to
return ESRCH instead of faulting, so the crash only shows on arm.

The agent hits this on every failed startup: HealthCheck() fails, main()
goes to done, ShutdownAgent() calls uninit_api_svc() although
StartupAgent() never ran. The unit restarts the agent after the failure, so
the crash repeats until the systemd start limit stops it, e.g. for as long
as the identity service cannot hand out connection info.

Take the running flag with atomic_exchange and skip the teardown when it
was already false. This also makes a second uninit a no-op, matching the
guard the timer thread already has in AducTimer_Stop().

Add regression coverage: the tests interpose pthread_join and assert that
no join is attempted on the zero handle, before init and after a completed
uninit.

Signed-off-by: Jan Zachmann 50990105+JanZachmann@users.noreply.github.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIGSEGV on every failed startup: uninit_api_svc joins a thread that was never created

1 participant