[flaky] "linkmgrd" crash observed during restart

[vkjammala-arista]

In some of the sonic-mgmt tests where linkmgrd process restart is being performed, we have observed that linkmgrd crashed due to SIGABRT or SIGSEGV during malloc.

We have seen two different backtrace path for the crash (but both of them are during malloc, so most likely related to memory corruption), but couldn't able to reproduce the issue.

Logs snippet for one of the failure:

138949 2025 Jul 18 07:10:13.593094 gd272 INFO mux#supervisord: linkmgrd malloc(): unaligned tcache chunk detected
138950 2025 Jul 18 07:10:13.605806 gd272 INFO swss#supervisord: orchagent
138951 2025 Jul 18 07:10:14.896500 gd272 INFO mux#supervisord 2025-07-18 07:10:14,896 WARN exited: linkmgrd (terminated by SIGABRT (core dumped); not expected)

GDB backtrace:

Program terminated with signal SIGABRT, Aborted.
#0  0x00007f443ca8eeec in ?? () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f44396cb6c0 (LWP 789))]
(gdb) bt
#0  0x00007f443ca8eeec in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f443ca3ffb2 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f443ca2a472 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f443ca8342f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f443ca9886a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f443ca9cc1c in malloc () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007f443ccae58c in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x0000555aab24d863 in link_prober::LinkProberBase::LinkProberBase (this=<optimized out>, muxPortConfig=..., ioService=..., linkProberStateMachinePtr=<optimized out>, this=<optimized out>, muxPortConfig=...,
    ioService=..., linkProberStateMachinePtr=<optimized out>) at src/link_prober/LinkProberBase.cpp:50
#8  0x0000555aab2834bb in link_prober::LinkProberSw::LinkProberSw (linkProberStateMachinePtr=0x7f442c075580, ioService=..., muxPortConfig=..., this=0x7f4424006dc0) at src/link_prober/LinkProberSw.cpp:50
#9  std::_Construct<link_prober::LinkProberSw, common::MuxPortConfig&, boost::asio::io_context&, link_prober::LinkProberStateMachineBase*> (__p=0x7f4424006dc0) at /usr/include/c++/12/bits/stl_construct.h:119
#10 std::allocator_traits<std::allocator<void> >::construct<link_prober::LinkProberSw, common::MuxPortConfig&, boost::asio::io_context&, link_prober::LinkProberStateMachineBase*> (__p=0x7f4424006dc0)
    at /usr/include/c++/12/bits/alloc_traits.h:635
#11 std::_Sp_counted_ptr_inplace<link_prober::LinkProberSw, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<common::MuxPortConfig&, boost::asio::io_context&, link_prober::LinkProberStateM
achineBase*> (__a=..., this=0x7f4424006db0) at /usr/include/c++/12/bits/shared_ptr_base.h:604
#12 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<link_prober::LinkProberSw, std::allocator<void>, common::MuxPortConfig&, boost::asio::io_context&, link_prober::LinkProberStateMachineBase*> (
    __a=..., __p=<optimized out>, this=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:971
#13 std::__shared_ptr<link_prober::LinkProberSw, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<void>, common::MuxPortConfig&, boost::asio::io_context&, link_prober::LinkProberStateMachineBase*> (
    __tag=..., this=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1712
#14 std::shared_ptr<link_prober::LinkProberSw>::shared_ptr<std::allocator<void>, common::MuxPortConfig&, boost::asio::io_context&, link_prober::LinkProberStateMachineBase*> (__tag=..., this=<optimized out>)
    at /usr/include/c++/12/bits/shared_ptr.h:464
#15 std::make_shared<link_prober::LinkProberSw, common::MuxPortConfig&, boost::asio::io_context&, link_prober::LinkProberStateMachineBase*> () at /usr/include/c++/12/bits/shared_ptr.h:1010
#16 link_manager::ActiveStandbyStateMachine::handleSwssBladeIpv4AddressUpdate (this=0x7f442c0747a0, address=...) at src/link_manager/LinkManagerStateMachineActiveStandby.cpp:351
#17 0x0000555aab2d23bb in boost::_mfi::mf1<void, link_manager::LinkManagerStateMachineBase, boost::asio::ip::address>::operator() (this=0x7f44396ca960, a1=..., p=0x7f442c0747a0)
    at /usr/include/boost/bind/mem_fn_template.hpp:165
#18 boost::_bi::list2<boost::_bi::value<link_manager::LinkManagerStateMachineBase*>, boost::_bi::value<boost::asio::ip::address> >::operator()<boost::_mfi::mf1<void, link_manager::LinkManagerStateMachineBase, boost::asio::ip::address>, boost::_bi::list0> (a=<synthetic pointer>..., f=..., this=0x7f44396ca970) at /usr/include/boost/bind/bind.hpp:319
#19 boost::_bi::bind_t<void, boost::_mfi::mf1<void, link_manager::LinkManagerStateMachineBase, boost::asio::ip::address>, boost::_bi::list2<boost::_bi::value<link_manager::LinkManagerStateMachineBase*>, boost::_bi::value<boost::asio::ip::address> > >::operator() (this=0x7f44396ca960) at /usr/include/boost/bind/bind.hpp:1294
#20 boost::asio::asio_handler_invoke<boost::_bi::bind_t<void, boost::_mfi::mf1<void, link_manager::LinkManagerStateMachineBase, boost::asio::ip::address>, boost::_bi::list2<boost::_bi::value<link_manager::LinkManagerStateMachineBase*>, boost::_bi::value<boost::asio::ip::address> > > > (function=...) at /usr/include/boost/asio/handler_invoke_hook.hpp:88
#21 boost_asio_handler_invoke_helpers::invoke<boost::_bi::bind_t<void, boost::_mfi::mf1<void, link_manager::LinkManagerStateMachineBase, boost::asio::ip::address>, boost::_bi::list2<boost::_bi::value<link_manager::LinkManagerStateMachineBase*>, boost::_bi::value<boost::asio::ip::address> > >, boost::_bi::bind_t<void, boost::_mfi::mf1<void, link_manager::LinkManagerStateMachineBase, boost::asio::ip::address>, boost::_bi::list
2<boost::_bi::value<link_manager::LinkManagerStateMachineBase*>, boost::_bi::value<boost::asio::ip::address> > > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:54
#22 boost::asio::detail::handler_work<boost::_bi::bind_t<void, boost::_mfi::mf1<void, link_manager::LinkManagerStateMachineBase, boost::asio::ip::address>, boost::_bi::list2<boost::_bi::value<link_manager::LinkMana
gerStateMachineBase*>, boost::_bi::value<boost::asio::ip::address> > >, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u>, void>::complete<boost::_bi::bind_t<void, boost::_mfi::mf1<void, link_m
anager::LinkManagerStateMachineBase, boost::asio::ip::address>, boost::_bi::list2<boost::_bi::value<link_manager::LinkManagerStateMachineBase*>, boost::_bi::value<boost::asio::ip::address> > > > (handler=...,
    function=..., this=<synthetic pointer>) at /usr/include/boost/asio/detail/handler_work.hpp:425
#23 boost::asio::detail::completion_handler<boost::_bi::bind_t<void, boost::_mfi::mf1<void, link_manager::LinkManagerStateMachineBase, boost::asio::ip::address>, boost::_bi::list2<boost::_bi::value<link_manager::Li
nkManagerStateMachineBase*>, boost::_bi::value<boost::asio::ip::address> > >, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> >::do_complete (owner=0x555ae9483cf0, base=0x7f442c068d40)
    at /usr/include/boost/asio/detail/completion_handler.hpp:74
#24 0x0000555aab245be5 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=<optimized out>, ec=..., owner=<optimized out>, this=<optimized out>, this=<optimized out>, owner=<optimized out>,
    ec=..., bytes_transferred=<optimized out>) at /usr/include/boost/asio/detail/scheduler_operation.hpp:40
#25 boost::asio::detail::strand_service::do_complete (owner=0x555ae9483cf0, base=0x7f442c073e60, ec=...) at /usr/include/boost/asio/detail/impl/strand_service.ipp:169
#26 0x0000555aab2e6db6 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=<optimized out>, ec=..., owner=<optimized out>, this=<optimized out>, this=<optimized out>, owner=<optimized out>,
    ec=..., bytes_transferred=<optimized out>) at /usr/include/boost/asio/detail/scheduler_operation.hpp:40
#27 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x555ae9483cf0) at /usr/include/boost/asio/detail/impl/scheduler.ipp:481
#28 boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] (this=0x555ae9483cf0, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:204
#29 0x0000555aab2cb3dc in boost::asio::io_context::run (this=<optimized out>) at /usr/include/boost/asio/impl/io_context.ipp:63
#30 0x00007f443d0ecb17 in ?? () from /lib/x86_64-linux-gnu/libboost_thread.so.1.74.0
#31 0x00007f443ca8d1f5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#32 0x00007f443cb0cb00 in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)```