`bc_stark_sdk` 2.0.2: `stark_node` leaks memory to OOM-kill within ~10-30s of dual-hand operation (reproducible 7/7)

[AnnaKn0rr]

Environment

• Robot: Unitree G1 (dual Revo2 Touch, capacitive, connected via dual RS-485 / dual USB-serial)
• SDK: bc_stark_sdk / libbc_stark_sdk.so v2.0.2 (linux ARM64)
• ROS2: Foxy, ros2_stark_controller package (based on this repo's ros2_stark_ws)
• Left hand: SN BCXTL2361J250002F, FW 1.0.16.U
• Right hand: SN BCXTR2361J250001A, FW 1.0.16.U
• Protocol: Modbus RTU, baud 460800

Symptom

stark_node process RSS grows to ~2GB+ within roughly 10-30 seconds of starting, and is killed (either by the kernel OOM killer or a cgroup memory cap we added as a mitigation). Reproduced on 7 consecutive runs under systemd with Restart=always, every single one died the same way.

Kernel OOM killer log from one occurrence (no memory cap in place at the time):

dockerd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-500
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/stark_node.service,task=stark_node,pid=1854,uid=1000
Out of memory: Killed process 1854 (stark_node) total-vm:31320256kB, anon-rss:13234216kB, file-rss:0kB, shmem-rss:16kB, UID:1000 pgtables:41340kB oom_score_adj:0
oom_reaper: reaped process 1854 (stark_node), now anon-rss:0kB, file-rss:0kB, shmem-rss:24kB

~13.2GB resident in well under a minute of runtime. This is severe enough that it starved unrelated processes on the machine (note dockerd is what invoked the OOM killer here, not our process).

Key evidence this is in the SDK, not our integration code

In one run, bad_alloc was thrown three times before our first log line printed before modbus_open() was ever called, before our ROS2 poll timer (50ms) was even constructed:

[INFO] [stark_node-1]: process started with pid [3158]
[stark_node-1] bad_alloc caught: std::bad_alloc
[stark_node-1] bad_alloc caught: std::bad_alloc
[stark_node-1] bad_alloc caught: std::bad_alloc
... (dies ~12s later, exit code -9)

Since nothing in our code has executed yet at that point, this rules out a leak in the ROS2 node integration and points at SDK-internal state (possibly a background thread spawned on library load or modbus_open).

We also see it happen after a fully clean initialization (no read timeouts, both hands report device info successfully, touch sensors enabled on both):

[LEFT] SN=BCXTL2361J250002F  FW=1.0.16.U  HW=11
[LEFT] Touch sensors enabled — waiting 1 s…
touch_sensor_setup: slave_id=126, bits=0b11111
touch_sensor_enable: slave_id=126, address=0x0FA0, bits=0b11111, values=[1, 1, 1, 1, 1]
[RIGHT] SN=BCXTR2361J250001A  FW=1.0.16.U  HW=11
[RIGHT] Touch sensors enabled — waiting 1 s…
touch_sensor_setup: slave_id=127, bits=0b11111
touch_sensor_enable: slave_id=127, address=0x0FA0, bits=0b11111, values=[1, 1, 1, 1, 1]
... (dies ~11s later, exit code -9)

So the leak isn't tied to the read-timeout/retry path either, it also happens in steady-state polling after a clean startup.

Our integration code (for reference, every returned pointer is freed)

void publish_motor(DeviceHandler* handle, uint8_t slave_id,
                   rclcpp::Publisher<ros2_stark_interfaces::msg::MotorStatus>::SharedPtr& pub,
                   const char* label) {
    auto* data = stark_get_motor_status(handle, slave_id);
    if (!data) { /* ... */ return; }
    auto msg = ros2_stark_interfaces::msg::MotorStatus();
    std::copy(data->positions, data->positions + 6, msg.positions.begin());
    std::copy(data->speeds,    data->speeds    + 6, msg.speeds.begin());
    std::copy(data->currents,  data->currents  + 6, msg.currents.begin());
    std::copy(data->states,    data->states    + 6, msg.states.begin());
    pub->publish(msg);
    free_motor_status_data(data);
}

void publish_touch(DeviceHandler* handle, uint8_t slave_id,
                   rclcpp::Publisher<ros2_stark_interfaces::msg::TouchStatus>::SharedPtr& pub,
                   const char* label) {
    auto* data = stark_get_touch_status(handle, slave_id);
    if (!data) { /* ... */ return; }
    auto msg = ros2_stark_interfaces::msg::TouchStatus();
    for (int i = 0; i < 5; ++i) { /* copy fields */ }
    pub->publish(msg);
    free_touch_finger_data(data);
}

Both are called from a 50ms wall_timer, alongside stark_get_voltage on a 30s timer (returns a scalar, nothing to free).

Repro steps

1. Dual-hand Modbus RTU setup, baud 460800, one hand per USB-serial adapter.
2. Call stark_set_hardware_type, stark_get_device_info, stark_enable_touch_sensor(handle, slave_id, 0x1F) on each hand at startup.
3. Poll stark_get_motor_status + stark_get_touch_status every 50ms per hand, freeing each returned pointer after use.
4. Observe process RSS via systemctl status <service> or ps grows unbounded within seconds, process is killed within 10-30s.

Questions

1. Is there a newer SDK build past v2.0.2 (linux ARM64) that addresses this?
2. Does bc_stark_sdk spawn any internal background threads on modbus_open() or library load that could account for allocation happening before the caller's own API calls return?
3. Is there a known workaround (e.g. a required cleanup/flush call we're missing, or a known-bad code path when both hands share one process)?

Happy to provide valgrind/heaptrack output against stark_node if that would help narrow down the allocation site.

[Yongle-Fu]

Thanks for the detailed report. I checked the v2.0.2 C FFI and Modbus paths, but the current evidence does not point to a leak in stark_get_motor_status or stark_get_touch_status themselves.

Both functions return a single Rust Box containing only fixed-size inline arrays, and free_motor_status_data / free_touch_finger_data reconstruct and drop that same Box. Even if none of those objects were freed, polling both hands at 20 Hz would only retain a few KB per second, which cannot explain 13 GB of RSS in under a minute.

The SDK uses one lazily initialized global Tokio runtime. It is created on the first SDK operation, not during library loading, and modbus_open() does not create one runtime or permanent polling thread per hand. stark_enable_touch_sensor() schedules a one-shot setup task, but the two status getters block until their Modbus operation completes and do not leave background tasks behind.

One important detail: the public launch file starts two separate stark_node processes, one under left_hand and one under right_hand. The code in this report appears to be a customized single-process dual-hand version: it uses ros2_stark_interfaces and a 50 ms timer, while the repository version at the time used ros2_stark_msgs, a 100 ms timer, and downloaded SDK v1.1.9. Since the reported process uses v2.0.2, please confirm that the C header and ARM64 shared library came from the same v2.0.2 artifact and that another installed library is not being loaded at runtime.

Could you provide the output of:

pgrep -af stark_node
ros2 node list | grep stark_node
grep libbc_stark_sdk /proc/<stark-node-pid>/maps
ldd /path/to/stark_node | grep bc_stark
sha256sum /actual/path/libbc_stark_sdk.so /actual/path/stark-sdk.h

The three std::bad_alloc messages before the first application log also do not by themselves show that Rust code threw the exception. Rust allocation failure aborts; it is not translated into C++ std::bad_alloc. That message must come from a C++ catch path, although earlier heap corruption could still make a later C++ allocation fail.

The fastest way to separate SDK behavior from the ROS 2 integration would be a minimal C/C++ process with no ROS 2 or DDS: open both ports, enable touch once, poll motor and touch for both hands, immediately call both free functions, and monitor RSS for several minutes. If that process still grows, a heaptrack trace from it would let us identify the actual allocation stack. If it stays flat, the next isolation step is to add ROS message construction and then DDS publishing separately.

Please also attach the exact customized stark_node.cpp and its matching generated message definitions. With those files plus a heap profile, we should be able to narrow this down without guessing.

[lihuawei-topsun]

PR #4 has now moved this repository to SDK v2.0.3, but I would keep this issue open until the reporter reruns the exact failing topology. A version upgrade is not yet evidence that the 13 GB/minute failure is fixed.

The maintainer's no-ROS binary is the right next step. I would make the isolation matrix explicit so one run can identify the layer instead of producing another ambiguous heap profile:

Cell	Process topology	Calls	Rate	Purpose
A	one process, left only	motor status only	10 Hz, then 20 Hz	baseline getter/free slope
B	one process, left only	touch status only	10 Hz, then 20 Hz	isolate touch path
C	one process, both hands	motor + touch, sequential	10 Hz, then 20 Hz	reproduce shared-runtime/dual-handle case
D	two processes, one hand each	same as C	20 Hz	separate per-process state from hardware/load
E	C + ROS message construction, no publish	same	20 Hz	isolate generated message/container allocation
F	E + DDS publish	same	20 Hz	isolate middleware/backpressure/subscriber effects

For every cell, freeze the header/library pair and record:

• SDK version, header SHA-256, loaded .so path/hash from /proc/<pid>/maps, and the exact executable hash;
• hand firmware/SN redacted consistently, port, slave ID and whether touch was enabled;
• RSS/PSS/anonymous memory from /proc/<pid>/smaps_rollup at 1 Hz, API call counts, failures/timeouts and process exit reason;
• a fixed duration (for example five minutes) or an explicit memory ceiling, not “until it looks stable.”

Run the lightweight smaps_rollup trace first. Then run heaptrack only on the smallest cell that still has a positive memory slope; profiling all ROS/DDS layers at once will add noise and may change timing. LD_DEBUG=libs or the /proc mapping is also important here because the repository moved from 1.1.9 to 2.0.3 while the report used a customized 2.0.2 binary.

Interpretation can then be deterministic:

• C grows but A/B/D stay flat → inspect shared SDK/global runtime or dual-handle serialization;
• A or B grows → getter/free or per-device SDK path;
• E grows after C is flat → message construction/copy path;
• only F grows → DDS queue/subscriber/backpressure or allocator retention;
• all cells stay flat on 2.0.3 → bisect the customized node around initialization and timers before closing the issue.

One field-safety addition: a cgroup kill/restart is useful containment, but the service should not restart into an active command state. On memory ceiling, stale state or process loss, enter a documented hand hold/open/disable state, invalidate the previous command generation and require a new operator authorization before resuming.

I work on real-hardware Unitree G1 integration and can help review a sanitized smaps_rollup/heaptrack bundle or the minimal dual-hand harness, but I do not have this exact Revo2 firmware/SDK combination and would not claim the upgrade is validated. Independent integrator; not affiliated with BrainCo or Unitree. Contact: li.huawei@topsunpower.cc.

Structured hardware-validation request: https://github.com/lihuawei-topsun/robotics-field-integration/issues/new?template=hardware-validation.yml. Blank redacted manifest: https://lihuawei-topsun.github.io/robotics-field-integration/assets/unitree-validation-manifest-template.yaml. Do not put hand serials, device paths, customer identifiers or private endpoints in a public issue.

[Yongle-Fu]

Thanks for the additional input.

PR #4 has been merged into main. The repository now uses SDK v2.0.3 and builds against the header and shared library from the same downloaded SDK package. This removes the previous risk of compiling against the vendored v1.1.9 header while loading a v2.x library at runtime.

This upgrade is not yet being claimed as a fix for the reported memory growth.

@AnnaKn0rr, could you please perform a clean rebuild from the current main branch and rerun the same single-process, dual-hand topology?

Before rebuilding, please remove the ROS workspace build, install, and log directories, as well as any previously cached SDK artifacts. For the test result, please include:

• the SDK version;
• the loaded libbc_stark_sdk.so path from /proc/<pid>/maps;
• the SHA-256 hashes of the loaded shared library, SDK header, and executable;
• RSS samples over time, preferably from /proc/<pid>/smaps_rollup;
• whether the failure still occurs with the same 50 ms polling interval;
• the exact customized stark_node.cpp and its build configuration, if they can be shared.

Please keep the existing memory limit or another safe test ceiling in place to avoid a system-wide OOM. A five-minute run is sufficient if memory remains stable.

If the process still shows a positive memory slope on v2.0.3, the next step will be a minimal non-ROS dual-hand reproducer, followed by heaptrack on the smallest configuration that still reproduces the growth.

Please redact serial numbers, device paths, and other deployment-specific identifiers from new attachments.