[BUG]: SIGSEGV in heap_recorder.c (st_object_record_update → ruby_obj_memsize_of → classext_memsize) on Ruby 4.0 with heap size profiling

[navidemad]

Tracer Version(s)

2.35.0

Ruby Version(s)

ruby 4.0.5 [x86_64-linux]

Relevant Library and Version(s)

libdatadog 33.0.0.1.0

Bug Report

Within hours of enabling experimental heap profiling (experimental_heap_enabled = true) on Ruby 4.0.5, our Sidekiq workers started segfaulting with two crash signatures that always appeared together. Roughly 15 crashes of each signature over ~10 hours, across several worker types. Both stopped completely the moment we disabled experimental_heap_enabled.

Signature A — crash inside the tracer:

• SIGSEGV (SEGV_MAPERR)
• frame: st_object_record_update in ext/datadog_profiling_native_extension/heap_recorder.c

Signature B — crash inside Ruby's GC:

• SIGSEGV (SI_KERNEL)
• frame: classext_memsize in Ruby's gc.c

These look like two faces of the same call chain. During a full heap update, heap_recorder_update calls st_foreach(object_records, st_object_record_update, ...). For each live object, when size profiling is on, st_object_record_update calls ruby_obj_memsize_of(ref) → rb_obj_memsize_of(obj). For T_CLASS / T_MODULE / T_ICLASS objects on Ruby 4.0, rb_obj_memsize_of walks the per-namespace class extensions and calls into classext_memsize (new in Ruby 4.0's namespace/classext implementation) — which is where the process dies. Depending on where the bad access lands, the crash is attributed either to the tracer frame (Signature A) or to the Ruby GC frame (Signature B).

ruby_obj_memsize_of (in ruby_helpers.c) already keeps a denylist of rb_obj_memsize_of paths that rb_bug/crash the VM (e.g. T_NODE is explicitly excluded), but it still forwards T_CLASS / T_MODULE / T_ICLASS straight through to rb_obj_memsize_of. On Ruby 4.0 that path is no longer safe.

This only triggers with heap size profiling: st_object_record_update calls ruby_obj_memsize_of only when size_enabled && update_include_old (full update). experimental_heap_size_enabled defaults to true whenever heap profiling is enabled, so anyone turning on heap profiling on Ruby 4.0 is exposed by default.

This appears to be the same Ruby 4.0 incompatibility that #5148 originally guarded against and that #5201 re-enabled behind the experimental flag — the class-object size-accounting path still seems unsafe on Ruby 4.0.

Suggested fix direction: skip T_CLASS / T_MODULE / T_ICLASS in ruby_obj_memsize_of on Ruby 4.0 (return 0, as is already done for unsupported types), or otherwise guard the classext memsize path — analogous to the existing T_NODE exclusion. Alternatively, gate experimental_heap_size_enabled off on Ruby 4.0 until the classext path is safe.

Reproduction Code

No minimal reproduction yet. It triggers under real workload on Sidekiq workers that allocate and retain many Class/Module objects, and reproduces reliably within hours of enabling the flag on Ruby 4.0.5. It disappears entirely once experimental_heap_enabled is set back to false.

Configuration Block

Datadog.configure do |c|
  c.profiling.enabled = true
  c.profiling.advanced.allocation_enabled = true
  c.profiling.advanced.experimental_heap_enabled = true
  # experimental_heap_size_enabled left at its default (true)
end

Error Logs

We have the full native crash dumps for both signatures and will attach them to this issue. The crashing frames are st_object_record_update (heap_recorder.c) for Signature A and classext_memsize (gc.c) for Signature B.

Operating System

Linux (x86_64, container)

[navidemad]

Native crash backtraces for both signatures (crashing thread only; the rest of each dump is ~15 idle threads parked in pthread_cond_wait/epoll_wait/sleep). Both occur while the profiler serializes the heap profile and walks live objects' sizes, and both converge on st_object_record_update → rb_obj_memsize_of → rb_class_classext_foreach → the Ruby 4.0 classext memsize path. Signature A dies one frame higher (iterating the classext table) than Signature B (inside rb_id_table_memsize).

Signature A — crash attributed to heap_recorder.c

SIGSEGV: Process terminated with SI_KERNEL (SIGSEGV)
rb_st_foreach                    (st.c:1661)
rb_class_classext_foreach        (class.c:461)
rb_obj_memsize_of                (gc.c:2320)
st_object_record_update          (datadog-2.35.0/ext/datadog_profiling_native_extension/heap_recorder.c:771)
rb_st_foreach                    (st.c:1661)
heap_recorder_update             (datadog-2.35.0/ext/datadog_profiling_native_extension/heap_recorder.c:587)
heap_recorder_prepare_iteration  (datadog-2.35.0/ext/datadog_profiling_native_extension/heap_recorder.c:605)
_native_serialize                (datadog-2.35.0/ext/datadog_profiling_native_extension/stack_recorder.c:527)
vm_call_cfunc_with_frame_        (vm_insnhelper.c:3901)
vm_sendish                       (vm_insnhelper.c:6134)
vm_exec_core                     (insns.def:904)
rb_vm_exec                       (vm.c:2801)
rb_ec_yield                      (vm_eval.c:1386)
rb_ec_ensure                     (eval.c:1146)
rb_mut_synchronize               (thread_sync.c:679)

Signature B — crash attributed to gc.c classext_memsize

SIGSEGV: Process terminated with SI_KERNEL (SIGSEGV)
rb_id_table_memsize              (id_table.c:133)
classext_memsize                 (gc.c:2271)
rb_class_classext_foreach        (class.c:462)
rb_obj_memsize_of                (gc.c:2320)
st_object_record_update          (datadog-2.35.0/ext/datadog_profiling_native_extension/heap_recorder.c:771)
rb_st_foreach                    (st.c:1661)
heap_recorder_update             (datadog-2.35.0/ext/datadog_profiling_native_extension/heap_recorder.c:587)
heap_recorder_prepare_iteration  (datadog-2.35.0/ext/datadog_profiling_native_extension/heap_recorder.c:605)
_native_serialize                (datadog-2.35.0/ext/datadog_profiling_native_extension/stack_recorder.c:527)
vm_call_cfunc_with_frame_        (vm_insnhelper.c:3901)
vm_sendish                       (vm_insnhelper.c:6134)
vm_exec_core                     (insns.def:904)
rb_vm_exec                       (vm.c:2801)
rb_ec_yield                      (vm_eval.c:1386)
rb_ec_ensure                     (eval.c:1146)
rb_mut_synchronize               (thread_sync.c:679)

Environment: Ruby 4.0.5 (/app/vendor/ruby-4.0.5, x86_64-linux), datadog 2.35.0, libdatadog 33.0.0.1.0, jemalloc, inside a container. The full per-thread Datadog Crashtracker dumps for both signatures are available if useful.

[navidemad]

Root-cause analysis

I dug into the dd-trace-rb side of this. Both signatures are the same operation, with the crash surfacing one frame apart. Here is the mechanism and where I think the fix belongs.

Shared mechanism

The heap recorder tracks committed objects weakly, by object_id only — an object_record stores just a long obj_id (heap_recorder.c), and the GC mark callback marks only pending recordings (heap_recorder_mark_pending_recordings), never committed ones. So a tracked object can be freed at any time, and its liveness is reconstructed at update time via ObjectSpace._id2ref (ruby_ref_from_id in ruby_helpers.c), relying on it to raise RangeError for dead ids.

During profile serialization, heap_recorder_prepare_iteration runs a full update under the GVL:

1. _id2ref(obj_id) resurrects each tracked object;
2. if size profiling is on (size_enabled, i.e. experimental_heap_size_enabled, which defaults to true when heap profiling is on), st_object_record_update calls ruby_obj_memsize_of(ref) → rb_obj_memsize_of(obj);
3. for a T_CLASS / T_MODULE / T_ICLASS, rb_obj_memsize_of walks Ruby 4.0's per-namespace class extensions and dies.

Two things this establishes:

• It runs under the GVL, and rb_obj_memsize_of never releases it, so there is no data race during the walk — the bad classext pointer pre-exists the call.
• The wrapper has already passed switch (rb_type(obj)), so the object header is a valid class type. The corruption is in the class's classext graph, not the object slot.

The two signatures are the same bug

• gc.c classext_memsize → rb_id_table_memsize: got into a classext, tried to size one of its id_tables (m_tbl/const_tbl) — that id_table pointer is invalid.
• heap_recorder.c → rb_st_foreach (via rb_class_classext_foreach): crashed one frame higher, iterating the namespace→classext table itself — that table is invalid.

Same cause, two surfaces (the classext container vs. one entry's sub-table).

Most likely root cause

The weak-by-id liveness model plus _id2ref resurrection is unsafe for class objects on Ruby 4.0's namespace/classext model: there is a window where _id2ref returns a class that is already unreachable / mid-teardown (id→obj mapping not yet purged, or a namespace classext being dismantled), and rb_obj_memsize_of then walks freed classext memory. The existing gen_age == 0 skip and the RangeError check do not close this on Ruby 4.0. A secondary factor is that classext_memsize / rb_class_classext_foreach are not necessarily robust to being called on every live class (e.g. T_ICLASS, or a lazily/partially-populated namespace classext).

This lines up with this path being the reason heap profiling was hard-disabled on Ruby 4 in #5148 and only re-enabled experimentally in #5201.

The rarity (~15 crashes / 10h for each signature) fits: classes are rarely sampled compared to String/Array/Hash, so a tracked class and a fragile classext at full-update time is a narrow but non-zero window on busy Sidekiq workers that churn many anonymous classes.

Where the fix belongs

ruby_obj_memsize_of (ext/datadog_profiling_native_extension/ruby_helpers.c) already keeps a denylist of rb_obj_memsize_of paths that crash the VM (T_NODE is explicitly excluded). The minimal fix is to also exclude T_CLASS / T_MODULE / T_ICLASS on Ruby 4.0+ (return 0, like other unsupported types). Class objects contribute negligibly to a heap size profile, so the lost signal is immaterial. I will open a PR for this.

[navidemad]

Opened #5938 with the minimal mitigation described above: skip rb_obj_memsize_of for T_CLASS/T_MODULE/T_ICLASS on Ruby 4.0+ (gated by a new NO_SAFE_CLASS_MEMSIZE define). Verified on Linux + Ruby 4.0: the tracked class is reported with size 0 and no SIGSEGV, and the full stack_recorder_spec passes.

[ivoanjo]

Hello there @navidemad!

Thanks for the report and analysis, and sorry you ran into this issue. There's a few notes in the analysis I'd like to dig deeper into, so give us some time to look into the details here 👀

Having said that, as a workaround there is a way to disable calling ruby_obj_memsize_of completely via disabling the "heap size enabled" profile type of the profiler -- the size is only used there, so all other features will remain unaffected.

This can be done via code:

Datadog.configure do |c|
  c.profiling.advanced.experimental_heap_size_enabled = false
end

or via environment variable DD_PROFILING_EXPERIMENTAL_HEAP_SIZE_ENABLED=false.