Debugging

Elio provides debugging tools to inspect coroutine (vthread) states, virtual call stacks, and scheduler information. These tools work with both live processes and coredump files.

Overview

Elio coroutines maintain debug metadata in each frame:

Unique ID for identification
State (created, running, suspended, completed, failed)
Source location (file, function, line)
Worker thread assignment
Parent pointer for virtual stack traversal

The debugger extensions find runnable coroutine frames through each worker's Chase-Lev deque. The transient MPSC inbox and rare overflow queue are not walked because concurrent producers make a lock-free debugger snapshot unreliable; their tasks appear after the worker drains them into the deque. This avoids a global frame registry and its always-on synchronization cost.

Virtual Stack

C++20 stackless coroutines allocate each frame independently on the heap. When a coroutine suspends, the native call stack unwinds completely, so traditional stack traces cannot show the logical call chain. Elio reconstructs this information through an intrusive virtual stack built into every coroutine frame.

How It Works

Each coroutine's promise type inherits from promise_base, which contains a parent_ pointer. When coroutine A co_awaits coroutine B, B's promise stores a pointer back to A's promise. This forms a singly-linked list from the innermost frame to the outermost caller, mirroring what a native call stack would look like if the coroutines were regular functions. Lazy task ownership alone does not keep a frame installed in thread-local stack state, so moving or destroying an unstarted task cannot leave creator-thread stack pointers behind.

The thread-local current_frame_ tracks which frame is currently executing. A task binds its parent_ to the actual awaiter when co_await starts it. Scheduler, synchronization, I/O, and affinity-migration resume paths preserve that ancestry and install the resumed frame while user code runs. Completion restores the logical parent before transferring control to the continuation. Separate initial spawn and spawn_to paths detach construction-time ancestry when independent work is handed to the scheduler.

Frame Validation

Each promise_base contains a frame_magic_ field set to 0x454C494F46524D45 (the ASCII string "ELIOFRME"). The debugger tools check this magic value when traversing memory to distinguish valid Elio coroutine frames from arbitrary data. This is especially important during coredump analysis, where the debugger walks raw memory without type information.

The stable raw-memory fallback is limited to frame_magic_ and parent_. Runtime policy is held through execution_context_, a shared_ptr<task_execution_context>. GDB/LLDB scripts detect that field when type information is available but intentionally do not assume a particular standard-library shared_ptr representation.

With type information available, inspect the context through normal C++ expressions when diagnosing worker-local I/O placement:

promise.execution_context()->has_active_io_pin()
promise.execution_context()->io_owner_worker()
promise.execution_context()->io_context_generation()
promise.execution_context()->active_io_pin_count()

An active pin must match the owning worker's io_context::owner_worker_id() and io_context::generation(). User affinity may differ while the operation is pending because effective affinity gives the I/O owner priority. The debugger scripts intentionally do not decode these fields from raw shared_ptr bytes.

Debug Metadata

Every frame carries the following debug metadata with no additional allocation:

Field	Description
`debug_id_`	Unique monotonic identifier assigned at creation
`debug_state_`	Current state: created, running, suspended, completed, or failed
`debug_worker_id_`	Index of the worker thread the frame is assigned to (or 0xFFFFFFFF if unassigned)
`debug_location_`	Source location struct (type `debug_location`) with members `file`, `function`, `line`, set via manual `set_location()`
`parent_`	Pointer to the calling frame's promise, forming the virtual stack chain
`frame_magic_`	Magic number for frame integrity validation

The debugger tools (elio-pstack, elio-gdb.py, elio_lldb.py) use this metadata to present coroutine state in a format familiar to anyone who has used pstack or thread apply all bt.

Tools

Tool	Description
`elio-pstack`	Command-line tool similar to `pstack`
`elio-gdb.py`	GDB Python extension
`elio_lldb.py`	LLDB import entrypoint (loads `elio-lldb.py`)

elio-pstack

A command-line tool that prints stack traces of all Elio coroutines, similar to pstack for threads.

Usage

# Attach to running process
elio-pstack <pid>

# Analyze coredump with executable
elio-pstack <executable> <corefile>

# Analyze coredump (auto-detect executable)
elio-pstack <corefile>

Options

Option	Description
`-h, --help`	Show help message
`-v, --verbose`	Show verbose output
`-l, --list`	Only list vthreads (no backtraces)
`-i, --info <id>`	Show detailed info for specific vthread
`-s, --stats`	Show statistics only

Examples

# Print all vthread backtraces for a running process
elio-pstack 12345

# List all vthreads without backtraces
elio-pstack -l 12345

# Get detailed info for vthread #42
elio-pstack -i 42 12345

# Analyze a coredump
elio-pstack ./myapp core.12345

# Show scheduler statistics from coredump
elio-pstack -s core.12345

GDB Extension

Loading

# From GDB command line
gdb -ex 'source /path/to/tools/elio-gdb.py' ./myapp

# Or in GDB session
(gdb) source /path/to/tools/elio-gdb.py

# Or add to ~/.gdbinit
source /path/to/tools/elio-gdb.py

Commands

Command	Description
`elio`	Show help
`elio list`	List all vthreads from worker queues
`elio bt [id]`	Show backtrace for vthread(s)
`elio info <id>`	Show detailed info for a vthread
`elio workers`	Show worker thread information
`elio stats`	Show scheduler statistics

Examples

(gdb) elio list
--------------------------------------------------------------------------------
ID       State        Worker   Function                       Location
--------------------------------------------------------------------------------
1        suspended    0        worker_task                    debug_test.cpp:84
2        suspended    1        process_data                   debug_test.cpp:73
3        running      2        compute_value                  debug_test.cpp:60

Total queued coroutines: 3

(gdb) elio bt 1
vthread #1 [suspended] (worker 0)
  #0   0x00007f1234567890 in worker_task at debug_test.cpp:84
  #1   0x00007f1234567abc in async_main at debug_test.cpp:112

(gdb) elio info 1
vthread #1
  State:    suspended
  Worker:   0
  Handle:   0x00007f1234567890
  Promise:  0x00007f12345678a0
  Function: worker_task
  Location: debug_test.cpp:84

  Virtual Call Stack:
    #0   worker_task at debug_test.cpp:84
    #1   async_main at debug_test.cpp:112

(gdb) elio workers
Scheduler: running
Worker threads: 4
------------------------------------------------------------
Worker   Status       Queue Size   Tasks Executed
------------------------------------------------------------
0        running      5            1234
1        running      3            1189
2        running      4            1201
3        running      2            1156

(gdb) elio stats
Scheduler: running
Worker threads: 4
Total queued coroutines: 14
Total tasks executed: 4780

LLDB Extension

Loading

# From LLDB command line
lldb -o 'command script import /path/to/tools/elio_lldb.py' ./myapp

# Or in LLDB session
(lldb) command script import /path/to/tools/elio_lldb.py

# Or add to ~/.lldbinit
command script import /path/to/tools/elio_lldb.py

Commands

The LLDB extension provides the same commands as GDB:

Command	Description
`elio`	Show help
`elio list`	List all vthreads from worker queues
`elio bt [id]`	Show backtrace for vthread(s)
`elio info <id>`	Show detailed info for a vthread
`elio workers`	Show worker thread information
`elio stats`	Show scheduler statistics

Setting Debug Location

For more accurate debugging information, you can manually set the debug location in your coroutines:

#include <elio/elio.hpp>

// Helper awaitable to get promise reference
struct get_promise {
    bool await_ready() const noexcept { return false; }
    
    template<typename Promise>
    bool await_suspend(std::coroutine_handle<Promise> h) noexcept {
        promise_ = &h.promise();
        return false; // Don't actually suspend
    }
    
    elio::coro::promise_base& await_resume() noexcept {
        return *promise_;
    }
    
    elio::coro::promise_base* promise_ = nullptr;
};

elio::coro::task<void> my_coroutine() {
    // Set debug location
    auto& p = co_await get_promise{};
    p.set_location(__FILE__, __FUNCTION__, __LINE__);
    p.set_state(elio::coro::coroutine_state::running);
    
    // ... coroutine body ...
    
    co_await some_operation();
    
    // Update state after suspension point
    p.set_state(elio::coro::coroutine_state::suspended);
    
    co_return;
}

Sanitizer Support

Elio's test suite builds with both AddressSanitizer (ASAN) and ThreadSanitizer (TSAN). Task coroutine frames use the standard heap allocation path in sanitizer and normal builds alike, allowing sanitizers to track the same allocation, cross-thread destruction, and lifetime behavior used in production.

No source changes or build flags are needed beyond enabling the sanitizer itself:

# Build and run with ASAN
cmake --build . --target elio_tests_asan
./tests/elio_tests_asan

# Build and run with TSAN
cmake --build . --target elio_tests_tsan
./tests/elio_tests_tsan

Limitations

Only queued coroutines are visible: Coroutines currently executing on a worker thread are not in any queue and cannot be found by the debugger. Use regular GDB/LLDB thread inspection for those.
Queue traversal is best-effort: The debugger reads queue data structures directly from memory. In rare cases of concurrent modification, some frames may be missed.
Parent chain traversal: The virtual stack display shows the parent pointer chain, but detailed info for parent frames may be limited.

Coredump Analysis

All tools work with coredump files for post-mortem debugging:

# Generate coredump on crash (ensure ulimit allows it)
ulimit -c unlimited

# Or trigger manually
kill -ABRT <pid>
gcore <pid>

# Analyze with elio-pstack
elio-pstack ./myapp core.12345

# Or with GDB
gdb ./myapp core.12345 -ex 'source tools/elio-gdb.py' -ex 'elio bt'

Troubleshooting

"No active scheduler found"

The debugger couldn't find the scheduler. Possible causes:

The process hasn't created a scheduler yet
The scheduler has been destroyed
Symbol information is not available (stripped binary)

"No queued vthreads found"

All coroutines are either completed or currently executing. Check:

Regular thread backtraces with bt or thread apply all bt
Scheduler statistics with elio stats

Symbols not found

Ensure the binary is compiled with debug symbols:

cmake -DCMAKE_BUILD_TYPE=Debug ..
# or
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..

Debugging

Debugging

Overview

Virtual Stack

How It Works

Frame Validation

Debug Metadata

Tools

elio-pstack

Usage

Options

Examples

GDB Extension

Loading

Commands

Examples

LLDB Extension

Loading

Commands

Setting Debug Location

Sanitizer Support

Limitations

Coredump Analysis

Troubleshooting

"No active scheduler found"

"No queued vthreads found"

Symbols not found

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally