Fix scheduler shrink issue causing I/O orphaned by retired workers

[Claude]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>[Bug] Scheduler shrink can orphan I/O started by retired workers during drain</issue_title>
<issue_description>## Bug Description

Code review found a scheduler resize/lifecycle race in runtime::scheduler::set_thread_count(). When shrinking the worker pool, a worker whose io_context().pending_count() == 0 is stopped and joined. However, worker_thread::run() enters a drain phase after running_ becomes false and continues resuming queued coroutine tasks on the retiring worker.

If one of those drained tasks starts a new async I/O operation, it binds to the retiring worker's io_context. The worker can then exit, leaving that newly submitted I/O without a polling worker. The suspended coroutine may never resume.

Relevant code paths:

• include/elio/runtime/scheduler.hpp: shrink path calls workers_[i]->stop() when pending_count() == 0, then redistribute_tasks().
• worker_thread::run() drain phase continues calling run_task(handle) after running_ is false.
• I/O awaitables bind to runtime::worker_thread::current()->io_context().

Environment

Please complete the following information:

• Elio Version: 0.5.2 (CMakeLists.txt project version)
• Compiler: Not compiler-specific; found by code review
• OS: Linux
• Kernel Version: Not kernel-specific
• Build Type: Not build-type-specific
• CMake Version: Not applicable

Steps to Reproduce

This was found by code review. A likely runtime scenario is:

1. Start a scheduler with more than one worker.
2. Queue a coroutine on a worker that has no pending I/O yet.
3. Shrink the scheduler so that worker is retired.
4. During the worker's shutdown drain phase, resume the queued coroutine.
5. The coroutine starts a new async operation such as co_await tcp.read(...) or co_await time::sleep_for(...).
6. The retiring worker exits; the new operation is attached to an io_context that is no longer polled.

Minimal Reproducible Example

No executable reproducer yet. Sketch of the shape:

#include <elio/elio.hpp>

using namespace elio;

coro::task<void> starts_io_during_retire() {
    // Needs to be resumed during the retiring worker's drain phase.
    co_await time::sleep_for(std::chrono::milliseconds(10));
}

int main() {
    runtime::scheduler sched(2);
    sched.start();
    sched.go(starts_io_during_retire);
    sched.set_thread_count(1);
    sched.shutdown();
}

A robust test likely needs worker affinity or instrumentation to force the task onto the worker being retired.

Expected Behavior

Shrinking the scheduler should not leave tasks or I/O operations attached to an io_context that no worker will continue polling.

Actual Behavior

The current drain phase can execute queued tasks on a retiring worker after running_ is false. Those tasks can submit fresh I/O to the retiring worker's io_context, after the shrink path already decided there was no pending I/O to drain.

Error Messages

None. This is a potential hang/silent leak rather than a direct error path.

ASAN/TSAN Output

Not available. This was found by code review, not by sanitizer execution.

Additional Context

The shrink path already has a separate draining-worker mode for workers that have pending I/O. The problematic case is when pending_count() == 0 before stop(), but drain-phase task execution creates new pending I/O afterward.

Possible approaches:

• Do not resume arbitrary queued tasks on a worker being retired; redistribute queued tasks before stopping it.
• Add a stop mode for shrink that drains/destroys or migrates queue entries without running user coroutine bodies on the retiring worker.
• Make I/O awaitables refuse to bind to a worker that is in shutdown/draining retirement mode and reschedule to an active worker instead.

Related Issues

• None found in currently open issues.

Possible Solution

// Conceptual direction only:
// 1. Mark worker as retiring / not accepting local execution.
// 2. Stop stealing from it and stop external scheduling to it.
// 3. Move queued handles to active workers before joining.
// 4. Keep the special I/O-draining path only for operations that were already pending.

Checklist

• I have searched existing issues to avoid duplicates
• I have provided all requested information available from code review
• I have included a minimal reproducible example (if possible)
• I have tested with the latest version of Elio
• I have included sanitizer output (for memory/threading issues)
  </issue_description>
• Closes [Bug] Scheduler shrink can orphan I/O started by retired workers during drain #244

<agent_instructions>Check if reported issue is really exists. If does, make associated PR to fix and add tests to make sure it is covered. if not, reply in issue.</agent_instructions>

Comments on the Issue (you are @claude[agent] in this section)

[Copilot]

Pull request overview

This PR fixes a scheduler resize/lifecycle race in runtime::scheduler where shrinking the worker pool could orphan freshly-started async I/O. Previously, a retiring worker's post-running_==false drain phase resumed queued coroutines locally; if such a coroutine started new async I/O (e.g. co_await sleep_for(...)), the operation bound to the retiring worker's io_context, which stops being polled once the thread exits, hanging the coroutine. The fix routes queued tasks back through scheduler_->spawn() when the worker is being retired (scheduler still running) rather than resuming them locally, while preserving local draining during a genuine full shutdown.

Changes:

• In worker_thread::run(), compute retiring = scheduler_->is_running() and, during the drain phase, respawn queued handles to surviving workers when retiring instead of executing them locally; still destroy done handles.
• Add a targeted regression test that forces a pinned I/O-starting task into a busy retiring worker's inbox and verifies the redistributed I/O completes on the surviving worker.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
include/elio/runtime/scheduler.hpp	Splits the worker drain phase into a retirement path (respawn queued tasks to active workers) vs. a full-shutdown path (drain locally), preventing orphaned I/O on retiring workers.
tests/unit/test_scheduler.cpp	Adds helper tasks and a regression test verifying I/O started by a task queued on a retiring worker completes on a surviving worker after shrink.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.