Experiment with indexed I/O ready workers

[yxanul]

Summary

Draft experiment PR. Do not merge for now.

This keeps the existing per-worker io_inbox task storage model, adds an optional ready-worker index, and keeps the single global I/O-ready queue as an env-gated control.

New knobs:

• ZMEU_TASK_GLOBAL_IO_READY=1: route resumed I/O tasks through one global I/O task queue.
• ZMEU_TASK_IO_READY_INDEX=1: keep resumed I/O tasks in the home worker inbox, index worker IDs whose inboxes transitioned empty -> non-empty, and use the normal gated wake instead of targeted home-worker wakes.

The index path is advisory: stale worker IDs are allowed, and indexed workers are requeued when a pop leaves more inbox work behind. Victim inbox scanning remains as a fallback.

Findings

Quiet reactor echo benchmark, 1024 connections x 1000 rounds:

mode	W	rt/s	p50	p95	p99
base	16	1.337M	703us	1275us	1599us
global	16	1.127M	897us	1103us	1323us
index	16	1.178M	860us	1068us	1352us
base	24	1.071M	913us	1512us	1758us
global	24	1.100M	920us	1099us	1306us
index	24	1.159M	867us	1094us	1373us
base	32	0.907M	1144us	1606us	1832us
global	32	1.099M	916us	1088us	1275us
index	32	1.104M	919us	1105us	1316us

One quiet base W24 run failed in the load generator with connection reset/refused and was excluded.

Counter run signal:

• W24 victim inbox attempts/round: base 15.6, global 5.1, index 5.2.
• W32 victim inbox attempts/round: base 23.2, global 6.7, index 7.8.
• W24 parks/round: base 0.228, global 0.053, index 0.068.
• W32 parks/round: base 0.227, global 0.037, index 0.052.

Interpretation:

• The ready index cuts most accidental O(workers) victim inbox discovery at W24/W32.
• It recovers W24 throughput/p50 better than the single global I/O task queue.
• W16 still has a p50/throughput tradeoff versus base.
• W8 does not justify enabling this path.
• This is not yet an obvious default; it is useful as a saved experiment and likely wants batch draining or per-worker listed flags next.

CPU sanity:

• zparallel_digest: neutral at W24/W32.
• runtime_wakeup_hell: index was about 1-2.5% slower in 200-run batches.

Validation

• cargo fmt --check
• cargo build --release --features mlir-backend
• cargo test --features mlir-backend deterministic_task_counters
• cargo test --features mlir-backend scheduler_gauntlet
• cargo test --features mlir-backend wakeup_hell_gauntlet
• cargo test --features mlir-backend batch_steals_worker_deque_backlog
• git diff --check

Follow-up Ideas

• Batch-drain a small number of tasks from indexed inboxes before returning to search.
• Add a per-worker io_ready_listed flag to reduce stale index entries.
• Move victim inbox scanning later or make it fallback-only once the index is strong enough.
• Compare against sharded ready-worker indexes before considering sharded task queues.