cass index deadlocks at a batch boundary on main: all page-prep workers futex-parked while jobs stay active, heartbeat keeps refreshing (reproduces across force-rebuild and incremental)

[nskidan]

Setup: cass 0.6.25 built from a6a202e3 (2026-08-22), Linux x86_64 (Ubuntu 6.8 kernel, 16 cores / 62 GB); corpus: 3,714 conversations / ~250K messages including five codex rollout files of 87–250 MB (2026-08-07..17); archive DB 2.1 GB.

Symptom: every indexing path stops making forward progress at a batch boundary and never recovers, while the index-run.lock heartbeat keeps refreshing (the in-binary hint points at #244/#258, both closed — this reproduces on current main):

• cass index --full --force-rebuild (default responsiveness budget): wedges at current=320/3714
• same with CASS_RESPONSIVENESS_MAX_INFLIGHT_BYTES=8589934592: wedges at current=1024/3714 (budget size relocates the wedge)
• plain cass index (incremental): wedges at current=320

All runs used CASS_INDEX_STALL_ABORT_SECS=0; without it the run exits 70 during the (legitimately slow) preparing phase on this DB before indexing even starts — the 300s abort default fires on a healthy slow open (secondary bug, see below).

Diagnostics at wedge (progress events byte-identical for 20+ minutes):

• producer_state: "waiting_turn" (force-rebuild) / "waiting_result" (incremental); active_page_prep_jobs: 7; queue_depth: 0; ordered_buffered_pages: 0
• inflight_message_bytes frozen (312145408 in one run, 796632055 in another — far below the 8 GiB budget, so not byte starvation)
• per-thread CPU over a 15 s window: ~0.04 s total across all 56–58 threads — nothing is computing; each cass-lexical-pa* worker accumulated ~57 s CPU before parking permanently
• kernel stacks: main thread in clock_nanosleep poll loop, every worker in futex_wait — jobs marked active while their threads sleep looks like a lost wakeup between page-prep completion and the ordered producer
• wedge onset correlates with reaching the first oversized (250 MB) conversation; possibly relevant: redact_secrets skips memoization above MAX_MEMOIZED_INPUT_BYTES, so the giant conversations take the uncached path right where the wedge occurs

Secondary findings:

1. stall_threshold_secs:120 / abort_threshold_secs:300 fires during the preparing phase on a 2.1 GB DB → exit 70 before indexing begins.
2. Weeks of 15-minute timer runs dying this way had accumulated 64 interrupted raw_mirror_capture artifacts that blocked doctor mutation — --cleanup-interrupted-artifacts --yes handled it cleanly (nice mechanism).

Thread-stack capture #1 (force-rebuild wedge, current=320)

captured 2026-08-22T23:47:35Z pid=2808659 during inflight-budget wedge (current=320/3714, ~50min no progress)
== thread 2808659 cass state=S
[<0>] hrtimer_nanosleep+0xa2/0x170
[<0>] common_nsleep_timens+0x4e/0x90
[<0>] __x64_sys_clock_nanosleep+0xf4/0x180
[<0>] x64_sys_call+0x1c2c/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2808690 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2808691 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2808692 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2808693 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2808694 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2808698 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2808704 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809879 cass-responsive state=S
[<0>] hrtimer_nanosleep+0xa2/0x170
[<0>] common_nsleep_timens+0x4e/0x90
[<0>] __x64_sys_clock_nanosleep+0xf4/0x180
[<0>] x64_sys_call+0x1c2c/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809880 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809881 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809882 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809883 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809884 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809885 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809886 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809887 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809888 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809889 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809890 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809891 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809892 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809893 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809894 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809895 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2809896 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810150 cass-lexical-pa state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810151 cass-lexical-pa state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810152 cass-lexical-pa state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810153 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810154 cass-lexical-pa state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810155 cass-lexical-pa state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810156 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810157 cass-lexical-pa state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810158 cass-lexical-pa state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810159 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810160 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810161 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810162 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810163 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810431 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810432 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810433 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810434 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810435 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810436 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810437 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810438 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810439 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810440 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810441 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810442 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810443 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810444 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810445 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2810446 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80

Thread-stack capture #2 (8GiB-budget wedge current=1024, + final incremental-wedge diagnostics)

second capture 2026-08-23T00:13:44Z pid=2826155 wedge at current=1024/3714
== thread 2826155 cass cpu=44.5 state=S
[<0>] hrtimer_nanosleep+0xa2/0x170
[<0>] common_nsleep_timens+0x4e/0x90
[<0>] __x64_sys_clock_nanosleep+0xf4/0x180
[<0>] x64_sys_call+0x1c2c/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2826187 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2826188 asupersync-work cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2826189 asupersync-work cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2826191 asupersync-work cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2826193 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2826196 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2826203 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2827734 cass-responsive cpu= state=S
[<0>] hrtimer_nanosleep+0xa2/0x170
[<0>] common_nsleep_timens+0x4e/0x90
[<0>] __x64_sys_clock_nanosleep+0xf4/0x180
[<0>] x64_sys_call+0x1c2c/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 2827735 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827736 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827737 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827738 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827739 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827740 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827741 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827742 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827743 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827744 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827745 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827746 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827747 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827748 cass-lexical-re cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827749 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2827750 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2827751 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2828164 cass-lexical-pa cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828165 cass-lexical-pa cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828166 cass-lexical-pa cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828167 cass-lexical-pa cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828168 cass-lexical-pa cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828169 cass-lexical-pa cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828170 cass-lexical-pa cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828171 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2828172 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2828173 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2828174 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2828175 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2828176 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2828177 asupersync-work cpu= state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
== thread 2828445 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828446 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828447 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828448 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828449 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828450 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828451 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828452 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828453 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828454 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828455 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828456 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828457 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828458 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828459 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
== thread 2828460 cass cpu= state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
=== final wedge evidence 2026-08-23T00:33:05Z — incremental path, current=320 ===
{"phase":"indexing","phase_code":2,"unit":"conversations","total":3714,"total_is_final":true,"current":320,"activity":334,"discovered_agents":0,"agent_names":[],"is_rebuilding":true,"finalizing":false,"persist_in_progress":false,"elapsed_ms":1095288,"rate_per_sec":0.292160600682195,"eta_seconds":11616.89835,"last_error":null,"quarantined_conversations":0,"lexical_update_deferred":false,"rebuild_pipeline":{"queue_depth":0,"inflight_message_bytes":796632055,"pending_batch_conversations":191,"pending_batch_message_bytes":3402704,"page_prep_workers":7,"active_page_prep_jobs":7,"ordered_buffered_pages":0,"budget_generation":2,"producer_budget_wait_count":1,"producer_budget_wait_ms":23,"producer_state":"waiting_result","reservation_next_sequence":10,"producer_budget_active_waiters":1,"producer_handoff_wait_count":1,"producer_handoff_wait_ms":597,"host_loadavg_1m":0.08,"host_available_memory_bytes":8240369664,"process_rss_bytes":48663314432,"controller_mode":"pressure_limited","controller_reason":"host_available_memory_bytes_8240369664_below_reserve_8428064768","staged_merge_workers_max":0,"staged_merge_allowed_jobs":0,"staged_merge_active_jobs":0,"staged_merge_ready_artifacts":0,"staged_merge_ready_groups":0,"staged_merge_controller_reason":null,"staged_shard_build_workers_max":0,"staged_shard_build_allowed_jobs":0,"staged_shard_build_active_jobs":0,"staged_shard_build_pending_jobs":0,"staged_shard_build_controller_reason":null,"staged_shard_build_memory_reserve_bytes":0,"staged_shard_build_emergency_memory_reserve_bytes":0,"staged_shard_build_completed_jobs":0,"staged_shard_build_last_shard_index":null,"staged_shard_build_last_message_bytes":0,"staged_shard_build_last_index_size_bytes":0,"staged_shard_build_last_duration_ms":0,"staged_shard_build_last_amplification_milli":null,"staged_shard_build_observed_amplification_milli":null},"event":"progress","ts_ms":1787445183605}

Last progress event of the incremental run (truncated)

{"phase":"indexing","phase_code":2,"unit":"conversations","total":3714,"total_is_final":true,"current":320,"activity":334,"discovered_agents":0,"agent_names":[],"is_rebuilding":true,"finalizing":false,"persist_in_progress":false,"elapsed_ms":1095288,"rate_per_sec":0.292160600682195,"eta_seconds":11616.89835,"last_error":null,"quarantined_conversations":0,"lexical_update_deferred":false,"rebuild_pipeline":{"queue_depth":0,"inflight_message_bytes":796632055,"pending_batch_conversations":191,"pending_batch_message_bytes":3402704,"page_prep_workers":7,"active_page_prep_jobs":7,"ordered_buffered_pages":0,"budget_generation":2,"producer_budget_wait_count":1,"producer_budget_wait_ms":23,"producer_state":"waiting_result","reservation_next_sequence":10,"producer_budget_active_waiters":1,"producer_handoff_wait_count":1,"producer_handoff_wait_ms":597,"host_loadavg_1m":0.08,"host_available_memory_bytes":8240369664,"process_rss_bytes":48663314432,"controller_mode":"pressure_limited","controller_reason":"host_available_memory_bytes_8240369664_below_reserve_8428064768","staged_merge_workers_max":0,"staged_merge_allowed_jobs":0,"staged_merge_active_jobs":0,"staged_merge_ready_artifacts":0,"staged_merge_ready_groups":0,"staged_merge_controller_reason":null,"staged_shard_build_workers_max":0,"staged_shard_build_allowed_jobs":0,"staged_shard_build_active_jobs":0,"staged_shard_build_pending_jobs":0,"staged_shard_build_controller_reason":null,"staged_shard_build_memory_reserve_bytes":0,"staged_shard_build_emergency_memory_reserve_bytes":0,"staged_shard_build_completed_jobs":0,"staged_shard_build_last_shard_index":null,"staged_shard_build_last_message_bytes":0,"staged_shard_build_last_index_size_bytes":0,"staged_shard_build_last_duration_ms":0,"staged_shard_build_last_amplification_milli":null,"staged_shard_build_observed_amplification_milli":null},"event":"progress","ts_ms":1787445183605}

[Dicklesworthstone]

Thank you — this is one of the best-instrumented reports this repo has received, and it is confirmed. Tracked as coding_agent_session_search-cjugu (P0).

Independent confirmation. I have a second, much smaller reproduction of the same wedge class: on a Linux test host, cass index --full over a tiny multi-agent fixture (no oversized conversations) wedges ~13 s in with the identical kernel-stack shape you captured — main thread in clock_nanosleep, every worker in futex_wait, asupersync workers in ep_poll, ~zero CPU, heartbeat alive. So the wedge does not require the 250 MB conversations; they just make it far more likely on your corpus.

Where your evidence points. Your budget experiment is the key discriminator: raising CASS_RESPONSIVENESS_MAX_INFLIGHT_BYTES relocates the wedge (320 → 1024), which means bytes are being retained, not that a wakeup is merely lost. The ordered pipeline reserves budget like this (src/indexer/mod.rs, reserve_lexical_rebuild_page_budget):

1. wait_for_turn(sequence) — strictly ordered;
2. acquire_with_wait(page_message_bytes) — while still holding the turn;
3. finish_turn(sequence) only after the acquisition succeeds.

That design makes one under-budget acquisition park the entire pipeline by construction: every later sequence parks in wait_for_turn (your waiting_turn), the ordered producer parks in waiting_result, and the run can only proceed if already-reserved bytes release without the turn-holder progressing. Your frozen inflight_message_bytes with ordered_buffered_pages: 0 says exactly that release is not happening — the retained bytes sit somewhere downstream of page-prep (consumed-but-not-yet-released), and the audit in the bead targets the release paths between producer consumption and shard commit for a hold that itself requires producer progress.

Your secondary findings, both accepted:

1. The 120 s/300 s stall/abort clock starting before the preparing phase completes is a real, separate bug — on a 2.1 GB archive a healthy open exceeds it. It gets a phase-aware fix (tracked in the same bead).
2. Glad --cleanup-interrupted-artifacts --yes handled the 64 accumulated capture artifacts — that mechanism exists for precisely the crash-loop you hit.

Workaround until the fix ships: none that's honest — the wedge is structural, and CASS_INDEX_STALL_ABORT_SECS=0 (which you already found) only stops the misfiring abort, not the wedge. Nothing you're doing is wrong, and force-rebuild vs incremental makes no difference, as you observed.

One note: v0.6.26 (released today) does not contain a fix for this — it's a dependency-refresh release. The wedge reproduces on current main as you said, and the fix will land against the bead above with your thread-stack captures as the acceptance evidence.

[Dicklesworthstone]

Fix landed on main: 424765b.

Root cause (matches every one of your observations). The ordered pipeline reserves each page's bytes in sequence order, and the sink retains every absorbed packet's reservation in its pending batch until a flush — but the only flush triggers were conversation count and planned-shard boundaries. Bytes were never a trigger. A page of few-but-huge conversations (your 87–250 MB rollouts) therefore parks most of the budget inside an idle sink; the next page-prep worker holds the ordered reservation turn and parks in the budget wait; the turn blocks every later page, so the sink's channel stays empty and no release can ever arrive. Your budget experiment was the fingerprint that ruled out a lost wakeup: raising CASS_RESPONSIVENESS_MAX_INFLIGHT_BYTES relocated the wedge (320 → 1024) because it takes longer for retained-bytes + next-request to exceed the cap.

The fix. The sink already wakes every 250 ms. On a tick where its channel is empty and the byte limiter has a parked waiter, it now flushes the pending batch, releasing the retained reservations — batching efficiency is untouched otherwise (an idle tick with no budget waiter never force-flushes). Liveness: parked acquirers are observed within ≤ 250 ms, handed-off pages drain without producer progress, and acquisitions are budget-capped, so inflight drains until any capped request fits. A regression test (gh413_sink_release_unparks_budget_waiter_holding_the_reservation_turn) pins the geometry.

A correction to my earlier comment. I wrote that my small-fixture reproduction was "the same wedge class" — the kernel-stack shape matched, but a discriminator run with CASS_PIPELINE_TRACE=1 showed my fixture wedges in the ingest phase with zero lexical-pipeline activity: it is a different bug, tracked separately. Your report's evidence alone drove this fix; I shouldn't have leaned on my repro as confirmation of the same mechanism.

Could you retest from main (≥ 424765b3)? Your corpus is the definitive validation:

git pull && cargo build --release
CASS_INDEX_STALL_ABORT_SECS=0 ./target/release/cass index --full --force-rebuild

With the fix you should see an INFO line like lexical rebuild sink starved while a page-prep worker waits for byte budget; flushing pending batch … (GH#413) right where the old runs wedged, and progress should continue past 320/3714. (Keep CASS_INDEX_STALL_ABORT_SECS=0 for now — your secondary finding, the stall/abort clock firing during a healthy slow preparing phase, is a separate pre-existing bug that already has its own open work item, referenced from this issue's tracker.)

[nskidan]

Retested on the original corpus — the fix is validated. Full results below, plus three follow-on findings from the same corpus you may want as separate work items.

Build: exactly 424765b3 (cass 0.6.26 (424765b3d3ef)). Note: current tip 154f36d9 doesn't build standalone — frankensqlite::AsyncConnection imports resolve against crates.io fsqlite =0.3.8 without its async-api feature (E0432 in src/search/query.rs:32, src/storage/sqlite.rs:16, plus two follow-on E0277s at query.rs:6901) — flagging in case it's mid-flight work rather than known.

Environment: Linux x86-64, 16 cores, 62 GB RAM. Archive: 3,714 sessions at rebuild time (2.24 GB canonical DB, now 3.0 GB), including the 87–251 MB codex rollouts from the report.

Validation: --full --force-rebuild

CASS_INDEX_STALL_ABORT_SECS=0 cass index --full --force-rebuild

• Completed 3714/3714 in ~14m08s, exit 0. Final WAL checkpoint clean (3221/3221 frames). cass health healthy; search returns correct hits.
• The GH#413 INFO line fired 23 times. First firing at ~9.5 min: pending_conversations=224 pending_message_bytes=4670342 inflight_message_bytes=939524096 — precisely the retained-bytes-at-budget geometry from the report, where every earlier run wedged permanently on first encounter (320/3714). This run sailed through all 23.
• The 120 s preparing stall warnings fired as you predicted and stayed report-only under ABORT_SECS=0 — consistent with the separate phase-aware-clock issue you're tracking.
• Observed peak RSS during the rebuild: ~38–40 GB (last stall snapshot: process_rss_bytes: 40914833408). Completed fine here; noting for sizing.

Thank you — this restored session search on this box after three weeks down.

Follow-on findings (post-fix, same corpus)

1. First incremental after the canonical-only force-rebuild carries a large deferred backlog, at single-threaded speed. Since --force-rebuild skips the raw rescan, the next plain cass index had to absorb ~3 weeks of never-ingested content (indexing had been silently wedged since Aug 3; the giant rollouts arrived Aug 7–17). Run uncapped, it completed cleanly in 8h06m (378 conversations, 306,675 messages, 0 quarantined, exit 0). The dominant cost is per-giant-conversation persist (map_to_internal_with_redactor + writer transaction): the ~70 MB class took ~13 min each, the 251 MB rollout 80+ minutes — cost looks super-linear in conversation size, and the phase is one-conversation-one-core while 15 cores idle. (The flow-limiter reservation gating made the progress totals advance in steps — took us a while to understand, but working as designed per the #413 fix's own model.) Feature-shaped ask: a way to route bulk catch-up through the parallel rebuild machinery, or parallelize the per-conversation persist.

2. An externally killed incremental appears to invalidate the incremental checkpoint and escalate the NEXT run to a full-archive reconcile. Our systemd unit SIGTERM'd one incremental at its 30-min ceiling (mea culpa — the unit predates the fix). The next cass index admitted 4,044 conversations (≈ the entire archive) to its indexing phase instead of a small delta — with nothing new on disk (verified by file mtimes: exactly one transcript had changed). That run then died by kernel OOM at ~23 min (RSS ballooning during giant-conversation persists on a 62 GB box; the successful 8h run had peaked ~35 GB). If SIGTERM-mid-run → full reconcile is intended crash-safety, it interacts badly with finding 1's cost profile: kill → full reconcile → too big to finish → kill/OOM → repeat. A bounded/resumable reconcile (or a checkpoint that survives SIGTERM between conversation commits) would break the cycle.

3. After the OOM-killed run, plain cass search (no --refresh) burned 100% CPU for minutes without returning. Timed out at 180 s with ~158 s CPU consumed; a second query behaved the same. Related observation from before the retest: a hook-invoked plain cass search had spawned a lexical_refresh index job (per the index-run lock's job_kind) and held the lock wedged for 5 h under the release binary — so some search path does take the index-run lock and do index work even without --refresh; under a poisoned/dirty state that work appears unbounded from the caller's perspective.

4. --full --force-rebuild now OOMs on this corpus — because the fix worked. Yesterday's 14-minute validation run predates the 8-hour ingest: back then the giant rollouts were NOT yet in the canonical DB, so the canonical-only rebuild never touched them. Now that they are, the parallel rebuild dies by kernel OOM (62 GB box) in the page-prep phase:

• default inflight budget: SIGKILL by the OOM killer ~17 min in;
• retried with CASS_RESPONSIVENESS_MAX_INFLIGHT_BYTES=268435456 (256 MB): RSS still shot past 57 GB (our watchdog SIGTERM'd it cleanly at that point to protect the DB). So the inflight-byte budget does not bound resident memory here — the blowup looks like per-page/per-conversation fetch expansion of the 87–251 MB conversations (page_conversation_limit=32, batch_fetch_message_limit=800000 per the "adopted pipeline budgets" line), possibly with the budget controller scaling up against 56 GB of free host memory.

Net state on this box: canonical DB intact and complete (clean SIGTERMs; WAL tiny), but no index-build path currently completes — the parallel rebuild OOMs on the giants, and the incremental path costs hours-per-giant (finding 1) and escalates to full-archive reconcile after any kill (finding 2). Search is down pending a way through. Most useful knobs from where we sit: size-aware paging in lexical-rebuild page-prep (a giant conversation should page alone, in bounded chunks), and/or a size-quarantine option for the lexical path.

Happy to test patches for any of these on this corpus — it reproduces the giant-rollout profile reliably. Thread-stack captures and full run logs banked if you want them attached to the tracker bead.

[Dicklesworthstone]

Thanks for the validation run and for the four follow-ons; that is exactly the evidence needed.

On the build note: current main compiles standalone again. The fsqlite dependency now enables its async-api feature explicitly in Cargo.toml, so the AsyncConnection imports resolve against crates.io 0.3.8. 424765b3 is on main but is not in the v0.6.26 tag, so the sink-starvation fix ships in the next release.

Your findings, cross-referenced so they do not get lost in this thread:

• Finding 3 (a plain cass search spawning a lexical_refresh job that holds index-run.lock with a frozen last_progress_at_ms, and cass search burning CPU without returning after an OOM-killed run) is the same wedge as 0.6.26: a foreground cass search can wedge lexical_refresh and hold index-run.lock indefinitely (heartbeat refreshes while progress stays frozen; #258 pattern) #422, filed independently the same day from a different corpus. Your detail that it appears after an interrupted/OOM-killed index run narrows the state that arms it; I have noted it there.
• Finding 2 (a SIGTERM'd incremental escalating the next run to a full-archive reconcile) overlaps an interrupted incremental cass index keeps its committed conversations but nothing else #426, which measured the same "kill mid-run, next run re-parses everything" behaviour and proposes a per-file ingest ledger plus a termination hook that finishes the in-flight chunk and writes watermarks.
• Findings 1 and 4 (per-giant-conversation persist being super-linear and single-threaded; page-prep RSS not bounded by the inflight-byte budget on 87-251 MB conversations, so --full --force-rebuild now OOMs precisely because the wedge no longer hides it) are not covered by an existing issue. They need a size-aware paging design in lexical-rebuild page-prep (a giant conversation paging alone, in bounded chunks) rather than another knob, so I am keeping them on this issue rather than splitting them out until that design exists. Your numbers are the acceptance data for it.

If you can attach the thread-stack capture from the OOM'd page-prep run (finding 4) here, that is the most useful single artifact for the next step.

[nskidan]

Here's the finding-4 capture — from a fresh controlled repro this morning rather than the original OOM'd run (the kernel SIGKILL left nothing to read), same geometry: built at 424765b3, CASS_RESPONSIVENESS_MAX_INFLIGHT_BYTES=268435456, --full --force-rebuild, otherwise idle 62 GB box. Two snapshots 108 seconds apart: RSS 30 GB → 46 GB (~150 MB/s allocation), at which point I SIGTERM'd it to protect the DB (the prior identical run at default budget was OOM-killed by the kernel).

The near-peak snapshot is the telling one: all 7 cass-lexical-pa (page-prep) worker threads are simultaneously in R state — concurrent giant-conversation expansion, multiplicative across workers, invisible to the inflight-byte budget. Consistent with your "a giant conversation should page alone, in bounded chunks" framing; happy to be the acceptance corpus when that design exists.

Capture 1 — RSS 30 GB (VmRSS/VmHWM 31,533,488 kB, 26 threads)

captured 2026-08-26T07:28:35Z pid=132406 during page-prep RSS-blowup repro, RSS=30GB (inflight budget 256MB, finding 4)
VmHWM:	31533488 kB VmRSS:	31533488 kB Threads:	26 
== thread 132406 cass state=S
[<0>] hrtimer_nanosleep+0xa2/0x170
[<0>] common_nsleep_timens+0x4e/0x90
[<0>] __x64_sys_clock_nanosleep+0xf4/0x180
[<0>] x64_sys_call+0x1c2c/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132459 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132460 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132461 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132462 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132474 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132486 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132496 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136471 cass-responsive state=S
[<0>] hrtimer_nanosleep+0xa2/0x170
[<0>] common_nsleep_timens+0x4e/0x90
[<0>] __x64_sys_clock_nanosleep+0xf4/0x180
[<0>] x64_sys_call+0x1c2c/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136473 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136474 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136475 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136476 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136477 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136478 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136479 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136480 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136481 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136482 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136483 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136484 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136485 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136486 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136487 cass state=R
== thread 136488 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136489 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80

Capture 2 — RSS 46 GB near-peak, 108 s later (7× cass-lexical-pa in R state)

captured 2026-08-26T07:30:23Z pid=132406 during page-prep RSS-blowup repro NEAR-PEAK, RSS=46GB (about to SIGTERM)
VmHWM:	49124620 kB VmRSS:	49124620 kB Threads:	40 
== thread 132406 cass state=S
[<0>] hrtimer_nanosleep+0xa2/0x170
[<0>] common_nsleep_timens+0x4e/0x90
[<0>] __x64_sys_clock_nanosleep+0xf4/0x180
[<0>] x64_sys_call+0x1c2c/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132459 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132460 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132461 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132462 asupersync-work state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132474 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132486 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 132496 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136471 cass-responsive state=S
[<0>] hrtimer_nanosleep+0xa2/0x170
[<0>] common_nsleep_timens+0x4e/0x90
[<0>] __x64_sys_clock_nanosleep+0xf4/0x180
[<0>] x64_sys_call+0x1c2c/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136473 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136474 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136475 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136476 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136477 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136478 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136479 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136480 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136481 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136482 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136483 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136484 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136485 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136486 cass-lexical-re state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136487 cass state=S
[<0>] futex_wait_queue+0x64/0xa0
[<0>] __futex_wait+0x155/0x1d0
[<0>] futex_wait+0x74/0x120
[<0>] do_futex+0x105/0x260
[<0>] __x64_sys_futex+0x12a/0x200
[<0>] x64_sys_call+0x111/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136488 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 136489 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 137874 cass-lexical-pa state=R
== thread 137875 cass-lexical-pa state=R
== thread 137876 cass-lexical-pa state=R
== thread 137877 cass-lexical-pa state=R
== thread 137878 cass-lexical-pa state=R
[<0>] srso_return_thunk+0x5/0x5f
== thread 137879 cass-lexical-pa state=R
== thread 137880 cass-lexical-pa state=R
== thread 137881 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 137882 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 137883 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 137884 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 137885 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 137886 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
== thread 137888 asupersync-work state=S
[<0>] ep_poll+0x342/0x390
[<0>] do_epoll_wait+0xdb/0x100
[<0>] do_compat_epoll_pwait.part.0+0xf/0x90
[<0>] __x64_sys_epoll_pwait+0x91/0x150
[<0>] x64_sys_call+0xd7e/0x25a0
[<0>] do_syscall_64+0x7f/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80

Also: confirmed on the cross-refs (#422 matches our post-kill search behavior; #426's ingest ledger + termination watermarks would break exactly the kill→full-reconcile cycle we hit), and thanks for the fsqlite async-api fix on main — we'll rebuild from main for the next round of testing.

[lifeofgurpreet]

Additional controlled retest against a large established archive:

• Candidate was built on Linux from v0.6.23 base 8844ba66e24d18c54ece51d06fd6fede6002a60e plus only 424765b3 (the sink-starvation patch); candidate SHA256 e25f0f963000ff76fd4564c1d0929d47a210a36764cd68271926e721b5626a94.
• Canonical archive is 35.67 GB / ~5.0M messages. I used one already-queued Codex transcript: 44,613,416 bytes, 27,850 JSONL records, largest line 141,932 bytes (so this is not a single multi-megabyte line). No source pull or full rebuild.
• index --watch-once <that one path> ran with CASS_INDEX_STALL_ABORT_SECS=0, read-witness cap 4096, one CPU, 12 GB cgroup cap, and a 1,800 s outer timeout. It entered watch-once, peaked at 1.13 GB RSS, then returned 124 at the outer bound; the durable queue remained 35/35 and the exact target was retained.
• Read-only lexical search through the released compatible reader remains healthy afterwards (one-result probe 2.16 s).

This means the 424 fix did not make this existing large-archive watch-once path bounded in our geometry. Because the target has many normal-sized records rather than one giant line, it may be a different large-conversation/re-ingest path from the retained-byte sink geometry. I will not retry or run a corpus rebuild. Is there a minimal compatible follow-up commit/configuration to test, short of moving the published archive to the current 0.7.x refactor?

[nskidan]

@lifeofgurpreet — a data point that may connect your run to finding 1 above (single-threaded per-conversation persist cost) rather than a distinct wedge: on our corpus, persisting one large codex transcript measured ~13 minutes for the ~70 MB class and 80+ minutes for a 251 MB one, on a full uncontended core, with cost growing worse than linear in size/record count. At one cgroup-limited CPU, a 44.6 MB / 27,850-record transcript not finishing inside 1,800 s sits on that same curve.

The discriminating signal from our runs: while inside one transcript, the process stays CPU-bound (~100% of its core) with conversation-level counters frozen but the per-message activity heartbeat ticking, and last_progress_at_ms in index-run.lock frozen the whole time. If your 1,800 s run showed that profile, it's likely finding 1's cost curve and would share the size-aware paging fix discussed above; if it sat near-idle CPU instead, that's a different (wedge-class) state worth its own capture.

(Caveat: we never tested --watch-once, and our memory-blowup variant — finding 4 — needs the parallel rebuild path with multiple workers, which your 1-CPU run wouldn't hit; your 1.13 GB RSS is consistent with that.)

[lifeofgurpreet]

That discriminator points away from finding 1 for this run. The canary cgroup had CPUQuota=100%; at elapsed ~94 s the process was only ~21.6% CPU and cgroup cumulative CPU was 31.0 s. At elapsed ~157 s, cumulative CPU was only 37.2 s. The main process was observed in hrtimer_nanosleep, while the child cass had S<l state. Peak cgroup memory was 1.13 GB.

The host was contended (CPU PSI ~43% at one sample), but this is still far below a continuously CPU-bound full core, and the cumulative CPU growth had flattened before the outer 1,800 s timeout. We did not enable CASS_PIPELINE_TRACE / robot ingest tracing, so I cannot recover the activity heartbeat or lock progress after termination. I will treat this as the distinct idle/wedge class and will not run another canonical attempt merely to collect tracing.

[nskidan]

Agreed — flattening cumulative CPU with the main thread in hrtimer_nanosleep and the workers parked is the idle/wedge profile, not the cost curve; your numbers settle it cleanly.

One possibly-useful connection for triage: in the first maintainer comment on this thread (2026-08-24), @Dicklesworthstone described a small-fixture reproduction that wedges in the ingest phase with zero lexical-pipeline activity — explicitly identified as a different bug from the sink-retention geometry fixed by 424765b3, tracked separately. A watch-once ingest of a single queued transcript going idle-wedged despite the sink fix sounds like it could be that bug rather than a third one — and would also explain why cherry-picking 424765b3 alone changed nothing in your geometry.

[ASRagab]

Reporting a second, distinct deadlock in the same StreamingByteLimiter that the sink-starvation fix (424765b3) does not close. It is a different mechanism from retained sink bytes, it reproduces on the v0.6.26 release binary, and reading current main it is still live there. The short version: on the steady budget, a page's byte request is exactly equal to the in-flight cap, so a page can be admitted only when bytes_in_flight is exactly zero.

Environment

• macOS 26.5 arm64, 12 cores. host_available_memory_bytes: 10701979648 (~10.0 GiB free at stall time).
• cass 0.6.26 (aa2023301a02), Homebrew release binary. I did not build 424765b3; the source analysis below is against current main (Cargo.toml version 0.7.1).
• Corpus: 5477 conversations, 3.9 GB canonical DB. No giant conversations — pages measured 12–18 MB of real message bytes for 32 conversations, so this is the opposite corpus profile from the original report.

The arithmetic

Three independent defaults compose into request == cap:

1. Steady commit_interval_message_bytes defaults to 512 MiB (lexical_rebuild_commit_interval_message_bytes).
2. That same value is passed as batch_fetch_message_bytes_limit — the flat per-page reservation. In lexical_rebuild_runtime_pipeline_budget_snapshot the steady budget is built with steady_commit_interval_message_bytes in both the byte-limit slot and the commit-interval slot.
3. The derived cap is batch_fetch_message_bytes_limit * (pipeline_channel_size + 1) = 512 MiB × 5 = 2.5 GiB, but scale_inflight_byte_limit then applies min(cfg.max_inflight_bytes), and default_max_inflight_bytes_for_available is clamp(available_memory / 32, 512 MiB, 16 GiB).

So the cap exceeds 512 MiB only when available_memory / 32 > 512 MiB, i.e. only on hosts with more than 16 GiB of available memory. Below that the clamp lands on the 512 MiB floor — exactly the per-page request. Then in acquire_with_wait:

let reservation = requested_bytes.min(max_bytes_in_flight);   // 512 MiB.min(512 MiB)
if state.bytes_in_flight.saturating_add(reservation) <= max_bytes_in_flight {

512 MiB + 512 MiB <= 512 MiB is false for any nonzero bytes_in_flight. The bounded 5 s re-arm re-evaluates the same false predicate forever.

This predicts the difference between the two corpora in this thread: on the 62 GiB Linux box available/32 ≈ 1.9 GiB, so request (512 MiB) < cap and three pages fit. On a ≤16 GiB host the steady budget self-deadlocks by construction, independently of conversation sizes.

Why the 424765b3 flush does not close it

The starvation flush fires on an idle tick when pipeline_rx.is_empty() and a waiter is parked. Draining the sink to zero rescues the case where the sink is the only holder. It does not help when a reservation is held by a page sitting in the ordered channel: the channel is then non-empty, so the flush does not fire, and the capped request still requires bytes_in_flight == 0.

This is also the one point where I think the fix comment's liveness argument does not hold as written — "acquisitions are budget-capped, so inflight drains until any capped request fits". A request equal to the cap does not fit at any nonzero in-flight level; it needs exactly zero, which nothing in the ordered pipeline guarantees.

Evidence

sample(1) on the wedged process, ~120 s after the last log line, 0.5% CPU. Every thread parked; one frame names the site:

### Thread_...: cass-lexical-page-prep-2
      app: StreamingByteLimiter::acquire_with_wait
     leaf: __psynch_cvwait
### Thread_...: cass-lexical-page-prep-0/1/5
      app: spawn_lexical_rebuild_page_prep_workers   (ordered turn)
     leaf: __psynch_cvwait
### Thread_...: cass-lexical-page-prep-3/4
      app: crossbeam_channel::flavors::array::Channel (recv, no work)
     leaf: semaphore_wait_trap
### Thread_... DispatchQueue_1: com.apple.main-thread
      app: run_index_with_data
     leaf: __semwait_signal            (nanosleep, 7222/7224 samples)

12 cass-lexical-rebuild-* writer-pool threads all in __psynch_cvwait. Matching status payload:

inflight_message_bytes=38562578  max_message_bytes_in_flight=536870912
pending_batch_conversations=0  queue_depth=0  budget_generation=0
controller_mode=pinned_steady  producer_budget_active_waiters=1

38 MiB in use against a 512 MiB cap, with a parked waiter — which is only possible if the request itself is ≥ the cap. That payload is the cleanest discriminator between this variant and the retained-bytes one: here in-flight is low, not near the cap.

What the startup budget looks like on the same host

For contrast, the startup budget uses the 128 MiB INITIAL_COMMIT_EVERY_MESSAGE_BYTES default, so request (128 MiB) < cap (512 MiB) and four pages fit. That path wedges in the retained-bytes way this issue already describes — my first two stalls were at 160/5477 and 3584/5477 with inflight pinned at exactly 536870912 and pending_batch_conversations of 128 and 236. So a ≤16 GiB host can hit both variants in one session depending on which budget is active.

One incidental note on the startup path: page_conversation_limit serves as both the page chunk size and the sink's flush threshold (pending_batch.len() >= current_batch_conversation_limit), and the controller rewrites it live. When the controller shrank it from the steady value back to 32 while 128 conversations were already pending, the batch was over-threshold but the threshold check only runs inside finish_conversation! on packet arrival — and no packet could arrive. Re-checking the flush predicate after a budget transition would close that particular ordering.

Suggested fix direction

The cap is derived from the per-page limit, then clamped independently, which breaks the invariant the derivation established. Re-establishing it after the clamp looks like the minimal change:

batch_fetch_message_bytes_limit = min(configured, max_message_bytes_in_flight / (pipeline_channel_size + 1))

applied after scale_inflight_byte_limit, so the per-page request can never reach the cap however the governor clamps it. A cheap independent guard would be a hard error at construction when batch_fetch_message_bytes_limit >= max_message_bytes_in_flight, since that configuration cannot make progress and currently fails silently as a hang.

Decoupling batch_fetch_message_bytes_limit from commit_interval_message_bytes seems worth considering separately — they are different quantities (a fetch guardrail vs. a commit fence) that currently share one value and one env var, so tuning either moves the other.

Working workaround for ≤16 GiB hosts

Contrary to "no workaround that's honest" earlier in the thread — true for the retained-bytes variant — this variant does have one, since it is purely a ratio problem. On the v0.6.26 release binary this took my rebuild from a hard stall at 160/5477 to 5477/5477:

CASS_TANTIVY_REBUILD_COMMIT_EVERY_MESSAGE_BYTES=33554432          # 32 MiB per-page reservation
CASS_TANTIVY_REBUILD_INITIAL_COMMIT_EVERY_MESSAGE_BYTES=33554432
CASS_TANTIVY_REBUILD_PIPELINE_MAX_MESSAGE_BYTES_IN_FLIGHT=2147483648   # 2 GiB
CASS_RESPONSIVENESS_MAX_INFLIGHT_BYTES=4294967296                 # lift the governor clamp
CASS_TANTIVY_REBUILD_BATCH_FETCH_CONVERSATIONS=32
CASS_TANTIVY_REBUILD_INITIAL_BATCH_FETCH_CONVERSATIONS=32
CASS_TANTIVY_REBUILD_CONTROLLER_MODE=steady

CASS_RESPONSIVENESS_MAX_INFLIGHT_BYTES is the load-bearing one and is easy to miss: without it, CASS_TANTIVY_REBUILD_PIPELINE_MAX_MESSAGE_BYTES_IN_FLIGHT is silently clipped to the governor ceiling. I set 6 GiB in an earlier attempt and the adopted-budgets log line still showed 512 MiB. The ratio is what matters — 32 MiB against 2 GiB gives 64 pages of headroom instead of one.

This also corroborates finding 1/4 from a small-conversation corpus: the inflight-byte budget does not bound resident memory here either. Peak RSS was 3.1 GB during page-prep and 7.9 GB during the fts5 publish, on a box with 10 GiB free — with no conversation larger than a few MB.

Two small things in the stall hint

The stall diagnostics hint points at #244 and #258, both closed, rather than this issue, and suggests sudo cat /proc/$(pgrep -f 'cass index')/stack and gdb, neither of which exists on macOS. sample <pid> 10 -f out.txt is the macOS equivalent and produced the capture above. Also worth noting for anyone reading a wedge on macOS: the no preparing progress for 120s warning fired repeatedly during a healthy fts5 publish running at 98% CPU, which matches the phase-aware-clock item already tracked.

Happy to test a patch on this corpus — it reproduces deterministically at 160/5477 with default settings on this host.

[lifeofgurpreet]

v0.7.1 / Quill compatibility follow-up from the established 35.67 GB legacy archive. This is a scratch-clone-only result; canonical DB, corpus, and published indexes were not modified.

Current main candidate: d959f035cc691e0559c4fa7d78664c196036fa9c (cass 0.7.1). A scratch clone contains the exact 35,673,329,664-byte canonical DB plus its existing v8/v9-derived state. Plain cass index --data-dir <scratch> --json never gets past phase=preparing before any discovery or lexical work.

There are two independent unbounded preparation paths before Quill can run:

1. bounded_canonical_db_corruption_probe is not bounded on this FrankenSQLite archive. Its SELECT id, conversation_id, role FROM messages ORDER BY id DESC LIMIT 1 premise is documented as an O(log n) B-tree edge read, but it read 8.4 GiB, reached a 6.0 GiB cgroup memory peak, and produced zero index progress in 5m56s CPU. This is a new pre-index regression, not an HNSW failure. It needs a genuinely bounded FrankenSQLite-safe edge probe, or a fail-closed bounded/observable alternative; the emergency malformed-DB bypass is not an acceptable production workaround.

2. With CASS_INDEX_ALLOW_MALFORMED_DB=1 set only in the scratch unit to isolate (1), it still remained at preparing, current=0, discovered_agents=0 for 85s, consuming 37.1 CPU-s and peaking at 1.0 GiB RSS. The path is nonresumable_pending_lexical_rebuild_status_from_readonly_db -> unconditional count_total_conversations_exact -> SELECT COUNT(*) FROM conversations. It performs that exact count before even checking the cheap checkpoint DB-path mismatch in the clone, and plain index has no preflight subphase breadcrumb or timeout. On the canonical path the v9 checkpoint count is older than the current corpus, so this exact count would still be required before Quill work begins.

Requested behavior: do not let routine incremental index silently spend unbounded work in phase 0. Check marker identity before any expensive count; use a persisted/cheap invariant where safe; when an exact legacy count is unavoidable, expose a named preflight phase, activity/bytes, deadline, and resumable outcome. The corruption probe needs the same treatment.

Acceptance test I can run: an isolated exact clone of this 35.67 GB archive must advance from preparation into named Quill lexical work under bounded CPU/memory, then produce a cold lexical query. Only after that should we consider semantic/HNSW; no live corpus migration or raw-source rebuild is needed for this test.