Skip to content

LRU eviction lets a dynamic resolution re-lane a target (the dynamic twin of #1582) #1599

Description

@Rotorsoft

Defect

The dynamic-target twin of #1582. Evicting a target's record from the subscription LRU makes it read as never-seen, and a never-seen target's first resolution wins — so a later, lower-priority resolution silently re-lanes a target that was already carrying a different lane.

libs/act/src/internal/correlate-cycle.ts:688-695

const recorded =
  this._static_subscriptions.get(resolved.target) ??
  this._dynamic_subscriptions.get(resolved.target);
const priority = resolved.priority ?? 0;
const upgraded = !recorded || priority > recorded.floor;
const carried = upgraded
  ? { priority, lane }                                   // this resolution's OWN lane
  : { priority: recorded.priority, lane: recorded.lane }; // the carried-forward lane

subscribe then applies existing.lane = lane unconditionally (priority is merged with bump_priority = max, so priority is safe; the lane is not).

#1582 fixed exactly this hazard for static targets by moving them into their own unbounded map — the comment at :686-687 even names the mechanism: "Statics are consulted first and from their own map: the LRU can evict, and a missing record reads as never-seen." Dynamic targets still live in the evicting LRU and still have the problem.

The LRU is documented as "a memory bound, not a correctness mechanism". That is true for a dynamic target's priority, and false for its lane.

Failure scenario

Two dynamic reactions resolving to one target at different priorities, with maxSubscribedStreams small enough that other targets evict the record between resolutions.

CONTROL  maxSubscribedStreams: 50   →  { lane: "fast", priority: 10 }
RED      maxSubscribedStreams: 1    →  { lane: "slow", priority: 10 }

CONTROL  onlyLanes: ["fast"], bound 50  →  ran ["hi", "hi", "lo"]
RED      onlyLanes: ["fast"], bound 1   →  ran ["hi", "hi"]        <-- "lo" starved

Under lane sharding the mis-laned stream is simply never drained by the worker that owns its declared lane.

Recovery is poor: it repairs only if a later higher-priority resolution happens to hit the same target, and a restart does not re-subscribe dynamic targets — so with no further high-priority event the stream stays mis-laned indefinitely.

Proof (red)

Control/red verdicts above are the wave-21 eviction hunter's, from a probe it ran and then deleted before going idle. I verified the root-cause mechanism in source (the three lines above, plus subscribe's unconditional existing.lane = lane) but did not re-run the probe in the main loop — so this one is filed on a source-confirmed mechanism plus the hunter's numbers, not on my own red. Worth re-establishing a red test as the first step of the fix.

Contract violated

Same one #1582 falsified for statics — the #1487 behavior-contracts row, "a mark never re-lanes or downgrades what it rides along with". #1582 closed the static half; the dynamic half is still open.

There is a second consequence worth noting in the fix: because eviction makes the target read as never-seen, it also suppresses the #1567 lane-conflict report for that target — the diagnostic that would otherwise surface the disagreement is the thing eviction removes. (See the related dynamic-guard normalization gap filed alongside this.)

Fix direction

The static fix (an unbounded side map) is not available here — dynamic targets are unbounded by nature, which is why the LRU exists. Options:

  1. Make subscribe merge the lane rather than overwrite it — i.e. give the lane the same "first assignment wins unless outranked" treatment priority already gets via bump_priority, resolved against the durable row rather than process-local memory. Puts the correctness in the durable state, where eviction cannot reach it.
  2. Treat an evicted record as unknown rather than as never-seen — read the current lane back from the store before deciding upgraded. Correct, but adds a read to the correlate hot path.
  3. Accept the re-laning and make it loud — report it, matching how the wave-19 family decided dynamic disagreements are reported rather than corrected. Cheapest, but it leaves the starvation.

Option 1 looks closest to the existing design (it is what priority already does), but this is a genuine design choice rather than a mechanical fix.

Found by debug wave 21 (bounded-collection eviction lens).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:sharedShared librariesbugSomething isn't workingpriority:mediumNice-to-have improvements

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions