fix(core): survive heterogeneous NUMA topologies in RDMA fetch

[xiaguan]

Problem

In a cluster mixing machines with different NUMA node counts (e.g. 2-node and 6-node hosts), the RDMA fetch path broke down in two compounding ways when a small-topology machine fetched blocks from a large-topology peer:

1. Eviction storm. A remote slot advertising a NUMA node with no local pool made StorageEngine::allocate enter its reclaim loop with largest_free_allocation_for_node() == 0 forever — the loop's only remaining exit was an empty read cache. Every failed fetch attempt evicted the machine's entire resident cache and flooded the MetaServer removal queue, which then cascaded into cluster-wide cache churn.

2. Guaranteed fetch failure. The advertised numa_node was used verbatim as a key into local pinned pools, so failed to allocate slab for NUMAx aborted the whole batch. On heterogeneous clusters, cross-topology fetch could never succeed.

Observed in production; the slab-allocation failures and full-cache eviction storms were confirmed live before p2p was disabled as a stopgap.

Fix

Guard (storm killer): StorageEngine::allocate now rejects requests that are provably unsatisfiable — larger than what the target pool could hold when completely empty (missing NUMA pool → 0, or size above per-shard capacity) — before any reclaim, with an error log and a new pegaflow_pool_alloc_unsatisfiable counter. The capacity bound is a construction-time constant read (allocatable_bytes), no allocator locks on the hot path.

Root cause (topology resolution): the fetch path now treats the advertised NUMA node as a placement hint, not a pool key. resolve_block_numas rewrites each slot onto local topology right after QueryBlocksForTransfer:

• local node → kept;
• foreign node id → deterministic spread by id modulo local nodes;
• UNKNOWN (no hint) → round-robin by slot position, so a batch spreads across pools instead of collapsing onto one node;
• no local NUMA pools (global allocator) → UNKNOWN.

Rebuilt SealedBlocks carry the resolved local node, so re-serving fetched blocks advertises real topology to peers (and resolution is idempotent on the next hop). Remap counts appear in the existing RDMA fetch summary log line (numa_remapped=N) and a new pegaflow_rdma_fetch_numa_remapped counter — expected steady state on heterogeneous clusters, so no per-fetch warn.

Also fixes the duplicate warn import in storage/mod.rs that broke pegaflow-core builds without the rdma feature.

Scope note

The guard closes the deterministic full-cache wipe (allocation that can never succeed). Reclaim can still evict deeply when a request is satisfiable in principle but the pool is fragmented or pinned by inflight/still-referenced blocks — out of scope here.

Tests (TDD)

All new tests were written first and observed failing on the old code:

• alloc_for_missing_numa_pool_fails_fast_without_evicting — the incident scenario: NUMA pool only for node 0, allocation for node 3 must fail without touching resident cache (previously wiped it).
• alloc_larger_than_node_capacity_fails_fast_without_evicting — oversized request, same contract.
• resolve_block_numas_maps_foreign_nodes_onto_local_topology — keep/remap/idempotency.
• resolve_block_numas_spreads_unknown_slots_across_local_nodes — UNKNOWN batches must not pile onto one node.
• resolve_block_numas_without_local_pools_collapses_to_unknown — global-allocator receiver.

Verified: cargo test -p pegaflow-core passes with cuda-13,rdma (128) and cuda-13 without rdma (123, previously did not compile); workspace clippy clean on both combos.

🤖 Generated with Claude Code