Support multi-stream concurrency and PDL in roofline analysis

[AHEADer]

Summary

The current roofline analysis assumes a single-stream, sequential execution model: each operator runs in isolation, and the total runtime is the sum of per-operator max(compute_cycles, memory_cycles). This does not capture the performance benefits of multi-stream concurrency or NVIDIA's Programmatic Dependent Launch (PDL) mechanism.

Current Behavior

• Single-stream assumption: All operators are analyzed sequentially. The three SOL models (unfused, fused, fused+prefetched) differ only in memory accounting, not in execution parallelism.
• No inter-operator overlap: Independent operators that could run concurrently on different streams are still summed serially.
• No PDL modeling: Blackwell's PDL feature, which enables dependent kernels to overlap execution by launching the next kernel before the current one finishes, is not considered.

Proposed Enhancement

1. Multi-Stream Modeling

• Build an operator dependency graph (DAG) from the einsum graph's producer-consumer relationships.
• Identify independent subgraphs that can be scheduled on separate CUDA streams.
• Model the total runtime as the critical path through the DAG rather than the sum of all operators.
• Account for shared resource contention (e.g., memory bandwidth, SM occupancy) when operators run concurrently.

2. PDL (Programmatic Dependent Launch) Modeling

• For dependent operator chains, model the overlap enabled by PDL where a downstream kernel can begin execution before its predecessor fully completes.
• Reduce the effective kernel launch overhead in sequential chains.
• This is particularly relevant for Blackwell (B200) and later architectures.

Impact

For models with significant operator-level parallelism (e.g., multi-head attention, parallel branches in inception-style architectures), the current single-stream model may significantly overestimate runtime. Adding multi-stream and PDL support would provide tighter performance bounds.

[hqjennynv]

Thanks for raising these points, @AHEADer.

Both optimizations are indeed important to consider in SOL derivation, and we’ve carefully evaluated their implications in CUDA.

TL;DR: the current fused runtime SOL remains valid even with these two optimizations.

Detailed explanation:
All three SOL models currently use a single whole-graph roofline:

max(total_compute_cycles, total_memory_cycles)

rather than a sum of per-op rooflines.

The three models differ only in memory accounting, specifically, which tensors are counted as DRAM traffic. Additionally, in the current release, fused and fused_prefetched produce identical results right not.

The originally intended distinction between per-op summation vs. whole-graph modeling is deprecated. We’ve updated SOL_GUIDE.md to reflect this.

Using a whole-graph roofline (i.e., max(Σ compute, Σ memory) instead of Σ max(computeᵢ, memoryᵢ)) implicitly assumes perfect overlap between compute-bound and memory-bound operations. This aligns with an idealized multi-stream execution model.

While this SOL is valid, it might not capture several important real-world behaviors for tighter bounds:

• DAG critical-path scheduling under dependency constraints
• Resource contention when operations overlap (e.g., shared SMs or memory bandwidth)
• PDL launch-gap reductions in dependent execution chains
• Per-op bottleneck diversity, where compute- and memory-bound ops interleave during execution