GPU Inference Performance

This repository contains three PyTorch GPU inference performance notebooks.

The exercises cover:

• Roofline analysis for memory-bound vs compute-bound GPU kernels
• CUDA-event timing and arithmetic intensity measurement
• Autoregressive decode-loop profiling
• KV-cache based decode optimization
• torch.compile graph capture and graph-break cleanup
• Manual CUDA graph capture/replay for repeated fixed-shape decode steps

The notebooks were executed on an NVIDIA H100 GPU.

Notebooks

1. hw1_roofline_completed.ipynb
   Measures low- and high-arithmetic-intensity workloads and plots them on a roofline model.

2. hw2_decode_optimization_completed.ipynb
   Profiles a slow greedy decode loop and optimizes it using KV cache, reduced host synchronization, and bf16 timed generation.

3. hw3_compile_cuda_graphs_completed.ipynb
   Measures kernel launch overhead, fixes graph breaks for torch.compile, and compares eager execution with manual CUDA graph replay.

Main results

• Estimated CUDA launch overhead: ~6.09 µs
• HW2 optimized decode speedup: ~2.10× vs baseline
• HW3 manual CUDA graph replay speedup: ~3.50× vs eager

Environment

• Python
• PyTorch
• CUDA
• NVIDIA H100 GPU
• Jupyter Notebook