A comprehensive educational resource highlighting the challenges and solutions of developing Triton kernels. This collection of notebooks demonstrates common pitfalls in parallel GPU programming and provides practical solutions for achieving efficiency and numerical correctness.
- Single-Kernel Global Sum: Using
tl.sum()incorrectly for global reductions - Atomic Operations Pitfalls: How autotuner can accumulate incorrect results
- Race Conditions: Multiple blocks writing to the same memory location
- CUDA-enabled GPU (or ROCm for AMD GPUs)
- Python 3.8+
- CUDA Toolkit or ROCm installation
We welcome contributions that improve the learning experience! Here's how to help:
- Bug fixes: Corrections to existing code or documentation
- New misadventures: Additional examples of common Triton kernel pitfalls and failures
- Documentation: Improvements to explanations of GPU programming concepts
- Performance improvements: Optimizations to existing kernels
- Fork the repository
- Create a feature branch:
git checkout -b feature/your-feature - Make your changes and test thoroughly
- Submit a pull request with a clear description
- For new misadventures: Focus on common Triton kernel mistakes that produce incorrect results or fail in non-obvious ways
- Include the "why": Explain what makes each misadventure a realistic trap for developers
- Show the failure: Demonstrate incorrect output, performance issues, or compilation errors
- Provide the fix: Include the correct approach with clear explanations
- Test thoroughly: Ensure all kernels produce correct results against PyTorch/NumPy equivalents
- Add clear comments: Explain GPU-specific concepts and Triton language features
- Include performance notes: Mention optimization considerations where relevant