Resources

Curated learning resources for GPU programming, MLIR, ML systems, inference optimization, and performance engineering. Each section is organized as its own learning track so compiler, kernel, and systems resources stay easy to scan.

CUDA / Triton / CuTe GPU Programming A structured path through GPU architecture, CUDA fundamentals, Triton, CUTLASS, CuTe DSL, GEMM, attention kernels, PTX, profiling, and modern ML systems performance work. CUDA Triton Performance

Compilers / IR / Lowering MLIR A segmented MLIR path covering official docs, hands-on tutorials, blogs, videos, compiler-stack case studies, papers, and research/social posts. Dialects Lowering IREE

GPU Programming

Start Here

GPU Performance Engineering Resources A broad curriculum for learning GPU performance engineering from fundamentals toward frontier AI-lab kernel work

Modern GPU Programming for ML Systems MLC course-style notes for writing modern GPU programs in the context of ML systems

Basic Facts About GPUs A concise mental model for GPU architecture, execution, memory hierarchy, and practical performance constraints

Modal GPU Glossary Quick reference for GPU terms, architecture vocabulary, serving concepts, and accelerator terminology

CUDA Foundations

Nick CUDA Crash Course Fast practical introduction to CUDA programming and kernel structure

Udacity Intro to Parallel Programming Classic foundations for parallel algorithms, CUDA kernels, scans, reductions, sorting, and GPU thinking

NVIDIA Modern CUDA C++ Programming Class Official NVIDIA CUDA training material covering modern CUDA C++ programming patterns

OLCF CUDA Training Series HPC-oriented CUDA lessons from Oak Ridge, useful for serious systems and performance practice

Cornell Virtual Workshop: CUDA Concepts Readable CUDA concept notes for kernels, grids, blocks, memory, and execution hierarchy

Triton, CuTe, CUTLASS, and DSLs

Triton GPU Kernels 101 Beginner-friendly Triton path with examples for writing custom ML kernels in Python

Triton Kernels Playlist Common deep learning operators implemented as Triton kernels

GPU Mode Lecture: CUTLASS CUTLASS concepts, abstractions, and how production GEMM kernels are structured

Learning CUTLASS the Hard Way Deep dive into understanding CUTLASS by working through its abstractions and performance model

Getting Started with CuTe DSL Practical entry point for NVIDIA CuTe DSL and layout-driven kernel programming

CuTe Layouts Notes on CuTe layout algebra, a key idea behind modern CUTLASS and CuTe DSL kernels

GEMM and Tensor Cores

How to Optimize a CUDA Matmul Kernel Canonical step-by-step CUDA SGEMM optimization path toward cuBLAS-like performance

Inside NVIDIA GPUs: High-Performance Matmul Kernels Architecture, PTX/SASS, warp tiling, asynchronous pipelines, and tensor-core matmul intuition

Modern GPU Matmul Optimization Register tiling, vectorized loads, shared-memory staging, cp.async, TMA, warp specialization, split-K, and tensor cores

Advanced Matrix Multiplication Optimization CUDA SGEMM optimization with inline PTX, asynchronous memory copies, double buffering, and bank-conflict avoidance

Tensor Cores for ML Systems Modern tensor-core programming concepts for ML workloads

Matrix Multiplication on Blackwell: Breaking SOTA Blackwell matmul optimization case study with cluster launch control and state-of-the-art performance targets

Attention and LLM Inference Kernels

FlashAttention from the Bottom Up Low-level explanation of FlashAttention and why memory movement dominates attention performance

FlashAttention in Triton from First Principles Derives and implements FlashAttention with Triton, including the reasoning behind the algorithm

CUTLASS and FlashAttention 3 GPU Mode lecture on CUTLASS and FlashAttention 3 implementation ideas

How FlashAttention 4 Works Walkthrough of newer FlashAttention ideas and reverse-engineering attention performance

PyTorch FlexAttention PyTorch-native attention flexibility and performance for custom attention masks and patterns

Making Dr GRPO Go Brrr Fused decode-attention kernel work in an RL training loop, with practical microbenchmarking context

PTX, SASS, and Hardware Internals

A Gentle Introduction to CUDA PTX CUDA compilation pipeline, PTX, SASS, and a hand-written PTX kernel playground

Tensor Core Intrinsics: PTXAS Reverse Engineering Notes Reference notes for tensor-core intrinsics, PTXAS behavior, register allocation, and instruction scheduling

Programming Hopper GPUs: Memory Consistency Model Hopper-era memory consistency and synchronization details for advanced kernel work

NVFP4 Explained Blackwell low-precision floating point, microscaling, and hardware-algorithm co-design

NVFP4 Blockscaled GEMM with CUTLASS CUTLASS tutorial for Blackwell-class NVFP4 blockscaled GEMM kernels

Profiling and Performance Analysis

Introduction to Performance Analysis for NVIDIA GPUs Profiler-driven workflow for identifying common GPU performance bottlenecks

NHR PerfLab Seminar Series Performance engineering talks across GPU programming, profiling, and HPC optimization

CUTLASS GEMM Performance Measurement Guidelines Benchmarking methodology for GEMM kernels, measurement hygiene, and comparing against library baselines

Kernel Fusion in CUDA How fusion reduces memory traffic and launch overhead, with practical CUDA optimization patterns

Deep Dives and Reference Collections

The Feynman GPU Lectures A bottom-up GPU masterclass from transistors and CUDA cores to Tensor Cores, Hopper, and Blackwell

Bob Crovella: NVIDIA Technical Blog Author archive with practical CUDA explanations, performance notes, and NVIDIA platform guidance

The CUDA Handbook Book-style CUDA resource for building a durable understanding of GPU programming

CUDA Notes Compact notes and examples for CUDA programming concepts

MLIR

Official Docs, Repos, and Reference Material

MLIR Related Publications Official MLIR publication index for the compiler framework, dialects, lowering, and research lineage

IREE Retargetable MLIR-based machine learning compiler and runtime toolkit

IREE MLIR Linalg Tutorial IREE learning material for MLIR Linalg and compiler lowering paths

MLIR Sparsifier Google guide for sparse tensor compilation concepts and MLIR sparsifier workflows

ONNX-MLIR Representation and reference lowering of ONNX models in MLIR compiler infrastructure

ONNX-MLIR GitHub Source repository for ONNX model lowering, compiler passes, and runtime integration

Tutorials and Hands-On Learning

MLIR: Getting Started Jeremy Kun’s beginner-friendly entry into MLIR concepts, IR reading, and compiler toolchain setup

MLIR for Beginners Tutorial Hands-on GitHub tutorial for building intuition around MLIR dialects, lowering, and passes

Last Week Notes: MLIR Compact MLIR notes useful for quickly revisiting syntax, concepts, and compiler structure

MLIR Optimization Passes Developer-oriented guide to writing and reasoning about MLIR optimization passes

TVM and MLIR Learning Notes Learning repository for connecting TVM, MLIR, tensor compilers, and lowering ideas

Blogs and Articles

MLIR: A Compiler Infrastructure for the End of Moore’s Law Cornell CS6120 article explaining MLIR’s motivation, dialect design, and role in heterogeneous computing

What About the MLIR Compiler Infrastructure? Modular article on MLIR’s role in AI compiler infrastructure and portable accelerator stacks

Increasing Development Velocity of Giant AI Models Modular compiler-engineering case study on scaling large AI model development

Increasing Development Velocity of Giant AI Models, Part 2 Follow-up on MLIR attribute design, model weights, and compiler infrastructure ergonomics

Learn by Doing: TorchInductor Reduction Kernels Walkthrough of TorchInductor reduction kernel generation and compiler-produced GPU code

FlyDSL: MLIR-Native GPU Kernel Development AMD ROCm article on a Python-first, MLIR-native DSL for expert GPU kernel development and tuning

Accelerating Long-Context Model Training in JAX and XLA NVIDIA technical blog on compiler/runtime techniques for long-context training in JAX and XLA

FTL Model Compiler Framework Nuro article on an ML compiler framework and production compiler design concerns

Videos and Talks

MLIR Video Playlist Playlist of MLIR talks and tutorials for building compiler infrastructure intuition

MLIR Beginner-Friendly Tutorial 2: Part 1 Alex Singer tutorial continuing MLIR motivation, IR reading, dialect lowering, and optimization workflows

MLIR Beginner-Friendly Tutorial 2: Part 2 Follow-up Alex Singer tutorial with more beginner-friendly MLIR lowering and optimization detail

Deep Dive into Cruise’s ML Compiler NVIDIA GTC talk on building an extensible modular ML compiler using open-source compiler technologies

ONNX and Python to C++ Graph Compilation Presentation on graph compilation for edge AI, ONNX, Python frontends, and C++ code generation

Papers and Research

MLIR-Based Compiler Research Paper Compiler paper saved from the MLIR channel for deeper reading on MLIR-style compiler systems

From Loop Nests to Silicon: MLIR-AIR Research on mapping AI workloads onto AMD NPUs with MLIR-AIR, explicit data movement, and spatial architectures

Scorch Research PDF from the MLIR resource channel for compiler and accelerator systems reading

Weight Block Sparsity Training, compilation, and AI engine accelerator resource focused on structured sparsity