Profiling

Profiling workflows for finding bottlenecks across Python, PyTorch operators, CUDA kernels, memory allocation, and full CPU/GPU timelines.

LLM Compiler Profiling Qwen2.5-0.5B FP32 on RTX 5050 with CUDA Graphs disabled: prefill kernels 1,385 → 482 for 44 tokens, decode kernels 8,968 → 3,280 for eight steps, and decode latency 97.9 → 66.8 ms. Traced fusion and attention changes; maximum logit error 6.1 × 10⁻⁵.

GPU Kernel Benchmarking PyTorch and cuBLAS/cuBLASLt baselines, CUDA Graph replay, and 80/80 backend-shape correctness checks covering tails and KN/NK weight layouts. Profiling with PyTorch Profiler and Nsight Systems/Compute.

Attention Occupancy Analysis Nsight Systems/Compute profiling of FA2-style forward attention: 255 registers/thread and 8.3% occupancy at D=128, alongside correctness and SDPA performance comparisons.

Further Explorations

Nsight Systems System-wide CPU/GPU timelines, CUDA API traces, NVTX ranges, and utilization gaps

Nsight Compute Kernel-level metrics, occupancy, memory throughput, warp stalls, and source correlation

PyTorch Profiler Operator costs, CPU/CUDA activity, stack traces, tensor shapes, memory, and traces

CUDA Memory Profiling Allocator snapshots, fragmentation, peak memory, and non-PyTorch allocation debugging