Transformers

Transformer projects covering from-scratch implementations, PyTorch models, attention mechanics, architecture variants, and efficiency-focused experiments.

Triton FlashAttention-2: Forward Attention FA2-style forward attention with online softmax, FP16 MHA/GQA at D=32/64/128, causal tile pruning, and masking for partial sequence tiles. Passed 60/60 CUDA cases against PyTorch SDPA with maximum absolute error 2.44 × 10⁻⁴. On RTX 5050 at causal B=1, H=16, N=511, D=64, measured 1.15×/1.13× versus eager/torch.compile SDPA using the default backend.

Further Explorations

Transformer in NumPy Tokenization, embeddings, attention, MLP blocks, layer norm, and training loop basics

Transformer in PyTorch Clean encoder/decoder implementation with modular attention and training utilities

Attention Variants MHA, MQA, GQA, sliding-window attention, rotary embeddings, and KV-cache-aware decoding

Efficient Architectures Long-context, sparse, linear-attention, and memory-efficient transformer experiments