Inference Optimization

Techniques for improving serving latency, throughput, memory efficiency, time-to-first-token, and sustained decode performance.

Continuous Batching & Chunked Prefill Token-budget scheduling in nano-vLLM with TTFT/TPOT/KV telemetry. Output throughput increased from 58 to 223 tokens/s (3.84×) versus single-sequence scheduling on 12 mixed-length Qwen3-0.6B requests.

Quantized Loading & Storage NF4 reduced loaded CUDA allocation from 1.11 to 0.50 GiB versus BF16. Evaluated INT8 loading and INT8/packed-INT4 storage prototypes, keeping memory measurements distinct from execution speedups.

Paged KV Cache & Prefix Reuse Paged KV tables, refcounts, and parent-hash prefix caching audited over 1,500 seeded operations. Warm 256-token prefix reuse avoided 3,072 prefill tokens and reduced median TTFT from 139 to 34 ms; all 12 greedy outputs matched.

Compiled LLM Inference Qwen2.5-0.5B FP32 eight-step decode improved from 97.9 to 66.8 ms with CUDA Graphs disabled. Analyzed generated fusion and attention changes, checked maximum logit error of 6.1 × 10⁻⁵, and documented a separate compiled MLP latency regression.

Edge Precision & Accuracy Tradeoffs MobileCLIP2-S2 FP16 deployment on Qualcomm QNN with device-output validation. Rejected calibrated W8A16 QDQ after image cosine similarity fell to 0.811, preserving retrieval accuracy as a deployment constraint.

Further Explorations

Speculative Decoding Draft-and-verify decoding, EAGLE-style methods, and acceptance-rate tuning

Disaggregated Serving Separating prefill and decode paths for scalable, latency-aware inference