Inference runtime systems and serving engines for efficient model deployment.
nano-vLLM: Single-GPU LLM Inference Runtime Token-budget continuous batching, chunked prefill, and OpenAI-style serving with TTFT/TPOT/KV telemetry. Paged KV tables, refcounts, and parent-hash prefix caching were audited over 1,500 seeded operations. On 12 mixed-length Qwen3-0.6B requests, output throughput rose from 58 to 223 tokens/s versus single-sequence scheduling; all 12 greedy outputs matched in the prefix-reuse experiment.
vLLM Paged attention and high-throughput LLM serving
SGLang Structured generation and serving workflows
TensorRT Optimized inference engines
TensorRT-LLM LLM-specific serving and engine optimization