Baseline — nine engines, one map

This is the inaugural release of the Inference Engines tracker. From here on, every week brings a fresh sourcing pass — version bumps, new features, status changes, and any newcomers worth watching — published as a dated post like this one.

The lineup

Nine engines, grouped into three tiers.

Server / production

• vLLM — v0.22.1 (Jun 5 2026). The broadest-ecosystem default: PagedAttention, the widest hardware coverage (NVIDIA, AMD, TPU, Trainium, Gaudi), and the backbone of most Kubernetes serving stacks. Recent work: DeepSeek V4 hardening with NVFP4 fused MoE, KV Offload + Hybrid Memory Allocator, and deprecation of Transformers v4.
• SGLang — v0.5.12.post1 (May 23 2026). The throughput leader for prefix-heavy traffic thanks to RadixAttention. Speculative Decoding V2 is now default; CUDA 13 + PyTorch 2.11 is the new build baseline.
• TensorRT-LLM — container 25.12. NVIDIA’s max-performance engine: highest ceiling, steepest setup, deep Triton integration. New B200 support and NVFP4 GEMM for Llama/Mixtral.
• LMDeploy — v0.13.0 (May 12 2026). TurboMind makes it the throughput champion for quantized serving — 4-bit runs ~2.4x faster than FP16. Added Qwen3.5 4-bit and Anthropic-compatible endpoints.

Local / edge

• llama.cpp — rolling (b87xx). Landed its largest rewrite since 2023 in April 2026: ~2.1x throughput on 70B quantized, backend-agnostic tensor parallelism, 1-bit quantization, and a Qualcomm Hexagon NPU backend.
• Ollama — rolling (early June 2026). The easiest local on-ramp, now with a Windows desktop GUI, cloud fallback for oversized models, and structured outputs.
• MLC-LLM — rolling. Compile-once, deploy-anywhere via Apache TVM — including in-browser (WebLLM/WebGPU) and on-device mobile.
• ExLlamaV3 + TabbyAPI — rolling. Best quality-per-bit on consumer GPUs (EXL3), often first to support brand-new quantized releases.

Legacy

• TGI — archived (read-only) March 21 2026 at v3.3.5. Hugging Face now points users to vLLM or SGLang. Tracked here only to flag the migration.

Benchmark snapshot

On H100, 2026 head-to-heads put SGLang (~16,215 tok/s) and LMDeploy (~16,132 tok/s) roughly 29% ahead of a fully-optimized vLLM (~12,553 tok/s). At ~1M requests/day that gap is worth about $15k/month in GPU spend — but vLLM still wins on ecosystem breadth and hardware coverage.

Starter watchlist

Three names being tracked but not yet promoted: llm-d (Kubernetes-native distributed vLLM serving), NanoFlow (research-grade intra-device parallelism), and DeepSpeed-MII (Microsoft’s serving layer, currently behind the leaders). Any that prove significant will graduate into the main list.

Next refresh runs Monday. Version numbers should always be confirmed against each project’s release page before you deploy.