The Inference Engine Room

Start with vLLM. It's the safest default — the broadest model and hardware coverage (NVIDIA, AMD, TPU, Trainium, Gaudi), the largest contributor base, and the backbone of most Kubernetes serving stacks. You give up ~29% peak throughput versus SGLang/LMDeploy, but you gain the deepest ecosystem and the least chance of hitting an unsupported edge case.

Go SGLang or LMDeploy — both land around ~16,200 tok/s on H100, roughly 29% ahead of vLLM (~12,500). That gap is real money: about $15k/month saved when serving ~1M requests/day. If you're willing to invest in a build/tuning step for the absolute ceiling, TensorRT-LLM goes higher still.

SGLang, decisively. Its RadixAttention automatically reuses the KV cache for shared prefixes (system prompts, retrieved context, conversation history), delivering up to 6.4x on prefix-heavy traffic. The more your requests overlap, the bigger the win — this is the workload SGLang was built for.

LMDeploy. Its TurboMind engine is the throughput leader for quantized serving — 4-bit inference runs ~2.4x faster than FP16, and it adds TurboQuant KV-cache quantization on top. Especially strong for InternLM/Qwen families and cost-optimized endpoints. Keep it patched — it had a recently-exploited CVE.

TensorRT-LLM has the highest ceiling on NVIDIA hardware, with first-class support for the newest features (NVFP4, Blackwell/B200) and tight Triton Inference Server integration. The trade-off is the steepest learning curve and an engine build/compile step — worth it for latency-critical, well-defined deployments, less so for fast-moving experimentation.

NVIDIA Dynamo. It's an orchestration layer that sits above vLLM, SGLang, or TensorRT-LLM — not a replacement for them. It splits prefill and decode into independently-scaled GPU pools, adds KV-aware routing to skip redundant prefill (~2x faster TTFT), tiers the KV cache from GPU down to CPU/SSD/object storage, and uses an SLA-driven Planner to autoscale at minimum TCO. For a Kubernetes-only, vLLM-centric alternative, llm-d (now in the CNCF Sandbox) is the one to watch. Both are overkill on a single node — reach for them when one box stops being enough.

Ollama if you want the lowest-friction experience — one command to pull and run, an OpenAI-compatible API, and now a desktop GUI plus cloud fallback for models too big to run locally. Drop down to llama.cpp directly when you want full control, edge hardware (Apple Metal, AMD CDNA4, Qualcomm Hexagon), or to embed inference in your own app — Ollama is built on it.

MLC-LLM / WebLLM. Its compile-once, deploy-anywhere model (via Apache TVM) is unmatched for breadth — run on WebGPU in a browser, on iOS/Android, or on embedded hardware, with no server needed. The compilation workflow is more involved than Ollama, but nothing else covers the browser + mobile space as well for privacy-first, no-backend apps.

ExLlamaV3 + TabbyAPI. The EXL3 format gives the best quality-per-bit on consumer NVIDIA GPUs, with tensor/expert parallelism so you can spread a large model across a multi-GPU rig. It's also often first to support brand-new quantized releases (it was the only stack for some Qwen3.5 quants), and TabbyAPI gives you an OpenAI-compatible endpoint for personal or small-team use.

Plan a migration. TGI was archived (read-only) in March 2026 and is no longer developed — Hugging Face itself now points users to vLLM or SGLang, both of which adopted the transformers-architecture approach TGI pioneered. Move to vLLM for the safest general-purpose path, or SGLang if throughput is your priority.