Week of Jun 15 — Dynamo Snapshot, Ollama Hermes Desktop, TokenSpeed on the watchlist

A quieter version week — no major stable bumps from the flagship server engines — but several notable developments worth tracking.

Engine updates

vLLM — confirmed at v0.22.1 (Jun 5 2026) via PyPI, unchanged. Active dev continues toward v0.23.x (multimodal TTS adapter framework, DeepSeek-V4 Mega-MoE EPLB hardening) but no new stable release landed this week.

SGLang — confirmed at v0.5.12.post1 (May 26 2026) via GitHub (still tagged Latest). The web reports Docker images for a v0.5.13 tag appeared Jun 14, but it isn’t yet marked as a stable GitHub release; holding at post1 until confirmed.

LMDeploy — holds at v0.13.0 (May 12 2026); documentation was refreshed on Jun 11 but no new release.

TensorRT-LLM — still rolling container 25.12. Noteworthy: a TensorRT-LLM 1.3.0rc15.post1 build (with DeepSeek-V4-Pro support — MLA, EPLB, NVFP4) is shipping inside Dynamo dev builds this week.

NVIDIA Dynamo — stable version remains v1.1.1, but two notable developments:

• Dynamo Snapshot (released early Jun 2026): a CRIU-based checkpoint/restore system for AI inference workloads on Kubernetes. A GPU Memory Service (GMS) handles asynchronous weight restoration from NVMe SSDs — cutting cold-start time for single-GPU workloads (e.g. Qwen3-0.6B) to under 5 seconds, and delivering a 21x speedup for large models like GPT-OSS-120B versus standard Kubernetes cold starts.
• v1.3.x dev cycle: pre-release builds this week added model-specific deployment recipes for DeepSeek-V4-Pro (TRT-LLM backend), Nemotron-3 Ultra and Super (vLLM backend), and Kimi K2.6 (vLLM backend). All are pre-release/experimental only.

llama.cpp — advanced to build b9631 (Jun 14 2026); still rolling. No architectural changes — routine build cadence.

Ollama — Jun 12 2026 release brings:

• Hermes Desktop native UI support
• /api/show response caching — 6.7x median latency improvement for integrations (VS Code, etc.) that call this endpoint repeatedly
• Gemma 4 MTP speculative decoding on Macs — over 2x speedup for Gemma 4 31B on coding tasks

MLC-LLM, ExLlamaV3 + TabbyAPI — no material change this week.

TGI — still archived/read-only since Mar 21 2026. No change.

Watchlist

Added — TokenSpeed (LightSeek Foundation): a speed-of-light LLM inference engine specifically targeting agentic and long-context workloads (50K+ token contexts, multi-turn conversations). The project claims one of the fastest MLA implementations on Blackwell and demonstrated 580 TPS on Qwen3.5-397B-A17B in May 2026. Credibility signal: its EAGLE-3 MLA kernels were integrated directly into SGLang v0.5.12 as an attention backend on SM100/B300. Still pre-production as of June 2026, with production hardening described as ongoing. Worth watching for a stable release.

llm-d — remains on the watchlist at v0.5 with an explicitly unstable API. No new release; the CNCF Sandbox status and Red Hat AI Inference deployments continue to build credibility, but still holding for a stable 1.0.

Aphrodite Engine, NanoFlow, DeepSpeed-MII — no new signal; retained as-is.

Benchmark notes

The H100 throughput picture is unchanged this week: SGLang (~16,215 tok/s) and LMDeploy (~16,132) sit ~29% ahead of vLLM (~12,553). Dynamo Snapshot’s gains are cold-start and orchestration-level, not single-GPU throughput. No new head-to-head benchmarks published this week.