Snapshot · 2026-06-14 · read the note ↗
The Inference Silicon Room
A weekly read on the chips powering AI inference — GPUs, wafer-scale, dataflow LPUs, inference ASICs, and RISC-V.
Architecture comparison
Side-by-side specs at a glance. Click any card for full detail. Use tabs to filter by architecture type.
| Chip | Architecture | Process | Memory / BW | Peak Compute | Status | Best for |
|---|---|---|---|---|---|---|
| NVIDIA H100 SXM5 NVIDIA | GPU · Datacenter | TSMC 4N (4nm-class) | 80 GB HBM3 · 3.35 TB/s | 989 TFLOPS FP16 sparse | GA | Production LLM inference where ecosystem and tooling matter most |
| NVIDIA B200 / GB200 NVIDIA | GPU · Datacenter | TSMC 4NP (4nm) | 192 GB HBM3e (per B200) · 8.0 TB/s | 4.5 PFLOPS FP4 | GA | Frontier model serving (70B–405B parameters) |
| AMD Instinct MI325X AMD | GPU · Datacenter | TSMC 5nm + 6nm (HBM package) | 288 GB HBM3e · 5.3 TB/s | 1,307 TFLOPS FP16 sparse | GA | Very large model inference where memory capacity trumps throughput |
| Cerebras WSE-3 Cerebras Systems | Wafer-Scale Engine | TSMC 5nm | 44 GB · 21 PB/s | 125 PFLOPS AI | GA | Ultra-low-latency inference for interactive applications |
| Groq LPU (Groq 3) Groq (now NVIDIA) | LPU · Dataflow | TSMC 4nm (estimated) | ~230 MB per chip | Sub-ms first-token latency | GA | Real-time voice and conversational AI (latency-critical) |
| SambaNova SN50 RDU SambaNova Systems | RDU · Reconfigurable Dataflow | TSMC 3nm (estimated) | — | 5× faster than competitive chips (claimed) | Sampling | Agentic AI systems with multiple models and rapid context switching |
| Etched Sohu Etched AI | Transformer ASIC | TSMC 3nm | — | 500,000 tokens/sec (claimed) | Pre-GA | High-volume transformer inference where the model architecture is frozen |
| Tenstorrent Blackhole Tenstorrent | RISC-V · Open Silicon | TSMC 6nm | 32 GB LPDDR5 (n150), 128 GB LPDDR5 (n300 dual) | Up to 768 TFLOPS BF16 | Sampling | Research teams wanting full hardware transparency and custom kernel development |
| Fractile (unnamed chip) Fractile | SRAM Inference ASIC | TBD (TSMC, likely 3nm) | — | 100× faster than GPU (claimed) | Pre-GA | If delivered: highest priority for frontier model API providers (Anthropic, OpenAI) |
| d-Matrix Corsair d-Matrix | In-Memory Compute ASIC | TSMC 12nm | 128 GB on-chip DRAM + compute | 100+ GB DRAM with co-located compute | Sampling | Large model (70B+) inference where capacity and efficiency both matter |
| Google Trillium (TPU v6) Google DeepMind / Cloud | Cloud-Only ASIC | TSMC 5nm (estimated) | HBM | 4.7× ML compute vs TPU v4 | GA | Gemini API workloads on Google Cloud |
| AWS Trainium2 / Inferentia3 Amazon Web Services | Cloud-Only ASIC | TSMC 3nm (Trainium2) | HBM3e (capacity not disclosed) | 3.7× throughput vs Trainium1 | GA | Amazon Bedrock API inference (Claude via Anthropic, Llama, Titan) |
Which chip should I use?
Quick decision guide by workload and requirements.
I need the highest tokens-per-second for batch inference at scale
Cerebras WSE-3 or SambaNova SN50 for the absolute throughput ceiling. WSE-3 runs trillion-parameter models 6.7× faster than the best GPU cloud. SN50 is optimised for agentic workloads with high-throughput continuous batching. NVIDIA B200 NVL72 is the safe GA choice.
I need the lowest latency for real-time / interactive applications
Groq LPU (now part of NVIDIA) is the benchmark for deterministic low-latency inference — no stalls, predictable execution. Cerebras WSE-3 also excels here due to massive on-chip SRAM eliminating off-chip fetches.
I'm running open models and want to avoid vendor lock-in
Tenstorrent Blackhole is the most open option — RISC-V cores, open-source TT-Metal SDK, open IP licensing. NVIDIA H100/B200 with vLLM/SGLang gives the richest ecosystem but CUDA is a de facto lock-in.
I need inference at the edge or on-device
Axelera Metis for edge AI deployment. For laptop/desktop, NVIDIA consumer GPUs (RTX 4090/5090) or Apple Silicon M-series are more practical than datacenter ASICs.
I'm a cloud-first shop and want managed inference
Google Trillium (TPU v6) on Vertex AI or AWS Trainium2 on SageMaker. Both are cloud-only but deeply integrated with their respective ecosystems and priced competitively for sustained workloads.
I want the lowest cost-per-token for a large deployment
Depends on model size. For frontier-scale (100B+): Cerebras or SambaNova. For mid-size (7–70B): NVIDIA H100 with quantisation and efficient batching (vLLM). Watch Fractile for the 90% cost-reduction promise in 2027.
In the pipeline
Announced, rumoured, or in development. Treat unconfirmed items with caution.
NVIDIA Rubin (GB300 successor) Announced
Next-gen GPU announced at GTC 2025. Rubin Ultra pairs two Rubin dies with 50% more HBM than B200. Expected 2026-2027 delivery. NVLink 6.0 at ~5.6 TB/s.
Tenstorrent Galaxy (server pod) Pre-release
Full server system with 32 Blackhole cards. Targeting hyperscaler and cloud inference deployments. Open-source TT-Metal stack differentiates from closed-compiler alternatives.
Cerebras WSE-4 Rumored
Next-generation wafer-scale engine. Expected to move to TSMC 3nm and extend on-chip SRAM beyond 44GB. No official confirmation; treat as speculative.
AWS Inferentia3 Expected 2026
Next-gen successor to Trainium2/Inferentia2. Expected focus on cost-per-token efficiency for Amazon Bedrock workloads. No official specs yet.
Fractile chip (unnamed) 2027
UK startup's SRAM memory-compute fusion ASIC. $220M raised May 2026. Architecture eliminates off-chip DRAM entirely — 100x speed, 90% cost reduction claimed vs GPU. Talks with Anthropic.
On the watchlist
Emerging chips and companies being tracked but not yet promoted to a full card.
Luminous Computing ↗
Photonic computing startup targeting 100x performance per watt vs electronic ASICs for AI. Optical computing for matrix multiplication. Very early-stage.
Why watch: If optical interconnects pan out, the memory bandwidth problem dissolves — photonic in-memory compute could leapfrog all current architectures.
Rain AI (Neuromorphic) ↗
In-memory neuromorphic ASIC targeting analog compute for sparse activations. Very different paradigm from current digital ASICs.
Why watch: Analog/neuromorphic compute could offer 1000x efficiency for certain workloads if it proves programmable enough for transformer architectures.
Intel Gaudi 3 ↗
Intel's latest AI accelerator. 128GB HBM2e at 3.7 TB/s, 1835 TFLOPS BF16 (sparse). Strong on multi-chip scaling with RoCE networking, avoiding NVLink.
Why watch: SambaNova-Intel partnership (Apr 2026) could turn Gaudi 3 into a serious inference platform with the right software stack. Worth watching the collaboration.
Qualcomm Cloud AI 100 Ultra ↗
LPDDR5x-based inference chip with 128GB memory and 100W TDP. Targets cost-sensitive inference deployments.
Why watch: Extremely power-efficient design (100W vs 700W H100). If the ecosystem matures, the TCO story is compelling for medium-scale inference.