Not every AI chip is a GPU.
Google has trained on its own TPUs since 2017, Amazon sells Trainium by the rack, and in 2026 Microsoft deployed an inference chip of its own and OpenAI unveiled one. At the other end, a 128 GB box now fits on a desk. Here is how they differ, what they’re good for and what they cost.
- Why do Google, Amazon, Microsoft and OpenAI design their own chips?
- What does a custom chip give up compared with a GPU?
- What can a desk-side machine really run?
Trade flexibility for efficiency.
A GPU has to run every kind of model, every kernel anyone writes, and graphics too. A company that serves its own models at hyperscale knows exactly what it runs, so it can spend the silicon on what those models need and skip the rest.
Matrix engines, sized to the job
A TPU is built around systolic arrays: grids of multiply-accumulate cells that pass operands to their neighbours, so a matrix multiply needs almost no instruction or register traffic. Google grew them from 128 × 128 to 256 × 256 in Trillium. A GPU instead spreads its tensor cores across many general-purpose SMs.
Only the numbers models use
No FP64 for science codes, no rasterizers or video engines. The die area goes to BF16, FP8 and now FP4 math, and to the scaling formats (MXFP8, MXFP4) that keep low-precision accuracy acceptable.
Memory placed for the dataflow
Inference chips spend area on large on-chip SRAM so weights and activations stay close to the math, and wire HBM in slices next to the cores that use them. Fewer trips to HBM means more tokens per joule.
A network designed with the chip
TPUs link directly to each other in a 3D torus with optical circuit switches; Trainium uses NeuronLink inside an UltraServer. Thousands of chips act as one machine without a separate switch fabric bolted on afterwards.
Cost, supply and leverage
Buying merchant GPUs means paying their margin and queuing for their allocation. Designing with Broadcom, Marvell or MediaTek and booking TSMC capacity directly gives a hyperscaler its own roadmap and a better negotiating position.
The catch: software
CUDA has two decades of kernels, libraries and engineers behind it. Custom chips live or die by their compilers: XLA and JAX for TPUs, the Neuron SDK and NKI for Trainium, PyTorch and Triton backends for the rest. That is why most of them serve the maker’s own models first.
The chips the hyperscalers build.
Tensor Processing Unit
A decade of systolic arrays, now split in twoGoogle designed the first TPU in 2015 for its own inference and has trained every Gemini on TPUs since. Each chip is built around matrix units (MXUs): systolic arrays where operands flow cell to cell, so a matrix multiply costs almost no instruction overhead. Chips talk to each other directly over the inter-chip interconnect (ICI) in a 3D torus, with optical circuit switches that rewire the pod in software.
Ironwood (TPU7x, 2025) was the first TPU built for inference first: 192 GB of HBM at 7.4 TB/s and native FP8, in pods of 9,216 chips that pool about 1.77 PB of HBM. In 2026 Google split the eighth generation in two: TPU 8t for training and TPU 8i for serving.
- TPU 8i carries 384 MB of on-chip SRAM, three times the 8t’s, to keep KV caches and activations close to the math.
- TPU 8i swaps the torus for a low-diameter “Boardfly” network of up to 1,152 chips, at most 7 hops apart, because inference cares about latency more than bisection bandwidth.
- Ironwood rents for $12 per chip-hour on demand, or $5.40 with a 3-year commitment. Anthropic and others have signed for large TPU capacity.
| Chip | Memory | Bandwidth | BF16 | FP8 | FP4 |
|---|---|---|---|---|---|
| TPU v5e2023 | 16 GB | 820 GB/s | 197 TF | — | — |
| TPU v5p2023 | 95 GB | 2.77 TB/s | 459 TF | — | — |
| TPU v6e Trillium2024 | 32 GB | 1.64 TB/s | 918 TF | — | — |
| TPU7x Ironwood2025 | 192 GB | 7.37 TB/s | 2.3 PF | 4.6 PF | — |
| TPU 8t2026 · announced | 216 GB | 6.53 TB/s | — | — | 13 PF |
| TPU 8i2026 · announced | 288 GB | 8.6 TB/s | — | — | 10 PF |
Google publishes only the FP4 peak for TPU 8t and 8i, which are not yet generally available. Power is not disclosed for any TPU.
Trainium
Amazon’s chip, rented by the rackAnnapurna Labs, the Israeli chip team Amazon bought in 2015, designs Trainium for training and Inferentia for serving. Each chip holds several NeuronCores with their own matrix, vector and scalar engines, programmed through the Neuron SDK or, for hand-written kernels, the NKI language.
Trainium2 is the chip behind Project Rainier, Anthropic’s cluster of about half a million of them. Trainium3, AWS’s first 3 nm chip, raised memory to 144 GB at 4.9 TB/s and joins 144 chips into one Trn3 UltraServer through NeuronSwitch.
- Trainium3 runs MXFP4 at the same rate as MXFP8, 2.5 PF: the 4-bit format saves memory and bandwidth, not math.
- A Trn3 UltraServer holds 20.7 TB of HBM3e across 144 chips, enough for a trillion-parameter model in FP8 with room for KV cache.
- Trainium4 is announced to support NVIDIA’s NVLink Fusion, among the first custom accelerators committed to it.
| Chip | Memory | Bandwidth | BF16 | FP8 | FP4 |
|---|---|---|---|---|---|
| Inferentia22023 | 32 GB | 820 GB/s | 190 TF | 190 TF | — |
| Trainium22024 | 96 GB | 2.9 TB/s | 667 TF | 1.3 PF | — |
| Trainium32025 | 144 GB | 4.9 TB/s | 671 TF | 2.5 PF | 2.5 PF |
Trainium2 rents for $2.24 per chip-hour through Capacity Blocks; Trainium3 has no public price yet.
Jalapeño
OpenAI’s first chip, built only for inferenceOpenAI announced Jalapeño with Broadcom on 24 June 2026 as the first step of a plan to deploy 10 GW of its own accelerators, and detailed it at Hot Chips in August. It does one thing: serve large language models. OpenAI says it went from design to tape-out in nine months, with volume production due in 2027, with its own models helping the engineers.
The headline is bandwidth per watt. 216 GiB of HBM4 deliver 15.4 TB/s, nearly twice a Blackwell GPU, from a chip rated at 700 W and measured at or below 550 W. The die is cut into core slices, each paired with its own slice of HBM, so most memory traffic never crosses the chip.
- Racks of 128 chips with 600 GB/s each between them, and pods of 2,048 chips on Broadcom Tomahawk 6 switches at 200 GB/s each.
- OpenAI compared it against GB200 NVL72 on gpt-oss-120b and GB300 NVL72 on DeepSeek-R1 and Kimi K2.5, claiming more tokens per watt and faster responses.
- Only the 13.4 PF MXFP4 rate is public; the process node is unconfirmed, with press reports pointing to TSMC 3 nm.
| Chip | Memory | Bandwidth | BF16 | FP8 | FP4 |
|---|---|---|---|---|---|
| Jalapeño2026 · internal | 232 GB | 15.4 TB/s | — | — | 13 PF |
Figures from OpenAI’s Hot Chips 2026 talk as reported by ServeTheHome, The Register and SemiWiki. OpenAI quotes memory in GiB.
Maia 200
Azure’s inference chip, on EthernetMaia 200, launched in January 2026 on TSMC 3 nm with about 140 billion transistors, serves OpenAI and Microsoft models on Azure. Microsoft claims 30% better performance per dollar than the rest of its fleet.
It pairs 216 GB of HBM3e at 7 TB/s with 272 MB of on-chip SRAM, and builds scale-up from plain Ethernet with an integrated NIC: four chips link directly on a tray, and clusters reach 6,144 chips.
- 10 PF of FP4 and 5 PF of FP8 on tensor units. BF16 runs on the vector units at 1.3 PF, so the chip is built for low-precision serving, not training.
- 2.8 TB/s of bidirectional chip-to-chip bandwidth over Ethernet rather than a proprietary link.
| Chip | Memory | Bandwidth | BF16 | FP8 | FP4 |
|---|---|---|---|---|---|
| Maia 2002026 · internal | 216 GB | 7 TB/s | 1.3 PF | 5.1 PF | 10 PF |
Figures from Microsoft’s launch post and its published table.
MTIA
From recommendation models to LLMsMeta’s Training and Inference Accelerator started as a chip for the ranking and recommendation models behind its feeds. MTIA 400, from Meta’s 2026 roadmap, is the first built for generative AI: two compute chiplets, 288 GB of HBM at 9.2 TB/s and 72 chips per rack on a switched backplane.
- The roadmap continues with MTIA 450 at 18.4 TB/s in early 2027, and MTIA 500 at 27.6 TB/s with up to 512 GB at about 1,700 W.
- Specifications come from press coverage of Meta’s roadmap presentation, not a Meta datasheet.
| Chip | Memory | Bandwidth | BF16 | FP8 | FP4 |
|---|---|---|---|---|---|
| MTIA 4002026 · internal | 288 GB | 9.2 TB/s | 3.0 PF | 6.0 PF | 12 PF |
Reported figures; Meta does not label them dense or sparse.
The SRAM machines
No HBM at all: everything on chipA different bet: skip HBM and hold the model in on-chip SRAM, which is far faster per byte but far smaller. Cerebras builds one chip from an entire wafer, 46,225 mm² with 44 GB of SRAM. Groq’s LPU spreads a model across hundreds of small SRAM chips; after a roughly $20 billion licensing deal in December 2025, NVIDIA sells the Groq 3 LPU in LPX racks alongside Vera Rubin.
Per-user speed is spectacular, often thousands of tokens per second. The cost is capacity: a 1T-parameter model needs dozens of wafers or thousands of LPUs, so these systems shine at fast decode for models that can be spread thin. They sit outside the HBM model in our calculator.
- An LPX rack of 256 LPUs holds 128 GB of SRAM in total, less than one Blackwell GPU’s HBM, but streams it at about 40 PB/s.
- NVIDIA positions LPX as a decode accelerator next to Rubin GPUs, which keep handling prefill and long-context memory.
| Chip | Memory | Bandwidth | BF16 | FP8 | FP4 |
|---|---|---|---|---|---|
| Cerebras WSE-32024 | 44 GB SRAM | 21 PB/s | 125 PF (sparse FP16) | ||
| Groq 3 LPU2026 | ≈0.5 GB SRAM | ≈150 TB/s | 1.2 PF FP8 | ||
Cerebras quotes its peak with sparsity; Groq 3 figures are as reported. These systems are not in the calculator, which models HBM-based chips.
Two numbers decide what a chip can serve.
Memory capacity sets which models fit; memory bandwidth sets how fast they decode, because every generated token streams the active weights through the chip. Datacenter accelerators sit top-right on HBM. Desk-side machines trade bandwidth for capacity with cheaper LPDDR or GDDR memory.
Hollow points are announced or internal-only parts. A diagonal is how many times per second a device can read its entire memory: a dense model that fills the memory can’t decode faster than that for one user. Hover or focus a point for its name.
A supercomputer under the desk, sort of.
Local machines split into two camps. Graphics cards have fast GDDR memory but not much of it. Unified-memory boxes like DGX Spark, Strix Halo and Macs hold far bigger models in slower shared memory. The estimates below assume one user and a 32k-token context.
DGX Spark
- Memory
- 128 GB LPDDR5X unified
- Bandwidth
- 273 GB/s
- Price
- $6,950 (was $3,999)
- Power
- 140 W
A Grace Blackwell chip with 128 GB of memory shared by CPU and GPU. Plenty of compute and capacity, but 273 GB/s caps decode speed. Two units link over ConnectX-7; partner GB10 boxes such as the ASUS Ascent GX10 cost about $6,000, and a 64 GB version is expected around $4,999 from partners.
DGX Station GB300
- Memory
- 252 GB HBM3e (+496 GB LPDDR5X)
- Bandwidth
- 7.1 TB/s
- Price
- ≈$85–95k (OEM listings)
- Power
- 1,600 W
A Blackwell Ultra GPU in a 1.6 kW tower. The 252 GB of HBM3e runs at 7.1 TB/s; another 496 GB of CPU memory is coherent but about 18× slower, so we count only the HBM.
RTX PRO 6000 Blackwell
- Memory
- 96 GB GDDR7
- Bandwidth
- 1.79 TB/s
- Price
- $17,063 (≈$8,500 in 2025)
- Power
- 600 W
The most memory on a single card, at RTX 5090 bandwidth. Four of them hold a 300B-parameter model at 4-bit. FP16 and FP8 rates are derived from the 5090’s; NVIDIA publishes only the sparse FP4 figure.
GeForce RTX 5090
- Memory
- 32 GB GDDR7
- Bandwidth
- 1.79 TB/s
- Price
- $6,480 (MSRP $1,999)
- Power
- 575 W
The fastest consumer card for decode at 1.79 TB/s, but 32 GB holds only a ~30B model at 8-bit or ~50B at 4-bit. GDDR7 shortages pushed street prices past 3× MSRP.
GeForce RTX 4090 (used)
- Memory
- 24 GB GDDR6X
- Bandwidth
- 1.01 TB/s
- Price
- ≈$3,200 used
- Power
- 450 W
Discontinued; used prices are now about twice its $1,599 launch price. 24 GB at 1 TB/s, with FP8 but no FP4.
GeForce RTX 3090 (used)
- Memory
- 24 GB GDDR6X
- Bandwidth
- 936 GB/s
- Price
- ≈$1,400 used
- Power
- 350 W
Still the cheapest way to 24 GB at nearly 1 TB/s, which is why multi-3090 rigs are a local-LLM staple. No FP8, so it runs quantized weights through 16-bit math.
Radeon AI PRO R9700
- Memory
- 32 GB GDDR6
- Bandwidth
- 640 GB/s
- Price
- $2,200 (MSRP $1,299)
- Power
- 300 W
The cheapest new 32 GB card, with a blower cooler made for stacking two to four. Runs on ROCm.
Arc Pro B70
- Memory
- 32 GB GDDR6
- Bandwidth
- 608 GB/s
- Price
- ≈$1,300–1,850 (MSRP $949)
- Power
- 230 W
Intel’s 2026 32 GB card, close to the R9700 on paper. The FP16 rate is derived from Intel’s INT8 figure.
Ryzen AI Max+ 395 (128 GB)
- Memory
- 128 GB LPDDR5X unified
- Bandwidth
- 256 GB/s
- Price
- $3,449 (Framework Desktop)
- Power
- 120 W
The budget way to 128 GB: decode speed in DGX Spark territory for half the price, but far less compute, so long prompts are slow to read. FP16 rate derived from 40 RDNA 3.5 CUs at 2.9 GHz.
Ryzen AI Max+ PRO 495 (192 GB)
- Memory
- 192 GB LPDDR5X unified
- Bandwidth
- 273 GB/s
- Price
- $6,799 pre-order (Framework)
- Power
- 120 W
The Strix Halo refresh: 50% more memory and slightly more bandwidth, enough for a 300B MoE at 4-bit. Pre-orders ship from late 2026.
Mac Studio M5 Ultra (512 GB)
- Memory
- 512 GB LPDDR5X unified
- Bandwidth
- 1.2 TB/s
- Price
- $22,299
- Power
- 480 W
The highest-bandwidth desk-side box short of HBM: 1.2 TB/s across 512 GB. macOS gives the GPU about 75% of memory by default (adjustable). The FP16 figure is an estimate; Apple publishes none.
Mac Studio M3 Ultra (512 GB)
- Memory
- 512 GB LPDDR5 unified
- Bandwidth
- 819 GB/s
- Price
- ≈$9,500 at launch
- Power
- 480 W
The machine that made 600B-parameter MoE models runnable at home. Apple retired the 512 GB option in March 2026. Without matrix units, reading long prompts is slow.
MacBook Pro M5 Max (128 GB)
- Memory
- 128 GB LPDDR5X unified
- Bandwidth
- 614 GB/s
- Price
- ≈$6,600 (est.)
- Power
- 140 W
A 128 GB laptop at 614 GB/s, with the GPU neural accelerators that speed up prompt reading. The price is an estimate after Apple’s June 2026 memory price rise.
Mac mini M5 Pro (64 GB)
- Memory
- 64 GB LPDDR5X unified
- Bandwidth
- 307 GB/s
- Price
- $2,699
- Power
- 155 W
The cheapest Mac with GPU matrix units. 64 GB fits a 35B model comfortably; 307 GB/s keeps small MoE models quick.
Try any model on any of these in the models calculator. Speeds are bandwidth-based estimates for a single user; prefill (reading a long prompt) is much slower on the bandwidth-rich, compute-poor boxes.
Chips, racks and rentals.
Datacenter accelerators have no public price list. Vendors sell them in servers and racks under negotiated contracts, and the hyperscalers’ own chips are only rented by the hour. These are the best public figures, marked by how solid they are.
| Accelerator | Per chip | As a system | Rent |
|---|---|---|---|
| H100NVIDIA · 2022 | $25–40kreported | — | $3.99 Lambda · $6.16 CoreWeave · $6.88 AWS |
| H200NVIDIA · 2024 | ≈$27kreported | — | $6.31 CoreWeave · $7.91 AWS |
| B200NVIDIA · 2024 | $30–50kreported | ≈$500–560kDGX / HGX B200, 8 GPUs | $6.69 Lambda · $8.60 CoreWeave · $14.24 AWS |
| B300NVIDIA · 2025 | $45–60kestimate | $400–550kHGX B300, 8 GPUs | $16.15 AWS Capacity Blocks · $17.80 AWS |
| GB200 NVL72NVIDIA · 2024 | ≈$42–54k per GPUestimate | $3.0–3.9MGB200 NVL72 rack, 72 GPUs | $10.50 CoreWeave · $10.58 AWS Capacity Blocks |
| GB300 NVL72NVIDIA · 2025 | ≈$51–56k per GPUestimate | ≈$3.7–4MGB300 NVL72 rack, 72 GPUs | — |
| Vera Rubin NVL72NVIDIA · 2026 | ≈$70–125k per GPUestimate | $5–9.1M (analysts disagree)Vera Rubin NVL72 rack | — |
| MI300XAMD · 2023 | $10–15kestimate | — | $6.00 Azure |
| MI325XAMD · 2024 | sold, price not publicnot public | — | — |
| MI355XAMD · 2025 | $25–30kestimate | — | — |
| MI455X (Helios)AMD · 2026 | ≈$70–76k per GPUestimate | $5–5.5MHelios rack, 72 GPUs | — |
| TPU v5eGoogle · 2023 | cloud onlycloud only | — | $1.20 Google Cloud |
| TPU v5pGoogle · 2023 | cloud onlycloud only | — | $4.20 Google Cloud |
| TPU v6e TrilliumGoogle · 2024 | cloud onlycloud only | — | $2.70 Google Cloud |
| TPU7x IronwoodGoogle · 2025 | cloud onlycloud only | — | $12.00 on demand · $5.40 with a 3-year commitment |
| TPU 8tGoogle · 2026 · announced | not priced yetcloud only | — | — |
| TPU 8iGoogle · 2026 · announced | not priced yetcloud only | — | — |
| Inferentia2AWS · 2023 | cloud onlycloud only | — | $1.08 AWS (inf2) |
| Trainium2AWS · 2024 | cloud onlycloud only | — | $2.24 AWS Capacity Blocks |
| Trainium3AWS · 2025 | cloud onlycloud only | — | no public price yet |
| JalapeñoOpenAI · 2026 · internal | not soldnot sold | — | — |
| Maia 200Microsoft · 2026 · internal | not soldnot sold | — | — |
| MTIA 400Meta · 2026 · internal | not soldnot sold | — | — |
| Gaudi 3Intel · 2024 | $15.6klist | $125k (Intel list)8-chip baseboard kit | — |
Specifications researched 30 October 2026 and re-checked 14 November 2026 from vendor pages, documentation and Hot Chips coverage. Compute is dense peak TFLOPS; vendor figures quoted with sparsity are halved. Datacenter chip and rack prices are not published by any vendor: they are press reports or analyst estimates, and rack-scale GPUs are priced as the rack estimate divided by 72. Rentals are the cheapest public on-demand list price per chip-hour. Desk-side prices are October 2026 retail, after the 2026 memory shortage pushed many to 1.5–3× their launch price.
- Google Cloud: TPU 8t and TPU 8i technical deep dive ↗
- Google Cloud: TPU7x (Ironwood) and Cloud TPU pricing ↗
- AWS Neuron: Trainium3 architecture ↗
- ServeTheHome: OpenAI Jalapeño at Hot Chips 2026 ↗
- OpenAI and Broadcom: Jalapeño announcement ↗
- Microsoft: Maia 200 ↗
- ServeTheHome: Meta MTIA roadmap ↗
- The Register: NVIDIA LPX (Groq 3) deep dive ↗
- NVIDIA Vera Rubin NVL72 and HGX specifications ↗
- AMD Instinct MI455X and Helios ↗
- NVIDIA DGX Spark and DGX Station ↗
- Apple: Mac Studio with M5 Max and M5 Ultra ↗
- Wing VC: what an NVL72 rack costs ↗