Beyond the GPUWhat exists · Stop 6 of 9

Not every AI chip is a GPU.

Google has trained on its own TPUs since 2017, Amazon sells Trainium by the rack, and in 2026 Microsoft deployed an inference chip of its own and OpenAI unveiled one. At the other end, a 128 GB box now fits on a desk. Here is how they differ, what they’re good for and what they cost.

After this page you can answer
  1. Why do Google, Amazon, Microsoft and OpenAI design their own chips?
  2. What does a custom chip give up compared with a GPU?
  3. What can a desk-side machine really run?
Data last revised November 2026
Why build your own

Trade flexibility for efficiency.

A GPU has to run every kind of model, every kernel anyone writes, and graphics too. A company that serves its own models at hyperscale knows exactly what it runs, so it can spend the silicon on what those models need and skip the rest.

01

Matrix engines, sized to the job

A TPU is built around systolic arrays: grids of multiply-accumulate cells that pass operands to their neighbours, so a matrix multiply needs almost no instruction or register traffic. Google grew them from 128 × 128 to 256 × 256 in Trillium. A GPU instead spreads its tensor cores across many general-purpose SMs.

02

Only the numbers models use

No FP64 for science codes, no rasterizers or video engines. The die area goes to BF16, FP8 and now FP4 math, and to the scaling formats (MXFP8, MXFP4) that keep low-precision accuracy acceptable.

03

Memory placed for the dataflow

Inference chips spend area on large on-chip SRAM so weights and activations stay close to the math, and wire HBM in slices next to the cores that use them. Fewer trips to HBM means more tokens per joule.

04

A network designed with the chip

TPUs link directly to each other in a 3D torus with optical circuit switches; Trainium uses NeuronLink inside an UltraServer. Thousands of chips act as one machine without a separate switch fabric bolted on afterwards.

05

Cost, supply and leverage

Buying merchant GPUs means paying their margin and queuing for their allocation. Designing with Broadcom, Marvell or MediaTek and booking TSMC capacity directly gives a hyperscaler its own roadmap and a better negotiating position.

06

The catch: software

CUDA has two decades of kernels, libraries and engineers behind it. Custom chips live or die by their compilers: XLA and JAX for TPUs, the Neuron SDK and NKI for Trainium, PyTorch and Triton backends for the rest. That is why most of them serve the maker’s own models first.

Custom silicon

The chips the hyperscalers build.

Google

Tensor Processing Unit

A decade of systolic arrays, now split in two
Google Cloud only

Google designed the first TPU in 2015 for its own inference and has trained every Gemini on TPUs since. Each chip is built around matrix units (MXUs): systolic arrays where operands flow cell to cell, so a matrix multiply costs almost no instruction overhead. Chips talk to each other directly over the inter-chip interconnect (ICI) in a 3D torus, with optical circuit switches that rewire the pod in software.

Ironwood (TPU7x, 2025) was the first TPU built for inference first: 192 GB of HBM at 7.4 TB/s and native FP8, in pods of 9,216 chips that pool about 1.77 PB of HBM. In 2026 Google split the eighth generation in two: TPU 8t for training and TPU 8i for serving.

  • TPU 8i carries 384 MB of on-chip SRAM, three times the 8t’s, to keep KV caches and activations close to the math.
  • TPU 8i swaps the torus for a low-diameter “Boardfly” network of up to 1,152 chips, at most 7 hops apart, because inference cares about latency more than bisection bandwidth.
  • Ironwood rents for $12 per chip-hour on demand, or $5.40 with a 3-year commitment. Anthropic and others have signed for large TPU capacity.
ChipMemoryBandwidthBF16FP8FP4
TPU v5e202316 GB820 GB/s197 TF——
TPU v5p202395 GB2.77 TB/s459 TF——
TPU v6e Trillium202432 GB1.64 TB/s918 TF——
TPU7x Ironwood2025192 GB7.37 TB/s2.3 PF4.6 PF—
TPU 8t2026 · announced216 GB6.53 TB/s——13 PF
TPU 8i2026 · announced288 GB8.6 TB/s——10 PF

Google publishes only the FP4 peak for TPU 8t and 8i, which are not yet generally available. Power is not disclosed for any TPU.

Amazon Web Services

Trainium

Amazon’s chip, rented by the rack
AWS only

Annapurna Labs, the Israeli chip team Amazon bought in 2015, designs Trainium for training and Inferentia for serving. Each chip holds several NeuronCores with their own matrix, vector and scalar engines, programmed through the Neuron SDK or, for hand-written kernels, the NKI language.

Trainium2 is the chip behind Project Rainier, Anthropic’s cluster of about half a million of them. Trainium3, AWS’s first 3 nm chip, raised memory to 144 GB at 4.9 TB/s and joins 144 chips into one Trn3 UltraServer through NeuronSwitch.

  • Trainium3 runs MXFP4 at the same rate as MXFP8, 2.5 PF: the 4-bit format saves memory and bandwidth, not math.
  • A Trn3 UltraServer holds 20.7 TB of HBM3e across 144 chips, enough for a trillion-parameter model in FP8 with room for KV cache.
  • Trainium4 is announced to support NVIDIA’s NVLink Fusion, among the first custom accelerators committed to it.
ChipMemoryBandwidthBF16FP8FP4
Inferentia2202332 GB820 GB/s190 TF190 TF—
Trainium2202496 GB2.9 TB/s667 TF1.3 PF—
Trainium32025144 GB4.9 TB/s671 TF2.5 PF2.5 PF

Trainium2 rents for $2.24 per chip-hour through Capacity Blocks; Trainium3 has no public price yet.

OpenAI × Broadcom

Jalapeño

OpenAI’s first chip, built only for inference
OpenAI’s own fleet

OpenAI announced Jalapeño with Broadcom on 24 June 2026 as the first step of a plan to deploy 10 GW of its own accelerators, and detailed it at Hot Chips in August. It does one thing: serve large language models. OpenAI says it went from design to tape-out in nine months, with volume production due in 2027, with its own models helping the engineers.

The headline is bandwidth per watt. 216 GiB of HBM4 deliver 15.4 TB/s, nearly twice a Blackwell GPU, from a chip rated at 700 W and measured at or below 550 W. The die is cut into core slices, each paired with its own slice of HBM, so most memory traffic never crosses the chip.

  • Racks of 128 chips with 600 GB/s each between them, and pods of 2,048 chips on Broadcom Tomahawk 6 switches at 200 GB/s each.
  • OpenAI compared it against GB200 NVL72 on gpt-oss-120b and GB300 NVL72 on DeepSeek-R1 and Kimi K2.5, claiming more tokens per watt and faster responses.
  • Only the 13.4 PF MXFP4 rate is public; the process node is unconfirmed, with press reports pointing to TSMC 3 nm.
ChipMemoryBandwidthBF16FP8FP4
Jalapeño2026 · internal232 GB15.4 TB/s——13 PF

Figures from OpenAI’s Hot Chips 2026 talk as reported by ServeTheHome, The Register and SemiWiki. OpenAI quotes memory in GiB.

Microsoft

Maia 200

Azure’s inference chip, on Ethernet
Microsoft’s own fleet

Maia 200, launched in January 2026 on TSMC 3 nm with about 140 billion transistors, serves OpenAI and Microsoft models on Azure. Microsoft claims 30% better performance per dollar than the rest of its fleet.

It pairs 216 GB of HBM3e at 7 TB/s with 272 MB of on-chip SRAM, and builds scale-up from plain Ethernet with an integrated NIC: four chips link directly on a tray, and clusters reach 6,144 chips.

  • 10 PF of FP4 and 5 PF of FP8 on tensor units. BF16 runs on the vector units at 1.3 PF, so the chip is built for low-precision serving, not training.
  • 2.8 TB/s of bidirectional chip-to-chip bandwidth over Ethernet rather than a proprietary link.
ChipMemoryBandwidthBF16FP8FP4
Maia 2002026 · internal216 GB7 TB/s1.3 PF5.1 PF10 PF

Figures from Microsoft’s launch post and its published table.

Meta

MTIA

From recommendation models to LLMs
Meta’s own fleet

Meta’s Training and Inference Accelerator started as a chip for the ranking and recommendation models behind its feeds. MTIA 400, from Meta’s 2026 roadmap, is the first built for generative AI: two compute chiplets, 288 GB of HBM at 9.2 TB/s and 72 chips per rack on a switched backplane.

  • The roadmap continues with MTIA 450 at 18.4 TB/s in early 2027, and MTIA 500 at 27.6 TB/s with up to 512 GB at about 1,700 W.
  • Specifications come from press coverage of Meta’s roadmap presentation, not a Meta datasheet.
ChipMemoryBandwidthBF16FP8FP4
MTIA 4002026 · internal288 GB9.2 TB/s3.0 PF6.0 PF12 PF

Reported figures; Meta does not label them dense or sparse.

Cerebras · NVIDIA (Groq)

The SRAM machines

No HBM at all: everything on chip
Sold, and rented as an API

A different bet: skip HBM and hold the model in on-chip SRAM, which is far faster per byte but far smaller. Cerebras builds one chip from an entire wafer, 46,225 mm² with 44 GB of SRAM. Groq’s LPU spreads a model across hundreds of small SRAM chips; after a roughly $20 billion licensing deal in December 2025, NVIDIA sells the Groq 3 LPU in LPX racks alongside Vera Rubin.

Per-user speed is spectacular, often thousands of tokens per second. The cost is capacity: a 1T-parameter model needs dozens of wafers or thousands of LPUs, so these systems shine at fast decode for models that can be spread thin. They sit outside the HBM model in our calculator.

  • An LPX rack of 256 LPUs holds 128 GB of SRAM in total, less than one Blackwell GPU’s HBM, but streams it at about 40 PB/s.
  • NVIDIA positions LPX as a decode accelerator next to Rubin GPUs, which keep handling prefill and long-context memory.
ChipMemoryBandwidthBF16FP8FP4
Cerebras WSE-3202444 GB SRAM21 PB/s125 PF (sparse FP16)
Groq 3 LPU2026≈0.5 GB SRAM≈150 TB/s1.2 PF FP8

Cerebras quotes its peak with sparsity; Groq 3 figures are as reported. These systems are not in the calculator, which models HBM-based chips.

Capacity × bandwidth

Two numbers decide what a chip can serve.

Memory capacity sets which models fit; memory bandwidth sets how fast they decode, because every generated token streams the active weights through the chip. Datacenter accelerators sit top-right on HBM. Desk-side machines trade bandwidth for capacity with cheaper LPDDR or GDDR memory.

datacenter desk-side full-memory reads per secondper device · both axes log
100 GB/s1 TB/s10 TB/s16 GB32 GB64 GB128 GB256 GB512 GB1.024 TBMEMORY CAPACITY · what fits →BANDWIDTH · how fast →1×/s10×/s100×/sJalapeñoVera Rubin NVL72MI455X (Helios)B200H100TPU7x IronwoodTrainium3GeForce RTX 5090DGX SparkMac Studio M5 Ultra (512 GB)Ryzen AI Max+ 395 (128 GB)RTX PRO 6000 BlackwellTPU v5e

Hollow points are announced or internal-only parts. A diagonal is how many times per second a device can read its entire memory: a dense model that fills the memory can’t decode faster than that for one user. Hover or focus a point for its name.

Desk-side AI

A supercomputer under the desk, sort of.

Local machines split into two camps. Graphics cards have fast GDDR memory but not much of it. Unified-memory boxes like DGX Spark, Strix Halo and Macs hold far bigger models in slower shared memory. The estimates below assume one user and a 32k-token context.

NVIDIA · GB10 mini PC

DGX Spark

Memory
128 GB LPDDR5X unified
Bandwidth
273 GB/s
Price
$6,950 (was $3,999)
Power
140 W

A Grace Blackwell chip with 128 GB of memory shared by CPU and GPU. Plenty of compute and capacity, but 273 GB/s caps decode speed. Two units link over ConnectX-7; partner GB10 boxes such as the ASUS Ascent GX10 cost about $6,000, and a 64 GB version is expected around $4,999 from partners.

Qwen3.6 35B A3B 4-bit: ~72 tok/sgpt-oss-120b: ~46 tok/s
NVIDIA · Deskside workstation

DGX Station GB300

Memory
252 GB HBM3e (+496 GB LPDDR5X)
Bandwidth
7.1 TB/s
Price
≈$85–95k (OEM listings)
Power
1,600 W

A Blackwell Ultra GPU in a 1.6 kW tower. The 252 GB of HBM3e runs at 7.1 TB/s; another 496 GB of CPU memory is coherent but about 18× slower, so we count only the HBM.

Qwen3.6 35B A3B 4-bit: ~352 tok/sgpt-oss-120b: ~344 tok/s
NVIDIA · Workstation card

RTX PRO 6000 Blackwell

Memory
96 GB GDDR7
Bandwidth
1.79 TB/s
Price
$17,063 (≈$8,500 in 2025)
Power
600 W

The most memory on a single card, at RTX 5090 bandwidth. Four of them hold a 300B-parameter model at 4-bit. FP16 and FP8 rates are derived from the 5090’s; NVIDIA publishes only the sparse FP4 figure.

Qwen3.6 35B A3B 4-bit: ~242 tok/sgpt-oss-120b: ~195 tok/s
NVIDIA · Graphics card

GeForce RTX 5090

Memory
32 GB GDDR7
Bandwidth
1.79 TB/s
Price
$6,480 (MSRP $1,999)
Power
575 W

The fastest consumer card for decode at 1.79 TB/s, but 32 GB holds only a ~30B model at 8-bit or ~50B at 4-bit. GDDR7 shortages pushed street prices past 3× MSRP.

Qwen3.6 35B A3B 4-bit: ~242 tok/sgpt-oss-120b: doesn’t fit
NVIDIA · Graphics card

GeForce RTX 4090 (used)

Memory
24 GB GDDR6X
Bandwidth
1.01 TB/s
Price
≈$3,200 used
Power
450 W

Discontinued; used prices are now about twice its $1,599 launch price. 24 GB at 1 TB/s, with FP8 but no FP4.

Qwen3.6 35B A3B 4-bit: ~182 tok/sgpt-oss-120b: doesn’t fit
NVIDIA · Graphics card

GeForce RTX 3090 (used)

Memory
24 GB GDDR6X
Bandwidth
936 GB/s
Price
≈$1,400 used
Power
350 W

Still the cheapest way to 24 GB at nearly 1 TB/s, which is why multi-3090 rigs are a local-LLM staple. No FP8, so it runs quantized weights through 16-bit math.

Qwen3.6 35B A3B 4-bit: ~175 tok/sgpt-oss-120b: doesn’t fit
AMD · Workstation card

Radeon AI PRO R9700

Memory
32 GB GDDR6
Bandwidth
640 GB/s
Price
$2,200 (MSRP $1,299)
Power
300 W

The cheapest new 32 GB card, with a blower cooler made for stacking two to four. Runs on ROCm.

Qwen3.6 35B A3B 4-bit: ~138 tok/sgpt-oss-120b: doesn’t fit
Intel · Workstation card

Arc Pro B70

Memory
32 GB GDDR6
Bandwidth
608 GB/s
Price
≈$1,300–1,850 (MSRP $949)
Power
230 W

Intel’s 2026 32 GB card, close to the R9700 on paper. The FP16 rate is derived from Intel’s INT8 figure.

Qwen3.6 35B A3B 4-bit: ~133 tok/sgpt-oss-120b: doesn’t fit
AMD · Strix Halo mini PC

Ryzen AI Max+ 395 (128 GB)

Memory
128 GB LPDDR5X unified
Bandwidth
256 GB/s
Price
$3,449 (Framework Desktop)
Power
120 W

The budget way to 128 GB: decode speed in DGX Spark territory for half the price, but far less compute, so long prompts are slow to read. FP16 rate derived from 40 RDNA 3.5 CUs at 2.9 GHz.

Qwen3.6 35B A3B 4-bit: ~69 tok/sgpt-oss-120b: ~44 tok/s
AMD · Gorgon Halo mini PC

Ryzen AI Max+ PRO 495 (192 GB)

Memory
192 GB LPDDR5X unified
Bandwidth
273 GB/s
Price
$6,799 pre-order (Framework)
Power
120 W

The Strix Halo refresh: 50% more memory and slightly more bandwidth, enough for a 300B MoE at 4-bit. Pre-orders ship from late 2026.

Qwen3.6 35B A3B 4-bit: ~72 tok/sgpt-oss-120b: ~46 tok/s
Apple · Mac

Mac Studio M5 Ultra (512 GB)

Memory
512 GB LPDDR5X unified
Bandwidth
1.2 TB/s
Price
$22,299
Power
480 W

The highest-bandwidth desk-side box short of HBM: 1.2 TB/s across 512 GB. macOS gives the GPU about 75% of memory by default (adjustable). The FP16 figure is an estimate; Apple publishes none.

Qwen3.6 35B A3B 4-bit: ~200 tok/sgpt-oss-120b: ~152 tok/s
Apple · Mac

Mac Studio M3 Ultra (512 GB)

Memory
512 GB LPDDR5 unified
Bandwidth
819 GB/s
Price
≈$9,500 at launch
Power
480 W

The machine that made 600B-parameter MoE models runnable at home. Apple retired the 512 GB option in March 2026. Without matrix units, reading long prompts is slow.

Qwen3.6 35B A3B 4-bit: ~161 tok/sgpt-oss-120b: ~116 tok/s
Apple · Laptop

MacBook Pro M5 Max (128 GB)

Memory
128 GB LPDDR5X unified
Bandwidth
614 GB/s
Price
≈$6,600 (est.)
Power
140 W

A 128 GB laptop at 614 GB/s, with the GPU neural accelerators that speed up prompt reading. The price is an estimate after Apple’s June 2026 memory price rise.

Qwen3.6 35B A3B 4-bit: ~134 tok/sgpt-oss-120b: ~93 tok/s
Apple · Mac

Mac mini M5 Pro (64 GB)

Memory
64 GB LPDDR5X unified
Bandwidth
307 GB/s
Price
$2,699
Power
155 W

The cheapest Mac with GPU matrix units. 64 GB fits a 35B model comfortably; 307 GB/s keeps small MoE models quick.

Qwen3.6 35B A3B 4-bit: ~80 tok/sgpt-oss-120b: doesn’t fit

Try any model on any of these in the models calculator. Speeds are bandwidth-based estimates for a single user; prefill (reading a long prompt) is much slower on the bandwidth-rich, compute-poor boxes.

What it costs

Chips, racks and rentals.

Datacenter accelerators have no public price list. Vendors sell them in servers and racks under negotiated contracts, and the hyperscalers’ own chips are only rented by the hour. These are the best public figures, marked by how solid they are.

AcceleratorPer chipAs a systemRent
H100NVIDIA · 2022$25–40kreported—$3.99 Lambda · $6.16 CoreWeave · $6.88 AWS
H200NVIDIA · 2024≈$27kreported—$6.31 CoreWeave · $7.91 AWS
B200NVIDIA · 2024$30–50kreported≈$500–560kDGX / HGX B200, 8 GPUs$6.69 Lambda · $8.60 CoreWeave · $14.24 AWS
B300NVIDIA · 2025$45–60kestimate$400–550kHGX B300, 8 GPUs$16.15 AWS Capacity Blocks · $17.80 AWS
GB200 NVL72NVIDIA · 2024≈$42–54k per GPUestimate$3.0–3.9MGB200 NVL72 rack, 72 GPUs$10.50 CoreWeave · $10.58 AWS Capacity Blocks
GB300 NVL72NVIDIA · 2025≈$51–56k per GPUestimate≈$3.7–4MGB300 NVL72 rack, 72 GPUs—
Vera Rubin NVL72NVIDIA · 2026≈$70–125k per GPUestimate$5–9.1M (analysts disagree)Vera Rubin NVL72 rack—
MI300XAMD · 2023$10–15kestimate—$6.00 Azure
MI325XAMD · 2024sold, price not publicnot public——
MI355XAMD · 2025$25–30kestimate——
MI455X (Helios)AMD · 2026≈$70–76k per GPUestimate$5–5.5MHelios rack, 72 GPUs—
TPU v5eGoogle · 2023cloud onlycloud only—$1.20 Google Cloud
TPU v5pGoogle · 2023cloud onlycloud only—$4.20 Google Cloud
TPU v6e TrilliumGoogle · 2024cloud onlycloud only—$2.70 Google Cloud
TPU7x IronwoodGoogle · 2025cloud onlycloud only—$12.00 on demand · $5.40 with a 3-year commitment
TPU 8tGoogle · 2026 · announcednot priced yetcloud only——
TPU 8iGoogle · 2026 · announcednot priced yetcloud only——
Inferentia2AWS · 2023cloud onlycloud only—$1.08 AWS (inf2)
Trainium2AWS · 2024cloud onlycloud only—$2.24 AWS Capacity Blocks
Trainium3AWS · 2025cloud onlycloud only—no public price yet
JalapeñoOpenAI · 2026 · internalnot soldnot sold——
Maia 200Microsoft · 2026 · internalnot soldnot sold——
MTIA 400Meta · 2026 · internalnot soldnot sold——
Gaudi 3Intel · 2024$15.6klist$125k (Intel list)8-chip baseboard kit—
Sources & notes

Specifications researched 30 October 2026 and re-checked 14 November 2026 from vendor pages, documentation and Hot Chips coverage. Compute is dense peak TFLOPS; vendor figures quoted with sparsity are halved. Datacenter chip and rack prices are not published by any vendor: they are press reports or analyst estimates, and rack-scale GPUs are priced as the rack estimate divided by 72. Rentals are the cheapest public on-demand list price per chip-hour. Desk-side prices are October 2026 retail, after the 2026 memory shortage pushed many to 1.5–3× their launch price.

Recap

The short version

  1. A custom chip trades the GPU’s flexibility for lower cost on a workload its owner already knows.
  2. The same two numbers decide what any of them can serve: memory capacity and memory bandwidth.
  3. Unified-memory boxes hold big models but stream them slowly; graphics cards are the reverse.
Check yourself

Why would a company that already buys every GPU it can get also spend years building its own chip?

Pick one. The reasoning shows either way.