What runs where, and how fast.
Pick an open-weights model and see which chips can hold it, how many you need, how fast each user’s answer streams, how many users it serves at once and what the hardware costs, from a desk-side DGX Spark to a rack of Blackwells. Every number is a first-order estimate built from the model’s real architecture.
- Will a given open model fit on a given chip, and how many chips does it need?
- How fast will it stream, and for how many users at once?
- What does that speed cost in hardware?
Fit it, then feed it.
Two questions decide everything. Does it fit? That’s the weights plus every user’s KV cache against memory capacity. How fast does it go? For decode, that’s mostly bytes moved per token against memory bandwidth. Quantization shrinks both; long contexts and many users grow the cache.
Or build your own Every control below updates the link in your address bar, so a setup can be shared.
- Parameters
- 117B · 5.1B active per token
- Released as
- MXFP4 experts · 65 GB checkpoint
- Context window
- 131k tokens
- Attention
- 18 full + 18 sliding-window (128) layers
- KV cache
- 21 MB per 1k tokens at FP8
- Quality index
- 45.8 benchleader ↗ weights ↗
| Hardware | Setup | Memory used | Tok/s per user | Total tok/s | Max users | Hardware | $ / M tokens |
|---|---|---|---|---|---|---|---|
| MI455X (Helios)432 GB · 23.3 TB/s | 1 × | 66 GB of 389 GB | 419memory-bound | 419 | 533 | $73k | $1.84 |
| Vera Rubin NVL72288 GB · 19.2 TB/s | 1 × | 66 GB of 259 GB | 410memory-bound | 410 | 319 | $97k | $2.50 |
| Jalapeño232 GB · 15.4 TB/s | 1 × | 66 GB of 209 GB | 399memory-bound | 399 | 236 | not sold | — |
| MTIA 400288 GB · 9.2 TB/s | 1 × | 66 GB of 259 GB | 365memory-bound | 365 | 319 | not sold | — |
| TPU 8i288 GB · 8.6 TB/s | 1 × | 66 GB of 259 GB | 360memory-bound | 360 | 319 | not priced yet | — |
| B200180 GB · 8 TB/s | 1 × | 66 GB of 162 GB | 354memory-bound | 354 | 159 | $40k | $1.31 |
| B300270 GB · 8 TB/s | 1 × | 66 GB of 243 GB | 354memory-bound | 354 | 293 | $53k | $1.71 |
| GB200 NVL72186 GB · 8 TB/s | 1 × | 66 GB of 167 GB | 354memory-bound | 354 | 168 | $48k | $1.57 |
| GB300 NVL72278 GB · 8 TB/s | 1 × | 66 GB of 250 GB | 354memory-bound | 354 | 305 | $53k | $1.75 |
| MI355X288 GB · 8 TB/s | 1 × | 66 GB of 259 GB | 354memory-bound | 354 | 319 | $28k | $0.99 |
| TPU7x Ironwood192 GB · 7.37 TB/s | 1 × | 66 GB of 173 GB | 347memory-bound | 347 | 177 | $12.00/hrent only | $9.60rented |
| DGX Station GB300252 GB · 7.1 TB/s | 1 × | 66 GB of 227 GB | 344memory-bound | 344 | 266 | $95k | $3.11 |
| Maia 200216 GB · 7 TB/s | 1 × | 66 GB of 194 GB | 343memory-bound | 343 | 213 | not sold | — |
| TPU 8t216 GB · 6.53 TB/s | 1 × | 66 GB of 194 GB | 336memory-bound | 336 | 213 | not priced yet | — |
| MI325X256 GB · 6 TB/s | 1 × | 66 GB of 230 GB | 328memory-bound | 328 | 272 | sold, price not public | — |
| MI300X192 GB · 5.3 TB/s | 1 × | 66 GB of 173 GB | 316memory-bound | 316 | 177 | $13k | $0.52 |
| Trainium3144 GB · 4.9 TB/s | 1 × | 66 GB of 130 GB | 308memory-bound | 308 | 106 | cloud only | — |
| H200141 GB · 4.8 TB/s | 1 × | 66 GB of 127 GB | 306memory-bound | 306 | 101 | $27k | $1.03 |
| Gaudi 3128 GB · 3.7 TB/s | 1 × | 66 GB of 115 GB | 278memory-bound | 278 | 82 | $16k | $0.73 |
| H10080 GB · 3.35 TB/s | 1 × | 66 GB of 72 GB | 267memory-bound | 267 | 11 | $33k | $1.40 |
| Trainium296 GB · 2.9 TB/s | 1 × | 66 GB of 86 GB | 251memory-bound | 251 | 34 | $2.23/hrent only | $2.48rented |
| TPU v5p95 GB · 2.77 TB/s | 1 × | 66 GB of 86 GB | 245memory-bound | 245 | 33 | $4.20/hrent only | $4.76rented |
| TPU v5e16 GB · 820 GB/s | 8 × | 66 GB of 115 GB | 227memory-bound | 227 | 82 | $9.60/hrent only | $12rented |
| TPU v6e Trillium32 GB · 1.64 TB/s | 4 × | 66 GB of 115 GB | 227memory-bound | 227 | 82 | $10.80/hrent only | $13rented |
| GeForce RTX 509032 GB · 1.79 TB/s | 3 × | 66 GB of 86 GB | 218memory-bound | 218 | 34 | $19k | $1.27 |
| GeForce RTX 4090 (used)24 GB · 1.01 TB/s | 4 × | 66 GB of 86 GB | 203memory-bound | 203 | 34 | $13k | $1.03 |
| GeForce RTX 3090 (used)24 GB · 936 GB/s | 4 × | 66 GB of 86 GB | 199memory-bound | 199 | 34 | $6k | $0.59 |
| RTX PRO 6000 Blackwell96 GB · 1.79 TB/s | 1 × | 66 GB of 86 GB | 195memory-bound | 195 | 34 | $17k | $1.05 |
| Inferentia232 GB · 820 GB/s | 4 × | 66 GB of 115 GB | 192memory-bound | 192 | 82 | $4.32/hrent only | $6.26rented |
| Radeon AI PRO R970032 GB · 640 GB/s | 3 × | 66 GB of 86 GB | 157memory-bound | 157 | 34 | $7k | $0.68 |
| Arc Pro B7032 GB · 608 GB/s | 3 × | 66 GB of 86 GB | 154memory-bound | 154 | 34 | $6k | $0.57 |
| Mac Studio M5 Ultra (512 GB)512 GB · 1.2 TB/s | 1 × | 66 GB of 384 GB | 152memory-bound | 152 | 525 | $22k | $1.68 |
| Mac Studio M3 Ultra (512 GB)512 GB · 819 GB/s | 1 × | 66 GB of 384 GB | 116memory-bound | 116 | 525 | $9k | $1.04 |
| MacBook Pro M5 Max (128 GB)128 GB · 614 GB/s | 1 × | 66 GB of 96 GB | 93memory-bound | 93 | 50 | $7k | $0.82 |
| DGX Spark128 GB · 273 GB/s | 1 × | 66 GB of 109 GB | 46memory-bound | 46 | 71 | $7k | $1.71 |
| Ryzen AI Max+ PRO 495 (192 GB)192 GB · 273 GB/s | 1 × | 66 GB of 159 GB | 46memory-bound | 46 | 155 | $7k | $1.66 |
| Ryzen AI Max+ 395 (128 GB)128 GB · 256 GB/s | 1 × | 66 GB of 109 GB | 44memory-bound | 44 | 71 | $3k | $0.95 |
| Doesn’t fit | |||||||
| Mac mini M5 Pro (64 GB) | Needs 66 GB; one device holds 48 GB. | ||||||
Estimates, not benchmarks: decode time per token = max(bytes read ÷ 65% of peak bandwidth, FLOPs ÷ 50% of peak compute) plus a fixed per-layer cost, with bandwidth pooled only across a fast scale-up link. Real engines usually land below these numbers, most of all at batch 1 on many GPUs. $ / M tokens = hardware amortized over 3 years running 24/7 plus power at $0.15/kWh, or the cheapest public hourly rental for chips you can only rent, at your number of users.
How the estimate works.
Memory: will it fit?
Weights are parameters times bytes per parameter. The KV cache depends on the attention design, which we read from each model’s config.json.
need = params × bits ÷ 8
+ users × KV per token × contextLatent attention (MLA) caches one small vector per layer; sliding-window layers stop growing at their window; linear-attention and Mamba layers keep a fixed state. That’s why two 30B models can differ 9× in cache size. Datacenter GPUs group in powers of two up to one 8-GPU server, then by whole servers; desk-side machines go one card at a time.
Speed: how fast per user?
Each decode step streams the active weights and every user’s KV cache through the chip once, and does 2 FLOPs per active parameter per user.
step = max(bytes ÷ bandwidth,
FLOPs ÷ compute) + per-layer costMixture-of-experts models read only the active experts for one user, but a batch touches more experts, so we count the share of weights a batch of b tokens hits: 1 − (1 − active/total)b. Bandwidth adds up only across a fast link like NVLink; beyond it, servers pipeline for throughput, not speed.
Cost: what does a token cost?
Hardware prices are street or reported prices per chip or card. Rack-scale GPUs are the rack’s price ÷ 72, which includes its CPUs, switches and cooling; desk-side machines are the whole system.
$/hour = price ÷ (3 years × 8,760 h)
+ watts × $0.15 per kWhDivide by tokens per hour at your number of users and you get $ per million tokens. One user wastes most of a big GPU; raise the users slider and watch the cost collapse until the KV cache runs out of room. Cloud rental prices are on the Silicon page.
What the estimate leaves out: prefill time (the first token), attention FLOPs, speculative decoding (which can add 1.5–3× per user), engine overheads that vary by framework, and quality loss from quantization. See the serving playbook for what each technique changes.
The architecture writes the bill.
Two models with similar scores can need very different hardware. Four choices in a model’s config decide it: how many parameters it stores, how many each token uses, how its attention remembers the conversation, and the precision of every number. Each one lands on a different line of the spec sheet.
- Seed-OSS-36B-Instruct45 tok/s1 × H100 to hold 40.3 GB
- Gemma 4 31B55 tok/s1 × H100 to hold 32.1 GB
- Qwen3.8 27B60 tok/s1 × H100 to hold 28.1 GB
- gpt-oss-120b204 tok/s2 × H100 to hold 118 GB
- Qwen3.5-122B-A10B139 tok/s2 × H100 to hold 122 GB
- DeepSeek V3.2120 tok/s16 × H100 to hold 672 GB
- DeepSeek V4 Flash172 tok/s4 × H100 to hold 284 GB
active weights that user’s KV cache · speeds are first-order estimates for one user, as in the calculator above
Show the arithmetic for gpt-oss-120b
weights read = 5.1B active params × 1 byte = 5.1 GBA mixture-of-experts token only passes through the experts its router picks, so it reads 5.1B of 117B.
KV read = 18,432 values × 32K tokens + 18,432 × 128 (window) = 606 MBHalf its layers only look back 128 tokens, so their cache stops growing.
tokens/s ≈ 2 × 3.35 TB/s × 65% ÷ 5.7 GB ≈ 763, about 204 after the fixed cost of each of its 36 layers2 chips split the model and read their shares at the same time, so their bandwidth adds up.
Everything has to be loaded.
Every parameter sits in memory whether a token uses it or not. DeepSeek V3.2 activates 37B parameters per token but stores 671B: about 690 GB at FP8, more than a whole 8-GPU H100 server holds. Capacity, not speed, is often what forces a second server.
Each token reads only what it uses.
Decoding reads the active weights once per step, and every token costs 2 FLOPs per active parameter. gpt-oss-120b reads 5.1B per token, so a DGX Spark’s 128 GB of slower memory runs it about as fast as a dense 5B model. The catch: a large batch wakes up most of the experts, so busy servers end up reading nearly the whole model every step.
Memory of the conversation.
Full attention keeps a key and a value for every head in every layer; grouped-query attention shares them across heads, often 8× smaller; latent attention keeps one compressed vector per layer; sliding-window layers stop growing at their window; linear and Mamba layers keep a fixed state. At 128K tokens Seed-OSS-36B keeps about 17 GB per user at FP8, DeepSeek V4 Flash under half a gigabyte. That decides how many conversations share a GPU, and how long ones slow down.
Fewer bits, fewer bytes.
FP8 halves every byte against BF16 and 4-bit halves it again, shrinking both what must fit and what each token reads. Fast maths in those formats needs hardware support: FP8 from Ada, Hopper and MI300, FP4 from Blackwell and MI355X. On older chips 4-bit weights still save memory and bandwidth, but are expanded to 16 bits for the maths, so prefill gets no faster.
So the right chip depends on the model: big mixtures-of-experts want capacity (MI355X and MI455X at 288 and 432 GB, or a 72-GPU rack); long conversations want capacity for the cache and bandwidth to read it; desk-side machines with lots of slower unified memory suit models with few active parameters.
39 open-weights models, by size.
The best-scoring open-weights releases on benchleader.com, with architecture read from each model’s Hugging Face config. Select a name to load it into the calculator. Model not on the list? Paste any Hugging Face model id into Will it fit?, our Hugging Face Space. It reads that model’s config straight from the Hub and ranks 13 chips with the same estimate.
| Model | Params | Active | Released as | Context | KV · 1k tok (FP8) | Attention | Index |
|---|---|---|---|---|---|---|---|
| Moonshot AI · 2026-07 | 2.8T | 104B | MXFP4 experts 1.6 TB | 1M | 14 MB | 24 MLA layers + 69 linear (KDA) layers | 61.5 |
| Moonshot AI · 2026-10 | 2.8T | 104B | MXFP4 experts 1.6 TB | 1M | 17 MB | 24 sparse MLA (DSA) layers + 69 linear (KDA) layers | 61.3 |
| Alibaba · 2026-08 | 2.4T | 95B | BF16 4.9 TB | 262k | 47 MB | 23 attention + 69 linear (Gated DeltaNet) layers | 51.4 |
| DeepSeek · 2026-08 | 1.6T | 49B | FP4 experts + FP8 865 GB | 1M | 9 MB | compressed attention (30 layers at 1/4, 31 at 1/128) + 128-token window (estimated) | 56.0 |
| Xiaomi · 2026-09 | 1.0T | 42B | FP4 experts + FP8 566 GB | 1M | 45 MB | 10 full + 60 sliding-window (128) layers | 56.8 |
| Moonshot AI · 2026-04 | 1T | 32B | INT4 experts 595 GB | 262k | 35 MB | MLA on all 61 layers | 59.3 |
| Mistral AI · 2026-10 | 1T | 52B | FP8 1.1 TB | 1M | 42 MB | MLA + indexer; 63 of 72 layers keep their own KV | 52.0 |
| Thinking Machines · 2026-07 | 975B | 41B | BF16 1.9 TB | 66k | 138 MB | 11 global + 55 local (512-token) layers | 50.0 |
| Zhipu AI · 2026-08 | 744B | 40B | FP8 756 GB | 1M | 48 MB | MLA on all 78 layers; sparse-attention indexer on 21 | 58.8 |
| Zhipu AI · 2026-02 | 744B | 40B | BF16 1.5 TB | 203k | 55 MB | MLA + sparse-attention indexer on all 78 layers | 60.2 |
| DeepSeek · 2025-12 | 671B | 37B | FP8 690 GB | 164k | 43 MB | MLA + sparse-attention indexer on all 61 layers | 56.3 |
| DeepSeek · 2026-10 | 552B | 16B | FP4 experts + FP8 510 GB | 1M | 4 MB | 4 shared compressed KV caches (3 at 1/2, 1 full) + 128-token window (estimated) | 58.0 |
| NVIDIA · 2026-06 | 550B | 55B | BF16 1.1 TB | 1M | 6 MB | 12 attention layers among 108 Mamba-2 / MoE layers | 55.4 |
| Reflection · 2026-10 | 501B | 23B | BF16 1.0 TB | 1M | 64 MB | 9 full + 43 sliding-window (512) layers | 54.2 |
| MiniMax · 2026-06 | 428B | 23B | BF16 854 GB | 1M | 61 MB | GQA on all 60 layers with block-sparse attention (estimated) | 56.1 |
| MiniMax · 2026-10 | 428B | 23B | NVFP4 experts + MXFP8 250 GB | 1M | 61 MB | GQA on all 60 layers with block-sparse attention (estimated) | 53.5 |
| Alibaba · 2026-02 | 397B | 17B | BF16 807 GB | 262k | 15 MB | 15 attention + 45 linear (Gated DeltaNet) layers | 56.4 |
| Zhipu AI · 2026-08 | 320B | 18B | FP8 328 GB | 1M | 7 MB | 11 sparse MLA layers + 34 linear (KDA) layers | 56.5 |
| Xiaomi · 2026-09 | 309B | 15B | FP4 experts + FP8 173 GB | 1M | 24 MB | 9 full + 39 sliding-window (128) layers | 54.8 |
| Tencent · 2026-07 | 295B | 21B | BF16 598 GB | 262k | 164 MB | GQA on all 80 layers (8 KV heads) | 55.7 |
| DeepSeek · 2026-07 | 284B | 13B | FP4 experts + FP8 160 GB | 1M | 6 MB | compressed attention (21 layers at 1/4, 20 at 1/128) + 128-token window (estimated) | 57.1 |
| StepFun · 2026-05 | 196B | 11B | BF16 403 GB | 262k | 59 MB | 12 full + 33 sliding-window (512) layers | 53.0 |
| Alibaba · 2026-08 | 125B | 6B | BF16 360 GB | 262k | 12 MB | 12 attention + 36 linear (Gated DeltaNet) layers (estimated) | 53.9 |
| Ant Group · 2026-08 | 124B | 5.1B | BF16 255 GB | 262k | 4 MB | about 7 MLA layers + 35 linear (KDA) layers (estimated) | 51.7 |
| Alibaba · 2026-02 | 122B | 10B | BF16 250 GB | 262k | 12 MB | 12 attention + 36 linear (Gated DeltaNet) layers | 54.1 |
| NVIDIA · 2026-03 | 120B | 12B | FP8 128 GB | 1M | 4 MB | 8 attention layers among 88 Mamba-2 / MoE layers | 51.1 |
| OpenAI · 2025-08 | 117B | 5.1B | MXFP4 experts 65 GB | 131k | 21 MB | 18 full + 18 sliding-window (128) layers | 45.8 |
| Zhipu AI · 2025-07 | 106B | 12B | BF16 221 GB | 131k | 94 MB | GQA on all 46 layers (8 KV heads) | 46.7 |
| ByteDance · 2025-08 | 36B | dense | BF16 72 GB | 524k | 131 MB | GQA on all 64 layers (8 KV heads) | 49.1 |
| Alibaba · 2026-04 | 35B | 3B | BF16 72 GB | 262k | 10 MB | 10 attention + 30 linear (Gated DeltaNet) layers | 51.8 |
| Google · 2026-04 | 31B | dense | BF16 63 GB | 262k | 430 MB | 10 global + 50 sliding-window (1024) layers | 49.8 |
| Zhipu AI · 2026-01 | 30B | 3B | BF16 62 GB | 203k | 27 MB | MLA on all 47 layers | 51.0 |
| NVIDIA · 2025-12 | 30B | 3.5B | BF16 63 GB | 1M | 3 MB | 6 attention layers among 52 Mamba-2 / MoE layers | 47.4 |
| Meta · 2026-08 | 29.6B | dense | BF16 60 GB | 131k | 27 MB | 13 full + 39 sliding-window (2048) layers | 54.4 |
| Alibaba · 2026-08 | 27B | dense | BF16 56 GB | 262k | 33 MB | 16 attention + 48 linear (Gated DeltaNet) layers | 53.1 |
| Google · 2026-04 | 25.2B | 3.8B | BF16 52 GB | 262k | 108 MB | 5 global + 25 sliding-window (1024) layers | 56.1 |
| OpenAI · 2025-08 | 21B | 3.6B | MXFP4 experts 14 GB | 131k | 14 MB | 12 full + 12 sliding-window (128) layers | 45.6 |
| Alibaba · 2026-02 | 9B | dense | BF16 19 GB | 262k | 16 MB | 8 attention + 24 linear (Gated DeltaNet) layers | 50.4 |
| Alibaba · 2026-02 | 4B | dense | BF16 9.3 GB | 262k | 16 MB | 8 attention + 24 linear (Gated DeltaNet) layers | 49.3 |
Model list, release dates and quality index from benchleader.com (refreshed 14 November 2026). Parameter counts are from each model card; checkpoint sizes, layer layouts and context windows from the Hugging Face repositories, checked 30 October and 14 November 2026. Throughput and cost figures are first-order estimates, not measurements: real engines differ by framework, batch shape and kernel support. KV-cache sizes marked estimated are our reading of a new architecture’s config (DeepSeek V4 and V4.1 compressed attention, MiniMax M3 sparse attention, Ling 3.0, Qwen3.8 Flash Next).