Which model can my GPU serve?
From 6 GB to 24 GB of VRAM: which open-weight models your card can run, what quantization means for quality, and how to measure your own machine instead of guessing.
A model is not one file. The same model ships in several quantizations — the same weights stored at lower precision — and each one needs a different amount of VRAM. That is how a 27B model reaches an 8 GB card at all.
The ladder
| Your card | What it can serve |
|---|---|
| 6–8 GB (GTX 1060, RX 580, RTX 3070) | Gemma 4 E4B, the small Qwen2.5 models |
| 10 GB (RTX 3080) | + Gemma 4 12B |
| 12 GB (RTX 3060 12G, RTX 4070) | + Qwen3.8 27B at Q2_K_XL |
| 16 GB (RTX 4060 Ti 16G, RX 6900 XT) | + gpt-oss 20B, Qwen3.8 at Q3_K_XL |
| 24 GB (RTX 3090/4090, RX 7900 XTX) | everything, incl. Gemma 4 26B-A4B and Qwen3.8 at Q4_K_M or Q5_K_M |
The full list, with prices and licences, is on the models page.
What quantization costs you
Fewer bits per weight means a smaller file and less VRAM, at some cost in quality. The catalogue labels every build so the trade-off is visible rather than hidden:
- reference — what we benchmark against and would run given free choice (Q4/Q5).
- balanced — near-reference, noticeably smaller (Q3).
- compact — fits small cards, with a real quality cost (Q2, IQ2).
A 27B model at 2-bit is often still better than a 4B model at 4-bit — but not always, and not for code. When a model publishes quantization-aware weights (Google does for Gemma), we use those: they lose far less than a quantization applied afterwards.
Don't guess — measure
VRAM figures in any catalogue are estimates: weights, plus the KV cache for the context and every parallel slot, plus room to compute. Your driver, your desktop and your other applications all take a share. We learned this the blunt way — a 12B model with four batching slots on a 16 GB card left 27 MiB free and decoded at 2 tokens per second — so the catalogue now scales the slot count to the build, and mahout measures rather than predicts:
$ mahout models # what the catalogue offers, and what fits your card
$ mahout bench --model qwen3.8-27b # load it for real, measure tokens/secondUntil a model has been benchmarked on your machine, your dashboard shows it as an estimate and never quotes an earnings figure for it. A measured number is the only kind worth showing.