one model, every card: how a 27b runs on 8 gb

A friend asked the question that decides whether any of this works: "I have a 3060 with 12 GB. Am I out?"

Six weeks ago the honest answer was yes. Our catalogue had two tiny models, and a model id and a quantization were the same thing — you either fit the one build we shipped, or you served nothing. Today his card serves Qwen3.8 27B, and this is what had to change.

a model is not a file

The fix was a distinction we should have drawn on day one. A model is what a customer asks for: qwen3.8-27b. A build is a concrete artifact — one quantization, one engine, one exact file from one exact upstream revision, with the SHA-256 published for it.

One model, several builds. The customer's request never changes; the host picks the build its card can actually run:

Card	Build	On disk
24 GB	Q5_K_M — reference	18.4 GB
20 GB	Q4_K_M — reference	15.3 GB
16 GB	Q3_K_XL — balanced	12.2 GB
12 GB	Q2_K_XL — compact	9.2 GB
9 GB	IQ2_XXS — compact	6.8 GB

We label the trade rather than hide it. A 27B at 2-bit is a real model with a real quality cost — often still better than a 4B at 4-bit, but not always, and not for code. Calling that "compact" and saying why is more useful than quietly serving it.

the flag pair that doubled our context

Sizing those builds taught us something embarrassing. Our first attempt loaded Qwen3.8 Q4_K_M onto a 24 GB RX 7900 XTX with 16K of context and four batching slots. The card had 345 MiB free and spent the run thrashing.

The fix came from reading a configuration that had been serving that exact model daily on that exact card: --flash-attn on with --cache-type-k q4_1 --cache-type-v q4_1. A quantized KV cache is roughly a quarter the size of the default. Same card, same model, after:

before:  16K context  →    345 MiB free   (thrashing)
after:   32K context  →  6,812 MiB free

Double the context and six gigabytes of headroom, from one flag pair. It is now the baseline every GGUF build launches with.

the estimate is not the promise

Every build carries a VRAM estimate: weights, plus KV cache for its context and slots, plus headroom. That estimate decides what is worth downloading — it is there so nobody pulls 15 GB to discover it does not fit.

It is not what gets advertised to customers. A build becomes claimable on a machine only after that machine has loaded it and benchmarked it. Estimates gate downloads; measurements decide routing. When your dashboard says your card does 37 tokens a second, that number came off your card, not off a table.

what your card can serve

Your card	Models
6–8 GB	Gemma 4 E4B, the small Qwen2.5 models
10 GB	+ Gemma 4 12B
12 GB	+ Qwen3.8 27B (Q2_K_XL)
16 GB	+ gpt-oss 20B, Qwen3.8 at Q3_K_XL
24 GB	everything, incl. Gemma 4 26B-A4B and Qwen3.8 at Q4/Q5

The catalogue has the full list with prices. Or install the agent and ask your own machine:

$ mahout models
this machine: 24 GiB VRAM detected

* qwen3.8-27b  (Qwen3.8 27B Instruct)
      > UD-Q5_K_M      18.4 GiB  needs ~23 GiB   32K ctx  reference   <- this machine

So: 12 GB is not out. It is, as of this week, enough to serve the flagship.

See the catalogue · Which model fits my GPU? · Install mahout