What makes your GPU fast, and what that pays

You are paid per token your machine produces. So throughput is not a benchmark score here — it is the hourly rate. A card that decodes twice as fast earns twice as much from the same electricity, the same wear and the same hours of being unavailable for anything else.

Most of what determines that is our job, and we do it once for everybody. Some of it is yours.

What we tune for you

The engine build is chosen by measurement

Every host runs the same inference engine: a specific llama.cpp release, pinned by digest inside mahout and verified on download. We do not simply track the newest tag. A candidate build is benchmarked against the current one on real consumer cards, and the pin moves only if the candidate is faster. Upstream performance on one backend and one class of hardware is not something release notes reliably describe, and the difference between two builds a few months apart can be several times the throughput.

We ship the upstream artifact rather than compiling our own, because a locally compiled build of the same release measures within about ten percent — not enough to justify putting an unsigned binary on other people's machines.

The right card, automatically

On a machine with more than one GPU, mahout picks the card with the most free memory and tells the engine to use exactly that one. It reports the choice before it downloads anything:

  [  PULLING] gpu Vulkan1 (Radeon RX 7900 XTX (RADV NAVI31) (24576 MiB, 22726 MiB free))

Free memory rather than total, which has a useful consequence: if you are already using the big card for something else, mahout takes the other one instead of competing with you for it.

Speculative decoding, where the model supports it

Some open models ship a small prediction head alongside the weights, trained to guess the next two tokens from the model's own hidden state. The engine can then verify several guesses in a single pass — when they are right, you get several tokens for roughly the cost of one. On a healthy build this is worth more than three times the plain decoding rate.

It depends on the weights, not on configuration, so it applies to the builds in our catalogue that were published with such a head. Where it is available, mahout enables it; where it is not, nothing is lost.

Memory spent on the right things

The KV cache — the model's memory of the conversation so far — is stored quantized, which cuts its cost roughly fourfold. That is what lets a 27B model hold a long context on a consumer card instead of spending all its VRAM on bookkeeping. Flash attention is on, the context length and the number of concurrent slots are set per build rather than globally, and the model is kept entirely on the GPU: a model that spills into system memory decodes several times slower, so the catalogue sizes builds to avoid it.

What is in your hands

• Which card, if you have several. mahout chooses well by default; mahout serve --gpu 1 overrides it.
• Your power cap. Lower is cooler and quieter and cheaper to run, and slower — so it earns less. A real trade, and yours to make. See deciding when your GPU works.
• What else is running. The engine competes with anything else on the card. vram_reserve_bytes keeps a slice free for your own use.
• Which models you serve. A smaller model decodes faster but sells for less per token; the balance differs by card. Your dashboard shows euros per serving hour for each one your machine has measured.

Seeing your own numbers

Nothing here is a promise about your hardware. Run the benchmark and find out:

$ mahout models                    # what fits this card
$ mahout bench --model qwen3.8-27b # load it, measure tokens per second

The result is signed on your machine and appears in your dashboard as a measured figure. Until a model has been benchmarked on your hardware it is shown as an estimate and never quoted as an earnings number — see what your GPU actually earns.