Every token per second is money

On most platforms, a slow dependency is a slow product. On ours it is somebody's income. Hosts are paid per token their machine produces, so throughput converts directly into euros per hour — and a card that decodes at half the rate earns half as much from the same electricity, the same wear and the same hours of not being available for anything else.

That changes how we treat performance. It is not a nice-to-have we get to when there is time. It is the number our hosts are paid on, so it gets measured before it gets shipped.

We choose the engine build by benchmark, not by version number

Every host runs the same inference engine, pinned by digest inside mahout and verified on download. The obvious policy is to track the newest upstream release. We do not, because "newest" is a guess about performance and we can simply check.

A candidate build gets benchmarked against the current one on real consumer cards, with the same model and the same flags, and the pin moves only if the candidate wins. Release notes for a large project cannot reliably tell you what happened to one backend on one class of hardware, and the gap between two builds a few months apart can be several times the throughput in either direction.

The tool that does the comparison ships in the mahout repository, so this is reproducible rather than a claim.

Speculative decoding, where the weights allow it

Some open models are published with a small prediction head that guesses the next couple of tokens from the model's own hidden state. The engine verifies those guesses in a single pass; when they are right, several tokens cost roughly what one would. On a healthy build that is worth more than three times the plain decoding rate — the single largest lever available on consumer hardware today.

It is a property of the weights rather than a setting, so it applies to the catalogue builds that ship such a head. Where it exists, mahout uses it.

The right card, without you configuring anything

Plenty of enthusiast machines have two GPUs. mahout picks the one with the most free memory, tells the engine to use exactly that card, and prints the choice before it downloads a single byte. Free memory rather than total, which means that if you are already using the big card, it quietly takes the other one instead of fighting you for it.

And the boring, decisive things

A quantized KV cache, which cuts the cost of holding a long conversation roughly fourfold. Flash attention on. Context length and concurrency set per build rather than globally. And above all, builds sized so the model stays entirely on the GPU — a model that spills into system memory decodes several times slower, and no amount of cleverness elsewhere recovers that.

What we will not do

Compile our own engine binaries. We measured it: a locally compiled build of the same release lands within about ten percent of the official archive. Ten percent is not worth becoming the distributor of an unsigned binary that runs on other people's computers, so we ship the upstream artifact and spend the effort on choosing it.

And we will not quote you a throughput figure for your hardware. Run mahout bench and get a measured one, signed on your own machine. The full reference is in the knowledge base.