How a request finds a GPU

Your request does not go to a server we own. It goes to a queue, and a machine somewhere in Europe picks it up. That indirection is the product, so it is worth knowing exactly how it works.

The path, step by step

1. The gateway authenticates you and estimates what the call will cost. If your prepaid balance or the key's spend cap will not cover it, you get a 402 before anything is queued — never a surprise bill.
2. The job is queued for one model. It carries your messages and nothing about you: no name, no email, no key beyond an internal id used for billing.
3. A host claims it. Hosts poll outward over HTTPS and take work for the models they serve. A host can only claim a model it has benchmarked on that machine — the catalogue's opinion that a build fits is not enough.
4. Tokens stream back through the gateway as they are produced. You see them arriving; we do not wait for the whole answer.
5. The job settles. Tokens are counted, your balance is debited to the micro-cent, the host is credited 80%, and the prompt is dropped from the record.

Why hosts poll instead of us calling them

Every host is behind a home router. If the pool had to reach in, every host would need port forwarding, a static address or a tunnel — and each of those is a support ticket and a security hole. Polling outward means a machine works the moment mahout starts, with nothing configured and no inbound port open. It is the single decision that makes consumer hardware practical.

What decides which host gets your job

Today: the models a host has qualified, whether it is idle, and whether it is paused. A machine its owner is actively using pauses itself and simply stops claiming.

What that deliberately does not yet include is reputation-weighted routing and latency classes. They are designed (see the technical specification, §5) and not built. We would rather say that than imply a scheduler cleverer than the one running.

When something goes wrong

• No host claims the job — you get a timeout and the job is marked failed. You are not billed for tokens nobody produced.
• A host dies mid-answer — today the request fails rather than silently truncating. Transparent resume across hosts is on the list; a half-answer presented as whole would be worse than an error.
• You run out of credit mid-stream — the current job completes and settles; the next one gets a 402.

What this costs you in latency

A poll-based queue adds a fraction of a second before work starts. In exchange, the pool is made of hardware that would otherwise be idle, which is why the price per token is what it is. For chat and batch work the trade is invisible. For hard real-time it is not the right architecture, and we would rather tell you that than sell it.