a GPU died mid-sentence. the sentence finished.

Our pool has an honest structural weakness: it is built from computers that are allowed to walk away. A host's machine pauses itself the moment its owner sits down, because that is the deal — the machine stays theirs. Breakers trip. Wi-Fi drops. A gaming PC in a living room is not a datacenter, and we have never pretended otherwise.

Until this week, that honesty had a sharp edge. If the machine serving your request died before the first token, we requeued the job and you never knew. If it died after the first token — mid-answer, mid-word — your stream ended in an error. Retrying yourself meant paying for the prompt again and losing everything already generated.

This week a GPU died mid-sentence in our test rig, and the sentence finished anyway. Here is how.

finishing someone else's sentence

The fix rests on a property of chat models that is easy to forget: a conversation that ends with an unfinished assistant message is a perfectly valid prompt. Hand the model its own half-written answer as the trailing message, and it does not start over — it keeps writing, from the exact character where the text stops.

So when a host disappears mid-stream, the gateway builds a continuation job: the original messages, plus everything already delivered to you, as a final assistant message. Any machine serving that model can claim it. The engine picks the sentence up where it broke — we verified it continues cleanly from half a word — and streams only the new tokens.

Your connection never closes. The two halves arrive on the same stream, stitched in order, and the answer reads as one answer, because it is one.

how the gateway notices

Two triggers. The obvious one: the host reports the job failed. The subtle one: it stops reporting anything at all. A stream that has produced no token for 20 seconds is treated as dead — a healthy engine emits many tokens a second, so twenty seconds of silence is not thinking, it is absence.

One continuation per request, deliberately. If the second machine dies too, you get an honest error, not an endless relay hiding a sick pool. Failover is for surviving bad luck; it must never become a way of not noticing bad health.

who pays for the dead half

The accounting is where a marketplace shows its character, so we wrote it down as rules:

• The dead host earns nothing. A receipt-backed payout requires a completed job; a stream it abandoned is not one.
• You never pay for the prompt twice. The continuation re-reads the original prompt plus the partial answer, but those input tokens are billed at zero. You pay for output — each token exactly once, whichever machine produced it.
• The rescuing machine earns its own output. It gets paid for the tokens it generated, not a discount for arriving second.

If you integrate against the API and want to know when this happened, the final chunk carries x_elephant: { resumed: true } (buffered responses get an x-elephant-resumed header). Most users will never look. The point of the feature is that they never have to.

what it does not fix

A model cannot un-say tokens. If the first machine emitted half an answer, those words are already on your screen; the continuation writes what follows, it does not revise. And a pool with one serving machine still has one point of failure — which is exactly why the next milestone on our list is boring, deliberate redundancy.

Consumer hardware disappearing is not an edge case for us; it is the substrate. The job was never to pretend the machines are reliable. It was to make the answer reliable anyway.

When a job goes wrong · How a request finds a GPU · Try the API