On-Chip LLM: Sampling Without Asking
The FPGA decoded a reply in 6.6ms. It landed in the browser in 100. The gap was one unmeasured loop, and the fix was the Gumbel-max trick.
Part of the on-chip LLM on a $250 FPGA series. A lesson in profiling the whole round trip, not the kernel you are proud of.
The faithful model decodes a full reply in about 6.6 milliseconds of fabric time. The reply used to land in the browser in about 100. Almost none of that gap was the fabric. Most of it was one embarrassing loop nobody had measured.
To get variety, the chat samples (Sampling knob that rescales the logits before a token is drawn: low values play it safe, high values take risks. Zero collapses to always picking the best token. 0.85, Top-k sampling: only the k most likely tokens are allowed into the draw, so the long tail of nonsense can never be picked. 40) instead of always picking the single best token. But the probabilities live in the fabric. So every token, the host read all 193 output The model's raw output scores, one per possible next token, before they are turned into probabilities. back over /dev/mem: the Linux device that lets a privileged process read/write physical memory directly, used here to poke the fabric's registers., ran a Turns a row of scores into a probability distribution. The fiddliest non-linear to build in hardware, and the one that grows with context length. and a draw on the Arm core, and only then knew the next character. Measured, that readback was about 58% of a reply. We were serving a ~19,000-token-per-second model through a 1,000-token-per-second straw.
The fix is a lovely identity. Sampling from softmax(logit / T) is exactly the same as taking the Picks the single highest-scoring option. Greedy decoding is just argmax over the output logits. of logit + T * g, where g is Gumbel noise. This is the Gumbel-max trick: sampling from softmax(logit/T) is exactly argmax(logit + T*noise). It lets the existing argmax hardware do temperature sampling with no readback. trick. The thing that picks the sample is an argmax, and the fabric already has an argmax. So the sampler is not new hardware: it is the existing argmax with a precomputed noise value added to each logit, and the host’s job collapses from 193 reads per token to one seed write per request. As a bonus, seed = 0 zeroes the noise, which makes the sampler bit-identical to Decoding by always taking the single most likely next token. Deterministic and repeatable, which makes it ideal for bit-exact verification, and boring, which is why the chat samples instead. decode. One datapath, two behaviours.
That moved the localhost round-trip ceiling from ~1,000 to ~5,600 tok/s, about 5.6x, with the fabric record completely untouched. The public number through the Cloudflare Cloudflare Tunnel: exposes a service to the internet without a static IP or open ports, with TLS and a DDoS buffer in front. sits around 1,658, and that remainder is honest network distance, not silicon.
The two ceilings
A live chatbot has two speeds, and conflating them is the easiest way to lie to yourself.
Fabric tok/s is how fast the silicon decodes: pure logic cycles over the clock. Round-trip tok/s is how fast a reply lands in your browser: the fabric plus the host loop plus the network plus the tunnel. They are wildly different numbers.
This is the part most write-ups skip. The big number is real, the small number is also real, and they measure different things. The lesson worth keeping: profile the whole round-trip, not the kernel you are proud of.
Next: Serving an FPGA to strangers, or back to the main post. The code is on GitHub.