Aug 9, 2026 · FPGA

On-Chip LLM: Inside the Chip

The datapath of a 3.16M-parameter transformer living in FPGA fabric: wide-word GEMV, the dual-port split-brain, and the proof that 16 streams is a hard wall.

FPGAHardware

Part of the on-chip LLM on a $250 FPGA series. This is the hardware. Read the main post first for why any of this exists.

Here is what is actually in the fabric.

inside the chip: 16 streams, zero DRAMArm A53 host/dev/mem MMIOwrites 1 seed reg per requestProgrammable Logic (PL)sequencer FSM - embed to blocks to head to sample to append-KV, then loop. CPU never in the loop.URAMweight image~12.6 MbitINT4 residenttrue dual-portcohort A - 8 streamsGEMV wide-word, LANES=128weights transposed, 2 MAC/DSPcohort B - 8 streamsindependent URAM portstream-desync dissolvesN = 16 split-brainnon-linear bricksLayerNorm (rsqrt)softmax (running max)GELU (LUT + interp)dequant per-channelP-wide, arbitratedhead + samplerargmax+ Gumbel noiseseed=0 -> greedy1 token outBRAM - on-chip working memoryactivations + per-stream scratch | KV cache (on-chip window)AXI-Lite regsAXI-DMA boot weightsport 1port 2tokenKVclock set by PS PLL via --fclk (snaps to 1000/N MHz: 125, 142.9, 166.7, 200...) | record build runs at 200 MHz
The whole model lives here. One resident INT4 weight image in UltraRAM is read through two independent ports by two 8-stream cohorts (the split-brain layout that gets to N=16), activations and the KV cache sit in BRAM, and a sequencer FSM drives the entire per-token forward with the CPU completely out of the loop. The only thing the host does per request is write a single random seed.

The pieces:

  • The weight image is one resident 4-bit integers (16 levels). The weights are stored as INT4, which is what makes the model small enough to fit on-chip. blob, ~12.6 Mbit, sitting in UltraRAM: the big, wide on-chip SRAM (~18 Mb, 64 blocks). Holds the resident INT4 weight image. Crucially, it is true dual-ported, which enables the two-cohort 'split-brain'.. It is loaded once at boot (UltraRAM cannot be initialised from the The compiled configuration file that programs the FPGA fabric, loaded onto the Kria at boot., so “resident,” not “baked in”). Every layer reads its slice of this one memory.
  • General Matrix-Vector multiply. Generating one token at a time makes every linear layer a matrix-times-vector, which is the bulk of the work. the wide-word way. The naive layout gives each compute Processing Element (a 'lane'): one MAC unit. More lanes means more multiply-accumulates per clock cycle. its own memory bank, which caps you at about 64 lanes before the fabric runs out of banks. Instead the weights are stored transposed, so a single wide UltraRAM word (1024 bits is 256 INT4 nibbles) feeds 256 lanes the same column, all sharing one activation. That one change is what unlocks the wide datapath. The record build runs 128 lanes, the faithful chat build runs 256.
  • Two cohorts, the split-brain. UltraRAM is genuinely dual-ported. So the design runs two independent 8-stream cohorts, each reading the same weight image through its own port, sharing only the weights and an arbitrated set of non-linear units. Eight plus eight is N=16. Because a cohort never shares a weight pass with the other, an entire class of stream-synchronisation bugs simply does not exist.
  • The non-linear bricks. LayerNorm: normalises a vector to zero mean and unit variance, then applies a learned gain. A standard transformer building block. (with a reciprocal-square-root done by a seed table plus Newton-Raphson), Turns a row of scores into a probability distribution. The fiddliest non-linear to build in hardware, and the one that grows with context length. (running-max, no overflow), GELU: the smooth activation function inside the MLP. Implemented in fabric as a lookup table plus interpolation. (lookup table plus linear interpolation), and per-channel dequant. Each is P-wide and each was proven bit-exact in simulation before it was allowed near silicon.
  • The sampler is the Picks the single highest-scoring option. Greedy decoding is just argmax over the output logits. hardware the Decoding by always taking the single most likely next token. Deterministic and repeatable, which makes it ideal for bit-exact verification, and boring, which is why the chat samples instead. decoder already had, plus on-chip Gumbel noise. That story is its own article: Sampling without asking.
  • The host interface is AXI-Lite: the simple register interface the CPU pokes one transaction at a time. Fine for control, too slow to be in the per-token loop. for registers (poked from the Arm side over /dev/mem: the Linux device that lets a privileged process read/write physical memory directly, used here to poke the fabric's registers.) and AXI-DMA: the high-throughput streaming interface, used here to load the weights into URAM at boot. to stream the weights in at boot. After boot, in the fast path, the host writes one register per request and otherwise stays out of the way.

Getting the CPU out of the loop

The single most important architectural move in the whole project was the jump from 11 tok/s to 44, and it is not about going wide or going fast. It is about getting the CPU out of the inner loop. A The in-fabric state machine that drives the whole forward pass (embed, blocks, head, sample, append-KV, loop) autonomously. The thing that turns ~100 tok/s into ~10k. state machine (Finite-State Machine: the hardware controller (the 'sequencer') that runs the entire per-token forward pass with zero CPU involvement.) in the fabric runs the entire per-token forward pass (embedding, four transformer blocks, final layernorm, the output head, the sample, appending to the KV cache: the stored per-layer Keys and Values for every past token, so each new token only computes its own position instead of re-reading the whole context., loop) with the Arm core touching nothing. The CPU-in-the-loop versions all asymptote to the The quad-core Arm Cortex-A53 CPU on the KV260 (~1.33 GHz). The baseline the fabric is compared against, and the orchestrator when the CPU is in the loop.’s own speed, because every handoff over the AXI-Lite register interface costs a full transaction. Once the fabric runs itself, the bandwidth wall is the only wall left.

Why it stops at 16 streams

The obvious next move from N=16 is N=32, or packing more multiply-accumulates into each DSP. Both are impossible on this chip, and the project proved it rather than assuming it.

Each DSP48E2: the FPGA's dedicated hardware multiplier blocks (1248 on this chip). Each can pack two INT4-by-INT8 multiply-accumulates. can do two INT4-by-8-bit integers. The activations are INT8. Multiply-ACcumulate: the one-multiply-one-add operation matrix multiplies are built from. The fundamental unit of compute here. using Xilinx’s Operand packing: fitting more than one multiply into a single DSP by placing two small operands side by side in its wide multiplier. trick. Could it do three? No, on two independent walls: three non-overlapping nibble products need 28 bits but the DSP port is 27 bits wide, and three 1024-element neurons hold 66 bits of accumulator against a 48-bit accumulator. There is a script, dsp3_pack_proof.py, that falsifies the three-MAC scheme over 1.2 million randomised lane products and confirms the two-MAC scheme with zero mismatches. Going wider on streams would need 2,048 DSPs the KV260 does not have.

So past the stream ceiling, the levers stopped being “more streams” and became “fewer cycles” and “higher clock.” This is the honest shape of optimisation on real silicon: you prove the doors that are closed, then you stop pushing on them.

The clock ceiling

The faithful build was stuck at 166.7 MHz because the timing report collapsed under routing congestion at the higher clocks, the router simply gave up and stopped optimising for speed. The fix was not cleverness, it was tidiness: the LayerNorm input buffers were stored as a pile of per-lane arrays, which synthesises into a thicket of address multiplexers and enormous fan-out nets. Rewriting them as one wide word per row (the layout the rest of the design already used) freed about 11,900 Flip-Flop: the FPGA's basic 1-bit storage primitive. and 3,800 Look-Up Table: the FPGA's basic logic primitive. (Also used loosely for a precomputed math table, e.g. for GELU; context disambiguates.) and killed two of the worst fan-out trees. That was enough for the router to close cleanly. The design went from failing timing to closing with margin, then overclocked bit-exact to 200 MHz on silicon: the same greedy token stream, identical, at 142.857, 166.7, and 200 MHz. The clock only comes in quantised steps of 1000/N MHz (N = 7, 6, 5), so 200 is the last rung before 250 MHz (N=4), which is a hard wall where the fabric hangs.

Next: Sampling without asking, or back to the main post. The code is on GitHub.