Accelerating Gemma 4: faster inference with multi-token prediction developers

Why speculative decoding?

The technical reality is that standard LLM inference is tied to memory bandwidth, which creates a significant latency bottleneck. The processor spends most of its time moving billions of parameters from VRAM to the processing units just to generate a single token. This leads to underutilization of computing power and high latency, especially on consumer hardware.

Speculative decoding decouples token generation from verification. By combining a heavy target model (e.g. Gemma 4 31B) with a light drafter (the MTP model), we can use idle machines to use the drafter to “predict” multiple future tokens at once – in less time than it takes the target model to process just one token. The target model then checks all of these proposed tokens in parallel.

How speculative decoding works

Standard models of large languages ​​generate text autoregressively and produce exactly one token at a time. While this process is effective, it requires the same computational effort for…