Speculative decoding

Why generation is slow

A large autoregressive language model produces text one token at a time. Every new token requires another full run of the model, so decoding K tokens takes K serial runs^[1]. Input can be processed in parallel, but generation itself is a chain of dependent steps: token number three cannot be computed before token number two exists. That serial structure is what makes inference from large autoregressive models slow^[1], and it is the bottleneck speculative decoding attacks.

The observation behind it

The starting point is that hard language modelling tasks often contain easier subtasks that a more efficient model can approximate well^[1]. Producing a whole sentence well is hard work, but much of what happens inside the sentence is routine. If a cheap model can handle the routine parts, the expensive model does not have to be consulted separately for every single token.

How it works

Speculative decoding turns that division of labour into an algorithm. A smaller approximation model proposes several tokens^[1]. The large model then runs in parallel on those proposals, using speculative execution^[1], and a novel sampling method decides how the proposals are used^[1]. Because the large model evaluates a whole block of candidate tokens at once instead of one token per run, its work is packed more densely and the number of serial steps falls.

A small illustration: suppose the small model drafts four tokens in a row. The large model can score all four positions in a single parallel pass, rather than being called four times. Where the draft matches what the large model itself would have produced, the tokens are kept; where it departs from that, the relevant part is not accepted. The sampling method is what makes this selection sound.

The guarantee is exactness. The algorithm produces exact decoding from the large model and does not change the output distribution^[1]. The text obtained is the text the large model alone would have produced; only the time taken differs.

Where it came from

Leviathan et al. introduced speculative decoding in November 2022 as an algorithm that samples from autoregressive models faster by computing several tokens in parallel^[1]. The proposal appeared in a paper titled "Fast Inference from Transformers via Speculative Decoding", and the technique it describes is named after speculative execution^[1].

Results

On T5-XXL the authors measured a 2X-3X acceleration compared with the standard T5X implementation, with identical outputs^[1]. The speedup therefore came without any change in what the model said.

What it is and is not

Because the output distribution is preserved, speculative decoding is a latency optimisation rather than an approximation, unlike quantization or pruning^[1]. Those methods trade some fidelity for smaller or cheaper computation; speculative decoding does not trade fidelity at all. It can also accelerate existing off-the-shelf models without retraining or architecture changes^[1], so it sits alongside a model rather than requiring a new one.

Limits

The large model still does the deciding. A smaller model only proposes candidates, and the large model's own computation remains the source of the final output^[1]. The saving comes from rearranging the work, not from replacing the expensive model with a cheap one: speculative decoding is not a way to make a model that answers roughly the same thing, it is a way to get the same answer sooner. How much time is actually saved depends on how closely the smaller model's proposals match what the larger model would have said, so the benefit is not a fixed number but a property of the pair of models involved.