Intro
Massive language fashions are extraordinarily power-hungry and require a major quantity of GPU assets to carry out nicely. Nonetheless, the transformer structure doesn’t take full benefit of the GPU.
GPUs, by design, can course of issues in parallel, however the transformer structure is auto-regressive. To ensure that the subsequent token to get generated it has to take a look at the entire earlier tokens that got here earlier than it. Transformers don’t will let you predict the subsequent n tokens in parallel. In the end, this makes the era part of LLMs fairly gradual as every new token should be produced sequentially. Speculative decoding is a novel optimization approach that goals to resolve this subject.
There are just a few completely different strategies for speculative decoding. The approach described on this article makes use of the 2 mannequin method.
Speculative Decoding
Speculative decoding works by having two fashions, a big most important mannequin and a…