NarrowSpec:推测解码中草稿隐藏宽度无需与目标匹配

媒体报道 Hugging Face Papers 原文语言:中文

本文介绍了NarrowSpec方法,证明了读取目标隐藏状态的草稿模型(如DFlash和EAGLE-3)在运行其残差流时,宽度降至一半或四分之一仍能保持接受率。在B200和GB10上的测试表明,这可将端到端解码速度提升1.06至1.44倍,且在更大批次下也没有性能倒退。

全文

Speculative decoding is widely used to speed up generation in large language models (LLMs), with a small draft model proposing tokens that one forward pass of the target model verifies. Draft models that read the target's hidden states, such as DFlash and EAGLE-3, typically inherit the target's hidden width. We show that the draft's whole residual stream can run at half or a quarter of that width while largely preserving acceptance. For DFlash, which borrows the target's vocabulary projection, the trained output adapter folds into it, giving one narrower projection with logits unchanged up to floating-point rounding. We test two drafter families, DFlash and EAGLE-3, on two targets, Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4), in SGLang on a B200 and a DGX Spark (GB10). At batch size 1 and temperature 0, narrowing the draft to half or a quarter width speeds up end-to-end decoding by 1.06-1.12x on B200 and 1.13-1.44x on GB10. The speedup holds at larger batch sizes. From batch 32 to the largest batch each GPU fits, narrowing reaches up to 1.21x on both GPUs, and no narrow draft is slower than full width.

View arXiv page View PDF Add to collection

正文以来源语言显示。

参考来源

← 返回每日AI报告