跳转至

Speculative Decoding（推测解码）

约 917 个字 预计阅读时间 4 分钟

Intuition (one sentence)

Speculative decoding first drafts several tokens with a cheap method, then verifies them in one pass with the large model: keep correct guesses, discard wrong ones. If guesses are good, one step can advance multiple tokens.

Why this helps

  • During decoding, GPU time is often dominated by loading model weights from memory (memory-bound), not pure compute.
  • The idea is: weight-loading cost is still paid, but we do extra verification work in the same round, so each round becomes more productive.
  • Guessing extra tokens increases KV cache usage, but usually much less than model-weight footprint.

How to draft guesses

  1. N-gram: continue based on patterns already seen in context (fast, simple).
  2. Draft model: use a smaller model to propose several next tokens (usually more accurate, but requires running another model).

Drafting can be parallel (faster, less accurate) or autoregressive (slower, more accurate).

Verify

  • Greedy / temperature 0: a tiny mismatch can collapse the whole drafted span, so speculative decoding often helps less.
  • Stochastic decoding: accept/reject can follow probability-based rules (details vary by implementation).

Connection to lossless sampling (optional detail): let draft distribution be q and target distribution be p. A common implementation uses rejection sampling: accept a candidate with probability min(p/q, 1), then sample rejected mass from the residual distribution norm(p - q). This preserves equivalence to sampling directly from p.

落地难点（工程向）

  • 草稿模型也要 KV，显存怎么分、怎么和主模型配合。
  • 草稿很小：不一定和主模型用同一套并行；多卡时可能出现 某几张卡闲着、显存却不好匀（例如 vLLM 一类约束）。
  • 给「待验证的一串词」预留 KV 时，可能 撞上块（block）边界，要处理丢弃或重排。
  • 猜几个词 要调参；有的请求 根本不跑 spec decode，要区分对待。
  • 小模型本身若强行上大并行，未必划算（相对它自己的规模）。

论文阅读

参考资料

kv 稀疏