The model that drafts its own next token.

We turned speculative decoding on, and Anton's Mac mini got slower. That was the morning of the 2nd of September, and it is why this story has a second day.

The idea first. A model generates one token at a time, and for a big model every token costs a full pass over the weights. Speculative decoding lets a smaller model (the drafter) propose a few tokens ahead and has the big model verify them all in one pass. Right guesses give you two or three tokens for the price of one pass and a bit; wrong guesses cost you the bit. The output does not change either way: the big model only keeps a drafted token it would have produced itself.

Two current models ship with their own drafter. Qwen 3.8 carries a NextN head, one extra layer that predicts the token after the next one. Gemma 4 ships a four-layer assistant model next to the 26B main model. Both now run in dasLLAMA on Metal. The drafter is a second file you load next to the model. We measured on Boris's new M5 Max laptop and on Anton's M4 Pro Mac mini, which answers over ssh from the other side of the globe.

What we tried

How deep to draft, first of all. One draft token means the big model computes two positions in one pass: the position it was at anyway, where its own answer either matches the draft or replaces it, and the position after the draft, which only counts if the draft matched. So a round always yields one token and, when the draft was right, a second. Two draft tokens mean three positions, and the second draft only pays if it is accepted often enough to cover that third row.

We tried one against two on each machine and let the tuner decide, the way it decides everything else on a box, and every machine we own settled on one. Three and four were so far behind on the development build that they never reached the ruler.

The Gemma assistant reads the main model's hidden state as its input, and there are two places to read it from, before or after the final norm. After won on acceptance, so after ships.

We also checked whether squeezing the main model harder hurts the drafter. The 8-bit Gemma accepted 76.5% of the drafts, the 4-bit one 74.6%. Not much, then.

Then the kernels, which turned out to be the real story.

How we measured

The board on daslang.io uses llama-bench, and llama-bench cannot measure speculative decoding. So we built a small ruler for ourselves. It takes four chat prompts, renders each through the model's own chat template with thinking off, and runs two engines on the identical rendered text for the same number of tokens - ours, and a stock llama.cpp built from upstream exactly as they ship it.

We focused on two models, gemma-4-26B-A4B-it-Q4_K_M with its Q8_0 assistant and Qwen3.8-27B-Q4_K_M with its Q8_0 NextN head, and both engines read the same drafter file.

The four prompts are the first turns of four tasks from hemingkx/Spec-Bench: a travel blog post to write, a news article to summarize, a GSM8K arithmetic problem, and a trivia question.

Two of the ruler's rules exist because each one cost us an afternoon.

Rule 1. Same thinking mode on both sides. Our very first llama.cpp run had thinking on while ours had it off, and its acceptance read 91%. A thinking-on answer is an outline, not a blog post - a different text with a different acceptance rate. The ruler now renders one prompt and hands the same bytes to both engines.

Rule 2. Let the box cool down. Surprisingly, a Mac that has been working hard for ten minutes decodes 13 to 18% slower than a cool one. Who knew. Every arm now waits three minutes on the 27B before it measures. The release build also runs before anything else: when the measuring program had already run the engine in-process, everything it measured afterwards came out 20% low, for reasons we still find funny.

What we measured

So we ran it, and the drafter earns its keep: about one extra token in five, and not a token more.

tokens/s	dasLLAMA off	1 draft	2 drafts	llama.cpp off	1 draft	2 drafts
Gemma, M5	122.7	145.9 (75.2%)	148.1 (62.6%)	97.7	133.7	142.0
Gemma, M4	59.2	67.5 (74.6%)	55.6 (62.9%)	52.8	67.2	62.5
Qwen, M5	27.6	33.0 (77.4%)	31.4 (75.1%)	25.2	34.1	33.5
Qwen, M4	12.7	14.0 (77.4%)	12.4 (75.1%)	11.5	13.8	12.5

In parentheses, the acceptance rate.

The same drafter drafts the same on both sides, 75 to 94% accepted at the first position depending on the prompt, in both engines. Whatever separates the two columns, it is not the drafting. The four prompts do not agree with each other at all. Gemma on the M5:

tokens/s	dasLLAMA off	1 draft	2 drafts	llama.cpp off	1 draft	2 drafts
writing	124.2	144.0 (69.3%)	138.3 (50.8%)	98.2	139.6 (81.4%)	130.8 (60.0%)
summarization	118.1	133.4 (69.3%)	124.6 (54.9%)	95.8	119.8 (70.3%)	126.0 (56.8%)
math	123.9	160.1 (89.6%)	187.9 (88.0%)	98.4	145.8 (89.6%)	163.4 (84.0%)
qa	124.8	148.6 (74.0%)	155.5 (63.4%)	98.2	129.5 (76.4%)	147.9 (70.5%)

Qwen on the M4, the machine that started the second day:

tokens/s	dasLLAMA off	1 draft	2 drafts	llama.cpp off	1 draft	2 drafts
writing	12.8	13.5 (70.7%)	10.9 (59.3%)	11.5	13.6 (75.0%)	12.3 (66.7%)
summarization	12.6	14.0 (80.3%)	12.0 (71.7%)	11.5	13.8 (81.4%)	12.4 (69.8%)
math	12.8	15.4 (93.9%)	13.6 (86.2%)	11.5	14.5 (93.8%)	13.9 (87.0%)
qa	12.8	13.2 (67.1%)	13.8 (88.0%)	11.5	13.5 (80.0%)	11.5 (62.5%)

The arithmetic of a round

Here is the part we most wanted to write down. A round with one draft token produces, on average, 1 + a tokens, where a is the acceptance rate. So its speedup over plain decoding is (1 + a) / c, with c the cost of the round in plain passes. Turn it around and the tables tell you what a round costs: c = (1 + a) / speedup.

round cost, in plain passes	dasLLAMA	llama.cpp
Gemma, M5	1.75 / 1.19 = 1.47	1.79 / 1.37 = 1.31
Gemma, M4	1.75 / 1.14 = 1.53	1.83 / 1.27 = 1.44
Qwen, M5	1.77 / 1.20 = 1.48	1.83 / 1.35 = 1.35
Qwen, M4	1.77 / 1.10 = 1.61	1.82 / 1.20 = 1.52

Timed on the M4, a round is the draft, the verify, the accept-and-commit and the state replay, and the verify is ninety-three percent of it. Checking two positions at once costs both engines about one and a half plain passes, and that, not the drafter, is what caps the gain at 1.1 to 1.2x on a dense 27B.

It is also why depth 2 falls behind on the M4 and only breaks even on the M5. A third row costs another half to a whole pass, and the second draft is accepted 51 to 66% of the time, which does not cover it. The math prompt is the exception: there the second draft lands 86 to 88%, and depth 2 comes out ahead on every machine, 187.9 against 160.1 for Gemma on the M5. Code sits in between: asked for a bubble sort in C++, Gemma on the M5 had 86% of first drafts and 68% of second drafts accepted, and depth 2 came out ahead again, 174.0 against 159.0 tokens per second, with llama.cpp at 153.7 against 140.0 on the same prompt.

Acceptance is a property of the text. The cost of a row is a property of the box. Reconciling the two is a tricky business left for future us.

The same arithmetic is why we never built the other idea, drafting wide: offering the two or three most likely tokens at one position instead of a chain. Every candidate is another row to verify, at half a pass to a whole pass each, and a runner-up only earns its row when the first candidate was wrong - one time in four on these prompts. It lost on paper before it could lose on the box.

Where the two days went

Back to that morning. The M4 read 12.7 tokens per second plain and 11.0 with the drafter on. llama.cpp on the same machine read 11.5 and 13.6. Without that second pair, a verify that cost two full passes for two rows would have looked like the price of a smaller GPU. With it, it looked like what it was. It is a lot easier when you have an oracle; otherwise how do you know you are slow?

The kernel that verifies two rows in one pass over the weights turned out to be a good fit for the M5 and a bad one for the M4, because the two chips have very different amounts of arithmetic per byte of memory bandwidth. So a new M4-specific kernel was written, the tuner picks it where it is the faster one, and the M4 row went to 13.9.

Then the depth-2 row, which was dropping like crazy, 7.9 tokens per second against 12.7 plain. Same story, one row up: three rows took a kernel nobody had ever tried on this chip. We already had all the pieces - two rows on the new kernel, one on the old single pass - and 1 + 2 turned out to be better than 3. Depth 2 went from 7.9 to 12.4, depth 1 to 14.0, by the evening of the 3rd. About a hundred lines and a test.

What a draft is worth

Exactly what the verify costs. The drafter's quality, the depth, the box, the quantization - each of them moves the number a few percent around that one fact. The models ship with the drafter built in, so the question was never whether to use it. The question was how cheap we could make the check, and the answer so far is one and a half passes for two tokens, on both engines, on both chips.