Sixteen speculative cycles before the first plain one made a short
generation on prose pay the losing kind for most of its length (C=16 at
32 steps: 174 tok/s against ~195 once the policy has settled). Eight
cycles give p over 128 draft rows at sixteen streams, enough to price.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
A draft row's cost is not per row: the seventeenth row of a batched cycle
opens a second tile of every dense projection, after which rows are
nearly free (16 rows 80 ms, 23 rows 126, 32 rows 149), so gating streams
by their draft's confidence bought nothing (prose 184.2 vs 183.9 at a
margin of 2, code 210.7 vs 213.6). A batch speculates for everyone or
for nobody, and it pays when (1 + p) c_plain > c_spec.
The engine keeps p and both cycle times as running averages (the same
shape as the session's depth policy), starts by pricing both kinds and
then probes the other kind for two cycles, every thirty-two while plain
and every hundred and twenty-eight while speculating. DS4_QWEN4_SPEC_TRACE
prints the state per cycle.
C=16 prose 184 -> 195 tok/s (plain ~200, mostly plain cycles now), code
213.6 -> 214.3; C=8 prose 131 plain -> 158, code 131 -> 179 (the drafts
fill the one tile a C=8 batch has). C=2 verifies lossless.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The batched cycle drafted with one predictor pass per session, sixteen
passes of the nextn layer over one or two rows each, every one reading
the layer's weights again. One pass now covers every row of the batch:
the trunk streams sit in the batch's R at the positions the trunk rows
had, so the layer's attention takes the same row table as the trunk's,
the staged inputs and residual go where the prompt priming puts them,
and the sessions' last rows are gathered for one pass of the head mixer
before the batched head and the per-session argmax. A rejected draft's
row runs along with its parent as the token that follows; the next pass
rewrites its cache entry before anything reads it.
C=16 code 204 -> 213.5 tok/s cumulative (cycle 156 -> 148 ms), prose
176 -> 183; C=2 verifies lossless against plain greedy.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The speculative batch ran the GDN convolution, scan and the attention of
each session through the single-session kernels, one to two rows at a
time: sixteen sessions, forty-eight layers, ~2300 short dispatches a
cycle. The GDN rows kernels now come in a variant addressed through a
table (state, history and the verify snapshots of each session by GPU
address, its first row and its row count), which walks a session's one or
two tokens in order and writes the snapshot after the first, with the
per-token arithmetic of the verify scan; the attention rows kernels take
one entry per token, so a draft row is simply a second entry at pos + 1
that sees the first token's keys. The trunk of a cycle with no drafts now
costs what a plain step does (89 ms against 92).
With drafts the cycle is 158 ms for 1.50 tokens per stream (152 tok/s at
C=16 on the prose corpus): every draft row costs its own MoE pass, so the
marginal row (~4.4 ms) is close to a plain token (5.6 ms) and greedy
speculation cannot pay at this width on this model unless acceptance is
high. Lossless against plain greedy decode (harness --spec --verify).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
ds4_sessions_eval_batch_speculative_argmax runs one MTP cycle for a batch:
every session contributes its next token and, when the predictor left it a
draft for that token, the draft as a second row. The dense work of the
trunk runs once over all the rows; the recurrent, n-gram and attention
steps run per session over its one or two rows with the single-session
verify kernels, snapshots included, so a session's own arithmetic is that
of the C=1 speculative cycle. A draft stands when it is the target's
argmax for its row; a rejected one restores the row-0 snapshot. The
predictor then runs its layer per session over the committed rows and
every session's head as one product. The Q8 batch GEMM serves 9..32 rows
in sixteen-row halves.
The harness gains --spec (with --verify checking the committed tokens
against plain greedy decode, which speculation must reproduce). On an M5
Max at C=16 the first cut commits 1.49 tokens per stream and cycle on the
prose corpus but the cycle costs 171 ms against 92 for a plain step: the
per-session kernels and the sequential predictor passes are the next
thing to batch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The fp32 tile for F16 batches was meant for decode batches, where the
generic tile's half-staged activations carried rounding into the logits;
its rule also caught every prefill chunk's narrow hyper-connection
projections. There it is slower (1305 vs 1410 tok/s at 4k) and it moved
a single stream's tokens off the reference's: with the generic tile back
for chunks past 64 rows, a C=1 stream generates the same 32 tokens as
upstream main, and prefill returns to 1408 tok/s at 4k, 1455 at 16k.
Decode batches keep the fp32 tile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The k-split of the dense GEMM counted only the row tiles, so a prefill
chunk's narrow projections split k thirteen ways although their 256
token tiles already filled the machine, and their partial planes grew
with the chunk: past the first context size of a run they no longer fit
the scratch allocated once, the GEMM failed mid-batch and every call fell
back to the per-token kernel, costing 8% of prefill at 8k and 16k. The
rule now counts both tile dimensions: a decode batch splits as before, a
prefill chunk does not.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The shared transients arena took the shape of the first session, and a
later session needing a wider one (a larger context, hence more indexer
blocks) silently fell back to private transients, ~4-5 GiB each: sixteen
of them at the next context size of a sweep exceeded memory and the
process died without a word. The engine now counts the sessions
borrowing the arena and, when a wider session arrives while none does,
frees the arena so that session builds the wider one; while sessions
borrow it, a wider one keeps private transients as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The plain batched decode sized its logits scratch to the batch width, so
the Q8 tile's 32-row padding did not fit and the output head, 248320 rows
read once per four tokens by the per-token matvec, cost 3.9 ms of a
76 ms step. The scratch now holds the tile's rows, which the speculative
path already did, and the head takes the tile like every other Q8
projection.
C=16 plain prose 196.9 -> 203 tok/s cumulative; batched vs sequential
unchanged (same worst logit deviation, tokens agree).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The batch tile took every F16 projection past one row, which also caught
the two- and three-row speculative verify of a single session: its
hyper-connection down products cost three times what the few-row matvec
charged them, and that was their original path. The tile now starts at
four rows. C=1 with MTP on the prose corpus: 50.5 -> 55.6 tok/s (plain
51.3); C=16 is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The grouped down pass ran its two rows one after the other, each a
dependent chain of block products. Both rows now walk their blocks in
the same loop, two independent chains per pair in flight; each chain's
own order is the per-token kernel's and the test pins it byte for bit.
+2.3% on the C=16 step (1.16x over the per-token kernels).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Same shape as the mid pass: a template on the pair count so the block
loop carries no per-pair branch, one named accumulator per pair, the
per-token kernel's pinned chain untouched and pinned byte for bit by the
test. +1.4% on the C=16 step (1.14x over the per-token kernels).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The grouped mid kernel tested the pair count inside its innermost loop,
three uniform branches per weight, and read each block's scales and
nibbles a byte at a time. A pass is now a template on the pair count
(one to four), so the inner loops carry no branch, and the block's words
come in as three loads (the byte values are the same); the per-token
arithmetic and its order are untouched, the test still pins it byte for
bit against the per-token kernel. +7% on the C=16 step of an M5 Max
(1.12x over the per-token kernels, from 1.047x).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqNkMw6d
The scratch for the k-split planes grew on demand, and the command buffer
holds unretained references: a plane buffer replaced while earlier
dispatches of the same batch still pointed at it was read after its
release. That is what made splits of sixteen and more look wrong (a
logit difference of 1.3 that never moved). The scratch is now allocated
once, 64 MiB or the first request if larger, and a request it cannot serve
fails instead of reallocating. Splits of 32 and 64 now measure exact; they
are no faster, so the default width stays.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
At sixteen rows the per-token Q8 matvec read every matrix once per four
rows and streamed the batch's activations once per four weight rows: a
third of the step for 3.4 GB of weights. The batch GEMM gives each
simdgroup 32 weight rows and every token: each lane stages one q8_0 block
of its row as floats (the matvec's d * q values) into the simdgroup's own
threadgroup memory, so simdgroups never wait on each other, and the rows
multiply on fp32 simdgroup matrices against token tiles read straight from
the activations. Narrow projections split k across the grid and sum their
planes with the tile reduce. The sums stay within 2e-6 of the matvec's;
tests pin the kernel against a double reference and the matvec.
+10% on the C=16 step of an M5 Max (paired A/B). Every simdgroup matrix
index is a compile-time constant: with the token-tile count a runtime
value the same kernel ran twice slower. DS4_QWEN4_NO_BATCH_MM keeps the
matvec for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Sixteen rows pick ten experts each but only about a third as many
distinct ones, and the per-row kernels read an expert once per row that
chose it. The grouped kernels give each expert to the threadgroup of the
first (row, slot) pair in its list, which walks the expert's rows once per
four pairs with a named accumulator per pair, so every row's block walk and
chain order are the per-row kernel's and its outputs match bit for bit
(tests/test_qwen4_kernels.c pins mid and down under heavy reuse). The
shared expert, read once per row as an extra slot, becomes dense
projections over the batch as in prefill.
Together +7% on the C=16 step of an M5 Max (paired A/B); single rows and
verify rows keep the per-row kernels. DS4_QWEN4_MOE_NO_GROUP and
DS4_QWEN4_MOE_SHARED_DENSE_MIN keep the old paths for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Tarjei Mandt
Salvatore Sanfilippo
davidandClaude Fable 5.1