Use a four-term log1p series for exp(x) below 1/32 in all four DeepSeek router transform sites. Clamp the polynomial input so its unselected branch stays finite, and compare Metal probabilities against the shared accurate reference without widening the tolerance.
Refs antirez/ds4#1039. Validated on M5 Max with a fresh default build, test_deepseek41_metal, test_deepseek41_graph --vision-routing, and full Metal test runs with DS4_METAL_DISABLE_PRE_M5_ROUTER_TRANSFORM_FINALIZE_FUSION=1 and DS4_METAL_MATH_SAFE=1. No model required. Pre-M5 validation and model-level speed/SSD checks remain outstanding.
(cherry picked from commit 7f5f1b77ae)
The speculative batch ran the GDN convolution, scan and the attention of
each session through the single-session kernels, one to two rows at a
time: sixteen sessions, forty-eight layers, ~2300 short dispatches a
cycle. The GDN rows kernels now come in a variant addressed through a
table (state, history and the verify snapshots of each session by GPU
address, its first row and its row count), which walks a session's one or
two tokens in order and writes the snapshot after the first, with the
per-token arithmetic of the verify scan; the attention rows kernels take
one entry per token, so a draft row is simply a second entry at pos + 1
that sees the first token's keys. The trunk of a cycle with no drafts now
costs what a plain step does (89 ms against 92).
With drafts the cycle is 158 ms for 1.50 tokens per stream (152 tok/s at
C=16 on the prose corpus): every draft row costs its own MoE pass, so the
marginal row (~4.4 ms) is close to a plain token (5.6 ms) and greedy
speculation cannot pay at this width on this model unless acceptance is
high. Lossless against plain greedy decode (harness --spec --verify).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The grouped down pass ran its two rows one after the other, each a
dependent chain of block products. Both rows now walk their blocks in
the same loop, two independent chains per pair in flight; each chain's
own order is the per-token kernel's and the test pins it byte for bit.
+2.3% on the C=16 step (1.16x over the per-token kernels).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Same shape as the mid pass: a template on the pair count so the block
loop carries no per-pair branch, one named accumulator per pair, the
per-token kernel's pinned chain untouched and pinned byte for bit by the
test. +1.4% on the C=16 step (1.14x over the per-token kernels).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The grouped mid kernel tested the pair count inside its innermost loop,
three uniform branches per weight, and read each block's scales and
nibbles a byte at a time. A pass is now a template on the pair count
(one to four), so the inner loops carry no branch, and the block's words
come in as three loads (the byte values are the same); the per-token
arithmetic and its order are untouched, the test still pins it byte for
bit against the per-token kernel. +7% on the C=16 step of an M5 Max
(1.12x over the per-token kernels, from 1.047x).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqNkMw6d
At sixteen rows the per-token Q8 matvec read every matrix once per four
rows and streamed the batch's activations once per four weight rows: a
third of the step for 3.4 GB of weights. The batch GEMM gives each
simdgroup 32 weight rows and every token: each lane stages one q8_0 block
of its row as floats (the matvec's d * q values) into the simdgroup's own
threadgroup memory, so simdgroups never wait on each other, and the rows
multiply on fp32 simdgroup matrices against token tiles read straight from
the activations. Narrow projections split k across the grid and sum their
planes with the tile reduce. The sums stay within 2e-6 of the matvec's;
tests pin the kernel against a double reference and the matvec.
+10% on the C=16 step of an M5 Max (paired A/B). Every simdgroup matrix
index is a compile-time constant: with the token-tile count a runtime
value the same kernel ran twice slower. DS4_QWEN4_NO_BATCH_MM keeps the
matvec for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Sixteen rows pick ten experts each but only about a third as many
distinct ones, and the per-row kernels read an expert once per row that
chose it. The grouped kernels give each expert to the threadgroup of the
first (row, slot) pair in its list, which walks the expert's rows once per
four pairs with a named accumulator per pair, so every row's block walk and
chain order are the per-row kernel's and its outputs match bit for bit
(tests/test_qwen4_kernels.c pins mid and down under heavy reuse). The
shared expert, read once per row as an extra slot, becomes dense
projections over the batch as in prefill.
Together +7% on the C=16 step of an M5 Max (paired A/B); single rows and
verify rows keep the per-row kernels. DS4_QWEN4_MOE_NO_GROUP and
DS4_QWEN4_MOE_SHARED_DENSE_MIN keep the old paths for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The decode batch ran attention as sixteen chains of per-row dispatches:
prep, block key, score, select, expand, decode and merge for each session
in turn. The rows kernels take the row as a grid axis and reach each
session's caches through a staged table of GPU addresses, so a layer costs
one dispatch per stage. Every kernel keeps the single-row arithmetic, the
row's own split count included, and the single-row kernels call the same
bodies; tests/test_qwen4_kernels.c pins the rows path byte-exact against
the per-row dispatches for dense, block-completing and sparse rows on both
selector widths.
On an M5 Max at C=16, ctx 4096 the step is unchanged (1.00x): those grids
were not latency-bound after all, the attention stages take ~6.5 ms of a
110 ms step. DS4_QWEN4_NO_BATCH_ATTN keeps the per-row path for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Qwen3.8 sessions were excluded from native session batching: each decode
step ran the whole graph once per session, so sixteen sessions cost
sixteen times the weight traffic. The batched encoder now runs the dense
work (hyper-connection mixers, projections, MoE, output head) once over
the stacked rows and keeps only the recurrent state, the caches and the
PLE history per session, with batched GDN scan and conv kernels reading
the pooled state through per-row slots.
Session transients (prefill scratch) move to one shared arena sized for
the largest session, so each further session costs its caches only
(memory per session divided by ~3.6 at ctx 4096). The dense GEMM gains a
split-K path for the narrow-batch regime, and the dense/MoE/mixer GEMM
thresholds, tuned for T=1 or T>=8192, are moved to serve decode batches
of 4..64 rows. Every step keeps the tokens of the sequential path;
float reduction order changes where the batched kernels do.
C=16 at ctx 4096 on an M5 Max goes from ~50 to ~143 tok/s aggregate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
Keep the existing MV, SIMDgroup and TensorOps arithmetic while reading selected experts from the bounded cache. Enable this by default for single-host eight-expert IQ2 batches below 256 tokens when a full layer fits; retain the existing larger-prefill and fallback paths.
Test exact intermediate and final outputs at both synthetic and full GLM dimensions, cache eviction, dispatch boundaries and undersized budgets. Full-model logits, coding sessions and MTP snapshot checks pass on M5 Max.
Use the model-defined 2051-token dense boundary, handle padded selections, and synchronize reused reduction storage in Metal, CUDA and ROCm, including vision normalization.
Enable bounded ROCm selected-attention GEMM for GLM 5.3 without RoPE. Add independent attention references, boundary tests and finite, repeatable vision checks. Focused Strix continued prefill improves from about 21 to 68 tokens per second; stored official scores are unchanged.
Use the M5 seed schedule under TP and evaluate verifier rows together instead of running each row separately. Reuse the existing static MXFP4 kernels with row-aware addressing, and enable packed MXFP4 activations for two-rank M5 prefill.
Keep streaming, exact-sampling scheduling and unmeasured device defaults unchanged. Add synthetic ownership/static-shape checks and a physical TP speculative oracle. Record focused quality, speed comparisons and transport caveats in the QA guide.
ds4_gpu_compressor_store_batch_tensor and the score-with-APE path used to
encode one f16->f32 copy per ratio-row segment (n_tokens/4 single-threadgroup
encoders per layer: 21.5k of the 25k encoders of a 2048-token prefill). A
tiled-row kernel (kernel_cpy_tile_rows_f16_f32 / _f32_f32) writes the whole
batch in one dispatch with identical values. README: updated two-Mac numbers
and the new opt-out env vars.
Two-Mac TP prefill, byte-exact vs the TensorOps-MoE hashes: 2048 tokens
785 -> 830 t/s, 128 tokens 242 -> 253 t/s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The routed-expert prefill GEMMs for MXFP4 experts now use the existing
kernel_mul_mm_id_mpp TensorOps kernel (new instantiations
kernel_mul_mm_id_mxfp4_f32_mpp / _f16_mpp), as IQ2_XXS and Q2_K experts
already did. Same half weights x half activations with f32 accumulation
as the simdgroup kernel; the accumulation order differs, so frontier
logits are no longer byte-identical to the previous kernel (2048-token
prefill on the two-Mac TP pair: mean |dlogit| 0.17, max 1.48, same
argmax, 9/10 top-10 overlap, KL 1e-4). DS4_METAL_DISABLE_MXFP4_MM_ID_MPP=1
restores the simdgroup kernels.
Two-Mac TP prefill: 2048 tokens 673 -> 785 t/s, 128 tokens 164 -> 242 t/s.
Decode unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The 8-group attention split adopted after the sweep saved 1 us in the
attention kernels but made the layer's later concurrent FFN block ~50 us
slower (114 -> 162 us; cause not understood, measured by toggling
DS4_METAL_FLASH_NWG on one build), costing ~4 t/s. Back to 32 groups;
the env override stays for experiments. With it the default output is
byte-identical to the pre-session reference again, which also validates
the kv-task fold and the two-rank router-select fusion against it.
The two-rank router-select branch no longer sets router_only_done (that
enabled the single-machine parallel FFN path and computed the whole
shared expert on the coordinator); it only skips the plain router matvec
and the standalone select.
Rank-speed lane bias, opt-in (DS4_TP_LANE_BIAS=1): the gate service
probes the peer's RDMA arrival while it spins on its own flag, giving a
signed per-gate skew (peer early > 0) without any protocol change; the
coordinator turns the FFN-gate skew EWMA into a lane bias (default 19
lanes per us of skew, smoothed, +-shared_dim/2, 64-lane steps), carries
it in the EVAL command's reserved word, and both ranks add it to the
shared-expert split. The logits-half frame carries one extra float (the
sender's FFN-gate wait) for future use. Measured on this pair: skew 2-5
us, 64-128 lanes moved, no token-rate change (48.0 vs 48.2, 47.0 vs
46.4), so it stays off by default because it makes the output depend on
timing. DS4_TP_LANE_BIAS_GAIN and DS4_TP_LANE_BIAS_DEBUG for experiments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The fused q/kv norm + KV RoPE + FP8 store dispatch ran its two tasks as
two threadgroups and cost 13 us per layer, most of it the serial kv task.
The kernel now takes a task mask; on the two-rank decode path the q task
runs alone (3.8 us) and the kv task is deferred and encoded in a
concurrent encoder together with the q_b projection, whose 17.8 MB stream
hides it completely (the section measures 35.8 us, q_b alone 36.6). The
KV staging and the raw-only attention path flush a still-pending task
before reading the cache, so every path sees the row. Same arithmetic,
byte-identical output. DS4_METAL_DISABLE_KV_NORM_DEFER=1 opts out.
The existing fused router projection + exact select kernel (128
threadgroups, last-arriving threadgroup selects) was gated to
single-machine runs; enable it for two ranks, where it replaces the f16
router matvec (7.6 us) + select (6.0) with one 11.1 us dispatch and the
shared expert keeps running in the concurrent FFN encoder. Byte-identical.
DS4_METAL_DISABLE_M5_ROUTER_PROJECT_SELECT_FUSE=1 opts out.
Validated on the M5 Max pair: identical text with every toggle at ctx 128
(256 tokens) and ctx 2050 (partial pad block), prefill logits unchanged;
attention-side command buffer 226.1 -> 214.7 us of GPU compute per layer.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The vec attention kernel's split-K work group count is a pipeline
constant; make it adaptive (8 groups per head for <= 256 keys, else 32;
DS4_METAL_FLASH_NWG overrides). Measured at 128 keys: 8 groups -1 us
net, 4 groups +9 us.
Add kernel_dsv4_hc_expand4_mix_norm_slices_f16: the HC expand + gate
matvec + collapse + input norm as 64 slice threadgroups with fixed-order
partial sums and a last-arriver tail, no spin barrier. Measured 16.7 us
against the six-threadgroup cluster kernel's 14.5 (the cluster kernel is
launch + serial tail bound, not barrier bound), so it stays behind
DS4_METAL_HC_EXPAND_SLICES=1 and the cluster kernel remains the default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Factor the one-dispatch touch-table encode out of the gate prefetch
(ds4_gpu_encode_touch_table). Add kernel_mul_mv_q8_0_f32_r4 and its
flag-folded twin (four rows per threadgroup, bit-exact with the two-row
kernels) behind DS4_METAL_Q8_MV_NR0=4 for the pass-through dispatch
sites; measured slower on the M5 Max pair (48.1 vs 49.3, 45.1 vs 48.4
t/s), so the default stays at two rows. DS4_TP_FOLD_DEBUG=1 prints the
first flag-fold request/take decisions.
An attention-window prefetch (touching this rank's out_a half in a
concurrent encoder with the flash-attention vec kernel) was measured and
not kept: the projection got 10 us faster but the latency-bound vec
kernel slowed by the touch's duration under saturated DRAM, token rate
unchanged. Notes in misc/MAC_TP_SPEED.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The checked poll-gate flag was its own kernel after each partial (4-7 us
per gate). The payload producers now publish it themselves from the
last-arriving threadgroup: kernel_dsv4_add2_f32_tp_flag_checked for the
FFN gate (local shared+routed sum into the slab slot) and
kernel_dsv4_mul_mv_q8_0_f32_tp_flag_checked for the attention gate (the
attention-output K-slice matvec). Each threadgroup adds the integer sum
of the words it stored to a per-slot accumulator, fences, then arrives;
the last arriver exchanges the accumulator, resets both counters, and
stores the checksum and the flag. Integer sums are order independent,
so the checksum equals the standalone kernel's over the same payload and
the stored values are the plain kernels' values: text byte-identical.
ds4_gpu_tp_gate_encode skips its flag kernel when the producer
prepublished the sequence number; ds4_gpu_add_tensor_tp_flag and
ds4_gpu_tp_flag_fold_request (taken by the K-slice matvec when its
output is the slot) are the host entry points. DS4_TP_DISABLE_FLAG_FOLD=1
opts out. Timeline: fused add+flag 3.2 us vs add 2.1 + flag 4.0.
Gate-time prefetch was one dispatch per plan range (up to 7) at ~3 us
fixed cost each, 16-22 us per gate and exposed whenever the peer's
partial was already there. It is now one dispatch over a table of up to
8 bindless views (gpuAddress + useResources), threads assigned to ranges
by cumulative line counts (kernel_touch_u8_stride_table). The
attention-gate plan lists the other shared half last for the dynamic
shared split. DS4_TP_PREFETCH_MARGIN_US / DS4_TP_PREFETCH_BYTES_PER_US
override the sizing constants for experiments.
Measured on two M5 Max over TB5 RDMA (README prompt, 256 tokens,
interleaved pairs): 47.35 vs 46.78 and 46.70 vs 45.86 t/s with the folds
off; eight repeated default runs identical in text and frontier logits.
Encoder timeline: attention-side command buffer 233.5 -> 226.1 us,
FFN-side 173.1 -> 168.6 us of GPU compute.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Routed experts are owned by expert id, so each rank streams 0..6 of the
6 selected experts per token while the shared expert was split in fixed
halves; the rank owning more selected experts was the critical path of
the layer and the other rank idled at the gate. The new split kernels
(kernel_dsv4_shared_gate_up_swiglu_q8_0_split and
kernel_dsv4_shared_down_q8_0_split) read the selected ids and take a
complementary lane range per rank sized to balance bytes: lanes0 =
shared_dim/2 + (n1 - n0) * shift * shared_dim with shift = routed expert
bytes / (2 * shared expert bytes), rounded to 64 lanes. On the V4 Flash
MXFP4/Q8 file one extra routed expert moves half of the shared expert to
the other rank. A rank with zero lanes writes a zero partial, so the
local routed+shared add is unchanged. The kernels keep the args in
constant memory with explicit row/K counts (a by-value args copy cost
+9.8 us per layer in the matvec bodies); with shift 0 they are bit-exact
with the static halves (+1.9 us per layer). The attention-gate prefetch
plan lists the other shared half last so a rank with a long measured
wait streams it in.
Default for two-rank Metal TP decode. DS4_TP_STATIC_SHARED_SPLIT=1
restores the fixed halves; DS4_TP_SHARED_SPLIT_SHIFT_Q16 overrides the
shift (0 = static halves through the split kernels, for exactness
checks). Measured on two M5 Max over TB5 RDMA (README prompt, 256
tokens, interleaved pairs): ctx 128 46.6-47.4 vs 45.2-46.7 t/s, ctx 2048
45.6-46.2 vs 45.1 t/s; both ranks wait less at FFN gates. Output is
deterministic and differs from the static split only by floating-point
grouping of the partial sums.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Two-Mac DeepSeek V4 Flash tensor-parallel decode on M5 Max pairs.
- Poll gates (default; DS4_TP_DISABLE_POLL_GATES opts out): each flag gate
ends the command buffer and the next queued buffer opens with a bounded
fresh-line poll released by the service thread, instead of parking on a
shared event whose CPU->GPU wake-up costs 27-43 us. Ring of 16 regions,
~600 ms budget, timeout reported at the next gate.
- Checksum-verified flags: kernel_dsv4_tp_flag_set_checked publishes an
exact integer checksum of the 16 KiB partial; the service thread verifies
the payload in memory before the RDMA post (the flag line can land ~35 us
before the buffer completes; unverified sends were nondeterministic).
- HC post/expand folded into the compound HC producer with a device-scope
barrier across its six threadgroups; the engine defers the post-gate
expand and flushes it before any other reader
(DS4_METAL_DISABLE_HC_EXPAND_PRODUCER_FUSE opts out).
- Gate-time prefetch: the next phase's weights (rank slices, priority
order, cut at this rank's measured peer-wait window) are touched into the
cache before the poll (DS4_TP_DISABLE_GATE_PREFETCH opts out).
- Concurrent routed/shared FFN branches, safe decode command-buffer split,
exact-shape TP MXFP4 static kernels, DSpark direct partial commit, and
split gate profiling from the morning session.
- Diagnostics: DS4_METAL_ENCODER_TIMELINE per-pass GPU timeline,
DS4_METAL_CB_TIMES per-command-buffer driver/GPU timing.
README ctx 128, 256 greedy tokens: 45.5 vs 43.3 t/s (event gates, same
build); frontier logits and generated text byte-identical throughout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
Tarjei Mandt
Emilian Bold
davidandClaude Fable 5.1
Salvatore Sanfilippo