The speculative batch ran the GDN convolution, scan and the attention of
each session through the single-session kernels, one to two rows at a
time: sixteen sessions, forty-eight layers, ~2300 short dispatches a
cycle. The GDN rows kernels now come in a variant addressed through a
table (state, history and the verify snapshots of each session by GPU
address, its first row and its row count), which walks a session's one or
two tokens in order and writes the snapshot after the first, with the
per-token arithmetic of the verify scan; the attention rows kernels take
one entry per token, so a draft row is simply a second entry at pos + 1
that sees the first token's keys. The trunk of a cycle with no drafts now
costs what a plain step does (89 ms against 92).
With drafts the cycle is 158 ms for 1.50 tokens per stream (152 tok/s at
C=16 on the prose corpus): every draft row costs its own MoE pass, so the
marginal row (~4.4 ms) is close to a plain token (5.6 ms) and greedy
speculation cannot pay at this width on this model unless acceptance is
high. Lossless against plain greedy decode (harness --spec --verify).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
At sixteen rows the per-token Q8 matvec read every matrix once per four
rows and streamed the batch's activations once per four weight rows: a
third of the step for 3.4 GB of weights. The batch GEMM gives each
simdgroup 32 weight rows and every token: each lane stages one q8_0 block
of its row as floats (the matvec's d * q values) into the simdgroup's own
threadgroup memory, so simdgroups never wait on each other, and the rows
multiply on fp32 simdgroup matrices against token tiles read straight from
the activations. Narrow projections split k across the grid and sum their
planes with the tile reduce. The sums stay within 2e-6 of the matvec's;
tests pin the kernel against a double reference and the matvec.
+10% on the C=16 step of an M5 Max (paired A/B). Every simdgroup matrix
index is a compile-time constant: with the token-tile count a runtime
value the same kernel ran twice slower. DS4_QWEN4_NO_BATCH_MM keeps the
matvec for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Sixteen rows pick ten experts each but only about a third as many
distinct ones, and the per-row kernels read an expert once per row that
chose it. The grouped kernels give each expert to the threadgroup of the
first (row, slot) pair in its list, which walks the expert's rows once per
four pairs with a named accumulator per pair, so every row's block walk and
chain order are the per-row kernel's and its outputs match bit for bit
(tests/test_qwen4_kernels.c pins mid and down under heavy reuse). The
shared expert, read once per row as an extra slot, becomes dense
projections over the batch as in prefill.
Together +7% on the C=16 step of an M5 Max (paired A/B); single rows and
verify rows keep the per-row kernels. DS4_QWEN4_MOE_NO_GROUP and
DS4_QWEN4_MOE_SHARED_DENSE_MIN keep the old paths for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The decode batch ran attention as sixteen chains of per-row dispatches:
prep, block key, score, select, expand, decode and merge for each session
in turn. The rows kernels take the row as a grid axis and reach each
session's caches through a staged table of GPU addresses, so a layer costs
one dispatch per stage. Every kernel keeps the single-row arithmetic, the
row's own split count included, and the single-row kernels call the same
bodies; tests/test_qwen4_kernels.c pins the rows path byte-exact against
the per-row dispatches for dense, block-completing and sparse rows on both
selector widths.
On an M5 Max at C=16, ctx 4096 the step is unchanged (1.00x): those grids
were not latency-bound after all, the attention stages take ~6.5 ms of a
110 ms step. DS4_QWEN4_NO_BATCH_ATTN keeps the per-row path for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Qwen3.8 sessions were excluded from native session batching: each decode
step ran the whole graph once per session, so sixteen sessions cost
sixteen times the weight traffic. The batched encoder now runs the dense
work (hyper-connection mixers, projections, MoE, output head) once over
the stacked rows and keeps only the recurrent state, the caches and the
PLE history per session, with batched GDN scan and conv kernels reading
the pooled state through per-row slots.
Session transients (prefill scratch) move to one shared arena sized for
the largest session, so each further session costs its caches only
(memory per session divided by ~3.6 at ctx 4096). The dense GEMM gains a
split-K path for the narrow-batch regime, and the dense/MoE/mixer GEMM
thresholds, tuned for T=1 or T>=8192, are moved to serve decode batches
of 4..64 rows. Every step keeps the tokens of the sequential path;
float reduction order changes where the batched kernels do.
C=16 at ctx 4096 on an M5 Max goes from ~50 to ~143 tok/s aggregate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Share the Metal session graph with CUDA for text and vision, MTP and exact sampling, steering, checkpoints, and disk-only BF16 n-grams. Add bounded concurrent Linux n-gram reads, vectorized dense and expert kernels, register-held recurrent prefill state, and efficient small speculative batches.
Keep Qwen out of DeepSeek batch dispatch and exclude disk n-grams from resident memory accounting. Use normal file-backed weight loading in diagnostics and link n-gram support into the CUDA quality scorer.
Include independent kernel and memory-safety tests, mixed-prefill and recovery checks, long-context server and agent coverage, and user documentation with measured DGX Spark performance.
Read one next layer into reserved slots of the admitted expert cache using two 8 MiB staging buffers. Protect current inputs, publish only completed copies, keep unused read-ahead cold, and join before cancellation or cache/model teardown.
Enable read-ahead from 2K tokens; smaller prefills were slower with full-layer reads. Add eviction, truncated-source, buffered-I/O, cancellation and early-exit tests. Validate exact logits and saved state on the Sparks, including Compute Sanitizer.
Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
Use the existing seed verifier for resident IQ2/Q2 Spark sessions. Avoid duplicate declined proposals, check short drafts after normal seed decode, and exclude the seed from scheduler acceptance. Pause longer for unproductive prose, but retry sooner after useful long drafts. Keep exact sampling, streaming and other hardware defaults unchanged.
- ds4_gpu_tp_big_gate_kick_flush: a kick that ends the command buffer so
the shared-event signal is published while the caller keeps encoding.
The service thread then waits for that buffer's completion before it
reads the payload: the event fires when the GPU is done, but the stores
only land for the CPU at buffer completion (measured: the first ~10 MB
of the second half's rows were stale without the wait). The attention
sub-gate pipeline (DS4_TP_SUBGATE_PIPELINE=1) now uses it too.
- FFN sub-gate (DS4_TP_FFN_SUBGATE=1, off by default): the routed partial
is produced in two row halves with this rank's shared-expert rows folded
per half before each kick, so the exchanged rows are what the peer adds.
Byte-identical to the single gate but net-negative on the M5 Max pair
(781 vs ~850 t/s at 2048 tokens: two 1024-row routed calls cost ~27%
more kernel time than one 2048-row call).
- Diagnostics: DS4_TP_BIG_GATE_DEBUG=2 logs every big gate (period since
the previous release, handshake, transfer); DS4_TP_FAKE_BIG_GATE=1 skips
the exchange for a compute-only timing bound (results are garbage);
DS4_TP_KICK_SETTLE_US adds a delay after the kick-buffer wait;
tp_ffn_recv / tp_ffn_sum tensor dumps after the FFN all-reduce.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
- ds4_tp.c: the prefill big-gate exchange now moves the partial through the
slab batch regions in windows of up to 256 16 KB messages, receives posted
in chains of 64 behind a one-byte control barrier, sends in chains of 64
with at most four signaled chains in flight (32 MB in 4-6 ms instead of
~26 ms). Queue pairs ask for 1024/1024 work requests (fallback 256/64)
and the device limits are printed at open. Payload-buffer registration
stays opt-in (DS4_TP_RDMA_REG_PAYLOAD): Apple's verbs cap a memory region
at 16 MB and registering Metal buffers gave LOC_PROT_ERR completions.
DS4_TP_BIG_GATE_DEBUG=1 prints handshake/transfer times every 32 gates.
- ds4_metal.m: a command buffer submitted after the engine's queue idled
for ~3 s starts 600-800 ms late on M5 Max (measured; other queues do not
prevent it). A main-queue keepalive submits one 1 KB kernel per second
(DS4_METAL_DISABLE_QUEUE_KEEPALIVE=1 opts out) and the sharded load pays
the first submission up front (ds4_gpu_warm_command_queue). The big-gate
arrival spin sleep-polls (DS4_TP_BIG_GATE_SPIN_US). Diagnostics: blit
passes in the encoder timeline, DS4_METAL_CB_TIMES prints encode/commit/
GPU-span per owned command buffer with a backtrace on stalls.
Two-Mac TP prefill, byte-exact frontier logits: 2048 tokens 565 -> 673 t/s,
128 tokens 93 -> 164 t/s; decode unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The 8-group attention split adopted after the sweep saved 1 us in the
attention kernels but made the layer's later concurrent FFN block ~50 us
slower (114 -> 162 us; cause not understood, measured by toggling
DS4_METAL_FLASH_NWG on one build), costing ~4 t/s. Back to 32 groups;
the env override stays for experiments. With it the default output is
byte-identical to the pre-session reference again, which also validates
the kv-task fold and the two-rank router-select fusion against it.
The two-rank router-select branch no longer sets router_only_done (that
enabled the single-machine parallel FFN path and computed the whole
shared expert on the coordinator); it only skips the plain router matvec
and the standalone select.
Rank-speed lane bias, opt-in (DS4_TP_LANE_BIAS=1): the gate service
probes the peer's RDMA arrival while it spins on its own flag, giving a
signed per-gate skew (peer early > 0) without any protocol change; the
coordinator turns the FFN-gate skew EWMA into a lane bias (default 19
lanes per us of skew, smoothed, +-shared_dim/2, 64-lane steps), carries
it in the EVAL command's reserved word, and both ranks add it to the
shared-expert split. The logits-half frame carries one extra float (the
sender's FFN-gate wait) for future use. Measured on this pair: skew 2-5
us, 64-128 lanes moved, no token-rate change (48.0 vs 48.2, 47.0 vs
46.4), so it stays off by default because it makes the output depend on
timing. DS4_TP_LANE_BIAS_GAIN and DS4_TP_LANE_BIAS_DEBUG for experiments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The fused q/kv norm + KV RoPE + FP8 store dispatch ran its two tasks as
two threadgroups and cost 13 us per layer, most of it the serial kv task.
The kernel now takes a task mask; on the two-rank decode path the q task
runs alone (3.8 us) and the kv task is deferred and encoded in a
concurrent encoder together with the q_b projection, whose 17.8 MB stream
hides it completely (the section measures 35.8 us, q_b alone 36.6). The
KV staging and the raw-only attention path flush a still-pending task
before reading the cache, so every path sees the row. Same arithmetic,
byte-identical output. DS4_METAL_DISABLE_KV_NORM_DEFER=1 opts out.
The existing fused router projection + exact select kernel (128
threadgroups, last-arriving threadgroup selects) was gated to
single-machine runs; enable it for two ranks, where it replaces the f16
router matvec (7.6 us) + select (6.0) with one 11.1 us dispatch and the
shared expert keeps running in the concurrent FFN encoder. Byte-identical.
DS4_METAL_DISABLE_M5_ROUTER_PROJECT_SELECT_FUSE=1 opts out.
Validated on the M5 Max pair: identical text with every toggle at ctx 128
(256 tokens) and ctx 2050 (partial pad block), prefill logits unchanged;
attention-side command buffer 226.1 -> 214.7 us of GPU compute per layer.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The checked poll-gate flag was its own kernel after each partial (4-7 us
per gate). The payload producers now publish it themselves from the
last-arriving threadgroup: kernel_dsv4_add2_f32_tp_flag_checked for the
FFN gate (local shared+routed sum into the slab slot) and
kernel_dsv4_mul_mv_q8_0_f32_tp_flag_checked for the attention gate (the
attention-output K-slice matvec). Each threadgroup adds the integer sum
of the words it stored to a per-slot accumulator, fences, then arrives;
the last arriver exchanges the accumulator, resets both counters, and
stores the checksum and the flag. Integer sums are order independent,
so the checksum equals the standalone kernel's over the same payload and
the stored values are the plain kernels' values: text byte-identical.
ds4_gpu_tp_gate_encode skips its flag kernel when the producer
prepublished the sequence number; ds4_gpu_add_tensor_tp_flag and
ds4_gpu_tp_flag_fold_request (taken by the K-slice matvec when its
output is the slot) are the host entry points. DS4_TP_DISABLE_FLAG_FOLD=1
opts out. Timeline: fused add+flag 3.2 us vs add 2.1 + flag 4.0.
Gate-time prefetch was one dispatch per plan range (up to 7) at ~3 us
fixed cost each, 16-22 us per gate and exposed whenever the peer's
partial was already there. It is now one dispatch over a table of up to
8 bindless views (gpuAddress + useResources), threads assigned to ranges
by cumulative line counts (kernel_touch_u8_stride_table). The
attention-gate plan lists the other shared half last for the dynamic
shared split. DS4_TP_PREFETCH_MARGIN_US / DS4_TP_PREFETCH_BYTES_PER_US
override the sizing constants for experiments.
Measured on two M5 Max over TB5 RDMA (README prompt, 256 tokens,
interleaved pairs): 47.35 vs 46.78 and 46.70 vs 45.86 t/s with the folds
off; eight repeated default runs identical in text and frontier logits.
Encoder timeline: attention-side command buffer 233.5 -> 226.1 us,
FFN-side 173.1 -> 168.6 us of GPU compute.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Routed experts are owned by expert id, so each rank streams 0..6 of the
6 selected experts per token while the shared expert was split in fixed
halves; the rank owning more selected experts was the critical path of
the layer and the other rank idled at the gate. The new split kernels
(kernel_dsv4_shared_gate_up_swiglu_q8_0_split and
kernel_dsv4_shared_down_q8_0_split) read the selected ids and take a
complementary lane range per rank sized to balance bytes: lanes0 =
shared_dim/2 + (n1 - n0) * shift * shared_dim with shift = routed expert
bytes / (2 * shared expert bytes), rounded to 64 lanes. On the V4 Flash
MXFP4/Q8 file one extra routed expert moves half of the shared expert to
the other rank. A rank with zero lanes writes a zero partial, so the
local routed+shared add is unchanged. The kernels keep the args in
constant memory with explicit row/K counts (a by-value args copy cost
+9.8 us per layer in the matvec bodies); with shift 0 they are bit-exact
with the static halves (+1.9 us per layer). The attention-gate prefetch
plan lists the other shared half last so a rank with a long measured
wait streams it in.
Default for two-rank Metal TP decode. DS4_TP_STATIC_SHARED_SPLIT=1
restores the fixed halves; DS4_TP_SHARED_SPLIT_SHIFT_Q16 overrides the
shift (0 = static halves through the split kernels, for exactness
checks). Measured on two M5 Max over TB5 RDMA (README prompt, 256
tokens, interleaved pairs): ctx 128 46.6-47.4 vs 45.2-46.7 t/s, ctx 2048
45.6-46.2 vs 45.1 t/s; both ranks wait less at FFN gates. Output is
deterministic and differs from the static split only by floating-point
grouping of the partial sums.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Two-Mac DeepSeek V4 Flash tensor-parallel decode on M5 Max pairs.
- Poll gates (default; DS4_TP_DISABLE_POLL_GATES opts out): each flag gate
ends the command buffer and the next queued buffer opens with a bounded
fresh-line poll released by the service thread, instead of parking on a
shared event whose CPU->GPU wake-up costs 27-43 us. Ring of 16 regions,
~600 ms budget, timeout reported at the next gate.
- Checksum-verified flags: kernel_dsv4_tp_flag_set_checked publishes an
exact integer checksum of the 16 KiB partial; the service thread verifies
the payload in memory before the RDMA post (the flag line can land ~35 us
before the buffer completes; unverified sends were nondeterministic).
- HC post/expand folded into the compound HC producer with a device-scope
barrier across its six threadgroups; the engine defers the post-gate
expand and flushes it before any other reader
(DS4_METAL_DISABLE_HC_EXPAND_PRODUCER_FUSE opts out).
- Gate-time prefetch: the next phase's weights (rank slices, priority
order, cut at this rank's measured peer-wait window) are touched into the
cache before the poll (DS4_TP_DISABLE_GATE_PREFETCH opts out).
- Concurrent routed/shared FFN branches, safe decode command-buffer split,
exact-shape TP MXFP4 static kernels, DSpark direct partial commit, and
split gate profiling from the morning session.
- Diagnostics: DS4_METAL_ENCODER_TIMELINE per-pass GPU timeline,
DS4_METAL_CB_TIMES per-command-buffer driver/GPU timing.
README ctx 128, 256 greedy tokens: 45.5 vs 43.3 t/s (event gates, same
build); frontier logits and generated text byte-identical throughout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
Every ratio-th decode token, each layer finalizes one freshly pooled
compressor row per compressor: RMS norm, RoPE tail, then the FP8
round-trip + F16 commit copy (attention row, 512 floats) or the
Hadamard+FP4 QAT (indexer row, 128 floats), plus the ratio-4 state
shifts for both. Those were eleven tiny single-row dispatches per
layer per emit token; kernel_dsv4_comp_row_finalize_f32 runs them as
ONE dispatch: threadgroup 0 finalizes the attention row, threadgroup 1
the indexer row, threadgroups 2-21 do both state shifts.
Bit-exact by construction: the norm phases replicate
kernel_rms_norm_mul_f32_4's float4/simd_sum/zero-padded cross-simdgroup
tree at its exact thread counts (128 for the 512-row, 32 for the
128-row); the RoPE phases call ds4_rope_tail_pair_affine_row verbatim
(lanes 0-63, nthreads=64); the FP8 phase replicates
kernel_dsv4_fp8_kv_quantize_f32's 64-lane shmem max tree; the QAT phase
replicates kernel_dsv4_indexer_hadamard_fp4_f32; the commit copy and
state shifts are elementwise. Threads outside a phase's virtual width
still execute every barrier, so barriers stay uniform.
The compressor update wrapper gains a defer_finalize flag that stops
after the pool (the shift is deferred too and rides the fused
dispatch). Default on M5 (DS4_METAL_ENABLE_COMP_FINALIZE_FUSE opts in
elsewhere; DS4_METAL_DISABLE_M5_COMP_FINALIZE_FUSE rolls back).
Evidence on M5 Max, ds4f-q2: six controlled inverse A/B runs, all
positive (+0.36%/+0.41%/+0.42%/+0.42%/+0.48%/+0.59%), every retained
block exact (exact_rows=1041 exact_floats=134580480
exact_selected_ids=1040); official frontier hashes canonical.
One dispatch now covers the q_a/kv Q8 pair projection and the four F16
compressor projections (attention + indexer) with their state-store
epilogue in the resident FULL-phase decode layer. Both stages read the
same normalized attention input and write disjoint outputs.
kernel_dsv4_qkv_pair_quad_compressor_store_q8_0 hosts two virtual NSG=4
cohorts per threadgroup for the Q8 pair range (exact replica of
kernel_mul_mv_q8_0_f32_pair per-lane K walk and reduction tree, same
cohort trick as kernel_dsv4_router_shared_gate_up_q8_0); the compressor
ranges run kernel_mul_mv_f16_f32_pair_4_impl<2> and the paired store
epilogue verbatim. Every output bit matches the separate dispatches:
controlled A/B exact_rows=1041 exact_floats=134580480
exact_selected_ids=1040 in every retained block, and the official
frontier hash stays eb794f497861d7d9e373665f6e115d8ebe4e17c13c431aa7e318282f16a0f21d.
Default on M5 Apple silicon only (DS4_METAL_ENABLE_QKV_PAIR_QUAD_FUSE
opts in elsewhere; DS4_METAL_DISABLE_M5_QKV_PAIR_QUAD_FUSE rolls back).
Measurements on M5 Max, ds4f-q2: inverse A/B (disable) -0.44%/-0.57%;
interleaved official replicas +0.40%/+1.48% decode.
Integrate the MXFP4 Metal decode/prefill optimizations (rounds 7-9) as one
commit. Keep both the MXFP4 half_lut/tail_cull down-pipeline selection and
main's mpp_mask override for IQ2_XXS gate/up and Q2_K/IQ2_XXS down; they
target disjoint quant types and coexist.
Port of the Entrpi/ds4 batched-serving fork's per-layer decode graph
design (fork commits abe26575, 2200c675, 9caaa508; author Entrpi
<entrpi@proton.me>) onto upstream's decode body. Two
position-independent islands of each decode layer are captured once
per activation-buffer variant and replayed on later tokens: the hc-pre
stage (exactly the TO_QKV phase prefix) and the
attention-output-to-layer-end tail (exactly the FROM_ATTN_TO_FFN phase
suffix). Capture and replay reuse the existing phase machinery, so
there is no duplicated encode path.
Island kernels ride cuda_decode_stream(): the legacy NULL stream in
eager mode (bit-identical behavior to before) or a dedicated blocking
stream while capturing/replaying, which preserves the legacy stream's
implicit ordering; cuBLAS is pointed at the capture stream for the
duration of a capture. First sight of a key runs eager to warm lazy
allocators, second sight captures + instantiates + launches, later
sights replay. Any failure retires the entry and re-encodes the island
eagerly, so a broken capture cannot lose a token's work. TP,
multi-tier, SSD streaming, debug dumps, profiling, reference kernels,
and the hash-router layers (token-dependent router argument) all
disqualify capture and stay eager. DS4_CUDA_DECODE_GRAPHS=0 disables;
DS4_CUDA_DECODE_GRAPH_LOG=1 traces captures.
Measured on DGX Spark GB10 (sm_121), DeepSeek V4 Flash q2 imatrix
gguf: 300-token greedy output byte-identical to main with graphs on
(86 island-0 + 80 island-1 graphs live); decode 17.2-17.9 t/s vs
17.0-17.5 baseline (+1-2%). The modest win shows launch overhead is
not the dominant serial-decode cost in upstream's body at 2k context;
the capture substrate is groundwork for extending coverage to the
position-dependent middle section (fork-style device-scalar
substrates), where the remaining launches live.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011CsqQzZEeuz9biUSEkGm8b
Tarjei Mandt
davidandClaude Fable 5.1
Salvatore Sanfilippo
Donato Capitella
Donato Capitella
Entrpi