Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Restore GLM recurrent and pooled-indexer state after rejected speculation. Invalidate unsafe DeepSeek rewinds and rebuild retained agent prefixes with their images before decoding again.
Rebuild saved TP token prefixes on both ranks. Give GLM speculative cycles an explicit command and shared token limit, with protocol version 10 requiring peers to update together.
Account for live and planned Metal GLM sessions when reserving graph and streaming memory. Add model-free state and protocol tests plus a small GPU rollback regression.
- ds4_tp.c: ds4_tp_batch_block_begin/end arm a receive window for a whole
speculative verify block (one batch gate per layer): the decode window is
drained once, the first layer's receives are posted behind a single
control-byte barrier, and every gate posts the next layer's receives
before sending its own rows, so no send can reach an unposted queue and
no per-gate TCP handshake is needed (the old path cost two round trips
per gate). Both ranks call it from the verify path (ds4.c, via the TP
context stored at ds4_engine_tp_bind). DS4_TP_DISABLE_VERIFY_WINDOW=1
keeps the old per-gate protocol. -4 to -5 ms per verify block.
- ds4.c: the worker's prefix commit no longer tries to preserve a DSpark
capture it never has (it fails with DS4_DSPARK_SEED_BATCH set); the fused
seed batch can be enabled under TP with DS4_DSPARK_SEED_BATCH_TP=1
(experiment: on this pair it verifies more rows per cycle and is slower).
- Diagnostics: DS4_DSPARK_CYCLE_TRACE=1 (per-cycle propose/verify/commit
timing), DS4_BENCH_SPEC_TRACE=1 (bench per-call cost by accepted count),
DS4_TP_GATE_PROFILE now also reports verify-block batch gates (arrival
wait, exchange, release per gate).
Measured on the M5 Max pair (README prompt, 128 context): a verify block
costs ~27 ms + ~5.7 ms per draft row of GPU time (each row re-streams its
own routed experts at bandwidth) plus ~3 ms of gates; DSpark decode stays
at ~35 t/s vs ~48 t/s plain on prose (24-32% draft acceptance).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
- ds4_gpu_tp_big_gate_kick_flush: a kick that ends the command buffer so
the shared-event signal is published while the caller keeps encoding.
The service thread then waits for that buffer's completion before it
reads the payload: the event fires when the GPU is done, but the stores
only land for the CPU at buffer completion (measured: the first ~10 MB
of the second half's rows were stale without the wait). The attention
sub-gate pipeline (DS4_TP_SUBGATE_PIPELINE=1) now uses it too.
- FFN sub-gate (DS4_TP_FFN_SUBGATE=1, off by default): the routed partial
is produced in two row halves with this rank's shared-expert rows folded
per half before each kick, so the exchanged rows are what the peer adds.
Byte-identical to the single gate but net-negative on the M5 Max pair
(781 vs ~850 t/s at 2048 tokens: two 1024-row routed calls cost ~27%
more kernel time than one 2048-row call).
- Diagnostics: DS4_TP_BIG_GATE_DEBUG=2 logs every big gate (period since
the previous release, handshake, transfer); DS4_TP_FAKE_BIG_GATE=1 skips
the exchange for a compute-only timing bound (results are garbage);
DS4_TP_KICK_SETTLE_US adds a delay after the kick-buffer wait;
tp_ffn_recv / tp_ffn_sum tensor dumps after the FFN all-reduce.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
- ds4_tp.c: the prefill big-gate exchange now moves the partial through the
slab batch regions in windows of up to 256 16 KB messages, receives posted
in chains of 64 behind a one-byte control barrier, sends in chains of 64
with at most four signaled chains in flight (32 MB in 4-6 ms instead of
~26 ms). Queue pairs ask for 1024/1024 work requests (fallback 256/64)
and the device limits are printed at open. Payload-buffer registration
stays opt-in (DS4_TP_RDMA_REG_PAYLOAD): Apple's verbs cap a memory region
at 16 MB and registering Metal buffers gave LOC_PROT_ERR completions.
DS4_TP_BIG_GATE_DEBUG=1 prints handshake/transfer times every 32 gates.
- ds4_metal.m: a command buffer submitted after the engine's queue idled
for ~3 s starts 600-800 ms late on M5 Max (measured; other queues do not
prevent it). A main-queue keepalive submits one 1 KB kernel per second
(DS4_METAL_DISABLE_QUEUE_KEEPALIVE=1 opts out) and the sharded load pays
the first submission up front (ds4_gpu_warm_command_queue). The big-gate
arrival spin sleep-polls (DS4_TP_BIG_GATE_SPIN_US). Diagnostics: blit
passes in the encoder timeline, DS4_METAL_CB_TIMES prints encode/commit/
GPU-span per owned command buffer with a backtrace on stalls.
Two-Mac TP prefill, byte-exact frontier logits: 2048 tokens 565 -> 673 t/s,
128 tokens 93 -> 164 t/s; decode unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The 8-group attention split adopted after the sweep saved 1 us in the
attention kernels but made the layer's later concurrent FFN block ~50 us
slower (114 -> 162 us; cause not understood, measured by toggling
DS4_METAL_FLASH_NWG on one build), costing ~4 t/s. Back to 32 groups;
the env override stays for experiments. With it the default output is
byte-identical to the pre-session reference again, which also validates
the kv-task fold and the two-rank router-select fusion against it.
The two-rank router-select branch no longer sets router_only_done (that
enabled the single-machine parallel FFN path and computed the whole
shared expert on the coordinator); it only skips the plain router matvec
and the standalone select.
Rank-speed lane bias, opt-in (DS4_TP_LANE_BIAS=1): the gate service
probes the peer's RDMA arrival while it spins on its own flag, giving a
signed per-gate skew (peer early > 0) without any protocol change; the
coordinator turns the FFN-gate skew EWMA into a lane bias (default 19
lanes per us of skew, smoothed, +-shared_dim/2, 64-lane steps), carries
it in the EVAL command's reserved word, and both ranks add it to the
shared-expert split. The logits-half frame carries one extra float (the
sender's FFN-gate wait) for future use. Measured on this pair: skew 2-5
us, 64-128 lanes moved, no token-rate change (48.0 vs 48.2, 47.0 vs
46.4), so it stays off by default because it makes the output depend on
timing. DS4_TP_LANE_BIAS_GAIN and DS4_TP_LANE_BIAS_DEBUG for experiments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Apple's Thunderbolt RDMA (UC queue pairs, two-sided SEND/RECV) has a
failure mode UC never reports: a send that reaches a queue pair with no
receive posted does not merely get lost, that direction of the pair
stops delivering for good (the pair then reads RTR on query and later
sends never complete). Runs used to discover this only at the first
bulk prefill round, as "timeout waiting for bulk RDMA round (0/64
recvs, send=0)", typically on the first run after a role swap.
After the ready barrier both sides now run a warm-up round: post one
full-size receive, exchange a "receives posted" control frame
(DS4_TP_FRAME_RDMA_POSTED), then resend a tagged 16 KB message until the
peer confirms receipt over the control channel (DS4_TP_FRAME_RDMA_WARM,
100 ms per attempt). A failed warm-up recreates the queue pair and
completion queue (new number, new PSN), redoes the info exchange and
warm-up, up to four setups; both sides take the same path because the
status exchange gives them the same outcome. Bulk-round receive
completions check the byte length, and a bulk timeout reports the queue
pair state and chunk layout. DS4_TP_DISABLE_RDMA_WARMUP=1 skips the
round.
Measured on the M5 Max pair: with the posted barrier the warm-up passes
on the first attempt in every run (10/10); without the barrier the
warm-up itself raced the peer's send and poisoned pairs. The bulk
round keeps its pre-existing race (a peer that finished the previous
round earlier can send before our receives are posted); a control-frame
barrier there collides with the eval protocol, so the fix is to pre-post
the next round's receives into a second staging area before sending the
current round. Noted in misc/MAC_TP_SPEED.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
- ds4_tp.c: under DS4_TP_GATE_PROFILE print per gate kind the RDMA post cost
and the wait for the peer's partial (the attention gate sits at the
~11 us wire floor; FFN-gate averages include prefill batch gates).
- misc/mxfp4_moe_bench.m: standalone microbench of the exact-shape TP MXFP4
pair-SwiGLU and sum6 kernels on rotating expert copies with a warm-up
pass; COMPACT=1 measures the unowned-slot launch cost.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Two-Mac DeepSeek V4 Flash tensor-parallel decode on M5 Max pairs.
- Poll gates (default; DS4_TP_DISABLE_POLL_GATES opts out): each flag gate
ends the command buffer and the next queued buffer opens with a bounded
fresh-line poll released by the service thread, instead of parking on a
shared event whose CPU->GPU wake-up costs 27-43 us. Ring of 16 regions,
~600 ms budget, timeout reported at the next gate.
- Checksum-verified flags: kernel_dsv4_tp_flag_set_checked publishes an
exact integer checksum of the 16 KiB partial; the service thread verifies
the payload in memory before the RDMA post (the flag line can land ~35 us
before the buffer completes; unverified sends were nondeterministic).
- HC post/expand folded into the compound HC producer with a device-scope
barrier across its six threadgroups; the engine defers the post-gate
expand and flushes it before any other reader
(DS4_METAL_DISABLE_HC_EXPAND_PRODUCER_FUSE opts out).
- Gate-time prefetch: the next phase's weights (rank slices, priority
order, cut at this rank's measured peer-wait window) are touched into the
cache before the poll (DS4_TP_DISABLE_GATE_PREFETCH opts out).
- Concurrent routed/shared FFN branches, safe decode command-buffer split,
exact-shape TP MXFP4 static kernels, DSpark direct partial commit, and
split gate profiling from the morning session.
- Diagnostics: DS4_METAL_ENCODER_TIMELINE per-pass GPU timeline,
DS4_METAL_CB_TIMES per-command-buffer driver/GPU timing.
README ctx 128, 256 greedy tokens: 45.5 vs 43.3 t/s (event gates, same
build); frontier logits and generated text byte-identical throughout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
Tarjei Mandt
Salvatore Sanfilippo
