Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Restore GLM recurrent and pooled-indexer state after rejected speculation. Invalidate unsafe DeepSeek rewinds and rebuild retained agent prefixes with their images before decoding again.
Rebuild saved TP token prefixes on both ranks. Give GLM speculative cycles an explicit command and shared token limit, with protocol version 10 requiring peers to update together.
Account for live and planned Metal GLM sessions when reserving graph and streaming memory. Add model-free state and protocol tests plus a small GPU rollback regression.
- ds4_tp.c: ds4_tp_batch_block_begin/end arm a receive window for a whole
speculative verify block (one batch gate per layer): the decode window is
drained once, the first layer's receives are posted behind a single
control-byte barrier, and every gate posts the next layer's receives
before sending its own rows, so no send can reach an unposted queue and
no per-gate TCP handshake is needed (the old path cost two round trips
per gate). Both ranks call it from the verify path (ds4.c, via the TP
context stored at ds4_engine_tp_bind). DS4_TP_DISABLE_VERIFY_WINDOW=1
keeps the old per-gate protocol. -4 to -5 ms per verify block.
- ds4.c: the worker's prefix commit no longer tries to preserve a DSpark
capture it never has (it fails with DS4_DSPARK_SEED_BATCH set); the fused
seed batch can be enabled under TP with DS4_DSPARK_SEED_BATCH_TP=1
(experiment: on this pair it verifies more rows per cycle and is slower).
- Diagnostics: DS4_DSPARK_CYCLE_TRACE=1 (per-cycle propose/verify/commit
timing), DS4_BENCH_SPEC_TRACE=1 (bench per-call cost by accepted count),
DS4_TP_GATE_PROFILE now also reports verify-block batch gates (arrival
wait, exchange, release per gate).
Measured on the M5 Max pair (README prompt, 128 context): a verify block
costs ~27 ms + ~5.7 ms per draft row of GPU time (each row re-streams its
own routed experts at bandwidth) plus ~3 ms of gates; DSpark decode stays
at ~35 t/s vs ~48 t/s plain on prose (24-32% draft acceptance).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
The 8-group attention split adopted after the sweep saved 1 us in the
attention kernels but made the layer's later concurrent FFN block ~50 us
slower (114 -> 162 us; cause not understood, measured by toggling
DS4_METAL_FLASH_NWG on one build), costing ~4 t/s. Back to 32 groups;
the env override stays for experiments. With it the default output is
byte-identical to the pre-session reference again, which also validates
the kv-task fold and the two-rank router-select fusion against it.
The two-rank router-select branch no longer sets router_only_done (that
enabled the single-machine parallel FFN path and computed the whole
shared expert on the coordinator); it only skips the plain router matvec
and the standalone select.
Rank-speed lane bias, opt-in (DS4_TP_LANE_BIAS=1): the gate service
probes the peer's RDMA arrival while it spins on its own flag, giving a
signed per-gate skew (peer early > 0) without any protocol change; the
coordinator turns the FFN-gate skew EWMA into a lane bias (default 19
lanes per us of skew, smoothed, +-shared_dim/2, 64-lane steps), carries
it in the EVAL command's reserved word, and both ranks add it to the
shared-expert split. The logits-half frame carries one extra float (the
sender's FFN-gate wait) for future use. Measured on this pair: skew 2-5
us, 64-128 lanes moved, no token-rate change (48.0 vs 48.2, 47.0 vs
46.4), so it stays off by default because it makes the output depend on
timing. DS4_TP_LANE_BIAS_GAIN and DS4_TP_LANE_BIAS_DEBUG for experiments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Apple's Thunderbolt RDMA (UC queue pairs, two-sided SEND/RECV) has a
failure mode UC never reports: a send that reaches a queue pair with no
receive posted does not merely get lost, that direction of the pair
stops delivering for good (the pair then reads RTR on query and later
sends never complete). Runs used to discover this only at the first
bulk prefill round, as "timeout waiting for bulk RDMA round (0/64
recvs, send=0)", typically on the first run after a role swap.
After the ready barrier both sides now run a warm-up round: post one
full-size receive, exchange a "receives posted" control frame
(DS4_TP_FRAME_RDMA_POSTED), then resend a tagged 16 KB message until the
peer confirms receipt over the control channel (DS4_TP_FRAME_RDMA_WARM,
100 ms per attempt). A failed warm-up recreates the queue pair and
completion queue (new number, new PSN), redoes the info exchange and
warm-up, up to four setups; both sides take the same path because the
status exchange gives them the same outcome. Bulk-round receive
completions check the byte length, and a bulk timeout reports the queue
pair state and chunk layout. DS4_TP_DISABLE_RDMA_WARMUP=1 skips the
round.
Measured on the M5 Max pair: with the posted barrier the warm-up passes
on the first attempt in every run (10/10); without the barrier the
warm-up itself raced the peer's send and poisoned pairs. The bulk
round keeps its pre-existing race (a peer that finished the previous
round earlier can send before our receives are posted); a control-frame
barrier there collides with the eval protocol, so the fix is to pre-post
the next round's receives into a second staging area before sending the
current round. Noted in misc/MAC_TP_SPEED.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Two-Mac DeepSeek V4 Flash tensor-parallel decode on M5 Max pairs.
- Poll gates (default; DS4_TP_DISABLE_POLL_GATES opts out): each flag gate
ends the command buffer and the next queued buffer opens with a bounded
fresh-line poll released by the service thread, instead of parking on a
shared event whose CPU->GPU wake-up costs 27-43 us. Ring of 16 regions,
~600 ms budget, timeout reported at the next gate.
- Checksum-verified flags: kernel_dsv4_tp_flag_set_checked publishes an
exact integer checksum of the 16 KiB partial; the service thread verifies
the payload in memory before the RDMA post (the flag line can land ~35 us
before the buffer completes; unverified sends were nondeterministic).
- HC post/expand folded into the compound HC producer with a device-scope
barrier across its six threadgroups; the engine defers the post-gate
expand and flushes it before any other reader
(DS4_METAL_DISABLE_HC_EXPAND_PRODUCER_FUSE opts out).
- Gate-time prefetch: the next phase's weights (rank slices, priority
order, cut at this rank's measured peer-wait window) are touched into the
cache before the poll (DS4_TP_DISABLE_GATE_PREFETCH opts out).
- Concurrent routed/shared FFN branches, safe decode command-buffer split,
exact-shape TP MXFP4 static kernels, DSpark direct partial commit, and
split gate profiling from the morning session.
- Diagnostics: DS4_METAL_ENCODER_TIMELINE per-pass GPU timeline,
DS4_METAL_CB_TIMES per-command-buffer driver/GPU timing.
README ctx 128, 256 greedy tokens: 45.5 vs 43.3 t/s (event gates, same
build); frontier logits and generated text byte-identical throughout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
Salvatore Sanfilippo
