Keep the table outside model mappings and Metal residency. Use bounded explicit reads, reject failed lookups before GPU consumption, and remove the runtime sidecar option and paging controls. Preserve Qwen hash and speculative state handling; cover malformed files and exact recovery after prefill, decode, and MTP read failures.
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
- ds4_tp.c: ds4_tp_batch_block_begin/end arm a receive window for a whole
speculative verify block (one batch gate per layer): the decode window is
drained once, the first layer's receives are posted behind a single
control-byte barrier, and every gate posts the next layer's receives
before sending its own rows, so no send can reach an unposted queue and
no per-gate TCP handshake is needed (the old path cost two round trips
per gate). Both ranks call it from the verify path (ds4.c, via the TP
context stored at ds4_engine_tp_bind). DS4_TP_DISABLE_VERIFY_WINDOW=1
keeps the old per-gate protocol. -4 to -5 ms per verify block.
- ds4.c: the worker's prefix commit no longer tries to preserve a DSpark
capture it never has (it fails with DS4_DSPARK_SEED_BATCH set); the fused
seed batch can be enabled under TP with DS4_DSPARK_SEED_BATCH_TP=1
(experiment: on this pair it verifies more rows per cycle and is slower).
- Diagnostics: DS4_DSPARK_CYCLE_TRACE=1 (per-cycle propose/verify/commit
timing), DS4_BENCH_SPEC_TRACE=1 (bench per-call cost by accepted count),
DS4_TP_GATE_PROFILE now also reports verify-block batch gates (arrival
wait, exchange, release per gate).
Measured on the M5 Max pair (README prompt, 128 context): a verify block
costs ~27 ms + ~5.7 ms per draft row of GPU time (each row re-streams its
own routed experts at bandwidth) plus ~3 ms of gates; DSpark decode stays
at ~35 t/s vs ~48 t/s plain on prose (24-32% draft acceptance).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
- ds4_tp.c: the prefill big-gate exchange now moves the partial through the
slab batch regions in windows of up to 256 16 KB messages, receives posted
in chains of 64 behind a one-byte control barrier, sends in chains of 64
with at most four signaled chains in flight (32 MB in 4-6 ms instead of
~26 ms). Queue pairs ask for 1024/1024 work requests (fallback 256/64)
and the device limits are printed at open. Payload-buffer registration
stays opt-in (DS4_TP_RDMA_REG_PAYLOAD): Apple's verbs cap a memory region
at 16 MB and registering Metal buffers gave LOC_PROT_ERR completions.
DS4_TP_BIG_GATE_DEBUG=1 prints handshake/transfer times every 32 gates.
- ds4_metal.m: a command buffer submitted after the engine's queue idled
for ~3 s starts 600-800 ms late on M5 Max (measured; other queues do not
prevent it). A main-queue keepalive submits one 1 KB kernel per second
(DS4_METAL_DISABLE_QUEUE_KEEPALIVE=1 opts out) and the sharded load pays
the first submission up front (ds4_gpu_warm_command_queue). The big-gate
arrival spin sleep-polls (DS4_TP_BIG_GATE_SPIN_US). Diagnostics: blit
passes in the encoder timeline, DS4_METAL_CB_TIMES prints encode/commit/
GPU-span per owned command buffer with a backtrace on stalls.
Two-Mac TP prefill, byte-exact frontier logits: 2048 tokens 565 -> 673 t/s,
128 tokens 93 -> 164 t/s; decode unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
Add shared help text across the CLI, server, agent, bench, and eval tools. Expand distributed-mode guidance, clean up endpoint naming, and use a TTY-only 256-color layout with clearer section titles, option arguments, separators, examples, and explanatory text.
Add DeepSeek V4 Pro support on top of main.
- add Flash/Pro shape selection and Pro Metal inference support
- fix shared SwiGLU clamp and indexer QAT behavior for Flash and Pro
- update eval/context tooling and imatrix quantization for Pro
- tag KV caches/sessions with model ids so Flash and Pro state cannot cross-load
- remove misc/PRO.md from tracked files while leaving it as a local scratch log
antirez
Donato Capitella
Donato Capitella
Salvatore Sanfilippo
user