ds4_sessions_eval_batch_speculative_argmax runs one MTP cycle for a batch:
every session contributes its next token and, when the predictor left it a
draft for that token, the draft as a second row. The dense work of the
trunk runs once over all the rows; the recurrent, n-gram and attention
steps run per session over its one or two rows with the single-session
verify kernels, snapshots included, so a session's own arithmetic is that
of the C=1 speculative cycle. A draft stands when it is the target's
argmax for its row; a rejected one restores the row-0 snapshot. The
predictor then runs its layer per session over the committed rows and
every session's head as one product. The Q8 batch GEMM serves 9..32 rows
in sixteen-row halves.
The harness gains --spec (with --verify checking the committed tokens
against plain greedy decode, which speculation must reproduce). On an M5
Max at C=16 the first cut commits 1.49 tokens per stream and cycle on the
prose corpus but the cycle costs 171 ms against 92 for a plain step: the
per-session kernels and the sequential predictor passes are the next
thing to batch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Serving several requests at once is a different regime from the single
session walk ds4-bench measures. session_concurrency_bench drives the
engine API directly over a concurrency x context grid and reports the
aggregate decode throughput, the per-stream loss as the width grows, the
cost a resumed prefill imposes on the decoders, and a teacher-forced check
that a batched step emits the tokens the sequential path emits. Cells are
priced from the engine's own memory accounting against a budget sampled
at startup, because macOS swaps long before an allocation fails.
concurrency_sweep.sh runs one process per cell; serve_concurrency_bench.py
measures the HTTP server with vLLM-style metrics.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Use one verified GGUF per quantization, remove sidecar setup, and document disk versus resident size. Record native-reader, model, agent/server, steering, fallback, and continued-prefill QA with explicit limits.
Keep the table outside model mappings and Metal residency. Use bounded explicit reads, reject failed lookups before GPU consumption, and remove the runtime sidecar option and paging controls. Preserve Qwen hash and speculative state handling; cover malformed files and exact recovery after prefill, decode, and MTP read failures.
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
Drop duplicate ignore entries, nest the Metal bench binaries, clarify the
historical Round-15 test note, and gate test-mxfp4-metal on the generator
--check so the checked-in MXFP4 half LUT cannot drift silently.
Promote the positive ordinary-Metal M5 decode compounds on the existing M1-M4 path: both QKV/compressor forms, compressor finalization and exact pool, packed FlashAttention, the HC producer continuation, and the exact-shape concurrent Q2 FFN. Keep all existing fallbacks and add dominant per-feature plus aggregate rollback controls.
Revalidate the resident IQ2XXS/Q2_K 2/32 command-buffer schedule and admit it only for the exact single-device tape and measured position window. Explicit split values, including second=0, remain authoritative.
On M3 Ultra, the official interleaved median moves from 41.08 to 44.18 tok/s (+7.55%; steady 41.44 to 44.52). The 1024-token full-vocabulary A/B is bit exact over 1041 rows, 134580480 F32 logits, and 1040 selected IDs. MXFP4 also improves 39.97 to 42.67 tok/s. Full make test and focused Metal/Q2 tests pass.
Integrate the MXFP4 Metal decode/prefill optimizations (rounds 7-9) as one
commit. Keep both the MXFP4 half_lut/tail_cull down-pipeline selection and
main's mpp_mask override for IQ2_XXS gate/up and Q2_K/IQ2_XXS down; they
target disjoint quant types and coexist.
DGX Spark (GB10, sm_121, 121 GiB UMA, driver 580+) sits in an unusual
spot for CUDA inference: ATS (Address Translation Service) lets the
GPU consume host-mmap'd weights directly, but at significantly lower
effective bandwidth than HBM-resident copies. For an 80 GB IQ2XXS
DeepSeek V4 Flash checkpoint, the difference is the model running
versus the model being usable.
This commit adds:
- Startup HBM cache that copies hot tensor spans (attn projections,
MoE shared experts, output projection) into device memory at engine
init, capped by a configurable budget (defaults sized to leave
headroom for KV cache and a second model load). Cold MoE routed
experts stay ATS-mapped.
- Factored the cudaMalloc+memcpy populate path into a helper and
reordered cuda_model_range_ptr so the HBM-resident lookup is a
single hash-keyed read that wins over the UVA-mapped pointer on
the hot decode path.
- GPU argmax kernel; the prior fallback misused indexer scoring as
an argmax which double-paid the dispatcher cost on N=1 decode.
- Pair-fused Q_A + KV_A matmuls in qkv_rms_fused decode path
(one shared weight load per row, two outputs).
- Parallelized matmul_q8_0_hc_expand epilogue across n_hc lanes
(n_hc parallel residual loads + writes vs n_hc^2 serial reads).
- HBM cache also populated for the MTP support model.
- Drop `cudaHostRegisterReadOnly` flag — unsupported on GB10.
- Drop `!mtp_ready` gate from accelerator_cache_model_tensors so
the MTP support model gets the same HBM-cache treatment.
Bench (DGX Spark / GB10, ds4flash, n=256, "knight" prompt, 3-run mean):
Plain decode before: ~13.9 t/s (ATS-mapped weights, all paths)
Plain decode after: ~16.13 t/s (HBM-resident hot spans + small-N kernel fuses)
Adds `speed-bench/gb10.csv` per CONTRIBUTING.md convention so the
2048..65536 sweep is preserved alongside the existing m2_ultra.csv
and m4_max.csv. Generated via:
./ds4-bench -m ds4flash.gguf \
--prompt-file speed-bench/promessi_sposi.txt \
--ctx-start 2048 --ctx-max 65536 --step-incr 2048 \
--gen-tokens 128 --csv speed-bench/gb10.csv
Hardware: NVIDIA DGX Spark (GB10 / sm_121), driver 580.142, CUDA 13.0
Model: DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
Expose Flash and PRO model IDs as server compatibility aliases for the loaded GGUF. Add download targets for the mixed Flash q2/q4 quant and PRO GGUFs, update README guidance, and add the M3 Ultra PRO benchmark data.
Salvatore Sanfilippo
davidandClaude Fable 5.1
Donato Capitella
kyuz0
Trevor Strieber