Qwen3.8 sessions were excluded from native session batching: each decode
step ran the whole graph once per session, so sixteen sessions cost
sixteen times the weight traffic. The batched encoder now runs the dense
work (hyper-connection mixers, projections, MoE, output head) once over
the stacked rows and keeps only the recurrent state, the caches and the
PLE history per session, with batched GDN scan and conv kernels reading
the pooled state through per-row slots.
Session transients (prefill scratch) move to one shared arena sized for
the largest session, so each further session costs its caches only
(memory per session divided by ~3.6 at ctx 4096). The dense GEMM gains a
split-K path for the narrow-batch regime, and the dense/MoE/mixer GEMM
thresholds, tuned for T=1 or T>=8192, are moved to serve decode batches
of 4..64 rows. Every step keeps the tokens of the sequential path;
float reduction order changes where the batched kernels do.
C=16 at ctx 4096 on an M5 Max goes from ~50 to ~143 tok/s aggregate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Accumulate dense component products directly in the output and fuse short F16 projections while retaining FP32 correction sums and cuBLAS fallbacks. Tune expert tile sizes by weight format and unpack IQ2 signs in packed bytes without changing represented weights. Keep BF16 n-grams disk-only.
Extend independent kernel references for large outputs, partial tiles, alignment, strides, repeated writes and output guards. Refresh Spark performance references and repeatable sanitizer checks.
Paired Spark runs improve Q2/Q4 large prefill from 685/670 to 745/755 tokens/s, Q2 decoding from 21.0 to 22.6 tokens/s, and Q2 English MTP from 26.8 to 30.3 tokens/s. Official continuation scores remain unchanged; kernel references, sanitizers, session/MTP, mixed-prefill and vision checks pass.
Collect references from Alibaba through OpenRouter with an explicit empty system message and no provider fallback. Validate token byte alignment and reject lossy Unicode metadata for API agreement while retaining continuation likelihood scoring.
Track 100 short and 12 long reference cases, Q2/Q4 default and quality controls, continued-prefill results, and independent long-context answers. Include collector and scorer regressions and document the hosted-checkpoint and precision limitations.
Share the Metal session graph with CUDA for text and vision, MTP and exact sampling, steering, checkpoints, and disk-only BF16 n-grams. Add bounded concurrent Linux n-gram reads, vectorized dense and expert kernels, register-held recurrent prefill state, and efficient small speculative batches.
Keep Qwen out of DeepSeek batch dispatch and exclude disk n-grams from resident memory accounting. Use normal file-backed weight loading in diagnostics and link n-gram support into the CUDA quality scorer.
Include independent kernel and memory-safety tests, mixed-prefill and recovery checks, long-context server and agent coverage, and user documentation with measured DGX Spark performance.
Use one verified GGUF per quantization, remove sidecar setup, and document disk versus resident size. Record native-reader, model, agent/server, steering, fallback, and continued-prefill QA with explicit limits.
Restore matching logits on verifier rewinds, replay stale recurrent state, discard old snapshots on restore, and invalidate partial payload reads. Keep lazy snapshots all-or-nothing and respect small prefill buffers and the unfused diagnostic path.
Preserve schema-declared string arguments. Repair all-row reference dumps and teacher-forced chunking. Add model-backed regressions and fix test-fixture leaks exposed by ASAN.
Document the measured README, fresh and continued prefill gains, exact state and continuation checks, memory bounds and agent regression. Include the small automatic-cache decode cost instead of claiming a decoding speedup.
Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
Preserve all five original commits by Donato Capitella. Resolve the DSpark fallback conflict by retaining the existing Metal/DGX Spark short-proposal behavior and adding the ROCm reference-alignment condition.
Closes#992
Pair the previous target feature with the next seed at their respective positions and construct target KV directly from the projected normalized feature. Merge trusted target captures into the model support window instead of exposing the full prefill or replacing history with each verifier batch. Keep temporary draft rows outside persistent history and reset safely across capture gaps.
Scope the correction to physical gfx1151 independently of diagnostic fast-path overrides. Add 162 CPU cache-range and physical-ring assertions, and document the tested five-draft/confidence-0.7 configuration. Exact-sampling history-gap performance remains unqualified.
Sampled coding continuation improves 14.17/14.38 to 25.10/24.46 tokens/s at 8K/16K. Long coding passes 24/24 checks, sampled coding 7/7, and an actual tool round trip passes. Both 100-case official reports remain identical; the 64K native continuation completes at 22.61 tokens/s.
Use the model-defined 2051-token dense boundary, handle padded selections, and synchronize reused reduction storage in Metal, CUDA and ROCm, including vision normalization.
Enable bounded ROCm selected-attention GEMM for GLM 5.3 without RoPE. Add independent attention references, boundary tests and finite, repeatable vision checks. Focused Strix continued prefill improves from about 21 to 68 tokens per second; stored official scores are unchanged.
Keep the introduction and everyday usage in the README, with setup guides near the top and detailed references at the end. Document Ada/L40S support explicitly and replace stale model, memory, caching and performance claims.
Retain the root Strix Halo page as a link to its new guide.
antirez
davidandClaude Fable 5.1
Donato Capitella