jpeg_prog_decode_ac_refine treated ZRL (run=15, size=0) as run=16 and
reused the skip loop, which kept consuming correction bits for non-zero
coefficients after the 16th zero-history coefficient. Those bits belong
after the next Huffman symbol (ITU T.81 G.1.2.3, libjpeg
decode_mcu_AC_refine), so the bitstream desynchronised: many progressive
JPEGs, including plain `cjpeg -progressive` output, failed to load or
decoded with wrong pixels.
Skip `run` zeros while refining non-zeros on the way, write the new value
(0 for ZRL) at the next zero, and stop, matching libjpeg's structure.
Add tests/test_image_decode with two cjpeg-generated fixtures that fail
on the old decoder (wrong pixels for grayscale, NULL for 4:2:0) and are
bit-exact with djpeg after the fix. Wire it into `make test`.
(cherry picked from commit 50ed0a4c73)
Serving several requests at once is a different regime from the single
session walk ds4-bench measures. session_concurrency_bench drives the
engine API directly over a concurrency x context grid and reports the
aggregate decode throughput, the per-stream loss as the width grows, the
cost a resumed prefill imposes on the decoders, and a teacher-forced check
that a batched step emits the tokens the sequential path emits. Cells are
priced from the engine's own memory accounting against a budget sampled
at startup, because macOS swaps long before an allocation fails.
concurrency_sweep.sh runs one process per cell; serve_concurrency_bench.py
measures the HTTP server with vLLM-style metrics.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Share the Metal session graph with CUDA for text and vision, MTP and exact sampling, steering, checkpoints, and disk-only BF16 n-grams. Add bounded concurrent Linux n-gram reads, vectorized dense and expert kernels, register-held recurrent prefill state, and efficient small speculative batches.
Keep Qwen out of DeepSeek batch dispatch and exclude disk n-grams from resident memory accounting. Use normal file-backed weight loading in diagnostics and link n-gram support into the CUDA quality scorer.
Include independent kernel and memory-safety tests, mixed-prefill and recovery checks, long-context server and agent coverage, and user documentation with measured DGX Spark performance.
Keep the table outside model mappings and Metal residency. Use bounded explicit reads, reject failed lookups before GPU consumption, and remove the runtime sidecar option and paging controls. Preserve Qwen hash and speculative state handling; cover malformed files and exact recovery after prefill, decode, and MTP read failures.
Include the image helper object used by the CUDA visual-attention code in standalone test binaries. Let the engine runtime test reuse the normal core dependencies, excluding its separately instrumented engine object.
Verified all affected targets link on DGX Spark sm_121a; CUDA long-context, Q8 row and DeepSeek 4.1 kernel regressions pass.
Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
Use the same cache clock for typed and generic GLM routes, including models with leading dense layers. Report every selected expert in routing diagnostics. Add an eviction oracle that checks all eight slots and byte-identical output across different cache capacities.
Do not HTML-unescape argument bodies. Share a narrow, reversible escape for each body closing delimiter, including its literal escaped spelling. Keep streaming fragments intact across those escapes. Preserve protocol-looking text inside GLM agent arguments until the actual argument terminator. Add round-trip, streaming and agent regressions.
Reuse the existing fused IQ2 gate/up and Q2 down paths for up to eight resident DeepSeek rows on GB10. Keep GLM, streaming and multi-GPU dispatch unchanged. Test overlapping experts, distinct gate/up weights, invalid IDs and exact agreement with separate rows; CUDA memory, race and synchronization checks pass.
Avoid per-call pool allocations in dense and paired aligned Q8 kernels. Reuse the existing device-checked buffer, retaining the pool fallback for missing or undersized scratch.
Three interleaved Spark Flash Q2 runs improve 2K/4K/8K steady decode from 17.90/15.23/15.01 to 19.25/16.22/15.97 t/s. Prefill is unchanged and full prefill and post-decode logits match exactly. Add scratch fallback and graph replay tests.
Use the model-defined 2051-token dense boundary, handle padded selections, and synchronize reused reduction storage in Metal, CUDA and ROCm, including vision normalization.
Enable bounded ROCm selected-attention GEMM for GLM 5.3 without RoPE. Add independent attention references, boundary tests and finite, repeatable vision checks. Focused Strix continued prefill improves from about 21 to 68 tokens per second; stored official scores are unchanged.
Exclude CMA pages from host availability, guard pinned and integrated-GPU allocations, and reserve space for the OS and required runtime buffers. Defer optional Q8 expansion and avoid wasting large model arenas.
Add memory parsing and allocation-admission tests. Focused Strix checks cover matched Flash/DSpark, resident GLM and automatic, oversized and one-expert SSD cache budgets.
Restore GLM recurrent and pooled-indexer state after rejected speculation. Invalidate unsafe DeepSeek rewinds and rebuild retained agent prefixes with their images before decoding again.
Rebuild saved TP token prefixes on both ranks. Give GLM speculative cycles an explicit command and shared token limit, with protocol version 10 requiring peers to update together.
Account for live and planned Metal GLM sessions when reserving graph and streaming memory. Add model-free state and protocol tests plus a small GPU rollback regression.
Use the M5 seed schedule under TP and evaluate verifier rows together instead of running each row separately. Reuse the existing static MXFP4 kernels with row-aware addressing, and enable packed MXFP4 activations for two-rank M5 prefill.
Keep streaming, exact-sampling scheduling and unmeasured device defaults unchanged. Add synthetic ownership/static-shape checks and a physical TP speculative oracle. Record focused quality, speed comparisons and transport caveats in the QA guide.
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
Add 'make test-rocm', mirroring 'make test' for ROCm-only hosts (the
CUDA-specific sampling/session oracles stay in 'make test'), and link
ds4_test/ds4_agent_test/score_official through DS4_LINK like the main
binaries so they build without nvcc. score_official now compiles its
translation unit in a separate step with ROCM_HOST_CFLAGS: hipcc's link
driver injects -x c ahead of the object list and would otherwise try to
parse relocatables as C source, and the Strix host objects are built
position-independent so the scorer TU must match.
Production kernel trace of the 10.5k-token prefill (rocprofv3 on the
coordinator) showed the MXFP4 routed-MoE tile kernels dominate prefill:
gate/up 247.7 ms and down 174.9 ms per layer-chunk, bound not by
compute (~0.6% of fp32 peak) but by expert-weight DRAM re-reads - every
8-token tile re-reads its experts' full gate+up rows.
Changes (all bit-exact; per-element accumulation order untouched):
- gate/up tile8 kernel grid swapped to (tile, row-block): consecutive
blocks re-walk the same 32-row weight slice across an expert's token
tiles, so the slice stays L2-resident. Production: 247.7 -> 222.3 ms
per layer-chunk (-10.3%), 10.5k prefill 61.5 s -> 59.9 s.
- A-activation LDS staging threshold raised 16 -> 28 q8_K blocks (the
7168-dim case; 56.9 KiB dynamic smem with the >48 KiB opt-in).
Neutral in production (L2 was already absorbing the re-reads), large
in the isolated bench; staged and unstaged dot helpers are bit-equal
(integer bsum order is exact; verified in-tree and at 7168 dims).
- down-kernel grid swap measured NEGATIVE (174.9 -> 184.8 ms) and was
reverted.
- Two experimental kernels parked default-off behind env gates with
honest numbers: tile32 B-register reuse (DS4_ROCM_ENABLE_MXFP4_TILE32;
register spills, 2155 ms bench) and row8-ldsB B-staging
(DS4_ROCM_ENABLE_MXFP4_LDSB; 154 s production - scattered A reads).
- tests/bench_mxfp4_rocm: production-shape MoE bench harness. NOTE:
it is a poor proxy for production kernel time (isolated process
memory/L2 behavior differs ~5x); production traces are the arbiter.
- DS4_ROCM_MOE_PATH_DEBUG: launch-path diagnostics.
Validation: tests/test_mxfp4_rocm PASS (default dims + 7168/top-8
variant), acceptance gate df07199e5a292872 holds, decode 15.38 t/s,
10.5k probe 59.9 s (176 t/s avg).
Implement resident MXFP4 gate/up and down paths for decode and batched routed MoE on ROCm. Add a synthetic CPU-oracle regression covering full, partial, and large expert tiles, report MXFP4 as a 4-bit routed quant, and make Strix host objects position independent for TheRock linking.
Drop duplicate ignore entries, nest the Metal bench binaries, clarify the
historical Round-15 test note, and gate test-mxfp4-metal on the generator
--check so the checked-in MXFP4 half LUT cannot drift silently.
Integrate the MXFP4 Metal decode/prefill optimizations (rounds 7-9) as one
commit. Keep both the MXFP4 half_lut/tail_cull down-pipeline selection and
main's mpp_mask override for IQ2_XXS gate/up and Q2_K/IQ2_XXS down; they
target disjoint quant types and coexist.
Compile RTX 50 and RTX PRO Blackwell targets as compute_120a/sm_120a so the vendored block-scaled MXFP4 instructions are accepted. Accept both sm_120 and sm_120a spellings.
Closes#672Closes#679
Mark Shank
korale77
davidandClaude Fable 5.1
Salvatore Sanfilippo
Donato Capitella
kyuz0
Jonathan Yates
Entrpi