Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Do not force full unrolling of runtime-bounded IQ2 tail loads. Keep the ROCm graph memory-baseline initialization out of DS4_NO_GPU builds, matching the declaration and allowing the model-free session-state suite to compile.
Preserve all five original commits by Donato Capitella. Resolve the DSpark fallback conflict by retaining the existing Metal/DGX Spark short-proposal behavior and adding the ROCm reference-alignment condition.
Closes#992
Use direct input loads for full IQ2_XXS tiles with width at least 32 on HIP/gfx1151. For partial tiles, initialize the invalid suffix once and reload only the valid prefix at each K stage. Preserve scratch lifetime, barriers and invalid-tail handling; other backend and quantization templates retain their existing path.
Keep small IQ2 and Q2_K templates on their established masked-load path, where the wider load-scheduling change altered compiler accumulation order during development. The specialization changes input staging without changing the selected matrix operations or their source-level accumulation.
Reuse the existing fused IQ2 gate/up and Q2 down paths for up to eight resident DeepSeek rows on GB10. Keep GLM, streaming and multi-GPU dispatch unchanged. Test overlapping experts, distinct gate/up weights, invalid IDs and exact agreement with separate rows; CUDA memory, race and synchronization checks pass.
Avoid per-call pool allocations in dense and paired aligned Q8 kernels. Reuse the existing device-checked buffer, retaining the pool fallback for missing or undersized scratch.
Three interleaved Spark Flash Q2 runs improve 2K/4K/8K steady decode from 17.90/15.23/15.01 to 19.25/16.22/15.97 t/s. Prefill is unchanged and full prefill and post-decode logits match exactly. Add scratch fallback and graph replay tests.
Exclude CMA pages from host availability, guard pinned and integrated-GPU allocations, and reserve space for the OS and required runtime buffers. Defer optional Q8 expansion and avoid wasting large model arenas.
Add memory parsing and allocation-admission tests. Focused Strix checks cover matched Flash/DSpark, resident GLM and automatic, oversized and one-expert SSD cache budgets.
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
Vendors the Entrpi/ds4 fork's cuda/mmq module at its v0.5.1 state
(llama.cpp mmq kernels pinned in cuda/mmq/VENDOR.md, MIT, plus the
fork's ds4_mmq C ABI, D2R kernels and repack library; fused prefill
pipeline portions carry Marco Palaferri's Portions-Copyright headers)
and wires the RAW-layout tier only: dense Q8_0 GEMMs and the IQ2_XXS
gate/up pair + Q2_K down routed-MoE pipeline at n_tok >= 2. The
aligned-SoA / D2R / producer-quantized tiers and the weight server are
deliberately not wired (they need the repack artifact pipeline).
Decode (n_tok == 1) is untouched. mmq changes FP32 reduction order vs
the cublas+dequant pipeline, so prefill logits drift at ULP scale:
gated on quality mode off, DS4_CUDA_MMQ=0 restores legacy dispatch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011CsqQzZEeuz9biUSEkGm8b
Salvatore Sanfilippo
Donato Capitella
Donato Capitella
Entrpi
