mtfyun/ds4
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: mtfyun/ds4:glm-5.3-flash
mtfyun/ds4:main mtfyun/ds4:glm-5.3-flash mtfyun/ds4:ds4f-mxfp4 mtfyun/ds4:laguna-s2.1 mtfyun/ds4:glm5.2 mtfyun/ds4:responses-api
...
pull from: mtfyun/ds4:main
mtfyun/ds4:main mtfyun/ds4:glm-5.3-flash mtfyun/ds4:ds4f-mxfp4 mtfyun/ds4:laguna-s2.1 mtfyun/ds4:glm5.2 mtfyun/ds4:responses-api
533 Commits
glm-5.3-flash ... main
Author SHA1 Message Date
antirez b3f50180ec Read half-aligned Qwen Q2 weight rows correctly on Metal 
The ordinary Q2 row dot treated packed codes as word-aligned. Metal read the wrong codes when the expert row started two bytes into a word. Read the same two words via packed bytes and preserve the accumulation order.

Compile the two production down kernels independently in the comparison test so the intervening fused reduction does not pull in unrelated dependencies. The remote M5 Max passes 19,955,180 comparisons across 1,650 fixtures, including half-aligned and address-table weights.
 2026-11-03 16:55:02 +01:00
antirez 5e3a3449e2 Check both staging widths in the Qwen compact expert dispatch test 
The half and float wrappers now forward a fourth template argument for the K tile. Exercise both widths, including negative controls for a dropped or incorrectly fixed width. The M5 Max passes all 3,200 schedule cases under ASan and UBSan.
 2026-11-03 16:52:44 +01:00
antirez cd1c5507b6 Preserve resident Qwen prefill speed with speculative decoding 
Prepare predictor history by default for SSD streaming. When weights fit in RAM, initialize historical draft rows to zero and keep target verification enabled. Full prompt history remains available with DS4_QWEN4_MTP_PREFILL=on.

With 7,809 prompt tokens and 64 generated tokens on an M5 Max, the resident default took 6.47 s versus 7.53 s with eager history. Generated text matched previous main in both modes. The same policy recovered prefill throughput on the DGX Spark.
 2026-11-03 16:50:59 +01:00
antirez 2cd0ae7064 Keep M6 Qwen tuning separate from DeepSeek and GLM kernel selection 2026-11-03 16:41:21 +01:00
antirez 5c9ed63c16 Keep generated Qwen research dumps out of the release tree 
The original experiments remain available in the contributor history. Keep the production code, regression tests and concise benchmarks in the release checkout.
 2026-11-03 16:38:47 +01:00
antirez 7bba0f7d2c Load Qwen version 2 checkpoints and rebuild missing predictor history on demand 2026-11-03 16:38:47 +01:00
antirez a2798605b7 Use resident Qwen arithmetic when experts are streamed from SSD 
Keep the resident projection, attention and shared-expert reductions at each batch size. The storage-only change now preserves full logits across 29-token prompts, appends and the sparse boundary on the M5 Max. Compare complete resident and streamed vocabularies in the model-backed generation test.
 2026-11-03 16:37:20 +01:00
antirez 0dbfd05e16 Keep Metal-only Qwen predictor kernel tests out of the CUDA binary 2026-11-03 16:32:40 +01:00
antirez afd33efab2 Keep current M5 expert tuning alongside the streamed Metal path 2026-11-03 16:29:53 +01:00
antirez c942a90fdd Restore Qwen predictor scratch views after the fused resident expert path 2026-11-03 16:28:41 +01:00
antirez 4ef9e4d0d0 Merge #1056: Giorgio Oppo’s Qwen3.8 Metal SSD expert streaming 
Integrate the streamed expert cache with the current resident IQ2 kernels, projection dispatch and asynchronous n-gram reads. Preserve the original contributor commits.
 2026-11-03 16:28:10 +01:00
Giorgio Oppo 9eb4a12f25 Merge upstream/main into qwen-kernel-opt 
Integrate antirez/main at 890898cf and resolve the shared Metal Q8 conflicts.
Keep Qwen's plain NR2/NR4 matvec and reduction arithmetic alongside upstream's
BF16, RoPE, HC and row variants. Preserve Qwen dispatch guards and prefill
unpack policy, with BF16 rounding disabled for Qwen and NR4 disabled for the
BF16 entry points. Keep upstream's concurrent-unpack and cluster2 shape guards.

Update the Q8 oracle source extraction for the dedicated plain implementation;
the independent frozen references remain unchanged.

Validated CPU/Metal builds; complete Qwen kernels and SSD expert tests;
resident/SSD arithmetic in four math/buffer modes; memory, MTP/device policies
and invalid-token staging; frontend, Engram and session/TP host tests.
Frozen Q8 oracles pass 3072 reduction cases and 1024 full-GEMV fixtures with
zero non-NaN bit differences.

V4.1 synthetic coverage passes up to its decode-topk dispatch-count assertion
at tests/test_deepseek41_metal.c:1618. The same assertion also fails on an
isolated build of unmodified upstream/main on this M1 Max. CUDA/ROCm, M5/M6
hardware and full-model inference were not tested.
 2026-11-02 21:35:23 +01:00
antirez e4709b44be Share DeepSeek V4 Metal index queries across four keys 
Adapt the query reuse idea from Tarjei Mandt and contributors in #1073 to the 64-head Flash scorer. Each SIMD group retains its original dot-product reduction. Store the rounded contribution from each head before summing them in their original order.

On M5 Max, V4 Flash 0731 Q2 steady decode improved from 39.34 to 40.27 tokens/s at 16K and 35.31 to 37.80 at 66K. All full-vocabulary logits matched on the same 96-token continuation at both lengths. Keep the one-key schedules for short rows, tensor parallelism and strict math.
 2026-11-02 11:36:48 +01:00
antirez 890898cfb4 Apply Metal weight tracking to the model being opened 
Select Mark Shank’s V4.1 read-only weight policy after validating the GGUF family. A fresh process otherwise used the V4 Flash default, and reopening the engine inherited the previous model’s setting.
 2026-11-02 11:26:14 +01:00
antirez 9fe88cd312 Integrate the per-model Metal weight policy fix 2026-11-02 11:26:14 +01:00
antirez a4bbb38ce4 Integrate the V4.1 memory-aware Metal TP fix 2026-11-02 11:17:03 +01:00
antirez c54841002d Keep V4.1 TP sharded when either Mac needs the smaller working set 2026-11-02 11:17:02 +01:00
antirez 8c54d7fb01 Merge #1073 with local Metal and Spark optimizations 2026-11-02 11:12:11 +01:00
antirez 0accf8b395 Merge #1073: speed up DeepSeek V4.1 Flash on Metal 
Integrate Tarjei Mandt (kernelpool) and contributors’ Metal decode, prefill and tensor-parallel work. Keep the existing four-row index scorer for short and non-FP4 inputs, and retain the Spark small-batch projection.
 2026-11-02 11:09:09 +01:00
antirez 6075227f1e Preserve the Spark small-batch shared-expert projection 2026-11-02 11:04:22 +01:00
antirez 224e77e791 Keep Metal-only V4.1 decode helpers out of the Spark build 2026-11-02 11:04:22 +01:00
antirez a88711246c Merge reviewed server PRs #1165 and #1245 from upstream 2026-11-01 18:07:54 +01:00
Salvatore Sanfilippo b4e3faf526 Merge #1245: allow up to 256 images per server request 
Allow up to 256 images per server request
 2026-11-01 18:07:47 +01:00
Salvatore Sanfilippo da2a102dd0 Merge #1165: advertise the loaded model modalities 
Server: advertise loaded model modalities
 2026-11-01 18:06:35 +01:00
antirez b7615c141f Keep the default CUDA unroll policy for multirow Qwen Q8 
Only the single-token Q8 loop needs a four-way unroll. Express the multirow loop without a pragma so NVCC does not warn about and ignore a zero unroll count. On GB10, all 552 executable sections and their constants and shared-memory metadata match the previous build byte for byte.
 2026-11-01 09:46:06 +01:00
antirez 36e2680676 Link ROCm fallback symbols into standalone GPU test tools 
The release QA link on Strix Halo exposed two unresolved V4.1 projection symbols from ds4_rocm.o. Add the fallback object already used by the engine to the four standalone consumers. All four now link on gfx1151 and the MXFP4, GLM attention and KDA tests pass.
 2026-11-01 09:43:04 +01:00
antirez ae40c5c4f0 Carry the complete V4.1 raw window through medium prefill pruning 
Keep all 128 terminal rows in the dependency calculation. A one-row seed was enough for the fresh benchmark but left early keys of the final decoder window different on a CUDA 4096-to-10240 append. The compressed caches matched; differences in the raw window also changed the next-token logits.

The same Spark append now matches the entire 74 MiB session and all five vocabulary vectors at 202.7 vs 154.0 tokens/s. Cover an exactly aligned 4096-token starting frontier, continued prefill, a ragged final chunk and ordinary decode in the regression.
 2026-10-31 20:43:40 +01:00
antirez c1bce729fc Cross the legacy GLM sparse boundary in one SSD layer sweep 
For full single-host compact IQ2 graphs, keep aligned dense and sparse ranges in the same layer sweep. Let the existing attention slices switch regimes and score only the suffix that consumes explicit IDs. Keep a split for partial tiles, two-row verification, TP, imatrix and quality graphs.

Full GLM 5.2/5.3 M5 Max IQ2 checkpoints with a 40 GiB cache: 4096-token 5.2 prefill reached 113 tokens/s with identical saved state and logits (the pre-change runs were 83–87). A full 5.3 1792-to-2304 append improved from 13.0 to 25.6 tokens/s, also byte-exact.
 2026-10-31 20:42:03 +01:00
antirez ef8e536501 Reuse Q2_K bit planes in Qwen Metal expert prefills 
Extend the Q4_K load-reuse approach from Matthijs van den Bos (#1198) to Q2_K. Four K steps select planes from the same 16 bytes, and eight share the same scale pair. Keep four-byte alignment and the original dequantization.

The M5 Max IQ2/Q2 prefill expert benchmark averaged 7114 to 7032 microseconds across four runs per binary. The Metal specialization fixture passed and an 8192-token Q2 prompt reproduced the complete saved session and continuation logits.
 2026-10-31 20:35:44 +01:00
antirez 524c874506 Skip wired GLM weights during full-layer SSD read-ahead 
Record successful locking of the complete static weight set. In whole-layer reads for GLM, prefetch only the three routed expert tensors when all static tensors are wired. Dense layers then need no worker. Preserve normal prefetch after partial or failed mlock.

M5 Max full GLM 5.2, 1024-token SSD prefill: the saved session and five full logit vectors were identical in four runs with a 40 GiB cache. This removes redundant pread traffic, including calls for weights that were already resident.
 2026-10-31 20:32:49 +01:00
antirez 46ade1bb3e Reuse rotated compact keys across legacy GLM prefill queries 
Prepare the live FP16 key cache once per sparse attention slice with the original RoPE helper, then share the FP32 results across queries and heads. Bound scratch at 16 MiB, retain the original path for small or unsupported launches, and fall back on allocation failure. Keep the serialized cache format unchanged.

On M5 Max the full GLM 5.3 IQ2 4096-to-6144 SSD append averaged 80.12 to 83.21 tokens/s. GLM 5.2 also preserved all frontier and continuation logits. Add a direct rollback comparison that catches differences from folding the rotation constants.
 2026-10-31 20:31:53 +01:00
antirez 1c6d058d9f Gather Qwen BF16 prompt embeddings directly on the GPU 
Reuse the idle token-routing and block-output tensors for batched BF16 gather and HC expansion. Send the token IDs in place of a host-expanded FP32 residual, preserving position handling and validation. Keep short inputs, other embedding formats, image spans and ROCm on their original paths.

Four-run 4096-token Q2/Q4 comparisons on M5 Max and GB10 matched the saved state and the prefill and four continuation logit vectors. Q2 on GB10 measured 725–734 tokens/s vs 629–680 before.
 2026-10-31 20:30:32 +01:00
antirez d9fc9cf30c Merge #1147: validate sparse selection prefixes in Metal GLM prefill 
Credit to Emilian Bold. Full eight-head groups can use the valid-row specialization up to the first padding ID when every later slot is invalid. Interior holes keep the guarded path.

On M5 Max the first four exact 8192-to-10240 GLM 5.3 Flash Q2 append runs measured 410–419 tokens/s with the prefix path and 382–385 with it disabled. The attention fixtures pass under Metal API validation.
 2026-10-31 20:30:04 +01:00
antirezandMarcello Costagliola b799c90e3a Resize independent image rows in parallel before vision prefill 
Keep every bicubic tap, its accumulation order, canvas stride and rounding. Partition large resizes across at most eight workers and finish unassigned rows synchronously if thread creation fails. Small images stay serial.

The flat padded-canvas regression comes from Marcello Costagliola (#1008). The tap-table implementation changed pixels on GB10. The row scheduler matches all 78 differential cases on M5 Max and DGX Spark, including adversarial rounding ties. A 4032x3024 to 1024x768 resize takes 29.7 vs 190.5 ms on M5 Max and 54.2 vs 227.7 ms on GB10.

Co-authored-by: Marcello Costagliola <marcello.costagliola1@gmail.com>
 2026-10-31 20:28:10 +01:00
antirez 773247a881 Skip whole decoder chunks in medium V4.1 prefills 
Adapt the aligned dependency suffix and 128-row ring warmup identified by HDYS-TTBYS in #1272 to 4096–8191-row single-host text sweeps. Preserve matrix partitions and keep TP, DSpark, multimodal, ROCm and the existing long-sweep policy on their prior routes.

At 6144 tokens the 1017 IQ2 SSD run matched the saved state and nine full logit vectors on M5 Max and GB10. Spark improved from 154.6 to 198.7 tokens/s. The default-kernel regression covers a ragged 4353-row prefill and a nonzero-start 6144-row append.
 2026-10-31 20:21:25 +01:00
antirez 8c9b1986a4 Merge #1127: overlap short callback-driven Metal Flash prefills 
Preserve emilianbold’s two-layer submission bound and report progress only after the queued layers finish. SSD streaming, tensor parallelism, visual input, profiling and large chunks keep their existing scheduling.

On M5 Max with Flash-0731 Q2, 12 warmed 256-token prefills per schedule averaged 296.09 to 300.08 tokens/s; all 3,102,720 checked logits were identical. Four 511-token fresh runs also matched the serialized cache, all logits after prefill and eight continuation steps, and all 44 progress notifications.
 2026-10-31 20:15:45 +01:00
antirez c2779bb4e8 Keep the four-head Flash indexer schedule on Metal TP 
The one-barrier schedule improves single-host decode but slowed an 8192-token Flash-0731 RDMA decode on two M5 Max machines. Reverse-order repeats gave 42.57/42.49 tok/s on the new schedule versus 43.56/42.95 on the previous schedule. Select a separate pipeline at runtime and preserve the four-head schedule for the pair.

After the guard, the final and baseline builds averaged 42.98 and 43.07 tok/s with byte-identical FP32 logits throughout all four runs. Single-host raw scores and guards still match at all five widths.
 2026-10-31 17:49:26 +01:00
antirez 32604bc6de Issue Qwen Q8 weights ahead during single-token CUDA decode 
Unroll the stock four-quant lane loop by four only for the one-row kernel. Inspired by matrixfede’s load-scheduling work in #1207. Keep the existing dot-product order and GGUF weights.

DGX Spark, a real 2048-token prefix and 96 teacher-forced steps, eight interleaved snapshot replays: warm median 21.95 to 22.20 tok/s on Q4 and 23.36 to 23.63 on Q2. All 248320 logits were identical after every token. The Qwen CUDA kernel suite and a rebuilt one-row-specialized binary passed.
 2026-10-31 17:46:11 +01:00
antirez dc9580daf4 Merge #1213: stop discarded GLM MTP drafts after their KV write 
Integrate Flor1an-B’s catch-up optimization and use a null draft destination to share normal completion and cleanup. The accepted pair still publishes its first next-token KV entry before the real draft consumes it.

On a 128 GiB M5 Max with a 40 GiB SSD expert cache, four runs of the full GLM 5.3 IQ2 checkpoint averaged 4.802 to 4.848 tok/s (+1.0%). All 80 output tokens and all 23 multi-token cycles matched. The same workload on GLM 5.2 matched its output and acceptance counts, with timing within noise. The final build also passed the streamed GLM 5.2 boundary case and GPU session-state suite.
 2026-10-31 17:38:36 +01:00
antirez babe24db9f Fill single-token GLM indexer tensor tiles with heads on Spark 
In the one-token path, put 16 of the 32 query heads in the WMMA tile that normally spans 16 tokens. Keep the FP16 conversion, eight ordered MMA steps and serial FP32 head-weight accumulation used by the batched kernel. Leave causal, quality and other GPU paths on their existing dispatch.

Sixteen FP16/FP32 cache tests compare every score to the original batched kernel, including ties, ragged key counts and an output guard. CUDA indexer regressions and Compute Sanitizer passed. Full GLM 5.3 Flash Q2 on one Spark, real 16384-token prefix and 96 decoded tokens: steady 14.94 to 15.06 tok/s (+0.8%) with bit-identical full logits in four ABBA runs.
 2026-10-31 17:33:15 +01:00
antirez db67455bed Reduce Metal Flash and Pro index scores with one barrier 
Each SIMD group retains its key and writes independent head contributions. Sum them in the original head order. On M5 Max the raw score arrays were byte-identical at five widths, including tied keys and a guarded tail. At 32768 prompt tokens on Flash-0731 Q2, ABBA runs averaged 38.81 to 39.38 steady tok/s (+1.5%); all logits matched across the 96 decode steps.
 2026-10-31 17:33:15 +01:00
antirezandtrueimage efd018cc43 Tile single-host GLM 5.3 Flash indexed attention on Metal 
Adapt trueimage’s four-phase attention kernels from f4023085 in #1090. Reuse the existing split-attention buffers for scores and normalization, and reuse the consumed low-rank query for value accumulation. Retain the established paths for TP, RoPE and unsupported cache and weight shapes.

M5 Max, full Q2 model, 2048-token prefix and 80 decode steps: four fresh runs averaged 29.91 to 33.43 steady tok/s (+11.8%). Raw logits matched at each step. The ragged, repeated-index GPU oracle also matches the generic kernel with the production scratch aliases under Metal validation.

Co-authored-by: trueimage <11846060+trueimage@users.noreply.github.com>
 2026-10-31 17:32:53 +01:00
antirez b00bbf42da Remove per-head barriers in the CUDA Flash and Pro decode indexer 
Each warp writes all its head contributions before the one block barrier. The final sum retains the original groups of four. On GB10, 32769 keys took 0.301 ms versus 0.409 ms with identical raw score bytes, including tied keys and an output guard. On Flash-0731 Q2, four fresh 32768-prompt runs averaged 18.47 to 18.65 tok/s; every logit matched at all 96 decode steps.
 2026-10-31 17:29:39 +01:00
antirez 79e59d7ac2 Merge #1270: keep legacy GLM MTP live across the sparse boundary 
Credit Luo Hanjun (Elvis-Luo). On an M5 Max with the real GLM 5.2 IQ2 checkpoint and a 40 GiB SSD expert cache, a 2046-token prompt reproduces the pre-merge verifier failure at position 2047. After the fix the next 32 greedy tokens match ordinary decoding exactly, including nine multi-token cycles. The GPU session-state tests pass.
 2026-10-31 17:26:35 +01:00
antirez 95976d737f Merge #1238: speed up Qwen3.8 Metal decode 
Integrate paradise-runner’s expert, attention, hyper-connection, and PLE scheduling work. Preserve main’s validation and chunk prefetch history. Limit the new PLE overlap to single-row Metal decode. Omit the process-lifetime router copy and speculative model-range prefetch.

On a 128 GiB M5 Max, four fresh runs with a 2048-token prompt and 80 decode steps measured 52.54 to 59.68 tok/s on Q2 (+13.6%) and 54.18 to 56.77 on Q4 (+4.8%). Every FP32 logit matched the pre-merge binary, including greedy Q4. The PLE prefetch and error-recovery fixture passed.
 2026-10-31 17:16:16 +01:00
antirez d61aa0c139 Reuse V4.1 index queries across eight keys per Spark warp 
Keep eight independent keys in each warp's registers and reuse the
query loads across them. The per-key FP32 operations, reduction tree,
mask sentinel and original score positions stay the same. Below 8192
keys the single-key warp supplies more parallelism. Multi-row and
partially visible inputs also use the original shape.

The Metal four-row scorer from HDYS-TTBYS (#1267) inspired the grouping.
On GB10, the isolated dense scorer goes from 0.37 to 0.24 ms at 65K
and 5.7 to 3.3 ms at 1M keys. On two physical Sparks over RDMA with a
real 65536-token prompt and full Q2 weights, steady ABBA runs measured
19.75/20.22/20.30/20.14 t/s (+1.6%). All 129280 logits match for all
64 steps in all four runs. One Spark at a 1048512-token synthetic cache
frontier with full streamed Flash-1017 weights went from 4.10 to 4.26
t/s (+3.9%, warm medians), with all logits identical for 32 steps.

Cover a ragged 8197-key row against the original shared reduction,
alternately retaining and rejecting the tail and checking an output
guard. The CUDA and ROCm V4.1 kernel suites pass, including CUDA memcheck.
 2026-10-31 16:27:39 +01:00
antirezandbadelon e3137ce6bc Batch disk Engram reads for Metal verifier rows 
Adapt the Engram read batching in #1240 to consecutive rows of one session. Read directly into the existing staging allocation with strided row IDs. Independent sessions and mixed batches keep their per-session reads.

M5 Max, streamed V4.1 Q2 with DSpark, 2048-token prompt and 192 generated tokens: four fresh ABBA runs averaged 14.07 to 14.28 steady tokens/s. Generated text matches, and all accepted-prefix, rewind, exact sampling and snapshot checks pass with Metal validation enabled.

Co-authored-by: badelon <211747211+badelon@users.noreply.github.com>
 2026-10-31 15:54:28 +01:00
antirez 23c333bd1c Skip rejected V4.1 index keys in Metal SSD decode 
A streaming single-token sweep owns the candidate blocks for exactly the same lifetime as a resident sweep. Enable the GPU-only scoring from #1262 here too.

On an M5 Max with Flash-1017 Q2, a 40 GiB expert budget and 131072 prompt tokens, the warm median across three paired 96-step replays goes from 16.65 to 16.99 tokens/s. All 129280 logits match bit for bit at every step.
 2026-10-31 15:52:26 +01:00
antirez c7bb2d7656 Skip masked V4.1 index scores on DGX Spark 
The layer-20 shortlist contains at most 2048 eight-key blocks. On later
reindex layers, one CUDA CTA owns exactly one of those blocks. Write the
existing rejection sentinel before loading its keys when the mask excludes
it. Admitted keys use the same warp reduction and original score slots.

On a GB10, Flash-1017 Q2 with a 40 GiB SSD expert budget, a real 65536-token
prefix and 64 teacher-forced decode steps, the warm control runs measured
12.65/12.64/12.64 t/s and the corresponding optimized runs 12.80/12.83/
12.85 t/s. All vocabulary logits matched at every step. The CUDA primitive
compares scores with a separate full score/filter and guards the final
partial block; Compute Sanitizer reported zero errors.

The corresponding Metal candidate-selection idea came from HDYS-TTBYS
in #1262. This CUDA variant consumes the existing GPU mask directly.
 2026-10-31 15:49:39 +01:00
antirez 4c29e82b3e Merge #1034: queue single-host V4.1 Metal SSD decode by default 
Bring in Dango233's bounded streaming schedule and forced-eviction
oracle. Expert-ID reads and slot replacement already order the stream.
The separate Engram inputs from #1261 remove the mid-token overwrite.
Retain the old waits for quality/imatrix and as an A/B control.

M5 Max 128 GiB, Flash-1017 Q2, 40 GiB SSD expert budget, 2048 prompt
tokens and 128 greedy output tokens, fresh processes in ABBA order:
steady control 14.65/14.51, queued 15.77/15.47 t/s (+7.1%).
The output matched byte for byte. In same-session teacher-forced replay,
all 129280 logits matched after each of 128 tokens. Metal API validation
also passed the forced-eviction oracle, including logits, history and
all live cache spans, and the public snapshot/cancellation fixture.
 2026-10-31 15:47:40 +01:00
antirez 47400ed008 Merge #1262: skip rejected V4.1 keys without changing Metal top-k ties 
Keep HDYS-TTBYS's prefix merge and alternating partial buffers, resolved
against the four-row scorer from #1267. A repeated-key oracle exposed a
difference in the PR's compact sort: equal scores produced different
original row IDs (3224 vs 3612 at rank 16). Ascending candidate indices
do not make the original bitonic sort stable.

Score layer 20's selected blocks directly on the GPU, writing scores at
their original cache indices. Reuse the usual mask and full-width top-k
so equal scores encounter the same comparison network as before. This
also removes the CPU readback, qsort, upload and remapping buffers.

M5 Max: raw scores and all ordered top-512 IDs match on a tied, ragged
32773-key fixture. The bounded resident graph matched every logit for
40 consecutive positions at 16K, 64K and 131K. Warm 131K throughput
increased from 27.39 to 28.74 t/s. The full resident checkpoint exceeds
this host's memory; the timing fixture shares one routed bank.
 2026-10-31 15:42:39 +01:00
antirez 32f4243a09 Merge #1261: pipeline resident V4.1 Metal decode submissions 
Preserve HDYS-TTBYS's four scheduling and Engram buffer commits. On an
M5 Max, a bounded resident 40-layer fixture (one shared routed-expert
bank) gave 23.04–23.08 t/s drained and 30.61–30.64 t/s queued after
warmup. All 129280 logits matched at each of 40 positions. Two- and
six-row fixtures also matched the pre-PR binary byte for byte. This is
a scheduling measurement, not throughput for the full resident model.
The real Q2 SSD session-state test and Metal kernel suite passed.
 2026-10-31 15:37:19 +01:00
Giorgio Oppo f2e60304a3 Merge upstream/main into qwen-kernel-opt 
Resolve conflicts while preserving Qwen SSD/MTP optimizations and the new
upstream PLE prefetch path. Validate tokens and repair the predictor boundary
before either input-staging path, and preserve projection phases for prefill,
decode and verification. Keep both sets of build targets and kernel tests,
adapting the raw16 fixture to the branch's M5/M6 device helper.

Validated Metal and CPU builds, Qwen kernels, SSD cache/read/fallback tests,
resident/SSD arithmetic in four math/buffer modes, memory/MTP/device policies,
PLE prefetch/cancel/fallback/recovery with the local synthetic fixture,
Metal buffer lifetimes, V4.1 indexer parity, and model-independent host tests.
The complete make test suite requires the unavailable ds4flash.gguf model;
CUDA/ROCm and M5/M6 hardware were not tested.
 2026-10-31 09:39:36 +01:00
antirez e14d7f8b1e Release cached engines between regression tests 
Close each test's cached engines before the next entry opens an engine with its own options. This prevents make test and combinations such as session-snapshot plus logprob-vectors from colliding with the runtime's single-instance lock.

Validated the formerly failing combined tests and the full make test suite on M5 Max with the release Flash 0731 GGUF and DSpark support. Legacy MTP passes separately because SSD cache-pressure tests intentionally use an incompatible engine configuration.
 2026-10-30 19:32:07 +01:00
antirez c97964b8aa Validate Qwen tokens before prefetched input staging 
Keep the existing vocabulary check ahead of embedding reads on both staging paths. Reject malformed chunks before changing live PLE history, and test invalid tokens in the first chunk, a prefetched chunk, and the partial tail with exact recovery.
 2026-10-30 18:10:33 +01:00
antirez 21178462db Test indexer row-group tails and output bounds 2026-10-30 17:57:19 +01:00
HDYS-TTBYS 345d7f44cb Score four indexer rows per SIMD group 
kernel_glm_indexer_score_one_direct spends one 128-thread threadgroup per row:
it stages the 512 B key row in threadgroup memory and closes every head group
with two threadgroup barriers. It also re-reads the whole query vector
(32 heads x 512 B = 16 KB) for every row, even though that vector is the same
for all rows of a token.

Give one SIMD group four rows instead. Each lane holds four float4 key values
in registers, and the head loop reads q4[lane] once and reuses it for all four
dots and all four simd_sum calls. No threadgroup memory and no barriers, and a
threadgroup covers 16 rows instead of 4.

On this tree (without candidate-row scoring) 1M index rows go from 7.93 ms to
2.89 ms, 68 -> 186 GB/s, 2.75x on the score pass alone. In a tree that also
carries #1262 the same kernel runs the stream at 304 GB/s instead of 143 and
the 130K-context score pass drops from 1413 to 657 us per token.

The arithmetic per row is untouched: dot(float4), simd_sum, then
max(s*scale,0)*w[head] accumulated on lane 0 in head order. The accumulate
carries #pragma clang fp contract(off) because the library builds with fast
math and the scalar kernel rounds the product through threadgroup memory.
Without it 334,121 of 1M rows differ by one fma rounding, and --dump-logits
still passes: these scores only feed a top-k, so a one-ulp change is invisible
until it flips a selection at the boundary. That is why this adds a kernel-level
memcmp test rather than relying on the logits gate.

tests/test_v41_indexer_score compares both kernels over 1M random rows,
fp4-representable keys (the cache holds fp4-dequantized values) and a
full-range f16 cache, and prints each variant's rate so the number reproduces
without a model. DS4_METAL_DISABLE_V41_SCORE_ROWS4=1 selects the scalar kernel
for A/B.

Prefill parity is exact at every frontier, the session fixture passes, and
--dump-logits plus generated text are byte-identical on and off at 32K.

Scope: the shared host entry point also serves GLM 5.2/5.3 indexer scoring
(same 32 heads x 128 dim shape), so those paths take the new kernel as well.
Bit-identity is established at kernel level including f16 caches; no GLM model
was available to run end to end.
 2026-10-30 17:56:53 +01:00
Matthijs van den Bos e73e516e75 metal: reuse Q4_K raw loads in Qwen NAX prefill 2026-10-30 17:56:53 +01:00
Matthijs van den Bos 64baaa0678 Test Qwen PLE prefetch parity and recovery 
Exercise 256-token prefill chunks and a partial tail with the real model,
checking exact logits at each frontier and after decode against rollback.
Verify cancellation/resume, background read failure, and unavailable worker
fallback with test-local injection.

Validated on Apple Metal with Qwen3.8-Flash-Next-Q4.gguf:
make -j4 tests/test_qwen4_ple_prefetch test-qwen4-ngrams
./tests/test_qwen4_ple_prefetch gguf/Qwen3.8-Flash-Next-Q4.gguf

CUDA not run.
 2026-10-30 17:56:10 +01:00
Matthijs van den Bos 347c2e1fbe Overlap Qwen PLE reads with prefill GPU work 2026-10-30 17:56:10 +01:00
Emilian Bold 82aa7e8e60 metal: use shuffle network for non-causal top-k block sort 
The non-causal kernel_argsort_f32_i32_desc used the threadgroup-memory
bitonic network; the simd_shuffle_xor variant added for DSv4.1 was only
wired to the causal path. Instantiate it for the plain kernel too.
Selected indices are bit-identical (154 shape/tie/-inf/odd-size cases).

The 1024-wide block sort is ~85% of top-k time, so whole
ds4_gpu_indexer_topk_tensor calls (M5 Max, 256 rows, wall clock incl.
~250us fixed commit/wait), main vs this change:

  8192/512     973 -> 754 us  (-22%)
  16384/1024  2024 -> 1565 us (-23%)
  32768/2048  4275 -> 3424 us (-20%)
  32768/512   3646 -> 2793 us (-23%)
 2026-10-30 17:56:04 +01:00
nomish 5382eac5e0 metal: process medium V4.1 SSD appends in one layer sweep 
e9e1baa made CUDA process medium SSD appends (2048 < N < 8192 with a
final tile of at least 256 rows) in one layer sweep that keeps the
2048-row arithmetic partitions. Enable the same rule on Metal, where
these appends still streamed the experts twice. Below 8192 rows a Metal
sweep already runs in 2048-row encoder chunks and skips the decoder
suffix, so the partitions are unchanged.

DS4_METAL_DISABLE_V41_MEDIUM_SWEEP turns it off. The dispatch test
expects the one-sweep count on Apple, and the Metal model check runs its
control session with the switch set so state_equal covers both paths.
 2026-10-30 17:55:43 +01:00
Tim Lawrenz a8ce74afd5 server: render blank reasoning as absent 
A must-echo client (DeepSeek-family) replays an assistant turn that produced
no reasoning as a single space, because the hosted API rejects an empty
string. The renderers echoed that body verbatim, so the prompt carried a token
between the think markers that the model never sampled.

KV reuse is a prefix match, so one extra token invalidates every turn after
it. Measured on a live 36k conversation: the token tier reports
first_mismatch at the space, and the text tier — which normally rescues a
token mismatch by comparing rendered text instead of ids — fails as well,
because the rendered text now differs by a byte. The request then falls back
to a 4,096-token disk checkpoint and re-prefills 32,818 tokens (3.4 min).
The same failure at 200k tokens costs 201,906 tokens of prefill (~19 min),
and 14 of 14 such turns in one day took that path.

Normalise a whitespace-only reasoning body to absent at parse time so every
renderer treats it exactly like a turn that never had reasoning. Measured
after the change with --ds41-render: " ", "" and an absent key now produce
byte-identical prompts.

Adds test_render_blank_reasoning_is_absent(), which asserts a padded turn and
a reasoning-free turn render identically.

Suite status: `./ds4_test --server` has one pre-existing failure
(test_json_int_handles_non_finite_values, json_int("NaN")) that reproduces on
5379c1c without this change and is unaffected by it.
 2026-10-30 17:52:11 +01:00
nomish 97e32768e6 metal: use exact divisions in the V4.1 FP4 activation quantizer 
kernel_dsv41_quantize computes the E4M3 block scale as amax / 6 and the E2M1
value as |x| / scale with the default fast-math division. On an M5 Max the
quotient can land one ulp past an exact E2M1 rounding midpoint, so the value
rounds up instead of to even and make test-deepseek41-metal stops at:

    quantization mode=3 index=5428: 11.25 != 7.5

(two adjacent E2M1 steps at that block scale; deterministic across runs).

Use precise::divide for those two divisions so the kernel matches the
round-to-nearest-even reference. The E2M1 division is shared with the
power-of-two scale mode, where dividing by a power of two was already exact;
the FP8 path is unchanged.
 2026-10-30 17:50:17 +01:00
antirez 381808d839 Test CPU-touched Metal buffer reclamation 
Exercise tensor contents, read and write from a C caller without an outer autorelease pool. Check physical footprint after warmup, since ordinary heap statistics miss retained shared Metal buffers.

The bounded 32-allocation regression grows by 512 MiB without the lifetime fix and passes with it. Include the check in the existing Metal command-memory target.
 2026-10-30 17:49:58 +01:00
Mark Shank a4c922c560 Fix a Metal session memory leak on /ctx 
Under ARC, -contents retains and autoreleases its buffer. The C threads
calling ds4_gpu_tensor_contents/write/read have no pool, so a freed
session kept its CPU-touched buffers alive. Every /ctx in the CLI leaked
about 0.14 GiB with GLM 5.3 Flash at 64K context, and about 1 GiB per
session with V4.1 after a 2K prefill. Numerics are unchanged.
 2026-10-30 17:48:55 +01:00
Giulio Davide Carparelli 2768d0ed3c fix(jpeg): validate SOS spectral parameters in progressive scans 
jpeg_load_mem copied Ss/Se/Ah/Al from the SOS segment unchecked. The
progressive AC decoders loop with k <= Se and index jpeg_zigzag[k], so a
scan with Se > 63 read past the 64-entry table and used the result as a
write index into the coefficient block. This was reachable from request
images on the server when --vision is enabled.

Reject progressive scans that violate T.81 G.1.1.1.1: Se > 63, Ss > Se,
DC scans with Se != 0, interleaved AC scans, and Ah/Al > 13.

Add the issue's reproducer bytes as a fixture plus an invalid DC scan
case to tests/test_image_decode.

Closes #1292
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-30 17:47:37 +01:00
Dango233 3459d5546b docs: revalidate streaming decode queue on Flash 1017 2026-10-30 18:57:29 +08:00
Dango233 fc69b626ae Merge remote-tracking branch 'origin/main' into dango-codex/queue-1017-validation 2026-10-30 15:33:45 +08:00
HDYS-TTBYS 1e6a0e9602 Decode: give batched Engram tables separate row buffers 2026-10-26 23:16:44 +09:00
paradise-runner 694620956b Qwen decode: DS4_QWEN4_NO_PLE_OVERLAP restores up-front PLE staging; name the moe down_reduce n_hc slot 2026-10-26 06:36:53 -06:00
HDYS-TTBYS 1be7b4db2a Decode: give each Engram table its own input buffer 
The queued single-token decode drained after layer 13 only because the
layer-14 Engram upload reused the buffer layer 1 was still reading. Give
table 1 a second scratch buffer and point the layer at the buffer that was
just written, so the queue runs from layer 0 to the final drain without a
mid-token wait. TP keeps its layer-13 drain: that path cannot be exercised
on this machine and retains the scheduling it was validated with.

Bit-identical: logits at 3776 tokens, 64-token greedy output, and
--decode-consistency 16 all match the previous binary exactly
(max_abs 1.63013864, rms 0.310800667 at token 16066).

ds4-bench, 64 forced tokens, alternating runs:
  4K:  gen tps 37.69 -> 38.07 (+1.0%), first token 28.95 -> 28.45 ms
  32K: gen tps 35.94 -> 36.67 (+2.0%), first token 33.4 -> 28.8 ms
Prefill unchanged.
 2026-10-26 21:14:32 +09:00
Luo Hanjun d9f2d15b40 glm: bridge MTP verification across the indexed attention boundary 2026-10-25 15:22:31 +08:00
HDYS-TTBYS 5fd33a4e53 Set the candidate-decode flag after the Engram reads 
Early returns before the layer loop left candidate_decode set, which would
re-enable the per-row candidate build in a following prefill sweep. Move the
assignment past those paths and disable candidate scoring for SSD streaming,
where the compact path's CPU/GPU round trip has not been exercised against
the expert cache.
 2026-10-24 00:00:24 +09:00
HDYS-TTBYS 9d7a00b4fb Restrict candidate-row scoring to single-token decode 
The candidate list is only valid for the token that built it, and one shared
buffer cannot describe several rows. Copied-row paths (prefill sweep
fallback, batch index, step_batch) could therefore see a stale candidate
count and score layers 24-36 against another token's rows; the prefill row
loop also paid a drain and CPU sort for every row past 16K.

Build candidates only while ds41_graph_step runs, record candidate_pos, and
require candidate_decode && candidate_pos == pos before taking the compact
path. Every other path keeps the full-index scoring it had before.
 2026-10-23 23:04:12 +09:00
HDYS-TTBYS 01772c087a Optimize V4.1 long-context decode 
Three independent changes, all verified byte-identical against the previous
path:

- Decode top-k uses the prefix-merge kernel for the 512-row selection, so
  each merge reads only a run's top 512 entries instead of the whole run.
- Layers after the block-selecting layer 20 score only the candidate rows:
  the layer-20 block ids are sorted, expanded into a 16K-row list, scored by
  a row-indexed variant of the direct score kernel, reduced with top-k over
  the compact scores, and mapped back to original row ids. Layers 2/8/14/20
  and short contexts keep the full-index path.
  DS4_V41_DISABLE_CANDIDATE_SCORING=1 restores it.
- The direct score kernel alternates two partial-sum buffers, removing one
  barrier per head group without changing the accumulation order.

ds4-bench with 64 forced tokens, M5 Ultra + 1017 IQ2: ctx 4K 37.90,
64K 35.95, 256K 33.06, 512K 30.73, 1M 27.19 t/s. The 1M drop versus 4K is
28.3%, down from 49.6%. Logits are byte-identical with candidate scoring
on/off at 32K/64K/256K, prefill-parity and session-fixture pass, and
decode-consistency reports the same max_abs/rms.
 2026-10-23 22:31:35 +09:00
HDYS-TTBYS 57df7a5bf4 Pipeline V4.1 single-host batched decode command buffers 
Commit each layer without waiting in the batched step too. SSD streaming and
TP are excluded; the layer-13 drain is unchanged. The batched step also
carries short prefill and DSpark verification, and prefill-parity plus
session-fixture pass with the new scheduling.

Session-batch oracle: same max_abs (3.8147e-06), 45.5 vs 43.5 rows/s.
ds4-server --batched-session 8: 43.5/43.7 vs 40.7/41.1 tok/s. DSpark output
unchanged. DS4_METAL_DISABLE_V41_BATCH_DECODE_PIPELINE=1 reverts.
 2026-10-23 18:24:30 +09:00
HDYS-TTBYS be2773bc6f Pipeline V4.1 single-host decode command buffers 
Commit each layer's command buffer without waiting so the GPU runs layer N
while the CPU encodes layer N+1. The required drains stay at the Engram
handoff (layer 13) and at the final layer, and streaming, imatrix and TP
paths keep their existing scheduling.

Measured on M5 Ultra with the 1017 IQ2 GGUF: ctx 4096, 256 tokens,
27.35 -> 38.10 t/s; 8K prompt with 512 tokens, 27.08 -> 37.53 t/s; ctx 64K,
25.43 -> 34.53 t/s. Logits and greedy output are byte-identical, and
session-fixture, prefill-parity, decode-consistency and MTL_DEBUG_LAYER
pass. DS4_METAL_DISABLE_V41_DECODE_PIPELINE=1 restores the per-layer wait.
 2026-10-23 18:24:30 +09:00
Tarjei Mandt 72ccf36317 Revert the V4.1 decode pre-encode: held command buffer, per-token GPU spin 2026-10-22 15:15:36 +11:00
Tarjei Mandt 51fef8784a Merge pull request #9 from IngeniousIdiocy/v41-decode-on-kp7 2026-10-22 15:04:09 +11:00
Tarjei Mandt 3650da359e Turn off the V4.1 TP concurrent shared expert with the fixed-route switch 2026-10-22 11:54:16 +11:00
Tarjei Mandt ce5b718be8 Start V4.1 short-sweep suffix pruning at 512 rows 2026-10-22 11:54:06 +11:00
Tarjei Mandt 5b83cdfe8c Merge pull request #7 from ivanfioravanti/v41-tp-decode-engram-warm 2026-10-22 11:53:51 +11:00
Mark Shank e1d8114167 Encode the next V4.1 decode token while this one finishes 
At the end of each decode step the CPU waits for the GPU to finish the last command buffer. It now uses that wait to encode the next token's first command buffer (embedding, RoPE table and layer 0) ahead of time. That buffer reads the token id from GPU memory, so it can be encoded before the token is known. The next step submits it if its inputs are unchanged (same graph and position, and no other pass over the graph in between) and throws it away otherwise. A small kernel queued behind it keeps the GPU from going idle between the two steps. Sessions with a DSpark drafter don't use it, because they draft between steps.

The kernels, their order and the command buffer boundaries are the same as without it, so the results are bit-identical. A test that needs the model file checks this across two sessions, a prompt append and a session restore.

DS4_METAL_DISABLE_V41_PREENCODE=1 turns it off.
 2026-10-21 15:50:18 -04:00
Mark Shank 4ff3af604f Speed up the V4.1 decode top-512 
V4.1 decode picks the 512 best index scores for each token. For that case (V4 Flash is unchanged), each block is sorted with SIMD shuffles, each merge of two sorted runs keeps only the best 512, the 512 winners of the radix select are sorted together in one pass, and when scoring is limited to candidate blocks, the candidate list is built once per token. The selected indices and their order are the same as before on any input without NaNs. This builds on Tarjei Mandt's radix select and candidate-block scoring (af7c02b5, 2d5ef0d3).

DS4_METAL_DISABLE_V41_DECODE_TOPK=1 uses the previous kernels.
 2026-10-21 15:50:18 -04:00
Mark Shank 64ceeea4f5 Score long V4.1 decode index rows from bf16 keys 
The indexer's keys are FP4 values times a power-of-two scale, and every kernel that writes them rounds them to bf16, so a bf16 copy holds them exactly in half the space of the f32 key cache. Those kernels now also write a bf16 copy, and once there are 8192 or more index keys, decode scores them from the copy. The scoring math and its order are unchanged, so the results are bit-identical. Restoring a saved session rebuilds the copy, and if any restored key isn't exact in bf16, scoring uses the f32 keys. This is Metal only.

DS4_METAL_DISABLE_V41_INDEX_BF16=1 skips the copy entirely.
 2026-10-21 15:50:17 -04:00
Mark Shank e99dc21751 Spread two single-core V4.1 decode reductions over the GPU 
Two reductions in V4.1 decode ran on a tiny part of the GPU: the Engram gate used one 32-thread SIMD group per stream, and the hyper-connection collapse used a single threadgroup. The gate now uses eight SIMD groups per stream, and the collapse is split across 42 threadgroups in two kernels. Each thread still adds its values in the original order, so the results are bit-identical.

DS4_METAL_DISABLE_V41_SPLIT_REDUCE=1 goes back to the single-group kernels.
 2026-10-21 15:50:13 -04:00
Mark Shank f0b4f55fcc Compute one RoPE table per V4.1 decode token 
Every layer computed the same RoPE cos/sin values for the current position, in both the attention and the indexer. Decode now computes them once per token into a small table at the start of the step, and the layers read it.

With the table, the kernel that finishes attention can also undo the RoPE rotation and round to bf16 as it writes its output, so the next projection becomes a plain Q8_0 matrix-vector product. In the indexer, applying RoPE and quantizing the query to FP4 are now one kernel instead of two. The results are bit-identical.

DS4_METAL_DISABLE_V41_ROPE_TABLE=1 computes RoPE in each layer as before.
 2026-10-21 15:49:56 -04:00
Mark Shank 55b1e75e64 Narrow more V4.1 decode weights and read them with wider loads 
The indexer's q_b weights are FP8 (E4M3) in the original checkpoint, but the GGUF widens them to F16, as it does for the Engram projection. Decode now reads an exact FP8 copy of them, built when the model loads, which halves the bytes read. If any layer can't be converted without loss, every layer keeps reading F16.

The BF16 router and FP8 matvec kernels now load 16 bytes at a time, and the FP8 kernel reads each scale once for both of its rows. They multiply and add in the same order as before, so the results are bit-identical.

The copies are allocated like V4.1's model weights, without Metal hazard tracking. This builds on Ivan Fioravanti's FP8 and BF16 weight copies.

DS4_METAL_DISABLE_V41_INDEXER_E4M3=1 keeps the F16 reads. DS4_METAL_MODEL_TRACKED=1 turns hazard tracking on for the model weights and the copies.
 2026-10-21 15:49:41 -04:00
Mark Shank b66c926046 Compile V4.1's fixed decode shapes into its decode matvecs 
Some V4.1 decode matrix-vector products always run with the same sizes: the attention q_a and kv projections, the MoE gate and up projections, the attention output projection and the FFN's hyper-connection merge. They now use kernel variants with those sizes written in as constants, so the compiler can unroll the loops and fold the index math. The results are bit-identical. This extends Ivan Fioravanti's fixed-size kernels to more calls.

DS4_METAL_DISABLE_V41_LITERAL_SHAPES=1 uses the general kernels.
 2026-10-21 15:49:18 -04:00
Mark Shank 95606b5045 Fix a data race in the V4.1 FP8 weight-copy builder 
The FP8 weight copies are built by several threads at once, and any of them can set job->failed when a block can't be converted exactly. The flag was a plain bool written and read across threads. Make its accesses atomic, and have the other threads stop early once one has failed.
 2026-10-21 15:49:03 -04:00
Mark Shank 26335c7044 Overlap V4.1 decode hyper-connection weights with the projections 
Each V4.1 block takes its input as a weighted sum of the four residual streams, with weights from the previous block, and also computes its own hyper-connection weights (the mixer projection and its Sinkhorn split), which are first needed after the block's attention or FFN output. Single-Mac decode now computes the input first and the weights alongside the next projections: in attention, inside the query projection's dispatch, which also stores the layer's KV row; in the FFN, beside the routed experts' gate and up projections. Each part does the same arithmetic as before, so the results are bit-identical.

This comes from our earlier V4.1 decode work, with the short-K Q8 row helper that its query kernel uses.

This commit also adds two things the later commits use. Each V4.1 decode option is read from the environment once at startup, and with the DS4_GPU_TEST_COUNT_KERNELS test flag a test can count how often a kernel ran, to check that the new code was actually used.

DS4_METAL_DISABLE_V41_DEFERRED_HC=1 turns it off.
 2026-10-21 15:49:00 -04:00
ivanfioravanti f5127dfc82 Allow up to 256 images per server request 
ds4-server rejected requests with more than 16 images ("too many images;
at most 16 are allowed"). Clients that keep screenshots in an agent
conversation pass that within a few turns. The 16 was only the server's
own cap and the size of two stack arrays; the engine, session
checkpoints, TP mirroring and the KV disk cache all size image state
dynamically.

SERVER_MAX_IMAGES (256) now sets the cap, the error message and both
arrays (about 22 KB of stack at most, in request threads).
 2026-10-21 19:43:58 +02:00
ivanfioravanti 5a588dbf12 Share weight loads across two rows in the exact TP K-slice matvec 
Two-Mac TP prefill on pre-M5 GPUs computes each rank's half of wo_b with
kernel_mul_mv_q8_0_f32_rows8, one threadgroup per activation row. The
new kernel_mul_mv_q8_0_f32_rows8_tok2 takes two activation rows per
threadgroup: each lane dequantizes a weight block once into registers
and runs the one-row kernel's walk for both rows, and each row then takes
the one-row reduction. The partials, and so the TP logits, are unchanged.

TP prefill, 2x M3 Ultra, Q4 MXFP4, 2K/4K/8K appends: 445/435/538 ->
475/462/574 t/s, decode logits bit-identical. Other shapes measured
(4x4, 4x2, 8x4, 8x8, 16x2, 2x8, rows grouped through the cache) were
slower or equal. DS4_METAL_DISABLE_V41_TP_KSLICE_TOK2=1 restores the
one-row kernel.
 2026-10-21 17:54:49 +02:00
ivanfioravanti 565db43559 Prune the V4.1 decoder suffix of short sweeps without changing the output 
Layer il of a sweep only needs its last 1 + (39 - il) * 127 rows. Sweeps
of 2541+ rows already skip whole 2048-row tiles of the rest; shorter
sweeps and appends ran every row through all 40 layers.

Under 8192 rows, a sweep now also prunes inside the plain schedule's
chunks while keeping each kept row's arithmetic:

- chunks before the cut are skipped, and later ones run unchanged;
- the cut chunk starts at a 128-row boundary of the sequence (the
  compressor blocks) and keeps at least 256 rows, below which other
  router and expert kernels run;
- its F32 router picks the kernel the whole chunk would use
  (ds4_gpu_matmul_f32_plan_tensor): the tiled kernel gives each row the
  same sums for any row count, the row-count switch was the only change.

Decode logits after 2048 -> 8192, 777 -> 8192, 1000 -> 8000, 4100,
6500 and 8192 -> 16384 sweeps hash identically to the previous build.
Q4 MXFP4 on M3 Ultra: 2K/4K/8K appends 406/390/484 -> 479/467/576 t/s.
DS4_METAL_DISABLE_V41_SUFFIX_ALL=1 restores whole-tile pruning only.
 2026-10-21 17:20:24 +02:00
ivanfioravanti 187f46b4fa ds4-bench: per-step decode logits hash and dump 
DS4_BENCH_DECODE_LOGITS_HASH=1 prints an FNV-1a hash of every decode
step's full logits, outside the timed region, to compare arms bit for bit;
DS4_BENCH_DECODE_LOGITS_DUMP=path also appends the raw F32 logits for KL
checks. The encoder timeline keeps longer kernel names.
 2026-10-21 16:03:25 +02:00
ivanfioravanti bdae7640ef Warm the V4.1 Engram page cache when the tables fit in memory 
Prefill reads 2 x 24 Engram rows per token, spread across 189 GiB. That
random pattern is the worst case for external SSDs: a USB drive served
about 6K rows/s over the whole span, uncached or buffered alike, so a 2K
prompt stalled about 12 s before layers 1 and 14. Q2 single-Mac prefill
ran at 205-290 t/s while its GPU work alone supported ~420.

When the resident model, graph buffers, a 64 GiB reserve and the tables
fit in physical memory (Q2 on a 512 GB Mac; not Q4), a low-priority
background reader streams the tables once into the kernel page cache at
the drive's sequential rate (~1 GB/s over USB, about 3.5 minutes). The
uncached row readers then hit memory. No table is allocated or wired, the
pages stay reclaimable, and the warmer stops at memory pressure. The same
bytes are read either way, so logits do not change.

Q2 from a USB SSD, cold prompts: 2K 198 -> 420 t/s, 8K 214 -> 518 t/s.
DS4_DISABLE_V41_ENGRAM_WARM=1 skips the warmer; DS4_V41_ENGRAM_WARM_WAIT=1
blocks until it finishes, for benchmarks.
 2026-10-21 16:03:25 +02:00
ivanfioravanti e91c453ec2 Overlap more of V4.1 two-Mac TP decode 
- A balanced TP rank, which binds every expert and runs three of the six
  routed, takes the fixed-route static MoE kernels; their per-row walks
  equal the generic ones for any slot count.
- The rank's half of the shared expert runs in the routed experts'
  concurrent encoder: its lanes' BF16 mid, then a K-slice down matvec that
  rounds its input to BF16 as it loads (kernel_mul_mv_q8_0_f32_bf16in).
- out_b's K slice likewise rounds low to BF16 on load, dropping a pass.

TP logits are bit-identical. Switches:
DS4_METAL_DISABLE_V41_TP_FIXED_ROUTE, DS4_METAL_DISABLE_V41_TP_CONCURRENT_FFN,
DS4_METAL_DISABLE_V41_TP_KSLICE_ROUND_IN.
 2026-10-21 16:03:25 +02:00
antirez ee7e840523 cuda: improve SSD-streamed prefill with ordered expert reads 
Read selected experts in file order and coalesce adjacent transfers during
batched prefill. Make V4.1 whole-next-layer prefetch opt-in, while preserving
the original order for single-token decode and cache seeding.

Keep router remapping and cache budgets unchanged. Publish cache entries only
after uploads complete, and drain partial uploads before releasing failed
reservations. Retain and document the switches for restoring the old policy.

Validate the SSD-only integration with Metal and CUDA builds, 44 CUDA cache
checks in each of three modes, outstanding-reader teardown, and exact
129280-value frontier logits for both 1017 GGUFs in paired 2K prefill runs.
The earlier two-Spark default qualification covers larger contexts, tight
caches, greedy output parity and sanitizer checks for the same loader.
 2026-10-21 09:30:33 +02:00
ivanfioravanti 0f9b03c9df Speed up exact V4.1 prefill matmuls and indexer scores on M3 
- V4.1 BF16 Q8 matmuls on M3 Ultra take the existing unpack-then-F16-GEMM
  path when the batch is wide: +0.6-0.8% prefill, bit-identical.
- kernel_dsv41_indexer_scores_mma keeps each simdgroup's K^T fragments in
  registers and reads Q and head weights from device memory, so the head
  loop runs without threadgroup memory or barriers. Products, block order
  and head sum order match the tiled F32 kernel: bit-identical scores,
  1.31 -> 0.77 s at 16K.

Together, cold prefill 32K 860 -> 892 t/s and 64K 853 -> 898 t/s.
DS4_QWEN4_Q8_PREFILL_UNPACK=0 and DS4_METAL_DISABLE_V41_INDEX_MMA=1 roll
them back.
 2026-10-21 08:10:02 +02:00
ivanfioravanti 7d9903b2ed Read narrow decode copies of the V4.1 router and Engram projection 
The GGUF widens two decode weights: the BF16-source router is stored as
F32 and the Engram projection, an E4M3 checkpoint, as F16. Decode now
reads a BF16 copy of the router and the exact E4M3 codes with their
32x32 block scales, built once at graph allocation and freed with the
engine. The E4M3 matvec maps each code to an F16 bit pattern (exactly
value / 256, subnormals included) instead of decoding it.

Logits are bit-identical. Decode at 2K on M3 Ultra: 38.5 -> 39.2 t/s;
the Engram projection drops from 849 to 658 us per token.

DS4_METAL_DISABLE_V41_ROUTER_BF16=1 and DS4_METAL_DISABLE_V41_ENGRAM_E4M3=1
restore the wide reads.
 2026-10-21 08:09:09 +02:00
paradise-runner b5aedb9b96 Qwen hyper-connections: load the norm squares and inject partials ahead of their in-order sums 
Result: {"status":"keep","speedup":1.1865,"cand_tps":60.36,"ctrl_tps":50.98,"ratio_sd":0.0024}
 2026-10-20 09:00:20 -06:00
paradise-runner f768d146e1 Qwen attention decode: two query heads per simdgroup over six simdgroups 
Result: {"status":"keep","speedup":1.1736,"cand_tps":60.07,"ctrl_tps":51.06,"ratio_sd":0.0056}
 2026-10-20 09:00:20 -06:00
paradise-runner 6605c3cefb Qwen attention merge: load the split maxima sixteen at a time 
Result: {"status":"keep","speedup":1.1637,"cand_tps":59.4,"ctrl_tps":51.03,"ratio_sd":0.0017}
 2026-10-20 09:00:20 -06:00
paradise-runner 0f12ea4577 Qwen prefetch: factor the pending-range binding 2026-10-20 09:00:20 -06:00
paradise-runner 9000fb74f9 Qwen decode: hc norm prefetches its down projection into the system cache 2026-10-20 09:00:20 -06:00
paradise-runner 6b7cb44613 Qwen decode router: read a BF16 copy of the BF16-valued F32 rows 2026-10-20 09:00:20 -06:00
paradise-runner a3a7e2de6b Qwen Q2_K row dot: two aligned word loads per lane instead of eight bytes 2026-10-20 09:00:20 -06:00
paradise-runner 4ed25e4f07 Qwen single-token MoE: fold the weighted reduce into the down kernel 2026-10-20 09:00:19 -06:00
paradise-runner b440c3c20e Qwen F32 row dot: unroll so decode GDN alpha/beta loads issue ahead 2026-10-20 09:00:19 -06:00
paradise-runner d2ae13eac0 Qwen decode: read PLE n-grams while the GPU runs layer 0; flush layer 1 separately 2026-10-20 09:00:19 -06:00
paradise-runner fd16a72f52 Specialize Qwen IQ2_XXS/Q2_K decode expert rows on M5 2026-10-20 09:00:19 -06:00
paradise-runner 47c34fb444 Qwen IQ2_XXS decode mid: stage grid/sign tables in threadgroup memory 2026-10-20 09:00:19 -06:00
Tarjei Mandt e2d09d1b05 Schedule DeepSeek V4.1 DSpark drafts with the V4 scheduler 2026-10-20 11:59:28 +11:00
Tarjei Mandt f41806dcd2 Keep the DeepSeek V4.1 prefill partition when a DSpark drafter is loaded 2026-10-20 11:59:28 +11:00
Tarjei Mandt 732af32f9c Merge remote-tracking branch 'origin/main' into ds41-perf 2026-10-20 09:57:24 +11:00
Tarjei Mandt 75408d41a6 Remove the DeepSeek V4.1 DSpark runtime before merging upstream's 2026-10-20 09:09:49 +11:00
antirez a101246338 Add verified V4.1 Flash MXFP4 downloads and resumable assembly 
Download all nine 1017 MXFP4 shards, verify their published sizes and SHA-256 hashes, and restore the original page-aligned GGUF before selecting it. Preserve the input shards and check peak disk space before downloading.

Correct the 1017 Q2 checksum and document both targets. Validation: 16 offline downloader tests, shell syntax, and exact original-header reconstruction from the actual 1017 shard metadata.
 2026-10-19 14:14:09 +02:00
antirez 5379c1c9ac Download script now defaults to new DS4.1 Flash checkpoint. 2026-10-19 12:55:24 +02:00
antirez 902baebb70 Merge PR #1135: fix Metal compressed-attention causality 2026-10-19 10:03:07 +02:00
Tarjei Mandt 31786cb594 Begin DeepSeek V4.1 TP verify blocks as two gates per layer 2026-10-18 23:39:36 +11:00
Tarjei Mandt 5958ac718c Keep a restored DeepSeek V4.1 session's drafter off the ring rows it did not write 2026-10-18 22:22:59 +11:00
Tarjei Mandt 8c2b536717 Align DeepSeek V4.1 DSpark sampling and --dspark-strict with the DSpark paper 2026-10-18 22:22:58 +11:00
Tarjei Mandt f547bf91ca Rewind DeepSeek V4.1 sessions inside the last DSpark block 2026-10-18 22:22:58 +11:00
Tarjei Mandt c25ff9a4d4 Cut DeepSeek V4.1 DSpark per-cycle overhead 2026-10-18 22:22:58 +11:00
Tarjei Mandt c91c968b52 Sample DeepSeek V4.1 tokens with position-keyed Gumbel noise 2026-10-18 22:22:58 +11:00
Tarjei Mandt a3a231ce40 Verify DeepSeek V4.1 DSpark rows on the decode attention kernels 2026-10-18 22:22:58 +11:00
Tarjei Mandt 62c6684eb2 Keep the DeepSeek V4.1 matrix rows kernels to DSpark proposals and verify 2026-10-18 22:22:58 +11:00
Tarjei Mandt 2d5ef0d3c1 Score only the candidate blocks of later DeepSeek V4.1 index layers 2026-10-18 22:22:58 +11:00
Tarjei Mandt b71e7bcb8c Decode DeepSeek V4.1 without DSpark proposals under --dspark-strict and --quality 2026-10-18 22:22:58 +11:00
Tarjei Mandt 78d5d4d7fa Keep the DeepSeek V4.1 prefill partition when a DSpark drafter is attached 2026-10-18 22:22:57 +11:00
Tarjei Mandt 0a02a3cd56 Project DeepSeek V4.1 DSpark prefill rows without half-precision inputs 2026-10-18 22:22:57 +11:00
Tarjei Mandt 15a3d710d3 End the tensor parallelism bind line 2026-10-18 22:22:57 +11:00
Tarjei Mandt cbb687462b Detach the TP model residency set from the queue only on request 2026-10-18 22:22:57 +11:00
Tarjei Mandt b90fce71f4 Keep DeepSeek V4.1 short-prefill tails under 256 rows on their own tile 2026-10-18 22:22:57 +11:00
Tarjei Mandt aa5692e985 Merge pull request #6 from ivanfioravanti/ds41-perf 2026-10-18 21:48:33 +11:00
ivanfioravanti 8f194c8158 Use the wider V4.1 short-prefill tiles under TP too 
Each wider encoder tile also crosses fewer TP partial-sum gates. On two
M3 Ultras over RDMA, 7168-token prefill improves 603 to 621 tok/s; 14336
tokens is unchanged (748 to 750). Frontier logits match the narrow-tile
control byte for byte at both sizes.
 2026-10-18 00:19:36 +02:00
ivanfioravanti 559616bb19 Merge branch 'pr4-wider-tiles' into ds41-perf 2026-10-18 00:17:19 +02:00
ivanfioravanti 724336c8e9 Give TP big-gate arrival its own grace 
Layers 1 and 14 of each V4.1 prefill block first read the Engram rows for
up to 8192 tokens from the disk table on the host. When one rank's read
outlasted the other's gate timeout plus 2 s, the peer failed the big-gate
header exchange (layer 1 gate 1) and the prefill aborted: with the default
750 ms timeout, 64K and 128K prompts failed on two M3 Ultras.

The arrival handshake now waits up to 30 s (DS4_TP_BIG_GATE_ARRIVAL_MS).
The decode timeout and the payload exchange stay strict, and a dead peer
still fails at once on the closed socket. 64K and 128K prompts complete at
767 and 724 tok/s with the default timeout.
 2026-10-18 00:09:29 +02:00
ivanfioravanti 52780d4165 Detach the model residency set from the queue under TP 
With the residency set attached to the command queue, TP prefill on two
512 GB M3 Ultras hit sporadic stalls in the V4.1 decoder suffix: a command
buffer started 2-2.6 s after commit (GPU span ~10 ms), the cost of a full
residency request for the 285 GB of model views, while the peer rank waited
at the next gate. 8K prefill swung between 130 and 460 tok/s.

The views are made resident at load, so keep the committed set but detach
it from the queue in ds4_gpu_tp_init. 8K prefill is now 697-701 tok/s with
no stalls. DS4_METAL_TP_QUEUE_RESIDENCY_SET=1 keeps the set attached.
Tested with iogpu.wired_limit_mb raised and the wired collector disabled.

Also add DS4_TP_STALL_TRACE=MS to report gates whose GPU wait or exchange
exceeds the threshold.
 2026-10-17 23:19:19 +02:00
ivanfioravanti e255ed02d8 Use the fused MXFP4 routed MoE kernels under the TP expert split 
The pair-SwiGLU mm_id kernels lacked expert ownership, so TP ranks fell
back to the unfused path along with the compact tile, map scatter, tail
culls, down half LUT and pair half-scale variants. Teach the pair kernels
to zero the mid rows of unowned experts and to index the rank-local expert
slice, as kernel_mul_mm_id does, and drop the g_tp_split_world gates.

TP logits match the previous build exactly; single-Mac logits are
unchanged. MoE time per 4096-row chunk drops from 72 to 65 ms.
 2026-10-17 23:19:19 +02:00
ivanfioravanti 7e7e595455 Merge remote-tracking branch 'upstream/main' into ds41-perf 
# Conflicts:
#	docs/SPECULATIVE_DECODING.md
#	ds4.c
#	ds4_tp.c
#	tests/test_deepseek41_metal.c
 2026-10-17 13:30:19 +02:00
LEFBE 9f365a1748 glm: skip MTP head/attention for discarded-output catch-up forwards 
The accept path calls glm_graph_mtp_step() twice: once to advance the
MTP sub-model's KV cache to the just-committed position (output
discarded into a variable never read), then again to produce the real
next draft. The first call ran the full pipeline -- Q/RoPE/indexed
attention/attn_out/ffn_norm/router/routed-experts/shared-expert/residual/
head_norm/lm_head -- for a result that was thrown away immediately.

Everything from qk_low onward only feeds *draft_out; nothing past
kv_store (which writes the only state that outlives the call, the MTP
compact KV cache at this position) is needed when the caller doesn't
read the output. Add a need_output parameter and skip straight to
end_commands/cleanup when false, matching llama.cpp's headless-graph
pruning for the equivalent GLM5-Next MTP catch-up forward.

Verified byte-exact: ds4_test --session-snapshot DS4_TEST_GLM_MTP=1
reports identical accept/reject sequence (cycles=16 single=11 double=5
tokens=21) on real GLM-5.3-Flash-Q2 before/after. Two live-server
scenarios (short deterministic list, long 400-token
prose+code+explanation) are byte-identical against the unpatched
binary with --mtp on.

Does not fix the known MTP decode regression (see memory): per-cycle
timing (--mtp-timing) shows the catch-up call this touches was already
cheap (~4ms) on the accept path; the dominant cost is the unavoidable
extra full forward pass on every cycle (reject: ~32-36ms; the verify2
batch-of-2 step itself: ~35-41ms, constant) -- both untouched by this
change since their output is genuinely needed. Kept anyway: real,
verified-correct removal of wasted GPU work at zero risk.
 2026-10-14 08:36:24 +02:00
antirez d329b3c523 Reuse the RDMA verification window for V4.1 DSpark 2026-10-10 13:40:50 +02:00
antirez f38445cf46 Use batched CUDA F16 projections in the V4.1 drafter 2026-10-10 13:15:21 +02:00
antirez 0df64b67a9 Batch V4.1 verification attention outputs on Metal and CUDA 2026-10-10 13:15:21 +02:00
antirez 97079d7dcf Enable DeepSeek V4.1 DSpark over Metal and CUDA TP 2026-10-10 13:10:32 +02:00
Giorgio Oppo 8d543ecc95 Merge antirez/main into qwen-kernel-opt 
Resolve the two ds4.c conflicts while retaining upstream CUDA TP batching
and Spark confidence defaults, the branch's M5/M6 device policy, and
Qwen SSD memory admission without reapplying generic cache limits.

Validation on Apple M1 Max, 32 GiB, in an isolated clean checkout:
- Full Metal application build and CPU application build.
- Qwen memory, MTP policy and Metal device-policy tests.
- Qwen resident arithmetic in all four math/command-buffer modes.
- Qwen kernels, exact Q2, prefill reuse, Q8 prefill variants, HC math,
  and SSD selected-expert/cache/failure-retry tests.
- DSpark confidence, window, EOS and rewind host contract tests.
- git diff --check.

CUDA/ROCm hardware, distributed transport and full-model performance
were not tested by this merge validation.
 2026-10-10 12:26:41 +02:00
antirez 792e7e6aad Include the pinned revision in DSpark artifact QA 2026-10-09 18:28:56 +02:00
antirez b290d83bc5 Skip V4.1 candidate sorting when every block is retained 
Build the identical all-allowed mask directly below the 2048-block cap. This avoids the costly arbitrary-size sort on ROCm without changing sparse attention or routing. Verify exact masks across the boundary on Metal, CUDA and ROCm.
 2026-10-09 15:35:18 +02:00
antirez 784617c8f7 Document V4.1 DSpark conversion, usage and quality checks 2026-10-09 15:22:37 +02:00
antirez 4c64ae44c1 Enable V4.1 DSpark on Metal, CUDA and ROCm 
Add single-host batched verification, exact stochastic acceptance, bounded prefix rollback and session restoration. Keep the native drafter resident alongside target SSD streaming and account for its memory. Enable the V4.1 ROCm target graph and reuse batched causal index selection. Add native-weight, runtime and official-score verification coverage. Network TP speculation remains explicitly unsupported.
 2026-10-09 15:07:43 +02:00
antirez f3ee00a1d2 Keep ROCm exact-row Q8 projections on decode arithmetic 
Do not switch explicit decode-row calls to prefill activation precision. Reuse batched Q8 activation kernels, or scalar rows in quality mode. Verified attention output boundaries and the 100-case Flash 0731 official continuation suite.
 2026-10-09 14:04:04 +02:00
antirez 5c3b0a49ea Preserve ROCm router probabilities and signed-zero sort ties 2026-10-09 13:49:12 +02:00
antirez 2dcead8ce2 Allow DSpark conversion from only the native draft shards 2026-10-09 13:18:45 +02:00
antirez 87a95e0c11 Add the V4.1 DSpark draft graph and native-weight reference tests 2026-10-09 12:51:07 +02:00
antirez da5a1272cb Convert native DeepSeek V4.1 Flash DSpark weights to GGUF 2026-10-09 12:35:04 +02:00
antirez a615eb6f07 Batch V4.1 CUDA TP attention output projections 
Reuse the exact grouped Q8 projection for short prefills and concurrent sessions while keeping causal attention row-local. Preserve BF16 boundaries and use the original per-row path for unsupported layouts. Physical five-, seven- and eight-session and mixed-batch tests retain exact logits.
 2026-10-08 13:34:02 +02:00
antirez cef4a4d906 Balance V4.1 shared experts across CUDA TP peers 
Assign the shared expert to the rank with fewer selected routed experts, with deterministic ties. Preserve the same addition grouping in short batches and reject mismatched ownership settings at TP startup. Cover graph replay, router scratch reuse, batched masking and physical mode negotiation.
 2026-10-08 13:12:38 +02:00
antirez 3920942d7a Overlap V4.1 disk n-gram reads with CUDA decode 
Use four persistent readers per graph and await each table only at its consuming layer. Keep tables disk-backed, disable Linux read-ahead, and join pending reads on every exit. Cover reader reuse, failures, cancellation by destruction and concurrent readers.
 2026-10-08 13:12:15 +02:00
antirez 68ecb1b0c4 Fuse V4.1 CUDA normalization and HC BF16 stores 
Preserve the original reductions and every BF16 rounding boundary while eliminating separate rounding launches. Keep the unfused path available for diagnostics.

Add bitwise kernel tests covering ragged widths, multiple rows, in-place normalization and output guards. Dual-Spark runs measured a repeatable roughly one percent decode gain; CUDA memory checks passed.
 2026-10-08 12:43:22 +02:00
antirez 8a522702bd Use exact Q8 matrix kernels for CUDA TP prefill slices 
Respect the parent weight row stride when computing a K slice. Reuse the existing integer matrix path for prefill-sized batches without changing scalar block reductions. Extend grouped and sliced projection tests through 129 rows and the V4.1 output dimensions.

Dual-Spark V4.1 Q2 warm measurements: 438 to 548 tokens/s at 32K, and 337 to 412 tokens/s for an 8K append. Frontier logits are byte-identical; focused bitwise tests and CUDA memcheck pass.
 2026-10-08 11:38:32 +02:00
Salvatore Sanfilippo fc80bd695d Avoid full context rebuilds at DSpark response boundaries 
Retain the latest verifier frontier and restore a bounded token prefix when a response ends inside an accepted block. Preserve valid next-token logits for sampling-mode changes, and keep the existing rebuild fallback for unavailable or stale snapshots.

Coordinate TP restores with an acknowledged state-only command before replay. Add rewind, EOS, transport, and two-turn streaming regression tests.
 2026-10-07 21:43:05 +02:00
Salvatore Sanfilippo d6b516abb4 Score API top choices independently of alternative order 
Select maximum reported logprobs, count tied maxima once, and skip unmappable maxima without substituting lower-ranked tokens. Cover ordering, ties and damaged token bytes in the quality fixture.
 2026-10-06 18:35:04 +02:00
Salvatore Sanfilippo b14bf6220f Repair DSpark and ROCm regression tests 
Compile the HIP MMVQ test separately from its object-only link and use the existing ROCm fold stub. Update the EOS contract fixture for TP proposals and split verifier heads while retaining the historical negative control.
 2026-10-06 17:48:04 +02:00
Tarjei Mandt cad79332b3 Keep the DeepSeek V4.1 matrix rows, idle prefill and DSpark checks off other models 2026-10-06 14:58:14 +11:00
Tarjei Mandt ff95f53e87 Reset the radix top-k scratch for new row counts and keep it to DeepSeek V4.1 2026-10-06 00:08:04 +11:00
antirez f0962e3757 Stop tensor-parallel inference after a transport failure 
Refuse commands and scalar, batched and bulk gates once the link is
failed. Keep failed command sends and verify-window setup/completion
failures fatal, rather than attempting replay on unsynchronized queues.

A one-sided head-row failure previously allowed later commands and
decode exchanges to consume peer drain traffic as model partial sums.
Recovery now requires a fresh connection. Healthy inference arithmetic
and protocol messages are unchanged.

Add failed-context no-I/O checks, mocked drain/post/completion failures,
and a physical mismatched-head-row regression in both link directions.
The old code fails the new unit regressions; ASAN/UBSAN, TCP/RDMA fallbacks,
peer exits, physical Q2/MXFP4 speculation and server coding checks pass.
 2026-10-05 14:17:41 +02:00
antirez b3afc8e991 Document dual-Spark inference and its release checks 
Describe Q2/MXFP4 network tensor parallelism, DSpark defaults, matching
peer settings, and measured performance with workload and timing scope.
Document native drafter conversion without changing the shipped default.

Extend future QA with physical RDMA, memory ownership, active-stream and
graph checks, official continuation scoring, context boundaries, server
coding sessions, cancellation and fallback coverage.
 2026-10-05 14:17:41 +02:00
antirez 63059b654c Optimize Spark attention and support native MXFP4 DSpark experts 
Coalesce bounded F16 projection copies, specialize Q8 vocabulary work,
tile indexer scoring and compact top-k sorts, and accelerate indexed
attention. The single-row three-pass path preserves scalar arithmetic,
including runtime reciprocal-square-root rounding and reduction order.

Fix divergent top-k compaction decisions. Admit native MXFP4 drafter
experts without changing the default support format. Keep unsupported
shapes and allocation failures on their existing fallbacks.

Strengthen expert-identity fixtures and wire the retained kernel and
history regressions into QA targets. Final single-row tests cover graph
replay, masks, tails and independent references; memory, race and sync
sanitizers pass. Official and continued-context full logits match the
preceding control. Physical Q2/MXFP4 speculation, server tool calls,
prefix reuse, cancellation and V4.1 regression checks are covered.
 2026-10-05 14:17:41 +02:00
antirez 43a91aaa3b Distribute and accelerate Flash speculative decoding on Spark 
Split the drafter and verifier vocabulary head across peers, negotiate
their state and memory requirements, and retain local drafting when the
split is unavailable. Use an ordered RDMA command mailbox for verification
and commits. Preserve accepted-prefix history through restores and skips.

Balance MXFP4 experts, stage short expert blocks, overlap independent
projections and shared experts, batch compressor and frontier copies, and
specialize measured Q8 projection shapes. Fix active-stream ordering,
paired projection row coverage and the model's indexer top-k threshold.

Add independent numerical references, graph and stream tests, transport
failure tests, and speculative session lifecycle coverage. The fused HC
path preserves the mathematical graph but changes rounding; official
continuation scores are mixed, not universally improved. A pre-existing
scalar-versus-batched book oracle discrepancy remains under investigation.
 2026-10-05 14:17:41 +02:00
antirez 77570b465c Add two-host CUDA tensor parallelism for DeepSeek V4 Flash 
Support resident Q2 and MXFP4 Flash across two CUDA peers. Split attention,
shared experts and short speculative work; reuse bounded CUDA scratch and
keep short embedding and Markov operations on the GPU.

Correct shared/routed contribution ordering and quantized vector bounds.
Keep eight-bit activations for MXFP4 matrix kernels. Align DSpark target
positions, confidence inputs and rolling history across GPU backends,
including continued prefill and skipped proposals. This extends the
reference alignment introduced by Donato Capitella in 394865c5.

Include kernel references, bounds checks, cache-history tests and physical
TP coverage. The cross-backend history changes still require physical
Metal and ROCm release QA.
 2026-10-05 14:17:41 +02:00
antirez 962ac1913d Improve V4.1 CUDA attention and Spark RDMA exchanges 
Batch causal attention selection, preserve exact FP8 Engram conversion,
and fuse shared-expert activation stages. Fix token-tile shared addressing.

Negotiate larger RDMA messages and let CUDA use registered exchange
buffers directly. Retain transport and allocation fallbacks, with focused
attention, buffer-ownership and physical-link regression tests.
 2026-10-05 14:17:41 +02:00
Tarjei Mandt 4f806387f7 Keep the DeepSeek V4.1 wide index scorer exact and off the GLM path 2026-10-05 11:21:26 +11:00
Giorgio Oppo 37f66ba3c9 Metal: restore main arithmetic for resident Qwen prefill 
Scope the legacy half/unsplit projection policy and parent-chunk attention
dispatch to SSD streaming graphs. Resident prompts again use main's FP32
small-batch F16/Q8 projections, F32 split-K reductions and per-range attention
at the dense/sparse boundary. This corrects a policy regression that changed
GDN state and subsequent greedy tokens. Decode and SSD arithmetic are retained.

Add a bounded regression test with a frozen 0aaea5a dispatch oracle, independent
attention references, negative controls, guard regions and immutable inputs
and weights. Exercise retained/unretained command buffers and fast/safe Metal
math. Require SSD mode for the existing legacy-unsplit generation reference.

Archive source provenance, replay helpers, a frozen HC shader oracle, the
resident fixture generator and full-vocabulary/greedy A/B captures. Exclude
models, executables, object files and generated source checkouts. Pending
experimental production kernel changes remain outside this commit.

Validation on Apple M1 Max, 32 GiB:
- Build ds4, ds4-server and regression tests from exactly the staged sources.
- Four Metal validation/math/command-buffer configurations pass; the old
  resident policy fails the regression test as expected.
- Resident repeated-weight Q4 fixture: all 248320 logits are bit-identical to
  main for 29, 575 and 5942-token prompts; both 64-step greedy dumps match.
- Full Q4 SSD: 29 and 575-token prompt dumps are byte-identical before/after.
- HC shader reference: 71793 comparisons, no differences.
- One-shot/session generation and SSD legacy-reference tests match all logits
  across the prompt frontier and eight continuation tokens.
- Memory planning and whitespace checks pass.

The resident fixture uses repeated real layer weights and zero PLE. Results
are arithmetic regressions, not quality or performance claims; the tester's
full Q2/M4 Max configuration still needs external confirmation.
 2026-10-03 12:02:43 +02:00
Giorgio Oppo 575369b19d Research: isolate Qwen prefill arithmetic and token drift 
Archive the investigation of PR #1056 comment 5963229415 comparing main
0aaea5a with 02313128 on M4 Max/Q2 resident. Trace the device-independent
prefill policy introduced by 2d6a207b: small F16/Q8 prompt batches change
operand precision, while GDN alpha/beta F32 projections stop using split-K.
Distinguish this policy from the numerically neutral M2-M4 tuning gate,
89986c3c attention partitions and the secondary HC arithmetic candidate.

Record ten full-vocabulary captures and four 64-step greedy comparisons
from one frozen HEAD snapshot on M1 Max/Q4/SSD, with inherited DS4 flags
removed. Restoring only F32 split-K changes the 575-token frontier by
0.41702342; restoring the complete dense policy produces identical logits
on that fixture. The 29/575 repeated HEAD controls are byte-identical.
The 5942-token tail-only split ablation changes logits by 0.33132219.

With ordinary T1 decode policy held constant, the dense-policy ablation
changes the first generated token at step 59 for Rome29 and step 20 for
book575. Preserve raw logits, top-20 logprobs, logs, token-ID fixtures,
commands, source hashes and independent provenance/numerical audits.
These fixtures do not reproduce the tester's exact M4/Q2 prompts or
attribute every component of the reported deltas; they are not quality
or performance evaluations.

Provide a preparation script that recreates 103 tracked source files from
0231312820, applies the isolated diagnostic
patch and verifies all source hashes. Add fresh snapshot/output arguments
to both runners and document serial reproduction using the local Q4 GGUF.
Keep generated snapshot copies, executables and object files outside Git.

Validation:
- Independent recalculation of all ten 248320-logit dumps and greedy
  common prefixes, selected token IDs and bytes passed.
- Preparation in a fresh temporary directory reproduced all 103 source
  hashes and applied only the reviewed diagnostic patch.
- Python AST/JSON parsing and both runner help commands passed.
- git diff --cached --check passed.

This commit archives research only; pending production optimization
experiments remain outside its scope. No production dispatch is changed.
 2026-10-03 09:21:17 +02:00
Donato Capitella 67f2ca912c Server: advertise loaded model modalities 2026-10-01 10:44:24 +01:00
Giorgio Oppo 0231312820 Metal: extend Qwen M1 Max tuning to M2, M3 and M4 
Share the established M1 Max dispatch policy with the M2-M4 families
without changing kernel arithmetic, supported tensor shapes, math-mode
guards or diagnostic overrides.

- Centralize device eligibility with generation-bounded name matching.
  Preserve M1 base/Pro/Ultra and M5/M6 defaults, and retain M3 Ultra's
  dedicated MoE specialization, geometry and prefill tile policy.
- Enable wide Q8 decode row reuse, HC gate/mix reuse, short MTP Q8
  projections and the existing IQ2/Q2/MXFP4 expert paths on M2-M4.
- Extend small HC prefill tiles, large Q8 SSD prefill, sparse IQ2/Q2
  SSD tiles, Q4/MXFP4 K32 tiles, large-batch half staging and Q4 decode
  scratch release under the same existing shape and batch guards.
- Update SSD dispatch probes, cover 47 device-name cases and document
  the supported families and the limits of current hardware evidence.

Validation: build all application targets and run the device-policy,
Qwen kernel, safe-math reuse/projection, MoE specialization and SSD
expert tests (retained and unretained command buffers) on an isolated
HEAD export containing only this commit. Local hardware is M1 Max;
M2/M3/M4 throughput still requires validation on those devices.
 2026-10-01 08:04:59 +02:00
Giorgio Oppo f32a16a561 Research: archive Qwen Q4 SSD scheduling and decode experiments 
Preserve the September 30 radical optimization prototypes and their measured
limits without enabling a new production path.

- Archive frozen MoE shaders and a standalone Metal oracle for compact tile
  scheduling with heavily used experts first. The paired screens measure
  +8.51% and +10.07% MoE throughput on captured layer 9/47 routing, with
  unchanged FP32 outputs. The projected whole-prefill benefit is only about
  2% and has not been demonstrated end to end.
- Include raw timing samples, exactness results, source hashes, and a lossless
  495 KB routing fixture containing indices only. Document direct rebuild
  commands and the limitations of the historical preparation script.
- Record twelve plain/MTP model runs with matching output bytes. MTP improves
  Roma decode by 12.59% versus the ordinary CLI; same-session comparisons
  show 19.44% on Roma and 24.24% on code. Preserve the extra memory/I/O costs
  and distinguish these results from untested architecture proposals.
- Document elastic prefill-to-decode workspace, inter-layer SSD prefetch,
  prompt-lookup drafting, and the rejected lossless weight-compression screen.
- Bundle both prompts and retain raw stdout bytes and frozen source hashes
  without line-ending or whitespace normalization.

Validation for this archive: standalone Objective-C oracle compiles; Python
syntax and JSON parse; frozen source SHA-256 values match; routing gzip,
record hashes and expert/pair membership validate; copied Roma prompt matches.
The GPU and model results are the recorded September 30 runs, not rerun here.

Earlier local production bundles, requantization work, executables and model
weights are outside this commit. Recorded model source/binary hashes identify
the working tree used; this archive alone does not reconstruct that binary.
 2026-10-01 05:18:07 +02:00
Emilian Bold 7172b249a8 Recover Metal GLM prefill speed with validated selection prefixes 
Scan selected IDs on the GPU before using the valid-row specialization. Trim only invalid tail slots, retain guarded handling for interior holes, and preserve the physical selection stride and attention order.

Enable the fast path for full head groups outside TP, with a diagnostic rollback. Add CPU-reference and bitwise rollback tests for padded, interior-invalid, empty and offset selections.

On M5 Max with resident GLM-5.3-Flash-Q2, main-based interleaved continued-prefill measurements improve 7.2% at 16K and 7.1% at 24K, with bit-exact full logits. Addresses the dispatch regression identified in #1029 without reverting invalid-row masking.
 2026-09-29 03:40:50 +03:00
Giorgio Oppo b96a12be7b Metal: optimize Qwen Q4 SSD prefill on M1 Max 
Add K32 NT4 expert matrix kernels for Q4_K/MXFP4 SSD prefill.
Reuse Q4 headers across eight staging steps and reduce threadgroup
storage while retaining the original sequence of FP32 MMA operations.
Keep the existing K64 entry points for other devices, resident inference
and other token tile sizes.

Use half activation shadows for the Q4/MXFP4 pair at 8192+ tokens,
preserving the public float intermediate. Enable the existing Q8 unpack
plus F16 GEMM path for large Qwen SSD batches on M1 Max. Retain scratch
allocations through unretained command buffers and fall back to direct
Q8 if optional allocation fails. Release prefill-only Q4 scratch at the
existing synchronized transition to row kernels, without adding a sync.

Extend validation with a frozen ee79f22 Metal oracle, exact Q4/MXFP4
float/half comparisons, SSD dispatch probes, failed-read retries,
scratch growth and reuse, prefill/decode/prefill transitions, and Q8
scratch lifetime checks in both command-buffer ownership modes.

Validation:
- Forced full build; no compiler warnings.
- Q4 oracle in default and safe math: 331,727,600 exact float values and
  121,446,912 exact half values per mode; IQ2/Q2 oracle also passes.
- SSD selected-expert and Q8 prefill variant suites pass.
- Resident K64 oracle remains exact and shows no observed slowdown.

Full-model results and diagnostic methodology are recorded in
speed-bench/qwen38-m1-q4-k32.{md,json}. All compared outputs are identical;
expert-cache capacity and requested SSD bytes are unchanged. The report
retains the earlier negative decode results and the subsequent controlled
comparison, rather than treating the exploratory gain as guaranteed.

Eight controlled ABBA+BAAB runs on M1 Max/32 GiB, same executable and
shader, toggling only the new prefill defaults in a diagnostic copy:
- Prefill: 154.31 -> 161.49 token/s (+4.65%).
- Decode: 4.0656 -> 4.0569 token/s (-0.21%), within observed variation.
- Later 50-forward decode window: 226.931 -> 221.873 ms/token (+2.28%).
The earlier -8.06% decode result did not reproduce in this control. Scratch
release is a verified memory reduction, not an established explanation
for that earlier result. No diagnostic rollback flag enters production.
 2026-09-27 17:23:11 +02:00
Giorgio Oppo ee79f22b90 Metal: accelerate Qwen MXFP4 prefill and batched down rows 
Decode each MXFP4 prefill half-block with one exponent read and four
byte-aligned packed loads. Keep the original LUT multiplication and final
half/float conversion, without changing accumulation or tile geometry.

Reuse activations across two MXFP4 down rows for M1 Max multi-token NR2
dispatches, preserving each row's block, FMA and reduction order. Use the
existing paired Q8 helper for shared experts and guard odd output tails.
Specialize this through function constant 907 and include it in the
pipeline cache key, compiling the extra register footprint out of T1.
Retain the existing NR1/NR4, other-device and safe-math fallback policies.

Add a standalone staging oracle against two unchanged stage8 decoders,
covering all exponent bytes, nibble values and byte alignments with both
storage types and math modes. Extend exact down tests with odd row counts,
block tails, T1/T2/T3, shared Q8/MXFP4 slots, NR1/2/4 and output guards.
Register the oracle build/test/clean targets and document measured gains.

Validation on Apple M1 Max:
- Build all binaries and the affected Metal test targets successfully.
- Staging oracle: 1,048,576 comparisons, zero NaN payload differences.
- Exact MV tests pass with DS4_METAL_MATH_SAFE=0 and 1.
- MoE MM specialization: 132 passing cases per math mode.
- SSD experts: 94 passing cases per math mode, including overlap,
  selected-only reads, cache reuse, failure/retry and fallback paths.
- Original/current intermediate output fingerprints match in both modes.
- Eight full Q4 SSD runs, including MTP, produce identical output within
  each workload; git diff --check passes.

Against 46425263, alternating resident prefill measurements reduce median
kernel time by 5.2-6.1%. Same-process two-token down measurements improve
throughput by 14.7-17.2%, with bit-identical results and no T1 GPU regression.
Full SSD prefill improves from 22.94 to 23.41 t/s in the measured workload;
median MTP generation stays at 7.65 t/s. These local results do not imply
the same gains for complete inference or other GPUs.
 2026-09-27 10:24:37 +02:00
Giorgio Oppo 4642526317 Metal: optimize Qwen Q4 decode and verify SSD parity 
Reduce Q4_K gate/up memory traffic by sharing Q8 activation reads across
the shared expert's two projections. Extract the existing IQ2 helper for
reuse, and widen routed activation and scale loads with packed float4 and
half2 operations. Preserve the scalar quantized-code loads and the original
per-lane FP32 accumulation and SIMD reduction order.

Enable MXFP4 down prefetch automatically on M1 Max for the measured
640-to-2560 projection. Preserve existing M5/M6 fast-math policies and the
diagnostic disable switch. Track the compiled library's math mode so
strict math keeps the plain noncontracted kernel even when prefetch is
explicitly requested; the prefetched kernel's explicit FMA otherwise
introduces a one-bit difference in that mode.

Strengthen tests with a plain-kernel oracle independent of automatic
prefetch, explicit/default dispatch checks, and Q4 SSD coverage at the
32/33 and 128-token boundaries. Compare addressed and masked Q4 decode
against the generic resident kernel across cache reuse, fallback and retry.
Add MXFP4 down microbenchmarks and document the full-model measurements.
Keep the existing prefill and row-dispatch policies: the experimental
smaller Q2 sparse-prefill tiles did not show a stable Q4 benefit.

Validation on Apple M1 Max 32 GiB:
- make -j2 all and the affected Metal test targets; git diff --check.
- Q4 ordered geometry and MoE decode exactness with MATH_SAFE=0 and 1.
- IQ2 shared-projection regression: 180 fixtures, 1,119,600 comparisons.
- MoE prefill specialization: 132 passing cases per math mode.
- SSD expert regression: 94 passing cases per math mode, including
  missing-only reads, overlap, output guards, failure and exact retry.
- Q4 SSD, ctx=32768, chunk=1024, temp=0, nothink, 100 generated tokens:
  three alternating pairs improve median decode from 6.07 to 6.42 t/s
  (+5.8%), with identical output in every run. Prefill medians are 6.94
  and 6.91 t/s. SSD/system variation remains substantial; these are local
  measurements and do not establish gains on other devices.
- Q4 MTP with two draft tokens: before/after 64-token output identical.
 2026-09-26 15:35:37 +02:00
Giorgio Oppo 81ee4badb8 Qwen: honor Metal's SSD streaming memory budget 
Avoid discounting Metal's recommended working set twice when sizing the
Qwen SSD expert cache. Use the smaller of 87.5% of physical RAM and the
reported recommendation, then retain the existing graph, staging and
2 GiB runtime reservations.

Let Qwen's complete admission calculation handle both automatic and
explicit cache budgets. Keep admitted static weights eligible for mlock
instead of applying the generic cache cap again after admission. Other
model families and backends retain their existing checks.

Add coverage for host- and Metal-limited budgets, unavailable capacities
and overflow-safe arithmetic, and document the resulting memory policy.

Validation:
- make test-qwen4-memory
- Qwen Q4 SSD runs on M1 Max 32 GiB: 6.32 GiB static weights locked,
  11.58 GiB expert cache, 2.59 GiB staging and 22.96 GiB planned total
  at ctx=32768 and prefill_chunk=1024.
 2026-09-26 15:35:21 +02:00
Sebastian Christiansen e5b19e280e Fix compressed-key causality in Metal visual attention 
Intersect compressed-key selection with each query's causal frontier while preserving raw image bidirectionality. Add an analytic regression covering selection patterns, compression boundaries, FP16/FP32 keys, and tensor view offsets.

Validated on M5 Max 128 GiB with the Vision-Exp IQ2XXS model: baseline fails the new regression; patched Metal kernels, 60 image generations with exact replay, resident/SSD session checks, long-context recall, tool-call quality, frontend tests, and CPU/Metal builds pass. Strict image grading remains 10/12 per arm due to screenshot wording; existing SSD output differences are unchanged. Balanced text throughput changed +0.0033% prefill and -0.2598% generation. Official Flash-0731 vectors, CUDA runtime, and distributed execution were not validated.
 2026-09-26 15:11:33 +02:00
Giorgio Oppo 089d966069 Qwen: make MTP prompt preparation configurable 
Resident short-context workloads can pay for preparing every historical
predictor row without recovering that cost during decode. Add an explicit
DS4_QWEN4_MTP_PREFILL policy so that this tradeoff can be controlled while
MTP draft generation and target verification remain enabled:

- on (default, also 1): prepare predictor history for all deployments.
- off (also 0): skip historical projections and initialize those rows to zero.
- auto: prepare history only when using SSD streaming.

Keep the existing default until resident acceptance and total wall time
are measured. A cold predictor prefix changes draft context and can reduce
acceptance; it is not equivalent to fully prepared history. Reject invalid
settings before mutating the Qwen session, and ignore the setting when MTP
is disabled.

Initialize skipped K/V and raw indexer rows deterministically, synchronize
before Metal host writes, and recompute completed pooled indexer blocks to
preserve real keys from an existing partial block. Defer each cold chunk's
last row to normal boundary preparation so uninterrupted prefill, append,
and checkpoint restore share the same predictor state. Apply the policy
only to new prompt rows; retain reused and restored cache contents.

Document the modes, acceptance tradeoff and checkpoint behavior. Add 64
model-free policy cases and extend the Metal prefill suite with poisoned
caches, reuse, invalid settings, append, mixed real/zero pooled blocks,
snapshots, cancellation/resume, target-logit comparisons and active MTP
including forced depth-3 verification. Keep the full-history oracle on.

Validation:
- Built ds4, ds4-server and the Qwen MTP prefill test.
- All 64 model-free policy cases passed; git diff --check passed.
- Short Metal model suite passed on M1 Max with SSD streaming and the local
  Qwen Q2-AProjQ4 GGUF, including greedy and exact-sampling continuations.
  The model test used a temporary source copy with existing Q4_K attention
  loader support and generic GEMV from the separate Q4 branch; those Q4
  compatibility changes are not included here.
- Resident 3.5k performance and CUDA execution were not measured; no
  throughput or acceptance improvement is claimed.
 2026-09-26 13:59:29 +02:00
Emilian Bold c71ca56d81 Overlap bounded Metal prefill layer submissions 2026-09-25 19:23:59 +03:00
Giorgio Oppo b1af94b438 Merge antirez/main into qwen-kernel-opt 2026-09-23 09:40:35 +02:00
Giorgio Oppo 028b43f3a1 Metal: stabilize Qwen HC arithmetic and extend M5 tuning to M6 2026-09-23 09:36:00 +02:00
Giorgio Oppo 47ca2eb5d9 Metal: enable Qwen SSD prefill overlap on all devices 2026-09-23 09:17:35 +02:00
Tarjei Mandt 3d3c83bbbd Merge remote-tracking branch 'origin/main' into ds41-perf 
# Conflicts:
#	ds4.c
#	ds4_engram.c
 2026-09-21 08:48:08 +10:00
Tarjei Mandt 798c64f444 Round a short DeepSeek V4.1 sweep down only for a token-major tail 2026-09-21 08:14:17 +10:00
ivanfioravanti 6bf43ee8da Amortize V4.1 short prefill with wider encoder tiles 
Port the short encoder tile selection from 4f9a2e0 onto KernelPool ds41-perf a15028e. Limit it to non-draft, single-node pre-M5 Metal inference and preserve decoder partitions.

Four alternating-order Q4 trials on M3 Ultra: 7956-token initial prefill improves 572.31 to 592.74 t/s (+3.57%); a 15894-token continuation improves 694.63 to 700.44 t/s (+0.84%). Exact live state, history and logits match in resident and SSD modes, including following decode. Add reproducible graph checks.

Builds and CPU syntax check pass. make test reproduces the same 14 V4 fixture assertions seen on previous baseline main; seven remaining recipe checks pass separately. CUDA hardware, distributed inference and DSpark were not rerun.
 2026-09-20 13:23:48 +02:00
Tarjei Mandt 92197579f8 Keep DeepSeek V4.1 CUDA row batches on the per-row decode kernels 2026-09-20 21:00:59 +10:00
Tarjei Mandt 2a55193649 Guard the DeepSeek V4.1 Metal-only helpers for CUDA builds 2026-09-20 21:00:59 +10:00
Tarjei Mandt 64240fb9c2 Prefill DeepSeek V4.1 prompts under 8192 tokens on the plain schedule 2026-09-20 21:00:59 +10:00
Salvatore Sanfilippo 0aaea5a238 Document physical Metal TP QA preflight and native batch coverage 
Check per-boot RDMA addressing and wired residency limits, match prefill chunk overrides on both ranks, and account for the extra sessions used by numerical oracles. Require at least three Metal TP rows when checking the native V4.1 path; two rows intentionally use ordered decoding.
 2026-09-20 03:21:24 +02:00
Salvatore Sanfilippo 6687c6f90b Guard ROCm memory baseline out of CPU-only builds 
The CPU-only placement tests also define DS4_ROCM_BUILD, but never use the GPU memory baseline. Exclude its declaration there, matching the existing guards around its uses.

Verified warning-free ROCm applications and test builds on Strix Halo, with placement, memory, session, server and agent tests passing. The production ds4.o is byte-identical after stripping debug metadata.
 2026-09-20 03:12:38 +02:00
GiorgioOppo a6ad6361ca test(qwen): add sparse and long-context MTP replay diagnostics 
Add an optional 17408-token synthetic fixture to the real-model Metal
MTP prefill regression. Run one trunk graph with SSD streaming and probe
actual residual rows at the first sparse query and near the final prefix.
Save only predictor state to keep the test practical on a 32 GiB M1 Max.

Replay cache-only, optimized last-row and recursive predictor paths
against the frozen full-layer reference using identical saved history,
trunk rows and lookahead IDs. Cover T1/T2/T3 and exact T3 verification;
poison destination cache rows before replay and compare complete logits,
EH projections, residuals and cache publication. Restore predictor state,
RoPE positions, verification policy and the device target head afterward.

Provide bounded chunk/prefix options and optional token, tensor and JSONL
dumps for comparisons across builds. Keep the original short regression.
Document usage, the actual 8192-row threshold of the M1 half-operand path,
measured results and limits of the acceptance investigation.

Validation on M1 Max, 32 GiB, Qwen3.8-Flash-Next-Q2, SSD cache 1024:
- Short MTP regression, memory tests and production build pass.
- The long fixture passes at chunks 128/2048 on 11811b9 and at chunk
  2048 on 04c0867 and 89986c3, with Metal API validation.
- At chunk 2048 all 36 captures from 89986c3 and 11811b9 are byte-identical.
  Differences from 04c0867 already occur in the corrected trunk inputs.
- Between current chunks 128/2048, captured trunk rows and final target
  logits match exactly. Maximum predictor delta is 0.0001266003; the
  16 full/recursive proposals retain the same argmax.
- A separate same-row grouping diagnostic finds pre-existing HC projection
  rounding differences, with identical stage reports on 04c0867/11811b9.
- The independent half-kernel oracle passes 31 fixtures including 8192
  rows under API validation. A 24-fixture NT1/2/4 subset passes shader
  validation; the NT8 mid stress case exceeds instrumented M1 shared
  memory capacity (36864 bytes versus 32768).
- git diff --check passes. No production arithmetic or dispatch changes.

These synthetic checks do not reproduce the reviewer's acceptance counts
without the original prompt. Full-model chunk-8192 acceptance is not
covered by these runs. Temporary diagnostics and captures are excluded.

Refs: https://github.com/antirez/ds4/pull/1056#issuecomment-5744306216
 2026-09-19 21:19:31 +02:00
GiorgioOppo fd0c24af54 test(qwen): repair compact MoE checks after half-kernel extraction 
The compact-dispatch regression still expected expert mapping in the
float gate/up entry point after that logic moved to the shared float/half
implementation. Follow the shared helper and validate both wrappers,
including template arguments, grid coordinates and shadow-buffer wiring.
Check that expert bounds precede counts reads and cover every half-down
NT instantiation. Ignore comments when inspecting shader source.

Add 29 negative controls for broken mappings, missing or late guards,
incorrect wrapper arguments and missing half-down variants. Extend the
CPU streaming stub with the half-pass association used by the extracted
production guards; verify invalidation on empty passes and preservation
on nonempty passes with and without active streaming state.

Validation: make test-qwen4-moe-mm-compact passes with ASan/UBSan across
1033 mapping cases, 3200 schedules, 961600 matching events, 3 pointer
lifetime cases, 12 NAX empty-state cases and 4 streamed half-state cases.
This is host/dispatch coverage; production kernels are unchanged.

Refs: https://github.com/antirez/ds4/pull/1056#issuecomment-5744306216
 2026-09-19 21:19:15 +02:00
Salvatore Sanfilippo 06f7135d0f Keep Metal-only Qwen batch tests out of the CUDA suite 
Guard native cross-session attention, grouped expert, and Q8 batch GEMM tests with __APPLE__. CUDA retains its supported scalar, prefill, and recurrent kernel oracles instead of failing to link against Metal-only entry points.
 2026-09-19 20:23:37 +02:00
Tarjei Mandt a15028ee40 Round DeepSeek V4.1 vectors with the shared bf16 helper 2026-09-20 01:51:24 +10:00
Tarjei Mandt 8be1005089 Fold the one-row DeepSeek V4.1 entry points into their rows form 2026-09-20 01:51:24 +10:00
Emilian Bold 0719a0baf8 Improve Metal router softplus accuracy for small logits 
Use a four-term log1p series for exp(x) below 1/32 in all four DeepSeek router transform sites. Clamp the polynomial input so its unselected branch stays finite, and compare Metal probabilities against the shared accurate reference without widening the tolerance.

Refs antirez/ds4#1039. Validated on M5 Max with a fresh default build, test_deepseek41_metal, test_deepseek41_graph --vision-routing, and full Metal test runs with DS4_METAL_DISABLE_PRE_M5_ROUTER_TRANSFORM_FINALIZE_FUSION=1 and DS4_METAL_MATH_SAFE=1. No model required. Pre-M5 validation and model-level speed/SSD checks remain outstanding.

(cherry picked from commit 7f5f1b77ae)
 2026-09-19 17:00:45 +02:00
Salvatore Sanfilippo d9bc45dbda Add fresh official V4.1 Flash continuations and router quality comparisons 2026-09-19 17:00:38 +02:00
Tarjei Mandt 0ac42d6fe0 Tidy the DeepSeek V4.1 comments and switches 2026-09-20 00:50:13 +10:00
Tarjei Mandt 8387625991 Drop the DeepSeek V4.1 tensor-parallel gate trace 2026-09-20 00:50:13 +10:00
Tarjei Mandt 5432223d60 Keep the resident Q4_K expert tiles to the DeepSeek V4.1 shape 2026-09-20 00:50:13 +10:00
Tarjei Mandt 7d89c59c4d Pick the DeepSeek V4.1 index kernel under test through the GPU test flags 2026-09-20 00:50:13 +10:00
Tarjei Mandt c6f4086f6d Fold the DeepSeek V4.1 RoPE quantize test into the Metal test 2026-09-20 00:50:13 +10:00
Tarjei Mandt ed43e1a56b Shorten the DeepSeek V4.1 CUDA stubs 2026-09-20 00:50:13 +10:00
Tarjei Mandt c4b66b9032 Remove the unused DeepSeek V4.1 split HC weighted sum 2026-09-20 00:50:13 +10:00
Tarjei Mandt 66dff86949 Remove the DeepSeek V4.1 exact rows attention path 2026-09-20 00:50:12 +10:00
Tarjei Mandt 258ad331dc Drop the embedded DSpark stages from the DeepSeek V4.1 engine 2026-09-20 00:50:12 +10:00
Tarjei Mandt 4fe6a78029 Guard the DeepSeek V4.1 tensor-parallel flag folds for non-Apple builds 2026-09-20 00:50:12 +10:00
GiorgioOppo 11811b96b2 perf(metal): reuse half operands for large Qwen SSD MoE prefill 
Add half-storage input and intermediate variants of the existing
simdgroup MoE gate/up/activation and down kernels. Preserve FP32 matrix
accumulation, activation arithmetic and the public float intermediate;
move the existing float-to-half operand conversions to a shared input
copy and a half shadow written by the mid epilogue.

Enable the path automatically only for M1 Max SSD streaming with
IQ2_XXS gate/up, Q2_K down and chunks of at least 8192 tokens. Keep the
original path for smaller chunks: a 2048-token experiment did not show
a reliable full-model benefit despite a faster isolated kernel pair.
No production opt-in flag is added.

Scope the half shadow to a single streaming invocation and validate the
producer tensors and dimensions before consuming it. Invalidate state
on entry/failure, separate resident and missing passes, retain scratch
buffers across submitted commands and reconvert reused input contents.
Allocate both optional buffers before encoding and fall back to float
storage on allocation failure. Account explicitly for scratch storage:
140 MiB at the tested 8192-token model shape, already covered by the
existing conservative workspace estimate.

Add a standalone frozen-source oracle/benchmark with generic weight
function constants, NT1/2/4/8, padded Q2_K rows, sparse routing, guarded
buffers and poisoned outputs. Check both FP32 outputs and the actual
half payloads against CPU binary16 conversion. Add Make targets and
extend SSD tests across the 8191/8192 policy boundary, scratch growth to
16384 and reuse, split resident/missing passes, failed reads and retries.

Validation on Apple M1 Max with Qwen3.8-Flash-Next-Q2:
- Build, full Metal kernel suite, CPU memory tests and SSD tests passed.
- Standalone oracle: 31 fixtures, 1015326720 float comparisons and
  100325376 half comparisons, all exact against the independent reference.
- Generic MoE pair at 8192 tokens: 322.620 -> 294.798 ms (+9.44%
  throughput), including conversion and dual writes; nine ABBA/BAAB rounds.
- Complete model, SSD cache 1024 experts, context 16384, MTP off,
  8192-token prompt plus 128 decode evaluations, four runs per arm:
  prefill 181.993 -> 198.252 t/s (+8.93% aggregate; +5.84% comparing
  per-run medians). All four prefill pairs improve; a slow baseline run
  raises the aggregate. A stable 10% overall gain is not demonstrated.
- Decode at chunk 8192 is inconclusive (+0.34% aggregate).
- All timed/validation replays preserve complete vocabulary logits bit
  for bit. Final Metal-debug replays separately validate the retained
  2048 fallback and 8192 optimized path, each comparing 32033280 logits.

The recorded 2048 optimized-path experiment is not enabled by this
commit. Only the automatic threshold/comment changed after measurement;
the retained 8192 execution path is unchanged. Other Apple chips keep
the previous dispatch and are unbenchmarked; CUDA/ROCm are not modified.
 2026-09-19 11:04:08 +02:00
GiorgioOppo 9bff1ca4e1 perf(metal): reuse Qwen decode activations across four Q8 output rows 
Add a four-row specialization of the existing Q8_0 matvec, preserving its
K traversal, eight-term dot product, SIMD-group count and reduction tree.
Select it for single-row Qwen projections on M1 Max with NSG=4, outside
TP2, for the measured 2560->6144/10240/12288 and 6144->2560 shapes.
Complete four-row tiles avoid out-of-bounds weight reads; vocabulary,
other shapes and other devices retain their existing dispatch.

Extend the frozen e37f185 oracle to compare two- and four-row variants
under default/safe math, NSG variants, offset views, guarded outputs and
explicitly padded tail weights. Add a public-dispatch regression covering
the four model shapes plus non-four-row and NSG fallback cases.

Validation on Apple M1 Max:
- 1024 frozen-reference fixtures, 571136 values, zero differing bits.
- Full Qwen Metal kernel suite and public-dispatch checks passed with
  Metal API validation enabled.
- Nine alternating GPU benchmark rounds with 32 dispatches per sample
  show median paired kernel throughput gains of 11.6-28.7%.
- Combined candidate/full-model replay retains bit-exact logits through
  8192 prompt tokens and 128 decode evaluations.

The isolated kernel speedup is not a claim of equivalent SSD decode
throughput: the combined 8192-chunk experiment measures only +0.34%
aggregate decode throughput, within substantial run-to-run variability.
 2026-09-19 11:03:41 +02:00
Tarjei Mandt b3d4a267c0 Open only the DSpark shards for a drafter-only DeepSeek V4.1 conversion 2026-09-19 09:46:34 +10:00
Tarjei Mandt c9bfdcc951 Release DeepSeek V4.1 tensor-parallel decode gates inside the command buffer 2026-09-19 09:46:34 +10:00
Salvatore Sanfilippo aafc65b444 Bound loopback packet size in the tiny-buffer TCP regression 
Negotiate 512-byte TCP segments on Linux before connecting. Otherwise shrinking loopback sockets to 1 KiB receive buffers leaves 32 KiB segments, making zero-window probing dominate large test transfers and sometimes trip the idle timeout. Retain the payload sizes, bidirectional backpressure, descriptor-mode checks and timeout/EOF cases. Linux and macOS transport tests pass; production transport is unchanged.
 2026-09-18 23:27:51 +02:00
Salvatore Sanfilippo 077a25760e Keep decode-sized n-gram concurrency on Apple dispatch workers 
Retain the measured Metal win without imposing per-token pthread creation on CUDA and ROCm. Tables remain disk-only. Warm M5 Max SSD decode improved from 12.62 to 13.39 tokens/s with byte-identical output; row and error tests also pass under ASAN/UBSAN.
 2026-09-18 23:27:51 +02:00
Jake Maness 6c00e2d535 Let V4.1 single-token decode use the concurrent Engram reader 
ds4_engram_read_batch() already fans row reads across ENGRAM_READERS
workers, with the comment "Fixed concurrency hides random-read latency
without caching the table" -- but only above a 256-row threshold.

ds41_graph_step() reads DS4_ENGRAM_COLS rows per table (48 rows for the
two tables) through the serial ds4_engram_read(), so ordinary decode
never reached that path. Those 48 preads run in series, and they run
before ds4_gpu_begin_commands(), so the GPU is idle for all of them.
Only bulk prefill ever cleared 256.

This routes the decode read through the batched entry point and lowers
the threshold to 8 so decode-sized batches engage the existing workers.
Same rows, same order, same values -- only the fetch is concurrent.

Measured on a Mac Studio M3 Ultra (80 GPU cores, 256 GB, macOS 26.6.2),
DeepSeek-V4.1-Flash-Q2, resident (no --ssd-streaming), --ctx 131072,
temperature 0, 200 generated tokens, this branch vs its merge-base:

  main            19.18, 19.14 tok/s                    median 19.16
  main + patch    20.42, 21.00, 21.11, 21.18 tok/s      median 21.06

  +9.9%, about 4.7 ms/token. Generated text was byte-identical between
  the two builds. (The 20.42 was a cold first run after rebuild.)

The underlying cost is storage latency, not bandwidth -- a standalone
pread benchmark against the same GGUF, 48 random 264-byte rows with
F_NOCACHE and readahead disabled, 150 batches:

  serial    5.177 ms/batch  (107.9 us/row)
  parallel  0.614 ms/batch  ( 12.8 us/row)

The saving is roughly constant per token, so it is worth more on a
faster tree: on ivanfioravanti's ds41f-optimizations branch, where this
was originally profiled, the same change measured 26.8 -> 30.5 tok/s
(Engram wall 5.55 -> 1.12 ms/token) over three alternating pairs, also
byte-identical.

Credit where it is due:

  - ivanfioravanti, whose V4.1 optimization work is where this was
    profiled. His ds41f-dspark branch already carries a better version
    of this fix, ds41_engram_parallel(), which reads both tables in a
    single dispatch -- but gated to speculative sessions, so ordinary
    decode still falls through to the serial loop there too. Ungating
    that is probably the better long-term shape of this change.

  - The oMLX / drowzeys V4.1-on-Mac work, which is what pointed at
    Engram as a decode bottleneck at all. Their approach was a RAM
    cache of hot rows; measured row reuse here was only 22.5% over a
    600-token coding generation, worth roughly +1 tok/s, so the win
    turned out to be concurrency rather than caching. Wrong idea,
    right place to look.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 92a61612a2)
 2026-09-18 23:27:51 +02:00
Salvatore Sanfilippo 5994f062de Validate image positions before token-prefix slot reuse 
Keep fingerprint-only matching for authenticated visible histories that rebase spans, but require exact historical image positions for token and rewind reuse. Reproduce the old false hit with a synthetic checkpoint. Fix the live eviction oracle to include all competing slots. Four-slot Qwen and GLM vision/tool/eviction and concurrent-request tests pass. Follow-up to #765.
 2026-09-18 23:27:51 +02:00
antirez 4067320f06 Merge remote-tracking branch 'pr-review/765' 2026-09-18 23:27:47 +02:00
Salvatore Sanfilippo 25334fef16 Test numeric reasoning strings at the API boundary 
Cover 0 through 100, malformed values, overflow-sized input and unchanged named controls. Follow-up to #1074.
 2026-09-18 23:27:31 +02:00
GiorgioOppo 89986c3c89 fix(qwen): retain prefill attention arithmetic across Metal partitions 
A Qwen prefill crossing the 512-block sparse-attention budget leaves three
internal dense rows at absolute positions 2048 through 2050. With chunk 128
or chunk 2048, these rows previously selected the short-tail decode kernel
because their subrange contained fewer than nine rows. A single-pass
prefill selected matrix attention for the same rows. The different
rounding introduced drift despite identical input tokens and queries.

Carry the parent prefill policy through both dense and sparse attention
subranges. Add a Metal prefill entry point backed by the existing shared
implementation so internal short partitions retain matrix attention when
the parent prefill has more than eight rows. Preserve the existing shape
guards and scalar diagnostic override. Genuine short prefill tails,
decode, speculative verification and non-Apple dispatch retain their
previous policies; shader arithmetic and cache storage are unchanged.

Add regression coverage comparing a whole 2051-row attention reference
with the three-row dense partition, and sparse partitions of one through
eight rows. Require bit-identical outputs, bounded tensor views and intact
output guards. Include the previous scalar dispatch as a negative control
and verify that the diagnostic override still selects its arithmetic.

Validation on Apple M1 Max, 32 GiB, Qwen3.8-Flash-Next-Q2, SSD streaming:
- Production and affected test executables build successfully.
- Complete Qwen Metal kernel suite passes with MTL_DEBUG_LAYER=1.
- Memory/workspace, generation and MTP prefill suites pass. Generation
  includes a genuine three-token tail and the legacy prefill reference;
  MTP checks use the suite's existing numerical tolerances.
- For a 5760-token prompt and 128 teacher-forced decode steps, requested
  chunks 128, 2048 and 8192 each match the original single-pass reference
  bit for bit at all 129 full-vocabulary frontiers: 96,099,840 logit-pair
  comparisons, with MTP disabled and a 1024-expert cache.
- The original chunked prefill had maximum absolute logit error 0.597853661
  against that single pass; the candidate has zero on this fixture.
- Per-layer traces locate the first divergence at layer 3 attention output
  for token 2048, after identical inputs and queries. Traced final captures
  reproduce the uninstrumented captures exactly.
- git diff --check passes.

This targets internal attention partitions rather than promising equality
for arbitrary chunk sizes: genuine small tails and very short sparse
indexer scoring retain their existing policies. The single-pass comparison
is a numerical consistency check, not a model-quality or throughput claim.

Refs: https://github.com/antirez/ds4/pull/1056#issuecomment-5731630396
 2026-09-18 17:38:43 +02:00
Tarjei Mandt af7c02b594 Select DeepSeek V4.1 index candidates with a radix select on Metal 2026-09-18 23:32:47 +10:00
Tarjei Mandt 4b1b1519ed Score DeepSeek V4.1 index keys with one thread per key on Metal 2026-09-18 23:32:47 +10:00
GiorgioOppo 04c08672b5 fix(metal): preserve Q8 dot arithmetic to eliminate late decode drift 
The paired char2 quant loop changed Q8 GEMV results under Metal's
standard fast-math compilation despite preserving the apparent source
accumulation order. Relative to e37f185, the Qwen long-prompt replay
amplified these differences to a maximum logit error of 0.513016701 at
decode step 65, with another increase at step 106.

Restore the original unrolled eight-term qs[i] dot loop while keeping
packed_float4 activation loads and the optimized two-stage reduction.
Buffer layouts, dispatch geometry, quantization formats, and cache policy
remain unchanged. HC and MoE ablations reproduced the unfixed outputs
exactly; isolating the paired quant loop removed the drift. CUDA and ROCm
are unchanged.

Add a complete Metal Q8 GEMV regression oracle frozen independently from
e37f185. Exercise the dot product, scale application, K traversal,
reduction, output tails, scalar-aligned views, varied scales, outliers,
cancellation, and signed zeros. Verify input immutability, output guards,
and independent CPU dot samples. Cover NSG 1/2/4/8 with default and safe
math. Add make targets for execution and compilation-only checks, plus
binary cleanup and ignore rules.

Validation on Apple M1 Max:
- Full Q8 oracle: 480 fixtures and 35,712 outputs match reference bits.
- Negative control using the parent shader: 1,549 output bit differences,
  confirming that the new regression test detects the original defect.
- Existing Q8 reduction oracle: 3,072 fixtures / 4,096 output rows pass.
- Complete Qwen kernel suite, production build, and diff checks pass.
- Three full-vocabulary Qwen replays with 29, 575, and 5,760 prompt tokens
  and 128 teacher-forced decode steps each match e37f185 byte for byte:
  387 frontiers / 96,099,840 logits, including the former late spikes.
  Configuration: Qwen3.8-Flash-Next-Q2, Metal SSD streaming, 1,024 cached
  experts, context 8,192, chunks 128/2,048, MTP off, API validation on.

These results establish parity for the tested M1 Max fixtures and model
trajectories. Kernel timing was variable; no end-to-end throughput claim
is made. Temporary reports and captures are not included in this commit.
 2026-09-18 15:29:17 +02:00
GiorgioOppo 2d6a207bbb fix(qwen): preserve reference prefill arithmetic with explicit projection phases 
Short prompt chunks and tails were selecting kernels intended for independent
decode batches. Float-operand F16/Q8 tiles and split-K FP32 reductions changed
prefill arithmetic and introduced substantial logit drift against e37f185,
including different results for the same prompt at different chunk sizes.

Track prefill/decode projection policy explicitly on each Qwen graph:
- Route contiguous prompt chunks, including single-token tails, through the
  reference projection paths and an unsplit Metal dense GEMM entry point.
- Preserve existing optimized projection dispatch for ordinary decode, native
  session batches, MTP verification, predictor cache updates and draft chains.
- Scope and restore the phase on success and failure; mark copied batch row
  graphs as decode rather than inferring phase from row count or all_rows.
- Keep CUDA dispatch and the separate vision encoder unchanged. The new CUDA
  prefill entry point delegates to the existing dense function.

Add --prefill-reference to the real-model generation regression test. Compare
normal one-shot/session dispatch against legacy unsplit direct prefill, then
restore automatic decode on both sides. Cover 29-token short prompts,
575-token prompts with a 63-token tail, and two complete 128-token chunks.
Reject inherited reference overrides that could hide a dispatch regression.
Update the private MTP oracle calls for the graph-aware projection helpers.

Validation on Apple M1 Max, 32 GiB, Metal SSD streaming:
- ds4/server and relevant test builds pass; memory planner, Metal kernel,
  generation/reference and MTP cache/continuation suites pass.
- All 27 generation/reference vocabulary frontiers are bit-exact. The same
  test compiled against original 0024ca0 fails at the first frontier, as
  expected, confirming that it detects the prefill regression.
- Five configurations compared with e37f185 cover 85 frontiers and 21,107,200
  FP32 logits, using identical prompts and 16 teacher-forced continuation IDs.
  Worst prefill error falls from 0.48730135 to 1.90734863e-6; worst decode
  error falls from 1.37674665 to 0.000828742981. No argmax disagreements.
  Residual numerical differences remain; this is not bit-exact parity with
  e37f185 or a model-quality evaluation.
- Patched 5,760-token captures are bit-exact across chunk sizes 128 and 2048
  at all 17 frontiers, including the subsequent decode steps.
- Sixteen alternating SSD timing runs against 0024ca0 (four per variant per
  workload) show median prefill changes of -0.71% at 575/chunk128 and +2.54%
  at 5760/chunk2048. Decode medians change +1.59% and +8.12%, respectively;
  short 16-step samples and overlapping ranges do not establish a general
  speedup or isolate decode kernel performance.

CUDA/ROCm GPU execution and resident-mode validation were not performed.
Generated reports, binary captures and temporary logs are excluded.
 2026-09-18 14:15:14 +02:00
GiorgioOppo 0024ca0803 fix(qwen): unify prefill chunk resolution and test generation parity 
An explicit prefill chunk at or above the context size previously fell back
to DS4_QWEN4_PREFILL_CHUNK or the default in one-shot and session allocation,
while the memory estimator clamped it to the context. This could change the
actual prefill boundaries and make execution disagree with memory admission.

Use one resolver for one-shot generation, session allocation, memory
estimation and SSD admission. Explicit requests always take precedence over
the environment and clamp to the context size. Preserve existing defaults
when no explicit chunk is supplied.

Add CPU coverage for chunk precedence and context-boundary cases. Add a
real-model Metal regression test that checks public one-shot token emission
and progress against sessions, then compares every vocabulary logit between
an independently replayed direct graph and the session through eight decode
steps. Cover conflicting environment settings, oversized requests and long
prompts with configurable token counts; bound the SSD expert cache.

Document the test and the upstream 8db1d1d one-shot behavior that ignores the
CLI prefill chunk. Cross-revision comparisons must also set the same
DS4_QWEN4_PREFILL_CHUNK value to align effective chunk boundaries.

Validation on Apple M1 Max, 32 GiB:
- Built ds4, ds4-server and both affected tests without warnings.
- CPU memory/chunk-resolution tests passed; git diff --check passed.
- Five Qwen Q2 SSD cases passed with Metal API validation, including 5,760
  prompt tokens at chunks 128 and 2,048 and an oversized explicit request.
- All public greedy tokens matched; 45 complete vocabulary frontiers were
  bit-exact between direct replay and sessions (11,174,400 float comparisons).
- The oversized-request regression test fails against f47db4f with the
  observed first frontier 32 rather than the expected 247, and passes here.

These checks establish same-build parity at identical chunk boundaries.
They do not establish main-versus-branch resident parity on the tester's
64 GiB machine, or token invariance across different chunk sizes.
 2026-09-18 13:22:40 +02:00
Alessandro Checco ecf7a93148 Enhance reasoning effort parsing logic 
With this fix the server with respond to e.g.
"reasoning_effort": "50"

so it becomes compatible with pi.dev that doens't accept numerical value (and cannot send think_level by default)

(cherry picked from commit de418c0861)
 2026-09-18 09:54:45 +02:00
Spencer Gilbert 63bbba4ba7 Advertise the loaded GLM model id in /v1/models 
(cherry picked from commit 28c06c88e0)
 2026-09-18 09:54:41 +02:00
Mark Shank 064bc229c6 Fix Metal object header dependencies 
(cherry picked from commit a32a030e1b)
 2026-09-18 09:54:38 +02:00
Salvatore Sanfilippo 62867e913a Make JPEG regression failure diagnostics safe 
Always compute the decoded fingerprint before checking dimensions, so the error path cannot print an uninitialized buffer. Follow-up to #981.
 2026-09-18 09:54:34 +02:00
korale77 ac43e900d8 Fix progressive JPEG AC refinement ZRL handling in Iris decoder 
jpeg_prog_decode_ac_refine treated ZRL (run=15, size=0) as run=16 and
reused the skip loop, which kept consuming correction bits for non-zero
coefficients after the 16th zero-history coefficient. Those bits belong
after the next Huffman symbol (ITU T.81 G.1.2.3, libjpeg
decode_mcu_AC_refine), so the bitstream desynchronised: many progressive
JPEGs, including plain `cjpeg -progressive` output, failed to load or
decoded with wrong pixels.

Skip `run` zeros while refining non-zeros on the way, write the new value
(0 for ZRL) at the next zero, and stop, matching libjpeg's structure.

Add tests/test_image_decode with two cjpeg-generated fixtures that fail
on the old decoder (wrong pixels for grayscale, NULL for 4:2:0) and are
bit-exact with djpeg after the fix. Wire it into `make test`.

(cherry picked from commit 50ed0a4c73)
 2026-09-18 09:53:23 +02:00
Salvatore Sanfilippo 39bc628cc3 Cover split Qwen reasoning markers in both streaming APIs 
Verify tool-enabled answers are emitted before generation finishes for OpenAI and Anthropic clients. Keep the existing DeepSeek repeated-reasoning regressions. Follow-up to #1077.
 2026-09-18 09:52:20 +02:00
Jeff d31089d5ad Fix Qwen tool content streaming 
(cherry picked from commit 77a39ebf0ea81ff2f7767af1bf9631890215482e)
 2026-09-18 09:50:23 +02:00
ayuan1129 fff391e6c7 GLM: render the tool call block inline, as the chat template does 
append_glm_tool_calls_text() wrapped the reconstructed block in a newline on
each side.  GLM's tokenizer.chat_template does not: it emits

  <|assistant|><think></think><tool_call>name<arg_key>k</arg_key>...</tool_call>

with no surrounding whitespace in any variant (empty content, non-empty
content, reasoning present, followed by an observation or by a user turn).
The block is only wrapped when it has to be rebuilt from parsed arguments;
when the raw sampled bytes are available they are emitted verbatim, so the two
paths disagreed.

That disagreement is invisible while a conversation stays in one process: the
live tool turn feeds the raw sampled bytes into the prompt, so the running KV
prefix still lines up.  It surfaces when a client replays the history, because
the replay goes through the reconstruction path and picks up the extra
newlines.  Any key or checkpoint derived from the visible transcript then fails
to match the replay: in the restart smoke test a conversation that should
re-prefill ~14 tokens re-prefilled 3800 of 11992, because the visible-key
checkpoint it wrote was three bytes off.

Cold prefill is affected too, and independently of any cache: a replayed GLM
tool conversation was being rendered in a shape the model never trained on.

Drop the two newlines.  test_glm_tool_calls_render_matches_template asserts the
template-true block and that the raw and reconstructed renders are byte
identical; test_render_glm_preserves_reasoning_with_tools had pinned the old
bytes and is corrected.

Reproduced with the restart-restore smoke test on GLM-5.3-Flash-Q2:
  before: FAIL, restored 8192 of 11992 tokens (3800 re-prefilled)
  after:  OK,   restored 11998 of 12012 tokens (14 re-prefilled)

(cherry picked from commit d186421d96)
 2026-09-18 09:49:08 +02:00
Salvatore Sanfilippo 9775e26ccd Test GLM pooled decode padding against an independent reference 
Cover all pool tail lengths after 2048 selected rows. Repeat the split Metal decode with padded IDs and compare its value projection to a double-precision attention reference. Follow-up to #1051.
 2026-09-18 09:48:57 +02:00
Mark Shank 582d809fd3 Fix GLM-5.3 Metal decode masking for padded selections 
Track the pooled selection’s guaranteed prefix instead of promising that all selected rows are valid. Padded selections use the existing bounds-checked kernel.

Adapted from 101d20c23f3a27456bdc61e0da35e2bea97eb04e.

Validation: make -j1 and git diff --check passed on macOS M3 Ultra (Metal build). No model was loaded; GPU runtime and performance tests were not rerun.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 4e166b1fb4)
 2026-09-18 09:45:20 +02:00
Tarjei Mandt fd7091916a Score DeepSeek V4.1 verify rows with the tiled indexer kernel 2026-09-18 12:53:34 +10:00
Tarjei Mandt 1c6920666c Stop proposing DeepSeek V4.1 drafts while they lose to plain decoding 2026-09-18 12:53:34 +10:00
Tarjei Mandt 0b2d6e28cb Write only the DeepSeek V4.1 text tensors from the converter 2026-09-18 12:53:34 +10:00
Mikhail Korobov 0edbd9cd64 Merge upstream/main: vision live-state accounting in the reuse probe 
Upstream 81a7658 ("Preserve live vision context across server tool calls")
rewrote the live-continuation helpers this branch replaced with
slot_probe_reuse(): image markers carry request-local nonces and are
normalized for visible replay, image tokens are spliced into continuations
instead of being tokenized as text, and image appends are validated by
fingerprint and row count rather than by exact span equality.  The merge
keeps the probe architecture and folds the new conditions into it, so slot
routing and the execution ladder still cannot disagree:

- The vision gate is fingerprint-prefix (new
  ds4_session_vision_fingerprint_prefix_matches): historical images must
  match by row count and fingerprint, positions may shift because
  build_live_prompt_suffix() rebases them, and appended images are allowed.
  An exact-state gate rejected valid appends, and a position-sensitive one
  rejected replays that omit hidden reasoning.
- Every live tier materializes through build_live_prompt_suffix(), so new
  images are spliced into the continuation and a failed rebase is a safe
  miss rather than a malformed prompt.
- responses-visible and thinking-visible compare in visible_prompt_key
  space, so a replay with fresh marker nonces still binds while a changed or
  shifted image does not.
- Tool-output-only continuations keep upstream's contract: if the selected
  live tier cannot be materialized, the request is answered with 409
  (live_continuation_unavailable) instead of being generated from a
  context-free prompt built from tool outputs alone.
- Single-session mode refreshes the slot's rendered-text cache like the
  batched worker does, so the memory-text tier is probe-authoritative on
  both worker paths; the redundant ladder fallback and the four superseded
  upstream helpers are removed.

Tests: test_slot_probe_vision_tiers covers image appends, pixel mismatches
and marker-nonce visible replay at the probe/routing level;
test_live_continuation_contract pins the 409 predicate;
tests/test_session_state.c gains fingerprint-prefix cases; and the new live
suite tests/test_server_vision_live_state.py checks slot routing, eviction
victims and the tool-result-only 409 against a real vision model.

Verified on DeepSeek V4 Flash Vision Experimental (M5 Max): ./ds4_test
--server, ds4_agent_test, test-session-state, live test_server_vision_cache.py
(1- and 4-session), test_server_story.py, test_server_batching.py and the new
vision live-state suite all pass.
 2026-09-18 04:07:30 +05:00
Tarjei Mandt aea1a0bae9 Document DeepSeek V4.1 DSpark support files, two-rank drafting and idle-slot prefill 2026-09-17 14:27:09 +10:00
Tarjei Mandt 5c2dac7e37 End the tensor parallelism bind line 2026-09-17 14:27:09 +10:00
Tarjei Mandt 29ce271717 Give DeepSeek V4.1 prefill rows their own second Engram table 2026-09-17 14:27:09 +10:00
Tarjei Mandt b7bcccdea2 Stop DeepSeek V4.1 drafts before the end-of-sequence token 2026-09-17 12:13:32 +10:00
Tarjei Mandt 7da9535f2d Sync idle server slots in the engine's prefill chunk 2026-09-17 12:13:32 +10:00
Tarjei Mandt bd85ba6aca Keep the row kernels on the exact DeepSeek V4.1 rows path 2026-09-17 11:20:10 +10:00
Tarjei Mandt 1e3bcf8d87 Release DeepSeek V4.1 verify-block gates through the spin kernel 2026-09-17 10:38:31 +10:00
Tarjei Mandt c674b198d4 Verify DeepSeek V4.1 DSpark blocks across two tensor-parallel nodes 2026-09-17 10:38:30 +10:00
Tarjei Mandt a6b50ff6c4 Multiply the DeepSeek V4.1 shared expert and attention output rows through simdgroup matrices 2026-09-17 08:54:56 +10:00
Tarjei Mandt 262b4a66e3 Multiply DeepSeek V4.1 verify rows through simdgroup matrices 2026-09-17 08:54:56 +10:00
Tarjei Mandt e49643fff9 Run the DeepSeek V4.1 shared expert beside the routed experts on Metal 2026-09-17 08:54:56 +10:00
Tarjei Mandt 867465cc66 Fold the DeepSeek V4.1 tensor-parallel gate sums into the hyper-connection expansion 2026-09-17 08:54:56 +10:00
Tarjei Mandt a13b513e63 Split the DeepSeek V4.1 shared expert across the tensor-parallel ranks 2026-09-17 08:54:56 +10:00
Tarjei Mandt 8d73a0effa Balance the DeepSeek V4.1 tensor-parallel decode experts by list position 2026-09-17 08:54:56 +10:00
Tarjei Mandt a3f6f313a6 Port the decode-control regression test and fix its router batch call 2026-09-17 08:54:55 +10:00
Tarjei Mandt 1923131d4c Match the rope contraction of the DeepSeek V4.1 fused rope-quantize store 2026-09-17 08:54:55 +10:00
Tarjei Mandt dc81436da2 Bind the owned expert range once when a DeepSeek V4.1 batch falls back to one-token MoE 2026-09-17 08:54:55 +10:00
Tarjei Mandt 9f3892d2fe Write DeepSeek V4.1 tensor-parallel partials into the slab slot and batch the kslice rows 2026-09-17 08:54:55 +10:00
Tarjei Mandt 4fbbc4a45b Encode the DeepSeek V4.1 vocabulary head before the last decode drain and start the wide sweep at 3072 tokens 2026-09-17 08:54:55 +10:00
Tarjei Mandt d8d1523b33 Speed up DeepSeek V4.1 Metal decode and run DSpark verify rows on its decode kernels 2026-09-17 08:54:55 +10:00
Tarjei Mandt a60edc61f5 Write the DeepSeek V4.1 DSpark stages to a separate support GGUF 2026-09-17 08:54:55 +10:00
Tarjei Mandt c016ea7730 Keep the released MXFP4 experts in the DeepSeek V4.1 converter 2026-09-17 08:54:55 +10:00
Tarjei Mandt 3b7f8f2242 Select every key for short DeepSeek V4.1 index rows 2026-09-17 08:54:55 +10:00
Tarjei Mandt d744ebb9e7 Run Q4_K DeepSeek V4.1 experts on the compact resident tiles 2026-09-17 08:54:55 +10:00
Tarjei Mandt ce5a812fd3 Select DeepSeek V4.1 candidate blocks in batches 2026-09-17 08:54:55 +10:00
Tarjei Mandt ccbf2c083e Run the DeepSeek V4.1 decoder suffix from 2541 tokens 2026-09-17 08:54:55 +10:00
Tarjei Mandt e768396179 Skip DeepSeek V4.1 candidate selection while every block is kept 2026-09-17 08:54:55 +10:00
Tarjei Mandt 10118742ae Publish DeepSeek V4.1 ratio-1 keys in batches on Metal 2026-09-17 08:54:54 +10:00
Tarjei Mandt edceb7ab7f Select DeepSeek V4.1 experts in one dispatch 2026-09-17 08:54:54 +10:00
Tarjei Mandt d95f8b6101 Round DeepSeek V4.1 activations inside the producing kernels 2026-09-17 08:54:54 +10:00
Tarjei Mandt 2a281b0802 Queue DeepSeek V4.1 decode layers on a single node 2026-09-17 08:54:54 +10:00
GiorgioOppo f47db4fb83 Merge antirez/main at 8db1d1d into qwen-kernel-opt 
Integrate upstream Qwen CUDA support, Metal multi-session decode and
speculative batching, shared prefill workspaces and recurrent-state pools,
grouped resident MoE kernels, dense/Q8 batch projections, and server state
handling. Retain the Qwen SSD-streaming and Metal kernel optimizations from
this branch.

Resolve the host/shader MoE argument layout by retaining the compact SSD
slot masks and appending the upstream grouped-list fields. Update layout
assertions and direct gate/up and down regression fixtures to the resulting
72-byte ABI. Keep SSD address resources and in-flight buffer lifetimes in
the common dispatch path alongside the upstream resident row cache.

Keep SSD sessions on the ordered ordinary/speculative fallback with private
streaming workspaces. Resident sessions use the new upstream shared arena.
Preserve the existing SSD prefill tile threshold and guard Metal-only SSD,
cache-only attention preparation, and MTP projection calls from CUDA builds.

Reconcile retained MTP tail rows and snapshot ownership with native batching:
prepare pending predictor boundaries, copy per-session tails and rejection
snapshots, maintain tail metadata after commit, and complete the historical
cache row after an accepted two-token verification when next-draft creation
is skipped. Preserve checkpoint payload compatibility within this branch.

Retain both upstream and branch-specific build/test targets. Extend the SSD
failure/recovery test to reject native resident dispatch explicitly and
verify exact logits across ordered ordinary and speculative batches. Clarify
resident versus SSD batching in the Qwen and server documentation.

Validation on Apple M1 Max, 32 GiB:
- Build ds4, ds4-server, and all selected test binaries without warnings.
- Pass Qwen memory planning and compact-MoE CPU tests with ASan/UBSan.
- Pass the complete Qwen kernel suite with Metal API validation, including
  upstream grouped MoE, batch projections, and attention paths.
- Pass direct MoE mid (180 fixtures, 1,119,600 comparisons), MoE down
  (1,650 fixtures, 19,955,180 comparisons), HC (92 fixtures), and Q8
  reduction (3,072 cases) regression suites.
- Pass SSD expert cache/read/retry tests and MoE specialization tests.
- Pass real Qwen Q2 SSD n-gram failure/recovery, ordered batch parity, MTP
  prefill, snapshot/payload replay, and continuation checks.
- Pass a 50-token greedy SSD generation for 'narrami la storia di roma'.

CUDA execution and full resident-model multi-session integration were not
run on this 32 GiB Mac. This merge does not make new performance claims.
 2026-09-16 19:51:07 +02:00
Salvatore Sanfilippo 8db1d1d155 Document Qwen Metal batching and its quality checks 2026-09-16 15:17:26 +02:00
Salvatore Sanfilippo ab38bc6b7a Serve Qwen batched MTP with safe session state and output limits 
Keep exact sampled requests on ordinary batches. Clear stale verification snapshots, honor logical context limits, and reset speculative timing estimates when batch width changes. Exercise forced rejection, mixed decode, snapshot restore, disk failures and arena replacement.
 2026-09-16 15:17:25 +02:00
Salvatore Sanfilippo d1620ba061 Reject incomplete concurrency benchmarks and oversized requests 2026-09-16 15:02:09 +02:00
Salvatore Sanfilippo 8e608a66e0 Trim Qwen batching controls and benchmark documentation 2026-09-16 14:46:28 +02:00
davidandClaude Fable 5.1 9ba464d529 Price the batched cycle kinds after eight drafted cycles, not sixteen 
Sixteen speculative cycles before the first plain one made a short
generation on prose pay the losing kind for most of its length (C=16 at
32 steps: 174 tok/s against ~195 once the policy has settled).  Eight
cycles give p over 128 draft rows at sixteen streams, enough to price.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:18:48 +02:00
davidandClaude Fable 5.1 307778657f Decide per cycle whether a speculative batch drafts 
A draft row's cost is not per row: the seventeenth row of a batched cycle
opens a second tile of every dense projection, after which rows are
nearly free (16 rows 80 ms, 23 rows 126, 32 rows 149), so gating streams
by their draft's confidence bought nothing (prose 184.2 vs 183.9 at a
margin of 2, code 210.7 vs 213.6).  A batch speculates for everyone or
for nobody, and it pays when (1 + p) c_plain > c_spec.

The engine keeps p and both cycle times as running averages (the same
shape as the session's depth policy), starts by pricing both kinds and
then probes the other kind for two cycles, every thirty-two while plain
and every hundred and twenty-eight while speculating.  DS4_QWEN4_SPEC_TRACE
prints the state per cycle.

C=16 prose 184 -> 195 tok/s (plain ~200, mostly plain cycles now), code
213.6 -> 214.3; C=8 prose 131 plain -> 158, code 131 -> 179 (the drafts
fill the one tile a C=8 batch has).  C=2 verifies lossless.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:18:47 +02:00
antirez 548cdaf897 Bound speculative batches and invalidate failed predictor state 2026-09-16 14:18:17 +02:00
davidandClaude Fable 5.1 e2cba0b7be Batch the predictor layer across a speculative batch's sessions 
The batched cycle drafted with one predictor pass per session, sixteen
passes of the nextn layer over one or two rows each, every one reading
the layer's weights again.  One pass now covers every row of the batch:
the trunk streams sit in the batch's R at the positions the trunk rows
had, so the layer's attention takes the same row table as the trunk's,
the staged inputs and residual go where the prompt priming puts them,
and the sessions' last rows are gathered for one pass of the head mixer
before the batched head and the per-session argmax.  A rejected draft's
row runs along with its parent as the token that follows; the next pass
rewrites its cache entry before anything reads it.

C=16 code 204 -> 213.5 tok/s cumulative (cycle 156 -> 148 ms), prose
176 -> 183; C=2 verifies lossless against plain greedy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:16:26 +02:00
davidandClaude Fable 5.1 d569d05aec Run the speculative batch's recurrent and attention steps as rows 
The speculative batch ran the GDN convolution, scan and the attention of
each session through the single-session kernels, one to two rows at a
time: sixteen sessions, forty-eight layers, ~2300 short dispatches a
cycle.  The GDN rows kernels now come in a variant addressed through a
table (state, history and the verify snapshots of each session by GPU
address, its first row and its row count), which walks a session's one or
two tokens in order and writes the snapshot after the first, with the
per-token arithmetic of the verify scan; the attention rows kernels take
one entry per token, so a draft row is simply a second entry at pos + 1
that sees the first token's keys.  The trunk of a cycle with no drafts now
costs what a plain step does (89 ms against 92).

With drafts the cycle is 158 ms for 1.50 tokens per stream (152 tok/s at
C=16 on the prose corpus): every draft row costs its own MoE pass, so the
marginal row (~4.4 ms) is close to a plain token (5.6 ms) and greedy
speculation cannot pay at this width on this model unless acceptance is
high.  Lossless against plain greedy decode (harness --spec --verify).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:15:22 +02:00
davidandClaude Fable 5.1 a504136766 Speculate over a batch of Qwen3.8 sessions 
ds4_sessions_eval_batch_speculative_argmax runs one MTP cycle for a batch:
every session contributes its next token and, when the predictor left it a
draft for that token, the draft as a second row.  The dense work of the
trunk runs once over all the rows; the recurrent, n-gram and attention
steps run per session over its one or two rows with the single-session
verify kernels, snapshots included, so a session's own arithmetic is that
of the C=1 speculative cycle.  A draft stands when it is the target's
argmax for its row; a rejected one restores the row-0 snapshot.  The
predictor then runs its layer per session over the committed rows and
every session's head as one product.  The Q8 batch GEMM serves 9..32 rows
in sixteen-row halves.

The harness gains --spec (with --verify checking the committed tokens
against plain greedy decode, which speculation must reproduce).  On an M5
Max at C=16 the first cut commits 1.49 tokens per stream and cycle on the
prose corpus but the cycle costs 171 ms against 92 for a plain step: the
per-session kernels and the sequential predictor passes are the next
thing to batch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:15:13 +02:00
antirez 467b6f05a0 Harden Qwen Metal batch integration and quality checks 2026-09-16 14:14:36 +02:00
davidandClaude Fable 5.1 9e91baccda Keep prefill chunks on the generic F16 tile 
The fp32 tile for F16 batches was meant for decode batches, where the
generic tile's half-staged activations carried rounding into the logits;
its rule also caught every prefill chunk's narrow hyper-connection
projections.  There it is slower (1305 vs 1410 tok/s at 4k) and it moved
a single stream's tokens off the reference's: with the generic tile back
for chunks past 64 rows, a C=1 stream generates the same 32 tokens as
upstream main, and prefill returns to 1408 tok/s at 4k, 1455 at 16k.
Decode batches keep the fp32 tile.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:07:24 +02:00
davidandClaude Fable 5.1 b4da720207 Count a prefill chunk's token tiles in the k-split rule 
The k-split of the dense GEMM counted only the row tiles, so a prefill
chunk's narrow projections split k thirteen ways although their 256
token tiles already filled the machine, and their partial planes grew
with the chunk: past the first context size of a run they no longer fit
the scratch allocated once, the GEMM failed mid-batch and every call fell
back to the per-token kernel, costing 8% of prefill at 8k and 16k.  The
rule now counts both tile dimensions: a decode batch splits as before, a
prefill chunk does not.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:07:24 +02:00
davidandClaude Fable 5.1 65d2eb3e15 Replace the shared session arena when a wider session finds it idle 
The shared transients arena took the shape of the first session, and a
later session needing a wider one (a larger context, hence more indexer
blocks) silently fell back to private transients, ~4-5 GiB each: sixteen
of them at the next context size of a sweep exceeded memory and the
process died without a word.  The engine now counts the sessions
borrowing the arena and, when a wider session arrives while none does,
frees the arena so that session builds the wider one; while sessions
borrow it, a wider one keeps private transients as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:07:24 +02:00
davidandClaude Fable 5.1 643d4cb5bc Give the batched output head the Q8 tile 
The plain batched decode sized its logits scratch to the batch width, so
the Q8 tile's 32-row padding did not fit and the output head, 248320 rows
read once per four tokens by the per-token matvec, cost 3.9 ms of a
76 ms step.  The scratch now holds the tile's rows, which the speculative
path already did, and the head takes the tile like every other Q8
projection.

C=16 plain prose 196.9 -> 203 tok/s cumulative; batched vs sequential
unchanged (same worst logit deviation, tokens agree).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:59 +02:00
davidandClaude Fable 5.1 891c0053c7 Keep verify rows on the few-row matvec 
The batch tile took every F16 projection past one row, which also caught
the two- and three-row speculative verify of a single session: its
hyper-connection down products cost three times what the few-row matvec
charged them, and that was their original path.  The tile now starts at
four rows.  C=1 with MTP on the prose corpus: 50.5 -> 55.6 tok/s (plain
51.3); C=16 is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:38 +02:00
davidandClaude Fable 5.1 a154af774b Walk both rows of a grouped down simdgroup together 
The grouped down pass ran its two rows one after the other, each a
dependent chain of block products.  Both rows now walk their blocks in
the same loop, two independent chains per pair in flight; each chain's
own order is the per-token kernel's and the test pins it byte for bit.
+2.3% on the C=16 step (1.16x over the per-token kernels).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:38 +02:00
davidandClaude Fable 5.1 3c0cfc3108 Specialize the grouped MoE down pass on its pair count 
Same shape as the mid pass: a template on the pair count so the block
loop carries no per-pair branch, one named accumulator per pair, the
per-token kernel's pinned chain untouched and pinned byte for bit by the
test.  +1.4% on the C=16 step (1.14x over the per-token kernels).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:38 +02:00
davidandClaude Fable 5.1 54a39d7fa2 Specialize the grouped MoE mid pass on its pair count 
The grouped mid kernel tested the pair count inside its innermost loop,
three uniform branches per weight, and read each block's scales and
nibbles a byte at a time.  A pass is now a template on the pair count
(one to four), so the inner loops carry no branch, and the block's words
come in as three loads (the byte values are the same); the per-token
arithmetic and its order are untouched, the test still pins it byte for
bit against the per-token kernel.  +7% on the C=16 step of an M5 Max
(1.12x over the per-token kernels, from 1.047x).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqNkMw6d
 2026-09-16 14:06:38 +02:00
davidandClaude Fable 5.1 d36dd5d592 Allocate the k-split scratch once 
The scratch for the k-split planes grew on demand, and the command buffer
holds unretained references: a plane buffer replaced while earlier
dispatches of the same batch still pointed at it was read after its
release.  That is what made splits of sixteen and more look wrong (a
logit difference of 1.3 that never moved).  The scratch is now allocated
once, 64 MiB or the first request if larger, and a request it cannot serve
fails instead of reallocating.  Splits of 32 and 64 now measure exact; they
are no faster, so the default width stays.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:24 +02:00
davidandClaude Fable 5.1 8378039eb4 Multiply Q8 decode batches on fp32 simdgroup matrices 
At sixteen rows the per-token Q8 matvec read every matrix once per four
rows and streamed the batch's activations once per four weight rows: a
third of the step for 3.4 GB of weights.  The batch GEMM gives each
simdgroup 32 weight rows and every token: each lane stages one q8_0 block
of its row as floats (the matvec's d * q values) into the simdgroup's own
threadgroup memory, so simdgroups never wait on each other, and the rows
multiply on fp32 simdgroup matrices against token tiles read straight from
the activations.  Narrow projections split k across the grid and sum their
planes with the tile reduce.  The sums stay within 2e-6 of the matvec's;
tests pin the kernel against a double reference and the matvec.

+10% on the C=16 step of an M5 Max (paired A/B).  Every simdgroup matrix
index is a compile-time constant: with the token-tile count a runtime
value the same kernel ran twice slower.  DS4_QWEN4_NO_BATCH_MM keeps the
matvec for A/B.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:24 +02:00
davidandClaude Fable 5.1 fe457518a6 Group a decode batch's routed experts and run its shared expert densely 
Sixteen rows pick ten experts each but only about a third as many
distinct ones, and the per-row kernels read an expert once per row that
chose it.  The grouped kernels give each expert to the threadgroup of the
first (row, slot) pair in its list, which walks the expert's rows once per
four pairs with a named accumulator per pair, so every row's block walk and
chain order are the per-row kernel's and its outputs match bit for bit
(tests/test_qwen4_kernels.c pins mid and down under heavy reuse).  The
shared expert, read once per row as an extra slot, becomes dense
projections over the batch as in prefill.

Together +7% on the C=16 step of an M5 Max (paired A/B); single rows and
verify rows keep the per-row kernels.  DS4_QWEN4_MOE_NO_GROUP and
DS4_QWEN4_MOE_SHARED_DENSE_MIN keep the old paths for A/B.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:24 +02:00
davidandClaude Fable 5.1 2c5b1aab14 Keep F16 decode-batch projections on the fp32 tile 
Past eight rows the F16 projections reached the generic tile, which stages
the activations as halves; the hyper-connection up projections carried
that rounding into the logits of a batched step, flipping near-tied tokens
against sequential decode.  The k-split fp32 tile keeps float operands at
the same cost (1.01x on the C=16 step), so it takes every F16 batch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:24 +02:00
davidandClaude Fable 5.1 4e01ef9dec Take F32 decode-batch projections through the tiled GEMM 
The router and the GDN gate projections are F32 and narrow (512 and 48
rows), so at sixteen rows the per-token kernel gives the machine a few
dozen threadgroups and reads the matrix once per four tokens.  The
k-split tile spreads them over the machine: +2% on the C=16 step (paired
A/B on an M5 Max).  Q8 weights measured the other way (the tile loses 5%)
and stay on the per-token kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:24 +02:00
davidandClaude Fable 5.1 3e068bb0e7 Batch Qwen3.8 attention rows into one dispatch per stage 
The decode batch ran attention as sixteen chains of per-row dispatches:
prep, block key, score, select, expand, decode and merge for each session
in turn.  The rows kernels take the row as a grid axis and reach each
session's caches through a staged table of GPU addresses, so a layer costs
one dispatch per stage.  Every kernel keeps the single-row arithmetic, the
row's own split count included, and the single-row kernels call the same
bodies; tests/test_qwen4_kernels.c pins the rows path byte-exact against
the per-row dispatches for dense, block-completing and sparse rows on both
selector widths.

On an M5 Max at C=16, ctx 4096 the step is unchanged (1.00x): those grids
were not latency-bound after all, the attention stages take ~6.5 ms of a
110 ms step.  DS4_QWEN4_NO_BATCH_ATTN keeps the per-row path for A/B.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:23 +02:00
davidandClaude Fable 5.1 6efb642d53 Report the logit distance when a batched verify fails 
The distance gathered up to the failing step says whether the flip was a
near tie under a changed reduction order or a wrong result.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:23 +02:00
davidandClaude Fable 5.1 ba3b0c23e3 Batch Qwen3.8 Flash Next decode natively across sessions 
Qwen3.8 sessions were excluded from native session batching: each decode
step ran the whole graph once per session, so sixteen sessions cost
sixteen times the weight traffic.  The batched encoder now runs the dense
work (hyper-connection mixers, projections, MoE, output head) once over
the stacked rows and keeps only the recurrent state, the caches and the
PLE history per session, with batched GDN scan and conv kernels reading
the pooled state through per-row slots.

Session transients (prefill scratch) move to one shared arena sized for
the largest session, so each further session costs its caches only
(memory per session divided by ~3.6 at ctx 4096).  The dense GEMM gains a
split-K path for the narrow-batch regime, and the dense/MoE/mixer GEMM
thresholds, tuned for T=1 or T>=8192, are moved to serve decode batches
of 4..64 rows.  Every step keeps the tokens of the sequential path;
float reduction order changes where the batched kernels do.

C=16 at ctx 4096 on an M5 Max goes from ~50 to ~143 tok/s aggregate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:06:23 +02:00
davidandClaude Fable 5.1 147b2632e0 Add a session-concurrency benchmark for the engine and the server 
Serving several requests at once is a different regime from the single
session walk ds4-bench measures.  session_concurrency_bench drives the
engine API directly over a concurrency x context grid and reports the
aggregate decode throughput, the per-stream loss as the width grows, the
cost a resumed prefill imposes on the decoders, and a teacher-forced check
that a batched step emits the tokens the sequential path emits.  Cells are
priced from the engine's own memory accounting against a budget sampled
at startup, because macOS swaps long before an allocation fails.
concurrency_sweep.sh runs one process per cell; serve_concurrency_bench.py
measures the HTTP server with vLLM-style metrics.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:05:35 +02:00
davidandClaude Fable 5.1 d0b74340bf Read a decode token's n-gram rows concurrently 
The PLE n-gram table is read from disk uncached, one 320-byte pread per
row, and a token needs sixteen rows.  Requests under 256 rows were read
in sequence to spare the worker dispatch, which put 1.3 ms of SSD latency
per token on the critical path before the GPU could start: 7% of a
19 ms decode step, 10% of a speculative cycle.  Any request of two rows
or more now goes through the parallel readers, capped at the row count.

C=1 plain 51.3 -> 54.5 tok/s (stage 1.31 -> 0.25 ms); C=1 MTP code
69.9 -> 78.9, prose 58.3 -> 66.0.  Logits bit-identical to the previous
build; prefill unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
 2026-09-16 14:04:25 +02:00
GiorgioOppo 9a8462a19f Metal: streamline Qwen Q2 down loads and shared expert row reuse 
Widen Q2_K activation, scale and quant loads with packed types while keeping
the original FP32 accumulation chain. Process the first two complete blocks
without per-lane tail checks, then process the final 128 live values only in
the lower sixteen lanes. Never form padded activation addresses for logical
K640 weights stored at physical K768.

Reuse shared Q8 activation loads across two output rows, preserving each
row's block order, four-term expression and simd_sum. Keep the existing host
dispatch and fallback for other shared formats.

Add a source-derived GPU regression against the generic row-dot kernel:
resident and separate SSD payloads, full/sparse/empty masks, T1/2/3, odd row
tails, minimum alignments, nonzero padding, default/safe math and edge values.
Validation: 1650 fixtures and 19955180 comparisons pass with Metal API
validation and zero NaN payload differences. Host Q2 dispatch tests pass
for T1/2/3, output2560/2561 and shared present/absent.

The tail/shared changes, measured against the preceding packed-load candidate,
improve isolated full-slot kernel throughput by 6.36%, 7.77% and 7.09% at
T1/T2/T3. Eight ordinary SSD requests show decode medians of 9.71 versus
9.67 t/s; no end-to-end gain is demonstrated. Output and cache work match.
 2026-09-16 08:21:04 +02:00
GiorgioOppo 13c53d922b Metal: specialize Qwen IQ2 decode and reuse shared Q8 gate/up inputs 
Export NR1 and NR2 variants from one IQ2XXS gate/up kernel body. On M1 Max,
select NR1 for the shipped single-token K2560/F640 shape with at least eight
active slots and the default four-SIMD-group policy. Preserve NR2 for small
split passes, multiple tokens and explicit generic-specialization settings.
Register both exports with the SSD address-table specialization.

For the shared Q8 expert, load each activation group once for gate and up
while keeping independent ordered accumulators and SIMD reductions. Preserve
the fallback for other shared weight types and the original slot addressing.

Add a standalone GPU test for both exports, resident/SSD addressing, compact
masks, tails, floating-point edge cases and default/safe math. Compare shared
outputs against separate row-dot projections, independently of both exports.

Validation: the current M1 Max run passes 180 fixtures and 1119600 comparisons,
including the independent shared oracle, with zero NaN payload differences.
These correctness results do not imply an end-to-end throughput gain.
 2026-09-16 08:21:04 +02:00
GiorgioOppo a37fd7f53d Metal: reuse IQ2 headers when staging MoE prefill tiles 
Handle IQ2XXS directly in qwen4_mm_stage16 instead of decoding two independent
eight-value halves. Read the shared 32-value header once through a packed
ushort4, then load aligned four-byte codebook slices for both halves.

Respect the two-byte alignment of 66-byte GGUF superblocks and avoid reading
past codebook entries. Preserve the stage8 scale expression, sign mapping,
element order and final destination conversion for each staged value.
Other quantization formats retain their existing paths.

Validation: strict/fast CPU decoding oracles and sanitizer checks from the
optimization review pass; the current Metal MoE MM specialization suite
also passes with exact mid/down output and intact padding on M1 Max.
 2026-09-16 08:21:03 +02:00
GiorgioOppo bbbc01207e Metal: tune Qwen hyper-connection dispatches for M1 Max 
HC-down's narrow 320-channel output leaves too few threadgroups in ordinary
prefill chunks. Add a 32-channel by 16-token F16 tile using 64 threads and
3072 bytes of shared memory, preserving half RHS staging and K32 MMA order.
Select it only on M1 Max for K=10240, output=320 and 9..128 tokens.

Enable the existing F16 RMS-reuse path for measured M1 Max chunks of 48..256
tokens at the shipped HC shape. Use sixteen SIMD groups for the single-token
reuse mixer without changing per-row lane mapping or reduction order.
Retain the established device and shape fallbacks.

Add a direct HC-down GPU oracle with partial tiles, bounds checks, default
and safe math; add host norm-dispatch boundary cases at 47/48 and 256/257
as well as ordinary chunk/tail sizes. Wire the test into Makefile and clean.

Validation: the direct HC suite passes all 16 pipelines and 92 fixtures on
M1 Max with Metal API validation. Existing host norm/mixer parity checks
cover the changed policies; no cross-device speedup is claimed.
 2026-09-16 08:21:03 +02:00
GiorgioOppo a50fecce6f Metal: streamline Q8 matrix-vector loads and reduction 
Use packed float4 activation loads and aligned char2 quant loads while
preserving each row's eight-operand FP32 accumulation sequence.

Give Q8 a single-use reduction helper: SIMD groups publish only their own
partials, zero padding stays in registers, and group zero performs the final
reduction. Preserve both simd_sum stages even for NSG=1, including positive
zero padding, while removing the shared-memory clear and its extra barrier.
Document the scratch lifetime requirement at the helper.

Add a standalone Metal regression using a frozen legacy reduction oracle,
with compile-only support and Makefile/clean integration. Exercise grouped
reductions, row tails and floating-point edge cases.

Validation: 3072 cases and 4096 output rows pass on M1 Max with Metal API
validation; non-NaN results match bitwise and NaN classifications match,
with zero NaN payload differences in this run.
 2026-09-16 08:21:03 +02:00
GiorgioOppo 2ba92ab234 Metal: compact streamed MoE prefill dispatches by active expert 
SSD split passes previously dispatched a full expert grid even when many
experts had no work in that pass. Copy the active original expert IDs into
each encoder's arguments and launch only those experts for mid/down, regular
and tail tiles, including the NAX variants.

Keep unfiltered routing frequencies for tile-policy decisions and preserve
original expert IDs for weights, counts, token lists and outputs. Resident
and missing passes carry immutable CPU pass-count views; GPU arguments
contain copied IDs rather than CPU pointers. Reject inconsistent pass counts
and skip empty passes while invalidating stale NAX half-mid associations.

Update host and shader layouts together, with size/offset assertions for the
2112-byte structure and the 4096-byte setBytes limit. Add a source-derived CPU
regression for mappings, schedules, split-pass lifetimes and empty-state
handling, including ASan/UBSan execution.

Validation: sanitized compact-dispatch regression and the Metal MoE MM
specialization suite pass. Existing full-model SSD comparisons retain
identical generated output and expert-cache work.
 2026-09-16 08:21:03 +02:00
GiorgioOppo 29607116c3 Qwen: skip unused indexer queries in dense attention prefixes 
Dense-prefix attention consumes every complete visible block, so its indexer
query projection and normalization have no consumer. Derive the first sparse
row from one shared, 64-bit causal-block boundary and omit only those unused
queries when an entire ordinary chunk remains dense.

Continue preparing and pooling every indexer key for later sparse attention.
Chunks crossing the boundary keep the original full query projection and
batch shape, preserving rounding. MTP query preparation and cache-only
predictor population retain their existing behavior.

Add row-by-row CPU boundary tests, including the shipped position-2051 cutoff
and unsigned limits. Extend Metal preparation tests to T=1/2/3/128, both RoPE
modes, poisoned IQ destinations, exact retained Q/gate/K/V/IK outputs, and
surrounding cache guards.

Validation: test_qwen4_memory and the targeted Metal attention-preparation
suite pass on M1 Max with Metal API validation enabled.
 2026-09-16 08:21:03 +02:00
Salvatore Sanfilippo f396751952 Optimize Qwen CUDA projections and expert kernels 
Accumulate dense component products directly in the output and fuse short F16 projections while retaining FP32 correction sums and cuBLAS fallbacks. Tune expert tile sizes by weight format and unpack IQ2 signs in packed bytes without changing represented weights. Keep BF16 n-grams disk-only.

Extend independent kernel references for large outputs, partial tiles, alignment, strides, repeated writes and output guards. Refresh Spark performance references and repeatable sanitizer checks.

Paired Spark runs improve Q2/Q4 large prefill from 685/670 to 745/755 tokens/s, Q2 decoding from 21.0 to 22.6 tokens/s, and Q2 English MTP from 26.8 to 30.3 tokens/s. Official continuation scores remain unchanged; kernel references, sanitizers, session/MTP, mixed-prefill and vision checks pass.
 2026-09-16 00:26:12 +02:00
antirez bb6a7485f3 Keep release QA focused on repeatable checks 2026-09-15 17:04:07 +02:00
antirez 442982dd87 Update Qwen Spark performance references 2026-09-15 17:04:06 +02:00
antirez f74d410962 Check Qwen CUDA split attention against CPU references 2026-09-15 16:44:01 +02:00
antirez 02a2558225 Share Qwen CUDA prefill attention across grouped query heads 
Use tensor-core products with scaled residual correction for FP32 queries and probabilities. Keep scalar and split attention fallbacks, and extend coverage with independent double-precision references, sparse masks, empty rows and head-group tails.
 2026-09-15 16:23:23 +02:00
antirez 7fe838b8cb Reuse Qwen CUDA prefill normalization across injection chunks 
Keep the decode reduction order and verify byte-exact normalized rows and injection partials against one-token controls, including long batches and guarded tensor views.
 2026-09-15 15:40:42 +02:00
Salvatore Sanfilippo 63b5dca93e Add official Qwen quality references and validated API scoring 
Collect references from Alibaba through OpenRouter with an explicit empty system message and no provider fallback. Validate token byte alignment and reject lossy Unicode metadata for API agreement while retaining continuation likelihood scoring.

Track 100 short and 12 long reference cases, Q2/Q4 default and quality controls, continued-prefill results, and independent long-context answers. Include collector and scorer regressions and document the hosted-checkpoint and precision limitations.
 2026-09-15 15:00:11 +02:00
Salvatore Sanfilippo df6eedd972 Add optimized CUDA inference for Qwen3.8 Flash Next 
Share the Metal session graph with CUDA for text and vision, MTP and exact sampling, steering, checkpoints, and disk-only BF16 n-grams. Add bounded concurrent Linux n-gram reads, vectorized dense and expert kernels, register-held recurrent prefill state, and efficient small speculative batches.

Keep Qwen out of DeepSeek batch dispatch and exclude disk n-grams from resident memory accounting. Use normal file-backed weight loading in diagnostics and link n-gram support into the CUDA quality scorer.

Include independent kernel and memory-safety tests, mixed-prefill and recovery checks, long-context server and agent coverage, and user documentation with measured DGX Spark performance.
 2026-09-15 15:00:11 +02:00
GiorgioOppo 5cbdb6ef81 Metal: pipeline Qwen SSD MoE prefill and reuse Q2_K down rows 
Overlap Qwen batched MoE work with reads of the selected experts on M1 Max.
Previously the MM path waited for the missing payloads before running its
routed projections. Run resident experts first, then submit missing gate/up
work as soon as those weights arrive while the read pool loads down weights.
Only the remaining down projection is submitted after the second read phase.

Keep per-expert token lists and routing frequencies unchanged. Separate
resident/missing count buffers and immutable address snapshots preserve tile
selection and caller-owned routing state. Skip zero-count experts before the
MM tile logic, including the NAX helpers.

Retain pending payloads and indirect-address resources across command-buffer
flushes, including unretained command buffers. Install cache entries only
after all projections have been read, protect them with the current inflight
sequence, and drain submitted work before reclaiming failed reservations.
Recycle uninstalled slab slots or explicit buffers after read/install errors.
Report the two read phases accurately in the streaming diagnostics.

Add a Q2_K down kernel that reuses each activation load for two output rows
at the Qwen width of 640. Preserve the lane mapping, block/element order and
SIMD reduction, and ignore the unused tail of the physical 768-column row.
Select it automatically on M1 Max with the existing two-row dispatch policy;
keep the generic fallback for other shapes, devices and diagnostic overrides.

Extend regression coverage for Q2_K T1/T2/T3, odd output widths, shared rows,
nonzero quantized padding and guarded tensor views. Add SSD MM cold/mixed
fixtures and 18 down-read failure cases across three quantization pairs and
T29/T33/T128. Each new case exercises four consecutive failures followed by
an exact retry and a warm-cache pass. Verify unfinished outputs, input lists,
counts, cache ownership and batch restoration on failure.

Validation on Apple M1 Max, 32 GiB:
- Metal builds and git diff --check passed.
- Full Qwen kernel and MoE specialization suites passed, including
  T8191/T8192/T8193 boundaries.
- SSD tests passed with ordinary and unretained command buffers and with
  separate expert buffers; full-model SSD MTP cache/logit checks passed.
- 96 exploratory/final performance runs retained identical output bytes.
  The final 32-run ABBA/BAAB comparison used Qwen3.8-Flash-Next-Q2,
  SSD streaming, ctx 4096, chunk 128, temp 0 and --nothink, with short/long
  prompts and MTP on/off (four observations per arm and configuration).
- Relative to the already optimized resident-overlap/Q2-reuse baseline,
  the additional SSD pipeline improved median prefill by 6.4-10.9%.
  Median decode changes ranged from -4.4% to +3.1%; this is a prefill
  optimization with a measured decode tradeoff, not a uniform speedup.
  Cache budgets, hits/misses and read volumes matched within each case.

The experimental Q8 row variants and FP16 activation staging were removed
after full-model measurements. No model-format or precision change, new
permanent opt-in, benchmark report or temporary artifact is included.
 2026-09-15 11:15:31 +02:00
GiorgioOppo 81b9ebbf7a Metal: overlap Qwen MoE down and MTP verification with SSD reads 
The existing SSD prefix handled resident/shared gate-up work for one token.
Two- and three-token MTP verification waited for all missing experts, and
the down projection remained after I/O even when its inputs were resident.

Give MoE launches independent slot masks for T1/T2/T3, with explicitly empty
rows and compact grids mapped back to the original routed/shared slots.
Apply the masks to generic, IQ2 and Q4 mid kernels and both down variants.
Submit resident/shared mid and down work before reading missing experts,
then finish only missing slots before the unchanged reduction and HC update.

Snapshot all three GPU-address tables so cache installation cannot race the
prefix. Retain their buffers and mark referenced expert resources inflight;
keep failure drains, batch ownership and off-size/full-layer fallback paths.
Numerical dot/reduction order and slot layout are preserved. No new opt-in
is required. Timing totals count shared submission/read intervals once.

Extend SSD fixtures with independent T2/T3 masks, empty prefix/suffix rows,
duplicates, shared/no-shared experts, Q2 padding and Q4/MXFP4/Q8 formats.
Verify exact mid/down/reduce/HC output, guards, missing-only reads, warm
operation with an unreadable fd, failure cleanup and retry. Also exercise
24 disjoint cache replacements through the public T128 streaming path.

Validation: make all; memory and cache-state tests; full Qwen Metal kernel
suite; SSD default, unretained, Q4-NR1, MXFP4-prefetch and specialized MV
variants; real-model MTP prefix/checkpoint/continuation and n-gram recovery
oracles all passed. SSD stage comparisons and recovery logits are exact.
The independent differently batched continuation oracle remained within its
existing tolerance (maximum absolute logit error 0.000974178).

Combined-series benchmark against b86c8ae: M1 Max 32 GiB, Qwen Q2, Metal SSD,
ctx 4096, chunk 128, temp 0, no thinking. Four runs per version/scenario,
ABBA/BAAB order, 32 runs total; every generated output was byte-identical.
Short prompt: 29 input/100 output tokens; long: 575 input/50 output tokens.
Median prefill/decode token rates, baseline -> complete series:
  short MTP:    9.695/8.250  -> 9.575/10.170  (-1.24%/+23.27%)
  long MTP:    39.865/8.405  -> 42.840/11.015 (+7.46%/+31.05%)
  short normal: 9.455/7.415  -> 9.375/7.955   (-0.85%/+7.28%)
  long normal:40.940/8.620  -> 44.890/9.445  (+9.65%/+9.57%)

These measure all three changes together on this host, not this commit in
isolation. Baseline decode was variable (short MTP 7.71-10.82 t/s; long MTP
8.25-10.78 t/s); filesystem caches were not purged. The short prefill shows
no stable gain. Intermediate reports, logs and model files are excluded.
 2026-09-15 09:19:17 +02:00
GiorgioOppo e2b4a47e07 Metal: plan Qwen SSD expert cache evictions once per batch 
A full Qwen expert cache scanned all 40,960 metadata entries for each missing
expert in a selected batch. Large prefill unions repeated that scan hundreds
of times before parallel reads could start.

Collect at most 512 eligible candidates with a bounded heap and sort them by
hotness, last use, layer and expert ID. Consume each victim at the original
buffer-allocation point, preserving the scalar selector's tie order, lazy
eviction, buffer offsets and accounting. Planning itself never evicts an
unused candidate if a subsequent allocation or read fails.

Limit the path to owned-buffer selected batches and invalid target entries.
Keep the existing selector for inflight/service-thread states, mapped
batches, live target replacement and recovery after a stale or exhausted
plan. GLM/DeepSeek mapped-batch callers retain their current path.

Add a CPU-only Metal-cache oracle test for 1/16/17/511/512 victims, a full
40,960-entry cache, protected IDs, numeric ties, sentinel values, interrupted
consumption, stale/inflight plans, live replacement and exact counters.
Register the make target and ignore its generated executable.

Validation: make test-metal-ssd-reuse passed. The full series additionally
passed the public Qwen SSD fixture with 24 disjoint victims. Combined-series
M1 Max measurements reduced long-prompt cache scan time by 81% without MTP
and 86% with MTP; these are nested preparation times, not end-to-end gains.
 2026-09-15 09:19:02 +02:00
GiorgioOppo 9e1429b8b6 Qwen: bound MTP staging to recover SSD expert cache capacity 
The memory admission path reserved two complete copies of the largest routed
layer, including the differently quantized MTP predictor. Ordinary draft and
chain calls select only ten predictor experts, so the reserve unnecessarily
reduced the trunk expert cache when MTP was enabled.

Compute the staging bound from the maximum of two full trunk windows, two
compact predictor payloads with their three address tables, and one full
predictor layer for the older-Metal/no-gpuAddress fallback. Preserve the full
fallback bound: streamed MoE drains previous transient allocations before
loading another payload, and discards compact buffers before full fallback.

For Qwen3.8-Flash-Next-Q2 this changes staging from 2,778,726,400 to
1,525,678,080 bytes. On the tested M1 Max/4096 context, the automatic expert
cache grows from 7,371 to 8,212 entries without increasing the planned budget.

Add metadata-only tests for real padded Q2 tensor dimensions, disabled or
absent MTP, compact/full fallback dominance, address tables, invalid shapes,
and saturated or overflowing sizes.

Validation: make all and test_qwen4_memory passed. The complete three-change
series also passed Metal kernel/SSD fixtures and the real-model MTP and
n-gram recovery oracles; performance of the series is recorded in the final
commit, rather than attributed to this memory change alone.
 2026-09-15 09:18:35 +02:00
GiorgioOppo b86c8ae1db Metal: remove redundant Qwen MTP projections and expert work 
Preparing historical predictor cache rows previously computed query/gate
and indexer-query projections that were never consumed. Catch-up batches
also ran the predictor MoE for every row even though only the final
residual feeds the vocabulary head or the next recursive draft.

Add a cache-only attention preparation kernel that writes K, V and raw
indexer keys using the original normalization, RoPE and cache arithmetic.
Skip query projections at graph level while retaining fused K/V/IK GEMV
for short batches and the existing pooled-key updates. Ordinary attention
continues through the complete preparation path.

Add a Q8 EH projection for the staged [embedding|0] and [0|hidden] rows.
Separate embedding and hidden dispatches within one encoder, traversing
only the active weight half while preserving full row strides and the
original float4 dot products and SIMD reduction order. Check the inactive
half's scale words to preserve NaN/Inf classification for zero products.
Validate tensor spans, alignment and output aliasing before dispatch.
Select this kernel on M1 Max for E2560, four HC streams and T1..3; keep
the existing path on other devices/layouts, non-Q8 weights and diagnostic
GEMM or Q8 prefill-profile selections. No quantization change, new buffer
allocation or production opt-in is introduced.

Keep batched attention, HC mixing and router arithmetic during MTP
catch-up, then bind last-row views for the independent routed/shared MoE
and residual reduction. Restore graph bindings and release views on all
exit paths. Earlier rows still build the full predictor KV prefix; only
the final residual is consumed. Ordinary trunk MoE remains unchanged.

Extend the focused Metal kernel regression with T1/T2/T3, actual Qwen
projection dimensions, cache boundaries and guards, nonzero multimodal
RoPE positions, signed zero, nonfinite Q8 scales, diagnostic fallbacks,
input immutability and invalid-span/alias rejection. Add a frozen original
MTP input/layer oracle to the model-backed prefix test, including recursive
drafts and exact three-row verification. Preserve diagnostic environment
settings and cover unfused execution without assuming speculation runs.

Validation on M1 Max 32 GiB with Qwen3.8-Flash-Next-Q2.gguf:
- Application and relevant test targets built; git diff --check passed.
- Focused synthetic cache/projection kernel tests and memory checks passed.
- Real SSD predictor comparisons matched EH, KV/pooled keys, final residual,
  all 248320 logits and recursive drafts bit for bit for T1/T2/T3.
- Prefix, chunk/append, payload and unfused tests passed. Session checkpoint,
  restore/rewind, exact-sampling continuation, n-gram failure recovery and
  small-buffer regressions passed. Independent target continuation stayed
  within the established batch tolerance (max absolute error 0.000974178).

Performance versus 92f57fc, frozen binaries/shaders, SSD streaming,
ctx4096/chunk128, temp0, no thinking, MTP enabled via --mtp-timing:
- Four runs per version and prompt in ABBA BAAB order, harmonic-mean rates.
- Rome, 29 input/100 output tokens: prefill 9.85 -> 9.88 t/s (+0.3%),
  decode 9.01 -> 9.57 t/s (+6.2%); mean process time increased 0.8%.
- Long prompt, 575 input/50 output tokens: prefill 41.28 -> 42.12 t/s
  (+2.0%), decode 9.40 -> 9.11 t/s (-3.0%); process time decreased 0.5%.
- Generated outputs and first-draft acceptance counts were identical.
- Isolated EH microbenchmark: 16 samples per arm, 64 dispatches per sample,
  including host encoding and GPU completion. T1: 161.28 -> 127.36 us;
  T2: 309.67 -> 188.75 us; T3: 438.84 -> 277.03 us (+27..64% throughput).
- A separate no-MTP control measured 7.58 -> 7.33 t/s decode (-3.3%).
  End-to-end variation remains material; no stable global 10% improvement
  is established. Final runs had no concurrent model processes, but OS
  file-cache and desktop background activity were not controlled.

CUDA/ROCm were not exercised; graph changes are in the Metal Qwen path.
Temporary benchmarks, logs, notes and model files are excluded.
 2026-09-14 20:53:23 +02:00
GiorgioOppo 92f57fc4d7 Qwen: prepare MTP prefix caches and preserve predictor state 
Qwen prompt sync previously ran only the trunk. The first predictor step
then wrote at the decode position while attention read the entire nextn
prefix, including unwritten rows or data left by a previous session.
Target verification still gated the generated tokens, but the drafter's
context was incomplete and could reduce proposal acceptance.

Build the predictor K/V, raw indexer keys and pooled block keys while each
prefill chunk's trunk hidden states are available. Share the existing MTP
input projection and attention preparation, using at most three rows of
draft scratch. Skip the predictor attention output, MoE and vocabulary
head, avoiding routed-expert SSD reads for this cache preparation.

Retain the last committed trunk row until its actual successor is known:
- Repair the last predictor row before prompt append, ordinary decode or
  verification, including a sampled token that differs from the argmax.
- Produce the first draft from this retained row rather than requiring an
  initial ordinary decode cycle to seed the predictor.
- Preserve the row in all three verifier snapshots and restore it during
  rejection, exact-sampling replacement and caller-requested rewind.
- Track a contiguous predictor prefix and invalidate provisional chain
  rows and staged inputs when restoring or preparing only the cache.
- Initialize future RoPE positions for recursive drafts without advancing
  the trunk's multimodal position state.

Qwen payload v3 stores only the canonical predictor prefix plus its pending
trunk row. Reject old payload tags that cannot prove a complete prefix;
old Qwen disk checkpoints must be rebuilt. Account for the retained row
and verifier copies in memory estimates: 40 KiB per row on this model,
up to 160 KiB including all snapshots. Clarify that timing statistics count
first-draft acceptance, and document MTP usage with Metal SSD streaming.

Add a model-backed Metal regression comparing cache-only preparation with
the complete MTP layer on identical trunk batches. Cover poisoned caches,
one-token prompts, chunk lookahead, non-argmax append, reused sessions,
MTP on/off payload interoperability, legacy-tag rejection and depth-2,
depth-3 and exact-sampling continuations. Restore FORCE_ACCEPT after the
rewind test and remove assumptions that the first MTP call commits only
one token. Allow the small-buffer CLI regression to use SSD streaming.

Validation on M1 Max 32 GiB, Qwen3.8-Flash-Next-Q2.gguf:
- Standard make and Qwen memory-estimate test passed.
- MTP prefix oracle passed with bitwise cache, residual and prefill logits
  agreement; continuation logits stayed within the established tolerance
  (maximum absolute error below 0.001 across 248320 vocabulary entries).
- Prefill checkpoints, reused restore, rewind, exact-sampling resample,
  n-gram read-failure recovery and small/unfused buffer tests passed.
- Two runs per arm in ABBA order, ctx4096/chunk128, SSD streaming, temp0:
  short-prompt decode 8.13 -> 8.09 t/s; 575-token-prompt decode
  8.87 -> 9.88 t/s (+11.4%), using harmonic means. Output was identical.
  Long-prompt first-draft acceptance rose from 45.5% to 56.2%; prefill
  throughput fell 2.5% on the long prompt and 6.4% on the short prompt.
  Mean total process time improved only 1.1% on the long prompt and
  increased 3.4% on the short prompt, including startup and prefill.
  These are local observations with uncontrolled OS file-cache/background
  variation, not a general 10% speed guarantee. CUDA/ROCm and vision were
  not exercised; the affected Qwen execution path is Metal.
 2026-09-14 19:45:36 +02:00
GiorgioOppo 496b153eed Metal: adapt Qwen SSD MoE tiles and specialize low-bit prefill 
Short Qwen SSD prefill batches route relatively few tokens to each of 512
experts. On M1 Max, the previous 32-token matrix tiles waste matrix work
on padding and the generic dequantizer retains run-time format selection
inside the K loop.

Use the CPU routing frequencies already validated against selected IDs and
GPU lists to choose eight-token tiles with one grid slice when their padded
token work is less than half that of the 32-token tiles. Preserve wider
tiles for concentrated routing so weights are reused across more tokens.
This adds no allocation, GPU readback, or synchronization.

Enable the existing type-specialized IQ2_XXS/Q2_K matrix kernels on this
measured SSD path. Function constant 900 removes dynamic format selection;
dequantization, FP16 operands, accumulation order, and output layouts stay
unchanged. Preserve the existing diagnostic overrides and PSO cache keys.

Scope automatic selection to M1 Max, 512 experts, and batches of 33..128
tokens. Batches of at most 32 tokens showed no stable end-to-end benefit
and retain their old pipelines. Resident inference, other devices/types,
large prefills, and the single-token decode path retain their prior policy.

Extend SSD regression coverage with a forced generic NT4/grid4 resident
oracle and repeated generic NT1/grid4 versus automatic SSD dispatch on the
same buffers. Verify frozen router-list membership, per-expert counts,
offsets and guards, skewed routing, the 32/33-token boundary, cache hits with
an unreadable backing fd, cache overflow, and full-selection fallback.
Compare mid, down, reduction, and HC outputs bit for bit; restore all five
diagnostic environment settings after each case.

Validation:
- make all; full Qwen Metal kernel suite and MoE specialization matrix.
- Qwen SSD fixture: default, unretained command buffers, Q4 NR1 fallback;
  final 32/33-token boundary fixture passes default and unretained modes.
- Metal SSD regression and real Qwen ngram state/failure recovery tests.
- Real SSD model: all 248,320 logits match the old generic NT4 path after
  a 342-token prefill and each of 25 subsequent decode steps.

Performance versus cdfc0d5, Qwen3.8-Flash-Next-Q2, M1 Max 32 GiB, SSD,
ctx 4096, prefill chunk 128, temp 0, no thinking, max 50 output tokens:
- 352-token prompt: 32.58 -> 36.92 t/s prefill (+13.3%).
- 575-token natural prompt: 38.96 -> 46.53 t/s prefill (+19.4%).
Each long prompt uses two alternated A/B pairs with frozen binaries and
identical generated output; rates aggregate equal token counts over elapsed
time. Application expert caches start empty; the OS cache is uncontrolled.
All measured long-prompt chunks are above 32 tokens and exercise the final
policy. No 10% claim is made for short prompts, decode, or other devices.
 2026-09-14 19:12:04 +02:00
GiorgioOppo cdfc0d51e1 Metal: reuse Qwen IQ2 and HC inputs on M1 Max 
Reduce repeated activation work in Qwen decode without changing the model's
quantization or per-row arithmetic. Select the new paths automatically on
M1 Max and retain the existing diagnostic overrides for direct comparison.

Add an IQ2_XXS gate/up kernel with two rows per SIMD group. Load each lane's
eight activation values once per quantization block and reuse them across
both rows and projections. Preserve the original partial sums, block order,
SIMD reduction, shared-expert fallback and output layout. Read 66-byte block
headers through aligned ushort loads. Carry the existing SSD address-table
function constant and compact slot mapping through the new dispatch, with
guards for odd rows and invalid expert IDs.

Add an F16 hyper-connection mixer that stores each low-rank input and its
sigmoid in threadgroup memory for reuse across streams and output rows.
Keep the two values separate to preserve (x * weight) * sigmoid(x) rounding;
pin reassociation and contraction at the original arithmetic boundaries.
Allocate 2560 bytes of scratch for rank 320 and place the barrier before
partial-group exits. Keep the paired two-token mixer and other-device
defaults unchanged. No public API or production environment flag is added.

Extend numerical tests with an explicit generic oracle versus automatic
dispatch, retaining all previous NR/NSG geometry checks. Cover T1/T2/T3,
F640/F641, shared and unshared experts, E33/E2560 HC outputs and prefix/suffix
guards. Add a focused 96 MiB test/benchmark arena using packed IQ2 weights
without dense shadows. Time seven alternating pairs on identical buffers,
after 20 warmups per arm, with 100 IQ2 or 1000 HC calls per measured batch.
Timings include CPU encoding and the final GPU completion wait.

Validation on Apple M1 Max, 32 GiB:
- Built all application targets and relevant Metal tests.
- Passed the complete Qwen kernel suite and MoE exact comparisons.
- Passed Qwen SSD fixtures in default, unretained-command-buffer and Q4_K
  NR1 configurations, plus existing Metal SSD expert regressions.
- Passed real Qwen3.8 Q2 n-gram failure/recovery tests for prefill, decode
  and MTP. Two paired model runs preserved all logits bit for bit across
  410 decode steps, with independent session state and alternating order.

Resident microbenchmarks measured IQ2 gate/up at 173.78 -> 163.91 us for
T1 (+6.0% throughput), with similar gains at T2/T3. The HC mixer measured
36.57 -> 17.64 us for T1 and 101.22 -> 41.28 us for T3.

Full-model gains are smaller and variable. Two paired runs with a shared
7708-expert SSD cache measured +2.51% and -0.08%, or +1.20% pooled over
400 measured steps after warmup. Independent CLI runs did not establish a
stable end-to-end speedup; these results do not claim a 10% gain, a prefill
improvement, or a benefit on untested devices. Temporary logs, model files
and intermediate benchmark harnesses are excluded from this commit.
 2026-09-14 18:18:33 +02:00
GiorgioOppo d6cbc77ca6 Metal: overlap Qwen decode expert reads with cached gate/up 
Qwen SSD decode previously waited for every selected expert payload before
starting the MoE. For cache-eligible single-token calls with misses, submit
gate/up and SwiGLU for cached routed slots and the shared expert before the
existing parallel pread batch. Compute missing slots after installation,
then execute the original full down projection and ordered reduction once.

Implementation:
- Add a private before-read callback after missing buffers are reserved and
  selected entries are protected. Existing DeepSeek/GLM callers pass NULL.
- Snapshot the resident gate/up GPU address tables so cache installation can
  update its global tables while the submitted prefix is still executing.
  Retain snapshots through GPU completion, including unretained command
  buffers, and mark cached payloads inflight before dispatch.
- Add a private slot mask to the generic and ordered Q4_K mid kernels.
  Compact the dispatch grid to popcount(mask), then recover original slot
  indices before reading IDs or weights. Preserve shared placement, output
  strides, dot-product order, and the zero-mask full-dispatch behavior.
- Drain submitted work before failure cleanup clears cache entries, and
  restore the caller's command-batch ownership for exact retries.
- Keep all-hit, batched prefill, off-size MTP and overflow staging schedules
  unchanged. No public API, CLI option, or quantization change is introduced.

Regression coverage:
- Extend the sparse-file SSD fixture with partial/cold/warm T1 selections,
  duplicate IDs, optional shared experts, and 10/16 routed slots.
- Compare mid/down/reduce/HC outputs and guards bit for bit with resident
  kernels. Restrict actual pread requests to unique missing expert ranges;
  verify failed-read prefix completion, untouched missing slots, recovery,
  and zero-read warm hits with an unreadable file descriptor.
- Passed the fixture in default, unretained-buffer and Q4_K NR1 modes;
  resident MoE specialization and ordered Q4_K tests; the existing Metal SSD
  expert regression; and real Qwen n-gram failure/recovery tests for prefill,
  decode and MTP with exact logits. Built all application targets.

Measured against 85d37ae on an M1 Max with 32 GiB, Qwen3.8 Q2, automatic
expert cache, ctx 4096, prefill chunk 128, 100 generated tokens, temp 0 and
nothink, using "narrami la storia di roma". Three alternating pairs yielded
decode medians 8.25 -> 8.44 t/s (+2.3%) and prefill 9.98 -> 10.05 t/s.
Generated output was identical. Profiled cache hits, misses, evictions and
12.80 GiB of expert reads were unchanged. Document the observed variability:
these warm-file measurements do not establish a 10% or cross-device gain.
 2026-09-14 17:50:40 +02:00
GiorgioOppo 85d37aec2a Metal: stage only selected Qwen experts for MTP and cache overflow 
Avoid reading all 512 experts when a Qwen layer has a different size from
the persistent cache or a prefill selection exceeds its capacity. The Q2
pack's Q4/MXFP4 MTP layer previously read about 1.29 GiB on each invocation,
even when the router selected only ten experts.

Allocate compact gate/up/down payloads for the distinct selected experts
and build private GPU-address tables indexed by their original IDs. Keep
the existing row/MM/NAX arithmetic, Q2 physical down-row padding and final
reductions unchanged. Preserve the trunk cache and its address tables.
Retain all six staging buffers through GPU completion, including batches
using unretained command buffers, and declare the indirect payloads as
encoder resources without registering nonexistent cache entries.

Use the original three contiguous reads when every expert is selected or
GPU buffer addresses are unavailable. Choose that fallback before I/O;
actual read failures propagate without rereading the complete layer.
Keep the existing conservative memory admission and staging reservation.

Extend the SSD fixture with a test-only pread interceptor, linked through a
separate Metal backend object. It performs real reads, rejects accesses to
unselected expert ranges, checks per-projection byte totals and verifies
zero reads for resident/cache-hit paths. Cover compact cache overflow,
off-size MTP with two/three tokens, full selection of all 512 experts,
failed reads, retries and retention of the original trunk cache.
Update the Qwen documentation to describe selective temporary staging.

Validation on Apple M1 Max, 32 GiB:
- Full build, host Qwen memory test and existing resident MoE exact tests.
- SSD bitwise mid/down/reduce/HC parity and read-range assertions, including
  DS4_METAL_UNRETAINED_COMMAND_BUFFERS=1 and existing Metal SSD regressions.
- Real Q2 n-gram recovery: exact logits after prefill/decode/MTP read errors
  across two SSD sessions.
- Three alternating before/after runs of the Rome prompt, ctx4096, chunk128,
  50 generated tokens, temperature0 and no thinking: median MTP generation
  4.55 -> 7.86 t/s (+72.7%). All generated output matched, with 27 verifier
  cycles and 21 accepted drafts in every MTP run.
- A separate profiling comparison reduced expert pread bytes from
  48.40 -> 12.02 GiB (-75.2%); these include filesystem-cache reads.

The speed result is specific to this short MTP workload. Expert I/O still
finishes before routed computation; no overlap or next-layer prefetch is
introduced, and CUDA/ROCm or M5 NAX execution was not tested here.
 2026-09-14 17:16:37 +02:00
GiorgioOppo e37f18576c Metal: add bounded SSD expert streaming for Qwen3.8 Flash Next 
Allow the existing --ssd-streaming mode to run the Qwen Metal graph without
making every routed expert resident. Keep dense, shared, GDN, attention and
hyper-connection weights mapped; retain the separate disk-only n-gram path.

Reuse the explicit-pread expert cache with 512 expert IDs and top-10 routing.
Dispatch the existing row, grouped-MM and NAX kernels through specialized
expert-address tables, preserving their accumulation and reduction order.
Keep Q2_K physical down-row padding separate from logical activations, and
pass the shared gate/up and down quantization types independently.

Stage complete layers in owned buffers when a batch exceeds cache capacity
or the layer has a different expert size, including the Q2 pack's Q4/MXFP4
MTP layer. Never fall back to mapping routed weights for Qwen. Retain these
buffers through GPU completion, including unretained command buffers.
Validate router IDs, lists, tensor ranges and aliases before dispatch, and
preserve command-batch ownership across read failures and retries.

Size the cache after accounting for static weights, context, recurrent and
speculative state, prefill workspace, staging and runtime headroom. Share
the Qwen estimator between GPU and CPU-only builds, fix its large-context
integer overflow, and check additional sessions and direct-generation
graphs against the same budget. Reject unsupported preload/full-layer
options instead of silently applying another model's policy.

Add sparse-file GPU parity tests for cold/warm caches, eviction, overflow,
mixed expert sizes, invalid input and failed-read recovery. Add host memory
regressions and enable the real-model n-gram recovery test in SSD mode.
Document cache sizing, commands and current I/O/MTP limitations.

Validation: full macOS build; Qwen SSD parity with ordinary and unretained
command buffers; existing Qwen resident decode/Q4 geometry tests; existing
Metal SSD IQ2/Q4/MXFP4 tests; host cache and Qwen memory tests. A real Q2
Metal SSD run completed the Rome prompt with a 50-token generation limit.
The real-model n-gram test also passed prefill, decode and MTP read-failure
recovery with exact logits across two SSD sessions.
 2026-09-14 17:00:10 +02:00
antirez 9139e2ae58 Document and download self-contained Qwen BF16 n-gram releases 
Use one verified GGUF per quantization, remove sidecar setup, and document disk versus resident size. Record native-reader, model, agent/server, steering, fallback, and continued-prefill QA with explicit limits.
 2026-09-14 13:46:19 +02:00
antirez 67bd55c0ec Honor steering in standalone greedy Qwen generation 
Pass the requested vector and scales into the standalone graph, as the session runner already does. The native-BF16 QA exposed the missing load because ordinary generation silently ignored attention steering. Require the enabled diagnostic and retain exact ordinary/MTP text checks for both Q2 and Q4.
 2026-09-14 13:41:49 +02:00
antirez 846428239d Keep short Qwen prefill tails on FP32-query attention 
The native-BF16 Q4 fixture exposed half-query rounding amplified by nearly tied MoE routing scores at a four-row tail. CPU double-precision attention checks confirmed the matrix formula, but using decode arithmetic for at most eight rows keeps this boundary stable without changing large-prefill kernels. Extend kernel and full-logit coverage through the eight-row tail.
 2026-09-14 13:33:34 +02:00
antirez 97ba327538 Read Qwen native BF16 n-grams directly from the GGUF 
Keep the table outside model mappings and Metal residency. Use bounded explicit reads, reject failed lookups before GPU consumption, and remove the runtime sidecar option and paging controls. Preserve Qwen hash and speculative state handling; cover malformed files and exact recovery after prefill, decode, and MTP read failures.
 2026-09-14 13:32:48 +02:00
antirez e68bcc8fab Package Qwen with original BF16 n-grams 2026-09-14 13:00:36 +02:00
Salvatore Sanfilippo 15e88cfe91 Link CUDA regression tests with their shared helpers 
Include the image helper object used by the CUDA visual-attention code in standalone test binaries. Let the engine runtime test reuse the normal core dependencies, excluding its separately instrumented engine object.

Verified all affected targets link on DGX Spark sm_121a; CUDA long-context, Q8 row and DeepSeek 4.1 kernel regressions pass.
 2026-09-14 11:44:47 +02:00
Salvatore Sanfilippo 948307e189 Keep Qwen steering capture compatible with Python 3.9 
Defer annotation evaluation so the new optional PLE argument can be imported by the system Python. Verified steering, MTP, all 48 layer captures, and prompt-only dumps on M5 Max.
 2026-09-14 11:13:31 +02:00
Salvatore Sanfilippo cbaf6cd336 Use explicit Qwen scoring inputs and remove an unused CUDA stub 
Add --ple to the official continuation scorer and use its existing --rendered-prompt option instead of rewriting hardcoded token IDs through an environment variable. Remove the unreferenced multi-shard mapping stub introduced by the PR.
 2026-09-14 11:06:16 +02:00
Salvatore Sanfilippo 923b819abf Harden Qwen checkpoints, speculative decode and tool calls 
Restore matching logits on verifier rewinds, replay stale recurrent state, discard old snapshots on restore, and invalidate partial payload reads. Keep lazy snapshots all-or-nothing and respect small prefill buffers and the unfused diagnostic path.

Preserve schema-declared string arguments. Repair all-row reference dumps and teacher-forced chunking. Add model-backed regressions and fix test-fixture leaks exposed by ASAN.
 2026-09-14 11:06:16 +02:00
Salvatore Sanfilippo efc27d6d86 Keep literal reasoning and tool tags inside Qwen arguments 
Use the structural reasoning delimiter on the server and wait for the parameter delimiter while streaming in ds4-agent. Add server and bytewise agent regressions; both fail before the fixes and pass afterward on M5 Max.
 2026-09-14 10:34:10 +02:00
Salvatore Sanfilippo bab202990e Keep the Qwen Metal graph out of CUDA and ROCm builds 
Reject unsupported Qwen execution modes before warming or sharding the model. Preserve the CPU first-token reference. The ROCm build now links without Metal-only symbols and is warning-free.
 2026-09-14 10:32:26 +02:00
Salvatore Sanfilippo f06d3ebd52 Merge Qwen3.8 Flash Next support from PR #991 
Closes #991. Preserve the original implementation commit and all author credits.
 2026-09-14 10:13:08 +02:00
Dango233 0a21d1abb8 bench: revalidate Metal decode queue against current upstream 2026-09-14 15:37:08 +08:00
Dango233 7a18d09149 Align with upstream DeepSeek V4.1 CUDA support 2026-09-14 15:11:44 +08:00
Salvatore Sanfilippo 6e4c285a92 Record CUDA SSD prefill gains and regression checks 
Document the measured README, fresh and continued prefill gains, exact state and continuation checks, memory bounds and agent regression. Include the small automatic-cache decode cost instead of claiming a decoding speedup.
 2026-09-14 00:13:55 +02:00
Salvatore Sanfilippo e9e1baaff1 Process medium CUDA SSD appends in one layer sweep 
Reuse the carry workspace for medium appends with a substantial final tile. Preserve the existing 2048-row arithmetic partitions instead of visiting every layer twice. Retain the old dispatch for tiny tails and explicitly smaller prefill chunks.

Extend dispatch and mixed-state controls. Full logits, caches, snapshots and next decode remain exact through 60K context and fresh 49K deferred prefill. Together with read-ahead, the 3241-token README prefill improves from about 55 to 98 tokens/s with a 64 GiB cache hint.
 2026-09-14 00:01:08 +02:00
Salvatore Sanfilippo c2c3ce324c Overlap CUDA SSD expert reads with V4.1 prefill 
Read one next layer into reserved slots of the admitted expert cache using two 8 MiB staging buffers. Protect current inputs, publish only completed copies, keep unused read-ahead cold, and join before cancellation or cache/model teardown.

Enable read-ahead from 2K tokens; smaller prefills were slower with full-layer reads. Add eviction, truncated-source, buffered-I/O, cancellation and early-exit tests. Validate exact logits and saved state on the Sparks, including Compute Sanitizer.
 2026-09-14 00:00:54 +02:00
Salvatore Sanfilippo a04f46fa42 DeepSeek v4.1 Flash support for CUDA 
Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.

Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
 2026-09-13 17:32:48 +02:00
ivanfioravantiAndreas ParuseldavidMichael KuebbelerOgilthorp3anemll
 		ccea768827 Add Qwen3.8 Flash Next model support 
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.

Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.

Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.

Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.

Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
 2026-09-13 14:57:29 +02:00
Dango233 9b50495847 DeepSeek V4.1 Flash: queue bounded single-host Metal SSD decode 2026-09-13 11:19:19 +08:00
Salvatore Sanfilippo bd66c40207 DeepSeek v4.1 Flash support for Metal 2026-09-12 11:47:54 +02:00
Salvatore Sanfilippo 6289c51627 Give agent hints a distinct terminal label 
Render the existing hint marker with a white-on-teal badge and a slim left rule. Keep body text, redirected output and the model transcript unchanged.

Test every stream split, unfinished markers, quoted continuations, literal code and both tool syntaxes. Verify terminal colors and UTF-8 at multiple widths, including redraws between fragments.
 2026-09-08 17:26:35 +02:00
Salvatore Sanfilippo 74fbe546ff Add vision client and compaction recovery release checks 
Record the focused M5 Max results, long-prefix image latency, actual Pi tasks across three APIs, and the remaining pre-existing DSpark stop-boundary rebuild. Distinguish this pass from untested release platforms.
 2026-09-08 14:12:56 +02:00
Salvatore Sanfilippo e33a5b2538 Keep agent tasks running across context compaction 
Reserve summary space, recover already-full conversations by summarizing a bounded prefix, and retain every unsummarized token. Reject oversized input without corrupting the current conversation.

Resume unfinished responses in the same assistant message while preserving thinking state and the output budget. Discard incomplete tool calls before compacting and retry without executing partial arguments.

Cover real-model compaction, generated-code correctness, budget accounting, and recovery in a PTY regression; add deterministic boundary tests.
 2026-09-08 14:12:35 +02:00
Salvatore Sanfilippo 81a76585ff Preserve live vision context across server tool calls 
Reuse bounded image embeddings, preserve exact image identities while appending new images, and splice image tokens into authenticated live continuations. Handle image tool results in Responses and Anthropic requests as well as Chat Completions. Normalize request-local image markers for visible replay without trusting text placeholders as image identities.

Add prefix and encoder-cache unit coverage plus real-model image replay and Pi coding-client regressions. Validated GLM 5.3 Flash and DeepSeek Vision-Exp on M5 Max, including 53K/56K prefixes, batched sessions, all three API formats, fresh logit replay, and frontend ASan/UBSan.
 2026-09-08 13:57:09 +02:00
Rowan TrollopeSalvatore SanfilippoClaude Fable 5
 		6546ae5473 agent: add opt-in programming hints 
Adapt teaching-mode PR #391 to /hints on|off. Append the full
instruction once per context and short state changes at safe
assistant boundaries. New and resumed sessions start off;
compaction reinstates the current policy.

Use ordinary Markdown asides and the existing renderer, without
a learner profile, new tool, or persistent preference.

Tested on M5 Max IT with DeepSeek Flash and GLM Flash coding sessions,
prefix reuse, queued changes, compaction, save/switch/new, frontend
and PTY tests, and agent ASAN/UBSAN checks.

Closes #391

Co-authored-by: Salvatore Sanfilippo <antirez@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-09-07 18:16:55 +02:00
antirez f62ca29a30 README: intro section updated. 2026-09-07 13:23:08 +02:00
Salvatore Sanfilippo b6af0adf8c Bound GLM SSD prefill cache priorities so decode can adapt 
The recent-token bonus selects the initial cache contents; it is not a count of observed decode routes. Passing that inflated score into eviction hotness kept the seed effectively pinned after long prefills, starving demand-loaded experts.

Use the existing bounded rank policy without changing the seeded expert list or inference arithmetic. On M5 Max IT, full GLM 5.3 SSD agent decode at 100K context improved from 0.48 to 4.38 t/s. Matched 128-token greedy replays stayed identical, and the Metal SSD cache tests passed under API validation.
 2026-09-07 01:39:49 +02:00
Salvatore Sanfilippo c0a6119f36 Preserve the final byte of in-memory session snapshots 
Reserve a separate byte for the terminator written by fmemopen, without changing the serialized payload length. Add model-free grow/reuse coverage and compare real-model snapshots against file serialization.
 2026-09-06 23:59:05 +02:00
Salvatore Sanfilippo 49ec955c8c Keep ROCm QA builds clean for partial tiles and CPU-only fixtures 
Do not force full unrolling of runtime-bounded IQ2 tail loads. Keep the ROCm graph memory-baseline initialization out of DS4_NO_GPU builds, matching the declaration and allowing the model-free session-state suite to compile.
 2026-09-06 23:41:42 +02:00
Salvatore Sanfilippo d12898f4e2 Initialize the terminal escape buffer for GCC dataflow checks 
Make the empty escape buffer explicit without changing incremental key handling. This removes the maybe-uninitialized warning found by the DGX Spark QA build.
 2026-09-06 23:41:41 +02:00
Salvatore Sanfilippo 78c522e96b Keep DSpark EOS fixture compatible with pre-fix source 
Use a macro for the scheduler policy mock so historical verifier extraction does not fail -Werror on an unused helper. Check the current verifier with sanitizers and confirm that the pre-fix source reproduces the excess committed EOS row.
 2026-09-06 23:06:12 +02:00
Salvatore Sanfilippo 5ad935d7bb Merge PR #996: validate saved quality scores and frontier logits 
Retain the original commit by Donato Capitella. Add backend-independent offline validation for complete scorer coverage and full-vector numerical comparisons without changing inference.

Closes #996
 2026-09-06 23:03:45 +02:00
Salvatore Sanfilippo bff84aec32 Adapt DSpark EOS regression fixture to current scheduler helper 
Mock the existing short-fallback policy referenced by the extracted verifier. Keep scheduler behavior outside this state-ownership fixture, and retain compatibility with pre-fix negative controls.
 2026-09-06 23:03:45 +02:00
Salvatore Sanfilippo 909d6560cb Merge PR #992: ROCm gfx1151 prefill and DSpark improvements 
Preserve all five original commits by Donato Capitella. Resolve the DSpark fallback conflict by retaining the existing Metal/DGX Spark short-proposal behavior and adding the ROCm reference-alignment condition.

Closes #992
 2026-09-06 22:58:22 +02:00
Salvatore Sanfilippo b36e26ed13 Make agent file tools and background shell jobs reliable 
Replace files atomically with metadata and concurrent-change checks. Stream bounded reads with exact continuation, report incomplete searches, preserve literal [upto] edits, and avoid silently truncating user text.

Drain shell output and enforce deadlines independently of model progress. Synchronize observations and cleanup, bound log tails, honor status refresh requests, and filter terminal commands from displayed shell output. Align tool prompts and schemas with actual behavior and vision availability.

Add file, job, schema and PTY regressions. Combined fixes passed frontend tests, ASan/UBSan, ThreadSanitizer and DeepSeek/GLM editing smoke tests on M5 Max IT. Linux-specific metadata preservation still needs native QA.
 2026-09-06 22:50:54 +02:00
Salvatore Sanfilippo db1942148c Fix agent terminal redraws, Unicode input and Markdown rendering 
Buffer complete terminal updates, repaint status without erasing live input, and retain pending redraws across throttling. Preserve fragmented UTF-8, navigation and paste input, and bound tab completion safely.

Use shared Unicode width handling for input, output and queued prompts. Preserve unmatched Markdown delimiters and publish code recoloring as one update. Add focused terminal regression tests.
 2026-09-06 22:50:20 +02:00
Donato Capitella 85a4f0e611 tests: validate complete score tables and frontier logits 
Add two offline standard-library utilities with 30 host tests. Require complete API-covered official-score manifests, finite consistent values and matched denominators; optionally enforce per-case identity. Validate every expected ds4-bench frontier and complete float32 vector, preserving signed zero and reporting drift separately from invalid artifacts.

The existing comparator silently accepts missing rows, duplicate rows and NaN in reproduced controls; the validator rejects them. Saved Flash/Vision 100-case pairs pass unchanged. The frontier tool detects full-vector drift despite matching argmax IDs at all four measured frontiers. Document use and scope without changing inference or introducing model/GPU dependencies.
 2026-09-06 21:37:42 +01:00
Donato Capitella 394865c5d1 rocm: align DSpark target features and retain their rolling window 
Pair the previous target feature with the next seed at their respective positions and construct target KV directly from the projected normalized feature. Merge trusted target captures into the model support window instead of exposing the full prefill or replacing history with each verifier batch. Keep temporary draft rows outside persistent history and reset safely across capture gaps.

Scope the correction to physical gfx1151 independently of diagnostic fast-path overrides. Add 162 CPU cache-range and physical-ring assertions, and document the tested five-draft/confidence-0.7 configuration. Exact-sampling history-gap performance remains unqualified.

Sampled coding continuation improves 14.17/14.38 to 25.10/24.46 tokens/s at 8K/16K. Long coding passes 24/24 checks, sampled coding 7/7, and an actual tool round trip passes. Both 100-case official reports remain identical; the 64K native continuation completes at 22.61 tokens/s.
 2026-09-06 21:26:09 +01:00
Donato Capitella e83b106309 rocm: accelerate gfx1151 DSpark verifier attention projection 
Use independent version-matched hipBLASLt 2379 plans for batched attention-A at verifier widths 2 through 6. Preserve handle-stream ordering, shape/model guards, failure isolation and the original fallback.

Production-shape full-output, input and canary checks pass with documented rounding differences. The 5/6-row operator is about 2.55x faster; native and objective coding checks retain the prefill gain.
 2026-09-06 21:26:09 +01:00
Donato Capitella 1ef9bba608 rocm: select version-matched gfx1151 Lt prefill plans 
Use zero-workspace F32-output hipBLASLt plans 2537/2539 for the qualified DeepSeek projection shapes. Require matching gfx1151, model, library versions and supported descriptors; retain the original fallback.

Warm 16K native prefill improves 262.64 to 301.30 tokens/s. Operator safety and official scoring pass; long-prompt NLL changes +0.65% at default 4K chunks while the four coding tasks pass 24/24 checks.
 2026-09-06 21:26:09 +01:00
Salvatore Sanfilippo 90aa4b560a Document full GLM SSD append gains and validated coding QA 2026-09-06 17:29:02 +02:00
Salvatore Sanfilippo d7ade9fd65 Reuse cached IQ2 experts for small Metal GLM prefills 
Keep the existing MV, SIMDgroup and TensorOps arithmetic while reading selected experts from the bounded cache. Enable this by default for single-host eight-expert IQ2 batches below 256 tokens when a full layer fits; retain the existing larger-prefill and fallback paths.

Test exact intermediate and final outputs at both synthetic and full GLM dimensions, cache eviction, dispatch boundaries and undersized budgets. Full-model logits, coding sessions and MTP snapshot checks pass on M5 Max.
 2026-09-06 17:28:31 +02:00
Salvatore Sanfilippo fb2abb29e4 Align Metal GLM attention scratch for arbitrary context lengths 
Metal API validation aborted on full-attention and indexed-decode scratch sizes that were not multiples of 16 bytes. Pad only the allocation; kernel indexing, precision and arithmetic remain unchanged. Cover all context-length remainders, both cache formats and invalid selected rows in the attention regression.
 2026-09-06 16:29:03 +02:00
Salvatore Sanfilippo 4814f44fed Remove managed-storage notifications from shared Metal buffers 
Shared buffers do not use didModifyRange. Metal API validation rejects these calls; remove them and their now-empty timing metric. Require the SSD primitive test under API validation.
 2026-09-06 15:49:05 +02:00
Salvatore Sanfilippo 31d1992633 Record Metal SSD cache gains and the outstanding full GLM server stall 2026-09-06 15:42:09 +02:00
Salvatore Sanfilippo 437dbcba8e Read full IQ2 GLM layers ahead during large continued prefills 
Keep selected-expert prefill for typed Flash layouts. Generic IQ2 GLM reuses the sequential whole-layer path with one-layer lookahead for continued chunks of at least 256 tokens, without changing prompt chunks or expert arithmetic.

On M5 Max, full GLM 5.3 Q2 at a fixed 35.36 GiB expert cache improves 8K/4K-append prefill from 63.23/61.56 to 90.70/84.31 t/s together with bounded model views. All 154880 logits at both frontiers are byte-identical. The 512-token append improves from 14.64 to 26.01 t/s in the larger-cache test.
 2026-09-06 13:49:51 +02:00
Salvatore Sanfilippo 660e1d4b64 Bound Metal streaming model views without losing auxiliary mappings 
Replace single-range target views just like disjoint layer spans. Preserve separately mapped vision and drafter weights. Explicitly map all indexer normalization weights before GLM compact-cache warmup instead of relying on old layers remaining mapped.

Add repeated mapping and auxiliary-weight checks to the SSD expert regression, and label wrapper allocation counters as cumulative. Full GLM 8K/12K streaming completes with a 61.35 GiB expert cache without the swap growth seen in the old mapping control; Flash full logits remain byte-identical.
 2026-09-06 13:44:00 +02:00
Salvatore Sanfilippo 715ed7a865 Age Metal expert caches from the first routed layer 
Use the same cache clock for typed and generic GLM routes, including models with leading dense layers. Report every selected expert in routing diagnostics. Add an eviction oracle that checks all eight slots and byte-identical output across different cache capacities.
 2026-09-06 13:16:43 +02:00
Salvatore Sanfilippo 91dda0bc88 Let Metal expert caching adapt beyond the built-in preload ranking 
Keep the preload list but bound its initial eviction preference. Fixed-code SSD reads fall from 94.65 to 78.18 GiB; matching Vision Exp continuation scores are unchanged. No routing, precision or arithmetic changes.
 2026-09-06 13:09:45 +02:00
Salvatore Sanfilippo 85837d02e1 Use more of the safe Metal working set for DeepSeek expert caching 2026-09-06 12:19:01 +02:00
Salvatore Sanfilippo edcafe4d57 Reuse GLM router readback during IQ2 expert streaming 2026-09-06 12:09:20 +02:00
Donato Capitella 7a8c1b27c8 dspark: stop verification at the first EOS 
A greedy draft can contain EOS followed by more tokens. The verifier previously retained that suffix even when the returned token vector stopped at EOS, leaving session state ahead of the caller. Bound the validated draft inclusively at its first EOS before verification, capture and commit. Preserve upstream seed accounting, repeated prefix restoration, invalid-token and budget checks, ignored-EOS handling, and the separate exact stochastic helper.

Add a CPU regression executing the actual greedy helper against checked target-state mocks. Unmodified main returns two tokens while advancing three on the interior-EOS probe; the repair returns and advances two. All 7,217 cases and 240,503 assertions pass on the repaired helper, covering positions through the 16-token capacity, partial commits, rollback/replay, fused and ordinary seeds, output/context limits and transmitted TP tokens. No-EOS and ignored-EOS subsets pass on both versions.

This is shared verification control flow without backend arithmetic edits. The host fixture does not establish GPU numerical, public-dispatch or multi-GPU qualification; batches containing a suffix after EOS may become shorter.
 2026-09-06 11:01:26 +01:00
Donato Capitella 64e490bc1a rocm: specialize gfx1151 IQ2 prefill input staging 
Use direct input loads for full IQ2_XXS tiles with width at least 32 on HIP/gfx1151. For partial tiles, initialize the invalid suffix once and reload only the valid prefix at each K stage. Preserve scratch lifetime, barriers and invalid-tail handling; other backend and quantization templates retain their existing path.

Keep small IQ2 and Q2_K templates on their established masked-load path, where the wider load-scheduling change altered compiler accumulation order during development. The specialization changes input staging without changing the selected matrix operations or their source-level accumulation.
 2026-09-06 11:01:25 +01:00
Salvatore Sanfilippo e965352906 Preserve RoPE in full GLM compact prefill 2026-09-06 12:00:03 +02:00
Salvatore Sanfilippo 54fc4f691c Document measured Metal SSD gains and scoped coding-session QA 2026-09-06 03:44:30 +02:00
Salvatore Sanfilippo 6ccc1e7e76 Use the available Metal cache and overlap the first GLM streaming prefill 2026-09-06 03:39:00 +02:00
Salvatore Sanfilippo 22e417286c Keep static Metal streaming weights resident within the runtime budget 2026-09-06 03:39:00 +02:00
Salvatore Sanfilippo d6580a5ffc Let parallel Metal prefill reads run without serial read-ahead 
Batch expert loads already submit all requested tensors to the pread worker pool. Avoid reading those pages serially first; keep decode read-ahead unchanged. M5 Max GLM Q4 initial/continued prefill improved from 73/77 to 93/97 tokens/s in the isolated comparison, with identical full frontier logits.
 2026-09-06 02:54:46 +02:00
Salvatore Sanfilippo aa8fef4a16 Require real server coding sessions and prefix reuse checks in QA 2026-09-06 01:59:27 +02:00
Salvatore Sanfilippo 46e46e7d7c Extend tool-call QA with speculative boundaries and real coding clients 2026-09-06 01:50:26 +02:00
Salvatore Sanfilippo 42da390e2f Run frontend regressions without loading a model 2026-09-06 01:37:21 +02:00
Salvatore Sanfilippo 6eac69e0aa Bound tool recovery by the request budget and explicit stops 2026-09-06 01:34:57 +02:00
Salvatore Sanfilippo 930ab73279 Keep valid opportunistic drafts across tool sampling boundaries 2026-09-06 01:27:29 +02:00
Salvatore Sanfilippo e1711c21d9 Report tool observation errors without compacting the conversation 
Distinguish image/rendering failures from results that exceed the context budget. Preserve the transcript and return the original error. Add an invalid-image regression and run the agent suite under ASan and UBSan.
 2026-09-06 01:18:40 +02:00
Salvatore Sanfilippo d108ae4e8d Preserve literal tool data across agent and server rendering 
Do not HTML-unescape argument bodies. Share a narrow, reversible escape for each body closing delimiter, including its literal escaped spelling. Keep streaming fragments intact across those escapes. Preserve protocol-looking text inside GLM agent arguments until the actual argument terminator. Add round-trip, streaming and agent regressions.
 2026-09-06 01:14:31 +02:00
Salvatore Sanfilippo fc6414cdfc Keep literal tool and thinking markers inside argument data 
Use the incremental grammar state to detect completed calls, and skip argument/key bodies when searching final control markers. Validate complete calls before recovering unclosed reasoning. Cover DeepSeek and GLM arguments containing both tool endings and thinking tags without truncating or losing the call.
 2026-09-06 01:07:50 +02:00
Salvatore Sanfilippo 759dd7cd69 Do not repair truncated tool actions or exceed output limits 
Only repair an omitted DeepSeek wrapper around already closed invocations. Never supply a missing argument or invocation terminator. Preserve length finishes and do not retry beyond the client output budget. Log the actual stop cause, token ID and available budget when a tool call is incomplete.
 2026-09-06 01:03:59 +02:00
Salvatore Sanfilippo 5b3cc8bfd1 Respect GLM tool syntax and speculative server boundaries 
Track GLM function names and argument bodies with the existing incremental recognizer. Keep reasoning quotes out of syntax sampling. Discard unconsumed speculative tokens at tool and control stops, and resample after a structural/payload boundary with restored logits. Rebuild invalid recurrent prefixes with their image conditioning. Add GLM fragment and byte-boundary coverage.
 2026-09-06 00:58:00 +02:00
Salvatore Sanfilippo d077fa67e5 Keep split tool delimiters in the server sampling tracker 
Retain a lone opening angle bracket until the next token, without forcing code or HTML greedy. Previously the tracker consumed it and missed a parameter close split across tokens. Add byte-by-byte regressions for all three DeepSeek spellings and string/JSON bodies; all twelve failing checks now pass.
 2026-09-06 00:50:14 +02:00
Salvatore Sanfilippo aa1e71cf03 Document DGX Spark DSpark speed and focused QA 2026-09-06 00:44:53 +02:00
Salvatore Sanfilippo 2803820dc5 Use aligned Q8 vector kernels for small Spark batches 
Extend the existing resident DeepSeek Q8 dispatch through eight rows on GB10. Keep GLM and other devices on their current paths. Check each batch against separate vector calls, including reused scratch and graph replay. The combined DSpark coding workload improves beyond the aligned expert dispatch; ordinary 100-case continuation scores remain unchanged.
 2026-09-06 00:42:59 +02:00
Salvatore Sanfilippo 45f54e9d11 Use aligned expert vector kernels for small Spark batches 
Reuse the existing fused IQ2 gate/up and Q2 down paths for up to eight resident DeepSeek rows on GB10. Keep GLM, streaming and multi-GPU dispatch unchanged. Test overlapping experts, distinct gate/up weights, invalid IDs and exact agreement with separate rows; CUDA memory, race and synchronization checks pass.
 2026-09-06 00:40:24 +02:00
Salvatore Sanfilippo 558cd34311 Restore text-only agent tool results for DeepSeek 
Use each model family’s normal message formatter when a multimodal observation contains no images. Non-vision Flash and PRO no longer reject every tool result as an apparent context overflow, and text results retain their native wrapper and escaping. Add model-free Flash/PRO/GLM regression coverage; CUDA coding/edit tasks pass with ordinary opportunistic and exact DSpark sampling.
 2026-09-06 00:36:26 +02:00
Salvatore Sanfilippo f6c169f825 Retain all five CUDA DSpark partial prefixes 
Save the fifth compressor frontier so accepting five of six verified rows does not replay the last token. Size verifier test sessions to their prompts and continuations instead of allocating two unnecessary 32K contexts. The prefix and serial-target oracles pass across continued contexts through 8K.
 2026-09-05 23:48:13 +02:00
Salvatore Sanfilippo df80a87da1 Batch DSpark seed tokens on resident DGX Spark 
Use the existing seed verifier for resident IQ2/Q2 Spark sessions. Avoid duplicate declined proposals, check short drafts after normal seed decode, and exclude the seed from scheduler acceptance. Pause longer for unproductive prose, but retry sooner after useful long drafts. Keep exact sampling, streaming and other hardware defaults unchanged.
 2026-09-05 23:48:13 +02:00
Salvatore Sanfilippo 9ab705347c Record long GLM quality controls and Spark decode gains 
Replace the expired-credential note with completed Z.AI FP8 long-context comparisons. Record matched Flash-0731 timing medians, exact before/after quality, scratch graph tests and warning-free backend builds.
 2026-09-05 20:55:01 +02:00
Salvatore Sanfilippo 4a14eb83ba Remove unused CUDA and ROCm Q8 matvec kernels 
Delete the two obsolete kernels that quantized activations separately for every output row. Neither has callers or a testing role; the active prequantized and reference paths remain unchanged. CUDA sm121a and ROCm gfx1151 builds pass without warnings.
 2026-09-05 20:53:22 +02:00
Salvatore Sanfilippo b42682f12f Reuse aligned Q8 scratch during CUDA decoding 
Avoid per-call pool allocations in dense and paired aligned Q8 kernels. Reuse the existing device-checked buffer, retaining the pool fallback for missing or undersized scratch.

Three interleaved Spark Flash Q2 runs improve 2K/4K/8K steady decode from 17.90/15.23/15.01 to 19.25/16.22/15.97 t/s. Prefill is unchanged and full prefill and post-decode logits match exactly. Add scratch fallback and graph replay tests.
 2026-09-05 20:36:08 +02:00
Salvatore Sanfilippo 189ff0b302 Add long GLM FP8 references with reasoning-aware scoring 2026-09-05 20:26:22 +02:00
Salvatore Sanfilippo eb847ac568 Fix official-score logprob parsing for null API responses 2026-09-05 20:03:38 +02:00
Salvatore Sanfilippo f4d03f6cf9 Record backend QA results and preserve failure diagnostics 
Keep failed DSpark command diagnostics and report snapshot prompt/context settings. Extend release checks for attention boundaries, usable ROCm memory, repeated vision embeddings and bounded speculative replay.

Record focused GLM and Flash quality, SSD, batching and performance results, including the cold-prefill cost and remaining long-context API and release-matrix gaps.
 2026-09-05 19:42:02 +02:00
Salvatore Sanfilippo 96f547a245 Remove unused ROCm kernels and helpers 
Drop unreferenced Q8, attention, MoE and GLM projection implementations while retaining active fallbacks, diagnostics and performance tests.
 2026-09-05 19:42:01 +02:00
Salvatore Sanfilippo 0e9cc2d43e Fix GLM attention masking and reductions across GPU backends 
Use the model-defined 2051-token dense boundary, handle padded selections, and synchronize reused reduction storage in Metal, CUDA and ROCm, including vision normalization.

Enable bounded ROCm selected-attention GEMM for GLM 5.3 without RoPE. Add independent attention references, boundary tests and finite, repeatable vision checks. Focused Strix continued prefill improves from about 21 to 68 tokens per second; stored official scores are unchanged.
 2026-09-05 19:42:01 +02:00
Salvatore Sanfilippo 9d9e1296d2 Budget ROCm allocations against usable pinned memory 
Exclude CMA pages from host availability, guard pinned and integrated-GPU allocations, and reserve space for the OS and required runtime buffers. Defer optional Q8 expansion and avoid wasting large model arenas.

Add memory parsing and allocation-admission tests. Focused Strix checks cover matched Flash/DSpark, resident GLM and automatic, oversized and one-expert SSD cache budgets.
 2026-09-05 19:41:28 +02:00
Salvatore Sanfilippo ade4d623f9 Fix Linux builds after Metal TP changes 
Guard Apple-only TP hooks and verbs-only timing variables. Link standalone CUDA and ROCm tests with their image and MMQ dependencies.
 2026-09-05 19:40:33 +02:00
Salvatore Sanfilippo 6c25bba129 Put common getting-started instructions in the README 
Keep platform-specific details in their guides, update the cross-links, and remove the release QA matrix link from the README.
 2026-09-05 19:38:18 +02:00
Salvatore Sanfilippo 2f0bc1e3bf Split README into platform setup and focused reference guides 
Keep the introduction and everyday usage in the README, with setup guides near the top and detailed references at the end. Document Ada/L40S support explicitly and replace stale model, memory, caching and performance claims.

Retain the root Strix Halo page as a link to its new guide.
 2026-09-05 14:29:52 +02:00
Salvatore Sanfilippo 233eeb85a7 Fix session rollback, TP restore and GLM memory accounting 
Restore GLM recurrent and pooled-indexer state after rejected speculation. Invalidate unsafe DeepSeek rewinds and rebuild retained agent prefixes with their images before decoding again.

Rebuild saved TP token prefixes on both ranks. Give GLM speculative cycles an explicit command and shared token limit, with protocol version 10 requiring peers to update together.

Account for live and planned Metal GLM sessions when reserving graph and streaming memory. Add model-free state and protocol tests plus a small GPU rollback regression.
 2026-09-05 14:29:18 +02:00
Salvatore Sanfilippo 932896fd0a Fix standalone Metal test linkage 
Link the KDA and MXFP4 Metal tests with the image helpers required by the Metal backend.
 2026-09-05 14:28:35 +02:00
Salvatore Sanfilippo bbec2006f8 Finish deferred Metal TP residuals before mixed-batch output heads 
The mixed prefill/decode path omitted the final deferred residual expansion used by ordinary TP decode. Finish each session head before another session can reuse transient exchange buffers.

The existing full-logit oracle reproduced a 23.7409 maximum difference on the baseline despite identical argmax tokens. Two, four and six MXFP4 TP sessions now match serial logits exactly, including continued prefill; mixed Q2/Q4 also passes with six sessions in TP and on one host. Document the required regression checks.
 2026-09-05 03:25:06 -07:00
Salvatore Sanfilippo 5806d0994d Batch Metal TP DSpark experts and reuse MXFP4 prefill activations 
Use the M5 seed schedule under TP and evaluate verifier rows together instead of running each row separately. Reuse the existing static MXFP4 kernels with row-aware addressing, and enable packed MXFP4 activations for two-rank M5 prefill.

Keep streaming, exact-sampling scheduling and unmeasured device defaults unchanged. Add synthetic ownership/static-shape checks and a physical TP speculative oracle. Record focused quality, speed comparisons and transport caveats in the QA guide.
 2026-09-05 03:24:49 -07:00
Salvatore Sanfilippo be76012927 Batch confident DSpark seeds on M5 and back off on weak drafts 2026-09-04 18:52:56 -07:00
Salvatore Sanfilippo 8d309ab062 Capture partial cache frontiers for six-token DSpark verification 2026-09-04 18:07:16 -07:00
Salvatore Sanfilippo 8fcd61d6bf Fix Metal scratch size for double-buffered dense kernels 2026-09-04 17:51:24 -07:00
Salvatore Sanfilippo 2ea3839ef3 Reuse routed activations in Metal tensor prefill 2026-09-04 17:33:17 -07:00
Salvatore Sanfilippo 29aa7267af Extend ds4-eval with hard capability suite 2026-09-03 15:28:24 -07:00
Mikhail Korobov 469ed2f4b5 Merge upstream/main: fold multimodal live-KV hardening into the reuse probe 
Upstream 98b9135 ('server: reuse exact multimodal live KV safely') and
b0982a1 ('Harden multimodal cache fallback') restructured the execution
ladder this PR replaced with slot_probe_reuse().  Resolution keeps the
probe architecture and moves the new safety conditions into it, so
execution and routing still cannot disagree:

- The live_vision_match gate (image fingerprints are invisible to text/
  token prefixes) is now checked once at the top of
  slot_probe_reuse_locked(); job_slot_score inherits it, subsuming
  upstream's -1 vision guard.
- The GLM memory-rewind materialization re-validates the frontier after
  ds4_session_rewind() (common prefix lands on the rewind target and the
  vision state still matches) and falls back to disk cache / cold prefill
  when validation fails, as upstream's ladder did.
- Upstream's ladder reorder (memory-text before disk cache) is subsumed
  by the probe's tier order; the deleted ladder helpers stay deleted.
- Tests from both sides kept: the slot probe/routing tests and upstream's
  test_multimodal_prefill_resume_frontier.
 2026-09-03 12:27:00 +05:00
Salvatore Sanfilippo b0a147a7fb Add conversation prefix conditioning 2026-09-03 01:25:25 +02:00
antirez 5d41a80fd2 Remove dead Metal TP experiments 2026-09-02 19:39:22 +02:00
antirez c185c7aabf Fix Metal TP ARC cleanup warnings 2026-09-02 15:12:29 +02:00
antirez 4886c56815 Merge remote-tracking branch 'macm5us/mac-tp-1-sep-2026' 2026-09-02 15:10:31 +02:00
Salvatore SanfilippoandClaude Fable 5.1 e0b8c8fdf9 Remove dead experiments, keep diagnostics 
Dropped the measured-no-benefit experiments: DS4_TP_FAKE_BIG_GATE (timing
diagnostic with garbage results), DS4_TP_KICK_SETTLE_US, the FFN sub-gate
row halves (DS4_TP_FFN_SUBGATE, byte-exact but net-negative), the fused
DSpark seed batch under TP (DS4_DSPARK_SEED_BATCH_TP, slower), and
DS4_METAL_MM_ID_MIN_ROWS (grouped GEMM below 32 rows, slower).  Kept:
ds4_gpu_tp_big_gate_kick_flush (used by the attention sub-gate option),
the worker-side prefix commit hardening, and all env-gated diagnostics
(DS4_METAL_CB_TIMES, DS4_TP_BIG_GATE_DEBUG, DS4_DSPARK_CYCLE_TRACE,
DS4_BENCH_SPEC_TRACE, DS4_DSPARK_EXPERT_OVERLAP, batch-gate profile,
tp_ffn_recv/tp_ffn_sum dumps).

Validated: frontier hashes e45a0e5953cb0e64 (2048) and 8f0470ff68881053
(128) unchanged; DSpark verify window ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 06:04:16 -07:00
Salvatore SanfilippoandClaude Fable 5.1 b941e98aad README: measured DSpark numbers on a single M5 Max with the resident Q2 model 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 05:23:31 -07:00
Salvatore SanfilippoandClaude Fable 5.1 cffcd64ae9 DSpark diagnostics: DS4_DSPARK_EXPERT_OVERLAP (distinct experts per verify block), DS4_METAL_MM_ID_MIN_ROWS (grouped-GEMM row threshold experiment), cycle trace requires stats timing 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 05:22:28 -07:00
Salvatore SanfilippoandClaude Fable 5.1 7fc640c1bb README: DSpark under two-Mac TP is slower than plain decode 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 05:17:19 -07:00
Salvatore SanfilippoandClaude Fable 5.1 b65ad7dfbc Metal TP DSpark: pre-posted RDMA verify window, seed-batch under TP, tracing 
- ds4_tp.c: ds4_tp_batch_block_begin/end arm a receive window for a whole
  speculative verify block (one batch gate per layer): the decode window is
  drained once, the first layer's receives are posted behind a single
  control-byte barrier, and every gate posts the next layer's receives
  before sending its own rows, so no send can reach an unposted queue and
  no per-gate TCP handshake is needed (the old path cost two round trips
  per gate).  Both ranks call it from the verify path (ds4.c, via the TP
  context stored at ds4_engine_tp_bind).  DS4_TP_DISABLE_VERIFY_WINDOW=1
  keeps the old per-gate protocol.  -4 to -5 ms per verify block.
- ds4.c: the worker's prefix commit no longer tries to preserve a DSpark
  capture it never has (it fails with DS4_DSPARK_SEED_BATCH set); the fused
  seed batch can be enabled under TP with DS4_DSPARK_SEED_BATCH_TP=1
  (experiment: on this pair it verifies more rows per cycle and is slower).
- Diagnostics: DS4_DSPARK_CYCLE_TRACE=1 (per-cycle propose/verify/commit
  timing), DS4_BENCH_SPEC_TRACE=1 (bench per-call cost by accepted count),
  DS4_TP_GATE_PROFILE now also reports verify-block batch gates (arrival
  wait, exchange, release per gate).

Measured on the M5 Max pair (README prompt, 128 context): a verify block
costs ~27 ms + ~5.7 ms per draft row of GPU time (each row re-streams its
own routed experts at bandwidth) plus ~3 ms of gates; DSpark decode stays
at ~35 t/s vs ~48 t/s plain on prose (24-32% draft acceptance).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 05:15:45 -07:00
Salvatore SanfilippoandClaude Fable 5.1 daa0324569 README: two-Mac prefill numbers after the TensorOps routed GEMM and queue keepalive 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 04:41:57 -07:00
Salvatore SanfilippoandClaude Fable 5.1 a06a1bed27 Metal TP prefill: pipelined big-gate kicks (opt-in FFN sub-gate), diagnostics 
- ds4_gpu_tp_big_gate_kick_flush: a kick that ends the command buffer so
  the shared-event signal is published while the caller keeps encoding.
  The service thread then waits for that buffer's completion before it
  reads the payload: the event fires when the GPU is done, but the stores
  only land for the CPU at buffer completion (measured: the first ~10 MB
  of the second half's rows were stale without the wait).  The attention
  sub-gate pipeline (DS4_TP_SUBGATE_PIPELINE=1) now uses it too.
- FFN sub-gate (DS4_TP_FFN_SUBGATE=1, off by default): the routed partial
  is produced in two row halves with this rank's shared-expert rows folded
  per half before each kick, so the exchanged rows are what the peer adds.
  Byte-identical to the single gate but net-negative on the M5 Max pair
  (781 vs ~850 t/s at 2048 tokens: two 1024-row routed calls cost ~27%
  more kernel time than one 2048-row call).
- Diagnostics: DS4_TP_BIG_GATE_DEBUG=2 logs every big gate (period since
  the previous release, handshake, transfer); DS4_TP_FAKE_BIG_GATE=1 skips
  the exchange for a compute-only timing bound (results are garbage);
  DS4_TP_KICK_SETTLE_US adds a delay after the kick-buffer wait;
  tp_ffn_recv / tp_ffn_sum tensor dumps after the FFN all-reduce.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 04:40:58 -07:00
Salvatore SanfilippoandClaude Fable 5.1 8cbdd7624f Metal prefill: expand the compressor position table with one tiled-row kernel 
ds4_gpu_compressor_store_batch_tensor and the score-with-APE path used to
encode one f16->f32 copy per ratio-row segment (n_tokens/4 single-threadgroup
encoders per layer: 21.5k of the 25k encoders of a 2048-token prefill).  A
tiled-row kernel (kernel_cpy_tile_rows_f16_f32 / _f32_f32) writes the whole
batch in one dispatch with identical values.  README: updated two-Mac numbers
and the new opt-out env vars.

Two-Mac TP prefill, byte-exact vs the TensorOps-MoE hashes: 2048 tokens
785 -> 830 t/s, 128 tokens 242 -> 253 t/s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 04:13:29 -07:00
Salvatore SanfilippoandClaude Fable 5.1 22402728c3 Metal prefill: MXFP4 routed experts on the TensorOps (nax) mm_id kernel 
The routed-expert prefill GEMMs for MXFP4 experts now use the existing
kernel_mul_mm_id_mpp TensorOps kernel (new instantiations
kernel_mul_mm_id_mxfp4_f32_mpp / _f16_mpp), as IQ2_XXS and Q2_K experts
already did.  Same half weights x half activations with f32 accumulation
as the simdgroup kernel; the accumulation order differs, so frontier
logits are no longer byte-identical to the previous kernel (2048-token
prefill on the two-Mac TP pair: mean |dlogit| 0.17, max 1.48, same
argmax, 9/10 top-10 overlap, KL 1e-4).  DS4_METAL_DISABLE_MXFP4_MM_ID_MPP=1
restores the simdgroup kernels.

Two-Mac TP prefill: 2048 tokens 673 -> 785 t/s, 128 tokens 164 -> 242 t/s.
Decode unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 04:07:56 -07:00
Salvatore SanfilippoandClaude Fable 5.1 e2c39074f4 Metal TP prefill: staged windowed RDMA big gates, main-queue idle keepalive 
- ds4_tp.c: the prefill big-gate exchange now moves the partial through the
  slab batch regions in windows of up to 256 16 KB messages, receives posted
  in chains of 64 behind a one-byte control barrier, sends in chains of 64
  with at most four signaled chains in flight (32 MB in 4-6 ms instead of
  ~26 ms).  Queue pairs ask for 1024/1024 work requests (fallback 256/64)
  and the device limits are printed at open.  Payload-buffer registration
  stays opt-in (DS4_TP_RDMA_REG_PAYLOAD): Apple's verbs cap a memory region
  at 16 MB and registering Metal buffers gave LOC_PROT_ERR completions.
  DS4_TP_BIG_GATE_DEBUG=1 prints handshake/transfer times every 32 gates.
- ds4_metal.m: a command buffer submitted after the engine's queue idled
  for ~3 s starts 600-800 ms late on M5 Max (measured; other queues do not
  prevent it).  A main-queue keepalive submits one 1 KB kernel per second
  (DS4_METAL_DISABLE_QUEUE_KEEPALIVE=1 opts out) and the sharded load pays
  the first submission up front (ds4_gpu_warm_command_queue).  The big-gate
  arrival spin sleep-polls (DS4_TP_BIG_GATE_SPIN_US).  Diagnostics: blit
  passes in the encoder timeline, DS4_METAL_CB_TIMES prints encode/commit/
  GPU-span per owned command buffer with a backtrace on stalls.

Two-Mac TP prefill, byte-exact frontier logits: 2048 tokens 565 -> 673 t/s,
128 tokens 93 -> 164 t/s; decode unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 04:02:10 -07:00
antirezandClaude Fable 5.1 1cb18a2c2b README: DeepSeek V4 Flash tensor parallelism across two MacBooks (setup, commands, measured rates, knobs) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 03:13:44 -07:00
antirezandClaude Fable 5.1 399e48a73b Metal TP: attention split count back to 32; opt-in rank-speed lane bias 
The 8-group attention split adopted after the sweep saved 1 us in the
attention kernels but made the layer's later concurrent FFN block ~50 us
slower (114 -> 162 us; cause not understood, measured by toggling
DS4_METAL_FLASH_NWG on one build), costing ~4 t/s.  Back to 32 groups;
the env override stays for experiments.  With it the default output is
byte-identical to the pre-session reference again, which also validates
the kv-task fold and the two-rank router-select fusion against it.

The two-rank router-select branch no longer sets router_only_done (that
enabled the single-machine parallel FFN path and computed the whole
shared expert on the coordinator); it only skips the plain router matvec
and the standalone select.

Rank-speed lane bias, opt-in (DS4_TP_LANE_BIAS=1): the gate service
probes the peer's RDMA arrival while it spins on its own flag, giving a
signed per-gate skew (peer early > 0) without any protocol change; the
coordinator turns the FFN-gate skew EWMA into a lane bias (default 19
lanes per us of skew, smoothed, +-shared_dim/2, 64-lane steps), carries
it in the EVAL command's reserved word, and both ranks add it to the
shared-expert split.  The logits-half frame carries one extra float (the
sender's FFN-gate wait) for future use.  Measured on this pair: skew 2-5
us, 64-128 lanes moved, no token-rate change (48.0 vs 48.2, 47.0 vs
46.4), so it stays off by default because it makes the output depend on
timing.  DS4_TP_LANE_BIAS_GAIN and DS4_TP_LANE_BIAS_DEBUG for experiments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 02:25:52 -07:00
antirezandClaude Fable 5.1 5024711a7d Metal TP: kv norm task concurrent with q_b; fused router select for two ranks 
The fused q/kv norm + KV RoPE + FP8 store dispatch ran its two tasks as
two threadgroups and cost 13 us per layer, most of it the serial kv task.
The kernel now takes a task mask; on the two-rank decode path the q task
runs alone (3.8 us) and the kv task is deferred and encoded in a
concurrent encoder together with the q_b projection, whose 17.8 MB stream
hides it completely (the section measures 35.8 us, q_b alone 36.6).  The
KV staging and the raw-only attention path flush a still-pending task
before reading the cache, so every path sees the row.  Same arithmetic,
byte-identical output.  DS4_METAL_DISABLE_KV_NORM_DEFER=1 opts out.

The existing fused router projection + exact select kernel (128
threadgroups, last-arriving threadgroup selects) was gated to
single-machine runs; enable it for two ranks, where it replaces the f16
router matvec (7.6 us) + select (6.0) with one 11.1 us dispatch and the
shared expert keeps running in the concurrent FFN encoder.  Byte-identical.
DS4_METAL_DISABLE_M5_ROUTER_PROJECT_SELECT_FUSE=1 opts out.

Validated on the M5 Max pair: identical text with every toggle at ctx 128
(256 tokens) and ctx 2050 (partial pad block), prefill logits unchanged;
attention-side command buffer 226.1 -> 214.7 us of GPU compute per layer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 02:06:00 -07:00
antirezandClaude Fable 5.1 aef6daebbd Metal TP: adaptive flash split count, slice-parallel HC step as experiment 
The vec attention kernel's split-K work group count is a pipeline
constant; make it adaptive (8 groups per head for <= 256 keys, else 32;
DS4_METAL_FLASH_NWG overrides).  Measured at 128 keys: 8 groups -1 us
net, 4 groups +9 us.

Add kernel_dsv4_hc_expand4_mix_norm_slices_f16: the HC expand + gate
matvec + collapse + input norm as 64 slice threadgroups with fixed-order
partial sums and a last-arriver tail, no spin barrier.  Measured 16.7 us
against the six-threadgroup cluster kernel's 14.5 (the cluster kernel is
launch + serial tail bound, not barrier bound), so it stays behind
DS4_METAL_HC_EXPAND_SLICES=1 and the cluster kernel remains the default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-02 01:46:49 -07:00
antirezandClaude Fable 5.1 313bebc639 TP RDMA: warm-up round with posted-receive barrier, queue pair recreation, diagnostics 
Apple's Thunderbolt RDMA (UC queue pairs, two-sided SEND/RECV) has a
failure mode UC never reports: a send that reaches a queue pair with no
receive posted does not merely get lost, that direction of the pair
stops delivering for good (the pair then reads RTR on query and later
sends never complete).  Runs used to discover this only at the first
bulk prefill round, as "timeout waiting for bulk RDMA round (0/64
recvs, send=0)", typically on the first run after a role swap.

After the ready barrier both sides now run a warm-up round: post one
full-size receive, exchange a "receives posted" control frame
(DS4_TP_FRAME_RDMA_POSTED), then resend a tagged 16 KB message until the
peer confirms receipt over the control channel (DS4_TP_FRAME_RDMA_WARM,
100 ms per attempt).  A failed warm-up recreates the queue pair and
completion queue (new number, new PSN), redoes the info exchange and
warm-up, up to four setups; both sides take the same path because the
status exchange gives them the same outcome.  Bulk-round receive
completions check the byte length, and a bulk timeout reports the queue
pair state and chunk layout.  DS4_TP_DISABLE_RDMA_WARMUP=1 skips the
round.

Measured on the M5 Max pair: with the posted barrier the warm-up passes
on the first attempt in every run (10/10); without the barrier the
warm-up itself raced the peer's send and poisoned pairs.  The bulk
round keeps its pre-existing race (a peer that finished the previous
round earlier can send before our receives are posted); a control-frame
barrier there collides with the eval protocol, so the fix is to pre-post
the next round's receives into a second staging area before sending the
current round.  Noted in misc/MAC_TP_SPEED.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-01 20:29:58 -07:00
antirezandClaude Fable 5.1 23bbbf36e3 Metal TP: touch-table helper, four-row Q8 matvec experiment knob, fold diagnostics 
Factor the one-dispatch touch-table encode out of the gate prefetch
(ds4_gpu_encode_touch_table).  Add kernel_mul_mv_q8_0_f32_r4 and its
flag-folded twin (four rows per threadgroup, bit-exact with the two-row
kernels) behind DS4_METAL_Q8_MV_NR0=4 for the pass-through dispatch
sites; measured slower on the M5 Max pair (48.1 vs 49.3, 45.1 vs 48.4
t/s), so the default stays at two rows.  DS4_TP_FOLD_DEBUG=1 prints the
first flag-fold request/take decisions.

An attention-window prefetch (touching this rank's out_a half in a
concurrent encoder with the flash-attention vec kernel) was measured and
not kept: the projection got 10 us faster but the latency-bound vec
kernel slowed by the touch's duration under saturated DRAM, token rate
unchanged.  Notes in misc/MAC_TP_SPEED.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-01 19:10:31 -07:00
antirezandClaude Fable 5.1 75787d9bb9 Metal TP: fold the gate flag into the payload producers; one-dispatch prefetch 
The checked poll-gate flag was its own kernel after each partial (4-7 us
per gate).  The payload producers now publish it themselves from the
last-arriving threadgroup: kernel_dsv4_add2_f32_tp_flag_checked for the
FFN gate (local shared+routed sum into the slab slot) and
kernel_dsv4_mul_mv_q8_0_f32_tp_flag_checked for the attention gate (the
attention-output K-slice matvec).  Each threadgroup adds the integer sum
of the words it stored to a per-slot accumulator, fences, then arrives;
the last arriver exchanges the accumulator, resets both counters, and
stores the checksum and the flag.  Integer sums are order independent,
so the checksum equals the standalone kernel's over the same payload and
the stored values are the plain kernels' values: text byte-identical.
ds4_gpu_tp_gate_encode skips its flag kernel when the producer
prepublished the sequence number; ds4_gpu_add_tensor_tp_flag and
ds4_gpu_tp_flag_fold_request (taken by the K-slice matvec when its
output is the slot) are the host entry points.  DS4_TP_DISABLE_FLAG_FOLD=1
opts out.  Timeline: fused add+flag 3.2 us vs add 2.1 + flag 4.0.

Gate-time prefetch was one dispatch per plan range (up to 7) at ~3 us
fixed cost each, 16-22 us per gate and exposed whenever the peer's
partial was already there.  It is now one dispatch over a table of up to
8 bindless views (gpuAddress + useResources), threads assigned to ranges
by cumulative line counts (kernel_touch_u8_stride_table).  The
attention-gate plan lists the other shared half last for the dynamic
shared split.  DS4_TP_PREFETCH_MARGIN_US / DS4_TP_PREFETCH_BYTES_PER_US
override the sizing constants for experiments.

Measured on two M5 Max over TB5 RDMA (README prompt, 256 tokens,
interleaved pairs): 47.35 vs 46.78 and 46.70 vs 45.86 t/s with the folds
off; eight repeated default runs identical in text and frontier logits.
Encoder timeline: attention-side command buffer 233.5 -> 226.1 us,
FFN-side 173.1 -> 168.6 us of GPU compute.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-01 18:48:54 -07:00
antirezandClaude Fable 5.1 c69e57fe91 Metal TP: GPU-decided shared-expert lane split for rank balancing 
Routed experts are owned by expert id, so each rank streams 0..6 of the
6 selected experts per token while the shared expert was split in fixed
halves; the rank owning more selected experts was the critical path of
the layer and the other rank idled at the gate.  The new split kernels
(kernel_dsv4_shared_gate_up_swiglu_q8_0_split and
kernel_dsv4_shared_down_q8_0_split) read the selected ids and take a
complementary lane range per rank sized to balance bytes: lanes0 =
shared_dim/2 + (n1 - n0) * shift * shared_dim with shift = routed expert
bytes / (2 * shared expert bytes), rounded to 64 lanes.  On the V4 Flash
MXFP4/Q8 file one extra routed expert moves half of the shared expert to
the other rank.  A rank with zero lanes writes a zero partial, so the
local routed+shared add is unchanged.  The kernels keep the args in
constant memory with explicit row/K counts (a by-value args copy cost
+9.8 us per layer in the matvec bodies); with shift 0 they are bit-exact
with the static halves (+1.9 us per layer).  The attention-gate prefetch
plan lists the other shared half last so a rank with a long measured
wait streams it in.

Default for two-rank Metal TP decode.  DS4_TP_STATIC_SHARED_SPLIT=1
restores the fixed halves; DS4_TP_SHARED_SPLIT_SHIFT_Q16 overrides the
shift (0 = static halves through the split kernels, for exactness
checks).  Measured on two M5 Max over TB5 RDMA (README prompt, 256
tokens, interleaved pairs): ctx 128 46.6-47.4 vs 45.2-46.7 t/s, ctx 2048
45.6-46.2 vs 45.1 t/s; both ranks wait less at FFN gates.  Output is
deterministic and differs from the static split only by floating-point
grouping of the partial sums.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-01 18:19:11 -07:00
antirezandClaude Fable 5.1 6cfc278706 TP: RDMA gate post/peer-wait profile, MXFP4 MoE microbench 
- ds4_tp.c: under DS4_TP_GATE_PROFILE print per gate kind the RDMA post cost
  and the wait for the peer's partial (the attention gate sits at the
  ~11 us wire floor; FFN-gate averages include prefill batch gates).
- misc/mxfp4_moe_bench.m: standalone microbench of the exact-shape TP MXFP4
  pair-SwiGLU and sum6 kernels on rotating expert copies with a warm-up
  pass; COMPACT=1 measures the unowned-slot launch cost.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-01 17:32:46 -07:00
antirezandClaude Fable 5.1 1a976b7dbf Metal TP: poll gates, checksum-verified flags, HC expand fusion, gate prefetch 
Two-Mac DeepSeek V4 Flash tensor-parallel decode on M5 Max pairs.

- Poll gates (default; DS4_TP_DISABLE_POLL_GATES opts out): each flag gate
  ends the command buffer and the next queued buffer opens with a bounded
  fresh-line poll released by the service thread, instead of parking on a
  shared event whose CPU->GPU wake-up costs 27-43 us. Ring of 16 regions,
  ~600 ms budget, timeout reported at the next gate.
- Checksum-verified flags: kernel_dsv4_tp_flag_set_checked publishes an
  exact integer checksum of the 16 KiB partial; the service thread verifies
  the payload in memory before the RDMA post (the flag line can land ~35 us
  before the buffer completes; unverified sends were nondeterministic).
- HC post/expand folded into the compound HC producer with a device-scope
  barrier across its six threadgroups; the engine defers the post-gate
  expand and flushes it before any other reader
  (DS4_METAL_DISABLE_HC_EXPAND_PRODUCER_FUSE opts out).
- Gate-time prefetch: the next phase's weights (rank slices, priority
  order, cut at this rank's measured peer-wait window) are touched into the
  cache before the poll (DS4_TP_DISABLE_GATE_PREFETCH opts out).
- Concurrent routed/shared FFN branches, safe decode command-buffer split,
  exact-shape TP MXFP4 static kernels, DSpark direct partial commit, and
  split gate profiling from the morning session.
- Diagnostics: DS4_METAL_ENCODER_TIMELINE per-pass GPU timeline,
  DS4_METAL_CB_TIMES per-command-buffer driver/GPU timing.

README ctx 128, 256 greedy tokens: 45.5 vs 43.3 t/s (event gates, same
build); frontier logits and generated text byte-identical throughout.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WjmBSeiH6tPDtJuiQNFSpq
 2026-09-01 17:26:50 -07:00
antirez 110afdd888 Use random completion response IDs 2026-09-01 17:32:21 +02:00
antirez b0982a1b4e Harden multimodal cache fallback 2026-09-01 13:33:00 +02:00
Jordi Posthumus 98b91351ad server: reuse exact multimodal live KV safely 2026-09-01 07:46:20 -03:00
antirez 3da2da96b2 Document DeepSeek Vision-Exp usage 2026-09-01 12:23:44 +02:00
antirez f9c459ce2f Fix ROCm MMQ scratch lifetime and tail reads 2026-09-01 03:26:23 +02:00
antirez 200b9f45bf Fix CUDA vision build 2026-09-01 00:32:06 +02:00
antirez 409080a623 Fix ROCm vision build 2026-09-01 00:23:37 +02:00
antirez e976a54331 Add DeepSeek Vision-Exp support to ROCm 2026-09-01 00:22:23 +02:00
Salvatore Sanfilippo fc8bf3c39c Add DeepSeek V4 Flash vision support 2026-09-01 00:11:58 +02:00
antirez ad4ac91a39 Speed up GLM 5.3 ROCm prefill 2026-08-31 23:58:50 +02:00
antirez ab06d196da Speed up GLM 5.3 decode on ROCm 2026-08-31 20:14:12 +02:00
antirez ec7642cdd9 README: GLM 5.3 is supported as well 2026-08-31 15:36:35 +02:00
antirez 4b321cf1ec Make ROCm release builds warning-free 2026-08-31 15:12:02 +02:00
antirez f5ee85b341 Make GLM SSD streaming memory budgets adaptive 
Measure the runtime decode map before selecting the global Metal path, account its mapped bytes once, and fall back to per-layer mapping when it cannot fit. Treat explicit cache sizes as performance targets that may be reduced after model, graph, context, and platform reserves are known. Share the final guard-budget calculation between fitting and refusal on Metal and ROCm.
 2026-08-31 14:54:16 +02:00
Salvatore Sanfilippo e4593c374f Fix GLM streaming resident layer maps 2026-08-31 13:45:02 +02:00
antirez 5da32061ba Merge GLM 5.3 SSD streaming and full checkpoint support 
# Conflicts:
#	ds4.c
#	tests/test_glm53_kda.c
 2026-08-31 13:25:05 +02:00
Salvatore Sanfilippo dcc70b41ef Generalize GLM quantization for full checkpoints 2026-08-31 03:33:22 -07:00
Salvatore Sanfilippo 81c8c39a5c Support FP32 GLM compact attention caches 2026-08-31 03:33:22 -07:00
Salvatore Sanfilippo c8b6e4f4d8 Improve GLM routed imatrix coverage 2026-08-31 03:33:21 -07:00
Salvatore Sanfilippo 01f3316c9c Fix full GLM SSD streaming on Metal 2026-08-31 03:33:00 -07:00
antirez a547fdff4a Merge pull request #887 from kyuz0/perf/rocm-gfx1151-mmq-kernel-lab 
# Conflicts:
#	Makefile
#	ds4.c
#	ds4_bench.c
#	ds4_help.c
#	tests/test_gpu_args_cli.sh
 2026-08-31 11:48:21 +02:00
Donato Capitella 9254c3daf4 tests: honor no-GPU ROCm test builds 2026-08-30 12:43:58 +01:00
Donato Capitella e1147cdcbd server: support fixed-length greedy decode 2026-08-30 12:40:52 +01:00
Donato Capitella 742adcaf14 docs: update Strix Halo ROCm setup 2026-08-29 21:51:45 +01:00
kyuz0 c012d5cedf fix(rocm): stabilize gfx1151 MMQ routing maps 2026-08-29 21:51:45 +01:00
Donato Capitella 17c2855987 Revert "bench: add untimed context baseline" 
This reverts commit e344bbef57b4713136ab703af392017ac80c7927.
 2026-08-29 21:51:45 +01:00
Donato Capitella 985cacd07b bench: add untimed context baseline 2026-08-29 21:51:45 +01:00
Donato Capitella d208b2b762 rocm: generalize validated gfx1151 DSpark defaults 2026-08-29 21:51:45 +01:00
Donato Capitella f9e4c0fe2c rocm: accelerate gfx1151 DSpark verification 2026-08-29 21:51:45 +01:00
Donato Capitella 9dfb291b0b rocm: tile large IQ2 MMQ prefill batches 2026-08-29 21:51:45 +01:00
Donato Capitella e532136631 fix(rocm): adapt DSpark scheduling on gfx1151 2026-08-29 21:51:45 +01:00
Donato Capitella 24665f8192 fix(rocm): preserve target cache with DSpark support 2026-08-29 21:51:45 +01:00
Donato Capitella e73b367db9 feat(bench): add basic DSpark throughput support 2026-08-29 21:51:45 +01:00
Donato Capitella d94a54e612 perf(rocm): retune DSpark F16 solutions for rocBLAS 5.6 2026-08-29 21:51:45 +01:00
Donato Capitella 0efd822b9b fix(rocm): stop retrying rejected DSpark GEMM solutions 2026-08-29 21:51:45 +01:00
Donato Capitella e8686b9d04 perf(dspark): fuse gfx1151 seed and draft verification 2026-08-29 21:51:45 +01:00
Donato Capitella 9ca8642d01 perf(dspark): skip zero-threshold confidence probe 2026-08-29 21:51:45 +01:00
Donato Capitella 227fa190aa perf(rocm): accelerate gfx1151 DSpark verification 2026-08-29 21:51:45 +01:00
Donato Capitella 307dc093d9 perf(rocm): specialize tiny Q8 verifier batches 2026-08-29 21:51:45 +01:00
Donato Capitella a4e015e004 perf(rocm): accelerate tiny-batch DSpark verification 2026-08-29 21:51:45 +01:00
Donato Capitella ab3346a695 bench: finalize gfx1151 prefill result 2026-08-29 21:51:45 +01:00
Donato Capitella 1ef0680b79 perf(rocm): default to tuned gfx1151 prefill paths 2026-08-29 21:51:45 +01:00
Donato Capitella 49a30025d6 perf(rocm): reduce gfx1151 prefill staging overhead 2026-08-29 21:51:45 +01:00
Donato Capitella b7f2f2d37f bench: close gfx1151 attention key tile sweep 2026-08-29 21:51:45 +01:00
Donato Capitella 86128aae00 bench: record 290.79 tok/s gfx1151 prefill lead 2026-08-29 21:51:45 +01:00
Donato Capitella 4e23cf1bcc rocm: use 80-key WMMA attention tiles on gfx1151 2026-08-29 21:51:45 +01:00
Donato Capitella becf9f96d8 bench: record large F16 gfx1151 win 2026-08-29 21:51:45 +01:00
Donato Capitella 8c1629c95c perf(rocm): accelerate large F16 prefill projection 2026-08-29 21:51:45 +01:00
Donato Capitella 491e7e3d3f perf(rocm): remove rejected paired IQ2 kernel 2026-08-29 21:51:45 +01:00
Donato Capitella 9fd5e399e9 bench: profile paired gfx1151 IQ2 kernel 2026-08-29 21:51:45 +01:00
Donato Capitella c40e8a8e09 perf(rocm): pair IQ2 gate and up tiles 2026-08-29 21:51:45 +01:00
Donato Capitella 11e1eb31b0 bench: record compact gfx1151 IQ2 launch 2026-08-29 21:51:45 +01:00
Donato Capitella 4bf7c47b71 perf(rocm): compact IQ2 MoE tile launches 2026-08-29 21:51:45 +01:00
Donato Capitella 540ccbafa7 bench: record 277 tok/s gfx1151 lead 2026-08-29 21:51:45 +01:00
Donato Capitella 9ab922f50e perf(rocm): tighten IQ2 MoE launch bound 2026-08-29 21:51:45 +01:00
Donato Capitella 45dbb31505 docs: record gfx1151 small-M F16 results 2026-08-29 21:51:45 +01:00
Donato Capitella 6bdc68bccf perf(rocm): specialize small-M F16 prefill GEMMs 2026-08-29 21:51:45 +01:00
kyuz0 b0d80b13c8 bench: record 263 tok/s gfx1151 lead 2026-08-29 21:51:45 +01:00
kyuz0 e4cd58705b rocm: specialize tiny-m f16 prefill gemm 2026-08-29 21:51:45 +01:00
kyuz0 d8d5fa679a bench: record 256 tok/s gfx1151 lead 2026-08-29 21:51:45 +01:00
kyuz0 4d89fe9e84 rocm: vectorize indexed attention kv staging 2026-08-29 21:51:45 +01:00
kyuz0 df1269414d bench: close unsafe f16 kv mirror experiment 2026-08-29 21:51:45 +01:00
kyuz0 64b22d74f1 bench: map remaining gfx1151 prefill pools 2026-08-29 21:51:45 +01:00
kyuz0 300c5814b7 bench: record indexed attention tile rejection 2026-08-29 21:51:45 +01:00
kyuz0 68e0ca1e44 bench: record 248 tok/s prefill lead 2026-08-29 21:51:45 +01:00
kyuz0 e43738a0ab rocm: specialize attention output B for gfx1151 2026-08-29 21:51:45 +01:00
kyuz0 d104a31b14 bench: map attention output GEMM 2026-08-29 21:51:44 +01:00
kyuz0 c2b67652e9 bench: record Q2 n4 rejection 2026-08-29 21:51:44 +01:00
kyuz0 f6022ee42d bench: record IQ2 sign-mask optimization 2026-08-29 21:51:44 +01:00
kyuz0 0610a6577e rocm: use cached IQ2 sign masks in MMQ 2026-08-29 21:51:44 +01:00
kyuz0 dd356c3fad bench: record gfx1151 prefill optimization results 2026-08-29 21:51:44 +01:00
kyuz0 7470dd60a7 build: link MMQ objects in ROCm regression target 2026-08-29 21:51:44 +01:00
kyuz0 c53b2ef6fa rocm: accelerate sparse attention with WMMA 2026-08-29 21:51:44 +01:00
kyuz0 b93fb08937 rocm: tile Q8 prefill WMMA by output shape 2026-08-29 21:51:44 +01:00
kyuz0 9933e01f50 rocm: wire tuned MMQ IQ2 prefill experiment 2026-08-29 21:51:44 +01:00
kyuz0 61e9b991c3 rocm: add gfx1151 MMQ kernel lab 2026-08-29 21:51:44 +01:00
kyuz0 e6e7e6a5d7 rocm: reuse LDS output tile in Q2_K MoE down kernel 2026-08-29 21:51:44 +01:00
Mikhail Korobov ada805ad59 server: staleness-aware eviction tiers in slot routing 
The reuse-aware router sent a request that reuses nothing to the slot
with the shortest resident checkpoint (-live).  On a long-running
server this leaks: a finished long conversation is never the cheapest
victim, so such checkpoints pile up, the pool of usable slots shrinks,
and subagent/parallel requests end up evicting shorter *active*
conversations - the prefill thrashing this routing set out to fix
comes back.

Split the no-reuse victim selection into staleness tiers:

- empty slots still win (nothing to evict);
- a checkpoint idle past SLOT_STALE_AFTER_SEC (1h) is most likely a
  finished conversation: victims are picked most-idle-first,
  regardless of size, so forgotten sessions get reclaimed;
- fresher checkpoints belong to active conversations: keep the -live
  rule (evict the shortest, lose the least prefilled work).

Staleness only lowers a checkpoint's priority as a *victim*; it never
blocks reuse - the owner of a stale checkpoint who continues the
conversation still pins the slot via the reuse tier (which now scores
the actual reusable prefix instead of the full checkpoint length).

Also log evictions at dispatch time (slot, evicted tokens, idle time,
stale/protected), so routing decisions are visible in the server log
instead of being inferable only from the next request's cache miss.
 2026-08-14 12:26:49 +05:00
Mikhail Korobov 4d7028e066 server: reuse-aware slot routing via shared live-state probe 
With --batched-session N, job_slot_score routed requests to the slot
with the longest raw common token prefix, with no check that the slot's
checkpoint is actually reusable. Parallel subagents sharing a system
prompt were attracted to the slot holding the main agent's long context,
evicting it with zero reuse benefit and forcing a full re-prefill when
main resumed (masked only by --kv-disk-dir round-trips). Even an alien
request sharing no prefix at all evicted slot[0] on the tie-break.

Split every RAM reuse tier into a cheap probe predicate and its
materialization, and define tier order and conditions exactly once in
slot_probe_reuse_locked(): responses-visible, responses/anthropic
tool-output, GLM memory-rewind, memory-token, thinking-visible,
memory-text. The execution ladder (generate_job) probes once and
materializes only the selected tier; job_slot_score consumes the same
probe, so routing and execution cannot disagree. A slot scores +live
when the request would reuse its live state, -live otherwise: an empty
slot always beats an eviction, and the cheapest (shortest) unusable
checkpoint becomes the victim. Text-tier bindings are part of the probe,
so single-session chats whose resent history retokenizes differently or
omits hidden thinking keep pinning to their slot.

The memory-text probe needs rendered checkpoint text; cache it per slot
and refresh in the slot worker after each job, so probing stays a memcmp
and never detokenizes under tool_mu.

Verified: unit tests for every probe tier plus dispatch routing
(./ds4_test --server), full make test green, and live checks showing the
subagent deflected to the empty slot, main resuming via memory-token,
and thinking-visible / anthropic-tool-output / responses-tool-output
continuations unchanged.
 2026-08-10 16:05:17 +05:00
1820 changed files with 2879460 additions and 8354 deletions
Expand all files Collapse all files

No files matched your search

.gitignore
+93 -1
View File
@@ -9,7 +9,7 @@
/ds4_test
/ds4flash.gguf
/TODO.md
/gguf/
/gguf
/core
/core.*
/tests/test_engine_mgpu_placement
@@ -25,12 +25,93 @@
/tests/cuda_long_context_smoke
/tests/test_layer_pack
/tests/test_metal_session_batch
/tests/test_metal_tp_spec
/tests/test_metal_tp_cancel
/tests/test_metal_moe_prefill
/tests/test_metal_ssd_experts
/tests/test_metal_ssd_reuse
/tests/test_metal_command_memory
/tests/test_metal_dense_mpp
/tests/test_mxfp4_cuda
/tests/test_mmvq_cuda
/tests/test_mmvq_rocm
/tests/test_mxfp4_dot
/tests/test_mxfp4_metal
/tests/test_q4k_dot
/tests/test_sampling
/tests/test_web_recovery
/tests/test_quality_api
/tests/test_cuda_q8_scratch
/tests/test_cuda_dspark_moe
/tests/test_session_state
/tests/test_session_state_gpu
/tests/test_tp_commands
/tests/test_tp_rdma
/tests/test_tp_tcp
/tests/test_tp_link
/tests/test_cuda_tp
/tests/test_cuda_tp_link
/tests/test_cuda_verify_head
/tests/test_cuda_shared_side
/tests/test_cuda_tokentile
/tests/test_cuda_attn_side
/tests/test_cuda_compressor
/tests/test_cuda_copy_spans
/tests/test_cuda_dspark
/tests/test_cuda_dspark_tp_markov
/tests/test_cuda_dspark_tp_split
/tests/test_cuda_embed
/tests/test_cuda_f16_chunks
/tests/test_cuda_grouped_q8
/tests/test_cuda_hc_norm
/tests/test_cuda_hc_pre
/tests/test_cuda_indexed_one
/tests/test_cuda_indexer
/tests/test_cuda_mxfp4_split
/tests/test_cuda_pool
/tests/test_cuda_q8_0_gemv
/tests/test_cuda_q8_pair
/tests/test_cuda_router_stream
/tests/test_cuda_topk_stream
/tests/test_deepseek_indexer
/tests/test_mxfp4_staged_cuda
/tests/test_cuda_tp_repack
/tests/test_metal_tp_bulk
/tests/test_glm53_kda
/tests/test_glm53_vision_engine
/tests/test_glm_attention
/tests/test_glm_attention_rocm
/tests/test_linux_memory
/tests/test_rocm_memory
/tests/test_qwen4_kernels
/tests/test_metal_device_policy
/tests/test_metal_q8_reduction
/tests/test_qwen4_moe_mm_specialize
/tests/test_qwen4_ssd_experts
/tests/test_qwen4_memory
/tests/test_qwen4_generation
/tests/test_qwen4_vision
/tests/test_deepseek4_vision_image
/tests/test_image_decode
/tests/test_deepseek41_metal
/tests/test_deepseek41_cuda
/tests/test_cuda_q8_rows
/tests/test_cuda_reductions
/tests/test_cuda_shared
/tests/test_cuda_ssd_cache
/tests/test_cuda_ssd_batch
/tests/test_cuda_ssd_repack
/tests/test_deepseek41_gguf
/tests/test_qwen4_ngrams
/tests/test_qwen4_ngram_state
/tests/test_qwen4_mtp_prefill
/tests/test_qwen4_mtp_policy
/tests/test_qwen4_ple_prefetch
/tests/test_deepseek41_graph
/tests/test_v41_indexer_score
/tests/test_deepseek41_prefill
/tests/test_deepseek41_cli
/tests/test_engram
*.o
*.dSYM/
__pycache__/
@@ -40,3 +121,14 @@ __pycache__/
/misc/
.*.swp
.DS_Store
/tests/test_qwen4_conv_parallel
/tests/test_q8_prefill_variants
/tests/test_metal_qwen4_hc
/tests/test_metal_qwen4_moe_mid
/tests/test_metal_qwen4_moe_down
/tests/test_metal_qwen4_mxfp4_stage
/tests/test_metal_q8_gemv_reference
/tests/test_metal_qwen4_moe_half
/tests/test_qwen4_resident_arithmetic
EVAL_DATA.md
+54
View File
@@ -0,0 +1,54 @@
# Evaluation data
`ds4-eval` contains small, curated subsets of external benchmarks. They are
integration tests for DwarfStar, not official benchmark distributions or
leaderboard scores. Case IDs are retained so each prompt can be checked against
its source.
## Hard suite
The 50-case `hard` suite was assembled on 2026-09-03:
| Source | Cases | Source revision | License |
|---|---:|---|---|
| [MMLU-Pro](https://github.com/TIGER-AI-Lab/MMLU-Pro) | 30 | `f418b116db00b065c2aea046518d8fcf74d39872` | Apache-2.0 |
| [OlympiadBench](https://huggingface.co/datasets/Hothan/OlympiadBench) | 10 | `91184b52131e7fc9455fef848035173aea8cc01a` | Apache-2.0 |
| [LiveBench](https://huggingface.co/datasets/livebench/reasoning) | 5 | `6fc6498a5dfba553f69f4413feabade1f1a2d384` | Apache-2.0 |
| [NIST Juliet 1.3](https://samate.nist.gov/SARD/test-suites/112) | 5 | archive SHA-256 `ada9d7e1c323d283446df3f55bdee0d00bda1fed786785fe98764d58688f38eb` | CC0-1.0 / public domain |
Selected source IDs:
- MMLU-Pro: `86`, `671`, `899`, `1467`, `1995`, `2398`, `2826`, `3024`,
`3546`, `3696`, `4674`, `4699`, `5078`, `5848`, `6034`, `6500`, `6844`,
`6955`, `7738`, `7754`, `8771`, `8863`, `9055`, `9487`, `10360`, `10567`,
`10775`, `11120`, `11292`, `11346`.
- OlympiadBench: `1606`, `1613`, `1614`, `1618`, `1697`, `1709`, `1716`,
`1736`, `1750`, `1766`.
- LiveBench: `c833574b0ff6e8609bee19b36fabc48501cdcd9a64b99cd0422f54ada1acc53d`,
`b7b7e331df4614775210ef5e524ed7f20c5fec6655fb0dd179c94e4d71170862`,
`0daa7ca38beec4441b9d5c04d0b98912322926f0a3ac28a5097889d4ed83506f`,
`9ee37b9a04ab050936c86b2c5bb7abbaa0bc0e737d59a7bff9ba11e9b4069c1d`,
`d7071c9ff5d9779e7ab955366d0ae8db40f785aadfe7ff0b5a7ede98c05c44ea`.
- NIST Juliet reductions use the local IDs beginning with `juliet-` in
`ds4_eval_cases.c`.
The MMLU-Pro choices and keys are unchanged. OlympiadBench cases are English,
text-only rows; only their published final answers are embedded. LiveBench's
benchmark-specific output wrapper was removed because `ds4-eval` supplies one
uniform final-answer instruction. The NIST cases are short, locally written
reductions of Juliet weakness patterns rather than copies of complete Juliet
test cases.
Multiple-choice order is retained for MMLU-Pro. Open answers declare their
grading type and, where needed, a small list of equivalent surface forms. The
12-case `hard-smoke` suite is a fixed subset of `hard`.
Apache-2.0 terms for redistributed benchmark material are included in
[`licenses/Apache-2.0.txt`](licenses/Apache-2.0.txt). DwarfStar's own source
remains under the repository's top-level MIT license.
## Core suite
The original 92-case `core` suite contains 25 GPQA Diamond, 25 SuperGPQA, 25
AIME 2025, and 17 reduced defensive code-review cases. Its existing source IDs,
ordering, prompts, and answer keys are unchanged by the hard-suite addition.
LICENSE
+1
View File
@@ -2,6 +2,7 @@ MIT License
Copyright (c) 2026 The ds4.c authors
Copyright (c) 2023-2026 The ggml authors
Copyright (c) 2023 DeepSeek
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Makefile
+1108 -64
View File
File diff suppressed because it is too large. Load diff
QA_BEFORE_RELEASES.md
+1423 -128
View File
File diff suppressed because it is too large. Load diff
README.md
+169 -1687
View File
File diff suppressed because it is too large. Load diff
STRIXHALO.md
+3 -143
View File
@@ -1,146 +1,6 @@
# DS4 on Strix Halo
This is the minimal setup for DS4 ROCm inference on a
Strix Halo machine with 128 GB RAM and Radeon 8060S (`gfx1151`).
The setup guide has moved to [docs/STRIX_HALO.md](docs/STRIX_HALO.md).
## 1. Install ROCm
On Ubuntu 26.04 LTS, install the ROCm compiler/runtime and libraries used by the Strix Halo backend:
```sh
sudo apt-get update
sudo apt-get install -y \
hipcc rocminfo rocm-smi \
libamdhip64-dev \
libhipblas-dev libhipblaslt-dev \
librocblas-dev \
librocwmma-dev \
libhipcub-dev
```
The backend uses rocWMMA. On this Ubuntu 26.04 setup, `librocwmma-dev`
installs the top-level rocWMMA headers but misses `rocwmma/internal/`.
No Ubuntu package currently provides those internal headers. Install a complete
matching rocWMMA header tree:
```sh
git clone --depth 1 --branch rocm-7.1.0 https://github.com/ROCm/rocWMMA.git /tmp/rocWMMA-rocm-7.1.0
sudo mkdir -p /usr/local/include
sudo cp -a /tmp/rocWMMA-rocm-7.1.0/library/include/rocwmma /usr/local/include/
```
If ROCm is installed under `/usr` but tooling expects `/opt/rocm`, add these
compatibility links:
```sh
sudo mkdir -p /opt/rocm/bin
sudo ln -sf /usr/bin/hipcc /opt/rocm/bin/hipcc
sudo ln -sfn /usr/lib/x86_64-linux-gnu /opt/rocm/lib
sudo ln -sfn /usr/include /opt/rocm/include
```
## 2. Enable ROCm access
The user running DS4 must be able to open `/dev/kfd` and the DRM render node:
```sh
sudo usermod -aG render,video "$USER"
```
Log out and back in, or reboot. Verify:
```sh
rocminfo | grep -A80 'Name: gfx1151'
```
If DS4 says `no ROCm-capable device is detected`, check that `rocminfo` can open
`/dev/kfd` and that `groups` includes `render`.
## 3. Increase GPU-visible memory
A 128 GB Strix Halo system may initially expose only about 62 GB of GPU-visible
memory. DS4 needs the larger GTT aperture for the 80.76 GiB model plus runtime
buffers.
Use these kernel parameters:
```text
amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856
```
On Ubuntu with GRUB:
```sh
sudo cp /etc/default/grub /etc/default/grub.bak
sudoedit /etc/default/grub
```
Set:
```text
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856"
```
Then:
```sh
sudo update-grub
sudo reboot
```
After reboot, verify:
```sh
cat /proc/cmdline
sudo dmesg | grep -Ei 'GTT|gttsize|TTM|VRAM'
rocminfo | grep -A80 'Name: gfx1151'
```
Expected signs:
```text
amdgpu: 126976M of GTT memory ready
rocminfo gfx1151 pool: 130023424 KB
```
## 4. Build DS4
Use the normal Strix Halo target. It builds the standard binary names:
```sh
make strix-halo -j"$(nproc)"
```
`make rocm` is an alias for `make strix-halo`.
After changes to the ROCm MXFP4 or routed-MoE kernels, run the standalone
CPU-oracle regression:
```sh
make test-mxfp4-rocm
```
This test does not require a full model GGUF. It covers resident decode and
batched routed-MoE execution at 1, 3, 32, 128, and 512 tokens.
## 5. Use the right GGUF
Use the standard IQ2XXS/Q2K/Q8 imatrix GGUF:
```text
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
```
Avoid the mixed IQ2/IQ4 or IQ2/Q4 GGUFs on this machine for now. They put much
more memory pressure on the ROCm path and can trigger system OOM instead of a
clean DS4 failure.
## 6. Run DS4
Run it normally:
```sh
./ds4 -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
```
The ROCm build uses the Strix Halo backend automatically.
It covers ROCm prerequisites, GPU-visible memory, current model downloads,
and first runs for DeepSeek Flash and GLM 5.3 Flash.
cuda/mmq/VENDOR.md
+8
View File
@@ -94,6 +94,14 @@ Symbols the vendored files reference, and how they resolve in this directory:
## Re-syncing with upstream
Local MMVQ bounds fix: pass the logical row count to the generic vector
kernel instead of using its output stride as a row bound. Both generic and
multi-token MoE kernels also guard tail-row weight loads.
MXFP4 MMQ uses Q8_1 activations on Blackwell as on older GPUs. Its tile
loader, K iteration and activation-block stride use the integer-MMA layout;
the stored MXFP4 weights are unchanged.
When upstream lands a bugfix or perf improvement we want, the procedure is:
```sh
cuda/mmq/ds4_ggml_stubs.cu
+85 -7
View File
@@ -4,19 +4,25 @@
// new_pool_for_device that the vendored common.cuh declares without
// defining.
//
// Phase 0: pool is plain cudaMallocAsync / cudaFreeAsync. Phase 4 swaps
// in ds4's existing cuda_tmp_alloc slab allocator.
// CUDA uses stream-ordered allocation. HIP retains stream-local allocations
// because returning live MMQ scratch to its asynchronous pool is unreliable.
#include "common.cuh" // pulls in ds4_ggml_stubs.h via redirect headers
#if defined(GGML_USE_HIP)
#include "vendors/hip.h"
#include "../../ds4_rocm_memory.h"
#else
#include <cuda_runtime.h>
#include <cublas_v2.h>
#endif
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <memory>
#include <mutex>
#include <vector>
// ----------------------------------------------------------------------------
// Device info singleton.
@@ -44,10 +50,23 @@ const ggml_cuda_device_info & ggml_cuda_info() {
for (int i = 0; i < count; i++) {
cudaDeviceProp p;
CUDA_CHECK(cudaGetDeviceProperties(&p, i));
#if defined(GGML_USE_HIP)
unsigned gfx_id = 0;
if (sscanf(p.gcnArchName, "gfx%x", &gfx_id) == 1) {
info.devices[i].cc = GGML_CUDA_CC_OFFSET_AMD + (int)gfx_id;
} else {
info.devices[i].cc = GGML_CUDA_CC_OFFSET_AMD + p.major * 0x100 + p.minor * 0x10;
}
#else
info.devices[i].cc = p.major * 100 + p.minor * 10;
#endif
info.devices[i].nsm = p.multiProcessorCount;
info.devices[i].smpb = p.sharedMemPerBlock;
#if defined(GGML_USE_HIP)
info.devices[i].smpbo = p.sharedMemPerBlock;
#else
info.devices[i].smpbo = p.sharedMemPerBlockOptin;
#endif
info.devices[i].integrated = p.integrated != 0;
info.devices[i].vmm = false;
info.devices[i].vmm_granularity = 0;
@@ -64,13 +83,13 @@ const ggml_cuda_device_info & ggml_cuda_info() {
int ggml_cuda_get_device() {
int dev = 0;
cudaGetDevice(&dev);
(void)cudaGetDevice(&dev);
return dev;
}
void ggml_cuda_set_device(int device) {
int cur = -1;
cudaGetDevice(&cur);
(void)cudaGetDevice(&cur);
if (cur != device) {
CUDA_CHECK(cudaSetDevice(device));
}
@@ -95,7 +114,7 @@ int64_t ggml_time_us() {
}
// ----------------------------------------------------------------------------
// Concrete pool wrapping cudaMallocAsync / cudaFreeAsync.
// Concrete device scratch pool.
// ----------------------------------------------------------------------------
namespace {
@@ -126,19 +145,78 @@ struct ds4_naive_pool : public ggml_cuda_pool {
explicit ds4_naive_pool(int device) : device(device) {}
#if defined(GGML_USE_HIP)
struct cached_block {
void *ptr;
size_t size;
cudaStream_t stream;
};
std::mutex mutex;
std::vector<cached_block> available;
std::vector<void *> owned;
#endif
void * alloc(size_t size, size_t * actual_size) override {
ggml_cuda_set_device(device);
#if defined(GGML_USE_HIP)
{
std::lock_guard<std::mutex> guard(mutex);
size_t best = available.size();
for (size_t i = 0; i < available.size(); i++) {
if (available[i].stream != t_ds4_pool_stream ||
available[i].size < size) {
continue;
}
if (best == available.size() ||
available[i].size < available[best].size) {
best = i;
}
}
if (best != available.size()) {
const cached_block block = available[best];
available[best] = available.back();
available.pop_back();
if (actual_size) *actual_size = block.size;
return block.ptr;
}
}
void * ptr = nullptr;
CUDA_CHECK(ds4_rocm_allocation_fits(size) ? hipSuccess : hipErrorOutOfMemory);
CUDA_CHECK(cudaMallocAsync(&ptr, size, t_ds4_pool_stream));
{
std::lock_guard<std::mutex> guard(mutex);
owned.push_back(ptr);
}
if (actual_size) *actual_size = size;
return ptr;
#else
void * ptr = nullptr;
CUDA_CHECK(cudaMallocAsync(&ptr, size, t_ds4_pool_stream));
if (actual_size) *actual_size = size;
return ptr;
#endif
}
void free(void * ptr, size_t /*size*/) override {
void free(void * ptr, size_t size) override {
if (!ptr) return;
ggml_cuda_set_device(device);
#if defined(GGML_USE_HIP)
std::lock_guard<std::mutex> guard(mutex);
available.push_back({ptr, size, t_ds4_pool_stream});
#else
CUDA_CHECK(cudaFreeAsync(ptr, t_ds4_pool_stream));
#endif
}
#if defined(GGML_USE_HIP)
~ds4_naive_pool() override {
ggml_cuda_set_device(device);
(void)cudaDeviceSynchronize();
for (void *ptr : owned) (void)cudaFree(ptr);
}
#endif
};
} // anonymous namespace
@@ -149,7 +227,7 @@ std::unique_ptr<ggml_cuda_pool> ggml_backend_cuda_context::new_pool_for_device(i
ggml_backend_cuda_context::~ggml_backend_cuda_context() {
if (copy_event) {
cudaEventDestroy(copy_event);
(void)cudaEventDestroy(copy_event);
copy_event = nullptr;
}
// streams[][], cublas_handles[], and pools[][] are owned-by-value
cuda/mmq/ds4_mmq.cu
+873 -59
View File
File diff suppressed because it is too large. Load diff
cuda/mmq/ds4_mmq.h
+11 -4
View File
@@ -14,7 +14,11 @@
#pragma once
#if defined(GGML_USE_HIP)
#include "vendors/hip.h"
#else
#include <cuda_runtime.h>
#endif
#include <stddef.h>
#include <stdint.h>
@@ -319,7 +323,8 @@ int ds4_mmq_iq2_xxs_moe_pair_soa(
* assignment map once, runs the paired gate/up MMQs, computes clamp +
* SwiGLU + router weighting in mid_f32, then gathers and quantizes those
* rows for the Q2_K down MMQ through the same ids_dst/expert_bounds. No
* second mm_ids_helper; gate/up/mid/down keep the pair-major layout. */
* second mm_ids_helper; gate/up/mid/down keep the pair-major layout.
* preserve_reduction retains the raw MMQ accumulation order instead of D2R. */
int ds4_mmq_iq2_xxs_q2_K_moe_fused_soa(
const void * W_gate_soa,
const void * W_up_soa,
@@ -338,6 +343,7 @@ int ds4_mmq_iq2_xxs_q2_K_moe_fused_soa(
int n_experts,
int n_expert_used,
float clamp,
int preserve_reduction,
cudaStream_t stream);
/* Aligned-artifact production fast path: gate/up stay in registers, weighted
@@ -878,11 +884,12 @@ int ds4_mmq_q4_K_dense_pair_vec(
int K,
cudaStream_t stream);
// Set the thread-local stream that the internal cuda pool uses for
// cudaMallocAsync / cudaFreeAsync. Defaults to cudaStreamPerThread.
// Set the thread-local stream used for internal stream-ordered scratch.
// Defaults to cudaStreamPerThread.
// Step 8 (CUDA Graphs) calls this with the capture stream so pool
// allocations land on the captured stream and don't invalidate capture.
// Pass NULL to reset to cudaStreamPerThread.
// Pass cudaStreamPerThread to restore the default; NULL selects the legacy
// default stream.
void ds4_pool_set_stream(cudaStream_t stream);
#ifdef __cplusplus
cuda/mmq/ds4_mmq_d2r.cu
+4
View File
@@ -14,8 +14,12 @@
#include "common.cuh"
#include "mmq.cuh"
#if defined(GGML_USE_HIP)
#include "vendors/hip.h"
#else
#include <cuda_fp16.h>
#include <cuda_runtime.h>
#endif
#include <climits>
#include <cstddef>
cuda/mmq/ds4_mmq_d2r.cuh
+4
View File
@@ -3,7 +3,11 @@
#pragma once
#if defined(GGML_USE_HIP)
#include "vendors/hip.h"
#else
#include <cuda_runtime.h>
#endif
#include <stddef.h>
#include <stdint.h>
cuda/mmq/ds4_repack.cu
+42 -4
View File
@@ -6,8 +6,12 @@
* lines are the gate. */
#include "ds4_repack.h"
#if defined(GGML_USE_HIP)
#include "vendors/hip.h"
#else
#include <cuda_runtime.h>
#include <cuda_fp16.h>
#endif
#include <atomic>
#include <cerrno>
@@ -201,6 +205,7 @@ static bool tensor_type_info(uint32_t type, uint64_t &block_elems, uint64_t &blo
case 28: block_elems = 1; block_bytes = 8; return true;
case 29: block_elems = 256; block_bytes = 56; return true;
case 30: block_elems = 1; block_bytes = 2; return true;
case 39: block_elems = 32; block_bytes = 17; return true;
default: return false;
}
}
@@ -337,12 +342,16 @@ __global__ static void repack_iq2_xxs_aligned_kernel(
__half *dq,
uint2 *qs,
const unsigned char *raw,
uint64_t nblk) {
uint64_t nblk,
const int32_t *slots = nullptr,
uint64_t expert_blocks = 0) {
const uint64_t i = (uint64_t)blockIdx.x * blockDim.x + threadIdx.x;
if (i >= nblk * 8ull) return;
const uint64_t blk = i >> 3;
const uint32_t p = (uint32_t)(i & 7u);
const unsigned char *src = raw + blk * 66ull;
const uint64_t source = slots ?
(uint64_t)slots[blk / expert_blocks] * expert_blocks + blk % expert_blocks : blk;
const unsigned char *src = raw + source * 66ull;
if (p == 0u) {
uint16_t h;
memcpy(&h, src, 2u);
@@ -373,7 +382,8 @@ __global__ static void repack_q2_k_aligned_kernel(
uint64_t g0, // absolute raw-block index of raw[0]
uint64_t cblk, // raw blocks in this chunk
uint32_t nb_row, // blocks per row = K/256
uint32_t nrows) { // rows per expert = M
uint32_t nrows, // rows per expert = M
const int32_t *slots = nullptr) {
const uint64_t i = (uint64_t)blockIdx.x * blockDim.x + threadIdx.x;
if (i >= cblk * 16ull) return;
const uint64_t j = i >> 4;
@@ -384,7 +394,10 @@ __global__ static void repack_q2_k_aligned_kernel(
const uint64_t e = g / ((uint64_t)nb_row * nrows);
const uint64_t pblk = ((uint64_t)e * (nrows/2u) + r/2u) * nb_row + b;
const uint32_t parity = r & 1u;
const unsigned char *src = raw + j * 84ull;
const uint64_t expert_blocks = (uint64_t)nb_row * nrows;
const uint64_t source = slots ?
(uint64_t)slots[j / expert_blocks] * expert_blocks + j % expert_blocks : j;
const unsigned char *src = raw + source * 84ull;
uint32_t w;
memcpy(&w, src + 16u + (uint64_t)p * 4u, 4u);
qs2[pblk * 32ull + (uint64_t)p * 2ull + parity] = w;
@@ -399,6 +412,31 @@ __global__ static void repack_q2_k_aligned_kernel(
}
}
bool ds4_repack_selected_experts(void *dst, const void *raw, const int32_t *slots,
uint32_t kind, uint32_t rows, uint32_t cols,
uint32_t count) {
if (!dst || !raw || !slots || !rows || !cols || !count || cols % 256u) return false;
const uint64_t expert_blocks = (uint64_t)rows * (cols / 256u);
if (expert_blocks > UINT64_MAX / count) return false;
const uint64_t blocks = expert_blocks * count;
if (blocks > (uint64_t)INT_MAX * 16u) return false;
if (kind == DS4_REPACK_IQ2_XXS_ALIGNED_MOE) {
const uint64_t scales = (blocks * 2u + 63u) & ~UINT64_C(63);
repack_iq2_xxs_aligned_kernel<<<(unsigned)((blocks * 8u + 255u) / 256u), 256>>>(
(__half *)dst, (uint2 *)((char *)dst + scales),
(const unsigned char *)raw, blocks, slots, expert_blocks);
} else if (kind == DS4_REPACK_Q2_K_ALIGNED_MOE && rows % 2u == 0u) {
const uint64_t pairs = blocks / 2u;
const uint64_t dm_bytes = (pairs * 8u + 63u) & ~UINT64_C(63);
const uint64_t sc_bytes = (pairs * 32u + 63u) & ~UINT64_C(63);
repack_q2_k_aligned_kernel<<<(unsigned)((blocks * 16u + 255u) / 256u), 256>>>(
(uint32_t *)dst, (uint32_t *)((char *)dst + dm_bytes),
(uint32_t *)((char *)dst + dm_bytes + sc_bytes),
(const unsigned char *)raw, 0, blocks, cols / 256u, rows, slots);
} else return false;
return cudaGetLastError() == cudaSuccess;
}
/* One thread per (block, 16B half of the 32B code payload); p==0 additionally
* splits out the block scale. Source raw block_q8_0 is 34 bytes =
* [half d][32 x int8 codes]. */
cuda/mmq/ds4_repack.h
+7
View File
@@ -106,6 +106,13 @@ void ds4_repack_set_hash(bool on);
uint64_t ds4_repack_fnv1a(const unsigned char *p, uint64_t n, uint64_t h);
/* Gather raw cached experts into one aligned artifact on the default stream.
* slots is a device array of count valid source expert indices. The caller
* sizes dst using the matching ds4_mmq_*_aligned_bytes layout contract. */
bool ds4_repack_selected_experts(void *dst, const void *raw, const int32_t *slots,
uint32_t kind, uint32_t rows, uint32_t cols,
uint32_t count);
/* Build every candidate's aligned artifact. On success appends the built
* artifacts to out, adds their byte total to *repacked_bytes_out (may be
* NULL), and returns true; on any failure releases every artifact this call
cuda/mmq/mmq.cuh
+99 -14
View File
@@ -109,13 +109,14 @@ struct tile_x_sizes {
};
static int get_mmq_x_max_host(const int cc) {
int base = (turing_mma_available(cc) || amd_wmma_available(cc)) ? 128 :
const int hardware_max = (turing_mma_available(cc) || amd_wmma_available(cc)) ? 128 :
GGML_CUDA_CC_IS_NVIDIA(cc) && ggml_cuda_highest_compiled_arch(cc) >= GGML_CUDA_CC_VOLTA ?
#ifdef GGML_CUDA_FORCE_MMQ
128 : 64;
#else
MMQ_DP4A_MAX_BATCH_SIZE : 64;
#endif // GGML_CUDA_FORCE_MMQ
int base = cc == GGML_CUDA_CC_OFFSET_AMD + 0x1151 ? 64 : hardware_max;
// ds4: optional Step-4 experiment hook. DS4_CUDA_MMQ_X_MAX=N clips the
// tile-width selector to N when sweeping for an sm_120-specific
// optimum. Value rounded down to a multiple of 8 (the iteration step
@@ -133,7 +134,7 @@ static int get_mmq_x_max_host(const int cc) {
}
}
}
if (g_override > 0 && g_override < base) base = g_override;
if (g_override > 0) base = std::min(g_override, hardware_max);
return base;
}
@@ -161,13 +162,18 @@ static constexpr __device__ int get_mmq_x_max_device() {
}
static int get_mmq_y_host(const int cc) {
#if defined(GGML_USE_HIP) && defined(DS4_HIP_MMQ_Y)
GGML_UNUSED(cc);
return DS4_HIP_MMQ_Y;
#else
return GGML_CUDA_CC_IS_AMD(cc) ? (GGML_CUDA_CC_IS_RDNA1(cc) ? 64 : 128) :
((GGML_CUDA_CC_IS_NVIDIA(cc) && ggml_cuda_highest_compiled_arch(cc) >= GGML_CUDA_CC_VOLTA) ? 128 : 64);
#endif
}
static constexpr __device__ int get_iter_k([[maybe_unused]] const ggml_type type) {
#if defined(BLACKWELL_MMA_AVAILABLE)
if (type == GGML_TYPE_NVFP4 || type == GGML_TYPE_MXFP4) {
if (type == GGML_TYPE_NVFP4) {
return MMQ_ITER_K_FP4;
}
#endif // defined(BLACKWELL_MMA_AVAILABLE)
@@ -176,11 +182,15 @@ if (type == GGML_TYPE_NVFP4 || type == GGML_TYPE_MXFP4) {
static constexpr __device__ int get_mmq_y_device() {
#if defined(GGML_USE_HIP)
#if defined(DS4_HIP_MMQ_Y)
return DS4_HIP_MMQ_Y;
#else
#if defined(RDNA1)
return 64;
#else
return 128;
#endif // defined RDNA1
#endif // defined(DS4_HIP_MMQ_Y)
#else
#if __CUDA_ARCH__ >= GGML_CUDA_CC_VOLTA
return 128;
@@ -317,7 +327,11 @@ static constexpr __device__ int mmq_get_granularity_device(const int /*mmq_x*/)
#if defined(GGML_USE_HIP)
static int mmq_get_nwarps_host(const int cc, const int warp_size) {
#if defined(DS4_HIP_MMQ_Y)
return amd_mfma_available(cc) ? 8 : DS4_HIP_MMQ_Y/16;
#else
return amd_mfma_available(cc) ? 8 : 256/warp_size;
#endif
}
#else
static int mmq_get_nwarps_host(const int /*cc*/, const int warp_size) {
@@ -327,7 +341,11 @@ static int mmq_get_nwarps_host(const int /*cc*/, const int warp_size) {
static constexpr __device__ int mmq_get_nwarps_device() {
#if defined(AMD_MFMA_AVAILABLE) || defined(AMD_WMMA_AVAILABLE)
#if defined(DS4_HIP_MMQ_Y)
return DS4_HIP_MMQ_Y/16;
#else
return 8;
#endif
#else
return 256/ggml_cuda_get_physical_warp_size();
#endif // AMD_MFMA_AVAILABLE
@@ -2966,12 +2984,13 @@ template <int mmq_y, bool need_check> static __device__ __forceinline__ void loa
#pragma unroll
for (int l = 0; l < QR2_XXS; ++l) {
const uint2 grid_pos = ((const uint2*)iq2xxs_grid)[aux8[l]];
const uint32_t signs = unpack_ksigns(aux32 >> (7 * l));
const uint2 sign_masks =
((const uint2 *) ksigns64)[(aux32 >> (7 * l)) & 0x7fu];
const int signs0 = __vcmpne4(signs & 0x08040201, 0);
const int signs0 = (int)sign_masks.x;
const int grid0 = __vsub4(grid_pos.x ^ signs0, signs0);
const int signs1 = __vcmpne4(signs & 0x80402010, 0);
const int signs1 = (int)sign_masks.y;
const int grid1 = __vsub4(grid_pos.y ^ signs1, signs1);
#if defined(AMD_MFMA_AVAILABLE) || defined(TURING_MMA_AVAILABLE) || defined(AMD_WMMA_AVAILABLE)
@@ -3511,13 +3530,10 @@ struct mmq_type_traits<mmq_x, mmq_y, need_check, GGML_TYPE_Q8_0> {
template <int mmq_x, int mmq_y, bool need_check>
struct mmq_type_traits<mmq_x, mmq_y, need_check, GGML_TYPE_MXFP4> {
static constexpr int vdr = VDR_MXFP4_Q8_1_MMQ;
#ifdef BLACKWELL_MMA_AVAILABLE
static constexpr load_tiles_mmq_t load_tiles = load_tiles_mxfp4_fp4<mmq_y, need_check>;
static constexpr vec_dot_mmq_t vec_dot_mma = vec_dot_fp4_fp4_mma<mmq_x, mmq_y, GGML_TYPE_MXFP4>;
#else
// Packed FP4 weights do not imply FP4 activations. Preserve Q8 activation
// precision on Blackwell too, using the exact integer weight mantissas.
static constexpr load_tiles_mmq_t load_tiles = load_tiles_mxfp4<mmq_y, need_check>;
static constexpr vec_dot_mmq_t vec_dot_mma = vec_dot_q8_0_q8_1_mma<mmq_x, mmq_y, MMQ_Q8_1_DS_LAYOUT_D4>;
#endif // BLACKWELL_MMA_AVAILABLE
static constexpr vec_dot_mmq_t vec_dot_dp4a = vec_dot_q8_0_q8_1_dp4a<mmq_x, mmq_y>;
};
@@ -3666,13 +3682,36 @@ static __device__ __forceinline__ void mul_mat_q_process_tile(
#if defined(BLACKWELL_MMA_AVAILABLE)
// FP4 tile stores 8 blocks
constexpr int ne_block = (type == GGML_TYPE_MXFP4 || type == GGML_TYPE_NVFP4) ? QK_K : 4 * QK8_1;
constexpr int ne_block = type == GGML_TYPE_NVFP4 ? QK_K : 4 * QK8_1;
#else
constexpr int ne_block = 4 * QK8_1;
#endif // defined(BLACKWELL_MMA_AVAILABLE)
constexpr int ITER_K = get_iter_k(type);
constexpr int blocks_per_iter = ITER_K / qk;
static_assert(type != GGML_TYPE_MXFP4 ||
(ITER_K == MMQ_ITER_K && ne_block == 4 * QK8_1),
"MXFP4 MMQ requires Q8_1 activation tiles");
// Invalid columns must be zero before the matrix instruction. Masking
// only write-back is too late: stale or NaN tail blocks can contaminate
// other values in the same hardware matrix fragment.
const int valid_y_words = min(mmq_x, tile_y_max_j + 1) * MMQ_TILE_Y_K;
#if defined(GGML_USE_HIP) && defined(__gfx1151__)
// Load scheduling can change fast-math accumulation order. Keep the
// established path outside the larger gfx1151 IQ2 prefill templates.
constexpr bool fast_y_loads = type == GGML_TYPE_IQ2_XXS && mmq_x >= 32;
const bool full_y_tile = tile_y_max_j >= mmq_x - 1;
const int linear_tid = threadIdx.y*warp_size + threadIdx.x;
constexpr int nthreads = nwarps * warp_size;
// Seed invalid columns once. Both K stages refresh only the valid prefix,
// and the existing barriers keep the suffix zero before every matrix op.
if (fast_y_loads && !full_y_tile) {
for (int l = valid_y_words + linear_tid; l < mmq_x * MMQ_TILE_Y_K; l += nthreads) {
tile_y[l] = 0;
}
}
#endif
float sum[mmq_x*mmq_y / (nwarps*warp_size)] = {0.0f};
@@ -3702,12 +3741,35 @@ static __device__ __forceinline__ void mul_mat_q_process_tile(
}
{
const int * by0 = y + ncols_y * (kb0 * qk / ne_block) * sz;
#if defined(GGML_USE_HIP) && defined(__gfx1151__)
if constexpr (fast_y_loads) {
if (full_y_tile) {
#pragma unroll
for (int l0 = 0; l0 < mmq_x * MMQ_TILE_Y_K; l0 += nthreads) {
const int l = l0 + linear_tid;
tile_y[l] = by0[l];
}
} else {
for (int l = linear_tid; l < valid_y_words; l += nthreads) {
tile_y[l] = by0[l];
}
}
} else {
#pragma unroll
for (int l0 = 0; l0 < mmq_x * MMQ_TILE_Y_K; l0 += nwarps * warp_size) {
int l = l0 + threadIdx.y*warp_size + threadIdx.x;
tile_y[l] = l < valid_y_words ? by0[l] : 0;
}
}
#else
#pragma unroll
for (int l0 = 0; l0 < mmq_x * MMQ_TILE_Y_K; l0 += nwarps * warp_size) {
int l = l0 + threadIdx.y*warp_size + threadIdx.x;
tile_y[l] = by0[l];
tile_y[l] = l < valid_y_words ? by0[l] : 0;
}
#endif
}
__syncthreads();
@@ -3718,12 +3780,35 @@ static __device__ __forceinline__ void mul_mat_q_process_tile(
{
const int * by0 = y + ncols_y * ((kb0 * qk / ne_block) * sz + sz);
#if defined(GGML_USE_HIP) && defined(__gfx1151__)
if constexpr (fast_y_loads) {
if (full_y_tile) {
#pragma unroll
for (int l0 = 0; l0 < mmq_x * MMQ_TILE_Y_K; l0 += nthreads) {
const int l = l0 + linear_tid;
tile_y[l] = by0[l];
}
} else {
for (int l = linear_tid; l < valid_y_words; l += nthreads) {
tile_y[l] = by0[l];
}
}
} else {
#pragma unroll
for (int l0 = 0; l0 < mmq_x * MMQ_TILE_Y_K; l0 += nwarps * warp_size) {
int l = l0 + threadIdx.y*warp_size + threadIdx.x;
tile_y[l] = l < valid_y_words ? by0[l] : 0;
}
}
#else
#pragma unroll
for (int l0 = 0; l0 < mmq_x * MMQ_TILE_Y_K; l0 += nwarps * warp_size) {
int l = l0 + threadIdx.y*warp_size + threadIdx.x;
tile_y[l] = by0[l];
tile_y[l] = l < valid_y_words ? by0[l] : 0;
}
#endif
}
__syncthreads();
cuda/mmq/mmvq.cu
+19 -16
View File
@@ -395,7 +395,7 @@ template <ggml_type type, int ncols_dst, bool has_fusion, bool small_k = false>
__launch_bounds__(calc_nwarps(type, ncols_dst, get_device_table_id())*ggml_cuda_get_physical_warp_size(), 1)
static __global__ void mul_mat_vec_q(
const void * __restrict__ vx, const void * __restrict__ vy, const int32_t * __restrict__ ids, const ggml_cuda_mm_fusion_args_device fusion, float * __restrict__ dst,
const uint32_t ncols_x, const uint3 nchannels_y, const uint32_t stride_row_x, const uint32_t stride_col_y,
const uint32_t ncols_x, const uint3 nchannels_y, const uint32_t nrows_x, const uint32_t stride_row_x, const uint32_t stride_col_y,
const uint32_t stride_col_dst, const uint3 channel_ratio, const uint32_t stride_channel_x,
const uint32_t stride_channel_y, const uint32_t stride_channel_dst, const uint3 sample_ratio,
const uint32_t stride_sample_x, const uint32_t stride_sample_y, const uint32_t stride_sample_dst,
@@ -463,7 +463,7 @@ static __global__ void mul_mat_vec_q(
// 1. Hide latency by prefetching bias and gate here
// 2. load only on threads that won't die after partial sum calculation
if (threadIdx.x < rows_per_cuda_block && threadIdx.y == 0 &&
(rows_per_cuda_block == 1 || uint32_t(row0 + threadIdx.x) < stride_col_dst)) {
uint32_t(row0 + threadIdx.x) < nrows_x) {
#pragma unroll
for (int j = 0; j < ncols_dst; ++j) {
x_biases[j] = x_bias[j * stride_col_dst + threadIdx.x];
@@ -473,7 +473,7 @@ static __global__ void mul_mat_vec_q(
if (use_gate_bias) {
gate_bias = gate_bias + sample_dst*stride_sample_dst + channel_bias*stride_channel_dst + row0;
if (threadIdx.x < rows_per_cuda_block && threadIdx.y == 0 &&
(rows_per_cuda_block == 1 || uint32_t(row0 + threadIdx.x) < stride_col_dst)) {
uint32_t(row0 + threadIdx.x) < nrows_x) {
#pragma unroll
for (int j = 0; j < ncols_dst; ++j) {
gate_biases[j] = gate_bias[j * stride_col_dst + threadIdx.x];
@@ -499,6 +499,8 @@ static __global__ void mul_mat_vec_q(
for (int j = 0; j < ncols_dst; ++j) {
#pragma unroll
for (int i = 0; i < rows_per_cuda_block; ++i) {
// Routed output strides include other experts, not just rows.
if (uint32_t(row0 + i) >= nrows_x) continue;
tmp[j][i] += vec_dot_q_cuda(
vx, &y[j*stride_col_y + kby], kbx_offset + i*stride_row_x + kbx, kqs);
if constexpr (has_fusion) {
@@ -562,7 +564,7 @@ static __global__ void mul_mat_vec_q(
}
}
if (threadIdx.x < rows_per_cuda_block && (rows_per_cuda_block == 1 || uint32_t(row0 + threadIdx.x) < stride_col_dst)) {
if (threadIdx.x < rows_per_cuda_block && uint32_t(row0 + threadIdx.x) < nrows_x) {
float result = tmp[j][threadIdx.x];
if constexpr (has_fusion) {
if (use_bias) {
@@ -654,6 +656,7 @@ static __global__ void mul_mat_vec_q_moe(
#pragma unroll
for (int i = 0; i < c_rows_per_block; ++i) {
if (uint32_t(row0 + i) >= nrows_x) continue;
tmp[i] += vec_dot_q_cuda(vx, &y[kby], kbx_offset + i*stride_row_x + kbx, kqs);
}
}
@@ -685,7 +688,7 @@ static std::pair<dim3, dim3> calc_launch_params(
template<ggml_type type, int c_ncols_dst, bool small_k = false>
static void mul_mat_vec_q_switch_fusion(
const void * vx, const void * vy, const int32_t * ids, const ggml_cuda_mm_fusion_args_device fusion, float * dst,
const uint32_t ncols_x, const uint3 nchannels_y, const uint32_t stride_row_x, const uint32_t stride_col_y,
const uint32_t ncols_x, const uint3 nchannels_y, const uint32_t nrows_x, const uint32_t stride_row_x, const uint32_t stride_col_y,
const uint32_t stride_col_dst, const uint3 channel_ratio, const uint32_t stride_channel_x,
const uint32_t stride_channel_y, const uint32_t stride_channel_dst, const uint3 sample_ratio,
const uint32_t stride_sample_x, const uint32_t stride_sample_y, const uint32_t stride_sample_dst,
@@ -696,7 +699,7 @@ static void mul_mat_vec_q_switch_fusion(
if constexpr (c_ncols_dst == 1) {
if (has_fusion) {
mul_mat_vec_q<type, c_ncols_dst, true, small_k><<<block_nums, block_dims, nbytes_shared, stream>>>
(vx, vy, ids, fusion, dst, ncols_x, nchannels_y, stride_row_x, stride_col_y, stride_col_dst,
(vx, vy, ids, fusion, dst, ncols_x, nchannels_y, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, ids_stride);
return;
@@ -706,7 +709,7 @@ static void mul_mat_vec_q_switch_fusion(
GGML_ASSERT(!has_fusion && "fusion only supported for ncols_dst=1");
mul_mat_vec_q<type, c_ncols_dst, false, small_k><<<block_nums, block_dims, nbytes_shared, stream>>>
(vx, vy, ids, fusion, dst, ncols_x, nchannels_y, stride_row_x, stride_col_y, stride_col_dst,
(vx, vy, ids, fusion, dst, ncols_x, nchannels_y, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, ids_stride);
}
@@ -819,7 +822,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst,
nsamples_dst, warp_size, table_id, true);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst, true>(
vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio_fd,
stride_sample_x, stride_sample_y, stride_sample_dst, dims.first, dims.second, 0, ids_stride,
stream);
@@ -827,7 +830,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst,
nsamples_dst, warp_size, table_id);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(
vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio_fd,
stride_sample_x, stride_sample_y, stride_sample_dst, dims.first, dims.second, 0, ids_stride,
stream);
@@ -836,7 +839,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
case 2: {
constexpr int c_ncols_dst = 2;
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst, nsamples_dst, warp_size, table_id);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio_fd, stride_sample_x, stride_sample_y, stride_sample_dst,
dims.first, dims.second, 0, ids_stride, stream);
@@ -844,7 +847,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
case 3: {
constexpr int c_ncols_dst = 3;
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst, nsamples_dst, warp_size, table_id);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio_fd, stride_sample_x, stride_sample_y, stride_sample_dst,
dims.first, dims.second, 0, ids_stride, stream);
@@ -852,7 +855,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
case 4: {
constexpr int c_ncols_dst = 4;
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst, nsamples_dst, warp_size, table_id);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio_fd, stride_sample_x, stride_sample_y, stride_sample_dst,
dims.first, dims.second, 0, ids_stride, stream);
@@ -860,7 +863,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
case 5: {
constexpr int c_ncols_dst = 5;
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst, nsamples_dst, warp_size, table_id);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio_fd, stride_sample_x, stride_sample_y, stride_sample_dst,
dims.first, dims.second, 0, ids_stride, stream);
@@ -868,7 +871,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
case 6: {
constexpr int c_ncols_dst = 6;
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst, nsamples_dst, warp_size, table_id);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio_fd, stride_sample_x, stride_sample_y, stride_sample_dst,
dims.first, dims.second, 0, ids_stride, stream);
@@ -876,7 +879,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
case 7: {
constexpr int c_ncols_dst = 7;
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst, nsamples_dst, warp_size, table_id);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio_fd, stride_sample_x, stride_sample_y, stride_sample_dst,
dims.first, dims.second, 0, ids_stride, stream);
@@ -884,7 +887,7 @@ static void mul_mat_vec_q_switch_ncols_dst(
case 8: {
constexpr int c_ncols_dst = 8;
std::pair<dim3, dim3> dims = calc_launch_params<type>(c_ncols_dst, nrows_x, nchannels_dst, nsamples_dst, warp_size, table_id);
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, stride_row_x, stride_col_y, stride_col_dst,
mul_mat_vec_q_switch_fusion<type, c_ncols_dst>(vx, vy, ids, fusion, dst, ncols_x, nchannels_y_fd, nrows_x, stride_row_x, stride_col_y, stride_col_dst,
channel_ratio_fd, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio_fd, stride_sample_x, stride_sample_y, stride_sample_dst,
dims.first, dims.second, 0, ids_stride, stream);
cuda/mmq/test/d2r_stubs.cu
+36
View File
@@ -0,0 +1,36 @@
#include "ds4_mmq_d2r.cuh"
extern "C" int ds4_cuda_q8_fold_take_q81(
const void *src, uint64_t in_dim, const void **q81) {
(void)src;
(void)in_dim;
(void)q81;
return 0;
}
// The D2R kernels use NVIDIA cp.async and MMA assembly. Disable them in ROCm
// builds and use the portable raw/SoA paths instead.
bool ds4_mmq_q2_K_moe_d2r_available(int) { return false; }
bool ds4_mmq_iq2_xxs_moe_d2r_available(int) { return false; }
bool ds4_mmq_q8_0_dense_d2r_available(int) { return false; }
size_t ds4_mmq_q2_K_moe_d2r_scratch_bytes(int64_t, int) { return 0; }
size_t ds4_mmq_iq2_xxs_moe_d2r_pair_scratch_bytes(int64_t, int) { return 0; }
size_t ds4_mmq_iq2_xxs_moe_d2r_fused_scratch_bytes(int64_t, int) { return 0; }
int ds4_mmq_q2_K_moe_d2r_launch(
const void *, int64_t, const void *, const int32_t *, const int32_t *,
float *, int, int, int64_t, int, void *, size_t, cudaStream_t) { return 1; }
int ds4_mmq_q8_0_dense_d2r_launch(
const void *, const void *, float *, int, int, int, cudaStream_t) { return 1; }
int ds4_mmq_iq2_xxs_moe_d2r_pair_launch(
const void *, const void *, int64_t, const void *, const int32_t *,
const int32_t *, float *, float *, int, int, int64_t, int, void *, size_t,
cudaStream_t) { return 1; }
int ds4_mmq_iq2_xxs_moe_d2r_fused_launch(
const void *, const void *, int64_t, const void *, const int32_t *, int,
const int32_t *, const int32_t *, const float *, void *, int, int, int64_t,
int, float, void *, size_t, cudaStream_t) { return 1; }
cuda/mmq/test/test_iq2_aligned_entry.cu
+132 -37
View File
@@ -1,18 +1,9 @@
// test_iq2_aligned_entry.cu — M1-Inc1 integration parity gate.
//
// A/B of the PRODUCTION aligned entry (ds4_mmq_iq2_xxs_aligned_moe_vec, fed
// the artifact layout the weight server builds under --repack-iq2-aligned)
// against the production raw-layout vec entry (ds4_mmq_iq2_xxs_moe_vec) at
// the decode shape. Both entries quantize the activation through
// quantize_row_q8_1_cuda, so only the float accumulation order differs.
//
// The kernel-level A/B with hand-rolled quantize lives in
// proto_iq2_aligned.cu (the original +12% proof); this test locks the
// integrated entry + artifact layout contract.
//
// Build (box):
// nvcc -O3 --use_fast_math -std=c++17 -arch=sm_121 -I.. \
// test_iq2_aligned_entry.cu ../*.o -lcudart -lcuda -o test_iq2_aligned_entry
// Production IQ2/Q2 MoE parity, including small verifier batches.
// Compare raw and aligned entries, fused gate/up and separate token rows,
// and fused down/sum against the sum of independent expert projections.
// Both entries use the same Q8_1 activation quantizer.
// Build/run: make test-cuda-dspark-moe CUDA_ARCH=sm_121
// Use --check-only to skip timing loops under Compute Sanitizer.
#include "ds4_mmq.h"
@@ -30,19 +21,79 @@
#define CK(x) do { cudaError_t e_ = (x); if (e_ != cudaSuccess) { \
printf("CUDA ERR %s @%d: %s\n", #x, __LINE__, cudaGetErrorString(e_)); exit(1); } } while (0)
static int test_q2_down(cudaStream_t stream, std::mt19937 &rng) {
const int rows = 256, columns = 2048, experts = 64;
const size_t pairs = (size_t)experts * (rows / 2) * (columns / 256);
std::vector<uint8_t> artifact(ds4_mmq_q2_k_aligned_bytes(rows, columns, experts));
for (auto &byte : artifact) byte = (uint8_t)rng();
for (size_t i = 0; i < pairs * 8; i += 2) {
const uint16_t scale = 0x2400;
memcpy(artifact.data() + i, &scale, 2);
}
void *aligned, *raw;
CK(cudaMalloc(&aligned, artifact.size()));
CK(cudaMalloc(&raw, (size_t)experts * rows * (columns / 256) * 84));
CK(cudaMemcpy(aligned, artifact.data(), artifact.size(), cudaMemcpyHostToDevice));
if (ds4_mmq_q2_K_aligned_derepack(aligned, raw, rows, columns, experts, stream)) return 1;
int bad = 0;
for (int tokens = 1; tokens <= 8; tokens++) {
const int assignments = tokens * 6;
std::vector<float> x((size_t)assignments * columns);
for (float &v : x) v = ((int)(rng() % 2000) - 1000) / 500.0f;
std::vector<int32_t> ids(assignments);
for (int i = 0; i < assignments; i++) ids[i] = (i * 7) % experts;
ids.back() = -1;
float *dx, *a, *b;
int32_t *di;
const size_t out_bytes = (size_t)assignments * rows * sizeof(float);
CK(cudaMalloc(&dx, x.size() * sizeof(float)));
CK(cudaMalloc(&di, ids.size() * sizeof(int32_t)));
CK(cudaMalloc(&a, out_bytes));
CK(cudaMalloc(&b, out_bytes));
CK(cudaMemcpyAsync(dx, x.data(), x.size() * sizeof(float), cudaMemcpyHostToDevice, stream));
CK(cudaMemcpyAsync(di, ids.data(), ids.size() * sizeof(int32_t), cudaMemcpyHostToDevice, stream));
if (ds4_mmq_q2_K_aligned_moe_vec(aligned, dx, di, a, rows, columns,
assignments, experts, 1, stream) ||
ds4_mmq_q2_K_moe_vec(raw, dx, di, b, rows, columns,
assignments, experts, 1, stream)) return 1;
CK(cudaStreamSynchronize(stream));
std::vector<float> got(out_bytes / sizeof(float)), ref(got.size());
CK(cudaMemcpy(got.data(), a, out_bytes, cudaMemcpyDeviceToHost));
CK(cudaMemcpy(ref.data(), b, out_bytes, cudaMemcpyDeviceToHost));
if (memcmp(got.data(), ref.data(), out_bytes)) bad++;
if (ds4_mmq_q2_K_moe_down_sum6_vec(raw, dx, di, a, rows, columns,
tokens, experts, 6, stream)) return 1;
CK(cudaStreamSynchronize(stream));
CK(cudaMemcpy(got.data(), a, (size_t)tokens * rows * sizeof(float), cudaMemcpyDeviceToHost));
for (int t = 0; t < tokens; t++) {
for (int row = 0; row < rows; row++) {
float expected = 0.0f;
for (int s = 0; s < 6; s++) expected += ref[((size_t)t * 6 + s) * rows + row];
const float actual = got[(size_t)t * rows + row];
if (!std::isfinite(actual) || fabsf(actual - expected) > 1e-3f + fabsf(expected) * 2e-5f) bad++;
}
}
CK(cudaFree(dx)); CK(cudaFree(di)); CK(cudaFree(a)); CK(cudaFree(b));
}
CK(cudaFree(aligned)); CK(cudaFree(raw));
printf("Q2 down assignments 6..48: layout/sum failures=%d\n", bad);
return bad;
}
int main(int argc, char **argv) {
const bool parity_only = argc > 1 && !strcmp(argv[1], "--check-only");
const int M = 2048, K = 4096;
const int n_experts = 256;
const int n_slots = 6;
const int n_idsets = 32;
const int iters = 2000;
const int iters = parity_only ? 1 : 2000;
const int nb = K / 256;
const long long row_bytes = (long long)nb * 66;
const double slot_gb = (double)n_slots * M * row_bytes / 1e9;
if (ds4_mmq_init(0) != 0) { printf("ds4_mmq_init failed\n"); return 1; }
std::mt19937 rng(argc > 1 ? (uint32_t)atoi(argv[1]) : 1234u);
std::mt19937 rng(argc > 1 && !parity_only ? (uint32_t)atoi(argv[1]) : 1234u);
// Random bytes are valid IQ2_XXS content; keep block scales sane.
std::vector<uint8_t> W((size_t)n_experts * M * row_bytes);
@@ -114,7 +165,7 @@ int main(int argc, char **argv) {
return 1;
}
CK(cudaEventRecord(e0, stream));
const int dr_iters = 50;
const int dr_iters = parity_only ? 1 : 50;
for (int i = 0; i < dr_iters; i++)
(void)ds4_mmq_iq2_xxs_aligned_derepack(dArt, dRaw2, M, K, n_experts, stream);
CK(cudaEventRecord(e1, stream));
@@ -221,6 +272,7 @@ int main(int argc, char **argv) {
double max_rel = 0.0, max_abs = 0.0; int bad = 0;
for (size_t i = 0; i < ref.size(); i++) {
const double a = ref[i], b = got[i];
if (!std::isfinite(a) || !std::isfinite(b)) { bad++; continue; }
const double ad = fabs(a - b);
const double rd = ad / (fabs(a) > 1e-3 ? fabs(a) : 1e-3);
if (ad > max_abs) max_abs = ad;
@@ -254,23 +306,34 @@ int main(int argc, char **argv) {
}
int bad_mid = count_bad(em, om, nullptr, nullptr);
// ---- multi-token parity (batch_eval-hang fix, 2026-07-03): aligned vs
// raw production entries at n_tokens=4, flat assignment layout
// [tok*n_slots+slot], including one -1 router id (NaN-path guard).
int bad4 = 0, bad4_mid = 0;
{
const int n_tok = 4, n_asg = n_tok * n_slots;
// Tiny verifier batches: distinct, shared and partly overlapping experts.
// Deduplicating gate/up must be exact against separate one-token calls.
// Use different gate/up matrices so swapped operands cannot pass.
std::vector<uint8_t> up_art = ART;
for (size_t i = dq_bytes; i < up_art.size(); i++) up_art[i] ^= 0x35;
uint8_t *dRawUp;
CK(cudaMalloc(&dRawUp, W.size()));
CK(cudaMemcpyAsync(dArt2, up_art.data(), up_art.size(), cudaMemcpyHostToDevice, stream));
if (ds4_mmq_iq2_xxs_aligned_derepack(dArt2, dRawUp, M, K, n_experts, stream)) return 1;
int bad4 = 0, bad4_mid = 0, bad_dedup = test_q2_down(stream, rng);
for (int n_tok = 1; n_tok <= 8; n_tok++) {
for (int overlap = 0; overlap < 3; overlap++) {
const int n_asg = n_tok * n_slots;
std::vector<float> X4((size_t)n_tok * K);
for (auto &v : X4) v = ((float)(rng() % 2000) - 1000.0f) / 500.0f;
std::vector<int32_t> ids4(n_asg);
for (int a = 0; a < n_asg; a++) ids4[a] = (a * 7 + 3) % n_experts;
ids4[13] = -1; // router NaN path: both entries must write a clean 0 row
for (int a = 0; a < n_asg; a++) {
ids4[a] = overlap == 0 ? (a * 7 + 3) % n_experts :
(a % n_slots + (overlap == 2 ? 2 * (a / n_slots) : 0)) % n_experts;
}
if (n_asg > 13) ids4[13] = -1;
std::vector<float> w4(n_asg);
for (int a = 0; a < n_asg; a++) w4[a] = 0.25f + 0.05f * a;
float *dX4, *dOut4B, *dOut4A, *dMid4, *dW4; int32_t *dIds4;
float *dX4, *dOut4B, *dOut4A, *dUp4B, *dMid4, *dW4; int32_t *dIds4;
CK(cudaMalloc(&dX4, sizeof(float) * X4.size()));
CK(cudaMalloc(&dOut4B, sizeof(float) * n_asg * M));
CK(cudaMalloc(&dOut4A, sizeof(float) * n_asg * M));
CK(cudaMalloc(&dUp4B, sizeof(float) * n_asg * M));
CK(cudaMalloc(&dMid4, sizeof(float) * n_asg * M));
CK(cudaMalloc(&dW4, sizeof(float) * n_asg));
CK(cudaMalloc(&dIds4, sizeof(int32_t) * n_asg));
@@ -281,8 +344,10 @@ int main(int argc, char **argv) {
int r2 = ds4_mmq_iq2_xxs_aligned_moe_vec(dArt, dX4, dIds4, dOut4A, M, K, n_tok, n_experts, n_slots, stream);
int r3 = ds4_mmq_iq2_xxs_aligned_moe_gate_up_mid_vec(dArt, dArt2, dX4, dIds4, dW4, dMid4,
M, K, n_tok, n_experts, n_slots, clampv, stream);
if (r1 != 0 || r2 != 0 || r3 != 0) {
printf("multi-token entries rc=%d/%d/%d\n", r1, r2, r3);
int r4 = ds4_mmq_iq2_xxs_moe_vec(dRawUp, dX4, dIds4, dUp4B,
M, K, n_tok, n_experts, n_slots, stream);
if (r1 != 0 || r2 != 0 || r3 != 0 || r4 != 0) {
printf("multi-token entries rc=%d/%d/%d/%d\n", r1, r2, r3, r4);
return 1;
}
CK(cudaStreamSynchronize(stream));
@@ -290,11 +355,24 @@ int main(int argc, char **argv) {
CK(cudaMemcpy(ob4.data(), dOut4B, sizeof(float) * ob4.size(), cudaMemcpyDeviceToHost));
CK(cudaMemcpy(oa4.data(), dOut4A, sizeof(float) * oa4.size(), cudaMemcpyDeviceToHost));
CK(cudaMemcpy(om4.data(), dMid4, sizeof(float) * om4.size(), cudaMemcpyDeviceToHost));
bad4 = count_bad(ob4, oa4, nullptr, nullptr);
std::vector<float> up4(om4.size());
CK(cudaMemcpy(up4.data(), dUp4B, sizeof(float) * up4.size(), cudaMemcpyDeviceToHost));
bad4 += count_bad(ob4, oa4, nullptr, nullptr);
for (int t = 0; t < n_tok; t++) {
if (ds4_mmq_iq2_xxs_aligned_moe_gate_up_mid_vec(
dArt, dArt2, dX4 + t * K, dIds4 + t * n_slots,
dW4 + t * n_slots, dMid4 + t * n_slots * M,
M, K, 1, n_experts, n_slots, clampv, stream) != 0) return 1;
}
CK(cudaStreamSynchronize(stream));
std::vector<float> serial_mid(om4.size());
CK(cudaMemcpy(serial_mid.data(), dMid4, sizeof(float) * serial_mid.size(),
cudaMemcpyDeviceToHost));
if (memcmp(om4.data(), serial_mid.data(), sizeof(float) * om4.size())) bad_dedup++;
std::vector<float> em4((size_t)n_asg * M);
for (int a = 0; a < n_asg; a++) {
for (int r = 0; r < M; r++) {
float g = ob4[(size_t)a * M + r], u = g;
float g = ob4[(size_t)a * M + r], u = up4[(size_t)a * M + r];
if (!std::isfinite(g)) g = 0.0f;
if (!std::isfinite(u)) u = 0.0f;
if (clampv > 1.0e-6f) {
@@ -306,10 +384,27 @@ int main(int argc, char **argv) {
em4[(size_t)a * M + r] = silu * u * w4[a];
}
}
bad4_mid = count_bad(em4, om4, nullptr, nullptr);
bad4_mid += count_bad(em4, om4, nullptr, nullptr);
if (ds4_mmq_iq2_xxs_moe_gate_up_mid_vec(
dW, dRawUp, dX4, dIds4, dW4, dMid4,
M, K, n_tok, n_experts, n_slots, clampv, stream)) return 1;
CK(cudaStreamSynchronize(stream));
CK(cudaMemcpy(om4.data(), dMid4, sizeof(float) * om4.size(), cudaMemcpyDeviceToHost));
bad4_mid += count_bad(em4, om4, nullptr, nullptr);
for (int t = 0; t < n_tok; t++) {
if (ds4_mmq_iq2_xxs_moe_gate_up_mid_vec(
dW, dRawUp, dX4 + t * K, dIds4 + t * n_slots,
dW4 + t * n_slots, dMid4 + t * n_slots * M,
M, K, 1, n_experts, n_slots, clampv, stream)) return 1;
}
CK(cudaStreamSynchronize(stream));
CK(cudaMemcpy(serial_mid.data(), dMid4, sizeof(float) * serial_mid.size(), cudaMemcpyDeviceToHost));
if (memcmp(om4.data(), serial_mid.data(), sizeof(float) * om4.size())) bad_dedup++;
CK(cudaFree(dX4)); CK(cudaFree(dOut4B)); CK(cudaFree(dOut4A));
CK(cudaFree(dMid4)); CK(cudaFree(dW4)); CK(cudaFree(dIds4));
CK(cudaFree(dMid4)); CK(cudaFree(dW4)); CK(cudaFree(dIds4)); CK(cudaFree(dUp4B));
}
}
CK(cudaFree(dRawUp));
printf("TEST_IQ2_ALIGNED_ENTRY M=%d K=%d slots=%d experts=%d iters=%d (weights/iter %.1f MB)\n",
M, K, n_slots, n_experts, iters, slot_gb * 1000.0);
@@ -322,8 +417,8 @@ int main(int argc, char **argv) {
printf(" fused entry : %.4f ms -> %6.1f GB/s (vs 2x aligned %+.1f%%)\n",
ms_fused, 2.0 * slot_gb / (ms_fused / 1e3), 100.0 * (2.0f * ms_al / ms_fused - 1.0));
printf(" parity: aligned max_rel=%.3e max_abs=%.3e bad=%d | pair bad=%d | fused-mid bad=%d"
" | ntok4 bad=%d ntok4-mid bad=%d -> %s\n",
max_rel, max_abs, bad, bad_pair, bad_mid, bad4, bad4_mid,
(bad == 0 && bad_pair == 0 && bad_mid == 0 && bad4 == 0 && bad4_mid == 0) ? "PASS" : "FAIL");
return (bad == 0 && bad_pair == 0 && bad_mid == 0 && bad4 == 0 && bad4_mid == 0) ? 0 : 2;
" | tiny-batch bad=%d mid bad=%d dedup-exact bad=%d -> %s\n",
max_rel, max_abs, bad, bad_pair, bad_mid, bad4, bad4_mid, bad_dedup,
(bad == 0 && bad_pair == 0 && bad_mid == 0 && bad4 == 0 && bad4_mid == 0 && bad_dedup == 0) ? "PASS" : "FAIL");
return (bad == 0 && bad_pair == 0 && bad_mid == 0 && bad4 == 0 && bad4_mid == 0 && bad_dedup == 0) ? 0 : 2;
}
cuda/mmq/test/test_mmq_parity.cu
+65 -20
View File
@@ -24,20 +24,28 @@
// Pull in the block_* struct definitions. We use the CUDA decl/impl mode
// so the field paths match what the vendored mmq code uses (anonymous
// outer union + named "data" inner struct). cuda_fp16.h is available
// because nvcc compiles this TU. Half-precision conversions go via
// outer union + named "data" inner struct). Half-precision conversions go via
// __half_raw <-> uint16_t bit patterns, which makes the CPU-side
// fp16<->float helpers below independent of any host-side fp16 ABI.
//
// We DON'T use the host IQ2 lookup tables from this mode (they'd be
// __device__). iq2_host_tables.h instead provides plain host const
// arrays generated directly from ggml-common.h's bit-for-bit contents.
#if defined(GGML_USE_HIP)
#define GGML_COMMON_DECL_HIP
#define GGML_COMMON_IMPL_HIP
#else
#define GGML_COMMON_DECL_CUDA
#define GGML_COMMON_IMPL_CUDA
#endif
#include "../ggml-common.h"
#if defined(GGML_USE_HIP)
#include "vendors/hip.h"
#else
#include <cuda_runtime.h>
#include <cuda_fp16.h>
#endif
#include <algorithm>
#include <cmath>
@@ -719,7 +727,9 @@ bool run_moe_pair_generic(
const int32_t *, float *, float *,
int, int, int, int, int, cudaStream_t),
int (*single_entry)(const void *, const float *, const int32_t *,
float *, int, int, int, int, int, cudaStream_t)) {
float *, int, int, int, int, int, cudaStream_t),
bool default_stream = false,
int repeats = 1) {
fprintf(stderr, "=== %s/PAIR M=%d K=%d ntok=%d nexp=%d nused=%d seed=%u ===\n",
tag, M, K, n_tokens, n_experts, n_expert_used, seed);
@@ -752,8 +762,10 @@ bool run_moe_pair_generic(
const int64_t ne_get_rows = (int64_t)n_tokens * n_expert_used;
const size_t out_count = (size_t)M * ne_get_rows;
const size_t pair_count = out_count * (size_t)repeats;
cudaStream_t stream; cudaStreamCreate(&stream);
cudaStream_t stream = nullptr;
if (!default_stream) cudaStreamCreate(&stream);
void * dWa = nullptr; void * dWb = nullptr;
float * dX = nullptr; int32_t * dIds = nullptr;
float * dYa_single = nullptr; float * dYb_single = nullptr;
@@ -764,45 +776,69 @@ bool run_moe_pair_generic(
cudaMalloc(&dIds, ids.size() * sizeof(int32_t));
cudaMalloc(&dYa_single, out_count * sizeof(float));
cudaMalloc(&dYb_single, out_count * sizeof(float));
cudaMalloc(&dYa_pair, out_count * sizeof(float));
cudaMalloc(&dYb_pair, out_count * sizeof(float));
cudaMalloc(&dYa_pair, pair_count * sizeof(float));
cudaMalloc(&dYb_pair, pair_count * sizeof(float));
cudaMemcpyAsync(dWa, W_a.data(), W_a.size() * sizeof(BlockT), cudaMemcpyHostToDevice, stream);
cudaMemcpyAsync(dWb, W_b.data(), W_b.size() * sizeof(BlockT), cudaMemcpyHostToDevice, stream);
cudaMemcpyAsync(dX, X.data(), X.size() * sizeof(float), cudaMemcpyHostToDevice, stream);
cudaMemcpyAsync(dIds, ids.data(), ids.size() * sizeof(int32_t), cudaMemcpyHostToDevice, stream);
cudaMemsetAsync(dYa_single, 0, out_count * sizeof(float), stream);
cudaMemsetAsync(dYb_single, 0, out_count * sizeof(float), stream);
cudaMemsetAsync(dYa_pair, 0, out_count * sizeof(float), stream);
cudaMemsetAsync(dYb_pair, 0, out_count * sizeof(float), stream);
cudaMemsetAsync(dYa_pair, 0, pair_count * sizeof(float), stream);
cudaMemsetAsync(dYb_pair, 0, pair_count * sizeof(float), stream);
int rc_sa = single_entry(dWa, dX, dIds, dYa_single, M, K, n_tokens, n_experts, n_expert_used, stream);
int rc_sb = single_entry(dWb, dX, dIds, dYb_single, M, K, n_tokens, n_experts, n_expert_used, stream);
int rc_p = pair_entry (dWa, dWb, dX, dIds, dYa_pair, dYb_pair,
M, K, n_tokens, n_experts, n_expert_used, stream);
int rc_p = 0;
for (int repeat = 0; repeat < repeats && rc_p == 0; repeat++) {
rc_p = pair_entry(
dWa, dWb, dX, dIds,
dYa_pair + (size_t)repeat * out_count,
dYb_pair + (size_t)repeat * out_count,
M, K, n_tokens, n_experts, n_expert_used, stream);
}
if (rc_sa != 0 || rc_sb != 0 || rc_p != 0) {
fprintf(stderr, "%s pair entry: rc_sa=%d rc_sb=%d rc_p=%d\n", tag, rc_sa, rc_sb, rc_p);
cudaFree(dWa); cudaFree(dWb); cudaFree(dX); cudaFree(dIds);
cudaFree(dYa_single); cudaFree(dYb_single); cudaFree(dYa_pair); cudaFree(dYb_pair);
cudaStreamDestroy(stream);
if (stream) cudaStreamDestroy(stream);
return false;
}
std::vector<float> ya_single(out_count, 0.0f), yb_single(out_count, 0.0f);
std::vector<float> ya_pair (out_count, 0.0f), yb_pair (out_count, 0.0f);
std::vector<float> ya_pair(pair_count, 0.0f), yb_pair(pair_count, 0.0f);
cudaMemcpyAsync(ya_single.data(), dYa_single, out_count * sizeof(float), cudaMemcpyDeviceToHost, stream);
cudaMemcpyAsync(yb_single.data(), dYb_single, out_count * sizeof(float), cudaMemcpyDeviceToHost, stream);
cudaMemcpyAsync(ya_pair.data(), dYa_pair, out_count * sizeof(float), cudaMemcpyDeviceToHost, stream);
cudaMemcpyAsync(yb_pair.data(), dYb_pair, out_count * sizeof(float), cudaMemcpyDeviceToHost, stream);
cudaMemcpyAsync(ya_pair.data(), dYa_pair, pair_count * sizeof(float), cudaMemcpyDeviceToHost, stream);
cudaMemcpyAsync(yb_pair.data(), dYb_pair, pair_count * sizeof(float), cudaMemcpyDeviceToHost, stream);
cudaStreamSynchronize(stream);
cudaFree(dWa); cudaFree(dWb); cudaFree(dX); cudaFree(dIds);
cudaFree(dYa_single); cudaFree(dYb_single); cudaFree(dYa_pair); cudaFree(dYb_pair);
cudaStreamDestroy(stream);
if (stream) cudaStreamDestroy(stream);
// Both pair outputs should be bit-identical to their single counterparts -
// same kernel, same Q8_1 buffer. Allow exactly zero abs/rel tolerance.
const bool ok_a = check_close(ya_pair, ya_single, 0.0f, 0.0f);
const bool ok_b = check_close(yb_pair, yb_single, 0.0f, 0.0f);
const bool ok = ok_a && ok_b;
// The pair path can choose a tighter tile width than two separate calls,
// which changes accumulation order slightly. Repeated pair calls must
// still be bit-identical to the first pair result.
#if defined(GGML_USE_HIP)
constexpr float pair_atol = 2.0e-4f;
constexpr float pair_rtol = 1.0e-5f;
#else
constexpr float pair_atol = 0.0f;
constexpr float pair_rtol = 0.0f;
#endif
const std::vector<float> ya_first(ya_pair.begin(), ya_pair.begin() + out_count);
const std::vector<float> yb_first(yb_pair.begin(), yb_pair.begin() + out_count);
bool ok_a = check_close(ya_first, ya_single, pair_atol, pair_rtol);
bool ok_b = check_close(yb_first, yb_single, pair_atol, pair_rtol);
for (int repeat = 1; repeat < repeats; repeat++) {
const auto a_first = ya_pair.begin() + (size_t)repeat * out_count;
const auto b_first = yb_pair.begin() + (size_t)repeat * out_count;
const std::vector<float> ya(a_first, a_first + out_count);
const std::vector<float> yb(b_first, b_first + out_count);
ok_a &= check_close(ya, ya_first, 0.0f, 0.0f);
ok_b &= check_close(yb, yb_first, 0.0f, 0.0f);
}
const bool ok = ok_a && ok_b;
fprintf(stderr, "%s\n\n", ok ? "PASS" : "FAIL");
return ok;
}
@@ -1197,6 +1233,15 @@ int main(int argc, char ** argv) {
"IQ2_XXS", QK_K_LOCAL, /*M=*/256, /*K=*/512, /*nt=*/8,
/*ne=*/16, /*nu=*/6, 0xC0FE10, gen_iq2,
ds4_mmq_iq2_xxs_moe_pair, ds4_mmq_iq2_xxs_moe);
// Production admission geometry. This uses the default stream so the
// persistent routing maps and tight expert-column bound are exercised
// together.
all_ok &= run_moe_pair_generic<block_iq2_xxs>(
"IQ2_XXS/PRODUCTION_ADMISSION", QK_K_LOCAL,
/*M=*/256, /*K=*/512, /*nt=*/371, /*ne=*/256, /*nu=*/6,
0xC0FE11, gen_iq2,
ds4_mmq_iq2_xxs_moe_pair, ds4_mmq_iq2_xxs_moe,
/*default_stream=*/true, /*repeats=*/32);
all_ok &= run_moe_pair_generic<block_q4_K>(
"Q4_K", QK_K_LOCAL, /*M=*/256, /*K=*/512, /*nt=*/8,
/*ne=*/16, /*nu=*/6, 0xC4FE10, gen_q4k,
cuda/mmq/test/test_mmq_soa_tiles.cu
+11 -6
View File
@@ -30,7 +30,11 @@
#include "ds4_mmq.h"
#if defined(GGML_USE_HIP)
#include "vendors/hip.h"
#else
#include <cuda_runtime.h>
#endif
// libds4mmq.a references this ds4_cuda.cu symbol from the q8-fold vec paths
// (C3 Inc4); the entries under test never reach it, so a "no fold available"
@@ -123,9 +127,9 @@ float time_ms(F &&fn, int reps) {
// ---------------------------------------------------------------------------
// Q2_K down leg.
// ---------------------------------------------------------------------------
void run_q2k_leg(std::mt19937 &rng, int reps) {
void run_q2k_leg(std::mt19937 &rng, int reps, int prompt_tokens) {
const int E = 256, M = 4096, K = 2048;
const int n_tokens = 4096 * 6; // assignments, n_expert_used == 1
const int n_tokens = prompt_tokens * 6; // assignments, n_expert_used == 1
const uint64_t nb_row = (uint64_t)K / 256;
const uint64_t nblk = (uint64_t)E * M * nb_row;
const uint64_t npair = nblk / 2;
@@ -192,9 +196,9 @@ void run_q2k_leg(std::mt19937 &rng, int reps) {
// ---------------------------------------------------------------------------
// IQ2_XXS gate/up pair leg.
// ---------------------------------------------------------------------------
void run_iq2_leg(std::mt19937 &rng, int reps) {
void run_iq2_leg(std::mt19937 &rng, int reps, int prompt_tokens) {
const int E = 256, M = 2048, K = 4096;
const int n_tokens = 4096, n_used = 6;
const int n_tokens = prompt_tokens, n_used = 6;
const uint64_t nb_row = (uint64_t)K / 256;
const uint64_t nblk = (uint64_t)E * M * nb_row;
const uint64_t soa_bytes = ds4_mmq_iq2_xxs_aligned_bytes(M, K, E);
@@ -277,9 +281,10 @@ void run_iq2_leg(std::mt19937 &rng, int reps) {
int main(int argc, char **argv) {
const int reps = (argc > 1) ? atoi(argv[1]) : 20;
const int prompt_tokens = (argc > 2) ? atoi(argv[2]) : 4096;
std::mt19937 rng(0x5A0Au);
run_q2k_leg(rng, reps);
run_iq2_leg(rng, reps);
run_q2k_leg(rng, reps, prompt_tokens);
run_iq2_leg(rng, reps, prompt_tokens);
printf(g_failures ? "SOA-TILES FAIL (%d)\n" : "SOA-TILES PASS\n", g_failures);
return g_failures ? 1 : 0;
}
cuda/mmq/vendors/hip.h
Vendored
+308
View File
@@ -0,0 +1,308 @@
#pragma once
#define HIP_DISABLE_WARP_SYNC_BUILTINS 1
#include <hip/hip_runtime.h>
#include <hipblas/hipblas.h>
#include <hip/hip_fp16.h>
#include <hip/hip_bf16.h>
#ifdef GGML_USE_NCCL
#include <rccl/rccl.h>
#endif // GGML_USE_NCCL
#define CUBLAS_GEMM_DEFAULT HIPBLAS_GEMM_DEFAULT
#define CUBLAS_GEMM_DEFAULT_TENSOR_OP HIPBLAS_GEMM_DEFAULT
#define CUBLAS_OP_N HIPBLAS_OP_N
#define CUBLAS_OP_T HIPBLAS_OP_T
#define CUBLAS_STATUS_SUCCESS HIPBLAS_STATUS_SUCCESS
#define CUBLAS_TF32_TENSOR_OP_MATH 0
#define CUDA_R_16F HIPBLAS_R_16F
#define CUDA_R_16BF HIPBLAS_R_16B
#define CUDA_R_32F HIPBLAS_R_32F
#define CUBLAS_SIDE_RIGHT HIPBLAS_SIDE_RIGHT
#define CUBLAS_FILL_MODE_UPPER HIPBLAS_FILL_MODE_UPPER
#define CUBLAS_DIAG_NON_UNIT HIPBLAS_DIAG_NON_UNIT
#define CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED hipDeviceAttributeVirtualMemoryManagementSupported
#define CU_MEM_ALLOC_GRANULARITY_RECOMMENDED hipMemAllocationGranularityRecommended
#define CU_MEM_ALLOCATION_TYPE_PINNED hipMemAllocationTypePinned
#define CU_MEM_LOCATION_TYPE_DEVICE hipMemLocationTypeDevice
#define CU_MEM_ACCESS_FLAGS_PROT_READWRITE hipMemAccessFlagsProtReadWrite
#define CU_CHECK(fn) {hipError_t err = fn; if(err != hipSuccess) { GGML_ABORT("HipVMM Failure: %s\n", hipGetErrorString(err)); }}
#define __shfl_sync(mask, var, laneMask, width) __shfl(var, laneMask, width)
#define __shfl_up_sync(mask, var, laneMask, width) __shfl_up(var, laneMask, width)
#define __shfl_down_sync(mask, var, laneMask, ...) __shfl_down(var, laneMask, ##__VA_ARGS__)
#define __shfl_xor_sync(mask, var, laneMask, width) __shfl_xor(var, laneMask, width)
#define __all_sync(mask, var) __all(var)
#define __any_sync(mask, var) __any(var)
#define cublasStrsmBatched hipblasStrsmBatched
#define cublasCreate hipblasCreate
#define cublasDestroy hipblasDestroy
#define cublasGemmEx hipblasGemmEx
#define cublasGemmBatchedEx hipblasGemmBatchedEx
#define cublasGemmStridedBatchedEx hipblasGemmStridedBatchedEx
#define cublasHandle_t hipblasHandle_t
#define cublasSetMathMode(handle, mode) CUBLAS_STATUS_SUCCESS
#define cublasSetStream hipblasSetStream
#define cublasSgemm hipblasSgemm
#define cublasSgemmBatched hipblasSgemmBatched
#define cublasSgemmStridedBatched hipblasSgemmStridedBatched
#define cublasStatus_t hipblasStatus_t
#define cublasOperation_t hipblasOperation_t
#define cudaDevAttrCooperativeLaunch hipDeviceAttributeCooperativeLaunch
#define cudaDeviceCanAccessPeer hipDeviceCanAccessPeer
#define cudaDeviceDisablePeerAccess hipDeviceDisablePeerAccess
#define cudaDeviceEnablePeerAccess hipDeviceEnablePeerAccess
#define cudaDeviceGetAttribute hipDeviceGetAttribute
#define cudaDeviceGetPCIBusId hipDeviceGetPCIBusId
#define cudaDeviceProp hipDeviceProp_t
#define cudaDeviceSynchronize hipDeviceSynchronize
#define cudaError_t hipError_t
#define cudaErrorMemoryAllocation hipErrorOutOfMemory
#define cudaErrorPeerAccessAlreadyEnabled hipErrorPeerAccessAlreadyEnabled
#define cudaErrorPeerAccessNotEnabled hipErrorPeerAccessNotEnabled
#define cudaEventCreateWithFlags hipEventCreateWithFlags
#define cudaEventDisableTiming hipEventDisableTiming
#define cudaEventRecord hipEventRecord
#define cudaEventSynchronize hipEventSynchronize
#define cudaEvent_t hipEvent_t
#define cudaEventDestroy hipEventDestroy
#define cudaFree hipFree
#define cudaFreeAsync hipFreeAsync
#define cudaFreeHost hipHostFree
#define cudaGetDevice hipGetDevice
#define cudaGetDeviceCount hipGetDeviceCount
#define cudaGetDeviceProperties hipGetDeviceProperties
#define cudaGetErrorString hipGetErrorString
#define cudaGetLastError hipGetLastError
#define cudaHostRegister hipHostRegister
#define cudaHostRegisterPortable hipHostRegisterPortable
#define cudaHostRegisterReadOnly hipHostRegisterReadOnly
#define cudaHostUnregister hipHostUnregister
#define cudaLaunchCooperativeKernel hipLaunchCooperativeKernel
#define cudaLaunchHostFunc hipLaunchHostFunc
#define cudaMalloc hipMalloc
#define cudaMallocAsync hipMallocAsync
#define cudaMallocHost(ptr, size) hipHostMalloc(ptr, size, hipHostMallocDefault)
#define cudaMallocManaged hipMallocManaged
#define cudaMemAdvise hipMemAdvise
#define cudaMemcpy hipMemcpy
#define cudaMemcpyAsync hipMemcpyAsync
#define cudaMemcpyPeerAsync hipMemcpyPeerAsync
#define cudaMemcpy2DAsync hipMemcpy2DAsync
#define cudaMemcpyDeviceToDevice hipMemcpyDeviceToDevice
#define cudaMemcpyDeviceToHost hipMemcpyDeviceToHost
#define cudaMemcpyHostToDevice hipMemcpyHostToDevice
#define cudaMemcpyKind hipMemcpyKind
#define cudaMemset hipMemset
#define cudaMemsetAsync hipMemsetAsync
#define cudaMemGetInfo hipMemGetInfo
#define cudaOccupancyMaxPotentialBlockSize hipOccupancyMaxPotentialBlockSize
#define cudaSetDevice hipSetDevice
#define cuDeviceGet hipDeviceGet
#define CUdevice hipDevice_t
#define CUdeviceptr hipDeviceptr_t
#define cuMemUnmap hipMemUnmap
#define CUmemAccessDesc hipMemAccessDesc
#define cuMemAddressFree hipMemAddressFree
#define cuMemRelease hipMemRelease
#define CUmemGenericAllocationHandle hipMemGenericAllocationHandle_t
#define cuMemCreate hipMemCreate
#define cuMemAddressReserve hipMemAddressReserve
#define cuMemMap hipMemMap
#define cuMemSetAccess hipMemSetAccess
#define cuMemGetAllocationGranularity hipMemGetAllocationGranularity
#define CUmemAllocationProp hipMemAllocationProp
#define cuDeviceGetAttribute hipDeviceGetAttribute
#define cudaStreamCreateWithFlags hipStreamCreateWithFlags
#define cudaStreamCreate hipStreamCreate
#define cudaStreamDestroy hipStreamDestroy
#define cudaStreamFireAndForget hipStreamFireAndForget
#define cudaStreamNonBlocking hipStreamNonBlocking
#define cudaStreamPerThread hipStreamPerThread
#define cudaStreamSynchronize hipStreamSynchronize
#define cudaStreamCaptureStatus hipStreamCaptureStatus
#define cudaStreamCaptureStatusNone hipStreamCaptureStatusNone
#define cudaStreamIsCapturing hipStreamIsCapturing
#define cudaStreamWaitEvent hipStreamWaitEvent
#define cudaGraphExec_t hipGraphExec_t
#define cudaGraphNode_t hipGraphNode_t
#define cudaKernelNodeParams hipKernelNodeParams
#define cudaKernelNodeParams hipKernelNodeParams
#define cudaGraphExecDestroy hipGraphExecDestroy
#define cudaGraphLaunch hipGraphLaunch
#define cudaErrorGraphExecUpdateFailure hipErrorGraphExecUpdateFailure
#define cudaGraphExecUpdateResult hipGraphExecUpdateResult
#define cudaGraphNodeType hipGraphNodeType
#define cudaGraphNodeTypeKernel hipGraphNodeTypeKernel
#define cudaGraphInstantiate hipGraphInstantiate
#define cudaStreamEndCapture hipStreamEndCapture
#define cudaGraphDestroy hipGraphDestroy
#define cudaGraphKernelNodeSetParams hipGraphKernelNodeSetParams
#define cudaErrorInvalidDeviceFunction hipErrorInvalidDeviceFunction
#define cudaGraphKernelNodeGetParams hipGraphKernelNodeGetParams
#define cudaGraphNodeGetType hipGraphNodeGetType
#define cudaGraphGetNodes hipGraphGetNodes
#define cudaGraphExecUpdate hipGraphExecUpdate
#define cudaStreamCaptureModeRelaxed hipStreamCaptureModeRelaxed
#define cudaStreamBeginCapture hipStreamBeginCapture
#define cudaGraph_t hipGraph_t
#define cudaStream_t hipStream_t
#define cudaSuccess hipSuccess
#define cudaOccupancyMaxActiveBlocksPerMultiprocessor hipOccupancyMaxActiveBlocksPerMultiprocessor
#define cudaFuncSetAttribute hipFuncSetAttribute
#define cudaFuncAttributeMaxDynamicSharedMemorySize hipFuncAttributeMaxDynamicSharedMemorySize
#define __trap() do { abort(); __builtin_unreachable(); } while(0)
#define CUBLAS_STATUS_SUCCESS HIPBLAS_STATUS_SUCCESS
#define CUBLAS_STATUS_NOT_INITIALIZED HIPBLAS_STATUS_NOT_INITIALIZED
#define CUBLAS_STATUS_ALLOC_FAILED HIPBLAS_STATUS_ALLOC_FAILED
#define CUBLAS_STATUS_INVALID_VALUE HIPBLAS_STATUS_INVALID_VALUE
#define CUBLAS_STATUS_ARCH_MISMATCH HIPBLAS_STATUS_ARCH_MISMATCH
#define CUBLAS_STATUS_MAPPING_ERROR HIPBLAS_STATUS_MAPPING_ERROR
#define CUBLAS_STATUS_EXECUTION_FAILED HIPBLAS_STATUS_EXECUTION_FAILED
#define CUBLAS_STATUS_INTERNAL_ERROR HIPBLAS_STATUS_INTERNAL_ERROR
#define CUBLAS_STATUS_NOT_SUPPORTED HIPBLAS_STATUS_NOT_SUPPORTED
#if HIP_VERSION >= 60500000
#define CUBLAS_COMPUTE_16F HIPBLAS_COMPUTE_16F
#define CUBLAS_COMPUTE_32F HIPBLAS_COMPUTE_32F
#define CUBLAS_COMPUTE_32F_FAST_16F HIPBLAS_COMPUTE_32F_FAST_16F
#define cublasComputeType_t hipblasComputeType_t
#define cudaDataType_t hipDataType
#else
#define CUBLAS_COMPUTE_16F HIPBLAS_R_16F
#define CUBLAS_COMPUTE_32F HIPBLAS_R_32F
#define CUBLAS_COMPUTE_32F_FAST_16F HIPBLAS_R_32F
#define cublasComputeType_t hipblasDatatype_t
#define cudaDataType_t hipblasDatatype_t
#endif // HIP_VERSION >= 6050000
#if !defined(__HIP_PLATFORM_AMD__)
#error "The HIP backend supports only AMD targets"
#endif // !defined(__HIP_PLATFORM_AMD__)
#define __CUDA_ARCH__ 1300
#if defined(__gfx900__) || defined(__gfx906__)
#define GCN5
#endif // defined(__gfx900__) || defined(__gfx906__)
#if defined(__gfx803__)
#define GCN4
#endif // defined(__gfx803__)
#if defined(GCN5) || defined(GCN4)
#define GCN
#endif // defined(GCN5) || defined(GCN4)
#if defined(__gfx950__)
#define CDNA4
#endif // defined(__gfx950__)
#if defined(__gfx942__)
#define CDNA3
#endif // defined(__gfx942__)
#if defined(__gfx90a__)
#define CDNA2
#endif // defined(__gfx90a__)
#if defined(__gfx908__)
#define CDNA1
#endif // defined(__gfx908__)
#if defined(CDNA4) || defined(CDNA3) || defined(CDNA2) || defined(CDNA1)
#define CDNA // For the entire family
#endif // defined(CDNA4) || defined(CDNA3) || defined(CDNA2) || defined(CDNA1)
#if defined(__GFX12__)
#define RDNA4
#endif // defined(__GFX12__)
#if defined(__GFX11__)
#define RDNA3
#endif // defined(__GFX11__)
#if defined(__gfx1150__) || defined(__gfx1151__) || defined(__gfx1152__) || defined(__gfx1153__)
#define RDNA3_5
#endif // defined(__gfx1150__) || defined(__gfx1151__) || defined(__gfx1152__) || defined(__gfx1153__)
#if defined(RDNA3) && !defined(RDNA3_5)
#define RDNA3_0
#endif // defined(RDNA3) && !defined(RDNA3_5)
#if defined(__gfx1030__) || defined(__gfx1031__) || defined(__gfx1032__) || defined(__gfx1033__) || \
defined(__gfx1034__) || defined(__gfx1035__) || defined(__gfx1036__) || defined(__gfx1037__)
#define RDNA2
#endif
#if defined(__gfx1010__) || defined(__gfx1012__)
#define RDNA1
#endif // defined(__gfx1010__) || defined(__gfx1012__)
#if defined(RDNA4) || defined(RDNA3) || defined(RDNA2) || defined(RDNA1)
#define RDNA // For the entire family
#endif // defined(RDNA4) || defined(RDNA3) || defined(RDNA2) || defined(RDNA1)
#ifndef __has_builtin
#define __has_builtin(x) 0
#endif
typedef __hip_bfloat16 nv_bfloat16;
typedef __hip_bfloat162 nv_bfloat162;
#if HIP_VERSION >= 60200000
#include <hip/hip_fp8.h>
typedef __hip_fp8_e4m3 __nv_fp8_e4m3;
#define FP8_AVAILABLE
#endif // HIP_VERSION >= 60200000
typedef int8_t int8x4_t __attribute__((ext_vector_type(4)));
typedef uint8_t uint8x4_t __attribute__((ext_vector_type(4)));
static __device__ __forceinline__ int __vsubss4(const int a, const int b) {
const int8x4_t va = reinterpret_cast<const int8x4_t&>(a);
const int8x4_t vb = reinterpret_cast<const int8x4_t&>(b);
#if __has_builtin(__builtin_elementwise_sub_sat)
const int8x4_t c = __builtin_elementwise_sub_sat(va, vb);
return reinterpret_cast<const int &>(c);
#else
int8x4_t c;
int16_t tmp;
#pragma unroll
for (int i = 0; i < 4; i++) {
tmp = va[i] - vb[i];
if(tmp > std::numeric_limits<int8_t>::max()) tmp = std::numeric_limits<int8_t>::max();
if(tmp < std::numeric_limits<int8_t>::min()) tmp = std::numeric_limits<int8_t>::min();
c[i] = tmp;
}
return reinterpret_cast<int &>(c);
#endif // __has_builtin(__builtin_elementwise_sub_sat)
}
static __device__ __forceinline__ int __vsub4(const int a, const int b) {
return __vsubss4(a, b);
}
static __device__ __forceinline__ unsigned int __vcmpeq4(unsigned int a, unsigned int b) {
const uint8x4_t& va = reinterpret_cast<const uint8x4_t&>(a);
const uint8x4_t& vb = reinterpret_cast<const uint8x4_t&>(b);
unsigned int c;
uint8x4_t& vc = reinterpret_cast<uint8x4_t&>(c);
#pragma unroll
for (int i = 0; i < 4; ++i) {
vc[i] = va[i] == vb[i] ? 0xff : 0x00;
}
return c;
}
static __device__ __forceinline__ unsigned int __vcmpne4(unsigned int a, unsigned int b) {
const uint8x4_t& va = reinterpret_cast<const uint8x4_t&>(a);
const uint8x4_t& vb = reinterpret_cast<const uint8x4_t&>(b);
unsigned int c;
uint8x4_t& vc = reinterpret_cast<uint8x4_t&>(c);
#pragma unroll
for (int i = 0; i < 4; ++i) {
vc[i] = va[i] == vb[i] ? 0x00 : 0xff;
}
return c;
}
dir-steering/README.md
+28
View File
@@ -16,6 +16,9 @@ The file shape depends on the model:
- DeepSeek V4 Flash: `43 x 4096`.
- GLM 5.3 Flash: `45 x 4096`. The separate MTP predictor layer is omitted.
- Qwen3.8 Flash Next: `48 x 2560`. FFN steering is applied to each
hyper-connection branch of the residual; dumps average those branches
at the last prompt token.
GLM 5.2 steering is not implemented.
@@ -179,3 +182,28 @@ Style control:
The method is not a fine-tune. It is a low-rank runtime edit, so it works best
for coarse behavior, topic, or style directions that are consistently present in
the activation captures.
## Qwen3.8 Flash Next
Capture uses `--think` / `--nothink`
(not `--think-high`). Dumps track the prompt phase explicitly, including
one-token tails, and retain the last prompt token during ordinary and MTP
decode. `attn_out` captures the output projection of both GDN and full-attention
layers, giving one row for each of the 48 trunk layers:
```sh
python3 dir-steering/tools/build_direction.py \
--profile qwen3.8-flash-next \
--ds4 ./ds4 \
--model gguf/Qwen3.8-Flash-Next-Q4.gguf \
--good-file /path/to/target-prompts.txt \
--bad-file /path/to/control-prompts.txt \
--out dir-steering/out/qwen38-direction.json \
--component ffn_out \
--ctx 512
```
Qwen steering is Metal-only. `--mtp-model`, SSD streaming, and `--power`
remain unsupported for this graph. The bank contains only the 48 trunk layers;
the embedded MTP predictor remains unsteered. Its drafts are verified by the
steered target trunk, so `--mtp` remains supported.
dir-steering/tools/build_direction.py
+7 -2
View File
@@ -10,6 +10,8 @@ At runtime ds4 applies:
Positive scale suppresses the target direction. Negative scale amplifies it.
"""
from __future__ import annotations
import argparse
import array
import json
@@ -23,6 +25,7 @@ from pathlib import Path
MODEL_PROFILES = {
"deepseek-v4-flash": (43, 4096),
"glm-5.3-flash": (45, 4096),
"qwen3.8-flash-next": (48, 2560),
}
@@ -72,6 +75,7 @@ def run_capture(
env["DS4_METAL_GRAPH_DUMP_PREFIX"] = str(dump_prefix)
env["DS4_METAL_GRAPH_DUMP_NAME"] = component
env["DS4_METAL_GRAPH_DUMP_POS"] = "0"
env["DS4_QWEN4_PREFILL_CHUNK"] = str(max(ctx, 1024))
cmd = [
str(ds4),
@@ -79,10 +83,11 @@ def run_capture(
"--ctx", str(ctx),
"--prompt-file", str(prompt_path),
"-n", "1",
"--prefill-chunk", str(max(ctx, 1024)),
]
if system:
cmd += ["--system", system]
cmd.append("--think-high" if think else "--nothink")
cmd.append("--think" if think else "--nothink")
result = subprocess.run(cmd, cwd=ds4.parent, env=env, check=False,
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
if result.returncode != 0:
@@ -128,7 +133,7 @@ def main() -> None:
ap.add_argument("--system", default="You are a helpful assistant.")
ap.add_argument("--component", default="ffn_out",
choices=("ffn_out", "attn_out"),
help="runtime-editable 4096-wide activation stream")
help="runtime-editable activation stream at the profile's embedding width")
ap.add_argument("--think", action="store_true",
help="capture after <think>; default captures direct answers")
ap.add_argument("--pair-normalize", action="store_true",
docs/CLIENTS.md
+125
View File
@@ -0,0 +1,125 @@
# Coding Agent Clients
[README](../README.md) | [Server](SERVER.md)
Start the server first:
```sh
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
```
Set the client's context limit no higher than the server's. Output tokens also
consume that context; a client limit does not enlarge the server allocation.
The examples use Flash and localhost. Change the address for a trusted remote
server. The `dsv4-local` strings are placeholders, not server authentication.
## Pi
Add this provider to `~/.pi/agent/models.json`:
```json
{
"providers": {
"ds4": {
"baseUrl": "http://127.0.0.1:8000/v1",
"api": "openai-completions",
"apiKey": "dsv4-local",
"compat": {
"supportsStore": false,
"supportsDeveloperRole": false,
"supportsReasoningEffort": true,
"supportsUsageInStreaming": true,
"maxTokensField": "max_tokens",
"supportsStrictMode": false,
"thinkingFormat": "deepseek",
"requiresReasoningContentOnAssistantMessages": true
},
"models": [{
"id": "deepseek-v4-flash",
"name": "DwarfStar Flash",
"reasoning": true,
"thinkingLevelMap": {
"off": null, "minimal": "low", "low": "low",
"medium": "medium", "high": "high", "xhigh": "xhigh"
},
"input": ["text"],
"contextWindow": 100000,
"maxTokens": 16384,
"cost": {"input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0}
}]
}
}
}
```
Select `ds4/deepseek-v4-flash` in Pi. This is a text-only client configuration;
image use also requires the appropriate client input declaration and server
vision encoder.
## OpenCode
Merge the provider into `~/.config/opencode/opencode.json`:
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"ds4": {
"name": "DwarfStar",
"npm": "@ai-sdk/openai-compatible",
"options": {
"baseURL": "http://127.0.0.1:8000/v1",
"apiKey": "dsv4-local"
},
"models": {
"deepseek-v4-flash": {
"name": "DwarfStar Flash",
"limit": {"context": 100000, "output": 16384}
}
}
}
}
}
```
Select `ds4/deepseek-v4-flash` as the model.
## Codex CLI
Use the Responses API. Add a provider to the Codex configuration:
```toml
[model_providers.ds4]
name = "DwarfStar"
base_url = "http://127.0.0.1:8000/v1"
wire_api = "responses"
stream_idle_timeout_ms = 1000000
```
```sh
codex --model deepseek-v4-flash -c model_provider=ds4
```
## Claude Code
Use the Anthropic-compatible endpoint. A shell wrapper can select the local
model for the main agent and its secondary model roles:
```sh
#!/bin/sh
unset ANTHROPIC_API_KEY
export ANTHROPIC_BASE_URL="http://127.0.0.1:8000"
export ANTHROPIC_AUTH_TOKEN="dsv4-local"
export ANTHROPIC_MODEL="deepseek-v4-flash"
export ANTHROPIC_DEFAULT_SONNET_MODEL="deepseek-v4-flash"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="deepseek-v4-flash"
export ANTHROPIC_DEFAULT_OPUS_MODEL="deepseek-v4-flash"
export CLAUDE_CODE_SUBAGENT_MODEL="deepseek-v4-flash"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export CLAUDE_STREAM_IDLE_TIMEOUT_MS=600000
exec claude "$@"
```
Give the wrapper a different name from `claude`. Agent clients can send large
initial prompts, so the first prefill may take time. Disk caching helps reuse
compatible prefixes on later sessions.
docs/CUDA_MULTI_GPU.md
+90
View File
@@ -0,0 +1,90 @@
# NVIDIA CUDA GPUs
[README](../README.md) | [Getting started](../README.md#start-here)
**Ada Lovelace is supported, including 48 GB L40S cards.** This is an important
use case: a model whose other inference implementations require newer GPUs
can still run here when DwarfStar supports its GGUF layout. The Flash kernels
do not require Blackwell's native FP4 instructions; Ada uses an appropriate
CUDA path. Support remains model-specific, not a promise to run arbitrary GGUFs.
## Build
Install the NVIDIA driver and CUDA toolkit, including `nvcc` and cuBLAS.
Use the local GPU architecture, or select Ada explicitly:
```sh
make cuda-generic
# For an Ada build:
make cuda CUDA_ARCH=sm_89
```
For a GB10 machine use the [DGX Spark](DGX_SPARK.md) target instead.
## Normal layer placement
Without tensor parallelism, DwarfStar places complete layers on the selected
GPUs. `--gpu-vram auto` uses reported free VRAM, reserving space for the graph
and context. Explicit budgets are comma-separated GiB values, one per device.
```sh
./download_model.sh ds4f-q2
./ds4 --cuda --gpu-devices 0,1,2,3 --gpu-vram auto --ctx 32768
```
This is also the multi-GPU placement mode for GLM. Startup refuses a layout
that would require unsupported CPU execution; reduce context or choose a
smaller model if necessary.
## Flash tensor parallelism
`--cuda-tensor-parallel` pairs GPUs and divides routed-expert work inside
each pair. Pairs also own successive layer ranges. This is an in-process
configuration, not the network `--role coordinator` / `--role worker` mode.
The device order matters. List all layer-home GPUs first, then their partners.
For physical pairs `(0,1)`, `(2,3)`, `(4,5)`, `(6,7)`, use:
```text
0,2,4,6,1,3,5,7
```
Check your machine's topology with `nvidia-smi topo -m`; do not assume this
ordering gives the best pairs on another server. Each pair stores one half
of the routed experts per GPU. Dense attention, routers, and shared experts
are replicated within the pair; the output head is vocabulary-sharded.
For the tested eight-L40S setup:
```sh
./download_model.sh ds4f-q4
./ds4-agent --cuda --cuda-tensor-parallel \
--gpu-devices 0,2,4,6,1,3,5,7 --gpu-vram auto --ctx 100000
```
Q4 has native grouped routed kernels for multi-user throughput. Q2 needs less
memory, but unsupported grouped shapes use a slower, correct fallback.
Four 48 GB cards with Q2 and eight with Q4 are tested configurations. An even
device count alone does not guarantee memory fit.
## Serve multiple users
```sh
./ds4-server --cuda --cuda-tensor-parallel \
--gpu-devices 0,2,4,6,1,3,5,7 --gpu-vram auto \
--ctx 100000 --batched-session 16 --host 0.0.0.0
```
The supported eight-L40S Flash configuration has reached roughly 126 aggregate
generation tokens/s with 16 decode rows. This is total throughput, not the
speed of one user. The recorded conditions and regression requirements are in
[the QA guide](../QA_BEFORE_RELEASES.md).
CUDA TP defaults to 2048-token prefill chunks. Keep that default for the
16-session, 100k-context setup; larger chunks need more scratch memory.
Reduce session count or context if all KV states do not fit.
The repository's [server launcher](../run-nvidia-tp-server.sh) is an example
from the L40S deployment, not a portable default: it uses host-specific model
and cache paths and defaults to MXFP4. The commands above need no launcher or
environment tuning. See [serving](SERVER.md) for disk caches and API access.
docs/DGX_SPARK.md
+191
View File
@@ -0,0 +1,191 @@
# DGX Spark
[README](../README.md) | [Getting started](../README.md#start-here)
Use the NVIDIA driver and CUDA development
toolkit for the machine; the build needs `nvcc` and cuBLAS.
Check that `nvidia-smi` sees the GPU before building.
## Build and run
```sh
make cuda-spark
./download_model.sh ds4f-q2
./ds4 --cuda
```
The build targets GB10 (`sm_121a`) and enables the Blackwell-specific kernels.
Do not use `--cuda-tensor-parallel` on a single Spark.
Stop other inference services before loading a model so they do not compete
for memory. Restore any services you stopped when finished.
## DeepSeek V4.1 Flash
Q2 needs SSD streaming on one 128 GB Spark:
```sh
./download_model.sh ds41f-q2
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
--ssd-streaming --ctx 32768
```
The file is 341 GiB, including 189 GiB of disk-only Engram tables. Keep it on
a fast local SSD. The expert cache is sized automatically; leave memory for
other programs and the context. Use the same options with `ds4-agent` or
`ds4-server`.
Two Sparks can instead keep half the experts each, using
[network tensor parallelism](DISTRIBUTED.md#tensor-parallelism-between-two-sparks).
Both need the complete GGUF on their local SSD. Single-host V4.1
[DSpark](SPECULATIVE_DECODING.md#deepseek-v41-flash-dspark) also works with SSD
streaming; TP speculation and CUDA vision are not supported for V4.1.
The [model guide](MODELS.md#deepseek-v41-flash) covers
thinking levels and the separate Metal configurations.
September 13-14, 2026, Q2 SSD with automatic cache sizing: a 3,241-token
`/read README.md` prefill reached 93-96 t/s, up from 49-54 t/s, with 32K
allocated context. No extra flag is needed. A separate 64 GiB cache-budget
run with 64K allocated context reached 384 t/s for a 32K initial prefill and
88 t/s for a 3.2K append. These are SSD-streamed V4.1 results, not resident
V4 Flash numbers; speed depends on prompt length and the expert cache.
The automatic-cache 256-token reply test decoded about 5% slower, but
prefill plus the reply fell from about 87 to 59 seconds.
See the [QA record](../QA_BEFORE_RELEASES.md#cuda-ssd-streaming) for conditions
and longer-context measurements.
On two Sparks over the direct RoCE link, Q2 measured **21.8 t/s** for one
session on October 2, 2026: a 1K prompt, 64K allocated context and 2,048
teacher-forced decode tokens, without speculation.
| Work | Speed |
| --- | ---: |
| Fresh 32K prefill | 411 t/s |
| Add 8K after 32K | 314 t/s |
Both runs used 64K allocated context and 512 decode tokens per frontier;
decode at these longer contexts was about 18 t/s. Engram tables stayed on disk.
Decode batching starts at five ready sessions; smaller groups run in order.
The earlier eight-session reference is about **28 aggregate t/s**, not per
client. See the [network TP QA record](../QA_BEFORE_RELEASES.md#cuda-network-tensor-parallelism)
for the workloads and memory limits.
## Flash 0731 on two Sparks
[Network tensor parallelism](DISTRIBUTED.md#tensor-parallelism-between-two-sparks)
also supports Flash 0731 Q2 and MXFP4, including DSpark with the matching
support file. Each host holds about 45 GiB of Q2 weights or 77 GiB of MXFP4,
plus context and runtime buffers. Download `ds4f-mxfp4` on both for MXFP4.
October 2, 2026, Q2, direct 200 Gb/s RoCE, 64K allocated context: a 105-token
English prose prompt decoded 512 tokens at **26.0 t/s** without speculation.
A fresh 32K prefill reached **618 t/s**; appending 8K reached **610 t/s**.
Ordinary decode at those longer frontiers was about 19 t/s. These are single
measurements, not release medians.
October 4, 2026, matching 0731 DSpark support, default speculation settings,
temperature zero, 64K allocated context and 512 generated tokens. Both formats
use the drafter on both ranks over RDMA. These are medians of two runs per case
(four for MXFP4 prose), including the first token or block. Benchmark processes
ran on the performance cores (`taskset -c 5-9,15-19`):
| Weights | Prompt | DSpark |
| --- | --- | ---: |
| Q2 | English prose, 105 tokens | 44.0 t/s |
| Q2 | C hash table, 31 tokens | 83.3 t/s |
| MXFP4 | English prose, 105 tokens | 41.7 t/s |
| MXFP4 | C hash table, 31 tokens | 75.6 t/s |
On one Spark, the same Q2 prose and coding prompts reached 23.4 and 44.0 t/s
with DSpark. These are workload-specific measurements, not a promised speedup;
poor draft acceptance can still make it slower. The measured settings are
automatic once DSpark is enabled; speculation itself remains opt-in.
## Qwen3.8 Flash Next
Both Q2 and Q4 fit on one Spark. After building:
```sh
./download_model.sh qwen38-q2
./ds4 --mtp --ctx 32768
```
Q2 keeps 41.73 GiB of weights resident; Q4 keeps 69.74 GiB. Both GGUFs also
contain 95.37 GiB of BF16 n-grams, read directly from the SSD. Leave room
for context and runtime buffers. Vision, the native agent, server APIs and
text-session checkpoints work on CUDA; see [Qwen setup](QWEN38_FLASH_NEXT.md).
September 15, 2026, Q2 on one Spark: a 16K prefill reached 250 t/s, extending
it to 30K reached 243 t/s, and ordinary decode stayed around 17.3 t/s.
These are single runs with 8,192-token prefill chunks and 32 decode tokens.
A separate 256-token prose run measured 17.1 t/s ordinarily and 19.9 t/s with
MTP, at temperature zero and 8K allocated context. Speculation is opt-in;
its benefit depends on the prompt.
## GLM 5.3 Flash
Q2 is the resident target for one Spark:
```sh
./download_model.sh glm53-q2
./ds4 --cuda -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 16384
```
Q4 does not fit resident. GLM Spark-to-Spark tensor parallelism is not
implemented; the two-Mac RDMA instructions do not apply to GLM on CUDA.
## Vision and speculative decoding
DeepSeek Vision Experimental needs its matching text model and encoder:
```sh
./download_model.sh ds4f-vision-q2
./ds4 --cuda --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf
```
Use `/read image.png` in the CLI. GLM vision also works on this backend; see
[models and vision](MODELS.md#vision).
For Flash 0731, [DSpark](SPECULATIVE_DECODING.md) uses the separate 0731 support
file. Vision Experimental has a different drafter. GLM uses its built-in MTP
block with `--mtp`. None is enabled by default.
## Flash Q2 with DSpark
September 6, 2026, fully resident Flash 0731: median generation speed from
three runs of 256 tokens, 4K allocated context and a 512-token prefill chunk.
| Prompt | Temperature | Ordinary decode | DSpark |
| --- | ---: | ---: | ---: |
| C hash table | 0 | 19.72 t/s | 31.41 t/s |
| C hash table | 1 | 19.53 t/s | 29.98 t/s |
| Unpredictable prose | 1 | 19.53 t/s | 18.81 t/s |
Longer coding runs reached about 33 t/s. Poor draft acceptance can still make
DSpark slower. Temperature 1 uses the default opportunistic policy; these are
not exact-sampling results. DSpark does not accelerate prefill.
To reproduce the greedy coding case after downloading `ds4f-q2` and
`ds4f-dspark`:
```sh
./ds4 --cuda -m ds4flash.gguf \
--dspark --mtp-model gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
--ctx 4096 --prefill-chunk 512 --nothink --temp 0 --seed 12345 -n 256 \
-p 'Write a complete C hash table implementation with string keys, insert, find, delete, and a test main. Output only C code.'
```
For the temperature-1 rows, use `--temp 1 --top-p 0.95 --min-p 0.05`.
The [QA record](../QA_BEFORE_RELEASES.md#16-speed-regression) includes the previous
implementation, continued-context checks and quality comparisons.
## Larger models and serving
CUDA also has [SSD streaming](SSD_STREAMING.md) paths for larger weights.
Memory fit and speed depend on the model layout; start with Q2 for normal use.
V4.1 Q2 SSD serving batches up to eight decode rows together. An eight-session
test reached 11.0 aggregate t/s versus 8.4 with ordered execution; this is
total throughput, not speed per client. Other single-Spark model paths retain
their existing scheduling.
See [serving](SERVER.md) and the recorded [benchmarks](PERFORMANCE.md).
docs/DISTRIBUTED.md
+208
View File
@@ -0,0 +1,208 @@
# Inference Across Machines
[README](../README.md)
There are two modes:
| Mode | Split | Main use |
| --- | --- | --- |
| Tensor parallelism | Routed experts and per-layer work across two Macs or two Sparks | Resident inference with lower per-token work on each GPU |
| Pipeline parallelism | Complete layer ranges across several machines | Fit larger models and overlap long prefills |
These are separate from [tensor parallelism across CUDA cards](CUDA_MULTI_GPU.md).
Network protocols have no authentication or encryption. Use trusted machines
and a trusted network; run the same commit on every peer. Model paths and
artifacts must agree. Update all TP peers together when changing versions.
## Tensor parallelism between two Macs
This is a 50/50 split with exactly one worker. Do not pass `--layers`.
Routed experts are sharded; attention partitioning depends on the model. Both
GPUs work on the same token and exchange partial results. This can reduce generation
latency, but the gain depends on the model, link, and comparison setup.
Two 128 GB Macs are useful for V4 Flash Q4/MXFP4 or GLM 5.3 Flash Q4.
GLM 5.2 IQ2_XXS is another tested capacity setup. V4.1 Flash Q2 also runs
on two 128 GB Macs, with disk-only Engram tables.
A larger quant may need larger machines even though its tensor layout is supported.
### Link setup
Use a Thunderbolt cable. RDMA requires an active verbs device with an
IPv4-mapped GID; a working ping alone does not establish that.
```sh
rdma_ctl status
ibv_devinfo -v
```
Addresses must be on the cabled member interfaces, not only the Thunderbolt
bridge. For example, after checking which interfaces are active:
```sh
# Machine A, example member interface en1.
sudo ifconfig en1 inet 10.99.0.2/30 alias
# Machine B, example member interface en6.
sudo ifconfig en6 inet 10.99.0.1/30 alias
```
For the large tested shards on otherwise idle 128 GB Macs, the setup raised
the per-boot GPU wired-memory limit on both machines:
```sh
sudo sysctl iogpu.wired_limit_mb=120000
```
This is specific to that memory configuration. It grants a larger GPU budget;
it does not create more RAM. Check model, context, and system headroom before
raising a limit on your machine.
### Start the pair
Download the same model on both machines. For GLM 5.3 Flash Q4:
```sh
./download_model.sh glm53-q4
```
Start the worker first; it retries while the coordinator loads:
```sh
# Machine B.
./ds4 --tensor-parallel --role worker \
--coordinator 10.99.0.2 9911 --transport rdma --ctx 8192
# Machine A.
./ds4 --tensor-parallel --role coordinator \
--listen 10.99.0.2 9911 --transport rdma --ctx 8192
```
The verbs device and GID are selected automatically. If ambiguous, specify
`--rdma-device` and `--rdma-gid-index` from `ibv_devinfo`. Use `--transport tcp`
on both peers when RDMA is unavailable. Do not probe the waiting coordinator
with `curl` or `nc`: it may treat the connection as a worker handshake.
Keep workers running in a terminal or managed session and retain both logs.
Do not treat repeated handshake or RDMA timeouts as successful QA merely
because a retry works.
The coordinator can be `ds4`, `ds4-agent`, `ds4-server`, or `ds4-bench`;
workers run `ds4`. For models with vision support, pass the same `--vision FILE`
to both for image input.
For GLM MTP, enable `--mtp` on both. For DeepSeek DSpark, both need the
matching support model and DSpark options. V4.1 supports vision but not
speculative decoding.
TP disk-cache restore currently rebuilds the exact saved token prefix on both
ranks rather than restoring the coordinator alone. Expect prefill on restore.
See [speculation](SPECULATIVE_DECODING.md) and [serving](SERVER.md).
## Tensor parallelism between two Sparks
V4 Flash 0731 Q2/MXFP4 and V4.1 Flash Q2 text inference support one GPU per rank,
with a 50/50 expert split. Each Spark holds about 45 GiB of Flash 0731 Q2
weights, 77 GiB of Flash MXFP4, or 81 GiB for V4.1, plus context and runtime
buffers. V4.1 Engram tables stay on disk. Do not add `--ssd-streaming` or
`--cuda-tensor-parallel`: those select different memory/execution modes.
Build the same commit with `make cuda-spark` on both machines and download
`ds41f-q2` on both. RDMA needs the libibverbs development headers at build time,
its runtime library, and an active RoCEv2 link. Check `ibv_devinfo -v` and use
the addresses of the directly connected ports, not the management network.
For example, with the coordinator at `172.31.250.1` on the direct link:
```sh
# Worker.
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf --ctx 32768 \
--tensor-parallel --role worker --coordinator 172.31.250.1 9911 \
--transport rdma
# Coordinator.
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf --ctx 32768 \
--tensor-parallel --role coordinator --listen 172.31.250.1 9911 \
--transport rdma
```
The link address selects the matching verbs device and GID. If selection is
ambiguous, use `--rdma-device` and `--rdma-gid-index`. TCP is also available
with `--transport tcp` on both peers.
For Flash 0731, download `ds4f-q2` or `ds4f-mxfp4` on both hosts and use its
GGUF path in the same commands. For DSpark, download `ds4f-dspark` on both and
add `--dspark --mtp-model PATH` to both commands. With MXFP4 over RDMA, both
GPUs also share the drafter work. This needs about 2.5 GiB extra per rank,
beside the support model and context. If the worker has no drafter, or the
configuration cannot use the split, drafting stays on the coordinator.
Speed depends on the prompt.
The coordinator can be `ds4-agent`, `ds4-server` or `ds4-bench`. Match context
sizes and any `--prefill-chunk` overrides on both sides. For V4.1, five or more
ready sessions use native batched decode; smaller groups run in order.
Use `--batched-session 8` on the server and allow memory for all eight contexts.
Flash 0731 network sessions currently run in order, without native batching.
CUDA network TP does not support vision, V4.1 DSpark or other model/quant layouts.
## Pipeline parallelism
Each process maps only its assigned layers, retaining that slice of the KV
state. Layer ranges are inclusive. `N:output` includes the final layer and
output head. Activations travel from one stage to the next over TCP.
For Flash Q4 on two machines, download `ds4f-q4` on both, then start each side.
Replace the example address with your coordinator's reachable address:
```sh
# Machine A.
./ds4 --role coordinator --layers 0:19 --listen 10.99.0.2 9911
# Machine B.
./ds4 --role worker --layers 20:output --coordinator 10.99.0.2 9911
```
Normally give the output head to the final worker. With several workers,
choose non-overlapping ranges covering the entire model. Workers register
their ranges with the coordinator; intermediate workers forward activations
directly to the next stage.
Long prefill chunks can occupy different stages simultaneously. A single
generation stream cannot use that overlap: each token must finish the route
before the next one is sampled. Use pipeline mode primarily for capacity and
long-prefill throughput, not as a guaranteed decode speedup.
### Full PRO Q4
For two 512 GB Mac Studios, use the split artifacts:
```sh
# Machine A.
./download_model.sh pro-q4-layers00-30
./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf \
--role coordinator --layers 0:30 --listen 10.99.0.2 9911
# Machine B.
./download_model.sh pro-q4-layers31-output
./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf \
--role worker --layers 31:output --coordinator 10.99.0.2 9911
```
These downloads do not change `ds4flash.gguf`. Startup is expensive because
each side must make its model slice resident.
### Tuning and recovery
Keep default chunk sizes first. `--dist-prefill-window N` controls the number
of chunks in flight; `--dist-prefill-chunk N` overrides the session-derived
chunk size. `--debug` shows route and per-hop timings.
Activations use 32-bit transport by default. `--dist-activation-bits 16` halves
the payload; `8` is more aggressive. These change numerical precision on the
wire, not weights or KV storage. Validate output when changing them.
A disconnected worker invalidates the route. In-flight work can fail; later
requests need a complete route before proceeding, and the coordinator can
replay the saved token prefix to rebuild worker state. Pipeline snapshots
serialize all layer slices into one payload and redistribute them when loaded.
For protocol details, see [ds4_distributed.c](../ds4_distributed.c)
and [ds4_tp.c](../ds4_tp.c).
docs/METAL.md
+71
View File
@@ -0,0 +1,71 @@
# Metal on Apple Silicon
[README](../README.md) | [Getting started](../README.md#start-here)
## Build and run
Install Apple's command-line developer tools if needed:
```sh
xcode-select --install
```
From the repository root:
```sh
make
./download_model.sh ds4f-q2
./ds4
```
The same build supports M3 and M5 Macs. Hardware-specific fast paths are
selected automatically; no environment variable is needed to enable them.
Leave other GPU and memory-heavy applications idle when comparing performance.
## Choose a model
| Memory | Starting point |
| --- | --- |
| 64 GB | Flash Q2 with `--ssd-streaming` |
| 96 GB | Flash Q2; leave room for the context and other applications |
| 128 GB | Flash Q2, or GLM 5.3 Flash Q2 with modest initial context |
| 256 GB | Flash Q4/MXFP4 or GLM 5.3 Flash Q4 |
| 512 GB | Larger models, including PRO Q2 |
Here, Flash means DeepSeek V4 Flash. V4.1 Flash has different memory
requirements; see its [model guide](MODELS.md#deepseek-v41-flash).
These are starting points, not guarantees that every context or session count
will fit. GLM 5.3 Flash Q2 is about 90 GiB before runtime allocations.
Stop other memory-heavy workloads before loading it resident.
```sh
./download_model.sh glm53-q2
./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 32768
```
For a model larger than RAM, start with automatic cache sizing:
```sh
./ds4 --ssd-streaming
```
See [SSD streaming](SSD_STREAMING.md) before increasing the expert cache.
Do not bypass the memory guard just to make an oversized resident model start.
## Two Macs
Two 128 GB Macs can run a larger model fully resident with a 50/50 routed-expert
split. Thunderbolt RDMA is the low-latency option; TCP is also supported.
Follow [tensor parallel setup](DISTRIBUTED.md#tensor-parallelism-between-two-macs).
For more than two machines, use [pipeline parallelism](DISTRIBUTED.md#pipeline-parallelism).
## Next steps
- [Vision and GLM models](MODELS.md)
- [DSpark and GLM MTP](SPECULATIVE_DECODING.md)
- [Batched serving](SERVER.md#multiple-sessions)
- [Benchmarking](PERFORMANCE.md)
For DeepSeek V4 Flash and PRO, `--power 70` trades throughput for lower
sustained GPU load. V4.1 and GLM currently require `--power 100`.
docs/MODELS.md
+273
View File
@@ -0,0 +1,273 @@
# Models and Vision
[README](../README.md) | [Getting started](../README.md#start-here)
DwarfStar is not a general GGUF runner. Use the download targets below: other
GGUFs may have unsupported tensor layouts, metadata, or quantization mixes.
Run `./download_model.sh --help` for filenames and all available targets.
Main-model downloads update `ds4flash.gguf`. Encoders, draft models, packaged
FP8 weights, and split PRO pieces do not. Pass `-m FILE` to avoid depending on
which model was downloaded last.
Some downloads require the Hugging Face CLI; the script prints installation
instructions when needed. Authentication is optional for public weights;
your cached Hugging Face token or `HF_TOKEN` is used when present.
## DeepSeek V4
| Target | Use |
| --- | --- |
| `ds4f-q2` | Flash 0731, about 81 GiB; starting point for 96/128 GB systems |
| `ds4f-q2-q4` | Mostly Q2, with the last six routed-expert layers at Q4; needs more memory |
| `ds4f-q4` | Flash 0731 Q4; larger-memory or distributed systems |
| `ds4f-mxfp4` | Native MXFP4 routed experts; larger-memory or distributed systems |
| `pro-q2-imatrix` | PRO 0813; 512 GB resident target, or SSD streaming |
| `pro-q4-split` | Both PRO Q4 pieces for pipeline execution |
```sh
./download_model.sh ds4f-q2
./ds4
```
The Flash Q2 recipe spends most of its compression on routed experts:
IQ2_XXS gate/up and Q2_K down. Other components use higher precision, including
Q8 projections, shared experts and output, plus F16/F32 tensors. They are not
all untouched source weights. The imatrix guides the routed quantization.
The MXFP4 recipe preserves DeepSeek's released MXFP4 routed experts without
requantizing them. Metal and CUDA support it. Blackwell uses native FP4 matrix
instructions and FP4 activations for batched expert work; CUDA decode and
other CUDA architectures use Q8 activations. ROCm also has a resident MXFP4
path, including pipeline execution for models too large for one host.
To build weights rather than download them, see [GGUF tools](../gguf-tools/README.md).
## DeepSeek V4.1 Flash
V4.1 Flash text and vision inference work on Metal. CUDA supports text with
Q2 SSD streaming on one Spark or resident shards across two Sparks. ROCm
supports single-host text inference with SSD streaming. It needs its own
GGUF, tokenizer and inference graph; V4 Flash weights and DSpark support files
are not interchangeable with it.
| Target | File size | Main weights |
| --- | ---: | ---: |
| `ds41f-q2` | 341 GiB | 152 GiB |
| `ds41f-q4` | 483 GiB | 294 GiB |
| `ds41f-mxfp4` | 467 GiB | 278 GiB |
Q2 and Q4 use imatrix-calibrated routed experts; MXFP4 uses OCP MXFP4 routed
experts with higher-precision attention, shared experts and output weights.
All include 189 GiB of Engram tables. Engram rows are read directly from the
file as needed in every mode, never loaded as a resident table. Keep the GGUF
on a fast local SSD.
On one 128 GB Mac, use SSD streaming. Leave the expert cache budget automatic:
```sh
./download_model.sh ds41f-q2
./ds4 -m gguf/DeepSeek-V4.1-Flash-1017-IQ2_XXS-Q2_K-imatrix.gguf \
--ssd-streaming --ctx 32768
```
Use `ds4-agent` or `ds4-server` with the same model and memory options.
On a DGX Spark, add `--cuda`; see the [Spark guide](DGX_SPARK.md#deepseek-v41-flash).
`--think-level 25` sets reasoning effort explicitly; the range is 1 to 100,
with 0 disabling thinking. `/think 25` changes it in the CLI or native agent.
`--think` selects 75 and `--think-max` selects 100.
For two 128 GB Macs or Sparks, follow the [TP/RDMA setup](DISTRIBUTED.md), passing this
GGUF with `-m` on both ranks and omitting `--ssd-streaming`. Each rank holds
about 81 GiB of main weights, plus context and runtime buffers. Both machines
need the complete GGUF on disk. A 256 GB or larger Mac can instead hold all
main weights; full residency has been tested on an M3 Ultra with 512 GB.
Q4 needs SSD streaming on smaller Macs, or a 512 GB Mac for full residency.
It does not fit resident TP across two 128 GB Macs. Download it with
`./download_model.sh ds41f-q4` and select `gguf/DeepSeek-V4.1-Flash-Q4.gguf`.
The download comes in two parts; the script joins and verifies them automatically.
Allow another 37 GiB of free disk space while joining. Rerun the command to resume an
interrupted download or join.
For the 1017 MXFP4 build, run:
```sh
./download_model.sh ds41f-mxfp4
```
The target downloads all nine standard GGUF shards, verifies each size and
SHA-256, and merges them into `gguf/DeepSeek-V4.1-Flash-1017-MXFP4-experts.gguf`.
DwarfStar's V4.1 loader needs this single file with page-aligned Engram tables;
the script restores the original 16 KiB alignment and verifies the original
whole-file checksum before updating `ds4flash.gguf`. The shard files are kept,
so allow about **935 GiB** for shards plus the merged output. Free space is
checked before downloading. Rerun the same command to resume downloads or an
interrupted merge; existing files and the previous default model are preserved
if verification fails. Q2 and MXFP4 use the 1017 checkpoint; Q4 remains the
previous release.
Large SSD prefills process layers in wide batches. Metal overlaps computation
with the next layer's reads; CUDA stages experts into its bounded device cache.
Short appends keep using the expert cache.
Resident and TP inference also batch continued prefills automatically.
For concurrent serving, see [session batching](SERVER.md#multiple-sessions).
Each slot needs its own context memory; start with `--ctx 4096` before
increasing both context and slot count. CUDA Q2 SSD mode batches up to eight
decode rows; CUDA network TP currently serves sessions in order.
[DSpark](SPECULATIVE_DECODING.md#deepseek-v41-flash-dspark) uses a
separate V4.1 support GGUF and currently requires one host. Pipeline execution
is not implemented for V4.1; vision requires Metal.
Scalar, batched and tensor-parallel execution are not numerically identical.
Q4 batched prefill shows a small probability-score loss on the short official
continuation test, with unchanged overall top-token agreement. See the
[QA results](../QA_BEFORE_RELEASES.md#17-deepseek-v41-flash) for details
and remaining differences.
For images, download the matching encoder and add it to the same command:
```sh
./download_model.sh ds41f-vision
./ds4-agent -m gguf/DeepSeek-V4.1-Flash-1017-IQ2_XXS-Q2_K-imatrix.gguf \
--ssd-streaming --vision gguf/DeepSeek-V4.1-Flash-Vision.gguf
```
Vision works with SSD streaming, full residency and two-Mac TP. Pass the encoder
on both TP ranks. Use `/read image.png` in `ds4`, `view_image` in `ds4-agent`,
or the [server image API](SERVER.md#images). V4 Flash vision encoders do not
work with V4.1. See [conversion](../gguf-tools/README.md#convert-deepseek-v41-flash)
to build the GGUFs from safetensors.
## Qwen3.8 Flash Next
`./download_model.sh qwen38-q2` downloads one **137.10 GiB** GGUF: 41.73 GiB
of main/MTP weights and 95.37 GiB of original BF16 n-grams kept on disk.
Its gate/up experts use IQ2_XXS; down experts use Q2_K with 640 logical inputs
padded to 768 in the weight file. It replaces the larger MXFP4-down IQ2 release.
For 64 GB Macs, start at 8K context with a 1,024-token prefill chunk; runtime
allocations add to the main weights, but the n-gram table is not mapped or
preloaded. Keep the GGUF on a fast local SSD.
The larger `qwen38-q4k` target uses 165.11 GiB on disk and 69.74 GiB for
resident weights, before runtime buffers.
This model runs on Metal and single-GPU CUDA, including DGX Spark.
The script links `ds4flash.gguf` to the combined GGUF:
```sh
./ds4 --ctx 8192 --prefill-chunk 1024
```
Add `--mtp` for speculation; no second file is needed.
See [Qwen setup](QWEN38_FLASH_NEXT.md)
for memory, conversion, vision, and sampling details.
Vision uses a separate encoder. `./download_model.sh qwen38-vision` downloads
llama.cpp's Q8_0 mmproj from
[ggml-org/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF);
pass it at runtime with `--vision`.
## GLM 5.3 Flash
| Target | Approximate file size | Use |
| --- | ---: | --- |
| `glm53-q2` | 90 GiB | One 128 GB Mac or DGX Spark; ROCm also supported |
| `glm53-q4` | 178 GiB | Larger Mac, two 128 GB Macs, or SSD streaming |
| `glm53-fp8` | 305 GiB | Packaged native weights only; inference not implemented |
```sh
./download_model.sh glm53-q2
./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 32768
```
GLM 5.3 Flash has recurrent KDA layers, sparse DSA attention, hyper-connections,
and a built-in MTP block. The Q2 file uses imatrix-guided IQ2_XXS gate/up and
Q2_K down experts. Q4 is the higher-precision alternative.
Q2 is close enough to a 128 GB machine's memory budget that other workloads
and context size matter. Follow the [Metal](METAL.md), [Spark](DGX_SPARK.md),
or [Strix Halo](STRIX_HALO.md) starting configuration for your host.
Ordinary decode is the default. Enable the embedded draft block with `--mtp`:
```sh
./ds4-agent -m gguf/GLM-5.3-Flash-Q2.gguf --mtp --ctx 50000
```
No second model file is needed. See [sampling behavior](SPECULATIVE_DECODING.md)
before choosing between the default opportunistic mode and exact sampling.
## Full GLM 5.3 and GLM 5.2
Full GLM 5.3 Q2 is about 197 GiB. Use a sufficiently large machine or streaming:
```sh
./download_model.sh glm53-full-q2
./ds4 --ssd-streaming
```
GLM 5.2 downloads are `glm-antirez-iq2xxs`, `glm-antirez-q2`,
`glm-antirez-q4`, and the 11-shard `glm-unsloth-q4`. They are much larger
than GLM 5.3 Flash; choose memory capacity before choosing the quantization.
GLM runs on Metal, CUDA and ROCm. The routed paths include IQ2_XXS, Q2_K, and
Q4_K, with additional mixed layouts supported by the tested GGUFs. Two-Mac
ownership-aware TP accepts IQ2_XXS, Q2_K, and Q4_K gate/up layouts; this does
not mean every Q4 model fits two 128 GB machines. Use a tested artifact, not
an arbitrary combination of supported tensor types.
GLM uses graph-selected prefill chunks and does not accept `--prefill-chunk`
or an external `--mtp-model`. It currently requires `--power 100`.
Directional steering is supported for GLM 5.3, not GLM 5.2.
## Vision
PNG and JPEG input works in the CLI, native agent, and HTTP server on Metal,
single-GPU CUDA, and ROCm. The encoder must match the model.
V4.1 Flash vision is currently Metal-only; its setup is [above](#deepseek-v41-flash).
### DeepSeek Flash Vision Experimental
Vision Experimental is a different language checkpoint from Flash 0731.
The main download includes its encoder:
```sh
./download_model.sh ds4f-vision-q2
./ds4 --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf
```
Larger targets are `ds4f-vision-q2-q4` and `ds4f-vision-mxfp4`.
`ds4f-vision-encoder` downloads just the encoder when the language GGUF is
already present. Vision Experimental has its own [DSpark drafter](SPECULATIVE_DECODING.md).
### GLM 5.3 Flash
The text GGUF stays the same. Download and add the encoder explicitly:
```sh
./download_model.sh glm53-vision
./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf \
--vision gguf/GLM-5.3-Flash-Vision-Encoder.gguf
```
Use `/read image.png` in `ds4`, or start `ds4-agent` with the same `--vision`
argument to enable `view_image`. Agent sessions containing images cannot yet
be saved with `/save`.
For two-Mac TP, pass the same encoder on both ranks. The coordinator encodes
the image and sends the projected visual tokens to the worker.
For HTTP image formats and limits, see [serving](SERVER.md#images).
### Qwen3.8 Flash Next
The text GGUF stays the same. Download and add the encoder explicitly:
```sh
./download_model.sh qwen38-vision
./ds4 --mtp \
--vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf
```
The encoder is llama.cpp's Q8_0 mmproj conversion of the model's Qwen3-VL
tower. Use `/read image.png` in `ds4` or `image_url` parts over HTTP; see
[Qwen setup](QWEN38_FLASH_NEXT.md) for image limits and resize behavior.
docs/PERFORMANCE.md
+72
View File
@@ -0,0 +1,72 @@
# Performance and Benchmarking
[README](../README.md)
Compare the same checkpoint, quantization, context, and sampling settings.
Record the commit and whether weights were resident, streamed, or distributed.
Keep other GPU workloads idle and repeat in alternating order: one favorable
run is not a speed result.
## Context sweeps
`ds4-bench` measures prefill and generation at successive context frontiers:
```sh
./ds4-bench -m ds4flash.gguf \
--prompt-file speed-bench/promessi_sposi.txt \
--ctx-start 2048 --ctx-max 65536 --step-incr 2048 --gen-tokens 128
```
Each prefill number measures the newly added interval. Generation uses a fixed
greedy, non-EOS probe. The benchmark normally restores a memory snapshot after
each probe; network TP/pipeline runs and snapshots beyond its memory limit use
prefix replay instead. Do not interpret replay time as continued-prefill speed.
Use `--step-mul F` for exponential context spacing. Output is CSV, including
prefill throughput, generation throughput, and snapshot size when available.
The prompt is the cleaned public-domain *I Promessi Sposi* text in
[speed-bench](../speed-bench/README.md).
Prefill defaults are configuration-specific: ordinary DeepSeek uses 4096-token
chunks, CUDA TP uses 2048, and long PRO prompts may use 8192. GLM chooses its
own chunks and rejects `--prefill-chunk`. The strict DeepSeek API-vector test
pins 2048; do not generalize that setting to every benchmark.
## Recorded Flash Q2 baseline
These existing sweeps use 2048-token intervals and 128 generation tokens per
frontier. They are recorded baselines, not measurements of every subsequent
commit. Full data: [M5 Max](../speed-bench/m5_max.csv) and
[DGX Spark](../speed-bench/gb10.csv).
| Machine | Context | Prefill | Generation |
| --- | ---: | ---: | ---: |
| M5 Max, 128 GB | 2048 | 790.18 t/s | 39.35 t/s |
| M5 Max, 128 GB | 16384 | 572.53 t/s | 36.14 t/s |
| M5 Max, 128 GB | 32768 | 557.04 t/s | 34.36 t/s |
| M5 Max, 128 GB | 65536 | 398.50 t/s | 27.64 t/s |
| DGX Spark, 128 GB | 2048 | 825.76 t/s | 18.05 t/s |
| DGX Spark, 128 GB | 16384 | 872.44 t/s | 15.10 t/s |
| DGX Spark, 128 GB | 32768 | 855.94 t/s | 14.43 t/s |
| DGX Spark, 128 GB | 65536 | 822.98 t/s | 13.84 t/s |
![M5 Max Flash Q2 throughput](../speed-bench/m5_max_ts.svg)
Historical PRO Q2 measurements on the M3 Ultra are retained in this chart:
![PRO Q2 on M3 Ultra](../speed-bench/pro_model_m3_ultra_ts.svg)
## What to compare next
- For SSD streaming, record the effective cache and distinguish cold startup
from a warm cache.
- For multiple sessions, report both individual latency and aggregate
throughput. Ordered fallback is not native batching.
- For DSpark/MTP, compare plain decode too. A faster drafter path can still be
slower than ordinary decoding on an unpredictable prompt.
- For TP, keep quantization and prompt equal; comparing resident Q4 on two
machines with streamed Q4 on one measures capacity benefits as well as parallelism.
Recent focused TP and DSpark comparisons, including their limitations, live in
[QA_BEFORE_RELEASES.md](../QA_BEFORE_RELEASES.md). Use its speed-regression
procedure rather than accumulating one-off timings in the main README.
docs/QWEN38_FLASH_NEXT.md
+551
View File
@@ -0,0 +1,551 @@
# Qwen3.8 Flash Next
[Back to README](../README.md)
Qwen3.8-Flash-Next uses the `qwen4exp` GGUF architecture and a dedicated
Metal/CUDA graph with gated delta-net, gated GQA, block-sparse attention,
hyper-connections, n-gram embeddings, MoE, and MTP.
## Download and run
```sh
make
./download_model.sh qwen38-q2
./ds4 --ctx 8192 --prefill-chunk 1024
```
The download installs one GGUF containing the main model, MTP and original
BF16 n-grams, and updates `ds4flash.gguf`:
| Target | File size | Resident weights |
| --- | ---: | ---: |
| `qwen38-q2` | 137.10 GiB | 41.73 GiB |
| `qwen38-q4k` | 165.11 GiB | 69.74 GiB |
The 95.37 GiB n-gram table stays on disk. The runtime reads selected rows
directly from the GGUF, without mapping or preloading the table. Keep the
file on a fast local SSD. No sidecar or n-gram option is needed.
The Q2 model uses IQ2_XXS gate/up experts and Q2_K down projections padded
from 640 to 768 columns, calibrated with an imatrix. Q4 uses calibrated
Q4_K gate/up and MXFP4 down experts. Context and runtime buffers still need
RAM beyond the resident weights; start Q2 with the context and chunk above
on a 64 GB machine. Q4 needs a larger machine.
On Metal, SSD streaming keeps a bounded cache of routed experts instead of
requiring all of them in RAM:
```sh
./ds4 --ssd-streaming --ctx 4096 --prefill-chunk 128
```
The Q2 and Q4 packs support streaming during prefill, decode and MTP. Dense
weights, shared experts, context and runtime buffers still need RAM; the
n-gram table continues to use its existing disk reads. Missing routed experts
are read into owned buffers. If a prefill batch needs more experts than the
cache can hold, or an MTP layer uses a different expert size, it uses explicit
temporary staging while retaining the normal kernel arithmetic. On devices
with GPU buffer addresses, this staging contains only the distinct selected
experts. Selecting every expert, or running without buffer-address support,
retains the full-layer fallback.
During single-token decode, cache misses overlap with gate/up computation
for cached experts and the shared expert. Once the reads finish, the missing
gate/up slots are computed, followed by one down pass and the original ordered
reduction. Private address snapshots keep the running GPU work independent of
cache updates. Fully cached selections use one gate/up pass. Cached batched
prefill uses overlap on every Metal device: resident experts run while missing
weights load, then missing gate/up computation overlaps the down-weight reads.
Temporary staging retains its existing schedule; an MTP layer whose
expert size differs from the main model is staged again on each invocation,
without replacing the main model's cache entries.
A local M1 Max 32 GiB check compared this compact split schedule with
`85d37ae`, using the Q2 pack, automatic expert cache, `--ctx 4096`,
`--prefill-chunk 128`, `-n 100`, `--temp 0 --nothink`, and the prompt
`narrami la storia di roma`. Three alternating pairs produced:
| Decode | Runs (t/s) | Median (t/s) |
| --- | --- | --- |
| Before | 8.40, 8.25, 7.26 | 8.25 |
| Compact split | 8.44, 8.51, 8.14 | 8.44 |
All generated output was identical. The observed median gain was 2.3%; the
variation between runs does not establish a 10% improvement. Prefill medians
were 9.98 and 10.05 t/s; its execution path is unchanged. These are local
warm-file measurements, not a guarantee for other Macs or cold SSD reads.
Q4_K gate/up shares activation reads between the shared expert's Q8
projections and uses packed activation/scale loads while preserving the
original FP32 accumulation order. On M1 Max, the 640-to-2560 MXFP4 down
projection also enables prefetch by default. `DS4_QWEN4_MOE_DOWN_PREFETCH=0`
restores the plain down kernel for diagnostics. Safe math always retains that
plain kernel, including when prefetch is explicitly requested, because its
noncontracted arithmetic differs from the prefetched kernel's explicit FMA.
A local Q4 SSD check on M1 Max 32 GiB used `--ctx 32768`,
`--prefill-chunk 1024`, `--temp 0 --nothink`, `-n 100`, and the prompt
`Narrami in italiano la storia di Roma`. Three alternating pairs comparing
the previous kernels with these changes produced:
| Decode | Runs (t/s) | Median (t/s) |
| --- | --- | --- |
| Before | 6.02, 6.07, 6.26 | 6.07 |
| Q4 kernel updates | 6.42, 6.03, 7.27 | 6.42 |
All generated output was identical. The observed median gain was 5.8%; SSD
and system variation remained substantial. Prefill medians were 6.94 and
6.91 t/s. These changes retain the existing prefill and row-dispatch policies;
Q2's smaller sparse-prefill tiles did not show a stable benefit for Q4 in
local comparisons. The measurements do not establish gains on other devices.
MXFP4 prefill also decodes each 16-value half-block with one exponent read
and packed byte loads, retaining the original multiplication and final
half/float conversion. Compared with `46425263`, alternating resident-kernel
benchmarks on M1 Max at 2048 tokens and 10 routed slots reduced median
gate/up-plus-down time from 70.65 to 66.31 ms with 32 experts and from 81.57
to 77.34 ms with 256 experts (6.1% and 5.2% less time).
The staging oracle checks all exponent bytes, nibble codes and byte
alignments against two original 8-value decoders in both math modes.
On M1 Max, batched MXFP4 down rows also reuse activations across two output
rows. A function constant enables this only for multi-token NR2 dispatches;
single-token decode and NR1/NR4 retain the original row loop. A warmed
same-process comparison with 16 alternating samples per variant, 128 calls
per sample, width 640, 2560 output rows and 10 routed slots measured:
| Two-token down kernel | Before GPU time | After GPU time |
| --- | --- | --- |
| Q8 shared expert | 176.62 us | 150.70 us |
| MXFP4 shared expert | 216.20 us | 188.48 us |
Outputs were bit-identical, and single-token GPU timings did not regress.
Full Q4 SSD runs remain sensitive to I/O: two alternating pairs at context
32768 and chunk 1024 gave prefill medians of 22.94 and 23.41 t/s on a
201-token prompt, with all 100 generated tokens unchanged. With MTP and two
draft tokens on the Rome prompt, median generation remained 7.65 t/s despite
the faster down kernel. These checks compare against `46425263`; they do
not imply the same percentage gain for complete inference or other devices.
Leave the expert-cache budget automatic initially. A plain count passed to
`--ssd-streaming-cache-experts` requests dynamic cache slots; an `NGB` budget
also includes the reserved layer-staging space. Startup accounts for the
context, prefill chunk, static weights, vision weights when present, and
staging before choosing the cache size. Reduce the context or prefill chunk
if those fixed requirements leave insufficient memory.
The total budget is the smaller of 87.5% of physical RAM and Metal's
recommended working set, with a further 2 GiB reserved for runtime overhead.
The recommendation is not discounted again, and admitted static weights stay
eligible for locking in RAM as the expert cache grows.
Qwen fills its expert cache on demand, so `--ssd-streaming-cold` does not
change its preload behavior. Popularity preloading and the GLM-specific
`--ssd-streaming-full-layers` option are not supported for Qwen.
On M1 Max, the shared half-operand MoE prefill path requires IQ2_XXS gate/up,
Q2_K down and at least 8192 actual tokens in one streamed batch. The configured
`--prefill-chunk` is only an upper limit: a 5760-token prompt with chunk 8192
does not activate this path. Comparing that prompt at chunks 2048 and 8192
measures the change in batching, not an isolated half-operand kernel gain.
On a DGX Spark, build with `make cuda-spark` instead of `make`. Both Q2 and
Q4 fit resident on an otherwise idle 128 GB Spark; the n-grams stay on SSD.
The same commands support text, vision, MTP, steering and text-session
checkpoints in `ds4`, `ds4-agent` and `ds4-server`. No CUDA-specific model
conversion or feature flags are needed.
Leave `--prefill-chunk` unset on the Spark to use the faster 8192-token
default. The 1024-token example above saves memory on smaller Macs.
Use the same model options with `ds4-agent` or `ds4-server`.
Add `--mtp` for speculative decoding using the built-in MTP weights.
`DS4_QWEN4_MTP_PREFILL` controls how prefill prepares the predictor’s prompt history.
| Value | Predictor history for new prompt rows |
| --- | --- |
| `auto` (default) | Prepare it only with SSD streaming |
| `on` (also `1`) | Prepare it for both resident and SSD streaming runs |
| `off` (also `0`) | Skip historical projections and initialize those cache rows to zero |
To prepare the full predictor history when the weights fit in RAM, run
```sh
DS4_QWEN4_MTP_PREFILL=on ./ds4 -m gguf/Qwen3.8-Flash-Next-Q2.gguf --mtp --ctx 8192
```
Use `on` for resident workloads where the decode
benefit outweighs the prefill cost, and compare total wall time on your prompts.
`off` and resident `auto` keep MTP drafts and target verification enabled, but
the cold predictor history can change drafts and reduce their acceptance rate.
Zero keys still participate in predictor attention. Target prefill caches and logits are unchanged.
Decode continues to prepare real predictor rows. Each prefill chunk also
retains its final row for real preparation at the next boundary, keeping
checkpoint restore consistent with uninterrupted prefill.
The setting applies to newly prefilled rows. Reused or restored checkpoints
retain their existing predictor cache contents.
Without `--mtp`, this setting has no effect.
`ds4-bench` benchmarks ordinary decoding; it does not accept `--mtp`.
`--nothink` disables thinking. The server exposes
`qwen3.8-flash-next`, `qwen3.8-flash-next-chat`, and
`qwen3.8-flash-next-reasoner` aliases. Tool calls use the native
`<tool_call><function=...><parameter=...>` format.
With resident weights on Metal, `ds4-server --batched-session N` decodes
the slots together. Work that
only reads weights runs once for the whole batch, while the delta-net
recurrence, the attention caches and the n-gram convolution stay per session.
The slots also share one prefill workspace, so an extra slot costs its caches
rather than another few GiB of transients. As with the other natively batched
models, grouping changes the order of floating point reductions, so a batched
reply can differ from the same prompt decoded alone.
Add `--mtp` to use batched speculative decoding as well. The server accepts
matching greedy drafts by default, including at nonzero temperature. With
`--mtp-exact-sampling`, sampled requests use ordinary batched decoding;
temperature-zero requests can still speculate. Default greedy acceptance also
means a seed need not reproduce a reply under a different batching schedule.
Speculative batches above 16 sessions, images and steering use the ordered
fallback. Metal SSD expert streaming also uses the ordered fallback and
keeps each session's streaming workspace; it does not use the resident batch
arena. CUDA currently decodes sessions in order.
Disk KV checkpoints include recurrent state. Rewinding to an earlier position
replays the retained prefix on the next evaluation. The native context is
262144 tokens; `DS4_QWEN4_YARN_FACTOR=2` or `=4` enables static YaRN for
longer contexts, with a possible quality cost on shorter prompts.
## Metal tuning on M1 Max and M2-M4
The established M1 Max simdgroup optimizations also apply to Apple M2, M3
and M4, including their Pro, Max and Ultra variants, within the same tensor
shapes, batch sizes, math modes and diagnostic overrides:
- Wide Q8 decode projections, HC gate/mix reuse and short MTP Q8 projections.
- IQ2 gate/up, Q2_K down-row reuse and the Q4 model's MXFP4 down prefetch.
- Small HC-down and HC-normalization prefill batches, and large Q8 SSD prefill.
- Sparse IQ2/Q2 SSD tiles, Q4/MXFP4 K32 tiles, optional half-operand staging
for large SSD batches and release of that scratch before Q4 decode.
M3 Ultra retains its dedicated MoE specialization, row/group geometry and
prefill tile policy; the other shared Q8, HC and MTP choices still apply.
M5/M6 retain their existing policy. The selection does not expand to M1
base/Pro/Ultra or to future generation names.
This widens device selection, not the kernel arithmetic or GGUF formats.
Local validation uses M1 Max. An independent M4 Max Q2 resident test reported
about 7% more decode throughput after widening the decode gates, with matching
outputs; it did not validate M2/M3, other M4 variants, or SSD performance.
See the [M4 Max test report](https://github.com/antirez/ds4/pull/1056#issuecomment-5922543210).
Device eligibility therefore does not imply a measured speedup on every chip.
## DGX Spark performance
Measured on a single Spark with resident weights and disk-only n-grams:
| Model | First 1024 tokens | Next 7168 tokens | Decode at 8192 tokens |
| --- | ---: | ---: | ---: |
| Q2 | 516 t/s | 745 t/s | 22.6 t/s |
| Q4 | 513 t/s | 755 t/s | 21.2 t/s |
These are averages of two runs using `speed-bench/promessi_sposi.txt`,
8192-token prefill chunks and 128 teacher-forced decode tokens, without MTP.
Loading the model is excluded. Q2 also reached about 771 t/s on a fresh
32K-token prefix with `--prefill-chunk 32768`. Small continuations have lower
throughput: adding 32 tokens after an 8K Q4 prefix took about 273 ms.
MTP speed depends on how often drafts are accepted; the table measures
ordinary decoding.
## Conversion
See [GGUF conversion](../gguf-tools/README.md) for conversion from the HF
checkpoint and calibrated low-bit expert quantization.
`gguf-tools/qwen4_pack_to_qwen4exp.py` migrates old DS4 fast packs to the
active `qwen4exp` schema. Finish older main-only conversions with
`gguf-tools/qwen4_native_ngrams.py`, using the original HF n-gram shards.
Expanding an old quantized table to BF16 does not restore its lost precision.
## Vision
Images go through the model's Qwen3-VL vision tower. Download the encoder
with `./download_model.sh qwen38-vision`, then use
`--vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf`. The CLI accepts
`/read image.png`; the server accepts `image_url` parts. Each image is resized to
multiples of 32 pixels within 64 to 1024 tokens (`DS4_QWEN4_IMAGE_MAX_TOKENS`
raises the cap), encoded on the GPU, and takes the model's 3D rope positions;
live KV reuse keys on the image fingerprints. `make test-qwen4-vision`
compares the tower with the Hugging Face implementation using the same GGUF
weights, dequantized to float32. It separately reports GGUF-versus-original
checkpoint quality and GPU-versus-original agreement. Both comparisons use
the unchanged minimum per-token cosine threshold of 0.99; implementation
parity gates the default exit status. Add `--require-quality` to also fail on
GGUF-versus-original quality loss. A passing implementation check alone does
not mean the quantized encoder matches the original checkpoint.
For a small suite including OCR, diagrams, a photograph, and resizing:
```sh
make tests/test_qwen4_vision
uv run --with numpy --with torch --with torchvision --with pillow \
--with safetensors --with transformers --with gguf tests/qwen4_vision_ref.py \
--snapshot /path/to/HF-checkpoint \
--mmproj gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf \
--image tests/vision-fixtures/qwen38/orbit.png \
--image tests/vision-fixtures/qwen38/maple.png \
--image tests/vision-fixtures/glm53/diagram.png \
--image tests/vision-fixtures/glm53/text.png \
--image tests/vision-fixtures/glm53/earth.jpg \
--image tests/vision-fixtures/glm53/screenshot.png \
--json-report /tmp/qwen38-vision-parity.json
```
Transformers must include `qwen4_exp`; the reference runs on CPU. Repeat
`--image` to add cases. The original single-image `--out` embedding dump is
still supported. Metric checks without model weights run with
`uv run --with numpy python -m unittest discover -s tests -p test_qwen4_vision_ref.py`.
JPEG decoder agreement with Pillow, including progressive scans, subsampling,
and image edges, is checked separately with
`uv run --with numpy --with pillow python -m unittest discover -s tests -p test_jpeg_decode.py`.
To check CLI `/read` image turns and text follow-ups with ordinary and MTP
decode, use two different PNG or JPEG images with the model-backed regression:
```sh
make ds4
uv run tests/test_qwen4_cli_vision.py --model /path/to/main-with-mtp.gguf \
--vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf \
--image /path/to/first.png --image /path/to/second.png
```
The test checks that all four turns complete in each mode, including errors
that the interactive CLI can report without a nonzero process exit. It saves
the responses and diagnostics for inspection; it does not grade image content.
The Metal and CUDA graphs accept Q8_0, Q4_0, F16, BF16 and F32
dense weights, Q8_0/MXFP4/Q4_0/Q4_K/Q2_K/IQ2_XXS experts, F16/F32/Q8_0
hyper-connection mixers and the original BF16 n-gram table.
Tensor parallelism and pipeline execution are not implemented for this model
yet. SSD expert streaming is supported on Metal only. ROCm is not supported.
CPU code is a correctness reference, not a general inference backend.
M6 uses the same Metal dispatch defaults as M5, including the Qwen indexer,
attention merge, HC mixers and MoE scheduling. Existing M5 diagnostic rollback
switches apply to both generations; tensor kernels still require runtime
capability support. The F16 HC mixers explicitly preserve their accumulation
order in both normal and safe shader math modes. This does not guarantee
bit-identical logits across GPU generations: tensor matmuls retain their own
accumulation order.
## Validation
```sh
make test-qwen4-kernels test-qwen4-q2 test-qwen4-prefill-reuse test-q8-prefill-variants
make test-qwen4-ssd-experts test-qwen4-memory
make test-qwen4-hc-math test-metal-device-policy
make test-frontends
make test-qwen4-ngrams
make tests/test_qwen4_ngram_state
make -B -C gguf-tools quants-shared
python3 -m unittest discover -s gguf-tools/tests -p test_qwen4_pack.py
python3 -m unittest discover -s gguf-tools/tests -p test_qwen4_native_ngrams.py
```
The kernel tests exercise the active Metal API. Vision and end-to-end model
checks additionally require the checkpoints described above.
The SSD test needs no GGUF: a sparse temporary file uses a 512-expert table,
including indices 383, 384 and 511, with cold and warm selections, eviction,
prefill staging, different-size MTP layers and failed reads followed by exact recovery. It
compares routed intermediates, reduced outputs and hyper-connection residuals
bit for bit against the resident kernels, including the padded Q2 down rows.
Its test-only backend checks actual `pread` requests against selected expert
ranges and counts returned bytes: ten experts for off-size MTP, 24 for cache
overflow, zero reads for cache hits, and the full 512-expert fallback.
Single-token tests also cover partially cached selections, duplicate IDs,
optional shared experts and all 16 routed slots. A failed SSD read must leave
the completed cached/shared gate/up slots exact, preserve the untouched slots,
and allow an exact retry after draining GPU work.
The memory-estimate test runs without a GPU or model.
To compare ordinary one-shot generation with the session path at identical
prefill boundaries, including every vocabulary logit during continuation:
```sh
make tests/test_qwen4_generation
./tests/test_qwen4_generation gguf/Qwen3.8-Flash-Next-Q2.gguf --ssd-streaming
# Exercise sparse attention and many chunk boundaries with a longer fixture:
./tests/test_qwen4_generation gguf/Qwen3.8-Flash-Next-Q2.gguf \
--ssd-streaming --chunk 128 --ctx 8192 --tokens 5760
```
Resident and SSD inference use the same arithmetic for prompt projections,
attention and shared experts. `make test-qwen4-resident-arithmetic` checks
these operations against the resident reference without loading a GGUF.
On a Mac that can hold the model, add `--prefill-reference` to the streamed
generation test to compare every vocabulary score with a resident run.
Omit `--ssd-streaming` on a machine that can keep the model resident. The
test checks explicit chunk precedence, progress frontiers, greedy tokens and
full logits. Different chunk sizes can change reduction order and routed
expert choices; compare the actual chunk boundaries before diagnosing token
drift. In particular, upstream `8db1d1d` ignores `--prefill-chunk` in the
ordinary one-shot Qwen path, although its sessions honor the option. For an
A/B comparison with that revision, also set `DS4_QWEN4_PREFILL_CHUNK` to the
same value in both binaries. This branch uses an explicit chunk in preference
to that environment setting and clamps it to the context size.
On CUDA, use `make test-qwen4-cuda` for the kernel tests. They compare the
active kernels with independent CPU references, without model weights.
Vision and end-to-end checks additionally require the checkpoints above.
Run `tests/test_qwen4_ngram_state MODEL.gguf` on either backend to check
failed disk reads during prefill, decode and MTP, then exact recovery.
For a smaller memory footprint, run
`tests/test_qwen4_ngram_state MODEL.gguf --ssd-streaming`. This uses a
256-token context, an eight-token prefill chunk and a 1024-expert cache request
while keeping both comparison sessions and MTP enabled.
To compare the predictor's cache-only prefill with a complete MTP pass:
```sh
make tests/test_qwen4_mtp_prefill
./tests/test_qwen4_mtp_prefill gguf/Qwen3.8-Flash-Next-Q2.gguf --ssd-streaming
```
This checks one-token prompts, chunk boundaries, non-argmax continuation,
stale cache contents and payload restore, followed by greedy, depth-three
and exact-sampling continuation checks. It uses a 128-token context and a
1024-expert cache request. The small-buffer MTP CLI regression also accepts
`--ssd-streaming`.
For the sparse-attention boundary and a 17408-token prefix, use the longer
predictor regression:
```sh
./tests/test_qwen4_mtp_prefill gguf/Qwen3.8-Flash-Next-Q2.gguf \
--ssd-streaming --long-context --prefill-chunk 2048 \
--dump-prefix /tmp/qwen-mtp-long-2048
```
Repeat with `--prefill-chunk 128` and a different dump prefix. The fixture
uses repeated, fixed chat token IDs, one graph, a 17416-token context and a
1024-expert cache request. It runs the trunk once and probes actual residual
rows across the first sparse query and at the final prefix. Each probe
compares cache-only preparation, the optimized last-row predictor and a
recursive draft with the frozen full-layer reference, using identical input
rows and lookahead tokens. It checks complete logits, cache contents, EH
projections, residuals and unchanged trunk state for T1/T2/T3 and exact T3
verification. Destination cache rows are poisoned before each replay.
Bit-identical comparisons are between the candidate and frozen reference
replayed from the same saved predictor state within each run. Equality between
chunk sizes is checked separately: chunk-dependent grouping can change the
arithmetic used to build the historical predictor cache.
`--dump-prefix` writes the fixture IDs, target/trunk captures, reference and
candidate predictor logits, and a JSONL manifest for comparisons between
builds. `--prefix-tokens N` changes the fixture length; it must be a multiple
of four covering the sparse boundary. These are implementation checks on a
synthetic sequence, not a quality evaluation or an acceptance-rate benchmark.
Comparing MTP acceptance between commits additionally requires the same
prompt, sampling settings and draft-depth policy. A trunk arithmetic change
can alter predictor inputs even when the predictor itself remains correct.
On an M1 Max with 32 GiB and Q2 SSD streaming, the 17408-token fixture passed
at chunks 128 and 2048 on `11811b9`, and at chunk 2048 on `04c0867` and
`89986c3`. At chunk 2048, all 36 captured tensors from `89986c3` and `11811b9`
were byte-identical. Differences from `04c0867` already appeared in the trunk
residuals after the attention fix in `89986c3`; both predictor implementations
followed those changed inputs.
Between chunks 128 and 2048 on `11811b9`, the two captured trunk slices and
final target logits were byte-identical. Predictor logits differed by at most
0.0001266003, with no argmax change in the 16 full/recursive comparisons. A
separate replay of identical rows as T3, T1+1+1, T2+1 and T1+2 found arithmetic
differences beginning at the HC down projection; its per-stage differences
were identical on `04c0867` and `11811b9`. This identifies an existing source
of grouping-dependent predictor cache values, not a demonstrated quality or
acceptance regression. The full-model runs above do not exercise the
8192-row half-operand path.
The independent `test_metal_qwen4_moe_half` comparison against the shader
source from `89986c3` also passed 31 fixtures, including 8192 rows, with Metal
API validation. Full shader validation exceeded the M1 threadgroup-memory
limit for the NT8 gate/up stress case (36864 bytes reported, 32768 available).
A separate NT1/2/4 subset passed with shader validation, including the
8192-row fixture at mid NT4/down NT4. These kernel checks do not replace an
8192-chunk full-model acceptance comparison.
Official Alibaba continuations are tracked for 100 short prompts and 12
archive/code prompts from 2K to 24K tokens. Build the quality scorer, then run
from the repository root:
```sh
gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/data/qwen38-flash-alibaba-100/manifest.tsv /tmp/qwen-short.tsv 4096
gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/data/qwen38-flash-alibaba-long/manifest.tsv /tmp/qwen-long.tsv 32768
```
Repeat with `--quality` and, for the long set, `--continued-prefill 1` and
`--continued-prefill 256`. These fixtures match the no-thinking template;
no rendered-prompt flag is needed. See [quality testing](../gguf-tools/quality-testing/README.md)
for collection settings, measurements and the hosted-checkpoint limitations.
[Checkpoint-fix benchmark charts and measurements](../speed-bench/qwen38-checkpoints/README.md)
compare prefill, ordinary decode, and MTP decode against the preceding PR head.
Qwen prefill checkpoints include matching logits at each completed chunk,
including cancellation frontiers. To test save/restore and continuation:
```sh
DS4_TEST_MODEL=/path/to/main-with-mtp.gguf \
DS4_TEST_GLM_MTP=1 DS4_TEST_MTP_EXACT=1 \
./ds4_test --qwen4-prefill-checkpoints --qwen4-restore-reuse \
--session-snapshot --session-rewind --session-rewind-resample
python3 tests/test_qwen4_checkpoint_replay.py \
--model /path/to/main-with-mtp.gguf
python3 tests/test_qwen4_mtp_limits.py \
--model /path/to/main-with-mtp.gguf
python3 tests/test_qwen4_logit_dump.py \
--model /path/to/main-with-mtp.gguf
```
The server regression checks continued disk saves, cache-budget eviction,
and text-answer history with and without tool schemas. It covers clients
that omit reasoning and clients that echo `reasoning_content`, including
server restart after a continuation. For tools-enabled histories, the first
answer keeps the exact disk key; after a client continues without reasoning,
the next checkpoint uses the visible history key for restart reuse.
The MTP tests cover small prefill buffers, rewinds, reused sessions, and
truncated checkpoints. The logit-dump test requires NumPy and compares all-row
prefill with teacher-forced decode across a chunk boundary. For official
continuation scoring, use `--rendered-prompt`
when a fixture already contains the complete model chat template.
`tests/test_qwen4_prefill MODEL PROMPT 8192` checks mixed prefill sizes,
progress callbacks and exact replay through the sparse-attention boundary.
It also reports differences against a fresh prefill followed by individual
decodes. Those schedules can round differently and exchange nearly tied
experts; their full logits need not match. Use model-quality checks as well,
not only state replay:
```sh
python3 tests/test_server_story.py --url http://127.0.0.1:8000 \
--model qwen3.8-flash-next --output /tmp/qwen-story
```
Start the server with at least `--ctx 49152`. This checks all sixteen facts
in a 31K-token story, then a correction turn that must reuse the prefix.
Qwen directional steering and activation capture support all 48 trunk layers;
see [directional steering](../dir-steering/README.md). Model-backed regressions:
```sh
python3 tests/test_qwen4_steering.py --model /path/to/main-with-mtp.gguf
./ds4_agent_test --full-context-save /path/to/main-with-mtp.gguf
make test-web-recovery
```
The agent can save an already-full text transcript without a KV payload.
Restart with a larger `--ctx`, then `/switch` to the saved session and
`/compact` to recover room. Sessions containing images still use the existing
save restriction. Automatic compaction reserves summary space and continues
interrupted generation through the existing agent compaction path.
docs/QWEN38_Q4_REGISTER_REUSE.md
+149
View File
@@ -0,0 +1,149 @@
# Qwen Q4_K raw-load register reuse
Measured on 9 October 2026: warmed fresh 16K prefill improved **2.38%** on the
standard text fixture and **2.27%** on C source, with exact old/new output checks.
## Change and provenance
The NAX gate/up and down lookahead loops previously reloaded a Q4_K header and
quant chunk for every 32-weight group. The new next-step loader retains the
header within each 256-weight superblock and the quant words across each
low/high-nibble pair. Initial loads and non-Q4 loaders are unchanged. The
existing dequantizer, arithmetic order, barriers and dispatch geometry are
unchanged. No additional buffer or runtime flag is introduced.
Inspired by the Q4 portion of [StarForge 08ea4d7](https://github.com/Chida82/sf-q3-8flash/commit/08ea4d7eb2e780e2d31690766600880e8dea8718).
Only this register reuse is adopted; the donor's other optimizations are excluded.
The measured host baseline was patched-source commit
`b51ae1ce23a59e5e5f7031c0144e76ac35f13e6c`. It already included
[PLE prefetch #1154](https://github.com/antirez/ds4/pull/1154), common to both arms.
**These gains are incremental to #1154, not a measurement of that PR.** The PR
is based on upstream `fc80bd695da76ee14bcb86820bf900a5cdfda806`; its unmodified
shader, kernel test and Metal host source match the campaign baseline. The
complete upstream host integration was built separately, but was not rerun
through the full model campaign.
Shader SHA256:
- Control: `13224aeae23802defef678877146ae8047a45c2d1be824361efa2f4f5aa25b28`
- Candidate: `dff0b9e48e5d55b68a818d4ccc28ea3468ecd464b8eff83cf426dcb9a102b4cf`
## Measurement
M5 Max, 128 GiB, macOS 26.6.2, resident Qwen3.8 Flash Next Q4 GGUF, Q4_K routed
gate/up and MXFP4 routed down, 8192-token chunks, native context allocation
262144, no batching. Each process warmed weights and a disposable complete
prefill/decode session, freed that session, then measured fresh KV on the same
engine/source. This avoids first-use specialized Metal pipeline compilation.
Only one model process/session ran at a time. Timed runs did not add stage
synchronization or state dumps. Decode measurements use 64 ordinary greedy
tokens; they do not measure MTP throughput.
| Corpus | Runs / balanced blocks | Prefill throughput gain | Conditional 95% block-bootstrap interval | Ordinary decode change |
| --- | --- | --- | --- | --- |
| Standard text, 16384 tokens | 16 / 4 | +2.376% | +2.172 to +2.660% | -0.061% |
| C source, 16384 tokens | 8 / 2 | +2.274% | +2.147 to +2.401% | -0.622% |
Each block uses ABBA or BAAB (A=control, B=candidate). Its ratio is mean B time
/ mean A time; the summary is the geometric mean of block ratios, converted
with `gain = 100 * (1 / ratio - 1)`. The interval enumerates ordered block
resamples with replacement. It is conditional on these few blocks, not a
complete estimate of host uncertainty. A warmed A/A block showed a false
-1.83% difference, and each arm's text prefill times drifted about 6–7%
first-to-last. All four text blocks and both code blocks favored the candidate.
Cold-process results are excluded from this estimate. Do not extrapolate this
gain to every context length.
Historical model SHA256:
`680944460a8cbe93ba8b6d7b6107213ffb7e22320bd913000e563ca0a0f25a8a`.
The 177280286720-byte model was not rehashed for this campaign; its recorded
size, inode and modification time matched the previously verified file.
### Per-run durations
Seconds reconstructed from CSV throughput rounded to two decimal places;
printed precision below is not measurement precision. Each row measures
16384 prefill tokens and 64 ordinary decode tokens.
| Corpus | Run | Arm | Block | Prefill seconds | Decode seconds |
| --- | --- | --- | --- | --- | --- |
| Text | 1 | A | 1 | 10.618895 | 1.210974 |
| Text | 2 | B | 1 | 10.405314 | 1.201652 |
| Text | 3 | B | 1 | 10.576054 | 1.200075 |
| Text | 4 | A | 1 | 10.947627 | 1.197829 |
| Text | 5 | B | 2 | 11.428253 | 1.231954 |
| Text | 6 | A | 2 | 11.613763 | 1.234806 |
| Text | 7 | A | 2 | 11.543149 | 1.238151 |
| Text | 8 | B | 2 | 11.247340 | 1.234806 |
| Text | 9 | A | 3 | 11.488353 | 1.211892 |
| Text | 10 | B | 3 | 11.199748 | 1.226054 |
| Text | 11 | B | 3 | 11.201050 | 1.218351 |
| Text | 12 | A | 3 | 11.421640 | 1.221141 |
| Text | 13 | B | 4 | 11.157419 | 1.220675 |
| Text | 14 | A | 4 | 11.397010 | 1.217193 |
| Text | 15 | A | 4 | 11.355461 | 1.213730 |
| Text | 16 | B | 4 | 11.078879 | 1.218120 |
| Code | 1 | A | 1 | 10.207209 | 1.199850 |
| Code | 2 | B | 1 | 10.157848 | 1.220210 |
| Code | 3 | B | 1 | 10.268109 | 1.225115 |
| Code | 4 | A | 1 | 10.657300 | 1.221840 |
| Code | 5 | B | 2 | 10.232773 | 1.218351 |
| Code | 6 | A | 2 | 10.552621 | 1.219048 |
| Code | 7 | A | 2 | 10.505931 | 1.217193 |
| Code | 8 | B | 2 | 10.331954 | 1.224646 |
## Correctness and regression checks
Campaign checks passed on the patched host baseline and final local integration:
- The focused fixture directly byte-compared 87,492,020 old/new output bytes,
covering both changed call sites, NAX levels 0–6, Q4/Q4, Q4/MXFP4, all-MXFP4
and IQ2XXS/Q2_K, K512/2560, signed/zero finite Q4 headers, shuffled routes and
counts around 8/16/32/64 boundaries. Live values must be finite and nonzero;
reserved output slots and trailing write sentinels must remain untouched.
- All 24 warmed runs matched each corpus's full 248320-float frontier row.
- Complete persisted payloads at 16384 and 16896 tokens, save/load/resave,
128 ordinary decode vocabulary rows, token IDs and positions matched exactly.
- Near-native retained prefixes at 131072, 245760 and 246272 tokens matched
full frontier logits, including the final 512-token append. These are
correctness observations, not statistically repeated speed measurements.
- `make`, `make test`, the full Qwen kernel suite, PLE rollback/prefetch/failure
tests, checkpoint/restore/snapshot/rewind with opportunistic MTP, ordinary/MTP
vision completion tests and synthetic Metal resident/SSD Q4 comparisons passed.
- An isolated 11-request sampled tool/history/restart fixture passed at
temperature 0.6 with non-exact MTP: 102 verification cycles, 94 drafts accepted.
This bounded smoke is not a multi-hour agent soak or an MTP speed comparison.
CUDA sources are untouched; no external CUDA or distributed runtime test is
claimed. The finite fixture is not exhaustive over all possible bit patterns.
Serialized state comparisons do not cover every ephemeral MTP scratch buffer.
The existing masked-row loaders can form inactive weight-row addresses past a
65-row logical tensor; this is present in both arms. Output sentinels check
writes, not logical weight-read bounds. Register reuse preserves/subsets those
read addresses; deployed row dimensions are multiples of 64. A bounds repair
needs separate validation and is not included in this patch.
### Reproduce the focused exact comparison
On M5, from the PR checkout, run the same test executable in separate processes
against the baseline and candidate shaders. A fixture pass alone is not the
exactness oracle; compare the complete dumps. Run GPU/model tests only when the
managed server is free and exclusively reserved.
```sh
make tests/test_qwen4_kernels
git show fc80bd695da76ee14bcb86820bf900a5cdfda806:metal/qwen4.metal > /tmp/qwen4-control.metal
DS4_METAL_QWEN4_SOURCE=/tmp/qwen4-control.metal DS4_TEST_QWEN4_RAW16_ONLY=1 DS4_TEST_QWEN4_RAW16_DUMP=/tmp/qwen4-control.raw ./tests/test_qwen4_kernels
DS4_METAL_QWEN4_SOURCE="$PWD/metal/qwen4.metal" DS4_TEST_QWEN4_RAW16_ONLY=1 DS4_TEST_QWEN4_RAW16_DUMP=/tmp/qwen4-candidate.raw ./tests/test_qwen4_kernels
cmp /tmp/qwen4-control.raw /tmp/qwen4-candidate.raw
```
Expected dump size: 87492020 bytes; expected SHA256 for each:
`6cd1882c9ede246f78cad98f0e6d13b63a045a5d0ba332e186e25e57e5dba554`.
Timing used a diagnostic warmed-session harness rather than the stock first
prefill invocation. Raw local manifests, shaders, dumps and generator scripts
are retained outside the PR; the durations above allow the reported timing
ratios to be independently recalculated.
docs/SERVER.md
+154
View File
@@ -0,0 +1,154 @@
# Serving Models
[README](../README.md) | [Client setup](CLIENTS.md)
## Start locally
```sh
./ds4-server --ctx 32768
```
The default address is `http://127.0.0.1:8000`. Use `--host 0.0.0.0` to listen
on other interfaces. Restrict access to trusted clients; for an Internet-facing
deployment, put authentication and TLS in front of the server.
`--cors` enables browser cross-origin headers. It does not change the listening
address or provide access control.
The selected build chooses the backend. Pass `-m FILE` for an explicit model.
Use `--chdir /path/to/ds4` when starting outside the project directory so
relative runtime files such as Metal kernels can be found.
## APIs
| Endpoint | Use |
| --- | --- |
| `GET /v1/models` | Loaded model information |
| `POST /v1/chat/completions` | OpenAI-style chat |
| `POST /v1/responses` | Responses-style requests and continuations |
| `POST /v1/completions` | Text completions |
| `POST /v1/messages` | Anthropic-style messages |
The Flash and PRO names accepted by the model endpoints are compatibility
aliases, not separate loaded models. The GGUF passed at startup selects the model.
`GET /v1/models` and `GET /v1/models/{id}` include `architecture.input_modalities` and `architecture.output_modalities`. Input modalities are `["text", "image"]` when the loaded engine has vision enabled, otherwise `["text"]`; output modalities are always `["text"]`. Compatibility aliases report the same loaded-engine capabilities.
```sh
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"Explain Redis streams."}],"stream":true}'
```
Chat, Responses, and Anthropic support tools and SSE streaming. Reasoning is
returned separately from visible text in each API's native form. Standard
sampling and output-budget fields are supported; explicit request parameters
take precedence over defaults.
The default sampling settings are temperature 1, top-p 1, and min-p 0.05.
For DeepSeek, thinking is on by default. `reasoning_effort=max` selects Think
Max only with sufficient context; otherwise it falls back to normal thinking.
`xhigh` maps to normal thinking, not Think Max. Use `think:false`, a disabled
thinking object, or a non-thinking model alias for direct answers.
## Multiple sessions
```sh
./ds4-server --ctx 4096 --batched-session 4
```
Without `--batched-session`, there is one resident session. With it, the server
preallocates independent KV states and queues requests when all slots are busy.
Choose context and slot count together: a context that fits once may not fit
four times. Idle slots can be cached before reuse; active requests are not evicted.
Where the model supports it, the slots share one prefill workspace instead of
each keeping its own, so an extra slot costs only its caches. That matters most
for Qwen3.8 Flash Next, whose transients are sized by the prefill chunk rather
than by the context: at the default chunk they run to several GiB per session.
The startup line reports both figures.
| Backend/model | Decode execution |
| --- | --- |
| Metal, resident Flash | Native shared-expert/QKV batching where supported |
| Metal, resident V4.1 Flash | Native decoding for 2-8 sessions |
| Metal RDMA TP, V4.1 Flash | Native decoding for 3-8 sessions; ordered fallback for two |
| Metal SSD streaming, V4.1 Flash | Ordered fallback |
| Metal, GLM 5.2 | Ordered fallback |
| Metal, GLM 5.3 | Native batching through 2051 visible tokens; ordered fallback afterward |
| Metal, Qwen3.8 Flash Next with resident weights | Native batching of the shared work; recurrent state, caches and PLE history stay per session |
| CUDA, supported multi-GPU Flash TP layout | Native grouped decode and mixed prefill/decode |
| Metal, Qwen3.8 Flash Next with SSD expert streaming | Ordered fallback with per-session streaming workspaces |
| Single-GPU CUDA, including Spark | Ordered fallback |
Fallback executes the rows separately. It provides concurrency and scheduling
fairness, not the aggregate speedup of native batching. Native grouping may
change floating-point reduction order slightly. V4.1 sessions containing images
use the ordered fallback.
Long prefills yield to active decoders in bounded intervals, normally 128
tokens. `--mixed-prefill-quantum N` changes that interval for testing. While
nothing is decoding, a V4.1 slot prefills in the engine's own chunk.
Session-batched serving uses ordinary target decoding, except Qwen3.8 on
Metal, where `--mtp` also batches speculative decoding with resident weights
and uses ordered speculative cycles with SSD expert streaming. Its
`--mtp-exact-sampling` mode uses ordinary batches for nonzero-temperature
requests. Other models do not use MTP/DSpark while session batching is active.
For the eight-L40S example, see [CUDA GPUs](CUDA_MULTI_GPU.md#serve-multiple-users).
## Images
Start with the matching language GGUF and `--vision FILE`; see
[model-specific instructions](MODELS.md#vision).
OpenAI chat and Responses accept inline PNG/JPEG data URIs. Anthropic accepts
base64 image sources. Remote URLs and server-side file paths are rejected.
Image blocks preserve their order in the request. The limit is 16 images and
a 64 MiB HTTP body.
## Disk KV cache
Disk caching saves useful prefixes across slot reuse and server restarts:
```sh
./ds4-server --ctx 100000 \
--kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
```
Clients can resend complete conversation histories. The server first tries
the live token prefix, then compatible rendered-text prefixes from disk, and
prefills the new suffix. With multiple slots, each slot has its own live state;
disk is the additional persistence layer, not the only way to retain sessions.
TP cache loading rebuilds the saved token prefix on both ranks. It is not an
instantaneous restoration of both GPUs. Pipeline loading redistributes the
saved layer state over its route.
Defaults are intended to avoid saving fragile token boundaries. For unusual
workloads, the controls are `--kv-cache-min-tokens`,
`--kv-cache-cold-max-tokens`, `--kv-cache-continued-interval-tokens`,
`--kv-cache-boundary-trim-tokens`, and `--kv-cache-boundary-align-tokens`.
Check `./ds4-server --help` for their defaults.
Quantization variants may share compatible prefixes. Add
`--kv-cache-reject-different-quant` for same-quant reuse only.
Cache files contain prompt text and model state: treat the directory as
private. It is disposable; stop the server before clearing it.
## Tool history and debugging
For DeepSeek, the server preserves sampled DSML tool blocks and assigns
unguessable tool IDs. Replaying those IDs avoids retokenizing a differently
formatted JSON history. The bounded replay map can be stored in cache files.
When exact replay is unavailable, canonical rendering may require rebuilding
part of the prefix.
`--tool-memory-max-ids` bounds this map.
`--disable-exact-dsml-tool-replay` disables it for diagnostic comparisons.
Use `--trace /tmp/ds4-trace.txt` to record prompt rendering, cache decisions,
generated text, and tool-parser events. Traces can contain sensitive content.
Cache formats are implementation details. The current header and extension
definitions are in [ds4_kvstore.h](../ds4_kvstore.h) and
[ds4_kvstore.c](../ds4_kvstore.c); model-specific payload handling is in
[ds4.c](../ds4.c).
docs/SPECULATIVE_DECODING.md
+179
View File
@@ -0,0 +1,179 @@
# Speculative Decoding
[README](../README.md)
Speculation proposes future tokens with a smaller draft block, then checks
them with the main model. An accepted prefix advances generation by several
tokens in one verification pass. It does not accelerate prefill.
It is opt-in. Gains depend on the prompt, model, backend, and context length;
poor acceptance can make it slower. Measure your workload rather than assuming
that a draft model always helps.
## DeepSeek Flash: DSpark
DSpark is a separate support GGUF, not a standalone language model. It proposes
up to five future tokens. Match its checkpoint to the main model:
| Main checkpoint | Download | Support file |
| --- | --- | --- |
| Flash 0731 | `ds4f-dspark` | `gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf` |
| Flash Vision Experimental | `ds4f-vision-dspark` | `gguf/DeepSeek-V4-Flash-Vision-Exp-DSpark-support.gguf` |
For the 0731 Q2 model:
```sh
./download_model.sh ds4f-q2
./download_model.sh ds4f-dspark
./ds4 --dspark --mtp-model gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf
```
For Vision Experimental, substitute its matching main model and support file.
Do not mix the two checkpoints. DSpark is not supported for PRO.
The same flags work in `ds4-agent` and non-batched `ds4-server` requests.
The support file adds about 5.6 GiB of weights plus runtime state. The main
model must be resident; external DSpark is not supported with SSD streaming.
DSpark replaces the legacy one-stage MTP drafter for that run; the two are
not stacked.
Resident M5 paths batch supported verifier expert rows, including two-Mac TP.
On DGX Spark, resident Flash 0731 Q2 and two-Spark MXFP4 also batch the seed
with longer drafts and use small-batch Q8 and expert kernels. No extra flags
are needed.
Single-Spark Q2 uses a confidence-pruning threshold of 0.4. Two-Spark
Flash 0731 Q2 and MXFP4 leave confidence pruning off and split the drafter
over RDMA when both peers load its support file. `--dspark-confidence`
overrides the threshold. Exact sampling keeps its default threshold of 0.8.
The scheduler can back off when drafting is unproductive. Defaults select the
fast paths; diagnostic environment variables are not needed for normal use.
Recorded comparisons are in [the QA guide](../QA_BEFORE_RELEASES.md).
For the tested Strix Halo coding configuration, use `--dspark --dspark-confidence 0.7` with the default five-token draft cap and scheduler. Client sampling is temperature `1.0`, `top_p=0.95`, `min_p=0`, and `top_k=0`; high reasoning was also checked on coding and tool-use requests. This uses opportunistic sampling as described below; exact-mode throughput is not qualified by these measurements. `--mtp-draft` controls legacy autoregressive MTP, not the DSpark draft width.
## DeepSeek V4.1 Flash: DSpark
V4.1 has its own 8.04 GiB support GGUF. It retains the native MXFP4 experts
and uses a different draft graph from V4 Flash. Do not interchange the files.
See [conversion](../gguf-tools/README.md#convert-v41-dspark) to build it.
```sh
./ds4 -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
--ssd-streaming --dspark \
--mtp-model gguf/DeepSeek-V4.1-Flash-DSpark.gguf
```
The same options work with `ds4-agent` and `ds4-server`. Metal, CUDA and ROCm
support single-host execution, with either resident target weights or SSD
streaming. The drafter stays resident, so leave another 8 GiB for it.
Two-host tensor parallelism works on Metal and CUDA: add the same `--dspark
--mtp-model FILE` options to both the coordinator and worker. Each peer keeps
the drafter resident; the target and its verification run across both GPUs.
Up to five drafts are checked together. The sampling choices below apply;
`--mtp-exact-sampling` uses sampled proposals and the full acceptance/correction
rule for V4.1. Restored sessions warm the draft window with 128 ordinary tokens
before speculation resumes. Measure speed on your prompts: acceptance and SSD
cache size both matter. V4.1 DSpark can still be slower than ordinary decoding,
including in TP mode.
## GLM: built-in MTP
GLM's draft block is already in its main GGUF:
```sh
./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf --mtp
```
`--mtp-timing` also enables it and prints acceptance and timing counters.
The current GLM cycle commits up to two tokens. No external support file is
needed, and ordinary decode remains the default.
## Qwen3.8: built-in MTP
Both Qwen downloads include MTP and native BF16 n-grams in the main GGUF:
```sh
./download_model.sh qwen38-q4k
./ds4 --mtp
```
Ordinary decode uses the same file with `--mtp` omitted. For non-zero
temperature, add `--mtp-exact-sampling` to preserve the target sampling
distribution. See [Qwen setup](QWEN38_FLASH_NEXT.md) for Metal and CUDA.
For the Q2 model with Metal SSD streaming:
```sh
./ds4 --metal -m gguf/Qwen3.8-Flash-Next-Q2.gguf \
--ssd-streaming --ctx 4096 --prefill-chunk 128 \
--mtp-timing --temp 0 --nothink -n 100 \
-p "narrami la storia di roma"
```
`--mtp-timing` enables MTP and reports verification cycles and first-draft
acceptance. Measure generation throughput against the same command without
that option: speculative decoding can lose time when too few proposals are
accepted, especially when loading the predictor's experts from SSD.
Qwen's [official model configuration](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json)
contains one MTP layer with no dedicated token embeddings. The same layer
can be called repeatedly to propose multiple tokens; the number of MTP
layers is not the draft length. The
[official blog](https://www.alibabacloud.com/blog/qwen3-8-flash-next-a-new-architecture-towardsultimate-cost-efficiency_603501)
describes training this module over multiple steps and using QSA in it.
These weights are already included in the GGUF; no separate draft model is
needed.
With SSD streaming, prompt prefill prepares the predictor's K/V and indexer
cache from the trunk hidden states and the next prompt token by default. It skips the predictor's
attention output, MoE and vocabulary head, so this preparation does not
stream predictor experts from SSD. The final hidden row is retained until
the actual next token is known. Snapshot restore and speculative rollback
retain that row too. This costs 40 KiB per row for this model, at most
160 KiB including the three verifier snapshots. Qwen payload version 3
stores the initialized prefix and retained row. Version 2 checkpoints and
plain checkpoints restore their target state directly. When MTP is enabled
without a saved predictor tail, the engine replays the saved tokens once
before continuing. With resident weights, prefill skips historical predictor
projections and initializes those cache rows to zero. MTP and target
verification remain enabled. Preparing the full history can improve draft
acceptance for long generations. Use `DS4_QWEN4_MTP_PREFILL=on` to enable
it for resident weights too. [Qwen MTP settings](QWEN38_FLASH_NEXT.md)
The cycle drafts one token ahead by default and engages a **second, chained
draft** (one extra nextn-layer step conditioned on the predictor's own
stream, verified in a 3-row pass) while recent first-draft acceptance is
perfect, disengaging after repeated second-draft rejections.
`DS4_QWEN4_MTP_DEPTH=2` or `=3` fixes the depth;
`0` (default) is the adaptive policy.
## Sampling and reproducibility
At temperature zero, accepted drafts must match the target's greedy
continuation. At non-zero temperature, the default mode is opportunistic:
ordinary tokens use the requested sampling settings, but matching greedy
drafts are accepted directly. Sampling resumes when the proposed suffix does
not match. This is deliberately more deterministic than ordinary sampling.
Use `--mtp-exact-sampling` to preserve the ordinary target sampling
distribution. Greedy proposals are accepted with their target probability;
sampled proposals use the target-to-draft probability ratio. Rejections sample
from the remaining target probability mass.
When a verified block crosses a tool sampling-mode boundary (for example
entering tool-call syntax during server decoding), the server rewinds to the
block start and re-evaluates the boundary token so the next sample uses the
new mode. Under exact sampling that rewind restores a pre-verify snapshot of
the recurrent state instead of resetting the graph, so long retained
contexts are not replayed at every boundary.
Accepted tokens keep the state produced by the batched verifier. Floating-point
reduction order can differ from one-token decode, so long greedy continuations
need not be byte-identical. For DeepSeek comparisons against the ordinary
target-only path, use `--quality` or `--dspark-strict`; these disable the
speculative acceptance path. They do not promise identical output across
different hardware or execution configurations.
Session-batched serving uses ordinary target decoding instead of combining
DSpark/MTP with the session batch. See [serving](SERVER.md#multiple-sessions).
docs/SSD_STREAMING.md
+118
View File
@@ -0,0 +1,118 @@
# Models Larger Than RAM
[README](../README.md)
Use resident inference when the model and runtime fit: it is faster.
SSD streaming keeps a bounded cache of routed experts and reads missing
experts from the GGUF. It trades speed for capacity; it does not remove the
memory needed for other weights, activations, scratch, and the context.
Metal supports streaming for DeepSeek, GLM and [Qwen3.8](QWEN38_FLASH_NEXT.md).
CUDA has streaming paths too, and ROCm supports GLM 5.2/5.3 streaming.
Do not infer support for every model
and tensor layout from the existence of the flag.
## Start with the automatic budget
```sh
./ds4 -m ds4flash.gguf --ssd-streaming
```
The startup report shows the effective cache, resident layers where applicable,
and memory requirements. Prefer the automatic budget for a first run. Use a
local SSD and leave memory for the OS and other applications.
Examples:
```sh
# Flash Q2 on a smaller Mac.
./download_model.sh ds4f-q2
./ds4 --ssd-streaming --ctx 32768 --nothink
# GLM 5.3 Flash Q4 on a 128 GB Mac.
./download_model.sh glm53-q4
./ds4 --ssd-streaming --ctx 4096
# PRO Q2 on a 128 GB Mac: usable for inspection, but slow.
./download_model.sh pro-q2-imatrix
./ds4 --ssd-streaming --ctx 32768 --nothink
```
Generation is usually more sensitive to cache misses than prefill. A large
model that starts successfully can still be too slow for interactive work.
Use a short generation before committing to a long task.
## Adjusting the cache
To leave more room for context or other sessions:
```sh
./ds4 --ssd-streaming --ssd-streaming-cache-experts 32GB
```
A byte budget is a target, not a guaranteed allocation. DwarfStar reserves
routed-prefill headroom and fits the cache to the remaining model, graph,
context, and backend budget. The effective value may be smaller than requested.
Non-routed weights and KV state are additional to that expert-cache budget.
A plain number, such as `--ssd-streaming-cache-experts 4000`, requests dynamic
expert slots rather than a byte budget. It is also subject to memory limits.
By default, GLM spends the cache budget on selected experts across all layers.
`--ssd-streaming-full-layers N` reserves full routed prefix layers instead.
Metal also keeps the non-routed weights resident when it can; they are needed
by every token, and paging them back in delays generation after a prompt.
An oversized expert cache can displace those weights and slow decoding.
More cache helps only while the rest of the working set still fits.
More context and more server sessions consume more memory. Metal GLM budgeting
includes planned and already resident sessions; a request can still be refused
when its mandatory allocations cannot fit. Disabling the memory guard is not
a remedy for insufficient RAM.
Leave expert preloading enabled for normal use. `--ssd-streaming-cold` and
`--ssd-streaming-preload-experts N` are mainly useful for controlled measurements.
See [benchmarking](PERFORMANCE.md) and the [release QA guide](../QA_BEFORE_RELEASES.md).
## CUDA expert reads
DeepSeek V4.1 CUDA prefill reads selected experts in file order and combines
adjacent reads by default, instead of prefetching every expert in the next
layer. Single-token decode retains its original read order, which avoids a
slowdown with small expert caches. The weight formats, routing, and cache
budget are unchanged; no extra flags are needed.
For comparisons on other storage or workloads,
`DS4_CUDA_ENABLE_SSD_PREFETCH=1` restores whole-next-layer prefetch.
`DS4_CUDA_DISABLE_SSD_SORT_SELECTED_READS=1` restores the original selected-read
order. Set both to reproduce the previous policy.
`DS4_CUDA_DISABLE_SSD_PREFETCH=1` still overrides prefetch enablement.
For experiments, `DS4_CUDA_SSD_SORT_SELECTED_READS=1` also orders single-token
loads; the explicit sort-disable switch takes precedence.
Unset that old experimental sort flag to use the new prefill-only default.
These switches use presence semantics: unset them to return to the default;
setting a switch to `0` still enables it.
## M5 Max measurements
On a 128 GB M5 Max, with automatic cache sizing and no speculative decoding:
| Model | Initial prefill | Continued prefill | Generation after each | Runs |
| --- | ---: | ---: | ---: | --- |
| GLM 5.3 Flash Q4_K, 177.77 GiB | 121 t/s | 104 t/s | 11.9 / 14.9 t/s | Three-run median |
| DeepSeek Flash Vision Exp MXFP4, 145.26 GiB | 300 t/s | 263 t/s | 11.9 / 19.3 t/s | Single run |
September 6, 2026. GLM used a 2K prompt and a 1K append; DeepSeek used 8K
and a 4K append. Both generated 128 tokens at each frontier. DeepSeek's latest
run uses an 86.2 GiB expert cache and updated eviction priorities; it is not
directly comparable to the earlier 64-token measurement. Generation includes
the first-token wait. Cache reuse depends on the prompt, so these are
workload references, not a speed
guarantee for every model larger than RAM.
Full GLM 5.3 IQ2_XXS (196.58 GiB) also benefits from caching short tool-result
prompts. With an 8K context and a 61.35 GiB effective expert cache, a 16-token
append fell from 30.8 to 2.9 seconds; generation afterward rose from 4.09 to
5.11 t/s (three-run medians, 64 generated tokens). All compared logits and
generated text were identical. This is a short-append improvement, not an
equivalent gain in initial prefill or every generation workload.
docs/STRIX_HALO.md
+101
View File
@@ -0,0 +1,101 @@
# AMD Strix Halo
[README](../README.md) | [Getting started](../README.md#start-here)
The reference system is a 128 GB Strix Halo with Radeon 8060S (`gfx1151`),
such as the Framework Desktop. The ROCm build uses the standard binary names
and selects the ROCm backend by default.
## Prerequisites
For a container setup, see the maintained
[ROCm toolbox](https://github.com/kyuz0/strix-halo-ds4-toolbox/blob/main/toolboxes/Dockerfile.rocm-10.0).
It can also be managed with
[AI Toolbox Cockpit](https://github.com/kyuz0/ai-toolbox-cockpit).
For a native Ubuntu build you need HIP, hipBLAS, hipBLASLt, rocBLAS, rocWMMA,
and hipCUB development files. The Ubuntu 26.04 setup used these packages:
```sh
sudo apt-get update
sudo apt-get install -y hipcc rocminfo rocm-smi \
libamdhip64-dev libhipblas-dev libhipblaslt-dev librocblas-dev \
librocwmma-dev libhipcub-dev
sudo usermod -aG render,video "$USER"
```
Log out and back in after changing groups. `rocminfo` must report `gfx1151`
and be able to open `/dev/kfd` before DwarfStar can run.
Some packaged rocWMMA headers omit `rocwmma/internal/`. If compilation fails
there, install the complete headers matching your ROCm installation, or use
the container. Do not mix header versions as a general workaround.
## GPU-visible memory
Check the memory pool reported by `rocminfo`. Some 128 GB configurations expose
only about 62 GB to the GPU, which is insufficient for resident Flash Q2 plus
runtime buffers. Firmware and kernel GTT/TTM settings control this limit.
The native reference setup used these memory parameters:
```text
amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856
```
They are a system-specific starting point, not an allocation budget for
DwarfStar. Preserve existing boot options and consult your kernel's settings
before changing them. Keep RAM available for the OS even when the GPU can
address most of it. Do not disable the IOMMU merely to copy another host's
configuration; doing so changes device isolation.
## Build and run Flash
```sh
make strix-halo
./download_model.sh ds4f-q2
./ds4 --rocm
```
`make rocm` is an alias. Use the current 0731 Q2 download for a first run;
larger mixed and Q4 models have substantially higher memory requirements.
Flash's ROCm resident and pipeline paths should not be confused with the GLM
SSD-streaming path.
## DeepSeek V4.1 Flash
Use Q2 with SSD streaming on a 128 GB machine:
```sh
./download_model.sh ds41f-q2
./ds4 --rocm -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
--ssd-streaming --ctx 4096
```
The 341 GiB GGUF includes disk-only Engram tables. Keep it on a fast local
SSD and start with automatic cache sizing. The matching
[V4.1 DSpark](SPECULATIVE_DECODING.md#deepseek-v41-flash-dspark) support file
is optional. V4.1 vision and network TP are not supported on ROCm.
Large prefills are not yet tuned on this backend and can be slow.
## GLM 5.3 Flash
The reference Q2 setup uses SSD streaming to leave room for its graph and KV
state. Begin with automatic cache sizing and a small context:
```sh
./download_model.sh glm53-q2
./ds4 --rocm -m gguf/GLM-5.3-Flash-Q2.gguf \
--ssd-streaming --ctx 4096
```
GLM 5.2 also supports ROCm streaming. Full-model GLM 5.2 inference requires it;
distributed layer slices can be resident. See [SSD streaming](SSD_STREAMING.md)
before adjusting the cache budget.
Both GLM 5.3 Flash and DeepSeek Flash Vision Experimental support images on
ROCm. Add the matching encoder with `--vision FILE`, as described in
[models and vision](MODELS.md#vision).
For a model-free routed-kernel check, use `make test-mxfp4-rocm`.
Full-model validation is described in [testing](TESTING.md).
docs/TESTING.md
+66
View File
@@ -0,0 +1,66 @@
# Testing and Development
[README](../README.md)
Read [CONTRIBUTING.md](../CONTRIBUTING.md) before proposing an inference change.
The [release QA guide](../QA_BEFORE_RELEASES.md) defines the hardware matrix,
checkpoint-matched quality tests, long-context agent tasks, and speed checks.
It also records which checks were not completed. A smoke test is not full QA.
## Focused checks
Model-free checks include:
```sh
make ds4_test ds4_agent_test test-session-state
./ds4_test --server
./ds4_agent_test
```
On Metal, small GPU tensor tests are available without loading a full GGUF:
```sh
make tests/test_session_state_gpu tests/test_glm53_kda tests/test_mxfp4_metal
./tests/test_session_state_gpu
./tests/test_glm53_kda
./tests/test_mxfp4_metal
```
`make test` also includes model-backed tests. Select the right GGUF and ensure
that it fits before running it; do not accidentally load a large model on a
single device during multi-GPU QA. ROCm has `make test-rocm`.
Official-vector tests must use continuations from the same checkpoint as the
GGUF. Flash 0731 and Vision Experimental are not interchangeable fixtures.
See [test vectors](../tests/test-vectors/README.md) and
[quality scoring](../gguf-tools/quality-testing/README.md).
## Investigating output
```sh
./ds4 --dump-tokens -p "..."
./ds4 --dump-logprobs /tmp/out.json --logprobs-top-k 20 --temp 0 -p "..."
./ds4 --dump-logits /tmp/logits.json --nothink --prompt-file prompt.txt
./ds4-server --trace /tmp/ds4-trace.txt
```
Token dumps catch template differences without inference. Logits and
continuations help distinguish sampling changes from graph errors. Server
traces include cache and tool-parser decisions. Keep traces private when they
contain real conversations.
## Tools and source references
- [GGUF conversion and quantization](../gguf-tools/README.md)
- [Imatrix collection](../gguf-tools/imatrix/README.md) and
[calibration corpus](../gguf-tools/imatrix/dataset/README.md)
- [Steering vectors](../dir-steering/README.md)
- [Benchmark scripts and charts](../speed-bench/README.md)
- [Evaluation data and licenses](../EVAL_DATA.md)
- [Session payload implementation](../ds4.c) and
[cache header definitions](../ds4_kvstore.h)
- [Pipeline protocol](../ds4_distributed.c) and [TP protocol](../ds4_tp.c)
For changes to state handling, cover rewind/replay, save/load, images, and
multiple sessions as well as a fresh prompt. For distributed changes, exercise
both ranks and failures; a local command-parser test is not physical TP QA.
download_model.sh
+415 -6
View File
@@ -4,7 +4,15 @@ set -e
GLM_UNSLOTH_REPO="unsloth/GLM-5.2-GGUF"
GLM_ANTIREZ_REPO="antirez/GLM-5.2-GGUF"
GLM53_REPO="antirez/glm-5.3-flash-gguf"
GLM53_FULL_REPO="antirez/glm-5.3-gguf"
QWEN38_REPO="antirez/qwen3.8-flash-next-gguf"
QWEN38_MMPROJ_REPO="ggml-org/Qwen3.8-Flash-Next-GGUF"
REPO="antirez/deepseek-v4-gguf"
DS41_REPO="antirez/deepseek-v4.1-flash-gguf"
DS41_Q2_FILE="DeepSeek-V4.1-Flash-1017-IQ2_XXS-Q2_K-imatrix.gguf"
DS41_Q4_FILE="DeepSeek-V4.1-Flash-Q4.gguf"
DS41_MXFP4_BASE="DeepSeek-V4.1-Flash-1017-MXFP4-experts"
DS41_VISION_FILE="DeepSeek-V4.1-Flash-Vision.gguf"
DS4F_Q2_FILE="DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf"
DS4F_Q4_FILE="DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf"
DS4F_MXFP4_FILE="DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.gguf"
@@ -13,16 +21,25 @@ PRO_Q2_IMATRIX_FILE="DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-
PRO_Q4_LAYERS00_30_FILE="DeepSeek-V4-Pro-Q4K-Layers00-30.gguf"
PRO_Q4_LAYERS31_OUTPUT_FILE="DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf"
DS4F_DSPARK_FILE="DeepSeek-V4-Flash-DSpark-support-0731.gguf"
DS4F_VISION_Q2_FILE="DeepSeek-V4-Flash-Vision-Exp-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8.gguf"
DS4F_VISION_Q2_Q4_FILE="DeepSeek-V4-Flash-Vision-Exp-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8.gguf"
DS4F_VISION_MXFP4_FILE="DeepSeek-V4-Flash-Vision-Exp-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out.gguf"
DS4F_VISION_ENCODER_FILE="DeepSeek-V4-Flash-Vision-Encoder.gguf"
DS4F_VISION_DSPARK_FILE="DeepSeek-V4-Flash-Vision-Exp-DSpark-support.gguf"
GLM_UNSLOTH_Q4_REMOTE_BASE="UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL"
GLM_UNSLOTH_Q4_LOCAL_BASE="GLM-5.2-UD-Q4_K_XL"
GLM_UNSLOTH_Q4_FIRST_FILE="$GLM_UNSLOTH_Q4_LOCAL_BASE-00001-of-00011.gguf"
GLM_ANTIREZ_IQ2XXS_FILE="GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf"
GLM_ANTIREZ_Q2_FILE="GLM-5.2-UD-Q2_K_RoutedQ2K.gguf"
GLM_ANTIREZ_Q4_FILE="GLM-5.2-UD-Q4_K_RoutedQ4K.gguf"
GLM53_FULL_Q2_FILE="GLM-5.3-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf"
GLM53_Q2_FILE="GLM-5.3-Flash-Q2.gguf"
GLM53_Q4_FILE="GLM-5.3-Flash-Q4_K.gguf"
GLM53_FP8_FILE="GLM-5.3-Flash-FP8.gguf"
GLM53_VISION_FILE="GLM-5.3-Flash-Vision-Encoder.gguf"
QWEN38_Q4_FILE="Qwen3.8-Flash-Next-Q4.gguf"
QWEN38_Q2_FILE="Qwen3.8-Flash-Next-Q2.gguf"
QWEN38_VISION_FILE="mmproj-Qwen3.8-Flash-Next-Q8_0.gguf"
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
OUT_DIR=${DS4_GGUF_DIR:-"$ROOT/gguf"}
@@ -42,6 +59,15 @@ Usage:
./download_model.sh ds4f-q4 [--token TOKEN]
./download_model.sh ds4f-mxfp4 [--token TOKEN]
./download_model.sh ds4f-dspark [--token TOKEN]
./download_model.sh ds4f-vision-q2 [--token TOKEN]
./download_model.sh ds4f-vision-q2-q4 [--token TOKEN]
./download_model.sh ds4f-vision-mxfp4 [--token TOKEN]
./download_model.sh ds4f-vision-encoder [--token TOKEN]
./download_model.sh ds4f-vision-dspark [--token TOKEN]
./download_model.sh ds41f-q2 [--token TOKEN]
./download_model.sh ds41f-q4 [--token TOKEN]
./download_model.sh ds41f-mxfp4 [--token TOKEN]
./download_model.sh ds41f-vision [--token TOKEN]
./download_model.sh pro-q2-imatrix [--token TOKEN]
./download_model.sh pro-q4-layers00-30 [--token TOKEN]
./download_model.sh pro-q4-layers31-output [--token TOKEN]
@@ -50,10 +76,14 @@ Usage:
./download_model.sh glm-antirez-iq2xxs [--token TOKEN]
./download_model.sh glm-antirez-q2 [--token TOKEN]
./download_model.sh glm-antirez-q4 [--token TOKEN]
./download_model.sh glm53-full-q2 [--token TOKEN]
./download_model.sh glm53-q2 [--token TOKEN]
./download_model.sh glm53-q4 [--token TOKEN]
./download_model.sh glm53-fp8 [--token TOKEN]
./download_model.sh glm53-vision [--token TOKEN]
./download_model.sh qwen38-q2 [--token TOKEN]
./download_model.sh qwen38-q4k [--token TOKEN]
./download_model.sh qwen38-vision [--token TOKEN]
Targets:
@@ -80,6 +110,47 @@ Targets:
Optional DSpark speculative decoding support GGUF for Flash 0731, about
6 GB. Enable it with --dspark and --mtp-model when running ds4 or ds4-server.
ds4f-vision-q2
DeepSeek V4 Flash Vision Experimental with 2-bit routed experts, about
81 GiB, plus its vision encoder. Recommended for 96 and 128 GB Macs.
ds4f-vision-q2-q4
Mixed Vision Experimental quant with routed experts in layers 37-42 at
Q4_K and the others at 2 bits, about 91 GiB, plus its vision encoder.
ds4f-vision-mxfp4
Native MXFP4 Vision Experimental model, about 145 GiB, plus its vision
encoder. Intended for CUDA or two 128 GB Macs using tensor parallelism.
ds4f-vision-encoder
Standalone Vision Experimental encoder, about 0.9 GiB. Use this when
the matching language GGUF is already present.
ds4f-vision-dspark
Matching DSpark speculative decoding support for Vision Experimental,
about 5.6 GiB. It is not compatible with the 0731 language checkpoint.
ds41f-q2
DeepSeek V4.1 Flash calibrated Q2, about 341 GiB on disk. Main weights
occupy 152 GiB; Engram tables stay on disk. Metal only: use SSD streaming
on one 128 GB Mac, tensor parallelism on two, or a larger resident Mac.
ds41f-vision
Matching V4.1 Flash vision encoder, about 0.9 GiB. Add --vision FILE
to the language-model command. Does not update ./ds4flash.gguf.
ds41f-q4
DeepSeek V4.1 Flash calibrated Q4, about 483 GiB on disk. Main weights
occupy 294 GiB; Engram tables stay on disk. Metal only: use SSD streaming
on smaller Macs or full residency on a 512 GB Mac. Downloads two parts
and joins them automatically; allow another 37 GiB of free disk space.
ds41f-mxfp4
DeepSeek V4.1 Flash 1017 MXFP4 routed experts, about 467 GiB on disk.
Downloads and verifies all nine GGUF shards, then restores the original
page-aligned GGUF for DwarfStar. Retains the shards: allow about 935 GiB
of disk space for shards plus output. Downloads and merging can resume.
pro-q2-imatrix
DeepSeek V4 PRO 0813 q2 imatrix quant, as a single GGUF file. About
430 GB on disk; intended for 512 GB RAM machines.
@@ -113,6 +184,10 @@ Targets:
GLM 5.2 antirez routed Q4_K GGUF from antirez/GLM-5.2-GGUF.
About 434 GB on disk.
glm53-full-q2
Full GLM 5.3 routed IQ2_XXS/Q2_K GGUF, about 197 GiB on disk.
Intended for 256 GB machines or SSD streaming on smaller systems.
glm53-q2
GLM 5.3 Flash imatrix Q2 GGUF, about 90 GiB on disk. Intended for
resident inference on 128 GB Macs.
@@ -130,6 +205,25 @@ Targets:
GLM 5.3 Flash vision encoder, about 1.1 GB on disk. Load it separately
with --vision; this target does not update ./ds4flash.gguf.
qwen38-q2 (alias: qwen38-iq2)
Qwen3.8-Flash-Next Q2: one 137.10 GiB GGUF. Main/MTP weights
occupy 41.73 GiB; original BF16 n-grams stay on disk.
Imatrix IQ2_XXS gate/up and Q2_K down experts (640 logical inputs,
padded to 768 on disk), with higher-precision dense/control tensors.
Smaller option for 64 GB Macs: start with --ctx 8192 and
--prefill-chunk 1024. Context buffers need additional RAM.
Keep the file on a local SSD. Add --mtp for speculation.
qwen38-q4k
Qwen3.8-Flash-Next Q4: one 165.11 GiB GGUF. Main/MTP weights
occupy 69.74 GiB; original BF16 n-grams stay on disk.
Imatrix Q4_K gate/up and MXFP4 down routed experts. Fits 128 GB
Macs with room for context buffers. Add --mtp for speculation.
qwen38-vision
Qwen3.8-Flash-Next vision encoder, about 0.6 GB on disk. Load it
with --vision; this target does not update ./ds4flash.gguf.
Options:
--token TOKEN Hugging Face token. Otherwise HF_TOKEN or the local HF token
cache is used if present.
@@ -145,10 +239,13 @@ Then the default commands work:
./ds4 -p "Hello"
./ds4-server --ctx 100000
After downloading DSpark support, enable it explicitly in greedy mode:
./ds4 --dspark --mtp-model <download directory>/$DS4F_DSPARK_FILE --temp 0
Qwen3.8 includes its n-grams and MTP; add --mtp to enable speculation:
./ds4 --mtp
PRO and GLM files are downloaded with the official Hugging Face downloader
After downloading DSpark support, enable it explicitly:
./ds4 --dspark --mtp-model <download directory>/$DS4F_DSPARK_FILE
PRO, V4.1, GLM and Qwen files use the official Hugging Face downloader
because they are too large, sharded, or nested for the curl path used by the
smaller DeepSeek Flash GGUF files.
EOF
@@ -172,6 +269,55 @@ case "$MODEL" in
ds4f-q4) MODEL_FILE=$DS4F_Q4_FILE ;;
ds4f-mxfp4) MODEL_FILE=$DS4F_MXFP4_FILE; FORCE_HF_DOWNLOAD=1 ;;
ds4f-dspark) MODEL_FILE=$DS4F_DSPARK_FILE; LINK_MODEL=0 ;;
ds4f-vision-q2)
MODEL_FILE=$DS4F_VISION_Q2_FILE
MODEL_FILES="$MODEL_FILE $DS4F_VISION_ENCODER_FILE"
FORCE_HF_DOWNLOAD=1
;;
ds4f-vision-q2-q4)
MODEL_FILE=$DS4F_VISION_Q2_Q4_FILE
MODEL_FILES="$MODEL_FILE $DS4F_VISION_ENCODER_FILE"
FORCE_HF_DOWNLOAD=1
;;
ds4f-vision-mxfp4)
MODEL_FILE=$DS4F_VISION_MXFP4_FILE
MODEL_FILES="$MODEL_FILE $DS4F_VISION_ENCODER_FILE"
FORCE_HF_DOWNLOAD=1
;;
ds4f-vision-encoder)
MODEL_FILE=$DS4F_VISION_ENCODER_FILE
FORCE_HF_DOWNLOAD=1
LINK_MODEL=0
;;
ds4f-vision-dspark)
MODEL_FILE=$DS4F_VISION_DSPARK_FILE
FORCE_HF_DOWNLOAD=1
LINK_MODEL=0
;;
ds41f-q2)
REPO=$DS41_REPO
MODEL_FILE=$DS41_Q2_FILE
FORCE_HF_DOWNLOAD=1
;;
ds41f-q4)
REPO=$DS41_REPO
MODEL_FILE=$DS41_Q4_FILE
FORCE_HF_DOWNLOAD=1
;;
ds41f-mxfp4)
REPO=$DS41_REPO
MODEL_FILE="$DS41_MXFP4_BASE.gguf"
for part in 00001 00002 00003 00004 00005 00006 00007 00008 00009; do
MODEL_FILES="$MODEL_FILES $DS41_MXFP4_BASE-${part}-of-00009.gguf"
done
FORCE_HF_DOWNLOAD=1
;;
ds41f-vision)
REPO=$DS41_REPO
MODEL_FILE=$DS41_VISION_FILE
FORCE_HF_DOWNLOAD=1
LINK_MODEL=0
;;
pro-q2-imatrix) MODEL_FILE=$PRO_Q2_IMATRIX_FILE ;;
pro-q4-layers00-30) MODEL_FILE=$PRO_Q4_LAYERS00_30_FILE; LINK_MODEL=0 ;;
pro-q4-layers31-output) MODEL_FILE=$PRO_Q4_LAYERS31_OUTPUT_FILE; LINK_MODEL=0 ;;
@@ -204,6 +350,11 @@ case "$MODEL" in
MODEL_FILE=$GLM_ANTIREZ_Q4_FILE
FORCE_HF_DOWNLOAD=1
;;
glm53-full-q2)
REPO=$GLM53_FULL_REPO
MODEL_FILE=$GLM53_FULL_Q2_FILE
FORCE_HF_DOWNLOAD=1
;;
glm53-q2)
REPO=$GLM53_REPO
MODEL_FILE=$GLM53_Q2_FILE
@@ -226,6 +377,22 @@ case "$MODEL" in
FORCE_HF_DOWNLOAD=1
LINK_MODEL=0
;;
qwen38-q2|qwen38-iq2)
REPO=$QWEN38_REPO
MODEL_FILE=$QWEN38_Q2_FILE
FORCE_HF_DOWNLOAD=1
;;
qwen38-q4k)
REPO=$QWEN38_REPO
MODEL_FILE=$QWEN38_Q4_FILE
FORCE_HF_DOWNLOAD=1
;;
qwen38-vision)
REPO=$QWEN38_MMPROJ_REPO
MODEL_FILE=$QWEN38_VISION_FILE
FORCE_HF_DOWNLOAD=1
LINK_MODEL=0
;;
-h|--help|help)
usage
exit 0
@@ -296,6 +463,99 @@ local_download_name() {
fi
}
artifact_identity() {
case "$1" in
"$QWEN38_Q2_FILE")
expected_bytes=147207127040
expected_sha=b1b93fa69aca5f187b0fb813aca8f3ec1beb5cf8cf0bd38cf041b93e0b6ccac9
;;
"$QWEN38_Q4_FILE")
expected_bytes=177280286720
expected_sha=680944460a8cbe93ba8b6d7b6107213ffb7e22320bd913000e563ca0a0f25a8a
;;
"$DS41_Q2_FILE")
expected_bytes=365713686528
expected_sha=5296e85ff64b42ecb62c8d0c0c54e1caa07e3df8e93112f2365ebbaed62ce93a
;;
"$DS41_Q4_FILE")
expected_bytes=518596067328
expected_sha=a5e2e2c3ada4b2e98d9f9e4b50f6d9c2a12c2c96f5da165c07e13aff9264984e
;;
"$DS41_Q4_FILE.part1")
expected_bytes=480000000000
expected_sha=6442b1f9224079662c02003c0ef9ef6be6e2aff509510f681dab9e6cc41df246
;;
"$DS41_Q4_FILE.part2")
expected_bytes=38596067328
expected_sha=7c3e10646c918eeaffbc39305a75ec96117450262c61454ff194cef00d7617f0
;;
"$DS41_MXFP4_BASE.gguf")
expected_bytes=501609136128
expected_sha=79608a69cd5bb919d8140c37105edc3a33419692d4e6caf4e68c901e4b359141
;;
"$DS41_MXFP4_BASE-00001-of-00009.gguf")
expected_bytes=49353784544
expected_sha=636918b09de6253bf41af29f06aac3ceeda220e46379d8a2058ef92c4e433ed8
;;
"$DS41_MXFP4_BASE-00002-of-00009.gguf")
expected_bytes=49412635712
expected_sha=b26d3fc1e6ae1be6c52665542f3c1fee6a33791478f26e6cd90b5a75940a9f96
;;
"$DS41_MXFP4_BASE-00003-of-00009.gguf")
expected_bytes=49582960352
expected_sha=20c7fe7adb0578336d8a5e5f9eed521aeb88e8c119ba46e0916cdaab28931c80
;;
"$DS41_MXFP4_BASE-00004-of-00009.gguf")
expected_bytes=49418206432
expected_sha=67592ba15870c4292a615666adcc2736b53cc345b2cccb1c36710a0a6caf15b4
;;
"$DS41_MXFP4_BASE-00005-of-00009.gguf")
expected_bytes=49412829632
expected_sha=ecd04e0aa71bfab8cfe79ea7fef7f35b6be0b6772455139e2081d6ce1dcc18ed
;;
"$DS41_MXFP4_BASE-00006-of-00009.gguf")
expected_bytes=49257603776
expected_sha=88214e0cc80fbae578b80c114527e6ce725199684e0b6849475c7d00b3d0f54c
;;
"$DS41_MXFP4_BASE-00007-of-00009.gguf")
expected_bytes=2406482112
expected_sha=38939faf3e4258f3ae333f0fa1954ce89515b75ceee90d37bf4ebe5fafe285e6
;;
"$DS41_MXFP4_BASE-00008-of-00009.gguf")
expected_bytes=101377628544
expected_sha=a9d1634da2ac57459a64b10b69ddedfadda7dcde0e71ca6e55bea7c293c4a9f0
;;
"$DS41_MXFP4_BASE-00009-of-00009.gguf")
expected_bytes=101380404256
expected_sha=b9832b355288c005da6ef444d02424464dd27f6ebdefeadbec2244264eb65285
;;
"$DS41_VISION_FILE")
expected_bytes=970555552
expected_sha=cc283f032b3e8b8d78aeb5fccaa14e97b859b0c53aae3cd6bffa690ddf0e9e15
;;
*) return 1 ;;
esac
}
verify_download() {
artifact_identity "$1" || return 0
if [ "$(wc -c < "$2")" -ne "$expected_bytes" ]; then
echo "Incorrect file size: $2. Move the incomplete file aside and retry." >&2
exit 1
fi
echo "Verifying SHA-256: $2"
if command -v sha256sum >/dev/null 2>&1; then
actual_sha=$(sha256sum < "$2")
else
actual_sha=$(shasum -a 256 < "$2")
fi
actual_sha=${actual_sha%% *}
if [ "$actual_sha" != "$expected_sha" ]; then
echo "Checksum mismatch: $2. The file was not accepted." >&2
exit 1
fi
}
download_one_hf() {
file=$1
local_file=$(local_download_name "$file")
@@ -306,6 +566,7 @@ download_one_hf() {
mkdir -p "$(dirname "$out")"
if [ -s "$out" ]; then
verify_download "$file" "$out"
echo "Already downloaded: $out"
return
fi
@@ -345,6 +606,7 @@ download_one_hf() {
echo "Hugging Face download finished but expected file is missing: $out" >&2
exit 1
fi
verify_download "$file" "$out"
}
download_one() {
@@ -386,7 +648,138 @@ download_one() {
mv "$part" "$out"
}
if [ -n "$MODEL_FILES" ]; then
download_ds41_q4() {
q4_out="$OUT_DIR/$DS41_Q4_FILE"
if [ -e "$q4_out" ]; then
verify_download "$DS41_Q4_FILE" "$q4_out"
echo "Already downloaded: $q4_out"
return
fi
if ! command -v python3 >/dev/null 2>&1; then
echo "Joining the Q4 download requires Python 3." >&2
exit 1
fi
if [ ! -e "$q4_out.assembling" ]; then
download_one_hf "$DS41_Q4_FILE.part1"
fi
download_one_hf "$DS41_Q4_FILE.part2"
artifact_identity "$DS41_Q4_FILE"
q4_bytes=$expected_bytes
q4_sha=$expected_sha
artifact_identity "$DS41_Q4_FILE.part1"
python3 - "$q4_out" "$expected_bytes" "$q4_bytes" "$q4_sha" <<'PY'
import fcntl
import hashlib
import os
from pathlib import Path
import shutil
import sys
out = Path(sys.argv[1])
boundary, total = map(int, sys.argv[2:4])
expected = sys.argv[4]
first, second, pending = (Path(str(out) + suffix)
for suffix in (".part1", ".part2", ".assembling"))
def verify(path):
if path.stat().st_size != total:
sys.exit("Incorrect file size: " + str(path))
print("Verifying SHA-256: " + str(path), flush=True)
digest = hashlib.sha256()
with path.open("rb") as fp:
for block in iter(lambda: fp.read(16 << 20), b""):
digest.update(block)
if digest.hexdigest() != expected:
sys.exit("Checksum mismatch: " + str(path) +
". Move this file aside before retrying; it was not accepted.")
# Keep this lock file: unlinking it could allow two different locks for the
# same download. The first part becomes the output without a second full copy.
with Path(str(pending) + ".lock").open("a") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
if out.exists():
verify(out)
sys.exit(0)
if not pending.exists():
if first.stat().st_size != boundary:
sys.exit("Incorrect file size: " + str(first))
first.rename(pending)
size = pending.stat().st_size
if not boundary <= size <= total:
sys.exit("Invalid partial assembly: " + str(pending) +
". Move it aside before retrying.")
if second.stat().st_size != total - boundary:
sys.exit("Incorrect file size: " + str(second))
# An interrupted append restarts only the smaller tail, never the prefix.
with pending.open("r+b") as dst:
dst.truncate(boundary)
dst.seek(boundary)
if shutil.disk_usage(out.parent).free < total - boundary + (1 << 30):
sys.exit("Not enough disk space to join Q4; keep the parts and retry.")
print("Joining Q4 download parts", flush=True)
with second.open("rb") as src:
shutil.copyfileobj(src, dst, 16 << 20)
dst.flush()
os.fsync(dst.fileno())
verify(pending)
pending.rename(out)
second.unlink()
PY
}
download_ds41_mxfp4() {
mx_out="$OUT_DIR/$MODEL_FILE"
if [ -e "$mx_out" ]; then
verify_download "$MODEL_FILE" "$mx_out"
echo "Already downloaded: $mx_out"
return
fi
if ! command -v python3 >/dev/null 2>&1; then
echo "Merging the MXFP4 shards requires Python 3." >&2
exit 1
fi
if [ ! -f "$ROOT/gguf-tools/merge_deepseek41_shards.py" ]; then
echo "Missing gguf-tools/merge_deepseek41_shards.py; use a complete checkout." >&2
exit 1
fi
mkdir -p "$OUT_DIR"
artifact_identity "$MODEL_FILE"
mx_bytes=$expected_bytes
mx_sha=$expected_sha
set -- "$mx_out.assembling" "$mx_bytes"
for shard in $MODEL_FILES; do
artifact_identity "$shard"
set -- "$@" "$OUT_DIR/$shard" "$expected_bytes"
done
# Check peak storage before downloading hundreds of GB. Retain the input
# shards and account for a resumable output; HF partial caches are extra.
python3 - "$OUT_DIR" "$@" <<'PY'
from pathlib import Path
import shutil
import sys
needed = 1 << 30
for name, size in zip(sys.argv[2::2], sys.argv[3::2]):
path = Path(name)
needed += max(0, int(size) - (path.stat().st_size if path.exists() else 0))
if shutil.disk_usage(sys.argv[1]).free < needed:
sys.exit("Not enough disk space for MXFP4 shards and merged GGUF: "
f"need {needed / (1 << 30):.1f} GiB free. Existing files were kept.")
PY
set --
for shard in $MODEL_FILES; do
download_one_hf "$shard"
set -- "$@" "$OUT_DIR/$shard"
done
python3 "$ROOT/gguf-tools/merge_deepseek41_shards.py" \
--output "$mx_out" --expected-bytes "$mx_bytes" --sha256 "$mx_sha" "$@"
}
if [ "$MODEL" = "ds41f-mxfp4" ]; then
download_ds41_mxfp4
elif [ "$MODEL" = "ds41f-q4" ]; then
download_ds41_q4
elif [ -n "$MODEL_FILES" ]; then
for file in $MODEL_FILES; do
download_one "$file"
done
@@ -396,8 +789,12 @@ fi
if [ "$MODEL" = "ds4f-dspark" ]; then
echo
echo "DSpark support downloaded. Enable it explicitly in greedy mode:"
echo " ./ds4 --dspark -m ./ds4flash.gguf --mtp-model $OUT_DIR/$DS4F_DSPARK_FILE --temp 0"
echo "DSpark support downloaded. Enable it explicitly:"
echo " ./ds4 --dspark -m ./ds4flash.gguf --mtp-model $OUT_DIR/$DS4F_DSPARK_FILE"
elif [ "$MODEL" = "ds4f-vision-dspark" ]; then
echo
echo "Vision Experimental DSpark support downloaded. Use it only with the matching checkpoint:"
echo " ./ds4 --dspark -m ./ds4flash.gguf --mtp-model $OUT_DIR/$DS4F_VISION_DSPARK_FILE"
elif [ "$MODEL" = "pro-q4-layers00-30" ] || [ "$MODEL" = "pro-q4-layers31-output" ] || [ "$MODEL" = "pro-q4-split" ]; then
echo
echo "Downloaded PRO Q4 distributed split file(s). Use them with --layers,"
@@ -410,3 +807,15 @@ fi
echo
echo "Done."
if [ "$MODEL" = qwen38-q2 ] || [ "$MODEL" = qwen38-iq2 ]; then
echo "Run with an 8K starting context:"
echo " ./ds4 --ctx 8192 --prefill-chunk 1024"
echo "Add --mtp to enable speculation."
elif [ "$MODEL" = qwen38-q4k ]; then
echo "Run ./ds4. Add --mtp to enable speculation."
fi
if [ "$MODEL" = qwen38-vision ]; then
echo
echo "Qwen3.8 vision encoder downloaded. Pass it with --vision, for example:"
printf ' ./ds4 --vision "%s/%s"\n' "$OUT_DIR" "$QWEN38_VISION_FILE"
fi
ds4.c
+18423 -1401
View File
File diff suppressed because it is too large. Load diff
ds4.h
+102 -6
View File
@@ -26,7 +26,11 @@ typedef enum {
DS4_THINK_NONE,
DS4_THINK_HIGH,
DS4_THINK_MAX,
DS4_THINK_LOW, /* Qwen3.8 reasoning_effort low/medium; other models render them as HIGH */
DS4_THINK_MEDIUM,
} ds4_think_mode;
/* Explicit numeric effort lives outside the stable named-mode values. */
#define DS4_THINK_LEVEL_BASE 1000
typedef enum {
DS4_LOG_DEFAULT,
@@ -177,6 +181,9 @@ typedef struct {
typedef struct {
float *data;
uint32_t token_count;
uint32_t layout;
uint32_t grid_width;
uint32_t grid_height;
uint32_t width;
uint32_t height;
uint32_t content_width;
@@ -300,8 +307,13 @@ bool ds4_engine_glm_layer_payload_bytes(ds4_engine *e,
int ds4_engine_model_id(ds4_engine *e);
bool ds4_engine_is_glm_dsa(ds4_engine *e);
bool ds4_engine_is_glm53(ds4_engine *e);
bool ds4_engine_is_qwen4(ds4_engine *e);
/* Qwen3.8 reasoning-effort system instruction for a think mode (NULL when none) */
const char *ds4_qwen4_reasoning_effort_text(ds4_think_mode mode);
const char *ds4_backend_name(ds4_backend backend);
bool ds4_think_mode_enabled(ds4_think_mode mode);
int ds4_think_mode_level(ds4_think_mode mode);
bool ds4_think_mode_parse_level(const char *text, ds4_think_mode *out);
const char *ds4_think_mode_name(ds4_think_mode mode);
const char *ds4_think_max_prefix(void);
const char *ds4_glm_reasoning_effort_text(ds4_think_mode mode);
@@ -333,16 +345,20 @@ int ds4_engine_collect_imatrix(ds4_engine *e,
const char *output_path,
int ctx_size,
int max_prompts,
int max_tokens);
int max_tokens,
int min_expert_samples);
void ds4_engine_dump_tokens(ds4_engine *e, const ds4_tokens *tokens);
int ds4_dump_text_tokenization(const char *model_path, const char *text, FILE *fp);
int ds4_dump_chat_tokenization(const char *model_path,
const char *system,
const char *prompt,
ds4_think_mode think_mode,
int ctx_size,
FILE *fp);
int ds4_engine_head_test(ds4_engine *e, const ds4_tokens *prompt);
bool ds4_engine_is_glm_dsa(ds4_engine *e);
bool ds4_engine_is_deepseek41(ds4_engine *e);
const char *ds4_deepseek41_reasoning_effort_text(ds4_think_mode mode);
int ds4_engine_first_token_test(ds4_engine *e, const ds4_tokens *prompt);
int ds4_engine_metal_graph_test(ds4_engine *e, const ds4_tokens *prompt);
int ds4_engine_metal_graph_full_test(ds4_engine *e, const ds4_tokens *prompt);
@@ -363,6 +379,7 @@ void ds4_encode_chat_prompt(
ds4_think_mode think_mode,
ds4_tokens *out);
void ds4_chat_append_max_effort_prefix(ds4_engine *e, ds4_tokens *tokens);
void ds4_chat_append_think_prefix(ds4_engine *e, ds4_tokens *tokens, ds4_think_mode mode);
void ds4_chat_append_message(ds4_engine *e, ds4_tokens *tokens, const char *role, const char *content);
void ds4_chat_append_assistant_prefix(ds4_engine *e, ds4_tokens *tokens, ds4_think_mode think_mode);
@@ -382,6 +399,9 @@ int ds4_token_assistant(ds4_engine *e);
* with the caller. */
struct ds4_tp;
int ds4_engine_tp_bind(ds4_engine *e, struct ds4_tp *tp, char *err, size_t errlen);
/* Release gate resources before freeing a bound transport. Sessions must
* already be closed. Also called by ds4_engine_close(). */
void ds4_engine_tp_unbind(ds4_engine *e);
int ds4_session_create(ds4_session **out, ds4_engine *e, int ctx_size);
void ds4_session_free(ds4_session *s);
@@ -396,9 +416,10 @@ void ds4_session_set_progress(ds4_session *s, ds4_session_progress_fn fn, void *
/* UI-only progress. It may report fine-grained progress inside a prefill chunk;
* callers must not treat it as a durable KV checkpoint boundary. */
void ds4_session_set_display_progress(ds4_session *s, ds4_session_progress_fn fn, void *ud);
/* Optional cooperative cancellation. ds4_session_sync() checks it only at
* safe boundaries where the live checkpoint is either unchanged or represents a
* valid token prefix, and returns DS4_SESSION_SYNC_INTERRUPTED when it stops. */
/* Cooperative cancellation for ds4_session_sync(), which drains pending work
* before returning DS4_SESSION_SYNC_INTERRUPTED. A complete prefix may remain
* usable, but a partial layer-major pass is invalidated and the next sync
* rebuilds it. Do not assume an interrupted session can be saved or decoded. */
void ds4_session_set_cancel(ds4_session *s, ds4_session_cancel_fn fn, void *ud);
void ds4_session_report_progress(ds4_session *s, const char *event, int current, int total);
/* Distributed coordinator sessions return 1 when the full layer route is
@@ -424,17 +445,57 @@ int ds4_session_sync_multimodal(ds4_session *s,
size_t image_count,
char *err,
size_t errlen);
/* A reusable image prefix has unchanged spans/fingerprints for all historical
* images, with any new images starting at or after the live token frontier.
* The caller must also check the token prefix. Invalid checkpoints never match. */
bool ds4_session_vision_prefix_matches(const ds4_session *s,
const ds4_vision_span *images,
size_t image_count);
/* Like the prefix check, but also require exactly the same image count. */
bool ds4_session_vision_state_matches(const ds4_session *s,
const ds4_vision_span *images,
size_t image_count);
/* Fingerprint-only image prefix: every historical image matches by row count
* and fingerprint, ignoring token positions (which rebase repairs) and
* allowing appended request images. Lets the server routing probe admit
* image-append continuations while still rejecting pixel mismatches. */
bool ds4_session_vision_fingerprint_prefix_matches(const ds4_session *s,
const ds4_vision_span *images,
size_t image_count);
/* Restore image positions from an independently authenticated live continuation
* (for example, matching tool-call IDs). Checks every fingerprint and row count;
* on failure, leaves spans unchanged. This does not verify the text history. */
bool ds4_session_rebase_vision_state(const ds4_session *s,
ds4_vision_span *images, size_t image_count);
/* True while a session contains, or is actively syncing, image-conditioned
* state. Such state must not be written to the text-keyed disk KV cache. */
bool ds4_session_has_vision_state(const ds4_session *s);
bool ds4_session_rewrite_requires_rebuild(int live_len, int canonical_len, int common);
ds4_session_rewrite_result ds4_session_rewrite_from_common(
ds4_session *s, const ds4_tokens *prompt, int common,
char *err, size_t errlen);
int ds4_session_common_prefix(ds4_session *s, const ds4_tokens *prompt);
bool ds4_session_checkpoint_valid(const ds4_session *s);
/* Test helpers (ds4-test): allocate a session shell holding only the given
* checkpoint tokens, for server-side routing/probe unit tests. Not usable
* for inference; free with ds4_session_free_test_checkpoint(). */
ds4_session *ds4_session_new_test_checkpoint(const int *tokens, int n);
void ds4_session_free_test_checkpoint(ds4_session *s);
/* Attach synthetic image identities to a test checkpoint (copies
* token_start/row-count/fingerprint per span). Not usable for inference. */
void ds4_session_set_test_images(ds4_session *s,
const ds4_vision_span *images, size_t n);
int ds4_session_argmax(ds4_session *s);
int ds4_session_argmax_excluding(ds4_session *s, int excluded_id);
int ds4_session_argmax_ignoring_eos(ds4_session *s,
ds4_think_mode think_mode);
int ds4_sample_logits(const float *logits, int n_vocab, float temperature,
int top_k, float top_p, float min_p, uint64_t *rng);
int ds4_session_sample(ds4_session *s, float temperature, int top_k, float top_p, float min_p, uint64_t *rng);
#ifdef DS4_TEST_HOOKS
/* Teacher-force a V4.1 verification block, returning logits after each token. */
int ds4_test_ds41_verify_tokens(ds4_session *s, const int *tokens, int count,
float *logits, char *err, size_t errlen);
int ds4_test_sample_logits(const float *logits, uint32_t n_vocab,
float temperature, int top_k,
float top_p, float min_p, uint64_t *rng,
@@ -464,6 +525,7 @@ int ds4_test_speculative_delta_sample(const float *target_logits,
int ds4_test_argmax_excluding_logits(const float *logits, uint32_t n_vocab,
int excluded_id);
uint64_t ds4_test_mixed_native_count(void);
uint64_t ds4_test_ds41_batch_count(void);
#endif
int ds4_session_top_logprobs(ds4_session *s, ds4_token_score *out, int k);
int ds4_session_token_logprob(ds4_session *s, int token, ds4_token_score *out);
@@ -484,6 +546,16 @@ typedef struct {
* sequential fallback. */
int ds4_sessions_eval_batch(ds4_decode_item *items, int count,
char *err, size_t errlen);
/* One speculative cycle for a batch of sessions (greedy acceptance, Qwen3.8
* with --mtp): each item feeds its token; a pending draft rides along as a
* second row and is committed when it is the target's argmax. accepted[i]
* lists the tokens committed for item i (the fed token, then the draft) and
* n_accepted[i] how many; the session's logits then follow its last
* committed token. Engines without native batching run one cycle per
* session in turn. */
int ds4_sessions_eval_batch_speculative_argmax(ds4_decode_item *items, int count,
int (*accepted)[2], int *n_accepted,
char *err, size_t errlen);
/* Advance one resumed prefill suffix and an independent decode batch as one
* scheduling step. Unsupported combinations use the ordinary serialized
* session operations. */
@@ -495,6 +567,10 @@ int ds4_session_eval_speculative_argmax(ds4_session *s, int first_token,
int max_tokens, int eos_token,
int *accepted, int accepted_cap,
char *err, size_t errlen);
int ds4_session_eval_speculative_argmax_ignoring_eos(
ds4_session *s, int first_token, int max_tokens, int eos_token,
ds4_think_mode think_mode,
int *accepted, int accepted_cap, char *err, size_t errlen);
/* Evaluate one already-sampled target token and speculatively extend it.
* Positive-temperature DSpark normally commits greedily verified draft
* tokens; dspark_exact_sampling selects exact stochastic p/q acceptance for
@@ -507,10 +583,29 @@ int ds4_session_eval_speculative(ds4_session *s, int first_token,
char *err, size_t errlen);
/* TP worker side of a mirrored speculative-verify block: run its half of the
* batch verify for KV side effects, then obey the leader's commit frame
* (keep, or roll back and replay). Only called from ds4_tp_worker_run. */
* (keep, or roll back and replay). flags are the VERIFY frame's
* DS4_TP_VERIFY_* bits. Only called from ds4_tp_worker_run. */
int ds4_session_tp_spec_cycle(ds4_session *s, const int *drafts, int draft_n,
char *err, size_t errlen);
uint32_t flags, char *err, size_t errlen);
int ds4_session_glm_tp_spec_cycle(ds4_session *s, int token, int limit,
char *err, size_t errlen);
/* TP worker side of a lockstep DSpark DRAFT/MAINTAIN frame (mode is a
* DS4_TP_DSPARK_* value). Only called from ds4_tp_worker_run. */
int ds4_session_tp_dspark_draft(ds4_session *s, int token, int pos,
uint32_t mode, char *err, size_t errlen);
/* Internal TP restore: no collectives and no logits exchange. Return 0 after
* restoring a recent verifier prefix, 1 if unavailable, -1 on copy failure.
* Logits are valid only at the block start; otherwise replay before sampling. */
int ds4_session_restore_speculative_prefix(ds4_session *s, int pos);
void ds4_session_invalidate(ds4_session *s);
/* Try a recent speculative rewind, including valid next-token logits. False
* means the caller must use ordinary rewind/sync before continuing; a failed
* restore may have invalidated the checkpoint. */
bool ds4_session_rewind_speculative(ds4_session *s, int pos);
/* Keep the token prefix, restoring recurrent state where possible. This may
* copy GPU state and replay a bounded speculative suffix on both TP ranks.
* Otherwise the checkpoint becomes invalid: sync the retained prefix before eval.
* Callers retaining images must use sync_multimodal for that rebuild. */
void ds4_session_rewind(ds4_session *s, int pos);
int ds4_session_pos(ds4_session *s);
int ds4_session_ctx(ds4_session *s);
@@ -519,6 +614,7 @@ int ds4_engine_routed_quant_bits(ds4_engine *e);
bool ds4_engine_has_output_head(ds4_engine *e);
bool ds4_engine_has_mtp(ds4_engine *e);
int ds4_engine_mtp_draft_tokens(ds4_engine *e);
bool ds4_engine_mtp_exact_sampling(ds4_engine *e);
const ds4_tokens *ds4_session_tokens(ds4_session *s);
/* Low-level graph slice entry points used by distributed inference. The
ds4_agent.c
+2209 -591
View File
File diff suppressed because it is too large. Load diff
ds4_bench.c
+131 -13
View File
@@ -36,6 +36,7 @@ extern int cudaProfilerStop(void) __attribute__((weak));
typedef struct {
const char *model_path;
const char *mtp_path;
const char *prompt_path;
const char *chat_prompt_path;
const char *system;
@@ -69,6 +70,9 @@ typedef struct {
bool cuda_tensor_parallel;
bool show_output;
bool teacher_forced_decode;
bool dspark;
bool dspark_confidence_threshold_set;
float dspark_confidence_threshold;
} bench_config;
static double bench_now_sec(void) {
@@ -264,6 +268,19 @@ static bench_config parse_options(int argc, char **argv) {
if (!strcmp(arg, "-m") || !strcmp(arg, "--model")) {
c.model_path = need_arg(&i, argc, argv, arg);
} else if (!strcmp(arg, "--mtp-model")) {
c.mtp_path = need_arg(&i, argc, argv, arg);
} else if (!strcmp(arg, "--dspark")) {
c.dspark = true;
} else if (!strcmp(arg, "--dspark-confidence")) {
const double v = parse_double_arg(need_arg(&i, argc, argv, arg), arg);
if (v < 0.0 || v > 1.0) {
fprintf(stderr, "ds4-bench: --dspark-confidence must be between 0 and 1\n");
exit(2);
}
c.dspark = true;
c.dspark_confidence_threshold = (float)v;
c.dspark_confidence_threshold_set = true;
} else if (!strcmp(arg, "--prompt-file")) {
c.prompt_path = need_arg(&i, argc, argv, arg);
} else if (!strcmp(arg, "--chat-prompt-file")) {
@@ -369,6 +386,16 @@ static bench_config parse_options(int argc, char **argv) {
fprintf(stderr, "ds4-bench: specify exactly one of --prompt-file or --chat-prompt-file\n");
exit(2);
}
if (c.dspark && !c.mtp_path) {
fprintf(stderr, "ds4-bench: --dspark requires --mtp-model FILE\n");
exit(2);
}
if (c.dspark && c.teacher_forced_decode) {
fprintf(stderr,
"ds4-bench: --dspark cannot be combined with "
"--teacher-forced-decode\n");
exit(2);
}
if (c.ctx_start > c.ctx_max) {
fprintf(stderr, "ds4-bench: --ctx-start must be <= --ctx-max\n");
exit(2);
@@ -620,6 +647,7 @@ int main(int argc, char **argv) {
ds4_engine_options opt = {
.model_path = cfg.model_path,
.mtp_path = cfg.mtp_path,
.backend = cfg.backend,
.n_threads = cfg.threads,
.context_size = cfg.ctx_alloc,
@@ -632,6 +660,9 @@ int main(int argc, char **argv) {
.power_percent = cfg.power_percent,
.warm_weights = cfg.warm_weights,
.quality = cfg.quality,
.dspark = cfg.dspark,
.dspark_confidence_threshold = cfg.dspark_confidence_threshold,
.dspark_confidence_threshold_set = cfg.dspark_confidence_threshold_set,
.cuda_tensor_parallel = cfg.cuda_tensor_parallel,
.ssd_streaming = cfg.ssd_streaming,
.ssd_streaming_cold = cfg.ssd_streaming_cold,
@@ -755,6 +786,20 @@ int main(int argc, char **argv) {
const bool distributed =
cfg.dist.role == DS4_DISTRIBUTED_COORDINATOR ||
cfg.tp.role == DS4_TP_LEADER;
const bool speculative = cfg.dspark && ds4_engine_mtp_draft_tokens(engine) > 1;
if (cfg.dspark && !speculative) {
fprintf(stderr, "ds4-bench: DSpark support model did not enable speculative decoding\n");
if (out != stdout) fclose(out);
ds4_session_free(session);
ds4_tokens_free(&prompt);
close_engine(engine, tp_leader);
return 1;
}
if (speculative) {
fprintf(stderr,
"ds4-bench: DSpark enabled with draft width %d; frontier restoration uses session snapshots\n",
ds4_engine_mtp_draft_tokens(engine));
}
ds4_session_snapshot snap = {0};
const uint64_t snapshot_max_bytes = bench_snapshot_max_bytes();
bool warned_large_snapshot = false;
@@ -788,6 +833,9 @@ int main(int argc, char **argv) {
}
const double prefill_t1 = bench_now_sec();
const double prefill_sec = prefill_t1 - prefill_t0;
if (getenv("DS4_METAL_CB_TIMES"))
fprintf(stderr, "ds4-bench: prefill window mono %.1f .. %.1f ms\n",
prefill_t0 * 1e3, prefill_t1 * 1e3);
const int prefill_tokens = frontier - previous;
if (write_frontier_logits_json(&cfg, engine, session, frontier, previous) != 0) {
@@ -825,6 +873,7 @@ int main(int argc, char **argv) {
const double gen_t0 = bench_now_sec();
double gen_first_sec = 0.0;
double gen_steady_sec = 0.0;
int gen_first_tokens = 0;
int gen_done = 0;
int *gen_token_buf = cfg.show_output && cfg.gen_tokens > 0
? malloc((size_t)cfg.gen_tokens * sizeof(gen_token_buf[0]))
@@ -845,14 +894,15 @@ int main(int argc, char **argv) {
cuda_profile_start = -1;
cuda_profile_tokens = 0;
}
for (int i = 0; i < cfg.gen_tokens; i++) {
bool generation_stop = false;
while (gen_done < cfg.gen_tokens && !generation_stop) {
if (ds4_session_pos(session) + 1 >= ds4_session_ctx(session)) {
fprintf(stderr, "ds4-bench: generation would exceed allocated context at frontier %d\n", frontier);
rc = 1;
break;
}
const int token = cfg.teacher_forced_decode
? prompt.v[frontier + i]
? prompt.v[frontier + gen_done]
: ds4_session_argmax_excluding(session, eos);
if (token < 0) {
fprintf(stderr, "ds4-bench: failed to choose non-EOS token at frontier %d\n", frontier);
@@ -861,27 +911,94 @@ int main(int argc, char **argv) {
}
const double token_t0 = bench_now_sec();
#if defined(DS4_BENCH_HAVE_CUDA_PROFILER)
if (i == cuda_profile_start && cudaProfilerStart) {
if (gen_done == cuda_profile_start && cudaProfilerStart) {
(void)cudaProfilerStart();
}
#endif
if (ds4_session_eval(session, token, err, sizeof(err)) != 0) {
fprintf(stderr, "ds4-bench: decode at frontier %d failed: %s\n", frontier, err);
rc = 1;
break;
int toks[17];
int ntok = 1;
if (speculative) {
const double spec_t0 = bench_now_sec();
ntok = ds4_session_eval_speculative_argmax(
session, token, cfg.gen_tokens - gen_done, eos,
toks, (int)(sizeof(toks) / sizeof(toks[0])),
err, sizeof(err));
if (getenv("DS4_BENCH_SPEC_TRACE") && ntok > 0 && ntok < 17) {
static double call_ms[17];
static unsigned call_n[17];
static unsigned calls_total;
call_ms[ntok] += (bench_now_sec() - spec_t0) * 1e3;
call_n[ntok]++;
if (++calls_total % 32u == 0u) {
fprintf(stderr, "ds4-bench: spec calls %u:", calls_total);
for (int k = 1; k < 17; k++)
if (call_n[k])
fprintf(stderr, " %d-tok x%u avg %.1f ms", k, call_n[k], call_ms[k] / call_n[k]);
fprintf(stderr, "\n");
}
}
if (ntok < 0) {
fprintf(stderr, "ds4-bench: DSpark decode at frontier %d failed: %s\n", frontier, err);
rc = 1;
break;
}
if (ntok == 0) {
fprintf(stderr, "ds4-bench: DSpark decode at frontier %d accepted no tokens\n", frontier);
rc = 1;
break;
}
} else {
toks[0] = token;
if (ds4_session_eval(session, token, err, sizeof(err)) != 0) {
fprintf(stderr, "ds4-bench: decode at frontier %d failed: %s\n", frontier, err);
rc = 1;
break;
}
}
#if defined(DS4_BENCH_HAVE_CUDA_PROFILER)
if (cuda_profile_start >= 0 &&
i + 1 == cuda_profile_start + cuda_profile_tokens &&
gen_done + ntok >=
cuda_profile_start + cuda_profile_tokens &&
cudaProfilerStop) {
(void)cudaProfilerStop();
}
#endif
const double token_t1 = bench_now_sec();
if (i == 0) gen_first_sec = token_t1 - token_t0;
else gen_steady_sec += token_t1 - token_t0;
if (gen_token_buf) gen_token_buf[gen_token_count++] = token;
gen_done++;
/* Diagnostic: DS4_BENCH_DECODE_LOGITS_HASH prints an FNV-1a hash of
* every decode step's full logits, to compare arms bit for bit. */
if (!speculative && getenv("DS4_BENCH_DECODE_LOGITS_HASH")) {
const int vocab = ds4_engine_vocab_size(engine);
float *lg = vocab > 0 ? malloc((size_t)vocab * sizeof(lg[0])) : NULL;
if (lg && ds4_session_copy_logits(session, lg, vocab) == vocab) {
uint64_t h = 1469598103934665603ull;
const unsigned char *b = (const unsigned char *)lg;
for (size_t k = 0; k < (size_t)vocab * sizeof(lg[0]); k++) h = (h ^ b[k]) * 1099511628211ull;
fprintf(stderr, "ds4-bench: decode-hash frontier=%d step=%d %016llx\n",
frontier, gen_done, (unsigned long long)h);
const char *dump = getenv("DS4_BENCH_DECODE_LOGITS_DUMP");
if (dump) {
FILE *fp = fopen(dump, "ab");
if (fp) { fwrite(lg, sizeof(lg[0]), (size_t)vocab, fp); fclose(fp); }
}
}
free(lg);
}
int cycle_tokens = 0;
for (int j = 0; j < ntok && gen_done < cfg.gen_tokens; j++) {
if (toks[j] == eos) {
generation_stop = true;
break;
}
if (gen_token_buf) gen_token_buf[gen_token_count++] = toks[j];
gen_done++;
cycle_tokens++;
}
if (gen_first_tokens == 0) {
gen_first_sec = token_t1 - token_t0;
gen_first_tokens = cycle_tokens;
} else {
gen_steady_sec += token_t1 - token_t0;
}
}
const double gen_t1 = bench_now_sec();
if (cfg.show_output && gen_token_buf && gen_token_count > 0) {
@@ -917,7 +1034,8 @@ int main(int argc, char **argv) {
}
const double gen_sec = gen_t1 - gen_t0;
const int gen_steady_tokens = gen_done > 1 ? gen_done - 1 : 0;
const int gen_steady_tokens = gen_done > gen_first_tokens ?
gen_done - gen_first_tokens : 0;
fprintf(out,
"%d,%d,%.2f,%d,%.2f,%.3f,%d,%.2f,%llu\n",
frontier,
ds4_cli.c
+154 -46
View File
@@ -3,6 +3,7 @@
#include "ds4_gpu_args.h"
#include "ds4_tp.h"
#include "ds4_help.h"
#include "ds4_prompt_prefix.h"
#include "linenoise.h"
/* ds4 CLI.
@@ -61,6 +62,7 @@ static bool cli_greedy_argmax_requested(bool speculative_requested) {
typedef struct {
const char *prompt;
const char *system;
ds4_prompt_prefix prefix;
bool raw_prompt;
int n_predict;
int ctx_size;
@@ -81,6 +83,7 @@ typedef struct {
const char *imatrix_output_path;
int imatrix_max_prompts;
int imatrix_max_tokens;
int imatrix_min_expert_samples;
ds4_think_mode think_mode;
bool head_test;
bool first_token_test;
@@ -497,14 +500,29 @@ static void print_generated_token(void *ud, int token) {
free(text);
}
static void build_chat_prompt(ds4_engine *engine,
const cli_generation_options *gen,
ds4_tokens *out) {
const ds4_think_mode think_mode = cli_effective_think_mode(gen);
ds4_chat_begin(engine, out);
ds4_chat_append_think_prefix(engine, out, think_mode);
if (gen->system && gen->system[0])
ds4_chat_append_message(engine, out, "system", gen->system);
ds4_prompt_prefix_append(engine, out, &gen->prefix);
ds4_chat_append_message(engine, out, "user", gen->prompt ? gen->prompt : "");
ds4_chat_append_assistant_prefix(engine, out, think_mode);
}
static void build_prompt(ds4_engine *engine, const cli_generation_options *gen, ds4_tokens *out) {
if (gen->raw_prompt) {
ds4_tokenize_text(engine, gen->prompt ? gen->prompt : "", out);
} else if (is_rendered_chat_prompt(gen->prompt)) {
ds4_tokenize_rendered_chat(engine, gen->prompt, out);
} else {
} else if (gen->prefix.count == 0) {
ds4_encode_chat_prompt(engine, gen->system, gen->prompt,
cli_effective_think_mode(gen), out);
} else {
build_chat_prompt(engine, gen, out);
}
}
@@ -1268,8 +1286,8 @@ static char *trim_inplace(char *s) {
static void print_repl_help(void) {
puts("Commands:");
puts(" /help Show this help.");
puts(" /think Use normal thinking mode.");
puts(" /think-max Use Think Max only when context is at least 393216 tokens.");
puts(" /think [N] Use normal thinking, or V4.1 effort 0..100 (0 disables thinking).");
puts(" /think-max Use maximum thinking (V4.1: 100; V4: requires ctx >= 393216).");
puts(" /nothink Disable thinking mode.");
puts(" /ctx N Set context size for following prompts.");
puts(" /power N Set GPU duty cycle percentage, 1..100.");
@@ -1314,8 +1332,11 @@ typedef struct {
int ctx_size;
int think_prefix_pos;
int think_prefix_tokens;
bool initial_system_text;
} repl_chat;
static void repl_chat_free(repl_chat *chat);
static void repl_chat_trim_images(repl_chat *chat, size_t count) {
while (chat->image_count > count) {
chat->image_count--;
@@ -1364,34 +1385,22 @@ static void tokens_remove(ds4_tokens *dst, int pos, int n) {
dst->len -= n;
}
static const char *repl_glm_reasoning_effort_text(ds4_think_mode mode) {
switch (mode) {
case DS4_THINK_HIGH: return "Reasoning Effort: High";
case DS4_THINK_MAX: return "Reasoning Effort: Max";
case DS4_THINK_NONE: return NULL;
}
return NULL;
}
static void repl_chat_build_think_prefix(ds4_engine *engine,
ds4_think_mode mode,
ds4_tokens *prefix) {
if (ds4_engine_is_glm_dsa(engine)) {
const char *effort = repl_glm_reasoning_effort_text(mode);
if (effort) ds4_chat_append_message(engine, prefix, "system", effort);
} else if (mode == DS4_THINK_MAX) {
ds4_chat_append_max_effort_prefix(engine, prefix);
}
ds4_chat_append_think_prefix(engine, prefix, mode);
}
/* Insert/replace the model-family thinking prefix inside the existing
* transcript. It lives immediately after the BOS sequence and before any
* user/system text, matching the GGUF chat templates. */
static void repl_chat_apply_think_prefix(ds4_engine *engine,
static bool repl_chat_apply_think_prefix(ds4_engine *engine,
repl_chat *chat,
ds4_think_mode mode) {
ds4_tokens prefix = {0};
repl_chat_build_think_prefix(engine, mode, &prefix);
if (ds4_engine_is_deepseek41(engine) && chat->initial_system_text && !prefix.len)
ds4_chat_append_message(engine, &prefix, "system", "");
bool same = chat->think_prefix_tokens == prefix.len;
if (same && prefix.len > 0) {
@@ -1401,11 +1410,16 @@ static void repl_chat_apply_think_prefix(ds4_engine *engine,
}
if (!same) {
const int old_prefix_tokens = chat->think_prefix_tokens;
const int shift = prefix.len - old_prefix_tokens;
if (chat->ctx_size > 0 && (int64_t)chat->transcript.len + shift >= chat->ctx_size) {
fprintf(stderr, "ds4: no context room to change the thinking prefix\n");
ds4_tokens_free(&prefix);
return false;
}
tokens_remove(&chat->transcript, chat->think_prefix_pos,
chat->think_prefix_tokens);
tokens_insert(&chat->transcript, chat->think_prefix_pos, &prefix);
chat->think_prefix_tokens = prefix.len;
const int shift = prefix.len - old_prefix_tokens;
for (size_t i = 0; i < chat->image_count; i++) {
chat->images[i].token_start =
(uint32_t)((int64_t)chat->images[i].token_start + shift);
@@ -1413,6 +1427,7 @@ static void repl_chat_apply_think_prefix(ds4_engine *engine,
if (chat->session) ds4_session_invalidate(chat->session);
}
ds4_tokens_free(&prefix);
return true;
}
static int repl_chat_create_session(ds4_engine *engine, repl_chat *chat, int ctx_size) {
@@ -1429,13 +1444,47 @@ static int repl_chat_create_session(ds4_engine *engine, repl_chat *chat, int ctx
static int repl_chat_init(ds4_engine *engine, repl_chat *chat, const cli_config *cfg) {
memset(chat, 0, sizeof(*chat));
chat->initial_system_text = cfg->gen.system && cfg->gen.system[0];
ds4_chat_begin(engine, &chat->transcript);
chat->think_prefix_pos = chat->transcript.len;
repl_chat_apply_think_prefix(engine, chat, cli_effective_think_mode(&cfg->gen));
if (cfg->gen.system && cfg->gen.system[0]) {
ds4_chat_append_message(engine, &chat->transcript, "system", cfg->gen.system);
}
return repl_chat_create_session(engine, chat, cfg->gen.ctx_size);
ds4_prompt_prefix_append(engine, &chat->transcript, &cfg->gen.prefix);
if (repl_chat_create_session(engine, chat, cfg->gen.ctx_size) != 0) {
ds4_tokens_free(&chat->transcript);
return 1;
}
if (cfg->gen.prefix.count == 0) return 0;
if (cli_wait_distributed_route(cfg, chat->session) != 0) {
repl_chat_free(chat);
return 1;
}
char err[160] = {0};
cli_prefill_progress progress = {
.base_tokens = 0,
.input_tokens = chat->transcript.len,
.use_color = ds4_log_is_tty(stderr),
};
ds4_session_set_progress(chat->session, cli_prefill_progress_cb, &progress);
ds4_session_set_display_progress(chat->session,
progress.use_color ? cli_prefill_progress_cb : NULL,
progress.use_color ? &progress : NULL);
cli_dist_busy_set(cfg, true);
int rc = ds4_session_sync(chat->session, &chat->transcript,
err, sizeof(err));
cli_dist_busy_set(cfg, false);
ds4_session_set_progress(chat->session, NULL, NULL);
ds4_session_set_display_progress(chat->session, NULL, NULL);
if (rc != 0) {
fprintf(stderr, "ds4: prefix prefill failed: %s\n",
err[0] ? err : "unknown error");
repl_chat_free(chat);
return 1;
}
return 0;
}
static void repl_chat_free(repl_chat *chat) {
@@ -1472,27 +1521,22 @@ static int run_chat_turn(ds4_engine *engine, cli_config *cfg, repl_chat *chat,
ds4_think_mode think_mode = ds4_think_mode_for_context(cfg->gen.think_mode,
chat->ctx_size);
repl_chat_apply_think_prefix(engine, chat, think_mode);
if (!repl_chat_apply_think_prefix(engine, chat, think_mode)) return 1;
const int rollback_len = chat->transcript.len;
const size_t rollback_images = chat->image_count;
if (image) {
ds4_tokens_push(&chat->transcript, ds4_token_user(engine));
ds4_vision_span *span = repl_chat_add_image(chat);
const char *text_parts[] = {"", user_text ? user_text : ""};
char image_error[160] = {0};
if (!span || !ds4_prompt_append_vision(engine, &chat->transcript,
span, image,
image_error,
sizeof(image_error))) {
if (!span || !ds4_chat_append_multimodal_message(
engine, &chat->transcript, "user", text_parts,
image, 1, span, image_error, sizeof(image_error))) {
repl_chat_trim_images(chat, rollback_images);
chat->transcript.len = rollback_len;
fprintf(stderr, "ds4: failed to add image: %s\n",
image_error[0] ? image_error : "out of memory");
return 1;
}
if (user_text && user_text[0]) {
ds4_tokenize_text(engine, "\n", &chat->transcript);
ds4_tokenize_text(engine, user_text, &chat->transcript);
}
} else {
ds4_chat_append_message(engine, &chat->transcript, "user", user_text);
}
@@ -1702,22 +1746,31 @@ static int run_repl(ds4_engine *engine, cli_config *cfg) {
if (!strcmp(cmd, "/help")) {
print_repl_help();
} else if (!strcmp(cmd, "/think")) {
cfg->gen.think_mode = DS4_THINK_HIGH;
repl_chat_apply_think_prefix(engine, &chat, DS4_THINK_HIGH);
puts("Thinking mode: high.");
} else if (!strncmp(cmd, "/think", 6) &&
(cmd[6] == '\0' || isspace((unsigned char)cmd[6]))) {
const char *arg = trim_inplace(cmd + 6);
ds4_think_mode mode = DS4_THINK_HIGH;
if (arg[0] && (!ds4_engine_is_deepseek41(engine) ||
!ds4_think_mode_parse_level(arg, &mode))) {
fprintf(stderr, "ds4: /think N requires V4.1 and a level from 0 to 100\n");
} else if (repl_chat_apply_think_prefix(engine, &chat, mode)) {
cfg->gen.think_mode = mode;
printf("Thinking mode: %s.\n", ds4_think_mode_name(mode));
}
} else if (!strcmp(cmd, "/think-max")) {
cfg->gen.think_mode = DS4_THINK_MAX;
bool active = ds4_think_mode_for_context(cfg->gen.think_mode,
bool active = ds4_think_mode_for_context(DS4_THINK_MAX,
chat.ctx_size) == DS4_THINK_MAX;
repl_chat_apply_think_prefix(engine, &chat,
active ? DS4_THINK_MAX : DS4_THINK_HIGH);
cli_warn_think_max_downgraded(&cfg->gen, "/think-max");
printf("Thinking mode: %s.\n", active ? "max" : "high (ctx below 393216)");
if (repl_chat_apply_think_prefix(engine, &chat,
active ? DS4_THINK_MAX : DS4_THINK_HIGH)) {
cfg->gen.think_mode = DS4_THINK_MAX;
cli_warn_think_max_downgraded(&cfg->gen, "/think-max");
printf("Thinking mode: %s.\n", active ? "max" : "high (ctx below 393216)");
}
} else if (!strcmp(cmd, "/nothink")) {
cfg->gen.think_mode = DS4_THINK_NONE;
repl_chat_apply_think_prefix(engine, &chat, DS4_THINK_NONE);
puts("Thinking mode: none.");
if (repl_chat_apply_think_prefix(engine, &chat, DS4_THINK_NONE)) {
cfg->gen.think_mode = DS4_THINK_NONE;
puts("Thinking mode: none.");
}
} else if (!strncmp(cmd, "/power", 6) && (cmd[6] == '\0' || isspace((unsigned char)cmd[6]))) {
char *arg = trim_inplace(cmd + 6);
if (!arg[0]) {
@@ -1942,6 +1995,19 @@ static cli_config parse_options(int argc, char **argv) {
}
c.prompt_owned = read_prompt_file(need_arg(&i, argc, argv, arg), true);
c.gen.prompt = c.prompt_owned;
} else if (!strcmp(arg, "--prefix-file")) {
if (c.gen.prefix.count != 0) {
fprintf(stderr, "ds4: specify --prefix-file only once\n");
exit(2);
}
const char *path = need_arg(&i, argc, argv, arg);
char error[256] = {0};
if (ds4_prompt_prefix_load(&c.gen.prefix, path,
error, sizeof(error)) != 0) {
fprintf(stderr, "ds4: %s\n",
error[0] ? error : "invalid prefix file");
exit(2);
}
} else if (!strcmp(arg, "-sys") || !strcmp(arg, "--system")) {
c.gen.system = need_arg(&i, argc, argv, arg);
} else if (!strcmp(arg, "--raw") || !strcmp(arg, "--raw-prompt")) {
@@ -2088,6 +2154,14 @@ static cli_config parse_options(int argc, char **argv) {
c.gen.imatrix_max_prompts = parse_int(need_arg(&i, argc, argv, arg), arg);
} else if (!strcmp(arg, "--imatrix-max-tokens")) {
c.gen.imatrix_max_tokens = parse_int(need_arg(&i, argc, argv, arg), arg);
} else if (!strcmp(arg, "--imatrix-min-expert-samples")) {
c.gen.imatrix_min_expert_samples =
parse_int(need_arg(&i, argc, argv, arg), arg);
} else if (!strcmp(arg, "--think-level")) {
if (!ds4_think_mode_parse_level(need_arg(&i, argc, argv, arg), &c.gen.think_mode)) {
fprintf(stderr, "ds4: --think-level requires an integer from 0 to 100\n");
exit(2);
}
} else if (!strcmp(arg, "--think")) {
c.gen.think_mode = DS4_THINK_HIGH;
} else if (!strcmp(arg, "--think-max")) {
@@ -2147,10 +2221,23 @@ static cli_config parse_options(int argc, char **argv) {
fprintf(stderr, "ds4: --imatrix-dataset requires --imatrix-out\n");
exit(2);
}
if (c.gen.imatrix_min_expert_samples < 0) {
fprintf(stderr, "ds4: --imatrix-min-expert-samples must not be negative\n");
exit(2);
}
if (c.gen.perplexity_file_path && c.gen.prompt) {
fprintf(stderr, "ds4: --perplexity-file does not use -p/--prompt-file\n");
exit(2);
}
if (c.gen.prefix.count != 0 && c.gen.raw_prompt) {
fprintf(stderr, "ds4: --prefix-file cannot be combined with --raw-prompt\n");
exit(2);
}
if (c.gen.prefix.count != 0 && is_rendered_chat_prompt(c.gen.prompt)) {
fprintf(stderr,
"ds4: --prefix-file cannot be combined with an already-rendered prompt\n");
exit(2);
}
char tp_err[256];
if (!ds4_tp_adopt_distributed_options(&c.engine.tp, c.dist,
tp_err, sizeof(tp_err))) {
@@ -2173,8 +2260,17 @@ static cli_config parse_options(int argc, char **argv) {
int main(int argc, char **argv) {
cli_config cfg = parse_options(argc, argv);
if (cfg.gen.dump_tokens) {
if (cfg.gen.prefix.count != 0) {
fprintf(stderr, "ds4: --dump-tokens does not support --prefix-file\n");
ds4_prompt_prefix_free(&cfg.gen.prefix);
ds4_dist_options_free(cfg.dist);
free(cfg.prompt_owned);
return 2;
}
if (cfg.gen.prompt == NULL) {
fprintf(stderr, "ds4: --dump-tokens requires -p or --prompt-file\n");
ds4_prompt_prefix_free(&cfg.gen.prefix);
ds4_dist_options_free(cfg.dist);
free(cfg.prompt_owned);
return 2;
}
@@ -2187,10 +2283,12 @@ int main(int argc, char **argv) {
rc = ds4_dump_chat_tokenization(cfg.engine.model_path,
cfg.gen.system,
cfg.gen.prompt,
cli_effective_think_mode(&cfg.gen),
cfg.gen.think_mode,
cfg.gen.ctx_size,
stdout);
}
ds4_dist_options_free(cfg.dist);
ds4_prompt_prefix_free(&cfg.gen.prefix);
free(cfg.prompt_owned);
return rc;
}
@@ -2241,6 +2339,14 @@ int main(int argc, char **argv) {
free(cfg.prompt_owned);
return 1;
}
if (ds4_think_mode_level(cfg.gen.think_mode) >= 0 && !ds4_engine_is_deepseek41(engine)) {
fprintf(stderr, "ds4: --think-level requires a DeepSeek V4.1 model\n");
ds4_engine_close(engine);
ds4_dist_options_free(cfg.dist);
ds4_prompt_prefix_free(&cfg.gen.prefix);
free(cfg.prompt_owned);
return 2;
}
cli_apply_model_sampling_defaults(engine, &cfg.gen);
if (cfg.engine.tp.role == DS4_TP_WORKER) {
int rc = ds4_tp_worker_run(engine, &cfg.engine.tp);
@@ -2313,7 +2419,8 @@ int main(int argc, char **argv) {
cfg.gen.imatrix_output_path,
cfg.gen.ctx_size,
cfg.gen.imatrix_max_prompts,
cfg.gen.imatrix_max_tokens);
cfg.gen.imatrix_max_tokens,
cfg.gen.imatrix_min_expert_samples);
} else if (cfg.gen.perplexity_file_path) {
rc = run_perplexity_file(engine, &cfg);
} else if (cfg.gen.prompt == NULL) {
@@ -2325,6 +2432,7 @@ int main(int argc, char **argv) {
ds4_engine_close(engine);
ds4_tp_free(tp_leader);
ds4_dist_options_free(cfg.dist);
ds4_prompt_prefix_free(&cfg.gen.prefix);
free(cfg.prompt_owned);
return rc;
}
ds4_cuda.cu
+4330 -736
View File
File diff suppressed because it is too large. Load diff
ds4_deepseek41_cuda.cuh
+1017
View File
File diff suppressed because it is too large. Load diff
ds4_deepseek41_dspark.h
+471
View File
@@ -0,0 +1,471 @@
/* Internal V4.1 drafter graph. Included after the target graph in ds4.c. */
#ifndef DS4_DEEPSEEK41_DSPARK_H
#define DS4_DEEPSEEK41_DSPARK_H
#define DS41_DRAFT_WINDOW 128u
#define DS41_DRAFT_ROWS 5u
#define DS41_VERIFY_ROWS (DS41_DRAFT_ROWS + 1u)
struct ds41_dspark {
const ds4_dspark_weights *weights;
ds41_prefill_row batch;
ds4_gpu_tensor *target[3], *window[3];
uint32_t positions[3][DS41_DRAFT_WINDOW];
uint32_t kv_end;
bool kv_valid;
ds4_gpu_tensor *mean_weights, *mean, *input, *projected, *main_x, *main_kv;
ds4_gpu_tensor *logits, *tokens, *markov_embed, *markov_logits, *confidence_input, *confidence;
float *host_logits, *draft_probs;
ds4_gpu_tensor *verify_residual, *verify_pre, *verify_logits, *verify_pool;
ds4_gpu_tensor *old_raw, *old_features;
uint32_t old_positions[3][DS41_VERIFY_ROWS];
ds41_gpu_graph verified[DS41_VERIFY_ROWS + 1u];
float *host_verified;
uint32_t verify_start, verify_count, rewind_end;
};
static void ds41_dspark_reset(ds41_dspark *d) {
if (!d) return;
memset(d->positions, 0xff, sizeof(d->positions));
d->kv_valid = false;
d->kv_end = 0;
d->verify_count = 0;
d->rewind_end = 0;
}
static void ds41_dspark_free(ds41_dspark *d) {
if (!d) return;
#define DS41_DRAFT_FREE(name, count) ds4_gpu_tensor_free(d->batch.name);
DS41_PREFILL_ROWS(DS41_DRAFT_FREE)
#undef DS41_DRAFT_FREE
for (unsigned i = 0; i < 3; i++) {
ds4_gpu_tensor_free(d->target[i]);
ds4_gpu_tensor_free(d->window[i]);
}
ds4_gpu_tensor_free(d->mean_weights); ds4_gpu_tensor_free(d->mean);
ds4_gpu_tensor_free(d->input); ds4_gpu_tensor_free(d->projected);
ds4_gpu_tensor_free(d->main_x); ds4_gpu_tensor_free(d->main_kv);
ds4_gpu_tensor_free(d->logits); ds4_gpu_tensor_free(d->markov_embed);
ds4_gpu_tensor_free(d->tokens);
ds4_gpu_tensor_free(d->markov_logits); ds4_gpu_tensor_free(d->confidence_input);
ds4_gpu_tensor_free(d->confidence);
for (unsigned i = 0; i <= DS41_VERIFY_ROWS; i++) {
ds41_gpu_graph *g = &d->verified[i];
ds4_gpu_tensor_free(g->residual); ds4_gpu_tensor_free(g->pre);
ds4_gpu_tensor_free(g->logits);
ds4_gpu_tensor_free(g->tp_logits_half);
for (unsigned j = 0; j < 4; j++) {
ds4_gpu_tensor_free(g->verify_previous_kv[j]);
ds4_gpu_tensor_free(g->verify_previous_score[j]);
}
}
ds4_gpu_tensor_free(d->verify_residual); ds4_gpu_tensor_free(d->verify_pre);
ds4_gpu_tensor_free(d->verify_logits); ds4_gpu_tensor_free(d->verify_pool);
ds4_gpu_tensor_free(d->old_raw); ds4_gpu_tensor_free(d->old_features);
free(d->host_verified);
free(d->host_logits);
free(d->draft_probs);
free(d);
}
static DS4_MAYBE_UNUSED ds41_dspark *ds41_dspark_alloc(const ds4_dspark_weights *dw) {
if (!dw || !dw->v41 || dw->n_stages != 3 || dw->block_size != DS41_DRAFT_ROWS ||
dw->missing_tensors || dw->invalid_tensors || dw->metadata_errors) return NULL;
ds41_dspark *d = calloc(1, sizeof(*d));
if (!d) return NULL;
d->weights = dw;
const ds41_gpu_graph shape = {.ctx = DS41_DRAFT_WINDOW}, *g = &shape;
#define DS41_DRAFT_ALLOC(name, width) \
if (!(d->batch.name = ds4_gpu_tensor_alloc((uint64_t)(width) * DS41_DRAFT_ROWS * sizeof(float)))) goto fail;
DS41_PREFILL_ROWS(DS41_DRAFT_ALLOC)
#undef DS41_DRAFT_ALLOC
for (unsigned i = 0; i < 3; i++) {
d->target[i] = ds4_gpu_tensor_alloc((uint64_t)DS41_DRAFT_WINDOW * DS4_N_EMBD * 4u);
d->window[i] = ds4_gpu_tensor_alloc((uint64_t)(DS41_DRAFT_WINDOW + DS41_DRAFT_ROWS) * DS4_N_HEAD_DIM * 4u);
if (!d->target[i] || !d->window[i]) goto fail;
}
#define DS41_DRAFT_A(name, count) \
if (!(d->name = ds4_gpu_tensor_alloc((uint64_t)(count) * sizeof(float)))) goto fail;
DS41_DRAFT_A(mean_weights, DS41_DRAFT_WINDOW * DS4_N_HC)
DS41_DRAFT_A(mean, DS41_DRAFT_WINDOW * DS4_N_EMBD)
DS41_DRAFT_A(input, DS41_DRAFT_WINDOW * 3u * DS4_N_EMBD)
DS41_DRAFT_A(projected, DS41_DRAFT_WINDOW * DS4_N_EMBD)
DS41_DRAFT_A(main_x, DS41_DRAFT_WINDOW * DS4_N_EMBD)
DS41_DRAFT_A(main_kv, DS41_DRAFT_WINDOW * DS4_N_HEAD_DIM)
DS41_DRAFT_A(logits, DS41_DRAFT_ROWS * DS4_N_VOCAB)
DS41_DRAFT_A(tokens, DS41_DRAFT_ROWS)
DS41_DRAFT_A(markov_embed, dw->markov_rank)
DS41_DRAFT_A(markov_logits, DS4_N_VOCAB)
DS41_DRAFT_A(confidence_input, DS4_N_EMBD + dw->markov_rank)
DS41_DRAFT_A(confidence, 1)
DS41_DRAFT_A(verify_residual, (DS41_VERIFY_ROWS + 1u) * DS4_N_HC * DS4_N_EMBD)
DS41_DRAFT_A(verify_pre, (DS41_VERIFY_ROWS + 1u) * DS4_N_HC)
DS41_DRAFT_A(verify_logits, (DS41_VERIFY_ROWS + 1u) * DS4_N_VOCAB)
DS41_DRAFT_A(verify_pool, (DS41_VERIFY_ROWS + 1u) * 8u * DS4_N_HEAD_DIM)
DS41_DRAFT_A(old_raw, 40u * DS41_VERIFY_ROWS * DS4_N_HEAD_DIM)
DS41_DRAFT_A(old_features, 3u * DS41_VERIFY_ROWS * DS4_N_EMBD)
#undef DS41_DRAFT_A
d->host_logits = malloc((uint64_t)DS41_DRAFT_ROWS * DS4_N_VOCAB * sizeof(float));
d->draft_probs = malloc((uint64_t)DS41_DRAFT_ROWS * DS4_N_VOCAB * sizeof(float));
d->host_verified = malloc((uint64_t)(DS41_VERIFY_ROWS + 1u) * DS4_N_VOCAB * sizeof(float));
if (!d->host_logits || !d->draft_probs || !d->host_verified || !ds4_gpu_tensor_fill_f32(d->mean_weights,
1.0f / DS4_N_HC, DS41_DRAFT_WINDOW * DS4_N_HC) ||
!ds4_gpu_tensor_fill_f32(d->tokens, 0, DS41_DRAFT_ROWS)) goto fail;
ds41_dspark_reset(d);
return d;
fail:
ds41_dspark_free(d);
return NULL;
}
static bool ds41_dspark_capture(ds41_dspark *d, uint32_t layer, uint32_t pos,
const ds4_gpu_tensor *residual, uint32_t rows) {
if (!d) return true;
d->rewind_end = 0;
unsigned stage = 0;
while (stage < 3 && d->weights->target_layers[stage] != layer) stage++;
if (stage == 3) return true;
if (!rows || rows > UINT32_MAX - pos) return false;
const uint32_t count = rows < DS41_DRAFT_WINDOW ? rows : DS41_DRAFT_WINDOW;
const uint32_t first = rows - count, start = pos + first;
const uint64_t bytes = (uint64_t)DS4_N_EMBD * sizeof(float);
ds4_gpu_tensor *src = ds4_gpu_tensor_view((ds4_gpu_tensor *)residual,
first * DS4_N_HC * bytes, count * DS4_N_HC * bytes);
ds4_gpu_tensor *out = ds4_gpu_tensor_view(d->mean, 0, count * bytes);
ds4_gpu_tensor *mix = ds4_gpu_tensor_view(d->mean_weights, 0, count * DS4_N_HC * sizeof(float));
bool ok = src && out && mix && ds4_gpu_hc_weighted_sum_tensor(out, src, mix, DS4_N_EMBD, DS4_N_HC) &&
ds4_gpu_dsv41_quantize(out, DS4_N_EMBD, count, DS4_V41_BF16);
const uint32_t slot = start % DS41_DRAFT_WINDOW;
const uint32_t part = count < DS41_DRAFT_WINDOW - slot ? count : DS41_DRAFT_WINDOW - slot;
if (ok) ok = ds4_gpu_tensor_copy(d->target[stage], slot * bytes, out, 0, part * bytes) &&
(part == count || ds4_gpu_tensor_copy(d->target[stage], 0, out, part * bytes, (count - part) * bytes));
ds4_gpu_tensor_free(mix); ds4_gpu_tensor_free(out); ds4_gpu_tensor_free(src);
if (ok) for (uint32_t i = 0; i < count; i++)
d->positions[stage][(start + i) % DS41_DRAFT_WINDOW] = start + i;
return ok;
}
static bool ds41_dspark_matmul_batch(ds4_gpu_tensor *out, const ds4_model *m,
const ds4_tensor *weight, const ds4_gpu_tensor *in, uint32_t count, bool round) {
#if !defined(__APPLE__) && !defined(DS4_ROCM_BUILD)
/* Draft rows share a GEMM. Target verification retains its scalar-reduction
* kernels; proposal arithmetic need not reproduce their reduction order. */
if (count > 1 && weight->type == DS4_TENSOR_F16 &&
!getenv("DS4_DISABLE_V41_DRAFT_GEMM"))
return metal_graph_matmul_plain_tensor(out, m, weight, weight->dim[0],
weight->dim[1], in, count) && (!round ||
ds4_gpu_dsv41_quantize(out, (uint32_t)weight->dim[1], count, DS4_V41_BF16));
#endif
return ds41_matmul_batch(out, m, weight, in, count, round);
}
/* Only accepted target features enter these rings. A fresh prefill seeds the
* last window once; subsequent calls project only the newly accepted rows. */
static bool ds41_dspark_seed(ds41_dspark *d, const ds4_model *m, uint32_t pos) {
if (!pos) return false;
const uint32_t oldest = pos > DS41_DRAFT_WINDOW ? pos - DS41_DRAFT_WINDOW : 0;
const uint32_t start = d->kv_valid && d->kv_end >= oldest && d->kv_end <= pos ? d->kv_end : oldest;
const uint32_t count = pos - start;
if (!count) return true;
const uint64_t row_bytes = (uint64_t)DS4_N_EMBD * sizeof(float);
for (uint32_t i = 0; i < count; i++) {
const uint32_t slot = (start + i) % DS41_DRAFT_WINDOW;
for (unsigned stage = 0; stage < 3; stage++) {
if (d->positions[stage][slot] != start + i ||
!ds4_gpu_tensor_copy(d->input, (i * 3u + stage) * row_bytes,
d->target[stage], slot * row_bytes, row_bytes)) return false;
}
}
const ds4_dspark_stage_weights *first = &d->weights->stage[0];
if (!ds41_dspark_matmul_batch(d->projected, m, first->main_proj, d->input, count, true) ||
!ds41_norm_batch(d->main_x, d->projected, m, first->main_norm, count)) return false;
const uint64_t kv_bytes = DS4_N_HEAD_DIM * sizeof(float);
for (unsigned stage = 0; stage < 3; stage++) {
const ds4_layer_weights *l = &d->weights->stage[stage].block;
if (!ds41_dspark_matmul_batch(d->main_kv, m, l->attn_kv, d->main_x, count, true) ||
!ds41_norm_batch(d->main_kv, d->main_kv, m, l->attn_kv_a_norm, count) ||
!ds4_gpu_dsv41_rope(d->main_kv, DS4_N_HEAD_DIM, 1, count, start, false, false) ||
!ds4_gpu_dsv41_quantize(d->main_kv, DS4_N_HEAD_DIM, count, DS4_V41_FP8_E8M0)) return false;
const uint32_t slot = start % DS41_DRAFT_WINDOW;
const uint32_t part = count < DS41_DRAFT_WINDOW - slot ? count : DS41_DRAFT_WINDOW - slot;
if (!ds4_gpu_tensor_copy(d->window[stage], slot * kv_bytes, d->main_kv, 0, part * kv_bytes) ||
(part < count && !ds4_gpu_tensor_copy(d->window[stage], 0, d->main_kv,
part * kv_bytes, (count - part) * kv_bytes))) return false;
}
d->kv_end = pos;
d->kv_valid = true;
return true;
}
static bool ds41_dspark_attention_output(ds41_prefill_row *b, const ds4_model *m,
const ds4_layer_weights *l) {
const uint32_t groups = DS4_N_OUT_GROUP, width = DS4_N_HEAD * DS4_N_HEAD_DIM / groups;
uint64_t weight_row;
if (!tensor_nbytes(l->attn_output_a->type, width, &weight_row)) return false;
for (uint32_t row = 0; row < DS41_DRAFT_ROWS; row++) {
for (uint32_t group = 0; group < groups; group++) {
ds4_tensor slice = *l->attn_output_a;
slice.abs_offset += group * DS4_N_LORA_O * weight_row;
slice.dim[1] = DS4_N_LORA_O;
ds4_gpu_tensor *x = ds4_gpu_tensor_view(b->heads,
((uint64_t)row * groups + group) * width * 4u, width * 4u);
ds4_gpu_tensor *y = ds4_gpu_tensor_view(b->low,
((uint64_t)row * groups + group) * DS4_N_LORA_O * 4u, DS4_N_LORA_O * 4u);
const bool ok = x && y && ds41_matmul(y, m, &slice, x, true);
ds4_gpu_tensor_free(y); ds4_gpu_tensor_free(x);
if (!ok) return false;
}
}
return ds41_dspark_matmul_batch(b->block, m, l->attn_output_b, b->low, DS41_DRAFT_ROWS, true);
}
static bool ds41_dspark_stage(ds41_dspark *d, const ds4_model *m, uint32_t stage, uint32_t pos) {
ds41_prefill_row *b = &d->batch;
const ds4_layer_weights *l = &d->weights->stage[stage].block;
const uint32_t n = DS41_DRAFT_ROWS, past = pos < DS41_DRAFT_WINDOW ? pos : DS41_DRAFT_WINDOW;
if (!ds41_hc_mix_batch(b, m, l, false, n) ||
!(stage ? ds4_gpu_hc_weighted_sum_split_tensor(b->x, b->residual, b->ffn_split, DS4_N_EMBD, DS4_N_HC) :
ds4_gpu_hc_weighted_sum_tensor(b->x, b->residual, b->pre, DS4_N_EMBD, DS4_N_HC)) ||
!ds4_gpu_dsv41_quantize(b->x, DS4_N_EMBD, n, DS4_V41_BF16) ||
!ds41_norm_batch(b->norm, b->x, m, l->attn_norm, n) ||
!ds41_dspark_matmul_batch(b->qr, m, l->attn_q_a, b->norm, n, true) ||
!ds41_norm_batch(b->qr, b->qr, m, l->attn_q_a_norm, n) ||
!ds41_dspark_matmul_batch(b->q, m, l->attn_q_b, b->qr, n, true) ||
!ds4_gpu_dsv41_rope(b->q, DS4_N_HEAD_DIM, DS4_N_HEAD, n, pos, false, false) ||
!ds41_dspark_matmul_batch(b->kv, m, l->attn_kv, b->norm, n, true) ||
!ds41_norm_batch(b->kv, b->kv, m, l->attn_kv_a_norm, n) ||
!ds4_gpu_dsv41_rope(b->kv, DS4_N_HEAD_DIM, 1, n, pos, false, false) ||
!ds4_gpu_dsv41_quantize(b->kv, DS4_N_HEAD_DIM, n, DS4_V41_FP8_E8M0) ||
!ds4_gpu_tensor_copy(d->window[stage], (uint64_t)past * DS4_N_HEAD_DIM * 4u,
b->kv, 0, (uint64_t)n * DS4_N_HEAD_DIM * 4u) ||
!ds4_gpu_attention_noncausal_raw_batch_heads_tensor(b->heads, m->map, m->size,
l->attn_sinks->abs_offset, b->q, d->window[stage], n, past + n,
DS41_DRAFT_WINDOW + n, 0, DS4_N_HEAD, DS4_N_HEAD_DIM) ||
!ds4_gpu_dsv41_quantize(b->heads, DS4_N_HEAD * DS4_N_HEAD_DIM, n, DS4_V41_BF16) ||
!ds4_gpu_dsv41_rope(b->heads, DS4_N_HEAD_DIM, DS4_N_HEAD, n, pos, false, true) ||
!ds41_dspark_attention_output(b, m, l) ||
!ds41_after_attention_batch(b, m, l, n, false)) {
fprintf(stderr, "ds4: V4.1 draft attention failed\n");
return false;
}
const uint32_t experts = d->weights->n_experts, active = d->weights->n_experts_used;
const uint64_t gate_row = routed_expert_row_bytes(l->ffn_gate_exps);
const uint64_t down_row = routed_expert_row_bytes(l->ffn_down_exps);
bool mid_f16 = false;
if (!(ds41_dspark_matmul_batch(b->route_logits, m, l->ffn_gate_inp, b->norm, n, false) &&
ds4_gpu_router_select_batch_tensor(b->selected, b->route_weights, b->route_probs,
m->map, m->size, l->ffn_exp_probs_b->abs_offset, 0, 0, 0, 0, true, false,
b->route_logits, d->tokens, experts, active, DS4_EXPERT_WEIGHT_SCALE, n))) {
fprintf(stderr, "ds4: V4.1 draft routing failed\n"); return false;
}
if (!(ds41_dspark_matmul_batch(b->shared_gate, m, l->ffn_gate_shexp, b->norm, n, true) &&
ds41_dspark_matmul_batch(b->shared_up, m, l->ffn_up_shexp, b->norm, n, true) &&
ds4_gpu_swiglu_tensor(b->shared_mid, b->shared_gate, b->shared_up,
n * DS4_N_FF_EXP, DS4_SWIGLU_CLAMP_EXP, 1.0f) &&
ds4_gpu_dsv41_quantize(b->shared_mid, DS4_N_FF_EXP, n, DS4_V41_BF16) &&
ds41_dspark_matmul_batch(b->shared, m, l->ffn_down_shexp, b->shared_mid, n, true))) {
fprintf(stderr, "ds4: V4.1 draft shared expert failed\n"); return false;
}
if (!ds4_gpu_routed_moe_batch_tensor(b->routed, b->gate, b->up, b->mid, b->experts,
m->map, m->size, l->ffn_gate_exps->abs_offset, l->ffn_up_exps->abs_offset,
l->ffn_down_exps->abs_offset, l->ffn_gate_exps->type, l->ffn_down_exps->type,
gate_row * DS4_N_FF_EXP, gate_row, down_row * DS4_N_EMBD, down_row,
DS4_N_EMBD, DS4_N_FF_EXP, DS4_N_EMBD, b->selected, b->route_weights,
experts, active, DS4_SWIGLU_CLAMP_EXP, b->norm, stage, n, &mid_f16, true)) {
fprintf(stderr, "ds4: V4.1 draft routed experts failed\n"); return false;
}
return ds4_gpu_add_tensor(b->block, b->routed, b->shared, n * DS4_N_EMBD) &&
ds4_gpu_dsv41_quantize(b->block, DS4_N_EMBD, n, DS4_V41_BF16) &&
ds4_gpu_hc_expand_split_tensor(b->residual, b->block, b->after_attn,
b->ffn_split, DS4_N_EMBD, DS4_N_HC) &&
ds4_gpu_dsv41_quantize(b->residual, DS4_N_HC * DS4_N_EMBD, n, DS4_V41_BF16);
}
/* Leaves the five base-logit rows and unnormalized collapsed features on the
* GPU. Markov sampling is sequential and performed after this stage chain. */
static DS4_MAYBE_UNUSED bool ds41_dspark_forward(ds41_dspark *d, const ds4_model *target,
const ds4_weights *tw, const ds4_model *draft,
uint32_t pos, int token, ds41_gpu_graph *target_graph) {
if (!d || !pos || token < 0 || (uint32_t)token >= DS4_N_VOCAB) return false;
ds41_prefill_row *b = &d->batch;
const bool local_experts = target_graph && target_graph->tp_world == 2;
if (local_experts) ds4_gpu_tp_suspend_expert_sharding(1);
bool ok = ds4_gpu_begin_commands() && ds41_dspark_seed(d, draft, pos);
if (!ok) fprintf(stderr, "ds4: V4.1 DSpark seed failed\n");
const float pre[DS41_DRAFT_ROWS][4] = {{1, 0, 0, 0}, {1, 0, 0, 0}, {1, 0, 0, 0}, {1, 0, 0, 0}, {1, 0, 0, 0}};
if (ok) ok = ds4_gpu_tensor_write(b->pre, 0, pre, sizeof(pre));
for (uint32_t row = 0; ok && row < DS41_DRAFT_ROWS; row++) {
const uint64_t bytes = (uint64_t)DS4_N_EMBD * DS4_N_HC * sizeof(float);
ds4_gpu_tensor *out = ds4_gpu_tensor_view(b->residual, row * bytes, bytes);
ok = out && ds4_gpu_embed_token_hc_tensor(out, target->map, target->size,
tw->token_embd->abs_offset, DS4_N_VOCAB, row ? d->weights->noise_token_id : (uint32_t)token,
DS4_N_EMBD, DS4_N_HC);
ds4_gpu_tensor_free(out);
}
for (uint32_t stage = 0; ok && stage < 3; stage++) {
ok = ds41_dspark_stage(d, draft, stage, pos);
if (!ok) fprintf(stderr, "ds4: V4.1 DSpark stage %u failed\n", stage);
}
if (ok) ok = ds4_gpu_hc_weighted_sum_split_tensor(b->x, b->residual,
b->ffn_split, DS4_N_EMBD, DS4_N_HC) &&
ds4_gpu_dsv41_quantize(b->x, DS4_N_EMBD, DS41_DRAFT_ROWS, DS4_V41_BF16) &&
ds41_norm_batch(b->norm, b->x, draft, d->weights->stage[2].norm, DS41_DRAFT_ROWS);
if (ok) ok = target_graph ?
ds41_output_projection(target_graph, d->logits, target, tw, b->norm, DS41_DRAFT_ROWS) :
ds41_matmul_batch(d->logits, target, tw->output, b->norm, DS41_DRAFT_ROWS, false);
if (ds4_gpu_commands_active() && !ds4_gpu_end_commands()) ok = false;
if (local_experts) ds4_gpu_tp_suspend_expert_sharding(0);
if (!ok) d->kv_valid = false;
return ok;
}
static DS4_MAYBE_UNUSED bool ds41_dspark_markov(ds41_dspark *d, const ds4_model *m,
uint32_t row, int previous, float *confidence) {
if (!d || row >= DS41_DRAFT_ROWS || previous < 0 ||
(uint32_t)previous >= DS4_N_VOCAB || !confidence) return false;
const ds4_dspark_stage_weights *last = &d->weights->stage[2];
float embedding[256];
const uint32_t rank = d->weights->markov_rank;
if (rank != 256 || !dspark_dense_row_to_f32(embedding, m, last->markov_w1, (uint32_t)previous))
return false;
const uint64_t bytes = (uint64_t)DS4_N_VOCAB * sizeof(float);
ds4_gpu_tensor *logits = ds4_gpu_tensor_view(d->logits, row * bytes, bytes);
bool ok = logits && ds4_gpu_begin_commands() &&
ds4_gpu_tensor_write(d->markov_embed, 0, embedding, sizeof(embedding)) &&
ds41_matmul(d->markov_logits, m, last->markov_w2, d->markov_embed, false) &&
ds4_gpu_add_tensor(logits, logits, d->markov_logits, DS4_N_VOCAB) &&
ds4_gpu_tensor_copy(d->confidence_input, 0, d->batch.x,
(uint64_t)row * DS4_N_EMBD * 4u, (uint64_t)DS4_N_EMBD * 4u) &&
ds4_gpu_tensor_copy(d->confidence_input, (uint64_t)DS4_N_EMBD * 4u,
d->markov_embed, 0, rank * 4u) &&
ds41_matmul(d->confidence, m, last->confidence_proj, d->confidence_input, false);
if (ds4_gpu_commands_active() && !ds4_gpu_end_commands()) ok = false;
if (ok) ok = ds4_gpu_tensor_read(logits, 0, d->host_logits + (uint64_t)row * DS4_N_VOCAB, bytes) &&
ds4_gpu_tensor_read(d->confidence, 0, confidence, sizeof(*confidence));
ds4_gpu_tensor_free(logits);
return ok;
}
/* Cache appends beyond the accepted frontier are invisible. Only the six
* overwritten raw-ring slots and pooling carries need rollback; no compressed
* cache proportional to the context is copied. */
static DS4_MAYBE_UNUSED bool ds41_dspark_verify(ds41_dspark *d, ds41_gpu_graph *g,
const ds4_model *m, const ds4_weights *w, const int *tokens, uint32_t count) {
if (!d || !g->valid || count < 2 || count > DS41_VERIFY_ROWS ||
g->pos > g->ctx || count > g->ctx - g->pos || !ds41_tp_batch_enabled(g)) return false;
/* Prepost the complete small-row exchange window, as in V4.0 verification.
* Bulk-prefill handshakes at every attention/FFN gate are too expensive. */
const bool tp_block = g->tp_world == 2;
uint32_t gates = 2;
if (tp_block && (!g->tp_batch_out || !g->tp_batch_in || !g_tp_block_ctx ||
!ds4_tp_batch_block_begin_gates(g_tp_block_ctx, count, DS4_N_LAYER, &gates) ||
gates != 2)) return false;
const uint64_t kv = DS4_N_HEAD_DIM * 4u, feature = DS4_N_EMBD * 4u;
ds41_gpu_graph *graphs[DS41_VERIFY_ROWS];
bool ok = ds4_gpu_begin_commands();
d->verify_start = g->pos;
d->verify_count = 0;
d->rewind_end = 0;
for (uint32_t i = 0; ok && i <= DS41_VERIFY_ROWS; i++) {
if (i >= count && i != DS41_VERIFY_ROWS) continue;
const uint32_t slot = (g->pos + i) % DS41_DRAFT_WINDOW;
for (uint32_t il = 0; ok && i < count && il < 40; il++)
ok = ds4_gpu_tensor_copy(d->old_raw, (il * DS41_VERIFY_ROWS + i) * kv,
g->window[il], slot * kv, kv);
for (uint32_t stage = 0; ok && i < count && stage < 3; stage++) {
d->old_positions[stage][i] = d->positions[stage][slot];
ok = ds4_gpu_tensor_copy(d->old_features, (stage * DS41_VERIFY_ROWS + i) * feature,
d->target[stage], slot * feature, feature);
}
ds41_gpu_graph *row = &d->verified[i];
ds4_gpu_tensor *res = row->residual, *pre = row->pre, *logits = row->logits;
ds4_gpu_tensor *half = row->tp_logits_half;
ds4_gpu_tensor *pool_k[4], *pool_s[4];
memcpy(pool_k, row->verify_previous_kv, sizeof(pool_k));
memcpy(pool_s, row->verify_previous_score, sizeof(pool_s));
*row = *g;
row->residual = res ? res : ds4_gpu_tensor_view(d->verify_residual, i * DS4_N_HC * feature, DS4_N_HC * feature);
row->pre = pre ? pre : ds4_gpu_tensor_view(d->verify_pre, i * DS4_N_HC * 4u, DS4_N_HC * 4u);
row->logits = logits ? logits : ds4_gpu_tensor_view(d->verify_logits,
(uint64_t)i * DS4_N_VOCAB * 4u, (uint64_t)DS4_N_VOCAB * 4u);
row->tp_logits_half = NULL;
if (g->tp_logits_half) {
const uint64_t bytes = (uint64_t)DS4_N_VOCAB * 2u;
row->tp_logits_half = half ? half : ds4_gpu_tensor_view(row->logits,
g->tp_rank * bytes, bytes);
ok = ok && row->tp_logits_half;
}
row->verify_prefix = true;
for (uint32_t owner = 0; owner < 4; owner++) {
const uint64_t off = (i * 8u + owner * 2u) * kv;
row->verify_previous_kv[owner] = pool_k[owner] ? pool_k[owner] :
ds4_gpu_tensor_view(d->verify_pool, off, kv);
row->verify_previous_score[owner] = pool_s[owner] ? pool_s[owner] :
ds4_gpu_tensor_view(d->verify_pool, off + kv, kv);
ok = ok && row->verify_previous_kv[owner] && row->verify_previous_score[owner];
}
ok = ok && row->residual && row->pre && row->logits;
if (i < count) graphs[i] = row;
else {
/* Keep the block start as well: exact sampling can change mode
* at the first token, before any verified token is consumed. */
ok = ok && ds4_gpu_tensor_copy(row->residual, 0, g->residual, 0, DS4_N_HC * feature) &&
ds4_gpu_tensor_copy(row->pre, 0, g->pre, 0, DS4_N_HC * 4u) &&
ds4_gpu_tensor_copy(row->logits, 0, g->logits, 0, (uint64_t)DS4_N_VOCAB * 4u);
for (uint32_t owner = 0; ok && owner < 4; owner++)
ok = ds4_gpu_tensor_copy(row->verify_previous_kv[owner], 0, g->previous_kv[owner], 0, kv) &&
ds4_gpu_tensor_copy(row->verify_previous_score[owner], 0, g->previous_score[owner], 0, kv);
}
}
if (ok) ok = ds41_graph_step_batch(graphs, tokens, (int)count, count, m, w);
if (ds4_gpu_commands_active() && !ds4_gpu_end_commands()) ok = false;
if (tp_block && (!ok || !ds4_tp_batch_block_end(g_tp_block_ctx))) {
ds4_tp_mark_failed(g_tp_block_ctx);
ok = false;
}
if (ok) ok = ds4_gpu_tensor_read(d->verify_logits, 0, d->host_verified,
(uint64_t)count * DS4_N_VOCAB * 4u) &&
ds4_gpu_tensor_read(d->verify_logits, (uint64_t)DS41_VERIFY_ROWS * DS4_N_VOCAB * 4u,
d->host_verified + (uint64_t)DS41_VERIFY_ROWS * DS4_N_VOCAB, DS4_N_VOCAB * 4u);
if (ok) d->verify_count = count;
else g->valid = false;
return ok;
}
static DS4_MAYBE_UNUSED bool ds41_dspark_commit(ds41_dspark *d, ds41_gpu_graph *g,
uint32_t keep, float *logits) {
if (!d || !d->verify_count || keep > d->verify_count ||
(g->pos != d->verify_start && g->pos != d->rewind_end) ||
(d->rewind_end && d->verify_start + keep > d->rewind_end)) return false;
const uint64_t kv = DS4_N_HEAD_DIM * 4u, feature = DS4_N_EMBD * 4u;
const uint32_t row = keep ? keep - 1u : DS41_VERIFY_ROWS;
ds41_gpu_graph *last = &d->verified[row];
bool ok = ds4_gpu_begin_commands();
for (uint32_t i = keep; ok && i < d->verify_count; i++) {
const uint32_t slot = (d->verify_start + i) % DS41_DRAFT_WINDOW;
for (uint32_t il = 0; ok && il < 40; il++)
ok = ds4_gpu_tensor_copy(g->window[il], slot * kv,
d->old_raw, (il * DS41_VERIFY_ROWS + i) * kv, kv);
for (uint32_t stage = 0; ok && stage < 3; stage++) {
ok = ds4_gpu_tensor_copy(d->target[stage], slot * feature,
d->old_features, (stage * DS41_VERIFY_ROWS + i) * feature, feature);
d->positions[stage][slot] = d->old_positions[stage][i];
}
}
for (uint32_t owner = 0; ok && owner < 4; owner++)
ok = ds4_gpu_tensor_copy(g->previous_kv[owner], 0, last->verify_previous_kv[owner], 0, kv) &&
ds4_gpu_tensor_copy(g->previous_score[owner], 0, last->verify_previous_score[owner], 0, kv);
if (ok) ok = ds4_gpu_tensor_copy(g->residual, 0, last->residual, 0, DS4_N_HC * feature) &&
ds4_gpu_tensor_copy(g->pre, 0, last->pre, 0, DS4_N_HC * 4u) &&
ds4_gpu_tensor_copy(g->logits, 0, last->logits, 0, (uint64_t)DS4_N_VOCAB * 4u);
if (ds4_gpu_commands_active() && !ds4_gpu_end_commands()) ok = false;
if (ok) {
g->pos = last->pos;
g->history = last->history;
memcpy(logits, d->host_verified + (uint64_t)row * DS4_N_VOCAB, DS4_N_VOCAB * 4u);
} else g->valid = false;
d->rewind_end = ok ? g->pos : 0;
return ok;
}
#endif
ds4_deepseek41_gpu.h
+315
View File
@@ -0,0 +1,315 @@
#ifndef DS4_DEEPSEEK41_GPU_H
#define DS4_DEEPSEEK41_GPU_H
#include <stdbool.h>
#include <stdint.h>
#ifndef DS4_GPU_TENSOR_DEFINED
#define DS4_GPU_TENSOR_DEFINED
typedef struct ds4_gpu_tensor ds4_gpu_tensor;
#endif
#ifdef __cplusplus
extern "C" {
#endif
/* V4.1 activation/cache formats. Buffers are float-addressable but the
* rounded values follow the released BF16/FP8/FP4 inference graph. */
typedef enum {
DS4_V41_BF16 = 0,
DS4_V41_FP8_E8M0 = 1,
DS4_V41_FP4_E8M0 = 2,
DS4_V41_FP4_E4M3 = 3,
} ds4_v41_activation_format;
/* One-token router: probabilities, biased top-k and normalised weights in one dispatch. */
int ds4_gpu_dsv41_router_one(ds4_gpu_tensor *selected, ds4_gpu_tensor *weights,
ds4_gpu_tensor *probs, const ds4_gpu_tensor *logits,
const void *model_map, uint64_t model_size, uint64_t bias_offset,
uint32_t n_expert, uint32_t top_k, float scale);
/* The same for a few rows (Metal only), and the rows' F32 logits with the
* single-row matvec's reduction. */
int ds4_gpu_dsv41_router_rows(ds4_gpu_tensor *selected, ds4_gpu_tensor *weights,
ds4_gpu_tensor *probs, const ds4_gpu_tensor *logits,
const void *model_map, uint64_t model_size, uint64_t bias_offset,
uint32_t n_expert, uint32_t top_k, float scale, uint32_t rows);
int ds4_gpu_dsv41_matmul_f32_rows(ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
uint64_t weight_offset, uint32_t in_dim, uint32_t out_dim,
const ds4_gpu_tensor *x, uint32_t rows);
int ds4_gpu_dsv41_quantize(ds4_gpu_tensor *x, uint32_t width, uint32_t rows,
ds4_v41_activation_format format);
#if !defined(__APPLE__) && !defined(DS4_ROCM_BUILD) && !defined(DS4_NO_GPU)
/* Same reductions as the ordinary kernels, with the required BF16 store
* folded into the operation. hc_sum reads either 4- or 24-float row weights. */
int ds4_gpu_dsv41_norm(ds4_gpu_tensor *out, const ds4_gpu_tensor *x,
const void *map, uint64_t size, uint64_t offset,
uint32_t width, uint32_t rows, float eps);
int ds4_gpu_dsv41_hc_sum(ds4_gpu_tensor *out, const ds4_gpu_tensor *x,
const ds4_gpu_tensor *weights, uint32_t width, uint32_t rows,
uint32_t weight_stride);
int ds4_gpu_dsv41_hc_expand(ds4_gpu_tensor *out, const ds4_gpu_tensor *block,
const ds4_gpu_tensor *residual, const ds4_gpu_tensor *split,
uint32_t width, uint32_t rows);
/* CUDA scalar Q8 shared expert. 1: queued; 0: unsupported, no work queued;
* -1: failure. After 1, keep the input/output tensors alive and unchanged
* until join, which orders the result before subsequent main-stream work.
* Other work may run between start and join using separate tensors.
* Optional ownership picks the rank with fewer selected routed experts.
* IDs are read on the main stream before later owned-expert filtering. */
typedef struct {
const ds4_gpu_tensor *selected;
uint32_t experts, active, rank, tie_owner;
} ds4_v41_shared_owner;
int ds4_gpu_dsv41_shared_start(
ds4_gpu_tensor *out, ds4_gpu_tensor *gate, ds4_gpu_tensor *up,
ds4_gpu_tensor *mid, const ds4_gpu_tensor *x,
const void *model_map, uint64_t model_size,
uint64_t gate_offset, uint64_t up_offset, uint64_t down_offset,
uint32_t width, uint32_t hidden, float clamp,
const ds4_v41_shared_owner *owner);
int ds4_gpu_dsv41_shared_join(void);
/* Mask a computed short batch before the owned router filter mutates IDs. */
int ds4_gpu_dsv41_shared_mask(ds4_gpu_tensor *shared, const ds4_gpu_tensor *selected,
uint32_t width, uint32_t rows, uint32_t experts, uint32_t active,
uint32_t rank, uint32_t tie_owner);
#endif
/* Full-head prefill, with BF16 rounding between the two Q8 projections. */
int ds4_gpu_dsv41_attention_output_batch(
ds4_gpu_tensor *out, ds4_gpu_tensor *low,
const void *model_map, uint64_t model_size,
uint64_t out_a_offset, uint64_t out_b_offset,
const ds4_gpu_tensor *heads, uint32_t n_tokens);
/* Packed local 32-head input and BF16 low projection; output is an unrounded
* rank partial. The graph sums ranks before rounding the attention block. */
int ds4_gpu_dsv41_attention_output_tp_batch(
ds4_gpu_tensor *out, ds4_gpu_tensor *low,
const void *model_map, uint64_t model_size,
uint64_t out_a_offset, uint64_t out_b_offset,
const ds4_gpu_tensor *heads, uint32_t n_tokens, uint32_t tp_rank);
/* Adjacent-pair, unit-magnitude RoPE with the released V4.1 frequencies. */
int ds4_gpu_dsv41_rope(ds4_gpu_tensor *x, uint32_t width, uint32_t heads,
uint32_t rows, uint32_t start, bool compressed, bool inverse);
/* Compressed pairs advance two absolute token positions per stored row. */
int ds4_gpu_dsv41_rope_bf16(ds4_gpu_tensor *x, uint32_t width, uint32_t heads,
uint32_t rows, uint32_t start, bool compressed, bool inverse);
int ds4_gpu_dsv41_rope_stride(ds4_gpu_tensor *x, uint32_t width, uint32_t heads,
uint32_t rows, uint32_t start, uint32_t stride,
bool compressed, bool inverse);
int ds4_gpu_dsv41_engram_add(ds4_gpu_tensor *residual,
const ds4_gpu_tensor *kv,
const ds4_gpu_tensor *q_weight,
const ds4_gpu_tensor *k_weight,
const ds4_gpu_tensor *mask,
uint32_t width, uint32_t rows, float eps);
/* Pool complete pairs and retain the last unpaired projection in previous_*.
* start is the absolute token position, including earlier chunks. */
int ds4_gpu_dsv41_pool2(ds4_gpu_tensor *out,
const ds4_gpu_tensor *kv, const ds4_gpu_tensor *scores,
ds4_gpu_tensor *previous_kv, ds4_gpu_tensor *previous_scores,
uint32_t width, uint32_t rows, uint32_t start);
/* Candidate blocks contain eight compressed positions. Produce causal block
* maxima, pinning the newest block; filter consumes a 0/-inf block mask. */
int ds4_gpu_dsv41_candidate_blocks(ds4_gpu_tensor *blocks,
const ds4_gpu_tensor *scores,
uint32_t width, uint32_t rows,
uint32_t start, uint32_t ratio);
int ds4_gpu_dsv41_candidate_filter(ds4_gpu_tensor *scores,
const ds4_gpu_tensor *block_mask,
uint32_t width, uint32_t rows,
uint32_t start, uint32_t ratio,
uint32_t mask_stride);
/* Per-row block top-2048 with the causal block count of ratio-1 rows, and
* the 0/-inf block masks it selects at mask_stride floats per row. */
int ds4_gpu_dsv41_candidate_topk_batch(ds4_gpu_tensor *selected,
const ds4_gpu_tensor *blocks,
uint32_t width, uint32_t rows, uint32_t start);
int ds4_gpu_dsv41_candidate_mask_batch(ds4_gpu_tensor *mask,
const ds4_gpu_tensor *selected,
uint32_t width, uint32_t rows,
uint32_t mask_stride);
/* Causal index scores over ratio-1/2 compressed keys, without an extra cast
* of the already quantized FP4 queries/keys. Scores have source_rows stride. */
int ds4_gpu_dsv41_indexer_scores_batch(ds4_gpu_tensor *scores,
const ds4_gpu_tensor *q,
const ds4_gpu_tensor *weights,
const ds4_gpu_tensor *keys,
uint32_t source_rows, uint32_t rows,
uint32_t start, uint32_t ratio);
#ifndef __APPLE__
int ds4_gpu_dsv41_indexer_scores_masked_one(ds4_gpu_tensor *scores,
const ds4_gpu_tensor *q, const ds4_gpu_tensor *weights, const ds4_gpu_tensor *keys,
const ds4_gpu_tensor *block_mask, uint32_t width);
#endif
/* One-token index scores of FP4 queries/keys: the GLM scorer, plus its wide
* kernel for long rows (Metal only). */
int ds4_gpu_dsv41_indexer_score_one_tensor(ds4_gpu_tensor *scores,
const ds4_gpu_tensor *q,
const ds4_gpu_tensor *weights,
const ds4_gpu_tensor *indexer_key_cache,
uint32_t n_rows, uint32_t n_head,
uint32_t head_dim, float scale,
bool cache_f16);
/* Rows [first, first + rows) of the index-key cache into its bf16 view; with flag, a key that is not bf16-exact
* sets it nonzero. score_row: one decode index row (32 heads x 128, scale 1/64) over the cache, reading view, when
* given, where the wide scorer runs (the same bits). */
int ds4_gpu_dsv41_index_keys_bf16(ds4_gpu_tensor *view, const ds4_gpu_tensor *cache,
uint32_t first, uint32_t rows, ds4_gpu_tensor *flag);
int ds4_gpu_dsv41_indexer_score_row(ds4_gpu_tensor *scores, const ds4_gpu_tensor *q, const ds4_gpu_tensor *weights,
const ds4_gpu_tensor *cache, const ds4_gpu_tensor *view, uint32_t rows);
/* The top-k of a row filtered by a block mask of more than 2048 blocks, from
* the scores of its 2048 candidate blocks only: the same ids in the same order
* (Metal only). Returns 0, leaving the full row to the caller, where unavailable. */
int ds4_gpu_dsv41_indexer_candidates_tensor(ds4_gpu_tensor *selected, ds4_gpu_tensor *scores,
ds4_gpu_tensor *list, const ds4_gpu_tensor *q,
const ds4_gpu_tensor *weights, const ds4_gpu_tensor *keys,
const ds4_gpu_tensor *block_mask, uint32_t n_comp,
float scale, uint32_t top_k, int list_ready);
/* list_ready != 0: list already holds block_mask's candidates (built by an earlier call since the mask was written),
* so it is not rebuilt. */
int ds4_gpu_dsv41_tensor_ops_available(void);
/* Reuse exact BF16 views of FP4 queries/keys across score tiles. Invalid
* casts retain the F32 arithmetic for the affected tile. */
uint64_t ds4_gpu_dsv41_indexer_packed_bytes(uint32_t source_rows, uint32_t rows);
int ds4_gpu_dsv41_indexer_pack(ds4_gpu_tensor *packed,
const ds4_gpu_tensor *q, const ds4_gpu_tensor *keys,
uint32_t source_rows, uint32_t rows);
int ds4_gpu_dsv41_indexer_scores_packed(ds4_gpu_tensor *scores,
const ds4_gpu_tensor *q,
const ds4_gpu_tensor *weights,
const ds4_gpu_tensor *keys,
const ds4_gpu_tensor *packed,
uint32_t source_rows, uint32_t rows,
uint32_t start, uint32_t ratio,
uint32_t packed_rows, uint32_t offset);
/* Exact row-sort ordering with independent causal widths; more than 512
* visible keys per row. Output stride is 512 indices. */
int ds4_gpu_dsv41_indexer_topk_batch(ds4_gpu_tensor *selected,
const ds4_gpu_tensor *scores,
uint32_t width, uint32_t rows,
uint32_t start, uint32_t ratio);
/* The indexer top-k, plus the radix select for wide rows (Metal); other models
* keep the merge ladder and its tie order. */
int ds4_gpu_dsv41_indexer_topk_tensor(ds4_gpu_tensor *selected,
const ds4_gpu_tensor *scores,
uint32_t n_comp, uint32_t n_tokens,
uint32_t top_k);
/* Rows with at most 512 visible keys select all of them, in key order. */
int ds4_gpu_dsv41_indexer_all_batch(ds4_gpu_tensor *selected, uint32_t rows,
uint32_t start, uint32_t ratio);
enum { DS4_V41_CARRY_BF16, DS4_V41_CARRY_MASK, DS4_V41_CARRY_F32 };
/* Lossless storage for already-BF16 activations or 0/-inf candidate masks.
* Packed rows are padded to whole uint32_t words. Plain rows remain F32. */
int ds4_gpu_dsv41_carry_copy(ds4_gpu_tensor *packed, uint32_t row_offset,
ds4_gpu_tensor *plain, uint32_t width, uint32_t rows,
uint32_t format, bool pack);
/* Batched F16 projections with the same arithmetic as individual matvecs. */
int ds4_gpu_dsv41_projection_rows(ds4_gpu_tensor *out,
const void *model_map, uint64_t model_size,
uint64_t weight_offset, uint32_t width,
uint32_t outputs, uint32_t rows,
const ds4_gpu_tensor *in);
/* The HC mixer projection of one row with its norm fused (0: not fused). */
int ds4_gpu_dsv41_hc_project(ds4_gpu_tensor *mix, const ds4_gpu_tensor *residual,
const void *model_map, uint64_t model_size, uint64_t fn_offset,
uint32_t n, uint32_t mix_dim, float eps);
/* One block's whole HC input in one dispatch: the mixer projection with its
* norm, then split, weighted sum with `pre`'s first n_hc weights and norm
* (0: not fused). */
/* The same for a few rows: `pre` advances pre_stride floats per row. */
int ds4_gpu_dsv41_hc_block_input_rows(ds4_gpu_tensor *mix, ds4_gpu_tensor *x, ds4_gpu_tensor *norm,
ds4_gpu_tensor *split, const ds4_gpu_tensor *stream,
const ds4_gpu_tensor *pre, const void *model_map, uint64_t model_size,
uint64_t fn_offset, uint64_t scale_offset, uint64_t base_offset,
uint64_t norm_offset, uint32_t n, uint32_t mix_dim, uint32_t n_embd,
uint32_t n_hc, uint32_t sinkhorn_iters, float hc_eps, float norm_eps,
uint32_t rows, uint32_t pre_stride);
/* One block's input: split, weighted sum with `pre`'s first n_hc weights, norm. */
int ds4_gpu_dsv41_hc_input(ds4_gpu_tensor *x, ds4_gpu_tensor *norm, ds4_gpu_tensor *split,
const ds4_gpu_tensor *mix, const ds4_gpu_tensor *residual,
const ds4_gpu_tensor *pre, const void *model_map, uint64_t model_size,
uint64_t scale_offset, uint64_t base_offset, uint64_t norm_offset,
uint32_t n_embd, uint32_t n_hc, uint32_t sinkhorn_iters,
float hc_eps, float norm_eps);
/* Two Q8_0 projections of one row, outputs rounded to bf16 (0: run them apart). */
int ds4_gpu_dsv41_project_pair_q8(ds4_gpu_tensor *out_a, ds4_gpu_tensor *out_b,
const void *model_map, uint64_t model_size,
uint64_t offset_a, uint64_t offset_b, uint32_t in_dim,
uint32_t out_a_dim, uint32_t out_b_dim, const ds4_gpu_tensor *x);
/* The shared expert's SwiGLU input from the Q8_0 gate and up weights, as the
* bf16 matvecs and the SwiGLU pass would produce it; gate/up are scratch. */
int ds4_gpu_dsv41_shared_swiglu(ds4_gpu_tensor *mid, ds4_gpu_tensor *gate, ds4_gpu_tensor *up,
const void *model_map, uint64_t model_size, uint64_t gate_offset,
uint64_t up_offset, uint32_t in_dim, uint32_t out_dim,
const ds4_gpu_tensor *x, float clamp, uint32_t rows);
/* Rope, quantize and store one row into dst at dst_offset bytes; x may be
* left roped and quantized in place. */
int ds4_gpu_dsv41_rope_quantize(const ds4_gpu_tensor *x, ds4_gpu_tensor *dst, uint64_t dst_offset,
uint32_t width, uint32_t start, bool compressed,
ds4_v41_activation_format format);
/* One row's F16 projection rounded to bf16 in the kernel (0: not fused). */
int ds4_gpu_dsv41_project_f16_bf16(ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
uint64_t weight_offset, uint32_t in_dim, uint32_t out_dim,
const ds4_gpu_tensor *x);
/* Decode attention over the raw ring and the compressed rows selected by
* ids, staged in one pass (0: not available; `selected` is gather scratch). */
int ds4_gpu_dsv41_attention_decode(ds4_gpu_tensor *heads, const void *model_map, uint64_t model_size,
uint64_t sinks_offset, const ds4_gpu_tensor *q,
const ds4_gpu_tensor *raw_kv, uint32_t n_raw, uint32_t raw_cap,
uint32_t raw_start, const ds4_gpu_tensor *comp_cache,
const ds4_gpu_tensor *ids, ds4_gpu_tensor *selected,
uint32_t n_comp, uint32_t attended, uint32_t n_head, uint32_t head_dim);
/* Single-Mac decode: the block input's collapse and norm now, then its hyper-connection weights inside the query
* projection's dispatch, which also ropes and stores the layer's KV row. 0 when DS4_GPU_V41_DEFERRED_HC is off. */
int ds4_gpu_dsv41_deferred_collapse(ds4_gpu_tensor *x, ds4_gpu_tensor *norm,
const ds4_gpu_tensor *residual, const ds4_gpu_tensor *pre,
const void *model_map, uint64_t model_size, uint64_t norm_offset, float eps);
int ds4_gpu_dsv41_query_deferred_hc(ds4_gpu_tensor *q, ds4_gpu_tensor *mix, ds4_gpu_tensor *split,
const ds4_gpu_tensor *qr, const ds4_gpu_tensor *residual,
const void *model_map, uint64_t model_size, uint64_t q_offset, uint64_t fn_offset,
uint64_t scale_offset, uint64_t base_offset, uint32_t pos, bool compressed,
uint32_t iters, float hc_eps, float norm_eps, ds4_gpu_tensor *x, ds4_gpu_tensor *norm,
const ds4_gpu_tensor *pre, uint64_t norm_offset,
const ds4_gpu_tensor *kv, ds4_gpu_tensor *cache, uint64_t cache_offset);
/* The Q8_0 q projection of one row with the layer's rope on its outputs. */
int ds4_gpu_dsv41_project_q(ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
uint64_t weight_offset, uint32_t in_dim, uint32_t out_dim,
const ds4_gpu_tensor *x, uint32_t pos, bool compressed);
/* A bf16 Q8_0 matvec of one row expanded straight into the n_hc residual
* streams; `add` (optional) joins the row before the expand. */
int ds4_gpu_dsv41_matmul_expand_rows(ds4_gpu_tensor *out_hc, const void *model_map, uint64_t model_size,
uint64_t weight_offset, uint32_t in_dim, uint32_t out_dim,
const ds4_gpu_tensor *x, const ds4_gpu_tensor *add,
const ds4_gpu_tensor *residual_hc, const ds4_gpu_tensor *split,
uint32_t n_hc, uint32_t rows);
/* The attention output low projection of one row with the heads' inverse
* rope (and bf16 rounding) folded into the load; heads may be roped in place. */
int ds4_gpu_dsv41_attention_low(ds4_gpu_tensor *low, const void *model_map, uint64_t model_size,
uint64_t out_a_offset, uint32_t group_dim, uint32_t rank,
uint32_t n_groups, ds4_gpu_tensor *heads, uint32_t pos, bool compressed);
/* Metal: one decode step's RoPE coefficient table (64 float2: (cos, sin) of float(pos) * frequency, entries 0..31
* raw, 32..63 compressed). rope_quantize_heads_tab: rope + quantization of heads rows of width in place, in one
* dispatch reading it. heads_once_arm: the next single-row decode reduce of 64 heads x 512 writes its heads already
* bf16-rounded and inverse-roped with the table; take returns 1 when it did (and disarms), and attention_low_plain
* then projects them. reduce_check (test entry): that reduce on tmp's partials, plain or transformed. */
int ds4_gpu_dsv41_rope_coeff_table(ds4_gpu_tensor *tab, uint32_t pos);
int ds4_gpu_dsv41_rope_quantize_heads_tab(ds4_gpu_tensor *x, uint32_t width, uint32_t heads, uint32_t start,
bool compressed, ds4_v41_activation_format format, const ds4_gpu_tensor *tab);
void ds4_gpu_dsv41_heads_once_arm(const ds4_gpu_tensor *tab, bool compressed);
int ds4_gpu_dsv41_heads_once_take(void);
int ds4_gpu_dsv41_attention_low_plain(ds4_gpu_tensor *low, const void *model_map, uint64_t model_size,
uint64_t out_a_offset, uint32_t group_dim, uint32_t rank,
uint32_t n_groups, const ds4_gpu_tensor *heads);
int ds4_gpu_dsv41_flash_attn_reduce_check(ds4_gpu_tensor *heads, const ds4_gpu_tensor *tmp, uint32_t nrows,
int transform, const ds4_gpu_tensor *tab, bool compressed);
/* Two weighted RMS norms rounded to bf16 in one dispatch. */
int ds4_gpu_dsv41_norm_pair_rows(ds4_gpu_tensor *out0, const ds4_gpu_tensor *x0, uint64_t weight0_offset, uint32_t n0,
ds4_gpu_tensor *out1, const ds4_gpu_tensor *x1, uint64_t weight1_offset, uint32_t n1,
const void *model_map, uint64_t model_size, float eps, uint32_t rows);
/* Gather 512-wide F32 KV rows; IDs must come from top-k over source_rows. */
int ds4_gpu_dsv41_gather_kv(ds4_gpu_tensor *out, const ds4_gpu_tensor *source,
const ds4_gpu_tensor *ids, uint32_t source_rows,
uint32_t selected_rows);
#ifdef __cplusplus
}
#endif
#endif
ds4_deepseek4_vision_gpu.cuh
+444
View File
@@ -0,0 +1,444 @@
/* DeepSeek V4 Flash Vision-Exp operations shared by CUDA and ROCm. */
#ifndef DS4_DEEPSEEK4_VISION_TYPES_DEFINED
#define DS4_DEEPSEEK4_VISION_TYPES_DEFINED
#define DS4_DEEPSEEK4_VISION_LAYERS 32u
#define DS4_DEEPSEEK4_LANGUAGE_LAYERS 43u
#define DS4_DEEPSEEK4_MTP_LAYERS 3u
typedef struct {
uint64_t norm1;
uint64_t qkv_weight;
uint64_t qkv_bias;
uint64_t attn_proj_weight;
uint64_t attn_proj_bias;
uint64_t norm2;
uint64_t mlp_w1;
uint64_t mlp_w2;
} ds4_deepseek4_vision_layer_weights;
typedef struct {
uint64_t patch_weight;
uint64_t patch_bias;
uint64_t post_norm;
uint64_t aligner_w1;
uint64_t aligner_w1_bias;
uint64_t aligner_w2;
uint64_t aligner_w2_bias;
uint64_t image_start;
uint64_t image_pad;
uint64_t image_newline;
uint64_t image_end;
uint64_t visual_router_bias[DS4_DEEPSEEK4_LANGUAGE_LAYERS];
uint64_t mtp_visual_router_bias[DS4_DEEPSEEK4_MTP_LAYERS];
uint64_t hash_router_bias[3];
ds4_deepseek4_vision_layer_weights layer[DS4_DEEPSEEK4_VISION_LAYERS];
} ds4_deepseek4_vision_weights;
#endif
#ifndef DS4_DEEPSEEK4_VISION_STREAM
#define DS4_DEEPSEEK4_VISION_STREAM 0
#endif
__device__ __forceinline__ static float deepseek4_vision_round_bf16_dev(
float value) {
uint32_t bits = __float_as_uint(value);
if ((bits & 0x7f800000u) == 0x7f800000u) return value;
bits += 0x00007fffu + ((bits >> 16u) & 1u);
return __uint_as_float(bits & 0xffff0000u);
}
__global__ static void deepseek4_vision_round_bf16_kernel(
float *x,
uint64_t count) {
const uint64_t i = (uint64_t)blockIdx.x * blockDim.x + threadIdx.x;
if (i < count) x[i] = deepseek4_vision_round_bf16_dev(x[i]);
}
__global__ static void deepseek4_vision_qkv_rope_kernel(
float *q,
float *k,
float *v,
const float *qkv,
const uint16_t *bias,
uint32_t rows,
uint32_t grid_w) {
const uint32_t row = blockIdx.x;
const uint32_t head = blockIdx.y;
const uint32_t lane = threadIdx.x;
if (row >= rows || head >= 16u || lane >= 32u) return;
const uint64_t qkv_base = (uint64_t)row * 3072u +
(uint64_t)head * 64u;
const uint64_t out_base = (uint64_t)row * 1024u +
(uint64_t)head * 64u;
const float q0 = deepseek4_vision_round_bf16_dev(
qkv[qkv_base + lane] +
glm53_vision_bf16(bias + (uint64_t)head * 64u + lane));
const float q1 = deepseek4_vision_round_bf16_dev(
qkv[qkv_base + lane + 32u] +
glm53_vision_bf16(bias + (uint64_t)head * 64u + lane + 32u));
const float k0 = deepseek4_vision_round_bf16_dev(
qkv[qkv_base + 1024u + lane] +
glm53_vision_bf16(bias + 1024u +
(uint64_t)head * 64u + lane));
const float k1 = deepseek4_vision_round_bf16_dev(
qkv[qkv_base + 1024u + lane + 32u] +
glm53_vision_bf16(bias + 1024u +
(uint64_t)head * 64u + lane + 32u));
const uint32_t y = row / grid_w;
const uint32_t x = row - y * grid_w;
const uint32_t pos = lane < 16u ? y : x;
const uint32_t freq = lane & 15u;
const float inv_freq = powf(10000.0f, -(float)freq / 16.0f);
const float angle = (float)pos * inv_freq;
const float cs = cosf(angle);
const float sn = sinf(angle);
q[out_base + lane] = deepseek4_vision_round_bf16_dev(q0 * cs - q1 * sn);
q[out_base + lane + 32u] =
deepseek4_vision_round_bf16_dev(q1 * cs + q0 * sn);
k[out_base + lane] = deepseek4_vision_round_bf16_dev(k0 * cs - k1 * sn);
k[out_base + lane + 32u] =
deepseek4_vision_round_bf16_dev(k1 * cs + k0 * sn);
v[out_base + lane] = deepseek4_vision_round_bf16_dev(
qkv[qkv_base + 2048u + lane] +
glm53_vision_bf16(bias + 2048u +
(uint64_t)head * 64u + lane));
v[out_base + lane + 32u] = deepseek4_vision_round_bf16_dev(
qkv[qkv_base + 2048u + lane + 32u] +
glm53_vision_bf16(bias + 2048u +
(uint64_t)head * 64u + lane + 32u));
}
__global__ static void deepseek4_vision_swiglu_split_kernel(
float *out,
const float *gate_up,
uint64_t count,
uint32_t width) {
const uint64_t i = (uint64_t)blockIdx.x * blockDim.x + threadIdx.x;
if (i >= count) return;
const uint32_t d = i % width;
const uint64_t row = i / width;
const uint64_t source = row * width * 2u + d;
const float gate = gate_up[source];
const float up = gate_up[source + width];
out[i] = deepseek4_vision_round_bf16_dev(
(gate / (1.0f + expf(-gate))) * up);
}
__global__ static void deepseek4_vision_add_residual_kernel(
float *x,
const float *residual,
uint64_t count) {
const uint64_t i = (uint64_t)blockIdx.x * blockDim.x + threadIdx.x;
if (i < count)
x[i] = deepseek4_vision_round_bf16_dev(x[i] + residual[i]);
}
__global__ static void deepseek4_vision_aligner_reorder_kernel(
float *out,
const float *x,
uint32_t grid_h,
uint32_t grid_w,
uint32_t output_rows) {
const uint32_t d = blockIdx.x * blockDim.x + threadIdx.x;
const uint32_t row = blockIdx.y;
if (d >= 9216u || row >= output_rows) return;
const uint32_t output_w = (grid_w + 2u) / 3u;
const uint32_t block_y = row / output_w;
const uint32_t block_x = row - block_y * output_w;
const uint32_t channel = d / 9u;
const uint32_t within = d - channel * 9u;
const uint32_t source_y = block_y * 3u + within / 3u;
const uint32_t source_x = block_x * 3u + within % 3u;
float value = 0.0f;
if (source_y < grid_h && source_x < grid_w) {
const uint64_t source_row = (uint64_t)source_y * grid_w + source_x;
value = x[source_row * 1024u + channel];
}
out[(uint64_t)row * 9216u + d] = value;
}
__global__ static void deepseek4_vision_gelu_bias_kernel(
float *out,
const float *x,
const uint16_t *bias,
uint64_t count,
uint32_t width) {
const uint64_t i = (uint64_t)blockIdx.x * blockDim.x + threadIdx.x;
if (i >= count) return;
const uint32_t d = i % width;
const float value = deepseek4_vision_round_bf16_dev(
x[i] + glm53_vision_bf16(bias + d));
out[i] = deepseek4_vision_round_bf16_dev(
0.5f * value *
(1.0f + glm53_vision_erf_approx(
value * 0.7071067811865475f)));
}
static int deepseek4_vision_round_tensor(
ds4_gpu_tensor *x,
uint64_t count,
const char *label) {
deepseek4_vision_round_bf16_kernel<<<
(unsigned)((count + 255u) / 256u), 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>((float *)x->ptr, count);
return glm53_vision_launch_ok(label);
}
extern "C" int ds4_gpu_deepseek4_vision_encode(
float *out,
const float *patches,
uint32_t grid_h,
uint32_t grid_w,
const void *model_map,
uint64_t model_size,
const ds4_deepseek4_vision_weights *weights) {
if (!out || !patches || !model_map || !weights || grid_h == 0u ||
grid_w == 0u || grid_h > UINT32_MAX / grid_w) return 0;
const uint32_t rows = grid_h * grid_w;
const uint32_t aligned_rows =
((grid_h + 2u) / 3u) * ((grid_w + 2u) / 3u);
const uint64_t row1024 = (uint64_t)rows * 1024u;
const uint64_t row2816 = (uint64_t)rows * 2816u;
const uint64_t row3072 = (uint64_t)rows * 3072u;
const uint64_t row5632 = (uint64_t)rows * 5632u;
const uint64_t aligned4096 = (uint64_t)aligned_rows * 4096u;
const uint64_t aligned9216 = (uint64_t)aligned_rows * 9216u;
if (row5632 > SIZE_MAX / sizeof(float) ||
aligned9216 > SIZE_MAX / sizeof(float)) return 0;
ds4_gpu_tensor *patch = NULL, *a = NULL, *b = NULL, *qkv = NULL;
ds4_gpu_tensor *q = NULL, *k = NULL, *v = NULL, *attn = NULL;
ds4_gpu_tensor *mlp_w1 = NULL, *mlp_mid = NULL;
ds4_gpu_tensor *align_in = NULL, *align_a = NULL, *align_b = NULL;
ds4_gpu_tensor *cur = NULL, *tmp = NULL;
const uint16_t *bias = NULL;
int ok = 0;
#define DSV4_VISION_ALLOC(name_, count_) do { \
name_ = ds4_gpu_tensor_alloc((count_) * sizeof(float)); \
if (!(name_)) goto cleanup; \
} while (0)
DSV4_VISION_ALLOC(patch, (uint64_t)rows * 588u);
DSV4_VISION_ALLOC(a, row1024);
DSV4_VISION_ALLOC(b, row1024);
DSV4_VISION_ALLOC(qkv, row3072);
DSV4_VISION_ALLOC(q, row1024);
DSV4_VISION_ALLOC(k, row1024);
DSV4_VISION_ALLOC(v, row1024);
DSV4_VISION_ALLOC(attn, row1024);
DSV4_VISION_ALLOC(mlp_w1, row5632);
DSV4_VISION_ALLOC(mlp_mid, row2816);
DSV4_VISION_ALLOC(align_in, aligned9216);
DSV4_VISION_ALLOC(align_a, aligned4096);
DSV4_VISION_ALLOC(align_b, aligned4096);
#undef DSV4_VISION_ALLOC
if (!ds4_gpu_tensor_write(
patch, 0, patches,
(uint64_t)rows * 588u * sizeof(float)) ||
!ds4_gpu_begin_commands()) goto cleanup;
ok = ds4_gpu_glm53_matmul_bf16(
a, model_map, model_size, weights->patch_weight,
588u, 1024u, patch, rows);
if (ok) {
bias = glm53_vision_weight(model_map, model_size,
weights->patch_bias, 1024u, "DeepSeek vision patch bias");
if (!bias) ok = 0;
}
if (ok) {
glm53_vision_bias_kernel<<<
(unsigned)((row1024 + 255u) / 256u), 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)a->ptr, bias, NULL, row1024, 1024u);
ok = glm53_vision_launch_ok("DeepSeek vision patch bias");
}
if (ok) ok = deepseek4_vision_round_tensor(
a, row1024, "DeepSeek vision patch round");
cur = a;
tmp = b;
for (uint32_t il = 0; ok && il < DS4_DEEPSEEK4_VISION_LAYERS; il++) {
const ds4_deepseek4_vision_layer_weights *w = &weights->layer[il];
const uint16_t *norm = glm53_vision_weight(
model_map, model_size, w->norm1, 1024u,
"DeepSeek vision norm1");
if (!norm) { ok = 0; break; }
glm53_vision_rms_kernel<<<rows, 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)tmp->ptr, (const float *)cur->ptr,
norm, 1024u, 1.0e-6f);
ok = glm53_vision_launch_ok("DeepSeek vision norm1");
if (ok) ok = deepseek4_vision_round_tensor(
tmp, row1024, "DeepSeek vision norm1 round");
if (ok) ok = ds4_gpu_glm53_matmul_bf16(
qkv, model_map, model_size, w->qkv_weight,
1024u, 3072u, tmp, rows);
const uint16_t *qkv_bias = NULL;
if (ok) {
qkv_bias = glm53_vision_weight(
model_map, model_size, w->qkv_bias, 3072u,
"DeepSeek vision QKV bias");
if (!qkv_bias) ok = 0;
}
if (ok) {
deepseek4_vision_qkv_rope_kernel<<<
dim3(rows, 16u, 1u), 32u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)q->ptr, (float *)k->ptr, (float *)v->ptr,
(const float *)qkv->ptr, qkv_bias, rows, grid_w);
ok = glm53_vision_launch_ok("DeepSeek vision QKV");
}
if (ok) {
glm53_vision_attention_kernel<<<
dim3(rows, 16u, 1u), 32u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)attn->ptr, (const float *)q->ptr,
(const float *)k->ptr, (const float *)v->ptr, rows);
ok = glm53_vision_launch_ok("DeepSeek vision attention");
}
if (ok) ok = deepseek4_vision_round_tensor(
attn, row1024, "DeepSeek vision attention round");
if (ok) ok = ds4_gpu_glm53_matmul_bf16(
tmp, model_map, model_size, w->attn_proj_weight,
1024u, 1024u, attn, rows);
if (ok) {
bias = glm53_vision_weight(
model_map, model_size, w->attn_proj_bias, 1024u,
"DeepSeek vision attention bias");
if (!bias) ok = 0;
}
if (ok) {
glm53_vision_bias_kernel<<<
(unsigned)((row1024 + 255u) / 256u), 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)tmp->ptr, bias, (const float *)cur->ptr,
row1024, 1024u);
ok = glm53_vision_launch_ok(
"DeepSeek vision attention residual");
}
if (ok) ok = deepseek4_vision_round_tensor(
tmp, row1024, "DeepSeek vision attention residual round");
ds4_gpu_tensor *swap = cur; cur = tmp; tmp = swap;
if (ok) {
norm = glm53_vision_weight(
model_map, model_size, w->norm2, 1024u,
"DeepSeek vision norm2");
if (!norm) ok = 0;
}
if (ok) {
glm53_vision_rms_kernel<<<rows, 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)tmp->ptr, (const float *)cur->ptr,
norm, 1024u, 1.0e-6f);
ok = glm53_vision_launch_ok("DeepSeek vision norm2");
}
if (ok) ok = deepseek4_vision_round_tensor(
tmp, row1024, "DeepSeek vision norm2 round");
if (ok) ok = ds4_gpu_glm53_matmul_bf16(
mlp_w1, model_map, model_size, w->mlp_w1,
1024u, 5632u, tmp, rows);
if (ok) ok = deepseek4_vision_round_tensor(
mlp_w1, row5632, "DeepSeek vision MLP input round");
if (ok) {
deepseek4_vision_swiglu_split_kernel<<<
(unsigned)((row2816 + 255u) / 256u), 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)mlp_mid->ptr, (const float *)mlp_w1->ptr,
row2816, 2816u);
ok = glm53_vision_launch_ok("DeepSeek vision SwiGLU");
}
if (ok) ok = ds4_gpu_glm53_matmul_bf16(
tmp, model_map, model_size, w->mlp_w2,
2816u, 1024u, mlp_mid, rows);
if (ok) ok = deepseek4_vision_round_tensor(
tmp, row1024, "DeepSeek vision MLP output round");
if (ok) {
deepseek4_vision_add_residual_kernel<<<
(unsigned)((row1024 + 255u) / 256u), 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)tmp->ptr, (const float *)cur->ptr, row1024);
ok = glm53_vision_launch_ok("DeepSeek vision MLP residual");
}
swap = cur; cur = tmp; tmp = swap;
}
if (ok) {
const uint16_t *norm = glm53_vision_weight(
model_map, model_size, weights->post_norm, 1024u,
"DeepSeek vision final norm");
if (!norm) ok = 0;
else {
glm53_vision_rms_kernel<<<rows, 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)tmp->ptr, (const float *)cur->ptr,
norm, 1024u, 1.0e-6f);
ok = glm53_vision_launch_ok("DeepSeek vision final norm");
}
}
if (ok) ok = deepseek4_vision_round_tensor(
tmp, row1024, "DeepSeek vision final norm round");
if (ok) {
dim3 grid((9216u + 255u) / 256u, aligned_rows, 1u);
deepseek4_vision_aligner_reorder_kernel<<<
grid, 256u, 0, DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)align_in->ptr, (const float *)tmp->ptr,
grid_h, grid_w, aligned_rows);
ok = glm53_vision_launch_ok("DeepSeek vision aligner reorder");
}
if (ok) ok = ds4_gpu_glm53_matmul_bf16(
align_a, model_map, model_size, weights->aligner_w1,
9216u, 4096u, align_in, aligned_rows);
if (ok) {
bias = glm53_vision_weight(
model_map, model_size, weights->aligner_w1_bias, 4096u,
"DeepSeek vision aligner hidden bias");
if (!bias) ok = 0;
}
if (ok) {
deepseek4_vision_gelu_bias_kernel<<<
(unsigned)((aligned4096 + 255u) / 256u), 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)align_b->ptr, (const float *)align_a->ptr,
bias, aligned4096, 4096u);
ok = glm53_vision_launch_ok("DeepSeek vision aligner GELU");
}
if (ok) ok = ds4_gpu_glm53_matmul_bf16(
align_a, model_map, model_size, weights->aligner_w2,
4096u, 4096u, align_b, aligned_rows);
if (ok) {
bias = glm53_vision_weight(
model_map, model_size, weights->aligner_w2_bias, 4096u,
"DeepSeek vision aligner output bias");
if (!bias) ok = 0;
}
if (ok) {
glm53_vision_bias_kernel<<<
(unsigned)((aligned4096 + 255u) / 256u), 256u, 0,
DS4_DEEPSEEK4_VISION_STREAM>>>(
(float *)align_a->ptr, bias, NULL, aligned4096, 4096u);
ok = glm53_vision_launch_ok(
"DeepSeek vision aligner output bias");
}
if (ok) ok = deepseek4_vision_round_tensor(
align_a, aligned4096, "DeepSeek vision aligner output round");
if (ds4_gpu_end_commands() == 0) ok = 0;
if (ok) ok = ds4_gpu_tensor_read(
align_a, 0, out, aligned4096 * sizeof(float));
cleanup:
ds4_gpu_tensor_free(align_b);
ds4_gpu_tensor_free(align_a);
ds4_gpu_tensor_free(align_in);
ds4_gpu_tensor_free(mlp_mid);
ds4_gpu_tensor_free(mlp_w1);
ds4_gpu_tensor_free(attn);
ds4_gpu_tensor_free(v);
ds4_gpu_tensor_free(k);
ds4_gpu_tensor_free(q);
ds4_gpu_tensor_free(qkv);
ds4_gpu_tensor_free(b);
ds4_gpu_tensor_free(a);
ds4_gpu_tensor_free(patch);
return ok;
}
#undef DS4_DEEPSEEK4_VISION_STREAM
ds4_engram.c
+680
View File
@@ -0,0 +1,680 @@
#define _DARWIN_C_SOURCE
#define _POSIX_C_SOURCE 200809L
#include "ds4_engram.h"
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <math.h>
#include <pthread.h>
#include <stdlib.h>
#include <string.h>
#include <stdatomic.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#ifdef __APPLE__
#include <dispatch/dispatch.h>
#include <pthread/qos.h>
#include <sys/sysctl.h>
#endif
bool ds4_engram_layout_valid(const ds4_engram_layout *l) {
if (!l || !l->token_map || !l->vocab_size ||
!l->compressed_vocab_size || l->compressed_vocab_size > INT32_MAX ||
l->pad_id >= l->compressed_vocab_size) return false;
for (uint32_t i = 0; i < l->vocab_size; i++)
if (l->token_map[i] >= l->compressed_vocab_size) return false;
for (int layer = 0; layer < DS4_ENGRAM_LAYERS; layer++) {
for (int i = 0; i < DS4_ENGRAM_NGRAM; i++) {
uint64_t m = l->multipliers[layer][i];
if (!(m & 1) || m > (uint64_t)INT64_MAX / l->compressed_vocab_size)
return false;
}
uint64_t total = 0;
for (int i = 0; i < DS4_ENGRAM_COLS; i++) {
if (l->primes[layer][i] < 2) return false;
total += l->primes[layer][i];
}
if (total != l->rows[layer]) return false;
}
return true;
}
void ds4_engram_history_reset(ds4_engram_history *h) {
for (int i = 0; i < DS4_ENGRAM_NGRAM - 1; i++) h->tail[i] = DS4_ENGRAM_DEAD;
}
bool ds4_engram_hash(const ds4_engram_layout *l, ds4_engram_history *h,
const int *tokens, const uint8_t *mask, size_t count,
uint32_t *rows) {
if (!l || !h || !l->token_map || (count && (!tokens || !rows)) ||
count > SIZE_MAX / (DS4_ENGRAM_LAYERS * DS4_ENGRAM_COLS * sizeof(*rows)))
return false;
for (int i = 0; i < DS4_ENGRAM_NGRAM - 1; i++) {
if (h->tail[i] < DS4_ENGRAM_DEAD ||
(h->tail[i] >= 0 && (uint32_t)h->tail[i] >= l->compressed_vocab_size))
return false;
}
for (size_t i = 0; i < count; i++) {
if (tokens[i] < 0 || (uint32_t)tokens[i] >= l->vocab_size) return false;
}
for (size_t i = 0; i < count; i++) {
int32_t current = mask && !mask[i] ? DS4_ENGRAM_DEAD :
(int32_t)l->token_map[tokens[i]];
uint32_t ids[DS4_ENGRAM_NGRAM];
bool blocked = false;
for (int j = 0; j < DS4_ENGRAM_NGRAM; j++) {
int32_t id = j ? h->tail[j - 1] : current;
blocked |= id == DS4_ENGRAM_DEAD;
ids[j] = blocked ? l->pad_id : (uint32_t)id;
}
for (int layer = 0; layer < DS4_ENGRAM_LAYERS; layer++) {
uint64_t hash = (uint64_t)ids[0] * l->multipliers[layer][0];
uint32_t offset = 0;
for (int j = 1; j < DS4_ENGRAM_NGRAM; j++) {
hash ^= (uint64_t)ids[j] * l->multipliers[layer][j];
for (int head = 0; head < DS4_ENGRAM_HEADS; head++) {
int col = (j - 1) * DS4_ENGRAM_HEADS + head;
uint32_t prime = l->primes[layer][col];
*rows++ = (uint32_t)(hash % prime) + offset;
offset += prime;
}
}
}
for (int j = DS4_ENGRAM_NGRAM - 2; j > 0; j--) h->tail[j] = h->tail[j - 1];
h->tail[0] = current;
}
return true;
}
bool ds4_engram_table_open(ds4_engram_table *t, const char *path,
uint64_t offset, uint32_t rows) {
if (!t) return false;
*t = (ds4_engram_table){.fd = -1};
uint64_t bytes = (uint64_t)rows * DS4_ENGRAM_ROW_BYTES;
if (!path || !rows || offset > INT64_MAX || bytes > INT64_MAX - offset) {
errno = EINVAL;
return false;
}
int fd = open(path, O_RDONLY | O_CLOEXEC);
if (fd < 0) return false;
struct stat st;
if (fstat(fd, &st) != 0) goto fail;
if (!S_ISREG(st.st_mode) || st.st_size < 0 || offset + bytes > (uint64_t)st.st_size) {
errno = EINVAL;
goto fail;
}
#ifdef __APPLE__
if (fcntl(fd, F_NOCACHE, 1) != 0 || fcntl(fd, F_RDAHEAD, 0) != 0) goto fail;
#elif defined(POSIX_FADV_RANDOM)
/* Hashed rows are scattered across the table. Read-ahead fetches unused
* pages and competes with the model's limited memory headroom. */
(void)posix_fadvise(fd, (off_t)offset, (off_t)bytes, POSIX_FADV_RANDOM);
#endif
*t = (ds4_engram_table){.fd = fd, .offset = offset, .rows = rows};
return true;
fail: {
int saved = errno;
close(fd);
errno = saved;
return false;
}
}
void ds4_engram_table_close(ds4_engram_table *t) {
if (!t) return;
if (t->fd >= 0) close(t->fd);
*t = (ds4_engram_table){.fd = -1};
}
static bool read_row(int fd, uint64_t offset, uint8_t row[DS4_ENGRAM_ROW_BYTES]) {
size_t done = 0;
while (done < DS4_ENGRAM_ROW_BYTES) {
ssize_t n = pread(fd, row + done, DS4_ENGRAM_ROW_BYTES - done,
(off_t)(offset + done));
if (n < 0 && errno == EINTR) continue;
if (n <= 0) {
if (n == 0) errno = EIO;
return false;
}
done += (size_t)n;
}
return true;
}
static float e4m3(uint8_t byte) {
int exponent = (byte >> 3) & 15, mantissa = byte & 7;
float value = exponent ? ldexpf((float)(8 + mantissa), exponent - 10) :
ldexpf((float)mantissa, -9);
return byte & 128 ? -value : value;
}
/* All code/scale pairs fit in 128 KiB; this is a format lookup, not a cache
* of model rows. Preserve the reference's F32 scaling and BF16 rounding. */
static uint16_t engram_bf16[256][256];
static pthread_once_t engram_bf16_once = PTHREAD_ONCE_INIT;
static void init_engram_bf16(void) {
for (unsigned scale = 0; scale < 256; scale++) {
for (unsigned code = 0; code < 256; code++) {
uint16_t result = UINT16_MAX;
if ((code & 127u) != 127u && scale != 255u) {
float value = ldexpf(e4m3((uint8_t)code), (int)scale - 127);
uint32_t bits;
memcpy(&bits, &value, sizeof(bits));
bits = (bits + 0x7fffu + ((bits >> 16) & 1u)) & 0xffff0000u;
if ((bits & 0x7f800000u) != 0x7f800000u) result = (uint16_t)(bits >> 16);
}
engram_bf16[scale][code] = result;
}
}
}
bool ds4_engram_read(const ds4_engram_table *t, const uint32_t *rows,
size_t count, float *out) {
if (!t || t->fd < 0 || (count && (!rows || !out)) ||
count > SIZE_MAX / (DS4_ENGRAM_DIM * sizeof(*out))) {
errno = EINVAL;
return false;
}
for (size_t i = 0; i < count; i++) {
if (rows[i] >= t->rows) {
errno = EINVAL;
return false;
}
}
pthread_once(&engram_bf16_once, init_engram_bf16);
uint8_t raw[DS4_ENGRAM_ROW_BYTES];
for (size_t i = 0; i < count; i++) {
if (!read_row(t->fd, t->offset + (uint64_t)rows[i] * sizeof(raw), raw)) return false;
for (int j = 0; j < DS4_ENGRAM_DIM; j++) {
const uint16_t value = engram_bf16[raw[DS4_ENGRAM_DIM + j / 32]][raw[j]];
if (value == UINT16_MAX) {
errno = EDOM;
return false;
}
const uint32_t bits = (uint32_t)value << 16;
memcpy(out + i * DS4_ENGRAM_DIM + j, &bits, sizeof(bits));
}
}
return true;
}
typedef struct {
uint32_t row, output;
} engram_request;
static int request_order(const void *a, const void *b) {
const engram_request *x = a, *y = b;
return (x->row > y->row) - (x->row < y->row);
}
enum { ENGRAM_READERS = 16 };
#ifdef __APPLE__
enum { ENGRAM_PARALLEL_MIN_ROWS = 8 };
#else
/* Unlike dispatch's shared pool, this path creates threads for each batch. */
enum { ENGRAM_PARALLEL_MIN_ROWS = 256 };
#endif
typedef struct {
const ds4_engram_table *table;
const engram_request *request;
float *out;
size_t count, readers;
int error[ENGRAM_READERS];
} engram_batch;
static void read_batch_part(void *context, size_t part) {
engram_batch *batch = context;
const engram_request *request = batch->request;
const size_t begin = batch->count * part / batch->readers;
const size_t end = batch->count * (part + 1) / batch->readers;
const float *previous = NULL;
for (size_t i = begin; i < end; i++) {
float *dst = batch->out + (size_t)request[i].output * DS4_ENGRAM_DIM;
if (i > begin && request[i].row == request[i - 1].row) {
memcpy(dst, previous, DS4_ENGRAM_DIM * sizeof(*dst));
} else {
if (!ds4_engram_read(batch->table, &request[i].row, 1, dst)) {
batch->error[part] = errno ? errno : EIO;
return;
}
previous = dst;
}
}
}
#ifndef __APPLE__
typedef struct {
engram_batch *batch;
size_t part;
} engram_reader;
static void *read_batch_thread(void *context) {
engram_reader *reader = context;
read_batch_part(reader->batch, reader->part);
return NULL;
}
#endif
bool ds4_engram_read_batch(const ds4_engram_table *t, const uint32_t *rows,
size_t tokens, size_t stride, float *out) {
if (!t || t->fd < 0 || (tokens && (!rows || !out || stride < DS4_ENGRAM_COLS)) ||
tokens > SIZE_MAX / (DS4_ENGRAM_COLS * DS4_ENGRAM_DIM * sizeof(*out)) ||
(tokens && tokens - 1 > (SIZE_MAX / sizeof(*rows) - DS4_ENGRAM_COLS) / stride)) {
errno = EINVAL;
return false;
}
for (size_t i = 0; i < tokens; i++) {
for (size_t j = 0; j < DS4_ENGRAM_COLS; j++) {
if (rows[i * stride + j] >= t->rows) {
errno = EINVAL;
return false;
}
}
}
if (!tokens) return true;
enum { BATCH_TOKENS = 2048 };
const size_t cap = tokens < BATCH_TOKENS ? tokens : BATCH_TOKENS;
engram_request *request = malloc(cap * DS4_ENGRAM_COLS * sizeof(*request));
if (!request) return false;
bool ok = true;
for (size_t start = 0; ok && start < tokens; start += cap) {
const size_t n = tokens - start < cap ? tokens - start : cap;
const size_t count = n * DS4_ENGRAM_COLS;
for (size_t i = 0; i < count; i++) {
request[i] = (engram_request){
rows[(start + i / DS4_ENGRAM_COLS) * stride + i % DS4_ENGRAM_COLS],
(uint32_t)i
};
}
qsort(request, count, sizeof(*request), request_order);
engram_batch batch = {.table = t, .request = request, .count = count,
.out = out + start * DS4_ENGRAM_COLS * DS4_ENGRAM_DIM, .readers = 1};
/* Fixed concurrency hides random-read latency without caching the table.
* Each worker owns disjoint output rows; all finish before GPU use. */
if (count >= ENGRAM_PARALLEL_MIN_ROWS) {
batch.readers = count < ENGRAM_READERS ? count : ENGRAM_READERS;
#ifdef __APPLE__
dispatch_apply_f(batch.readers,
dispatch_get_global_queue(QOS_CLASS_USER_INITIATED, 0), &batch, read_batch_part);
#else
pthread_t threads[ENGRAM_READERS - 1];
engram_reader readers[ENGRAM_READERS - 1];
size_t started = 0;
for (size_t part = 1; part < batch.readers; part++) {
readers[started] = (engram_reader){&batch, part};
if (pthread_create(&threads[started], NULL, read_batch_thread,
&readers[started])) break;
started++;
}
read_batch_part(&batch, 0);
/* Thread exhaustion only reduces concurrency, not correctness. */
for (size_t part = started + 1; part < batch.readers; part++)
read_batch_part(&batch, part);
for (size_t part = 0; part < started; part++)
if (pthread_join(threads[part], NULL)) abort();
#endif
} else
read_batch_part(&batch, 0);
for (size_t i = 0; i < batch.readers; i++) {
if (batch.error[i]) {
errno = batch.error[i];
ok = false;
break;
}
}
}
int saved = errno;
free(request);
errno = saved;
return ok;
}
struct ds4_engram_prefetch {
const ds4_engram_table *table;
const uint32_t *rows;
size_t tokens, stride;
float *out;
bool ok;
int error;
#ifdef __APPLE__
dispatch_group_t group;
#else
pthread_t thread;
bool threaded;
#endif
};
static void prefetch_run(void *context) {
ds4_engram_prefetch *p = context;
p->ok = ds4_engram_read_batch(p->table, p->rows, p->tokens, p->stride, p->out);
if (!p->ok) p->error = errno ? errno : EIO;
}
#ifndef __APPLE__
static void *prefetch_thread(void *context) {
prefetch_run(context);
return NULL;
}
#endif
ds4_engram_prefetch *ds4_engram_read_batch_start(const ds4_engram_table *table, const uint32_t *rows,
size_t tokens, size_t stride, float *out) {
ds4_engram_prefetch *p = calloc(1, sizeof(*p));
if (!p) return NULL;
*p = (ds4_engram_prefetch){.table = table, .rows = rows, .tokens = tokens, .stride = stride, .out = out};
#ifdef __APPLE__
p->group = dispatch_group_create();
dispatch_group_async_f(p->group, dispatch_get_global_queue(QOS_CLASS_USER_INITIATED, 0), p, prefetch_run);
#else
p->threaded = pthread_create(&p->thread, NULL, prefetch_thread, p) == 0;
if (!p->threaded) prefetch_run(p);
#endif
return p;
}
/* Page-cache warmer. Sequential reads fill the kernel's cache with the table
* once; the uncached row readers then hit it instead of the device. Spread
* prefill rows are the worst access pattern for external SSDs: a USB drive
* served ~6K random rows/s over the 189 GiB span, ~12 s per 2K prefill, but
* streams the whole span at ~1 GB/s. Clean cached pages stay reclaimable, so
* the warmer is a hint, never a reservation: it stops at memory pressure. */
enum { WARM_THREADS = 4 };
#define WARM_CHUNK ((uint64_t)8 << 20)
struct ds4_engram_warm {
int fd;
uint64_t begin, end, next;
_Atomic bool stop;
bool done, ok;
double started, seconds;
pthread_mutex_t lock;
pthread_cond_t cond;
unsigned running;
pthread_t thread[WARM_THREADS];
unsigned threads;
};
static double warm_now(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (double)ts.tv_sec + ts.tv_nsec * 1e-9;
}
static bool warm_pressure(void) {
#ifdef __APPLE__
int level = 0;
size_t len = sizeof(level);
/* 1 normal, 2 warn, 4 critical. */
return sysctlbyname("kern.memorystatus_vm_pressure_level", &level, &len, NULL, 0) == 0 &&
level > 1;
#else
return false;
#endif
}
static void *warm_thread(void *arg) {
ds4_engram_warm *w = arg;
#ifdef __APPLE__
pthread_set_qos_class_self_np(QOS_CLASS_UTILITY, 0);
#endif
uint8_t *buf = malloc(WARM_CHUNK);
bool ok = buf != NULL;
for (unsigned n = 0; ok && !w->stop; n++) {
const uint64_t off = __atomic_fetch_add(&w->next, WARM_CHUNK, __ATOMIC_RELAXED);
if (off >= w->end) break;
if (!(n & 63) && warm_pressure()) {
w->stop = true;
ok = false;
break;
}
size_t left = (size_t)(w->end - off < WARM_CHUNK ? w->end - off : WARM_CHUNK);
const uint64_t end = off + left;
while (left) {
/* A short read still warms those pages; only the file offset advances. */
const ssize_t got = pread(w->fd, buf, left, (off_t)(end - left));
if (got < 0 && errno == EINTR) continue;
if (got <= 0 || (size_t)got > left) {
ok = false;
break;
}
left -= (size_t)got;
}
}
free(buf);
pthread_mutex_lock(&w->lock);
if (!ok) w->ok = false;
if (--w->running == 0) {
w->done = true;
w->seconds = warm_now() - w->started;
pthread_cond_broadcast(&w->cond);
}
pthread_mutex_unlock(&w->lock);
return NULL;
}
uint64_t ds4_engram_physical_memory(void) {
#ifdef __APPLE__
uint64_t bytes = 0;
size_t len = sizeof(bytes);
return sysctlbyname("hw.memsize", &bytes, &len, NULL, 0) == 0 ? bytes : 0;
#else
const long pages = sysconf(_SC_PHYS_PAGES), page = sysconf(_SC_PAGESIZE);
return pages > 0 && page > 0 ? (uint64_t)pages * (uint64_t)page : 0;
#endif
}
ds4_engram_warm *ds4_engram_warm_start(const char *path, uint64_t offset, uint64_t bytes) {
if (!path || !bytes || offset > INT64_MAX || bytes > INT64_MAX - offset) return NULL;
ds4_engram_warm *w = calloc(1, sizeof(*w));
if (!w) return NULL;
w->fd = open(path, O_RDONLY | O_CLOEXEC);
if (w->fd < 0) {
free(w);
return NULL;
}
w->begin = w->next = offset;
w->end = offset + bytes;
w->ok = true;
w->started = warm_now();
pthread_mutex_init(&w->lock, NULL);
pthread_cond_init(&w->cond, NULL);
for (unsigned i = 0; i < WARM_THREADS; i++) {
pthread_mutex_lock(&w->lock);
w->running++;
pthread_mutex_unlock(&w->lock);
if (pthread_create(&w->thread[i], NULL, warm_thread, w)) {
pthread_mutex_lock(&w->lock);
w->running--;
pthread_mutex_unlock(&w->lock);
break;
}
w->threads++;
}
if (!w->threads) {
ds4_engram_warm_stop(w);
return NULL;
}
return w;
}
bool ds4_engram_warm_wait(ds4_engram_warm *w, double *seconds) {
if (!w) return false;
pthread_mutex_lock(&w->lock);
while (!w->done) pthread_cond_wait(&w->cond, &w->lock);
const bool ok = w->ok;
if (seconds) *seconds = w->seconds;
pthread_mutex_unlock(&w->lock);
return ok;
}
void ds4_engram_warm_stop(ds4_engram_warm *w) {
if (!w) return;
w->stop = true;
for (unsigned i = 0; i < w->threads; i++)
if (pthread_join(w->thread[i], NULL)) abort();
pthread_mutex_destroy(&w->lock);
pthread_cond_destroy(&w->cond);
close(w->fd);
free(w);
}
bool ds4_engram_read_batch_finish(ds4_engram_prefetch *p) {
if (!p) return false;
#ifdef __APPLE__
dispatch_group_wait(p->group, DISPATCH_TIME_FOREVER);
dispatch_release(p->group);
#else
if (p->threaded && pthread_join(p->thread, NULL)) abort();
#endif
const bool ok = p->ok;
if (!ok) errno = p->error;
free(p);
return ok;
}
enum { TOKEN_READERS = 4 };
typedef struct {
pthread_t thread;
ds4_engram_reader *reader;
unsigned index;
} engram_read_worker;
struct ds4_engram_reader {
engram_read_worker workers[TOKEN_READERS];
unsigned threads;
pthread_mutex_t mutex;
pthread_cond_t work, done;
bool active, shutdown;
uint64_t generation;
unsigned completed[2];
int errors[2];
ds4_engram_table tables[2];
uint32_t ids[2][DS4_ENGRAM_COLS];
float *out;
};
static void *read_token_thread(void *context) {
const engram_read_worker *worker = context;
ds4_engram_reader *r = worker->reader;
const unsigned index = worker->index;
uint64_t seen = 0;
pthread_mutex_lock(&r->mutex);
for (;;) {
while (r->generation == seen && !r->shutdown) pthread_cond_wait(&r->work, &r->mutex);
if (r->shutdown) break;
seen = r->generation;
pthread_mutex_unlock(&r->mutex);
for (unsigned table = 0; table < 2; table++) {
const unsigned first = index * DS4_ENGRAM_COLS / TOKEN_READERS;
const unsigned end = (index + 1u) * DS4_ENGRAM_COLS / TOKEN_READERS;
const bool ok = ds4_engram_read(&r->tables[table], r->ids[table] + first,
end - first, r->out + (table * DS4_ENGRAM_COLS + first) * DS4_ENGRAM_DIM);
const int error = ok ? 0 : errno ? errno : EIO;
pthread_mutex_lock(&r->mutex);
if (!r->errors[table]) r->errors[table] = error;
if (++r->completed[table] == TOKEN_READERS) {
if (table == 1) r->active = false;
pthread_cond_broadcast(&r->done);
}
pthread_mutex_unlock(&r->mutex);
}
pthread_mutex_lock(&r->mutex);
}
pthread_mutex_unlock(&r->mutex);
return NULL;
}
ds4_engram_reader *ds4_engram_reader_create(void) {
ds4_engram_reader *r = calloc(1, sizeof(*r));
if (!r) return NULL;
int error = pthread_mutex_init(&r->mutex, NULL);
if (error) goto fail;
error = pthread_cond_init(&r->work, NULL);
if (error) goto mutex;
error = pthread_cond_init(&r->done, NULL);
if (error) goto work;
for (unsigned i = 0; i < TOKEN_READERS; i++) {
r->workers[i].reader = r;
r->workers[i].index = i;
error = pthread_create(&r->workers[i].thread, NULL, read_token_thread, &r->workers[i]);
if (error) break;
r->threads++;
}
if (!error) return r;
pthread_mutex_lock(&r->mutex);
r->shutdown = true;
pthread_cond_broadcast(&r->work);
pthread_mutex_unlock(&r->mutex);
for (unsigned i = 0; i < r->threads; i++)
if (pthread_join(r->workers[i].thread, NULL)) abort();
pthread_cond_destroy(&r->done);
work:
pthread_cond_destroy(&r->work);
mutex:
pthread_mutex_destroy(&r->mutex);
fail:
free(r);
errno = error;
return NULL;
}
bool ds4_engram_reader_start(ds4_engram_reader *r,
const ds4_engram_table tables[2], const uint32_t ids[2][DS4_ENGRAM_COLS], float *out) {
if (!r || !tables || !ids || !out) { errno = EINVAL; return false; }
for (unsigned t = 0; t < 2; t++) {
if (tables[t].fd < 0) { errno = EINVAL; return false; }
for (unsigned col = 0; col < DS4_ENGRAM_COLS; col++)
if (ids[t][col] >= tables[t].rows) { errno = EINVAL; return false; }
}
pthread_mutex_lock(&r->mutex);
if (r->active || r->shutdown) {
pthread_mutex_unlock(&r->mutex);
errno = EBUSY;
return false;
}
memcpy(r->tables, tables, sizeof(r->tables));
memcpy(r->ids, ids, sizeof(r->ids));
memset(r->errors, 0, sizeof(r->errors));
r->out = out;
memset(r->completed, 0, sizeof(r->completed));
r->generation++;
r->active = true;
pthread_cond_broadcast(&r->work);
pthread_mutex_unlock(&r->mutex);
return true;
}
bool ds4_engram_reader_wait(ds4_engram_reader *r, unsigned table) {
if (!r || table >= 2) { errno = EINVAL; return false; }
pthread_mutex_lock(&r->mutex);
if (!r->active && !r->completed[1]) {
pthread_mutex_unlock(&r->mutex);
errno = EINVAL;
return false;
}
while (r->completed[table] < TOKEN_READERS) pthread_cond_wait(&r->done, &r->mutex);
const int error = r->errors[table];
pthread_mutex_unlock(&r->mutex);
if (error) errno = error;
return !error;
}
void ds4_engram_reader_free(ds4_engram_reader *r) {
if (!r) return;
pthread_mutex_lock(&r->mutex);
r->shutdown = true;
pthread_cond_broadcast(&r->work);
pthread_mutex_unlock(&r->mutex);
for (unsigned i = 0; i < r->threads; i++)
if (pthread_join(r->workers[i].thread, NULL)) abort();
pthread_cond_destroy(&r->done);
pthread_cond_destroy(&r->work);
pthread_mutex_destroy(&r->mutex);
free(r);
}
ds4_engram.h
+90
View File
@@ -0,0 +1,90 @@
#ifndef DS4_ENGRAM_H
#define DS4_ENGRAM_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
enum {
DS4_ENGRAM_LAYERS = 2,
DS4_ENGRAM_NGRAM = 4,
DS4_ENGRAM_HEADS = 8,
DS4_ENGRAM_COLS = 24,
DS4_ENGRAM_DIM = 256,
DS4_ENGRAM_ROW_BYTES = 264,
DS4_ENGRAM_DEAD = -1
};
typedef struct {
const uint32_t *token_map;
uint32_t vocab_size, compressed_vocab_size, pad_id;
uint32_t rows[DS4_ENGRAM_LAYERS];
uint64_t multipliers[DS4_ENGRAM_LAYERS][DS4_ENGRAM_NGRAM];
uint32_t primes[DS4_ENGRAM_LAYERS][DS4_ENGRAM_COLS];
} ds4_engram_layout;
/* Newest first; copying this value also snapshots the complete hash state. */
typedef struct {
int32_t tail[DS4_ENGRAM_NGRAM - 1];
} ds4_engram_history;
bool ds4_engram_layout_valid(const ds4_engram_layout *layout);
void ds4_engram_history_reset(ds4_engram_history *history);
/* Validate the layout once at model load. Mask zero breaks all n-grams that
* cross that position. Output is [token][layer][column], including masked rows;
* the graph must suppress Engram at masked positions, as in the reference. */
bool ds4_engram_hash(const ds4_engram_layout *layout,
ds4_engram_history *history,
const int *tokens, const uint8_t *mask, size_t count,
uint32_t *rows);
typedef struct {
int fd;
uint64_t offset;
uint32_t rows;
} ds4_engram_table;
/* A separate uncached file descriptor, never an mmap or Metal model view.
* Each GGUF I8 row is 256 E4M3 bytes followed by 8 original E8M0 scales. */
bool ds4_engram_table_open(ds4_engram_table *table, const char *path,
uint64_t offset, uint32_t rows);
void ds4_engram_table_close(ds4_engram_table *table);
/* Output uses F32 storage for the reference's BF16-rounded values. No whole
* table allocation; caller owns count * DIM floats. Failure invalidates output. */
bool ds4_engram_read(const ds4_engram_table *table, const uint32_t *rows,
size_t count, float *out);
/* Read COLS rows per token, restoring token order after deduplicated disk reads.
* Input stride is in row IDs; output is packed [token][COLS][DIM]. Temporary
* storage is bounded to 384 KiB, independent of the table and prefix size.
* On macOS, large batches use bounded concurrent pread readers. */
bool ds4_engram_read_batch(const ds4_engram_table *table, const uint32_t *rows,
size_t tokens, size_t stride, float *out);
/* The same read on a worker: `rows` and `out` stay the caller's until finish,
* which joins the worker and returns its result. */
typedef struct ds4_engram_prefetch ds4_engram_prefetch;
ds4_engram_prefetch *ds4_engram_read_batch_start(const ds4_engram_table *table, const uint32_t *rows,
size_t tokens, size_t stride, float *out);
bool ds4_engram_read_batch_finish(ds4_engram_prefetch *prefetch);
/* One reusable reader for a token's two tables. Tables/IDs are copied at
* start; descriptors and output must remain valid until wait(reader, 1) or
* free, even if wait(reader, 0) reports an error. No table data is cached.
* One caller owns the reader. Creation failure permits synchronous fallback. */
typedef struct ds4_engram_reader ds4_engram_reader;
ds4_engram_reader *ds4_engram_reader_create(void);
bool ds4_engram_reader_start(ds4_engram_reader *reader,
const ds4_engram_table tables[2], const uint32_t ids[2][DS4_ENGRAM_COLS], float *out);
bool ds4_engram_reader_wait(ds4_engram_reader *reader, unsigned table);
void ds4_engram_reader_free(ds4_engram_reader *reader);
/* Background sequential read of [offset, offset + bytes) into the kernel page
* cache, so the uncached row readers above hit memory. No table allocation;
* stops early at memory pressure. Wait reports whether it covered the span. */
typedef struct ds4_engram_warm ds4_engram_warm;
uint64_t ds4_engram_physical_memory(void);
ds4_engram_warm *ds4_engram_warm_start(const char *path, uint64_t offset, uint64_t bytes);
bool ds4_engram_warm_wait(ds4_engram_warm *warm, double *seconds);
void ds4_engram_warm_stop(ds4_engram_warm *warm);
#endif
ds4_eval.c
+787 -101
View File
File diff suppressed because it is too large. Load diff
ds4_eval_cases.c
+1118
View File
File diff suppressed because it is too large. Load diff
ds4_eval_cases.h
+51
View File
@@ -0,0 +1,51 @@
#ifndef DS4_EVAL_CASES_H
#define DS4_EVAL_CASES_H
#include <stddef.h>
#include <stdint.h>
#define EVAL_MAX_CHOICES 10
#define EVAL_MAX_ALIASES 4
typedef enum {
EVAL_ANSWER_AUTO = 0,
EVAL_ANSWER_CHOICE,
EVAL_ANSWER_INTEGER,
EVAL_ANSWER_RATIONAL,
EVAL_ANSWER_EXACT_TEXT,
EVAL_ANSWER_ORDERED_SEQUENCE,
EVAL_ANSWER_LINE_SET,
} eval_answer_kind;
enum {
EVAL_SUITE_CORE = 1u << 0,
EVAL_SUITE_HARD = 1u << 1,
EVAL_SUITE_HARD_SMOKE = 1u << 2,
};
typedef struct {
const char *source;
const char *id;
const char *domain;
const char *title;
const char *question;
const char *choice[EVAL_MAX_CHOICES];
const char *answer;
const char *alias[EVAL_MAX_ALIASES];
eval_answer_kind answer_kind;
uint32_t suites;
uint32_t max_tokens;
} eval_case;
typedef struct {
const char *name;
const char *url;
const char *license;
} eval_source;
extern const eval_case eval_hard_cases[];
extern const size_t eval_hard_case_count;
extern const eval_source eval_hard_sources[];
extern const size_t eval_hard_source_count;
#endif
ds4_glm53_vision_gpu.cuh
+1
View File
@@ -261,6 +261,7 @@ __global__ static void glm53_vision_layernorm_gelu_kernel(
__syncthreads();
}
const float mean = partial[0] / (float)width;
__syncthreads();
float var = 0.0f;
for (uint32_t d = tid; d < width; d += blockDim.x) {
const float centered = xr[d] - mean;
ds4_gpu.h
+868 -67
View File
File diff suppressed because it is too large. Load diff
ds4_gpu_copy.h
+24
View File
@@ -0,0 +1,24 @@
#ifndef DS4_GPU_COPY_H
#define DS4_GPU_COPY_H
#include <stdint.h>
/* Device copy spans, shared by ds4_gpu.h and the CUDA backend, which does
* not include ds4_gpu.h. The tensor stays opaque here. */
#ifndef DS4_GPU_TENSOR_DEFINED
#define DS4_GPU_TENSOR_DEFINED
typedef struct ds4_gpu_tensor ds4_gpu_tensor;
#endif
/* One device copy, with the arguments of ds4_gpu_tensor_copy. */
typedef struct {
ds4_gpu_tensor *dst;
uint64_t dst_offset;
const ds4_gpu_tensor *src;
uint64_t src_offset;
uint64_t bytes;
} ds4_gpu_copy_span;
#define DS4_GPU_COPY_SPANS_MAX 316u
#endif
ds4_gpu_tp.h
+63
View File
@@ -0,0 +1,63 @@
#ifndef DS4_GPU_TP_H
#define DS4_GPU_TP_H
#include <stdint.h>
#ifndef DS4_GPU_TENSOR_DEFINED
#define DS4_GPU_TENSOR_DEFINED
typedef struct ds4_gpu_tensor ds4_gpu_tensor;
#endif
#ifdef __cplusplus
extern "C" {
#endif
/* Metal queues each gate on its service thread and orders GPU arrival/release
* with events or flags. CUDA waits for its local stream and calls the exchange
* synchronously. Callbacks return nonzero on success; both ranks must issue
* the same gate sequence. Shutdown precedes transport/slab destruction. */
typedef int (*ds4_gpu_tp_exchange_fn)(void *ud, uint32_t layer, uint32_t gate, uint64_t seq);
typedef int (*ds4_gpu_tp_batch_exchange_fn)(void *ud, uint32_t layer,
uint32_t rows, uint64_t seq);
typedef int (*ds4_gpu_tp_big_exchange_fn)(void *ud, uint32_t layer,
uint64_t seq, const void *out,
void *in, uint64_t bytes);
/* CUDA: optional pinned slab shared by an integrated GPU and the transport.
* NULL requests the ordinary device tensor plus host staging fallback. */
ds4_gpu_tensor *ds4_gpu_tp_slab_alloc(uint64_t bytes);
int ds4_gpu_tp_init(uint32_t rank, ds4_gpu_tensor *slab,
uint64_t gpu_flags_off, uint64_t out_off, uint64_t vec_bytes,
ds4_gpu_tp_exchange_fn fn, void *ud);
void ds4_gpu_tp_shutdown(void);
int ds4_gpu_tp_failed(void);
int ds4_gpu_tp_gate_encode(uint32_t layer, uint32_t gate);
void ds4_gpu_tp_set_batch_exchange(ds4_gpu_tp_batch_exchange_fn fn);
int ds4_gpu_tp_batch_gate_encode(uint32_t layer, uint32_t rows);
void ds4_gpu_tp_set_big_exchange(ds4_gpu_tp_big_exchange_fn fn);
int ds4_gpu_tp_big_gate_encode(uint32_t layer, uint32_t rows,
const ds4_gpu_tensor *out_t,
ds4_gpu_tensor *in_t, uint64_t bytes);
/* Metal multi-session tapes reuse slab slots and therefore require event
* arrival. Single-session flag gates may flush in layer order. */
void ds4_gpu_tp_set_session_batch_mode(int enabled);
/* Metal decode gates publish and wait inside the command buffer (no flush per gate). */
int ds4_gpu_tp_decode_inline_gates(void);
int ds4_gpu_tp_decode_split_flush_safe(void);
/* Weight ranges consumed by the next Metal poll gate of the given kind. */
int ds4_gpu_tp_gate_prefetch_plan(uint32_t gate,
const void *model_map, uint64_t model_size,
const uint64_t *offsets, const uint64_t *bytes,
uint32_t count);
/* Coordinator-only drafting is replicated; verification remains sharded. */
void ds4_gpu_tp_suspend_expert_sharding(int suspend);
/* No-op when TP is not bound. */
void ds4_gpu_tp_keepalive_pause(int paused);
/* GLM attention head ownership; the caller zeros unowned heads and exchanges
* the output-projection partials at the big gate. */
void ds4_gpu_tp_set_attn_head_split(int enabled);
#ifdef __cplusplus
}
#endif
#endif
ds4_help.c
+31 -14
View File
@@ -148,7 +148,7 @@ static void print_model_runtime(FILE *fp, const help_colors *c,
title(fp, c, "Model And Runtime");
opt(fp, c, "-m, --model FILE", "GGUF model path. Default: ds4flash.gguf");
if (tool == DS4_HELP_DS4 || tool == DS4_HELP_AGENT || tool == DS4_HELP_SERVER) {
opt(fp, c, "--vision FILE", "GLM 5.3 vision encoder GGUF.");
opt(fp, c, "--vision FILE", "Vision encoder GGUF for the selected model.");
}
#ifdef DS4_ROCM_BUILD
opt(fp, c, "--metal | --rocm | --cpu", "Select the backend explicitly.");
@@ -172,13 +172,13 @@ static void print_model_runtime(FILE *fp, const help_colors *c,
opt(fp, c, "--power N", "GPU duty-cycle target, 1..100. Default: 100");
opt(fp, c, "--ssd-streaming", "Metal/CUDA/ROCm: opt in to SSD-backed model streaming instead of full residency.");
opt(fp, c, "--ssd-streaming-cold", "SSD streaming: skip default popularity-based expert-cache preload.");
opt(fp, c, "--ssd-streaming-cache-experts N|NGB", "SSD streaming: N is an exact dynamic expert count; NGB is a routed memory budget that also reserves two full prefill layers. Auto: 80% working set minus non-routed weights; GLM Metal caps lower.");
opt(fp, c, "--ssd-streaming-cache-experts N|NGB", "SSD streaming cache target. N requests dynamic expert slots; NGB also reserves two full prefill layers. Either may be reduced to fit the model, graph, context, and backend working set.");
opt(fp, c, "--ssd-streaming-full-layers N", "GLM Metal streaming: keep the first N routed layers fully resident. Default: auto from NGB expert budget; use 0 to disable.");
opt(fp, c, "--ssd-streaming-preload-experts N", "SSD streaming: upfront popularity preload count. DeepSeek auto-seeds by default; GLM demand-fills unless N is explicit.");
opt(fp, c, "--simulate-used-memory NGB", "Diagnostic: lock N GiB before model load to simulate a smaller-memory machine.");
opt(fp, c, "--prefill-chunk N", "Graph prefill chunk size. Default: CUDA TP 2048; PRO long prompts 8192; others 4096.");
if (full) {
if (tool == DS4_HELP_EVAL) {
if (tool == DS4_HELP_EVAL || tool == DS4_HELP_BENCH) {
opt(fp, c, "--mtp-model FILE", "External MTP or DSpark support GGUF.");
}
if (tool == DS4_HELP_DS4 || tool == DS4_HELP_AGENT || tool == DS4_HELP_SERVER) {
@@ -188,12 +188,15 @@ static void print_model_runtime(FILE *fp, const help_colors *c,
opt(fp, c, "--mtp-margin F", "Verifier confidence margin for fast MTP acceptance. Default: 3");
opt(fp, c, "--mtp-timing", "Enable embedded MTP and print acceptance/timing counters.");
opt(fp, c, "--dspark", "Enable DSpark using the support GGUF passed with --mtp-model.");
opt(fp, c, "--dspark-confidence F", "Enable DSpark with confidence pruning threshold 0..1. Greedy/opportunistic default: Metal 0.6, CUDA/ROCm 0.7; exact sampling: 0.8");
opt(fp, c, "--dspark-confidence F", "Enable DSpark with confidence pruning threshold 0..1. Greedy/opportunistic default: resident Spark Q2 0.4, two-Spark Flash Q2/MXFP4 0, Metal 0.6, other CUDA/ROCm 0.7; exact sampling: 0.8");
opt(fp, c, "--mtp-exact-sampling", "Preserve the ordinary temperature distribution instead of accepting target-matching greedy drafts directly.");
opt(fp, c, "--dspark-strict", "Load DSpark support but keep target-only decode.");
} else if (tool == DS4_HELP_BENCH) {
opt(fp, c, "--dspark", "Benchmark greedy DSpark using the support GGUF passed with --mtp-model.");
opt(fp, c, "--dspark-confidence F", "DSpark confidence pruning threshold 0..1.");
}
opt(fp, c, "--quality", "Prefer exact kernels where faster approximate paths exist.");
opt(fp, c, "--warm-weights", "Touch mapped tensor pages at startup to reduce first-use stalls.");
opt(fp, c, "--warm-weights", "Touch resident weights at startup to reduce first-use stalls.");
if (tool == DS4_HELP_DS4 || tool == DS4_HELP_BENCH) {
opt(fp, c, "--expert-profile FILE", "Metal-only: write routed expert locality/cache simulation JSON.");
}
@@ -201,7 +204,7 @@ static void print_model_runtime(FILE *fp, const help_colors *c,
fputc('\n', fp);
}
static void print_sampling(FILE *fp, const help_colors *c, bool full) {
static void print_sampling(FILE *fp, const help_colors *c, bool full, ds4_help_tool tool) {
title(fp, c, "Prompt And Sampling");
opt(fp, c, "-n, --tokens N", "Maximum generated tokens.");
opt(fp, c, "--temp F", "Sampling temperature. 0 is greedy/deterministic.");
@@ -209,8 +212,10 @@ static void print_sampling(FILE *fp, const help_colors *c, bool full) {
opt(fp, c, "--min-p F", "Keep tokens scoring at least F times the top token.");
opt(fp, c, "--seed N", "Sampling seed for reproducible non-greedy runs.");
para(fp, c, "GLM CLI and agent runs default to temperature 1.0, top-p 0.95, and min-p 0 unless those options are set explicitly.");
opt(fp, c, "--think", "Use normal thinking mode.");
opt(fp, c, "--think-max", "Use Think Max when context is large enough.");
opt(fp, c, "--think", "Use normal thinking mode (V4.1: effort 75).");
opt(fp, c, "--think-max", "Use maximum thinking (V4.1: 100; V4: requires ctx >= 393216).");
if (tool == DS4_HELP_DS4 || tool == DS4_HELP_AGENT)
opt(fp, c, "--think-level N", "V4.1 thinking effort, 1..100; 0 disables thinking.");
opt(fp, c, "--nothink", "Disable thinking and ask for direct replies.");
if (full) {
opt(fp, c, "-sys, --system TEXT", "System prompt. Empty string disables the default where supported.");
@@ -264,6 +269,7 @@ static void print_cli_specific(FILE *fp, const help_colors *c, bool full) {
opt(fp, c, "ds4", "Start the interactive prompt.");
opt(fp, c, "ds4 -p TEXT", "Run one prompt and exit.");
opt(fp, c, "ds4 --prompt-file FILE", "Run a long prompt from a file and exit.");
opt(fp, c, "--prefix-file FILE", "Preload complete alternating USER:/ASSISTANT: turns before the live conversation.");
fputc('\n', fp);
if (full) {
print_cli_diagnostics(fp, c);
@@ -284,6 +290,7 @@ static void print_cli_diagnostics(FILE *fp, const help_colors *c) {
opt(fp, c, "--imatrix-out FILE", "Write llama-compatible routed-MoE imatrix .dat.");
opt(fp, c, "--imatrix-max-prompts N", "Stop imatrix collection after N prompts.");
opt(fp, c, "--imatrix-max-tokens N", "Stop imatrix collection after N prompt tokens.");
opt(fp, c, "--imatrix-min-expert-samples N", "Continue until every routed expert has N samples.");
opt(fp, c, "--head-test", "Run the output HC/logits head after the native slice.");
opt(fp, c, "--first-token-test", "Run exact CPU whole-model pass for the first prompt token.");
opt(fp, c, "--metal-graph-test", "Compare first GPU-resident graph stages with CPU.");
@@ -295,7 +302,7 @@ static void print_cli_diagnostics(FILE *fp, const help_colors *c) {
static void print_cli_commands(FILE *fp, const help_colors *c) {
title_red(fp, c, "Interactive Commands");
opt(fp, c, "/help", "Show interactive commands.");
opt(fp, c, "/think, /think-max, /nothink", "Switch thinking mode.");
opt(fp, c, "/think [N], /think-max, /nothink", "Switch thinking mode; V4.1 accepts effort 0..100.");
opt(fp, c, "/ctx N", "Restart the interactive session with a new context size.");
opt(fp, c, "/power N", "Set GPU duty cycle percentage, 1..100.");
opt(fp, c, "/read FILE", "Submit a text file, PNG, or JPEG as the next user message.");
@@ -308,6 +315,7 @@ static void print_agent_specific(FILE *fp, const help_colors *c) {
title(fp, c, "Agent Options");
opt(fp, c, "-p, --prompt TEXT", "Submit an initial prompt after startup.");
opt(fp, c, "--prompt-file FILE", "Read the initial prompt from FILE.");
opt(fp, c, "--prefix-file FILE", "Preload complete alternating USER:/ASSISTANT: turns before the live task.");
opt(fp, c, "--non-interactive", "Run without TUI. With an initial prompt: one turn; otherwise: repeated stdin prompts.");
opt(fp, c, "--raw-prompt", "Non-interactive initial prompt only: omit agent chat/tool text.");
opt(fp, c, "--edit-upto", "Enable anchored [upto] edits and automatic marker insertion.");
@@ -326,6 +334,8 @@ static void print_agent_sessions(FILE *fp, const help_colors *c) {
opt(fp, c, "/del ID", "Delete a saved session.");
opt(fp, c, "/strip ID", "Remove KV payload; the text history can be rebuilt later.");
opt(fp, c, "/history [N]", "Show N recent user turns from the current session.");
opt(fp, c, "/hints on|off", "Enable or disable brief programming hints. New and resumed sessions start off.");
opt(fp, c, "/think [N]", "V4.1: set effort 0..100 (default 75) after this turn; rebuilds the prefix.");
opt(fp, c, "/power N", "Set GPU duty cycle percentage, 1..100.");
opt(fp, c, "/new", "Start a fresh session from the system prompt.");
opt(fp, c, "/quit, /exit", "Exit.");
@@ -391,9 +401,16 @@ static void print_bench_specific(FILE *fp, const help_colors *c) {
static void print_eval_specific(FILE *fp, const help_colors *c) {
title(fp, c, "Evaluation");
opt(fp, c, "-n, --tokens N", "Max generated tokens per question. Default: 16000");
opt(fp, c, "--questions N", "Run only the first N embedded questions.");
opt(fp, c, "--suite NAME", "core, hard, all, or hard-smoke. Default: core");
opt(fp, c, "--source NAME", "Run only cases from this source.");
opt(fp, c, "--domain NAME", "Run only cases in this domain.");
opt(fp, c, "--case-id ID", "Run the case with this source ID.");
opt(fp, c, "--list-cases", "List selected cases without loading a model.");
opt(fp, c, "--validate-cases", "Validate all embedded cases and exit.");
opt(fp, c, "-n, --tokens N", "Override the generation budget for every question.");
opt(fp, c, "--questions N", "Run only the first N selected questions.");
opt(fp, c, "--case-sequence LIST", "Run 1-based case numbers in this comma-separated order.");
opt(fp, c, "--retry-incomplete", "Retry a missing final answer once with twice the budget.");
opt(fp, c, "--trace FILE", "Write questions, outputs, and grading decisions.");
opt(fp, c, "--regrade-trace FILE", "Regrade a prior trace without loading the model.");
opt(fp, c, "--soft-limit-reply-budget N", "Soft close thinking near the end of reply budget. Default: 1024");
@@ -514,7 +531,7 @@ static void print_examples(FILE *fp, const help_colors *c, ds4_help_tool tool, c
static void print_topic(FILE *fp, const help_colors *c, ds4_help_tool tool, const char *topic) {
if (streq(topic, "all")) {
print_model_runtime(fp, c, tool, true);
if (tool_has_topic(tool, "sampling")) print_sampling(fp, c, true);
if (tool_has_topic(tool, "sampling")) print_sampling(fp, c, true, tool);
if (tool_has_topic(tool, "steering")) print_steering(fp, c);
print_distributed(fp, c);
if (tool == DS4_HELP_DS4) {
@@ -536,7 +553,7 @@ static void print_topic(FILE *fp, const help_colors *c, ds4_help_tool tool, cons
}
if (streq(topic, "runtime")) print_model_runtime(fp, c, tool, true);
else if (streq(topic, "sampling")) print_sampling(fp, c, true);
else if (streq(topic, "sampling")) print_sampling(fp, c, true, tool);
else if (streq(topic, "steering")) print_steering(fp, c);
else if (streq(topic, "distributed")) print_distributed(fp, c);
else if (tool == DS4_HELP_DS4 && streq(topic, "diagnostics")) print_cli_diagnostics(fp, c);
@@ -561,7 +578,7 @@ static void print_default(FILE *fp, const help_colors *c, ds4_help_tool tool) {
if (tool == DS4_HELP_DS4) {
print_cli_specific(fp, c, true);
print_sampling(fp, c, false);
print_sampling(fp, c, false, tool);
} else if (tool == DS4_HELP_SERVER) {
print_server_api(fp, c);
print_kv_cache(fp, c);
ds4_image.c
+643 -9
View File
@@ -2,6 +2,8 @@
#include <errno.h>
#include <math.h>
#include <pthread.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -387,14 +389,20 @@ static double ds4_cubic(double x, double a) {
return 0.0;
}
static void ds4_resize_rgb_bicubic(
const uint8_t *src,
uint32_t src_width,
uint32_t src_height,
float *dst,
uint32_t dst_width,
uint32_t dst_height,
uint32_t dst_stride) {
typedef struct {
const uint8_t *src;
float *dst;
uint32_t src_width, src_height, dst_width, dst_height, dst_stride;
uint32_t first_y, end_y;
} ds4_resize_rows;
static void *ds4_resize_bicubic_rows(void *context) {
const ds4_resize_rows *rows = context;
const uint8_t *src = rows->src;
float *dst = rows->dst;
const uint32_t src_width = rows->src_width, src_height = rows->src_height;
const uint32_t dst_width = rows->dst_width, dst_height = rows->dst_height;
const uint32_t dst_stride = rows->dst_stride;
double scale_x = (double)src_width / dst_width;
double scale_y = (double)src_height / dst_height;
double filter_x = scale_x >= 1.0 ? 1.0 / scale_x : 1.0;
@@ -402,7 +410,7 @@ static void ds4_resize_rgb_bicubic(
double support_x = scale_x >= 1.0 ? 2.0 * scale_x : 2.0;
double support_y = scale_y >= 1.0 ? 2.0 * scale_y : 2.0;
for (uint32_t dy = 0; dy < dst_height; dy++) {
for (uint32_t dy = rows->first_y; dy < rows->end_y; dy++) {
double center_y = scale_y * ((double)dy + 0.5);
int y0 = (int)(center_y - support_y + 0.5);
int y1 = (int)(center_y + support_y + 0.5);
@@ -440,6 +448,44 @@ static void ds4_resize_rgb_bicubic(
}
}
}
return NULL;
}
static void ds4_resize_rgb_bicubic(
const uint8_t *src,
uint32_t src_width,
uint32_t src_height,
float *dst,
uint32_t dst_width,
uint32_t dst_height,
uint32_t dst_stride) {
enum { MAX_WORKERS = 8 };
/* Separate output row ranges use the original horizontal and vertical
* coordinates, including when the destination is a padded canvas. */
const uint64_t work = (uint64_t)(src_width > dst_width ? src_width : dst_width) *
(src_height > dst_height ? src_height : dst_height);
uint32_t workers = work / 65536u < MAX_WORKERS ? (uint32_t)(work / 65536u) : MAX_WORKERS;
if (workers > dst_height) workers = dst_height;
ds4_resize_rows whole = {src, dst, src_width, src_height, dst_width, dst_height,
dst_stride, 0, dst_height};
if (workers < 2u) {
ds4_resize_bicubic_rows(&whole);
return;
}
ds4_resize_rows rows[MAX_WORKERS];
pthread_t threads[MAX_WORKERS - 1];
uint32_t started = 0;
for (; started + 1u < workers; started++) {
rows[started] = whole;
rows[started].first_y = (uint64_t)dst_height * started / workers;
rows[started].end_y = (uint64_t)dst_height * (started + 1u) / workers;
if (pthread_create(&threads[started], NULL, ds4_resize_bicubic_rows, &rows[started])) break;
}
whole.first_y = (uint64_t)dst_height * started / workers;
ds4_resize_bicubic_rows(&whole);
for (uint32_t i = 0; i < started; i++) {
if (pthread_join(threads[i], NULL)) abort();
}
}
int ds4_image_preprocess_glm53(
@@ -563,8 +609,596 @@ int ds4_image_preprocess_glm53(
return 1;
}
int ds4_image_preprocess_qwen4(
ds4_image_patches *out,
const ds4_image *image,
uint32_t min_image_tokens,
uint32_t max_image_tokens,
char *error,
size_t error_cap) {
const uint32_t patch = 16, factor = 32;
if (!out) return 0;
memset(out, 0, sizeof(*out));
if (!image || !image->rgb || image->width == 0 || image->height == 0 ||
min_image_tokens == 0 || max_image_tokens < min_image_tokens) {
ds4_image_error(error, error_cap, "invalid Qwen3.8 image preprocessing parameters");
return 0;
}
const double h = image->height, w = image->width;
if (fmax(h, w) / fmin(h, w) > 200.0) {
ds4_image_error(error, error_cap, "image aspect ratio must be below 200");
return 0;
}
/* Qwen2VL smart_resize: round to the factor, then scale into [min, max] pixels */
const double min_pixels = (double)min_image_tokens * factor * factor;
const double max_pixels = (double)max_image_tokens * factor * factor;
double h_bar = fmax(factor, round(h / factor) * factor);
double w_bar = fmax(factor, round(w / factor) * factor);
if (h_bar * w_bar > max_pixels) {
const double beta = sqrt(h * w / max_pixels);
h_bar = fmax(factor, floor(h / beta / factor) * factor);
w_bar = fmax(factor, floor(w / beta / factor) * factor);
} else if (h_bar * w_bar < min_pixels) {
const double beta = sqrt(min_pixels / (h * w));
h_bar = ceil(h * beta / factor) * factor;
w_bar = ceil(w * beta / factor) * factor;
}
const uint32_t target_height = (uint32_t)h_bar, target_width = (uint32_t)w_bar;
if (target_width > DS4_IMAGE_MAX_DIMENSION || target_height > DS4_IMAGE_MAX_DIMENSION ||
(uint64_t)target_width * target_height > DS4_IMAGE_MAX_PIXELS) {
ds4_image_error(error, error_cap, "resized image is too large");
return 0;
}
float *canvas = malloc((size_t)target_height * target_width * 3 * sizeof(float));
if (!canvas) {
ds4_image_error(error, error_cap, "unable to allocate resized image");
return 0;
}
if (target_width == image->width && target_height == image->height) {
for (size_t i = 0; i < (size_t)target_height * target_width * 3; i++) canvas[i] = image->rgb[i];
} else {
ds4_resize_rgb_bicubic(image->rgb, image->width, image->height,
canvas, target_width, target_height, target_width);
}
for (size_t i = 0; i < (size_t)target_height * target_width * 3; i++) {
canvas[i] = (canvas[i] / 255.0f - 0.5f) / 0.5f;
}
const uint32_t grid_height = target_height / patch, grid_width = target_width / patch;
const uint32_t patch_count = grid_height * grid_width;
const size_t patch_values = (size_t)patch_count * 3 * patch * patch;
float *patches = malloc(patch_values * sizeof(float));
if (!patches) {
free(canvas);
ds4_image_error(error, error_cap, "unable to allocate vision patches");
return 0;
}
size_t index = 0;
for (uint32_t block_y = 0; block_y < grid_height / 2; block_y++) {
for (uint32_t block_x = 0; block_x < grid_width / 2; block_x++) {
for (uint32_t merge_y = 0; merge_y < 2; merge_y++) {
for (uint32_t merge_x = 0; merge_x < 2; merge_x++) {
const uint32_t patch_y = block_y * 2 + merge_y, patch_x = block_x * 2 + merge_x;
for (uint32_t channel = 0; channel < 3; channel++) {
for (uint32_t y = 0; y < patch; y++) {
for (uint32_t x = 0; x < patch; x++) {
patches[index++] = canvas[((size_t)(patch_y * patch + y) * target_width +
patch_x * patch + x) * 3 + channel];
}
}
}
}
}
}
}
free(canvas);
out->content_width = target_width;
out->content_height = target_height;
out->padded_width = target_width;
out->padded_height = target_height;
out->grid_width = grid_width;
out->grid_height = grid_height;
out->patch_count = patch_count;
out->image_token_count = patch_count / 4;
out->patches = patches;
return 1;
}
void ds4_image_patches_free(ds4_image_patches *patches) {
if (!patches) return;
free(patches->patches);
memset(patches, 0, sizeof(*patches));
}
static uint32_t ds4_deepseek4_grid_tokens(
uint32_t height,
uint32_t width,
uint32_t *grid_height,
uint32_t *grid_width,
bool v41) {
const uint32_t h = (height / 14u + 2u) / 3u;
const uint32_t w = (width / 14u + 2u) / 3u;
uint64_t count = (uint64_t)h * (w + 1u) + 2u;
if (!v41) {
if (h & 1u) count += w + 1u;
if ((((h + 1u) / 2u) * (w + 1u)) & 1u) count += 2u;
}
if (grid_height) *grid_height = h;
if (grid_width) *grid_width = w;
return count <= UINT32_MAX ? (uint32_t)count : UINT32_MAX;
}
static int ds4_deepseek4_solve_resize(
uint32_t height,
uint32_t width,
uint32_t budget,
uint32_t *best_height,
uint32_t *best_width,
bool v41) {
const double ratio = (double)height / width;
const double max_w_float = sqrt(((double)budget - 2.0) / ratio + 0.25) - 0.5;
const double max_h_float = max_w_float * ratio;
uint32_t max_h, max_w;
if (max_w_float < 1.0) {
max_w = 1;
max_h = (budget - 2u) / (max_w + 1u);
if (!v41) max_h &= ~1u;
if (max_h == 0) return 0;
*best_width = max_w * 42u;
*best_height = max_h * 42u;
} else if (max_h_float < (v41 ? 1.0 : 2.0)) {
max_h = v41 ? 1u : 2u;
max_w = (budget - 2u) / max_h - 1u;
if (max_w <= 1u) return 0;
*best_width = max_w * 42u;
*best_height = max_h * 42u;
} else {
max_w = (uint32_t)floor(max_w_float);
max_h = (uint32_t)floor(max_h_float);
if (!v41) max_h &= ~1u;
if (max_w == 0 || max_h == 0) return 0;
const double scale = fmin((double)max_w * 42.0 / width,
(double)max_h * 42.0 / height);
*best_width = (uint32_t)floor(width * scale / 14.0) * 14u;
*best_height = (uint32_t)floor(height * scale / 14.0) * 14u;
}
return *best_width != 0 && *best_height != 0;
}
static int ds4_deepseek4_safe_resize(
uint32_t height,
uint32_t width,
uint32_t *best_height,
uint32_t *best_width,
uint32_t *llm_height,
uint32_t *llm_width,
bool v41) {
const uint32_t max_tokens = v41 ? 1024u : 381u;
uint32_t budget = max_tokens;
uint32_t tokens = ds4_deepseek4_grid_tokens(
*best_height, *best_width, llm_height, llm_width, v41);
while (tokens > max_tokens) {
if (budget <= 4u ||
!ds4_deepseek4_solve_resize(height, width, budget,
best_height, best_width, v41)) {
return 0;
}
tokens = ds4_deepseek4_grid_tokens(
*best_height, *best_width, llm_height, llm_width, v41);
budget--;
}
return 1;
}
static int ds4_image_preprocess_deepseek(
ds4_deepseek4_image_patches *out,
const ds4_image *image,
char *error,
size_t error_cap,
bool v41) {
if (!out) return 0;
memset(out, 0, sizeof(*out));
if (!image || !image->rgb || image->width == 0 || image->height == 0) {
ds4_image_error(error, error_cap,
"invalid DeepSeek V4 image preprocessing input");
return 0;
}
uint32_t planned_width = image->width;
uint32_t planned_height = image->height;
if (!v41 && (uint64_t)planned_width > (uint64_t)planned_height * 8u) {
planned_width = planned_height * 8u;
}
const uint64_t planned_pixels =
(uint64_t)planned_width * planned_height;
const uint32_t min_pixels = v41 ? 295936u : 147456u;
if (planned_pixels < min_pixels) {
const double scale = sqrt((double)min_pixels / planned_pixels);
planned_width = (uint32_t)(planned_width * scale);
planned_height = (uint32_t)(planned_height * scale);
if (planned_width == 0) planned_width = 1;
if (planned_height == 0) planned_height = 1;
}
uint32_t best_width = ds4_align_u32(planned_width, 14u);
uint32_t best_height = ds4_align_u32(planned_height, 14u);
uint32_t llm_height = 0, llm_width = 0;
if (!ds4_deepseek4_safe_resize(planned_height, planned_width,
&best_height, &best_width,
&llm_height, &llm_width, v41)) {
ds4_image_error(error, error_cap,
"unable to fit image within the vision token budget");
return 0;
}
if (best_width > DS4_IMAGE_MAX_DIMENSION ||
best_height > DS4_IMAGE_MAX_DIMENSION ||
(uint64_t)best_width * best_height > DS4_IMAGE_MAX_PIXELS) {
ds4_image_error(error, error_cap,
"resized image exceeds the decoded image limits");
return 0;
}
size_t canvas_values = (size_t)best_width * best_height * 3u;
float *canvas = malloc(canvas_values * sizeof(*canvas));
if (!canvas) {
ds4_image_error(error, error_cap, "unable to allocate resized image");
return 0;
}
for (size_t i = 0; i < canvas_values; i++) canvas[i] = 127.0f;
uint32_t content_width = best_width;
uint32_t content_height = best_height;
uint32_t offset_x = 0, offset_y = 0;
const bool force_resize =
!v41 && (uint64_t)image->width >= (uint64_t)image->height * 8u;
if (!force_resize) {
const double scale = fmin((double)best_width / image->width,
(double)best_height / image->height);
content_width = (uint32_t)lrint(image->width * scale);
content_height = (uint32_t)lrint(image->height * scale);
if (content_width < 1) content_width = 1;
if (content_height < 1) content_height = 1;
if (content_width > best_width) content_width = best_width;
if (content_height > best_height) content_height = best_height;
offset_x = (uint32_t)lrint((best_width - content_width) * 0.5);
offset_y = (uint32_t)lrint((best_height - content_height) * 0.5);
}
float *resized = canvas + ((size_t)offset_y * best_width + offset_x) * 3u;
ds4_resize_rgb_bicubic(image->rgb, image->width, image->height,
resized, content_width, content_height, best_width);
for (size_t i = 0; i < canvas_values; i++) {
canvas[i] = canvas[i] / 127.5f - 1.0f;
}
const uint32_t grid_height = best_height / 14u;
const uint32_t grid_width = best_width / 14u;
const uint32_t patch_count = grid_height * grid_width;
const size_t patch_values = (size_t)patch_count * 3u * 14u * 14u;
float *patches = malloc(patch_values * sizeof(*patches));
if (!patches) {
free(canvas);
ds4_image_error(error, error_cap, "unable to allocate vision patches");
return 0;
}
size_t index = 0;
for (uint32_t patch_y = 0; patch_y < grid_height; patch_y++) {
for (uint32_t patch_x = 0; patch_x < grid_width; patch_x++) {
for (uint32_t channel = 0; channel < 3u; channel++) {
for (uint32_t y = 0; y < 14u; y++) {
for (uint32_t x = 0; x < 14u; x++) {
const float *pixel = canvas +
((size_t)(patch_y * 14u + y) * best_width +
patch_x * 14u + x) * 3u;
patches[index++] = pixel[channel];
}
}
}
}
}
free(canvas);
if (index != patch_values) {
free(patches);
ds4_image_error(error, error_cap,
"internal DeepSeek vision patch layout mismatch");
return 0;
}
out->content_width = content_width;
out->content_height = content_height;
out->padded_width = best_width;
out->padded_height = best_height;
out->grid_width = grid_width;
out->grid_height = grid_height;
out->llm_grid_width = llm_width;
out->llm_grid_height = llm_height;
out->patch_count = patch_count;
out->patches = patches;
return 1;
}
int ds4_image_preprocess_deepseek4(
ds4_deepseek4_image_patches *out, const ds4_image *image,
char *error, size_t error_cap) {
return ds4_image_preprocess_deepseek(out, image, error, error_cap, false);
}
int ds4_image_preprocess_deepseek41(
ds4_deepseek4_image_patches *out, const ds4_image *image,
char *error, size_t error_cap) {
return ds4_image_preprocess_deepseek(out, image, error, error_cap, true);
}
int ds4_deepseek41_image_layout_build(
ds4_deepseek4_image_layout *out, uint32_t height, uint32_t width,
char *error, size_t error_cap) {
if (!out) return 0;
memset(out, 0, sizeof(*out));
const uint64_t count = (uint64_t)height * ((uint64_t)width + 1u) + 2u;
if (!height || !width || count > 1024u) {
ds4_image_error(error, error_cap, "invalid V4.1 image token grid");
return 0;
}
uint8_t *types = malloc((size_t)count);
uint32_t *perm = malloc((size_t)height * width * sizeof(*perm));
if (!types || !perm) {
free(types); free(perm);
ds4_image_error(error, error_cap, "unable to allocate V4.1 image layout");
return 0;
}
uint32_t token = 0, image = 0;
types[token++] = DS4_DEEPSEEK4_IMAGE_START;
for (uint32_t row = 0; row < height; row++) {
for (uint32_t col = 0; col < width; col++) {
types[token++] = DS4_DEEPSEEK4_IMAGE;
perm[image] = image;
image++;
}
types[token++] = DS4_DEEPSEEK4_IMAGE_NEWLINE;
}
types[token++] = DS4_DEEPSEEK4_IMAGE_END;
*out = (ds4_deepseek4_image_layout){token, image, types, perm};
return 1;
}
int ds4_deepseek4_image_layout_build(
ds4_deepseek4_image_layout *out,
uint32_t grid_height,
uint32_t grid_width,
uint32_t start_pos,
char *error,
size_t error_cap) {
if (!out) return 0;
memset(out, 0, sizeof(*out));
if (grid_height == 0 || grid_width == 0) {
ds4_image_error(error, error_cap, "invalid DeepSeek image token grid");
return 0;
}
const uint32_t pad_height = grid_height & 1u;
const uint32_t rows = grid_height + pad_height;
const uint32_t row_length = grid_width + 1u;
const uint32_t compress_pad = 3u - start_pos % 4u;
const uint32_t pad_last = (((rows / 2u) * row_length) & 1u) * 2u;
const uint64_t body_count64 = (uint64_t)rows * row_length;
const uint64_t token_count64 =
compress_pad + 1u + body_count64 + pad_last + 1u;
const uint64_t image_count64 = (uint64_t)grid_height * grid_width;
if (token_count64 > 384u || token_count64 > UINT32_MAX ||
image_count64 > UINT32_MAX) {
ds4_image_error(error, error_cap,
"DeepSeek image token layout exceeds its model limit");
return 0;
}
const uint32_t body_count = (uint32_t)body_count64;
uint8_t *body_types = malloc(body_count);
int32_t *body_images = malloc((size_t)body_count * sizeof(*body_images));
uint8_t *types = malloc((size_t)token_count64);
uint32_t *perm = malloc((size_t)image_count64 * sizeof(*perm));
if (!body_types || !body_images || !types || !perm) {
free(body_types);
free(body_images);
free(types);
free(perm);
ds4_image_error(error, error_cap,
"unable to allocate DeepSeek image token layout");
return 0;
}
for (uint32_t row = 0; row < rows; row++) {
for (uint32_t column = 0; column < row_length; column++) {
const uint32_t index = row * row_length + column;
if (row < grid_height && column < grid_width) {
body_types[index] = DS4_DEEPSEEK4_IMAGE;
body_images[index] = (int32_t)(row * grid_width + column);
} else {
body_types[index] = row < grid_height
? DS4_DEEPSEEK4_IMAGE_NEWLINE
: DS4_DEEPSEEK4_IMAGE_PAD;
body_images[index] = -1;
}
}
}
uint32_t type_pos = 0, perm_pos = 0;
for (uint32_t i = 0; i < compress_pad; i++)
types[type_pos++] = DS4_DEEPSEEK4_IMAGE_PAD;
types[type_pos++] = DS4_DEEPSEEK4_IMAGE_START;
for (uint32_t pair = 0; pair < rows / 2u; pair++) {
for (uint32_t column = 0; column < row_length; column++) {
for (uint32_t row_in_pair = 0; row_in_pair < 2u; row_in_pair++) {
const uint32_t source =
(pair * 2u + row_in_pair) * row_length + column;
types[type_pos++] = body_types[source];
if (body_images[source] >= 0)
perm[perm_pos++] = (uint32_t)body_images[source];
}
}
}
for (uint32_t i = 0; i < pad_last; i++)
types[type_pos++] = DS4_DEEPSEEK4_IMAGE_PAD;
types[type_pos++] = DS4_DEEPSEEK4_IMAGE_END;
free(body_images);
free(body_types);
if (type_pos != token_count64 || perm_pos != image_count64) {
free(types);
free(perm);
ds4_image_error(error, error_cap,
"internal DeepSeek image token layout mismatch");
return 0;
}
out->token_count = type_pos;
out->image_count = perm_pos;
out->types = types;
out->perm = perm;
return 1;
}
void ds4_deepseek4_image_patches_free(ds4_deepseek4_image_patches *patches) {
if (!patches) return;
free(patches->patches);
memset(patches, 0, sizeof(*patches));
}
void ds4_deepseek4_image_layout_free(ds4_deepseek4_image_layout *layout) {
if (!layout) return;
free(layout->types);
free(layout->perm);
memset(layout, 0, sizeof(*layout));
}
int ds4_deepseek4_next_image_span(
const int *tokens,
uint32_t token_count,
uint32_t vocab_size,
uint32_t *cursor,
uint32_t *block_start,
uint32_t *image_start,
uint32_t *image_end) {
if (!tokens || !cursor || !block_start || !image_start || !image_end ||
vocab_size == 0 || *cursor > token_count) return -1;
uint32_t i = *cursor;
while (i < token_count && tokens[i] >= 0 &&
(uint32_t)tokens[i] < vocab_size) i++;
if (i == token_count) {
*cursor = i;
return 0;
}
if (tokens[i] < 0) return -1;
const uint32_t block = i;
const int pad = (int)vocab_size + DS4_DEEPSEEK4_IMAGE_PAD;
while (i < token_count && tokens[i] == pad) i++;
if (i == token_count ||
tokens[i] != (int)vocab_size + DS4_DEEPSEEK4_IMAGE_START) {
return -1;
}
const uint32_t start = i++;
while (i < token_count) {
const int token = tokens[i];
if (token < (int)vocab_size ||
token > (int)vocab_size + DS4_DEEPSEEK4_IMAGE_END ||
token == (int)vocab_size + DS4_DEEPSEEK4_IMAGE_START) {
return -1;
}
if (token == (int)vocab_size + DS4_DEEPSEEK4_IMAGE_END) {
*block_start = block;
*image_start = start;
*image_end = i;
*cursor = i + 1u;
return 1;
}
i++;
}
return -1;
}
int ds4_deepseek4_prefill_chunk(
const int *tokens,
uint32_t token_count,
uint32_t vocab_size,
uint32_t pos0,
uint32_t cap,
uint32_t *chunk) {
if (!tokens || !chunk || vocab_size == 0 || cap == 0 ||
pos0 >= token_count) return 0;
uint32_t limit = pos0 +
(token_count - pos0 < cap ? token_count - pos0 : cap);
uint32_t cursor = pos0;
for (;;) {
uint32_t block_start, image_start, image_end;
const int parsed = ds4_deepseek4_next_image_span(
tokens, token_count, vocab_size, &cursor,
&block_start, &image_start, &image_end);
(void)image_start;
if (parsed < 0) return 0;
if (parsed == 0 || block_start >= limit) break;
const uint32_t after_image = image_end + 1u;
if (after_image > limit) {
if (after_image - pos0 > cap) {
if (block_start == pos0) return 0;
limit = block_start;
break;
}
limit = after_image;
}
}
*chunk = limit - pos0;
return *chunk != 0;
}
int ds4_deepseek4_attention_bounds(
const int *tokens,
uint32_t token_count,
uint32_t vocab_size,
uint32_t pos0,
uint32_t n_raw,
uint32_t window,
uint32_t *bounds) {
if (!tokens || !bounds || token_count == 0 || vocab_size == 0 ||
pos0 > UINT32_MAX - token_count || n_raw == 0 ||
n_raw > pos0 + token_count) return 0;
const uint32_t first_raw_pos = pos0 + token_count - n_raw;
const uint32_t last_raw_pos = first_raw_pos + n_raw - 1u;
for (uint32_t q = 0; q < token_count; q++) {
const uint32_t qpos = pos0 + q;
uint32_t lo = first_raw_pos;
if (window != 0 && qpos + 1u > window) {
const uint32_t window_lo = qpos + 1u - window;
if (window_lo > lo) lo = window_lo;
}
bounds[2u * q] = lo;
bounds[2u * q + 1u] = qpos < last_raw_pos ? qpos : last_raw_pos;
}
uint32_t cursor = 0;
for (;;) {
uint32_t block_start, image_start, image_end;
const int parsed = ds4_deepseek4_next_image_span(
tokens, token_count, vocab_size, &cursor,
&block_start, &image_start, &image_end);
(void)block_start;
if (parsed < 0) return 0;
if (parsed == 0) break;
for (uint32_t q = image_start; q <= image_end; q++) {
const uint32_t qpos = pos0 + q;
const uint32_t left = q - image_start < 383u
? q - image_start : 383u;
const uint32_t right = image_end - q < 384u
? image_end - q : 384u;
const uint32_t left_add = window != 0 && left >= window
? left - (window - 1u) : 0u;
uint32_t lo = 0;
if (window != 0) {
const uint32_t back = window - 1u + left_add;
lo = qpos > back ? qpos - back : 0u;
}
uint32_t hi = qpos + right;
if (lo < first_raw_pos) lo = first_raw_pos;
if (hi > last_raw_pos) hi = last_raw_pos;
bounds[2u * q] = lo;
bounds[2u * q + 1u] = hi;
}
}
return 1;
}
ds4_image.h
+102
View File
@@ -4,6 +4,10 @@
#include <stddef.h>
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
#define DS4_IMAGE_MAX_ENCODED_BYTES (64u * 1024u * 1024u)
#define DS4_IMAGE_MAX_DIMENSION 16384u
#define DS4_IMAGE_MAX_PIXELS (64u * 1024u * 1024u)
@@ -27,6 +31,34 @@ typedef struct {
float *patches;
} ds4_image_patches;
typedef struct {
uint32_t content_width;
uint32_t content_height;
uint32_t padded_width;
uint32_t padded_height;
uint32_t grid_width;
uint32_t grid_height;
uint32_t llm_grid_width;
uint32_t llm_grid_height;
uint32_t patch_count;
float *patches;
} ds4_deepseek4_image_patches;
typedef enum {
DS4_DEEPSEEK4_IMAGE_START = 0,
DS4_DEEPSEEK4_IMAGE_PAD = 1,
DS4_DEEPSEEK4_IMAGE = 2,
DS4_DEEPSEEK4_IMAGE_NEWLINE = 3,
DS4_DEEPSEEK4_IMAGE_END = 4,
} ds4_deepseek4_image_token_type;
typedef struct {
uint32_t token_count;
uint32_t image_count;
uint8_t *types;
uint32_t *perm;
} ds4_deepseek4_image_layout;
int ds4_image_decode_memory(
ds4_image *out,
const uint8_t *encoded,
@@ -50,6 +82,76 @@ int ds4_image_preprocess_glm53(
char *error,
size_t error_cap);
/* Qwen3-VL style: resize to multiples of 32 within the token budget, normalize
* with mean/std 0.5, emit 3*16*16 patches in 2x2 merge-window order. */
int ds4_image_preprocess_qwen4(
ds4_image_patches *out,
const ds4_image *image,
uint32_t min_image_tokens,
uint32_t max_image_tokens,
char *error,
size_t error_cap);
void ds4_image_patches_free(ds4_image_patches *patches);
int ds4_image_preprocess_deepseek4(
ds4_deepseek4_image_patches *out,
const ds4_image *image,
char *error,
size_t error_cap);
int ds4_image_preprocess_deepseek41(
ds4_deepseek4_image_patches *out, const ds4_image *image,
char *error, size_t error_cap);
int ds4_deepseek41_image_layout_build(
ds4_deepseek4_image_layout *out, uint32_t height, uint32_t width,
char *error, size_t error_cap);
int ds4_deepseek4_image_layout_build(
ds4_deepseek4_image_layout *out,
uint32_t grid_height,
uint32_t grid_width,
uint32_t start_pos,
char *error,
size_t error_cap);
void ds4_deepseek4_image_patches_free(ds4_deepseek4_image_patches *patches);
void ds4_deepseek4_image_layout_free(ds4_deepseek4_image_layout *layout);
/* Find the next DeepSeek image block at or after *cursor. A block includes
* the alignment PAD tokens immediately before IMAGE_START. Returns 1 when a
* block was found, 0 at end of input, and -1 for malformed synthetic tokens. */
int ds4_deepseek4_next_image_span(
const int *tokens,
uint32_t token_count,
uint32_t vocab_size,
uint32_t *cursor,
uint32_t *block_start,
uint32_t *image_start,
uint32_t *image_end);
/* Choose a prefill chunk without splitting a DeepSeek image block. */
int ds4_deepseek4_prefill_chunk(
const int *tokens,
uint32_t token_count,
uint32_t vocab_size,
uint32_t pos0,
uint32_t cap,
uint32_t *chunk);
/* Fill [lo, hi] absolute raw-key bounds for every query row. */
int ds4_deepseek4_attention_bounds(
const int *tokens,
uint32_t token_count,
uint32_t vocab_size,
uint32_t pos0,
uint32_t n_raw,
uint32_t window,
uint32_t *bounds);
#ifdef __cplusplus
}
#endif
#endif
ds4_kvstore.c
+9 -3
View File
@@ -414,6 +414,12 @@ void ds4_kvstore_fill_header(uint8_t h[DS4_KVSTORE_FIXED_HEADER],
kv_le_put64(h + 40, payload_bytes);
}
/* routed-expert quantizations whose checkpoints the store keys on */
bool ds4_kvstore_quant_bits_supported(int quant_bits) {
return quant_bits == 2 || quant_bits == 4 || quant_bits == 5 ||
quant_bits == 6 || quant_bits == 8;
}
bool ds4_kvstore_read_header(FILE *fp, ds4_kvstore_entry *e,
uint32_t *text_bytes) {
uint8_t h[DS4_KVSTORE_FIXED_HEADER];
@@ -436,7 +442,7 @@ bool ds4_kvstore_read_header(FILE *fp, ds4_kvstore_entry *e,
if (fread(tb, 1, sizeof(tb), fp) != sizeof(tb)) return false;
*text_bytes = ds4_kvstore_le_get32(tb);
e->text_bytes = *text_bytes;
return e->tokens != 0 && (e->quant_bits == 2 || e->quant_bits == 4);
return e->tokens != 0 && ds4_kvstore_quant_bits_supported(e->quant_bits);
}
bool ds4_kvstore_read_entry_file(const char *path, const char sha[41],
@@ -940,7 +946,7 @@ bool ds4_kvstore_store_live_prefix_text(ds4_kvstore *kc,
ds4_kvstore_tokens_copy_prefix(&store_tokens, tokens, store_len);
const int quant_bits = ds4_engine_routed_quant_bits(engine);
if (quant_bits != 2 && quant_bits != 4) {
if (!ds4_kvstore_quant_bits_supported(quant_bits)) {
ds4_tokens_free(&store_tokens);
return false;
}
@@ -1224,7 +1230,7 @@ int ds4_kvstore_try_load_text(ds4_kvstore *kc,
if (effective_prompt) effective_prompt->len = 0;
if (!kc->enabled || !prompt_text) return 0;
const int quant_bits = ds4_engine_routed_quant_bits(engine);
if (quant_bits != 2 && quant_bits != 4) return 0;
if (!ds4_kvstore_quant_bits_supported(quant_bits)) return 0;
const int model_id = ds4_engine_model_id(engine);
const size_t prompt_bytes = strlen(prompt_text);
int idx = ds4_kvstore_find_text_prefix(kc, prompt_text, model_id, quant_bits,
ds4_kvstore.h
+2
View File
@@ -214,4 +214,6 @@ char *ds4_kvstore_path_for_sha(ds4_kvstore *kc, const char sha[41]);
void ds4_kvstore_le_put32(uint8_t *p, uint32_t v);
uint32_t ds4_kvstore_le_get32(const uint8_t *p);
bool ds4_kvstore_quant_bits_supported(int quant_bits);
#endif
ds4_linux_memory.h
+37
View File
@@ -0,0 +1,37 @@
#ifndef DS4_LINUX_MEMORY_H
#define DS4_LINUX_MEMORY_H
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
/* CMA pages can back movable allocations, but not ordinary pinned GPU pages. */
static inline bool ds4_linux_nonmovable_memory_parse(FILE *fp, uint64_t *bytes) {
char line[256];
uint64_t available = 0, cma = 0;
bool found = false;
while (fgets(line, sizeof(line), fp)) {
unsigned long long kib;
if (sscanf(line, "MemAvailable: %llu kB", &kib) == 1) {
if (kib > UINT64_MAX / 1024u) return false;
available = (uint64_t)kib * 1024u;
found = true;
} else if (sscanf(line, "CmaFree: %llu kB", &kib) == 1) {
if (kib > UINT64_MAX / 1024u) return false;
cma = (uint64_t)kib * 1024u;
}
}
if (!found || ferror(fp)) return false;
*bytes = available > cma ? available - cma : 0;
return true;
}
static inline bool ds4_linux_nonmovable_memory(uint64_t *bytes) {
FILE *fp = fopen("/proc/meminfo", "r");
if (!fp) return false;
const bool ok = ds4_linux_nonmovable_memory_parse(fp, bytes);
fclose(fp);
return ok;
}
#endif
ds4_metal.m
+10442 -742
View File
File diff suppressed because it is too large. Load diff
ds4_metal_device.h
+31
View File
@@ -0,0 +1,31 @@
#ifndef DS4_METAL_DEVICE_H
#define DS4_METAL_DEVICE_H
#include <string.h>
/* Share the Qwen M1 Max defaults with M2-M4 while keeping other M1 devices
* and newer generations on their existing paths. Bound the generation token
* so names such as M20 do not inherit these defaults. */
static inline int ds4_metal_device_name_uses_qwen_legacy_tuning(const char *name) {
return name && (strcmp(name, "Apple M1 Max") == 0 ||
(strncmp(name, "Apple M", 7) == 0 &&
name[7] >= '2' && name[7] <= '4' &&
(name[8] == '\0' || name[8] == ' ')));
}
/* M3 Ultra already has dedicated MoE defaults; preserve those selections. */
static inline int ds4_metal_device_name_uses_qwen_legacy_moe_tuning(const char *name) {
return ds4_metal_device_name_uses_qwen_legacy_tuning(name) &&
strcmp(name, "Apple M3 Ultra") != 0;
}
/* Share the M5 tuning policy with M6. Keep the generation token bounded so
* untested future names such as M50 and M60 do not inherit these defaults.
* Callers must still check runtime and kernel capabilities where required. */
static inline int ds4_metal_device_name_is_m5_or_m6(const char *name) {
return name && strncmp(name, "Apple M", 7) == 0 &&
(name[7] == '5' || name[7] == '6') &&
(name[8] == '\0' || name[8] == ' ');
}
#endif
ds4_prompt_prefix.c
+249
View File
@@ -0,0 +1,249 @@
#include "ds4_prompt_prefix.h"
#include <errno.h>
#include <stdarg.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
static void prefix_error(char *error, size_t error_cap, const char *fmt, ...) {
if (!error || error_cap == 0) return;
va_list ap;
va_start(ap, fmt);
vsnprintf(error, error_cap, fmt, ap);
va_end(ap);
}
static bool prefix_marker_at(const char *data, size_t len, size_t pos,
ds4_prompt_prefix_role *role,
size_t *marker_len) {
static const char user[] = "USER:";
static const char assistant[] = "ASSISTANT:";
if (len - pos >= sizeof(user) - 1 &&
memcmp(data + pos, user, sizeof(user) - 1) == 0) {
*role = DS4_PROMPT_PREFIX_USER;
*marker_len = sizeof(user) - 1;
return true;
}
if (len - pos >= sizeof(assistant) - 1 &&
memcmp(data + pos, assistant, sizeof(assistant) - 1) == 0) {
*role = DS4_PROMPT_PREFIX_ASSISTANT;
*marker_len = sizeof(assistant) - 1;
return true;
}
return false;
}
static char *prefix_copy_content(const char *data, size_t start, size_t end) {
size_t len = end - start;
char *content = malloc(len + 1);
if (!content) return NULL;
memcpy(content, data + start, len);
content[len] = '\0';
return content;
}
void ds4_prompt_prefix_free(ds4_prompt_prefix *prefix) {
if (!prefix) return;
for (size_t i = 0; i < prefix->count; i++)
free(prefix->turns[i].content);
free(prefix->turns);
memset(prefix, 0, sizeof(*prefix));
}
int ds4_prompt_prefix_parse(ds4_prompt_prefix *out,
const char *data,
size_t len,
char *error,
size_t error_cap) {
if (!out) {
prefix_error(error, error_cap, "invalid prefix output");
return -1;
}
memset(out, 0, sizeof(*out));
if (!data || len == 0) {
prefix_error(error, error_cap, "prefix file is empty");
return -1;
}
if (memchr(data, '\0', len)) {
prefix_error(error, error_cap, "prefix file contains a NUL byte");
return -1;
}
size_t pos = 0;
if (len >= 3 && (unsigned char)data[0] == 0xef &&
(unsigned char)data[1] == 0xbb &&
(unsigned char)data[2] == 0xbf) {
pos = 3;
}
size_t line = 1;
while (pos < len) {
ds4_prompt_prefix_role role;
size_t marker_len = 0;
if (!prefix_marker_at(data, len, pos, &role, &marker_len)) {
prefix_error(error, error_cap,
"line %zu must start with USER: or ASSISTANT:", line);
ds4_prompt_prefix_free(out);
return -1;
}
const ds4_prompt_prefix_role expected =
(out->count % 2 == 0) ? DS4_PROMPT_PREFIX_USER
: DS4_PROMPT_PREFIX_ASSISTANT;
if (role != expected) {
prefix_error(error, error_cap, "line %zu: expected %s:", line,
expected == DS4_PROMPT_PREFIX_USER ? "USER"
: "ASSISTANT");
ds4_prompt_prefix_free(out);
return -1;
}
size_t content_start = pos + marker_len;
if (content_start < len &&
(data[content_start] == ' ' || data[content_start] == '\t')) {
content_start++;
}
size_t next = len;
size_t next_line = line;
size_t scan_line = line;
for (size_t i = content_start; i < len; i++) {
if (data[i] != '\n') continue;
scan_line++;
ds4_prompt_prefix_role ignored_role;
size_t ignored_len;
if (i + 1 < len &&
prefix_marker_at(data, len, i + 1,
&ignored_role, &ignored_len)) {
next = i + 1;
next_line = scan_line;
break;
}
}
size_t content_end = next;
if (content_end > content_start && data[content_end - 1] == '\n') {
content_end--;
if (content_end > content_start && data[content_end - 1] == '\r')
content_end--;
}
if (content_end == content_start) {
prefix_error(error, error_cap, "line %zu has an empty %s turn",
line, role == DS4_PROMPT_PREFIX_USER ? "USER"
: "ASSISTANT");
ds4_prompt_prefix_free(out);
return -1;
}
if (out->count == SIZE_MAX / sizeof(out->turns[0])) {
prefix_error(error, error_cap, "too many turns in prefix file");
ds4_prompt_prefix_free(out);
return -1;
}
ds4_prompt_prefix_turn *turns =
realloc(out->turns, (out->count + 1) * sizeof(out->turns[0]));
if (!turns) {
prefix_error(error, error_cap, "out of memory reading prefix file");
ds4_prompt_prefix_free(out);
return -1;
}
out->turns = turns;
char *content = prefix_copy_content(data, content_start, content_end);
if (!content) {
prefix_error(error, error_cap, "out of memory reading prefix file");
ds4_prompt_prefix_free(out);
return -1;
}
out->turns[out->count++] = (ds4_prompt_prefix_turn) {
.role = role,
.content = content,
};
if (next == len) {
pos = len;
} else {
pos = next;
line = next_line;
}
}
if (out->count == 0) {
prefix_error(error, error_cap, "prefix file contains no turns");
return -1;
}
if (out->turns[out->count - 1].role != DS4_PROMPT_PREFIX_ASSISTANT) {
prefix_error(error, error_cap,
"prefix file must end with an ASSISTANT turn");
ds4_prompt_prefix_free(out);
return -1;
}
return 0;
}
int ds4_prompt_prefix_load(ds4_prompt_prefix *out,
const char *path,
char *error,
size_t error_cap) {
if (!out || !path || !path[0]) {
prefix_error(error, error_cap, "invalid prefix file path");
return -1;
}
memset(out, 0, sizeof(*out));
FILE *fp = fopen(path, "rb");
if (!fp) {
prefix_error(error, error_cap, "cannot open prefix file %s: %s",
path, strerror(errno));
return -1;
}
if (fseek(fp, 0, SEEK_END) != 0) {
prefix_error(error, error_cap, "cannot seek prefix file %s: %s",
path, strerror(errno));
fclose(fp);
return -1;
}
long size = ftell(fp);
if (size < 0 || (uintmax_t)size > SIZE_MAX - 1) {
prefix_error(error, error_cap, "cannot size prefix file %s", path);
fclose(fp);
return -1;
}
if (fseek(fp, 0, SEEK_SET) != 0) {
prefix_error(error, error_cap, "cannot rewind prefix file %s: %s",
path, strerror(errno));
fclose(fp);
return -1;
}
size_t len = (size_t)size;
char *data = malloc(len + 1);
if (!data) {
prefix_error(error, error_cap, "out of memory reading prefix file");
fclose(fp);
return -1;
}
if (len && fread(data, 1, len, fp) != len) {
prefix_error(error, error_cap, "cannot read prefix file %s", path);
free(data);
fclose(fp);
return -1;
}
if (fclose(fp) != 0) {
prefix_error(error, error_cap, "cannot close prefix file %s: %s",
path, strerror(errno));
free(data);
return -1;
}
data[len] = '\0';
int rc = ds4_prompt_prefix_parse(out, data, len, error, error_cap);
free(data);
if (rc != 0 && error && error_cap && error[0]) {
char detail[256];
snprintf(detail, sizeof(detail), "%s", error);
prefix_error(error, error_cap, "%s: %s", path, detail);
}
return rc;
}
ds4_prompt_prefix.h
+49
View File
@@ -0,0 +1,49 @@
#ifndef DS4_PROMPT_PREFIX_H
#define DS4_PROMPT_PREFIX_H
#include "ds4.h"
#include <stddef.h>
typedef enum {
DS4_PROMPT_PREFIX_USER,
DS4_PROMPT_PREFIX_ASSISTANT,
} ds4_prompt_prefix_role;
typedef struct {
ds4_prompt_prefix_role role;
char *content;
} ds4_prompt_prefix_turn;
typedef struct {
ds4_prompt_prefix_turn *turns;
size_t count;
} ds4_prompt_prefix;
int ds4_prompt_prefix_parse(ds4_prompt_prefix *out,
const char *data,
size_t len,
char *error,
size_t error_cap);
int ds4_prompt_prefix_load(ds4_prompt_prefix *out,
const char *path,
char *error,
size_t error_cap);
void ds4_prompt_prefix_free(ds4_prompt_prefix *prefix);
static inline void ds4_prompt_prefix_append(ds4_engine *engine,
ds4_tokens *tokens,
const ds4_prompt_prefix *prefix) {
if (!prefix) return;
for (size_t i = 0; i < prefix->count; i++) {
const ds4_prompt_prefix_turn *turn = &prefix->turns[i];
const bool assistant = turn->role == DS4_PROMPT_PREFIX_ASSISTANT;
ds4_chat_append_message(engine, tokens,
assistant ? "assistant" : "user",
turn->content);
if (assistant && !ds4_engine_is_glm_dsa(engine))
ds4_tokens_push(tokens, ds4_token_eos(engine));
}
}
#endif
ds4_qwen4_cuda.cuh
+2356
View File
File diff suppressed because it is too large. Load diff
ds4_qwen4_unicode.inc
+310
View File
@@ -0,0 +1,310 @@
/* Generated by gguf-tools/gen_qwen4_unicode.py from the `regex` module's
* Unicode tables (regex 2026.9.3); codepoint classes for the
* Qwen3.8 pre-tokenizer. */
typedef struct { uint32_t lo, hi; } ds4_qwen4_urange;
static const ds4_qwen4_urange ds4_qwen4_letter_ranges[] = {
{0x00041, 0x0005a}, {0x00061, 0x0007a}, {0x000aa, 0x000aa}, {0x000b5, 0x000b5},
{0x000ba, 0x000ba}, {0x000c0, 0x000d6}, {0x000d8, 0x000f6}, {0x000f8, 0x002c1},
{0x002c6, 0x002d1}, {0x002e0, 0x002e4}, {0x002ec, 0x002ec}, {0x002ee, 0x002ee},
{0x00370, 0x00374}, {0x00376, 0x00377}, {0x0037a, 0x0037d}, {0x0037f, 0x0037f},
{0x00386, 0x00386}, {0x00388, 0x0038a}, {0x0038c, 0x0038c}, {0x0038e, 0x003a1},
{0x003a3, 0x003f5}, {0x003f7, 0x00481}, {0x0048a, 0x0052f}, {0x00531, 0x00556},
{0x00559, 0x00559}, {0x00560, 0x00588}, {0x005d0, 0x005ea}, {0x005ef, 0x005f2},
{0x00620, 0x0064a}, {0x0066e, 0x0066f}, {0x00671, 0x006d3}, {0x006d5, 0x006d5},
{0x006e5, 0x006e6}, {0x006ee, 0x006ef}, {0x006fa, 0x006fc}, {0x006ff, 0x006ff},
{0x00710, 0x00710}, {0x00712, 0x0072f}, {0x0074d, 0x007a5}, {0x007b1, 0x007b1},
{0x007ca, 0x007ea}, {0x007f4, 0x007f5}, {0x007fa, 0x007fa}, {0x00800, 0x00815},
{0x0081a, 0x0081a}, {0x00824, 0x00824}, {0x00828, 0x00828}, {0x00840, 0x00858},
{0x00860, 0x0086a}, {0x00870, 0x00887}, {0x00889, 0x0088f}, {0x008a0, 0x008c9},
{0x00904, 0x00939}, {0x0093d, 0x0093d}, {0x00950, 0x00950}, {0x00958, 0x00961},
{0x00971, 0x00980}, {0x00985, 0x0098c}, {0x0098f, 0x00990}, {0x00993, 0x009a8},
{0x009aa, 0x009b0}, {0x009b2, 0x009b2}, {0x009b6, 0x009b9}, {0x009bd, 0x009bd},
{0x009ce, 0x009ce}, {0x009dc, 0x009dd}, {0x009df, 0x009e1}, {0x009f0, 0x009f1},
{0x009fc, 0x009fc}, {0x00a05, 0x00a0a}, {0x00a0f, 0x00a10}, {0x00a13, 0x00a28},
{0x00a2a, 0x00a30}, {0x00a32, 0x00a33}, {0x00a35, 0x00a36}, {0x00a38, 0x00a39},
{0x00a59, 0x00a5c}, {0x00a5e, 0x00a5e}, {0x00a72, 0x00a74}, {0x00a85, 0x00a8d},
{0x00a8f, 0x00a91}, {0x00a93, 0x00aa8}, {0x00aaa, 0x00ab0}, {0x00ab2, 0x00ab3},
{0x00ab5, 0x00ab9}, {0x00abd, 0x00abd}, {0x00ad0, 0x00ad0}, {0x00ae0, 0x00ae1},
{0x00af9, 0x00af9}, {0x00b05, 0x00b0c}, {0x00b0f, 0x00b10}, {0x00b13, 0x00b28},
{0x00b2a, 0x00b30}, {0x00b32, 0x00b33}, {0x00b35, 0x00b39}, {0x00b3d, 0x00b3d},
{0x00b5c, 0x00b5d}, {0x00b5f, 0x00b61}, {0x00b71, 0x00b71}, {0x00b83, 0x00b83},
{0x00b85, 0x00b8a}, {0x00b8e, 0x00b90}, {0x00b92, 0x00b95}, {0x00b99, 0x00b9a},
{0x00b9c, 0x00b9c}, {0x00b9e, 0x00b9f}, {0x00ba3, 0x00ba4}, {0x00ba8, 0x00baa},
{0x00bae, 0x00bb9}, {0x00bd0, 0x00bd0}, {0x00c05, 0x00c0c}, {0x00c0e, 0x00c10},
{0x00c12, 0x00c28}, {0x00c2a, 0x00c39}, {0x00c3d, 0x00c3d}, {0x00c58, 0x00c5a},
{0x00c5c, 0x00c5d}, {0x00c60, 0x00c61}, {0x00c80, 0x00c80}, {0x00c85, 0x00c8c},
{0x00c8e, 0x00c90}, {0x00c92, 0x00ca8}, {0x00caa, 0x00cb3}, {0x00cb5, 0x00cb9},
{0x00cbd, 0x00cbd}, {0x00cdc, 0x00cde}, {0x00ce0, 0x00ce1}, {0x00cf1, 0x00cf2},
{0x00d04, 0x00d0c}, {0x00d0e, 0x00d10}, {0x00d12, 0x00d3a}, {0x00d3d, 0x00d3d},
{0x00d4e, 0x00d4e}, {0x00d54, 0x00d56}, {0x00d5f, 0x00d61}, {0x00d7a, 0x00d7f},
{0x00d85, 0x00d96}, {0x00d9a, 0x00db1}, {0x00db3, 0x00dbb}, {0x00dbd, 0x00dbd},
{0x00dc0, 0x00dc6}, {0x00e01, 0x00e30}, {0x00e32, 0x00e33}, {0x00e40, 0x00e46},
{0x00e81, 0x00e82}, {0x00e84, 0x00e84}, {0x00e86, 0x00e8a}, {0x00e8c, 0x00ea3},
{0x00ea5, 0x00ea5}, {0x00ea7, 0x00eb0}, {0x00eb2, 0x00eb3}, {0x00ebd, 0x00ebd},
{0x00ec0, 0x00ec4}, {0x00ec6, 0x00ec6}, {0x00edc, 0x00edf}, {0x00f00, 0x00f00},
{0x00f40, 0x00f47}, {0x00f49, 0x00f6c}, {0x00f88, 0x00f8c}, {0x01000, 0x0102a},
{0x0103f, 0x0103f}, {0x01050, 0x01055}, {0x0105a, 0x0105d}, {0x01061, 0x01061},
{0x01065, 0x01066}, {0x0106e, 0x01070}, {0x01075, 0x01081}, {0x0108e, 0x0108e},
{0x010a0, 0x010c5}, {0x010c7, 0x010c7}, {0x010cd, 0x010cd}, {0x010d0, 0x010fa},
{0x010fc, 0x01248}, {0x0124a, 0x0124d}, {0x01250, 0x01256}, {0x01258, 0x01258},
{0x0125a, 0x0125d}, {0x01260, 0x01288}, {0x0128a, 0x0128d}, {0x01290, 0x012b0},
{0x012b2, 0x012b5}, {0x012b8, 0x012be}, {0x012c0, 0x012c0}, {0x012c2, 0x012c5},
{0x012c8, 0x012d6}, {0x012d8, 0x01310}, {0x01312, 0x01315}, {0x01318, 0x0135a},
{0x01380, 0x0138f}, {0x013a0, 0x013f5}, {0x013f8, 0x013fd}, {0x01401, 0x0166c},
{0x0166f, 0x0167f}, {0x01681, 0x0169a}, {0x016a0, 0x016ea}, {0x016f1, 0x016f8},
{0x01700, 0x01711}, {0x0171f, 0x01731}, {0x01740, 0x01751}, {0x01760, 0x0176c},
{0x0176e, 0x01770}, {0x01780, 0x017b3}, {0x017d7, 0x017d7}, {0x017dc, 0x017dc},
{0x01820, 0x01878}, {0x01880, 0x01884}, {0x01887, 0x018a8}, {0x018aa, 0x018aa},
{0x018b0, 0x018f5}, {0x01900, 0x0191e}, {0x01950, 0x0196d}, {0x01970, 0x01974},
{0x01980, 0x019ab}, {0x019b0, 0x019c9}, {0x01a00, 0x01a16}, {0x01a20, 0x01a54},
{0x01aa7, 0x01aa7}, {0x01b05, 0x01b33}, {0x01b45, 0x01b4c}, {0x01b83, 0x01ba0},
{0x01bae, 0x01baf}, {0x01bba, 0x01be5}, {0x01c00, 0x01c23}, {0x01c4d, 0x01c4f},
{0x01c5a, 0x01c7d}, {0x01c80, 0x01c8a}, {0x01c90, 0x01cba}, {0x01cbd, 0x01cbf},
{0x01ce9, 0x01cec}, {0x01cee, 0x01cf3}, {0x01cf5, 0x01cf6}, {0x01cfa, 0x01cfa},
{0x01d00, 0x01dbf}, {0x01e00, 0x01f15}, {0x01f18, 0x01f1d}, {0x01f20, 0x01f45},
{0x01f48, 0x01f4d}, {0x01f50, 0x01f57}, {0x01f59, 0x01f59}, {0x01f5b, 0x01f5b},
{0x01f5d, 0x01f5d}, {0x01f5f, 0x01f7d}, {0x01f80, 0x01fb4}, {0x01fb6, 0x01fbc},
{0x01fbe, 0x01fbe}, {0x01fc2, 0x01fc4}, {0x01fc6, 0x01fcc}, {0x01fd0, 0x01fd3},
{0x01fd6, 0x01fdb}, {0x01fe0, 0x01fec}, {0x01ff2, 0x01ff4}, {0x01ff6, 0x01ffc},
{0x02071, 0x02071}, {0x0207f, 0x0207f}, {0x02090, 0x0209c}, {0x02102, 0x02102},
{0x02107, 0x02107}, {0x0210a, 0x02113}, {0x02115, 0x02115}, {0x02119, 0x0211d},
{0x02124, 0x02124}, {0x02126, 0x02126}, {0x02128, 0x02128}, {0x0212a, 0x0212d},
{0x0212f, 0x02139}, {0x0213c, 0x0213f}, {0x02145, 0x02149}, {0x0214e, 0x0214e},
{0x02183, 0x02184}, {0x02c00, 0x02ce4}, {0x02ceb, 0x02cee}, {0x02cf2, 0x02cf3},
{0x02d00, 0x02d25}, {0x02d27, 0x02d27}, {0x02d2d, 0x02d2d}, {0x02d30, 0x02d67},
{0x02d6f, 0x02d6f}, {0x02d80, 0x02d96}, {0x02da0, 0x02da6}, {0x02da8, 0x02dae},
{0x02db0, 0x02db6}, {0x02db8, 0x02dbe}, {0x02dc0, 0x02dc6}, {0x02dc8, 0x02dce},
{0x02dd0, 0x02dd6}, {0x02dd8, 0x02dde}, {0x02e2f, 0x02e2f}, {0x03005, 0x03006},
{0x03031, 0x03035}, {0x0303b, 0x0303c}, {0x03041, 0x03096}, {0x0309d, 0x0309f},
{0x030a1, 0x030fa}, {0x030fc, 0x030ff}, {0x03105, 0x0312f}, {0x03131, 0x0318e},
{0x031a0, 0x031bf}, {0x031f0, 0x031ff}, {0x03400, 0x04dbf}, {0x04e00, 0x0a48c},
{0x0a4d0, 0x0a4fd}, {0x0a500, 0x0a60c}, {0x0a610, 0x0a61f}, {0x0a62a, 0x0a62b},
{0x0a640, 0x0a66e}, {0x0a67f, 0x0a69d}, {0x0a6a0, 0x0a6e5}, {0x0a717, 0x0a71f},
{0x0a722, 0x0a788}, {0x0a78b, 0x0a7dc}, {0x0a7f1, 0x0a801}, {0x0a803, 0x0a805},
{0x0a807, 0x0a80a}, {0x0a80c, 0x0a822}, {0x0a840, 0x0a873}, {0x0a882, 0x0a8b3},
{0x0a8f2, 0x0a8f7}, {0x0a8fb, 0x0a8fb}, {0x0a8fd, 0x0a8fe}, {0x0a90a, 0x0a925},
{0x0a930, 0x0a946}, {0x0a960, 0x0a97c}, {0x0a984, 0x0a9b2}, {0x0a9cf, 0x0a9cf},
{0x0a9e0, 0x0a9e4}, {0x0a9e6, 0x0a9ef}, {0x0a9fa, 0x0a9fe}, {0x0aa00, 0x0aa28},
{0x0aa40, 0x0aa42}, {0x0aa44, 0x0aa4b}, {0x0aa60, 0x0aa76}, {0x0aa7a, 0x0aa7a},
{0x0aa7e, 0x0aaaf}, {0x0aab1, 0x0aab1}, {0x0aab5, 0x0aab6}, {0x0aab9, 0x0aabd},
{0x0aac0, 0x0aac0}, {0x0aac2, 0x0aac2}, {0x0aadb, 0x0aadd}, {0x0aae0, 0x0aaea},
{0x0aaf2, 0x0aaf4}, {0x0ab01, 0x0ab06}, {0x0ab09, 0x0ab0e}, {0x0ab11, 0x0ab16},
{0x0ab20, 0x0ab26}, {0x0ab28, 0x0ab2e}, {0x0ab30, 0x0ab5a}, {0x0ab5c, 0x0ab69},
{0x0ab70, 0x0abe2}, {0x0ac00, 0x0d7a3}, {0x0d7b0, 0x0d7c6}, {0x0d7cb, 0x0d7fb},
{0x0f900, 0x0fa6d}, {0x0fa70, 0x0fad9}, {0x0fb00, 0x0fb06}, {0x0fb13, 0x0fb17},
{0x0fb1d, 0x0fb1d}, {0x0fb1f, 0x0fb28}, {0x0fb2a, 0x0fb36}, {0x0fb38, 0x0fb3c},
{0x0fb3e, 0x0fb3e}, {0x0fb40, 0x0fb41}, {0x0fb43, 0x0fb44}, {0x0fb46, 0x0fbb1},
{0x0fbd3, 0x0fd3d}, {0x0fd50, 0x0fd8f}, {0x0fd92, 0x0fdc7}, {0x0fdf0, 0x0fdfb},
{0x0fe70, 0x0fe74}, {0x0fe76, 0x0fefc}, {0x0ff21, 0x0ff3a}, {0x0ff41, 0x0ff5a},
{0x0ff66, 0x0ffbe}, {0x0ffc2, 0x0ffc7}, {0x0ffca, 0x0ffcf}, {0x0ffd2, 0x0ffd7},
{0x0ffda, 0x0ffdc}, {0x10000, 0x1000b}, {0x1000d, 0x10026}, {0x10028, 0x1003a},
{0x1003c, 0x1003d}, {0x1003f, 0x1004d}, {0x10050, 0x1005d}, {0x10080, 0x100fa},
{0x10280, 0x1029c}, {0x102a0, 0x102d0}, {0x10300, 0x1031f}, {0x1032d, 0x10340},
{0x10342, 0x10349}, {0x10350, 0x10375}, {0x10380, 0x1039d}, {0x103a0, 0x103c3},
{0x103c8, 0x103cf}, {0x10400, 0x1049d}, {0x104b0, 0x104d3}, {0x104d8, 0x104fb},
{0x10500, 0x10527}, {0x10530, 0x10563}, {0x10570, 0x1057a}, {0x1057c, 0x1058a},
{0x1058c, 0x10592}, {0x10594, 0x10595}, {0x10597, 0x105a1}, {0x105a3, 0x105b1},
{0x105b3, 0x105b9}, {0x105bb, 0x105bc}, {0x105c0, 0x105f3}, {0x10600, 0x10736},
{0x10740, 0x10755}, {0x10760, 0x10767}, {0x10780, 0x10785}, {0x10787, 0x107b0},
{0x107b2, 0x107ba}, {0x10800, 0x10805}, {0x10808, 0x10808}, {0x1080a, 0x10835},
{0x10837, 0x10838}, {0x1083c, 0x1083c}, {0x1083f, 0x10855}, {0x10860, 0x10876},
{0x10880, 0x1089e}, {0x108e0, 0x108f2}, {0x108f4, 0x108f5}, {0x10900, 0x10915},
{0x10920, 0x10939}, {0x10940, 0x10959}, {0x10980, 0x109b7}, {0x109be, 0x109bf},
{0x10a00, 0x10a00}, {0x10a10, 0x10a13}, {0x10a15, 0x10a17}, {0x10a19, 0x10a35},
{0x10a60, 0x10a7c}, {0x10a80, 0x10a9c}, {0x10ac0, 0x10ac7}, {0x10ac9, 0x10ae4},
{0x10b00, 0x10b35}, {0x10b40, 0x10b55}, {0x10b60, 0x10b72}, {0x10b80, 0x10b91},
{0x10c00, 0x10c48}, {0x10c80, 0x10cb2}, {0x10cc0, 0x10cf2}, {0x10d00, 0x10d23},
{0x10d4a, 0x10d65}, {0x10d6f, 0x10d85}, {0x10e80, 0x10ea9}, {0x10eb0, 0x10eb1},
{0x10ec2, 0x10ec7}, {0x10f00, 0x10f1c}, {0x10f27, 0x10f27}, {0x10f30, 0x10f45},
{0x10f70, 0x10f81}, {0x10fb0, 0x10fc4}, {0x10fe0, 0x10ff6}, {0x11003, 0x11037},
{0x11071, 0x11072}, {0x11075, 0x11075}, {0x11083, 0x110af}, {0x110d0, 0x110e8},
{0x11103, 0x11126}, {0x11144, 0x11144}, {0x11147, 0x11147}, {0x11150, 0x11172},
{0x11176, 0x11176}, {0x11183, 0x111b2}, {0x111c1, 0x111c4}, {0x111da, 0x111da},
{0x111dc, 0x111dc}, {0x11200, 0x11211}, {0x11213, 0x1122b}, {0x1123f, 0x11240},
{0x11280, 0x11286}, {0x11288, 0x11288}, {0x1128a, 0x1128d}, {0x1128f, 0x1129d},
{0x1129f, 0x112a8}, {0x112b0, 0x112de}, {0x11305, 0x1130c}, {0x1130f, 0x11310},
{0x11313, 0x11328}, {0x1132a, 0x11330}, {0x11332, 0x11333}, {0x11335, 0x11339},
{0x1133d, 0x1133d}, {0x11350, 0x11350}, {0x1135d, 0x11361}, {0x11380, 0x11389},
{0x1138b, 0x1138b}, {0x1138e, 0x1138e}, {0x11390, 0x113b5}, {0x113b7, 0x113b7},
{0x113d1, 0x113d1}, {0x113d3, 0x113d3}, {0x11400, 0x11434}, {0x11447, 0x1144a},
{0x1145f, 0x11461}, {0x11480, 0x114af}, {0x114c4, 0x114c5}, {0x114c7, 0x114c7},
{0x11580, 0x115ae}, {0x115d8, 0x115db}, {0x11600, 0x1162f}, {0x11644, 0x11644},
{0x11680, 0x116aa}, {0x116b8, 0x116b8}, {0x11700, 0x1171a}, {0x11740, 0x11746},
{0x11800, 0x1182b}, {0x118a0, 0x118df}, {0x118ff, 0x11906}, {0x11909, 0x11909},
{0x1190c, 0x11913}, {0x11915, 0x11916}, {0x11918, 0x1192f}, {0x1193f, 0x1193f},
{0x11941, 0x11941}, {0x119a0, 0x119a7}, {0x119aa, 0x119d0}, {0x119e1, 0x119e1},
{0x119e3, 0x119e3}, {0x11a00, 0x11a00}, {0x11a0b, 0x11a32}, {0x11a3a, 0x11a3a},
{0x11a50, 0x11a50}, {0x11a5c, 0x11a89}, {0x11a9d, 0x11a9d}, {0x11ab0, 0x11af8},
{0x11bc0, 0x11be0}, {0x11c00, 0x11c08}, {0x11c0a, 0x11c2e}, {0x11c40, 0x11c40},
{0x11c72, 0x11c8f}, {0x11d00, 0x11d06}, {0x11d08, 0x11d09}, {0x11d0b, 0x11d30},
{0x11d46, 0x11d46}, {0x11d60, 0x11d65}, {0x11d67, 0x11d68}, {0x11d6a, 0x11d89},
{0x11d98, 0x11d98}, {0x11db0, 0x11ddb}, {0x11ee0, 0x11ef2}, {0x11f02, 0x11f02},
{0x11f04, 0x11f10}, {0x11f12, 0x11f33}, {0x11fb0, 0x11fb0}, {0x12000, 0x12399},
{0x12480, 0x12543}, {0x12f90, 0x12ff0}, {0x13000, 0x1342f}, {0x13441, 0x13446},
{0x13460, 0x143fa}, {0x14400, 0x14646}, {0x16100, 0x1611d}, {0x16800, 0x16a38},
{0x16a40, 0x16a5e}, {0x16a70, 0x16abe}, {0x16ad0, 0x16aed}, {0x16b00, 0x16b2f},
{0x16b40, 0x16b43}, {0x16b63, 0x16b77}, {0x16b7d, 0x16b8f}, {0x16d40, 0x16d6c},
{0x16e40, 0x16e7f}, {0x16ea0, 0x16eb8}, {0x16ebb, 0x16ed3}, {0x16f00, 0x16f4a},
{0x16f50, 0x16f50}, {0x16f93, 0x16f9f}, {0x16fe0, 0x16fe1}, {0x16fe3, 0x16fe3},
{0x16ff2, 0x16ff3}, {0x17000, 0x18cd5}, {0x18cff, 0x18d1e}, {0x18d80, 0x18df2},
{0x1aff0, 0x1aff3}, {0x1aff5, 0x1affb}, {0x1affd, 0x1affe}, {0x1b000, 0x1b122},
{0x1b132, 0x1b132}, {0x1b150, 0x1b152}, {0x1b155, 0x1b155}, {0x1b164, 0x1b167},
{0x1b170, 0x1b2fb}, {0x1bc00, 0x1bc6a}, {0x1bc70, 0x1bc7c}, {0x1bc80, 0x1bc88},
{0x1bc90, 0x1bc99}, {0x1d400, 0x1d454}, {0x1d456, 0x1d49c}, {0x1d49e, 0x1d49f},
{0x1d4a2, 0x1d4a2}, {0x1d4a5, 0x1d4a6}, {0x1d4a9, 0x1d4ac}, {0x1d4ae, 0x1d4b9},
{0x1d4bb, 0x1d4bb}, {0x1d4bd, 0x1d4c3}, {0x1d4c5, 0x1d505}, {0x1d507, 0x1d50a},
{0x1d50d, 0x1d514}, {0x1d516, 0x1d51c}, {0x1d51e, 0x1d539}, {0x1d53b, 0x1d53e},
{0x1d540, 0x1d544}, {0x1d546, 0x1d546}, {0x1d54a, 0x1d550}, {0x1d552, 0x1d6a5},
{0x1d6a8, 0x1d6c0}, {0x1d6c2, 0x1d6da}, {0x1d6dc, 0x1d6fa}, {0x1d6fc, 0x1d714},
{0x1d716, 0x1d734}, {0x1d736, 0x1d74e}, {0x1d750, 0x1d76e}, {0x1d770, 0x1d788},
{0x1d78a, 0x1d7a8}, {0x1d7aa, 0x1d7c2}, {0x1d7c4, 0x1d7cb}, {0x1df00, 0x1df1e},
{0x1df25, 0x1df2a}, {0x1e030, 0x1e06d}, {0x1e100, 0x1e12c}, {0x1e137, 0x1e13d},
{0x1e14e, 0x1e14e}, {0x1e290, 0x1e2ad}, {0x1e2c0, 0x1e2eb}, {0x1e4d0, 0x1e4eb},
{0x1e5d0, 0x1e5ed}, {0x1e5f0, 0x1e5f0}, {0x1e6c0, 0x1e6de}, {0x1e6e0, 0x1e6e2},
{0x1e6e4, 0x1e6e5}, {0x1e6e7, 0x1e6ed}, {0x1e6f0, 0x1e6f4}, {0x1e6fe, 0x1e6ff},
{0x1e7e0, 0x1e7e6}, {0x1e7e8, 0x1e7eb}, {0x1e7ed, 0x1e7ee}, {0x1e7f0, 0x1e7fe},
{0x1e800, 0x1e8c4}, {0x1e900, 0x1e943}, {0x1e94b, 0x1e94b}, {0x1ee00, 0x1ee03},
{0x1ee05, 0x1ee1f}, {0x1ee21, 0x1ee22}, {0x1ee24, 0x1ee24}, {0x1ee27, 0x1ee27},
{0x1ee29, 0x1ee32}, {0x1ee34, 0x1ee37}, {0x1ee39, 0x1ee39}, {0x1ee3b, 0x1ee3b},
{0x1ee42, 0x1ee42}, {0x1ee47, 0x1ee47}, {0x1ee49, 0x1ee49}, {0x1ee4b, 0x1ee4b},
{0x1ee4d, 0x1ee4f}, {0x1ee51, 0x1ee52}, {0x1ee54, 0x1ee54}, {0x1ee57, 0x1ee57},
{0x1ee59, 0x1ee59}, {0x1ee5b, 0x1ee5b}, {0x1ee5d, 0x1ee5d}, {0x1ee5f, 0x1ee5f},
{0x1ee61, 0x1ee62}, {0x1ee64, 0x1ee64}, {0x1ee67, 0x1ee6a}, {0x1ee6c, 0x1ee72},
{0x1ee74, 0x1ee77}, {0x1ee79, 0x1ee7c}, {0x1ee7e, 0x1ee7e}, {0x1ee80, 0x1ee89},
{0x1ee8b, 0x1ee9b}, {0x1eea1, 0x1eea3}, {0x1eea5, 0x1eea9}, {0x1eeab, 0x1eebb},
{0x20000, 0x2a6df}, {0x2a700, 0x2b81d}, {0x2b820, 0x2cead}, {0x2ceb0, 0x2ebe0},
{0x2ebf0, 0x2ee5d}, {0x2f800, 0x2fa1d}, {0x30000, 0x3134a}, {0x31350, 0x33479},
};
static const ds4_qwen4_urange ds4_qwen4_mark_ranges[] = {
{0x00300, 0x0036f}, {0x00483, 0x00489}, {0x00591, 0x005bd}, {0x005bf, 0x005bf},
{0x005c1, 0x005c2}, {0x005c4, 0x005c5}, {0x005c7, 0x005c7}, {0x00610, 0x0061a},
{0x0064b, 0x0065f}, {0x00670, 0x00670}, {0x006d6, 0x006dc}, {0x006df, 0x006e4},
{0x006e7, 0x006e8}, {0x006ea, 0x006ed}, {0x00711, 0x00711}, {0x00730, 0x0074a},
{0x007a6, 0x007b0}, {0x007eb, 0x007f3}, {0x007fd, 0x007fd}, {0x00816, 0x00819},
{0x0081b, 0x00823}, {0x00825, 0x00827}, {0x00829, 0x0082d}, {0x00859, 0x0085b},
{0x00897, 0x0089f}, {0x008ca, 0x008e1}, {0x008e3, 0x00903}, {0x0093a, 0x0093c},
{0x0093e, 0x0094f}, {0x00951, 0x00957}, {0x00962, 0x00963}, {0x00981, 0x00983},
{0x009bc, 0x009bc}, {0x009be, 0x009c4}, {0x009c7, 0x009c8}, {0x009cb, 0x009cd},
{0x009d7, 0x009d7}, {0x009e2, 0x009e3}, {0x009fe, 0x009fe}, {0x00a01, 0x00a03},
{0x00a3c, 0x00a3c}, {0x00a3e, 0x00a42}, {0x00a47, 0x00a48}, {0x00a4b, 0x00a4d},
{0x00a51, 0x00a51}, {0x00a70, 0x00a71}, {0x00a75, 0x00a75}, {0x00a81, 0x00a83},
{0x00abc, 0x00abc}, {0x00abe, 0x00ac5}, {0x00ac7, 0x00ac9}, {0x00acb, 0x00acd},
{0x00ae2, 0x00ae3}, {0x00afa, 0x00aff}, {0x00b01, 0x00b03}, {0x00b3c, 0x00b3c},
{0x00b3e, 0x00b44}, {0x00b47, 0x00b48}, {0x00b4b, 0x00b4d}, {0x00b55, 0x00b57},
{0x00b62, 0x00b63}, {0x00b82, 0x00b82}, {0x00bbe, 0x00bc2}, {0x00bc6, 0x00bc8},
{0x00bca, 0x00bcd}, {0x00bd7, 0x00bd7}, {0x00c00, 0x00c04}, {0x00c3c, 0x00c3c},
{0x00c3e, 0x00c44}, {0x00c46, 0x00c48}, {0x00c4a, 0x00c4d}, {0x00c55, 0x00c56},
{0x00c62, 0x00c63}, {0x00c81, 0x00c83}, {0x00cbc, 0x00cbc}, {0x00cbe, 0x00cc4},
{0x00cc6, 0x00cc8}, {0x00cca, 0x00ccd}, {0x00cd5, 0x00cd6}, {0x00ce2, 0x00ce3},
{0x00cf3, 0x00cf3}, {0x00d00, 0x00d03}, {0x00d3b, 0x00d3c}, {0x00d3e, 0x00d44},
{0x00d46, 0x00d48}, {0x00d4a, 0x00d4d}, {0x00d57, 0x00d57}, {0x00d62, 0x00d63},
{0x00d81, 0x00d83}, {0x00dca, 0x00dca}, {0x00dcf, 0x00dd4}, {0x00dd6, 0x00dd6},
{0x00dd8, 0x00ddf}, {0x00df2, 0x00df3}, {0x00e31, 0x00e31}, {0x00e34, 0x00e3a},
{0x00e47, 0x00e4e}, {0x00eb1, 0x00eb1}, {0x00eb4, 0x00ebc}, {0x00ec8, 0x00ece},
{0x00f18, 0x00f19}, {0x00f35, 0x00f35}, {0x00f37, 0x00f37}, {0x00f39, 0x00f39},
{0x00f3e, 0x00f3f}, {0x00f71, 0x00f84}, {0x00f86, 0x00f87}, {0x00f8d, 0x00f97},
{0x00f99, 0x00fbc}, {0x00fc6, 0x00fc6}, {0x0102b, 0x0103e}, {0x01056, 0x01059},
{0x0105e, 0x01060}, {0x01062, 0x01064}, {0x01067, 0x0106d}, {0x01071, 0x01074},
{0x01082, 0x0108d}, {0x0108f, 0x0108f}, {0x0109a, 0x0109d}, {0x0135d, 0x0135f},
{0x01712, 0x01715}, {0x01732, 0x01734}, {0x01752, 0x01753}, {0x01772, 0x01773},
{0x017b4, 0x017d3}, {0x017dd, 0x017dd}, {0x0180b, 0x0180d}, {0x0180f, 0x0180f},
{0x01885, 0x01886}, {0x018a9, 0x018a9}, {0x01920, 0x0192b}, {0x01930, 0x0193b},
{0x01a17, 0x01a1b}, {0x01a55, 0x01a5e}, {0x01a60, 0x01a7c}, {0x01a7f, 0x01a7f},
{0x01ab0, 0x01add}, {0x01ae0, 0x01aeb}, {0x01b00, 0x01b04}, {0x01b34, 0x01b44},
{0x01b6b, 0x01b73}, {0x01b80, 0x01b82}, {0x01ba1, 0x01bad}, {0x01be6, 0x01bf3},
{0x01c24, 0x01c37}, {0x01cd0, 0x01cd2}, {0x01cd4, 0x01ce8}, {0x01ced, 0x01ced},
{0x01cf4, 0x01cf4}, {0x01cf7, 0x01cf9}, {0x01dc0, 0x01dff}, {0x020d0, 0x020f0},
{0x02cef, 0x02cf1}, {0x02d7f, 0x02d7f}, {0x02de0, 0x02dff}, {0x0302a, 0x0302f},
{0x03099, 0x0309a}, {0x0a66f, 0x0a672}, {0x0a674, 0x0a67d}, {0x0a69e, 0x0a69f},
{0x0a6f0, 0x0a6f1}, {0x0a802, 0x0a802}, {0x0a806, 0x0a806}, {0x0a80b, 0x0a80b},
{0x0a823, 0x0a827}, {0x0a82c, 0x0a82c}, {0x0a880, 0x0a881}, {0x0a8b4, 0x0a8c5},
{0x0a8e0, 0x0a8f1}, {0x0a8ff, 0x0a8ff}, {0x0a926, 0x0a92d}, {0x0a947, 0x0a953},
{0x0a980, 0x0a983}, {0x0a9b3, 0x0a9c0}, {0x0a9e5, 0x0a9e5}, {0x0aa29, 0x0aa36},
{0x0aa43, 0x0aa43}, {0x0aa4c, 0x0aa4d}, {0x0aa7b, 0x0aa7d}, {0x0aab0, 0x0aab0},
{0x0aab2, 0x0aab4}, {0x0aab7, 0x0aab8}, {0x0aabe, 0x0aabf}, {0x0aac1, 0x0aac1},
{0x0aaeb, 0x0aaef}, {0x0aaf5, 0x0aaf6}, {0x0abe3, 0x0abea}, {0x0abec, 0x0abed},
{0x0fb1e, 0x0fb1e}, {0x0fe00, 0x0fe0f}, {0x0fe20, 0x0fe2f}, {0x101fd, 0x101fd},
{0x102e0, 0x102e0}, {0x10376, 0x1037a}, {0x10a01, 0x10a03}, {0x10a05, 0x10a06},
{0x10a0c, 0x10a0f}, {0x10a38, 0x10a3a}, {0x10a3f, 0x10a3f}, {0x10ae5, 0x10ae6},
{0x10d24, 0x10d27}, {0x10d69, 0x10d6d}, {0x10eab, 0x10eac}, {0x10efa, 0x10eff},
{0x10f46, 0x10f50}, {0x10f82, 0x10f85}, {0x11000, 0x11002}, {0x11038, 0x11046},
{0x11070, 0x11070}, {0x11073, 0x11074}, {0x1107f, 0x11082}, {0x110b0, 0x110ba},
{0x110c2, 0x110c2}, {0x11100, 0x11102}, {0x11127, 0x11134}, {0x11145, 0x11146},
{0x11173, 0x11173}, {0x11180, 0x11182}, {0x111b3, 0x111c0}, {0x111c9, 0x111cc},
{0x111ce, 0x111cf}, {0x1122c, 0x11237}, {0x1123e, 0x1123e}, {0x11241, 0x11241},
{0x112df, 0x112ea}, {0x11300, 0x11303}, {0x1133b, 0x1133c}, {0x1133e, 0x11344},
{0x11347, 0x11348}, {0x1134b, 0x1134d}, {0x11357, 0x11357}, {0x11362, 0x11363},
{0x11366, 0x1136c}, {0x11370, 0x11374}, {0x113b8, 0x113c0}, {0x113c2, 0x113c2},
{0x113c5, 0x113c5}, {0x113c7, 0x113ca}, {0x113cc, 0x113d0}, {0x113d2, 0x113d2},
{0x113e1, 0x113e2}, {0x11435, 0x11446}, {0x1145e, 0x1145e}, {0x114b0, 0x114c3},
{0x115af, 0x115b5}, {0x115b8, 0x115c0}, {0x115dc, 0x115dd}, {0x11630, 0x11640},
{0x116ab, 0x116b7}, {0x1171d, 0x1172b}, {0x1182c, 0x1183a}, {0x11930, 0x11935},
{0x11937, 0x11938}, {0x1193b, 0x1193e}, {0x11940, 0x11940}, {0x11942, 0x11943},
{0x119d1, 0x119d7}, {0x119da, 0x119e0}, {0x119e4, 0x119e4}, {0x11a01, 0x11a0a},
{0x11a33, 0x11a39}, {0x11a3b, 0x11a3e}, {0x11a47, 0x11a47}, {0x11a51, 0x11a5b},
{0x11a8a, 0x11a99}, {0x11b60, 0x11b67}, {0x11c2f, 0x11c36}, {0x11c38, 0x11c3f},
{0x11c92, 0x11ca7}, {0x11ca9, 0x11cb6}, {0x11d31, 0x11d36}, {0x11d3a, 0x11d3a},
{0x11d3c, 0x11d3d}, {0x11d3f, 0x11d45}, {0x11d47, 0x11d47}, {0x11d8a, 0x11d8e},
{0x11d90, 0x11d91}, {0x11d93, 0x11d97}, {0x11ef3, 0x11ef6}, {0x11f00, 0x11f01},
{0x11f03, 0x11f03}, {0x11f34, 0x11f3a}, {0x11f3e, 0x11f42}, {0x11f5a, 0x11f5a},
{0x13440, 0x13440}, {0x13447, 0x13455}, {0x1611e, 0x1612f}, {0x16af0, 0x16af4},
{0x16b30, 0x16b36}, {0x16f4f, 0x16f4f}, {0x16f51, 0x16f87}, {0x16f8f, 0x16f92},
{0x16fe4, 0x16fe4}, {0x16ff0, 0x16ff1}, {0x1bc9d, 0x1bc9e}, {0x1cf00, 0x1cf2d},
{0x1cf30, 0x1cf46}, {0x1d165, 0x1d169}, {0x1d16d, 0x1d172}, {0x1d17b, 0x1d182},
{0x1d185, 0x1d18b}, {0x1d1aa, 0x1d1ad}, {0x1d242, 0x1d244}, {0x1da00, 0x1da36},
{0x1da3b, 0x1da6c}, {0x1da75, 0x1da75}, {0x1da84, 0x1da84}, {0x1da9b, 0x1da9f},
{0x1daa1, 0x1daaf}, {0x1e000, 0x1e006}, {0x1e008, 0x1e018}, {0x1e01b, 0x1e021},
{0x1e023, 0x1e024}, {0x1e026, 0x1e02a}, {0x1e08f, 0x1e08f}, {0x1e130, 0x1e136},
{0x1e2ae, 0x1e2ae}, {0x1e2ec, 0x1e2ef}, {0x1e4ec, 0x1e4ef}, {0x1e5ee, 0x1e5ef},
{0x1e6e3, 0x1e6e3}, {0x1e6e6, 0x1e6e6}, {0x1e6ee, 0x1e6ef}, {0x1e6f5, 0x1e6f5},
{0x1e8d0, 0x1e8d6}, {0x1e944, 0x1e94a}, {0xe0100, 0xe01ef},
};
static const ds4_qwen4_urange ds4_qwen4_number_ranges[] = {
{0x00030, 0x00039}, {0x000b2, 0x000b3}, {0x000b9, 0x000b9}, {0x000bc, 0x000be},
{0x00660, 0x00669}, {0x006f0, 0x006f9}, {0x007c0, 0x007c9}, {0x00966, 0x0096f},
{0x009e6, 0x009ef}, {0x009f4, 0x009f9}, {0x00a66, 0x00a6f}, {0x00ae6, 0x00aef},
{0x00b66, 0x00b6f}, {0x00b72, 0x00b77}, {0x00be6, 0x00bf2}, {0x00c66, 0x00c6f},
{0x00c78, 0x00c7e}, {0x00ce6, 0x00cef}, {0x00d58, 0x00d5e}, {0x00d66, 0x00d78},
{0x00de6, 0x00def}, {0x00e50, 0x00e59}, {0x00ed0, 0x00ed9}, {0x00f20, 0x00f33},
{0x01040, 0x01049}, {0x01090, 0x01099}, {0x01369, 0x0137c}, {0x016ee, 0x016f0},
{0x017e0, 0x017e9}, {0x017f0, 0x017f9}, {0x01810, 0x01819}, {0x01946, 0x0194f},
{0x019d0, 0x019da}, {0x01a80, 0x01a89}, {0x01a90, 0x01a99}, {0x01b50, 0x01b59},
{0x01bb0, 0x01bb9}, {0x01c40, 0x01c49}, {0x01c50, 0x01c59}, {0x02070, 0x02070},
{0x02074, 0x02079}, {0x02080, 0x02089}, {0x02150, 0x02182}, {0x02185, 0x02189},
{0x02460, 0x0249b}, {0x024ea, 0x024ff}, {0x02776, 0x02793}, {0x02cfd, 0x02cfd},
{0x03007, 0x03007}, {0x03021, 0x03029}, {0x03038, 0x0303a}, {0x03192, 0x03195},
{0x03220, 0x03229}, {0x03248, 0x0324f}, {0x03251, 0x0325f}, {0x03280, 0x03289},
{0x032b1, 0x032bf}, {0x0a620, 0x0a629}, {0x0a6e6, 0x0a6ef}, {0x0a830, 0x0a835},
{0x0a8d0, 0x0a8d9}, {0x0a900, 0x0a909}, {0x0a9d0, 0x0a9d9}, {0x0a9f0, 0x0a9f9},
{0x0aa50, 0x0aa59}, {0x0abf0, 0x0abf9}, {0x0ff10, 0x0ff19}, {0x10107, 0x10133},
{0x10140, 0x10178}, {0x1018a, 0x1018b}, {0x102e1, 0x102fb}, {0x10320, 0x10323},
{0x10341, 0x10341}, {0x1034a, 0x1034a}, {0x103d1, 0x103d5}, {0x104a0, 0x104a9},
{0x10858, 0x1085f}, {0x10879, 0x1087f}, {0x108a7, 0x108af}, {0x108fb, 0x108ff},
{0x10916, 0x1091b}, {0x109bc, 0x109bd}, {0x109c0, 0x109cf}, {0x109d2, 0x109ff},
{0x10a40, 0x10a48}, {0x10a7d, 0x10a7e}, {0x10a9d, 0x10a9f}, {0x10aeb, 0x10aef},
{0x10b58, 0x10b5f}, {0x10b78, 0x10b7f}, {0x10ba9, 0x10baf}, {0x10cfa, 0x10cff},
{0x10d30, 0x10d39}, {0x10d40, 0x10d49}, {0x10e60, 0x10e7e}, {0x10f1d, 0x10f26},
{0x10f51, 0x10f54}, {0x10fc5, 0x10fcb}, {0x11052, 0x1106f}, {0x110f0, 0x110f9},
{0x11136, 0x1113f}, {0x111d0, 0x111d9}, {0x111e1, 0x111f4}, {0x112f0, 0x112f9},
{0x11450, 0x11459}, {0x114d0, 0x114d9}, {0x11650, 0x11659}, {0x116c0, 0x116c9},
{0x116d0, 0x116e3}, {0x11730, 0x1173b}, {0x118e0, 0x118f2}, {0x11950, 0x11959},
{0x11bf0, 0x11bf9}, {0x11c50, 0x11c6c}, {0x11d50, 0x11d59}, {0x11da0, 0x11da9},
{0x11de0, 0x11de9}, {0x11f50, 0x11f59}, {0x11fc0, 0x11fd4}, {0x12400, 0x1246e},
{0x16130, 0x16139}, {0x16a60, 0x16a69}, {0x16ac0, 0x16ac9}, {0x16b50, 0x16b59},
{0x16b5b, 0x16b61}, {0x16d70, 0x16d79}, {0x16e80, 0x16e96}, {0x16ff4, 0x16ff6},
{0x1ccf0, 0x1ccf9}, {0x1d2c0, 0x1d2d3}, {0x1d2e0, 0x1d2f3}, {0x1d360, 0x1d378},
{0x1d7ce, 0x1d7ff}, {0x1e140, 0x1e149}, {0x1e2f0, 0x1e2f9}, {0x1e4f0, 0x1e4f9},
{0x1e5f1, 0x1e5fa}, {0x1e8c7, 0x1e8cf}, {0x1e950, 0x1e959}, {0x1ec71, 0x1ecab},
{0x1ecad, 0x1ecaf}, {0x1ecb1, 0x1ecb4}, {0x1ed01, 0x1ed2d}, {0x1ed2f, 0x1ed3d},
{0x1f100, 0x1f10c}, {0x1fbf0, 0x1fbf9},
};
static const ds4_qwen4_urange ds4_qwen4_space_ranges[] = {
{0x00009, 0x0000d}, {0x00020, 0x00020}, {0x00085, 0x00085}, {0x000a0, 0x000a0},
{0x01680, 0x01680}, {0x02000, 0x0200a}, {0x02028, 0x02029}, {0x0202f, 0x0202f},
{0x0205f, 0x0205f}, {0x03000, 0x03000},
};
ds4_qwen4_vision.h
+19
View File
@@ -0,0 +1,19 @@
#ifndef DS4_QWEN4_VISION_H
#define DS4_QWEN4_VISION_H
#include <stdint.h>
#define DS4_QWEN4_VISION_LAYERS 27
typedef struct {
uint64_t ln1_w, ln1_b, qkv_w, qkv_b, out_w, out_b, ln2_w, ln2_b, up_w, up_b, down_w, down_b;
uint32_t qkv_type, out_type, up_type, down_type;
} ds4_qwen4_vision_layer_weights;
typedef struct {
uint64_t patch_w0, patch_w1, patch_b, pos_embd, post_ln_w, post_ln_b, mm0_w, mm0_b, mm2_w, mm2_b;
uint32_t mm0_type, mm2_type, patch_type;
uint32_t n_embd, n_ff, n_head, n_patch, n_merge, n_pos_side, n_out;
float eps;
ds4_qwen4_vision_layer_weights layer[DS4_QWEN4_VISION_LAYERS];
} ds4_qwen4_vision_weights;
#endif
ds4_rocm.cu
+28
View File
@@ -1,6 +1,7 @@
#ifdef __HIP_PLATFORM_AMD__
#include "ds4_rocm.h"
#include <hipblaslt/hipblaslt.h>
#include <rocblas/rocblas.h>
#define FULL_WARP_MASK 0xFFFFFFFFFFFFFFFFULL
#define MASK_T uint64_t
@@ -39,7 +40,27 @@
#include <vector>
#include "ds4_gpu.h"
#include "ds4_image.h"
static thread_local bool g_dspark_verify_mode;
extern "C" void ds4_gpu_set_dspark_verify_mode(bool enabled) {
g_dspark_verify_mode = enabled;
}
extern "C" int ds4_mmq_init(int device);
extern "C" int ds4_mmq_iq2_xxs_moe_pair(
const void *W_a, const void *W_b, const float *X_f32,
const int32_t *ids, float *out_a, float *out_b,
int M, int K, int n_tokens, int n_experts, int n_expert_used,
cudaStream_t stream);
extern "C" int ds4_mmq_q8_0_dense_vec(
const void *W, const float *X_f32, float *out_f32,
int M, int N, int K, cudaStream_t stream);
extern "C" int ds4_mmq_q2_K_moe_down_sum6_vec(
const void *W, const float *X, const int32_t *ids, float *out,
int M, int K, int n_tokens, int n_experts, int n_expert_used,
cudaStream_t stream);
#ifndef M_PI
#define M_PI 3.14159265358979323846
#endif
@@ -107,6 +128,10 @@ __device__ __constant__ static const int8_t cuda_mxfp4_values_x2[16] = {
#include "rocm/ds4_rocm_common.cuh"
extern "C" int ds4_gpu_dspark_gfx1151_fast_path(void) {
return ds4_rocm_is_gfx1151();
}
#include "rocm/ds4_rocm_q8.cuh"
#include "rocm/ds4_rocm_norm_rope.cuh"
@@ -147,6 +172,9 @@ __device__ __constant__ static const int8_t cuda_mxfp4_values_x2[16] = {
#include "rocm/ds4_rocm_current_api_compat.cuh"
#include "ds4_glm53_vision_gpu.cuh"
#include "ds4_deepseek4_vision_gpu.cuh"
#include "rocm/ds4_rocm_deepseek4_vision.cuh"
#include "ds4_deepseek41_cuda.cuh"
/* Tensor-parallel gates are Metal-only; stubs keep shared graph code
* linkable (TP option validation rejects non-Metal backends). */
ds4_rocm.h
+11 -5
View File
@@ -6,6 +6,7 @@
#include <hipcub/hipcub.hpp>
#include <rocwmma/rocwmma-version.hpp>
#include <rocwmma/rocwmma.hpp>
#include "ds4_rocm_memory.h"
#define cudaError_t hipError_t
#define cudaStream_t hipStream_t
@@ -30,9 +31,9 @@
#define cudaFuncSetAttribute(func, attr, value) hipFuncSetAttribute((const void *)(func), (attr), (value))
#define cudaMemLocationTypeDevice hipMemLocationTypeDevice
#define cudaMalloc hipMalloc
#define cudaMallocHost hipHostMalloc
#define cudaMallocManaged hipMallocManaged
#define cudaMalloc ds4_rocm_malloc
#define cudaMallocHost ds4_rocm_malloc_host
#define cudaMallocManaged ds4_rocm_malloc_managed
#define cudaFree hipFree
#define cudaFreeHost hipFreeHost
#define cudaMemset hipMemset
@@ -41,7 +42,7 @@
#define cudaMemcpyHostToDevice hipMemcpyHostToDevice
#define cudaMemcpyDeviceToHost hipMemcpyDeviceToHost
#define cudaMemcpyDeviceToDevice hipMemcpyDeviceToDevice
#define cudaMemGetInfo hipMemGetInfo
#define cudaMemGetInfo ds4_rocm_mem_get_info
#define cudaMemsetAsync hipMemsetAsync
#define cudaHostRegister hipHostRegister
@@ -138,5 +139,10 @@ extern "C" __device__ __attribute__((pure)) float __ocml_log1p_f32(float);
extern "C" __device__ __attribute__((const)) float __ocml_sqrt_f32(float);
static __device__ __forceinline__ float ds4_precise_expf(float x) { return __ocml_exp_f32(x); }
static __device__ __forceinline__ float ds4_precise_log1pf(float x) { return __ocml_log1p_f32(x); }
static __device__ __forceinline__ float ds4_precise_log1pf(float x) {
/* OCML's unsafe-math mode can reduce log1p(x) to log(1+x), losing
* small router probabilities. The cubic's relative error here is < 2^-31. */
if (fabsf(x) < 0x1p-10f) return x * (1.0f - x * (0.5f - x / 3.0f));
return __ocml_log1p_f32(x);
}
static __device__ __forceinline__ float ds4_precise_sqrtf(float x) { return __ocml_sqrt_f32(x); }
ds4_rocm_compat.cu
+6 -13
View File
@@ -1,4 +1,5 @@
#include <hip/hip_runtime.h>
#include "ds4_rocm_memory.h"
#include <stdio.h>
#include <stdlib.h>
@@ -57,7 +58,7 @@ extern "C" int ds4_gpu_tensor_alloc_on(ds4_gpu_tensor *t, int tier,
if (!t) return 1;
if (!rocm_tier_valid(tier)) return 2;
if (bytes == 0) bytes = 1;
if (hipMalloc(&t->ptr, (size_t)bytes) != hipSuccess) return 3;
if (ds4_rocm_malloc(&t->ptr, (size_t)bytes) != hipSuccess) return 3;
t->bytes = bytes;
t->owner = 1;
t->device_id = 0;
@@ -224,23 +225,15 @@ extern "C" int ds4_gpu_set_decode_score_vec4(int enabled) {
return 0;
}
extern "C" int ds4_gpu_matmul_q8_0_decode_rows_exact_tensor(
ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
uint64_t weight_offset, uint64_t in_dim, uint64_t out_dim,
const ds4_gpu_tensor *x, uint32_t n_rows) {
return ds4_gpu_matmul_q8_0_tensor(out, model_map, model_size,
weight_offset, in_dim, out_dim, x,
n_rows);
}
extern "C" int ds4_gpu_matmul_q8_0_pair_decode_rows_exact_tensor(
ds4_gpu_tensor *out0, ds4_gpu_tensor *out1, const void *model_map,
uint64_t model_size, uint64_t weight0_offset,
uint64_t weight1_offset, uint64_t in_dim, uint64_t out0_dim,
uint64_t out1_dim, const ds4_gpu_tensor *x, uint32_t n_rows) {
return ds4_gpu_matmul_q8_0_pair_tensor(
out0, out1, model_map, model_size, weight0_offset, weight1_offset,
in_dim, out0_dim, out1_dim, x, n_rows);
return ds4_gpu_matmul_q8_0_decode_rows_exact_tensor(out0, model_map,
model_size, weight0_offset, in_dim, out0_dim, x, n_rows) &&
ds4_gpu_matmul_q8_0_decode_rows_exact_tensor(out1, model_map,
model_size, weight1_offset, in_dim, out1_dim, x, n_rows);
}
extern "C" int ds4_gpu_matmul_f16_router_rows_exact_tensor(
ds4_rocm_memory.h
+62
View File
@@ -0,0 +1,62 @@
#ifndef DS4_ROCM_MEMORY_H
#define DS4_ROCM_MEMORY_H
#include <hip/hip_runtime.h>
#include "ds4_linux_memory.h"
static inline uint64_t ds4_rocm_model_arena_bytes(uint64_t need) {
/* Large tensor spans should not strand hundreds of MiB each. Small
* tensors still share an arena to avoid thousands of driver allocations. */
return need >= (256ull << 20) ? need : (1792ull << 20);
}
static inline bool ds4_rocm_uses_host_ram(void) {
int device;
hipDeviceProp_t prop;
return hipGetDevice(&device) == hipSuccess &&
hipGetDeviceProperties(&prop, device) == hipSuccess && prop.integrated;
}
static inline bool ds4_rocm_allocation_fits(size_t bytes, bool host = false) {
if (!host && !ds4_rocm_uses_host_ram()) return true;
uint64_t available;
const uint64_t reserve = 2ull << 30;
if (!ds4_linux_nonmovable_memory(&available)) {
fprintf(stderr, "ds4: ROCm cannot read usable host memory; allocation refused\n");
return false;
}
if (available >= reserve && bytes <= available - reserve) return true;
fprintf(stderr, "ds4: ROCm allocation refused: %.2f MiB requested, "
"%.2f GiB usable after excluding CMA, 2 GiB OS reserve\n",
(double)bytes / 1048576.0, (double)available / 1073741824.0);
return false;
}
template <typename T>
static inline hipError_t ds4_rocm_malloc(T **ptr, size_t bytes) {
if (!ds4_rocm_allocation_fits(bytes)) return hipErrorOutOfMemory;
return hipMalloc((void **)ptr, bytes);
}
template <typename T>
static inline hipError_t ds4_rocm_malloc_host(T **ptr, size_t bytes) {
if (!ds4_rocm_allocation_fits(bytes, true)) return hipErrorOutOfMemory;
return hipHostMalloc((void **)ptr, bytes);
}
template <typename T>
static inline hipError_t ds4_rocm_malloc_managed(T **ptr, size_t bytes) {
if (!ds4_rocm_allocation_fits(bytes, true)) return hipErrorOutOfMemory;
return hipMallocManaged((void **)ptr, bytes);
}
static inline hipError_t ds4_rocm_mem_get_info(size_t *free_b, size_t *total_b) {
hipError_t err = hipMemGetInfo(free_b, total_b);
if (err != hipSuccess || !ds4_rocm_uses_host_ram()) return err;
uint64_t available;
if (!ds4_linux_nonmovable_memory(&available)) return hipErrorUnknown;
if (available < *free_b) *free_b = (size_t)available;
return hipSuccess;
}
#endif
ds4_rocm_unavailable.cu
+3
View File
@@ -37,3 +37,6 @@ ROCM_UNAVAILABLE_INT(ds4_gpu_shared_mid_swiglu_q8_0_decode_exact_tensor)
extern "C" uint64_t ds4_gpu_tp_big_gate_kick(...) { return 0; }
ROCM_UNAVAILABLE_INT(ds4_gpu_tp_big_gate_wait)
ROCM_UNAVAILABLE_INT(ds4_gpu_tp_init)
extern "C" void ds4_gpu_tp_shutdown(void) {}
extern "C" int ds4_gpu_tp_failed(void) { return 1; }
ds4_server.c
+4436 -549
View File
File diff suppressed because it is too large. Load diff
ds4_ssd.c
+9 -4
View File
@@ -77,7 +77,7 @@ uint32_t ds4_ssd_cache_experts_for_byte_budget(uint64_t bytes,
return (uint32_t)experts;
}
static uint64_t ds4_ssd_auto_cache_percent(void) {
static uint64_t ds4_ssd_auto_cache_percent(uint32_t default_percent) {
const char *env = getenv("DS4_SSD_AUTO_CACHE_PCT");
if (env && env[0]) {
errno = 0;
@@ -102,22 +102,27 @@ static uint64_t ds4_ssd_auto_cache_percent(void) {
* the machine out of RAM and trips the Linux OOM killer. 80% was measured
* safe here; opt into more only deliberately via DS4_SSD_AUTO_CACHE_PCT.
*/
return 80;
return default_percent;
}
bool ds4_ssd_auto_cache_plan(uint64_t recommended_bytes,
uint32_t default_percent,
uint64_t model_limit_bytes,
uint64_t non_routed_bytes,
uint64_t per_expert_bytes,
uint64_t max_model_experts,
ds4_ssd_cache_plan *out) {
if (!out) return false;
memset(out, 0, sizeof(*out));
if (recommended_bytes == 0 || per_expert_bytes == 0) return false;
if (recommended_bytes == 0 || per_expert_bytes == 0 ||
default_percent < 50 || default_percent > 95) return false;
const uint64_t pct = ds4_ssd_auto_cache_percent();
const uint64_t pct = ds4_ssd_auto_cache_percent(default_percent);
out->model_target_bytes =
recommended_bytes > UINT64_MAX / pct ?
UINT64_MAX : (recommended_bytes * pct) / 100ull;
if (model_limit_bytes != 0 && out->model_target_bytes > model_limit_bytes)
out->model_target_bytes = model_limit_bytes;
if (out->model_target_bytes > non_routed_bytes) {
out->cache_bytes = out->model_target_bytes - non_routed_bytes;
}
ds4_ssd.h
+2
View File
@@ -24,6 +24,8 @@ bool ds4_parse_streaming_cache_experts_arg(const char *s,
uint32_t ds4_ssd_cache_experts_for_byte_budget(uint64_t bytes,
uint64_t per_expert_bytes);
bool ds4_ssd_auto_cache_plan(uint64_t recommended_bytes,
uint32_t default_percent,
uint64_t model_limit_bytes,
uint64_t non_routed_bytes,
uint64_t per_expert_bytes,
uint64_t max_model_experts,
ds4_tool_text.h
+59
View File
@@ -0,0 +1,59 @@
#ifndef DS4_TOOL_TEXT_H
#define DS4_TOOL_TEXT_H
#include <stdbool.h>
#include <stddef.h>
#include <string.h>
/* Tool bodies are not HTML. Escape only their own closing delimiter and
* spellings that would otherwise collide with that escape. */
static inline bool ds4_tool_text_escaped_close(const char *s, const char *end) {
if (*s != '&') return false;
s++;
while (!strncmp(s, "amp;", 4)) s += 4;
return !strncmp(s, "lt;", 3) && !strncmp(s + 3, end + 1, strlen(end + 1));
}
static inline bool ds4_tool_text_needs_escape(const char *s, const char *end) {
return !strncmp(s, end, strlen(end)) || ds4_tool_text_escaped_close(s, end);
}
static inline void ds4_tool_text_unescape(char *s, const char *end) {
char *out = s;
while (*s) {
if (ds4_tool_text_escaped_close(s, end)) {
if (!strncmp(s, "&amp;", 5)) {
*out++ = '&';
s += 5;
} else {
*out++ = '<';
s += 4;
}
} else {
*out++ = *s++;
}
}
*out = '\0';
}
/* A streaming fragment must not split an escape that could become a closing
* delimiter. Ordinary entities pass through unchanged. */
static inline bool ds4_tool_text_partial_escape(const char *s, size_t n,
const char *end) {
if (!n || *s != '&') return false;
s++;
n--;
while (n >= 4 && !memcmp(s, "amp;", 4)) {
s += 4;
n -= 4;
}
if (n < 4 && !memcmp(s, "amp;", n)) return true;
if (n < 3) return !memcmp(s, "lt;", n);
if (memcmp(s, "lt;", 3)) return false;
s += 3;
n -= 3;
size_t tail = strlen(end + 1);
return n < tail && !memcmp(s, end + 1, n);
}
#endif
ds4_tp.c
+1761 -272
View File
File diff suppressed because it is too large. Load diff
ds4_tp.h
+113 -7
View File
@@ -30,8 +30,9 @@ enum {
DS4_TP_GATE_ATTN = 0,
DS4_TP_GATE_FFN = 1,
DS4_TP_GATES_PER_LAYER = 2,
/* Max rows in a verify-block batch gate (speculative blocks are <=5). */
/* Includes the seed row as well as drafted tokens. */
DS4_TP_BATCH_MAX_ROWS = 8,
DS4_TP_BATCH_GATES_PER_LAYER = 2,
};
/* Engine identity exchanged in the hello so a mismatched pair aborts before
@@ -103,7 +104,20 @@ void ds4_tp_free(ds4_tp *tp);
int ds4_tp_rank(const ds4_tp *tp);
bool ds4_tp_is_rdma(const ds4_tp *tp);
/* True once both peers agreed on the RDMA command mailbox (RC queue pairs,
* not disabled with DS4_TP_DISABLE_RDMA_MAILBOX on either rank). */
bool ds4_tp_command_mailbox_active(const ds4_tp *tp);
uint32_t ds4_tp_peer_ctx(const ds4_tp *tp);
enum {
DS4_TP_MODE_EXPERT_INTERMEDIATE_SPLIT = 1u,
DS4_TP_MODE_V41_SHARED_OWNER = 2u,
DS4_TP_MODE_V41_SHARED_BALANCE = 4u,
DS4_TP_MODE_V41_DSPARK = 8u,
};
/* Call on both engines before binding the GPU slab. Resident layouts and
* shared-expert ownership must match before either rank can reduce results.
* full_experts is a local capability on entry, the intersection on return. */
int ds4_tp_agree_execution_mode(ds4_tp *tp, uint32_t flags, bool *full_experts, char *err, size_t errlen);
bool ds4_tp_failed(const ds4_tp *tp);
void ds4_tp_mark_failed(ds4_tp *tp);
@@ -115,7 +129,8 @@ void ds4_tp_mark_failed(ds4_tp *tp);
* in vectors S * vec_bytes RDMA/TCP-written with the peer partials
* in seq flags S * 8 written strictly after each in vector
* token slot 16 {seq u64, token i32, pad} leader->worker
* (gpu flags, then batch out/in: n_layer * BATCH_MAX_ROWS * vec_bytes
* (gpu flags, then batch out/in:
* n_layer * BATCH_GATES_PER_LAYER * BATCH_MAX_ROWS * vec_bytes
* each, row partials for the speculative verify-block gates)
*
* vec_bytes = n_embd * 4 (f32 partials, never quantized on the wire). */
@@ -126,6 +141,9 @@ uint64_t ds4_tp_slab_batch_out_offset(const ds4_tp *tp, uint32_t layer);
uint64_t ds4_tp_slab_batch_in_offset(const ds4_tp *tp, uint32_t layer);
uint64_t ds4_tp_slab_gpu_flags_offset(const ds4_tp *tp);
int ds4_tp_attach_slab(ds4_tp *tp, void *base, char *err, size_t errlen);
/* Stop the data plane before freeing its registered buffers. No more gates
* may be issued; free the transport after the engine has been unbound. */
void ds4_tp_detach_slab(ds4_tp *tp);
/* Exchange one gate: send out[layer][gate] to the peer's in[layer][gate]
* and wait until the peer's partial for `seq` has fully landed locally.
@@ -137,6 +155,15 @@ int ds4_tp_gate_exchange(ds4_tp *tp, uint32_t layer, uint32_t gate, uint64_t seq
* the GPU gate service thread. */
int ds4_tp_batch_gate_exchange(ds4_tp *tp, uint32_t layer, uint32_t rows,
uint64_t seq);
/* Verify-block RDMA window (speculative decoding): call on both ranks right
* before/after a verify block with one batch gate per layer. */
int ds4_tp_batch_block_begin(ds4_tp *tp, uint32_t rows, uint32_t n_layers);
/* Negotiate one or two batch gates per layer before encoding the block.
* Both ranks receive the smaller offered count. Slots are then consecutive
* gate indices, not layer indices. Works with RDMA and TCP. */
int ds4_tp_batch_block_begin_gates(ds4_tp *tp, uint32_t rows, uint32_t n_layers,
uint32_t *gates_per_layer);
int ds4_tp_batch_block_end(ds4_tp *tp);
/* Prefill batch gate: arbitrary-size symmetric payload exchange over bulk
* RDMA, with interleaved 2MB TCP rounds as fallback (see ds4_tp.c). */
@@ -160,7 +187,13 @@ int ds4_tp_send_sync_multimodal(ds4_tp *tp, uint64_t session_id,
uint32_t image_count);
int ds4_tp_send_eval(ds4_tp *tp, uint64_t session_id,
uint64_t seq, int token);
int ds4_tp_send_glm_mtp(ds4_tp *tp, uint64_t session_id,
uint64_t seq, int token, int limit);
int ds4_tp_send_rewind(ds4_tp *tp, uint64_t session_id, int pos);
/* State-only restore: ACK status 0 restored, 1 unavailable, -1 failed (closes).
* No TP collectives or logits reply. Await ds4_tp_wait_command_status() before
* mirroring replay EVALs; status 1 requires invalidation and full sync. */
int ds4_tp_send_spec_restore(ds4_tp *tp, uint64_t session_id, int pos);
int ds4_tp_send_invalidate(ds4_tp *tp, uint64_t session_id);
int ds4_tp_send_eval_batch(ds4_tp *tp, const ds4_tp_batch_item *items,
uint32_t count);
@@ -171,6 +204,12 @@ int ds4_tp_send_mixed_batch(ds4_tp *tp, uint64_t prefill_session_id,
int ds4_tp_send_command_ack(ds4_tp *tp, uint64_t session_id, int status);
int ds4_tp_wait_command_ack(ds4_tp *tp, uint64_t session_id,
const char *operation, char *err, size_t errlen);
int ds4_tp_wait_command_status(ds4_tp *tp, uint64_t session_id, int *status,
const char *operation, char *err, size_t errlen);
/* Both ranks call at matching prefill boundaries. Cancellation is agreed
* here, never sampled independently while either rank is inside a GPU gate. */
int ds4_tp_sync_checkpoint(ds4_tp *tp, uint32_t point, int current, int total,
bool requested, bool *cancelled);
int ds4_tp_send_stop(ds4_tp *tp);
/* Worker: blocks for the next mirrored command. Frame types below; for
@@ -196,6 +235,12 @@ typedef enum {
DS4_TP_FRAME_MIXED_BATCH = 16,
DS4_TP_FRAME_COMMAND_ACK = 17,
DS4_TP_FRAME_SYNC_MULTIMODAL = 18,
DS4_TP_FRAME_RDMA_WARM = 19,
DS4_TP_FRAME_RDMA_POSTED = 20,
DS4_TP_FRAME_GLM_MTP = 21,
DS4_TP_FRAME_SYNC_CHECKPOINT = 22,
DS4_TP_FRAME_DSPARK_DRAFT = 23,
DS4_TP_FRAME_SPEC_RESTORE = 24,
} ds4_tp_frame_type;
typedef struct {
@@ -203,6 +248,7 @@ typedef struct {
uint64_t session_id;
uint64_t seq;
int value;
int limit;
int *tokens;
uint32_t n_tokens;
ds4_tp_batch_item *items;
@@ -229,13 +275,73 @@ int ds4_tp_recv_logits_half(ds4_tp *tp, float *half, uint32_t count);
/* Speculative verify mirroring. The leader announces a draft block right
* before both ranks run the expert-split batch verify; the worker then blocks
* on the commit frame, which carries the leader's decision: full_accept keeps
* the pushed rows, otherwise both sides roll back and replay replay_n tokens
* through the gated single-token decode in lockstep. */
* on the commit frame. A full commit keeps the entire verified block, a prefix
* commit restores the matching verifier prefix on both ranks, and rollback
* restores the original frontier before replaying token_count tokens. */
typedef enum {
DS4_TP_VERIFY_ROLLBACK_REPLAY = 0,
DS4_TP_VERIFY_COMMIT_FULL = 1,
DS4_TP_VERIFY_COMMIT_PREFIX = 2,
} ds4_tp_verify_commit_mode;
int ds4_tp_send_verify(ds4_tp *tp, uint64_t session_id,
const int *drafts, uint32_t n);
int ds4_tp_send_verify_commit(ds4_tp *tp, int32_t full_accept, int32_t replay_n);
int ds4_tp_recv_verify_commit(ds4_tp *tp, int32_t *full_accept, int32_t *replay_n);
/* VERIFY flags. DS4_TP_VERIFY_HEAD_SPLIT: greedy block, each rank computes
* the verify head on its vocabulary half and the ranks combine row top-1s
* (ds4_tp_small_exchange); the worker then sends its half of the one row
* the leader commits (ds4_tp_head_row_send/recv). */
#define DS4_TP_VERIFY_HEAD_SPLIT 1u
/* DS4_TP_VERIFY_DSPARK_TP: the block holds at least one row of the last
* lockstep DRAFT's proposal, which the worker also holds and checks (a block
* trimmed to its seed is unmarked); DS4_TP_VERIFY_DSPARK_SEED: the block
* starts with the seed, so the proposal rows are drafts[1..]. */
#define DS4_TP_VERIFY_DSPARK_TP 2u
#define DS4_TP_VERIFY_DSPARK_SEED 4u
#define DS4_TP_VERIFY_FLAGS \
(DS4_TP_VERIFY_HEAD_SPLIT | DS4_TP_VERIFY_DSPARK_TP | DS4_TP_VERIFY_DSPARK_SEED)
int ds4_tp_send_verify_flags(ds4_tp *tp, uint64_t session_id,
const int *drafts, uint32_t n, uint32_t flags);
/* Lockstep TP drafter. The leader sends DSPARK_DRAFT where it would prepare
* a DSpark proposal. DRAFT: both ranks run the proposal for (token, pos) on
* their halves of the drafter. MAINTAIN: both only bring the drafter history
* up to pos; the leader may still draft on its own with the full drafter.
* The worker receives the token in value, pos in limit and the mode in seq. */
enum {
DS4_TP_DSPARK_MAINTAIN = 1,
DS4_TP_DSPARK_DRAFT = 2,
};
int ds4_tp_send_dspark_draft(ds4_tp *tp, uint64_t session_id, int token,
int pos, uint32_t mode);
/* Startup agreement, after the execution mode: enabled only when both ranks
* opted in with byte-identical metadata. When only one opted in, both keep
* the leader-local drafter (*agreed false, *peer_on reports the peer);
* different metadata fails. */
#define DS4_TP_DSPARK_META_BYTES 64u
int ds4_tp_agree_dspark_split(ds4_tp *tp, bool mine_on, const void *meta,
bool *agreed, bool *peer_on,
char *err, size_t errlen);
/* Lockstep symmetric exchange of up to DS4_TP_SMALL_EXCHANGE_MAX bytes over
* polled slab records (RDMA command mailbox only; 0 without it). Both
* ranks must pass the same byte count, which the record carries and the
* receiver checks. Failure marks the context failed. */
#define DS4_TP_SMALL_EXCHANGE_MAX 192u
int ds4_tp_small_exchange(ds4_tp *tp, const void *mine, void *peer, uint32_t bytes);
/* Worker -> leader float row of up to DS4_TP_HEAD_ROW_MAX floats (mailbox
* only), sent after the current verify block's begin. The worker writes the
* row into ds4_tp_head_row_out() (host memory inside the registered slab),
* then sends; the leader receives it into dst. The record carries the
* verify block, row and float count and the leader requires all three, so
* a row sent for a block the leader then rolled back without reading is
* never mistaken for a later one. */
#define DS4_TP_HEAD_ROW_MAX 131072u
float *ds4_tp_head_row_out(ds4_tp *tp);
int ds4_tp_head_row_send(ds4_tp *tp, uint32_t row, uint32_t floats);
int ds4_tp_head_row_recv(ds4_tp *tp, uint32_t row, float *dst, uint32_t floats);
int ds4_tp_send_verify_commit(ds4_tp *tp, int32_t mode, int32_t token_count);
int ds4_tp_recv_verify_commit(ds4_tp *tp, int32_t *mode, int32_t *token_count);
/* Standalone worker mode entry. Loads nothing itself: the engine is already
* open. */
ds4_web.c
+75 -6
View File
@@ -1144,6 +1144,46 @@ static char *web_browser_ws_url(ds4_web *web, char *err, size_t err_len) {
return ws;
}
/* Count page-type CDP targets. Once the last tab is closed a visible Chrome
has no window left, and a windowless Chrome rejects Target.createTarget
requests that ask for a background tab in the (now absent) window. */
static int web_page_target_count(ds4_web *web) {
char err[160] = {0};
char *body = web_http_request("GET", web->port, "/json/list", err, sizeof(err));
if (!body) return -1;
int n = 0;
const char *p = body;
while ((p = strstr(p, "\"type\"")) != NULL) {
p += 6;
while (*p == ' ' || *p == '\t' || *p == '\r' || *p == '\n') p++;
if (*p++ != ':') continue;
while (*p == ' ' || *p == '\t' || *p == '\r' || *p == '\n') p++;
if (strncmp(p, "\"page\"", 6) == 0) n++;
}
free(body);
return n;
}
/* Fallback target creation over plain HTTP. PUT /json/new works even when
Chrome has no window, which is exactly when Target.createTarget fails.
The verb must be PUT: modern Chrome answers GET /json/new with 405. */
static char *web_create_target_http(ds4_web *web, const char *url,
char *err, size_t err_len) {
char *enc = web_url_encode(url);
web_buf path = {0};
web_buf_puts(&path, "/json/new?");
web_buf_puts(&path, (enc && enc[0]) ? enc : "about%3Ablank");
free(enc);
char *path_s = web_buf_take(&path);
char *body = web_http_request("PUT", web->port, path_s, err, err_len);
free(path_s);
if (!body) return NULL;
char *id = web_json_get_string(body, "id");
free(body);
if (!id) web_set_err(err, err_len, "PUT /json/new returned no target id");
return id;
}
static bool web_open_tab(ds4_web *web, const char *url, web_tab *tab,
char *err, size_t err_len) {
memset(tab, 0, sizeof(*tab));
@@ -1168,14 +1208,33 @@ static bool web_open_tab(ds4_web *web, const char *url, web_tab *tab,
params_s, err, err_len);
free(params_s);
web_ws_close(&browser);
if (!resp) return false;
tab->id = web_json_get_string(resp, "targetId");
free(resp);
if (resp) {
tab->id = web_json_get_string(resp, "targetId");
if (!tab->id) {
char *why = web_json_get_string(resp, "message");
web_log(web, why ? why : "Target.createTarget returned no target id");
free(why);
}
free(resp);
}
/* A windowless Chrome refuses Target.createTarget with newWindow:false,
which is the state we land in the moment a previous tab is closed.
Recover through the HTTP endpoint instead of failing the request. */
if (!tab->id) {
web_tab_free(tab);
web_set_err(err, err_len, "Chrome did not return a page target id");
return false;
char herr[160] = {0};
tab->id = web_create_target_http(web, url, herr, sizeof(herr));
if (!tab->id) {
web_tab_free(tab);
if (herr[0])
web_set_err(err, err_len,
"Chrome did not return a page target id (%s)", herr);
else
web_set_err(err, err_len, "Chrome did not return a page target id");
return false;
}
web_log(web, "recovered Chrome target creation over HTTP");
}
char ws_url[PATH_MAX + 128];
@@ -1187,6 +1246,16 @@ static bool web_open_tab(ds4_web *web, const char *url, web_tab *tab,
static void web_close_tab(ds4_web *web, const web_tab *tab) {
if (!web || !tab || !tab->id || !tab->id[0]) return;
/* Never leave Chrome with zero page targets: that means no window, and a
windowless browser is what breaks the next Target.createTarget. Keeping
one tab alive is enough to hold the window open. */
int remaining = web_page_target_count(web);
if (remaining >= 0 && remaining <= 1) {
web_log(web, "keeping last Chrome page target open");
return;
}
char *enc = web_url_encode(tab->id);
web_buf path = {0};
web_buf_puts(&path, "/json/close/");
gguf-tools/.gitignore
+6
View File
@@ -12,3 +12,9 @@ quality-testing/data/*
!quality-testing/data/flash/**
!quality-testing/data/glm53-flash-openrouter-zai-fp8-100/
!quality-testing/data/glm53-flash-openrouter-zai-fp8-100/**
!quality-testing/data/flash-vision-exp-openrouter-novita-100/
!quality-testing/data/flash-vision-exp-openrouter-novita-100/**
!quality-testing/data/qwen38-flash-alibaba-100/
!quality-testing/data/qwen38-flash-alibaba-100/**
!quality-testing/data/qwen38-flash-alibaba-long/
!quality-testing/data/qwen38-flash-alibaba-long/**
gguf-tools/Makefile
+8 -6
View File
@@ -12,9 +12,9 @@ NATIVE_CPU_FLAG ?= -mcpu=native
QUANTS_SHARED := libds4quants.dylib
SHARED_FLAGS := -dynamiclib
QUALITY_LDLIBS := -lm -pthread -framework Foundation -framework Metal
QUALITY_TARGETS := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o ds4_image.o rax.o ds4_gpu_args.o
QUALITY_OBJS := ../ds4.o ../ds4_metal.o ../ds4_distributed.o ../ds4_tp.o ../ds4_ssd.o ../ds4_layer_pack.o ../ds4_image.o ../rax.o ../ds4_gpu_args.o
QUALITY_LINK := $(CC) $(CFLAGS) -I.. -o
QUALITY_TARGETS := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o ds4_engram.o ds4_image.o rax.o ds4_gpu_args.o
QUALITY_OBJS := ../ds4.o ../ds4_metal.o ../ds4_distributed.o ../ds4_tp.o ../ds4_ssd.o ../ds4_layer_pack.o ../ds4_engram.o ../ds4_image.o ../rax.o ../ds4_gpu_args.o
QUALITY_LINK = $(CC) $(CFLAGS) -I.. -o
LLAMA_CPP_LDLIBS ?= -L$(LLAMA_CPP_BIN) -Wl,-rpath,$(abspath $(LLAMA_CPP_BIN)) \
-lllama -lggml -lggml-base -lggml-cpu -lggml-blas -lggml-metal \
-framework Foundation -framework Metal -framework Accelerate
@@ -31,10 +31,10 @@ NVCC_ARCH_FLAGS := -arch=$(CUDA_ARCH)
endif
NVCCFLAGS ?= -O3 --use_fast_math $(NVCC_ARCH_FLAGS) -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
QUALITY_LDLIBS ?= -lm -Xcompiler -pthread -L$(CUDA_HOME)/targets/sbsa-linux/lib -L$(CUDA_HOME)/lib64 -lcudart -lcublas
QUALITY_TARGETS := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o ds4_image.o rax.o ds4_gpu_args.o \
QUALITY_TARGETS := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o ds4_engram.o ds4_image.o rax.o ds4_gpu_args.o \
cuda/mmq/ds4_ggml_stubs.o cuda/mmq/ds4_mmq.o cuda/mmq/ds4_mmq_d2r.o \
cuda/mmq/quantize.o cuda/mmq/mmid.o cuda/mmq/mmvq.o cuda/mmq/ds4_repack.o
QUALITY_OBJS := ../ds4.o ../ds4_cuda.o ../ds4_distributed.o ../ds4_tp.o ../ds4_ssd.o ../ds4_layer_pack.o ../ds4_image.o ../rax.o ../ds4_gpu_args.o \
QUALITY_OBJS := ../ds4.o ../ds4_cuda.o ../ds4_distributed.o ../ds4_tp.o ../ds4_ssd.o ../ds4_layer_pack.o ../ds4_engram.o ../ds4_image.o ../rax.o ../ds4_gpu_args.o \
../cuda/mmq/ds4_ggml_stubs.o ../cuda/mmq/ds4_mmq.o ../cuda/mmq/ds4_mmq_d2r.o \
../cuda/mmq/quantize.o ../cuda/mmq/mmid.o ../cuda/mmq/mmvq.o ../cuda/mmq/ds4_repack.o
QUALITY_LINK := $(NVCC) $(NVCCFLAGS) -I.. -o
@@ -45,10 +45,12 @@ endif
CFLAGS ?= -O3 -Wall -Wextra -std=c11 $(NATIVE_CPU_FLAG)
CPPFLAGS ?= -D_GNU_SOURCE
.PHONY: all clean quality-score quality-llama-score
.PHONY: all clean quants-shared quality-score quality-llama-score
all: deepseek4-quantize $(QUANTS_SHARED)
quants-shared: $(QUANTS_SHARED)
deepseek4-quantize: deepseek4-quantize.c quants.c quants.h
$(CC) $(CFLAGS) $(CPPFLAGS) -o $@ deepseek4-quantize.c quants.c -lm -pthread
gguf-tools/README.md
+190 -1
View File
@@ -14,6 +14,96 @@ The important pieces are:
- `quality-testing/`: prompts and scripts used to compare local GGUF variants
against official DeepSeek V4 Flash continuations.
## Qwen3.8 IQ2_XXS experiment
`qwen4_iq2.py` builds mixed IQ2_XXS/MXFP4 main weights with
embedded MTP. It quantizes the 48 trunk layers' gate/up experts directly from
the official BF16 checkpoint using the same pinned imatrix as the Q4_K pack.
It then copies every other tensor from an existing combined Q4_K `qwen4exp`
GGUF, including MXFP4 down projections, dense/control weights and MTP.
This isolates the gate/up precision change for a quality comparison.
The Python environment needs NumPy and `huggingface_hub`. Build the existing
native quantizer library with `make -C gguf-tools quants-shared`, then run:
```sh
python3 gguf-tools/qwen4_iq2.py quantize \
--source /path/to/official-bf16-checkpoint \
--imatrix /path/to/imatrix_unsloth.gguf_file \
--library gguf-tools/libds4quants.dylib \
--experts-dir /path/to/iq2-experts --threads 8
python3 gguf-tools/qwen4_iq2.py assemble \
--template /path/to/combined-q4k-mtp-pleext.gguf \
--experts-dir /path/to/iq2-experts \
--out /path/to/Qwen3.8-Flash-Next-IQ2XXSImatrix-MXFP4Down-MTP.gguf
```
The first stage can run beside the source checkpoint on another machine;
transfer the complete experts directory, including `experts.json`, for
assembly. Repeating the first command verifies completed tensors before
resuming. Assembly refuses existing outputs and verifies every written tensor
before publishing the completed GGUF and its JSON manifest.
The intermediate main model is approximately 50.3 GB. Finish it with the
[native n-gram packer](#native-qwen-n-grams) before inference. Quantization
changes model outputs; fitting weights on disk does not establish a 64 GB
runtime fit or acceptable reasoning quality. Compare with the same evaluator
settings using `ds4-eval --suite hard`.
### Experimental padded Q2_K down projections
Use `--projection down` on both stages to quantize the 48 trunk down
projections directly from BF16, with the pinned imatrix. Their 640 logical
inputs are zero-padded to 768 for three Q2_K blocks per row. Assemble against
the existing IQ2_XXS/MXFP4 combined model, using a new experts directory and
output path. All other tensors, including MTP, are copied unchanged:
```sh
uv run --with numpy --with huggingface_hub python gguf-tools/qwen4_iq2.py quantize \
--projection down --source /path/to/official-bf16-checkpoint \
--imatrix /path/to/imatrix_unsloth.gguf_file \
--library gguf-tools/libds4quants.dylib \
--experts-dir /path/to/q2-down-experts --threads 8
uv run --with numpy --with huggingface_hub python gguf-tools/qwen4_iq2.py assemble \
--projection down --template /path/to/IQ2XXSImatrix-MXFP4Down-MTP.gguf \
--experts-dir /path/to/q2-down-experts \
--out /path/to/IQ2XXSImatrix-Q2KDownPad768-MTP.gguf
```
This layout requires the padded-down Qwen runtime support. The GGUF stores
physical dimensions `[768, 2560, 512]`; the architecture's expert width remains
640. Metal decode and prefill use the padded weight stride while reading only
640 activation columns. The CPU reference supplies zero activation padding.
The shared expert and MTP retain their original dimensions and formats.
Saving 88 bytes per trunk down row reduces the main model by 5.15625 GiB;
quality and speed must be evaluated for this new recipe.
### Native Qwen n-grams
Finish main-only packs with the original BF16 n-gram shards, not the old
quantized sidecar:
```sh
python3 gguf-tools/qwen4_native_ngrams.py \
--model /path/to/IQ2XXSImatrix-Q2KDownPad768-MTP.gguf \
--source /path/to/Qwen3.8-Flash-Next \
--source-revision de4b8e4d43b917e7706784d8bb445c9af86a3540 \
--output gguf/Qwen3.8-Flash-Next-Q2.gguf
```
The source directory needs its index and the safetensors shards containing
`ple.ple_embedding` tensors. The packer checks the hash constants, copies all
main/MTP tensor bytes unchanged, and appends the original BF16 table at a
page-aligned offset. It verifies every copied payload before publishing the
file and its checksum report. Allow space for the complete new file plus
16 GiB spare. An incomplete output is never a runnable release artifact.
For the Q4 pack, use its main GGUF as `--model`. `--source` can also be a
previously verified native n-gram GGUF from the same pinned checkpoint.
This avoids keeping a second copy of the original safetensors. The final
Q2 and Q4 files include 95.37 GiB of disk-only n-grams; no table precision
is lost, and the main-model quantization is unchanged.
## Build
```sh
@@ -119,6 +209,67 @@ gguf-tools/deepseek4-quantize \
`--compare-tensor` regenerates a single tensor and byte-compares it against the
template or `--compare-gguf`. `--threads N` controls routed-expert workers.
## Convert DeepSeek V4.1 Flash
V4.1 uses its own converter; a V4 template is not compatible. Install NumPy,
tokenizers and SymPy, then build the quantizer library:
```sh
make -C gguf-tools libds4quants.dylib
python3 gguf-tools/deepseek41_quantize.py \
--hf models/DeepSeek-V4.1-Flash \
--source-revision df42c109f1defefcbfcedbe7d905718a12266e40 \
--out gguf/DeepSeek-V4.1-Flash-IQ2_XXS-Q2_K-bootstrap.gguf --dry-run
```
Omit `--dry-run` to write the file. Add `--resume` after an interrupted conversion.
Use `libds4quants.so` on Linux. Gate/up experts use IQ2_XXS and down experts use
Q2_K; attention, shared experts and the output head use Q8_0. `--quant q4` uses
Q4_K experts instead and `--quant mxfp4` keeps the released MXFP4 experts
unchanged, repacked into GGUF blocks without calibration. Engram rows retain
their original FP8 values and scales, packed together at the end of the GGUF for
disk lookups. Vision and DSpark weights are not included.
The first conversion uses weight-energy importance for IQ2_XXS. After runtime
calibration, add `--imatrix FILE` and choose a new output filename to regenerate
from the original safetensors. Do not requantize the first GGUF.
For Q4, add `--quant q4` and use `DeepSeek-V4.1-Flash-Q4.gguf` as the output.
This changes only the routed experts to Q4_K; the other tensor formats and
disk-only Engram layout stay the same. The same imatrix works for both recipes.
Check the finished artifact against the pinned source before running it:
```sh
python3 gguf-tools/deepseek41_validate_gguf.py \
--hf models/DeepSeek-V4.1-Flash \
--source-revision df42c109f1defefcbfcedbe7d905718a12266e40 \
--gguf gguf/DeepSeek-V4.1-Flash-IQ2_XXS-Q2_K-bootstrap.gguf --payload
```
For a calibrated file, pass the same `--imatrix FILE` used during conversion.
Pass `--quant q4` to the audit as well when checking a Q4 file.
The audit checks the complete layout, all non-expert tensors, sampled experts
and native Engram rows. It does not replace [inference quality tests](quality-testing/deepseek-v4.1-flash-20260910/README.md).
### Convert V4.1 DSpark
The V4.1 support converter needs NumPy, the official config and tensor index,
and only the shards containing `mtp.*` tensors (44, 45 and 46 of the pinned
checkpoint). No imatrix or main-model GGUF is needed:
```sh
python3 gguf-tools/deepseek41_dspark.py \
--hf models/DeepSeek-V4.1-Flash \
--source-revision df42c109f1defefcbfcedbe7d905718a12266e40 \
--out gguf/DeepSeek-V4.1-Flash-DSpark.gguf
```
The result is 8.04 GiB. Routed experts keep their native MXFP4 payloads;
dense weights expand losslessly to F16 or F32. Add `--dry-run` to inspect the
layout, or `--verify` to compare every tensor in an existing output against
the source. This file works only with the V4.1 Flash target.
## Convert A DSpark Support Checkpoint
The DSpark Flash checkpoint is published as Hugging Face safetensors and stores
@@ -136,7 +287,7 @@ The manifest reads only `model.safetensors.index.json`; it does not require the
large shard files to be present. The final summary should report three DSpark
stages and zero unknown DSpark tensors before attempting a full conversion.
To build the support GGUF used by `ds4 --mtp`, run the DSpark support mode. This
To build the support GGUF used by `ds4 --dspark`, run the DSpark support mode. This
mode writes standalone DSpark metadata plus the packed `mtp.*` tensor payloads;
it does not require a base-model GGUF template:
@@ -147,6 +298,12 @@ gguf-tools/deepseek4-quantize \
--out DeepSeek-V4-Flash-DSpark-support-0731.gguf
```
Add `--experts mxfp4` to preserve the checkpoint's routed expert weights
without requantization. Use a different output filename, such as
`DeepSeek-V4-Flash-DSpark-MXFP4-support-0731.gguf`. This takes about 10.1 GiB
instead of 5.6 GiB; the other tensors keep the same formats. A more accurate
drafter can improve acceptance, but does not guarantee faster generation.
`--dspark-support --dry-run` reads safetensors shard headers to derive exact
GGUF shapes and types, but it does not read tensor payloads. The DSpark metadata
defaults match the published Flash DSpark config: block size 5, target layers
@@ -191,3 +348,35 @@ make -C gguf-tools quality-score
gguf-tools/quality-testing/score_official MODEL.gguf gguf-tools/quality-testing/data/manifest.tsv /tmp/model.tsv 4096
python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv
```
## Qwen3.8 Flash Next
`qwen4_exp_convert.py` wraps llama.cpp's `convert_hf_to_gguf.py` and writes
the `qwen4exp` schema DS4 loads, including the MTP block as `blk.<n>.nextn.*`.
It needs a llama.cpp master checkout (b96806d96 or newer, found through
`--llama-cpp` or `$LLAMA_CPP`) and a Python with torch, safetensors and
transformers, such as llama.cpp's own venv:
```sh
python gguf-tools/qwen4_exp_convert.py --src /path/to/Qwen3.8-Flash-Next \
--source-revision de4b8e4d43b917e7706784d8bb445c9af86a3540 \
--out Qwen3.8-Flash-Next-Q8.gguf --outtype q8_0
python gguf-tools/qwen4_exp_convert.py --src /path/to/Qwen3.8-Flash-Next \
--source-revision de4b8e4d43b917e7706784d8bb445c9af86a3540 \
--out Qwen3.8-Flash-Next-MXFP4.gguf --outtype q8_0 --experts mxfp4
```
Options: `--outtype q8_0|f32` (dense projections), `--experts
q8_0|mxfp4|q4_k|f32` (routed experts; `--experts-down` picks the 640-wide down
projection type when the gate/up type needs 256-wide rows), `--hc-type f16|f32|q8_0`
(hyper-connection mixers), `--indexer bf16|f16|q8_0|f32` (the QSA indexer
projections, kept at the released BF16 by default), `--no-mtp` and
`--dry-run`. Norms, conv kernels, `ssm_a`, dt biases and the routers stay F32.
N-grams always retain the original BF16 bytes. The converter finishes through
the native packer above; its temporary main-weight file is removed on success.
`gen_qwen4_unicode.py` regenerates `ds4_qwen4_unicode.inc` for the `qwen35`
pre-tokenizer from a current `regex` release.
`make test-qwen4-kernels` runs the Metal kernel tests and
`make test-qwen4-vision` checks the vision tower against the HF implementation
(`tests/qwen4_vision_ref.py`).
gguf-tools/deepseek4-quantize.c
+177 -15
View File
@@ -46,8 +46,19 @@
#define DS4_KV_QUANTIZE_IMATRIX_N_ENTRIES "quantize.imatrix.entries_count"
#define DS4_KV_QUANTIZE_IMATRIX_N_CHUNKS "quantize.imatrix.chunks_count"
#define DS4_KV_COMPRESS_RATIOS "deepseek4.attention.compress_ratios"
#define DS4_KV_RMS_EPS "deepseek4.attention.layer_norm_rms_epsilon"
#define DS4_KV_CHECKPOINT_VARIANT "deepseek4.checkpoint_variant"
#define DS4_KV_VISION_SIDECAR_REQUIRED "deepseek4.vision.sidecar_required"
#define DS4_KV_SOURCE_URL "general.source.url"
#define DS4_KV_SOURCE_REVISION "general.source.revision"
#define DS4_GGUF_DEFAULT_ALIGNMENT 32
#define DS4_VISION_EXP_SOURCE_URL \
"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"
#define DS4_VISION_EXP_SOURCE_REVISION \
"e46e16bf6035c6f317eb2ac7458eb0362926d402"
#define DS4_VISION_EXP_NAME "DeepSeek V4 Flash Vision Experimental"
typedef enum {
GGUF_TYPE_UINT8 = 0,
GGUF_TYPE_INT8 = 1,
@@ -351,12 +362,31 @@ static int64_t json_i64(const json_doc *d, int tok) {
return strtoll(tmp, NULL, 10);
}
static double json_f64(const json_doc *d, int tok) {
char tmp[128];
const int n = d->v[tok].end - d->v[tok].start;
if (n <= 0 || n >= (int)sizeof(tmp)) die("bad JSON number");
memcpy(tmp, d->js + d->v[tok].start, (size_t)n);
tmp[n] = '\0';
errno = 0;
char *end = NULL;
double value = strtod(tmp, &end);
if (errno != 0 || end == tmp || *end != '\0' || !isfinite(value)) {
die("bad JSON number");
}
return value;
}
typedef struct {
uint32_t *compress_ratios;
uint64_t n_compress_ratios;
float rms_eps;
bool vision_exp;
char *source_revision;
} hf_model_metadata;
static hf_model_metadata load_hf_model_metadata(const char *hf_dir) {
static hf_model_metadata load_hf_model_metadata(const char *hf_dir,
const char *source_revision) {
hf_model_metadata m = {0};
char *path = path_join(hf_dir, "config.json");
size_t len = 0;
@@ -386,6 +416,36 @@ static hf_model_metadata load_hf_model_metadata(const char *hf_dir) {
m.compress_ratios[j++] = (uint32_t)value;
}
int rms_eps = json_obj_get(&d, 0, "rms_norm_eps");
if (rms_eps < 0) {
fprintf(stderr, "error: missing rms_norm_eps in %s\n", path);
exit(1);
}
double rms = json_f64(&d, rms_eps);
if (rms <= 0.0 || rms > 1.0) {
fprintf(stderr, "error: invalid rms_norm_eps in %s: %.9g\n", path, rms);
exit(1);
}
m.rms_eps = (float)rms;
int vision_layers = json_obj_get(&d, 0, "vision_n_layers");
if (vision_layers >= 0) {
int64_t count_layers = json_i64(&d, vision_layers);
if (count_layers != 32) {
fprintf(stderr, "error: unsupported vision_n_layers in %s: %" PRId64 "\n",
path, count_layers);
exit(1);
}
m.vision_exp = true;
if (!source_revision || strcmp(source_revision, DS4_VISION_EXP_SOURCE_REVISION) != 0) {
fprintf(stderr,
"error: Vision-Exp conversion requires --source-revision %s\n",
DS4_VISION_EXP_SOURCE_REVISION);
exit(1);
}
}
if (source_revision) m.source_revision = xstrdup(source_revision);
json_free(&d);
free(text);
free(path);
@@ -394,6 +454,7 @@ static hf_model_metadata load_hf_model_metadata(const char *hf_dir) {
static void free_hf_model_metadata(hf_model_metadata *m) {
free(m->compress_ratios);
free(m->source_revision);
memset(m, 0, sizeof(*m));
}
@@ -1669,6 +1730,15 @@ static void write_u64(FILE *fp, uint64_t v) {
if (fwrite(&v, sizeof(v), 1, fp) != 1) die("write u64 failed");
}
static void write_f32(FILE *fp, float v) {
if (fwrite(&v, sizeof(v), 1, fp) != 1) die("write f32 failed");
}
static void write_bool(FILE *fp, bool v) {
const uint8_t value = v ? 1u : 0u;
if (fwrite(&value, sizeof(value), 1, fp) != 1) die("write bool failed");
}
static void write_gguf_string(FILE *fp, const char *s) {
uint64_t n = strlen(s);
write_u64(fp, n);
@@ -1733,6 +1803,7 @@ static gguf_file load_gguf_metadata_with_override(const char *path,
byte_span *kv_keep = xcalloc((size_t)g.n_kv, sizeof(kv_keep[0]));
uint64_t n_kv_keep = 0;
bool found_compress_ratios = false;
bool found_rms_eps = false;
off_t kv_start = ftello(fp);
if (kv_start < 0) die("GGUF ftell failed");
@@ -1742,6 +1813,7 @@ static gguf_file load_gguf_metadata_with_override(const char *path,
char *key = read_gguf_string_fp(fp);
uint32_t type = read_u32_le_fp(fp, "GGUF KV type");
if (strcmp(key, DS4_KV_COMPRESS_RATIOS) == 0) found_compress_ratios = true;
if (strcmp(key, DS4_KV_RMS_EPS) == 0) found_rms_eps = true;
if (strcmp(key, "general.alignment") == 0 && type == GGUF_TYPE_UINT32) {
uint32_t a = read_u32_le_fp(fp, "GGUF alignment");
if (a) g.alignment = a;
@@ -1765,7 +1837,15 @@ static gguf_file load_gguf_metadata_with_override(const char *path,
* otherwise the output can contain duplicate GGUF metadata with stale
* and new values.
*/
const bool replace_from_config = metadata && strcmp(key, DS4_KV_COMPRESS_RATIOS) == 0;
const bool replace_from_config = metadata &&
(strcmp(key, DS4_KV_COMPRESS_RATIOS) == 0 ||
strcmp(key, DS4_KV_RMS_EPS) == 0 ||
(metadata->vision_exp &&
(strcmp(key, "general.name") == 0 ||
strcmp(key, DS4_KV_SOURCE_URL) == 0 ||
strcmp(key, DS4_KV_SOURCE_REVISION) == 0 ||
strcmp(key, DS4_KV_CHECKPOINT_VARIANT) == 0 ||
strcmp(key, DS4_KV_VISION_SIDECAR_REQUIRED) == 0)));
if (!is_imatrix_kv_key(key) && !replace_from_config) {
kv_keep[n_kv_keep++] = (byte_span){
.start = (size_t)(rec_start - kv_start),
@@ -1777,6 +1857,9 @@ static gguf_file load_gguf_metadata_with_override(const char *path,
if (metadata && !found_compress_ratios) {
die("template has no deepseek4.attention.compress_ratios metadata");
}
if (metadata && !found_rms_eps) {
die("template has no deepseek4.attention.layer_norm_rms_epsilon metadata");
}
if (metadata && g.n_layers > metadata->n_compress_ratios) {
die("config.json compress_ratios is shorter than the template layer count");
}
@@ -1852,8 +1935,25 @@ static uint64_t fnv1a64_bytes(const uint8_t *data, size_t n) {
}
static size_t extra_hf_metadata_kv_size(const hf_model_metadata *metadata) {
return gguf_string_size(DS4_KV_COMPRESS_RATIOS) + 4 + 4 + 8 +
(size_t)metadata->n_compress_ratios * 4;
size_t n = gguf_string_size(DS4_KV_COMPRESS_RATIOS) + 4 + 4 + 8 +
(size_t)metadata->n_compress_ratios * 4;
n += gguf_string_size(DS4_KV_RMS_EPS) + 4 + sizeof(float);
if (metadata->vision_exp) {
n += gguf_string_size("general.name") + 4 +
gguf_string_size(DS4_VISION_EXP_NAME);
n += gguf_string_size(DS4_KV_SOURCE_URL) + 4 +
gguf_string_size(DS4_VISION_EXP_SOURCE_URL);
n += gguf_string_size(DS4_KV_SOURCE_REVISION) + 4 +
gguf_string_size(metadata->source_revision);
n += gguf_string_size(DS4_KV_CHECKPOINT_VARIANT) + 4 +
gguf_string_size("vision-exp");
n += gguf_string_size(DS4_KV_VISION_SIDECAR_REQUIRED) + 4 + 1;
}
return n;
}
static uint64_t extra_hf_metadata_kv_count(const hf_model_metadata *metadata) {
return 2u + (metadata->vision_exp ? 5u : 0u);
}
static void write_hf_metadata_kvs(FILE *fp, const hf_model_metadata *metadata) {
@@ -1864,6 +1964,26 @@ static void write_hf_metadata_kvs(FILE *fp, const hf_model_metadata *metadata) {
for (uint64_t i = 0; i < metadata->n_compress_ratios; i++) {
write_u32(fp, metadata->compress_ratios[i]);
}
write_gguf_string(fp, DS4_KV_RMS_EPS);
write_u32(fp, GGUF_TYPE_FLOAT32);
write_f32(fp, metadata->rms_eps);
if (metadata->vision_exp) {
write_gguf_string(fp, "general.name");
write_u32(fp, GGUF_TYPE_STRING);
write_gguf_string(fp, DS4_VISION_EXP_NAME);
write_gguf_string(fp, DS4_KV_SOURCE_URL);
write_u32(fp, GGUF_TYPE_STRING);
write_gguf_string(fp, DS4_VISION_EXP_SOURCE_URL);
write_gguf_string(fp, DS4_KV_SOURCE_REVISION);
write_u32(fp, GGUF_TYPE_STRING);
write_gguf_string(fp, metadata->source_revision);
write_gguf_string(fp, DS4_KV_CHECKPOINT_VARIANT);
write_u32(fp, GGUF_TYPE_STRING);
write_gguf_string(fp, "vision-exp");
write_gguf_string(fp, DS4_KV_VISION_SIDECAR_REQUIRED);
write_u32(fp, GGUF_TYPE_BOOL);
write_bool(fp, true);
}
}
static output_context build_output_context(const gguf_file *tmpl, const quant_policy *policy,
@@ -1871,7 +1991,8 @@ static output_context build_output_context(const gguf_file *tmpl, const quant_po
const hf_model_metadata *metadata) {
output_context out = {0};
out.n_tensors = tmpl->n_tensors;
out.n_kv_extra = 1 + extra_imatrix_kv_count(im);
out.n_kv_extra = extra_hf_metadata_kv_count(metadata) +
extra_imatrix_kv_count(im);
out.alignment = tmpl->alignment;
out.tensors = xcalloc((size_t)out.n_tensors, sizeof(out.tensors[0]));
size_t tensor_info = 0;
@@ -2013,6 +2134,7 @@ typedef struct {
char *compare_gguf;
char *compare_tensor;
char *imatrix_file;
char *source_revision;
quant_policy policy;
int n_experts;
int n_threads;
@@ -2118,6 +2240,7 @@ static const dspark_name_rule dspark_stage_rules[] = {
{"ffn.gate.weight", "ffn_gate_inp.weight", "emit"},
{"ffn.gate.bias", "exp_probs_b.bias", "emit"},
{"ffn.gate.bias_vl", "visual_router_bias", "consume_visual_sidecar"},
{"ffn_norm.weight", "ffn_norm.weight", "emit"},
{"ffn.shared_experts.w1.weight", "ffn_gate_shexp.weight", "emit"},
{"ffn.shared_experts.w1.scale", "ffn_gate_shexp.weight", "consume_scale"},
@@ -2385,10 +2508,13 @@ static void dspark_shape_reversed_from_info(tensor_meta *m, const st_info *info)
}
static void dspark_plan_set_size(dspark_tensor_plan *tp) {
if (tp->meta.type != DS4Q_TYPE_I32 && !is_quantizable_target(tp->meta.type)) {
const bool preserved_mxfp4 =
tp->kind == DSPARK_PLAN_EXPERT && tp->meta.type == DS4Q_TYPE_MXFP4;
if (tp->meta.type != DS4Q_TYPE_I32 &&
!is_quantizable_target(tp->meta.type) && !preserved_mxfp4) {
die("unsupported DSpark planned tensor type");
}
if (ds4q_can_quantize(tp->meta.type) &&
if ((ds4q_can_quantize(tp->meta.type) || preserved_mxfp4) &&
tp->meta.ne[0] % ds4q_block_size(tp->meta.type) != 0) {
fprintf(stderr,
"error: DSpark tensor %s ne[0]=%" PRId64
@@ -2502,13 +2628,17 @@ static void write_gguf_kv_u32_array(FILE *fp, const char *key, const uint32_t *v
}
static void dspark_plan_finalize(dspark_support_plan *plan,
const dspark_support_options *opt) {
const dspark_support_options *opt,
const hf_model_metadata *metadata) {
qsort(plan->tensors, (size_t)plan->len, sizeof(plan->tensors[0]), dspark_plan_cmp);
plan->alignment = DS4_GGUF_DEFAULT_ALIGNMENT;
plan->n_kv = 9;
const char *name = metadata->vision_exp ?
"DeepSeek V4 Flash Vision Experimental DSpark support" :
"DeepSeek V4 Flash DSpark support";
plan->n_kv = 9 + (metadata->vision_exp ? 3 : 0);
plan->kv_bytes =
gguf_kv_size_string("general.architecture", "deepseek4-dspark") +
gguf_kv_size_string("general.name", "DeepSeek V4 Flash DSpark support") +
gguf_kv_size_string("general.name", name) +
gguf_kv_size_u32("general.alignment") +
gguf_kv_size_u32("dspark.block_size") +
gguf_kv_size_u32("dspark.markov_rank") +
@@ -2516,6 +2646,15 @@ static void dspark_plan_finalize(dspark_support_plan *plan,
gguf_kv_size_u32_array("dspark.target_layer_ids", opt->target_layer_count) +
gguf_kv_size_u32("dspark.stage_count") +
gguf_kv_size_u32("dspark.n_layers");
if (metadata->vision_exp) {
plan->kv_bytes +=
gguf_kv_size_string(DS4_KV_SOURCE_URL,
DS4_VISION_EXP_SOURCE_URL) +
gguf_kv_size_string(DS4_KV_SOURCE_REVISION,
metadata->source_revision) +
gguf_kv_size_string(DS4_KV_CHECKPOINT_VARIANT,
"vision-exp");
}
size_t tensor_info = 0;
size_t off = 0;
@@ -2538,7 +2677,8 @@ static void dspark_plan_finalize(dspark_support_plan *plan,
static dspark_support_plan build_dspark_support_plan(st_db *db,
const quant_policy *policy,
const dspark_support_options *opt,
int requested_n_experts) {
int requested_n_experts,
const hf_model_metadata *metadata) {
(void)opt;
dspark_support_plan plan = {0};
str_list names = load_index_weight_names(db->hf_dir);
@@ -2589,7 +2729,7 @@ static dspark_support_plan build_dspark_support_plan(st_db *db,
}
}
}
dspark_plan_finalize(&plan, opt);
dspark_plan_finalize(&plan, opt, metadata);
return plan;
}
@@ -2646,6 +2786,7 @@ static byte_buf generate_dspark_tensor(st_db *db, const dspark_tensor_plan *tp,
static void write_dspark_support_gguf(st_db *db,
const dspark_support_plan *plan,
const dspark_support_options *opt,
const hf_model_metadata *metadata,
const char *out_path,
int n_threads,
const imatrix_store *imatrix) {
@@ -2656,7 +2797,11 @@ static void write_dspark_support_gguf(st_db *db,
write_u64(fp, (uint64_t)plan->len);
write_u64(fp, plan->n_kv);
write_gguf_kv_string(fp, "general.architecture", "deepseek4-dspark");
write_gguf_kv_string(fp, "general.name", "DeepSeek V4 Flash DSpark support");
write_gguf_kv_string(
fp, "general.name",
metadata->vision_exp ?
"DeepSeek V4 Flash Vision Experimental DSpark support" :
"DeepSeek V4 Flash DSpark support");
write_gguf_kv_u32(fp, "general.alignment", (uint32_t)plan->alignment);
write_gguf_kv_u32(fp, "dspark.block_size", opt->block_size);
write_gguf_kv_u32(fp, "dspark.markov_rank", opt->markov_rank);
@@ -2664,6 +2809,14 @@ static void write_dspark_support_gguf(st_db *db,
write_gguf_kv_u32_array(fp, "dspark.target_layer_ids", opt->target_layers, opt->target_layer_count);
write_gguf_kv_u32(fp, "dspark.stage_count", (uint32_t)plan->stages);
write_gguf_kv_u32(fp, "dspark.n_layers", (uint32_t)plan->stages);
if (metadata->vision_exp) {
write_gguf_kv_string(fp, DS4_KV_SOURCE_URL,
DS4_VISION_EXP_SOURCE_URL);
write_gguf_kv_string(fp, DS4_KV_SOURCE_REVISION,
metadata->source_revision);
write_gguf_kv_string(fp, DS4_KV_CHECKPOINT_VARIANT,
"vision-exp");
}
for (int i = 0; i < plan->len; i++) {
const tensor_meta *t = &plan->tensors[i].meta;
@@ -2731,6 +2884,7 @@ static void usage(const char *argv0) {
printf(" --dspark-target-layers CSV DSpark target layer ids metadata, default 40,41,42\n");
printf(" --imatrix FILE legacy .dat imatrix from ds4 --imatrix-out\n");
printf(" --imatrix-strict fail if a quantized tensor has no matching imatrix vector\n");
printf(" --source-revision SHA pin the Hugging Face source revision in output metadata\n");
printf(" --experts TYPE set routed w1/w2/w3 expert tensors to TYPE\n");
printf(" --routed-w1 TYPE routed gate expert tensor type\n");
printf(" --routed-w2 TYPE routed down expert tensor type\n");
@@ -2843,6 +2997,8 @@ static params parse_args(int argc, char **argv) {
p.imatrix_file = need_value(argc, argv, &i, arg);
} else if (strcmp(arg, "--imatrix-strict") == 0) {
p.imatrix_strict = true;
} else if (strcmp(arg, "--source-revision") == 0) {
p.source_revision = need_value(argc, argv, &i, arg);
} else if (strcmp(arg, "--experts") == 0 || strcmp(arg, "--routed") == 0) {
ds4q_type t = parse_type(need_value(argc, argv, &i, arg));
p.policy.routed_w1 = p.policy.routed_w2 = p.policy.routed_w3 = t;
@@ -3013,10 +3169,13 @@ int main(int argc, char **argv) {
if (p.imatrix_file) imatrix_load(&imatrix, p.imatrix_file, p.imatrix_strict);
if (p.dspark_support) {
hf_model_metadata metadata = load_hf_model_metadata(p.hf_dir,
p.source_revision);
st_db db;
db_open(&db, p.hf_dir);
dspark_support_plan plan =
build_dspark_support_plan(&db, &p.policy, &p.dspark, p.n_experts);
build_dspark_support_plan(&db, &p.policy, &p.dspark,
p.n_experts, &metadata);
print_dspark_support_plan(&plan, &p.dspark);
if (p.dry_run) {
/* Plan only: build_dspark_support_plan reads shard headers, not tensor payloads. */
@@ -3026,6 +3185,7 @@ int main(int argc, char **argv) {
write_dspark_support_gguf(&db,
&plan,
&p.dspark,
&metadata,
p.out_gguf,
p.n_threads,
&imatrix);
@@ -3033,13 +3193,15 @@ int main(int argc, char **argv) {
}
free_dspark_support_plan(&plan);
db_close(&db);
free_hf_model_metadata(&metadata);
imatrix_free(&imatrix);
for (int i = 0; i < p.policy.n_overrides; i++) free(p.policy.overrides[i].prefix);
free(p.policy.overrides);
return 0;
}
hf_model_metadata metadata = load_hf_model_metadata(p.hf_dir);
hf_model_metadata metadata = load_hf_model_metadata(p.hf_dir,
p.source_revision);
gguf_file tmpl = load_gguf_metadata_with_override(p.template_gguf, &metadata);
if (p.n_experts <= 0) {
if (tmpl.n_experts > 0) {
gguf-tools/deepseek41_dspark.py
+310
View File
@@ -0,0 +1,310 @@
#!/usr/bin/env python3
"""Extract DeepSeek V4.1 Flash DSpark without further weight quantization.
Routed experts retain their native MXFP4 codes and scales. Dense weights are
expanded to F16 only when every value is exactly representable, otherwise F32.
The target model supplies the token embedding and output projection at runtime.
"""
import argparse
import hashlib
import json
import os
import re
import shutil
import struct
import sys
import time
from deepseek41_quantize import NativeQuantizer, scale_name
from glm53_quantize import (
SourceDB, TensorPlan, QTYPE_F16, QTYPE_F32, QTYPE_MXFP4, align,
conversion_signature, kv_string, kv_u32, kv_u32_array, load_resume_state,
print_plan, qtype_nbytes, save_resume_state, tensor_header,
read_exact,
)
ALIGNMENT = 16384
class LosslessWeights(NativeQuantizer):
def __init__(self):
# No C quantizer or imatrix is needed: this converter changes layouts,
# never weight values.
import numpy as np
self.np = np
self.fp8_lut = self._build_fp8_lut()
def dense_type(self, db, name):
values = self.to_f32(db, name)
with self.np.errstate(over="ignore", under="ignore"):
restored = values.astype(self.np.float16).astype(self.np.float32)
return QTYPE_F16 if self.np.array_equal(values, restored) else QTYPE_F32
def dense_bytes(self, db, item):
values = self.to_f32(db, item.source)
if item.qtype == QTYPE_F16:
with self.np.errstate(over="ignore", under="ignore"):
restored = values.astype(self.np.float16).astype(self.np.float32)
if not self.np.array_equal(values, restored):
raise ValueError(f"{item.name}: F16 would change source values")
return self.encode(values, item.qtype)
def expert_bytes(self, db, name):
np = self.np
info, sinfo = db.info(name), db.info(scale_name(name))
rows, packed_cols = info["shape"]
if (info["dtype"] != "I8" or packed_cols % 16 or
sinfo["dtype"] != "F8_E8M0" or
sinfo["shape"] != [rows, packed_cols // 16]):
raise ValueError(f"{name}: invalid native FP4 tensor or scales")
codes = np.frombuffer(db.read(name), dtype=np.uint8).reshape(-1, 16)
scales = np.frombuffer(db.read(scale_name(name)), dtype=np.uint8)
if np.any(scales == 255):
raise ValueError(f"{name}: nonfinite MXFP4 scale")
# Source nibbles are consecutive. GGUF puts values 0..15 in the low
# nibbles, 16..31 in the high nibbles, after the unchanged E8M0 byte.
unpacked = np.empty((codes.shape[0], 32), dtype=np.uint8)
unpacked[:, 0::2], unpacked[:, 1::2] = codes & 15, codes >> 4
packed = np.empty((codes.shape[0], 17), dtype=np.uint8)
packed[:, 0] = scales
packed[:, 1:] = unpacked[:, :16] | (unpacked[:, 16:] << 4)
if not (np.array_equal(packed[:, 1:] & 15, unpacked[:, :16]) and
np.array_equal(packed[:, 1:] >> 4, unpacked[:, 16:])):
raise ValueError(f"{name}: MXFP4 repacking changed a code")
return packed.tobytes()
def build_plan(db, config, weights):
if config.get("model_type") != "deepseek_v41":
raise ValueError("expected a DeepSeek V4.1 checkpoint")
c = config["text_config"]
if config["quantization_config"]["weight_block_size"] != [32, 32]:
raise ValueError("expected native 32x32 FP8 blocks")
dim, inter, hc = c["hidden_size"], c["moe_intermediate_size"], c["hc_mult"]
heads, hd = c["num_attention_heads"], c["head_dim"]
qrank, orank, groups = c["q_lora_rank"], c["o_lora_rank"], c["o_groups"]
experts = c["dspark_n_routed_experts"]
stages, vocab, rank = c["num_nextn_predict_layers"], c["vocab_size"], c["dspark_markov_rank"]
targets = c["dspark_target_layer_ids"]
if (stages != 3 or len(targets) != 3 or len(set(targets)) != 3 or
any(not 0 <= layer < c["num_hidden_layers"] for layer in targets) or
not 0 < c["dspark_num_experts_per_tok"] <= experts or
not 0 <= c["dspark_noise_token_id"] < vocab or
not 0 < c["dspark_block_size"] <= c["sliding_window"]):
raise ValueError("invalid DSpark dimensions or target layers")
plan, consumed = [], set()
def claim(name, shape, dtype=None):
info = db.info(name)
if info["shape"] != list(shape) or (dtype and info["dtype"] != dtype):
raise ValueError(f"{name}: unexpected {info['dtype']} {info['shape']}")
consumed.add(name)
if info["dtype"] in ("I8", "F8_E4M3"):
expected = ([shape[0], shape[1] // 16] if info["dtype"] == "I8"
else [(n + 31) // 32 for n in shape])
scale = db.info(scale_name(name))
if scale["dtype"] != "F8_E8M0" or scale["shape"] != expected:
raise ValueError(f"{name}: invalid native scale shape")
consumed.add(scale_name(name))
def regular(dst, src, shape, role):
claim(src, shape)
# Keep norms, routers, HC and confidence in F32 for the runtime's
# FP32 reductions. Matrix F16 is selected only after a lossless scan.
qt = weights.dense_type(db, src) if role in ("projection", "markov") else QTYPE_F32
plan.append(TensorPlan(dst, tuple(reversed(shape)), qt, role, source=src))
for stage in range(stages):
prefix = f"mtp.{stage}"
for site in ("attn", "ffn"):
for part, shape in (("fn", (hc * (hc + 2), hc * dim)),
("base", (hc * (hc + 2),)), ("scale", (3,))):
regular(f"{prefix}.hc_{site}_{part}.weight", f"{prefix}.hc_{site}_{part}", shape, "hc")
regular(f"{prefix}.{site}_norm.weight", f"{prefix}.{site}_norm.weight", (dim,), "norm")
for dst, src, shape, role in (
("attn_sinks.weight", "attn_sink", (heads,), "norm"),
("attn_q_a.weight", "wq_a.weight", (qrank, dim), "projection"),
("attn_q_b.weight", "wq_b.weight", (heads * hd, qrank), "projection"),
("attn_q_a_norm.weight", "q_norm.weight", (qrank,), "norm"),
("attn_kv.weight", "wkv.weight", (hd, dim), "projection"),
("attn_kv_a_norm.weight", "kv_norm.weight", (hd,), "norm"),
("attn_output_a.weight", "wo_a.weight", (groups * orank, heads * hd // groups), "projection"),
("attn_output_b.weight", "wo_b.weight", (dim, groups * orank), "projection"),
):
regular(f"{prefix}.{dst}", f"{prefix}.attn.{src}", shape, role)
regular(f"{prefix}.ffn_gate_inp.weight", f"{prefix}.ffn.gate.weight", (experts, dim), "router")
for suffix, dst in (("bias", "exp_probs_b.bias"), ("bias_vl", "exp_probs_b_vl.bias")):
regular(f"{prefix}.{dst}", f"{prefix}.ffn.gate.{suffix}", (experts,), "router")
for part, src, shape in (("gate", "w1", (inter, dim)),
("up", "w3", (inter, dim)), ("down", "w2", (dim, inter))):
regular(f"{prefix}.ffn_{part}_shexp.weight", f"{prefix}.ffn.shared_experts.{src}.weight", shape, "projection")
pattern = f"{prefix}.ffn.experts.{{expert}}.{src}.weight"
for expert in range(experts):
claim(pattern.format(expert=expert), (shape[0], shape[1] // 2), "I8")
plan.append(TensorPlan(f"{prefix}.ffn_{part}_exps.weight", (*reversed(shape), experts),
QTYPE_MXFP4, "experts", source=pattern, expert_layer=stage,
expert_part=part, expert_count=experts))
if stage == 0:
regular(f"{prefix}.main_proj.weight", f"{prefix}.main_proj.weight", (dim, dim * len(targets)), "projection")
regular(f"{prefix}.main_norm.weight", f"{prefix}.main_norm.weight", (dim,), "norm")
if stage == stages - 1:
regular(f"{prefix}.norm.weight", f"{prefix}.norm.weight", (dim,), "norm")
for src, dst in (("embed", "markov_w1"), ("head", "markov_w2")):
regular(f"{prefix}.markov_head.{dst}.weight", f"{prefix}.markov_head.{src}.weight", (vocab, rank), "markov")
regular(f"{prefix}.confidence_head.proj.weight", f"{prefix}.confidence_head.proj.weight", (1, dim + rank), "confidence")
unknown = {name for name in db.tensors if name.startswith("mtp.")} - consumed
if unknown:
raise ValueError(f"unclaimed DSpark tensors: {sorted(unknown)[:10]}")
offset = 0
for item in plan:
item.offset, item.nbytes = offset, qtype_nbytes(item.qtype, item.shape)
offset += align(item.nbytes, ALIGNMENT)
return plan
def metadata(config, revision):
c = config["text_config"]
records = [
kv_string("general.architecture", "deepseek41-dspark"),
kv_string("general.name", "DeepSeek V4.1 Flash DSpark"),
kv_string("general.source.url", "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash"),
kv_string("general.source.revision", revision),
kv_u32("general.alignment", ALIGNMENT),
kv_string("dspark.target_architecture", "deepseek41"),
kv_string("dspark.source_config", json.dumps(config, sort_keys=True, separators=(",", ":"))),
kv_string("dspark.weight_encoding", "native MXFP4; lossless F16/F32 expansion"),
kv_u32_array("dspark.target_layer_ids", c["dspark_target_layer_ids"]),
]
for key in ("block_size", "markov_rank", "noise_token_id", "n_routed_experts", "num_experts_per_tok"):
records.append(kv_u32(f"dspark.{key}", c[f"dspark_{key}"]))
return records
def source_fingerprint(db):
shards = sorted({info["shard"] for name, info in db.tensors.items() if name.startswith("mtp.")})
result = []
for shard in shards:
h = hashlib.sha256()
with open(os.path.join(db.hf_dir, shard), "rb") as fp:
for chunk in iter(lambda: fp.read(8 << 20), b""):
h.update(chunk)
result.append((shard, h.hexdigest()))
return result
def write_gguf(args, db, weights, plan, records):
data_start, data_bytes = print_plan(plan, records, [], ALIGNMENT)
if os.path.exists(args.out):
raise ValueError(f"refusing to overwrite {args.out}")
partial, journal = args.out + ".partial", args.out + ".partial.json"
fingerprint = source_fingerprint(db)
signature = hashlib.sha256((conversion_signature(plan, records, [], None) +
json.dumps(fingerprint)).encode()).hexdigest()
completed = 0
if os.path.exists(partial) or os.path.exists(journal):
if not args.resume or not (os.path.exists(partial) and os.path.exists(journal)):
raise ValueError("partial file and journal require --resume")
completed = load_resume_state(journal, signature, plan)
end = data_start + (plan[completed - 1].offset + align(plan[completed - 1].nbytes, ALIGNMENT) if completed else 0)
if shutil.disk_usage(os.path.dirname(os.path.abspath(args.out))).free < data_start + data_bytes - end + (32 << 30):
raise ValueError("insufficient disk space for output plus 32 GiB reserve")
header = b"GGUF" + struct.pack("<IQQ", 3, len(plan), len(records))
header += b"".join(records) + b"".join(tensor_header(item) for item in plan)
header += bytes(data_start - len(header))
if os.path.exists(partial):
with open(partial, "rb") as fp:
if fp.read(data_start) != header or os.fstat(fp.fileno()).st_size < end:
raise ValueError("partial GGUF is truncated or has a different header")
else:
with open(partial, "xb") as fp:
fp.write(header)
fp.flush()
os.fsync(fp.fileno())
save_resume_state(journal, signature, 0)
with open(partial, "r+b") as fp:
fp.truncate(end)
fp.seek(end)
for i in range(completed, len(plan)):
item, start = plan[i], time.monotonic()
if fp.tell() != data_start + item.offset:
raise ValueError(f"incorrect offset for {item.name}")
if item.is_expert:
for expert in range(item.expert_count):
fp.write(weights.expert_bytes(db, item.source.format(expert=expert)))
else:
fp.write(weights.dense_bytes(db, item))
if fp.tell() != data_start + item.offset + item.nbytes:
raise ValueError(f"incorrect payload size for {item.name}")
fp.write(bytes(align(item.nbytes, ALIGNMENT) - item.nbytes))
fp.flush()
os.fsync(fp.fileno())
save_resume_state(journal, signature, i + 1)
print(f"[{i + 1}/{len(plan)}] {item.name}: {time.monotonic() - start:.1f}s", flush=True)
# Refuse publication if an in-place source edit occurred during conversion.
if source_fingerprint(db) != fingerprint:
raise ValueError("source shards changed during conversion")
os.rename(partial, args.out)
os.unlink(journal)
def verify_gguf(path, db, weights, plan, records):
"""Check the complete layout and every payload, not a sample of experts."""
with open(path, "rb") as fp:
if read_exact(fp, 24, "header") != b"GGUF" + struct.pack("<IQQ", 3, len(plan), len(records)):
raise ValueError("GGUF header differs from source plan")
encoded = b"".join(records) + b"".join(tensor_header(t) for t in plan)
if read_exact(fp, len(encoded), "metadata and tensors") != encoded:
raise ValueError("GGUF metadata or tensor layout differs from source plan")
start = align(fp.tell(), ALIGNMENT)
size = start + plan[-1].offset + align(plan[-1].nbytes, ALIGNMENT)
if os.fstat(fp.fileno()).st_size != size:
raise ValueError("incorrect GGUF file size")
for item in plan:
fp.seek(start + item.offset)
if item.is_expert:
for expert in range(item.expert_count):
expected = weights.expert_bytes(db, item.source.format(expert=expert))
if read_exact(fp, len(expected), item.name) != expected:
raise ValueError(f"{item.name}: expert {expert} payload differs from source")
else:
expected = weights.dense_bytes(db, item)
if read_exact(fp, len(expected), item.name) != expected:
raise ValueError(f"{item.name}: dense payload differs from source")
print(f"PASS: all {len(plan)} tensor layouts and payloads match the source", flush=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--hf", required=True)
parser.add_argument("--out", required=True)
parser.add_argument("--source-revision", required=True)
parser.add_argument("--resume", action="store_true")
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--verify", action="store_true", help="audit an existing --out file without writing it")
args = parser.parse_args()
if not re.fullmatch("[0-9a-f]{40}", args.source_revision):
parser.error("source revision must be a full commit hash")
with open(os.path.join(args.hf, "config.json"), "rb") as fp:
config = json.load(fp)
db = SourceDB(args.hf, index_validator=lambda _: None, scale_validator=lambda _: None,
tensor_prefix="mtp.")
try:
weights = LosslessWeights()
plan = build_plan(db, config, weights)
records = metadata(config, args.source_revision)
if args.verify:
verify_gguf(args.out, db, weights, plan, records)
elif args.dry_run:
print_plan(plan, records, [], ALIGNMENT)
else:
write_gguf(args, db, weights, plan, records)
finally:
db.close()
if __name__ == "__main__":
try:
main()
except (ValueError, OSError) as error:
sys.exit(f"deepseek41-dspark: {error}")
gguf-tools/deepseek41_metadata.py
+154
View File
@@ -0,0 +1,154 @@
#!/usr/bin/env python3
"""DeepSeek V4.1 tokenizer and Engram metadata, derived from the released config.
The normalization, prime allocation and seeded hash multipliers follow DeepSeek's
MIT-licensed inference/engram.py. The runtime consumes the resulting arrays, not
a second implementation of Unicode normalization or NumPy's random generator.
"""
import json
import os
import struct
from glm53_quantize import (
GGUF_ARRAY, GGUF_STRING, GGUF_UINT32, GGUF_UINT64, kv_bool, kv_string,
kv_f32, kv_u32, kv_u32_array, pack_string,
)
# Apple Silicon VM pages are 16 KiB. The Engram extent must be removable
# without also unmapping bytes of the preceding resident tensor.
GGUF_ALIGNMENT = 16384
def array_record(key, kind, values):
header = pack_string(key) + struct.pack("<IIQ", GGUF_ARRAY, kind, len(values))
if kind == GGUF_STRING:
return header + b"".join(pack_string(value) for value in values)
fmt = {GGUF_UINT32: "I", GGUF_UINT64: "Q", 5: "i", 6: "f"}[kind]
return header + struct.pack(f"<{len(values)}{fmt}", *values)
def compressed_token_map(tokenizer):
from tokenizers import Regex, normalizers
sentinel = "\ue000"
normalizer = normalizers.Sequence([
normalizers.NFKC(), normalizers.NFD(), normalizers.StripAccents(),
normalizers.Lowercase(), normalizers.Replace(Regex(r"[ \t\r\n]+"), " "),
normalizers.Replace(Regex(r"^ $"), sentinel), normalizers.Strip(),
normalizers.Replace(sentinel, " "),
])
unique, result = {}, []
for token_id in range(tokenizer.get_vocab_size(with_added_tokens=True)):
text = tokenizer.decode([token_id], skip_special_tokens=False)
if "\ufffd" in text:
key = tokenizer.id_to_token(token_id)
else:
key = normalizer.normalize_str(text) or text
result.append(unique.setdefault(key, len(unique)))
return result, len(unique)
def engram_layout(config, tokenizer):
import numpy as np
from sympy import nextprime
token_map, vocab_size = compressed_token_map(tokenizer)
if vocab_size != config["engram_compressed_vocab_size"]:
raise ValueError(f"Engram vocabulary mismatch: {vocab_size}")
if len(token_map) != config["vocab_size"]:
raise ValueError("Engram requires every released tokenizer ID")
layers = config["engram_layer_ids"]
if len(layers) != 2 or config["engram_max_ngram_size"] != 4 or \
config["engram_n_heads"] != 8 or config["engram_head_dim"] != 256:
raise ValueError("unsupported Engram shape")
seen, primes, multipliers = set(), [], []
bound = (np.iinfo(np.int64).max // vocab_size) // 2
for layer in layers:
row = []
for _ in range(config["engram_max_ngram_size"] - 1):
current = config["engram_vocab_size"] - 1
for _ in range(config["engram_n_heads"]):
current = int(nextprime(current))
while current in seen:
current = int(nextprime(current))
seen.add(current)
row.append(current)
primes.append(row)
rng = np.random.default_rng(10007 * layer)
multipliers.append((rng.integers(0, bound, size=4, dtype=np.int64) * 2 + 1).tolist())
if [sum(row) for row in primes] != config["engram_num_embeddings"]:
raise ValueError("Engram table size differs from hash bucket layout")
return dict(token_map=token_map, compressed_vocab_size=vocab_size,
pad_id=token_map[config["engram_pad_token_id"]],
rows=config["engram_num_embeddings"], layers=layers,
primes=primes, multipliers=multipliers)
def metadata(hf_dir, revision):
from tokenizers import Tokenizer
with open(os.path.join(hf_dir, "config.json"), "rb") as fp:
config = json.load(fp)
if config["model_type"] != "deepseek_v41":
raise ValueError("not a DeepSeek V4.1 source checkpoint")
text = config["text_config"]
tokenizer = Tokenizer.from_file(os.path.join(hf_dir, "tokenizer.json"))
layout = engram_layout(text, tokenizer)
with open(os.path.join(hf_dir, "tokenizer.json"), "rb") as fp:
tok = json.load(fp)
if tok["model"]["type"] != "BPE":
raise ValueError("unsupported tokenizer")
# This tokenizer lists special tokens in both the BPE vocabulary and
# added_tokens. Its parsed ID table resolves that intentional overlap.
tokens = [tokenizer.id_to_token(i) for i in range(text["vocab_size"])]
if any(token is None for token in tokens):
raise ValueError("tokenizer has missing IDs")
special = {item["id"] for item in tok["added_tokens"] if item["special"]}
merges = [" ".join(pair) if isinstance(pair, list) else pair for pair in tok["model"]["merges"]]
records = [
kv_string("general.architecture", "deepseek41"),
kv_string("general.name", "DeepSeek V4.1 Flash"),
kv_string("general.source.url", "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash"),
kv_string("general.source.revision", revision),
kv_u32("general.alignment", GGUF_ALIGNMENT),
kv_string("deepseek41.config", json.dumps(config, sort_keys=True, separators=(",", ":"))),
kv_string("deepseek41.engram.encoding", "e4m3_e8m0_32_row264"),
kv_u32_array("deepseek41.engram.layer_ids", layout["layers"]),
kv_u32_array("deepseek41.engram.rows", layout["rows"]),
kv_u32("deepseek41.engram.compressed_vocab_size", layout["compressed_vocab_size"]),
kv_u32("deepseek41.engram.pad_id", layout["pad_id"]),
kv_u32_array("deepseek41.engram.token_map", layout["token_map"]),
kv_u32_array("deepseek41.engram.primes", sum(layout["primes"], [])),
array_record("deepseek41.engram.multipliers", GGUF_UINT64, sum(layout["multipliers"], [])),
kv_string("tokenizer.ggml.model", "gpt2"),
kv_string("tokenizer.ggml.pre", "joyai-llm"),
array_record("tokenizer.ggml.tokens", GGUF_STRING, tokens),
array_record("tokenizer.ggml.token_type", 5, [3 if i in special else 1 for i in range(len(tokens))]),
array_record("tokenizer.ggml.merges", GGUF_STRING, merges),
kv_u32("tokenizer.ggml.bos_token_id", config["bos_token_id"]),
kv_u32("tokenizer.ggml.eos_token_id", config["eos_token_id"]),
kv_bool("tokenizer.ggml.add_bos_token", True),
kv_bool("tokenizer.ggml.add_eos_token", False),
]
for key in (
"vocab_size", "hidden_size", "moe_intermediate_size", "num_hidden_layers",
"num_attention_heads", "num_key_value_heads", "head_dim", "qk_rope_head_dim",
"q_lora_rank", "o_lora_rank", "o_groups", "n_routed_experts", "n_shared_experts",
"num_experts_per_tok", "max_position_embeddings", "sliding_window",
"index_n_heads", "index_head_dim", "index_topk", "candidate_source_layer_id",
"candidate_topk_blocks", "candidate_block_size", "hc_mult", "hc_sinkhorn_iters",
"rope_theta", "compress_rope_theta",
):
records.append(kv_u32(f"deepseek41.{key}", text[key]))
for key in ("rms_norm_eps", "hc_eps", "swiglu_limit", "routed_scaling_factor"):
records.append(kv_f32(f"deepseek41.{key}", text[key]))
for key in ("scoring_func", "hidden_act", "topk_method"):
records.append(kv_string(f"deepseek41.{key}", text[key]))
records.append(kv_bool("deepseek41.norm_topk_prob", text["norm_topk_prob"]))
records.append(kv_u32_array("deepseek41.compress_ratios", text["compress_ratios"][:text["num_hidden_layers"]]))
for key in ("kv_source_layer_ids", "index_source_layer_ids"):
records.append(kv_u32_array(f"deepseek41.{key}", text[key]))
for key in ("factor", "beta_fast", "beta_slow", "original_max_position_embeddings"):
records.append(kv_f32(f"deepseek41.rope_scaling.{key}", text["rope_scaling"][key]))
return config, records
gguf-tools/deepseek41_quantize.py
+357
View File
@@ -0,0 +1,357 @@
#!/usr/bin/env python3
"""Convert DeepSeek V4.1 Flash to DwarfStar's Q2 or Q4_K GGUF.
Uses the project's C quantizers. Engram FP8 rows and their scales are packed
losslessly at the end of the file, outside the main model's resident extent.
Vision and DSpark are separate models and are not part of this text GGUF.
"""
import argparse
import concurrent.futures
import dataclasses
import hashlib
import json
import os
import re
import shutil
import struct
import sys
import time
from deepseek41_metadata import GGUF_ALIGNMENT, metadata
from glm53_quantize import (
SourceDB, TensorPlan, Quantizer, Imatrix, QTYPE_F32, QTYPE_F16,
QTYPE_Q8_0, QTYPE_Q2_K, QTYPE_Q4_K, QTYPE_IQ2_XXS, QTYPE_I8, QTYPE_MXFP4, align,
conversion_signature, kv_string, load_resume_state, print_plan,
qtype_nbytes, save_resume_state, tensor_header,
)
QUANTIZATION = {
"q2": "IQ2_XXS gate/up; Q2_K down; Q8_0 attention/shared/head",
"q4": "Q4_K gate/up/down; Q8_0 attention/shared/head",
"mxfp4": "released MXFP4 gate/up/down; Q8_0 attention/shared/head",
}
def scale_name(name):
return name.removesuffix(".weight") + ".scale"
def validate_scales(tensors):
for name, info in tensors.items():
if not name.startswith(("layers.", "embed.", "head.", "norm.")):
continue
dtype, shape = info["dtype"], info["shape"]
if dtype not in ("F8_E4M3", "I8") or not name.endswith(".weight"):
continue
if len(shape) != 2:
raise ValueError(f"{name}: expected a matrix")
if dtype == "I8":
expected = [shape[0], shape[1] // 16]
elif ".engram.embed." in name:
expected = [shape[0], shape[1] // 32]
else:
expected = [(dim + 31) // 32 for dim in shape]
scale = tensors.get(scale_name(name))
if not scale or scale["dtype"] != "F8_E8M0" or scale["shape"] != expected:
raise ValueError(f"{name}: expected E8M0 scales {expected}")
def build_plan(db, config, quant="q2"):
if quant not in QUANTIZATION:
raise ValueError(f"unknown quantization recipe: {quant}")
c = config["text_config"]
if config["quantization_config"]["weight_block_size"] != [32, 32]:
raise ValueError("expected native 32x32 FP8 blocks")
dim, inter = c["hidden_size"], c["moe_intermediate_size"]
heads, hd = c["num_attention_heads"], c["head_dim"]
qrank, orank, groups = c["q_lora_rank"], c["o_lora_rank"], c["o_groups"]
hc, experts, vocab = c["hc_mult"], c["n_routed_experts"], c["vocab_size"]
ih, idim = c["index_n_heads"], c["index_head_dim"]
plan, disk, consumed = [], [], set()
def claim(name, expected, dtype=None):
info = db.info(name)
if info["shape"] != list(expected) or (dtype and info["dtype"] != dtype):
raise ValueError(f"{name}: unexpected {info['dtype']} {info['shape']}, expected {dtype} {expected}")
consumed.add(name)
if info["dtype"] in ("I8", "F8_E4M3"):
consumed.add(scale_name(name))
return info
def regular(name, source, shape, qtype, role):
claim(source, shape)
plan.append(TensorPlan(name, tuple(reversed(shape)), qtype, role, source=source))
regular("token_embd.weight", "embed.weight", (vocab, dim), QTYPE_F16, "embedding")
regular("output_norm.weight", "norm.weight", (dim,), QTYPE_F32, "norm")
regular("output.weight", "head.weight", (vocab, dim), QTYPE_Q8_0, "output")
for layer in range(c["num_hidden_layers"]):
src, dst = f"layers.{layer}", f"blk.{layer}"
for site in ("attn", "ffn"):
for part, shape, qt in (("fn", (hc * (hc + 2), hc * dim), QTYPE_F16),
("base", (hc * (hc + 2),), QTYPE_F32),
("scale", (3,), QTYPE_F32)):
regular(f"{dst}.hc_{site}_{part}.weight", f"{src}.hc_{site}_{part}", shape, qt, "mhc")
regular(f"{dst}.{site}_norm.weight", f"{src}.{site}_norm.weight", (dim,), QTYPE_F32, "norm")
for target, source, shape, qt in (
("attn_sinks.weight", "attn_sink", (heads,), QTYPE_F32),
("attn_q_a.weight", "wq_a.weight", (qrank, dim), QTYPE_Q8_0),
("attn_q_b.weight", "wq_b.weight", (heads * hd, qrank), QTYPE_Q8_0),
("attn_q_a_norm.weight", "q_norm.weight", (qrank,), QTYPE_F32),
("attn_kv.weight", "wkv.weight", (hd, dim), QTYPE_Q8_0),
("attn_kv_a_norm.weight", "kv_norm.weight", (hd,), QTYPE_F32),
("attn_output_a.weight", "wo_a.weight", (groups * orank, heads * hd // groups), QTYPE_Q8_0),
("attn_output_b.weight", "wo_b.weight", (dim, groups * orank), QTYPE_Q8_0),
):
regular(f"{dst}.{target}", f"{src}.attn.{source}", shape, qt, "attention")
if layer in c["kv_source_layer_ids"]:
for target, source, shape, qt in (
("attn_compressor_kv.weight", "compressor.wkv.weight", (hd, dim), QTYPE_F16),
("attn_compressor_norm.weight", "compressor.norm.weight", (hd,), QTYPE_F32),
("indexer.attn_k.weight", "indexer.wk.weight", (idim, hd), QTYPE_F16),
("indexer.k_norm.weight", "indexer.k_norm.weight", (idim,), QTYPE_F32),
):
regular(f"{dst}.{target}", f"{src}.attn.{source}", shape, qt, "compressor")
if c["compress_ratios"][layer] > 1:
regular(f"{dst}.attn_compressor_gate.weight", f"{src}.attn.compressor.wgate.weight",
(hd, dim), QTYPE_F16, "compressor")
if layer in c["index_source_layer_ids"]:
regular(f"{dst}.indexer.attn_q_b.weight", f"{src}.attn.indexer.wq_b.weight",
(ih * idim, qrank), QTYPE_F16, "indexer")
regular(f"{dst}.indexer.proj.weight", f"{src}.attn.indexer.weights_proj.weight",
(ih, dim), QTYPE_F16, "indexer")
regular(f"{dst}.ffn_gate_inp.weight", f"{src}.ffn.gate.weight", (experts, dim), QTYPE_F32, "router")
for suffix, target in (("bias", "exp_probs_b.bias"), ("bias_vl", "exp_probs_b_vl.bias")):
regular(f"{dst}.{target}", f"{src}.ffn.gate.{suffix}", (experts,), QTYPE_F32, "router")
for part, source, shape, qt in (("gate", "w1", (inter, dim), QTYPE_IQ2_XXS),
("up", "w3", (inter, dim), QTYPE_IQ2_XXS),
("down", "w2", (dim, inter), QTYPE_Q2_K)):
regular(f"{dst}.ffn_{part}_shexp.weight", f"{src}.ffn.shared_experts.{source}.weight",
shape, QTYPE_Q8_0, "shared")
pattern = f"{src}.ffn.experts.{{expert}}.{source}.weight"
for expert in range(experts):
claim(pattern.format(expert=expert), (shape[0], shape[1] // 2), "I8")
plan.append(TensorPlan(f"{dst}.ffn_{part}_exps.weight", (*reversed(shape), experts),
QTYPE_MXFP4 if quant == "mxfp4" else QTYPE_Q4_K if quant == "q4" else qt,
"experts", source=pattern, expert_layer=layer,
expert_part=part, expert_count=experts))
if layer in c["engram_layer_ids"]:
index = c["engram_layer_ids"].index(layer)
rows = c["engram_num_embeddings"][index]
engram = f"{src}.engram"
claim(f"{engram}.embed.weight", (rows, 256), "F8_E4M3")
claim(f"{engram}.embed.scale", (rows, 8), "F8_E8M0")
disk.append(TensorPlan(f"{dst}.engram_embd.weight", (264, rows), QTYPE_I8,
"engram_disk", source=f"{engram}.embed.weight", raw_copy=True))
for part in ("q", "k"):
regular(f"{dst}.engram_{part}_norm.weight", f"{engram}.{part}_weight", (hc, dim), QTYPE_F32, "engram")
regular(f"{dst}.engram_kv.weight", f"{engram}.wkv.weight", ((hc + 1) * dim, 24 * 256), QTYPE_F16, "engram")
omitted = {name for name in db.tensors if name.startswith(("mtp.", "vision.", "aligner.", "image_"))}
if consumed | omitted != set(db.tensors):
raise ValueError(f"unclaimed source tensors: {sorted(set(db.tensors) - consumed - omitted)[:10]}")
offset = 0
for item in plan + disk:
item.offset = offset
item.nbytes = qtype_nbytes(item.qtype, item.shape)
offset += align(item.nbytes, GGUF_ALIGNMENT)
return plan + disk
class NativeQuantizer(Quantizer):
def to_f32(self, db, name, row_start=0, row_count=None):
if ".engram.embed." in name:
raise ValueError("Engram must never be materialized as a whole float tensor")
np = self.np
info = db.info(name)
if info["dtype"] not in ("I8", "F8_E4M3"):
result = super().to_f32(db, name, row_start, row_count)
else:
shape = info["shape"]
codes = np.frombuffer(db.read(name), dtype=np.uint8).reshape(shape)
sinfo = db.info(scale_name(name))
scales = np.frombuffer(db.read(scale_name(name)), dtype=np.uint8).reshape(sinfo["shape"])
if np.any(scales == 255):
raise ValueError(f"{name}: nonfinite scale")
scales = np.ldexp(np.ones_like(scales, dtype=np.float32), scales.astype(np.int32) - 127)
if info["dtype"] == "I8":
lut = np.array([0, .5, 1, 1.5, 2, 3, 4, 6, -0., -.5, -1, -1.5, -2, -3, -4, -6], np.float32)
result = np.empty((shape[0], shape[1] * 2), dtype=np.float32)
result[:, 0::2] = lut[codes & 15]
result[:, 1::2] = lut[codes >> 4]
result = (result.reshape(shape[0], -1, 32) * scales[:, :, None]).reshape(result.shape)
else:
if np.any((codes & 127) == 127):
raise ValueError(f"{name}: nonfinite FP8 weight")
result = self.fp8_lut[codes]
for row in range(0, shape[0], 32):
for col in range(0, shape[1], 32):
result[row:row + 32, col:col + 32] *= scales[row // 32, col // 32]
if row_count is not None:
result = result[row_start:row_start + row_count]
if not np.all(np.isfinite(result)):
raise ValueError(f"{name}: nonfinite dequantized weight")
return np.ascontiguousarray(result, dtype=np.float32)
def to_mxfp4(self, db, name):
"""Repack released E2M1 pairs into GGUF MXFP4 blocks without touching the values."""
np = self.np
info = db.info(name)
if info["dtype"] != "I8" or info["shape"][1] % 16:
raise ValueError(f"{name}: not a packed MXFP4 tensor")
rows, blocks = info["shape"][0], info["shape"][1] // 16
codes = np.frombuffer(db.read(name), dtype=np.uint8).reshape(rows, blocks, 16)
scales = np.frombuffer(db.read(scale_name(name)), dtype=np.uint8).reshape(rows, blocks)
if np.any(scales == 255):
raise ValueError(f"{name}: nonfinite scale")
values = np.empty((rows, blocks, 32), dtype=np.uint8)
values[:, :, 0::2] = codes & 15
values[:, :, 1::2] = codes >> 4
packed = np.empty((rows, blocks, 17), dtype=np.uint8)
packed[:, :, 0] = scales
packed[:, :, 1:] = values[:, :, :16] | (values[:, :, 16:] << 4)
return packed.tobytes()
def encode(self, array, qtype, imatrix=None):
if qtype == QTYPE_F16 and self.np.any(self.np.abs(array) > 65504):
raise ValueError("F16 tensor would overflow; preserve this family in BF16/F32")
return super().encode(array, qtype, imatrix)
def write_engram(fp, item, db, np):
# Keep weights and scales together without changing either byte. A chunk
# holds 16K rows (4 MiB of weights), independent of the 94 GiB table size.
rows = item.shape[1]
for start in range(0, rows, 16384):
count = min(16384, rows - start)
weights = b"".join(db.iter_read(item.source, start * 256, count * 256))
scales = b"".join(db.iter_read(scale_name(item.source), start * 8, count * 8))
w = np.frombuffer(weights, dtype=np.uint8).reshape(count, 256)
s = np.frombuffer(scales, dtype=np.uint8).reshape(count, 8)
if np.any((w & 127) == 127) or np.any(s == 255):
raise ValueError(f"{item.name}: nonfinite Engram row near {start}")
packed = np.empty((count, 264), dtype=np.uint8)
packed[:, :256], packed[:, 256:] = w, s
fp.write(packed.tobytes())
def write_gguf(args, plan, records, db):
quantizer = NativeQuantizer(args.quants_library)
imatrix = Imatrix(args.imatrix, quantizer.np)
if args.imatrix:
for item in plan:
if item.is_expert and item.name not in imatrix.entries:
raise ValueError(f"missing imatrix tensor {item.name}")
if any(quantizer.np.any(values < 0) for values in imatrix.entries.values()):
raise ValueError("negative importance values")
data_start, data_bytes = print_plan(plan, records, [], GGUF_ALIGNMENT)
partial, journal = args.out + ".partial", args.out + ".partial.json"
signature = conversion_signature(plan, records, [], args.imatrix)
# A metadata/recipe match alone cannot distinguish two source downloads.
source_identity = [(name, db.info(name)) for name in sorted(db.tensors)]
signature = hashlib.sha256((signature + json.dumps(source_identity, sort_keys=True)).encode()).hexdigest()
if os.path.exists(args.out):
raise ValueError(f"refusing to overwrite {args.out}")
completed = 0
if os.path.exists(partial) or os.path.exists(journal):
if not args.resume or not (os.path.exists(partial) and os.path.exists(journal)):
raise ValueError("partial file and journal require --resume")
completed = load_resume_state(journal, signature, plan)
end = data_start + (plan[completed - 1].offset +
align(plan[completed - 1].nbytes, GGUF_ALIGNMENT) if completed else 0)
free = shutil.disk_usage(os.path.dirname(os.path.abspath(args.out))).free
if free < data_start + data_bytes - end + (32 << 30):
raise ValueError("insufficient disk space for remaining output plus 32 GiB reserve")
header = b"GGUF" + struct.pack("<IQQ", 3, len(plan), len(records))
header += b"".join(records) + b"".join(tensor_header(item) for item in plan)
header += bytes(data_start - len(header))
if os.path.exists(partial):
with open(partial, "rb") as fp:
if fp.read(data_start) != header or os.fstat(fp.fileno()).st_size < end:
raise ValueError("partial GGUF is truncated or has a different header")
else:
with open(partial, "xb") as fp:
fp.write(header)
fp.flush()
os.fsync(fp.fileno())
save_resume_state(journal, signature, 0)
with open(partial, "r+b") as fp, concurrent.futures.ThreadPoolExecutor(max_workers=args.threads) as pool:
fp.truncate(end)
fp.seek(end)
for index in range(completed, len(plan)):
item = plan[index]
started = time.monotonic()
missing = 0
if fp.tell() != data_start + item.offset:
raise ValueError(f"incorrect offset for {item.name}")
if item.role == "engram_disk":
write_engram(fp, item, db, quantizer.np)
elif item.is_expert:
def convert(expert):
if item.qtype == QTYPE_MXFP4:
return quantizer.to_mxfp4(db, item.source.format(expert=expert)), False
values = quantizer.to_f32(db, item.source.format(expert=expert))
importance = imatrix.expert(item.name, expert, item.shape[0], item.expert_count)
return quantizer.encode(values, item.qtype, importance), importance is None
for start in range(0, item.expert_count, args.threads):
futures = [pool.submit(convert, e) for e in range(start, min(start + args.threads, item.expert_count))]
for future in futures:
data, fallback = future.result()
if len(data) != item.nbytes // item.expert_count:
raise ValueError("wrong encoded expert size")
fp.write(data)
missing += int(fallback)
else:
fp.write(quantizer.encode(quantizer.to_f32(db, item.source), item.qtype))
if fp.tell() != data_start + item.offset + item.nbytes:
raise ValueError(f"incorrect payload size for {item.name}")
fp.write(bytes(align(item.nbytes, GGUF_ALIGNMENT) - item.nbytes))
fp.flush()
os.fsync(fp.fileno())
save_resume_state(journal, signature, index + 1)
print(f"[{index + 1}/{len(plan)}] {item.name}: {item.nbytes / (1 << 30):.3f} GiB, "
f"{time.monotonic() - started:.1f}s, uncalibrated_experts={missing}", flush=True)
os.rename(partial, args.out)
os.unlink(journal)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--hf", required=True)
parser.add_argument("--out", required=True)
parser.add_argument("--source-revision", required=True)
parser.add_argument("--quant", choices=QUANTIZATION, default="q2")
parser.add_argument("--imatrix")
parser.add_argument("--threads", type=int, default=8)
parser.add_argument("--resume", action="store_true")
parser.add_argument("--dry-run", action="store_true")
suffix = "dylib" if sys.platform == "darwin" else "so"
parser.add_argument("--quants-library", default=os.path.join(os.path.dirname(__file__), f"libds4quants.{suffix}"))
args = parser.parse_args()
if not re.fullmatch(r"[0-9a-f]{40}", args.source_revision):
parser.error("source revision must be a full commit hash")
if not 1 <= args.threads <= 32:
parser.error("threads must be between 1 and 32")
config, records = metadata(args.hf, args.source_revision)
db = SourceDB(args.hf, index_validator=lambda _: None, scale_validator=validate_scales)
try:
plan = build_plan(db, config, args.quant)
records.append(kv_string("deepseek41.quantization", QUANTIZATION[args.quant]))
records.append(kv_string("deepseek41.calibration", "imatrix" if args.imatrix else "weight-energy bootstrap"))
if args.imatrix:
records.append(kv_string("quantize.imatrix.file", os.path.basename(args.imatrix)))
if args.dry_run:
print_plan(plan, records, [], GGUF_ALIGNMENT)
for item in plan:
print(json.dumps(dataclasses.asdict(item), sort_keys=True))
else:
write_gguf(args, plan, records, db)
finally:
db.close()
if __name__ == "__main__":
try:
main()
except (OSError, ValueError) as error:
sys.exit(f"deepseek41-quantize: {error}")
gguf-tools/deepseek41_validate_gguf.py
+125
View File
@@ -0,0 +1,125 @@
#!/usr/bin/env python3
"""Audit a completed V4.1 GGUF and sample its payloads against the source.
Checks the complete tensor layout, then re-encodes selected expert tensors and
samples native Engram rows. This is artifact validation, not inference QA.
"""
import argparse
import json
import os
from pathlib import Path
import random
import sys
from deepseek41_metadata import GGUF_ALIGNMENT
from deepseek41_quantize import (SourceDB, NativeQuantizer, Imatrix, build_plan,
validate_scales, scale_name, QUANTIZATION, QTYPE_MXFP4)
from glm53_quantize import (align, read_exact, read_u32, read_u64,
read_gguf_string, skip_gguf_value)
from glm53_validate_gguf import read_selected_metadata
def check_payload(fp, offset, item, db, quantizer, imatrix):
if item.role == "engram_disk":
rows = item.shape[1]
rng = random.Random(41 + rows)
selected = {0, 1, rows - 1, 16383, 16384, (1 << 32) // 264}
selected.update(rng.randrange(rows) for _ in range(128))
for row in sorted(selected):
if row >= rows:
continue
expected = b"".join(db.iter_read(item.source, row * 256, 256))
expected += b"".join(db.iter_read(scale_name(item.source), row * 8, 8))
fp.seek(offset + row * 264)
if read_exact(fp, 264, item.name) != expected:
raise ValueError(f"{item.name}: Engram row {row} differs from source")
elif item.is_expert:
selected = {0, (item.expert_layer * 17 + 41) % item.expert_count, item.expert_count - 1}
stride = item.nbytes // item.expert_count
for expert in sorted(selected):
if item.qtype == QTYPE_MXFP4:
expected = quantizer.to_mxfp4(db, item.source.format(expert=expert))
else:
values = quantizer.to_f32(db, item.source.format(expert=expert))
importance = imatrix.expert(item.name, expert, item.shape[0], item.expert_count)
expected = quantizer.encode(values, item.qtype, importance)
fp.seek(offset + expert * stride)
if len(expected) != stride or read_exact(fp, stride, item.name) != expected:
raise ValueError(f"{item.name}: encoded expert {expert} differs from source recipe")
else:
values = quantizer.to_f32(db, item.source)
expected = quantizer.encode(values, item.qtype)
fp.seek(offset)
if len(expected) != item.nbytes or read_exact(fp, item.nbytes, item.name) != expected:
raise ValueError(f"{item.name}: payload differs from source recipe")
def validate(args):
config = json.loads((Path(args.hf) / "config.json").read_text())
db = SourceDB(args.hf, index_validator=lambda _: None, scale_validator=validate_scales)
try:
plan = build_plan(db, config, args.quant)
with open(args.gguf, "rb") as fp:
if read_exact(fp, 4, "magic") != b"GGUF" or read_u32(fp, "version") != 3:
raise ValueError("expected GGUF v3")
if read_u64(fp, "tensor count") != len(plan):
raise ValueError("tensor count differs from source plan")
metadata = {}
for _ in range(read_u64(fp, "metadata count")):
key = read_gguf_string(fp, "metadata key")
kind = read_u32(fp, "metadata type")
if key in {"general.architecture", "general.alignment", "general.source.revision",
"deepseek41.calibration", "deepseek41.quantization"}:
if key in metadata:
raise ValueError(f"duplicate metadata: {key}")
metadata[key] = read_selected_metadata(fp, kind)
else:
skip_gguf_value(fp, kind)
expected = {"general.architecture": "deepseek41", "general.alignment": GGUF_ALIGNMENT,
"general.source.revision": args.source_revision,
"deepseek41.quantization": QUANTIZATION[args.quant],
"deepseek41.calibration": "imatrix" if args.imatrix else "weight-energy bootstrap"}
if metadata != expected:
raise ValueError(f"metadata mismatch: {metadata}")
for item in plan:
name = read_gguf_string(fp, "tensor name")
rank = read_u32(fp, "tensor rank")
if rank != len(item.shape):
raise ValueError(f"{name}: unexpected rank {rank}")
shape = tuple(read_u64(fp, "dimension") for _ in range(rank))
kind, offset = read_u32(fp, "tensor type"), read_u64(fp, "tensor offset")
if (name, shape, kind, offset) != (item.name, item.shape, item.qtype, item.offset):
raise ValueError(f"{item.name}: tensor layout mismatch")
start = align(fp.tell(), GGUF_ALIGNMENT)
size = start + plan[-1].offset + align(plan[-1].nbytes, GGUF_ALIGNMENT)
if os.fstat(fp.fileno()).st_size != size:
raise ValueError(f"expected file size {size}")
print(f"PASS: {len(plan)} tensor layouts; complete {size}-byte GGUF", flush=True)
if args.payload:
q = NativeQuantizer(args.quants_library)
imatrix = Imatrix(args.imatrix, q.np)
if args.imatrix and any(t.is_expert and t.name not in imatrix.entries for t in plan):
raise ValueError("imatrix is missing expert tensors")
for index, item in enumerate(plan):
check_payload(fp, start + item.offset, item, db, q, imatrix)
if (index + 1) % 25 == 0 or index + 1 == len(plan):
print(f"PASS: source payload checks {index + 1}/{len(plan)}", flush=True)
finally:
db.close()
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--hf", required=True)
parser.add_argument("--gguf", required=True)
parser.add_argument("--source-revision", required=True)
parser.add_argument("--quant", choices=QUANTIZATION, default="q2")
parser.add_argument("--imatrix")
parser.add_argument("--payload", action="store_true")
suffix = "dylib" if sys.platform == "darwin" else "so"
parser.add_argument("--quants-library", default=str(Path(__file__).with_name(f"libds4quants.{suffix}")))
args = parser.parse_args()
try:
validate(args)
except (OSError, ValueError) as error:
sys.exit(f"deepseek41-validate: {error}")
gguf-tools/deepseek41_vision.py
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env python3
"""Copy the released DeepSeek V4.1 Flash vision encoder into a separate GGUF.
All 266 tensors remain BF16, byte for byte. Visual router biases are already
part of the language GGUF produced by deepseek41_quantize.py.
"""
import argparse
import json
from pathlib import Path
import sys
from deepseek4_vision import create_gguf, print_summary, validate_gguf
from glm53_quantize import (
GGUF_ALIGNMENT, QTYPE_BF16, SourceDB, TensorPlan, align,
kv_f32, kv_string, kv_u32, qtype_nbytes,
)
SOURCE_REVISION = "df42c109f1defefcbfcedbe7d905718a12266e40"
SOURCE_URL = "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash"
def source_shapes():
shapes = {
"vision.patch_embed.proj.weight": (1024, 588),
"vision.patch_embed.proj.bias": (1024,),
"vision.norm.weight": (1024,),
"aligner.w1.weight": (5120, 9216),
"aligner.w1.bias": (5120,),
"aligner.w2.weight": (5120, 5120),
"aligner.w2.bias": (5120,),
"image_start": (5120,),
"image_newline": (5120,),
"image_end": (5120,),
}
for layer in range(32):
for suffix, shape in {
"norm1.weight": (1024,), "norm2.weight": (1024,),
"attn.wqkv.weight": (3072, 1024), "attn.wqkv.bias": (3072,),
"attn.wo.weight": (1024, 1024), "attn.wo.bias": (1024,),
"mlp.w1.weight": (5632, 1024), "mlp.w2.weight": (1024, 2816),
}.items():
shapes[f"vision.blocks.{layer}.{suffix}"] = shape
return shapes
def validate_inventory(tensors):
selected = {name for name in tensors if name.startswith(("vision.", "aligner.", "image_"))}
expected = set(source_shapes())
if selected != expected:
raise ValueError(f"vision inventory mismatch: missing {sorted(expected - selected)}, "
f"unexpected {sorted(selected - expected)}")
def build_plan(db):
validate_inventory(db.tensors)
plan, offset = [], 0
for name, shape in sorted(source_shapes().items()):
info = db.info(name)
if info["dtype"] != "BF16" or tuple(info["shape"]) != shape:
raise ValueError(f"{name}: expected BF16 {shape}, found {info['dtype']} {info['shape']}")
role = "vision" if name.startswith("vision.") else (
"aligner" if name.startswith("aligner.") else "image_embedding")
item = TensorPlan(name, tuple(reversed(shape)), QTYPE_BF16, role,
source=name, raw_copy=True)
item.nbytes = qtype_nbytes(item.qtype, item.shape)
if item.nbytes != info["nbytes"]:
raise ValueError(f"{name}: source byte size mismatch")
item.offset = offset
offset += align(item.nbytes)
plan.append(item)
return plan
def sidecar_metadata(hf_dir):
config = json.loads((Path(hf_dir) / "config.json").read_text())
if config.get("model_type") != "deepseek_v41" or config.get("image_token_id") != 129264:
raise ValueError("expected the DeepSeek V4.1 Flash checkpoint")
for section, expected in {
"text_config": {"hidden_size": 5120, "num_hidden_layers": 40, "n_routed_experts": 384},
"vision_config": {"num_hidden_layers": 32, "hidden_size": 1024,
"num_attention_heads": 16, "intermediate_size": 2816, "patch_size": 14,
"rope_theta": 10000, "downsample_ratio": 3, "max_image_tokens": 1024,
"min_pixels": 295936, "max_wh_ratio": None},
}.items():
for key, value in expected.items():
if key not in config[section] or config[section][key] != value:
raise ValueError(f"unexpected {section}.{key}: {config[section].get(key)!r}")
prefix = "deepseek4-vision"
records = [
kv_string("general.architecture", prefix),
kv_string("general.name", "DeepSeek V4.1 Flash Vision Encoder"),
kv_u32("general.alignment", GGUF_ALIGNMENT),
kv_string("general.source.url", SOURCE_URL),
kv_string("general.source.revision", SOURCE_REVISION),
kv_string(f"{prefix}.checkpoint_variant", "v4.1-flash"),
kv_f32(f"{prefix}.rope.freq_base", 10000),
kv_f32(f"{prefix}.attention.layer_norm_rms_epsilon", 1e-6),
]
for key, value in {
"block_count": 32, "embedding_length": 1024, "feed_forward_length": 2816,
"attention.head_count": 16, "projection_length": 5120, "patch_size": 14,
"downsample_ratio": 3, "image.max_tokens": 1024, "image.min_pixels": 295936,
"image.max_width_height_ratio": 0, "image.token_id": 129264,
"language.block_count": 40, "language.expert_count": 384,
}.items():
records.append(kv_u32(f"{prefix}.{key}", value))
return records
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--hf", required=True, help="official V4.1 Flash snapshot")
output = parser.add_mutually_exclusive_group(required=True)
output.add_argument("--out", help="write this vision GGUF")
output.add_argument("--validate", metavar="GGUF")
output.add_argument("--dry-run", action="store_true")
parser.add_argument("--verify-payload", action="store_true")
parser.add_argument("--expected-sha256")
args = parser.parse_args()
if (args.verify_payload or args.expected_sha256) and not args.validate:
parser.error("--verify-payload and --expected-sha256 require --validate")
records = sidecar_metadata(args.hf)
db = SourceDB(args.hf, validate_inventory, lambda _: None)
try:
plan = build_plan(db)
if args.out:
create_gguf(args.out, plan, records, db, overwrite=False)
elif args.validate:
validate_gguf(args.validate, plan, records, db,
args.verify_payload, args.expected_sha256)
else:
print_summary(plan, records)
finally:
db.close()
if __name__ == "__main__":
try:
main()
except (KeyError, OSError, ValueError) as error:
print(f"deepseek41-vision: {error}", file=sys.stderr)
raise SystemExit(1)
gguf-tools/deepseek4_vision.py
+415
View File
@@ -0,0 +1,415 @@
#!/usr/bin/env python3
"""Create or validate the standalone DeepSeek V4 Flash Vision sidecar GGUF."""
from __future__ import annotations
import argparse
import hashlib
import io
import json
import os
import re
import shutil
import struct
import sys
from glm53_quantize import (
GGUF_ALIGNMENT,
GGUF_FLOAT32,
GGUF_STRING,
GGUF_UINT32,
GGUF_VERSION,
QTYPE_BF16,
QTYPE_F32,
QTYPE_NAMES,
SourceDB,
TensorPlan,
align,
fail,
kv_f32,
kv_string,
kv_u32,
pack_string,
qtype_nbytes,
read_exact,
read_gguf_string,
read_u32,
read_u64,
skip_gguf_value,
tensor_header,
)
SOURCE_URL = "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"
SOURCE_REVISION = "e46e16bf6035c6f317eb2ac7458eb0362926d402"
ARCHITECTURE = "deepseek4-vision"
ROUTER_RE = re.compile(r"^(layers\.(\d+)|mtp\.(\d+))\.ffn\.gate\.bias_vl$")
HASH_BIAS_RE = re.compile(r"^layers\.([0-2])\.ffn\.gate\.bias$")
def validate_deepseek4_index(weight_map):
if len(weight_map) != 72633:
fail(f"expected 72,633 source tensors, found {len(weight_map)}")
shards = set(weight_map.values())
expected_shards = {
f"model-{index:05d}-of-00048.safetensors"
for index in range(1, 49)
}
if shards != expected_shards:
fail("source shard inventory differs from the pinned checkpoint")
selected = {name for name in weight_map if selected_role(name)}
validate_inventory(selected)
if "embed.weight" not in weight_map or "head.weight" not in weight_map:
fail("source language tensor inventory is incomplete")
def validate_deepseek4_fp8_scales(tensors):
fp8 = {name: info for name, info in tensors.items()
if info["dtype"] == "F8_E4M3"}
if len(fp8) != 390:
fail(f"expected 390 FP8 language tensors, found {len(fp8)}")
for name, info in fp8.items():
if not name.endswith(".weight") or len(info["shape"]) != 2:
fail(f"unexpected FP8 tensor layout: {name}")
scale_name = name.removesuffix(".weight") + ".scale"
scale = tensors.get(scale_name)
expected_shape = [
(info["shape"][0] + 127) // 128,
(info["shape"][1] + 127) // 128,
]
if not scale or scale["dtype"] != "F8_E8M0" or \
scale["shape"] != expected_shape:
fail(f"{name}: invalid native FP8 scale tensor {scale_name}")
def load_config(hf_dir):
with open(os.path.join(hf_dir, "config.json"), encoding="utf-8") as fp:
config = json.load(fp)
expected = {
"model_type": "deepseek_v4",
"hidden_size": 4096,
"n_routed_experts": 256,
"num_hidden_layers": 43,
"vision_n_layers": 32,
"vision_dim": 1024,
"vision_n_heads": 16,
"vision_inter_dim": 2816,
"vision_patch_size": 14,
"vision_rope_theta": 10000.0,
"vision_downsample_ratio": 3,
"vision_max_n_token": 384,
"vision_min_pixels": 147456,
"vision_max_wh_ratio": 8,
}
for key, value in expected.items():
if config.get(key) != value:
fail(f"unexpected config {key}: {config.get(key)!r}")
if config.get("rms_norm_eps") != 1e-20:
fail(f"unexpected language rms_norm_eps: {config.get('rms_norm_eps')!r}")
return config
def selected_role(name):
if name.startswith("vision."):
return "vision"
if name.startswith("aligner."):
return "aligner"
if name in {"image_start", "image_pad", "image_newline", "image_end"}:
return "image_embedding"
if ROUTER_RE.match(name):
return "visual_router_bias"
if HASH_BIAS_RE.match(name):
return "hash_router_bias"
return None
def validate_inventory(names):
names = set(names)
vision = {name for name in names if name.startswith("vision.")}
aligner = {name for name in names if name.startswith("aligner.")}
image = {name for name in names if name.startswith("image_")}
visual_bias = {name for name in names if ROUTER_RE.match(name)}
hash_bias = {name for name in names if HASH_BIAS_RE.match(name)}
if len(vision) != 259:
fail(f"expected 259 ViT tensors, found {len(vision)}")
if len(aligner) != 4:
fail(f"expected 4 aligner tensors, found {len(aligner)}")
if image != {"image_start", "image_pad", "image_newline", "image_end"}:
fail(f"unexpected image embedding inventory: {sorted(image)}")
expected_visual = {
*(f"layers.{layer}.ffn.gate.bias_vl" for layer in range(43)),
*(f"mtp.{stage}.ffn.gate.bias_vl" for stage in range(3)),
}
if visual_bias != expected_visual:
fail("visual router bias inventory differs from the pinned checkpoint")
expected_hash = {f"layers.{layer}.ffn.gate.bias" for layer in range(3)}
if hash_bias != expected_hash:
fail("hash router bias inventory differs from the pinned checkpoint")
if len(names) != 316:
fail(f"expected 316 sidecar tensors, found {len(names)}")
def build_plan(db):
selected = sorted(name for name in db.tensors if selected_role(name))
validate_inventory(selected)
plan = []
offset = 0
for name in selected:
info = db.info(name)
role = selected_role(name)
expected_dtype = "BF16" if role in {
"vision", "aligner", "image_embedding"
} else "F32"
if info["dtype"] != expected_dtype:
fail(f"{name}: expected {expected_dtype}, got {info['dtype']}")
qtype = QTYPE_BF16 if expected_dtype == "BF16" else QTYPE_F32
item = TensorPlan(
name=name,
shape=tuple(reversed(info["shape"])),
qtype=qtype,
role=role,
source=name,
raw_copy=True,
)
item.nbytes = qtype_nbytes(item.qtype, item.shape)
if item.nbytes != info["nbytes"]:
fail(f"{name}: planned {item.nbytes} bytes, source has {info['nbytes']}")
item.offset = offset
offset += align(item.nbytes)
plan.append(item)
return plan
def sidecar_metadata(hf_dir, source_revision):
config = load_config(hf_dir)
prefix = ARCHITECTURE
return [
kv_string("general.architecture", ARCHITECTURE),
kv_string("general.name", "DeepSeek V4 Flash Vision Encoder"),
kv_u32("general.alignment", GGUF_ALIGNMENT),
kv_string("general.source.url", SOURCE_URL),
kv_string("general.source.revision", source_revision),
kv_string(f"{prefix}.checkpoint_variant", "vision-exp"),
kv_u32(f"{prefix}.block_count", config["vision_n_layers"]),
kv_u32(f"{prefix}.embedding_length", config["vision_dim"]),
kv_u32(f"{prefix}.feed_forward_length", config["vision_inter_dim"]),
kv_u32(f"{prefix}.attention.head_count", config["vision_n_heads"]),
kv_u32(f"{prefix}.projection_length", config["hidden_size"]),
kv_u32(f"{prefix}.patch_size", config["vision_patch_size"]),
kv_f32(f"{prefix}.rope.freq_base", config["vision_rope_theta"]),
kv_f32(f"{prefix}.attention.layer_norm_rms_epsilon", 1e-6),
kv_u32(f"{prefix}.downsample_ratio", config["vision_downsample_ratio"]),
kv_u32(f"{prefix}.image.max_tokens", config["vision_max_n_token"]),
kv_u32(f"{prefix}.image.min_pixels", config["vision_min_pixels"]),
kv_u32(f"{prefix}.image.max_width_height_ratio", config["vision_max_wh_ratio"]),
kv_u32(f"{prefix}.language.block_count", config["num_hidden_layers"]),
kv_u32(f"{prefix}.language.expert_count", config["n_routed_experts"]),
]
def layout(plan, metadata):
header_bytes = 4 + 4 + 8 + 8
header_bytes += sum(len(record) for record in metadata)
header_bytes += sum(len(tensor_header(item)) for item in plan)
data_offset = align(header_bytes)
data_bytes = sum(align(item.nbytes) for item in plan)
return data_offset, data_bytes
def print_summary(plan, metadata):
data_offset, data_bytes = layout(plan, metadata)
print(f"tensors: {len(plan)}")
print(f"metadata_records: {len(metadata)}")
print(f"metadata_bytes: {data_offset}")
print(f"tensor_bytes: {sum(item.nbytes for item in plan)}")
print(f"file_bytes: {data_offset + data_bytes}")
roles = {}
for item in plan:
roles[item.role] = roles.get(item.role, 0) + item.nbytes
for role, size in sorted(roles.items()):
print(f"role_bytes: {role} {size}")
return data_offset, data_bytes
def create_gguf(path, plan, metadata, db, overwrite):
data_offset, data_bytes = print_summary(plan, metadata)
required = data_offset + data_bytes + 4 * (1 << 30)
free = shutil.disk_usage(os.path.dirname(os.path.abspath(path))).free
if free < required:
fail(f"insufficient free space: need output plus reserve {required}, have {free}")
if os.path.exists(path) and not overwrite:
fail(f"output exists: {path}; use --overwrite")
partial = path + ".partial"
if os.path.exists(partial):
if not overwrite:
fail(f"partial output exists: {partial}; use --overwrite")
os.unlink(partial)
with open(partial, "wb") as fp:
fp.write(b"GGUF")
fp.write(struct.pack("<IQQ", GGUF_VERSION, len(plan), len(metadata)))
for record in metadata:
fp.write(record)
for item in plan:
fp.write(tensor_header(item))
if fp.tell() > data_offset:
fail("GGUF header exceeds its planned data offset")
fp.write(bytes(data_offset - fp.tell()))
for index, item in enumerate(plan, 1):
if fp.tell() != data_offset + item.offset:
fail(f"{item.name}: output offset mismatch")
written = 0
for chunk in db.iter_read(item.source):
fp.write(chunk)
written += len(chunk)
if written != item.nbytes:
fail(f"{item.name}: copied {written} bytes, expected {item.nbytes}")
fp.write(bytes(align(item.nbytes) - item.nbytes))
if index % 25 == 0 or index == len(plan):
print(f"copied tensors: {index}/{len(plan)}", file=sys.stderr, flush=True)
fp.flush()
os.fsync(fp.fileno())
os.replace(partial, path)
def read_metadata_value(fp, value_type):
if value_type == GGUF_STRING:
return read_gguf_string(fp, "GGUF metadata string")
if value_type == GGUF_UINT32:
return read_u32(fp, "GGUF metadata uint32")
if value_type == GGUF_FLOAT32:
return struct.unpack("<f", read_exact(fp, 4, "GGUF metadata float32"))[0]
skip_gguf_value(fp, value_type)
return None
def file_sha256(path):
digest = hashlib.sha256()
with open(path, "rb") as fp:
while chunk := fp.read(16 << 20):
digest.update(chunk)
return digest.hexdigest()
def validate_gguf(path, plan, expected_metadata, db, verify_payload, expected_sha256):
expected_values = {}
for record in expected_metadata:
reader = io.BytesIO(record)
key = read_gguf_string(reader, "expected metadata key")
value_type = read_u32(reader, "expected metadata type")
expected_values[key] = read_metadata_value(reader, value_type)
with open(path, "rb") as fp:
if read_exact(fp, 4, "GGUF magic") != b"GGUF":
fail(f"{path}: not a GGUF file")
version = read_u32(fp, "GGUF version")
if version != GGUF_VERSION:
fail(f"expected GGUF v{GGUF_VERSION}, got v{version}")
tensor_count = read_u64(fp, "GGUF tensor count")
metadata_count = read_u64(fp, "GGUF metadata count")
if tensor_count != len(plan):
fail(f"tensor count {tensor_count} != expected {len(plan)}")
if metadata_count != len(expected_metadata):
fail(f"metadata count {metadata_count} != expected {len(expected_metadata)}")
actual_values = {}
for _ in range(metadata_count):
key = read_gguf_string(fp, "GGUF metadata key")
value_type = read_u32(fp, "GGUF metadata type")
actual_values[key] = read_metadata_value(fp, value_type)
if actual_values != expected_values:
fail("GGUF metadata differs from the pinned conversion metadata")
for index, item in enumerate(plan):
name = read_gguf_string(fp, f"tensor {index} name")
rank = read_u32(fp, f"tensor {index} rank")
shape = tuple(read_u64(fp, f"tensor {index} dimension") for _ in range(rank))
qtype = read_u32(fp, f"tensor {index} type")
offset = read_u64(fp, f"tensor {index} offset")
if (name, shape, qtype, offset) != (item.name, item.shape, item.qtype, item.offset):
fail(f"tensor {index} header differs from the conversion plan: {name}")
data_offset, data_bytes = layout(plan, expected_metadata)
actual_size = os.fstat(fp.fileno()).st_size
if actual_size != data_offset + data_bytes:
fail(f"file size {actual_size} != expected {data_offset + data_bytes}")
verified = 0
if verify_payload:
for index, item in enumerate(plan, 1):
fp.seek(data_offset + item.offset)
for source in db.iter_read(item.source):
if read_exact(fp, len(source), item.name) != source:
fail(f"{item.name}: payload differs from the official source")
verified += len(source)
padding = read_exact(fp, align(item.nbytes) - item.nbytes, f"{item.name} padding")
if any(padding):
fail(f"{item.name}: nonzero alignment padding")
if index % 25 == 0 or index == len(plan):
print(f"verified payloads: {index}/{len(plan)}", file=sys.stderr, flush=True)
digest = file_sha256(path)
if expected_sha256 and digest.lower() != expected_sha256.lower():
fail(f"SHA-256 {digest} != expected {expected_sha256.lower()}")
print(f"validated {path}: {len(plan)} tensors, {actual_size} bytes")
print(f"SHA-256: {digest}")
if verify_payload:
print(f"source payloads matched byte for byte: {verified} bytes")
def parse_args():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--hf", required=True, help="official Vision-Exp snapshot")
output = parser.add_mutually_exclusive_group()
output.add_argument("--out", help="create this vision GGUF")
output.add_argument("--validate", metavar="GGUF", help="validate an existing vision GGUF")
parser.add_argument("--source-revision", default=SOURCE_REVISION)
parser.add_argument("--verify-payload", action="store_true")
parser.add_argument("--expected-sha256")
parser.add_argument("--overwrite", action="store_true")
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
if args.source_revision != SOURCE_REVISION:
parser.error(f"unsupported source revision: {args.source_revision}")
if not args.dry_run and not args.out and not args.validate:
parser.error("one of --out, --validate, or --dry-run is required")
if args.verify_payload and not args.validate:
parser.error("--verify-payload requires --validate")
if args.expected_sha256 and not args.validate:
parser.error("--expected-sha256 requires --validate")
return args
def main():
args = parse_args()
db = SourceDB(
args.hf,
validate_deepseek4_index,
validate_deepseek4_fp8_scales,
)
try:
plan = build_plan(db)
metadata = sidecar_metadata(args.hf, args.source_revision)
if args.dry_run:
print_summary(plan, metadata)
elif args.out:
create_gguf(args.out, plan, metadata, db, args.overwrite)
print(f"deepseek4-vision: wrote {args.out}", file=sys.stderr)
else:
validate_gguf(
args.validate,
plan,
metadata,
db,
args.verify_payload,
args.expected_sha256,
)
finally:
db.close()
if __name__ == "__main__":
try:
main()
except (KeyError, OSError, ValueError, json.JSONDecodeError) as error:
print(f"deepseek4-vision: error: {error}", file=sys.stderr)
raise SystemExit(1)
gguf-tools/gen_qwen4_unicode.py
+50
View File
@@ -0,0 +1,50 @@
#!/usr/bin/env python3
"""Regenerate ds4_qwen4_unicode.inc, the codepoint classes of the Qwen3.8
("qwen35") pre-tokenizer regex, from the `regex` module's Unicode tables.
Use a current `regex` release; the generated header records its version.
Usage: python gen_qwen4_unicode.py > ../ds4_qwen4_unicode.inc
"""
import sys
import regex
CLASSES = {
"letter": r"\p{L}",
"mark": r"\p{M}",
"number": r"\p{N}",
"space": r"\s",
}
def main() -> None:
out = sys.stdout
out.write("/* Generated by gguf-tools/gen_qwen4_unicode.py from the `regex` module's\n")
out.write(f" * Unicode tables (regex {regex.__version__}); codepoint classes for the\n")
out.write(" * Qwen3.8 pre-tokenizer. */\n\n")
out.write("typedef struct { uint32_t lo, hi; } ds4_qwen4_urange;\n\n")
for index, (name, cls) in enumerate(CLASSES.items()):
if index:
out.write("\n")
pat = regex.compile(cls, regex.V1)
ranges = []
lo = prev = None
for cp in range(0x110000):
m = False if 0xD800 <= cp <= 0xDFFF else pat.match(chr(cp)) is not None
if m:
if lo is None:
lo = cp
prev = cp
elif lo is not None:
ranges.append((lo, prev))
lo = None
if lo is not None:
ranges.append((lo, prev))
out.write(f"static const ds4_qwen4_urange ds4_qwen4_{name}_ranges[] = {{\n")
for i in range(0, len(ranges), 4):
out.write(" " + " ".join(f"{{0x{a:05x}, 0x{b:05x}}}," for a, b in ranges[i:i + 4]) + "\n")
out.write("};\n")
if __name__ == "__main__":
main()
gguf-tools/glm53_full_quantize.py
Executable
+324
View File
@@ -0,0 +1,324 @@
#!/usr/bin/env python3
"""Build DwarfStar's asymmetric Q2 GGUF for the full GLM-5.3 model."""
from __future__ import annotations
import argparse
import json
import os
import sys
import glm53_quantize as q
from glm53_manifest import validate_glm53_full_index
EXPERT_COUNT = 256
BLOCK_COUNT = 79
MTP_BLOCK = 78
def source_prefix(layer):
return f"model.layers.{layer}"
def add_experts(plan, db, layer, part, qtype):
source = f"{source_prefix(layer)}.mlp.experts.{{expert}}.{part}_proj.weight"
shape = db.info(source.format(expert=0))["shape"]
if len(shape) != 2:
q.fail(f"expert source is not a matrix: {source.format(expert=0)}")
for expert in range(EXPERT_COUNT):
name = source.format(expert=expert)
if db.info(name)["shape"] != shape:
q.fail(f"expert shape mismatch: {name}")
item = q.TensorPlan(
f"blk.{layer}.ffn_{part}_exps.weight",
(shape[1], shape[0], EXPERT_COUNT),
qtype,
f"routed_{part}",
source=source,
expert_layer=layer,
expert_part=part,
expert_count=EXPERT_COUNT,
)
item.nbytes = q.qtype_nbytes(qtype, item.shape)
plan.append(item)
def indexer_owner(layer, indexer_types):
if layer == MTP_BLOCK:
return layer
owner = None
for current in range(layer + 1):
if indexer_types[current] == "full":
owner = current
if owner is None:
q.fail(f"layer {layer} has no indexer owner")
return owner
def add_attention(plan, db, layer, indexer_types):
prefix = f"{source_prefix(layer)}.self_attn"
owner_prefix = f"{source_prefix(indexer_owner(layer, indexer_types))}.self_attn"
mapping = (
("attn_q_a", prefix, "q_a_proj.weight", q.QTYPE_Q8_0),
("attn_q_a_norm", prefix, "q_a_layernorm.weight", q.QTYPE_F32),
("attn_q_b", prefix, "q_b_proj.weight", q.QTYPE_Q8_0),
("attn_kv_a_mqa", prefix, "kv_a_proj_with_mqa.weight", q.QTYPE_Q8_0),
("attn_kv_a_norm", prefix, "kv_a_layernorm.weight", q.QTYPE_F32),
("attn_output", prefix, "o_proj.weight", q.QTYPE_Q8_0),
("indexer.attn_q_b", owner_prefix, "indexer.wq_b.weight", q.QTYPE_Q8_0),
("indexer.attn_k", owner_prefix, "indexer.wk.weight", q.QTYPE_Q8_0),
("indexer.k_norm", owner_prefix, "indexer.k_norm.weight", q.QTYPE_F32),
("indexer.k_norm", owner_prefix, "indexer.k_norm.bias", q.QTYPE_F32),
("indexer.proj", owner_prefix, "indexer.weights_proj.weight", q.QTYPE_F32),
)
for target, source_base, source_tail, qtype in mapping:
suffix = ".bias" if source_tail.endswith(".bias") else ".weight"
target_name = target + (".bias" if suffix == ".bias" else ".weight")
q.add_regular(
plan,
db,
f"blk.{layer}.{target_name}",
f"{source_base}.{source_tail}",
qtype,
"dsa",
)
combined = f"{prefix}.kv_b_proj.weight"
expected = [64 * (192 + 256), 512]
if db.info(combined)["shape"] != expected:
q.fail(f"unexpected kv_b shape for layer {layer}: {db.info(combined)['shape']}")
q.add_regular(
plan,
db,
f"blk.{layer}.attn_k_b.weight",
combined,
q.QTYPE_Q8_0,
"dsa",
shape=(192, 512, 64),
transform="kv_b_k",
)
q.add_regular(
plan,
db,
f"blk.{layer}.attn_v_b.weight",
combined,
q.QTYPE_Q8_0,
"dsa",
shape=(512, 256, 64),
transform="kv_b_v",
)
def add_ffn(plan, db, layer, provisional_q2k=False):
prefix = f"{source_prefix(layer)}.mlp"
if layer < 3:
for part in ("gate", "up", "down"):
q.add_regular(
plan,
db,
f"blk.{layer}.ffn_{part}.weight",
f"{prefix}.{part}_proj.weight",
q.QTYPE_Q8_0,
"dense_ffn",
)
return
q.add_regular(
plan,
db,
f"blk.{layer}.ffn_gate_inp.weight",
f"{prefix}.gate.weight",
q.QTYPE_F32,
"router",
)
q.add_regular(
plan,
db,
f"blk.{layer}.exp_probs_b.bias",
f"{prefix}.gate.e_score_correction_bias",
q.QTYPE_F32,
"router",
)
expert_qtype = (
q.QTYPE_Q2_K
if provisional_q2k or layer == MTP_BLOCK
else q.QTYPE_IQ2_XXS
)
for part in ("gate", "up", "down"):
add_experts(plan, db, layer, part, expert_qtype)
for part in ("gate", "up", "down"):
q.add_regular(
plan,
db,
f"blk.{layer}.ffn_{part}_shexp.weight",
f"{prefix}.shared_experts.{part}_proj.weight",
q.QTYPE_Q8_0,
"shared_expert",
)
def build_plan(db, config, provisional_q2k=False):
plan = []
q.add_regular(plan, db, "token_embd.weight", "model.embed_tokens.weight", q.QTYPE_Q8_0, "embedding")
indexer_types = config["indexer_types"]
if len(indexer_types) != MTP_BLOCK:
q.fail(f"expected {MTP_BLOCK} indexer types, got {len(indexer_types)}")
for layer in range(BLOCK_COUNT):
prefix = source_prefix(layer)
q.add_regular(
plan,
db,
f"blk.{layer}.attn_norm.weight",
f"{prefix}.input_layernorm.weight",
q.QTYPE_F32,
"norm",
)
add_attention(plan, db, layer, indexer_types)
q.add_regular(
plan,
db,
f"blk.{layer}.ffn_norm.weight",
f"{prefix}.post_attention_layernorm.weight",
q.QTYPE_F32,
"norm",
)
add_ffn(plan, db, layer, provisional_q2k)
if layer == MTP_BLOCK:
q.add_regular(
plan,
db,
f"blk.{layer}.nextn.eh_proj.weight",
f"{prefix}.eh_proj.weight",
q.QTYPE_Q8_0,
"mtp",
)
for target, source in (
("enorm", "enorm.weight"),
("hnorm", "hnorm.weight"),
("shared_head_norm", "shared_head.norm.weight"),
):
q.add_regular(
plan,
db,
f"blk.{layer}.nextn.{target}.weight",
f"{prefix}.{source}",
q.QTYPE_F32,
"mtp",
)
q.add_regular(plan, db, "output_norm.weight", "model.norm.weight", q.QTYPE_F32, "output")
q.add_regular(plan, db, "output.weight", "lm_head.weight", q.QTYPE_Q8_0, "output")
names = [item.name for item in plan]
if len(names) != 1809:
q.fail(f"full GLM-5.3 plan has {len(names)} tensors, expected 1809")
if len(names) != len(set(names)):
q.fail("duplicate GGUF tensor names in full GLM-5.3 plan")
offset = 0
for item in plan:
item.offset = offset
offset += q.align(item.nbytes)
return plan
def model_metadata(hf_dir, source_revision):
with open(os.path.join(hf_dir, "chat_template.jinja"), "rb") as fp:
chat_template = fp.read()
return [
q.kv_string("general.architecture", "glm-dsa"),
q.kv_string("general.name", "GLM-5.3"),
q.kv_string("general.basename", "GLM-5.3"),
q.kv_string("general.version", "5.3"),
q.kv_string("general.license", "glm-5.3"),
q.kv_string("general.source.repo_url", "https://huggingface.co/zai-org/GLM-5.3"),
q.kv_string("general.source.revision", source_revision),
q.kv_u32("general.alignment", q.GGUF_ALIGNMENT),
q.kv_u32("general.quantization_version", 2),
q.kv_u32("glm-dsa.block_count", BLOCK_COUNT),
q.kv_u32("glm-dsa.nextn_predict_layers", 1),
q.kv_u64("glm-dsa.context_length", 1048576),
q.kv_u32("glm-dsa.embedding_length", 6144),
q.kv_u32("glm-dsa.vocab_size", 154880),
q.kv_u32("glm-dsa.feed_forward_length", 12288),
q.kv_u32("glm-dsa.expert_feed_forward_length", 2048),
q.kv_u32("glm-dsa.expert_count", EXPERT_COUNT),
q.kv_u32("glm-dsa.expert_used_count", 8),
q.kv_u32("glm-dsa.expert_shared_count", 1),
q.kv_u32("glm-dsa.leading_dense_block_count", 3),
q.kv_u32("glm-dsa.expert_group_count", 1),
q.kv_u32("glm-dsa.expert_group_used_count", 1),
q.kv_u32("glm-dsa.expert_gating_func", 2),
q.kv_f32("glm-dsa.expert_weights_scale", 2.5),
q.kv_bool("glm-dsa.expert_weights_norm", True),
q.kv_f32("glm-dsa.attention.layer_norm_rms_epsilon", 1.0e-5),
q.kv_u32("glm-dsa.attention.head_count", 64),
q.kv_u32("glm-dsa.attention.head_count_kv", 1),
q.kv_u32("glm-dsa.attention.key_length", 576),
q.kv_u32("glm-dsa.attention.value_length", 512),
q.kv_u32("glm-dsa.attention.key_length_mla", 256),
q.kv_u32("glm-dsa.attention.value_length_mla", 256),
q.kv_u32("glm-dsa.attention.q_lora_rank", 2048),
q.kv_u32("glm-dsa.attention.kv_lora_rank", 512),
q.kv_u32("glm-dsa.attention.indexer.head_count", 32),
q.kv_u32("glm-dsa.attention.indexer.key_length", 128),
q.kv_u32("glm-dsa.attention.indexer.top_k", 2048),
q.kv_u32("glm-dsa.rope.dimension_count", 64),
q.kv_f32("glm-dsa.rope.freq_base", 8000000.0),
q.kv_string("tokenizer.chat_template", chat_template),
]
def parse_args():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--hf", required=True, help="official zai-org/GLM-5.3 FP8 snapshot")
parser.add_argument("--tokenizer-template", required=True, help="compatible GLM GGUF tokenizer source")
parser.add_argument("--out", help="output GGUF")
parser.add_argument("--imatrix", help="DwarfStar/llama.cpp importance matrix")
parser.add_argument(
"--provisional-q2k",
action="store_true",
help="use Q2_K for every routed expert to build a fast calibration model",
)
parser.add_argument("--threads", type=int, default=8)
parser.add_argument("--source-revision", default="e0b07fd2751b42d5efa199cc02c2b271deadc516")
parser.add_argument("--quants-library", help="path to libds4quants")
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--resume", action="store_true")
parser.add_argument("--overwrite", action="store_true")
args = parser.parse_args()
if args.threads < 1 or args.threads > 64:
parser.error("--threads must be between 1 and 64")
if not args.dry_run and not args.out:
parser.error("--out is required unless --dry-run is used")
return args
def main():
args = parse_args()
with open(os.path.join(args.hf, "config.json"), "rb") as fp:
config = json.load(fp)
if config.get("architectures") != ["GlmMoeDsaForCausalLM"]:
q.fail(f"unexpected architecture: {config.get('architectures')!r}")
db = q.SourceDB(args.hf, validate_glm53_full_index)
try:
plan = build_plan(db, config, args.provisional_q2k)
tokenizer_records, template_tokens = q.load_tokenizer_records(args.tokenizer_template)
q.validate_tokenizer_template(args.hf, template_tokens, 154880)
kv_records = model_metadata(args.hf, args.source_revision)
if args.dry_run:
q.print_plan(plan, kv_records, tokenizer_records)
else:
q.write_gguf(args, plan, kv_records, tokenizer_records, db)
print(f"glm53-full-quantize: wrote {args.out}", file=sys.stderr)
finally:
db.close()
if __name__ == "__main__":
try:
main()
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"glm53-full-quantize: error: {error}", file=sys.stderr)
sys.exit(1)
gguf-tools/glm53_manifest.py
+56
View File
@@ -240,6 +240,62 @@ def validate_glm53_index(weight_map):
fail(f"missing required tensors: {missing}")
def validate_glm53_full_index(weight_map):
names = set(weight_map)
layers = set()
indexer_layers = set()
sparse_layers = set()
expert_ids = {}
expert_parts = {}
layer_re = re.compile(r"^model\.layers\.(\d+)\.(.+)$")
expert_re = re.compile(r"^mlp\.experts\.(\d+)\.(gate|up|down)_proj\.weight$")
allowed_top = {"model.embed_tokens.weight", "model.norm.weight", "lm_head.weight"}
for name in names:
match = layer_re.match(name)
if not match:
if name not in allowed_top and not name.endswith("_scale_inv"):
fail(f"unknown full GLM-5.3 tensor: {name}")
continue
layer = int(match.group(1))
tail = match.group(2)
layers.add(layer)
if tail == "self_attn.indexer.wk.weight":
indexer_layers.add(layer)
expert = expert_re.match(tail)
if expert:
sparse_layers.add(layer)
expert_id = int(expert.group(1))
expert_ids.setdefault(layer, set()).add(expert_id)
expert_parts.setdefault((layer, expert_id), set()).add(expert.group(2))
expect_equal(layers, set(range(79)), "full GLM-5.3 layer set")
expect_equal(sparse_layers, set(range(3, 79)), "full GLM-5.3 sparse FFN layer set")
expected_indexers = {0, 1, 2, 78} | set(range(6, 78, 4))
expect_equal(indexer_layers, expected_indexers, "full GLM-5.3 indexer owner layers")
for layer in range(3, 79):
expect_equal(expert_ids.get(layer), set(range(256)), f"layer {layer} expert ids")
for expert in range(256):
expect_equal(
expert_parts.get((layer, expert)),
{"gate", "up", "down"},
f"layer {layer} expert {expert} projections",
)
required = {
"model.embed_tokens.weight",
"model.norm.weight",
"lm_head.weight",
"model.layers.78.eh_proj.weight",
"model.layers.78.enorm.weight",
"model.layers.78.hnorm.weight",
"model.layers.78.shared_head.norm.weight",
}
missing = sorted(required - names)
if missing:
fail(f"missing required full GLM-5.3 tensors: {missing}")
def validate_fp8_scales(tensors):
for name, info in tensors.items():
if info["dtype"] != "F8_E4M3" or not name.endswith(".weight"):
gguf-tools/glm53_quantize.py
+85 -39
View File
@@ -39,6 +39,7 @@ QTYPE_Q8_K = 15
QTYPE_IQ2_XXS = 16
QTYPE_I8 = 24
QTYPE_BF16 = 30
QTYPE_MXFP4 = 39
QTYPE_NAMES = {
QTYPE_F32: "F32",
@@ -50,6 +51,7 @@ QTYPE_NAMES = {
QTYPE_IQ2_XXS: "IQ2_XXS",
QTYPE_I8: "I8",
QTYPE_BF16: "BF16",
QTYPE_MXFP4: "MXFP4",
}
QTYPE_LAYOUT = {
@@ -62,6 +64,7 @@ QTYPE_LAYOUT = {
QTYPE_IQ2_XXS: (256, 66),
QTYPE_I8: (1, 1),
QTYPE_BF16: (1, 2),
QTYPE_MXFP4: (32, 17),
}
LAYER_PREFIX = "model.language_model.layers"
@@ -276,6 +279,7 @@ class TensorPlan:
row_count: int | None = None
expert_layer: int | None = None
expert_part: str | None = None
expert_count: int | None = None
transform: str | None = None
raw_copy: bool = False
expert_scale: bool = False
@@ -288,11 +292,22 @@ class TensorPlan:
class SourceDB:
def __init__(self, hf_dir):
def __init__(
self,
hf_dir,
index_validator=validate_glm53_index,
scale_validator=validate_fp8_scales,
tensor_prefix=None,
):
self.hf_dir = hf_dir
index_path = os.path.join(hf_dir, "model.safetensors.index.json")
document, self.weight_map = load_index(index_path)
validate_glm53_index(self.weight_map)
index_validator(self.weight_map)
if tensor_prefix is not None:
self.weight_map = {name: shard for name, shard in self.weight_map.items()
if name.startswith(tensor_prefix)}
if not self.weight_map:
fail(f"source index has no tensors matching {tensor_prefix!r}")
self.declared_bytes = document.get("metadata", {}).get("total_size")
self.tensors = {}
self._fds = {}
@@ -303,6 +318,8 @@ class SourceDB:
if not os.path.isfile(path):
fail(f"missing source shard {path}")
for name, info in load_safetensors_header(path).items():
if tensor_prefix is not None and not name.startswith(tensor_prefix):
continue
if self.weight_map.get(name) != shard:
fail(f"index assigns {name} to {self.weight_map.get(name)!r}, not {shard}")
if name in self.tensors:
@@ -312,7 +329,7 @@ class SourceDB:
if set(self.tensors) != set(self.weight_map):
missing = sorted(set(self.weight_map) - set(self.tensors))
fail(f"source headers are incomplete; first missing tensor is {missing[0]}")
validate_fp8_scales(self.tensors)
scale_validator(self.tensors)
def info(self, name):
try:
@@ -405,8 +422,10 @@ def add_experts(plan, db, layer, part, qtype):
(shape[1], shape[0], 288),
qtype,
f"routed_{part}",
source=f"{source_prefix(layer)}.mlp.experts.{{expert}}.{part}_proj.weight",
expert_layer=layer,
expert_part=part,
expert_count=288,
)
item.nbytes = qtype_nbytes(qtype, item.shape)
plan.append(item)
@@ -545,10 +564,8 @@ def native_fp8_plan(db, plan):
covered = set()
for item in plan:
if item.is_expert:
first = (
f"{source_prefix(item.expert_layer)}.mlp.experts.0."
f"{item.expert_part}_proj.weight"
)
expert_count = item.expert_count or 288
first = item.source.format(expert=0)
info = db.info(first)
if info["dtype"] != "F8_E4M3":
fail(f"native expert source is not FP8 E4M3: {first}")
@@ -561,21 +578,20 @@ def native_fp8_plan(db, plan):
fail(f"native FP8 scale is not F32: {first}_scale_inv")
scale = TensorPlan(
item.name + "_scale_inv",
(scale_info["shape"][1], scale_info["shape"][0], 288),
(scale_info["shape"][1], scale_info["shape"][0], expert_count),
QTYPE_F32,
item.role + "_scale",
source=item.source,
expert_layer=item.expert_layer,
expert_part=item.expert_part,
expert_count=expert_count,
raw_copy=True,
expert_scale=True,
)
scale.nbytes = qtype_nbytes(scale.qtype, scale.shape)
native.append(scale)
for expert in range(288):
source = (
f"{source_prefix(item.expert_layer)}.mlp.experts.{expert}."
f"{item.expert_part}_proj.weight"
)
for expert in range(expert_count):
source = item.source.format(expert=expert)
if db.info(source)["dtype"] != "F8_E4M3":
fail(f"native expert source is not FP8 E4M3: {source}")
if db.info(source)["shape"] != info["shape"]:
@@ -796,7 +812,7 @@ class Quantizer:
elif info["dtype"] == "F16":
array = np.frombuffer(raw, dtype="<f2").astype(np.float32).reshape(shape)
elif info["dtype"] == "F8_E4M3":
if len(shape) != 2 or shape[0] % 128 or shape[1] % 128:
if len(shape) != 2:
fail(f"unsupported FP8 shape for {name}: {shape}")
codes = np.frombuffer(raw, dtype=np.uint8).reshape(shape)
if np.any((codes & 0x7F) == 0x7F):
@@ -805,8 +821,15 @@ class Quantizer:
scale_info = db.info(scale_name)
if scale_info["dtype"] != "F32":
fail(f"{scale_name} is not F32")
expected_scale_shape = [(dim + 127) // 128 for dim in shape]
if scale_info["shape"] != expected_scale_shape:
fail(
f"{scale_name} has shape {scale_info['shape']}, "
f"expected {expected_scale_shape}"
)
scales = np.frombuffer(db.read(scale_name), dtype="<f4").reshape(scale_info["shape"])
expanded = np.repeat(np.repeat(scales, 128, axis=0), 128, axis=1)
expanded = expanded[: shape[0], : shape[1]]
array = self.fp8_lut[codes] * expanded
else:
fail(f"unsupported source dtype {info['dtype']} for {name}")
@@ -879,11 +902,11 @@ class Imatrix:
fail(f"nonfinite imatrix values for {name}")
self.entries[name] = values
def expert(self, tensor_name, expert, width):
def expert(self, tensor_name, expert, width, expert_count=288):
values = self.entries.get(tensor_name)
if values is None:
return None
expected = 288 * width
expected = expert_count * width
if values.size != expected:
fail(f"imatrix {tensor_name} has {values.size} values, expected {expected}")
result = values[expert * width : (expert + 1) * width]
@@ -901,7 +924,7 @@ def tensor_header(item):
)
def print_plan(plan, kv_records, tokenizer_records):
def print_plan(plan, kv_records, tokenizer_records, alignment=GGUF_ALIGNMENT):
by_type = {}
by_role = {}
for item in plan:
@@ -909,8 +932,8 @@ def print_plan(plan, kv_records, tokenizer_records):
by_role[item.role] = by_role.get(item.role, 0) + item.nbytes
kv_bytes = sum(map(len, kv_records)) + sum(map(len, tokenizer_records))
tensor_info_bytes = sum(len(tensor_header(item)) for item in plan)
data_offset = align(4 + 4 + 8 + 8 + kv_bytes + tensor_info_bytes)
data_bytes = sum(align(item.nbytes) for item in plan)
data_offset = align(4 + 4 + 8 + 8 + kv_bytes + tensor_info_bytes, alignment)
data_bytes = sum(align(item.nbytes, alignment) for item in plan)
print(f"tensors: {len(plan)}")
print(f"metadata_records: {len(kv_records) + len(tokenizer_records)}")
print(f"metadata_bytes: {data_offset}")
@@ -928,23 +951,32 @@ def transform_regular(values, item):
return values
if item.transform not in ("kv_b_k", "kv_b_v"):
fail(f"unknown transform {item.transform} for {item.name}")
if values.shape != (64 * 512, 512):
fail(f"{item.name}: expected combined kv_b shape (32768, 512), got {values.shape}")
per_head = values.reshape(64, 512, 512)
if values.ndim != 2 or len(item.shape) != 3:
fail(f"{item.name}: invalid combined kv_b shapes {values.shape} -> {item.shape}")
n_head = item.shape[2]
rank = item.shape[1] if item.transform == "kv_b_k" else item.shape[0]
selected = item.shape[0] if item.transform == "kv_b_k" else item.shape[1]
if values.shape[1] != rank or values.shape[0] % n_head:
fail(f"{item.name}: invalid combined kv_b shape {values.shape}")
combined = values.shape[0] // n_head
other = combined - selected
if other <= 0:
fail(f"{item.name}: invalid combined kv_b dimensions {combined} and {selected}")
k_dim = selected if item.transform == "kv_b_k" else other
v_dim = other if item.transform == "kv_b_k" else selected
per_head = values.reshape(n_head, k_dim + v_dim, rank)
if item.transform == "kv_b_k":
return per_head[:, :256, :].transpose(0, 2, 1).copy()
return per_head[:, 256:, :].copy()
return per_head[:, :k_dim, :].transpose(0, 2, 1).copy()
return per_head[:, k_dim:, :].copy()
def iter_native_tensor_bytes(item, db, np):
if not item.raw_copy:
fail(f"{item.name}: tensor is not a native payload")
if item.is_expert:
for expert in range(288):
source = (
f"{source_prefix(item.expert_layer)}.mlp.experts.{expert}."
f"{item.expert_part}_proj.weight"
)
expert_count = item.expert_count or 288
for expert in range(expert_count):
source = item.source.format(expert=expert)
if item.expert_scale:
source += "_scale_inv"
yield from db.iter_read(source)
@@ -981,26 +1013,25 @@ def write_regular(fp, item, db, quantizer):
def write_experts(fp, item, db, quantizer, imatrix, threads):
layer = item.expert_layer
part = item.expert_part
expert_count = item.expert_count or 288
width = item.shape[0]
def convert(expert):
source = f"{source_prefix(layer)}.mlp.experts.{expert}.{part}_proj.weight"
source = item.source.format(expert=expert)
if item.raw_copy:
if item.expert_scale:
source += "_scale_inv"
return db.read(source), False
values = quantizer.to_f32(db, source)
weights = imatrix.expert(item.name, expert, width)
weights = imatrix.expert(item.name, expert, width, expert_count)
return quantizer.encode(values, item.qtype, weights), weights is None
per_expert = item.nbytes // 288
per_expert = item.nbytes // expert_count
fallback_count = 0
with concurrent.futures.ThreadPoolExecutor(max_workers=threads) as executor:
window = max(threads * 2, 1)
for start in range(0, 288, window):
futures = [executor.submit(convert, expert) for expert in range(start, min(start + window, 288))]
for start in range(0, expert_count, window):
futures = [executor.submit(convert, expert) for expert in range(start, min(start + window, expert_count))]
for future in futures:
data, fallback = future.result()
fallback_count += int(fallback)
@@ -1010,12 +1041,22 @@ def write_experts(fp, item, db, quantizer, imatrix, threads):
if imatrix.entries and fallback_count:
print(
f"glm53-quantize: {item.name} used weight-based fallback for "
f"{fallback_count}/288 unobserved experts",
f"{fallback_count}/{expert_count} unobserved experts",
file=sys.stderr,
)
def conversion_signature(plan, kv_records, tokenizer_records):
def file_sha256(path):
if not path:
return None
digest = hashlib.sha256()
with open(path, "rb") as fp:
while chunk := fp.read(8 << 20):
digest.update(chunk)
return digest.digest()
def conversion_signature(plan, kv_records, tokenizer_records, imatrix_path=None):
digest = hashlib.sha256()
for record in kv_records + tokenizer_records:
digest.update(record)
@@ -1023,7 +1064,12 @@ def conversion_signature(plan, kv_records, tokenizer_records):
digest.update(tensor_header(item))
digest.update((item.source or "").encode())
digest.update((item.transform or "").encode())
digest.update(struct.pack("<I", item.expert_count or 0))
digest.update(bytes((item.raw_copy, item.expert_scale)))
imatrix_digest = file_sha256(imatrix_path)
if imatrix_digest is not None:
digest.update(b"imatrix\0")
digest.update(imatrix_digest)
return digest.hexdigest()
@@ -1066,7 +1112,7 @@ def write_gguf(args, plan, kv_records, tokenizer_records, db):
partial = args.out + ".partial"
resume_path = partial + ".resume.json"
signature = conversion_signature(plan, kv_records, tokenizer_records)
signature = conversion_signature(plan, kv_records, tokenizer_records, args.imatrix)
if os.path.exists(args.out) and not args.overwrite:
fail(f"output exists: {args.out}; use --overwrite")
if args.overwrite:
gguf-tools/merge_deepseek41_shards.py
+238
View File
@@ -0,0 +1,238 @@
#!/usr/bin/env python3
"""Restore a page-aligned V4.1 GGUF from verified standard GGUF split files.
The downloader verifies each input's published SHA-256. This helper preserves
all tensor bytes and metadata, removes split metadata, restores 16 KiB alignment,
and verifies the original whole-file checksum before publishing the output.
Inputs are retained; an interrupted .assembling file can be resumed.
"""
import argparse
import fcntl
import hashlib
import math
import os
from pathlib import Path
import shutil
import struct
import sys
ALIGNMENT = 16384
CHUNK = 16 << 20
SCALARS = {0: "B", 1: "b", 2: "H", 3: "h", 4: "I", 5: "i", 6: "f",
7: "?", 10: "Q", 11: "q", 12: "d"}
TYPES = {0: (1, 4), 1: (1, 2), 8: (32, 34), 24: (1, 1), 39: (32, 17)}
SPLIT_KEYS = {"split.no", "split.count", "split.tensors.count"}
def align(n, alignment):
return (n + alignment - 1) // alignment * alignment
def read(fp, n):
data = fp.read(n)
if len(data) != n:
raise ValueError("Truncated GGUF")
return data
def scalar(fp, kind):
return struct.unpack("<" + kind, read(fp, struct.calcsize("<" + kind)))[0]
def string(fp):
size = scalar(fp, "Q")
if size > 128 << 20:
raise ValueError("GGUF string exceeds header limit")
return read(fp, size).decode("utf-8")
def skip_value(fp, kind, file_size, depth=0):
if depth > 4:
raise ValueError("Nested GGUF metadata exceeds limit")
if kind == 8:
fp.seek(scalar(fp, "Q"), 1)
elif kind == 9:
element, count = scalar(fp, "I"), scalar(fp, "Q")
if element in SCALARS:
fp.seek(count * struct.calcsize("<" + SCALARS[element]), 1)
else:
if count > file_size:
raise ValueError("Invalid GGUF array length")
for _ in range(count):
skip_value(fp, element, file_size, depth + 1)
elif kind in SCALARS:
fp.seek(struct.calcsize("<" + SCALARS[kind]), 1)
else:
raise ValueError("Unsupported GGUF metadata type")
if fp.tell() > min(file_size, 128 << 20):
raise ValueError("Invalid GGUF metadata extent")
def read_header(fp, file_size):
if read(fp, 4) != b"GGUF" or scalar(fp, "I") != 3:
raise ValueError("Expected GGUF v3")
nt, nk = scalar(fp, "Q"), scalar(fp, "Q")
if nt > 100000 or nk > 100000:
raise ValueError("Invalid GGUF directory count")
metadata, values, kinds, tensors = [], {}, {}, []
for _ in range(nk):
start = fp.tell()
key, kind = string(fp), scalar(fp, "I")
if key in kinds:
raise ValueError("Duplicate GGUF metadata key")
kinds[key] = kind
value_start = fp.tell()
skip_value(fp, kind, file_size)
end = fp.tell()
if kind in SCALARS or kind == 8:
fp.seek(value_start)
values[key] = string(fp) if kind == 8 else scalar(fp, SCALARS[kind])
fp.seek(start)
metadata.append((key, read(fp, end - start)))
alignment = values.get("general.alignment", 32)
if (not isinstance(alignment, int) or not 1 <= alignment <= ALIGNMENT or
alignment & (alignment - 1)):
raise ValueError("Invalid GGUF alignment")
offset = 0
for _ in range(nt):
name, rank = string(fp), scalar(fp, "I")
if not 1 <= rank <= 4:
raise ValueError("Invalid tensor rank")
shape = [scalar(fp, "Q") for _ in range(rank)]
kind, relative = scalar(fp, "I"), scalar(fp, "Q")
if kind not in TYPES or any(n == 0 for n in shape):
raise ValueError("Unsupported MXFP4 model tensor")
block, size = TYPES[kind]
if shape[0] % block or relative != offset:
raise ValueError("Invalid tensor shape or offset")
nbytes = math.prod(shape) // block * size
tensors.append((name, shape, kind, relative, nbytes))
offset += align(nbytes, alignment)
data_offset = align(fp.tell(), alignment)
if data_offset + offset != file_size:
raise ValueError("Incorrect shard file size")
return metadata, values, kinds, tensors, data_offset
def encoded_string(value):
data = value.encode("utf-8")
return struct.pack("<Q", len(data)) + data
def merge_plan(headers):
all_tensors, names = [], set()
for number, (metadata, values, kinds, tensors, _) in enumerate(headers):
if (kinds.get("split.no") != 2 or kinds.get("split.count") != 2 or
kinds.get("split.tensors.count") != 5 or
values.get("split.no") != number or
values.get("split.count") != len(headers)):
raise ValueError("Incorrect shard order/count or split metadata")
if number and any(key not in SPLIT_KEYS for key, _ in metadata):
raise ValueError("Unexpected metadata outside first shard")
for tensor in tensors:
if tensor[0] in names:
raise ValueError("Duplicate tensor across shards")
names.add(tensor[0])
all_tensors.append(tensor)
if any(h[1].get("split.tensors.count") != len(all_tensors) for h in headers):
raise ValueError("Incomplete GGUF shard set")
first = headers[0]
if (first[1].get("general.architecture") != "deepseek41" or
first[2].get("general.alignment") != 4):
raise ValueError("Expected DeepSeek V4.1 metadata and explicit alignment")
metadata = []
for key, raw in first[0]:
if key in SPLIT_KEYS:
continue
if key.startswith("split."):
raise ValueError("Unexpected split metadata")
if key == "general.alignment":
raw = encoded_string(key) + struct.pack("<II", 4, ALIGNMENT)
metadata.append(raw)
header = bytearray(b"GGUF" + struct.pack("<IQQ", 3, len(all_tensors), len(metadata)))
header.extend(b"".join(metadata))
offset = 0
for name, shape, kind, _, nbytes in all_tensors:
header.extend(encoded_string(name) + struct.pack("<I", len(shape)))
header.extend(struct.pack("<" + "Q" * len(shape), *shape))
header.extend(struct.pack("<IQ", kind, offset))
offset += align(nbytes, ALIGNMENT)
header.extend(bytes(align(len(header), ALIGNMENT) - len(header)))
return bytes(header), len(header) + offset
def verify(path, expected_bytes, expected_sha):
if path.stat().st_size != expected_bytes:
raise ValueError("Incorrect file size: " + str(path))
print("Verifying SHA-256: " + str(path), flush=True)
digest = hashlib.sha256()
with path.open("rb") as fp:
for data in iter(lambda: fp.read(CHUNK), b""):
digest.update(data)
if digest.hexdigest() != expected_sha:
raise ValueError("Checksum mismatch: " + str(path))
def merge(paths, output, expected_bytes, expected_sha):
pending = Path(str(output) + ".assembling")
with Path(str(pending) + ".lock").open("a") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
if output.exists():
verify(output, expected_bytes, expected_sha)
return
headers = []
for path in paths:
with path.open("rb") as fp:
headers.append(read_header(fp, os.fstat(fp.fileno()).st_size))
header, total = merge_plan(headers)
if total != expected_bytes:
raise ValueError("Merged layout does not match the original GGUF size")
completed = pending.stat().st_size if pending.exists() else 0
if completed > total:
raise ValueError("Invalid partial assembly; move it aside before retrying")
if shutil.disk_usage(output.parent).free < total - completed + (1 << 30):
raise ValueError("Not enough disk space to merge MXFP4; keep the shards and retry")
print("Merging MXFP4 shards (retaining inputs; restoring 16 KiB alignment)", flush=True)
with pending.open("r+b" if pending.exists() else "x+b") as dst:
def append(data):
# Verify an existing prefix before appending. Never truncate an
# interrupted or unrelated file to make it fit this assembly.
overlap = min(len(data), max(0, completed - dst.tell()))
if overlap and read(dst, overlap) != data[:overlap]:
raise ValueError("Partial assembly differs from shards; move it aside before retrying")
dst.write(data[overlap:])
append(header)
for path, h in zip(paths, headers):
with path.open("rb") as src:
for _, _, _, offset, nbytes in h[3]:
src.seek(h[4] + offset)
left = nbytes
while left:
data = read(src, min(CHUNK, left))
append(data)
left -= len(data)
append(bytes(align(nbytes, ALIGNMENT) - nbytes))
dst.flush()
os.fsync(dst.fileno())
verify(pending, expected_bytes, expected_sha)
# Publish without overwriting an output created by another process.
os.link(pending, output)
pending.unlink()
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--expected-bytes", required=True, type=int)
parser.add_argument("--sha256", required=True)
parser.add_argument("shards", nargs="+", type=Path)
args = parser.parse_args()
try:
merge(args.shards, args.output, args.expected_bytes, args.sha256)
except (OSError, ValueError, OverflowError) as exc:
sys.exit(str(exc))
if __name__ == "__main__":
main()
gguf-tools/quality-testing/README.md
+211 -6
View File
@@ -3,7 +3,7 @@
This directory contains the prompts, tracked official fixtures, and scripts used
to compare local GGUF variants against hosted-model continuations.
The metric is target-token negative log likelihood: collect a deterministic
The main metric is target-token negative log likelihood: collect an
official continuation, then ask each local GGUF how much probability it assigns
to that exact continuation token by token. This avoids judging quality from one
sampled answer.
@@ -18,12 +18,23 @@ calling hosted APIs:
- `data/glm53-flash-openrouter-zai-fp8-100`: 100 GLM 5.3 Flash continuations
from OpenRouter's pinned Z.AI FP8 endpoint. The endpoint does not expose
logprobs, so these are deterministic continuation fixtures.
- `data/glm53-flash-openrouter-zai-fp8-long`: eight code-context continuations
from the same FP8 endpoint, with 3K-7K prompts that exercise sparse attention.
Use the rendered prompts below to preserve the API's reasoning prefix.
- `data/flash`: 100 DeepSeek V4 Flash 0731 continuations collected from the
official DeepSeek API with `top_logprobs=20`.
- `data/pro`: 100 DeepSeek V4 PRO preview continuations collected from the
official DeepSeek API with `top_logprobs=20`.
- `data/pro-0813`: 100 DeepSeek V4 PRO 0813 continuations collected from the
official DeepSeek API with `top_logprobs=20`.
- `data/qwen38-flash-alibaba-100`: 100 non-thinking Qwen3.8 Flash
continuations from Alibaba through OpenRouter, with top-five logprobs.
- `data/qwen38-flash-alibaba-long`: 12 archive/code continuations from the
same endpoint, with prompts from 2K to 24K tokens.
- [deepseek-v4.1-flash-20260919-router](deepseek-v4.1-flash-20260919-router/README.md):
112 fresh official V4.1 Flash continuations, including 12 longer prompts up
to 23K tokens. Temperature-1 samples with top-20 logprobs, intended for
paired numerical comparisons. These are not V4 Flash or Vision-Exp vectors.
DeepSeek V4 Flash also has tracked official smoke vectors in
`tests/test-vectors/`. Those vectors drive `./ds4_test --logprob-vectors` and
@@ -32,17 +43,25 @@ include short prompts plus long-prompt attention cases.
The hosted APIs expose output-token logprobs and top-logprob alternatives, not
full vocabulary logits.
The scorer verifies each API token's bytes against the local token boundaries,
not just the number of tokens. If they differ, it still scores the continuation
text but skips API logprob comparisons for that case. Alternatives containing
Unicode replacement characters are excluded because the original token bytes
may have been lost by the provider.
## 2. Collect Official Continuations
For the tracked DeepSeek V4 Flash 0731 fixture:
The Flash 0731 fixture is historical. The official API now redirects the old
Flash names to V4.1; do not overwrite `data/flash` with new calls. Collect
current V4.1 references in a separate directory:
```sh
export DEEPSEEK_API_KEY=...
python3 gguf-tools/quality-testing/collect_official.py \
--model deepseek-v4-flash \
--model deepseek-flash \
--endpoint https://api.deepseek.com/chat/completions \
--prompts gguf-tools/quality-testing/prompts.jsonl \
--out gguf-tools/quality-testing/data/flash \
--out /tmp/deepseek-v4.1-flash-new \
--count 100 \
--max-tokens 24 \
--top-logprobs 20 \
@@ -113,13 +132,45 @@ The prompt list is tracked in `prompts.jsonl`. Curated fixture directories are
also tracked after review; ad-hoc API collection directories should stay
untracked until they are intentionally promoted into the release QA set.
Use `--system TEXT` when the reference needs an explicit system message.
`--system ''` sends an empty message, which is different from omitting it:
some providers insert a default system prompt when none is supplied. The
collector records this setting and rejects a changed system message on resume.
For a nonempty system message, render the matching local chat prompt and use
`--rendered-prompt`; the scorer's ordinary prompt mode adds no system message.
For Qwen, use the following settings in a new output directory:
```sh
python3 gguf-tools/quality-testing/collect_official.py \
--out /tmp/qwen-official \
--model qwen/qwen3.8-flash \
--endpoint https://openrouter.ai/api/v1/chat/completions \
--provider-order alibaba --require-parameters \
--require-response-provider Alibaba --require-response-model qwen/qwen3.8-flash \
--system '' --thinking omit --reasoning-effort none \
--count 100 --max-tokens 64 --top-logprobs 5
```
This reads `OPENROUTER_API_KEY` from the environment and disables provider
fallbacks. Omitting `--system ''` caused Alibaba to add 13 prompt tokens in
the September 2026 checks. The empty message matches Qwen's published
no-thinking template and the scorer's ordinary prompt mode.
Alibaba does not disclose this endpoint's precision. The
[official model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) describes
hosted Flash as based on Flash Next, not as a byte-identical checkpoint.
Agreement with this service is an external quality check, not proof of exact
native-weight inference.
## 3. Build The Local Scorer
```sh
make -C gguf-tools quality-score
```
The scorer links against the DS4 runtime and uses Metal by default.
The scorer links against the DS4 runtime, using Metal on macOS and CUDA on
Linux. On a DGX Spark, pass `CUDA_ARCH=sm_121` to the build command.
Build the optional llama.cpp control scorer with:
@@ -151,6 +202,71 @@ Use `data/flash/manifest.tsv` for Flash GGUFs and
care which model produced the manifest; the manifest path selects the
continuation set.
Add `--continued-prefill N` to process each prompt in two calls, leaving its
last `N` tokens for the second call. This checks continued-prefill quality
against the same official answers. Every prompt must contain more than `N`
tokens; otherwise the scorer fails instead of silently skipping the test.
Add `--session-batch N` (2 to 8) to score an official continuation alongside
unrelated, independently advancing sessions. The scorer rotates their row
order on every step. Compare against a run without this option using the same
manifest, model and context; budget memory for all `N` sessions.
Check session isolation separately on a dedicated Metal host:
```sh
DS4_TEST_MODEL=MODEL.gguf DS4_TEST_BATCH_ISOLATION=1 \
DS4_TEST_SESSION_COUNT=4 DS4_TEST_DECODE_STEPS=32 \
MTL_DEBUG_LAYER=1 ./tests/test_metal_session_batch
```
This requires exact logits when companion prompts and row order change, then
checks a continued prefill and resumed serial decoding. It also checks that
invalid batches leave the target unchanged. It does not replace serial/batch
quality comparisons: different arithmetic can round differently without
sessions contaminating one another. Use `DS4_TEST_PROMPT_FILE` and
`DS4_TEST_CONTEXT_SIZE` to repeat at longer prefixes. For physical TP, the test
also accepts `DS4_TEST_TP_RDMA_DEVICE` and `DS4_TEST_TP_GID_INDEX` alongside
its coordinator/worker settings. Throughput measurements must run separately
without Metal API validation.
For Metal V4.1 prefill scheduling changes, also run:
```sh
make tests/test_deepseek41_prefill
./tests/test_deepseek41_prefill --dispatch
MTL_DEBUG_LAYER=1 ./tests/test_deepseek41_prefill MODEL.gguf speed-bench/promessi_sposi.txt
```
The model test uses SSD streaming and two 128K sessions. Run it alone on a
dedicated host with at least 128 GiB RAM. It alternates small and large appends
through 113K context, checking dispatch, progress, unchanged-prefix reuse,
saved state and subsequent decoding against a control without decoder deferral.
Keep the official continuation checks too: this scheduling test does not judge
quality across different floating-point operation orders. Measure speed without
Metal API validation, separately for initial and continued prefills.
Run the same mixed-prefix test over physical RDMA on two dedicated Metal hosts.
Start the worker with the same model and a 131072-token context, then the test
on the coordinator (replace the host and RDMA device names):
```sh
# Worker
MTL_DEBUG_LAYER=1 ./ds4 -m MODEL.gguf --ctx 131072 \
--tensor-parallel --role worker --coordinator COORDINATOR 19455 \
--transport rdma --rdma-device WORKER_DEVICE --rdma-gid-index 1
# Coordinator
MTL_DEBUG_LAYER=1 ./tests/test_deepseek41_prefill --tensor-parallel \
MODEL.gguf speed-bench/promessi_sposi.txt COORDINATOR 19455 COORDINATOR_DEVICE 1
```
This compares equal TP prefill partitions and queued versus synchronous decode,
including full logits and cache state. TP snapshots rebuild both ranks from
tokens, so restoration is compared with an equivalent fresh replay. Also run
the official scorer in TP mode for initial and continued prompts; scheduling
parity alone does not establish model quality.
Add `--quality` to disable DS4's speed-oriented numerical shortcuts. For an
independent llama.cpp comparison of a DeepSeek V4 GGUF, use the same manifest
and the token-identical DS4 prompt renderer:
@@ -164,6 +280,25 @@ gguf-tools/quality-testing/score_llama \
deepseek-ds4
```
GLM's hosted endpoint requires reasoning. A no-thinking local prompt does not
match a reply generated after reasoning, even at temperature zero. For the long
GLM fixtures, render the official `Reasoning Effort: Low` template and include
the returned reasoning before scoring the answer:
```sh
python3 gguf-tools/quality-testing/render_glm_references.py \
gguf-tools/quality-testing/data/glm53-flash-openrouter-zai-fp8-long
gguf-tools/quality-testing/score_official /path/to/GLM-5.3-Flash-Q2.gguf \
gguf-tools/quality-testing/data/glm53-flash-openrouter-zai-fp8-long/manifest-rendered.tsv \
/tmp/glm-long.tsv 8192 --rendered-prompt
```
`--rendered-prompt` reads the prompt file with its existing chat markers; it
does not add another chat template. The renderer can also be applied to the
older 100-case GLM 5.3 Flash fixture. Keep rendered and legacy no-thinking
scores separate: their prefixes differ. Z.AI supplies no token logprobs, so
judge these runs by continuation NLL and greedy agreement, not logprob parity.
For a full-residency vs SSD-streaming comparison, score the same model twice and
add the streaming flags to one run:
@@ -191,7 +326,10 @@ Output fields:
- `first_token_matches`: how often the local greedy first token matches the
official first token.
- `avg_greedy_lcp`: average greedy longest common prefix against the official
continuation.
continuation. This measures exact agreement, not general answer quality;
one early mismatch discards all later agreement. Do not use it alone as a
quality gate, especially for sampled references. Compare paired NLL and
API probability agreement under matching execution settings as well.
- `api_target_mae`: when the manifest includes `response_file`, absolute
local-vs-API logprob delta for aligned official output tokens.
- `api_top_coverage`: fraction of API top-logprob alternatives that map exactly
@@ -202,3 +340,70 @@ Output fields:
local top-N for the same position.
- `api_top_mae`: local-vs-API logprob MAE over mapped API top alternatives.
- `api_pair_rate`: pairwise ordering agreement among mapped API alternatives.
## 6. Validate regression artifacts
Two standard-library Python tools check saved artifacts without running inference. For matched `score_official` runs with API logprob coverage, require the complete intended manifest and finite, consistent counts and scores:
```sh
python3 gguf-tools/quality-testing/validate_scores.py \
--manifest gguf-tools/quality-testing/data/flash/manifest.tsv \
--baseline /tmp/old.tsv --candidate /tmp/new.tsv --strict-identical > /tmp/scores.json
```
Exit 0 with `--strict-identical` requires every per-case value to match; omit it to report valid differences without deciding their acceptability. Unlike `compare_scores.py`, validation rejects missing/duplicate cases, malformed fields and changed coverage denominators. This stricter mode requires the current four-column manifest and scorer TSV format with nonzero API coverage; use the existing comparator for continuation-only references without API logprobs.
Compare complete vocabulary dumps from `ds4-bench --dump-frontier-logits-dir`:
```sh
python3 gguf-tools/quality-testing/compare_frontier_logits.py \
/tmp/old-logits /tmp/new-logits --frontiers 2048 4096 --ctx 8192 \
--model /models/model.gguf --backend rocm --quality false --quant-bits 2 --vocab 129280 \
--output /tmp/frontiers.json
```
Supply the exact expected metadata: `ctx` is allocated capacity; each frontier is total context, and prefill is its increase from the previous frontier (initially zero). Each directory must contain exactly the requested dumps. Exit 0 requires complete finite float32 bit identity; exit 1 reports drift, invalid artifacts or an operational error. JSON distinguishes `identical`, `drift` and `invalid`, with full-vector hashes, differing counts and maximum absolute differences. The output path must be new. The parser follows the writer's nine-significant-digit float format and preserves signed zero.
Matching argmax is insufficient: a measured 8K–64K comparison retained all four argmax IDs while full vectors differed, with maximum absolute difference 7.45228. This is numerical drift, not by itself proof of worse generation quality. Record prompt/weight hashes, source/build identity, settings and successful run completion separately; these tools cannot recover that provenance from score tables or dumps, and do not replace model-quality checks.
Run the host-only positive and negative controls:
```sh
python3 -m unittest discover -s gguf-tools/quality-testing/tests -v
```
## 7. Qwen3.8 Spark Baseline
Measured on 2026-09-15, using one DGX Spark per quant. The short set has 100
cases and 5,696 target tokens; the long set has 12 cases and 766 target tokens,
with prefixes from 1,971 to 23,985 tokens. These are ordinary decoding and
prefill checks, without MTP or vision.
| Suite | GGUF | API top-token agreement | NLL, default | NLL, `--quality` |
| --- | --- | ---: | ---: | ---: |
| Short | Q2 | 88.90% | 0.352980 | 0.352915 |
| Short | Q4 | 92.08% | 0.290297 | 0.290538 |
| Long | Q2 | 93.99% | 0.154199 | 0.154106 |
| Long | Q4 | 97.26% | 0.124611 | 0.125546 |
Agreement is measured after feeding the same official continuation prefix,
not as a percentage of correct answers. The two short cases with damaged
emoji metadata contribute to NLL but not API agreement, which covers 5,568
short-set tokens. All 766 long-set tokens align. Every local prompt and target
token ID was also checked against Qwen's published tokenizer.
The default fast path shows no material loss relative to `--quality` in these
sets. The paired case-bootstrap intervals include zero for both quants and
both suites. Q4 is closer to the hosted reference than Q2 on average.
These measurements do not prove identical native weights or correctness at
every context length.
Leaving the last 1 or 256 prompt tokens for a separate continued prefill
changed aggregate NLL by less than 0.0008 for either quant. Independent CLI
answers at about 2K and 24K context also retrieved the correct archive owner
and revision, and gave the correct clamp outputs and boundary tests, for
both Q2 and Q4.
Raw per-case scores, weight/source hashes and comparison details are retained
in [short results](data/qwen38-flash-alibaba-100/cuda-spark-20260915/results.json)
and [long results](data/qwen38-flash-alibaba-long/cuda-spark-20260915/results.json).
gguf-tools/quality-testing/collect_official.py
+47 -2
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import argparse
import json
import math
import os
import sys
import time
@@ -134,13 +135,17 @@ def request_one(
provider_order: list[str],
provider_allow_fallbacks: bool,
provider_require_parameters: bool,
temperature: float = 0,
system: str | None = None,
) -> dict:
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
"temperature": temperature,
"stream": False,
}
if system is not None:
payload["messages"].insert(0, {"role": "system", "content": system})
if top_logprobs > 0:
payload["logprobs"] = True
payload["top_logprobs"] = top_logprobs
@@ -181,6 +186,8 @@ def fetch_with_retry(
provider_order: list[str],
provider_allow_fallbacks: bool,
provider_require_parameters: bool,
temperature: float = 0,
system: str | None = None,
) -> dict:
delay = 1.0
for attempt in range(6):
@@ -198,6 +205,8 @@ def fetch_with_retry(
provider_order,
provider_allow_fallbacks,
provider_require_parameters,
temperature,
system,
)
except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", "replace")
@@ -218,11 +227,17 @@ def main() -> int:
ap.add_argument("--out", default="gguf-tools/quality-testing/data")
ap.add_argument("--prompts", default="gguf-tools/quality-testing/prompts.jsonl")
ap.add_argument("--model", default=MODEL)
ap.add_argument("--system", default=None,
help="explicit system message, including an empty string; "
"some providers insert a default when this is omitted")
ap.add_argument("--endpoint", default=ENDPOINT)
ap.add_argument("--api-key-env", default=None)
ap.add_argument("--count", type=int, default=100)
ap.add_argument("--max-tokens", type=int, default=24)
ap.add_argument("--top-logprobs", type=int, default=5)
ap.add_argument("--temperature", type=float, default=0,
help="sampling temperature, 0..2; use 1 for unscaled logprobs "
"from providers reporting probabilities after sampling transforms")
ap.add_argument("--thinking", choices=("disabled", "enabled", "omit"), default="disabled")
ap.add_argument("--reasoning-effort",
choices=("xhigh", "high", "medium", "low", "minimal", "none", "omit"),
@@ -235,11 +250,17 @@ def main() -> int:
ap.add_argument("--allow-provider-fallbacks", action="store_true")
ap.add_argument("--require-parameters", action="store_true",
help="for OpenRouter, route only to endpoints advertising the requested parameters")
ap.add_argument("--require-response-model",
help="reject a response that reports a different model ID")
ap.add_argument("--require-response-provider",
help="reject a response from a different provider")
ap.add_argument("--resume", action="store_true",
help="reuse complete cases already present in the output directory")
args = ap.parse_args()
if args.top_logprobs < 0 or args.top_logprobs > 20:
raise SystemExit("--top-logprobs must be between 0 and 20")
if not math.isfinite(args.temperature) or not 0 <= args.temperature <= 2:
raise SystemExit("--temperature must be between 0 and 2")
openrouter = "openrouter.ai" in args.endpoint
api_key_env = args.api_key_env or ("OPENROUTER_API_KEY" if openrouter else "DEEPSEEK_API_KEY")
@@ -265,6 +286,12 @@ def main() -> int:
(out / "prompts").mkdir(parents=True, exist_ok=True)
(out / "continuations").mkdir(parents=True, exist_ok=True)
(out / "responses").mkdir(parents=True, exist_ok=True)
system_path = out / "request_system.json"
if args.resume and (system_path.exists() or any((out / "responses").glob("*.json"))):
saved_system = json.loads(system_path.read_text()) if system_path.exists() else None
if saved_system != args.system:
raise RuntimeError("saved system message differs from --system; use a new output directory")
system_path.write_text(json.dumps(args.system, ensure_ascii=False) + "\n", encoding="utf-8")
manifest = out / "manifest.tsv"
rows = []
@@ -308,6 +335,8 @@ def main() -> int:
provider_order,
args.allow_provider_fallbacks,
provider_require_parameters,
args.temperature,
args.system,
)
choice = response["choices"][0]
content = choice.get("message", {}).get("content")
@@ -319,6 +348,19 @@ def main() -> int:
else:
raise RuntimeError(f"{case_id}: provider returned three empty continuations")
choice = response["choices"][0]
if (args.require_response_model and
response.get("model") != args.require_response_model):
raise RuntimeError(
f"{case_id}: expected model {args.require_response_model!r}, "
f"got {response.get('model')!r}"
)
if (args.require_response_provider and
response.get("provider") != args.require_response_provider):
raise RuntimeError(
f"{case_id}: expected provider "
f"{args.require_response_provider!r}, got "
f"{response.get('provider')!r}"
)
if response.get("model"):
observed_models.add(response["model"])
if response.get("provider"):
@@ -342,10 +384,11 @@ def main() -> int:
"schema": "ds4-official-continuations-v1",
"created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"model": args.model,
"system": args.system,
"endpoint": args.endpoint,
"observed_models": sorted(observed_models),
"observed_providers": sorted(observed_providers),
"temperature": 0,
"temperature": args.temperature,
"max_tokens": args.max_tokens,
"top_logprobs": args.top_logprobs,
"thinking": thinking,
@@ -354,6 +397,8 @@ def main() -> int:
"provider_order": provider_order,
"allow_provider_fallbacks": args.allow_provider_fallbacks,
"require_parameters": provider_require_parameters,
"required_response_model": args.require_response_model,
"required_response_provider": args.require_response_provider,
"case_count": len(rows),
}
(out / "collection.json").write_text(
Loaded 100 of 1820 files, more files were not shown because too many files have changed in this diff. Show more