Tune Laguna sampling defaults

Load Poolside's standalone DFlash GGUF, draft and verify greedy Laguna tokens on Metal, expose CLI/agent/server controls, pause speculation when measured throughput loses, and sanitize/test non-finite target features.

.gitattributes

@@ -0,0 +1,3 @@
+# Preserve hosted-model continuation bytes exactly; these fixtures may contain
+# model-generated trailing whitespace that is meaningful to tokenizer scoring.
+gguf-tools/quality-testing/data/*/continuations/*.txt -diff

.gitignore

@@ -1,14 +1,35 @@
 /ds4
 /ds4-server
 /ds4-bench
+/ds4-eval
+/ds4-agent
 /ds4_native
+/ds4_agent_test
 /ds4_server_test
 /ds4_test
 /ds4flash.gguf
 /TODO.md
 /gguf/
+/core
+/core.*
+/tests/test_engine_mgpu_placement
+/tests/test_engine_mgpu_runtime
+/tests/test_engine_mgpu_refusal
+/tests/test_engine_correctness
+/tests/test_gpu_args
+/tests/test_gpu_lookup_cache_strict
+/tests/test_gpu_model_cache
+/tests/test_gpu_xdev
+/tests/test_cuda_mixed_batch
+/tests/test_cuda_session_batch
+/tests/test_layer_pack
+/tests/test_metal_session_batch
+/tests/test_q4k_dot
+/tests/test_sampling
 *.o
 *.dSYM/
+__pycache__/
+*.pyc
 /misc/
 .*.swp
 .DS_Store

AGENT.md

@@ -7,32 +7,26 @@ Objective-C only where Metal requires it and Metal kernels under `metal/`.
 ## Goals
 
 - Keep the production path as whole-model Metal graph inference.
-- Keep model loading mmap-backed; do not eagerly copy the full GGUF.
+- Always make sure that the SSD streaming, CUDA, distributed inference, Metal default inference are not affected by fixes to other parts of the code.
+- Keep model loading mmap-backed for the Metal default case; do not eagerly copy the full GGUF. Keep the model loading for SSD streaming of routed experts explicit: allocated buffers, fast reads from disk, always try to hide loading of missing routed experts by loading them while performing the inference of the shared expert and routed experts already in RAM. Always try to hide loading of layers for prefill in SSD streaming mode using the inference time of the current layer as the next one is loaded.
 - Keep the CPU backend CPU-only and use it only as reference/debug code.
-- Preserve correctness before speed. Do not keep a faster path with unexplained
-  attention, KV cache, or logits drift.
-- Make long local agent sessions practical through live KV reuse and disk KV
-  checkpoints.
+- Preserve correctness before speed. Do not keep a faster path with unexplained attention, KV cache, or logits drift.
+- Make long local agent sessions practical through live KV reuse and disk KV checkpoints.
 
 ## Quality Rules
 
-- Comment important inference code where the model mechanics, cache lifetime,
-  memory policy, or API orchestration are not obvious from the local code.
+- Keep the implementation small, sharp, easy to understand. Try to write elegant code in a state of grace. Don't settle for the first thing that comes to mind, try to find the most minimal and better working design. Don't introduce slop: very fragile code that just patches specific cases, dead code, useless code and code ways more complicated of how it should be.
+- Comment important inference code where the model mechanics, cache lifetime, memory policy, or API orchestration are not obvious from the local code.
 - Prefer comments beside the implementation over separate design documents.
-- Keep comments instructive and compact: explain why a shape, ordering, cache
-  boundary, or memory choice exists.
+- Keep comments instructive and compact: explain why a shape, ordering, cache boundary, or memory choice exists.
 - Keep public APIs narrow. CLI/server code should not know tensor internals.
-- Do not add permanent semantic variants behind flags. Diagnostic switches are
-  fine when they validate the one release path.
+- Do not add permanent semantic variants behind flags. Diagnostic switches are fine when they validate the one release path.
 - Do not introduce C++.
 
 ## Safety
 
-- Avoid large CPU inference runs on macOS; the CPU path has previously exposed
-  kernel VM failures with very large mappings.
-- Do not run multiple huge model processes concurrently. The instance lock is
-  intentional.
-- Prefer short Metal smoke tests for build verification.
+- Avoid large CPU inference runs on macOS; the CPU path has previously exposed kernel VM failures with very large mappings.
+- Do not run multiple huge model processes concurrently. The instance lock is intentional.
 
 ## Layout
 
@@ -46,8 +40,17 @@ Objective-C only where Metal requires it and Metal kernels under `metal/`.
 - `tests/`: unit and live integration tests.
 - `misc/`: ignored notes, experiments, and old planning material.
 
+This list is not complete, check the files for more info.
+
 ## Testing
 
 Use `make` for build validation. Use `make test` for unit/regression tests when a
 model and Metal are available. Use live server tests only when intentionally
 testing the API surface.
+
+At every major change where one of the following could be affected, make sure to:
+
+1. Test the normal Metal path and that speed is still at the level it was.
+2. Test the SSD streaming path.
+3. Test the distributed inference if it could be affected, but ask the user before doing so.
+4. Check if CUDA could be broken after the change, and ask the user to give you access to the CUDA machine to actually test if everything is still fine.

MODEL_CARD.md

@@ -0,0 +1,243 @@
+# DeepSeek v4 model card synopsis
+
+This document extracts the most important information from the official
+DeepSeek-V4-Flash Hugging Face model card, with emphasis on facts that matter
+for local inference, DS4 development, and benchmark interpretation.
+
+Source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash
+
+## Model Family
+
+DeepSeek-V4 is a preview model family with two Mixture-of-Experts language
+models:
+
+| Model | Total parameters | Active parameters | Context length |
+|---|---:|---:|---:|
+| DeepSeek-V4-Flash | 284B | 13B | 1M tokens |
+| DeepSeek-V4-Pro | 1.6T | 49B | 1M tokens |
+
+Flash is the smaller and more efficient model. The model card says Flash-Max can
+approach Pro reasoning performance when given a larger thinking budget, while
+remaining behind Pro on pure knowledge and the most complex agentic tasks.
+
+## Architecture
+
+DeepSeek-V4 uses long-context compressed attention. The model card calls the
+hybrid design Compressed Sparse Attention (CSA) plus Heavily Compressed
+Attention (HCA). In DS4 terms, each layer keeps a raw sliding-window KV cache
+for the latest 128 tokens. This is the high-resolution local context.
+
+After that raw window, the model uses layer-dependent compressed KV rows:
+
+| 0-based layer indexes | DS4 ratio | Extra state | Meaning |
+|---|---:|---|---|
+| 0, 1 | none | none | Raw 128-token sliding window only |
+| even layers from 2 onward | 4 | compressed KV + indexer KV | One compressed row per 4 tokens, with an indexer selecting visible compressed rows |
+| odd layers from 3 onward | 128 | compressed KV | One compressed row per 128 tokens |
+
+So, after the first two layers, the model alternates ratio-4 and ratio-128
+compressed attention. A token in a compressed layer attends over both the raw
+latest-128-token window and the older compressed history. The compression here
+is time-axis compression: several token positions are pooled into one KV row.
+The attention rows still use the model attention/value dimensions, so raw and
+compressed rows can be consumed by the same mixed-attention computation.
+
+Ratio-4 layers are the selective compressed-attention layers. They maintain a
+second compressed stream for the indexer, and when the compressed history is
+larger than the configured top-k, DS4 scores the compressed rows and selects up
+to 512 of them for attention. Ratio-128 layers are the heavily compressed path:
+they do not have the indexer stream and use the available ratio-128 compressed
+rows directly.
+
+DS4 validates these details from the GGUF metadata. The relevant fixed
+implementation constants are:
+
+- Layers: 43
+- Raw sliding-window attention: 128 tokens
+- Indexer heads: 64
+- Indexer head dimension: 128
+- Indexer top-k: 512
+
+This is the practical reason the model can expose a 1M-token context without a
+standard full KV cache for every token in every layer. The model card reports
+that, at 1M tokens, DeepSeek-V4-Pro needs much less single-token inference
+compute and KV cache than DeepSeek-V3.2.
+
+The family also uses:
+
+- Manifold-Constrained Hyper-Connections (mHC), intended to improve signal
+  propagation stability across layers.
+- The Muon optimizer, used for faster convergence and training stability.
+- A post-training pipeline with domain expert cultivation followed by unified
+  consolidation via on-policy distillation.
+
+## Precision And Weights
+
+Official download entries include:
+
+| Model | Precision |
+|---|---|
+| DeepSeek-V4-Flash-Base | FP8 Mixed |
+| DeepSeek-V4-Flash | FP4 + FP8 Mixed |
+| DeepSeek-V4-Pro-Base | FP8 Mixed |
+| DeepSeek-V4-Pro | FP4 + FP8 Mixed |
+
+For the instruct models, the model card describes FP4 + FP8 Mixed as using FP4
+for MoE expert parameters and FP8 for most other parameters.
+
+## Reasoning Modes
+
+The instruct models support three reasoning-effort modes:
+
+| Mode | Intended behavior | Output shape |
+|---|---|---|
+| Non-think | Fast, intuitive replies | `</think>` summary |
+| High | Deliberate reasoning for harder tasks | `<think>... </think>` summary |
+| Max | Largest reasoning budget | Special system prompt plus thinking and summary |
+
+The model card recommends using at least a 384K-token context window for Think
+Max.
+
+## Important Flash Benchmarks
+
+### DeepSeek-V4-Flash Across Reasoning Modes
+
+| Benchmark | Non-Think | High | Max |
+|---|---:|---:|---:|
+| GPQA Diamond Pass@1 | 71.2 | 87.4 | 88.1 |
+| MMLU-Pro EM | 83.0 | 86.4 | 86.2 |
+| SimpleQA-Verified Pass@1 | 23.1 | 28.9 | 34.1 |
+| Chinese-SimpleQA Pass@1 | 71.5 | 73.2 | 78.9 |
+| HLE Pass@1 | 8.1 | 29.4 | 34.8 |
+| LiveCodeBench Pass@1 | 55.2 | 88.4 | 91.6 |
+| HMMT 2026 Feb Pass@1 | 40.8 | 91.9 | 94.8 |
+| IMOAnswerBench Pass@1 | 41.9 | 85.1 | 88.4 |
+| SWE Verified Resolved | 73.7 | 78.6 | 79.0 |
+| Terminal Bench 2.0 Acc | 49.1 | 56.6 | 56.9 |
+| MCPAtlas Pass@1 | 64.0 | 67.4 | 69.0 |
+| Toolathlon Pass@1 | 40.7 | 43.5 | 47.8 |
+
+### DeepSeek-V4-Flash-Base
+
+The base-model table reports these Flash-Base scores:
+
+| Benchmark | Shots | Score |
+|---|---:|---:|
+| SuperGPQA EM | 5-shot | 46.5 |
+| MMLU EM | 5-shot | 88.7 |
+| MMLU-Pro EM | 5-shot | 68.3 |
+| Simple-QA verified EM | 25-shot | 30.1 |
+| HumanEval Pass@1 | 0-shot | 69.5 |
+| GSM8K EM | 8-shot | 90.8 |
+| LongBench-V2 EM | 1-shot | 44.7 |
+
+The model card reports SuperGPQA for the base model table, not in the instruct
+reasoning-mode comparison table.
+
+## Chat Template And Encoding
+
+The release does not use a Jinja chat template as the source of truth. The
+official prompt renderer is the Python code in
+`encoding/encoding_dsv4.py`, with examples and tests in the same `encoding`
+directory:
+
+- https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/raw/main/encoding/encoding_dsv4.py
+- https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/raw/main/encoding/test_encoding_dsv4.py
+
+The important special tokens are:
+
+| Purpose | Token |
+|---|---|
+| Beginning of sequence | `<｜begin▁of▁sentence｜>` |
+| End of assistant turn | `<｜end▁of▁sentence｜>` |
+| User turn prefix | `<｜User｜>` |
+| Assistant turn prefix | `<｜Assistant｜>` |
+| Latest reminder prefix | `<｜latest_reminder｜>` |
+| Thinking start | `<think>` |
+| Thinking end / non-thinking marker | `</think>` |
+| DSML tool markup marker | `｜DSML｜` |
+
+The renderer accepts `system`, `user`, `assistant`, `tool`,
+`latest_reminder`, and `developer` roles. The `developer` role is described in
+the Python comments as an internal search-agent role, not as a normal public
+chat role.
+
+Normal chat mode starts with the BOS token, then system text if present, then
+alternating user and assistant markers. In non-thinking chat mode, a new
+assistant generation is opened with:
+
+```text
+<｜Assistant｜></think>
+```
+
+That immediate `</think>` tells the model to skip hidden reasoning and produce
+the visible answer. In thinking mode, a new assistant generation is opened with:
+
+```text
+<｜Assistant｜><think>
+```
+
+Completed assistant thinking turns are rendered as reasoning content inside
+`<think>...</think>`, followed by the visible answer and the EOS token.
+
+By default, the Python renderer drops earlier assistant reasoning content before
+the last user message. If tools are present on any message, it disables that
+reasoning drop and keeps the full reasoning/tool context. `reasoning_effort=max`
+also prepends a special high-effort instruction prefix before the first rendered
+message in thinking mode.
+
+Tool definitions are passed in OpenAI-compatible function schema form, but the
+model is instructed to emit DSML. A tool call is rendered as a DSML
+`tool_calls` block containing one or more `invoke` entries, each with named
+parameters. Parameters carry a `string="true"` flag for raw strings and
+`string="false"` for JSON values such as numbers, booleans, arrays, or objects.
+
+DeepSeek-V4 does not render standalone `tool` role messages. The Python
+preprocessor converts tool results into user content blocks and renders each
+result as:
+
+```text
+<tool_result>...</tool_result>
+```
+
+Tool-result bodies are rendered as raw text. Literal `<`, `>`, and `&` from
+file contents or shell output are preserved; only the exact closing sentinel
+`</tool_result>` is escaped so the wrapper cannot be terminated by data.
+
+When there are multiple tool results, the renderer sorts them to match the
+order of the preceding assistant tool calls.
+
+The same script also defines special task tokens for internal quick tasks such
+as title generation, search-query generation, action selection, authority
+classification, domain classification, and URL-read decisions. Those are
+separate from normal chat/tool rendering.
+
+## Local Running Notes
+
+The model card lists vLLM and SGLang examples for OpenAI-compatible serving.
+For local deployment, it recommends:
+
+- `temperature = 1.0`
+- `top_p = 1.0`
+- At least 384K context for Think Max
+
+These are deployment recommendations from the model card, not necessarily the
+same settings used for deterministic benchmarking. DS4 keeps `top_p=1.0` but
+adds a local `min_p=0.05` default to avoid sampling tokens whose probability is
+far below the best token.
+
+## Licensing
+
+The repository and model weights are licensed under the MIT License.
+
+## Citation
+
+The model card cites:
+
+```bibtex
+@misc{deepseekai2026deepseekv4,
+      title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
+      author={DeepSeek-AI},
+      year={2026},
+}
+```

Makefile

@@ -3,20 +3,29 @@ UNAME_S := $(shell uname -s)
 
 ifeq ($(UNAME_S),Darwin)
 NATIVE_CPU_FLAG ?= -mcpu=native
+SAMPLING_TEST :=
 else
 NATIVE_CPU_FLAG ?= -march=native
+SAMPLING_TEST := tests/test_sampling
 endif
 
-CFLAGS ?= -O3 -ffast-math $(NATIVE_CPU_FLAG) -Wall -Wextra -std=c99
-OBJCFLAGS ?= -O3 -ffast-math $(NATIVE_CPU_FLAG) -Wall -Wextra -fobjc-arc
+DEBUG_FLAGS ?= -g
+CFLAGS ?= -O3 -ffast-math $(DEBUG_FLAGS) $(NATIVE_CPU_FLAG) -Wall -Wextra -std=c99
+OBJCFLAGS ?= -O3 -ffast-math $(DEBUG_FLAGS) $(NATIVE_CPU_FLAG) -Wall -Wextra -fobjc-arc
+QUALITY_CFLAGS ?= -O3 $(DEBUG_FLAGS) $(NATIVE_CPU_FLAG) -Wall -Wextra -std=c11
 
 LDLIBS ?= -lm -pthread
 METAL_SRCS := $(wildcard metal/*.metal)
+ROCM_SRCS := $(wildcard rocm/*.cuh)
+DS4_TEST_MODEL ?= ds4flash.gguf
+DS4_TEST_MTP ?= gguf/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf
+DS4_DSPARK_MODEL ?= $(DS4_TEST_MODEL)
+DS4_DSPARK_SUPPORT ?= gguf/DeepSeek-V4-Flash-DSpark-support.gguf
 
 ifeq ($(UNAME_S),Darwin)
 METAL_LDLIBS := $(LDLIBS) -framework Foundation -framework Metal
-CORE_OBJS = ds4.o ds4_metal.o
-CPU_CORE_OBJS = ds4_cpu.o
+CORE_OBJS = ds4.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_metal.o ds4_layer_pack.o
+CPU_CORE_OBJS = ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
 CFLAGS += -D_GNU_SOURCE -fno-finite-math-only
 CUDA_HOME ?= /usr/local/cuda
@@ -25,38 +34,66 @@ CUDA_ARCH ?=
 ifneq ($(strip $(CUDA_ARCH)),)
 NVCC_ARCH_FLAGS := -arch=$(CUDA_ARCH)
 endif
-NVCCFLAGS ?= -O3 --use_fast_math $(NVCC_ARCH_FLAGS) -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
+NVCCFLAGS ?= -O3 -g -lineinfo --use_fast_math $(NVCC_ARCH_FLAGS) -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
+CORE_OBJS = ds4.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_cuda.o ds4_layer_pack.o
+CPU_CORE_OBJS = ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 CUDA_LDLIBS ?= -lm -Xcompiler -pthread -L$(CUDA_HOME)/targets/sbsa-linux/lib -L$(CUDA_HOME)/lib64 -lcudart -lcublas
-CORE_OBJS = ds4.o ds4_cuda.o
-CPU_CORE_OBJS = ds4_cpu.o
+HIPCC ?= $(shell command -v hipcc 2>/dev/null || echo /opt/rocm/bin/hipcc)
+ROCM_ARCH ?= gfx1151
+ROCM_CFLAGS ?= -O3 -ffast-math -g -fno-finite-math-only -pthread -D__HIP_PLATFORM_AMD__ -Wno-unused-command-line-argument --offload-arch=$(ROCM_ARCH)
+ROCM_LDLIBS ?= -lm -pthread -lhipblas -lhipblaslt
+DS4_LINK ?= $(NVCC) $(NVCCFLAGS)
+DS4_LINK_LIBS ?= $(CUDA_LDLIBS)
 METAL_LDLIBS := $(LDLIBS)
 endif
 
-.PHONY: all help clean test cpu cuda cuda-spark cuda-generic cuda-regression
+.PHONY: all help clean test test-metal-session-batch test-cuda-session-batch test-cuda-mixed-batch dspark-acceptance dspark-verify-depth mtp-verify-depth cpu cuda cuda-spark cuda-generic cuda-regression strix-halo rocm
 
 ifeq ($(UNAME_S),Darwin)
-all: ds4 ds4-server ds4-bench
+all: ds4 ds4-server ds4-bench ds4-eval ds4-agent
 
 help:
 	@echo "DS4 build targets:"
-	@echo "  make              Build Metal ./ds4, ./ds4-server, and ./ds4-bench"
-	@echo "  make cpu          Build CPU-only ./ds4, ./ds4-server, and ./ds4-bench"
+	@echo "  make              Build Metal ./ds4, ./ds4-server, ./ds4-bench, ./ds4-eval, and ./ds4-agent"
+	@echo "  make cpu          Build CPU-only ./ds4, ./ds4-server, ./ds4-bench, ./ds4-eval, and ./ds4-agent"
 	@echo "  make test         Build and run tests"
+	@echo "  make dspark-verify-depth  Run DSpark speculative verification smoke if support GGUF is present"
+	@echo "  make mtp-verify-depth  Run legacy MTP speculative verification smoke if MTP GGUF is present"
 	@echo "  make clean        Remove build outputs"
 
-ds4: ds4_cli.o linenoise.o $(CORE_OBJS)
-	$(CC) $(CFLAGS) -o $@ ds4_cli.o linenoise.o $(CORE_OBJS) $(METAL_LDLIBS)
+ds4: ds4_cli.o ds4_help.o linenoise.o ds4_gpu_args.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_cli.o ds4_help.o linenoise.o ds4_gpu_args.o $(CORE_OBJS) $(METAL_LDLIBS)
 
-ds4-server: ds4_server.o rax.o $(CORE_OBJS)
-	$(CC) $(CFLAGS) -o $@ ds4_server.o rax.o $(CORE_OBJS) $(METAL_LDLIBS)
+ds4-server: ds4_server.o ds4_help.o ds4_kvstore.o rax.o ds4_gpu_args.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_server.o ds4_help.o ds4_kvstore.o rax.o ds4_gpu_args.o $(CORE_OBJS) $(METAL_LDLIBS)
 
-ds4-bench: ds4_bench.o $(CORE_OBJS)
-	$(CC) $(CFLAGS) -o $@ ds4_bench.o $(CORE_OBJS) $(METAL_LDLIBS)
+ds4-bench: ds4_bench.o ds4_help.o ds4_gpu_args.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_bench.o ds4_help.o ds4_gpu_args.o $(CORE_OBJS) $(METAL_LDLIBS)
 
-cpu: ds4_cli_cpu.o ds4_server_cpu.o ds4_bench_cpu.o linenoise.o rax.o $(CPU_CORE_OBJS)
-	$(CC) $(CFLAGS) -o ds4 ds4_cli_cpu.o linenoise.o $(CPU_CORE_OBJS) $(LDLIBS)
-	$(CC) $(CFLAGS) -o ds4-server ds4_server_cpu.o rax.o $(CPU_CORE_OBJS) $(LDLIBS)
-	$(CC) $(CFLAGS) -o ds4-bench ds4_bench_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+ds4-eval: ds4_eval.o ds4_help.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_eval.o ds4_help.o $(CORE_OBJS) $(METAL_LDLIBS)
+
+ds4-agent: ds4_agent.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o ds4_gpu_args.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_agent.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o ds4_gpu_args.o $(CORE_OBJS) $(METAL_LDLIBS)
+
+gguf-tools/quality-testing/score_official: gguf-tools/quality-testing/score_official.c ds4.h $(CORE_OBJS) rax.o
+	$(CC) $(QUALITY_CFLAGS) -I. -o $@ gguf-tools/quality-testing/score_official.c $(CORE_OBJS) rax.o $(METAL_LDLIBS)
+
+tests/test_metal_session_batch.o: tests/test_metal_session_batch.c ds4.h
+	$(CC) $(CFLAGS) -I. -c -o $@ tests/test_metal_session_batch.c
+
+tests/test_metal_session_batch: tests/test_metal_session_batch.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ $^ $(METAL_LDLIBS)
+
+test-metal-session-batch: tests/test_metal_session_batch
+	DS4_TEST_MODEL="$(DS4_TEST_MODEL)" ./tests/test_metal_session_batch
+
+cpu: ds4_cli_cpu.o ds4_server_cpu.o ds4_bench_cpu.o ds4_eval_cpu.o ds4_agent_cpu.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o rax.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS)
+	$(CC) $(CFLAGS) -o ds4 ds4_cli_cpu.o ds4_help.o linenoise.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-server ds4_server_cpu.o ds4_help.o ds4_kvstore.o rax.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-bench ds4_bench_cpu.o ds4_help.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-eval ds4_eval_cpu.o ds4_help.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-agent ds4_agent_cpu.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
 
 cuda-regression:
 	@echo "cuda-regression requires a CUDA build"
@@ -68,15 +105,19 @@ help:
 	@echo "  make cuda-spark          Build CUDA for DGX Spark / GB10"
 	@echo "  make cuda-generic        Build CUDA for a generic local CUDA GPU"
 	@echo "  make cuda CUDA_ARCH=sm_N Build CUDA with an explicit nvcc -arch value"
-	@echo "  make cpu                 Build CPU-only ./ds4, ./ds4-server, and ./ds4-bench"
+	@echo "  make strix-halo          Build ROCm for Strix Halo / gfx1151"
+	@echo "  make rocm                Alias for make strix-halo"
+	@echo "  make cpu                 Build CPU-only ./ds4, ./ds4-server, ./ds4-bench, ./ds4-eval, and ./ds4-agent"
 	@echo "  make test                Build and run tests"
+	@echo "  make dspark-verify-depth Run DSpark speculative verification smoke if support GGUF is present"
+	@echo "  make mtp-verify-depth    Run legacy MTP speculative verification smoke if MTP GGUF is present"
 	@echo "  make clean               Remove build outputs"
 
 cuda-spark:
-	$(MAKE) ds4 ds4-server ds4-bench CUDA_ARCH=
+	$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent CUDA_ARCH=
 
 cuda-generic:
-	$(MAKE) ds4 ds4-server ds4-bench CUDA_ARCH=native
+	$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent CUDA_ARCH=native
 
 cuda:
 	@if [ -z "$(strip $(CUDA_ARCH))" ]; then \
@@ -84,41 +125,91 @@ cuda:
 		echo "       or use make cuda-spark / make cuda-generic"; \
 		exit 2; \
 	fi
-	$(MAKE) ds4 ds4-server ds4-bench CUDA_ARCH="$(CUDA_ARCH)"
+	$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent CUDA_ARCH="$(CUDA_ARCH)"
 
-ds4: ds4_cli.o linenoise.o $(CORE_OBJS)
-	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+strix-halo:
+	$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent \
+		CORE_OBJS="ds4.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_rocm.o ds4_rocm_compat.o ds4_rocm_unavailable.o ds4_layer_pack.o" \
+		CFLAGS="$(CFLAGS) -DDS4_ROCM_BUILD" \
+		DS4_LINK="$(HIPCC) $(ROCM_CFLAGS)" \
+		DS4_LINK_LIBS="$(ROCM_LDLIBS)"
 
-ds4-server: ds4_server.o rax.o $(CORE_OBJS)
-	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+rocm: strix-halo
 
-ds4-bench: ds4_bench.o $(CORE_OBJS)
-	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+ds4: ds4_cli.o ds4_help.o linenoise.o ds4_gpu_args.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
 
-cpu: ds4_cli_cpu.o ds4_server_cpu.o ds4_bench_cpu.o linenoise.o rax.o $(CPU_CORE_OBJS)
-	$(CC) $(CFLAGS) -o ds4 ds4_cli_cpu.o linenoise.o $(CPU_CORE_OBJS) $(LDLIBS)
-	$(CC) $(CFLAGS) -o ds4-server ds4_server_cpu.o rax.o $(CPU_CORE_OBJS) $(LDLIBS)
-	$(CC) $(CFLAGS) -o ds4-bench ds4_bench_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+ds4-server: ds4_server.o ds4_help.o ds4_kvstore.o rax.o ds4_gpu_args.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
+
+ds4-bench: ds4_bench.o ds4_help.o ds4_gpu_args.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
+
+ds4-eval: ds4_eval.o ds4_help.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
+
+ds4-agent: ds4_agent.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o ds4_gpu_args.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
+
+gguf-tools/quality-testing/score_official: gguf-tools/quality-testing/score_official.c ds4.h $(CORE_OBJS) rax.o
+	$(DS4_LINK) $(filter-out -ffast-math,$(QUALITY_CFLAGS)) -I. -o $@ gguf-tools/quality-testing/score_official.c $(CORE_OBJS) rax.o $(DS4_LINK_LIBS)
+
+cpu: ds4_cli_cpu.o ds4_server_cpu.o ds4_bench_cpu.o ds4_eval_cpu.o ds4_agent_cpu.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o rax.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS)
+	$(CC) $(CFLAGS) -o ds4 ds4_cli_cpu.o ds4_help.o linenoise.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-server ds4_server_cpu.o ds4_help.o ds4_kvstore.o rax.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-bench ds4_bench_cpu.o ds4_help.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-eval ds4_eval_cpu.o ds4_help.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-agent ds4_agent_cpu.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o ds4_gpu_args_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
 
 cuda-regression: tests/cuda_long_context_smoke
 	./tests/cuda_long_context_smoke
 endif
 
-ds4.o: ds4.c ds4.h ds4_gpu.h
+ds4.o: ds4.c ds4.h ds4_ssd.h ds4_distributed.h ds4_gpu.h
 	$(CC) $(CFLAGS) -c -o $@ ds4.c
 
-ds4_cli.o: ds4_cli.c ds4.h linenoise.h
+ds4_ssd.o: ds4_ssd.c ds4_ssd.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_ssd.c
+
+ds4_cli.o: ds4_cli.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h linenoise.h
 	$(CC) $(CFLAGS) -c -o $@ ds4_cli.c
 
-ds4_server.o: ds4_server.c ds4.h rax.h
+ds4_distributed.o: ds4_distributed.c ds4_distributed.h ds4.h ds4_ssd.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_distributed.c
+
+ds4_tp.o: ds4_tp.c ds4_tp.h ds4.h ds4_ssd.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_tp.c
+
+ds4_help.o: ds4_help.c ds4_help.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_help.c
+
+ds4_gpu_args.o: ds4_gpu_args.c ds4_gpu_args.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_gpu_args.c
+
+ds4_server.o: ds4_server.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h rax.h
 	$(CC) $(CFLAGS) -c -o $@ ds4_server.c
 
-ds4_bench.o: ds4_bench.c ds4.h
+ds4_bench.o: ds4_bench.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h
 	$(CC) $(CFLAGS) -c -o $@ ds4_bench.c
 
-ds4_test.o: tests/ds4_test.c ds4_server.c ds4.h rax.h
+ds4_eval.o: ds4_eval.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_eval.c
+
+ds4_agent.o: ds4_agent.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h ds4_web.h linenoise.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_agent.c
+
+ds4_web.o: ds4_web.c ds4_web.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_web.c
+
+ds4_kvstore.o: ds4_kvstore.c ds4_kvstore.h ds4.h ds4_ssd.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_kvstore.c
+
+ds4_test.o: tests/ds4_test.c ds4_server.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h rax.h
 	$(CC) $(CFLAGS) -Wno-unused-function -c -o $@ tests/ds4_test.c
 
+ds4_agent_test.o: tests/ds4_agent_test.c ds4_agent.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h ds4_web.h linenoise.h
+	$(CC) $(CFLAGS) -Wno-unused-function -c -o $@ tests/ds4_agent_test.c
+
 tests/cuda_long_context_smoke.o: tests/cuda_long_context_smoke.c ds4_gpu.h
 	$(CC) $(CFLAGS) -I. -c -o $@ tests/cuda_long_context_smoke.c
 
@@ -128,36 +219,188 @@ rax.o: rax.c rax.h rax_malloc.h
 linenoise.o: linenoise.c linenoise.h
 	$(CC) $(CFLAGS) -c -o $@ linenoise.c
 
-ds4_cpu.o: ds4.c ds4.h ds4_gpu.h
-	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4.c
+ds4_cpu.o: ds4.c ds4.h ds4_ssd.h ds4_distributed.h ds4_gpu.h
+	$(CC) $(CFLAGS) -Wno-unused-function -DDS4_NO_GPU -c -o $@ ds4.c
 
-ds4_cli_cpu.o: ds4_cli.c ds4.h linenoise.h
+ds4_cli_cpu.o: ds4_cli.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h linenoise.h
 	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_cli.c
 
-ds4_server_cpu.o: ds4_server.c ds4.h rax.h
+ds4_gpu_args_cpu.o: ds4_gpu_args.c ds4_gpu_args.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_gpu_args.c
+
+ds4_server_cpu.o: ds4_server.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h rax.h
 	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_server.c
 
-ds4_bench_cpu.o: ds4_bench.c ds4.h
+ds4_bench_cpu.o: ds4_bench.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h
 	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_bench.c
 
+ds4_eval_cpu.o: ds4_eval.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h
+	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_eval.c
+
+ds4_agent_cpu.o: ds4_agent.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h ds4_web.h linenoise.h
+	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_agent.c
+
 ds4_metal.o: ds4_metal.m ds4_gpu.h $(METAL_SRCS)
 	$(CC) $(OBJCFLAGS) -c -o $@ ds4_metal.m
 
-ds4_cuda.o: ds4_cuda.cu ds4_gpu.h ds4_iq2_tables_cuda.inc
+ds4_cuda.o: ds4_cuda.cu ds4_gpu.h ds4_gpu_mgpu.h ds4_iq2_tables_cuda.inc \
+	rocm/ds4_rocm_laguna.cuh rocm/ds4_rocm_dflash.cuh
 	$(NVCC) $(NVCCFLAGS) -c -o $@ ds4_cuda.cu
 
+ds4_rocm.o: ds4_rocm.cu ds4_gpu.h ds4_iq2_tables_cuda.inc $(ROCM_SRCS)
+	$(HIPCC) $(ROCM_CFLAGS) -c -o $@ ds4_rocm.cu
+
+ds4_rocm_compat.o: ds4_rocm_compat.cu ds4_gpu.h ds4_gpu_mgpu.h ds4_gpu_args.h
+	$(HIPCC) $(ROCM_CFLAGS) -c -o $@ ds4_rocm_compat.cu
+
+ds4_rocm_unavailable.o: ds4_rocm_unavailable.cu
+	$(HIPCC) $(ROCM_CFLAGS) -c -o $@ ds4_rocm_unavailable.cu
+
 tests/cuda_long_context_smoke: tests/cuda_long_context_smoke.o ds4_cuda.o
 	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
 
-ds4_test: ds4_test.o rax.o $(CORE_OBJS)
-ifeq ($(UNAME_S),Darwin)
-	$(CC) $(CFLAGS) -o $@ ds4_test.o rax.o $(CORE_OBJS) $(METAL_LDLIBS)
-else
-	$(NVCC) $(NVCCFLAGS) -o $@ ds4_test.o rax.o $(CORE_OBJS) $(CUDA_LDLIBS)
+tests/test_layer_pack.o: tests/test_layer_pack.c ds4_layer_pack.h
+	$(CC) $(CFLAGS) -I. -c -o $@ $<
+
+tests/test_layer_pack: tests/test_layer_pack.o ds4_layer_pack.o
+	$(CC) $(CFLAGS) -o $@ $^ $(LDLIBS)
+
+tests/test_gpu_args.o: tests/test_gpu_args.c ds4_gpu_args.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -I. -DDS4_NO_GPU -c -o $@ $<
+
+tests/test_gpu_args: tests/test_gpu_args.o ds4_gpu_args_cpu.o
+	$(CC) $(CFLAGS) -o $@ $^ $(LDLIBS)
+
+ds4_cpu_test_hooks.o: ds4.c ds4.h ds4_gpu.h ds4_gpu_mgpu.h ds4_layer_pack.h
+	$(CC) $(CFLAGS) -Wno-unused-function -DDS4_NO_GPU -DDS4_TEST_HOOKS -c -o $@ ds4.c
+
+tests/test_engine_mgpu_placement.o: tests/test_engine_mgpu_placement.c ds4.h ds4_gpu_mgpu.h ds4_layer_pack.h
+	$(CC) $(CFLAGS) -I. -c -o $@ $<
+
+tests/test_engine_mgpu_placement: tests/test_engine_mgpu_placement.o ds4_cpu_test_hooks.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	$(CC) $(CFLAGS) -o $@ $^ $(LDLIBS)
+
+ifneq ($(UNAME_S),Darwin)
+tests/test_gpu_xdev.o: tests/test_gpu_xdev.c ds4_gpu.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -I. -I$(CUDA_HOME)/include -c -o $@ $<
+
+tests/test_gpu_xdev: tests/test_gpu_xdev.o ds4_cuda.o
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+tests/test_gpu_model_cache.o: tests/test_gpu_model_cache.c ds4_gpu.h
+	$(CC) $(CFLAGS) -I. -I$(CUDA_HOME)/include -c -o $@ $<
+
+tests/test_gpu_model_cache: tests/test_gpu_model_cache.o ds4_cuda.o
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+tests/test_gpu_lookup_cache_strict.o: tests/test_gpu_lookup_cache_strict.c ds4_gpu.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -I. -I$(CUDA_HOME)/include -c -o $@ $<
+
+tests/test_gpu_lookup_cache_strict: tests/test_gpu_lookup_cache_strict.o ds4_cuda.o
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+ds4_cuda_test_hooks.o: ds4.c ds4.h ds4_gpu.h ds4_gpu_mgpu.h ds4_layer_pack.h
+	$(CC) $(CFLAGS) -Wno-unused-function -DDS4_TEST_HOOKS -I$(CUDA_HOME)/include -c -o $@ ds4.c
+
+tests/test_engine_mgpu_refusal.o: tests/test_engine_mgpu_refusal.c ds4.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -I. -I$(CUDA_HOME)/include -c -o $@ $<
+
+tests/test_engine_mgpu_refusal: tests/test_engine_mgpu_refusal.o ds4_gpu_args.o ds4_kvstore.o rax.o $(CORE_OBJS)
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+tests/test_engine_mgpu_runtime.o: tests/test_engine_mgpu_runtime.c ds4.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -DDS4_TEST_HOOKS -I. -I$(CUDA_HOME)/include -c -o $@ $<
+
+tests/test_engine_mgpu_runtime: tests/test_engine_mgpu_runtime.o ds4_cuda_test_hooks.o ds4_gpu_args.o ds4_kvstore.o rax.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_cuda.o ds4_layer_pack.o
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+tests/test_engine_correctness.o: tests/test_engine_correctness.c ds4.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -I. -I$(CUDA_HOME)/include -c -o $@ $<
+
+tests/test_engine_correctness: tests/test_engine_correctness.o ds4_gpu_args.o ds4_kvstore.o rax.o $(CORE_OBJS)
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+tests/test_sampling.o: tests/test_sampling.c ds4.h
+	$(CC) $(CFLAGS) -DDS4_TEST_HOOKS -I. -c -o $@ $<
+
+tests/test_sampling: tests/test_sampling.o ds4_cuda_test_hooks.o ds4_gpu_args.o ds4_kvstore.o rax.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_cuda.o ds4_layer_pack.o
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+tests/test_cuda_session_batch.o: tests/test_cuda_session_batch.c ds4.h ds4_gpu_args.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -I. -I$(CUDA_HOME)/include -c -o $@ $<
+
+tests/test_cuda_session_batch: tests/test_cuda_session_batch.o ds4_gpu_args.o ds4_kvstore.o rax.o $(CORE_OBJS)
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+test-cuda-session-batch: tests/test_cuda_session_batch
+	DS4_TEST_MODEL="$(DS4_TEST_MODEL)" ./tests/test_cuda_session_batch
+
+tests/test_cuda_mixed_batch.o: tests/test_cuda_mixed_batch.c ds4.h ds4_gpu_args.h ds4_gpu_mgpu.h
+	$(CC) $(CFLAGS) -DDS4_TEST_HOOKS -I. -I$(CUDA_HOME)/include -c -o $@ $<
+
+tests/test_cuda_mixed_batch: tests/test_cuda_mixed_batch.o ds4_cuda_test_hooks.o ds4_gpu_args.o ds4_kvstore.o rax.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_cuda.o ds4_layer_pack.o
+	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+
+test-cuda-mixed-batch: tests/test_cuda_mixed_batch
+	DS4_TEST_MODEL="$(DS4_TEST_MODEL)" ./tests/test_cuda_mixed_batch
 endif
 
-test: ds4_test
+ds4_test: ds4_test.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS)
+ifeq ($(UNAME_S),Darwin)
+	$(CC) $(CFLAGS) -o $@ ds4_test.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS) $(METAL_LDLIBS)
+else
+	$(NVCC) $(NVCCFLAGS) -o $@ ds4_test.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS) $(CUDA_LDLIBS)
+endif
+
+ds4_agent_test: ds4_agent_test.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS)
+ifeq ($(UNAME_S),Darwin)
+	$(CC) $(CFLAGS) -o $@ ds4_agent_test.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS) $(METAL_LDLIBS)
+else
+	$(NVCC) $(NVCCFLAGS) -o $@ ds4_agent_test.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS) $(CUDA_LDLIBS)
+endif
+
+test: ds4_test ds4_agent_test ds4-eval q4k-dot-test \
+	tests/test_layer_pack tests/test_engine_mgpu_placement tests/test_gpu_args \
+	$(SAMPLING_TEST) ds4 ds4-server ds4-bench ds4-agent
+	./ds4-eval --self-test-extractors
+	./ds4_agent_test
 	./ds4_test
+	./tests/test_layer_pack
+	./tests/test_engine_mgpu_placement
+	./tests/test_gpu_args
+	./tests/test_gpu_args_cli.sh
+ifneq ($(UNAME_S),Darwin)
+	./tests/test_sampling
+endif
+
+dspark-acceptance: ds4
+	DS4_DSPARK_MODEL="$(DS4_DSPARK_MODEL)" \
+	DS4_DSPARK_SUPPORT="$(DS4_DSPARK_SUPPORT)" \
+	sh tests/dspark_acceptance_fixture.sh
+
+dspark-verify-depth: ds4_test
+	@if [ ! -f "$(DS4_TEST_MODEL)" ]; then \
+		echo "dspark-verify-depth: skipped, missing model $(DS4_TEST_MODEL)"; \
+	elif [ ! -f "$(DS4_DSPARK_SUPPORT)" ]; then \
+		echo "dspark-verify-depth: skipped, missing DSpark support $(DS4_DSPARK_SUPPORT)"; \
+		echo "dspark-verify-depth: run make dspark-support or set DS4_DSPARK_SUPPORT=FILE"; \
+	else \
+		DS4_TEST_MODEL="$(DS4_TEST_MODEL)" DS4_TEST_DSPARK="$(DS4_DSPARK_SUPPORT)" ./ds4_test --dspark-verify-depth; \
+	fi
+
+mtp-verify-depth: ds4_test
+	@if [ ! -f "$(DS4_TEST_MODEL)" ]; then \
+		echo "mtp-verify-depth: skipped, missing model $(DS4_TEST_MODEL)"; \
+	elif [ ! -f "$(DS4_TEST_MTP)" ]; then \
+		echo "mtp-verify-depth: skipped, missing MTP support $(DS4_TEST_MTP)"; \
+		echo "mtp-verify-depth: run ./download_model.sh mtp or set DS4_TEST_MTP=FILE"; \
+	else \
+		DS4_TEST_MODEL="$(DS4_TEST_MODEL)" DS4_TEST_MTP="$(DS4_TEST_MTP)" ./ds4_test --mtp-verify-depth; \
+	fi
+
+q4k-dot-test: tests/test_q4k_dot.c
+	$(CC) -O2 -Wall -Wextra -std=c99 -o tests/test_q4k_dot tests/test_q4k_dot.c -lm -pthread
+	./tests/test_q4k_dot
 
 clean:
-	rm -f ds4 ds4-server ds4-bench ds4_cpu ds4_native ds4_server_test ds4_test *.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o
+	rm -f ds4 ds4-server ds4-bench ds4-eval ds4-agent ds4_cpu ds4_native ds4_server_test ds4_test ds4_agent_test gguf-tools/quality-testing/score_official tests/test_q4k_dot tests/test_metal_session_batch tests/test_gpu_xdev tests/test_gpu_model_cache tests/test_gpu_lookup_cache_strict tests/test_engine_mgpu_refusal tests/test_engine_mgpu_runtime tests/test_engine_correctness tests/test_sampling tests/test_cuda_session_batch tests/test_cuda_mixed_batch tests/*.o *.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o

QA_BEFORE_RELEASES.md

@@ -0,0 +1,494 @@
+# QA Before Releases
+
+This is the release gate for DwarfStar.  Run it before tagging or pushing a
+release build.  The goal is not to prove every code path exhaustively; it is to
+exercise the paths that have historically regressed: Metal graph inference,
+CUDA, ROCm, SSD streaming, distributed execution, disk KV cache, server APIs, and the
+agent TUI/tool state machine.
+
+Do not run multiple huge model processes at the same time.  Record the commit,
+hardware, GGUF file, context size, and any non-default flags for every manual
+run.
+
+Preferred release test hosts:
+
+- CUDA / DGX Spark: `toor@192.168.60.184`.
+- Metal / distributed Mac testing: `mac-m5max-it` and `mac-m5max-us`.
+- ROCm: The Strix Halo system at antirez@strixhalo (Framework Desktop).
+
+The Mac hosts have DNS entries and are reached through an internet VPN.  They
+are connected to each other over WiFi and also through a Thunderbolt 5
+point-to-point link.  The TB5 route is the preferred distributed-inference
+network when it is available, but it can be fragile and sometimes only works
+when `ds4` is executed in the foreground.  Prefer these machines for release
+testing, especially distributed inference.  Local fallback testing on this
+machine is acceptable when needed; it is an M3 Max with 128 GB RAM.
+The Strix Halo system is reachable via the VPN as well and has a local WiFi
+address in the same lan of the M5 Max systems. The CUDA hosts are in a
+different remote lan and are accessible via a different VPN active
+in this system.
+
+## 1. Repository And Build Sanity
+
+- Start from a clean tree except intentional release notes:
+  `git status --short`.
+- Build the normal local target:
+  `make clean && make`.
+- Build CPU-only binaries as a compile check only:
+  `make clean && make cpu`.
+- Treat compiler warnings as build failures. Save each release and test build's
+  complete output and require no `warning:` or NVCC `warning #` lines. Fix the
+  source when possible; use a narrow target-specific suppression only when a
+  test deliberately compiles a partial translation unit.
+- Repeat the warning-free build gate on the release hardware:
+  `make clean && make` on Metal,
+  `make clean && make cuda-spark` on DGX Spark,
+  `make clean && make cuda-generic CUDA_HOME=/usr` on the multi-GPU CUDA host,
+  `make clean && make strix-halo` on Strix Halo.
+- Run whitespace checks before committing:
+  `git diff --check`.
+- Confirm `./ds4 --help`, `./ds4-server --help`, and `./ds4-agent --help` render
+  cleanly, with readable section colors and no broken wrapping.
+
+## 2. Core Regression Tests
+
+- Run the default suite:
+  `make test`.
+- Run `tests/test_gpu_args_cli.sh` explicitly after changing executable option
+  parsing or multi-GPU placement. Invalid values and device/budget count
+  mismatches must reach the shared GPU parser in all four binaries; an
+  `unknown option` response from a binary that advertises the flag is a
+  release blocker. On CUDA, also start `ds4-server` once with
+  `--gpu-vram auto` and the intended `--gpu-devices` list and preserve the
+  resolved layout line.
+- Run the vector checks explicitly after any tokenizer, template, KV, kernel,
+  quantization, or prompt-rendering change:
+  `./ds4_test --logprob-vectors`
+  and `./ds4_test --local-golden-vectors`.
+- Run server tests when HTTP, SSE, prompt rendering, cache policy, or tool-call
+  replay changed:
+  `./ds4_test --server`.
+- Run `./ds4-eval --self-test-extractors`.
+
+## 3. Official Continuation Quality Gates
+
+These tests are release-blocking after tokenizer, template, KV-cache, attention,
+MoE routing, quantization, logit, or model-graph changes.  They are
+teacher-forced continuation checks against hosted-model output and API
+top-logprob slices, so do not replace them with one sampled chat answer.
+
+- Build the scorer:
+  `make -C gguf-tools quality-score`.
+- Run the tracked DeepSeek V4 Flash smoke vectors:
+  `./ds4_test --logprob-vectors`.
+  This covers the official Flash API vectors in `tests/test-vectors/`, including
+  short prompts and long-prompt attention cases.
+- Run the 100-case DeepSeek V4 Flash fixture for every released Flash GGUF:
+  `gguf-tools/quality-testing/score_official /path/to/deepseek-v4-flash.gguf gguf-tools/quality-testing/data/flash/manifest.tsv /tmp/flash.tsv 4096`.
+- Run the 100-case GLM 5.2 OpenRouter fixture for every released GLM GGUF:
+  `gguf-tools/quality-testing/score_official models/GLM-5.2-UD-Q4_K_XL.gguf gguf-tools/quality-testing/data/glm52-openrouter-100/manifest.tsv /tmp/glm52-q4.tsv 4096`.
+  Current Q4 XL reference band: first-token match `95/100`, API top-1 agreement
+  about `0.942`, and API pair-order agreement about `0.880`.
+- Run the same GLM fixture for reduced-precision GLM release files.  The Q2
+  routed reference is lower quality but should stay near first-token match
+  `92/100`, API top-1 agreement about `0.890`, and API pair-order agreement
+  about `0.800` unless the quantization changed deliberately.
+- Run the 100-case DeepSeek V4 PRO fixture for every released PRO GGUF:
+  `gguf-tools/quality-testing/score_official /path/to/deepseek-v4-pro.gguf gguf-tools/quality-testing/data/pro/manifest.tsv /tmp/pro.tsv 4096`.
+- For SSD streaming, run the same official-continuation scorer once with full
+  residency and once with `--ssd-streaming` for the release model.  The summary
+  and API agreement should stay in the same quality band.
+- Compare any candidate against the previous release or last-known-good output:
+  `python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv`.
+  Treat a large first-token-match drop, a clear NLL regression, or a material
+  API top-1/pair-order regression as a blocker unless the release notes call out
+  an intentional quality tradeoff.
+- Keep the raw `summary` and `api_summary` lines in the release notes or QA log.
+  Do not use stale manifests from `misc/` as release evidence.
+
+## 4. Metal Flash Path
+
+Use the normal Flash GGUF that 128 GB users run.
+
+- One-shot CLI:
+  `./ds4 -m ds4flash.gguf --ctx 32768 --nothink -p "Explain C pointers in one paragraph."`
+- Thinking and max-thinking prompts:
+  run one short coding prompt with default thinking and one with max thinking.
+- Long-context recall:
+  run the long name/number or archive recall test used for catching attention
+  and MoE routing drift.
+- Logprob sanity:
+  `./ds4 --nothink --temp 0 --dump-logprobs /tmp/ds4-logprobs.json --logprobs-top-k 20 -p "..."`
+  and inspect that the continuation is sane.
+- Speed sanity:
+  run `ds4-bench` with `speed-bench/promessi_sposi.txt` and compare prefill,
+  generation speed, and KV bytes with the last known good numbers for the same
+  machine.
+
+### DSpark / DeepSpec Runtime
+
+DSpark is opt-in, but it mutates the verifier, target-hidden capture, support
+model loading, and scheduler paths.  Run these whenever DSpark support,
+speculative verification, confidence/scheduler policy, target hidden capture,
+tiny routed-MoE verifier kernels, or shared `--mtp` support-model code changes:
+
+- Default greedy acceptance fixture:
+  `DS4_DSPARK_MODEL=/Users/antirez/ds4/ds4flash.gguf DS4_DSPARK_SUPPORT=/Users/antirez/ds4/gguf/DeepSeek-V4-Flash-DSpark-support.gguf make dspark-acceptance`.
+- 64-token guardrail:
+  `DS4_DSPARK_FIXTURE_TOKENS=64 DS4_DSPARK_MODEL=/Users/antirez/ds4/ds4flash.gguf DS4_DSPARK_SUPPORT=/Users/antirez/ds4/gguf/DeepSeek-V4-Flash-DSpark-support.gguf make dspark-acceptance`.
+- Fixed-block partial-accept fallback:
+  `DS4_DSPARK_FIXTURE_CONFIDENCE=0 DS4_DSPARK_FIXTURE_TOKENS=8 DS4_DSPARK_FIXTURE_REQUIRE_PARTIAL=1 DS4_DSPARK_MODEL=/Users/antirez/ds4/ds4flash.gguf DS4_DSPARK_SUPPORT=/Users/antirez/ds4/gguf/DeepSeek-V4-Flash-DSpark-support.gguf make dspark-acceptance`.
+- DSpark verifier invariant smoke:
+  `DS4_TEST_MODEL=/Users/antirez/ds4/ds4flash.gguf DS4_DSPARK_SUPPORT=/Users/antirez/ds4/gguf/DeepSeek-V4-Flash-DSpark-support.gguf make dspark-verify-depth`.
+- If shared support-model or verifier structures changed, also run legacy MTP:
+  `make mtp-verify-depth` with `DS4_TEST_MTP` set to a one-stage MTP support
+  GGUF, or confirm the target skips only because the optional file is missing.
+- Record `c_add` `accepted_draft`, `errors=0`, `verify_layer`, and `net_saved`
+  for both 32-token and 64-token runs.  A faster run with lower proposal
+  quality is a regression unless it was an intentional scheduler change.
+- If verifier MoE kernels changed, run one diagnostic `c_add` profile with
+  `DS4_DSPARK_VERIFY_SELECTED_PROFILE=1` or the Metal MoE stage profiler and
+  record the selected-expert footprint or stage timing in the DSpark log.
+
+### Session Microbatching And Metal TP
+
+Run these gates whenever session scheduling, batched decode, mixed
+prefill/decode, QKV projection, shared or routed experts, tensor parallelism,
+or backend fallback selection changes.
+
+- On a single Metal machine, run the full-vocabulary exact-logit oracle with
+  2, 4, 8, and 16 sessions:
+  `DS4_TEST_MODEL=/path/to/ds4flash.gguf DS4_TEST_SESSION_COUNT=N make test-metal-session-batch`.
+  Compatible resident Q8 runs must report `native_shared=1 native_qkv=1` at
+  every tested count. The 16-session run covers row counts above the old
+  artificial eight-row limit.
+- Repeat the four-session oracle with
+  `DS4_METAL_SESSION_BATCH_SHARED=0` and with
+  `DS4_METAL_SESSION_BATCH_QKV=0`. The first run must use the complete fallback;
+  the second may batch the shared expert only. Both must remain bit-exact.
+- The oracle must cover reversed row ordering, at least six decode steps, and a
+  mixed prefill/decode call. Any nonzero differing-logit count is a blocker;
+  argmax-only agreement is insufficient.
+- Benchmark 1, 2, 4, 8, and 16 simultaneous resident sessions on the same host
+  and model. Record model-step latency and aggregate decode tokens/second, not
+  only request completion speed. The current Metal path batches QKV and part of
+  the shared expert, but still runs attention, routed experts, shared down, and
+  the output head per session. Treat flat aggregate scaling as unfinished
+  implementation work, not evidence that Metal cannot benefit from batching.
+- On `mac-m5max-it` and `mac-m5max-us`, run the same oracle in physical TP mode
+  over explicit `tcp` and `rdma` transports. Set `DS4_TEST_TP_MODE=leader` on
+  the leader and `DS4_TEST_TP_MODE=worker DS4_TEST_TP_LEADER_HOST=HOST` on the
+  worker, with a unique `DS4_TEST_TP_PORT`. Run at least 2 and 4 sessions and
+  preserve both logs. TP currently uses the ordered per-session fallback, so
+  native single-machine row-grid flags must remain off in the TP logs.
+- For the current TB5 MacBook link, US is `10.99.0.2` on `en1`/`rdma_en1` and
+  IT is `10.99.0.1` on `en6`/`rdma_en6`; both use GID index 1. Before testing,
+  require `rdma_ctl status` to report `enabled` and `ibv_devinfo -v` to show
+  `PORT_ACTIVE` plus the corresponding `::ffff:10.99.0.x` GID. Force the
+  device and GID with `--rdma-device NAME --rdma-gid-index 1` if automatic
+  selection is ambiguous. A working TB IP ping alone is not RDMA evidence.
+- Kill the TP worker during one batch with `DS4_TEST_TP_DISCONNECT=1` on the
+  leader. The operation must fail cleanly, invalidate every affected session,
+  and return control without hanging.
+- Verify unsupported combinations explicitly: GLM, DSpark/MTP support models,
+  SSD streaming, quality/reference modes, steering, resident Q4 expert
+  overlap, and CPU-router modes must use the established exact fallback or
+  reject the combination before evaluation. They must not partially activate
+  native batching.
+
+## 5. Metal PRO Path
+
+PRO support is experimental, but release builds must not break it silently.
+
+- If a PRO-capable machine is available, run a short PRO q2 prompt and verify
+  the correct template, thinking behavior, and endpoint aliases.
+- For PRO Q4 distributed builds, test only on the intended high-memory machines.
+- If PRO cannot be run locally, at least build all binaries and review changes
+  touching model shape, tensor lookup, routed expert mapping, template logic,
+  and KV payload compatibility.
+
+## 6. GLM 5.2
+
+GLM has a different template, model shape, MTP block, attention layout,
+tensor-parallel gate width, and streaming policy. Flash or PRO success does not
+substitute for this matrix.
+
+- On a 512 GB Metal machine, run short greedy prompts with both the Q4 XL and
+  reduced-precision Q2 release GGUFs. Cover thinking and no-thinking templates,
+  and verify the server reports the GLM model family rather than a DeepSeek
+  alias internally.
+- Run the OpenRouter smoke vectors explicitly:
+  `DS4_TEST_MODEL=/path/to/glm.gguf
+  DS4_TEST_VECTOR_FILE=tests/test-vectors/glm-openrouter/official.vec
+  ./ds4_test --logprob-vectors`.
+  Preserve the report as a diagnostic. The hosted vectors include very
+  low-probability top-20 tails whose membership is not stable after GLM routed
+  expert quantization, so an individual `official top token missing locally`
+  assertion is not by itself a release blocker. Selected-token mismatches must
+  remain consistent with the model's 100-case first-token band, and the
+  section 3 scorer is the release gate for aggregate GLM quality.
+- Run the 100-case Q4 XL and Q2 official fixtures from section 3 and preserve
+  both `summary` and `api_summary` lines. Compare against the documented Q4 and
+  Q2 reference bands independently.
+- Run `tests/glm_long_context_smoke.sh` with the release-advertised context on
+  the 512 GB Metal host. The generated continuation must begin with `>` and
+  contain none of the known token-corruption markers.
+- Exercise integrated GLM MTP with `--glm-mtp-timing` on a deterministic
+  prompt. Compare the greedy text to
+  a non-MTP run, require clean speculative cycles, and record acceptance and
+  timing. Also run once with MTP disabled to prove ordinary decode remains the
+  default.
+- Run the Metal session oracle with 2 and 4 GLM sessions. It must report
+  `family=glm native_shared=0 native_qkv=0` and remain exact, including mixed
+  prefill/decode; the DeepSeek-only row-grid kernels must not activate.
+- Run resident and SSD-streaming GLM Q2 prompts with the same greedy input.
+  Compare first token and top-logprob sanity, and record the selected full-layer
+  prefix and dynamic expert-cache budget.
+- Run physical two-machine GLM TP over TCP and RDMA with short and long prompts.
+  Record prefill/decode speed, transport, rank residency, and clean shutdown.
+  Repeat one run with `--tensor-parallel-token-prefill` as the exact-arithmetic
+  diagnostic. Use a GGUF whose routed-expert type has ownership-aware GLM TP
+  kernels. Also test a Q4-routed GLM file as a negative gate: until Q4 ownership
+  kernels are implemented, both ranks must reject it clearly before evaluation
+  rather than loading a partial split or hanging.
+- On the eight-GPU CUDA host, run one resident GLM Q2 prompt, a long-context
+  prompt, integrated GLM MTP, and concurrent server requests. Use ordinary
+  eight-GPU layer placement for GLM; do not pass the Flash-specific
+  `--cuda-tensor-parallel` option. Multi-tier GLM prefill must
+  report progress through the tier-switching token-major path, and decode,
+  cache updates, and output-head/logit assembly must complete without CPU spill.
+  Auto-placement must reserve each layer's compact DSA/indexer cache and the
+  graph workspace before loading weights; a late graph-allocation failure is a
+  release blocker. Confirm the long-context layout stays within every device's
+  budget and uses additional tiers when the cache no longer fits on the earlier
+  ones.
+  The long-context harness can select this backend with
+  `DS4_GLM_BACKEND=cuda` and pass placement flags through
+  `DS4_GLM_EXTRA_ARGS="--gpu-vram auto --gpu-devices 0,2,4,6,1,3,5,7"`.
+- Through `ds4-server`, exercise OpenAI chat, Responses, and Anthropic requests
+  against GLM, including thinking and SSE. DeepSeek compatibility endpoint
+  aliases may resolve to the loaded model, but rendered prompts and generated
+  text must use the GLM template.
+
+## 7. SSD Streaming
+
+SSD streaming is a capacity path, so test both correctness and user experience.
+
+- Flash q2/q2-q4 streaming:
+  `./ds4 -m ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB -p "..."`
+- Regression test mixed-quant Flash SSD streaming. Use the mixed q2/q4 GGUF
+  with boosted Q4 routed-expert layers and a prompt long enough to exercise the
+  selected-address prefill path; it must not fail with "model range is not
+  covered by mapped model views":
+  `./ds4 -m gguf/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf --ssd-streaming --ssd-streaming-cache-experts 16GB --ctx 4096 --tokens 1 --nothink --prompt-file /tmp/ds4_600tok_prompt.txt`.
+- Cold streaming measurement:
+  run once with `--ssd-streaming-cold` and verify no deadlock, missing expert,
+  or impossible slowdown.
+- Confirm startup reports cache budget and that generation does not stall on
+  repeated expert misses for a small interactive prompt.
+- If streaming cache internals changed, test the same prompt twice and compare
+  first-token/logprob sanity between runs.
+
+## 8. CUDA / DGX Spark
+
+Before a release, ask the user for CUDA access if it is not already configured.
+Use the DGX Spark / GB10 host `toor@192.168.60.184`.  Do not claim CUDA is
+release-ready without this pass.
+
+- Fetch or push the exact release commit to the CUDA machine.
+- Build:
+  `make clean && make cuda-spark`.
+- Require both the DGX Spark build and the eight-GPU CUDA build to complete
+  without compiler warnings.
+- Run:
+  `make cuda-regression`.
+- Run a short CLI prompt with the Flash GGUF and record generation t/s.
+- Run a longer prompt that exercises routed experts past a few thousand tokens.
+- On the eight-GPU CUDA host, run the full-vocabulary decode oracle:
+  `DS4_TEST_MODEL=/path/to/flash.gguf make test-cuda-session-batch`.
+  Preserve the per-batch timing for 2, 4, and 8 rows and require
+  `nonexact_logits=0`. Run the released Q4 file and the reduced-precision Q2
+  file: Q4 exercises grouped routed/shared stages, while unsupported Q2 native
+  MoE shapes must retain the ordered exact fallback.
+- With CUDA TP attention enabled, compatible Q4 runs must use grouped
+  attention-core, QKV, KV-store, and attention-post by default and remain
+  full-vocabulary exact against isolated decode. On the eight-L40S host, the
+  16-row decode step must remain above 110 aggregate tokens/s. Repeat once with
+  `DS4_CUDA_TP_ATTN=0` only as rollback coverage; it is not the production
+  configuration.
+- Run native mixed prefill/decode at the default frontier and at compressed
+  context:
+  `DS4_TEST_MODEL=/path/to/flash.gguf make test-cuda-mixed-batch` and
+  `DS4_TEST_CONTEXT=4096 DS4_TEST_MIXED_INITIAL=2048 DS4_TEST_MIXED_ROUNDS=8
+  DS4_TEST_MODEL=/path/to/flash.gguf make test-cuda-mixed-batch`.
+  Every round must report exact logits and `mode=native`; a serialized fallback
+  is a failure for the eight-GPU TP/EP topology. Under CUDA TP attention, the
+  native mixed step must use the same exact grouped decode stages when their
+  capability checks pass; record correctness and speedup separately. Also
+  force an 800-row prefill quantum with
+  `DS4_TEST_ALLOW_FALLBACK=1`; it must report the serialized safety fallback.
+- Start `ds4-server` with 8 and 16 batched sessions and issue at least that many
+  simultaneous requests with mixed prompt lengths. Verify no session mix-up,
+  deadlock, or starvation and record aggregate generation throughput.
+- On DGX Spark, verify the same public batch API and server concurrency use the
+  single-GPU fallback without creating peer-only TP/EP state. The eight-GPU
+  native oracle is not a valid Spark test because its topology is intentionally
+  unavailable there.
+- If CUDA Q4, distributed, streaming hooks, tensor span loading, or model cache
+  code changed, test the specific GGUF and split mode that uses that path.
+- Verify that any CUDA-only warning fixes are also clean on macOS and do not
+  change Metal behavior.
+
+## 9. ROCm / Strix Halo
+
+Use the Strix Halo Framework Desktop via the VPN hostname `strixhalo`
+(`antirez@strixhalo`).  This host validates the ROCm backend; do not use it as
+a substitute for CUDA or Metal release testing.
+
+- Fetch or push the exact release commit to the Strix Halo machine.
+- Build:
+  `make clean && make strix-halo`.
+- Require the ROCm build to complete without compiler warnings.
+- Use the q2 Flash imatrix GGUF for release smoke tests:
+  `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf`.
+- Do not use the mixed q2-q4 or Q4 Flash GGUFs for routine Strix Halo QA yet.
+  They are dangerous on this machine for now because the ROCm path can hit
+  system OOM instead of failing cleanly.
+- Run a short CLI prompt:
+  `./ds4 -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf --ctx 4096 --nothink -p "Reply with exactly: OK"`.
+- Run one longer prompt if ROCm kernels, backend hooks, tensor loading, model
+  cache, KV cache, or graph prefill code changed.
+- Run the GLM Q2 release model through ROCm SSD streaming with at least four
+  generated tokens:
+  `./ds4 --rocm -m gguf/GLM-5.2-UD-Q2_K_RoutedQ2K.gguf --ssd-streaming --ctx 4096 --nothink --tokens 4 -p "Reply with exactly: OK"`.
+  Startup must select a cache budget that passes the memory guard without an
+  override, and both compact indexed prefill and decode must complete.
+- Run one longer GLM prompt with the release-advertised Strix context after
+  changes to GLM attention, typed quantized projections, streaming expert
+  caches, or memory budgeting. Record the context, cache split, and whether
+  the continuation stays free of token-corruption markers.
+- Run the same GLM model with `--glm-mtp-timing --temp 0`. At least one draft
+  verification cycle must complete without a `glm mtp step failed` message.
+- Record startup memory/cache messages, prefill speed, generation speed, and
+  whether the backend reports `ROCm backend initialized`.
+
+## 10. Distributed Inference
+
+Distributed code has regressed around route setup, KV snapshots, request IDs,
+and split model loading.  Test it whenever distributed, KV, session, or model
+loading code changes.
+
+- Prefer `mac-m5max-it` and `mac-m5max-us` for Metal distributed tests.  Use the
+  TB5 point-to-point link when it is working; otherwise note that the run used
+  WiFi/VPN routing.
+- Start workers first, then the coordinator.
+- Test a small prompt and a longer prompt.
+- Verify the coordinator waits for a complete route and exits cleanly.
+- Verify `Ctrl+C` returns control after the current distributed token or chunk
+  drains.
+- Save and restore a distributed KV snapshot if that code changed.
+- If CUDA distributed is relevant, test across the CUDA hosts and record
+  generation speed, not just "it works".
+
+## 11. Disk KV Cache
+
+Disk KV cache bugs are high impact for server users.
+
+- Start the server with:
+  `./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192`.
+- Run the same request twice and verify the second request hits cache.
+- Fill the cache enough to trigger eviction; verify the newly-written entry is
+  not evicted and useful anchors are retained.
+- Test rejection of incompatible checkpoints when model, quantization, context,
+  or raw/compressed KV layout changes.
+- Test stripped agent sessions: `/strip <id>` then `/switch <id>` should rebuild
+  by prefill and render sane history.
+
+## 12. Server APIs
+
+The server must keep compatibility across OpenAI, Responses, and Anthropic
+clients.
+
+- `GET /v1/models/deepseek-v4-flash` and `GET /v1/models/deepseek-v4-pro`
+  should both serve whichever GGUF is loaded.
+- Test OpenAI chat completion, OpenAI Responses, and Anthropic messages.
+- Test SSE streaming with thinking enabled and disabled.
+- Test keepalive during long prefill and confirm clients do not time out.
+- On the eight-L40S CUDA TP target, start `ds4-server` with the release TP
+  options and verify all 16 100k-context sessions allocate. Startup must report
+  a 2048-token prefill cap; a silent fallback to 4096 is an OOM regression.
+- Test `--trace` and confirm rendered prompts, cache decisions, generated text,
+  and tool-parser events are useful without leaking unrelated state.
+
+## 13. ds4-agent
+
+The agent is the most stateful component.  Test it manually, not only by build.
+
+- Startup banner, status bar, help, `/power`, `/save`, `/list`, `/switch`,
+  `/history`, `/compact`, `/new`, `/del`, and `/strip`.
+- Ctrl+C during generation, during prefill, during a web fetch, and during a
+  long tool call.  After `Stopped by user`, typing a new prompt must work.
+- Queue messages while the model is busy.  Queued messages must not skip tool
+  execution; after tool results, the queued user text must be provided.
+- Read/search/edit/write tools:
+  create a temp project, ask for edits, verify old/new and `[upto]` anchored
+  edits fail safely on ambiguous matches and do not require retyping whole files.
+- Real coding edit loop:
+  delete `/tmp/mymandel`, ask ds4-agent to create a small C ASCII Mandelbrot
+  program there, build and run it, then in a second user turn ask for a small
+  modification that should naturally use the edit tool, such as changing the
+  ASCII character ramp or output dimensions.  Verify the agent edits the
+  existing file instead of rewriting the whole project, and that the final
+  program still builds and runs.
+- Bash tools:
+  test short output, large output truncation, non-zero exit output, long-running
+  jobs, `bash_status`, and `bash_stop`.
+- Web tools:
+  `google_search` and `visit_page` should ask for visible Chrome approval with
+  timeout, open pages without stealing focus when possible, extract Markdown,
+  close tabs, and handle consent/privacy walls as tool errors the model can see.
+- TUI:
+  test multiline prompt editing, history navigation, queued prompt display,
+  status bar fill to terminal width, syntax highlighting in Markdown/code blocks,
+  and SSH/remote terminal flicker.
+
+## 14. Download Script And Model Files
+
+- Test `download_model.sh` in a temporary directory so local weights are not
+  overwritten.
+- Test one Flash target and one PRO target enough to verify URL, resume, Hugging
+  Face CLI/curl behavior, file naming, and symlink policy.
+- Verify legacy removed targets fail clearly.
+- Verify README model names match the script and Hugging Face repository.
+
+## 15. Performance And Power
+
+- Run `ds4-bench` on the release machine and compare with tracked CSV baselines.
+- Test `--power 100` is not throttled.
+- Test `--power 50` visibly reduces duty cycle in CLI, server, agent, eval, and
+  bench where practical.
+- Confirm context buffer size, raw KV rows, compressed KV rows, and mmap behavior
+  match expectations for 32k, 100k, and any release-advertised context size.
+
+## 16. Release Sign-off
+
+Do not sign off until:
+
+- macOS Metal Flash passed.
+- GLM 5.2 Metal, official-quality, MTP, batching-fallback, and applicable TP or
+  CUDA gates passed.
+- Official continuation quality gates passed for every released model family.
+- CUDA was tested on the CUDA machine or the release notes explicitly say CUDA
+  was not validated.
+- ROCm was tested on Strix Halo or the release notes explicitly say ROCm was
+  not validated.
+- Metal, CUDA, ROCm, CPU-only, and test builds completed without compiler
+  warnings on every release target that was validated.
+- Disk KV cache was exercised.
+- Server API streaming was exercised.
+- Agent interruption and tool loops were exercised manually.
+- Speed is within expected variance for the same hardware and model.
+- Metal 2/4/8/16-session exactness and forced fallback gates passed.
+- Physical Metal TP batching and CUDA native decode/mixed batching passed when
+  those backends are part of the release.
+- Any skipped item is written down with the reason.

STRIXHALO.md

@@ -0,0 +1,136 @@
+# DS4 on Strix Halo
+
+This is the minimal setup for DS4 ROCm inference on a
+Strix Halo machine with 128 GB RAM and Radeon 8060S (`gfx1151`).
+
+## 1. Install ROCm
+
+On Ubuntu 26.04 LTS, install the ROCm compiler/runtime and libraries used by the Strix Halo backend:
+
+```sh
+sudo apt-get update
+sudo apt-get install -y \
+  hipcc rocminfo rocm-smi \
+  libamdhip64-dev \
+  libhipblas-dev libhipblaslt-dev \
+  librocblas-dev \
+  librocwmma-dev \
+  libhipcub-dev
+```
+
+The backend uses rocWMMA. On this Ubuntu 26.04 setup, `librocwmma-dev`
+installs the top-level rocWMMA headers but misses `rocwmma/internal/`.
+No Ubuntu package currently provides those internal headers. Install a complete
+matching rocWMMA header tree:
+
+```sh
+git clone --depth 1 --branch rocm-7.1.0 https://github.com/ROCm/rocWMMA.git /tmp/rocWMMA-rocm-7.1.0
+sudo mkdir -p /usr/local/include
+sudo cp -a /tmp/rocWMMA-rocm-7.1.0/library/include/rocwmma /usr/local/include/
+```
+
+If ROCm is installed under `/usr` but tooling expects `/opt/rocm`, add these
+compatibility links:
+
+```sh
+sudo mkdir -p /opt/rocm/bin
+sudo ln -sf /usr/bin/hipcc /opt/rocm/bin/hipcc
+sudo ln -sfn /usr/lib/x86_64-linux-gnu /opt/rocm/lib
+sudo ln -sfn /usr/include /opt/rocm/include
+```
+
+## 2. Enable ROCm access
+
+The user running DS4 must be able to open `/dev/kfd` and the DRM render node:
+
+```sh
+sudo usermod -aG render,video "$USER"
+```
+
+Log out and back in, or reboot. Verify:
+
+```sh
+rocminfo | grep -A80 'Name:                    gfx1151'
+```
+
+If DS4 says `no ROCm-capable device is detected`, check that `rocminfo` can open
+`/dev/kfd` and that `groups` includes `render`.
+
+## 3. Increase GPU-visible memory
+
+A 128 GB Strix Halo system may initially expose only about 62 GB of GPU-visible
+memory. DS4 needs the larger GTT aperture for the 80.76 GiB model plus runtime
+buffers.
+
+Use these kernel parameters:
+
+```text
+amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856
+```
+
+On Ubuntu with GRUB:
+
+```sh
+sudo cp /etc/default/grub /etc/default/grub.bak
+sudoedit /etc/default/grub
+```
+
+Set:
+
+```text
+GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856"
+```
+
+Then:
+
+```sh
+sudo update-grub
+sudo reboot
+```
+
+After reboot, verify:
+
+```sh
+cat /proc/cmdline
+sudo dmesg | grep -Ei 'GTT|gttsize|TTM|VRAM'
+rocminfo | grep -A80 'Name:                    gfx1151'
+```
+
+Expected signs:
+
+```text
+amdgpu:  126976M of GTT memory ready
+rocminfo gfx1151 pool: 130023424 KB
+```
+
+## 4. Build DS4
+
+Use the normal Strix Halo target. It builds the standard binary names:
+
+```sh
+make strix-halo -j"$(nproc)"
+```
+
+`make rocm` is an alias for `make strix-halo`.
+
+## 5. Use the right GGUF
+
+Use the standard IQ2XXS/Q2K/Q8 imatrix GGUF:
+
+```text
+DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
+```
+
+Avoid the mixed IQ2/IQ4 or IQ2/Q4 GGUFs on this machine for now. They put much
+more memory pressure on the ROCm path and can trigger system OOM instead of a
+clean DS4 failure.
+
+## 6. Run DS4
+
+Run it normally:
+
+```sh
+./ds4 -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
+```
+
+The ROCm build uses the Strix Halo backend automatically.

download_model.sh

@@ -1,12 +1,29 @@
 #!/bin/sh
 set -e
 
+GLM_UNSLOTH_REPO="unsloth/GLM-5.2-GGUF"
+GLM_ANTIREZ_REPO="antirez/GLM-5.2-GGUF"
+LAGUNA_REPO="poolside/Laguna-S-2.1-GGUF"
+LAGUNA_ANTIREZ_REPO="antirez/Laguna-S-2.1-GGUF"
+LAGUNA_REVISION="706fa69799926b6afde1af9e24ca2a4923f110a1"
 REPO="antirez/deepseek-v4-gguf"
-Q2_FILE="DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf"
 Q2_IMATRIX_FILE="DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf"
-Q4_FILE="DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf"
 Q4_IMATRIX_FILE="DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf"
+Q2_Q4_IMATRIX_FILE="DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf"
+PRO_Q2_IMATRIX_FILE="DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf"
+PRO_Q4_LAYERS00_30_FILE="DeepSeek-V4-Pro-Q4K-Layers00-30.gguf"
+PRO_Q4_LAYERS31_OUTPUT_FILE="DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf"
 MTP_FILE="DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf"
+DSPARK_SUPPORT_FILE="DeepSeek-V4-Flash-DSpark-support.gguf"
+GLM_UNSLOTH_Q4_REMOTE_BASE="UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL"
+GLM_UNSLOTH_Q4_LOCAL_BASE="GLM-5.2-UD-Q4_K_XL"
+GLM_UNSLOTH_Q4_FIRST_FILE="$GLM_UNSLOTH_Q4_LOCAL_BASE-00001-of-00011.gguf"
+GLM_ANTIREZ_IQ2XXS_FILE="GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf"
+GLM_ANTIREZ_Q2_FILE="GLM-5.2-UD-Q2_K_RoutedQ2K.gguf"
+GLM_ANTIREZ_Q4_FILE="GLM-5.2-UD-Q4_K_RoutedQ4K.gguf"
+LAGUNA_Q4_FILE="laguna-s-2.1-Q4_K_M.gguf"
+LAGUNA_Q2_Q3_FILE="laguna-s-2.1-RoutedQ2_K-Last27Q3_K.gguf"
+LAGUNA_DFLASH_FILE="laguna-s-2.1-DFlash-Q8_0.gguf"
 
 ROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
 OUT_DIR=${DS4_GGUF_DIR:-"$ROOT/gguf"}
@@ -15,43 +32,103 @@ case "$OUT_DIR" in
     *) OUT_DIR="$ROOT/$OUT_DIR" ;;
 esac
 TOKEN=${HF_TOKEN:-}
+HF_REVISION=
 
 usage() {
     cat <<EOF
-DeepSeek V4 Flash GGUF downloader
+DwarfStar GGUF downloader
 
 Usage:
   ./download_model.sh q2-imatrix [--token TOKEN]
+  ./download_model.sh q2-q4-imatrix [--token TOKEN]
   ./download_model.sh q4-imatrix [--token TOKEN]
-  ./download_model.sh q2 [--token TOKEN]
-  ./download_model.sh q4 [--token TOKEN]
+  ./download_model.sh pro-q2-imatrix [--token TOKEN]
+  ./download_model.sh pro-q4-layers00-30 [--token TOKEN]
+  ./download_model.sh pro-q4-layers31-output [--token TOKEN]
+  ./download_model.sh pro-q4-split [--token TOKEN]
   ./download_model.sh mtp [--token TOKEN]
+  ./download_model.sh dspark-support [--token TOKEN]
+  ./download_model.sh glm-unsloth-q4 [--token TOKEN]
+  ./download_model.sh glm-antirez-iq2xxs [--token TOKEN]
+  ./download_model.sh glm-antirez-q2 [--token TOKEN]
+  ./download_model.sh glm-antirez-q4 [--token TOKEN]
+  ./download_model.sh laguna-q4 [--token TOKEN]
+  ./download_model.sh laguna-q2-q3 [--token TOKEN]
+  ./download_model.sh laguna-dflash [--token TOKEN]
 
 Targets:
-  *** PREFERRED GGUF FILES: USE THE IMATRIX VERSIONS BELOW ***
 
   q2-imatrix
        2-bit routed experts, about 81 GB on disk.
        Recommended model for 96 and 128 GB RAM machines.
 
+  q2-q4-imatrix
+       Mixed Flash quant: mostly q2 routed experts, with the last 6 layers
+       using q4 routed experts. About 98 GB on disk. Good for higher
+       quality inference for 128 GB MacBooks. Works on DGX Spark but loading
+       may struggle compared to q2-imatrix.
+
   q4-imatrix
        4-bit routed experts, about 153 GB on disk.
        Recommended model for machines with 256 GB RAM or more.
 
-  Legacy GGUF files:
+  pro-q2-imatrix
+       DeepSeek V4 PRO q2 imatrix quant, as a single GGUF file. About 430 GB
+       on disk; intended for 512 GB RAM machines.
 
-  q2   2-bit routed experts, about 81 GB on disk.
-       Older non-imatrix model for 96 and 128 GB RAM machines. Prefer
-       q2-imatrix unless you specifically need the legacy quant.
+  pro-q4-layers00-30
+       First half of the DeepSeek V4 PRO Q4 routed-expert quant, layers 0..30.
+       Use on the coordinator in a two-Mac-Studio distributed run. About 426 GB.
 
-  q4   4-bit routed experts, about 153 GB on disk.
-       Older non-imatrix model for machines with 256 GB RAM or more. Prefer
-       q4-imatrix unless you specifically need the legacy quant.
+  pro-q4-layers31-output
+       Second half of the DeepSeek V4 PRO Q4 routed-expert quant, layers
+       31..output. Use on the worker in a two-Mac-Studio distributed run.
+       About 412 GB.
+
+  pro-q4-split
+       Downloads both PRO Q4 split files into the download directory. About
+       838 GB total. This target does not update ./ds4flash.gguf.
 
   mtp  Optional speculative decoding component, about 3.5 GB on disk.
-       It is useful with q2-imatrix, q4-imatrix, q2, and q4, but must be
+       It is useful with q2-imatrix, q2-q4-imatrix, and q4-imatrix, but must be
        enabled explicitly with --mtp when running ds4 or ds4-server.
 
+  dspark-support
+       Optional DSpark speculative decoding support GGUF, about 6 GB. Enable it
+       with --dspark and --mtp when running ds4 or ds4-server.
+
+  glm-unsloth-q4
+       GLM 5.2 Unsloth UD-Q4_K_XL quant from unsloth/GLM-5.2-GGUF.
+       Downloads all 11 shards and links ./ds4flash.gguf to the first shard.
+
+  glm-antirez-iq2xxs
+       GLM 5.2 antirez routed IQ2_XXS GGUF from antirez/GLM-5.2-GGUF.
+       Includes Q2_K block 78 and is intended for reduced-memory testing.
+
+  glm-antirez-q2
+       GLM 5.2 antirez routed Q2_K GGUF from antirez/GLM-5.2-GGUF.
+       About 262 GB on disk.
+
+  glm-antirez-q4
+       GLM 5.2 antirez routed Q4_K GGUF from antirez/GLM-5.2-GGUF.
+       About 434 GB on disk.
+
+  laguna-q4
+       Official imatrix-quantized Laguna S 2.1 Q4_K_M GGUF from Poolside.
+       About 68 GB on disk; supported by Metal, CUDA, and ROCm with full model
+       residency.
+
+  laguna-q2-q3
+       Mixed Laguna S 2.1 routed-expert quant for 64 GB systems. Routed
+       layers 1..20 use Q2_K and layers 21..47 use Q3_K; all other tensors
+       retain the official Q4_K_M layout. 44.95 GiB on disk; supported by
+       Metal, CUDA, and ROCm with full model residency.
+
+  laguna-dflash
+       Q8_0 Laguna S 2.1 DFlash speculative support GGUF, quantized from
+       Poolside's official support model. About 1.04 GiB on disk. This is an
+       optional support model and does not replace or relink the main model.
+
 Options:
   --token TOKEN  Hugging Face token. Otherwise HF_TOKEN or the local HF token
                  cache is used if present.
@@ -60,7 +137,7 @@ Environment:
   DS4_GGUF_DIR   Directory used for downloaded GGUF files.
                  Default: ./gguf
 
-After q2-imatrix/q4-imatrix/q2/q4 downloads the script updates:
+After main-model downloads the script updates:
   ./ds4flash.gguf -> <download directory>/<selected model>
 
 Then the default commands work:
@@ -69,6 +146,13 @@ Then the default commands work:
 
 After downloading mtp, enable it explicitly, for example:
   ./ds4 --mtp <download directory>/$MTP_FILE --mtp-draft 2
+
+After downloading DSpark support, enable it explicitly in greedy mode:
+  ./ds4 --dspark --mtp <download directory>/$DSPARK_SUPPORT_FILE --temp 0
+
+Large PRO, GLM, and Laguna files use the official Hugging Face downloader
+because they are too large, sharded, or nested for the curl path used by the
+smaller DeepSeek Flash GGUF files.
 EOF
 }
 
@@ -79,13 +163,66 @@ fi
 
 MODEL=$1
 shift
+MODEL_FILES=
+LINK_MODEL=1
+FORCE_HF_DOWNLOAD=0
+FLATTEN_DOWNLOADS=0
 
 case "$MODEL" in
     q2-imatrix) MODEL_FILE=$Q2_IMATRIX_FILE ;;
+    q2-q4-imatrix) MODEL_FILE=$Q2_Q4_IMATRIX_FILE ;;
     q4-imatrix) MODEL_FILE=$Q4_IMATRIX_FILE ;;
-    q2) MODEL_FILE=$Q2_FILE ;;
-    q4) MODEL_FILE=$Q4_FILE ;;
-    mtp) MODEL_FILE=$MTP_FILE ;;
+    pro-q2-imatrix) MODEL_FILE=$PRO_Q2_IMATRIX_FILE ;;
+    pro-q4-layers00-30) MODEL_FILE=$PRO_Q4_LAYERS00_30_FILE; LINK_MODEL=0 ;;
+    pro-q4-layers31-output) MODEL_FILE=$PRO_Q4_LAYERS31_OUTPUT_FILE; LINK_MODEL=0 ;;
+    pro-q4-split)
+        MODEL_FILES="$PRO_Q4_LAYERS00_30_FILE $PRO_Q4_LAYERS31_OUTPUT_FILE"
+        LINK_MODEL=0
+        ;;
+    mtp) MODEL_FILE=$MTP_FILE; LINK_MODEL=0 ;;
+    dspark-support) MODEL_FILE=$DSPARK_SUPPORT_FILE; LINK_MODEL=0 ;;
+    glm-unsloth-q4)
+        REPO=$GLM_UNSLOTH_REPO
+        MODEL_FILE=$GLM_UNSLOTH_Q4_FIRST_FILE
+        MODEL_FILES=
+        for part in 00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011; do
+            MODEL_FILES="$MODEL_FILES $GLM_UNSLOTH_Q4_REMOTE_BASE-${part}-of-00011.gguf"
+        done
+        FORCE_HF_DOWNLOAD=1
+        FLATTEN_DOWNLOADS=1
+        ;;
+    glm-antirez-q2)
+        REPO=$GLM_ANTIREZ_REPO
+        MODEL_FILE=$GLM_ANTIREZ_Q2_FILE
+        FORCE_HF_DOWNLOAD=1
+        ;;
+    glm-antirez-iq2xxs)
+        REPO=$GLM_ANTIREZ_REPO
+        MODEL_FILE=$GLM_ANTIREZ_IQ2XXS_FILE
+        FORCE_HF_DOWNLOAD=1
+        ;;
+    glm-antirez-q4)
+        REPO=$GLM_ANTIREZ_REPO
+        MODEL_FILE=$GLM_ANTIREZ_Q4_FILE
+        FORCE_HF_DOWNLOAD=1
+        ;;
+    laguna-q4)
+        REPO=$LAGUNA_REPO
+        MODEL_FILE=$LAGUNA_Q4_FILE
+        FORCE_HF_DOWNLOAD=1
+        HF_REVISION=$LAGUNA_REVISION
+        ;;
+    laguna-q2-q3)
+        REPO=$LAGUNA_ANTIREZ_REPO
+        MODEL_FILE=$LAGUNA_Q2_Q3_FILE
+        FORCE_HF_DOWNLOAD=1
+        ;;
+    laguna-dflash)
+        REPO=$LAGUNA_ANTIREZ_REPO
+        MODEL_FILE=$LAGUNA_DFLASH_FILE
+        FORCE_HF_DOWNLOAD=1
+        LINK_MODEL=0
+        ;;
     -h|--help|help)
         usage
         exit 0
@@ -120,14 +257,114 @@ if [ -z "$TOKEN" ] && [ -s "$HOME/.cache/huggingface/token" ]; then
     TOKEN=$(cat "$HOME/.cache/huggingface/token")
 fi
 
+needs_hf_download() {
+    if [ "${FORCE_HF_DOWNLOAD:-0}" -eq 1 ]; then
+        return 0
+    fi
+    case "$1" in
+        "$PRO_Q2_IMATRIX_FILE"|"$PRO_Q4_LAYERS00_30_FILE"|"$PRO_Q4_LAYERS31_OUTPUT_FILE")
+            return 0
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
+find_hf_command() {
+    if command -v hf >/dev/null 2>&1; then
+        printf '%s\n' hf
+        return 0
+    fi
+    for dir in "$HOME"/Library/Python/*/bin "$HOME"/.local/bin; do
+        if [ -x "$dir/hf" ]; then
+            printf '%s\n' "$dir/hf"
+            return 0
+        fi
+    done
+    return 1
+}
+
+local_download_name() {
+    if [ "${FLATTEN_DOWNLOADS:-0}" -eq 1 ]; then
+        basename "$1"
+    else
+        printf '%s\n' "$1"
+    fi
+}
+
+download_one_hf() {
+    file=$1
+    local_file=$(local_download_name "$file")
+    out="$OUT_DIR/$local_file"
+    hf_out="$OUT_DIR/$file"
+    part="$out.part"
+
+    mkdir -p "$(dirname "$out")"
+
+    if [ -s "$out" ]; then
+        echo "Already downloaded: $out"
+        return
+    fi
+
+    if [ -e "$part" ]; then
+        echo "Found curl partial download: $part" >&2
+        echo "The Hugging Face downloader cannot resume curl .part files." >&2
+        echo "Move or remove that partial download before retrying this target." >&2
+        exit 1
+    fi
+
+    HF_CMD=$(find_hf_command || true)
+    if [ -z "$HF_CMD" ]; then
+        echo "Large GGUF downloads require the official Hugging Face CLI." >&2
+        echo "Install it with:" >&2
+        echo "  python3 -m pip install -U huggingface_hub hf_xet" >&2
+        exit 1
+    fi
+
+    echo "Downloading $file"
+    echo "from https://huggingface.co/$REPO"
+    if [ -n "$HF_REVISION" ]; then
+        echo "revision $HF_REVISION"
+    fi
+    echo "using $HF_CMD download"
+    echo "If the download stops, run the same command again to resume it."
+
+    if [ -n "$TOKEN" ] && [ -n "$HF_REVISION" ]; then
+        "$HF_CMD" download "$REPO" "$file" --revision "$HF_REVISION" --repo-type model --local-dir "$OUT_DIR" --token "$TOKEN"
+    elif [ -n "$TOKEN" ]; then
+        "$HF_CMD" download "$REPO" "$file" --repo-type model --local-dir "$OUT_DIR" --token "$TOKEN"
+    elif [ -n "$HF_REVISION" ]; then
+        "$HF_CMD" download "$REPO" "$file" --revision "$HF_REVISION" --repo-type model --local-dir "$OUT_DIR"
+    else
+        "$HF_CMD" download "$REPO" "$file" --repo-type model --local-dir "$OUT_DIR"
+    fi
+
+    if [ "$hf_out" != "$out" ] && [ -s "$hf_out" ]; then
+        mv "$hf_out" "$out"
+        rmdir "$(dirname "$hf_out")" 2>/dev/null || true
+    fi
+
+    if [ ! -s "$out" ]; then
+        echo "Hugging Face download finished but expected file is missing: $out" >&2
+        exit 1
+    fi
+}
+
 download_one() {
     file=$1
-    out="$OUT_DIR/$file"
+    local_file=$(local_download_name "$file")
+    out="$OUT_DIR/$local_file"
     part="$out.part"
     aria2_part="$out.aria2"
     url="https://huggingface.co/$REPO/resolve/main/$file"
 
-    mkdir -p "$OUT_DIR"
+    if needs_hf_download "$file"; then
+        download_one_hf "$file"
+        return
+    fi
+
+    mkdir -p "$(dirname "$out")"
 
     if [ -e "$aria2_part" ]; then
         echo "Found incomplete aria2 download sidecar: $aria2_part" >&2
@@ -153,14 +390,32 @@ download_one() {
     mv "$part" "$out"
 }
 
-download_one "$MODEL_FILE"
+if [ -n "$MODEL_FILES" ]; then
+    for file in $MODEL_FILES; do
+        download_one "$file"
+    done
+else
+    download_one "$MODEL_FILE"
+fi
 
 if [ "$MODEL" = "mtp" ]; then
     echo
-    echo "MTP is an optional component for q2-imatrix, q4-imatrix, q2, and q4."
+    echo "MTP is an optional component for q2-imatrix, q2-q4-imatrix, and q4-imatrix."
     echo "Enable it explicitly, for example:"
     echo "  ./ds4 --mtp $OUT_DIR/$MTP_FILE --mtp-draft 2"
-else
+elif [ "$MODEL" = "dspark-support" ]; then
+    echo
+    echo "DSpark support downloaded. Enable it explicitly in greedy mode:"
+    echo "  ./ds4 --dspark -m ./ds4flash.gguf --mtp $OUT_DIR/$DSPARK_SUPPORT_FILE --temp 0"
+elif [ "$MODEL" = "laguna-dflash" ]; then
+    echo
+    echo "Laguna DFlash support downloaded. Enable it explicitly in greedy mode:"
+    echo "  ./ds4 -m <laguna-model.gguf> --dflash $OUT_DIR/$LAGUNA_DFLASH_FILE --temp 0"
+elif [ "$MODEL" = "pro-q4-layers00-30" ] || [ "$MODEL" = "pro-q4-layers31-output" ] || [ "$MODEL" = "pro-q4-split" ]; then
+    echo
+    echo "Downloaded PRO Q4 distributed split file(s). Use them with --layers,"
+    echo "for example coordinator layers 0:30 and worker layers 31:output."
+elif [ "$LINK_MODEL" -eq 1 ]; then
     cd "$ROOT"
     ln -sfn "$OUT_DIR/$MODEL_FILE" ds4flash.gguf
     echo "Linked ./ds4flash.gguf -> $OUT_DIR/$MODEL_FILE"

ds4.h

@@ -6,6 +6,8 @@
 #include <stdint.h>
 #include <stdio.h>
 
+#include "ds4_ssd.h"
+
 /* Public engine boundary.
  *
  * The CLI and server should treat ds4_engine as the loaded model and
@@ -50,23 +52,126 @@ typedef struct {
     float logprob;
 } ds4_token_score;
 
+#define DS4_DEFAULT_TEMPERATURE 1.0f
+#define DS4_DEFAULT_TOP_P 1.0f
+#define DS4_DEFAULT_MIN_P 0.05f
+
 typedef struct ds4_engine ds4_engine;
 typedef struct ds4_session ds4_session;
 
 typedef void (*ds4_session_progress_fn)(void *ud, const char *event, int current, int total);
+typedef bool (*ds4_session_cancel_fn)(void *ud);
+
+#define DS4_SESSION_SYNC_INTERRUPTED 2
+
+typedef enum {
+    DS4_DISTRIBUTED_NONE = 0,
+    DS4_DISTRIBUTED_COORDINATOR,
+    DS4_DISTRIBUTED_WORKER,
+} ds4_distributed_role;
+
+typedef struct {
+    uint32_t start;
+    uint32_t end;
+    bool has_output;
+    bool set;
+} ds4_distributed_layers;
+
+typedef struct {
+    ds4_distributed_role role;
+    ds4_distributed_layers layers;
+    const char *listen_host;
+    int listen_port;
+    const char *coordinator_host;
+    int coordinator_port;
+    uint32_t prefill_chunk;
+    uint32_t prefill_window;
+    uint32_t activation_bits;
+    bool replay_check;
+    bool debug;
+} ds4_distributed_options;
+
+/* Tensor parallelism: two identical machines run the model in lockstep and
+ * split the heavy per-layer matvecs, exchanging partial sums at gates inside
+ * the graph (see misc/METAL_TENSOR_PARALLELISM.md).  Each rank keeps one
+ * contiguous half of the routed experts resident; dense and shared weights
+ * remain replicated.  The leader owns prompt/sampling and listens; the worker
+ * dials in and mirrors every session sync/eval. */
+typedef enum {
+    DS4_TP_NONE = 0,
+    DS4_TP_LEADER,
+    DS4_TP_WORKER,
+} ds4_tp_role;
+
+typedef enum {
+    DS4_TP_TRANSPORT_AUTO = 0,
+    DS4_TP_TRANSPORT_RDMA,
+    DS4_TP_TRANSPORT_TCP,
+} ds4_tp_transport;
+
+typedef struct {
+    ds4_tp_role role;
+    bool requested;             /* --tensor-parallel with shared role options */
+    const char *listen_host;    /* leader listens here for the worker */
+    int listen_port;
+    const char *leader_host;    /* worker dials the leader */
+    int leader_port;
+    ds4_tp_transport transport;
+    const char *rdma_device;
+    int rdma_gid_index;
+    bool rdma_gid_index_set;
+    bool glm_token_prefill;
+    int debug_hash;             /* cross-check hidden state every N tokens */
+} ds4_tp_options;
 
 typedef struct {
     const char *model_path;
     const char *mtp_path;
+    const char *dflash_path;
     ds4_backend backend;
     int n_threads;
+    int context_size;
+    uint32_t prefill_chunk;
     int mtp_draft_tokens;
+    int dflash_draft_tokens;
+    float dflash_p_min;
     float mtp_margin;
+    float dspark_confidence_threshold;
     const char *directional_steering_file;
+    const char *expert_profile_path;
     float directional_steering_attn;
     float directional_steering_ffn;
+    int power_percent;
+    uint32_t ssd_streaming_cache_experts;
+    uint64_t ssd_streaming_cache_bytes;
+    uint32_t ssd_streaming_full_layers;
+    uint32_t ssd_streaming_preload_experts;
+    uint64_t simulate_used_memory_bytes;
     bool warm_weights;
     bool quality;
+    bool glm_mtp;
+    bool glm_mtp_timing;
+    bool dspark;
+    bool dspark_strict;
+    bool dflash_p_min_set;
+    bool dspark_confidence_threshold_set;
+    bool cuda_tensor_parallel;
+    bool ssd_streaming;
+    bool ssd_streaming_cold;
+    bool ssd_streaming_full_layers_set;
+    bool inspect_only;
+    /* Multi-GPU placement uses this to price per-layer KV storage. */
+    int placement_ctx_hint;
+    /* Server batch mode serializes execution and can share prefill scratch. */
+    bool share_session_prefill_workspace;
+    bool first_token_test;
+    bool metal_graph_test;
+    bool load_slice;
+    uint32_t load_layer_start;
+    uint32_t load_layer_end;
+    bool load_output;
+    ds4_distributed_options distributed;
+    ds4_tp_options tp;
 } ds4_engine_options;
 
 typedef void (*ds4_token_emit_fn)(void *ud, int token);
@@ -88,16 +193,87 @@ typedef struct {
     uint64_t cap;
 } ds4_session_snapshot;
 
+typedef struct {
+    char *path;
+    uint64_t bytes;
+} ds4_session_payload_file;
+
 int ds4_engine_open(ds4_engine **out, const ds4_engine_options *opt);
+
+/* Multi-GPU pipeline-parallel entry point (wave 2).
+ *
+ * Accepts an optional ds4_gpu_config (defined in ds4_gpu_mgpu.h) that
+ * lets callers describe a multi-GPU placement target. Passing NULL is
+ * back-compatible with ds4_engine_open and produces identical engine
+ * state — bit-equivalent execution at runtime.
+ *
+ * When a non-NULL config is supplied AND the computed placement spans
+ * more than one tier (either multiple GPUs or any CPU-spill), this
+ * wave-2 implementation prints the layout and refuses to open: full
+ * multi-tier execution wiring lands in a follow-up task
+ * (mgpu-graph-session-execution). Callers receive a non-zero return
+ * and a documented stderr notice. */
+/* ds4_gpu_config is declared in ds4_gpu_mgpu.h, which callers should
+ * include separately. We forward-declare it here so this header can be
+ * used as-is (callers passing NULL don't need the struct definition). */
+struct ds4_gpu_config;
+int ds4_engine_create_with_gpu_config(ds4_engine **out,
+                                       const ds4_engine_options *opt,
+                                       const struct ds4_gpu_config *gpu_cfg);
 void ds4_engine_close(ds4_engine *e);
 void ds4_engine_summary(ds4_engine *e);
+int ds4_engine_vocab_size(ds4_engine *e);
+uint32_t ds4_engine_prefill_chunk(ds4_engine *e);
+int ds4_engine_power(ds4_engine *e);
+int ds4_engine_set_power(ds4_engine *e, int power_percent);
+const char *ds4_engine_model_name(ds4_engine *e);
+int ds4_engine_layer_count(ds4_engine *e);
+/* Decode gate schedule for the TP transport; see ds4_tp_identity. */
+void ds4_engine_tp_gate_schedule(ds4_engine *e,
+                                 uint32_t *start,
+                                 uint32_t *step,
+                                 uint32_t *per_token);
+uint32_t ds4_engine_layer_compress_ratio(ds4_engine *e, uint32_t layer);
+uint64_t ds4_engine_hidden_f32_values(ds4_engine *e);
+int ds4_engine_embd_dim(ds4_engine *e);
+uint64_t ds4_engine_model_bytes(ds4_engine *e);
+int ds4_engine_tp_vocab_split(ds4_engine *e);
+bool ds4_engine_glm_layer_payload_bytes(ds4_engine *e,
+                                        uint32_t layer,
+                                        uint32_t full_live,
+                                        uint32_t key_dim,
+                                        uint32_t value_dim,
+                                        uint32_t compact_live,
+                                        uint32_t index_live,
+                                        uint64_t *out);
+/* Stable id for cache compatibility.  0 is the original Flash shape, so old
+ * KV files with the previously-zero reserved byte remain Flash-compatible;
+ * Pro and later shapes must use nonzero ids. */
+int ds4_engine_model_id(ds4_engine *e);
+bool ds4_engine_is_glm_dsa(ds4_engine *e);
+bool ds4_engine_is_laguna(ds4_engine *e);
+const char *ds4_engine_default_system_prompt(ds4_engine *e);
+void ds4_engine_sampling_defaults(ds4_engine *e, float *temperature,
+                                  int *top_k, float *top_p, float *min_p);
 const char *ds4_backend_name(ds4_backend backend);
 bool ds4_think_mode_enabled(ds4_think_mode mode);
 const char *ds4_think_mode_name(ds4_think_mode mode);
 const char *ds4_think_max_prefix(void);
+const char *ds4_glm_reasoning_effort_text(ds4_think_mode mode);
 uint32_t ds4_think_max_min_context(void);
 ds4_think_mode ds4_think_mode_for_context(ds4_think_mode mode, int ctx_size);
+/* Uses the active model shape selected by ds4_engine_open(); call after opening
+ * the GGUF so Flash/Pro dimensions are known. */
 ds4_context_memory ds4_context_memory_estimate(ds4_backend backend, int ctx_size);
+ds4_context_memory ds4_context_memory_estimate_with_prefill(
+        ds4_backend backend,
+        int ctx_size,
+        uint32_t prefill_chunk);
+ds4_context_memory ds4_context_memory_estimate_with_prefill_mode(
+        ds4_backend backend,
+        int ctx_size,
+        uint32_t prefill_chunk,
+        bool ssd_streaming);
 bool ds4_log_is_tty(FILE *fp);
 void ds4_log(FILE *fp, ds4_log_type type, const char *fmt, ...);
 int ds4_engine_generate_argmax(ds4_engine *e, const ds4_tokens *prompt,
@@ -138,13 +314,46 @@ void ds4_encode_chat_prompt(
 void ds4_chat_append_max_effort_prefix(ds4_engine *e, ds4_tokens *tokens);
 void ds4_chat_append_message(ds4_engine *e, ds4_tokens *tokens, const char *role, const char *content);
 void ds4_chat_append_assistant_prefix(ds4_engine *e, ds4_tokens *tokens, ds4_think_mode think_mode);
+void ds4_chat_append_assistant_end(ds4_engine *e, ds4_tokens *tokens);
 
 char *ds4_token_text(ds4_engine *e, int token, size_t *len);
 int ds4_token_eos(ds4_engine *e);
+bool ds4_token_is_stop(ds4_engine *e, int token);
+bool ds4_token_is_thinking_control(ds4_engine *e, int token);
+bool ds4_token_is_stop_for_think_mode(ds4_engine *e,
+                                      int token,
+                                      ds4_think_mode mode);
+int ds4_token_user(ds4_engine *e);
+int ds4_token_assistant(ds4_engine *e);
+
+/* Tensor-parallel binding: allocates the GPU gate slab, registers it with
+ * the transport and arms the per-layer gate machinery.  Call once, after
+ * ds4_tp_create() and before any session work.  Transport lifecycle stays
+ * with the caller. */
+struct ds4_tp;
+int ds4_engine_tp_bind(ds4_engine *e, struct ds4_tp *tp, char *err, size_t errlen);
 
 int ds4_session_create(ds4_session **out, ds4_engine *e, int ctx_size);
 void ds4_session_free(ds4_session *s);
+int ds4_session_power(ds4_session *s);
+int ds4_session_set_power(ds4_session *s, int power_percent);
+bool ds4_session_is_distributed(ds4_session *s);
 void ds4_session_set_progress(ds4_session *s, ds4_session_progress_fn fn, void *ud);
+/* UI-only progress. It may report fine-grained progress inside a prefill chunk;
+ * callers must not treat it as a durable KV checkpoint boundary. */
+void ds4_session_set_display_progress(ds4_session *s, ds4_session_progress_fn fn, void *ud);
+/* Optional cooperative cancellation.  ds4_session_sync() checks it only at
+ * safe boundaries where the live checkpoint is either unchanged or represents a
+ * valid token prefix, and returns DS4_SESSION_SYNC_INTERRUPTED when it stops. */
+void ds4_session_set_cancel(ds4_session *s, ds4_session_cancel_fn fn, void *ud);
+/* Enable optional speculative support-model state for the next sync/decode
+ * sequence. Frontends disable it for sampling modes that cannot use a greedy
+ * verifier, avoiding support-model work on every ordinary decode token. */
+void ds4_session_set_speculative_enabled(ds4_session *s, bool enabled);
+void ds4_session_report_progress(ds4_session *s, const char *event, int current, int total);
+/* Distributed coordinator sessions return 1 when the full layer route is
+ * available, 0 when it is still incomplete, and -1 for a local API error. */
+int ds4_session_distributed_route_ready(ds4_session *s, char *err, size_t errlen);
 
 typedef enum {
     DS4_SESSION_REWRITE_ERROR = -1,
@@ -157,6 +366,7 @@ typedef enum {
 /* Synchronize the live session to a full prompt token prefix.  If the current
  * checkpoint is a prefix, only the suffix is evaluated; otherwise the backend
  * state is refilled from scratch. */
+#define DS4_SESSION_SYNC_INTERRUPTED 2
 int ds4_session_sync(ds4_session *s, const ds4_tokens *prompt, char *err, size_t errlen);
 bool ds4_session_rewrite_requires_rebuild(int live_len, int canonical_len, int common);
 ds4_session_rewrite_result ds4_session_rewrite_from_common(
@@ -165,30 +375,115 @@ ds4_session_rewrite_result ds4_session_rewrite_from_common(
 int ds4_session_common_prefix(ds4_session *s, const ds4_tokens *prompt);
 int ds4_session_argmax(ds4_session *s);
 int ds4_session_argmax_excluding(ds4_session *s, int excluded_id);
+int ds4_sample_logits(const float *logits, int n_vocab, float temperature,
+                      int top_k, float top_p, float min_p, uint64_t *rng);
 int ds4_session_sample(ds4_session *s, float temperature, int top_k, float top_p, float min_p, uint64_t *rng);
+#ifdef DS4_TEST_HOOKS
+int ds4_test_sample_logits(const float *logits, uint32_t n_vocab,
+                           float temperature, int top_k,
+                           float top_p, float min_p, uint64_t *rng,
+                           float *prob_scratch);
+uint64_t ds4_test_mixed_native_count(void);
+#endif
 int ds4_session_top_logprobs(ds4_session *s, ds4_token_score *out, int k);
 int ds4_session_token_logprob(ds4_session *s, int token, ds4_token_score *out);
+int ds4_session_copy_logits(ds4_session *s, float *out, int cap);
+int ds4_session_set_logits(ds4_session *s, const float *logits, int n);
+/* Pay the one-time first-submission GPU cost outside any measured window;
+ * used by the TP worker right after session create (no-op on CPU/GLM). */
+void ds4_session_gpu_warmup(ds4_session *s);
 int ds4_session_eval(ds4_session *s, int token, char *err, size_t errlen);
+
+typedef struct {
+    ds4_session *session;
+    int token;
+} ds4_decode_item;
+
+/* Advance independent sessions by one token each. Batch size one is exactly
+ * ds4_session_eval(). Backends without native batching use a correctness-first
+ * sequential fallback. */
+int ds4_sessions_eval_batch(ds4_decode_item *items, int count,
+                            char *err, size_t errlen);
+/* Advance one resumed prefill suffix and an independent decode batch as one
+ * scheduling step. Unsupported combinations use the ordinary serialized
+ * session operations. */
+int ds4_sessions_eval_batch_with_prefill(
+        ds4_decode_item *items, int count,
+        ds4_session *prefill_session, const ds4_tokens *prefill_prompt,
+        char *err, size_t errlen);
 int ds4_session_eval_speculative_argmax(ds4_session *s, int first_token,
                                         int max_tokens, int eos_token,
                                         int *accepted, int accepted_cap,
                                         char *err, size_t errlen);
+/* TP worker side of a mirrored speculative-verify block: run its half of the
+ * batch verify for KV side effects, then obey the leader's commit frame
+ * (keep, or roll back and replay). Only called from ds4_tp_worker_run. */
+int ds4_session_tp_spec_cycle(ds4_session *s, const int *drafts, int draft_n,
+                              char *err, size_t errlen);
 void ds4_session_invalidate(ds4_session *s);
 void ds4_session_rewind(ds4_session *s, int pos);
 int ds4_session_pos(ds4_session *s);
 int ds4_session_ctx(ds4_session *s);
+int ds4_session_prefill_cap(ds4_session *s);
 int ds4_engine_routed_quant_bits(ds4_engine *e);
+bool ds4_engine_has_output_head(ds4_engine *e);
 bool ds4_engine_has_mtp(ds4_engine *e);
 int ds4_engine_mtp_draft_tokens(ds4_engine *e);
 const ds4_tokens *ds4_session_tokens(ds4_session *s);
 
-/* Disk KV cache payload helpers.  The server owns the outer file header and
- * policy; the engine owns the DS4-specific serialized graph state. */
+/* Low-level graph slice entry points used by distributed inference.  The
+ * transport/session routing logic lives in ds4_distributed.c. */
+int ds4_session_layer_slice_reset(ds4_session *s, char *err, size_t errlen);
+int ds4_session_eval_layer_slice(ds4_session *s,
+                                 const int *tokens,
+                                 uint32_t n_tokens,
+                                 uint32_t pos0,
+                                 uint32_t layer_start,
+                                 uint32_t layer_end,
+                                 const float *input_hc,
+                                 float *output_hc,
+                                 bool output_logits,
+                                 float *logits,
+                                 char *err,
+                                 size_t errlen);
+int ds4_session_eval_output_head_from_hc(ds4_session *s,
+                                         const float *hidden_hc,
+                                         uint32_t n_tokens,
+                                         float *logits,
+                                         char *err,
+                                         size_t errlen);
+
+/* Disk KV payload helpers.  HTTP/agent code owns the outer file header and
+ * persistence policy; the engine owns the DS4-specific serialized graph state. */
+#define DS4_SESSION_PAYLOAD_MAGIC UINT32_C(0x34565344) /* "DSV4" */
+#define DS4_SESSION_PAYLOAD_VERSION UINT32_C(2)
+#define DS4_SESSION_PAYLOAD_U32_FIELDS 13u
+#define DS4_SESSION_LAYER_PAYLOAD_MAGIC UINT32_C(0x4c565344) /* "DSVL" */
+#define DS4_SESSION_LAYER_PAYLOAD_VERSION UINT32_C(1)
+#define DS4_SESSION_LAYER_PAYLOAD_U32_FIELDS 14u
+
 uint64_t ds4_session_payload_bytes(ds4_session *s);
+int ds4_session_stage_payload(ds4_session *s, ds4_session_payload_file *out,
+                              char *err, size_t errlen);
+int ds4_session_write_staged_payload(const ds4_session_payload_file *payload,
+                                     FILE *fp, char *err, size_t errlen);
+void ds4_session_payload_file_free(ds4_session_payload_file *payload);
 int ds4_session_save_payload(ds4_session *s, FILE *fp, char *err, size_t errlen);
 int ds4_session_load_payload(ds4_session *s, FILE *fp, uint64_t payload_bytes, char *err, size_t errlen);
 int ds4_session_save_snapshot(ds4_session *s, ds4_session_snapshot *snap, char *err, size_t errlen);
 int ds4_session_load_snapshot(ds4_session *s, const ds4_session_snapshot *snap, char *err, size_t errlen);
 void ds4_session_snapshot_free(ds4_session_snapshot *snap);
 
+uint64_t ds4_session_layer_payload_bytes(ds4_session *s,
+                                         uint32_t layer_start,
+                                         uint32_t layer_end);
+int ds4_session_save_layer_payload(ds4_session *s, FILE *fp,
+                                   uint32_t layer_start, uint32_t layer_end,
+                                   char *err, size_t errlen);
+int ds4_session_load_layer_payload(ds4_session *s, FILE *fp,
+                                   uint64_t payload_bytes,
+                                   const int *tokens, uint32_t n_tokens,
+                                   uint32_t layer_start, uint32_t layer_end,
+                                   char *err, size_t errlen);
+
 #endif

ds4_bench.c

@@ -1,13 +1,16 @@
 #include "ds4.h"
+#include "ds4_distributed.h"
+#include "ds4_gpu_args.h"
+#include "ds4_help.h"
 
 /* Purpose-built throughput benchmark.
  *
  * The benchmark walks one fixed token sequence to configurable context
  * frontiers, measuring only the newest prefill interval at each frontier.  It
- * then snapshots the live session in memory, performs a fixed greedy decode
- * run without allowing EOS, restores the snapshot, and continues to the next
- * frontier.  Snapshot save/restore time is intentionally outside both timing
- * windows.
+ * then snapshots the live session in memory when the payload is small enough,
+ * performs a fixed greedy decode run without allowing EOS, restores the
+ * snapshot or replays the prefix, and continues to the next frontier.  Snapshot
+ * save/restore time is intentionally outside both timing windows.
  */
 
 #include <errno.h>
@@ -20,12 +23,17 @@
 #include <string.h>
 #include <time.h>
 
+#define DS4_BENCH_DEFAULT_SNAPSHOT_MAX_BYTES (UINT64_C(1) << 30)
+
 typedef struct {
     const char *model_path;
     const char *prompt_path;
     const char *chat_prompt_path;
     const char *system;
     const char *csv_path;
+    const char *expert_profile_path;
+    const char *gpu_vram_arg;
+    const char *gpu_devices_arg;
     ds4_backend backend;
     int threads;
     int ctx_start;
@@ -33,9 +41,23 @@ typedef struct {
     int ctx_alloc;
     int step_incr;
     int gen_tokens;
+    int power_percent;
+    uint32_t prefill_chunk;
+    uint32_t ssd_streaming_cache_experts;
+    uint64_t ssd_streaming_cache_bytes;
+    uint32_t ssd_streaming_full_layers;
+    uint32_t ssd_streaming_preload_experts;
+    uint64_t simulate_used_memory_bytes;
     double step_mul;
+    const char *dump_frontier_logits_dir;
+    ds4_dist_options dist;
     bool warm_weights;
     bool quality;
+    bool ssd_streaming;
+    bool ssd_streaming_cold;
+    bool ssd_streaming_full_layers_set;
+    bool cuda_tensor_parallel;
+    bool show_output;
 } bench_config;
 
 static double bench_now_sec(void) {
@@ -44,42 +66,31 @@ static double bench_now_sec(void) {
     return (double)ts.tv_sec + (double)ts.tv_nsec / 1000000000.0;
 }
 
-static void usage(FILE *fp) {
-    fprintf(fp,
-        "Usage: ds4-bench --prompt-file FILE [options]\n"
-        "\n"
-        "Benchmarks instantaneous prefill and generation throughput at context\n"
-        "frontiers such as 2048, 4096, 6144, ... . Generation is always greedy,\n"
-        "runs for exactly --gen-tokens tokens, and skips EOS so every row is\n"
-        "comparable.\n"
-        "\n"
-        "Input:\n"
-        "  --prompt-file FILE\n"
-        "      Raw benchmark text. The fixed token sequence is sliced at each frontier.\n"
-        "  --chat-prompt-file FILE\n"
-        "      Render FILE as one no-thinking chat user message, then slice that sequence.\n"
-        "  -sys, --system TEXT\n"
-        "      System prompt used only with --chat-prompt-file.\n"
-        "\n"
-        "Model and backend:\n"
-        "  -m, --model FILE       GGUF model path. Default: ds4flash.gguf\n"
-        "  --metal | --cuda | --cpu | --backend NAME\n"
-        "      Select backend explicitly. Defaults to Metal on macOS, CUDA elsewhere.\n"
-        "  -t, --threads N        CPU helper threads.\n"
-        "  --quality              Prefer exact kernels where applicable.\n"
-        "  --warm-weights         Touch mapped tensor pages before benchmarking.\n"
-        "\n"
-        "Sweep:\n"
-        "  --ctx-start N          First measured frontier. Default: 2048\n"
-        "  --ctx-max N            Last measured frontier. Default: 32768\n"
-        "  --ctx-alloc N          Allocated context. Default: ctx-max + gen-tokens + 1\n"
-        "  --step-mul F           Multiplicative step. Default: 1\n"
-        "  --step-incr N          Linear step when --step-mul is 1. Default: 2048\n"
-        "  --gen-tokens N         Greedy decode tokens per frontier. Default: 128\n"
-        "\n"
-        "Output:\n"
-        "  --csv FILE             Write CSV there instead of stdout.\n"
-        "  -h, --help             Show this help.\n");
+static uint64_t bench_snapshot_max_bytes(void) {
+    const char *env = getenv("DS4_BENCH_SNAPSHOT_MAX_BYTES");
+    if (!env || env[0] == '\0') return DS4_BENCH_DEFAULT_SNAPSHOT_MAX_BYTES;
+    if (!strcmp(env, "unlimited") || !strcmp(env, "UNLIMITED") ||
+        !strcmp(env, "inf") || !strcmp(env, "INF")) {
+        return UINT64_MAX;
+    }
+    char *end = NULL;
+    unsigned long long v = strtoull(env, &end, 10);
+    if (env[0] == '\0' || !end || *end != '\0') {
+        fprintf(stderr,
+                "ds4-bench: invalid DS4_BENCH_SNAPSHOT_MAX_BYTES=%s; using default %llu\n",
+                env,
+                (unsigned long long)DS4_BENCH_DEFAULT_SNAPSHOT_MAX_BYTES);
+        return DS4_BENCH_DEFAULT_SNAPSHOT_MAX_BYTES;
+    }
+    return (uint64_t)v;
+}
+
+static double bytes_to_gib(uint64_t bytes) {
+    return (double)bytes / (1024.0 * 1024.0 * 1024.0);
+}
+
+static void usage(FILE *fp, const char *topic) {
+    ds4_help_print(fp, DS4_HELP_BENCH, topic);
 }
 
 static int parse_int(const char *s, const char *opt) {
@@ -92,6 +103,16 @@ static int parse_int(const char *s, const char *opt) {
     return (int)v;
 }
 
+static int parse_nonnegative_int(const char *s, const char *opt) {
+    char *end = NULL;
+    long v = strtol(s, &end, 10);
+    if (s[0] == '\0' || *end != '\0' || v < 0 || v > INT_MAX) {
+        fprintf(stderr, "ds4-bench: invalid value for %s: %s\n", opt, s);
+        exit(2);
+    }
+    return (int)v;
+}
+
 static double parse_double_arg(const char *s, const char *opt) {
     char *end = NULL;
     double v = strtod(s, &end);
@@ -112,10 +133,18 @@ static const char *need_arg(int *i, int argc, char **argv, const char *opt) {
 
 static ds4_backend parse_backend(const char *s, const char *opt) {
     if (!strcmp(s, "metal")) return DS4_BACKEND_METAL;
+#ifdef DS4_ROCM_BUILD
+    if (!strcmp(s, "rocm")) return DS4_BACKEND_CUDA;
+#else
     if (!strcmp(s, "cuda")) return DS4_BACKEND_CUDA;
+#endif
     if (!strcmp(s, "cpu")) return DS4_BACKEND_CPU;
     fprintf(stderr, "ds4-bench: invalid value for %s: %s\n", opt, s);
+#ifdef DS4_ROCM_BUILD
+    fprintf(stderr, "ds4-bench: valid backends are: metal, rocm, cpu\n");
+#else
     fprintf(stderr, "ds4-bench: valid backends are: metal, cuda, cpu\n");
+#endif
     exit(2);
 }
 
@@ -183,9 +212,29 @@ static bench_config parse_options(int argc, char **argv) {
     for (int i = 1; i < argc; i++) {
         const char *arg = argv[i];
         if (!strcmp(arg, "-h") || !strcmp(arg, "--help")) {
-            usage(stdout);
+            const char *topic = (i + 1 < argc && argv[i + 1][0] != '-') ?
+                argv[i + 1] : NULL;
+            usage(stdout, topic);
             exit(0);
-        } else if (!strcmp(arg, "-m") || !strcmp(arg, "--model")) {
+        }
+        char dist_parse_err[256] = {0};
+        ds4_dist_cli_parse_result dist_parse =
+            ds4_dist_parse_cli_arg(arg,
+                                   &i,
+                                   argc,
+                                   argv,
+                                   &c.dist,
+                                   dist_parse_err,
+                                   sizeof(dist_parse_err));
+        if (dist_parse == DS4_DIST_CLI_ERROR) {
+            fprintf(stderr,
+                    "ds4-bench: %s\n",
+                    dist_parse_err[0] ? dist_parse_err : "invalid distributed option");
+            exit(2);
+        }
+        if (dist_parse == DS4_DIST_CLI_MATCHED) continue;
+
+        if (!strcmp(arg, "-m") || !strcmp(arg, "--model")) {
             c.model_path = need_arg(&i, argc, argv, arg);
         } else if (!strcmp(arg, "--prompt-file")) {
             c.prompt_path = need_arg(&i, argc, argv, arg);
@@ -204,26 +253,84 @@ static bench_config parse_options(int argc, char **argv) {
         } else if (!strcmp(arg, "--step-mul")) {
             c.step_mul = parse_double_arg(need_arg(&i, argc, argv, arg), arg);
         } else if (!strcmp(arg, "--gen-tokens") || !strcmp(arg, "--tokens") || !strcmp(arg, "-n")) {
-            c.gen_tokens = parse_int(need_arg(&i, argc, argv, arg), arg);
+            c.gen_tokens = parse_nonnegative_int(need_arg(&i, argc, argv, arg), arg);
         } else if (!strcmp(arg, "--csv")) {
             c.csv_path = need_arg(&i, argc, argv, arg);
+        } else if (!strcmp(arg, "--dump-frontier-logits-dir")) {
+            c.dump_frontier_logits_dir = need_arg(&i, argc, argv, arg);
+        } else if (!strcmp(arg, "--expert-profile")) {
+            c.expert_profile_path = need_arg(&i, argc, argv, arg);
         } else if (!strcmp(arg, "-t") || !strcmp(arg, "--threads")) {
             c.threads = parse_int(need_arg(&i, argc, argv, arg), arg);
         } else if (!strcmp(arg, "--backend")) {
             c.backend = parse_backend(need_arg(&i, argc, argv, arg), arg);
         } else if (!strcmp(arg, "--metal")) {
             c.backend = DS4_BACKEND_METAL;
+#ifdef DS4_ROCM_BUILD
+        } else if (!strcmp(arg, "--rocm")) {
+            c.backend = DS4_BACKEND_CUDA;
+#else
         } else if (!strcmp(arg, "--cuda")) {
             c.backend = DS4_BACKEND_CUDA;
+#endif
+        } else if (!strcmp(arg, "--gpu-vram")) {
+            c.gpu_vram_arg = need_arg(&i, argc, argv, arg);
+        } else if (!strcmp(arg, "--gpu-devices")) {
+            c.gpu_devices_arg = need_arg(&i, argc, argv, arg);
+        } else if (!strcmp(arg, "--cuda-tensor-parallel")) {
+            c.cuda_tensor_parallel = true;
         } else if (!strcmp(arg, "--cpu")) {
             c.backend = DS4_BACKEND_CPU;
         } else if (!strcmp(arg, "--quality")) {
             c.quality = true;
+        } else if (!strcmp(arg, "--ssd-streaming")) {
+            c.ssd_streaming = true;
+        } else if (!strcmp(arg, "--ssd-streaming-cold")) {
+            c.ssd_streaming_cold = true;
+        } else if (!strcmp(arg, "--ssd-streaming-cache-experts")) {
+            uint32_t experts = 0;
+            uint64_t bytes = 0;
+            if (!ds4_parse_streaming_cache_experts_arg(
+                    need_arg(&i, argc, argv, arg), &experts, &bytes)) {
+                fprintf(stderr,
+                        "ds4-bench: --ssd-streaming-cache-experts must be a positive count or <number>GB\n");
+                exit(2);
+            }
+            c.ssd_streaming_cache_experts = experts;
+            c.ssd_streaming_cache_bytes = bytes;
+        } else if (!strcmp(arg, "--ssd-streaming-full-layers")) {
+            int v = parse_nonnegative_int(need_arg(&i, argc, argv, arg), arg);
+            c.ssd_streaming_full_layers = (uint32_t)v;
+            c.ssd_streaming_full_layers_set = true;
+        } else if (!strcmp(arg, "--ssd-streaming-preload-experts")) {
+            int v = parse_int(need_arg(&i, argc, argv, arg), arg);
+            if (v <= 0) {
+                fprintf(stderr, "ds4-bench: --ssd-streaming-preload-experts must be positive\n");
+                exit(2);
+            }
+            c.ssd_streaming_preload_experts = (uint32_t)v;
+        } else if (!strcmp(arg, "--simulate-used-memory")) {
+            if (!ds4_parse_gib_arg(need_arg(&i, argc, argv, arg),
+                                   &c.simulate_used_memory_bytes)) {
+                fprintf(stderr,
+                        "ds4-bench: --simulate-used-memory must be a positive GiB value, e.g. 64GB\n");
+                exit(2);
+            }
+        } else if (!strcmp(arg, "--prefill-chunk")) {
+            c.prefill_chunk = (uint32_t)parse_int(need_arg(&i, argc, argv, arg), arg);
+        } else if (!strcmp(arg, "--power")) {
+            c.power_percent = parse_int(need_arg(&i, argc, argv, arg), arg);
+            if (c.power_percent < 1 || c.power_percent > 100) {
+                fprintf(stderr, "ds4-bench: --power must be between 1 and 100\n");
+                exit(2);
+            }
         } else if (!strcmp(arg, "--warm-weights")) {
             c.warm_weights = true;
+        } else if (!strcmp(arg, "--show-output")) {
+            c.show_output = true;
         } else {
             fprintf(stderr, "ds4-bench: unknown option: %s\n", arg);
-            usage(stderr);
+            usage(stderr, NULL);
             exit(2);
         }
     }
@@ -253,9 +360,114 @@ static bench_config parse_options(int argc, char **argv) {
         fprintf(stderr, "ds4-bench: --ctx-alloc must be greater than ctx-max + gen-tokens\n");
         exit(2);
     }
+    char dist_err[256];
+    if (ds4_dist_prepare_engine_options(&c.dist, NULL, dist_err, sizeof(dist_err)) != 0) {
+        fprintf(stderr, "ds4-bench: %s\n", dist_err);
+        exit(2);
+    }
+    if (c.dist.role == DS4_DISTRIBUTED_WORKER) {
+        fprintf(stderr, "ds4-bench: --role worker is a serving mode; start workers with ./ds4\n");
+        exit(2);
+    }
     return c;
 }
 
+static void json_write_string(FILE *fp, const char *s) {
+    fputc('"', fp);
+    if (s) {
+        for (const unsigned char *p = (const unsigned char *)s; *p; p++) {
+            switch (*p) {
+            case '"':  fputs("\\\"", fp); break;
+            case '\\': fputs("\\\\", fp); break;
+            case '\b': fputs("\\b", fp); break;
+            case '\f': fputs("\\f", fp); break;
+            case '\n': fputs("\\n", fp); break;
+            case '\r': fputs("\\r", fp); break;
+            case '\t': fputs("\\t", fp); break;
+            default:
+                if (*p < 0x20) fprintf(fp, "\\u%04x", (unsigned)*p);
+                else fputc((char)*p, fp);
+                break;
+            }
+        }
+    }
+    fputc('"', fp);
+}
+
+static int write_frontier_logits_json(
+        const bench_config *cfg,
+        ds4_engine         *engine,
+        ds4_session        *session,
+        int                 frontier,
+        int                 previous) {
+    if (!cfg->dump_frontier_logits_dir) return 0;
+
+    const int vocab = ds4_engine_vocab_size(engine);
+    float *logits = malloc((size_t)vocab * sizeof(logits[0]));
+    if (!logits) {
+        fprintf(stderr, "ds4-bench: out of memory copying frontier logits\n");
+        return 1;
+    }
+    if (ds4_session_copy_logits(session, logits, vocab) != vocab) {
+        fprintf(stderr, "ds4-bench: failed to copy frontier logits at %d\n", frontier);
+        free(logits);
+        return 1;
+    }
+
+    char path[PATH_MAX];
+    const int n = snprintf(path,
+                           sizeof(path),
+                           "%s/frontier_%06d.logits.json",
+                           cfg->dump_frontier_logits_dir,
+                           frontier);
+    if (n <= 0 || (size_t)n >= sizeof(path)) {
+        fprintf(stderr, "ds4-bench: frontier logits path is too long\n");
+        free(logits);
+        return 1;
+    }
+
+    FILE *fp = fopen(path, "wb");
+    if (!fp) {
+        fprintf(stderr, "ds4-bench: failed to open %s: %s\n", path, strerror(errno));
+        free(logits);
+        return 1;
+    }
+
+    const int argmax = ds4_session_argmax(session);
+    fprintf(fp, "{\n  \"source\":\"ds4-bench\",\n  \"model\":");
+    json_write_string(fp, cfg->model_path);
+    fprintf(fp,
+            ",\n  \"backend\":\"%s\",\n  \"quality\":%s,\n"
+            "  \"quant_bits\":%d,\n  \"prompt_tokens\":%d,\n"
+            "  \"frontier_tokens\":%d,\n  \"prefill_tokens\":%d,\n"
+            "  \"ctx\":%d,\n  \"vocab\":%d,\n"
+            "  \"argmax_id\":%d,\n  \"argmax_logit\":%.9g,\n  \"logits\":[",
+            ds4_backend_name(cfg->backend),
+            cfg->quality ? "true" : "false",
+            ds4_engine_routed_quant_bits(engine),
+            frontier,
+            frontier,
+            frontier - previous,
+            cfg->ctx_alloc,
+            vocab,
+            argmax,
+            logits[argmax]);
+    for (int i = 0; i < vocab; i++) {
+        if (i) fputc(',', fp);
+        if ((i % 8) == 0) fputs("\n    ", fp);
+        if (isfinite(logits[i])) fprintf(fp, "%.9g", logits[i]);
+        else fputs("null", fp);
+    }
+    fputs("\n  ]\n}\n", fp);
+    if (fclose(fp) != 0) {
+        fprintf(stderr, "ds4-bench: failed to close %s\n", path);
+        free(logits);
+        return 1;
+    }
+    free(logits);
+    return 0;
+}
+
 static int next_frontier(const bench_config *c, int cur) {
     if (cur >= c->ctx_max) return c->ctx_max;
     int next;
@@ -271,8 +483,15 @@ static int next_frontier(const bench_config *c, int cur) {
     return next;
 }
 
-static void log_context_memory(ds4_backend backend, int ctx_size) {
-    ds4_context_memory m = ds4_context_memory_estimate(backend, ctx_size);
+static void log_context_memory(ds4_backend backend,
+                               int         ctx_size,
+                               uint32_t    prefill_chunk,
+                               bool        ssd_streaming) {
+    ds4_context_memory m =
+        ds4_context_memory_estimate_with_prefill_mode(backend,
+                                                      ctx_size,
+                                                      prefill_chunk,
+                                                      ssd_streaming);
     fprintf(stderr,
             "ds4-bench: context buffers %.2f MiB (ctx=%d, backend=%s, prefill_chunk=%u, raw_kv_rows=%u, compressed_kv_rows=%u)\n",
             (double)m.total_bytes / (1024.0 * 1024.0),
@@ -283,19 +502,124 @@ static void log_context_memory(ds4_backend backend, int ctx_size) {
             m.comp_cap);
 }
 
+static int wait_distributed_route(ds4_session *session) {
+    char err[256] = {0};
+    char last[256] = {0};
+    unsigned ticks = 0;
+    const struct timespec delay = {0, 250000000L};
+
+    for (;;) {
+        int ready = ds4_session_distributed_route_ready(session, err, sizeof(err));
+        if (ready > 0) {
+            if (ticks) fprintf(stderr, "ds4-bench: distributed route ready\n");
+            return 0;
+        }
+        if (ready < 0) {
+            fprintf(stderr,
+                    "ds4-bench: distributed route readiness failed: %s\n",
+                    err[0] ? err : "unknown error");
+            return 1;
+        }
+        const char *why = err[0] ? err : "route incomplete";
+        if (strcmp(last, why) != 0 || (ticks % 20u) == 0) {
+            fprintf(stderr, "ds4-bench: waiting for distributed route: %s\n", why);
+            snprintf(last, sizeof(last), "%s", why);
+        }
+        nanosleep(&delay, NULL);
+        ticks++;
+    }
+}
+
+static void maybe_warn_distributed_step_shape(const bench_config *cfg, ds4_session *session) {
+    if (!cfg || !session || cfg->dist.role != DS4_DISTRIBUTED_COORDINATOR) return;
+    uint32_t chunk = cfg->dist.prefill_chunk;
+    if (chunk == 0) {
+        const int cap = ds4_session_prefill_cap(session);
+        if (cap > 0) chunk = (uint32_t)cap;
+    }
+    if (chunk == 0) return;
+    if (cfg->step_mul == 1.0 &&
+        cfg->step_incr > 0 &&
+        (uint32_t)cfg->step_incr < chunk &&
+        cfg->ctx_start < cfg->ctx_max)
+    {
+        fprintf(stderr,
+                "ds4-bench: note: --step-incr=%d is smaller than distributed prefill chunk %u; "
+                "suffix rows will not show multi-chunk pipeline overlap\n",
+                cfg->step_incr,
+                chunk);
+    }
+}
+
 int main(int argc, char **argv) {
     bench_config cfg = parse_options(argc, argv);
-    log_context_memory(cfg.backend, cfg.ctx_alloc);
+
+    /* Hint the packer at the largest ctx this bench run will exercise
+     * so per-layer KV bytes are priced for the real session size, not
+     * a stale 4096 default. Single-tier and CPU paths ignore this. */
+    int placement_ctx_hint = cfg.ctx_max;
+    if (cfg.ctx_alloc > placement_ctx_hint) placement_ctx_hint = cfg.ctx_alloc;
+
+    ds4_gpu_config gpu_cfg = {0};
+    bool skip_cuda = false;
+    const bool have_gpu_config = cfg.gpu_vram_arg || cfg.gpu_devices_arg;
+    if (have_gpu_config) {
+        char gpu_err[256];
+        if (parse_gpu_vram_arg(cfg.gpu_vram_arg, cfg.gpu_devices_arg,
+                               &gpu_cfg, &skip_cuda,
+                               gpu_err, sizeof(gpu_err)) != 0) {
+            fprintf(stderr, "ds4-bench: %s\n", gpu_err);
+            return 2;
+        }
+        cfg.backend = skip_cuda ? DS4_BACKEND_CPU : DS4_BACKEND_CUDA;
+    }
 
     ds4_engine_options opt = {
         .model_path = cfg.model_path,
         .backend = cfg.backend,
         .n_threads = cfg.threads,
+        .context_size = cfg.ctx_alloc,
+        .prefill_chunk = cfg.prefill_chunk,
+        .ssd_streaming_cache_experts = cfg.ssd_streaming_cache_experts,
+        .ssd_streaming_cache_bytes = cfg.ssd_streaming_cache_bytes,
+        .ssd_streaming_full_layers = cfg.ssd_streaming_full_layers,
+        .ssd_streaming_preload_experts = cfg.ssd_streaming_preload_experts,
+        .simulate_used_memory_bytes = cfg.simulate_used_memory_bytes,
+        .power_percent = cfg.power_percent,
         .warm_weights = cfg.warm_weights,
         .quality = cfg.quality,
+        .cuda_tensor_parallel = cfg.cuda_tensor_parallel,
+        .ssd_streaming = cfg.ssd_streaming,
+        .ssd_streaming_cold = cfg.ssd_streaming_cold,
+        .ssd_streaming_full_layers_set = cfg.ssd_streaming_full_layers_set,
+        .expert_profile_path = cfg.expert_profile_path,
+        .distributed = cfg.dist,
     };
+    char dist_err[256];
+    if (ds4_dist_prepare_engine_options(&cfg.dist, &opt, dist_err, sizeof(dist_err)) != 0) {
+        fprintf(stderr, "ds4-bench: %s\n", dist_err);
+        return 2;
+    }
     ds4_engine *engine = NULL;
-    if (ds4_engine_open(&engine, &opt) != 0) return 1;
+    if (have_gpu_config && !skip_cuda) {
+        const bool was_auto =
+            (cfg.gpu_vram_arg && !strcmp(cfg.gpu_vram_arg, "auto")) ||
+            (!cfg.gpu_vram_arg && cfg.gpu_devices_arg);
+        char layout[256];
+        if (format_gpu_layout_line(&gpu_cfg, was_auto,
+                                   layout, sizeof(layout)) > 0) {
+            fprintf(stdout, "%s\n", layout);
+            fflush(stdout);
+        }
+        if (ds4_engine_create_with_gpu_config(
+                &engine, &opt, &gpu_cfg) != 0) return 1;
+    } else if (ds4_engine_open(&engine, &opt) != 0) {
+        return 1;
+    }
+    log_context_memory(opt.backend,
+                       cfg.ctx_alloc,
+                       ds4_engine_prefill_chunk(engine),
+                       cfg.ssd_streaming);
 
     char *text = read_file(cfg.prompt_path ? cfg.prompt_path : cfg.chat_prompt_path);
     ds4_tokens prompt = {0};
@@ -323,6 +647,15 @@ int main(int argc, char **argv) {
         ds4_engine_close(engine);
         return 1;
     }
+    if (cfg.dist.role == DS4_DISTRIBUTED_COORDINATOR &&
+        wait_distributed_route(session) != 0)
+    {
+        ds4_session_free(session);
+        ds4_tokens_free(&prompt);
+        ds4_engine_close(engine);
+        return 1;
+    }
+    maybe_warn_distributed_step_shape(&cfg, session);
 
     FILE *out = stdout;
     if (cfg.csv_path) {
@@ -335,11 +668,14 @@ int main(int argc, char **argv) {
             return 1;
         }
     }
-    fprintf(out, "ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes\n");
+    fprintf(out, "ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes\n");
     fflush(out);
 
     const int eos = ds4_token_eos(engine);
+    const bool distributed = cfg.dist.role == DS4_DISTRIBUTED_COORDINATOR;
     ds4_session_snapshot snap = {0};
+    const uint64_t snapshot_max_bytes = bench_snapshot_max_bytes();
+    bool warned_large_snapshot = false;
     char err[256];
     int previous = 0;
     int rc = 0;
@@ -361,13 +697,46 @@ int main(int argc, char **argv) {
         const double prefill_sec = prefill_t1 - prefill_t0;
         const int prefill_tokens = frontier - previous;
 
-        if (ds4_session_save_snapshot(session, &snap, err, sizeof(err)) != 0) {
-            fprintf(stderr, "ds4-bench: snapshot at %d failed: %s\n", frontier, err);
+        if (write_frontier_logits_json(&cfg, engine, session, frontier, previous) != 0) {
             rc = 1;
             break;
         }
 
+        const bool need_restore_after_generation =
+            cfg.gen_tokens > 0 && frontier < cfg.ctx_max;
+        bool have_snapshot = false;
+        if (need_restore_after_generation && !distributed &&
+            getenv("DS4_BENCH_DISABLE_SNAPSHOT") == NULL) {
+            const uint64_t payload_bytes = ds4_session_payload_bytes(session);
+            const bool large_snapshot_forced =
+                getenv("DS4_BENCH_FORCE_SNAPSHOT") != NULL;
+            if (payload_bytes > snapshot_max_bytes && !large_snapshot_forced) {
+                if (!warned_large_snapshot) {
+                    fprintf(stderr,
+                            "ds4-bench: session payload snapshot is %.2f GiB, above the %.2f GiB benchmark limit; "
+                            "replaying prefixes instead (set DS4_BENCH_FORCE_SNAPSHOT=1 to force snapshots)\n",
+                            bytes_to_gib(payload_bytes),
+                            bytes_to_gib(snapshot_max_bytes));
+                    warned_large_snapshot = true;
+                }
+            } else if (payload_bytes > 0) {
+                if (ds4_session_save_snapshot(session, &snap, err, sizeof(err)) != 0) {
+                    fprintf(stderr, "ds4-bench: snapshot at %d failed: %s\n", frontier, err);
+                    rc = 1;
+                    break;
+                }
+                have_snapshot = true;
+            }
+        }
+
         const double gen_t0 = bench_now_sec();
+        double gen_first_sec = 0.0;
+        double gen_steady_sec = 0.0;
+        int gen_done = 0;
+        int *gen_token_buf = cfg.show_output && cfg.gen_tokens > 0
+            ? malloc((size_t)cfg.gen_tokens * sizeof(gen_token_buf[0]))
+            : NULL;
+        int gen_token_count = 0;
         for (int i = 0; i < cfg.gen_tokens; i++) {
             if (ds4_session_pos(session) + 1 >= ds4_session_ctx(session)) {
                 fprintf(stderr, "ds4-bench: generation would exceed allocated context at frontier %d\n", frontier);
@@ -380,30 +749,64 @@ int main(int argc, char **argv) {
                 rc = 1;
                 break;
             }
+            const double token_t0 = bench_now_sec();
             if (ds4_session_eval(session, token, err, sizeof(err)) != 0) {
                 fprintf(stderr, "ds4-bench: decode at frontier %d failed: %s\n", frontier, err);
                 rc = 1;
                 break;
             }
+            const double token_t1 = bench_now_sec();
+            if (i == 0) gen_first_sec = token_t1 - token_t0;
+            else gen_steady_sec += token_t1 - token_t0;
+            if (gen_token_buf) gen_token_buf[gen_token_count++] = token;
+            gen_done++;
         }
         const double gen_t1 = bench_now_sec();
+        if (cfg.show_output && gen_token_buf && gen_token_count > 0) {
+            fprintf(stderr, "ds4-bench: gen[ctx=%d] decoded text: \"", frontier);
+            for (int i = 0; i < gen_token_count; i++) {
+                size_t tlen = 0;
+                char *txt = ds4_token_text(engine, gen_token_buf[i], &tlen);
+                if (txt) {
+                    fwrite(txt, 1, tlen, stderr);
+                    free(txt);
+                }
+            }
+            fprintf(stderr, "\"\n");
+            fflush(stderr);
+        }
+        free(gen_token_buf);
         if (rc != 0) break;
 
-        if (ds4_session_load_snapshot(session, &snap, err, sizeof(err)) != 0) {
-            fprintf(stderr, "ds4-bench: restore at %d failed: %s\n", frontier, err);
-            rc = 1;
-            break;
+        if (!need_restore_after_generation) {
+            /* Nothing later depends on the frontier state. */
+        } else if (distributed || !have_snapshot) {
+            if (ds4_session_sync(session, &prefix, err, sizeof(err)) != 0) {
+                fprintf(stderr, "ds4-bench: replay restore at %d failed: %s\n", frontier, err);
+                rc = 1;
+                break;
+            }
+        } else {
+            if (ds4_session_load_snapshot(session, &snap, err, sizeof(err)) != 0) {
+                fprintf(stderr, "ds4-bench: restore at %d failed: %s\n", frontier, err);
+                rc = 1;
+                break;
+            }
         }
 
         const double gen_sec = gen_t1 - gen_t0;
+        const int gen_steady_tokens = gen_done > 1 ? gen_done - 1 : 0;
         fprintf(out,
-                "%d,%d,%.2f,%d,%.2f,%llu\n",
+                "%d,%d,%.2f,%d,%.2f,%.3f,%d,%.2f,%llu\n",
                 frontier,
                 prefill_tokens,
                 prefill_sec > 0.0 ? (double)prefill_tokens / prefill_sec : 0.0,
-                cfg.gen_tokens,
-                gen_sec > 0.0 ? (double)cfg.gen_tokens / gen_sec : 0.0,
-                (unsigned long long)snap.len);
+                gen_done,
+                gen_sec > 0.0 ? (double)gen_done / gen_sec : 0.0,
+                gen_first_sec * 1000.0,
+                gen_steady_tokens,
+                gen_steady_sec > 0.0 ? (double)gen_steady_tokens / gen_steady_sec : 0.0,
+                (unsigned long long)(have_snapshot ? snap.len : 0));
         fflush(out);
 
         previous = frontier;

ds4_distributed.h

@@ -0,0 +1,125 @@
+#ifndef DS4_DISTRIBUTED_H
+#define DS4_DISTRIBUTED_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#include "ds4.h"
+
+/* Distributed inference is an engine backend, not a separate frontend API.
+ * Programs parse distributed options here, then keep using the normal
+ * ds4_session_* calls. Only worker/coordinator serving modes call
+ * ds4_dist_run() directly.
+ */
+typedef ds4_distributed_options ds4_dist_options;
+typedef struct ds4_dist_session ds4_dist_session;
+
+/* Options used by standalone `./ds4 --role coordinator -p ...` generation.
+ * Interactive tools and the server go through the normal ds4_session API.
+ */
+typedef struct {
+    const char *prompt;
+    const char *system;
+    const char *dump_logits_path;
+    const char *dump_logprobs_path;
+    int dump_logprobs_top_k;
+    int n_predict;
+    int ctx_size;
+    float temperature;
+    float top_p;
+    float min_p;
+    uint64_t seed;
+    ds4_think_mode think_mode;
+} ds4_dist_generation_options;
+
+typedef enum {
+    DS4_DIST_CLI_ERROR = -1,
+    DS4_DIST_CLI_NOT_MATCHED = 0,
+    DS4_DIST_CLI_MATCHED = 1,
+} ds4_dist_cli_parse_result;
+
+/* Shared option parsing. */
+bool ds4_dist_enabled(const ds4_dist_options *opt);
+ds4_dist_options *ds4_dist_options_create(void);
+void ds4_dist_options_free(ds4_dist_options *opt);
+void ds4_dist_usage(FILE *fp);
+ds4_dist_cli_parse_result ds4_dist_parse_cli_arg(
+        const char *arg,
+        int *index,
+        int argc,
+        char **argv,
+        ds4_dist_options *opt,
+        char *err,
+        size_t errlen);
+
+/* Applies distributed layer-loading choices to the engine options before the
+ * model is loaded.
+ */
+int ds4_dist_prepare_engine_options(
+        const ds4_dist_options *opt,
+        ds4_engine_options *engine,
+        char *err,
+        size_t errlen);
+
+/* Coordinator session backend used by ds4.c. These mirror the normal session
+ * operations; callers outside the engine should not need to call them directly.
+ */
+int ds4_dist_session_create(
+        ds4_dist_session **out,
+        ds4_engine *engine,
+        const ds4_dist_options *opt,
+        ds4_session *owner,
+        int ctx_size,
+        char *err,
+        size_t errlen);
+void ds4_dist_session_free(ds4_dist_session *d);
+
+/* Returns 1 when the coordinator has full layer coverage, 0 when workers are
+ * still missing, and -1 for configuration or internal errors.
+ */
+int ds4_dist_session_route_ready(ds4_dist_session *d, char *err, size_t errlen);
+
+/* Synchronize the distributed KV state to the requested prompt timeline. */
+int ds4_dist_session_sync(
+        ds4_dist_session *d,
+        ds4_session *owner,
+        const ds4_tokens *checkpoint,
+        const ds4_tokens *prompt,
+        float *logits,
+        char *err,
+        size_t errlen);
+
+/* Evaluate one additional token across the current distributed route. */
+int ds4_dist_session_eval(
+        ds4_dist_session *d,
+        ds4_session *owner,
+        const ds4_tokens *checkpoint,
+        int token,
+        float *logits,
+        char *err,
+        size_t errlen);
+
+/* Save/load use the normal DSV4 payload format. The coordinator gathers or
+ * pushes remote layer shards internally so saved files are topology-neutral.
+ */
+int ds4_dist_session_save_payload(
+        ds4_dist_session *d,
+        ds4_session *owner,
+        FILE *fp,
+        char *err,
+        size_t errlen);
+int ds4_dist_session_load_payload(
+        ds4_dist_session *d,
+        ds4_session *owner,
+        FILE *fp,
+        uint64_t payload_bytes,
+        char *err,
+        size_t errlen);
+
+/* Standalone distributed mode. Workers stay in this loop; coordinator one-shot
+ * mode uses it for `./ds4 --role coordinator -p ...`.
+ */
+int ds4_dist_run(ds4_engine *engine, const ds4_dist_options *opt, const ds4_dist_generation_options *gen);
+
+#endif

ds4_gpu_args.c

@@ -0,0 +1,230 @@
+/* ds4_gpu_args.c — shared parser for --gpu-vram and --gpu-devices.
+ *
+ * See ds4_gpu_args.h. The actual CUDA probing for the "auto" branch
+ * lives in ds4_cuda.cu so this .c file never includes <cuda_runtime.h>
+ * directly (mirroring the rest of the tree's separation between .c and
+ * .cu compilation units).
+ */
+#include "ds4_gpu_args.h"
+
+#include <ctype.h>
+#include <errno.h>
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#define DS4_GPU_ARGS_DEFAULT_SAFETY_MARGIN ((size_t)512 * 1024 * 1024)
+
+/* Helper: copy at most outlen-1 bytes of msg into out and NUL-terminate. */
+static void errbuf_set(char *out, size_t outlen, const char *msg) {
+    if (!out || outlen == 0) return;
+    size_t n = strlen(msg);
+    if (n >= outlen) n = outlen - 1;
+    memcpy(out, msg, n);
+    out[n] = '\0';
+}
+
+static void errbuf_setf(char *out, size_t outlen, const char *fmt, ...) {
+    if (!out || outlen == 0) return;
+    va_list ap;
+    va_start(ap, fmt);
+    vsnprintf(out, outlen, fmt, ap);
+    va_end(ap);
+}
+
+/* Maximum per-entry values. Generous, but small enough to keep all
+ * downstream arithmetic safely inside size_t/int. */
+#define DS4_GPU_ARGS_MAX_VRAM_GB        16384   /* 16 TiB per device */
+#define DS4_GPU_ARGS_MAX_DEVICE_INDEX   1023    /* CUDA visible-device ID cap */
+
+/* Parse a comma-separated list of nonnegative integers into out_count
+ * entries of out[]. Returns 0 on success, nonzero on parse error.
+ * Empty entries (",,") or trailing/leading commas error. Per-entry
+ * upper bound enforced via max_value (use LONG_MAX for "no cap"). */
+static int parse_csv_int_list(const char *s,
+                              long       *out,
+                              int        *out_count,
+                              int         max_count,
+                              long        max_value,
+                              char       *errbuf,
+                              size_t      errbuflen,
+                              const char *what) {
+    *out_count = 0;
+    if (!s || !*s) {
+        errbuf_setf(errbuf, errbuflen, "%s: empty value", what);
+        return 1;
+    }
+    const char *p = s;
+    while (*p) {
+        /* Disallow leading whitespace inside the list. */
+        while (*p == ' ' || *p == '\t') p++;
+        char *end = NULL;
+        errno = 0;
+        long v = strtol(p, &end, 10);
+        if (end == p) {
+            errbuf_setf(errbuf, errbuflen, "%s: not a number: '%s'", what, p);
+            return 1;
+        }
+        if (errno != 0 || v < 0 || v > max_value) {
+            errbuf_setf(errbuf, errbuflen,
+                        "%s: out-of-range value: '%s' (allowed 0..%ld)",
+                        what, p, max_value);
+            return 1;
+        }
+        if (*out_count >= max_count) {
+            errbuf_setf(errbuf, errbuflen, "%s: too many entries (max %d)", what, max_count);
+            return 1;
+        }
+        out[*out_count] = v;
+        (*out_count)++;
+        while (*end == ' ' || *end == '\t') end++;
+        if (*end == ',') {
+            end++;
+            if (*end == '\0' || *end == ',') {
+                errbuf_setf(errbuf, errbuflen, "%s: empty entry near '%s'", what, end);
+                return 1;
+            }
+        } else if (*end != '\0') {
+            errbuf_setf(errbuf, errbuflen, "%s: junk after value: '%s'", what, end);
+            return 1;
+        }
+        p = end;
+    }
+    if (*out_count == 0) {
+        errbuf_setf(errbuf, errbuflen, "%s: no values parsed", what);
+        return 1;
+    }
+    return 0;
+}
+
+int parse_gpu_vram_arg(const char     *vram_arg,
+                       const char     *devices_arg,
+                       ds4_gpu_config *out,
+                       bool           *out_skip_cuda,
+                       char           *errbuf,
+                       size_t          errbuflen) {
+    if (!out || !out_skip_cuda) {
+        errbuf_set(errbuf, errbuflen, "internal: NULL out parameter");
+        return 1;
+    }
+    *out_skip_cuda = false;
+    /* Caller zeroed *out — just confirm. */
+    out->n_gpus = 0;
+    out->safety_margin_bytes = DS4_GPU_ARGS_DEFAULT_SAFETY_MARGIN;
+
+    /* Parse devices_arg first so we have the filter when handling "auto". */
+    long dev_list[DS4_MAX_GPUS] = {0};
+    int  dev_count = 0;
+    if (devices_arg) {
+        if (parse_csv_int_list(devices_arg, dev_list, &dev_count,
+                                DS4_MAX_GPUS,
+                                DS4_GPU_ARGS_MAX_DEVICE_INDEX,
+                                errbuf, errbuflen,
+                                "--gpu-devices") != 0) {
+            return 1;
+        }
+    }
+
+    /* Case 1: vram_arg unset. */
+    if (vram_arg == NULL) {
+        if (devices_arg == NULL) {
+            errbuf_set(errbuf, errbuflen, "internal: parse_gpu_vram_arg called with both args NULL");
+            return 1;
+        }
+        /* --gpu-devices given without --gpu-vram: implicit auto with filter. */
+        vram_arg = "auto";
+    }
+
+    /* Case 2: --gpu-vram 0 — short-circuit. */
+    if (!strcmp(vram_arg, "0")) {
+        if (devices_arg) {
+            errbuf_set(errbuf, errbuflen,
+                       "--gpu-vram 0 conflicts with --gpu-devices");
+            return 1;
+        }
+        *out_skip_cuda = true;
+        return 0;
+    }
+
+    /* Case 3: --gpu-vram auto. */
+    if (!strcmp(vram_arg, "auto")) {
+#if !defined(DS4_NO_GPU) && !defined(__APPLE__)
+        int filter_buf[DS4_MAX_GPUS];
+        const int *filter_ptr = NULL;
+        if (dev_count > 0) {
+            for (int i = 0; i < dev_count; i++) filter_buf[i] = (int)dev_list[i];
+            filter_ptr = filter_buf;
+        }
+        if (ds4_gpu_args_probe_auto_cuda(filter_ptr, dev_count, out,
+                                          out->safety_margin_bytes,
+                                          errbuf, errbuflen) != 0) {
+            return 1;
+        }
+        return 0;
+#else
+        (void)dev_list; (void)dev_count;
+        errbuf_set(errbuf, errbuflen,
+                   "--gpu-vram auto requires a CUDA build");
+        return 1;
+#endif
+    }
+
+    /* Case 4: explicit GB budgets — "N[,N...]" */
+    long vram_list[DS4_MAX_GPUS] = {0};
+    int  vram_count = 0;
+    if (parse_csv_int_list(vram_arg, vram_list, &vram_count,
+                            DS4_MAX_GPUS,
+                            DS4_GPU_ARGS_MAX_VRAM_GB,
+                            errbuf, errbuflen,
+                            "--gpu-vram") != 0) {
+        return 1;
+    }
+    if (dev_count > 0 && dev_count != vram_count) {
+        errbuf_setf(errbuf, errbuflen,
+                    "--gpu-devices count (%d) does not match --gpu-vram count (%d)",
+                    dev_count, vram_count);
+        return 1;
+    }
+    out->n_gpus = vram_count;
+    for (int i = 0; i < vram_count; i++) {
+        int dev_id = (dev_count > 0) ? (int)dev_list[i] : i;
+        out->device_indices[i] = dev_id;
+        out->vram_bytes[i] = (size_t)vram_list[i] * (size_t)1024 * 1024 * 1024;
+    }
+    return 0;
+}
+
+int format_gpu_layout_line(const ds4_gpu_config *cfg,
+                           bool                  was_auto,
+                           char                 *out,
+                           size_t                outlen) {
+    if (!out || outlen == 0 || !cfg) return -1;
+    /* "ds4: GPU config: N devices [d0,d1,...] requested, budgets G0,G1,... GB; auto=<bool>" */
+    int written = snprintf(out, outlen,
+                           "ds4: GPU config: %d device%s [",
+                           cfg->n_gpus, cfg->n_gpus == 1 ? "" : "s");
+    if (written < 0 || (size_t)written >= outlen) return written;
+    for (int i = 0; i < cfg->n_gpus; i++) {
+        int n = snprintf(out + written, outlen - (size_t)written,
+                         "%s%d", i ? "," : "", cfg->device_indices[i]);
+        if (n < 0 || (size_t)(written + n) >= outlen) return written + n;
+        written += n;
+    }
+    int n = snprintf(out + written, outlen - (size_t)written,
+                     "] requested, budgets ");
+    if (n < 0 || (size_t)(written + n) >= outlen) return written + n;
+    written += n;
+    for (int i = 0; i < cfg->n_gpus; i++) {
+        double gb = (double)cfg->vram_bytes[i] / (1024.0 * 1024.0 * 1024.0);
+        int m = snprintf(out + written, outlen - (size_t)written,
+                         "%s%.0f", i ? "," : "", gb);
+        if (m < 0 || (size_t)(written + m) >= outlen) return written + m;
+        written += m;
+    }
+    int t = snprintf(out + written, outlen - (size_t)written,
+                     " GB; auto=%s", was_auto ? "true" : "false");
+    if (t < 0) return t;
+    written += t;
+    return written;
+}

ds4_gpu_args.h

@@ -0,0 +1,84 @@
+/* ds4_gpu_args.h — shared CLI argument parser for --gpu-vram and
+ * --gpu-devices flags. Used by ds4_cli, ds4_server, ds4_agent, and
+ * ds4_bench. See mgpu-cli-wiring task (wave 2) for design.
+ *
+ * The parser does not link CUDA headers directly. On CUDA builds the
+ * "auto" path is gated and dispatches to ds4_gpu_args_probe_auto_cuda,
+ * which is defined in ds4_cuda.cu. On Metal/CPU builds, "auto" returns
+ * an error.
+ */
+#ifndef DS4_GPU_ARGS_H
+#define DS4_GPU_ARGS_H
+
+#include <stdbool.h>
+#include <stddef.h>
+
+#include "ds4_gpu_mgpu.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Single canonical parser. Populates *out and *out_skip_cuda on
+ * success; returns 0 on success, nonzero on parse error (errbuf
+ * populated). Caller zeroes out and out_skip_cuda before calling.
+ *
+ *   vram_arg may be:
+ *     NULL       -- flag unset
+ *     "0"        -- skip CUDA entirely; sets *out_skip_cuda = true
+ *     "auto"     -- query free VRAM via CUDA (CUDA build only)
+ *     "N[,N...]" -- explicit GB budgets per device
+ *
+ *   devices_arg may be:
+ *     NULL       -- 0..n-1 of visible devices (or 0..vram_count-1)
+ *     "N[,N...]" -- explicit CUDA device indices
+ *
+ * Constraints:
+ *   - If both args are given and counts differ, error.
+ *   - Negative or non-integer values error.
+ *   - Counts > DS4_MAX_GPUS error.
+ */
+int parse_gpu_vram_arg(
+    const char     *vram_arg,
+    const char     *devices_arg,
+    ds4_gpu_config *out,
+    bool           *out_skip_cuda,
+    char           *errbuf,
+    size_t          errbuflen);
+
+/* Format the layout echo line written to stdout right before
+ * ds4_engine_create_with_gpu_config is called.
+ *
+ * Example output:
+ *   ds4: GPU config: 2 devices [0,1] requested, budgets 40,12 GB; auto=true
+ *
+ * Returns the number of bytes written (excluding the trailing NUL), or
+ * -1 if out is NULL/outlen is 0.
+ */
+int format_gpu_layout_line(const ds4_gpu_config *cfg,
+                           bool                  was_auto,
+                           char                 *out,
+                           size_t                outlen);
+
+/* CUDA-only auto probe. Defined in ds4_cuda.cu under the same gate.
+ * Returns 0 on success, nonzero on failure (errbuf populated).
+ * device_filter may be NULL (use all visible devices) or a list of
+ * length filter_len (use only those device indices).
+ *
+ * Declared here for the parser's use; not called by application code
+ * directly. The Mac/CPU builds skip this branch via the
+ * #if !defined(DS4_NO_GPU) && !defined(__APPLE__) gate in the parser. */
+#if !defined(DS4_NO_GPU) && !defined(__APPLE__)
+int ds4_gpu_args_probe_auto_cuda(const int      *device_filter,
+                                  int             filter_len,
+                                  ds4_gpu_config *out,
+                                  size_t          safety_margin_bytes,
+                                  char           *errbuf,
+                                  size_t          errbuflen);
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* DS4_GPU_ARGS_H */

ds4_gpu_mgpu.h

@@ -0,0 +1,237 @@
+/* ds4_gpu_mgpu.h — multi-GPU plumbing types and APIs (v0).
+ *
+ * This header carries the new multi-GPU additions for the multi-GPU plumbing PP work
+ * (device-aware CUDA). It is included from ds4_cuda.cu and from
+ * downstream tasks that need access to g_gpu[], g_n_gpus, g_gpu_peer_ok[],
+ * the ds4_gpu_config struct, and the new tensor APIs.
+ *
+ * Why not in ds4_gpu.h? The legacy ds4_gpu.h is included from C-only
+ * callers (ds4.c, ds4_cli.c, etc.) and from the Metal build, but is NOT
+ * included from ds4_cuda.cu historically. That asymmetry hid pre-existing
+ * signature mismatches between the legacy header and ds4_cuda.cu. We keep
+ * the legacy header opaque and put the new shared types here, so this
+ * file is the single source of truth for both ds4_cuda.cu and downstream
+ * multi-GPU tasks without disturbing the legacy contract.
+ *
+ * The struct definitions reference CUDA-specific handle types via void *
+ * placeholders so the header is safe to include from C builds, Metal
+ * builds, and the CUDA build (where ds4_cuda.cu casts the void * back
+ * to cudaStream_t / cublasHandle_t / cudaEvent_t internally).
+ */
+#ifndef DS4_GPU_MGPU_H
+#define DS4_GPU_MGPU_H
+
+#include <stddef.h>
+#include <stdint.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define DS4_MAX_GPUS 16
+
+/* Complete definition of the previously-opaque ds4_gpu_tensor, plus a
+ * typedef so the new API prototypes below can use the bare name
+ * `ds4_gpu_tensor *` in both C and C++ without forcing callers to
+ * include ds4_gpu.h first. Callers that include this header can
+ * stack-allocate or struct-embed tensors and pass to
+ * ds4_gpu_tensor_alloc_on. */
+struct ds4_gpu_tensor {
+    void    *ptr;
+    uint64_t bytes;
+    int      owner;
+    int      device_id;   /* -1 means legacy/untagged → treat as device 0 */
+};
+#ifndef DS4_GPU_TENSOR_DEFINED
+#define DS4_GPU_TENSOR_DEFINED
+typedef struct ds4_gpu_tensor ds4_gpu_tensor;
+#endif
+
+#ifndef DS4_GPU_ATTENTION_DECODE_ROW_DEFINED
+#define DS4_GPU_ATTENTION_DECODE_ROW_DEFINED
+#define DS4_GPU_ATTENTION_DECODE_BATCH_MAX 32u
+typedef struct {
+    uint64_t raw_kv;
+    uint64_t comp_kv;
+    uint64_t topk;
+    uint32_t pos;
+    uint32_t n_raw;
+    uint32_t raw_cap;
+    uint32_t raw_start;
+    uint32_t n_comp;
+    uint32_t top_k;
+    uint32_t window;
+    uint32_t ratio;
+    uint32_t indexed;
+} ds4_gpu_attention_decode_row;
+#endif
+
+/* Tagged so headers (notably ds4.h) can forward-declare `struct
+ * ds4_gpu_config` without dragging in this entire header. */
+typedef struct ds4_gpu_config {
+    int    device_indices[DS4_MAX_GPUS];   /* CUDA device IDs to use */
+    /* Explicit per-device budget in bytes. The engine does NOT auto-fill
+     * missing budgets - a value of 0 means "zero bytes of budget for
+     * this slot" and (combined with reserves) will push placement to
+     * CPU spill. Auto-detection (e.g. mapping --gpu-vram auto to
+     * cudaMemGetInfo) is the caller's job; see CLI flag wiring for the
+     * canonical CLI path. The engine emits a clear stderr and refuses
+     * if n_gpus > 0 and every vram_bytes[] is 0 (almost certainly a
+     * caller bug from zero-initializing the struct). */
+    size_t vram_bytes[DS4_MAX_GPUS];
+    int    n_gpus;
+    size_t safety_margin_bytes;            /* per-device reserve */
+} ds4_gpu_config;
+
+typedef struct {
+    int    device_id;
+    void  *stream;             /* cudaStream_t under CUDA */
+    void  *cublas;             /* cublasHandle_t under CUDA */
+    int    cublas_ready;
+    void  *scratch;
+    size_t scratch_bytes;
+    size_t budget_bytes;
+    size_t used_bytes;
+    void  *boundary_event;     /* cudaEvent_t under CUDA */
+} ds4_gpu_ctx;
+
+extern ds4_gpu_ctx g_gpu[DS4_MAX_GPUS];
+extern int         g_n_gpus;
+extern int         g_gpu_peer_ok[DS4_MAX_GPUS][DS4_MAX_GPUS];
+
+/* Primary multi-device init. The existing ds4_gpu_init (declared in
+ * ds4_gpu.h) is a thin shim that builds a single-device config for
+ * device 0 and calls this. */
+int ds4_gpu_init_multi(const ds4_gpu_config *cfg);
+
+/* Caller-supplied struct alloc on a specific device. Returns 0 on
+ * success, nonzero on error. Pair with ds4_gpu_tensor_free_in_place. */
+int  ds4_gpu_tensor_alloc_on(ds4_gpu_tensor *t, int device_id, uint64_t bytes);
+void ds4_gpu_tensor_free_in_place(ds4_gpu_tensor *t);
+
+/* Heap-allocated tensor on a specific logical tier; mirrors the legacy
+ * ds4_gpu_tensor_alloc ABI (returns ds4_gpu_tensor *) but with a tier
+ * parameter. Returns NULL on failure. Used by the multi-tier graph
+ * allocations in ds4.c. Single-tier callers can continue using the
+ * legacy ds4_gpu_tensor_alloc(bytes) which is equivalent to
+ * ds4_gpu_tensor_alloc_ptr_on(0, bytes). */
+ds4_gpu_tensor *ds4_gpu_tensor_alloc_ptr_on(int tier, uint64_t bytes);
+int ds4_gpu_tensor_copy_async(ds4_gpu_tensor *dst, const ds4_gpu_tensor *src, uint64_t bytes);
+void ds4_gpu_enable_q8_dequant_gemm(void);
+
+/* Heap-allocated MANAGED-memory tensor on a specific logical tier. Used
+ * for the KV-cache pool. Stamps tier on the tensor so subsequent
+ * ds4_gpu_tensor_free runs under the correct device. Note managed memory
+ * pages on first-touch and is not strictly device-bound; the stamped
+ * tier records the home tier for accounting + free. */
+ds4_gpu_tensor *ds4_gpu_tensor_alloc_managed_on(int tier, uint64_t bytes);
+
+/* Cross-device tensor copy. Same-device → cudaMemcpyAsync; peer-capable
+ * cross-device → cudaMemcpyPeerAsync with event sync; non-peer → pinned
+ * host bounce (per src→dst pair). Honors DS4_FORCE_HOST_BOUNCE=1. */
+int ds4_gpu_tensor_copy_xdev(ds4_gpu_tensor *dst,
+                              const ds4_gpu_tensor *src,
+                              uint64_t bytes);
+int ds4_gpu_tensor_copy_xdev_default(ds4_gpu_tensor *dst,
+                                      const ds4_gpu_tensor *src,
+                                      uint64_t bytes);
+
+/* Grouped default-stream handoff whose copies execute on the destination
+ * device. The source default stream records readiness; the destination waits,
+ * performs all three copies, then naturally orders its following kernels.
+ * This allows source-device work submitted afterward to overlap the handoff. */
+int ds4_gpu_tensor_copy_xdev3_default_dst(
+        ds4_gpu_tensor       *dst0,
+        const ds4_gpu_tensor *src0,
+        uint64_t              bytes0,
+        ds4_gpu_tensor       *dst1,
+        const ds4_gpu_tensor *src1,
+        uint64_t              bytes1,
+        ds4_gpu_tensor       *dst2,
+        const ds4_gpu_tensor *src2,
+        uint64_t              bytes2);
+
+/* Cross-device copy of three buffers between the same source/destination
+ * tiers, recording a single readiness event for the destination stream. This
+ * is meant for tiny grouped activation handoffs where three independent
+ * ds4_gpu_tensor_copy_xdev calls would spend more time in event plumbing than
+ * in the copies themselves. Falls back internally when a grouped peer copy is
+ * not applicable. */
+int ds4_gpu_tensor_copy_xdev3(ds4_gpu_tensor       *dst0,
+                              const ds4_gpu_tensor *src0,
+                              uint64_t              bytes0,
+                              ds4_gpu_tensor       *dst1,
+                              const ds4_gpu_tensor *src1,
+                              uint64_t              bytes1,
+                              ds4_gpu_tensor       *dst2,
+                              const ds4_gpu_tensor *src2,
+                              uint64_t              bytes2);
+
+/* Cross-device copy that also orders against prior work on the destination
+ * stream before overwriting dst. This is needed for pipelined prefill slots
+ * where the next producer reuses a destination buffer that the destination
+ * stage just consumed. */
+int ds4_gpu_tensor_copy_xdev_ordered(ds4_gpu_tensor *dst,
+                                      const ds4_gpu_tensor *src,
+                                      uint64_t bytes);
+
+/* Order a peer read without copying: records readiness on src's stream and
+ * makes dst_tier's stream wait for it. */
+int ds4_gpu_tensor_wait_xdev(const ds4_gpu_tensor *src, int dst_tier);
+int ds4_gpu_tensor_wait_xdev_default(const ds4_gpu_tensor *src, int dst_tier);
+
+/* Cross-device float add used by CUDA tensor-parallel reductions.
+ * See ds4_gpu.h for the full contract. */
+int ds4_gpu_add_xdev_tensor(ds4_gpu_tensor       *out,
+                             const ds4_gpu_tensor *local,
+                             const ds4_gpu_tensor *remote,
+                             ds4_gpu_tensor       *remote_tmp,
+                             uint32_t              n);
+
+/* Returns the device_id recorded on the tensor; -1 if untagged. */
+int ds4_gpu_tensor_device(const ds4_gpu_tensor *t);
+
+/* Set the current CUDA device by LOGICAL tier index (0..g_n_gpus-1).
+ *
+ * This is the canonical shim for per-layer device routing in the
+ * multi-tier execution path. The caller passes a logical tier index;
+ * the shim internally indexes g_gpu[tier].device_id and calls
+ * cudaSetDevice. Returns 0 on success, nonzero on error or if the
+ * tier index is out of range.
+ *
+ * Wave-2 multi-GPU placement scaffolding adds this shim but does not
+ * exercise the multi-tier execution path; multi-GPU execution
+ * (follow-up) is its first caller. */
+int ds4_gpu_set_current_device(int logical_tier);
+int ds4_gpu_set_current_device_fenced(int logical_tier);
+
+/* Register the mmap'd host model pointer for selective-cache lookups
+ * WITHOUT triggering any device-side copy. This bypasses the
+ * DS4_CUDA_COPY_MODEL environment-variable branch that
+ * ds4_gpu_set_model_map normally honors, which is essential for
+ * multi-tier startup: we want only per-device selective tensor caches,
+ * never the whole-model copy. Returns 1 on success, 0 on error. */
+int ds4_gpu_register_model_map_no_copy(const void *model_map, uint64_t model_size);
+
+/* Strict per-device selective-cache lookup (no fallback).
+ *
+ * Returns 1 only if a covering entry exists whose device_id matches the
+ * caller-supplied expected_device (a PHYSICAL CUDA device id). Otherwise
+ * returns 0 — no host-pointer fallback, no different-device match. The
+ * caller is expected to have cudaSetDevice'd to expected_device before
+ * invoking; the returned pointer is valid to consume from that device's
+ * kernel.
+ *
+ * Used by multi-tier kernel-dispatch resolvers in
+ * multi-GPU execution (multi-GPU execution). Single-tier callers should
+ * keep using ds4_gpu_lookup_cache for back-compat behavior. */
+int ds4_gpu_lookup_cache_strict(uint64_t source_offset,
+                                 uint64_t bytes,
+                                 int      expected_device,
+                                 void   **out_device_ptr);
+
+#ifdef __cplusplus
+} /* extern "C" */
+#endif
+
+#endif /* DS4_GPU_MGPU_H */

ds4_help.c

@@ -0,0 +1,590 @@
+#include "ds4_help.h"
+
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+typedef struct {
+    const char *off;
+    const char *cyan;
+    const char *title;
+    const char *yellow;
+    const char *grey;
+    const char *red;
+    const char *white;
+    const char *bright;
+} help_colors;
+
+static help_colors help_make_colors(FILE *fp) {
+    bool color = isatty(fileno(fp));
+    help_colors c = {0};
+    if (!color) return c;
+    c.off = "\x1b[0m";
+    c.cyan = "\x1b[38;5;81m";
+    c.title = "\x1b[1;38;5;250m";
+    c.yellow = "\x1b[38;5;179m";
+    c.grey = "\x1b[38;5;240m";
+    c.red = "\x1b[38;5;203m";
+    c.white = "\x1b[38;5;252m";
+    c.bright = "\x1b[1;38;5;231m";
+    return c;
+}
+
+static void title(FILE *fp, const help_colors *c, const char *s) {
+    fprintf(fp, "%s%s%s\n", c->title ? c->title : "", s, c->off ? c->off : "");
+}
+
+static void title_red(FILE *fp, const help_colors *c, const char *s) {
+    fprintf(fp, "%s%s%s\n", c->red ? c->red : "", s, c->off ? c->off : "");
+}
+
+static bool option_name_has_switch(const char *name) {
+    bool word_start = true;
+    while (*name) {
+        if (word_start && (*name == '-' || *name == '/')) return true;
+        word_start = (*name == ' ');
+        name++;
+    }
+    return false;
+}
+
+static void print_colored_option_name(FILE *fp, const help_colors *c, const char *name) {
+    bool has_switch = option_name_has_switch(name);
+    bool word_start = true;
+    while (*name) {
+        const char *start = name;
+        while (*name && *name != ' ') name++;
+        bool is_option = !has_switch || *start == '-' || *start == '/' ||
+                         (word_start && has_switch && *start != '[');
+        const char *color = is_option ? c->cyan : c->bright;
+        if (color) fputs(color, fp);
+        fwrite(start, 1, (size_t)(name - start), fp);
+        if (color && c->off) fputs(c->off, fp);
+        if (*name == ' ') {
+            fputc(*name++, fp);
+            word_start = false;
+        }
+    }
+}
+
+static void opt(FILE *fp, const help_colors *c, const char *name, const char *desc) {
+    if (c->cyan) {
+        fputs("  ", fp);
+        print_colored_option_name(fp, c, name);
+        fprintf(fp, " %s|%s ", c->grey ? c->grey : "", c->grey ? c->off : "");
+        fprintf(fp, "%s%s%s\n", c->white ? c->white : "", desc,
+                c->white ? c->off : "");
+        return;
+    }
+
+    const int col = 30;
+    int n = (int)strlen(name);
+    if (n > col) {
+        fprintf(fp, "  %s\n      %s\n", name, desc);
+    } else {
+        fprintf(fp, "  %-30s %s\n", name, desc);
+    }
+}
+
+static void para(FILE *fp, const help_colors *c, const char *s) {
+    fprintf(fp, "%s%s%s\n",
+            c->yellow ? c->yellow : "", s, c->yellow ? c->off : "");
+}
+
+static bool streq(const char *a, const char *b) {
+    return a && b && strcmp(a, b) == 0;
+}
+
+static bool topic_is(const char *topic, const char *name) {
+    return topic && strcmp(topic, name) == 0;
+}
+
+static const char *tool_name(ds4_help_tool tool) {
+    switch (tool) {
+    case DS4_HELP_DS4: return "ds4";
+    case DS4_HELP_SERVER: return "ds4-server";
+    case DS4_HELP_AGENT: return "ds4-agent";
+    case DS4_HELP_BENCH: return "ds4-bench";
+    case DS4_HELP_EVAL: return "ds4-eval";
+    }
+    return "ds4";
+}
+
+static const char *tool_usage(ds4_help_tool tool) {
+    switch (tool) {
+    case DS4_HELP_DS4:
+        return "Usage: ds4 [(-p PROMPT | --prompt-file FILE)] [options]";
+    case DS4_HELP_SERVER:
+        return "Usage: ds4-server [options]";
+    case DS4_HELP_AGENT:
+        return "Usage: ds4-agent [options]";
+    case DS4_HELP_BENCH:
+        return "Usage: ds4-bench (--prompt-file FILE | --chat-prompt-file FILE) [options]";
+    case DS4_HELP_EVAL:
+        return "Usage: ds4-eval [options]";
+    }
+    return "Usage: ds4 [options]";
+}
+
+static const char *tool_summary(ds4_help_tool tool) {
+    switch (tool) {
+    case DS4_HELP_DS4:
+        return "Chat with a local DwarfStar model, run one-shot prompts, inspect models, or coordinate distributed inference.";
+    case DS4_HELP_SERVER:
+        return "Serve one loaded DwarfStar model through OpenAI, Responses, Anthropic, and completion-compatible HTTP APIs.";
+    case DS4_HELP_AGENT:
+        return "Run the native terminal coding agent with live tools, session save/restore, and a responsive prompt while the model works.";
+    case DS4_HELP_BENCH:
+        return "Measure prefill, decode, context growth, and KV-cache size across repeatable context frontiers.";
+    case DS4_HELP_EVAL:
+        return "Run the built-in reasoning, math, science, and security evaluation harness with a live terminal UI.";
+    }
+    return "";
+}
+
+static void print_model_runtime(FILE *fp, const help_colors *c,
+                                ds4_help_tool tool, bool full) {
+    title(fp, c, "Model And Runtime");
+    opt(fp, c, "-m, --model FILE", "GGUF model path. Default: ds4flash.gguf");
+#ifdef DS4_ROCM_BUILD
+    opt(fp, c, "--metal | --rocm | --cpu", "Select the backend explicitly.");
+    opt(fp, c, "--backend NAME", "Backend name: metal, rocm, or cpu.");
+#else
+    opt(fp, c, "--metal | --cuda | --cpu", "Select the backend explicitly.");
+    opt(fp, c, "--backend NAME", "Backend name: metal, cuda, or cpu.");
+    opt(fp, c, "--gpu-vram N[,N,...]|auto", "CUDA VRAM budgets per device, in GiB, or auto-detect free VRAM.");
+    opt(fp, c, "--gpu-devices N[,N,...]", "CUDA device indices used by multi-GPU placement.");
+    if (tool != DS4_HELP_EVAL) {
+        opt(fp, c, "--cuda-tensor-parallel", "Enable the paired DeepSeek tensor/expert path on an even multi-GPU CUDA placement.");
+    }
+#endif
+    if (tool != DS4_HELP_BENCH) {
+        opt(fp, c, "-c, --ctx N", "Allocated context tokens.");
+    }
+    if (tool == DS4_HELP_SERVER) {
+        opt(fp, c, "-n, --tokens N", "Default max output tokens when clients omit a limit.");
+    }
+    opt(fp, c, "-t, --threads N", "CPU helper threads for host-side/reference work.");
+    opt(fp, c, "--power N", "GPU duty-cycle target, 1..100. Default: 100");
+    opt(fp, c, "--ssd-streaming", "Metal/CUDA/ROCm: opt in to SSD-backed model streaming instead of full residency.");
+    opt(fp, c, "--ssd-streaming-cold", "SSD streaming: skip default popularity-based expert-cache preload.");
+    opt(fp, c, "--ssd-streaming-cache-experts N|NGB", "SSD streaming: N is an exact dynamic expert count; NGB is a routed memory budget that also reserves two full prefill layers. Auto: 80% working set minus non-routed weights; GLM Metal caps lower.");
+    opt(fp, c, "--ssd-streaming-full-layers N", "GLM Metal streaming: keep the first N routed layers fully resident. Default: auto from NGB expert budget; use 0 to disable.");
+    opt(fp, c, "--ssd-streaming-preload-experts N", "SSD streaming: upfront popularity preload count. DeepSeek auto-seeds by default; GLM demand-fills unless N is explicit.");
+    opt(fp, c, "--simulate-used-memory NGB", "Diagnostic: lock N GiB before model load to simulate a smaller-memory machine.");
+    opt(fp, c, "--prefill-chunk N", "Graph prefill chunk size. Default: CUDA TP 2048; PRO 8192; Laguna 16384; others 4096.");
+    if (full) {
+        if (tool != DS4_HELP_BENCH) {
+            opt(fp, c, "--mtp FILE", "Optional MTP support GGUF used for draft-token probes.");
+        }
+        if (tool == DS4_HELP_DS4 || tool == DS4_HELP_AGENT || tool == DS4_HELP_SERVER) {
+            opt(fp, c, "--dflash FILE", "Laguna DFlash support GGUF for greedy speculative decoding.");
+            opt(fp, c, "--dflash-draft N", "Maximum adaptive DFlash draft positions, 1..15. CUDA default: 15; other backends: 3");
+            opt(fp, c, "--dflash-p-min P", "Stop before proposals below probability P, 0..1. Default: 0.4; 0 keeps fixed verifier width");
+            opt(fp, c, "--mtp-draft N", "Maximum autoregressive MTP draft tokens. Default: 1");
+            opt(fp, c, "--mtp-margin F", "Verifier confidence margin for fast MTP acceptance. Default: 3");
+            opt(fp, c, "--glm-mtp", "Enable integrated greedy GLM MTP speculation.");
+            opt(fp, c, "--glm-mtp-timing", "Enable GLM MTP and print acceptance/timing counters.");
+            opt(fp, c, "--dspark", "Enable DSpark using the support GGUF passed with --mtp.");
+            opt(fp, c, "--dspark-confidence F", "Enable DSpark with confidence pruning threshold 0..1. Default: 0.9");
+            opt(fp, c, "--dspark-strict", "Load DSpark support but keep target-only decode.");
+        }
+        opt(fp, c, "--quality", "Prefer exact kernels where faster approximate paths exist.");
+        opt(fp, c, "--warm-weights", "Touch mapped tensor pages at startup to reduce first-use stalls.");
+        if (tool == DS4_HELP_DS4 || tool == DS4_HELP_BENCH) {
+            opt(fp, c, "--expert-profile FILE", "Metal-only: write routed expert locality/cache simulation JSON.");
+        }
+    }
+    fputc('\n', fp);
+}
+
+static void print_sampling(FILE *fp, const help_colors *c, bool full) {
+    title(fp, c, "Prompt And Sampling");
+    opt(fp, c, "-n, --tokens N", "Maximum generated tokens.");
+    opt(fp, c, "--temp F", "Sampling temperature. 0 is greedy/deterministic.");
+    opt(fp, c, "--top-k N", "Sample only from the N highest-scoring tokens. 0 disables.");
+    opt(fp, c, "--top-p F", "Nucleus sampling probability.");
+    opt(fp, c, "--min-p F", "Keep tokens scoring at least F times the top token.");
+    opt(fp, c, "--seed N", "Sampling seed for reproducible non-greedy runs.");
+    para(fp, c, "GLM defaults to temperature 1.0, top-p 0.95, and min-p 0. Laguna defaults to temperature 0.7, top-k 20, top-p 0.95, and min-p 0.05. Explicit options always win.");
+    opt(fp, c, "--think", "Use normal thinking mode.");
+    opt(fp, c, "--think-max", "Use Think Max when context is large enough.");
+    opt(fp, c, "--nothink", "Disable thinking and ask for direct replies.");
+    if (full) {
+        opt(fp, c, "-sys, --system TEXT", "System prompt. Empty string disables the default where supported.");
+        opt(fp, c, "-p, --prompt TEXT", "One-shot prompt text.");
+        opt(fp, c, "--prompt-file FILE", "Read one-shot prompt text from FILE.");
+        opt(fp, c, "--raw-prompt", "Tokenize the one-shot prompt without chat markers.");
+    }
+    fputc('\n', fp);
+}
+
+static void print_steering(FILE *fp, const help_colors *c) {
+    title(fp, c, "Directional Steering");
+    opt(fp, c, "--dir-steering-file FILE", "Load one f32 direction vector per layer.");
+    opt(fp, c, "--dir-steering-ffn F", "Apply steering after FFN outputs. Default with file: 1");
+    opt(fp, c, "--dir-steering-attn F", "Apply steering after attention outputs. Default: 0");
+    fputc('\n', fp);
+}
+
+static void print_distributed(FILE *fp, const help_colors *c) {
+    title(fp, c, "Distributed Inference");
+    fputc('\n', fp);
+    para(fp, c, "Distributed mode runs one logical session across several machines by assigning contiguous model layer ranges to workers. Workers own their layer slice and KV-cache shard; the coordinator owns the prompt, sampling loop, and client/API flow. Start workers first, then start the coordinator. The coordinator waits for a complete route and streams hidden states through the workers.");
+    fputc('\n', fp);
+    opt(fp, c, "--role ROLE", "Distributed role: coordinator or worker.");
+    opt(fp, c, "--layers A:B", "Inclusive layer slice, e.g. 0:20 or 21:output.");
+    opt(fp, c, "--listen HOST PORT", "Coordinator listen address; workers may use it for their data listener.");
+    opt(fp, c, "--coordinator HOST PORT", "Coordinator address for --role worker.");
+    opt(fp, c, "--dist-prefill-chunk N", "Coordinator prefill pipeline chunk size. Default: session cap.");
+    opt(fp, c, "--dist-prefill-window N", "Max prefill chunks in flight. Default: workers+2, capped at 8.");
+    opt(fp, c, "--dist-activation-bits N", "Hidden-state transport width: 32, 16, or 8. Default: 32");
+    opt(fp, c, "--dist-replay-check", "Diagnostic: reset and replay prompt, then compare logits.");
+    opt(fp, c, "--debug", "Print coordinator route/debug logs.");
+    fputc('\n', fp);
+    title(fp, c, "Tensor Parallelism");
+    fputc('\n', fp);
+    para(fp, c, "Tensor parallelism uses the same coordinator/worker addresses as distributed mode, but always runs one 50/50 worker. Add --tensor-parallel, omit --layers, start the worker, then start the coordinator.");
+    fputc('\n', fp);
+    opt(fp, c, "--tensor-parallel", "Switch --role/--listen/--coordinator to two-machine tensor parallelism.");
+    opt(fp, c, "--transport auto|rdma|tcp", "Tensor gate transport. Default: auto");
+    opt(fp, c, "--rdma-device NAME", "Select a verbs device when auto-detection is ambiguous.");
+    opt(fp, c, "--rdma-gid-index N", "Select the local verbs GID index.");
+    opt(fp, c, "--tensor-parallel-token-prefill", "GLM diagnostic: prefill one token at a time for exact arithmetic.");
+    opt(fp, c, "--debug-hash N", "Cross-check hidden state every N tokens.");
+    fputc('\n', fp);
+}
+
+static void print_cli_diagnostics(FILE *fp, const help_colors *c);
+
+static void print_cli_specific(FILE *fp, const help_colors *c, bool full) {
+    title(fp, c, "CLI Modes");
+    opt(fp, c, "ds4", "Start the interactive prompt.");
+    opt(fp, c, "ds4 -p TEXT", "Run one prompt and exit.");
+    opt(fp, c, "ds4 --prompt-file FILE", "Run a long prompt from a file and exit.");
+    fputc('\n', fp);
+    if (full) {
+        print_cli_diagnostics(fp, c);
+    }
+}
+
+static void print_cli_diagnostics(FILE *fp, const help_colors *c) {
+    title(fp, c, "Diagnostics And Data Collection");
+    opt(fp, c, "--inspect", "Load the model and print a summary only.");
+    opt(fp, c, "--dump-tokens", "Tokenize the prompt exactly as written, then exit.");
+    opt(fp, c, "--dump-logits FILE", "Write full next-token logits as JSON.");
+    opt(fp, c, "--dump-logprobs FILE", "Write greedy continuation top-logprobs as JSON.");
+    opt(fp, c, "--logprobs-top-k N", "Alternatives stored by --dump-logprobs. Default: 20");
+    opt(fp, c, "--decode-consistency N", "Compare N-token decode logits with a fresh full prefill.");
+    opt(fp, c, "--expert-profile FILE", "Metal-only: write routed expert locality/cache simulation JSON.");
+    opt(fp, c, "--perplexity-file FILE", "Score raw text with teacher-forced NLL.");
+    opt(fp, c, "--imatrix-dataset FILE", "Rendered prompt dataset for imatrix collection.");
+    opt(fp, c, "--imatrix-out FILE", "Write llama-compatible routed-MoE imatrix .dat.");
+    opt(fp, c, "--imatrix-max-prompts N", "Stop imatrix collection after N prompts.");
+    opt(fp, c, "--imatrix-max-tokens N", "Stop imatrix collection after N prompt tokens.");
+    opt(fp, c, "--head-test", "Run the output HC/logits head after the native slice.");
+    opt(fp, c, "--first-token-test", "Run exact CPU whole-model pass for the first prompt token.");
+    opt(fp, c, "--metal-graph-test", "Compare first GPU-resident graph stages with CPU.");
+    opt(fp, c, "--metal-graph-full-test", "Run the GPU-resident self-token graph across all layers.");
+    opt(fp, c, "--metal-graph-prompt-test", "Compare CPU and GPU graph logits for the full prompt.");
+    fputc('\n', fp);
+}
+
+static void print_cli_commands(FILE *fp, const help_colors *c) {
+    title_red(fp, c, "Interactive Commands");
+    opt(fp, c, "/help", "Show interactive commands.");
+    opt(fp, c, "/think, /think-max, /nothink", "Switch thinking mode.");
+    opt(fp, c, "/ctx N", "Restart the interactive session with a new context size.");
+    opt(fp, c, "/power N", "Set GPU duty cycle percentage, 1..100.");
+    opt(fp, c, "/read FILE", "Read FILE and submit it as the next user message.");
+    opt(fp, c, "/quit, /exit", "Leave the prompt.");
+    opt(fp, c, "Ctrl+C", "Stop current generation and return to ds4>.");
+    fputc('\n', fp);
+}
+
+static void print_agent_specific(FILE *fp, const help_colors *c) {
+    title(fp, c, "Agent Options");
+    opt(fp, c, "-p, --prompt TEXT", "Submit an initial prompt after startup.");
+    opt(fp, c, "--non-interactive", "Run without TUI. With -p: one turn; without -p: repeated stdin prompts.");
+    opt(fp, c, "--raw-prompt", "Non-interactive -p only: tokenize prompt without agent chat/tool text.");
+    opt(fp, c, "-sys, --system TEXT", "Extra system prompt. Empty disables extra text.");
+    opt(fp, c, "--trace FILE", "Write prompt, token, and DSML debug trace.");
+    opt(fp, c, "--chdir DIR", "Change working directory before loading runtime assets.");
+    fputc('\n', fp);
+}
+
+static void print_agent_sessions(FILE *fp, const help_colors *c) {
+    title(fp, c, "Agent Runtime Commands");
+    opt(fp, c, "/save", "Save the current session in ~/.ds4/kvcache.");
+    opt(fp, c, "/compact", "Compact the current session context now.");
+    opt(fp, c, "/list", "List saved sessions, sorted by recent update time.");
+    opt(fp, c, "/switch ID", "Load a saved session and show recent history.");
+    opt(fp, c, "/del ID", "Delete a saved session.");
+    opt(fp, c, "/strip ID", "Remove KV payload; the text history can be rebuilt later.");
+    opt(fp, c, "/history [N]", "Show N recent user turns from the current session.");
+    opt(fp, c, "/power N", "Set GPU duty cycle percentage, 1..100.");
+    opt(fp, c, "/new", "Start a fresh session from the system prompt.");
+    opt(fp, c, "/quit, /exit", "Exit.");
+    fputc('\n', fp);
+}
+
+static void print_server_api(FILE *fp, const help_colors *c) {
+    title(fp, c, "HTTP API");
+    opt(fp, c, "--host HOST", "Bind address. Default: 127.0.0.1");
+    opt(fp, c, "--port N", "Bind port. Default: 8000");
+    opt(fp, c, "--cors", "Add Access-Control-Allow-* headers for browser JS clients.");
+    opt(fp, c, "--trace FILE", "Write prompts, cache decisions, output, and tool calls.");
+    opt(fp, c, "--batched-session N", "Keep N resident sessions and batch decode-ready requests.");
+    para(fp, c, "Endpoints: /v1/chat/completions, /v1/responses, /v1/completions, and /v1/messages.");
+    para(fp, c, "Model aliases are available for DeepSeek V4, GLM-5.2, and Laguna-S-2.1; every alias serves the loaded GGUF.");
+    fputc('\n', fp);
+}
+
+static void print_server_thinking(FILE *fp, const help_colors *c) {
+    title(fp, c, "Server Thinking Defaults");
+    para(fp, c, "DeepSeek-compatible chat requests default to high-effort thinking.");
+    para(fp, c, "reasoning_effort=max or output_config.effort=max requests Think Max.");
+    para(fp, c, "Think Max requires --ctx >= 393216; smaller contexts use high.");
+    para(fp, c, "thinking={type:disabled}, think=false, or model=deepseek-chat selects non-thinking mode.");
+    para(fp, c, "In thinking mode, client sampling knobs are ignored like the official API.");
+    fputc('\n', fp);
+}
+
+static void print_kv_cache(FILE *fp, const help_colors *c) {
+    title(fp, c, "Disk KV Cache");
+    opt(fp, c, "--kv-disk-dir DIR", "Enable disk KV checkpoints in DIR.");
+    opt(fp, c, "--kv-disk-space-mb N", "Disk budget. Default when enabled: 4096");
+    opt(fp, c, "--kv-cache-min-tokens N", "Do not save/load checkpoints shorter than N. Default: 512");
+    opt(fp, c, "--kv-cache-cold-max-tokens N", "Save cold first prompts up to N tokens. 0 disables. Default: 30000");
+    opt(fp, c, "--kv-cache-continued-interval-tokens N", "Save aligned continued frontiers. 0 disables. Default: 10000");
+    opt(fp, c, "--kv-cache-boundary-trim-tokens N", "Trim tail tokens for cold boundary saves. Default: 32");
+    opt(fp, c, "--kv-cache-boundary-align-tokens N", "Align cold boundary saves to this multiple. Default: 2048");
+    opt(fp, c, "--kv-cache-reject-different-quant", "Reject checkpoints written with different routed-expert quantization.");
+    opt(fp, c, "--disable-exact-dsml-tool-replay", "Disable exact sampled DSML tool replay map.");
+    opt(fp, c, "--tool-memory-max-ids N", "Exact tool-call IDs kept in RAM. Default: 100000");
+    fputc('\n', fp);
+}
+
+static void print_bench_specific(FILE *fp, const help_colors *c) {
+    title(fp, c, "Benchmark Input");
+    opt(fp, c, "--prompt-file FILE", "Raw benchmark text; token sequence is sliced at each frontier.");
+    opt(fp, c, "--chat-prompt-file FILE", "Render FILE as one no-thinking chat user message.");
+    opt(fp, c, "-sys, --system TEXT", "System prompt used only with --chat-prompt-file.");
+    fputc('\n', fp);
+    title(fp, c, "Benchmark Sweep");
+    opt(fp, c, "--ctx-start N", "First measured frontier. Default: 2048");
+    opt(fp, c, "--ctx-max N", "Last measured frontier. Default: 32768");
+    opt(fp, c, "--ctx-alloc N", "Allocated context. Default: ctx-max + gen-tokens + 1");
+    opt(fp, c, "--step-mul F", "Multiplicative step. Default: 1");
+    opt(fp, c, "--step-incr N", "Linear step when --step-mul is 1. Default: 2048");
+    opt(fp, c, "--gen-tokens N", "Greedy decode tokens per frontier. 0 for pure prefill. Default: 128");
+    opt(fp, c, "--csv FILE", "Write CSV there instead of stdout.");
+    opt(fp, c, "--dump-frontier-logits-dir DIR", "Write one full-logit JSON file per frontier.");
+    fputc('\n', fp);
+}
+
+static void print_eval_specific(FILE *fp, const help_colors *c) {
+    title(fp, c, "Evaluation");
+    opt(fp, c, "-n, --tokens N", "Max generated tokens per question. Default: 16000");
+    opt(fp, c, "--questions N", "Run only the first N embedded questions.");
+    opt(fp, c, "--case-sequence LIST", "Run 1-based case numbers in this comma-separated order.");
+    opt(fp, c, "--trace FILE", "Write questions, outputs, and grading decisions.");
+    opt(fp, c, "--regrade-trace FILE", "Regrade a prior trace without loading the model.");
+    opt(fp, c, "--soft-limit-reply-budget N", "Soft close thinking near the end of reply budget. Default: 1024");
+    opt(fp, c, "--hard-limit-reply-budget N", "Force </think> with N tokens left. Default: 512");
+    opt(fp, c, "--soft-limit-think-close-rank N", "Soft-close when </think> is in top N tokens. Default: 3");
+    opt(fp, c, "--pause-ms N", "Pause after each result in the TTY UI. Default: 350");
+    opt(fp, c, "--plain", "Disable split-screen ANSI UI.");
+    opt(fp, c, "--self-test-extractors", "Run answer-extractor self-tests and exit.");
+    fputc('\n', fp);
+}
+
+static bool tool_has_topic(ds4_help_tool tool, const char *topic) {
+    if (!topic) return true;
+    if (streq(topic, "all")) return true;
+    if (streq(topic, "runtime") || streq(topic, "distributed")) return true;
+    if (streq(topic, "sampling"))
+        return tool == DS4_HELP_DS4 || tool == DS4_HELP_AGENT || tool == DS4_HELP_EVAL;
+    if (streq(topic, "steering"))
+        return tool == DS4_HELP_DS4 || tool == DS4_HELP_SERVER || tool == DS4_HELP_AGENT;
+    switch (tool) {
+    case DS4_HELP_DS4:
+        return streq(topic, "diagnostics") || streq(topic, "commands");
+    case DS4_HELP_SERVER:
+        return streq(topic, "api") || streq(topic, "kv-cache") || streq(topic, "thinking");
+    case DS4_HELP_AGENT:
+        return streq(topic, "sessions") || streq(topic, "commands") || streq(topic, "tools");
+    case DS4_HELP_BENCH:
+        return streq(topic, "benchmark");
+    case DS4_HELP_EVAL:
+        return streq(topic, "evaluation");
+    }
+    return false;
+}
+
+static void more_line(FILE *fp, const help_colors *c, const char *label, const char *topic) {
+    static const char *colors[] = {
+        "\x1b[38;5;81m", "\x1b[38;5;114m", "\x1b[38;5;179m",
+        "\x1b[38;5;141m", "\x1b[38;5;147m"
+    };
+    static size_t idx;
+    const char *on = c->cyan ? colors[idx++ % (sizeof(colors) / sizeof(colors[0]))] : "";
+    if (streq(label, "Interactive commands:") && c->red) on = c->red;
+    const char *off = c->off ? c->off : "";
+    fprintf(fp, "    %s%-26s%s --help %s\n", on, label, off, topic);
+}
+
+static void print_more_info(FILE *fp, const help_colors *c, ds4_help_tool tool) {
+    title(fp, c, "More Info");
+    more_line(fp, c, "Runtime full info:", "runtime");
+    if (tool_has_topic(tool, "sampling"))
+        more_line(fp, c, "Sampling full info:", "sampling");
+    more_line(fp, c, "Distributed inference:", "distributed");
+    if (tool_has_topic(tool, "steering"))
+        more_line(fp, c, "Steering full info:", "steering");
+    if (tool == DS4_HELP_DS4) {
+        more_line(fp, c, "Interactive commands:", "commands");
+        more_line(fp, c, "Diagnostics:", "diagnostics");
+    } else if (tool == DS4_HELP_SERVER) {
+        more_line(fp, c, "HTTP API:", "api");
+        more_line(fp, c, "Disk KV cache:", "kv-cache");
+        more_line(fp, c, "Thinking behavior:", "thinking");
+    } else if (tool == DS4_HELP_AGENT) {
+        more_line(fp, c, "Agent sessions:", "sessions");
+        more_line(fp, c, "Agent commands:", "commands");
+        more_line(fp, c, "Agent tool system:", "tools");
+    } else if (tool == DS4_HELP_BENCH) {
+        more_line(fp, c, "Benchmark sweep:", "benchmark");
+    } else if (tool == DS4_HELP_EVAL) {
+        more_line(fp, c, "Evaluation options:", "evaluation");
+    }
+    fputc('\n', fp);
+}
+
+static void print_examples(FILE *fp, const help_colors *c, ds4_help_tool tool, const char *topic) {
+    title(fp, c, "Examples");
+    if (topic_is(topic, "distributed")) {
+        opt(fp, c, "worker", "./ds4 --role worker --layers 21:output --coordinator 192.168.0.181 9000 -m ds4flash.gguf");
+        opt(fp, c, "coordinator", "./ds4 --role coordinator --layers 0:20 --listen 0.0.0.0 9000 -p \"Hello\" -m ds4flash.gguf");
+    } else if (topic_is(topic, "runtime")) {
+        if (tool == DS4_HELP_SERVER) {
+            opt(fp, c, "Metal API", "./ds4-server -m ds4flash.gguf --metal --ctx 100000");
+            opt(fp, c, "quiet API", "./ds4-server --power 60 --host 127.0.0.1 --port 8000");
+        } else if (tool == DS4_HELP_AGENT) {
+            opt(fp, c, "agent", "./ds4-agent -m ds4flash.gguf --ctx 100000");
+            opt(fp, c, "quiet agent", "./ds4-agent --power 50");
+        } else if (tool == DS4_HELP_BENCH) {
+            opt(fp, c, "bench", "./ds4-bench --prompt-file long.txt --ctx-max 32768");
+            opt(fp, c, "quiet bench", "./ds4-bench --prompt-file long.txt --power 70");
+        } else if (tool == DS4_HELP_EVAL) {
+            opt(fp, c, "eval", "./ds4-eval --questions 10 --ctx 100000");
+            opt(fp, c, "CPU debug", "./ds4-eval --cpu --questions 1 --tokens 32");
+        } else {
+            opt(fp, c, "Metal", "./ds4 -m ds4flash.gguf --metal -c 100000");
+            opt(fp, c, "quiet thermals", "./ds4 -p \"Summarize README\" --power 50");
+        }
+    } else if (topic_is(topic, "steering")) {
+        opt(fp, c, "steer FFN", "./ds4 -p \"Write tersely\" --dir-steering-file dir.bin --dir-steering-ffn 0.8");
+    } else if (tool == DS4_HELP_SERVER || topic_is(topic, "api") || topic_is(topic, "kv-cache")) {
+        opt(fp, c, "local API", "./ds4-server --ctx 100000 --kv-disk-dir ~/.ds4/server-kv --kv-disk-space-mb 8192");
+        opt(fp, c, "curl", "curl http://127.0.0.1:8000/v1/models");
+    } else if (tool == DS4_HELP_AGENT || topic_is(topic, "sessions") || topic_is(topic, "tools")) {
+        opt(fp, c, "interactive", "./ds4-agent");
+        opt(fp, c, "one shot", "./ds4-agent --non-interactive -p \"Create /tmp/hello.c\"");
+    } else if (tool == DS4_HELP_BENCH || topic_is(topic, "benchmark")) {
+        opt(fp, c, "csv", "./ds4-bench --prompt-file long.txt --ctx-max 32768 --csv speed.csv");
+        opt(fp, c, "prefill only", "./ds4-bench --prompt-file long.txt --gen-tokens 0");
+    } else if (tool == DS4_HELP_EVAL || topic_is(topic, "evaluation")) {
+        opt(fp, c, "first 10", "./ds4-eval --questions 10 --trace eval.trace");
+        opt(fp, c, "plain", "./ds4-eval --plain --nothink --tokens 512");
+    } else {
+        opt(fp, c, "chat", "./ds4");
+        opt(fp, c, "one shot", "./ds4 -p \"Explain mmap in C\"");
+        opt(fp, c, "long prompt", "./ds4 --think-max --prompt-file prompt.txt --ctx 393216");
+    }
+    fputc('\n', fp);
+}
+
+static void print_topic(FILE *fp, const help_colors *c, ds4_help_tool tool, const char *topic) {
+    if (streq(topic, "all")) {
+        print_model_runtime(fp, c, tool, true);
+        if (tool_has_topic(tool, "sampling")) print_sampling(fp, c, true);
+        if (tool_has_topic(tool, "steering")) print_steering(fp, c);
+        print_distributed(fp, c);
+        if (tool == DS4_HELP_DS4) {
+            print_cli_specific(fp, c, true);
+            print_cli_commands(fp, c);
+        } else if (tool == DS4_HELP_SERVER) {
+            print_server_api(fp, c);
+            print_server_thinking(fp, c);
+            print_kv_cache(fp, c);
+        } else if (tool == DS4_HELP_AGENT) {
+            print_agent_specific(fp, c);
+            print_agent_sessions(fp, c);
+        } else if (tool == DS4_HELP_BENCH) {
+            print_bench_specific(fp, c);
+        } else if (tool == DS4_HELP_EVAL) {
+            print_eval_specific(fp, c);
+        }
+        return;
+    }
+
+    if (streq(topic, "runtime")) print_model_runtime(fp, c, tool, true);
+    else if (streq(topic, "sampling")) print_sampling(fp, c, true);
+    else if (streq(topic, "steering")) print_steering(fp, c);
+    else if (streq(topic, "distributed")) print_distributed(fp, c);
+    else if (tool == DS4_HELP_DS4 && streq(topic, "diagnostics")) print_cli_diagnostics(fp, c);
+    else if (tool == DS4_HELP_DS4 && streq(topic, "commands")) print_cli_commands(fp, c);
+    else if (tool == DS4_HELP_SERVER && streq(topic, "api")) print_server_api(fp, c);
+    else if (tool == DS4_HELP_SERVER && streq(topic, "kv-cache")) print_kv_cache(fp, c);
+    else if (tool == DS4_HELP_SERVER && streq(topic, "thinking")) print_server_thinking(fp, c);
+    else if (tool == DS4_HELP_AGENT && streq(topic, "sessions")) print_agent_sessions(fp, c);
+    else if (tool == DS4_HELP_AGENT && streq(topic, "commands")) print_agent_sessions(fp, c);
+    else if (tool == DS4_HELP_AGENT && streq(topic, "tools")) {
+        title(fp, c, "Agent Tool System");
+        para(fp, c, "The agent can read, search, write, edit, run bash, and browse through Chrome-backed web tools.");
+        para(fp, c, "DeepSeek-family models emit DSML tool calls; GLM and Laguna models use native <tool_call> syntax. Both are rendered live in the terminal.");
+        para(fp, c, "Edit uses exact old/new replacement; [upto] can bridge a unique head and tail for large anchored edits.");
+        fputc('\n', fp);
+    } else if (tool == DS4_HELP_BENCH && streq(topic, "benchmark")) print_bench_specific(fp, c);
+    else if (tool == DS4_HELP_EVAL && streq(topic, "evaluation")) print_eval_specific(fp, c);
+}
+
+static void print_default(FILE *fp, const help_colors *c, ds4_help_tool tool) {
+    print_model_runtime(fp, c, tool, false);
+
+    if (tool == DS4_HELP_DS4) {
+        print_cli_specific(fp, c, true);
+        print_sampling(fp, c, false);
+    } else if (tool == DS4_HELP_SERVER) {
+        print_server_api(fp, c);
+        print_kv_cache(fp, c);
+    } else if (tool == DS4_HELP_AGENT) {
+        print_agent_specific(fp, c);
+        print_agent_sessions(fp, c);
+    } else if (tool == DS4_HELP_BENCH) {
+        print_bench_specific(fp, c);
+    } else if (tool == DS4_HELP_EVAL) {
+        print_eval_specific(fp, c);
+    }
+}
+
+void ds4_help_print(FILE *fp, ds4_help_tool tool, const char *topic) {
+    help_colors c = help_make_colors(fp);
+    if (topic && !tool_has_topic(tool, topic)) {
+        fprintf(fp, "%s: unknown help topic '%s'\n\n", tool_name(tool), topic);
+        topic = NULL;
+    }
+
+    fprintf(fp, "%s%s%s\n", c.bright ? c.bright : "", tool_name(tool), c.off ? c.off : "");
+    fprintf(fp, "%s\n\n", tool_summary(tool));
+    fprintf(fp, "%s\n\n", tool_usage(tool));
+
+    if (topic) print_topic(fp, &c, tool, topic);
+    else {
+        print_default(fp, &c, tool);
+        print_more_info(fp, &c, tool);
+    }
+    print_examples(fp, &c, tool, topic);
+}

ds4_help.h

@@ -0,0 +1,16 @@
+#ifndef DS4_HELP_H
+#define DS4_HELP_H
+
+#include <stdio.h>
+
+typedef enum {
+    DS4_HELP_DS4,
+    DS4_HELP_SERVER,
+    DS4_HELP_AGENT,
+    DS4_HELP_BENCH,
+    DS4_HELP_EVAL,
+} ds4_help_tool;
+
+void ds4_help_print(FILE *fp, ds4_help_tool tool, const char *topic);
+
+#endif

ds4_kvstore.h

@@ -0,0 +1,218 @@
+#ifndef DS4_KVSTORE_H
+#define DS4_KVSTORE_H
+
+#include "ds4.h"
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+
+#define DS4_KVSTORE_FIXED_HEADER 48u
+#define DS4_KVSTORE_DEFAULT_MB 4096
+#define DS4_KVSTORE_HIT_HALF_LIFE_SECONDS (6ull * 60ull * 60ull)
+
+#define DS4_KVSTORE_EXT_TOOL_MAP          (1u << 0)
+#define DS4_KVSTORE_EXT_RESPONSES_VISIBLE (1u << 1)
+#define DS4_KVSTORE_EXT_THINKING_VISIBLE  (1u << 2)
+#define DS4_KVSTORE_EXT_SESSION_TITLE     (1u << 3)
+
+typedef enum {
+    DS4_KVSTORE_REASON_UNKNOWN   = 0,
+    DS4_KVSTORE_REASON_COLD      = 1,
+    DS4_KVSTORE_REASON_CONTINUED = 2,
+    DS4_KVSTORE_REASON_EVICT     = 3,
+    DS4_KVSTORE_REASON_SHUTDOWN  = 4,
+    DS4_KVSTORE_REASON_AGENT_SYSTEM  = 5,
+    DS4_KVSTORE_REASON_AGENT_SESSION = 6,
+} ds4_kvstore_reason;
+
+typedef enum {
+    DS4_KVSTORE_LOG_DEFAULT,
+    DS4_KVSTORE_LOG_KVCACHE,
+    DS4_KVSTORE_LOG_WARNING,
+} ds4_kvstore_log_type;
+
+typedef struct {
+    /* The file name is the rendered byte prefix, not the token sequence. The
+     * payload still carries the exact tokens and graph state; the hash only
+     * answers "does this checkpoint represent the bytes at the front of the
+     * incoming prompt?" */
+    char sha[41];
+    char *path;
+    uint8_t quant_bits;
+    /* Stored in header byte 7.  Flash is 0 for backward compatibility with
+     * older cache files where this reserved byte was always written as zero. */
+    uint8_t model_id;
+    uint8_t reason;
+    uint32_t tokens;
+    uint32_t hits;
+    uint32_t ctx_size;
+    uint8_t ext_flags;
+    uint64_t created_at;
+    uint64_t last_used;
+    uint64_t payload_bytes;
+    uint64_t text_bytes;
+    uint64_t file_size;
+} ds4_kvstore_entry;
+
+typedef struct {
+    int min_tokens;
+    int cold_max_tokens;
+    int continued_interval_tokens;
+    int boundary_trim_tokens;
+    int boundary_align_tokens;
+} ds4_kvstore_options;
+
+typedef struct {
+    bool enabled;
+    char *dir;
+    uint64_t budget_bytes;
+    bool reject_different_quant;
+    ds4_kvstore_options opt;
+    int continued_last_store_tokens;
+    ds4_kvstore_entry *entry;
+    int len;
+    int cap;
+    const char *log_name;
+    void *log_ud;
+    void (*log)(void *ud, ds4_kvstore_log_type type, const char *msg);
+} ds4_kvstore;
+
+typedef struct {
+    const char *text;
+    size_t text_len;
+    uint8_t model_id;
+    uint8_t quant_bits;
+    uint32_t ctx_size;
+    bool reject_different_quant;
+} ds4_kvstore_eviction_context;
+
+typedef struct {
+    void *ud;
+    uint8_t ext_flag;
+    bool (*serialized_size)(void *ud, const char *text, uint64_t *bytes_out);
+    bool (*write)(void *ud, FILE *fp, const char *text, uint64_t *written_bytes);
+    int (*load)(void *ud, FILE *fp, const void *wanted);
+    const void *load_wanted;
+} ds4_kvstore_trailer_hooks;
+
+typedef struct {
+    int tokens;
+    uint32_t text_bytes;
+    uint8_t quant_bits;
+    uint8_t ext_flags;
+    double load_ms;
+    bool consumed;
+    char *path;
+} ds4_kvstore_load_result;
+
+ds4_kvstore_options ds4_kvstore_default_options(void);
+uint8_t ds4_kvstore_reason_code(const char *reason);
+const char *ds4_kvstore_key_kind(uint8_t ext_flags);
+
+bool ds4_kvstore_open(ds4_kvstore *kc, const char *dir, uint64_t budget_mb,
+                      bool reject_different_quant, ds4_kvstore_options opt,
+                      const char *log_name,
+                      void (*log)(void *ud, ds4_kvstore_log_type type, const char *msg),
+                      void *log_ud);
+void ds4_kvstore_close(ds4_kvstore *kc);
+void ds4_kvstore_clear(ds4_kvstore *kc);
+void ds4_kvstore_entry_free(ds4_kvstore_entry *e);
+
+char *ds4_kvstore_render_tokens_text(ds4_engine *engine,
+                                     const ds4_tokens *tokens,
+                                     size_t *out_len);
+bool ds4_kvstore_byte_prefix_match(const char *text, size_t text_len,
+                                   const char *prefix, size_t prefix_len);
+void ds4_kvstore_tokens_copy_prefix(ds4_tokens *dst, const ds4_tokens *src, int n);
+void ds4_kvstore_build_prompt_from_exact_prefix_and_text_suffix(
+        ds4_engine *engine,
+        const ds4_tokens *exact_prefix,
+        const char *suffix_text,
+        ds4_tokens *out);
+
+int ds4_kvstore_store_len(const ds4_kvstore *kc, int tokens);
+int ds4_kvstore_chat_anchor_pos(const ds4_kvstore *kc,
+                                const ds4_tokens *prompt,
+                                int user_token_id,
+                                int assistant_token_id);
+int ds4_kvstore_continued_store_target(const ds4_kvstore *kc, int live_tokens);
+void ds4_kvstore_note_store(ds4_kvstore *kc, int tokens);
+int ds4_kvstore_suppress_continued_store(ds4_kvstore *kc, int tokens);
+void ds4_kvstore_restore_suppressed_continued(ds4_kvstore *kc,
+                                              int old_tokens,
+                                              int suppressed_tokens);
+
+bool ds4_kvstore_file_size_fits(const ds4_kvstore *kc,
+                                uint64_t text_bytes,
+                                uint64_t payload_bytes,
+                                uint64_t trailer_bytes,
+                                uint64_t *file_bytes_out,
+                                uint64_t *required_bytes_out);
+double ds4_kvstore_entry_eviction_score(const ds4_kvstore_entry *e,
+                                        const ds4_tokens *live,
+                                        uint64_t now,
+                                        const ds4_kvstore_eviction_context *incoming);
+void ds4_kvstore_evict(ds4_kvstore *kc, const ds4_tokens *live,
+                       uint64_t extra_bytes,
+                       const ds4_kvstore_eviction_context *incoming);
+int ds4_kvstore_find_text_prefix(ds4_kvstore *kc, const char *prompt_text,
+                                 int model_id, int quant_bits, int ctx_size);
+
+bool ds4_kvstore_store_live_prefix_text(ds4_kvstore *kc,
+                                        ds4_engine *engine,
+                                        ds4_session *session,
+                                        const ds4_tokens *tokens,
+                                        int store_len,
+                                        const char *reason,
+                                        const char *cache_text_override,
+                                        uint8_t cache_text_ext,
+                                        const char *cache_text_key,
+                                        const ds4_kvstore_trailer_hooks *hooks,
+                                        char *err,
+                                        size_t err_len);
+bool ds4_kvstore_store_live_prefix(ds4_kvstore *kc,
+                                   ds4_engine *engine,
+                                   ds4_session *session,
+                                   const ds4_tokens *tokens,
+                                   int store_len,
+                                   const char *reason,
+                                   const ds4_kvstore_trailer_hooks *hooks,
+                                   char *err,
+                                   size_t err_len);
+bool ds4_kvstore_maybe_store_continued(ds4_kvstore *kc,
+                                       ds4_engine *engine,
+                                       ds4_session *session,
+                                       const ds4_kvstore_trailer_hooks *hooks,
+                                       char *err,
+                                       size_t err_len);
+int ds4_kvstore_try_load_text(ds4_kvstore *kc,
+                              ds4_engine *engine,
+                              ds4_session *session,
+                              const char *prompt_text,
+                              ds4_tokens *effective_prompt,
+                              ds4_kvstore_load_result *result,
+                              const ds4_kvstore_trailer_hooks *hooks,
+                              bool responses_protocol);
+void ds4_kvstore_load_result_free(ds4_kvstore_load_result *result);
+
+bool ds4_kvstore_read_header(FILE *fp, ds4_kvstore_entry *e,
+                             uint32_t *text_bytes);
+bool ds4_kvstore_read_entry_file(const char *path, const char sha[41],
+                                 ds4_kvstore_entry *out);
+void ds4_kvstore_fill_header(uint8_t h[DS4_KVSTORE_FIXED_HEADER],
+                             uint8_t model_id, uint8_t quant_bits,
+                             uint8_t reason, uint8_t ext_flags,
+                             uint32_t tokens, uint32_t hits, uint32_t ctx_size,
+                             uint64_t created_at, uint64_t last_used,
+                             uint64_t payload_bytes);
+bool ds4_kvstore_touch_file(const char *path, uint32_t hits);
+bool ds4_kvstore_sha_hex_name(const char *name, char sha[41]);
+void ds4_kvstore_sha1_bytes_hex(const void *ptr, size_t len, char out[41]);
+char *ds4_kvstore_path_join(const char *dir, const char *name);
+char *ds4_kvstore_path_for_sha(ds4_kvstore *kc, const char sha[41]);
+void ds4_kvstore_le_put32(uint8_t *p, uint32_t v);
+uint32_t ds4_kvstore_le_get32(const uint8_t *p);
+
+#endif

ds4_layer_pack.c

@@ -0,0 +1,149 @@
+/* Monotonic-contiguous multi-tier layer placement packer.
+ *
+ * Pure C99: no CUDA, no platform-specific code. Used by both the CUDA and
+ * Metal/CPU builds. See ds4_layer_pack.h for the API contract and the
+ * design doc docs/superpowers/specs/2026-05-26-multi-gpu-pp-v0-design.md
+ * for the rationale. */
+
+#include "ds4_layer_pack.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+int ds4_compute_layer_placement(const size_t *entry_bytes,
+                                int n_entries,
+                                const ds4_layer_pack_config *cfg,
+                                int *device_for_entry) {
+    if (!entry_bytes || !cfg || !device_for_entry) return 1;
+    if (n_entries < 0) return 2;
+    if (cfg->n_gpus < 0 || cfg->n_gpus > DS4_LAYER_PACK_MAX_GPUS) return 3;
+
+    /* Work over a local copy of budgets so the caller's struct is untouched. */
+    size_t budget[DS4_LAYER_PACK_MAX_GPUS];
+    for (int d = 0; d < cfg->n_gpus; d++) {
+        budget[d] = cfg->gpu_budget_bytes[d];
+    }
+
+    int d = 0;
+    for (int e = 0; e < n_entries; e++) {
+        /* Advance until the current device can hold this entry, or we run
+         * out of devices. Strict greater-than: exact fits stay on d. */
+        while (d < cfg->n_gpus && entry_bytes[e] > budget[d]) {
+            d++;
+        }
+        if (d < cfg->n_gpus) {
+            device_for_entry[e] = d;
+            budget[d] -= entry_bytes[e];
+        } else {
+            device_for_entry[e] = DS4_LAYER_PACK_CPU;
+            /* d stays at cfg->n_gpus so every subsequent entry also lands
+             * on the CPU tier. */
+        }
+    }
+    return 0;
+}
+
+/* Internal helper: append " + embedding" or " + output head" to a buffer if
+ * the matching pseudo-layer lives on this tier. */
+static void append_pseudo_layer_tags(char *buf, size_t buflen,
+                                      const int *device_for_entry,
+                                      int tier,
+                                      int n_entries,
+                                      int n_layers) {
+    /* entry 0 -> embedding, entry n_layers+1 -> output head */
+    if (n_entries > 0 && device_for_entry[0] == tier) {
+        strncat(buf, " + embedding", buflen - strlen(buf) - 1);
+    }
+    if (n_entries > 1 && n_layers + 1 < n_entries &&
+        device_for_entry[n_layers + 1] == tier) {
+        strncat(buf, " + output head", buflen - strlen(buf) - 1);
+    }
+}
+
+/* Emit "L-R" for a contiguous transformer-layer range assigned to a tier.
+ * Returns 1 if anything was printed (i.e. tier owned at least one layer). */
+static int print_layer_range_for_tier(FILE *out, const int *device_for_entry,
+                                      int tier, int n_entries, int n_layers) {
+    /* Transformer layers live at indices 1..n_layers in device_for_entry.
+     * Their human-facing numbers are 0..n_layers-1 (i.e. entry index - 1). */
+    int first = -1;
+    int last  = -1;
+    for (int i = 1; i <= n_layers && i < n_entries; i++) {
+        if (device_for_entry[i] == tier) {
+            if (first < 0) first = i - 1;  /* human-facing index */
+            last = i - 1;
+        }
+    }
+    if (first < 0) {
+        fprintf(out, "(no transformer layers)");
+        return 0;
+    }
+    if (first == last) {
+        fprintf(out, "layer %d", first);
+    } else {
+        fprintf(out, "layers %d-%d", first, last);
+    }
+    return 1;
+}
+
+void ds4_layer_pack_print(FILE *out,
+                          const int *device_for_entry,
+                          int n_entries,
+                          int n_layers,
+                          const size_t *entry_bytes,
+                          const size_t *gpu_used_bytes,
+                          const size_t *gpu_budget_bytes,
+                          int n_gpus) {
+    if (!out || !device_for_entry) return;
+    (void)entry_bytes; /* not needed for the textual layout */
+
+    fprintf(out, "multi-GPU layout:\n");
+    for (int d = 0; d < n_gpus; d++) {
+        fprintf(out, "  GPU%d: ", d);
+        int have_layers = print_layer_range_for_tier(out, device_for_entry,
+                                                     d, n_entries, n_layers);
+        /* Pseudo-layer tags. */
+        char tags[64];
+        tags[0] = '\0';
+        append_pseudo_layer_tags(tags, sizeof(tags), device_for_entry, d,
+                                  n_entries, n_layers);
+        if (tags[0]) {
+            if (!have_layers) {
+                /* Erase the "(no transformer layers)" stub and just write
+                 * the tag with leading "+ " stripped. */
+                /* The stub was already written; appending " + embedding"
+                 * is still informative. */
+            }
+            fprintf(out, "%s", tags);
+        }
+        /* Usage line. */
+        if (gpu_used_bytes && gpu_budget_bytes) {
+            const double used_gb   = (double)gpu_used_bytes[d]   / 1073741824.0;
+            const double budget_gb = (double)gpu_budget_bytes[d] / 1073741824.0;
+            /* Two-space pad before the parens matches the design doc. */
+            fprintf(out, "  (%.1f / %.1f GB)", used_gb, budget_gb);
+        }
+        fputc('\n', out);
+    }
+    /* CPU tier: only print if at least one entry is on CPU. */
+    int cpu_present = 0;
+    for (int i = 0; i < n_entries; i++) {
+        if (device_for_entry[i] == DS4_LAYER_PACK_CPU) { cpu_present = 1; break; }
+    }
+    if (cpu_present) {
+        fprintf(out, "  CPU : ");
+        int have_layers = print_layer_range_for_tier(out, device_for_entry,
+                                                      DS4_LAYER_PACK_CPU,
+                                                      n_entries, n_layers);
+        char tags[64];
+        tags[0] = '\0';
+        append_pseudo_layer_tags(tags, sizeof(tags), device_for_entry,
+                                  DS4_LAYER_PACK_CPU, n_entries, n_layers);
+        if (tags[0]) {
+            (void)have_layers;
+            fprintf(out, "%s", tags);
+        }
+        fputc('\n', out);
+    }
+}

ds4_layer_pack.h

@@ -0,0 +1,77 @@
+#ifndef DS4_LAYER_PACK_H
+#define DS4_LAYER_PACK_H
+
+#include <stddef.h>
+#include <stdio.h>
+
+#define DS4_LAYER_PACK_MAX_GPUS 16
+#define DS4_LAYER_PACK_CPU      (-1)
+
+typedef struct {
+    /* Per-device budget in BYTES (already net of per-device overhead such as
+     * cuBLAS workspace, global scratch, logits buffer, MTP, steering, and the
+     * configured safety margin). The packer treats these as raw capacity. */
+    size_t gpu_budget_bytes[DS4_LAYER_PACK_MAX_GPUS];
+    int    n_gpus;
+} ds4_layer_pack_config;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Compute a monotonic-contiguous layer placement.
+ *
+ * Inputs:
+ *   entry_bytes[]     — per-entry byte footprint, in FORWARD ORDER:
+ *                         entry 0           = embedding pseudo-layer
+ *                         entry 1..n_layers = transformer layers
+ *                         entry n_layers+1  = output-head pseudo-layer
+ *   n_entries         — total number of entries (typically n_layers + 2)
+ *   cfg               — per-device budgets
+ *
+ * Output:
+ *   device_for_entry[] — filled with target device for each entry; CPU is
+ *                        DS4_LAYER_PACK_CPU. Caller-owned, must be
+ *                        large enough for n_entries int slots.
+ *
+ * Returns:
+ *   0 on success.
+ *   Nonzero on configuration errors (null pointer, n_entries < 0,
+ *   n_gpus out of range).
+ *
+ * Per the design doc, an entry that exceeds every GPU budget spills to CPU;
+ * by the monotonicity rule every entry after it also goes to CPU. There is
+ * no error path for "entry too large" — the CPU tier is always available. */
+int ds4_compute_layer_placement(const size_t *entry_bytes,
+                                int n_entries,
+                                const ds4_layer_pack_config *cfg,
+                                int *device_for_entry);
+
+/* Print a human-readable layout summary matching the design doc's format:
+ *
+ *   multi-GPU layout:
+ *     GPU0: layers 0-21 + embedding   (38.4 / 40.0 GB)
+ *     GPU1: layers 22-31              (11.7 / 12.0 GB)
+ *     CPU : layers 32-42 + output head
+ *
+ * Entry-naming convention:
+ *   entry 0          -> "embedding"
+ *   entry i in 1..n_layers -> transformer layer (numbered i-1)
+ *   entry n_layers+1 -> "output head"
+ *
+ * gpu_used_bytes[] and gpu_budget_bytes[] may be NULL — in that case the
+ * "(used / budget)" line is omitted for the GPU lines. */
+void ds4_layer_pack_print(FILE *out,
+                          const int *device_for_entry,
+                          int n_entries,
+                          int n_layers,
+                          const size_t *entry_bytes,
+                          const size_t *gpu_used_bytes,
+                          const size_t *gpu_budget_bytes,
+                          int n_gpus);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* DS4_LAYER_PACK_H */

ds4_rocm.cu

@@ -0,0 +1,229 @@
+#ifdef __HIP_PLATFORM_AMD__
+#include "ds4_rocm.h"
+#include <hipblaslt/hipblaslt.h>
+
+#define FULL_WARP_MASK 0xFFFFFFFFFFFFFFFFULL
+#define MASK_T uint64_t
+#define DS4_GPU_BACKEND_NAME "ROCm"
+#define DS4_GPU_LOG_PREFIX "ds4: ROCm "
+#define DS4_GPU_BLAS_NAME "hipBLAS"
+#else
+#include <cuda_runtime.h>
+#include <cuda_fp16.h>
+#include <mma.h>
+#include <cublas_v2.h>
+#include <cub/block/block_radix_sort.cuh>
+
+#define FULL_WARP_MASK 0xFFFFFFFFu
+#define MASK_T uint32_t
+#define DS4_GPU_BACKEND_NAME "CUDA"
+#define DS4_GPU_LOG_PREFIX "ds4: CUDA "
+#define DS4_GPU_BLAS_NAME "cuBLAS"
+#endif
+
+#include <stdint.h>
+#include <ctype.h>
+#include <errno.h>
+#include <limits.h>
+#include <math.h>
+#include <fcntl.h>
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <time.h>
+#include <unistd.h>
+#include <unordered_map>
+#include <vector>
+
+#include "ds4_gpu.h"
+
+#ifndef M_PI
+#define M_PI 3.14159265358979323846
+#endif
+
+#define CUDA_QK_K 256
+#define DS4_ROCM_UNUSED __attribute__((unused))
+
+enum {
+    /* attention_decode_mixed_kernel stores raw-window scores plus visible
+     * compressed scores in shared memory.  The host routes larger unmasked
+     * decode calls to the online attention kernel so this fixed buffer never
+     * becomes an out-of-bounds write at long context. */
+    DS4_ROCM_ATTENTION_SCORE_CAP = 8192u,
+    DS4_ROCM_ATTENTION_RAW_SCORE_CAP = 256u,
+    DS4_ROCM_TOPK_MERGE_GROUP = 8u
+};
+
+struct ds4_gpu_tensor {
+    void *ptr;
+    uint64_t bytes;
+    int owner;
+};
+
+typedef struct {
+    uint8_t scales[CUDA_QK_K / 16];
+    uint8_t qs[CUDA_QK_K / 4];
+    uint16_t d;
+    uint16_t dmin;
+} cuda_block_q2_K;
+
+typedef struct {
+    uint8_t hmask[CUDA_QK_K / 8];
+    uint8_t qs[CUDA_QK_K / 4];
+    uint8_t scales[12];
+    uint16_t d;
+} cuda_block_q3_K;
+
+typedef struct {
+    uint16_t d;
+    uint16_t dmin;
+    uint8_t scales[12];
+    uint8_t qs[CUDA_QK_K / 2];
+} cuda_block_q4_K;
+
+typedef struct {
+    float d;
+    int8_t qs[CUDA_QK_K];
+    int16_t bsums[CUDA_QK_K / 16];
+} cuda_block_q8_K;
+
+typedef struct {
+    uint16_t d;
+    uint16_t qs[CUDA_QK_K / 8];
+} cuda_block_iq2_xxs;
+
+static_assert(sizeof(cuda_block_q3_K) == 110, "Q3_K block layout mismatch");
+
+#include "ds4_iq2_tables_cuda.inc"
+
+#include "rocm/ds4_rocm_runtime.cuh"
+
+#include "rocm/ds4_rocm_common.cuh"
+
+#include "rocm/ds4_rocm_q8.cuh"
+
+#include "rocm/ds4_rocm_norm_rope.cuh"
+
+#include "rocm/ds4_rocm_laguna.cuh"
+
+#include "rocm/ds4_rocm_dflash.cuh"
+
+#include "rocm/ds4_rocm_fp8_kv.cuh"
+
+#include "rocm/ds4_rocm_attention.cuh"
+
+#include "rocm/ds4_rocm_hc.cuh"
+
+#include "rocm/ds4_rocm_output.cuh"
+
+#include "rocm/ds4_rocm_indexer.cuh"
+
+#include "rocm/ds4_rocm_embedding_launch.cuh"
+
+#include "rocm/ds4_rocm_matmul.cuh"
+
+#include "rocm/ds4_rocm_fp8_kv_launch.cuh"
+
+#include "rocm/ds4_rocm_compressor.cuh"
+
+#include "rocm/ds4_rocm_attention_launch.cuh"
+
+#include "rocm/ds4_rocm_shared_expert.cuh"
+
+#include "rocm/ds4_rocm_misc_launch.cuh"
+#include "rocm/ds4_rocm_router.cuh"
+
+#include "rocm/ds4_rocm_moe.cuh"
+
+#include "rocm/ds4_rocm_moe_launch.cuh"
+
+#include "rocm/ds4_rocm_glm.cuh"
+
+#include "rocm/ds4_rocm_hc_output_launch.cuh"
+
+#include "rocm/ds4_rocm_current_api_compat.cuh"
+
+/* Tensor-parallel gates are Metal-only; stubs keep shared graph code
+ * linkable (TP option validation rejects non-Metal backends). */
+extern "C" int ds4_gpu_tp_gate_encode(uint32_t layer, uint32_t gate) {
+    (void)layer; (void)gate;
+    fprintf(stderr, DS4_GPU_LOG_PREFIX "tensor parallelism is Metal-only\n");
+    return 0;
+}
+
+extern "C" void ds4_gpu_tp_set_batch_exchange(ds4_gpu_tp_batch_exchange_fn fn) {
+    (void)fn;
+}
+
+extern "C" void ds4_gpu_tp_suspend_expert_sharding(int suspend) {
+    (void)suspend;
+}
+
+extern "C" void ds4_gpu_tp_keepalive_pause(int paused) {
+    (void)paused;
+}
+
+extern "C" void ds4_gpu_tp_set_attn_head_split(int enabled) {
+    (void)enabled;
+}
+
+extern "C" void ds4_gpu_model_residency_skip(int skip) {
+    (void)skip;
+}
+
+extern "C" void ds4_gpu_tp_set_big_exchange(ds4_gpu_tp_big_exchange_fn fn) {
+    (void)fn;
+}
+
+extern "C" int ds4_gpu_tp_big_gate_encode(uint32_t layer, uint32_t rows,
+                                          const ds4_gpu_tensor *out_t,
+                                          ds4_gpu_tensor *in_t,
+                                          uint64_t bytes) {
+    (void)layer; (void)rows; (void)out_t; (void)in_t; (void)bytes;
+    return 0;
+}
+
+extern "C" int ds4_gpu_tp_batch_gate_encode(uint32_t layer, uint32_t rows) {
+    (void)layer; (void)rows;
+    fprintf(stderr, DS4_GPU_LOG_PREFIX "tensor parallelism is Metal-only\n");
+    return 0;
+}
+
+extern "C" int ds4_gpu_matmul_q8_0_kslice_tensor(
+        ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
+        uint64_t weight_offset, uint64_t full_in_dim, uint64_t k_off,
+        uint64_t k_cnt, uint64_t out_dim, const ds4_gpu_tensor *x,
+        uint64_t x_elem_off) {
+    (void)out; (void)model_map; (void)model_size; (void)weight_offset;
+    (void)full_in_dim; (void)k_off; (void)k_cnt; (void)out_dim; (void)x;
+    (void)x_elem_off;
+    fprintf(stderr, DS4_GPU_LOG_PREFIX "tensor parallelism is Metal-only\n");
+    return 0;
+}
+
+extern "C" int ds4_gpu_attention_output_q8_tp_tensor(
+        ds4_gpu_tensor *out, ds4_gpu_tensor *low, const void *model_map,
+        uint64_t model_size, uint64_t out_a_offset, uint64_t out_b_offset,
+        uint64_t group_dim, uint64_t rank, uint32_t n_groups_total,
+        uint32_t group0, uint32_t group_cnt, uint64_t out_dim,
+        const ds4_gpu_tensor *heads) {
+    (void)out; (void)low; (void)model_map; (void)model_size;
+    (void)out_a_offset; (void)out_b_offset; (void)group_dim; (void)rank;
+    (void)n_groups_total; (void)group0; (void)group_cnt; (void)out_dim;
+    (void)heads;
+    fprintf(stderr, DS4_GPU_LOG_PREFIX "tensor parallelism is Metal-only\n");
+    return 0;
+}
+
+extern "C" int ds4_gpu_hc_expand_add_tensor(
+        ds4_gpu_tensor *out_hc, const ds4_gpu_tensor *block_out,
+        const ds4_gpu_tensor *block_add, const ds4_gpu_tensor *residual_hc,
+        const ds4_gpu_tensor *post, const ds4_gpu_tensor *comb,
+        uint32_t n_embd, uint32_t n_hc) {
+    (void)out_hc; (void)block_out; (void)block_add; (void)residual_hc;
+    (void)post; (void)comb; (void)n_embd; (void)n_hc;
+    fprintf(stderr, DS4_GPU_LOG_PREFIX "tensor parallelism is Metal-only\n");
+    return 0;
+}

ds4_rocm.h

@@ -0,0 +1,142 @@
+#pragma once
+
+#include <hip/hip_runtime.h>
+#include <hipblas/hipblas.h>
+#include <hip/hip_fp16.h>
+#include <hipcub/hipcub.hpp>
+#include <rocwmma/rocwmma-version.hpp>
+#include <rocwmma/rocwmma.hpp>
+
+#define cudaError_t hipError_t
+#define cudaStream_t hipStream_t
+#define cudaEvent_t hipEvent_t
+#define cudaDeviceProp hipDeviceProp_t
+#define cudaMemLocation hipMemLocation
+
+#define cudaSuccess hipSuccess
+#define cudaErrorNotSupported hipErrorNotSupported
+#define cudaErrorInvalidValue hipErrorInvalidValue
+#define cudaGetLastError hipGetLastError
+#define cudaGetErrorString hipGetErrorString
+
+#define cudaGetDevice hipGetDevice
+#define cudaSetDevice hipSetDevice
+#define cudaDeviceSynchronize hipDeviceSynchronize
+#define cudaDeviceGetAttribute hipDeviceGetAttribute
+#define cudaGetDeviceProperties hipGetDeviceProperties
+#define cudaDevAttrPageableMemoryAccess hipDeviceAttributePageableMemoryAccess
+#define cudaDevAttrMaxSharedMemoryPerBlockOptin hipDeviceAttributeSharedMemPerBlockOptin
+#define cudaFuncAttributeMaxDynamicSharedMemorySize hipFuncAttributeMaxDynamicSharedMemorySize
+#define cudaFuncSetAttribute(func, attr, value) hipFuncSetAttribute((const void *)(func), (attr), (value))
+#define cudaMemLocationTypeDevice hipMemLocationTypeDevice
+
+#define cudaMalloc hipMalloc
+#define cudaMallocHost hipHostMalloc
+#define cudaMallocManaged hipMallocManaged
+#define cudaFree hipFree
+#define cudaFreeHost hipFreeHost
+#define cudaMemset hipMemset
+#define cudaMemcpy hipMemcpy
+#define cudaMemcpyAsync hipMemcpyAsync
+#define cudaMemcpyHostToDevice hipMemcpyHostToDevice
+#define cudaMemcpyDeviceToHost hipMemcpyDeviceToHost
+#define cudaMemcpyDeviceToDevice hipMemcpyDeviceToDevice
+#define cudaMemGetInfo hipMemGetInfo
+#define cudaMemsetAsync hipMemsetAsync
+
+#define cudaHostRegister hipHostRegister
+#define cudaHostUnregister hipHostUnregister
+#define cudaHostGetDevicePointer hipHostGetDevicePointer
+#define cudaHostRegisterMapped hipHostRegisterMapped
+#define cudaHostRegisterReadOnly hipHostRegisterReadOnly
+
+#define cudaMemAdvise(p1, p2, p3, p4) hipMemAdvise(p1, p2, p3, p4.id)
+#define cudaMemPrefetchAsync(devPtr, count, location, flags, stream) hipMemPrefetchAsync(devPtr, count, location.id, stream)
+#define cudaMemAdviseSetReadMostly hipMemAdviseSetReadMostly
+#define cudaMemAdviseSetPreferredLocation hipMemAdviseSetPreferredLocation
+
+#define cudaStreamCreateWithFlags hipStreamCreateWithFlags
+#define cudaStreamSynchronize hipStreamSynchronize
+#define cudaStreamDestroy hipStreamDestroy
+#define cudaStreamNonBlocking hipStreamNonBlocking
+
+#define cudaEventCreate hipEventCreate
+#define cudaEventCreateWithFlags hipEventCreateWithFlags
+#define cudaEventDestroy hipEventDestroy
+#define cudaEventRecord hipEventRecord
+#define cudaEventSynchronize hipEventSynchronize
+#define cudaEventElapsedTime hipEventElapsedTime
+#define cudaEventDisableTiming hipEventDisableTiming
+
+#define cublasHandle_t hipblasHandle_t
+#define cublasStatus_t hipblasStatus_t
+#define cublasMath_t hipblasMath_t
+
+#define CUBLAS_STATUS_SUCCESS HIPBLAS_STATUS_SUCCESS
+#define CUBLAS_OP_N HIPBLAS_OP_N
+#define CUBLAS_OP_T HIPBLAS_OP_T
+#define CUBLAS_GEMM_DEFAULT HIPBLAS_GEMM_DEFAULT
+#define CUBLAS_DEFAULT_MATH HIPBLAS_DEFAULT_MATH
+#define CUBLAS_COMPUTE_32F HIPBLAS_COMPUTE_32F
+#define CUBLAS_TF32_TENSOR_OP_MATH HIPBLAS_TF32_TENSOR_OP_MATH
+#define CUDA_R_16F HIPBLAS_R_16F
+#define CUDA_R_32F HIPBLAS_R_32F
+
+#define cublasCreate hipblasCreate
+#define cublasDestroy hipblasDestroy
+#define cublasSetMathMode hipblasSetMathMode
+#define cublasSgemm hipblasSgemm
+#define cublasSgemmStridedBatched hipblasSgemmStridedBatched
+#define cublasGemmEx hipblasGemmEx
+#define cublasGemmStridedBatchedEx hipblasGemmStridedBatchedEx
+
+namespace cub = hipcub;
+
+static __device__ __forceinline__ int32_t __vcmpne4(uint32_t a, uint32_t b) {
+    // For each byte: 0xFF if a != b, 0x00 if a == b
+    uint32_t diff = a ^ b;
+    // Spread any set bit in each byte to fill the whole byte
+    diff |= (diff >> 1); diff |= (diff >> 2); diff |= (diff >> 4);
+    diff &= 0x01010101u;
+    diff *= 0xFFu; // 0x01 -> 0xFF per byte
+    return (int32_t)diff;
+}
+
+static __device__ __forceinline__ int32_t __vsub4(int32_t a, int32_t b) {
+    // Per-byte subtraction (wrapping, not saturating)
+    uint32_t ua = (uint32_t)a, ub = (uint32_t)b;
+    // Trick: subtract bytes in parallel avoiding cross-byte borrows
+    uint32_t diff = ((ua | 0x80808080u) - (ub & 0x7F7F7F7Fu)) ^ ((ua ^ ~ub) & 0x80808080u);
+    return (int32_t)diff;
+}
+
+// __dp4a: dot product of 4 signed int8s packed in an int32.
+// gfx11-class AMD GPUs expose this as a single v_dot4_i32_i8 instruction;
+// using the clang builtin avoids expanding every Q8/Q8_K dot into scalar byte
+// multiplies in the ROCm compatibility layer.
+static __device__ __forceinline__ int32_t __dp4a(int32_t a, int32_t b, int32_t c) {
+#if defined(__HIP_PLATFORM_AMD__) || defined(__HIPCC__)
+    union ds4_i8x4_bits { int32_t i; char4 v; } av, bv;
+    av.i = a;
+    bv.i = b;
+    return amd_mixed_dot(av.v, bv.v, c, false);
+#else
+    const int8_t *a_bytes = reinterpret_cast<const int8_t*>(&a);
+    const int8_t *b_bytes = reinterpret_cast<const int8_t*>(&b);
+    return c + (int32_t)a_bytes[0] * b_bytes[0]
+             + (int32_t)a_bytes[1] * b_bytes[1]
+             + (int32_t)a_bytes[2] * b_bytes[2]
+             + (int32_t)a_bytes[3] * b_bytes[3];
+#endif
+}
+
+// Precise transcendentals for the MoE router top-k scores, immune to -fapprox-func.
+// These functions are to be used on paths where small error can be translated to
+// some macro effect - like expert selection kernels
+extern "C" __device__ __attribute__((pure))  float __ocml_exp_f32(float);
+extern "C" __device__ __attribute__((pure))  float __ocml_log1p_f32(float);
+extern "C" __device__ __attribute__((const)) float __ocml_sqrt_f32(float);
+
+static __device__ __forceinline__ float ds4_precise_expf(float x)   { return __ocml_exp_f32(x); }
+static __device__ __forceinline__ float ds4_precise_log1pf(float x) { return __ocml_log1p_f32(x); }
+static __device__ __forceinline__ float ds4_precise_sqrtf(float x)  { return __ocml_sqrt_f32(x); }

ds4_rocm_compat.cu

@@ -0,0 +1,423 @@
+#include <hip/hip_runtime.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "ds4_gpu_mgpu.h"
+#include "ds4_gpu.h"
+#include "ds4_gpu_args.h"
+
+ds4_gpu_ctx g_gpu[DS4_MAX_GPUS] = {};
+int g_n_gpus = 1;
+int g_gpu_peer_ok[DS4_MAX_GPUS][DS4_MAX_GPUS] = {{1}};
+
+static int rocm_tier_valid(int tier) {
+    return tier == 0 && g_n_gpus == 1;
+}
+
+extern "C" int ds4_gpu_init_multi(const ds4_gpu_config *cfg) {
+    if (!cfg || cfg->n_gpus != 1) {
+        fprintf(stderr, "ds4: ROCm supports one GPU per process\n");
+        return 0;
+    }
+    g_gpu[0].device_id = cfg->device_indices[0];
+    if (hipSetDevice(g_gpu[0].device_id) != hipSuccess) return 0;
+    return ds4_gpu_init();
+}
+
+extern "C" int ds4_gpu_set_current_device(int tier) {
+    if (!rocm_tier_valid(tier)) return 1;
+    return hipSetDevice(g_gpu[0].device_id) == hipSuccess ? 0 : 1;
+}
+
+extern "C" int ds4_gpu_set_current_device_fenced(int tier) {
+    return ds4_gpu_set_current_device(tier);
+}
+
+extern "C" int ds4_gpu_tensor_alloc_on(ds4_gpu_tensor *t, int tier,
+                                         uint64_t bytes) {
+    if (!t) return 1;
+    if (!rocm_tier_valid(tier)) return 2;
+    if (bytes == 0) bytes = 1;
+    if (hipMalloc(&t->ptr, (size_t)bytes) != hipSuccess) return 3;
+    t->bytes = bytes;
+    t->owner = 1;
+    t->device_id = 0;
+    return 0;
+}
+
+extern "C" ds4_gpu_tensor *ds4_gpu_tensor_alloc_ptr_on(int tier,
+                                                         uint64_t bytes) {
+    if (!rocm_tier_valid(tier)) return NULL;
+    return ds4_gpu_tensor_alloc(bytes);
+}
+
+extern "C" ds4_gpu_tensor *ds4_gpu_tensor_alloc_managed_on(int tier,
+                                                             uint64_t bytes) {
+    if (!rocm_tier_valid(tier)) return NULL;
+    return ds4_gpu_tensor_alloc_managed(bytes);
+}
+
+extern "C" void ds4_gpu_tensor_free_in_place(ds4_gpu_tensor *t) {
+    if (!t) return;
+    if (t->owner && t->ptr) (void)hipFree(t->ptr);
+    memset(t, 0, sizeof(*t));
+}
+
+extern "C" int ds4_gpu_tensor_device(const ds4_gpu_tensor *t) {
+    return t ? 0 : -1;
+}
+
+extern "C" int ds4_gpu_tensor_copy_async(ds4_gpu_tensor *dst,
+                                           const ds4_gpu_tensor *src,
+                                           uint64_t bytes) {
+    if (!dst || !src || bytes > dst->bytes || bytes > src->bytes) return 0;
+    if (bytes == 0) return 1;
+    return hipMemcpyAsync(dst->ptr, src->ptr, (size_t)bytes,
+                          hipMemcpyDeviceToDevice, 0) == hipSuccess;
+}
+
+extern "C" int ds4_gpu_tensor_copy_xdev(ds4_gpu_tensor *dst,
+                                          const ds4_gpu_tensor *src,
+                                          uint64_t bytes) {
+    return ds4_gpu_tensor_copy(dst, 0, src, 0, bytes);
+}
+
+extern "C" int ds4_gpu_tensor_copy_xdev_default(ds4_gpu_tensor *dst,
+                                                  const ds4_gpu_tensor *src,
+                                                  uint64_t bytes) {
+    return ds4_gpu_tensor_copy_xdev(dst, src, bytes);
+}
+
+extern "C" int ds4_gpu_tensor_copy_xdev_ordered(ds4_gpu_tensor *dst,
+                                                  const ds4_gpu_tensor *src,
+                                                  uint64_t bytes) {
+    return ds4_gpu_tensor_copy_xdev(dst, src, bytes);
+}
+
+extern "C" int ds4_gpu_tensor_copy_xdev3(
+        ds4_gpu_tensor *dst0, const ds4_gpu_tensor *src0, uint64_t bytes0,
+        ds4_gpu_tensor *dst1, const ds4_gpu_tensor *src1, uint64_t bytes1,
+        ds4_gpu_tensor *dst2, const ds4_gpu_tensor *src2, uint64_t bytes2) {
+    return (bytes0 == 0 || ds4_gpu_tensor_copy_xdev(dst0, src0, bytes0)) &&
+           (bytes1 == 0 || ds4_gpu_tensor_copy_xdev(dst1, src1, bytes1)) &&
+           (bytes2 == 0 || ds4_gpu_tensor_copy_xdev(dst2, src2, bytes2));
+}
+
+extern "C" int ds4_gpu_tensor_copy_xdev3_default_dst(
+        ds4_gpu_tensor *dst0, const ds4_gpu_tensor *src0, uint64_t bytes0,
+        ds4_gpu_tensor *dst1, const ds4_gpu_tensor *src1, uint64_t bytes1,
+        ds4_gpu_tensor *dst2, const ds4_gpu_tensor *src2, uint64_t bytes2) {
+    return ds4_gpu_tensor_copy_xdev3(dst0, src0, bytes0, dst1, src1, bytes1,
+                                     dst2, src2, bytes2);
+}
+
+extern "C" int ds4_gpu_tensor_wait_xdev(const ds4_gpu_tensor *src,
+                                          int dst_tier) {
+    return src && rocm_tier_valid(dst_tier);
+}
+
+extern "C" int ds4_gpu_tensor_wait_xdev_default(const ds4_gpu_tensor *src,
+                                                  int dst_tier) {
+    return ds4_gpu_tensor_wait_xdev(src, dst_tier);
+}
+
+extern "C" uint64_t ds4_gpu_tier_free_vram(int tier) {
+    size_t free_bytes = 0;
+    size_t total_bytes = 0;
+    if (!rocm_tier_valid(tier) ||
+        hipMemGetInfo(&free_bytes, &total_bytes) != hipSuccess) {
+        return 0;
+    }
+    return (uint64_t)free_bytes;
+}
+
+extern "C" int ds4_gpu_args_probe_auto_cuda(
+        const int *device_filter, int filter_len, ds4_gpu_config *out,
+        size_t safety_margin_bytes, char *errbuf, size_t errbuflen) {
+    if (!out) {
+        if (errbuf && errbuflen) snprintf(errbuf, errbuflen, "internal: NULL out");
+        return 1;
+    }
+    int visible = 0;
+    hipError_t rc = hipGetDeviceCount(&visible);
+    if (rc != hipSuccess || visible <= 0) {
+        if (errbuf && errbuflen) {
+            snprintf(errbuf, errbuflen, "hipGetDeviceCount failed: %s",
+                     rc == hipSuccess ? "no devices" : hipGetErrorString(rc));
+        }
+        return 1;
+    }
+    if (filter_len > 1 || (!device_filter && visible > 1)) {
+        if (errbuf && errbuflen) {
+            snprintf(errbuf, errbuflen,
+                     "ROCm supports one GPU per process; select one device");
+        }
+        return 1;
+    }
+    const int device = device_filter && filter_len == 1 ? device_filter[0] : 0;
+    if (device < 0 || device >= visible || hipSetDevice(device) != hipSuccess) {
+        if (errbuf && errbuflen) {
+            snprintf(errbuf, errbuflen, "invalid ROCm device %d", device);
+        }
+        return 1;
+    }
+    size_t free_bytes = 0;
+    size_t total_bytes = 0;
+    rc = hipMemGetInfo(&free_bytes, &total_bytes);
+    if (rc != hipSuccess) {
+        if (errbuf && errbuflen) {
+            snprintf(errbuf, errbuflen, "hipMemGetInfo failed: %s",
+                     hipGetErrorString(rc));
+        }
+        return 1;
+    }
+    const size_t reserve_floor = (size_t)2ull * 1024ull * 1024ull * 1024ull;
+    const size_t reserve_pct = free_bytes / 20u;
+    const size_t reserve = reserve_floor > reserve_pct ? reserve_floor : reserve_pct;
+    memset(out, 0, sizeof(*out));
+    out->device_indices[0] = device;
+    out->vram_bytes[0] = free_bytes > reserve ? free_bytes - reserve : 0;
+    out->n_gpus = 1;
+    out->safety_margin_bytes = safety_margin_bytes;
+    return 0;
+}
+
+extern "C" void ds4_gpu_enable_q8_dequant_gemm(void) {
+}
+
+static int g_rocm_q8_cache_suppressed = 0;
+
+extern "C" int ds4_gpu_q8_cache_suppressed(void) {
+    return g_rocm_q8_cache_suppressed;
+}
+
+extern "C" void ds4_gpu_set_q8_cache_suppressed(int suppressed) {
+    g_rocm_q8_cache_suppressed = suppressed != 0;
+}
+
+extern "C" int ds4_gpu_set_decode_fast_attention(int enabled) {
+    (void)enabled;
+    return 0;
+}
+
+extern "C" int ds4_gpu_set_decode_score_vec4(int enabled) {
+    (void)enabled;
+    return 0;
+}
+
+
+extern "C" int ds4_gpu_matmul_f16_router_rows_exact_tensor(
+        ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
+        uint64_t weight_offset, const ds4_gpu_tensor *x, uint32_t n_rows) {
+    return ds4_gpu_matmul_f16_tensor(out, model_map, model_size, weight_offset,
+                                     4096u, 256u, x, n_rows);
+}
+
+extern "C" int ds4_gpu_dsv4_qkv_rms_norm_rows_kv_rope_tensor(
+        ds4_gpu_tensor *q_out, const ds4_gpu_tensor *q,
+        const void *model_map, uint64_t model_size,
+        uint64_t q_weight_offset, uint32_t q_n,
+        ds4_gpu_tensor *kv_out, const ds4_gpu_tensor *kv,
+        uint64_t kv_weight_offset, uint32_t kv_n, uint32_t rows,
+        uint32_t kv_n_head, uint32_t kv_head_dim, uint32_t n_rot,
+        uint32_t pos0, uint32_t n_ctx_orig, bool inverse,
+        float freq_base, float freq_scale, float ext_factor,
+        float attn_factor, float beta_fast, float beta_slow, float eps) {
+    return ds4_gpu_dsv4_qkv_rms_norm_rows_tensor(
+                   q_out, q, model_map, model_size, q_weight_offset, q_n,
+                   kv_out, kv, kv_weight_offset, kv_n, rows, eps) != 0 &&
+           ds4_gpu_rope_tail_tensor(
+                   kv_out, rows, kv_n_head, kv_head_dim, n_rot, pos0,
+                   n_ctx_orig, inverse, freq_base, freq_scale, ext_factor,
+                   attn_factor, beta_fast, beta_slow) != 0;
+}
+
+extern "C" int ds4_gpu_embed_token_quant_tensor(
+        ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
+        uint64_t weight_offset, uint32_t weight_type, uint32_t n_vocab,
+        uint32_t token, uint32_t n_embd) {
+    if (weight_type != 8u) return 0;
+    return ds4_gpu_embed_token_q8_0_tensor(out, model_map, model_size,
+                                           weight_offset, n_vocab, token,
+                                           n_embd);
+}
+
+extern "C" int ds4_gpu_embed_tokens_quant_tensor(
+        ds4_gpu_tensor *out, const ds4_gpu_tensor *tokens,
+        const void *model_map, uint64_t model_size, uint64_t weight_offset,
+        uint32_t weight_type, uint32_t n_vocab, uint32_t n_tokens,
+        uint32_t n_embd) {
+    if (weight_type != 8u) return 0;
+    return ds4_gpu_embed_tokens_q8_0_tensor(out, tokens, model_map, model_size,
+                                            weight_offset, n_vocab, n_tokens,
+                                            n_embd);
+}
+
+extern "C" int ds4_gpu_matmul_quant_tensor(
+        ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
+        uint64_t weight_offset, uint32_t weight_type, uint64_t in_dim,
+        uint64_t out_dim, const ds4_gpu_tensor *x, uint64_t n_tok) {
+    if (weight_type == 8u) {
+        return ds4_gpu_matmul_q8_0_tensor(out, model_map, model_size,
+                                          weight_offset, in_dim, out_dim, x,
+                                          n_tok);
+    }
+    if (weight_type == 1u) {
+        return ds4_gpu_matmul_f16_tensor(out, model_map, model_size,
+                                         weight_offset, in_dim, out_dim, x,
+                                         n_tok);
+    }
+    return 0;
+}
+
+extern "C" int ds4_gpu_matmul_quant_decode_mpp_model_view_tensor(
+        ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
+        uint64_t weight_offset, uint32_t weight_type, uint64_t in_dim,
+        uint64_t out_dim, const ds4_gpu_tensor *x, uint64_t n_tok) {
+    if (weight_type == 8u) {
+        return ds4_gpu_matmul_q8_0_decode_mpp_model_view_tensor(
+                out, model_map, model_size, weight_offset, in_dim, out_dim,
+                x, n_tok);
+    }
+    return ds4_gpu_matmul_quant_tensor(out, model_map, model_size,
+                                       weight_offset, weight_type, in_dim,
+                                       out_dim, x, n_tok);
+}
+
+extern "C" int ds4_gpu_glm_k_b_project_typed_tensor(
+        ds4_gpu_tensor *out, const ds4_gpu_tensor *kv_norm,
+        const void *model_map, uint64_t model_size, uint64_t weight_offset,
+        uint32_t weight_type, uint32_t n_tokens, uint32_t kv_lora_dim,
+        uint32_t qk_nope, uint32_t n_head) {
+    if (weight_type != 8u) return 0;
+    return ds4_gpu_glm_k_b_project_tensor(out, kv_norm, model_map, model_size,
+                                           weight_offset, n_tokens,
+                                           kv_lora_dim, qk_nope, n_head);
+}
+
+extern "C" int ds4_gpu_glm_qk_lowrank_typed_tensor(
+        ds4_gpu_tensor *qk_low, const ds4_gpu_tensor *q,
+        const void *model_map, uint64_t model_size, uint64_t weight_offset,
+        uint32_t weight_type, uint32_t n_head, uint32_t kv_lora_dim,
+        uint32_t qk_nope, uint32_t qk_dim) {
+    if (weight_type != 8u) return 0;
+    return ds4_gpu_glm_qk_lowrank_q8_0_tensor(
+            qk_low, q, model_map, model_size, weight_offset, n_head,
+            kv_lora_dim, qk_nope, qk_dim);
+}
+
+extern "C" int ds4_gpu_glm_qk_lowrank_typed_batch_tensor(
+        ds4_gpu_tensor *qk_low, const ds4_gpu_tensor *q,
+        const void *model_map, uint64_t model_size, uint64_t weight_offset,
+        uint32_t weight_type, uint32_t n_tokens, uint32_t n_head,
+        uint32_t kv_lora_dim, uint32_t qk_nope, uint32_t qk_dim) {
+    if (weight_type != 8u) return 0;
+    return ds4_gpu_glm_qk_lowrank_q8_0_batch_tensor(
+            qk_low, q, model_map, model_size, weight_offset, n_tokens, n_head,
+            kv_lora_dim, qk_nope, qk_dim);
+}
+
+extern "C" int ds4_gpu_glm_value_project_typed_batch_heads_tensor(
+        ds4_gpu_tensor *heads, const ds4_gpu_tensor *lora,
+        const void *model_map, uint64_t model_size, uint64_t weight_offset,
+        uint32_t weight_type, uint32_t n_tokens, uint32_t n_head,
+        uint32_t kv_lora_dim, uint32_t value_dim) {
+    if (weight_type != 8u) return 0;
+    return ds4_gpu_glm_value_project_q8_0_batch_heads_tensor(
+            heads, lora, model_map, model_size, weight_offset, n_tokens,
+            n_head, kv_lora_dim, value_dim);
+}
+
+extern "C" int ds4_gpu_glm_attention_indexed_decode_typed_tensor(
+        ds4_gpu_tensor *heads, const ds4_gpu_tensor *q,
+        const ds4_gpu_tensor *qk_low, const ds4_gpu_tensor *kv_lora_cache,
+        const ds4_gpu_tensor *k_rope_cache, const void *model_map,
+        uint64_t model_size, uint64_t value_weight_offset,
+        uint32_t value_weight_type, const ds4_gpu_tensor *selected,
+        uint32_t n_selected, uint32_t cache_cap, bool cache_f16,
+        uint32_t n_head, uint32_t kv_lora_dim, uint32_t qk_nope,
+        uint32_t qk_rope, uint32_t value_dim, uint32_t n_ctx_orig,
+        float freq_base, float freq_scale, float ext_factor,
+        float attn_factor, float beta_fast, float beta_slow) {
+    if (value_weight_type != 8u) return 0;
+    return ds4_gpu_glm_attention_indexed_decode_tensor(
+            heads, q, qk_low, kv_lora_cache, k_rope_cache, model_map,
+            model_size, value_weight_offset, selected, n_selected, cache_cap,
+            cache_f16, n_head, kv_lora_dim, qk_nope, qk_rope, value_dim,
+            n_ctx_orig, freq_base, freq_scale, ext_factor, attn_factor,
+            beta_fast, beta_slow);
+}
+
+extern "C" int ds4_gpu_glm_attention_indexed_batch_typed_tensor(
+        ds4_gpu_tensor *heads, const ds4_gpu_tensor *q,
+        const ds4_gpu_tensor *qk_low, const ds4_gpu_tensor *kv_lora_cache,
+        const ds4_gpu_tensor *k_rope_cache, const void *model_map,
+        uint64_t model_size, uint64_t value_weight_offset,
+        uint32_t value_weight_type, const ds4_gpu_tensor *selected,
+        uint32_t n_tokens, uint32_t n_selected, uint32_t cache_cap,
+        bool cache_f16, uint32_t n_head, uint32_t kv_lora_dim,
+        uint32_t qk_nope, uint32_t qk_rope, uint32_t value_dim,
+        uint32_t n_ctx_orig, float freq_base, float freq_scale,
+        float ext_factor, float attn_factor, float beta_fast,
+        float beta_slow) {
+    if (value_weight_type != 8u) return 0;
+    return ds4_gpu_glm_attention_indexed_batch_tensor(
+            heads, q, qk_low, kv_lora_cache, k_rope_cache, model_map,
+            model_size, value_weight_offset, selected, n_tokens, n_selected,
+            cache_cap, cache_f16, n_head, kv_lora_dim, qk_nope, qk_rope,
+            value_dim, n_ctx_orig, freq_base, freq_scale, ext_factor,
+            attn_factor, beta_fast, beta_slow);
+}
+
+extern "C" int ds4_gpu_matmul_q6_K_tensor(
+        ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
+        uint64_t weight_offset, uint64_t in_dim, uint64_t out_dim,
+        const ds4_gpu_tensor *x, uint64_t n_tok) {
+    (void)out; (void)model_map; (void)model_size; (void)weight_offset;
+    (void)in_dim; (void)out_dim; (void)x; (void)n_tok;
+    return 0;
+}
+
+extern "C" int ds4_gpu_laguna_routed_shared_moe_one_tensor(
+        ds4_gpu_tensor *routed_out, ds4_gpu_tensor *routed_mid,
+        ds4_gpu_tensor *shared_out, ds4_gpu_tensor *shared_mid,
+        const void *model_map, uint64_t model_size,
+        const ds4_gpu_laguna_moe_desc *routed,
+        const ds4_gpu_laguna_moe_desc *shared,
+        uint32_t expert_in_dim, uint32_t expert_mid_dim, uint32_t out_dim,
+        const ds4_gpu_tensor *selected, const ds4_gpu_tensor *weights,
+        uint32_t n_total_expert, uint32_t n_expert,
+        const ds4_gpu_tensor *shared_selected,
+        const ds4_gpu_tensor *shared_weight, const ds4_gpu_tensor *x) {
+    (void)routed_out; (void)routed_mid; (void)shared_out; (void)shared_mid;
+    (void)model_map; (void)model_size; (void)routed; (void)shared;
+    (void)expert_in_dim; (void)expert_mid_dim; (void)out_dim;
+    (void)selected; (void)weights; (void)n_total_expert; (void)n_expert;
+    (void)shared_selected; (void)shared_weight; (void)x;
+    return 0;
+}
+
+extern "C" int ds4_gpu_laguna_qkvg_f16_tensor(
+        ds4_gpu_tensor *q, ds4_gpu_tensor *k, ds4_gpu_tensor *v,
+        ds4_gpu_tensor *gate, const void *model_map, uint64_t model_size,
+        uint64_t q_weight_offset, uint64_t k_weight_offset,
+        uint64_t v_weight_offset, uint64_t gate_weight_offset,
+        uint32_t in_dim, uint32_t q_dim, uint32_t kv_dim,
+        uint32_t gate_dim, const ds4_gpu_tensor *x) {
+    (void)q; (void)k; (void)v; (void)gate; (void)model_map; (void)model_size;
+    (void)q_weight_offset; (void)k_weight_offset; (void)v_weight_offset;
+    (void)gate_weight_offset; (void)in_dim; (void)q_dim; (void)kv_dim;
+    (void)gate_dim; (void)x;
+    return 0;
+}
+
+extern "C" int ds4_gpu_laguna_attn_output_residual_f16_tensor(
+        ds4_gpu_tensor *out, const void *model_map, uint64_t model_size,
+        uint64_t weight_offset, uint32_t in_dim, uint32_t out_dim,
+        const ds4_gpu_tensor *x, const ds4_gpu_tensor *residual) {
+    (void)out; (void)model_map; (void)model_size; (void)weight_offset;
+    (void)in_dim; (void)out_dim; (void)x; (void)residual;
+    return 0;
+}

ds4_rocm_unavailable.cu

@@ -0,0 +1,41 @@
+/* Shared graph code references CUDA-only fused and multi-device hooks even
+ * when ROCm validation guarantees that those paths cannot be selected.  Keep
+ * the unavailable surface in one place.  These C-linkage stubs intentionally
+ * accept any argument list because they never inspect their arguments. */
+
+#include <stdint.h>
+
+#define ROCM_UNAVAILABLE_INT(name) extern "C" int name(...) { return 0; }
+
+ROCM_UNAVAILABLE_INT(ds4_gpu_add_xdev_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_attention_decode_rows_rope_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_attention_noncausal_raw_batch_heads_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_attention_output_low_q4_K_slice_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_attention_output_low_q8_rows_exact_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_attention_output_q4_K_batch_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_attention_prefill_raw_heads_range_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_attention_prefill_static_mixed_heads_range_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_device_cache_support_tensors)
+ROCM_UNAVAILABLE_INT(ds4_gpu_device_cache_tensors)
+ROCM_UNAVAILABLE_INT(ds4_gpu_dspark_markov_argmax_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_indexer_top1_value_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_kv_fp8_store_raw_decode_rows_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_matmul_q8_0_kslice_hc_expand_add_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_matmul_q8_0_kslice_rows_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_matmul_q8_0_top1_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_matmul_quant_kslice_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_moe_handoff_pack_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_register_model_map_no_copy)
+ROCM_UNAVAILABLE_INT(ds4_gpu_register_support_map)
+ROCM_UNAVAILABLE_INT(ds4_gpu_rope_tail_decode_rows_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_routed_moe_batch_owned_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_routed_moe_one_owned_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_routed_moe_owned_packed_combine_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_routed_moe_owned_slots_combine_rows_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_routed_moe_owned_slots_combine_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_shared_down_hc_expand_add_q8_0_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_shared_down_hc_expand_owned_q8_0_tensor)
+ROCM_UNAVAILABLE_INT(ds4_gpu_shared_mid_swiglu_q8_0_decode_exact_tensor)
+
+extern "C" uint64_t ds4_gpu_tp_big_gate_kick(...) { return 0; }
+ROCM_UNAVAILABLE_INT(ds4_gpu_tp_big_gate_wait)

ds4_ssd.c

@@ -0,0 +1,210 @@
+#include "ds4_ssd.h"
+
+#include <ctype.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#ifndef MAP_ANONYMOUS
+#define MAP_ANONYMOUS MAP_ANON
+#endif
+
+static const uint64_t DS4_GIB = 1024ull * 1024ull * 1024ull;
+
+bool ds4_parse_gib_arg(const char *s, uint64_t *bytes) {
+    if (bytes) *bytes = 0;
+    if (!s || !s[0] || !bytes) return false;
+
+    size_t len = strlen(s);
+    if (len > 2 &&
+        (s[len - 2] == 'g' || s[len - 2] == 'G') &&
+        (s[len - 1] == 'b' || s[len - 1] == 'B')) {
+        len -= 2;
+    }
+    if (len == 0) return false;
+    for (size_t i = 0; i < len; i++) {
+        if (!isdigit((unsigned char)s[i])) return false;
+    }
+
+    char numbuf[32];
+    if (len >= sizeof(numbuf)) return false;
+    memcpy(numbuf, s, len);
+    numbuf[len] = '\0';
+
+    errno = 0;
+    unsigned long long v = strtoull(numbuf, NULL, 10);
+    if (errno != 0 || v == 0 || v > UINT64_MAX / DS4_GIB) return false;
+
+    *bytes = (uint64_t)v * DS4_GIB;
+    return true;
+}
+
+bool ds4_parse_streaming_cache_experts_arg(const char *s,
+                                           uint32_t   *experts,
+                                           uint64_t   *bytes) {
+    if (experts) *experts = 0;
+    if (bytes) *bytes = 0;
+    if (!s || !s[0] || !experts || !bytes) return false;
+
+    const size_t len = strlen(s);
+    if (len > 2 &&
+        (s[len - 2] == 'g' || s[len - 2] == 'G') &&
+        (s[len - 1] == 'b' || s[len - 1] == 'B')) {
+        return ds4_parse_gib_arg(s, bytes);
+    }
+
+    for (size_t i = 0; i < len; i++) {
+        if (!isdigit((unsigned char)s[i])) return false;
+    }
+
+    errno = 0;
+    unsigned long v = strtoul(s, NULL, 10);
+    if (errno != 0 || v == 0 || v > UINT32_MAX) return false;
+
+    *experts = (uint32_t)v;
+    return true;
+}
+
+uint32_t ds4_ssd_cache_experts_for_byte_budget(uint64_t bytes,
+                                               uint64_t per_expert_bytes) {
+    if (bytes == 0 || per_expert_bytes == 0) return 0;
+    const uint64_t experts = bytes / per_expert_bytes;
+    if (experts == 0 || experts > UINT32_MAX) return 0;
+    return (uint32_t)experts;
+}
+
+static uint64_t ds4_ssd_auto_cache_percent(void) {
+    const char *env = getenv("DS4_SSD_AUTO_CACHE_PCT");
+    if (env && env[0]) {
+        errno = 0;
+        char *end = NULL;
+        unsigned long v = strtoul(env, &end, 10);
+        if (end != env && *end == '\0' && errno == 0 &&
+            v >= 50 && v <= 95) {
+            return (uint64_t)v;
+        }
+        fprintf(stderr,
+                "ds4: invalid DS4_SSD_AUTO_CACHE_PCT=%s (want 50..95); "
+                "using default\n",
+                env);
+    }
+    /*
+     * Decode on the ROCm streaming path is SSD-bandwidth bound: every routed
+     * expert miss is a random NVMe read, so a larger resident expert cache is
+     * the biggest decode-throughput lever.  BUT the expert cache lives in the
+     * same physical RAM as the OS on a unified-memory APU, and transient
+     * spikes (pinned read/upload staging, the prefill headroom, cache growth)
+     * ride on top of the steady-state plan.  Pushing the split too high runs
+     * the machine out of RAM and trips the Linux OOM killer.  80% was measured
+     * safe here; opt into more only deliberately via DS4_SSD_AUTO_CACHE_PCT.
+     */
+    return 80;
+}
+
+bool ds4_ssd_auto_cache_plan(uint64_t            recommended_bytes,
+                             uint64_t            non_routed_bytes,
+                             uint64_t            per_expert_bytes,
+                             uint64_t            max_model_experts,
+                             ds4_ssd_cache_plan *out) {
+    if (!out) return false;
+    memset(out, 0, sizeof(*out));
+    if (recommended_bytes == 0 || per_expert_bytes == 0) return false;
+
+    const uint64_t pct = ds4_ssd_auto_cache_percent();
+    out->model_target_bytes =
+        recommended_bytes > UINT64_MAX / pct ?
+            UINT64_MAX : (recommended_bytes * pct) / 100ull;
+    if (out->model_target_bytes > non_routed_bytes) {
+        out->cache_bytes = out->model_target_bytes - non_routed_bytes;
+    }
+
+    uint64_t cache_experts = out->cache_bytes / per_expert_bytes;
+    if (cache_experts == 0) cache_experts = 1;
+    if (max_model_experts != 0 && cache_experts > max_model_experts) {
+        cache_experts = max_model_experts;
+    }
+    if (cache_experts > UINT32_MAX) cache_experts = UINT32_MAX;
+
+    out->cache_experts = (uint32_t)cache_experts;
+    out->effective_cache_bytes = cache_experts * per_expert_bytes;
+    return out->cache_experts != 0;
+}
+
+bool ds4_ssd_memory_lock_acquire(ds4_ssd_memory_lock *lock,
+                                 uint64_t             bytes) {
+    if (!lock) return false;
+    lock->ptr = NULL;
+    lock->bytes = 0;
+    if (bytes == 0) return true;
+    if (bytes > (uint64_t)SIZE_MAX) {
+        fprintf(stderr,
+                "ds4: --simulate-used-memory is too large for this process\n");
+        return false;
+    }
+
+    void *ptr = mmap(NULL,
+                     (size_t)bytes,
+                     PROT_READ | PROT_WRITE,
+                     MAP_PRIVATE | MAP_ANONYMOUS,
+                     -1,
+                     0);
+    if (ptr == MAP_FAILED) {
+        fprintf(stderr,
+                "ds4: --simulate-used-memory mmap %.2f GiB failed: %s\n",
+                (double)bytes / (double)DS4_GIB,
+                strerror(errno));
+        return false;
+    }
+
+    const long page_long = sysconf(_SC_PAGESIZE);
+    const uint64_t page = page_long > 0 ? (uint64_t)page_long : 4096ull;
+    const uint64_t chunk_bytes = 256ull * 1024ull * 1024ull;
+    volatile unsigned char *p = (volatile unsigned char *)ptr;
+
+    /*
+     * Touch and lock in bounded chunks.  A single very large mlock() is harder
+     * to diagnose when it fails and can create long uninterruptible VM work on
+     * macOS; chunking mirrors the standalone diagnostic utility.
+     */
+    uint64_t locked = 0;
+    for (uint64_t off = 0; off < bytes; off += chunk_bytes) {
+        uint64_t len = bytes - off;
+        if (len > chunk_bytes) len = chunk_bytes;
+
+        for (uint64_t pos = off; pos < off + len; pos += page) {
+            p[pos] = (unsigned char)(pos / page);
+        }
+        if (len != 0) p[off + len - 1u] = 1;
+
+        if (mlock((void *)(p + off), (size_t)len) != 0) {
+            fprintf(stderr,
+                    "ds4: --simulate-used-memory mlock failed after %.2f/%.2f GiB: %s\n",
+                    (double)locked / (double)DS4_GIB,
+                    (double)bytes / (double)DS4_GIB,
+                    strerror(errno));
+            if (locked != 0) munlock(ptr, (size_t)locked);
+            munmap(ptr, (size_t)bytes);
+            return false;
+        }
+        locked += len;
+    }
+
+    lock->ptr = ptr;
+    lock->bytes = bytes;
+    fprintf(stderr,
+            "ds4: simulated used memory: locked %.2f GiB before model load\n",
+            (double)bytes / (double)DS4_GIB);
+    return true;
+}
+
+void ds4_ssd_memory_lock_release(ds4_ssd_memory_lock *lock) {
+    if (!lock || !lock->ptr || lock->bytes == 0) return;
+    munlock(lock->ptr, (size_t)lock->bytes);
+    munmap(lock->ptr, (size_t)lock->bytes);
+    lock->ptr = NULL;
+    lock->bytes = 0;
+}

ds4_ssd.h

@@ -0,0 +1,36 @@
+#ifndef DS4_SSD_H
+#define DS4_SSD_H
+
+#include <stdbool.h>
+#include <stdint.h>
+
+typedef struct {
+    void *ptr;
+    uint64_t bytes;
+} ds4_ssd_memory_lock;
+
+typedef struct {
+    uint64_t model_target_bytes;
+    uint64_t cache_bytes;
+    uint64_t effective_cache_bytes;
+    uint32_t cache_experts;
+} ds4_ssd_cache_plan;
+
+bool ds4_parse_gib_arg(const char *s, uint64_t *bytes);
+bool ds4_parse_streaming_cache_experts_arg(const char *s,
+                                           uint32_t   *experts,
+                                           uint64_t   *bytes);
+
+uint32_t ds4_ssd_cache_experts_for_byte_budget(uint64_t bytes,
+                                               uint64_t per_expert_bytes);
+bool ds4_ssd_auto_cache_plan(uint64_t            recommended_bytes,
+                             uint64_t            non_routed_bytes,
+                             uint64_t            per_expert_bytes,
+                             uint64_t            max_model_experts,
+                             ds4_ssd_cache_plan *out);
+
+bool ds4_ssd_memory_lock_acquire(ds4_ssd_memory_lock *lock,
+                                 uint64_t             bytes);
+void ds4_ssd_memory_lock_release(ds4_ssd_memory_lock *lock);
+
+#endif

ds4_tp.h

@@ -0,0 +1,234 @@
+#ifndef DS4_TP_H
+#define DS4_TP_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+
+#include "ds4.h"
+
+/* Tensor-parallel transport and lockstep protocol.
+ *
+ * Two ranks run the same logical model, each with one contiguous half of the
+ * routed experts resident. Rank 0 (leader) is a normal frontend session that
+ * mirrors every ds4_session_sync()/ds4_session_eval() call to rank 1 (worker)
+ * over a TCP control socket, so both engines execute the identical graph
+ * sequence.
+ * Inside each decoded token, partial block outputs are exchanged through a
+ * registered memory slab: two-sided RDMA SEND/RECV when RDMA over
+ * Thunderbolt is available, or a full-duplex TCP exchange as fallback.
+ *
+ * Layering: ds4.c calls the session-mirroring and slab entry points;
+ * ds4_metal.m only ever sees ds4_tp_gate_exchange() through a callback
+ * registered with the GPU gate machinery.  Nothing here touches tensors.
+ */
+
+typedef struct ds4_tp ds4_tp;
+
+enum {
+    DS4_TP_GATE_ATTN = 0,
+    DS4_TP_GATE_FFN = 1,
+    DS4_TP_GATES_PER_LAYER = 2,
+    /* Max rows in a verify-block batch gate (speculative blocks are <=5). */
+    DS4_TP_BATCH_MAX_ROWS = 8,
+};
+
+/* Engine identity exchanged in the hello so a mismatched pair aborts before
+ * any inference runs. */
+typedef struct {
+    uint64_t gguf_bytes;
+    uint32_t model_id;
+    uint32_t n_layer;
+    uint32_t n_embd;
+    uint32_t n_vocab;
+    uint32_t quant_bits;
+    uint32_t ctx_size;
+    /* Decode gate schedule, used to place RDMA recvs into the right slab
+     * slot: slot(seq) = start + ((seq-1) % per_token) * step.
+     * per_token 0 falls back to the identity mapping over all slots
+     * (DS4: every layer fires ATTN then FFN). GLM fires one FFN gate per
+     * sparse layer only, so its schedule skips the dense prefix and the
+     * ATTN slots. Exchanged in the hello; both sides must agree. */
+    uint32_t gate_slot_start;
+    uint32_t gate_slot_step;
+    uint32_t gates_per_token;
+} ds4_tp_identity;
+
+bool ds4_tp_enabled(const ds4_tp_options *opt);
+
+typedef enum {
+    DS4_TP_CLI_ERROR = -1,
+    DS4_TP_CLI_NOT_MATCHED = 0,
+    DS4_TP_CLI_MATCHED = 1,
+} ds4_tp_cli_parse_result;
+
+/* CLI parsing, same contract as ds4_dist_parse_cli_arg(): returns 1 when the
+ * argument was consumed, 0 when not matched, -1 on error (err filled). */
+int ds4_tp_parse_cli_arg(
+        const char *arg,
+        int *index,
+        int argc,
+        char **argv,
+        ds4_tp_options *opt,
+        char *err,
+        size_t errlen);
+int ds4_tp_adopt_distributed_options(
+        ds4_tp_options *tp,
+        ds4_distributed_options *dist,
+        char *err,
+        size_t errlen);
+void ds4_tp_usage(FILE *fp);
+
+/* Validates option combinations that TP cannot run with (SSD streaming,
+ * distributed mode, MTP drafting, CPU backend). */
+int ds4_tp_validate_engine_options(
+        const ds4_engine_options *opt,
+        char *err,
+        size_t errlen);
+
+/* Connection bring-up.  The leader listens and accepts one worker; the
+ * worker dials with retry.  Both then exchange and validate identities.
+ * Blocking; call after the engine is loaded (identity needs the shape). */
+int ds4_tp_create(
+        ds4_tp **out,
+        const ds4_tp_options *opt,
+        const ds4_tp_identity *id,
+        char *err,
+        size_t errlen);
+void ds4_tp_free(ds4_tp *tp);
+
+int ds4_tp_rank(const ds4_tp *tp);
+bool ds4_tp_is_rdma(const ds4_tp *tp);
+uint32_t ds4_tp_peer_ctx(const ds4_tp *tp);
+bool ds4_tp_failed(const ds4_tp *tp);
+void ds4_tp_mark_failed(ds4_tp *tp);
+
+/* Gate slab.  The engine allocates one shared GPU-visible block and hands
+ * its base VA here; ds4_tp registers it with the NIC (RDMA) and exchanges
+ * remote keys.  Layout, all offsets from base, S = n_layer * 2 slots:
+ *
+ *   out vectors   S * vec_bytes   written by local GPU kernels
+ *   in  vectors   S * vec_bytes   RDMA/TCP-written with the peer partials
+ *   in  seq flags S * 8           written strictly after each in vector
+ *   token slot    16              {seq u64, token i32, pad} leader->worker
+ *   (gpu flags, then batch out/in: n_layer * BATCH_MAX_ROWS * vec_bytes
+ *    each, row partials for the speculative verify-block gates)
+ *
+ * vec_bytes = n_embd * 4 (f32 partials, never quantized on the wire). */
+uint64_t ds4_tp_slab_bytes(uint32_t n_layer, uint32_t n_embd);
+uint64_t ds4_tp_slab_out_offset(const ds4_tp *tp, uint32_t layer, uint32_t gate);
+uint64_t ds4_tp_slab_in_offset(const ds4_tp *tp, uint32_t layer, uint32_t gate);
+uint64_t ds4_tp_slab_batch_out_offset(const ds4_tp *tp, uint32_t layer);
+uint64_t ds4_tp_slab_batch_in_offset(const ds4_tp *tp, uint32_t layer);
+uint64_t ds4_tp_slab_gpu_flags_offset(const ds4_tp *tp);
+int ds4_tp_attach_slab(ds4_tp *tp, void *base, char *err, size_t errlen);
+
+/* Exchange one gate: send out[layer][gate] to the peer's in[layer][gate]
+ * and wait until the peer's partial for `seq` has fully landed locally.
+ * Called from the GPU gate service thread.  Returns 0 on failure. */
+int ds4_tp_gate_exchange(ds4_tp *tp, uint32_t layer, uint32_t gate, uint64_t seq);
+
+/* Verify-block batch gate: exchange `rows` row partials for one layer in one
+ * bulk RDMA transfer, with a symmetric TCP transfer as fallback. Called from
+ * the GPU gate service thread. */
+int ds4_tp_batch_gate_exchange(ds4_tp *tp, uint32_t layer, uint32_t rows,
+                               uint64_t seq);
+
+/* Prefill batch gate: arbitrary-size symmetric payload exchange over bulk
+ * RDMA, with interleaved 2MB TCP rounds as fallback (see ds4_tp.c). */
+int ds4_tp_big_gate_exchange(ds4_tp *tp, uint32_t layer, uint64_t seq,
+                             const void *out, void *in, uint64_t bytes);
+
+/* Lockstep mirroring (leader side) and worker loop primitives. */
+typedef struct {
+    uint64_t session_id;
+    int32_t token;
+    uint32_t reserved;
+} ds4_tp_batch_item;
+
+int ds4_tp_send_session_create(ds4_tp *tp, uint64_t session_id, int ctx_size);
+int ds4_tp_send_session_destroy(ds4_tp *tp, uint64_t session_id);
+int ds4_tp_send_sync(ds4_tp *tp, uint64_t session_id,
+                     const int *tokens, uint32_t n_tokens);
+int ds4_tp_send_eval(ds4_tp *tp, uint64_t session_id,
+                     uint64_t seq, int token);
+int ds4_tp_send_rewind(ds4_tp *tp, uint64_t session_id, int pos);
+int ds4_tp_send_invalidate(ds4_tp *tp, uint64_t session_id);
+int ds4_tp_send_eval_batch(ds4_tp *tp, const ds4_tp_batch_item *items,
+                           uint32_t count);
+int ds4_tp_send_mixed_batch(ds4_tp *tp, uint64_t prefill_session_id,
+                            const int *prompt, uint32_t prompt_count,
+                            const ds4_tp_batch_item *items,
+                            uint32_t count);
+int ds4_tp_send_command_ack(ds4_tp *tp, uint64_t session_id, int status);
+int ds4_tp_wait_command_ack(ds4_tp *tp, uint64_t session_id,
+                            const char *operation, char *err, size_t errlen);
+int ds4_tp_send_stop(ds4_tp *tp);
+
+/* Worker: blocks for the next mirrored command.  Frame types below; for
+ * DS4_TP_FRAME_SYNC the token array is returned in *tokens / *n_tokens
+ * (malloc'd, caller frees), for DS4_TP_FRAME_EVAL seq/token are filled. */
+typedef enum {
+    DS4_TP_FRAME_ERROR = -1,
+    DS4_TP_FRAME_SYNC = 1,
+    DS4_TP_FRAME_EVAL = 2,
+    DS4_TP_FRAME_REWIND = 3,
+    DS4_TP_FRAME_INVALIDATE = 4,
+    DS4_TP_FRAME_STOP = 5,
+    DS4_TP_FRAME_HASH = 6,
+    DS4_TP_FRAME_RDMA_INFO = 7,
+    DS4_TP_FRAME_SYNC_ACK = 8,
+    DS4_TP_FRAME_RDMA_READY = 9,
+    DS4_TP_FRAME_LOGITS = 10,
+    DS4_TP_FRAME_VERIFY = 11,
+    DS4_TP_FRAME_VERIFY_COMMIT = 12,
+    DS4_TP_FRAME_SESSION_CREATE = 13,
+    DS4_TP_FRAME_SESSION_DESTROY = 14,
+    DS4_TP_FRAME_EVAL_BATCH = 15,
+    DS4_TP_FRAME_MIXED_BATCH = 16,
+    DS4_TP_FRAME_COMMAND_ACK = 17,
+} ds4_tp_frame_type;
+
+typedef struct {
+    ds4_tp_frame_type type;
+    uint64_t session_id;
+    uint64_t seq;
+    int value;
+    int *tokens;
+    uint32_t n_tokens;
+    ds4_tp_batch_item *items;
+    uint32_t n_items;
+} ds4_tp_command;
+
+int ds4_tp_recv_command(
+        ds4_tp *tp,
+        ds4_tp_command *command,
+        char *err,
+        size_t errlen);
+void ds4_tp_command_free(ds4_tp_command *command);
+
+/* Debug lockstep check: both sides send their hidden-state hash for a token
+ * and compare.  Returns 0 on transport failure, -1 on hash mismatch. */
+int ds4_tp_hash_check(ds4_tp *tp, uint64_t seq, uint64_t hash, char *err, size_t errlen);
+
+/* Vocab-split output head: the worker ships its logits half to the leader
+ * after every eval (and after a sync) on the control socket. */
+int ds4_tp_send_logits_half(ds4_tp *tp, const float *half, uint32_t count);
+int ds4_tp_recv_logits_half(ds4_tp *tp, float *half, uint32_t count);
+
+/* Speculative verify mirroring.  The leader announces a draft block right
+ * before both ranks run the expert-split batch verify; the worker then blocks
+ * on the commit frame, which carries the leader's decision: full_accept keeps
+ * the pushed rows, otherwise both sides roll back and replay replay_n tokens
+ * through the gated single-token decode in lockstep. */
+int ds4_tp_send_verify(ds4_tp *tp, uint64_t session_id,
+                       const int *drafts, uint32_t n);
+int ds4_tp_send_verify_commit(ds4_tp *tp, int32_t full_accept, int32_t replay_n);
+int ds4_tp_recv_verify_commit(ds4_tp *tp, int32_t *full_accept, int32_t *replay_n);
+
+/* Standalone worker mode entry. Loads nothing itself: the engine is already
+ * open. */
+int ds4_tp_worker_run(ds4_engine *engine, const ds4_tp_options *opt);
+
+#endif

ds4_web.h

@@ -0,0 +1,33 @@
+#ifndef DS4_WEB_H
+#define DS4_WEB_H
+
+#include <stddef.h>
+#include <stdbool.h>
+
+typedef int (*ds4_web_confirm_fn)(void *privdata, const char *message,
+                                  char *err, size_t err_len);
+typedef void (*ds4_web_log_fn)(void *privdata, const char *message);
+typedef bool (*ds4_web_cancel_fn)(void *privdata);
+
+typedef struct {
+    const char *home_dir;
+    int port;
+    ds4_web_confirm_fn confirm;
+    void *confirm_privdata;
+    ds4_web_log_fn log;
+    void *log_privdata;
+    ds4_web_cancel_fn cancel;
+    void *cancel_privdata;
+} ds4_web_config;
+
+typedef struct ds4_web ds4_web;
+
+ds4_web *ds4_web_create(const ds4_web_config *cfg);
+void ds4_web_free(ds4_web *web);
+
+char *ds4_web_google_search(ds4_web *web, const char *query,
+                            char *err, size_t err_len);
+char *ds4_web_visit_page(ds4_web *web, const char *url,
+                         char *err, size_t err_len);
+
+#endif

gguf-tools/.gitignore

@@ -1,3 +1,12 @@
 deepseek4-quantize
+gguf-requantize-dense
 quality-testing/score_official
-quality-testing/data/
+quality-testing/score_llama
+quality-testing/data/*
+!quality-testing/data/README.md
+!quality-testing/data/glm52-openrouter-100/
+!quality-testing/data/glm52-openrouter-100/**
+!quality-testing/data/pro/
+!quality-testing/data/pro/**
+!quality-testing/data/flash/
+!quality-testing/data/flash/**

gguf-tools/Makefile

@@ -1,12 +1,21 @@
 CC ?= cc
+CXX ?= c++
 UNAME_S := $(shell uname -s)
+LLAMA_CPP_DIR ?= ../../deepseek-v4-quants/llama.cpp
+LLAMA_CPP_BUILD ?= $(LLAMA_CPP_DIR)/build-remote
+LLAMA_CPP_BIN ?= $(LLAMA_CPP_BUILD)/bin
+LLAMA_CPP_CXXFLAGS ?= -O3 -Wall -Wextra -std=c++17 \
+	-I$(LLAMA_CPP_DIR)/include -I$(LLAMA_CPP_DIR)/ggml/include
 
 ifeq ($(UNAME_S),Darwin)
 NATIVE_CPU_FLAG ?= -mcpu=native
 QUALITY_LDLIBS := -lm -pthread -framework Foundation -framework Metal
-QUALITY_TARGETS := ds4.o ds4_metal.o
-QUALITY_OBJS := ../ds4.o ../ds4_metal.o
+QUALITY_TARGETS := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o rax.o
+QUALITY_OBJS := ../ds4.o ../ds4_metal.o ../ds4_distributed.o ../ds4_tp.o ../ds4_ssd.o ../ds4_layer_pack.o ../rax.o
 QUALITY_LINK := $(CC) $(CFLAGS) -I.. -o
+LLAMA_CPP_LDLIBS ?= -L$(LLAMA_CPP_BIN) -Wl,-rpath,$(abspath $(LLAMA_CPP_BIN)) \
+	-lllama -lggml -lggml-base -lggml-cpu -lggml-blas -lggml-metal \
+	-framework Foundation -framework Metal -framework Accelerate
 else
 NATIVE_CPU_FLAG ?= -march=native
 CUDA_HOME ?= /usr/local/cuda
@@ -17,15 +26,17 @@ NVCC_ARCH_FLAGS := -arch=$(CUDA_ARCH)
 endif
 NVCCFLAGS ?= -O3 --use_fast_math $(NVCC_ARCH_FLAGS) -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
 QUALITY_LDLIBS ?= -lm -Xcompiler -pthread -L$(CUDA_HOME)/targets/sbsa-linux/lib -L$(CUDA_HOME)/lib64 -lcudart -lcublas
-QUALITY_TARGETS := ds4.o ds4_cuda.o
-QUALITY_OBJS := ../ds4.o ../ds4_cuda.o
+QUALITY_TARGETS := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o rax.o
+QUALITY_OBJS := ../ds4.o ../ds4_cuda.o ../ds4_distributed.o ../ds4_tp.o ../ds4_ssd.o ../ds4_layer_pack.o ../rax.o
 QUALITY_LINK := $(NVCC) $(NVCCFLAGS) -I.. -o
+LLAMA_CPP_LDLIBS ?= -L$(LLAMA_CPP_BIN) -Wl,-rpath,$(abspath $(LLAMA_CPP_BIN)) \
+	-lllama -lggml -lggml-base -lggml-cpu -ldl -lpthread
 endif
 
 CFLAGS ?= -O3 -Wall -Wextra -std=c11 $(NATIVE_CPU_FLAG)
 CPPFLAGS ?= -D_GNU_SOURCE
 
-.PHONY: all clean quality-score
+.PHONY: all clean quality-score quality-llama-score
 
 all: deepseek4-quantize
 
@@ -36,5 +47,8 @@ quality-score:
 	$(MAKE) -C .. $(QUALITY_TARGETS)
 	$(QUALITY_LINK) quality-testing/score_official quality-testing/score_official.c $(QUALITY_OBJS) $(QUALITY_LDLIBS)
 
+quality-llama-score:
+	$(CXX) $(LLAMA_CPP_CXXFLAGS) -o quality-testing/score_llama quality-testing/score_llama.cpp $(LLAMA_CPP_LDLIBS)
+
 clean:
-	rm -f deepseek4-quantize quality-testing/score_official
+	rm -f deepseek4-quantize quality-testing/score_official quality-testing/score_llama

gguf-tools/README.md

@@ -8,7 +8,7 @@ The important pieces are:
 - `deepseek4-quantize.c`: C HF-safetensors to GGUF quantizer.
 - `quants.[ch]`: the deliberately small local quantization implementation used
   by the quantizer.  It implements the DS4 output formats we actually ship:
-  `q8_0`, `q4_K`, `q2_K`, and `iq2_xxs`.
+  `q8_0`, `q8_K`, `q4_K`, `q2_K`, and `iq2_xxs`.
 - `imatrix/`: dataset and instructions for collecting routed-MoE activation
   importance with `ds4`.
 - `quality-testing/`: prompts and scripts used to compare local GGUF variants
@@ -86,6 +86,17 @@ gguf-tools/deepseek4-quantize \
   --imatrix gguf/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4.dat
 ```
 
+True Q8_K routed experts:
+
+```sh
+gguf-tools/deepseek4-quantize \
+  --hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash \
+  --template gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf \
+  --out gguf/DeepSeek-V4-Flash-Q8KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf \
+  --experts q8_K \
+  --threads 8
+```
+
 You can override tensor families:
 
 ```sh
@@ -108,6 +119,54 @@ gguf-tools/deepseek4-quantize \
 `--compare-tensor` regenerates a single tensor and byte-compares it against the
 template or `--compare-gguf`.  `--threads N` controls routed-expert workers.
 
+## Convert A DSpark Support Checkpoint
+
+The DSpark Flash checkpoint is published as Hugging Face safetensors and stores
+the draft module under `mtp.0`, `mtp.1`, and `mtp.2`.  Before writing a support
+GGUF, inspect the official index and verify that every DSpark tensor name is
+understood by the converter:
+
+```sh
+gguf-tools/deepseek4-quantize \
+  --hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash-DSpark \
+  --dspark-manifest > /tmp/dspark-manifest.tsv
+```
+
+The manifest reads only `model.safetensors.index.json`; it does not require the
+large shard files to be present.  The final summary should report three DSpark
+stages and zero unknown DSpark tensors before attempting a full conversion.
+
+To build the support GGUF used by `ds4 --mtp`, run the DSpark support mode.  This
+mode writes standalone DSpark metadata plus the packed `mtp.*` tensor payloads;
+it does not require a base-model GGUF template:
+
+```sh
+gguf-tools/deepseek4-quantize \
+  --hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash-DSpark \
+  --dspark-support \
+  --out DeepSeek-V4-Flash-DSpark-support.gguf
+```
+
+`--dspark-support --dry-run` reads safetensors shard headers to derive exact
+GGUF shapes and types, but it does not read tensor payloads.  The DSpark metadata
+defaults match the published Flash DSpark config: block size 5, target layers
+40,41,42, Markov rank 256, and noise token 128799.  Override them with
+`--dspark-block-size`, `--dspark-target-layers`, `--dspark-markov-rank`, and
+`--dspark-noise-token-id` if converting a different checkpoint.
+
+Before a full write, regenerate one support tensor and record its checksum:
+
+```sh
+gguf-tools/deepseek4-quantize \
+  --hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash-DSpark \
+  --dspark-support \
+  --compare-tensor mtp.0.main_proj.weight
+```
+
+This reads only the payloads needed for that tensor.  Add `--compare-gguf
+DeepSeek-V4-Flash-DSpark-support.gguf` to byte-compare against an existing
+support GGUF.
+
 ## When No Imatrix Is Given
 
 `iq2_xxs` requires an importance vector.  If `--imatrix` is not provided and

gguf-tools/imatrix/README.md

@@ -1,10 +1,12 @@
 # DS4 Imatrix Pipeline
 
 This directory contains the calibration dataset and instructions used to build
-activation importance matrices for DeepSeek V4 Flash GGUF quantization.
+activation importance matrices for DeepSeek V4 Flash and Pro GGUF
+quantization.
 
-The current imatrix target is the routed MoE path.  DS4 has 43 layers, 256
-routed experts per layer, and three routed expert tensors per layer:
+The current imatrix target is the routed MoE path.  Flash has 43 layers and
+256 routed experts per layer.  Pro has 61 layers and 384 routed experts per
+layer.  Both variants expose three routed expert tensors per layer:
 
 - `blk.N.ffn_gate_exps.weight`
 - `blk.N.ffn_up_exps.weight`
@@ -36,17 +38,22 @@ It contains DS4-rendered chat prompts, separated by visible
 - C/Metal source-review prompts from this repository.
 - Long-context snippets.
 - Agent/tool-call prompts using DS4's DSML syntax.
-- English and Italian prompts.
+- Language/prose rewriting, summarization, extraction, and translation prompts.
+- `ds4-eval` GPQA Diamond, SuperGPQA, and AIME2025 benchmark prompts.
 - Both thinking and non-thinking assistant prefixes.
 
-The current tracked dataset has 1952 rendered prompts and roughly 1.44M tokens
+The current tracked dataset has 4682 rendered prompts and roughly 2.91M tokens
 by the coarse bytes/4 estimate.  Check
 `gguf-tools/imatrix/dataset/manifest.json` for the exact generated-file
 summary.
 
 ## 2. Collect The Imatrix
 
-Use the DS4 runtime itself to collect routed MoE activation statistics:
+Use the DS4 runtime itself to collect routed MoE activation statistics.  The
+collector uses the loaded GGUF metadata, so the same command shape works for
+Flash and Pro.
+
+Flash example:
 
 ```sh
 ./ds4 \
@@ -56,6 +63,18 @@ Use the DS4 runtime itself to collect routed MoE activation statistics:
   --ctx 32768
 ```
 
+Pro example with a smaller calibration budget:
+
+```sh
+./ds4 \
+  -m ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct.gguf \
+  --imatrix-dataset gguf-tools/imatrix/dataset/rendered_prompts.txt \
+  --imatrix-out ../deepseek-v4-quants/imatrix/DeepSeek-V4-Pro-Instruct-routed-moe-ds4-small.dat \
+  --imatrix-max-prompts 16 \
+  --imatrix-max-tokens 32768 \
+  --ctx 32768
+```
+
 Useful smoke-test limits:
 
 ```sh
@@ -79,6 +98,8 @@ entry length = n_expert * n_columns
 ```
 
 The quantizer slices the right expert's segment when quantizing each expert.
+For Pro, that means each routed tensor entry contains 384 vectors instead of
+Flash's 256.
 
 ## 3. Generate GGUF Files With The Imatrix
 
@@ -109,6 +130,21 @@ gguf-tools/deepseek4-quantize \
   --imatrix ../deepseek-v4-quants/imatrix/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4-1p5m.dat
 ```
 
+Example Pro Q2 regeneration from a Pro template:
+
+```sh
+gguf-tools/deepseek4-quantize \
+  --hf ../deepseek-v4-quants/hf/DeepSeek-V4-Pro \
+  --template ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct.gguf \
+  --out ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix-small.gguf \
+  --imatrix ../deepseek-v4-quants/imatrix/DeepSeek-V4-Pro-Instruct-routed-moe-ds4-small.dat \
+  --threads 24
+```
+
+`deepseek4-quantize` reads the routed expert count from the template GGUF.  Use
+`--n-experts` only when working with an older template that lacks
+`deepseek4.expert_count`.
+
 For Q4, the imatrix does not change the runtime tensor type: routed experts
 remain `Q4_K`.  It changes how quantization error is weighted while choosing
 scales and codes.  For Q2, it replaces the previous synthetic weight-energy

gguf-tools/imatrix/dataset/README.md

@@ -26,11 +26,12 @@ The renderer mirrors the server prompt shape:
 
 Some records include DSML tool schemas, sampled DSML tool calls, and tool-result
 turns so the imatrix sees the same special-token patterns used by agent clients.
-The corpus is provider-neutral and also includes multi-language programming
-prompts, Bash scripting, algorithm recall, English-to-European-language
-translation, English-to-Chinese translation, reverse translation to English,
-long-context source excerpts, Metal/C code review tasks, and
-inference-specific debugging tasks.
+The corpus is provider-neutral and also includes language/prose rewriting,
+summarization, copy-editing, extraction, multilingual translation, programming
+prompts, Bash scripting, algorithm recall, `ds4-eval` benchmark-reasoning
+prompts, long-context code synthesis, agent transcript replay, log diagnosis,
+prose fact recovery, delayed-constraint and small needle tasks, Metal/C code
+review tasks, and inference-specific debugging tasks.
 
 For normal imatrix collection, use `rendered_prompts.txt` so calibration covers
 both thinking and non-thinking modes. Split files are provided for ablations.

gguf-tools/imatrix/dataset/build_ds4_imatrix_dataset.py

@@ -3,8 +3,8 @@
 
 The quantizer needs activation statistics from prompts that look like the real
 workload.  This script creates deterministic DS4-rendered chat prompts from the
-repo itself, agent/tool conversations, multilingual translation tasks,
-programming prompts, and long-context code reviews.  The output is
+repo itself, agent/tool conversations, language and translation tasks,
+benchmark-reasoning prompts, programming prompts, and long-context code reviews.  The output is
 intentionally plain JSONL/text so the imatrix collector can consume it without
 depending on this script.
 
@@ -18,6 +18,7 @@ provider-neutral rather than tuned around a single client brand.
 from __future__ import annotations
 
 import argparse
+import ast
 import hashlib
 import json
 import random
@@ -99,13 +100,9 @@ class Record:
 
 
 def escape_tool_result(text: str) -> str:
-    """Match ``append_dsml_text_escaped`` for tool result bodies."""
+    """Match ``append_tool_result_text`` for tool result bodies."""
 
-    return (
-        text.replace("&", "&")
-        .replace("<", "&lt;")
-        .replace(">", "&gt;")
-    )
+    return text.replace("</tool_result>", "&lt;/tool_result>")
 
 
 def escape_dsml_parameter(text: str) -> str:
@@ -234,11 +231,22 @@ def stable_id(category: str, source: str, mode: str, text: str) -> str:
     return f"{category}-{h.hexdigest()[:12]}"
 
 
+def normalize_rendered_text(text: str) -> str:
+    """Keep generated calibration files free of accidental trailing spaces.
+
+    Some imported benchmark questions include LaTeX/TikZ lines with trailing
+    blanks.  Those bytes do not carry useful calibration signal, and removing
+    them avoids churny generated files and whitespace-check failures.
+    """
+
+    return "\n".join(line.rstrip() for line in text.split("\n"))
+
+
 def add_record(records: list[Record], category: str, source: str,
                messages: list[dict], *, tools_schema: list[dict] | None = None,
                modes: Iterable[str] = ("nothink", "think")) -> None:
     for mode in modes:
-        rendered = render(messages, mode, tools_schema=tools_schema)
+        rendered = normalize_rendered_text(render(messages, mode, tools_schema=tools_schema))
         rid = stable_id(category, source, mode, rendered)
         records.append(Record(rid, category, mode, source, messages, rendered))
 
@@ -482,22 +490,792 @@ def make_translation_records(records: list[Record]) -> None:
         ])
 
 
+LANGUAGE_PASSAGES = [
+    (
+        "harbor-library",
+        "At seven each morning, Mira unlocked the little harbor library before "
+        "the fish market opened. The room smelled of salt, paper, and wet rope. "
+        "Retired captains came in to read newspapers, apprentices borrowed repair "
+        "manuals, and schoolchildren used the atlas to trace routes they hoped to "
+        "sail one day. When a storm damaged the pier, the library became the only "
+        "quiet place with a working radio, a kettle, and a table large enough for "
+        "the volunteers to spread their maps.",
+    ),
+    (
+        "city-heat",
+        "The city planted two thousand street trees after a summer in which the "
+        "hospital recorded an unusual rise in heat exhaustion. Sensors showed "
+        "that shaded sidewalks were often eight degrees cooler than exposed ones. "
+        "Shop owners first worried that construction would block their windows, "
+        "but many later said people lingered longer outside their stores. The "
+        "largest challenge was not planting the trees; it was watering them "
+        "during their first three dry seasons.",
+    ),
+    (
+        "museum-letter",
+        "The museum received a letter from a visitor who had recognized her "
+        "grandmother in the background of a 1934 photograph. Until then, the "
+        "archive listed the woman only as an unnamed station worker. The visitor "
+        "sent a copy of a diary page, a payroll stub, and a recipe card written "
+        "in the same hand. The curator updated the catalog entry, then wrote a "
+        "short exhibit label about how public memory can repair an incomplete "
+        "institutional record.",
+    ),
+    (
+        "clinic-schedule",
+        "A rural clinic changed its appointment schedule after noticing that "
+        "patients who worked in the fields missed morning visits twice as often "
+        "as afternoon ones. The new calendar reserved early slots for school "
+        "vaccinations and late slots for adults who could not leave work before "
+        "sunset. Missed appointments fell within a month, but the nurses also "
+        "found that reminder calls mattered most when they came from someone "
+        "who spoke the patient's first language.",
+    ),
+    (
+        "mountain-train",
+        "The mountain train was built to carry ore, not tourists. Its narrow "
+        "tracks followed the river, crossed three timber bridges, and climbed "
+        "through tunnels cut by hand. After the mine closed, the town nearly "
+        "removed the line, but teachers argued it was the clearest way to show "
+        "students how geography shaped local industry. Today the train runs "
+        "only on weekends, slowly enough for passengers to see the old signal "
+        "posts and the terraces where workers once kept vegetable gardens.",
+    ),
+    (
+        "software-release",
+        "The release note sounded simple: reduce startup time by warming fewer "
+        "buffers. In practice the change required a careful audit of which pages "
+        "were touched by the first request, which buffers were only needed after "
+        "a tool call, and which allocations could be delayed without moving a "
+        "latency spike into the user's first visible token. The team kept the "
+        "patch small, but the test plan became longer than the code.",
+    ),
+    (
+        "school-garden",
+        "The school garden began as a science project and became a language "
+        "project by accident. Students labeled the herbs in Arabic, Italian, "
+        "Mandarin, Spanish, and English because those were the languages spoken "
+        "at home. Parents started leaving recipes in a notebook by the gate. By "
+        "spring, the teacher used the garden to discuss migration, measurement, "
+        "soil chemistry, and why a word can carry a memory as well as a meaning.",
+    ),
+    (
+        "warehouse-robot",
+        "The warehouse robot failed only on rainy days. Engineers first blamed "
+        "the wheel encoders, then the floor markers, then the routing software. "
+        "A night-shift operator finally noticed that wet umbrellas left small "
+        "puddles near the entrance, and the robot slowed down whenever its front "
+        "sensor saw reflected ceiling lights. The fix was a mat, a hood over the "
+        "sensor, and a dashboard note that credited the operator by name.",
+    ),
+    (
+        "river-council",
+        "At the council meeting, the argument was not whether the river should "
+        "be cleaned, but how to measure success. Anglers wanted fish counts, "
+        "parents wanted safer banks, farmers wanted predictable irrigation, and "
+        "the budget office wanted milestones it could audit. The final plan used "
+        "four measures and a public map updated every quarter, which made the "
+        "project slower to announce but easier to defend.",
+    ),
+    (
+        "bakery-queue",
+        "The bakery changed its queue after the owner realized that customers "
+        "waiting for coffee blocked the bread shelves. A painted line on the "
+        "floor helped for a week, then failed during the Saturday rush. The best "
+        "solution was less decorative: move the sugar, cups, and lids away from "
+        "the register. Average waiting time fell, and the staff stopped having "
+        "to apologize for a problem that had never been caused by impatience.",
+    ),
+    (
+        "radio-weather",
+        "The weather bulletin was read in three languages because the fishing "
+        "boats did not all share the same crew language. The broadcaster learned "
+        "to keep numbers in the same order every time: wind direction, wind "
+        "speed, wave height, visibility, and warnings. Crews said the repeated "
+        "structure helped even when reception was poor, because a missing word "
+        "could often be inferred from its position in the sequence.",
+    ),
+    (
+        "archive-index",
+        "When the archive digitized its index cards, the scanner captured every "
+        "coffee stain and pencil mark. Some volunteers wanted to clean the "
+        "images automatically, but the archivist objected. The stains sometimes "
+        "showed which boxes had been handled during a flood, and the pencil "
+        "marks showed corrections made before the typed catalog existed. The "
+        "project therefore stored both a clean transcription and the original "
+        "image side by side.",
+    ),
+    (
+        "neighborhood-map",
+        "A neighborhood map drawn by residents looked inaccurate to professional "
+        "surveyors because distances were distorted. The bus stop, pharmacy, "
+        "laundromat, and playground appeared much larger than the courthouse or "
+        "bank. The planner kept the distortion because it revealed what official "
+        "maps often hide: the city experienced by walking with a stroller, "
+        "carrying groceries, or looking for shade is not the same city measured "
+        "from above.",
+    ),
+    (
+        "repair-cafe",
+        "At the repair cafe, people brought lamps, radios, torn jackets, and a "
+        "clock that had stopped after forty years. The volunteers did not promise "
+        "success. They promised to explain what they were trying, label the parts "
+        "they removed, and say when replacement would cost more than repair. The "
+        "most popular table was not electronics or sewing, but the one where "
+        "children took apart broken toys to see how gears and springs worked.",
+    ),
+    (
+        "forest-trail",
+        "The forest trail reopened with fewer signs than before. Rangers had "
+        "learned that visitors ignored crowded notice boards but noticed a single "
+        "clear instruction at the exact point where a decision was needed. The "
+        "new signs named the next landmark, the distance, and one safety warning. "
+        "Complaints about being lost dropped, and the trail looked less like a "
+        "corridor of rules.",
+    ),
+    (
+        "market-receipt",
+        "A market cooperative printed receipts that showed the farmer's name, "
+        "the distance traveled, and the date of harvest. Some customers found it "
+        "charming; others found it distracting. Sales data later showed that the "
+        "extra information mattered most for items people could not inspect "
+        "directly, such as flour, oil, and honey. For tomatoes and peaches, smell "
+        "still outweighed every sentence on the receipt.",
+    ),
+    (
+        "night-ferry",
+        "The night ferry crossed the lake even when the road around it was clear. "
+        "Commuters liked the twenty quiet minutes between phone calls and office "
+        "lights, and nurses used the benches to rest after long shifts. When the "
+        "transport agency proposed canceling the route, its spreadsheet counted "
+        "only tickets sold, not the ambulance trips avoided during winter, the "
+        "students who reached evening classes, or the small shops that opened "
+        "early because dock workers bought breakfast there.",
+    ),
+    (
+        "orchard-ledger",
+        "In the orchard ledger, every tree had a row for pruning, pests, harvest "
+        "weight, and weather notes. The oldest entries were written by a farmer "
+        "who never used abbreviations; later entries became shorter as the farm "
+        "grew. A new manager wanted to replace the ledger with a tablet form, but "
+        "kept one blank field labeled 'odd observations' after discovering that "
+        "several good decisions had started as remarks that did not fit any box.",
+    ),
+    (
+        "community-theater",
+        "The community theater sold out after it moved rehearsals into the front "
+        "room where passersby could hear the music. People who would never read a "
+        "poster stopped to ask about costumes, borrowed folding chairs, or offered "
+        "old curtains for the set. The director said the change worked because it "
+        "made preparation visible. The audience felt invited before anyone asked "
+        "them to buy a ticket.",
+    ),
+    (
+        "desert-well",
+        "The desert well was marked on maps as unreliable, but the older guides "
+        "still visited it after winter rains. They knew the water rose slowly "
+        "through cracked stone and disappeared before summer. A research team "
+        "installed a sensor and confirmed the pattern, yet the guides' calendar "
+        "was more useful than the graph because it included wind, animal tracks, "
+        "and stories about years when the rains arrived late.",
+    ),
+    (
+        "newspaper-correction",
+        "The newspaper correction was longer than the original mistake. It named "
+        "the incorrect statistic, explained how the reporter had misread the "
+        "table, printed the correct number, and described how future data stories "
+        "would be checked. Some readers mocked the length, but subscriptions rose "
+        "after the editor began treating corrections as public maintenance rather "
+        "than embarrassment.",
+    ),
+    (
+        "apartment-elevator",
+        "Residents complained that the apartment elevator was slow, but the repair "
+        "company found no mechanical fault. The building manager placed a mirror "
+        "beside the doors, then added a board with local notices and lost keys. "
+        "Complaints dropped because waiting felt shorter when people had something "
+        "useful to read. The elevator had not changed; the experience of waiting "
+        "had.",
+    ),
+    (
+        "field-notebook",
+        "A biologist's field notebook used color more than prose: blue dots for "
+        "water, green lines for moss, orange circles for nests. Her assistant "
+        "wanted to transcribe everything into sentences, but she kept the diagrams "
+        "because they showed proximity at a glance. Years later, the sketches "
+        "helped another team see that the birds nested near temporary pools only "
+        "after a particular moss appeared.",
+    ),
+    (
+        "town-clock",
+        "The town clock gained four minutes every week. For decades no one fixed "
+        "it because train schedules had moved to phones, and the clock remained "
+        "mostly ceremonial. Then the school used it for a lesson about measurement "
+        "error. Students logged the drift, predicted when it would be wrong by an "
+        "hour, and convinced the council that preservation sometimes means making "
+        "an old thing accurate again.",
+    ),
+    (
+        "kitchen-radio",
+        "In the restaurant kitchen, the radio was tuned low enough that customers "
+        "could not hear it and high enough that cooks could recognize the song. "
+        "The owner thought it was a distraction until a busy night when the radio "
+        "failed and the staff lost their rhythm. The music had been acting as a "
+        "shared clock, marking prep, service, cleanup, and the moment everyone "
+        "could finally sit down.",
+    ),
+    (
+        "wetland-boardwalk",
+        "The wetland boardwalk was raised by only twelve centimeters, but that was "
+        "enough to keep spring water from covering the planks. Visitors noticed "
+        "the new railings and benches, not the height. The maintenance crew "
+        "noticed something else: fewer people stepped off the path, so seedlings "
+        "returned near the edges. A small engineering change had become a habitat "
+        "change.",
+    ),
+    (
+        "festival-tickets",
+        "The festival changed ticket prices from three tiers to one suggested "
+        "price with a quiet option to pay less. Revenue stayed almost the same, "
+        "but attendance became more mixed by age and neighborhood. Volunteers "
+        "said the new wording mattered: people did not feel they were asking for "
+        "a discount; they felt they were choosing the price that let them attend "
+        "and still leave space for someone else.",
+    ),
+    (
+        "sewing-manual",
+        "The sewing manual used drawings instead of photographs because thread "
+        "and shadows confused beginners. Each drawing exaggerated the needle, "
+        "the fold, and the direction of tension. Advanced tailors found the pages "
+        "plain, but new learners made fewer mistakes. The designer concluded that "
+        "realism is not always the friend of instruction; sometimes the useful "
+        "picture is the one that lies about scale.",
+    ),
+    (
+        "island-power",
+        "The island power station added batteries before it added more solar "
+        "panels. Tourists expected the opposite, but engineers explained that "
+        "the island already produced enough midday electricity and lost too much "
+        "after sunset. The batteries did not make the photographs prettier, yet "
+        "they reduced diesel use more than another row of panels would have. The "
+        "best improvement was the least visible one.",
+    ),
+    (
+        "language-class",
+        "The language class stopped beginning with grammar charts and started "
+        "with lost-and-found objects: a glove, a ticket, a key, a cracked phone "
+        "case. Students described who might own each item and what had happened "
+        "before it was lost. Grammar still appeared, but as a tool for making "
+        "the story precise. Attendance improved because every lesson began with "
+        "a small mystery.",
+    ),
+    (
+        "bridge-inspection",
+        "The bridge inspection report listed no urgent danger, but it changed the "
+        "maintenance plan anyway. Hairline cracks near the drainage holes showed "
+        "that water was collecting where the original drawings assumed it would "
+        "flow away. The fix was inexpensive: clean the channels twice a year and "
+        "seal one joint. The important discovery was not damage, but the pattern "
+        "that would eventually create it.",
+    ),
+    (
+        "bookshop-window",
+        "The bookshop window once displayed bestsellers in perfect rows. A new "
+        "bookseller replaced them with pairs: a cookbook beside a travel memoir, "
+        "a history beside a novel, a children's atlas beside a book about birds. "
+        "Sales became less predictable but more varied. Customers said the window "
+        "felt like a conversation instead of a ranking.",
+    ),
+]
+
+
+LANGUAGE_TASKS = [
+    ("summary", "Summarize the passage in three concise bullet points."),
+    ("plain", "Rewrite the passage in plain English for a careful twelve-year-old reader."),
+    ("formal", "Rewrite the passage as a formal memo while preserving every factual detail."),
+    ("copyedit", "Copy-edit the passage for clarity. Keep the meaning, but improve flow and sentence rhythm."),
+    ("entities", "Extract people, places, organizations, problems, actions, and outcomes as compact JSON."),
+    ("tone", "Identify the tone, intended audience, and two implied assumptions in the passage."),
+    ("headline", "Write a headline, a one-sentence deck, and five search keywords for the passage."),
+    ("faq", "Turn the passage into a short FAQ with four questions and direct answers."),
+    ("compress", "Compress the passage to at most 90 words without losing the causal chain."),
+    ("compare", "Compare the passage with a similar situation from another field, making the analogy explicit."),
+    ("dialogue", "Turn the passage into a short dialogue between two people who disagree but listen carefully."),
+    ("critique", "Write a constructive critique of the decision or design described in the passage."),
+    ("abstract", "Write an academic-style abstract for the passage in 120 words or fewer."),
+    ("it", "Translate the passage into natural Italian, preserving names and concrete details."),
+    ("es", "Translate the passage into natural Spanish, preserving names and concrete details."),
+    ("zh", "Translate the passage into Simplified Chinese, preserving names and concrete details."),
+]
+
+
+def make_language_records(records: list[Record]) -> None:
+    """Add non-code prose tasks.
+
+    The passages above are original public-domain-compatible calibration text
+    written for this repository.  They cover summarization, rewriting, editing,
+    information extraction, and multilingual prose so the imatrix is not shaped
+    almost entirely by source-code and agent transcripts.
+    """
+
+    system = (
+        "You are a careful language assistant. Preserve meaning, names, numbers, "
+        "and causal relationships. Do not invent facts not supported by the text."
+    )
+    for source, passage in LANGUAGE_PASSAGES:
+        for label, instruction in LANGUAGE_TASKS:
+            add_record(records, "language", f"{source}:{label}", [
+                {"role": "system", "content": system},
+                {"role": "user", "content": f"{instruction}\n\nPassage:\n{passage}"},
+            ])
+
+
+def c_scan_string_end(text: str, pos: int) -> int:
+    """Return the index after a C string literal starting at ``pos``."""
+
+    assert text[pos] == '"'
+    pos += 1
+    while pos < len(text):
+        ch = text[pos]
+        if ch == "\\":
+            pos += 2
+            continue
+        if ch == '"':
+            return pos + 1
+        pos += 1
+    raise ValueError("unterminated C string literal")
+
+
+def c_string_value_at(text: str, pos: int) -> tuple[str, int]:
+    """Parse one or more adjacent C string literals.
+
+    ``ds4_eval.c`` uses ordinary escaped C strings for embedded questions.  We
+    reuse those definitions instead of duplicating 75 benchmark prompts in this
+    generator.  Python's literal parser handles the escaped newlines, quotes,
+    and backslashes used by these string initializers.
+    """
+
+    parts: list[str] = []
+    while True:
+        while pos < len(text) and text[pos].isspace():
+            pos += 1
+        if pos >= len(text) or text[pos] != '"':
+            break
+        end = c_scan_string_end(text, pos)
+        parts.append(ast.literal_eval(text[pos:end]))
+        pos = end
+    if not parts:
+        raise ValueError("expected C string literal")
+    return "".join(parts), pos
+
+
+def eval_case_blocks(eval_source: str) -> list[str]:
+    marker = "static const eval_case eval_cases[]"
+    start = eval_source.find(marker)
+    if start < 0:
+        raise ValueError("missing eval_cases array")
+    pos = eval_source.find("{", start)
+    if pos < 0:
+        raise ValueError("missing eval_cases initializer")
+
+    blocks: list[str] = []
+    depth = 1
+    case_start: int | None = None
+    pos += 1
+    in_string = False
+    in_char = False
+    line_comment = False
+    block_comment = False
+    escape = False
+    while pos < len(eval_source):
+        ch = eval_source[pos]
+        nxt = eval_source[pos + 1] if pos + 1 < len(eval_source) else ""
+
+        if line_comment:
+            if ch == "\n":
+                line_comment = False
+            pos += 1
+            continue
+        if block_comment:
+            if ch == "*" and nxt == "/":
+                block_comment = False
+                pos += 2
+            else:
+                pos += 1
+            continue
+        if in_string:
+            if escape:
+                escape = False
+            elif ch == "\\":
+                escape = True
+            elif ch == '"':
+                in_string = False
+            pos += 1
+            continue
+        if in_char:
+            if escape:
+                escape = False
+            elif ch == "\\":
+                escape = True
+            elif ch == "'":
+                in_char = False
+            pos += 1
+            continue
+
+        if ch == "/" and nxt == "/":
+            line_comment = True
+            pos += 2
+            continue
+        if ch == "/" and nxt == "*":
+            block_comment = True
+            pos += 2
+            continue
+        if ch == '"':
+            in_string = True
+            pos += 1
+            continue
+        if ch == "'":
+            in_char = True
+            pos += 1
+            continue
+        if ch == "{":
+            depth += 1
+            if depth == 2:
+                case_start = pos
+        elif ch == "}":
+            if depth == 2 and case_start is not None:
+                blocks.append(eval_source[case_start:pos + 1])
+                case_start = None
+            depth -= 1
+            if depth == 0:
+                break
+        pos += 1
+
+    return blocks
+
+
+def c_field_string(block: str, field: str) -> str:
+    m = re.search(rf"\.{re.escape(field)}\s*=", block)
+    if not m:
+        return ""
+    value, _ = c_string_value_at(block, m.end())
+    return value
+
+
+def c_choice_strings(block: str) -> list[str]:
+    choices: dict[int, str] = {}
+    for m in re.finditer(r"\.choice\[(\d+)\]\s*=", block):
+        value, _ = c_string_value_at(block, m.end())
+        choices[int(m.group(1))] = value
+    return [choices[i] for i in sorted(choices)]
+
+
+def build_eval_imatrix_prompt(case: dict) -> str:
+    out = [case["question"], ""]
+    choices = case["choices"]
+    if choices:
+        out.append("Choices:")
+        for i, choice in enumerate(choices):
+            out.append(f"{chr(ord('A') + i)}. {choice}")
+        out.append("")
+        out.append(
+            "Solve the question. At the end, write exactly one final line in "
+            "this format and do not write anything after it:"
+        )
+        out.append("Answer: <letter>")
+    else:
+        out.append(
+            "Solve the problem. At the end, write exactly one final line in "
+            "this format and do not write anything after it:"
+        )
+        out.append("Answer: <integer>")
+    return "\n".join(out)
+
+
+def make_eval_reasoning_records(root: Path, records: list[Record]) -> None:
+    """Include the exact benchmark prompt family used by ``ds4-eval``.
+
+    The benchmark cases are already embedded in ``ds4_eval.c`` with source
+    licensing notes.  This parser keeps the imatrix corpus synchronized with the
+    integration harness while excluding answer keys from model-visible text.
+    """
+
+    text = read_text(root / "ds4_eval.c")
+    system = (
+        "You are solving a hard benchmark question. Reason carefully. "
+        "The final answer must follow the requested format exactly."
+    )
+    for idx, block in enumerate(eval_case_blocks(text)):
+        case = {
+            "source": c_field_string(block, "source"),
+            "id": c_field_string(block, "id"),
+            "domain": c_field_string(block, "domain"),
+            "title": c_field_string(block, "title"),
+            "question": c_field_string(block, "question"),
+            "choices": c_choice_strings(block),
+        }
+        prompt = build_eval_imatrix_prompt(case)
+        source = f"{case['source']}:{case['id'] or idx}"
+        title = f"{case['source']} / {case['domain']} / {case['title']}"
+        add_record(records, "eval_reasoning", source, [
+            {"role": "system", "content": system},
+            {"role": "user", "content": f"{title}\n\n{prompt}"},
+        ])
+
+
 def make_long_context_records(root: Path, records: list[Record]) -> None:
+    """Add realistic long-context prompts.
+
+    These records are meant to expose routed FFNs to activations produced after
+    attention has integrated distant evidence.  They are not pure length padding:
+    every prompt asks the model to recover, compare, diagnose, or obey details
+    that occur far apart in the context.
+    """
+
     sources = [
         ("README.md", read_text(root / "README.md")),
         ("AGENT.md", read_text(root / "AGENT.md")),
         ("METAL.md", read_text(root / "METAL.md")),
+        ("gguf-tools/imatrix/README.md", read_text(root / "gguf-tools/imatrix/README.md")),
+        ("gguf-tools/imatrix/dataset/README.md", read_text(root / "gguf-tools/imatrix/dataset/README.md")),
+        ("gguf-tools/quality-testing/README.md", read_text(root / "gguf-tools/quality-testing/README.md")),
         ("ds4_server.c", read_text(root / "ds4_server.c")),
         ("ds4_metal.m", read_text(root / "ds4_metal.m")),
         ("metal/dsv4_hc.metal", read_text(root / "metal/dsv4_hc.metal")),
         ("metal/moe.metal", read_text(root / "metal/moe.metal")),
+        ("metal/dense.metal", read_text(root / "metal/dense.metal")),
     ]
-    blocks = []
+    by_name: dict[str, list[str]] = {}
     for name, text in sources:
-        for i, chunk in enumerate(chunk_text(text, size=5200, overlap=120)[:6]):
-            blocks.append(f"### {name} chunk {i}\n{chunk}")
+        chunks = chunk_text(text, size=3600, overlap=120)[:8]
+        if chunks:
+            by_name[name] = [f"### {name} chunk {i}\n{chunk}" for i, chunk in enumerate(chunks)]
+
+    def block(name: str, idx: int) -> str:
+        chunks = by_name.get(name) or []
+        if not chunks:
+            return f"### {name}\n(unavailable)"
+        return chunks[idx % len(chunks)]
+
+    def add_long(label: str, instruction: str, parts: list[str]) -> None:
+        body = "\n\n".join(p for p in parts if p)
+        add_record(records, "long_context", label, [
+            {"role": "system", "content": DEFAULT_SYSTEM},
+            {"role": "user", "content": instruction + "\n\n" + body},
+        ])
+
+    # Codebase synthesis: several distant implementation excerpts that require
+    # understanding cross-file contracts rather than a single local snippet.
+    add_long("codebase:sampling-cache", (
+        "Read these repository excerpts as one long context. Explain how prompt "
+        "rendering, KV disk-cache lookup, and tool replay interact. Identify two "
+        "ways a future patch could accidentally cause a useless re-prefill."
+    ), [
+        block("ds4_server.c", 1),
+        block("ds4_server.c", 4),
+        block("ds4.c", 2),
+        block("README.md", 1),
+    ])
+    add_long("codebase:metal-moe", (
+        "Read these long code excerpts and summarize the dataflow from routed "
+        "expert selection through MoE execution. Name the tensors whose value "
+        "distribution matters for imatrix collection."
+    ), [
+        block("ds4.c", 5),
+        block("ds4_metal.m", 3),
+        block("metal/moe.metal", 0),
+        block("metal/dsv4_hc.metal", 1),
+    ])
+    add_long("codebase:server-protocols", (
+        "Compare the protocol and server excerpts. What state must survive across "
+        "tool calls, process restarts, and client replay? Return a concise list of "
+        "invariants and the first log line you would add when debugging."
+    ), [
+        block("README.md", 0),
+        block("ds4_server.c", 0),
+        block("ds4_server.c", 3),
+        block("AGENT.md", 0),
+    ])
+    add_long("codebase:eval-imatrix", (
+        "Use the following excerpts to explain how the evaluation harness and the "
+        "imatrix dataset differ in purpose. Mention which parts affect model "
+        "quality measurement and which parts affect quantization calibration."
+    ), [
+        block("gguf-tools/imatrix/README.md", 0),
+        block("gguf-tools/imatrix/dataset/README.md", 0),
+        block("ds4.c", 6),
+        block("README.md", 2),
+    ])
+
+    # Agent transcript replay: the model must connect early user intent, later
+    # tool outputs, and repeated status/error text before deciding the next step.
+    transcript_a = "\n".join([
+        "USER: The server resumed an old coding session and got slower after a tool call.",
+        "ASSISTANT TOOL_CALL bash: rg -n \"canonicalization|responses replay|kv cache hit\" ds4_server.c",
+        "TOOL_RESULT:",
+        TOOL_RESULTS["grep-hit"],
+        "ASSISTANT: The first grep hit points to the renderer; I need logs next.",
+        "USER: Here is the trace excerpt. Do not change protocol semantics, only avoid useless rebuilds.",
+        "TRACE:",
+        "\n".join(f"0514 00:{30+i:02d}:14 ds4-server: chat ctx={12288+i*257}..{13000+i*257}:712 RESPPROTO TOOLS prefill chunk {i*128}/712 ({i*18}.0%) chunk=151.{i} t/s avg=149.{i} t/s" for i in range(6)),
+        "0514 00:36:37 ds4-server: tool checkpoint canonicalized ctx=13257..16846:3589 common=16846 live=16994 canonical=16937 via=rebuild",
+        "USER: What minimal next patch would you inspect first?",
+    ])
+    add_long("agent-transcript:responses-rebuild", (
+        "Read the long agent transcript and trace. Diagnose the likely source of "
+        "the wasted work and propose the next file/function to inspect first."
+    ), [transcript_a, block("ds4_server.c", 2), block("ds4_server.c", 5)])
+
+    transcript_b = "\n".join([
+        "USER: Please test CUDA on the remote machine, then check whether Metal has the same bug.",
+        "ASSISTANT TOOL_CALL bash: ssh 0.180 'cd ds4.c && make clean && make cuda && ./tests/cuda_long_context_smoke'",
+        "TOOL_RESULT:",
+        "nvcc -O3 --use_fast_math -o ds4-server ...\nlong_context_smoke: PASS tokens=81920 prefill=412.5s gen=128\n",
+        "ASSISTANT TOOL_CALL read_file: ds4_cuda.c lines 500..680",
+        "TOOL_RESULT:",
+        "cuda kernel uses kv_row_kind to branch raw/compressed writes; host path checks ctx rows before launch.\n",
+        "USER: Do not assume. Compare with Metal and tell me whether the issue exists there too.",
+        "ASSISTANT TOOL_CALL read_file: ds4_metal.m lines 900..1120",
+        "TOOL_RESULT:",
+        "Metal path stores raw and compressed KV through distinct command encoders and asserts the row range before dispatch.\n",
+        "USER: Summarize the conclusion and what regression test would catch this later.",
+    ])
+    add_long("agent-transcript:cuda-metal", (
+        "Read the long tool transcript. Produce a conclusion that distinguishes "
+        "confirmed facts from inferences, and name the regression test that would "
+        "make the issue findable in the future."
+    ), [transcript_b, block("ds4_metal.m", 2), block("metal/dense.metal", 0)])
+
+    transcript_c = "\n".join([
+        "USER: The benchmark TUI flickers after I press arrows.",
+        "ASSISTANT TOOL_CALL bash: git show 5b92e9b:ds4_eval.c | sed -n '1100,1260p'",
+        "TOOL_RESULT:",
+        "old renderer writes left panel only at safe boundaries and streams sampled bytes only in the right panel.\n",
+        "USER: Return to that model. Add raw mode and a stdin thread, but do not interleave writes.",
+        "ASSISTANT TOOL_CALL bash: ./ds4-eval --questions 2 --trace /tmp/eval.trace",
+        "TOOL_RESULT:",
+        "screen ok; arrows move selection; running case continues without left-panel corruption.\n",
+        "USER: Commit.",
+    ])
+    add_long("agent-transcript:tui-state", (
+        "Given this transcript and the code excerpts, explain the rendering state "
+        "model that avoids flicker and why input handling must be consumed only "
+        "at safe points."
+    ), [transcript_c, block("ds4.c", 0), block("ds4_server.c", 6)])
+
+    # Log/trace diagnosis: repeated operational logs with relevant details
+    # buried among normal progress lines.
+    cache_trace = "\n".join(
+        [f"0515 19:{10+i//6:02d}:{10+i%6*8:02d} ds4-server: chat ctx={20480+i*256}..{22528+i*256}:2048 TOOLS prefill chunk {min(2048, i*96)}/2048 ({min(100, i*4)}.0%) chunk={180-i%7*3}.12 t/s avg={176-i%5*2}.44 t/s" for i in range(28)] +
+        [
+            "0515 19:49:03 ds4-server: kv cache stored tokens=32768 trimmed=0 reason=continued key=token-text size=475.01 MiB save=83.4 ms",
+            "0515 19:49:04 ds4-server: kv cache evicted reason=disk-cache-full tokens=10240 hits=31 size=157.34 MiB file=/tmp/kvcache/old-hot.kv",
+            "0515 19:49:04 ds4-server: kv cache hit text tokens=32768 text=129332 quant=2 key=token-text load=41.7 ms file=/tmp/kvcache/new-frontier.kv",
+        ]
+    )
+    add_long("trace:kv-eviction", (
+        "Analyze this long cache trace. Explain whether eviction favored a useful "
+        "frontier or a stale once-hot file, and state which signal in the log "
+        "supports your conclusion."
+    ), [cache_trace])
+
+    tool_trace = "\n".join(
+        [f"0514 22:{16+i//10:02d}:{10+i%10*5:02d} ds4-server: chat ctx=13695..22840:9145 gen={5000+i*25} TOOLS DSML_START decoding chunk=16.{i%9}6 t/s avg=17.{i%5}0 t/s" for i in range(36)] +
+        [
+            "0514 22:17:06 ds4-server: chat ctx=13695..22840:9145 gen=6080 TOOLS DSML_START DSML_END finish=tool_calls",
+            "0514 22:17:06 ds4-server: tool parse recovered loose nested parameter name=write",
+            "0514 22:17:06 ds4-server: tool calls ctx=13695..22840:9145 n=1 names=[write]",
+        ]
+    )
+    add_long("trace:tool-recovery", (
+        "Read the long tool-call trace. Explain why returning a recoverable tool "
+        "result is better than failing the server request, and identify the first "
+        "line showing the model intended a tool call."
+    ), [tool_trace])
+
+    speed_trace = "\n".join(
+        [f"ctx_tokens={2048+i*2048},prefill_tps={232.4-i*2.1:.2f},gen_tps={19.0-i*0.08:.2f},kvcache_bytes={52184460+i*28188672}" for i in range(32)] +
+        ["note: one outlier at ctx=69632 has prefill_tps=124.32 and gen_tps=14.47"]
+    )
+    add_long("trace:speed-curve", (
+        "Given this long benchmark CSV excerpt, describe the trend without "
+        "overfitting to single-row noise. Name one plausible reason prefill and "
+        "generation slopes differ."
+    ), [speed_trace])
+
+    # Prose fact recovery: natural long story, not an artificial isolated
+    # needle, with facts spelled in words and requested later as structured data.
+    people = [
+        ("Alice", "fifty-two", 52, "kept the ferry accounts"),
+        ("Bruno", "seventeen", 17, "repaired the school roof"),
+        ("Caterina", "eighty-one", 81, "organized the archive"),
+        ("Diego", "thirty-four", 34, "mapped the old orchard"),
+        ("Elena", "sixty-eight", 68, "ran the clinic calendar"),
+        ("Farid", "twenty-nine", 29, "maintained the radio tower"),
+        ("Giulia", "ninety-three", 93, "cataloged the harbor tools"),
+        ("Hana", "forty-six", 46, "translated the notices"),
+        ("Ivan", "seventy-five", 75, "surveyed the river path"),
+        ("Marta", "eleven", 11, "tracked the bakery deliveries"),
+    ]
+    story_parts = []
+    for idx, (name, word, _num, job) in enumerate(people):
+        story_parts.append(
+            f"During week {idx + 1}, {name} {job}. The committee did not write "
+            f"{name}'s reference as digits; in the minutes it was spelled out as "
+            f"{word}, because the secretary wanted all assignments to look uniform."
+        )
+        story_parts.append(LANGUAGE_PASSAGES[idx % len(LANGUAGE_PASSAGES)][1])
+    add_long("story:assigned-numbers", (
+        "Read the long community story. Extract only the assignment references "
+        "for named people and return them as Name=number pairs, one per line."
+    ), ["\n\n".join(story_parts)])
+
+    # Document comparison: long policy/spec style context from multiple docs.
+    add_long("document-comparison:cache-eval-imatrix", (
+        "Compare these documents. Identify three tensions between benchmark "
+        "quality, imatrix representativeness, and disk-cache behavior, then "
+        "propose a practical compromise for each."
+    ), [
+        block("README.md", 3),
+        block("gguf-tools/imatrix/README.md", 0),
+        block("gguf-tools/quality-testing/README.md", 0),
+        block("AGENT.md", 1),
+    ])
+
+    # Needle-in-haystack: keep one small slice. The useful signal is the long
+    # context plus exact retrieval, not the artificial format by itself.
+    hay = []
+    for i in range(24):
+        hay.append(LANGUAGE_PASSAGES[(i * 5) % len(LANGUAGE_PASSAGES)][1])
+        if i == 17:
+            hay.append("Inspection note: the emergency calibration phrase is LANTERN-4827-CEDAR.")
+    add_long("needle:calibration-phrase", (
+        "Most of the following context is ordinary prose. Find the emergency "
+        "calibration phrase and quote it exactly, then explain where it appeared."
+    ), ["\n\n".join(hay)])
+
+    # Delayed constraint following: early requirements must survive a large body
+    # of intervening realistic context.
+    early_constraints = (
+        "Initial requirements, written before the long context: answer in exactly "
+        "four bullets; each bullet must start with SAFE:, FAST:, CLEAR:, or TEST:; "
+        "do not mention implementation details unless they appear in the context."
+    )
+    add_long("delayed-constraints:server-plan", (
+        "Follow the initial requirements after reading all context. Produce the "
+        "requested four-bullet plan and preserve the required prefixes."
+    ), [
+        early_constraints,
+        block("ds4_server.c", 1),
+        block("ds4_server.c", 5),
+        block("README.md", 4),
+        block("metal/moe.metal", 1),
+    ])
+
+    # Keep the older shuffled source-excerpt shape as a small background slice.
+    blocks = [chunk for chunks in by_name.values() for chunk in chunks[:4]]
     random.Random(7).shuffle(blocks)
-    for i in range(0, len(blocks), 4):
+    for i in range(0, min(len(blocks), 16), 4):
         group = blocks[i:i + 4]
         if len(group) < 2:
             continue
@@ -1392,7 +2170,7 @@ def write_outputs(outdir: Path, records: list[Record]) -> None:
 
     rendered_bytes = sum(len(r.rendered.encode("utf-8")) for r in records)
     manifest = {
-        "version": 3,
+        "version": 4,
         "purpose": "DeepSeek V4 Flash imatrix calibration prompts",
         "record_count": len(records),
         "rendered_utf8_bytes": rendered_bytes,
@@ -1426,7 +2204,9 @@ def main() -> None:
     make_general_records(records)
     make_programming_records(records)
     make_algorithm_records(records)
+    make_language_records(records)
     make_translation_records(records)
+    make_eval_reasoning_records(root, records)
     make_long_context_records(root, records)
     write_outputs(outdir, records)
 

gguf-tools/imatrix/dataset/manifest.json

@@ -1,29 +1,33 @@
 {
-  "version": 3,
+  "version": 4,
   "purpose": "DeepSeek V4 Flash imatrix calibration prompts",
-  "record_count": 3310,
-  "rendered_utf8_bytes": 10167295,
-  "rough_token_estimate_bytes_div_4": 2541823,
+  "record_count": 4690,
+  "rendered_utf8_bytes": 11670191,
+  "rough_token_estimate_bytes_div_4": 2917547,
   "categories": {
     "agent": 1106,
     "algorithms": 32,
+    "eval_reasoning": 150,
     "general": 40,
-    "long_context": 12,
+    "language": 1024,
+    "long_context": 36,
     "programming": 48,
-    "source": 1892,
+    "source": 2074,
     "translation": 180
   },
   "modes": {
-    "nothink": 1655,
-    "think": 1655
+    "nothink": 2345,
+    "think": 2345
   },
   "bytes_by_category": {
-    "agent": 4432105,
+    "agent": 4428393,
     "algorithms": 9388,
+    "eval_reasoning": 144853,
     "general": 11902,
-    "long_context": 248628,
+    "language": 725152,
+    "long_context": 357114,
     "programming": 14320,
-    "source": 5394802,
+    "source": 5922919,
     "translation": 56150
   },
   "files": {

gguf-tools/mixed/README.md

@@ -0,0 +1,40 @@
+# Mixed GGUF Splicing
+
+This directory contains a small GGUF splicer for experiments where only some
+routed-expert layers are upgraded from one quantization to another.
+
+The tool does not dequantize or requantize weights. It reads two compatible
+GGUF files, uses the first one as the base, and copies selected routed-expert
+tensors from the second one. All other tensors remain byte-for-byte identical to
+the base file.
+
+## Last Six Layers Q4 Experiment
+
+The best mixed Flash experiment so far keeps the fixed-imatrix 2-bit GGUF as the
+base, then replaces only the routed experts in layers `37-42` with the same
+tensors from the fixed-imatrix Q4 GGUF.
+
+This produces a file around 91 GB instead of the full Q4 file around 153 GB. In
+the local output-agreement checks, this last-six-layers file was statistically
+closer to the full Q4 GGUF than the plain 2-bit GGUF. This is not the same as a
+benchmark score, but it is a useful signal that the mixed file behaves more like
+the stronger Q4 model while keeping most of the 2-bit size.
+
+## Usage
+
+```bash
+python3 gguf-tools/mixed/splice_mixed_expert_layers_gguf.py \
+  --base /path/to/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf \
+  --donor /path/to/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-fixed.gguf \
+  --q4-layers 37-42 \
+  --out /path/to/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf
+```
+
+Use `--dry-run` first to check the selected tensors and expected size delta.
+Use `--force` only when you really want to overwrite the output file.
+
+Layer ranges can be changed without editing the script:
+
+```bash
+--q4-layers 0-2,40-42
+```

gguf-tools/mixed/splice_mixed_expert_layers_gguf.py

@@ -0,0 +1,404 @@
+#!/usr/bin/env python3
+"""Build a mixed DeepSeek V4 Flash GGUF by splicing routed-expert layers.
+
+The base GGUF supplies metadata and all tensors by default.  For selected layer
+IDs, this copies the routed expert tensors from a donor GGUF, rewriting the GGUF
+tensor directory and streaming tensor payloads without dequantizing or
+requantizing.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import re
+import struct
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+from typing import BinaryIO
+
+
+GGUF_HEADER_SIZE = 24
+GGUF_DEFAULT_ALIGNMENT = 32
+
+GGUF_VALUE_UINT8 = 0
+GGUF_VALUE_INT8 = 1
+GGUF_VALUE_UINT16 = 2
+GGUF_VALUE_INT16 = 3
+GGUF_VALUE_UINT32 = 4
+GGUF_VALUE_INT32 = 5
+GGUF_VALUE_FLOAT32 = 6
+GGUF_VALUE_BOOL = 7
+GGUF_VALUE_STRING = 8
+GGUF_VALUE_ARRAY = 9
+GGUF_VALUE_UINT64 = 10
+GGUF_VALUE_INT64 = 11
+GGUF_VALUE_FLOAT64 = 12
+
+GGUF_SCALAR_SIZES = {
+    GGUF_VALUE_UINT8: 1,
+    GGUF_VALUE_INT8: 1,
+    GGUF_VALUE_UINT16: 2,
+    GGUF_VALUE_INT16: 2,
+    GGUF_VALUE_UINT32: 4,
+    GGUF_VALUE_INT32: 4,
+    GGUF_VALUE_FLOAT32: 4,
+    GGUF_VALUE_BOOL: 1,
+    GGUF_VALUE_UINT64: 8,
+    GGUF_VALUE_INT64: 8,
+    GGUF_VALUE_FLOAT64: 8,
+}
+
+# GGML quant type -> (block elements, bytes per block, display name).
+# This intentionally includes the formats used by the current DeepSeek V4 Flash
+# GGUFs.  Add entries here if future recipes introduce new tensor types.
+GGML_QUANT_SIZES = {
+    0: (1, 4, "F32"),
+    1: (1, 2, "F16"),
+    8: (32, 34, "Q8_0"),
+    10: (256, 84, "Q2_K"),
+    12: (256, 144, "Q4_K"),
+    16: (256, 66, "IQ2_XXS"),
+    26: (1, 4, "I32"),
+}
+
+EXPERT_TENSOR_RE = re.compile(r"^blk\.(\d+)\.ffn_(gate|up|down)_exps\.weight$")
+
+
+@dataclass(frozen=True)
+class TensorInfo:
+    name: str
+    dims: tuple[int, ...]
+    ggml_type: int
+    rel_offset: int
+    data_offset: int
+    n_bytes: int
+
+
+@dataclass(frozen=True)
+class GGUFInfo:
+    path: Path
+    version: int
+    tensor_count: int
+    kv_count: int
+    kv_blob: bytes
+    alignment: int
+    tensors: list[TensorInfo]
+    tensor_by_name: dict[str, TensorInfo]
+
+
+@dataclass(frozen=True)
+class SplicePlan:
+    name: str
+    source: str
+    tensor: TensorInfo
+    new_rel_offset: int
+
+
+def read_u32(data: bytes, offset: int) -> int:
+    return struct.unpack_from("<I", data, offset)[0]
+
+
+def read_u64(data: bytes, offset: int) -> int:
+    return struct.unpack_from("<Q", data, offset)[0]
+
+
+def pad_to(value: int, alignment: int) -> int:
+    return ((value + alignment - 1) // alignment) * alignment
+
+
+def read_exact(src: BinaryIO, n_bytes: int) -> bytes:
+    data = src.read(n_bytes)
+    if len(data) != n_bytes:
+        raise EOFError("short read while parsing GGUF")
+    return data
+
+
+def read_u32_file(src: BinaryIO) -> int:
+    return struct.unpack("<I", read_exact(src, 4))[0]
+
+
+def read_u64_file(src: BinaryIO) -> int:
+    return struct.unpack("<Q", read_exact(src, 8))[0]
+
+
+def skip_gguf_string_file(src: BinaryIO) -> None:
+    n = read_u64_file(src)
+    src.seek(n, os.SEEK_CUR)
+
+
+def read_gguf_string_file(src: BinaryIO) -> str:
+    n = read_u64_file(src)
+    return read_exact(src, n).decode("utf-8")
+
+
+def skip_value_payload_file(src: BinaryIO, value_type: int) -> None:
+    if value_type == GGUF_VALUE_STRING:
+        skip_gguf_string_file(src)
+        return
+    if value_type == GGUF_VALUE_ARRAY:
+        subtype = read_u32_file(src)
+        count = read_u64_file(src)
+        if subtype == GGUF_VALUE_STRING:
+            for _ in range(count):
+                skip_gguf_string_file(src)
+            return
+        if subtype == GGUF_VALUE_ARRAY:
+            for _ in range(count):
+                skip_value_payload_file(src, subtype)
+            return
+        size = GGUF_SCALAR_SIZES.get(subtype)
+        if size is None:
+            raise ValueError(f"unsupported GGUF array subtype {subtype}")
+        src.seek(count * size, os.SEEK_CUR)
+        return
+    size = GGUF_SCALAR_SIZES.get(value_type)
+    if size is None:
+        raise ValueError(f"unsupported GGUF value type {value_type}")
+    src.seek(size, os.SEEK_CUR)
+
+
+def tensor_nbytes(dims: tuple[int, ...], ggml_type: int) -> int:
+    q = GGML_QUANT_SIZES.get(ggml_type)
+    if q is None:
+        raise ValueError(f"unsupported GGML tensor type {ggml_type}; add it to GGML_QUANT_SIZES")
+    block_elems, block_bytes, _name = q
+    n_elems = 1
+    for dim in dims:
+        n_elems *= dim
+    if n_elems % block_elems != 0:
+        raise ValueError(f"tensor element count {n_elems} is not divisible by block size {block_elems}")
+    return (n_elems // block_elems) * block_bytes
+
+
+def parse_gguf(path: Path) -> GGUFInfo:
+    with path.open("rb") as src:
+        if read_exact(src, 4) != b"GGUF":
+            raise ValueError(f"{path} is not a GGUF file")
+        version = read_u32_file(src)
+        tensor_count = read_u64_file(src)
+        kv_count = read_u64_file(src)
+
+        alignment = GGUF_DEFAULT_ALIGNMENT
+        for _ in range(kv_count):
+            key = read_gguf_string_file(src)
+            value_type = read_u32_file(src)
+            value_start = src.tell()
+            skip_value_payload_file(src, value_type)
+            value_end = src.tell()
+            if key == "general.alignment" and value_type == GGUF_VALUE_UINT32:
+                src.seek(value_start)
+                alignment = read_u32_file(src)
+                src.seek(value_end)
+
+        kv_end = src.tell()
+        src.seek(GGUF_HEADER_SIZE)
+        kv_blob = read_exact(src, kv_end - GGUF_HEADER_SIZE)
+        src.seek(kv_end)
+
+        raw_tensors: list[tuple[str, tuple[int, ...], int, int]] = []
+        for _ in range(tensor_count):
+            name = read_gguf_string_file(src)
+            n_dims = read_u32_file(src)
+            dims = tuple(read_u64_file(src) for _ in range(n_dims))
+            ggml_type = read_u32_file(src)
+            rel_offset = read_u64_file(src)
+            raw_tensors.append((name, dims, ggml_type, rel_offset))
+
+        data_start = pad_to(src.tell(), alignment)
+    tensors: list[TensorInfo] = []
+    for name, dims, ggml_type, rel_offset in raw_tensors:
+        n_bytes = tensor_nbytes(dims, ggml_type)
+        tensors.append(TensorInfo(name, dims, ggml_type, rel_offset, data_start + rel_offset, n_bytes))
+
+    return GGUFInfo(
+        path=path,
+        version=version,
+        tensor_count=tensor_count,
+        kv_count=kv_count,
+        kv_blob=kv_blob,
+        alignment=alignment,
+        tensors=tensors,
+        tensor_by_name={t.name: t for t in tensors},
+    )
+
+
+def parse_layer_set(spec: str) -> set[int]:
+    layers: set[int] = set()
+    for part in spec.split(","):
+        part = part.strip()
+        if not part:
+            continue
+        if "-" in part:
+            lo_s, hi_s = part.split("-", 1)
+            lo = int(lo_s)
+            hi = int(hi_s)
+            if hi < lo:
+                raise ValueError(f"bad descending range {part!r}")
+            layers.update(range(lo, hi + 1))
+        else:
+            layers.add(int(part))
+    if not layers:
+        raise ValueError("no layers selected")
+    return layers
+
+
+def should_take_donor(name: str, q4_layers: set[int]) -> bool:
+    match = EXPERT_TENSOR_RE.match(name)
+    return match is not None and int(match.group(1)) in q4_layers
+
+
+def qtype_name(ggml_type: int) -> str:
+    return GGML_QUANT_SIZES.get(ggml_type, (0, 0, f"type_{ggml_type}"))[2]
+
+
+def build_plan(base: GGUFInfo, donor: GGUFInfo, q4_layers: set[int]) -> list[SplicePlan]:
+    if base.version != donor.version:
+        raise ValueError(f"GGUF version mismatch: base={base.version} donor={donor.version}")
+    if base.tensor_count != donor.tensor_count:
+        raise ValueError(f"tensor count mismatch: base={base.tensor_count} donor={donor.tensor_count}")
+    if base.alignment != donor.alignment:
+        raise ValueError(f"alignment mismatch: base={base.alignment} donor={donor.alignment}")
+
+    plan: list[SplicePlan] = []
+    next_rel = 0
+    for base_tensor in base.tensors:
+        donor_tensor = donor.tensor_by_name.get(base_tensor.name)
+        if donor_tensor is None:
+            raise ValueError(f"donor is missing tensor {base_tensor.name}")
+        if base_tensor.dims != donor_tensor.dims:
+            raise ValueError(f"shape mismatch for {base_tensor.name}: {base_tensor.dims} vs {donor_tensor.dims}")
+        use_donor = should_take_donor(base_tensor.name, q4_layers)
+        source_tensor = donor_tensor if use_donor else base_tensor
+        plan.append(SplicePlan(
+            name=base_tensor.name,
+            source="donor" if use_donor else "base",
+            tensor=source_tensor,
+            new_rel_offset=next_rel,
+        ))
+        next_rel += pad_to(source_tensor.n_bytes, base.alignment)
+    return plan
+
+
+def pack_string(value: str) -> bytes:
+    raw = value.encode("utf-8")
+    return struct.pack("<Q", len(raw)) + raw
+
+
+def write_tensor_info(out: BinaryIO, item: SplicePlan) -> None:
+    t = item.tensor
+    out.write(pack_string(item.name))
+    out.write(struct.pack("<I", len(t.dims)))
+    for dim in t.dims:
+        out.write(struct.pack("<Q", dim))
+    out.write(struct.pack("<I", t.ggml_type))
+    out.write(struct.pack("<Q", item.new_rel_offset))
+
+
+def copy_exact(src: BinaryIO, dst: BinaryIO, n_bytes: int, bufsize: int = 64 * 1024 * 1024) -> None:
+    remaining = n_bytes
+    while remaining:
+        chunk = src.read(min(bufsize, remaining))
+        if not chunk:
+            raise EOFError("short read while copying tensor payload")
+        dst.write(chunk)
+        remaining -= len(chunk)
+
+
+def write_padding(out: BinaryIO, n_bytes: int, alignment: int) -> None:
+    pad = pad_to(n_bytes, alignment) - n_bytes
+    if pad:
+        out.write(b"\0" * pad)
+
+
+def write_mixed(base: GGUFInfo, donor: GGUFInfo, plan: list[SplicePlan], out_path: Path, force: bool) -> None:
+    tmp_path = out_path.with_name(out_path.name + ".tmp")
+    if out_path.exists() and not force:
+        raise FileExistsError(f"{out_path} already exists; pass --force to overwrite")
+    if tmp_path.exists():
+        raise FileExistsError(f"temporary file already exists: {tmp_path}")
+
+    total_payload = sum(item.tensor.n_bytes for item in plan)
+    copied = 0
+    next_report = 0
+
+    with base.path.open("rb") as base_file, donor.path.open("rb") as donor_file, tmp_path.open("wb") as out:
+        out.write(b"GGUF")
+        out.write(struct.pack("<I", base.version))
+        out.write(struct.pack("<Q", base.tensor_count))
+        out.write(struct.pack("<Q", base.kv_count))
+        out.write(base.kv_blob)
+        for item in plan:
+            write_tensor_info(out, item)
+        write_padding(out, out.tell(), base.alignment)
+
+        for item in plan:
+            src = donor_file if item.source == "donor" else base_file
+            src.seek(item.tensor.data_offset)
+            copy_exact(src, out, item.tensor.n_bytes)
+            write_padding(out, item.tensor.n_bytes, base.alignment)
+            copied += item.tensor.n_bytes
+            pct = int(copied * 100 / total_payload) if total_payload else 100
+            if pct >= next_report:
+                print(f"copied {copied / (1024 ** 3):.2f} GiB / {total_payload / (1024 ** 3):.2f} GiB ({pct}%)", flush=True)
+                next_report += 5
+
+        out.flush()
+        os.fsync(out.fileno())
+
+    os.replace(tmp_path, out_path)
+
+
+def summarize(base: GGUFInfo, donor: GGUFInfo, plan: list[SplicePlan]) -> None:
+    selected = [item for item in plan if item.source == "donor"]
+    base_bytes = sum(t.n_bytes for t in base.tensors)
+    out_bytes = sum(item.tensor.n_bytes for item in plan)
+    delta = out_bytes - base_bytes
+
+    print(f"base:  {base.path}")
+    print(f"donor: {donor.path}")
+    print(f"selected donor tensors: {len(selected)}")
+    by_type: dict[str, int] = {}
+    for item in selected:
+        by_type[qtype_name(item.tensor.ggml_type)] = by_type.get(qtype_name(item.tensor.ggml_type), 0) + 1
+    print("selected donor types:", ", ".join(f"{name}:{count}" for name, count in sorted(by_type.items())) or "none")
+    print(f"base tensor payload: {base_bytes:,} bytes ({base_bytes / (1024 ** 3):.2f} GiB)")
+    print(f"mixed tensor payload: {out_bytes:,} bytes ({out_bytes / (1024 ** 3):.2f} GiB)")
+    print(f"payload delta: {delta:,} bytes ({delta / (1024 ** 3):.2f} GiB)")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Splice selected DeepSeek V4 Flash routed-expert layers from a donor GGUF.")
+    parser.add_argument("--base", required=True, type=Path, help="base GGUF used for metadata and default tensors")
+    parser.add_argument("--donor", required=True, type=Path, help="donor GGUF used for selected routed expert layers")
+    parser.add_argument("--out", required=True, type=Path, help="output mixed GGUF")
+    parser.add_argument("--q4-layers", required=True, help="comma-separated layer IDs/ranges to take from donor, e.g. 37-42")
+    parser.add_argument("--dry-run", action="store_true", help="print the plan without writing the output")
+    parser.add_argument("--force", action="store_true", help="overwrite --out if it already exists")
+    args = parser.parse_args()
+
+    q4_layers = parse_layer_set(args.q4_layers)
+    print("q4 layers:", ",".join(str(x) for x in sorted(q4_layers)))
+
+    base = parse_gguf(args.base)
+    donor = parse_gguf(args.donor)
+    plan = build_plan(base, donor, q4_layers)
+    summarize(base, donor, plan)
+
+    if args.dry_run:
+        return 0
+
+    write_mixed(base, donor, plan, args.out, args.force)
+    final_size = args.out.stat().st_size
+    print(f"wrote: {args.out}")
+    print(f"file size: {final_size:,} bytes ({final_size / (1024 ** 3):.2f} GiB)")
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        raise SystemExit(main())
+    except Exception as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        raise SystemExit(1)

gguf-tools/quality-testing/README.md

@@ -1,35 +1,118 @@
 # Official-Continuation Quality Testing
 
-This directory contains the 100 prompts and scripts used to compare local GGUF
-variants against official DeepSeek V4 Flash continuations.
+This directory contains the prompts, tracked official fixtures, and scripts used
+to compare local GGUF variants against hosted-model continuations.
 
 The metric is target-token negative log likelihood: collect a deterministic
 official continuation, then ask each local GGUF how much probability it assigns
 to that exact continuation token by token.  This avoids judging quality from one
 sampled answer.
 
-## 1. Collect Official Continuations
+## 1. Tracked Fixture Sets
+
+Curated fixtures are kept in the repository so release QA can run without
+calling hosted APIs:
+
+- `data/glm52-openrouter-100`: 100 GLM 5.2 continuations collected through
+  OpenRouter `z-ai/glm-5.2` with `top_logprobs=20`.
+- `data/laguna-openrouter-100`: 100 Laguna S 2.1 continuations collected
+  through OpenRouter `poolside/laguna-s-2.1`. Poolside's endpoint does not
+  expose output-token logprobs.
+- `data/flash`: 100 DeepSeek V4 Flash continuations collected from the official
+  DeepSeek API with `top_logprobs=20`.
+- `data/pro`: 100 DeepSeek V4 PRO continuations collected from the official
+  DeepSeek API with `top_logprobs=20`.
+
+DeepSeek V4 Flash also has tracked official smoke vectors in
+`tests/test-vectors/`.  Those vectors drive `./ds4_test --logprob-vectors` and
+include short prompts plus long-prompt attention cases.
+
+The hosted APIs expose output-token logprobs and top-logprob alternatives, not
+full vocabulary logits.
+
+## 2. Collect Official Continuations
 
 ```sh
 export DEEPSEEK_API_KEY=...
 python3 gguf-tools/quality-testing/collect_official.py \
   --prompts gguf-tools/quality-testing/prompts.jsonl \
-  --out gguf-tools/quality-testing/data \
+  --out gguf-tools/quality-testing/data/flash \
   --count 100 \
   --max-tokens 24
 ```
 
+For GLM 5.2 through OpenRouter:
+
+```sh
+export OPENROUTER_API_KEY=...
+python3 gguf-tools/quality-testing/collect_official.py \
+  --model z-ai/glm-5.2 \
+  --endpoint https://openrouter.ai/api/v1/chat/completions \
+  --api-key-env OPENROUTER_API_KEY \
+  --prompts gguf-tools/quality-testing/prompts.jsonl \
+  --out gguf-tools/quality-testing/data/glm52-openrouter \
+  --count 100 \
+  --max-tokens 24 \
+  --top-logprobs 20 \
+  --token-limit-field max_tokens \
+  --provider-order parasail/fp8 \
+  --require-parameters \
+  --thinking omit \
+  --reasoning-effort none
+```
+
+For Laguna S 2.1 through OpenRouter:
+
+```sh
+export OPENROUTER_API_KEY=...
+python3 gguf-tools/quality-testing/collect_official.py \
+  --model poolside/laguna-s-2.1 \
+  --endpoint https://openrouter.ai/api/v1/chat/completions \
+  --api-key-env OPENROUTER_API_KEY \
+  --prompts gguf-tools/quality-testing/prompts.jsonl \
+  --out gguf-tools/quality-testing/data/laguna-openrouter-100 \
+  --count 100 \
+  --max-tokens 24 \
+  --top-logprobs 0 \
+  --token-limit-field max_tokens \
+  --thinking omit \
+  --reasoning-effort none
+```
+
+The Laguna fixture supports local target-token NLL, first-token agreement, and
+greedy-prefix comparison. API logprob-delta and top-N agreement fields remain
+zero because the Poolside endpoint does not return logprobs. Its raw response
+files are retained for provenance but omitted from `manifest.tsv`, avoiding
+attempts to parse unavailable API logprobs during local scoring.
+
+Use one output directory per official model.  The default model is Flash, so
+`data/flash` is the recommended path for Flash continuations.  For PRO:
+
+```sh
+python3 gguf-tools/quality-testing/collect_official.py \
+  --model deepseek-v4-pro \
+  --prompts gguf-tools/quality-testing/prompts.jsonl \
+  --out gguf-tools/quality-testing/data/pro \
+  --count 100 \
+  --max-tokens 24 \
+  --top-logprobs 20
+```
+
 The script writes:
 
-- `data/prompts/case_*.txt`
-- `data/continuations/case_*.txt`
-- `data/responses/case_*.json`
-- `data/manifest.tsv`
+- `data/<model>/prompts/case_*.txt`
+- `data/<model>/continuations/case_*.txt`
+- `data/<model>/responses/case_*.json`
+- `data/<model>/manifest.tsv`
 
-The prompt list is tracked in `prompts.jsonl`; the official responses are not
-tracked because they are derived from an external API.
+The response path is the optional fourth manifest column and is included only
+when `--top-logprobs` is greater than zero.
 
-## 2. Build The Local Scorer
+The prompt list is tracked in `prompts.jsonl`.  Curated fixture directories are
+also tracked after review; ad-hoc API collection directories should stay
+untracked until they are intentionally promoted into the release QA set.
+
+## 3. Build The Local Scorer
 
 ```sh
 make -C gguf-tools quality-score
@@ -37,23 +120,42 @@ make -C gguf-tools quality-score
 
 The scorer links against the DS4 runtime and uses Metal by default.
 
-## 3. Score GGUF Variants
+## 4. Score GGUF Variants
 
 ```sh
 gguf-tools/quality-testing/score_official \
   ../deepseek-v4-quants/gguf/OLD.gguf \
-  gguf-tools/quality-testing/data/manifest.tsv \
+  gguf-tools/quality-testing/data/pro/manifest.tsv \
   /tmp/old.tsv \
   4096
 
 gguf-tools/quality-testing/score_official \
   ../deepseek-v4-quants/gguf/NEW.gguf \
-  gguf-tools/quality-testing/data/manifest.tsv \
+  gguf-tools/quality-testing/data/pro/manifest.tsv \
   /tmp/new.tsv \
   4096
 ```
 
-## 4. Compare
+Use `data/flash/manifest.tsv` for Flash GGUFs,
+`data/glm52-openrouter-100/manifest.tsv` for GLM 5.2 GGUFs, and
+`data/laguna-openrouter-100/manifest.tsv` for Laguna S 2.1 GGUFs, and
+`data/pro/manifest.tsv` for PRO GGUFs. The scorer and comparator do not care
+which model produced the manifest; the manifest path selects the continuation
+set.
+
+For a full-residency vs SSD-streaming comparison, score the same model twice and
+add the streaming flags to one run:
+
+```sh
+gguf-tools/quality-testing/score_official \
+  /path/to/model.gguf \
+  gguf-tools/quality-testing/data/glm52-openrouter-100/manifest.tsv \
+  /tmp/streaming.tsv \
+  4096 \
+  --ssd-streaming
+```
+
+## 5. Compare
 
 ```sh
 python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv
@@ -69,3 +171,13 @@ Output fields:
   official first token.
 - `avg_greedy_lcp`: average greedy longest common prefix against the official
   continuation.
+- `api_target_mae`: when the manifest includes `response_file`, absolute
+  local-vs-API logprob delta for aligned official output tokens.
+- `api_top_coverage`: fraction of API top-logprob alternatives that map exactly
+  to one local tokenizer token.
+- `api_top1_rate`: how often the API top alternative equals the local greedy
+  token.
+- `api_topn_recall`: fraction of mapped API top-N alternatives found in the
+  local top-N for the same position.
+- `api_top_mae`: local-vs-API logprob MAE over mapped API top alternatives.
+- `api_pair_rate`: pairwise ordering agreement among mapped API alternatives.

gguf-tools/quality-testing/collect_official.py

@@ -1,11 +1,12 @@
 #!/usr/bin/env python3
-"""Collect official DeepSeek V4 Flash continuations for local quant scoring."""
+"""Collect hosted-model continuations for local quant scoring."""
 
 from __future__ import annotations
 
 import argparse
 import json
 import os
+import sys
 import time
 import urllib.error
 import urllib.request
@@ -120,19 +121,42 @@ PROMPTS = [
 ]
 
 
-def request_one(api_key: str, prompt: str, max_tokens: int) -> dict:
+def request_one(
+    api_key: str,
+    endpoint: str,
+    model: str,
+    prompt: str,
+    max_tokens: int,
+    top_logprobs: int,
+    thinking: str,
+    reasoning_effort: str,
+    token_limit_field: str,
+    provider_order: list[str],
+    provider_allow_fallbacks: bool,
+    provider_require_parameters: bool,
+) -> dict:
     payload = {
-        "model": MODEL,
+        "model": model,
         "messages": [{"role": "user", "content": prompt}],
         "temperature": 0,
-        "max_tokens": max_tokens,
-        "logprobs": True,
-        "top_logprobs": 5,
-        "thinking": {"type": "disabled"},
         "stream": False,
     }
+    if top_logprobs > 0:
+        payload["logprobs"] = True
+        payload["top_logprobs"] = top_logprobs
+    payload[token_limit_field] = max_tokens
+    if thinking != "omit":
+        payload["thinking"] = {"type": thinking}
+    if reasoning_effort != "omit":
+        payload["reasoning"] = {"effort": reasoning_effort}
+    if provider_order or provider_require_parameters:
+        provider = {"require_parameters": provider_require_parameters}
+        if provider_order:
+            provider["order"] = provider_order
+            provider["allow_fallbacks"] = provider_allow_fallbacks
+        payload["provider"] = provider
     req = urllib.request.Request(
-        ENDPOINT,
+        endpoint,
         data=json.dumps(payload).encode("utf-8"),
         headers={
             "Authorization": f"Bearer {api_key}",
@@ -144,11 +168,37 @@ def request_one(api_key: str, prompt: str, max_tokens: int) -> dict:
         return json.loads(fp.read().decode("utf-8"))
 
 
-def fetch_with_retry(api_key: str, prompt: str, max_tokens: int) -> dict:
+def fetch_with_retry(
+    api_key: str,
+    endpoint: str,
+    model: str,
+    prompt: str,
+    max_tokens: int,
+    top_logprobs: int,
+    thinking: str,
+    reasoning_effort: str,
+    token_limit_field: str,
+    provider_order: list[str],
+    provider_allow_fallbacks: bool,
+    provider_require_parameters: bool,
+) -> dict:
     delay = 1.0
     for attempt in range(6):
         try:
-            return request_one(api_key, prompt, max_tokens)
+            return request_one(
+                api_key,
+                endpoint,
+                model,
+                prompt,
+                max_tokens,
+                top_logprobs,
+                thinking,
+                reasoning_effort,
+                token_limit_field,
+                provider_order,
+                provider_allow_fallbacks,
+                provider_require_parameters,
+            )
         except urllib.error.HTTPError as e:
             body = e.read().decode("utf-8", "replace")
             if e.code < 500 and e.code != 429:
@@ -167,13 +217,46 @@ def main() -> int:
     ap = argparse.ArgumentParser()
     ap.add_argument("--out", default="gguf-tools/quality-testing/data")
     ap.add_argument("--prompts", default="gguf-tools/quality-testing/prompts.jsonl")
+    ap.add_argument("--model", default=MODEL)
+    ap.add_argument("--endpoint", default=ENDPOINT)
+    ap.add_argument("--api-key-env", default=None)
     ap.add_argument("--count", type=int, default=100)
     ap.add_argument("--max-tokens", type=int, default=24)
+    ap.add_argument("--top-logprobs", type=int, default=5)
+    ap.add_argument("--thinking", choices=("disabled", "enabled", "omit"), default="disabled")
+    ap.add_argument("--reasoning-effort",
+                    choices=("xhigh", "high", "medium", "low", "minimal", "none", "omit"),
+                    default=None)
+    ap.add_argument("--token-limit-field",
+                    choices=("auto", "max_tokens", "max_completion_tokens"),
+                    default="auto")
+    ap.add_argument("--provider-order",
+                    help="comma-separated OpenRouter provider slugs, for example ambient/fp8")
+    ap.add_argument("--allow-provider-fallbacks", action="store_true")
+    ap.add_argument("--require-parameters", action="store_true",
+                    help="for OpenRouter, route only to endpoints advertising the requested parameters")
     args = ap.parse_args()
+    if args.top_logprobs < 0 or args.top_logprobs > 20:
+        raise SystemExit("--top-logprobs must be between 0 and 20")
 
-    api_key = os.environ.get("DEEPSEEK_API_KEY")
+    openrouter = "openrouter.ai" in args.endpoint
+    api_key_env = args.api_key_env or ("OPENROUTER_API_KEY" if openrouter else "DEEPSEEK_API_KEY")
+    api_key = os.environ.get(api_key_env)
     if not api_key:
-        raise SystemExit("DEEPSEEK_API_KEY is not set")
+        raise SystemExit(f"{api_key_env} is not set")
+    thinking = args.thinking
+    reasoning_effort = args.reasoning_effort
+    if reasoning_effort is None:
+        reasoning_effort = "none" if openrouter else "omit"
+    if openrouter and args.thinking == "disabled":
+        thinking = "omit"
+    token_limit_field = args.token_limit_field
+    if token_limit_field == "auto":
+        token_limit_field = "max_tokens"
+    provider_order = []
+    if args.provider_order:
+        provider_order = [item.strip() for item in args.provider_order.split(",") if item.strip()]
+    provider_require_parameters = args.require_parameters
     prompts = load_prompts(Path(args.prompts))
 
     out = Path(args.out)
@@ -184,14 +267,37 @@ def main() -> int:
     manifest = out / "manifest.tsv"
     rows = []
     total = min(args.count, len(prompts))
+    print(f"model={args.model} endpoint={args.endpoint}", file=sys.stderr)
+    print(f"key_env={api_key_env} token_field={token_limit_field} thinking={thinking} reasoning={reasoning_effort}",
+          file=sys.stderr)
+    if provider_order or provider_require_parameters:
+        print(f"provider_order={','.join(provider_order) or '-'} "
+              f"provider_fallbacks={args.allow_provider_fallbacks} "
+              f"require_parameters={provider_require_parameters}", file=sys.stderr)
     for i, prompt in enumerate(prompts[: args.count]):
         case_id = f"case_{i:03d}"
         print(f"official {i + 1}/{total}: {case_id}", file=sys.stderr, flush=True)
-        response = fetch_with_retry(api_key, prompt, args.max_tokens)
+        response = fetch_with_retry(
+            api_key,
+            args.endpoint,
+            args.model,
+            prompt,
+            args.max_tokens,
+            args.top_logprobs,
+            thinking,
+            reasoning_effort,
+            token_limit_field,
+            provider_order,
+            args.allow_provider_fallbacks,
+            provider_require_parameters,
+        )
         choice = response["choices"][0]
         content = choice.get("message", {}).get("content", "")
         if not content:
             print(f"warning: empty continuation for {case_id}", file=sys.stderr)
+        logprob_items = (choice.get("logprobs") or {}).get("content", []) or []
+        if args.top_logprobs > 0 and not logprob_items:
+            raise RuntimeError(f"{case_id}: response did not include output-token logprobs")
 
         prompt_path = out / "prompts" / f"{case_id}.txt"
         cont_path = out / "continuations" / f"{case_id}.txt"
@@ -199,13 +305,20 @@ def main() -> int:
         prompt_path.write_text(prompt, encoding="utf-8")
         cont_path.write_text(content, encoding="utf-8")
         resp_path.write_text(json.dumps(response, ensure_ascii=False, indent=2), encoding="utf-8")
-        rows.append((case_id, prompt_path, cont_path, resp_path))
+        rows.append((case_id, prompt_path, cont_path,
+                     resp_path if args.top_logprobs > 0 else None))
         time.sleep(0.05)
 
     with manifest.open("w", encoding="utf-8") as fp:
-        fp.write("# id\tprompt_file\tcontinuation_file\tresponse_file\n")
+        fp.write("# id\tprompt_file\tcontinuation_file")
+        if args.top_logprobs > 0:
+            fp.write("\tresponse_file")
+        fp.write("\n")
         for row in rows:
-            fp.write("\t".join([row[0], str(row[1]), str(row[2]), str(row[3])]) + "\n")
+            fields = [row[0], str(row[1]), str(row[2])]
+            if row[3] is not None:
+                fields.append(str(row[3]))
+            fp.write("\t".join(fields) + "\n")
     print(f"wrote {manifest}", file=sys.stderr)
     return 0
 

gguf-tools/quality-testing/compare_scores.py

@@ -8,20 +8,89 @@ import sys
 from pathlib import Path
 
 
-def load(path: Path) -> dict[str, dict[str, float]]:
+def parse_cell(value: str) -> float | int:
+    v = float(value)
+    if v.is_integer():
+        return int(v)
+    return v
+
+
+def load(path: Path) -> dict[str, dict[str, float | int]]:
     with path.open(newline="", encoding="utf-8") as fp:
         rows = {}
         for row in csv.DictReader(fp, delimiter="\t"):
-            rows[row["id"]] = {
+            parsed = {
                 "target_tokens": int(row["target_tokens"]),
                 "nll": float(row["nll"]),
                 "avg_nll": float(row["avg_nll"]),
                 "first_match": int(row["first_match"]),
                 "greedy_lcp": int(row["greedy_lcp"]),
             }
+            for key, value in row.items():
+                if key is None or key == "id" or key in parsed or value is None or value == "":
+                    continue
+                parsed[key] = parse_cell(value)
+            rows[row["id"]] = parsed
         return rows
 
 
+def has_api(rows: dict[str, dict[str, float | int]], ids: list[str]) -> bool:
+    return all(
+        "api_target_tokens" in rows[case_id] and "api_top_items" in rows[case_id]
+        for case_id in ids
+    )
+
+
+def sum_key(rows: dict[str, dict[str, float | int]], ids: list[str], key: str) -> float:
+    return sum(float(rows[case_id].get(key, 0.0)) for case_id in ids)
+
+
+def weighted_avg(
+    rows: dict[str, dict[str, float | int]], ids: list[str], value_key: str, count_key: str
+) -> float:
+    count = sum_key(rows, ids, count_key)
+    if count == 0:
+        return 0.0
+    total = sum(
+        float(rows[case_id].get(value_key, 0.0)) * float(rows[case_id].get(count_key, 0.0))
+        for case_id in ids
+    )
+    return total / count
+
+
+def ratio(num: float, den: float) -> float:
+    return num / den if den else 0.0
+
+
+def print_api_summary(label: str, rows: dict[str, dict[str, float | int]], ids: list[str]) -> None:
+    target_tokens = sum_key(rows, ids, "api_target_tokens")
+    top_items = sum_key(rows, ids, "api_top_items")
+    top_mapped = sum_key(rows, ids, "api_top_mapped")
+    top1_count = sum_key(rows, ids, "api_top1_count")
+    top1_match = sum_key(rows, ids, "api_top1_match")
+    topn_ref = sum_key(rows, ids, "api_topn_ref")
+    topn_hit = sum_key(rows, ids, "api_topn_hit")
+    top_logprob_count = sum_key(rows, ids, "api_top_logprob_count")
+    pair_total = sum_key(rows, ids, "api_pair_total")
+    pair_agree = sum_key(rows, ids, "api_pair_agree")
+
+    print(f"{label}_api_target_tokens\t{int(target_tokens)}")
+    print(f"{label}_api_target_mae\t{weighted_avg(rows, ids, 'api_target_mae', 'api_target_tokens'):.9f}")
+    print(
+        f"{label}_api_target_mean_delta\t"
+        f"{weighted_avg(rows, ids, 'api_target_mean_delta', 'api_target_tokens'):.9f}"
+    )
+    print(f"{label}_api_top_coverage\t{ratio(top_mapped, top_items):.9f}")
+    print(f"{label}_api_top1_rate\t{ratio(top1_match, top1_count):.9f}")
+    print(f"{label}_api_topn_recall\t{ratio(topn_hit, topn_ref):.9f}")
+    print(f"{label}_api_top_mae\t{weighted_avg(rows, ids, 'api_top_mae', 'api_top_logprob_count'):.9f}")
+    print(
+        f"{label}_api_top_mean_delta\t"
+        f"{weighted_avg(rows, ids, 'api_top_mean_delta', 'api_top_logprob_count'):.9f}"
+    )
+    print(f"{label}_api_pair_rate\t{ratio(pair_agree, pair_total):.9f}")
+
+
 def main() -> int:
     if len(sys.argv) != 3:
         print(f"usage: {sys.argv[0]} OLD.tsv NEW.tsv", file=sys.stderr)
@@ -73,6 +142,9 @@ def main() -> int:
     print(f"case_wins_new_old_ties\t{new_case_wins}\t{old_case_wins}\t{ties}")
     print(f"first_token_matches_old_new\t{old_first}\t{new_first}")
     print(f"avg_greedy_lcp_old_new\t{old_lcp / len(ids):.3f}\t{new_lcp / len(ids):.3f}")
+    if has_api(old, ids) and has_api(new, ids):
+        print_api_summary("old", old, ids)
+        print_api_summary("new", new, ids)
 
     print("\nnew best cases:")
     for delta, case_id, t, old_avg, new_avg in sorted(deltas)[:8]:

gguf-tools/quality-testing/data/README.md

@@ -0,0 +1,26 @@
+# Official Quality Fixtures
+
+This directory contains curated hosted-model continuation fixtures that are
+safe to commit and use in release QA.
+
+- `glm52-openrouter-100`: 100 GLM 5.2 OpenRouter continuations with API
+  top-logprob slices.
+- `laguna-openrouter-100`: 100 Laguna S 2.1 OpenRouter continuations. Poolside's
+  endpoint does not expose output-token logprobs, so these support
+  teacher-forced continuation scoring only.
+- `flash`: 100 DeepSeek V4 Flash official continuations with API top-logprob
+  slices.
+- `pro`: 100 DeepSeek V4 PRO official continuations with API top-logprob
+  slices.
+
+Each fixture directory contains:
+
+- `prompts/case_*.txt`: exact user prompts.
+- `continuations/case_*.txt`: deterministic hosted-model continuations.
+- `responses/case_*.json`: raw hosted responses, including logprob slices when
+  the endpoint provides them.
+- `manifest.tsv`: paths consumed by `score_official`; the optional raw-response
+  path is present only for fixtures with API logprobs.
+
+DeepSeek V4 Flash smoke vectors are also tracked in `tests/test-vectors/` and
+are run by `./ds4_test --logprob-vectors`.

gguf-tools/quality-testing/data/flash/continuations/case_000.txt

@@ -0,0 +1 @@
+Here is a detailed explanation of B-tree insertion, covering the standard algorithm, node splits, and the special case of splitting

gguf-tools/quality-testing/data/flash/continuations/case_001.txt

@@ -0,0 +1,5 @@
+Here's a concise design for a TCP echo server that handles slow clients:
+
+## Architecture
+
+**Event-driven, non-block

gguf-tools/quality-testing/data/flash/continuations/case_002.txt

@@ -0,0 +1 @@
+Here's a comprehensive comparison of memory-mapped (mmap) model weights versus copying all weights into private buffers on macOS

gguf-tools/quality-testing/data/flash/continuations/case_003.txt

@@ -0,0 +1 @@
+This is an excellent question because it gets to the heart of why **RMSNorm** (Root Mean Square Normalization)

gguf-tools/quality-testing/data/flash/continuations/case_004.txt

@@ -0,0 +1 @@
+This is an excellent question that gets at a core optimization challenge in modern LLM deployment. Let's break down why **

gguf-tools/quality-testing/data/flash/continuations/case_005.txt

@@ -0,0 +1,5 @@
+Here's pseudocode for a binary heap push (insert) operation:
+
+```
+function push(heap, value):
+    //

gguf-tools/quality-testing/data/flash/continuations/case_006.txt

@@ -0,0 +1 @@
+This is a great question that gets at the core of how modern LLMs generate text. Both **temperature** and **

gguf-tools/quality-testing/data/flash/continuations/case_007.txt

@@ -0,0 +1 @@
+Here is a summary of how speculative decoding preserves exact greedy output (i.e., the same tokens that would be generated by

gguf-tools/quality-testing/data/flash/continuations/case_008.txt

@@ -0,0 +1 @@
+This is an excellent question that gets to the heart of how Large Language Models (LLMs) are served efficiently. The

gguf-tools/quality-testing/data/flash/continuations/case_009.txt

@@ -0,0 +1,5 @@
+Here are three invariants for a special-token table used in a tokenizer:
+
+---
+
+### 1. **Un

gguf-tools/quality-testing/data/flash/continuations/case_010.txt

@@ -0,0 +1 @@
+Ecco una spiegazione dettagliata del funzionamento dell'inserimento in un **B-tree**, con

gguf-tools/quality-testing/data/flash/continuations/case_011.txt

@@ -0,0 +1,3 @@
+Ecco un progetto conciso per un server TCP echo con gestione di client lenti:
+
+## Architettura

gguf-tools/quality-testing/data/flash/continuations/case_012.txt

@@ -0,0 +1 @@
+Ecco un confronto dettagliato tra **mmap** (memory-mapped file) e **copia

gguf-tools/quality-testing/data/flash/continuations/case_013.txt

@@ -0,0 +1 @@
+Certamente. Ecco una derivazione dettagliata del perché **RMSNorm** (Root Mean Square Normalization

gguf-tools/quality-testing/data/flash/continuations/case_014.txt

@@ -0,0 +1 @@
+La cache KV (Key-Value) su disco è particolarmente utile nelle **sessioni agentiche** (d

gguf-tools/quality-testing/data/flash/continuations/case_015.txt

@@ -0,0 +1 @@
+Ecco lo pseudocodice per inserire un elemento in un heap binario (min-heap o max-he

gguf-tools/quality-testing/data/flash/continuations/case_016.txt

@@ -0,0 +1 @@
+Questa è un'ottima domanda che tocca il cuore del controllo della generazione nei modelli linguistici

gguf-tools/quality-testing/data/flash/continuations/case_017.txt

@@ -0,0 +1 @@
+La decodifica speculativa mantiene l'output esatto perché utilizza un **meccanismo di ver

gguf-tools/quality-testing/data/flash/continuations/case_018.txt

@@ -0,0 +1 @@
+Questa è un'ottima domanda che tocca un aspetto fondamentale dell'ottimizzazione dell'inferenza

gguf-tools/quality-testing/data/flash/continuations/case_019.txt

@@ -0,0 +1 @@
+Ecco tre invarianti per una tabella di token speciali (ad esempio in un parser o in un sistema di

gguf-tools/quality-testing/data/flash/continuations/case_020.txt

@@ -0,0 +1,3 @@
+## Binary Search for Insertion Point
+
+Binary search finds the insertion point for a target value in a sorted array by maintaining

gguf-tools/quality-testing/data/flash/continuations/case_021.txt

@@ -0,0 +1,4 @@
+Here's a short C function that clamps an integer to a range:
+
+```c
+int clamp(int value, int

gguf-tools/quality-testing/data/flash/continuations/case_022.txt

@@ -0,0 +1 @@
+This is a fundamental question in concurrent programming. The core difference lies in **what they protect** and **how they coordinate

gguf-tools/quality-testing/data/flash/continuations/case_023.txt

@@ -0,0 +1 @@
+In the context of a network server, **backpressure** is a mechanism for **flow control**. It refers to the

gguf-tools/quality-testing/data/flash/continuations/case_024.txt

@@ -0,0 +1 @@
+Here is a safe and robust format for storing a model checkpoint header, designed for clarity, extensibility, and error prevention

gguf-tools/quality-testing/data/flash/continuations/case_025.txt

@@ -0,0 +1 @@
+A **ring buffer** (also called a circular buffer) is a fixed-size data structure that uses a contiguous block of

gguf-tools/quality-testing/data/flash/continuations/case_026.txt

@@ -0,0 +1,4 @@
+Here is a brief, structured plan for testing long-context prompt chunking.
+
+### Goal
+Validate that chunking a

gguf-tools/quality-testing/data/flash/continuations/case_027.txt

@@ -0,0 +1 @@
+This is an excellent question that gets to the heart of why modern low-bit quantization (like 4-bit and 3

gguf-tools/quality-testing/data/flash/continuations/case_028.txt

@@ -0,0 +1 @@
+This is a great question that gets at a key performance optimization in modern large language model (LLM) inference, particularly

gguf-tools/quality-testing/data/flash/continuations/case_029.txt

@@ -0,0 +1 @@
+This is an excellent question because comparing quantizations is rarely a simple "A is better than B" scenario. The "

gguf-tools/quality-testing/data/flash/continuations/case_030.txt

@@ -0,0 +1,3 @@
+## Ricerca Binaria del Punto di Inserimento
+
+La ricerca binaria del punto di inserimento è una

gguf-tools/quality-testing/data/flash/continuations/case_031.txt

@@ -0,0 +1,4 @@
+Ecco una breve funzione C che limita un intero a un intervallo specificato:
+
+```c
+int clamp

gguf-tools/quality-testing/data/flash/continuations/case_032.txt

@@ -0,0 +1 @@
+Certamente. La differenza fondamentale tra un **mutex** (mutual exclusion) e un **contatore atom

gguf-tools/quality-testing/data/flash/continuations/case_033.txt

@@ -0,0 +1 @@
+In un server di rete, **backpressure** (contro-pressione o contropressione) è un me

gguf-tools/quality-testing/data/flash/continuations/case_034.txt

@@ -0,0 +1 @@
+Ecco un formato sicuro per salvare l'header di un checkpoint modello, progettato per garantire integ

gguf-tools/quality-testing/data/flash/continuations/case_035.txt

@@ -0,0 +1 @@
+Ti spiegherò il funzionamento di un **ring buffer** (o buffer circolare) e le tec

gguf-tools/quality-testing/data/flash/continuations/case_036.txt

@@ -0,0 +1 @@
+Ecco un piano breve e strutturato per testare il **chunking** (frammentazione) di

gguf-tools/quality-testing/data/flash/continuations/case_037.txt

@@ -0,0 +1 @@
+Nella quantizzazione low-bit (a bassa precisione) dei modelli di deep learning, una **matrice

gguf-tools/quality-testing/data/flash/continuations/case_038.txt

@@ -0,0 +1 @@
+Questa è un'ottima domanda che tocca un aspetto cruciale dell'ottimizzazione dei modelli

gguf-tools/quality-testing/data/flash/continuations/case_039.txt

@@ -0,0 +1 @@
+Per confrontare due quantizzazioni (ad esempio, due modelli di compressione di un segnale, due