Document Qwen Metal batching and its quality checks

QA_BEFORE_RELEASES.md

@@ -1960,6 +1960,28 @@ paired comparisons, also with decode graphs disabled. Separately run the
   record different-schedule probability differences separately. Nearly tied
   experts can amplify normal rounding, so a max-logit difference alone does
   not establish a state bug or a quality regression.
+- For Metal native batching, run `tests/test_metal_session_batch` with
+  `DS4_TEST_BATCH_ISOLATION=1` at two, four and eight sessions, including a
+  sparse prefix. Reordered rows and changed companions must leave the target's
+  complete logits identical. Separately score both official manifests with
+  `--session-batch 1`, `4` and `8`; compare paired losses and API agreement.
+  The scalar and batched reductions need not be bit-identical. Investigate
+  greedy mismatches with their logit margins, not by loosening kernel tests.
+  Run `tests/test_qwen4_kernels` under Metal validation and the real-model
+  `tests/test_qwen4_ngram_state` for mixed ordinary/MTP cycles, failed batch
+  reads, exact recovery and the final context slot.
+- Start a four-slot Qwen Metal server with and without `--mtp`. Check concurrent
+  tool calls, prefix reuse, cancellation, stop strings and one/two-token output
+  limits. Repeat with `--mtp-exact-sampling`, mixing temperature-zero and
+  sampled requests. Only the former may use greedy speculative acceptance.
+  Compare seeded sampled replies in exact mode; default greedy MTP changes
+  random draws with draft scheduling, so equal seeds need not give equal text.
+  Benchmark ordinary and speculative batches on both prose and code, using
+  `speed-bench/session_concurrency_bench`; record aggregate and per-session
+  throughput. Keep n-grams on disk and monitor memory during arena growth,
+  session destruction/recreation and engine cleanup.
+  Run `python3 -m unittest discover -s tests -p test_serve_concurrency_bench.py`
+  so truncated/error streams cannot silently enter the throughput results.
 - Run `tests/test_server_story.py` with at least 49K server context: all sixteen
   story facts, the correction turn and cached-prefix reuse must pass. Also
   run `tests/test_agent_vision.py` with a long archive and

docs/QWEN38_FLASH_NEXT.md

@@ -48,13 +48,21 @@ Add `--mtp` for speculative decoding using the built-in MTP weights.
 `qwen3.8-flash-next-reasoner` aliases. Tool calls use the native
 `<tool_call><function=...><parameter=...>` format.
 
-`ds4-server --batched-session N` decodes the slots together: everything that
+On Metal, `ds4-server --batched-session N` decodes the slots together. Work that
 only reads weights runs once for the whole batch, while the delta-net
 recurrence, the attention caches and the n-gram convolution stay per session.
 The slots also share one prefill workspace, so an extra slot costs its caches
 rather than another few GiB of transients. As with the other natively batched
 models, grouping changes the order of floating point reductions, so a batched
 reply can differ from the same prompt decoded alone.
+Add `--mtp` to use batched speculative decoding as well. The server accepts
+matching greedy drafts by default, including at nonzero temperature. With
+`--mtp-exact-sampling`, sampled requests use ordinary batched decoding;
+temperature-zero requests can still speculate. Default greedy acceptance also
+means a seed need not reproduce a reply under a different batching schedule.
+Speculative batches above 16 sessions, images and steering use the ordered
+fallback. CUDA currently
+decodes sessions in order.
 
 Disk KV checkpoints include recurrent state. Rewinding to an earlier position
 replays the retained prefix on the next evaluation. The native context is
@@ -146,9 +154,8 @@ the responses and diagnostics for inspection; it does not grade image content.
 The Metal and CUDA graphs accept Q8_0, Q4_0, F16, BF16 and F32
 dense weights, Q8_0/MXFP4/Q4_0/Q4_K/Q2_K/IQ2_XXS experts, F16/F32/Q8_0
 hyper-connection mixers and the original BF16 n-gram table.
-Tensor parallelism, pipeline execution, SSD expert streaming and native
-multi-session decode batching are not implemented for this model yet.
-The server keeps separate session state and decodes ready sessions in order.
+Tensor parallelism, pipeline execution and SSD expert streaming are not
+implemented for this model yet.
 ROCm is not supported. CPU code is a correctness reference, not a general
 inference backend.
 

docs/SERVER.md

@@ -85,7 +85,10 @@ use the ordered fallback.
 
 Long prefills yield to active decoders in bounded intervals, normally 128
 tokens. `--mixed-prefill-quantum N` changes that interval for testing.
-Session-batched serving uses ordinary target decoding, not MTP/DSpark.
+Session-batched serving uses ordinary target decoding, except Qwen3.8 on
+Metal, where `--mtp` also batches speculative decoding. Its
+`--mtp-exact-sampling` mode uses ordinary batches for nonzero-temperature
+requests. Other models do not use MTP/DSpark while session batching is active.
 For the eight-L40S example, see [CUDA GPUs](CUDA_MULTI_GPU.md#serve-multiple-users).
 
 ## Images