diff --git a/QA_BEFORE_RELEASES.md b/QA_BEFORE_RELEASES.md index dfad65ba..3410ec08 100644 --- a/QA_BEFORE_RELEASES.md +++ b/QA_BEFORE_RELEASES.md @@ -1960,6 +1960,28 @@ paired comparisons, also with decode graphs disabled. Separately run the record different-schedule probability differences separately. Nearly tied experts can amplify normal rounding, so a max-logit difference alone does not establish a state bug or a quality regression. +- For Metal native batching, run `tests/test_metal_session_batch` with + `DS4_TEST_BATCH_ISOLATION=1` at two, four and eight sessions, including a + sparse prefix. Reordered rows and changed companions must leave the target's + complete logits identical. Separately score both official manifests with + `--session-batch 1`, `4` and `8`; compare paired losses and API agreement. + The scalar and batched reductions need not be bit-identical. Investigate + greedy mismatches with their logit margins, not by loosening kernel tests. + Run `tests/test_qwen4_kernels` under Metal validation and the real-model + `tests/test_qwen4_ngram_state` for mixed ordinary/MTP cycles, failed batch + reads, exact recovery and the final context slot. +- Start a four-slot Qwen Metal server with and without `--mtp`. Check concurrent + tool calls, prefix reuse, cancellation, stop strings and one/two-token output + limits. Repeat with `--mtp-exact-sampling`, mixing temperature-zero and + sampled requests. Only the former may use greedy speculative acceptance. + Compare seeded sampled replies in exact mode; default greedy MTP changes + random draws with draft scheduling, so equal seeds need not give equal text. + Benchmark ordinary and speculative batches on both prose and code, using + `speed-bench/session_concurrency_bench`; record aggregate and per-session + throughput. Keep n-grams on disk and monitor memory during arena growth, + session destruction/recreation and engine cleanup. + Run `python3 -m unittest discover -s tests -p test_serve_concurrency_bench.py` + so truncated/error streams cannot silently enter the throughput results. - Run `tests/test_server_story.py` with at least 49K server context: all sixteen story facts, the correction turn and cached-prefix reuse must pass. Also run `tests/test_agent_vision.py` with a long archive and diff --git a/docs/QWEN38_FLASH_NEXT.md b/docs/QWEN38_FLASH_NEXT.md index e2cbe9c8..091f08c6 100644 --- a/docs/QWEN38_FLASH_NEXT.md +++ b/docs/QWEN38_FLASH_NEXT.md @@ -48,13 +48,21 @@ Add `--mtp` for speculative decoding using the built-in MTP weights. `qwen3.8-flash-next-reasoner` aliases. Tool calls use the native `` format. -`ds4-server --batched-session N` decodes the slots together: everything that +On Metal, `ds4-server --batched-session N` decodes the slots together. Work that only reads weights runs once for the whole batch, while the delta-net recurrence, the attention caches and the n-gram convolution stay per session. The slots also share one prefill workspace, so an extra slot costs its caches rather than another few GiB of transients. As with the other natively batched models, grouping changes the order of floating point reductions, so a batched reply can differ from the same prompt decoded alone. +Add `--mtp` to use batched speculative decoding as well. The server accepts +matching greedy drafts by default, including at nonzero temperature. With +`--mtp-exact-sampling`, sampled requests use ordinary batched decoding; +temperature-zero requests can still speculate. Default greedy acceptance also +means a seed need not reproduce a reply under a different batching schedule. +Speculative batches above 16 sessions, images and steering use the ordered +fallback. CUDA currently +decodes sessions in order. Disk KV checkpoints include recurrent state. Rewinding to an earlier position replays the retained prefix on the next evaluation. The native context is @@ -146,9 +154,8 @@ the responses and diagnostics for inspection; it does not grade image content. The Metal and CUDA graphs accept Q8_0, Q4_0, F16, BF16 and F32 dense weights, Q8_0/MXFP4/Q4_0/Q4_K/Q2_K/IQ2_XXS experts, F16/F32/Q8_0 hyper-connection mixers and the original BF16 n-gram table. -Tensor parallelism, pipeline execution, SSD expert streaming and native -multi-session decode batching are not implemented for this model yet. -The server keeps separate session state and decodes ready sessions in order. +Tensor parallelism, pipeline execution and SSD expert streaming are not +implemented for this model yet. ROCm is not supported. CPU code is a correctness reference, not a general inference backend. diff --git a/docs/SERVER.md b/docs/SERVER.md index 941e6377..62ba48eb 100644 --- a/docs/SERVER.md +++ b/docs/SERVER.md @@ -85,7 +85,10 @@ use the ordered fallback. Long prefills yield to active decoders in bounded intervals, normally 128 tokens. `--mixed-prefill-quantum N` changes that interval for testing. -Session-batched serving uses ordinary target decoding, not MTP/DSpark. +Session-batched serving uses ordinary target decoding, except Qwen3.8 on +Metal, where `--mtp` also batches speculative decoding. Its +`--mtp-exact-sampling` mode uses ordinary batches for nonzero-temperature +requests. Other models do not use MTP/DSpark while session batching is active. For the eight-L40S example, see [CUDA GPUs](CUDA_MULTI_GPU.md#serve-multiple-users). ## Images