Check per-boot RDMA addressing and wired residency limits, match prefill chunk overrides on both ranks, and account for the extra sessions used by numerical oracles. Require at least three Metal TP rows when checking the native V4.1 path; two rows intentionally use ordered decoding.
The CPU-only placement tests also define DS4_ROCM_BUILD, but never use the GPU memory baseline. Exclude its declaration there, matching the existing guards around its uses.
Verified warning-free ROCm applications and test builds on Strix Halo, with placement, memory, session, server and agent tests passing. The production ds4.o is byte-identical after stripping debug metadata.
Guard native cross-session attention, grouped expert, and Q8 batch GEMM tests with __APPLE__. CUDA retains its supported scalar, prefill, and recurrent kernel oracles instead of failing to link against Metal-only entry points.
Use a four-term log1p series for exp(x) below 1/32 in all four DeepSeek router transform sites. Clamp the polynomial input so its unselected branch stays finite, and compare Metal probabilities against the shared accurate reference without widening the tolerance.
Refs antirez/ds4#1039. Validated on M5 Max with a fresh default build, test_deepseek41_metal, test_deepseek41_graph --vision-routing, and full Metal test runs with DS4_METAL_DISABLE_PRE_M5_ROUTER_TRANSFORM_FINALIZE_FUSION=1 and DS4_METAL_MATH_SAFE=1. No model required. Pre-M5 validation and model-level speed/SSD checks remain outstanding.
(cherry picked from commit 7f5f1b77ae)
Negotiate 512-byte TCP segments on Linux before connecting. Otherwise shrinking loopback sockets to 1 KiB receive buffers leaves 32 KiB segments, making zero-window probing dominate large test transfers and sometimes trip the idle timeout. Retain the payload sizes, bidirectional backpressure, descriptor-mode checks and timeout/EOF cases. Linux and macOS transport tests pass; production transport is unchanged.
Retain the measured Metal win without imposing per-token pthread creation on CUDA and ROCm. Tables remain disk-only. Warm M5 Max SSD decode improved from 12.62 to 13.39 tokens/s with byte-identical output; row and error tests also pass under ASAN/UBSAN.
ds4_engram_read_batch() already fans row reads across ENGRAM_READERS
workers, with the comment "Fixed concurrency hides random-read latency
without caching the table" -- but only above a 256-row threshold.
ds41_graph_step() reads DS4_ENGRAM_COLS rows per table (48 rows for the
two tables) through the serial ds4_engram_read(), so ordinary decode
never reached that path. Those 48 preads run in series, and they run
before ds4_gpu_begin_commands(), so the GPU is idle for all of them.
Only bulk prefill ever cleared 256.
This routes the decode read through the batched entry point and lowers
the threshold to 8 so decode-sized batches engage the existing workers.
Same rows, same order, same values -- only the fetch is concurrent.
Measured on a Mac Studio M3 Ultra (80 GPU cores, 256 GB, macOS 26.6.2),
DeepSeek-V4.1-Flash-Q2, resident (no --ssd-streaming), --ctx 131072,
temperature 0, 200 generated tokens, this branch vs its merge-base:
main 19.18, 19.14 tok/s median 19.16
main + patch 20.42, 21.00, 21.11, 21.18 tok/s median 21.06
+9.9%, about 4.7 ms/token. Generated text was byte-identical between
the two builds. (The 20.42 was a cold first run after rebuild.)
The underlying cost is storage latency, not bandwidth -- a standalone
pread benchmark against the same GGUF, 48 random 264-byte rows with
F_NOCACHE and readahead disabled, 150 batches:
serial 5.177 ms/batch (107.9 us/row)
parallel 0.614 ms/batch ( 12.8 us/row)
The saving is roughly constant per token, so it is worth more on a
faster tree: on ivanfioravanti's ds41f-optimizations branch, where this
was originally profiled, the same change measured 26.8 -> 30.5 tok/s
(Engram wall 5.55 -> 1.12 ms/token) over three alternating pairs, also
byte-identical.
Credit where it is due:
- ivanfioravanti, whose V4.1 optimization work is where this was
profiled. His ds41f-dspark branch already carries a better version
of this fix, ds41_engram_parallel(), which reads both tables in a
single dispatch -- but gated to speculative sessions, so ordinary
decode still falls through to the serial loop there too. Ungating
that is probably the better long-term shape of this change.
- The oMLX / drowzeys V4.1-on-Mac work, which is what pointed at
Engram as a decode bottleneck at all. Their approach was a RAM
cache of hot rows; measured row reuse here was only 22.5% over a
600-token coding generation, worth roughly +1 tok/s, so the win
turned out to be concurrency rather than caching. Wrong idea,
right place to look.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 92a61612a2)
Keep fingerprint-only matching for authenticated visible histories that rebase spans, but require exact historical image positions for token and rewind reuse. Reproduce the old false hit with a synthetic checkpoint. Fix the live eviction oracle to include all competing slots. Four-slot Qwen and GLM vision/tool/eviction and concurrent-request tests pass. Follow-up to #765.
With this fix the server with respond to e.g.
"reasoning_effort": "50"
so it becomes compatible with pi.dev that doens't accept numerical value (and cannot send think_level by default)
(cherry picked from commit de418c0861)
jpeg_prog_decode_ac_refine treated ZRL (run=15, size=0) as run=16 and
reused the skip loop, which kept consuming correction bits for non-zero
coefficients after the 16th zero-history coefficient. Those bits belong
after the next Huffman symbol (ITU T.81 G.1.2.3, libjpeg
decode_mcu_AC_refine), so the bitstream desynchronised: many progressive
JPEGs, including plain `cjpeg -progressive` output, failed to load or
decoded with wrong pixels.
Skip `run` zeros while refining non-zeros on the way, write the new value
(0 for ZRL) at the next zero, and stop, matching libjpeg's structure.
Add tests/test_image_decode with two cjpeg-generated fixtures that fail
on the old decoder (wrong pixels for grayscale, NULL for 4:2:0) and are
bit-exact with djpeg after the fix. Wire it into `make test`.
(cherry picked from commit 50ed0a4c73)
Verify tool-enabled answers are emitted before generation finishes for OpenAI and Anthropic clients. Keep the existing DeepSeek repeated-reasoning regressions. Follow-up to #1077.
append_glm_tool_calls_text() wrapped the reconstructed block in a newline on
each side. GLM's tokenizer.chat_template does not: it emits
<|assistant|><think></think><tool_call>name<arg_key>k</arg_key>...</tool_call>
with no surrounding whitespace in any variant (empty content, non-empty
content, reasoning present, followed by an observation or by a user turn).
The block is only wrapped when it has to be rebuilt from parsed arguments;
when the raw sampled bytes are available they are emitted verbatim, so the two
paths disagreed.
That disagreement is invisible while a conversation stays in one process: the
live tool turn feeds the raw sampled bytes into the prompt, so the running KV
prefix still lines up. It surfaces when a client replays the history, because
the replay goes through the reconstruction path and picks up the extra
newlines. Any key or checkpoint derived from the visible transcript then fails
to match the replay: in the restart smoke test a conversation that should
re-prefill ~14 tokens re-prefilled 3800 of 11992, because the visible-key
checkpoint it wrote was three bytes off.
Cold prefill is affected too, and independently of any cache: a replayed GLM
tool conversation was being rendered in a shape the model never trained on.
Drop the two newlines. test_glm_tool_calls_render_matches_template asserts the
template-true block and that the raw and reconstructed renders are byte
identical; test_render_glm_preserves_reasoning_with_tools had pinned the old
bytes and is corrected.
Reproduced with the restart-restore smoke test on GLM-5.3-Flash-Q2:
before: FAIL, restored 8192 of 11992 tokens (3800 re-prefilled)
after: OK, restored 11998 of 12012 tokens (14 re-prefilled)
(cherry picked from commit d186421d96)
Cover all pool tail lengths after 2048 selected rows. Repeat the split Metal decode with padded IDs and compare its value projection to a double-precision attention reference. Follow-up to #1051.
Track the pooled selection’s guaranteed prefix instead of promising that all selected rows are valid. Padded selections use the existing bounds-checked kernel.
Adapted from 101d20c23f3a27456bdc61e0da35e2bea97eb04e.
Validation: make -j1 and git diff --check passed on macOS M3 Ultra (Metal build). No model was loaded; GPU runtime and performance tests were not rerun.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 4e166b1fb4)
Upstream 81a7658 ("Preserve live vision context across server tool calls")
rewrote the live-continuation helpers this branch replaced with
slot_probe_reuse(): image markers carry request-local nonces and are
normalized for visible replay, image tokens are spliced into continuations
instead of being tokenized as text, and image appends are validated by
fingerprint and row count rather than by exact span equality. The merge
keeps the probe architecture and folds the new conditions into it, so slot
routing and the execution ladder still cannot disagree:
- The vision gate is fingerprint-prefix (new
ds4_session_vision_fingerprint_prefix_matches): historical images must
match by row count and fingerprint, positions may shift because
build_live_prompt_suffix() rebases them, and appended images are allowed.
An exact-state gate rejected valid appends, and a position-sensitive one
rejected replays that omit hidden reasoning.
- Every live tier materializes through build_live_prompt_suffix(), so new
images are spliced into the continuation and a failed rebase is a safe
miss rather than a malformed prompt.
- responses-visible and thinking-visible compare in visible_prompt_key
space, so a replay with fresh marker nonces still binds while a changed or
shifted image does not.
- Tool-output-only continuations keep upstream's contract: if the selected
live tier cannot be materialized, the request is answered with 409
(live_continuation_unavailable) instead of being generated from a
context-free prompt built from tool outputs alone.
- Single-session mode refreshes the slot's rendered-text cache like the
batched worker does, so the memory-text tier is probe-authoritative on
both worker paths; the redundant ladder fallback and the four superseded
upstream helpers are removed.
Tests: test_slot_probe_vision_tiers covers image appends, pixel mismatches
and marker-nonce visible replay at the probe/routing level;
test_live_continuation_contract pins the 409 predicate;
tests/test_session_state.c gains fingerprint-prefix cases; and the new live
suite tests/test_server_vision_live_state.py checks slot routing, eviction
victims and the tool-result-only 409 against a real vision model.
Verified on DeepSeek V4 Flash Vision Experimental (M5 Max): ./ds4_test
--server, ds4_agent_test, test-session-state, live test_server_vision_cache.py
(1- and 4-session), test_server_story.py, test_server_batching.py and the new
vision live-state suite all pass.
Tarjei Mandt
Salvatore Sanfilippo
Emilian Bold
Jake Maness
Alessandro Checco
Spencer Gilbert
Mark Shank
korale77
Jeff
ayuan1129
Mikhail Korobov