Upstream 81a7658 ("Preserve live vision context across server tool calls")
rewrote the live-continuation helpers this branch replaced with
slot_probe_reuse(): image markers carry request-local nonces and are
normalized for visible replay, image tokens are spliced into continuations
instead of being tokenized as text, and image appends are validated by
fingerprint and row count rather than by exact span equality. The merge
keeps the probe architecture and folds the new conditions into it, so slot
routing and the execution ladder still cannot disagree:
- The vision gate is fingerprint-prefix (new
ds4_session_vision_fingerprint_prefix_matches): historical images must
match by row count and fingerprint, positions may shift because
build_live_prompt_suffix() rebases them, and appended images are allowed.
An exact-state gate rejected valid appends, and a position-sensitive one
rejected replays that omit hidden reasoning.
- Every live tier materializes through build_live_prompt_suffix(), so new
images are spliced into the continuation and a failed rebase is a safe
miss rather than a malformed prompt.
- responses-visible and thinking-visible compare in visible_prompt_key
space, so a replay with fresh marker nonces still binds while a changed or
shifted image does not.
- Tool-output-only continuations keep upstream's contract: if the selected
live tier cannot be materialized, the request is answered with 409
(live_continuation_unavailable) instead of being generated from a
context-free prompt built from tool outputs alone.
- Single-session mode refreshes the slot's rendered-text cache like the
batched worker does, so the memory-text tier is probe-authoritative on
both worker paths; the redundant ladder fallback and the four superseded
upstream helpers are removed.
Tests: test_slot_probe_vision_tiers covers image appends, pixel mismatches
and marker-nonce visible replay at the probe/routing level;
test_live_continuation_contract pins the 409 predicate;
tests/test_session_state.c gains fingerprint-prefix cases; and the new live
suite tests/test_server_vision_live_state.py checks slot routing, eviction
victims and the tool-result-only 409 against a real vision model.
Verified on DeepSeek V4 Flash Vision Experimental (M5 Max): ./ds4_test
--server, ds4_agent_test, test-session-state, live test_server_vision_cache.py
(1- and 4-session), test_server_story.py, test_server_batching.py and the new
vision live-state suite all pass.
Reuse bounded image embeddings, preserve exact image identities while appending new images, and splice image tokens into authenticated live continuations. Handle image tool results in Responses and Anthropic requests as well as Chat Completions. Normalize request-local image markers for visible replay without trusting text placeholders as image identities.
Add prefix and encoder-cache unit coverage plus real-model image replay and Pi coding-client regressions. Validated GLM 5.3 Flash and DeepSeek Vision-Exp on M5 Max, including 53K/56K prefixes, batched sessions, all three API formats, fresh logit replay, and frontend ASan/UBSan.
Reserve a separate byte for the terminator written by fmemopen, without changing the serialized payload length. Add model-free grow/reuse coverage and compare real-model snapshots against file serialization.
Use each model family’s normal message formatter when a multimodal observation contains no images. Non-vision Flash and PRO no longer reject every tool result as an apparent context overflow, and text results retain their native wrapper and escaping. Add model-free Flash/PRO/GLM regression coverage; CUDA coding/edit tasks pass with ordinary opportunistic and exact DSpark sampling.
Use the model-defined 2051-token dense boundary, handle padded selections, and synchronize reused reduction storage in Metal, CUDA and ROCm, including vision normalization.
Enable bounded ROCm selected-attention GEMM for GLM 5.3 without RoPE. Add independent attention references, boundary tests and finite, repeatable vision checks. Focused Strix continued prefill improves from about 21 to 68 tokens per second; stored official scores are unchanged.
Restore GLM recurrent and pooled-indexer state after rejected speculation. Invalidate unsafe DeepSeek rewinds and rebuild retained agent prefixes with their images before decoding again.
Rebuild saved TP token prefixes on both ranks. Give GLM speculative cycles an explicit command and shared token limit, with protocol version 10 requiring peers to update together.
Account for live and planned Metal GLM sessions when reserving graph and streaming memory. Add model-free state and protocol tests plus a small GPU rollback regression.
Mikhail Korobov
Salvatore Sanfilippo