Keep fingerprint-only matching for authenticated visible histories that rebase spans, but require exact historical image positions for token and rewind reuse. Reproduce the old false hit with a synthetic checkpoint. Fix the live eviction oracle to include all competing slots. Four-slot Qwen and GLM vision/tool/eviction and concurrent-request tests pass. Follow-up to #765.
With this fix the server with respond to e.g.
"reasoning_effort": "50"
so it becomes compatible with pi.dev that doens't accept numerical value (and cannot send think_level by default)
(cherry picked from commit de418c0861)
Verify tool-enabled answers are emitted before generation finishes for OpenAI and Anthropic clients. Keep the existing DeepSeek repeated-reasoning regressions. Follow-up to #1077.
append_glm_tool_calls_text() wrapped the reconstructed block in a newline on
each side. GLM's tokenizer.chat_template does not: it emits
<|assistant|><think></think><tool_call>name<arg_key>k</arg_key>...</tool_call>
with no surrounding whitespace in any variant (empty content, non-empty
content, reasoning present, followed by an observation or by a user turn).
The block is only wrapped when it has to be rebuilt from parsed arguments;
when the raw sampled bytes are available they are emitted verbatim, so the two
paths disagreed.
That disagreement is invisible while a conversation stays in one process: the
live tool turn feeds the raw sampled bytes into the prompt, so the running KV
prefix still lines up. It surfaces when a client replays the history, because
the replay goes through the reconstruction path and picks up the extra
newlines. Any key or checkpoint derived from the visible transcript then fails
to match the replay: in the restart smoke test a conversation that should
re-prefill ~14 tokens re-prefilled 3800 of 11992, because the visible-key
checkpoint it wrote was three bytes off.
Cold prefill is affected too, and independently of any cache: a replayed GLM
tool conversation was being rendered in a shape the model never trained on.
Drop the two newlines. test_glm_tool_calls_render_matches_template asserts the
template-true block and that the raw and reconstructed renders are byte
identical; test_render_glm_preserves_reasoning_with_tools had pinned the old
bytes and is corrected.
Reproduced with the restart-restore smoke test on GLM-5.3-Flash-Q2:
before: FAIL, restored 8192 of 11992 tokens (3800 re-prefilled)
after: OK, restored 11998 of 12012 tokens (14 re-prefilled)
(cherry picked from commit d186421d96)
Upstream 81a7658 ("Preserve live vision context across server tool calls")
rewrote the live-continuation helpers this branch replaced with
slot_probe_reuse(): image markers carry request-local nonces and are
normalized for visible replay, image tokens are spliced into continuations
instead of being tokenized as text, and image appends are validated by
fingerprint and row count rather than by exact span equality. The merge
keeps the probe architecture and folds the new conditions into it, so slot
routing and the execution ladder still cannot disagree:
- The vision gate is fingerprint-prefix (new
ds4_session_vision_fingerprint_prefix_matches): historical images must
match by row count and fingerprint, positions may shift because
build_live_prompt_suffix() rebases them, and appended images are allowed.
An exact-state gate rejected valid appends, and a position-sensitive one
rejected replays that omit hidden reasoning.
- Every live tier materializes through build_live_prompt_suffix(), so new
images are spliced into the continuation and a failed rebase is a safe
miss rather than a malformed prompt.
- responses-visible and thinking-visible compare in visible_prompt_key
space, so a replay with fresh marker nonces still binds while a changed or
shifted image does not.
- Tool-output-only continuations keep upstream's contract: if the selected
live tier cannot be materialized, the request is answered with 409
(live_continuation_unavailable) instead of being generated from a
context-free prompt built from tool outputs alone.
- Single-session mode refreshes the slot's rendered-text cache like the
batched worker does, so the memory-text tier is probe-authoritative on
both worker paths; the redundant ladder fallback and the four superseded
upstream helpers are removed.
Tests: test_slot_probe_vision_tiers covers image appends, pixel mismatches
and marker-nonce visible replay at the probe/routing level;
test_live_continuation_contract pins the 409 predicate;
tests/test_session_state.c gains fingerprint-prefix cases; and the new live
suite tests/test_server_vision_live_state.py checks slot routing, eviction
victims and the tool-result-only 409 against a real vision model.
Verified on DeepSeek V4 Flash Vision Experimental (M5 Max): ./ds4_test
--server, ds4_agent_test, test-session-state, live test_server_vision_cache.py
(1- and 4-session), test_server_story.py, test_server_batching.py and the new
vision live-state suite all pass.
Qwen3.8 sessions were excluded from native session batching: each decode
step ran the whole graph once per session, so sixteen sessions cost
sixteen times the weight traffic. The batched encoder now runs the dense
work (hyper-connection mixers, projections, MoE, output head) once over
the stacked rows and keeps only the recurrent state, the caches and the
PLE history per session, with batched GDN scan and conv kernels reading
the pooled state through per-row slots.
Session transients (prefill scratch) move to one shared arena sized for
the largest session, so each further session costs its caches only
(memory per session divided by ~3.6 at ctx 4096). The dense GEMM gains a
split-K path for the narrow-batch regime, and the dense/MoE/mixer GEMM
thresholds, tuned for T=1 or T>=8192, are moved to serve decode batches
of 4..64 rows. Every step keeps the tokens of the sequential path;
float reduction order changes where the batched kernels do.
C=16 at ctx 4096 on an M5 Max goes from ~50 to ~143 tok/s aggregate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Keep the table outside model mappings and Metal residency. Use bounded explicit reads, reject failed lookups before GPU consumption, and remove the runtime sidecar option and paging controls. Preserve Qwen hash and speculative state handling; cover malformed files and exact recovery after prefill, decode, and MTP read failures.
Restore matching logits on verifier rewinds, replay stale recurrent state, discard old snapshots on restore, and invalidate partial payload reads. Keep lazy snapshots all-or-nothing and respect small prefill buffers and the unfused diagnostic path.
Preserve schema-declared string arguments. Repair all-row reference dumps and teacher-forced chunking. Add model-backed regressions and fix test-fixture leaks exposed by ASAN.
Use the structural reasoning delimiter on the server and wait for the parameter delimiter while streaming in ds4-agent. Add server and bytewise agent regressions; both fail before the fixes and pass afterward on M5 Max.
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
Reuse bounded image embeddings, preserve exact image identities while appending new images, and splice image tokens into authenticated live continuations. Handle image tool results in Responses and Anthropic requests as well as Chat Completions. Normalize request-local image markers for visible replay without trusting text placeholders as image identities.
Add prefix and encoder-cache unit coverage plus real-model image replay and Pi coding-client regressions. Validated GLM 5.3 Flash and DeepSeek Vision-Exp on M5 Max, including 53K/56K prefixes, batched sessions, all three API formats, fresh logit replay, and frontend ASan/UBSan.
Do not HTML-unescape argument bodies. Share a narrow, reversible escape for each body closing delimiter, including its literal escaped spelling. Keep streaming fragments intact across those escapes. Preserve protocol-looking text inside GLM agent arguments until the actual argument terminator. Add round-trip, streaming and agent regressions.
Use the incremental grammar state to detect completed calls, and skip argument/key bodies when searching final control markers. Validate complete calls before recovering unclosed reasoning. Cover DeepSeek and GLM arguments containing both tool endings and thinking tags without truncating or losing the call.
Only repair an omitted DeepSeek wrapper around already closed invocations. Never supply a missing argument or invocation terminator. Preserve length finishes and do not retry beyond the client output budget. Log the actual stop cause, token ID and available budget when a tool call is incomplete.
Track GLM function names and argument bodies with the existing incremental recognizer. Keep reasoning quotes out of syntax sampling. Discard unconsumed speculative tokens at tool and control stops, and resample after a structural/payload boundary with restored logits. Rebuild invalid recurrent prefixes with their image conditioning. Add GLM fragment and byte-boundary coverage.
Retain a lone opening angle bracket until the next token, without forcing code or HTML greedy. Previously the tracker consumed it and missed a parameter close split across tokens. Add byte-by-byte regressions for all three DeepSeek spellings and string/JSON bodies; all twelve failing checks now pass.
Upstream 98b9135 ('server: reuse exact multimodal live KV safely') and
b0982a1 ('Harden multimodal cache fallback') restructured the execution
ladder this PR replaced with slot_probe_reuse(). Resolution keeps the
probe architecture and moves the new safety conditions into it, so
execution and routing still cannot disagree:
- The live_vision_match gate (image fingerprints are invisible to text/
token prefixes) is now checked once at the top of
slot_probe_reuse_locked(); job_slot_score inherits it, subsuming
upstream's -1 vision guard.
- The GLM memory-rewind materialization re-validates the frontier after
ds4_session_rewind() (common prefix lands on the rewind target and the
vision state still matches) and falls back to disk cache / cold prefill
when validation fails, as upstream's ladder did.
- Upstream's ladder reorder (memory-text before disk cache) is subsumed
by the probe's tier order; the deleted ladder helpers stay deleted.
- Tests from both sides kept: the slot probe/routing tests and upstream's
test_multimodal_prefill_resume_frontier.
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
The reuse-aware router sent a request that reuses nothing to the slot
with the shortest resident checkpoint (-live). On a long-running
server this leaks: a finished long conversation is never the cheapest
victim, so such checkpoints pile up, the pool of usable slots shrinks,
and subagent/parallel requests end up evicting shorter *active*
conversations - the prefill thrashing this routing set out to fix
comes back.
Split the no-reuse victim selection into staleness tiers:
- empty slots still win (nothing to evict);
- a checkpoint idle past SLOT_STALE_AFTER_SEC (1h) is most likely a
finished conversation: victims are picked most-idle-first,
regardless of size, so forgotten sessions get reclaimed;
- fresher checkpoints belong to active conversations: keep the -live
rule (evict the shortest, lose the least prefilled work).
Staleness only lowers a checkpoint's priority as a *victim*; it never
blocks reuse - the owner of a stale checkpoint who continues the
conversation still pins the slot via the reuse tier (which now scores
the actual reusable prefix instead of the full checkpoint length).
Also log evictions at dispatch time (slot, evicted tokens, idle time,
stale/protected), so routing decisions are visible in the server log
instead of being inferable only from the next request's cache miss.
With --batched-session N, job_slot_score routed requests to the slot
with the longest raw common token prefix, with no check that the slot's
checkpoint is actually reusable. Parallel subagents sharing a system
prompt were attracted to the slot holding the main agent's long context,
evicting it with zero reuse benefit and forcing a full re-prefill when
main resumed (masked only by --kv-disk-dir round-trips). Even an alien
request sharing no prefix at all evicted slot[0] on the tie-break.
Split every RAM reuse tier into a cheap probe predicate and its
materialization, and define tier order and conditions exactly once in
slot_probe_reuse_locked(): responses-visible, responses/anthropic
tool-output, GLM memory-rewind, memory-token, thinking-visible,
memory-text. The execution ladder (generate_job) probes once and
materializes only the selected tier; job_slot_score consumes the same
probe, so routing and execution cannot disagree. A slot scores +live
when the request would reuse its live state, -live otherwise: an empty
slot always beats an eviction, and the cheapest (shortest) unusable
checkpoint becomes the victim. Text-tier bindings are part of the probe,
so single-session chats whose resent history retokenizes differently or
omits hidden thinking keep pinning to their slot.
The memory-text probe needs rendered checkpoint text; cache it per slot
and refresh in the slot worker after each job, so probing stays a memcmp
and never detokenizes under tool_mu.
Verified: unit tests for every probe tier plus dispatch routing
(./ds4_test --server), full make test green, and live checks showing the
subagent deflected to the empty slot, main resuming via memory-token,
and thinking-visible / anthropic-tool-output / responses-tool-output
continuations unchanged.
Stop relying on the model to restart a call after an injected reasoning close. Wait for a complete call, keep the preceding prose as reasoning, return the call structurally, and keep its protocol text out of streaming output.
Closes#675
Stop asking tool-enabled models to open a reasoning block that the prompt already opened. Hold the tentative answer until a tool call, response end, or second close marker proves whether it is answer text or another reasoning pass, for both OpenAI and Anthropic streams.
Closes#678
When an incoming GLM prompt exactly matches the beginning of the live session, rewind one token before its boundary and reevaluate that final prompt token. This restores the correct logits without rebuilding a long prompt.
Closes#668
Tarjei Mandt
Salvatore Sanfilippo
Alessandro Checco
Spencer Gilbert
Jeff
ayuan1129
Mikhail Korobov
davidandClaude Fable 5.1
Jordi Posthumus
Donato Capitella
Matteo Collina