Upstream 81a7658 ("Preserve live vision context across server tool calls")
rewrote the live-continuation helpers this branch replaced with
slot_probe_reuse(): image markers carry request-local nonces and are
normalized for visible replay, image tokens are spliced into continuations
instead of being tokenized as text, and image appends are validated by
fingerprint and row count rather than by exact span equality. The merge
keeps the probe architecture and folds the new conditions into it, so slot
routing and the execution ladder still cannot disagree:
- The vision gate is fingerprint-prefix (new
ds4_session_vision_fingerprint_prefix_matches): historical images must
match by row count and fingerprint, positions may shift because
build_live_prompt_suffix() rebases them, and appended images are allowed.
An exact-state gate rejected valid appends, and a position-sensitive one
rejected replays that omit hidden reasoning.
- Every live tier materializes through build_live_prompt_suffix(), so new
images are spliced into the continuation and a failed rebase is a safe
miss rather than a malformed prompt.
- responses-visible and thinking-visible compare in visible_prompt_key
space, so a replay with fresh marker nonces still binds while a changed or
shifted image does not.
- Tool-output-only continuations keep upstream's contract: if the selected
live tier cannot be materialized, the request is answered with 409
(live_continuation_unavailable) instead of being generated from a
context-free prompt built from tool outputs alone.
- Single-session mode refreshes the slot's rendered-text cache like the
batched worker does, so the memory-text tier is probe-authoritative on
both worker paths; the redundant ladder fallback and the four superseded
upstream helpers are removed.
Tests: test_slot_probe_vision_tiers covers image appends, pixel mismatches
and marker-nonce visible replay at the probe/routing level;
test_live_continuation_contract pins the 409 predicate;
tests/test_session_state.c gains fingerprint-prefix cases; and the new live
suite tests/test_server_vision_live_state.py checks slot routing, eviction
victims and the tool-result-only 409 against a real vision model.
Verified on DeepSeek V4 Flash Vision Experimental (M5 Max): ./ds4_test
--server, ds4_agent_test, test-session-state, live test_server_vision_cache.py
(1- and 4-session), test_server_story.py, test_server_batching.py and the new
vision live-state suite all pass.
ds4_sessions_eval_batch_speculative_argmax runs one MTP cycle for a batch:
every session contributes its next token and, when the predictor left it a
draft for that token, the draft as a second row. The dense work of the
trunk runs once over all the rows; the recurrent, n-gram and attention
steps run per session over its one or two rows with the single-session
verify kernels, snapshots included, so a session's own arithmetic is that
of the C=1 speculative cycle. A draft stands when it is the target's
argmax for its row; a rejected one restores the row-0 snapshot. The
predictor then runs its layer per session over the committed rows and
every session's head as one product. The Q8 batch GEMM serves 9..32 rows
in sixteen-row halves.
The harness gains --spec (with --verify checking the committed tokens
against plain greedy decode, which speculation must reproduce). On an M5
Max at C=16 the first cut commits 1.49 tokens per stream and cycle on the
prose corpus but the cycle costs 171 ms against 92 for a plain step: the
per-session kernels and the sequential predictor passes are the next
thing to batch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Keep the table outside model mappings and Metal residency. Use bounded explicit reads, reject failed lookups before GPU consumption, and remove the runtime sidecar option and paging controls. Preserve Qwen hash and speculative state handling; cover malformed files and exact recovery after prefill, decode, and MTP read failures.
Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
Reuse bounded image embeddings, preserve exact image identities while appending new images, and splice image tokens into authenticated live continuations. Handle image tool results in Responses and Anthropic requests as well as Chat Completions. Normalize request-local image markers for visible replay without trusting text placeholders as image identities.
Add prefix and encoder-cache unit coverage plus real-model image replay and Pi coding-client regressions. Validated GLM 5.3 Flash and DeepSeek Vision-Exp on M5 Max, including 53K/56K prefixes, batched sessions, all three API formats, fresh logit replay, and frontend ASan/UBSan.
Restore GLM recurrent and pooled-indexer state after rejected speculation. Invalidate unsafe DeepSeek rewinds and rebuild retained agent prefixes with their images before decoding again.
Rebuild saved TP token prefixes on both ranks. Give GLM speculative cycles an explicit command and shared token limit, with protocol version 10 requiring peers to update together.
Account for live and planned Metal GLM sessions when reserving graph and streaming memory. Add model-free state and protocol tests plus a small GPU rollback regression.
Upstream 98b9135 ('server: reuse exact multimodal live KV safely') and
b0982a1 ('Harden multimodal cache fallback') restructured the execution
ladder this PR replaced with slot_probe_reuse(). Resolution keeps the
probe architecture and moves the new safety conditions into it, so
execution and routing still cannot disagree:
- The live_vision_match gate (image fingerprints are invisible to text/
token prefixes) is now checked once at the top of
slot_probe_reuse_locked(); job_slot_score inherits it, subsuming
upstream's -1 vision guard.
- The GLM memory-rewind materialization re-validates the frontier after
ds4_session_rewind() (common prefix lands on the rewind target and the
vision state still matches) and falls back to disk cache / cold prefill
when validation fails, as upstream's ladder did.
- Upstream's ladder reorder (memory-text before disk cache) is subsumed
by the probe's tier order; the deleted ladder helpers stay deleted.
- Tests from both sides kept: the slot probe/routing tests and upstream's
test_multimodal_prefill_resume_frontier.
Implement GLM 5.3 Q2 and Q4 inference across Metal and CUDA, including tensor parallelism, RDMA, batching, MTP, snapshots, SSD streaming, and GLM 5.2 shared-path improvements.
Add quantization and validation tools, official quality fixtures, model downloads, regression coverage, release QA, and user documentation.
With --batched-session N, job_slot_score routed requests to the slot
with the longest raw common token prefix, with no check that the slot's
checkpoint is actually reusable. Parallel subagents sharing a system
prompt were attracted to the slot holding the main agent's long context,
evicting it with zero reuse benefit and forcing a full re-prefill when
main resumed (masked only by --kv-disk-dir round-trips). Even an alien
request sharing no prefix at all evicted slot[0] on the tie-break.
Split every RAM reuse tier into a cheap probe predicate and its
materialization, and define tier order and conditions exactly once in
slot_probe_reuse_locked(): responses-visible, responses/anthropic
tool-output, GLM memory-rewind, memory-token, thinking-visible,
memory-text. The execution ladder (generate_job) probes once and
materializes only the selected tier; job_slot_score consumes the same
probe, so routing and execution cannot disagree. A slot scores +live
when the request would reuse its live state, -live otherwise: an empty
slot always beats an eviction, and the cheapest (shortest) unusable
checkpoint becomes the victim. Text-tier bindings are part of the probe,
so single-session chats whose resent history retokenizes differently or
omits hidden thinking keep pinning to their slot.
The memory-text probe needs rendered checkpoint text; cache it per slot
and refresh in the slot worker after each job, so probing stays a memcmp
and never detokenizes under tool_mu.
Verified: unit tests for every probe tier plus dispatch routing
(./ds4_test --server), full make test green, and live checks showing the
subagent deflected to the empty slot, main resuming via memory-token,
and thinking-visible / anthropic-tool-output / responses-tool-output
continuations unchanged.
Integrate the MXFP4 Metal decode/prefill optimizations (rounds 7-9) as one
commit. Keep both the MXFP4 half_lut/tail_cull down-pipeline selection and
main's mpp_mask override for IQ2_XXS gate/up and Q2_K/IQ2_XXS down; they
target disjoint quant types and coexist.
Metal execution
Add the PRO Q4 routed-expert Metal path and distributed runtime changes needed to run the full model across two hosts.
Sliced GGUF loading
Allow distributed engines to bind only their local layer range, with token embeddings and the output head required only on the sides that need them.
Split artifacts
Document the two-file PRO Q4 setup and add download_model.sh targets for the coordinator half, worker half, and combined split download.
(cherry picked from commit a782cfba894c6a44af11e1b5fc69ccfc000ab39d)
Add DeepSeek V4 Pro support on top of main.
- add Flash/Pro shape selection and Pro Metal inference support
- fix shared SwiGLU clamp and indexer QAT behavior for Flash and Pro
- update eval/context tooling and imatrix quantization for Pro
- tag KV caches/sessions with model ids so Flash and Pro state cannot cross-load
- remove misc/PRO.md from tracked files while leaving it as a local scratch log
Use a shared DS4_DEFAULT_MIN_P of 0.05 for sampled generation. The default keeps top_p at 1.0 and uses min_p as the active filter, preventing very low relative-probability tokens from entering the sampling set while preserving plausible alternatives.
Apply the same default to the CLI, server, and ds4-eval, expose --min-p in CLI/eval, and make the server thinking-mode fixed sampler use min_p=0.05 instead of disabling min-p filtering. Explicit non-thinking API min_p values continue to be honored.
Builds checked with make -B ds4 ds4-server ds4-eval. Server unit tests pass; full make test could not run because another DS4 process was already holding /tmp/ds4.lock.
Cold disk KV checkpoints should maximize reuse across independent agent sessions, not preserve the variable user task. The reusable boundary is the last rendered <User> special token before the first <Assistant> special token: this keeps stable user-role scaffolding such as Codex environment_context, while avoiding later multi-turn user markers that belong to conversation history and are better handled by live/continued checkpoints.
Implement the anchor at token level, using the exact chat special token IDs, so the saved prefix is the same boundary the model sees. If no valid chat anchor is present, keep the existing trim/alignment heuristic for non-chat or too-short prompts.
Checked with ./ds4_test --server and real fresh-session pairs for pi, Claude Code, opencode, and Codex. The second run reused disk prefixes of 1188/1215 tokens for pi, 23164/23973 for Claude Code, 520/551 plus 10725/10756 for opencode's two stages, and 11554/11584 for Codex Responses, confirming that Codex's stable environment_context is now included while task text is left out.
Add routed-MoE imatrix collection to the DS4 graph path, including dataset generation docs and the tracked calibration prompt corpus.
Add a standalone DeepSeek V4 Flash HF-to-GGUF quantizer that reuses GGUF template metadata, emits DS4 Q2/Q4 recipes, supports imatrix-aware routed expert quantization, and internalizes the required Q8_0, Q4_K, Q2_K, and IQ2_XXS quantizers instead of vendoring GGML.
Document quantizer provenance, output formats, and the DS4-specific imatrix mapping used by the generated GGUF files.
Mikhail Korobov
davidandClaude Fable 5.1
antirez
Jordi Posthumus
Donato Capitella
Salvatore Sanfilippo
user
Codex