Stop relying on the model to restart a call after an injected reasoning close. Wait for a complete call, keep the preceding prose as reasoning, return the call structurally, and keep its protocol text out of streaming output.
Closes#675
Stop asking tool-enabled models to open a reasoning block that the prompt already opened. Hold the tentative answer until a tool call, response end, or second close marker proves whether it is answer text or another reasoning pass, for both OpenAI and Anthropic streams.
Closes#678
When an incoming GLM prompt exactly matches the beginning of the live session, rewind one token before its boundary and reevaluate that final prompt token. This restores the correct logits without rebuilding a long prompt.
Closes#668
Build the nearest prior call lookup while scanning forward so large Responses and Anthropic histories do not trigger repeated full-history searches.
Closes#547
Parse duplicate request fields into temporary storage before releasing the previous value, and cover malformed duplicates across OpenAI, Anthropic, Responses, and tool schemas.
Closes#433
Guard dynamic buffer growth against capacity wraparound, normalize malformed surrogate escapes to replacement characters without consuming unrelated escapes, and add a tool-heavy OpenCode-shaped parser stress test for issue #405.
Closes#405.
The model sometimes opens a DSML stanza without closing its thinking
first. The generation loop deliberately ignores tool markers inside
<think>, so such turns ran to the token limit and the call was dropped
at parse time, stalling agent sessions (#318).
Recover forward instead of rewriting sampled context: when a complete
stanza opening appears inside an unclosed think block, force-feed
"</think>" plus a blank line and let the model continue. Measured on
the real model, that position predicts a fresh stanza opening strongly
enough that the call restarts cleanly on the executable side; the
dangling opening stays harmlessly inside reasoning. Re-emitting the
stanza opening ourselves was tried and rejected: the duplicated opening
reads as an already-made call and the model ends the turn.
Detection is on accumulated text, so marker tokenization does not
matter, and only a complete opening triggers: quoted fragments and a
lone "<" keep decoding untouched. DS4_SERVER_DISABLE_THINK_TOOL_RECOVERY
restores the old behavior.
The new think-tool-recovery test replays a teacher-forced malformed
prefix token by token, asserts the recovery triggers exactly when the
opening completes, and verifies the live model then emits a valid,
parseable call.
Add shared help text across the CLI, server, agent, bench, and eval tools. Expand distributed-mode guidance, clean up endpoint naming, and use a TTY-only 256-color layout with clearer section titles, option arguments, separators, examples, and explanatory text.
Reject stale KVC graph payloads before loading them, bump the session payload ABI after recent runtime layout changes, and keep PR #253's prefill-failure recovery as a final safety net.
The server now discards the exact disk checkpoint that produced a restored-state prefill failure, resets the disk continuation marker, and logs unlink errors instead of silently ignoring them.
Tests cover stale payload ABI rejection.
Closes#251
Evict before writing a new KV checkpoint so the incoming entry cannot be selected as its own victim. Pass incoming-checkpoint context into eviction scoring and devalue compatible continued-prefix waypoints when making room for a longer checkpoint.
Inspired by the KV-cache observations in #174/#175 and #176/#177.
Co-authored-by: Salvatore Sanfilippo <antirez@gmail.com>
Co-authored-by: unsaltedbutter-ai <261676361+unsaltedbutter-ai@users.noreply.github.com>
After an unclean shutdown the on-disk KV checkpoint can be intact
(header, hash, token count all valid) but leave Metal in a state
where prefill fails. Since the file keeps passing load-time checks
it gets reloaded on every request, looping forever until the user
manually deletes the cache directory.
On prefill failure, if the prefix came from a disk entry, unlink it
and invalidate the session. Next request gets a clean cache miss.
Closes#251
Expose Flash and PRO model IDs as server compatibility aliases for the loaded GGUF. Add download targets for the mixed Flash q2/q4 quant and PRO GGUFs, update README guidance, and add the M3 Ultra PRO benchmark data.
Add DeepSeek V4 Pro support on top of main.
- add Flash/Pro shape selection and Pro Metal inference support
- fix shared SwiGLU clamp and indexer QAT behavior for Flash and Pro
- update eval/context tooling and imatrix quantization for Pro
- tag KV caches/sessions with model ids so Flash and Pro state cannot cross-load
- remove misc/PRO.md from tracked files while leaving it as a local scratch log
Long prefills (large prompts, no cache hit) can take minutes on local
hardware. ds4-server was silent on the socket the whole time, so HTTP
and TCP idle timeouts on the client side would close the connection
before the first response byte was written -- see the "sse headers
failed" log line that appeared at the very end of a multi-minute
prefill in real agent runs.
Stream the SSE response headers from the prefill_chunk progress
callback, then emit a ":" comment line (ignored by SSE clients per the
spec) at most every five seconds while prefill is still running. The
keepalive is best-effort: a closed socket simply fails the writes and
the outer code discovers the dead connection when it tries to stream
a real event, matching the existing error path. The tool-checkpoint
rebuild path pre-arms headers_sent because it only runs after the
response stream is already in flight, so it never tries to re-send
the SSE header line.
Verified on macOS Metal, q2-imatrix GGUF, ctx=200000:
- ./ds4_test --server passes
- 35s fresh prefill: client receives ": prefill" lines at +6/+12/
+18/+25/+31s, then SSE content events at +35s, no client disconnect
- 1s cached prompt: unchanged (sse_headers is still emitted from the
request handler when prefill never fires)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
try_repair_dsml scanned the full generated text for DSML tags. When the
model discusses DSML syntax in its reasoning (e.g. explaining the DSML
tags), those text mentions inflate the tag counts, causing false positive
repairs (appending unnecessary closing tags).
Fix: find the last </thinking> boundary and start counting only from
there. DSML mentioned inside reasoning is model text, not executable
tool calls — matches the same approach used by parse_generated_message_ex
(commit 037ee39).
Also updated the hallucinated strip path to copy the thinking section
verbatim and only strip from the post-thinking region.
Real-world validation: observed this exact false positive in production.
The model was explaining how try_repair_dsml works and quoted the DSML
tag syntax in its explanation. The parser mistook the quote for a real
tool call and the tag counting inflated, causing a failed repair.
Metrics from production use (after all fixes in this branch):
Tool calls: 169 | Invalid: 2 | Repaired: 38 | Orphan: 3
Only 2 cases remain unrecoverable; the other 41 (38 repaired + 3 orphan)
are now gracefully handled instead of causing finish=error.
Signed-off-by: Rui Gu <jackygurui@gmail.com>
When parse_generated_message_ex is called with require_thinking_closed=true
and the model never outputs </thinking>, the entire generation is treated
as reasoning and any DSML inside is silently ignored. This stderr log
makes the gate visible for debugging.
Refs: #167, commit 037ee39 (Ignore tool calls emitted inside thinking)
Signed-off-by: Rui Gu <jackygurui@gmail.com>
antirez
ivanfioravanti
Matteo Collina
Luigi Colluto
AAB
Kingsley Wong
Salvatore Sanfilippo
Giovanni Montana
user
Allen091080andClaude Opus 4.7
Rui Gu