Resolves the residual multiple-choice false negative tracked in #321
(follow-up to #319). find_answer_letter returned the first
boundary-isolated in-range capital on the answer line, so when the model
rejects a distractor before stating its pick the rejected letter won:
"Answer: It is not B, the answer is D" -> B (should be D)
"Answer: rules out C, leaving D" -> C (should be D)
The forward scan now skips an in-range capital that is the object of an
explicit, adjacent rejection -- "not B", "isn't B", "rule(s/d) out C",
"eliminate/reject/except E". The cue set is deliberately small and
high-precision: it rewrites only clear elimination phrasing and makes no
attempt at general selection-vs-rejection sentence parsing, which is the
principled limit noted in #321. It never looks before the answer marker
or across a newline, so a leading pick is accepted before any later
rejected distractor is inspected -- "Answer: D, not B" still grades D,
the regression that blocks the naive "take the last letter" fix.
Locked by new --self-test-extractors cases (model-free, no model/GPU),
including the "D, not B" guard. All prior extractor, integer, and
regrade self-tests continue to pass.
Tested: make ds4-eval (Metal) and make cpu build clean (-Wall -Wextra);
./ds4-eval --self-test-extractors passes on both the Metal and CPU
binaries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
find_answer_letter returned the first boundary-isolated in-range capital
after "Answer:", so on 10-choice (A-J) cases a leading English pronoun or
article was graded as the choice:
"Answer: I think it is C" -> I (should be C)
"Answer: I'll go with C." -> I (should be C)
"Answer: A careful reading ... C" -> A (should be C)
24 embedded cases are 10-choice, so this is reachable. The forward scan
now skips a standalone capital that begins a same-line word or a
contraction; the reverse scan still recovers it when it is the only
candidate ("Answer: A is correct"), and a genuine standalone answer
("Answer: I.") is unchanged.
Known limitation: a distractor explicitly rejected *before* the chosen
letter on the same line ("... rules out C, leaving D") is still misread;
that needs sentence-level parsing and is left for discussion.
Locked by new --self-test-extractors cases (model-free). All prior
self-tests and the integer/regrade cases continue to pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two model-free correctness fixes in the answer grader, locked by new
--self-test-extractors cases (no model/GPU needed):
- find_integer_answer: the answer-marker path took the first integer in
the window, so an answer line that shows its arithmetic
("Answer: m+n = 256+37 = 293") was graded as the first summand (256)
instead of the stated total (293) -> false negative. Reachable on the
many embedded AIME2025 "Find m+n / a+b+c" cases. The scan is now bound
to the answer line and, when it shows arithmetic, reads the value after
the last '='. "Final answer: 082" -> "82" and the loose fallback are
preserved.
- regrade_trace_file: every case with a MODEL_OUTPUT block was graded,
including interrupted ones (STOPPED/SKIPPED/SWITCHED/ERROR). Their
partial output inflated passed/failed and raised spurious "changed"
drift. Grading is now factored into regrade_case_outcome(), which only
grades PASSED/FAILED (and legacy status-less) traces; others are
reported via a new not_graded counter.
Tested: make (Metal) and make cpu build clean (-Wall -Wextra);
./ds4-eval --self-test-extractors passes; regrade of a 3-case trace
(1 PASSED, 2 STOPPED) reports passed=1 not_graded=2 changed=0
(was passed=2 failed=1 changed=1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add shared help text across the CLI, server, agent, bench, and eval tools. Expand distributed-mode guidance, clean up endpoint naming, and use a TTY-only 256-color layout with clearer section titles, option arguments, separators, examples, and explanatory text.
Add DeepSeek V4 Pro support on top of main.
- add Flash/Pro shape selection and Pro Metal inference support
- fix shared SwiGLU clamp and indexer QAT behavior for Flash and Pro
- update eval/context tooling and imatrix quantization for Pro
- tag KV caches/sessions with model ids so Flash and Pro state cannot cross-load
- remove misc/PRO.md from tracked files while leaving it as a local scratch log
Use a shared DS4_DEFAULT_MIN_P of 0.05 for sampled generation. The default keeps top_p at 1.0 and uses min_p as the active filter, preventing very low relative-probability tokens from entering the sampling set while preserving plausible alternatives.
Apply the same default to the CLI, server, and ds4-eval, expose --min-p in CLI/eval, and make the server thinking-mode fixed sampler use min_p=0.05 instead of disabling min-p filtering. Explicit non-thinking API min_p values continue to be honored.
Builds checked with make -B ds4 ds4-server ds4-eval. Server unit tests pass; full make test could not run because another DS4 process was already holding /tmp/ds4.lock.
Add a q key path that exits the TUI cleanly and prints the normal plain-text report, show the pause/quit hints in the TUI title, and keep the final report compact by omitting question titles and ANSI escapes. Also clamp per-case generation to the remaining context budget so small-ctx runs stop or skip individual cases instead of aborting the whole benchmark on KV capacity errors.
Remove SuperGPQA rows with broken upstream keys from ds4-eval instead of locally re-keying them. Replace them with cleaner self-contained SuperGPQA items, document the audited SuperGPQA slice, and regenerate the imatrix calibration prompts so benchmark-derived calibration no longer includes the broken items.
antirez
rinaldofestaandClaude Opus 4.8
Salvatore Sanfilippo
user
Ivan Fioravanti