Retain the measured Metal win without imposing per-token pthread creation on CUDA and ROCm. Tables remain disk-only. Warm M5 Max SSD decode improved from 12.62 to 13.39 tokens/s with byte-identical output; row and error tests also pass under ASAN/UBSAN.
ds4_engram_read_batch() already fans row reads across ENGRAM_READERS
workers, with the comment "Fixed concurrency hides random-read latency
without caching the table" -- but only above a 256-row threshold.
ds41_graph_step() reads DS4_ENGRAM_COLS rows per table (48 rows for the
two tables) through the serial ds4_engram_read(), so ordinary decode
never reached that path. Those 48 preads run in series, and they run
before ds4_gpu_begin_commands(), so the GPU is idle for all of them.
Only bulk prefill ever cleared 256.
This routes the decode read through the batched entry point and lowers
the threshold to 8 so decode-sized batches engage the existing workers.
Same rows, same order, same values -- only the fetch is concurrent.
Measured on a Mac Studio M3 Ultra (80 GPU cores, 256 GB, macOS 26.6.2),
DeepSeek-V4.1-Flash-Q2, resident (no --ssd-streaming), --ctx 131072,
temperature 0, 200 generated tokens, this branch vs its merge-base:
main 19.18, 19.14 tok/s median 19.16
main + patch 20.42, 21.00, 21.11, 21.18 tok/s median 21.06
+9.9%, about 4.7 ms/token. Generated text was byte-identical between
the two builds. (The 20.42 was a cold first run after rebuild.)
The underlying cost is storage latency, not bandwidth -- a standalone
pread benchmark against the same GGUF, 48 random 264-byte rows with
F_NOCACHE and readahead disabled, 150 batches:
serial 5.177 ms/batch (107.9 us/row)
parallel 0.614 ms/batch ( 12.8 us/row)
The saving is roughly constant per token, so it is worth more on a
faster tree: on ivanfioravanti's ds41f-optimizations branch, where this
was originally profiled, the same change measured 26.8 -> 30.5 tok/s
(Engram wall 5.55 -> 1.12 ms/token) over three alternating pairs, also
byte-identical.
Credit where it is due:
- ivanfioravanti, whose V4.1 optimization work is where this was
profiled. His ds41f-dspark branch already carries a better version
of this fix, ds41_engram_parallel(), which reads both tables in a
single dispatch -- but gated to speculative sessions, so ordinary
decode still falls through to the serial loop there too. Ungating
that is probably the better long-term shape of this change.
- The oMLX / drowzeys V4.1-on-Mac work, which is what pointed at
Engram as a decode bottleneck at all. Their approach was a RAM
cache of hot rows; measured row reuse here was only 22.5% over a
600-token coding generation, worth roughly +1 tok/s, so the win
turned out to be concurrency rather than caching. Wrong idea,
right place to look.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 92a61612a2)
Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Tarjei Mandt
Salvatore Sanfilippo
Jake Maness