Salvatore Sanfilippo
df6eedd972
Share the Metal session graph with CUDA for text and vision, MTP and exact sampling, steering, checkpoints, and disk-only BF16 n-grams. Add bounded concurrent Linux n-gram reads, vectorized dense and expert kernels, register-held recurrent prefill state, and efficient small speculative batches. Keep Qwen out of DeepSeek batch dispatch and exclude disk n-grams from resident memory accounting. Use normal file-backed weight loading in diagnostics and link n-gram support into the CUDA quality scorer. Include independent kernel and memory-safety tests, mixed-prefill and recovery checks, long-context server and agent coverage, and user documentation with measured DGX Spark performance.
151 lines
6.0 KiB
Markdown
151 lines
6.0 KiB
Markdown
# DGX Spark
|
|
|
|
[README](../README.md) | [Getting started](../README.md#start-here)
|
|
|
|
Use the NVIDIA driver and CUDA development
|
|
toolkit for the machine; the build needs `nvcc` and cuBLAS.
|
|
Check that `nvidia-smi` sees the GPU before building.
|
|
|
|
## Build and run
|
|
|
|
```sh
|
|
make cuda-spark
|
|
./download_model.sh ds4f-q2
|
|
./ds4 --cuda
|
|
```
|
|
|
|
The build selects `sm_121` and enables the Blackwell-specific kernels.
|
|
Do not use `--cuda-tensor-parallel` on a single Spark.
|
|
Stop other inference services before loading a model so they do not compete
|
|
for memory. Restore any services you stopped when finished.
|
|
|
|
## DeepSeek V4.1 Flash
|
|
|
|
Q2 needs SSD streaming on one 128 GB Spark:
|
|
|
|
```sh
|
|
./download_model.sh ds41f-q2
|
|
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
|
|
--ssd-streaming --ctx 32768
|
|
```
|
|
|
|
The file is 341 GiB, including 189 GiB of disk-only Engram tables. Keep it on
|
|
a fast local SSD. The expert cache is sized automatically; leave memory for
|
|
other programs and the context. Use the same options with `ds4-agent` or
|
|
`ds4-server`.
|
|
|
|
Two Sparks can instead keep half the experts each, using
|
|
[network tensor parallelism](DISTRIBUTED.md#tensor-parallelism-between-two-sparks).
|
|
Both need the complete GGUF on their local SSD. V4.1 CUDA vision and DSpark
|
|
are not supported. The [model guide](MODELS.md#deepseek-v41-flash) covers
|
|
thinking levels and the separate Metal configurations.
|
|
|
|
September 13-14, 2026, Q2 SSD with automatic cache sizing: a 3,241-token
|
|
`/read README.md` prefill reached 93-96 t/s, up from 49-54 t/s, with 32K
|
|
allocated context. No extra flag is needed. A separate 64 GiB cache-budget
|
|
run with 64K allocated context reached 384 t/s for a 32K initial prefill and
|
|
88 t/s for a 3.2K append. These are SSD-streamed V4.1 results, not resident
|
|
V4 Flash numbers; speed depends on prompt length and the expert cache.
|
|
The automatic-cache 256-token reply test decoded about 5% slower, but
|
|
prefill plus the reply fell from about 87 to 59 seconds.
|
|
See the [QA record](../QA_BEFORE_RELEASES.md#cuda-ssd-streaming) for conditions
|
|
and longer-context measurements.
|
|
|
|
On two Sparks over RoCE, September 13 measurements averaged **21.9 t/s** for
|
|
one session: Q2, a 1K prompt, 64K allocated context and 2,048 teacher-forced
|
|
decode tokens. This does not use speculative decoding. Separately, eight
|
|
ready sessions reached about **28 aggregate t/s**, not 28 t/s per client. Decode batching
|
|
starts at five ready sessions; smaller groups run in order. Separate long
|
|
runs measured about 400 t/s for a 32K prefill and 290 t/s for an 8K append. The
|
|
[network TP QA record](../QA_BEFORE_RELEASES.md#cuda-network-tensor-parallelism)
|
|
has the timing controls and memory limits.
|
|
|
|
## Qwen3.8 Flash Next
|
|
|
|
Both Q2 and Q4 fit on one Spark. After building:
|
|
|
|
```sh
|
|
./download_model.sh qwen38-q2
|
|
./ds4 --mtp --ctx 32768
|
|
```
|
|
|
|
Q2 keeps 41.73 GiB of weights resident; Q4 keeps 69.74 GiB. Both GGUFs also
|
|
contain 95.37 GiB of BF16 n-grams, read directly from the SSD. Leave room
|
|
for context and runtime buffers. Vision, the native agent, server APIs and
|
|
text-session checkpoints work on CUDA; see [Qwen setup](QWEN38_FLASH_NEXT.md).
|
|
|
|
September 15, 2026, Q2 on one Spark: a 16K prefill reached 250 t/s, extending
|
|
it to 30K reached 243 t/s, and ordinary decode stayed around 17.3 t/s.
|
|
These are single runs with 8,192-token prefill chunks and 32 decode tokens.
|
|
A separate 256-token prose run measured 17.1 t/s ordinarily and 19.9 t/s with
|
|
MTP, at temperature zero and 8K allocated context. Speculation is opt-in;
|
|
its benefit depends on the prompt.
|
|
|
|
## GLM 5.3 Flash
|
|
|
|
Q2 is the resident target for one Spark:
|
|
|
|
```sh
|
|
./download_model.sh glm53-q2
|
|
./ds4 --cuda -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 16384
|
|
```
|
|
|
|
Q4 does not fit resident. GLM Spark-to-Spark tensor parallelism is not
|
|
implemented; the two-Mac RDMA instructions do not apply to GLM on CUDA.
|
|
|
|
## Vision and speculative decoding
|
|
|
|
DeepSeek Vision Experimental needs its matching text model and encoder:
|
|
|
|
```sh
|
|
./download_model.sh ds4f-vision-q2
|
|
./ds4 --cuda --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf
|
|
```
|
|
|
|
Use `/read image.png` in the CLI. GLM vision also works on this backend; see
|
|
[models and vision](MODELS.md#vision).
|
|
|
|
For Flash 0731, [DSpark](SPECULATIVE_DECODING.md) uses the separate 0731 support
|
|
file. Vision Experimental has a different drafter. GLM uses its built-in MTP
|
|
block with `--mtp`. None is enabled by default.
|
|
|
|
## Flash Q2 with DSpark
|
|
|
|
September 6, 2026, fully resident Flash 0731: median generation speed from
|
|
three runs of 256 tokens, 4K allocated context and a 512-token prefill chunk.
|
|
|
|
| Prompt | Temperature | Ordinary decode | DSpark |
|
|
| --- | ---: | ---: | ---: |
|
|
| C hash table | 0 | 19.72 t/s | 31.41 t/s |
|
|
| C hash table | 1 | 19.53 t/s | 29.98 t/s |
|
|
| Unpredictable prose | 1 | 19.53 t/s | 18.81 t/s |
|
|
|
|
Longer coding runs reached about 33 t/s. Poor draft acceptance can still make
|
|
DSpark slower. Temperature 1 uses the default opportunistic policy; these are
|
|
not exact-sampling results. DSpark does not accelerate prefill.
|
|
|
|
To reproduce the greedy coding case after downloading `ds4f-q2` and
|
|
`ds4f-dspark`:
|
|
|
|
```sh
|
|
./ds4 --cuda -m ds4flash.gguf \
|
|
--dspark --mtp-model gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
|
|
--ctx 4096 --prefill-chunk 512 --nothink --temp 0 --seed 12345 -n 256 \
|
|
-p 'Write a complete C hash table implementation with string keys, insert, find, delete, and a test main. Output only C code.'
|
|
```
|
|
|
|
For the temperature-1 rows, use `--temp 1 --top-p 0.95 --min-p 0.05`.
|
|
The [QA record](../QA_BEFORE_RELEASES.md#16-speed-regression) includes the previous
|
|
implementation, continued-context checks and quality comparisons.
|
|
|
|
## Larger models and serving
|
|
|
|
CUDA also has [SSD streaming](SSD_STREAMING.md) paths for larger weights.
|
|
Memory fit and speed depend on the model layout; start with Q2 for normal use.
|
|
|
|
V4.1 Q2 SSD serving batches up to eight decode rows together. An eight-session
|
|
test reached 11.0 aggregate t/s versus 8.4 with ordered execution; this is
|
|
total throughput, not speed per client. Other single-Spark model paths retain
|
|
their existing scheduling.
|
|
See [serving](SERVER.md) and the recorded [benchmarks](PERFORMANCE.md).
|