Mark Shank e1d8114167 Encode the next V4.1 decode token while this one finishes ...

At the end of each decode step the CPU waits for the GPU to finish the last command buffer. It now uses that wait to encode the next token's first command buffer (embedding, RoPE table and layer 0) ahead of time. That buffer reads the token id from GPU memory, so it can be encoded before the token is known. The next step submits it if its inputs are unchanged (same graph and position, and no other pass over the graph in between) and throws it away otherwise. A small kernel queued behind it keeps the GPU from going idle between the two steps. Sessions with a DSpark drafter don't use it, because they draft between steps.

The kernels, their order and the command buffer boundaries are the same as without it, so the results are bit-identical. A test that needs the model file checks this across two sessions, a prompt append and a session restore.

DS4_METAL_DISABLE_V41_PREENCODE=1 turns it off.

2026-10-21 15:50:18 -04:00

..

test-vectors

Add DeepSeek V4 Flash vision support

2026-09-01 00:11:58 +02:00

vision-fixtures

Fix progressive JPEG AC refinement ZRL handling in Iris decoder

2026-09-18 09:53:23 +02:00

bench_mxfp4_rocm.c

MXFP4 prefill MoE: L2-friendly grid order + wider A-staging (r9)

2026-08-20 18:45:13 -07:00

compare_glm53_vision_embeddings.py

Fix GLM attention masking and reductions across GPU backends

2026-09-05 19:42:01 +02:00

cuda_long_context_smoke.c

cuda: warm up top-k regression timing

2026-06-02 09:07:28 +02:00

deepseek41_dspark_reference.py

Enable V4.1 DSpark on Metal, CUDA and ROCm

2026-10-09 15:07:43 +02:00

ds4_agent_terminal_test.py

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00

ds4_agent_test.c

Add optimized CUDA inference for Qwen3.8 Flash Next

2026-09-15 15:00:11 +02:00

ds4_test.c

Fix compressed-key causality in Metal visual attention

2026-09-26 15:11:33 +02:00

dspark_acceptance_fixture.sh

Record backend QA results and preserve failure diagnostics

2026-09-05 19:42:02 +02:00

generate_long_context_story_prompt.py

Add long-context story recall regression

2026-05-13 19:11:16 +02:00

glm_long_context_smoke.sh

Unify distributed inference CLI and documentation

2026-07-18 18:50:24 +02:00

long_context_security_prompt.txt

DS4 initial release

2026-05-07 17:16:28 +02:00

long_context_story_prompt.txt

Add long-context story recall regression

2026-05-13 19:11:16 +02:00

qwen4_vision_ref.py

Add optimized CUDA inference for Qwen3.8 Flash Next

2026-09-15 15:00:11 +02:00

run_glm53_vision_quality.py

Add DeepSeek V4 Flash vision support

2026-09-01 00:11:58 +02:00

test_agent_compaction.py

Read Qwen native BF16 n-grams directly from the GGUF

2026-09-14 13:32:48 +02:00

test_agent_vision.py

Read Qwen native BF16 n-grams directly from the GGUF

2026-09-14 13:32:48 +02:00

test_collect_official.py

Add official Qwen quality references and validated API scoring

2026-09-15 15:00:11 +02:00

test_compare_glm53_vision_embeddings.py

Fix GLM attention masking and reductions across GPU backends

2026-09-05 19:42:01 +02:00

test_cuda_attn_side.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_compressor.c

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_copy_spans.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_dspark_tp_markov.c

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_dspark_tp_split.c

Optimize Spark attention and support native MXFP4 DSpark experts

2026-10-05 14:17:41 +02:00

test_cuda_dspark.c

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_embed.c

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_cuda_f16_chunks.cu

Optimize Spark attention and support native MXFP4 DSpark experts

2026-10-05 14:17:41 +02:00

test_cuda_grouped_q8.cu

Use exact Q8 matrix kernels for CUDA TP prefill slices

2026-10-08 11:38:32 +02:00

test_cuda_hc_norm.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_hc_pre.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_indexed_one.cu

Optimize Spark attention and support native MXFP4 DSpark experts

2026-10-05 14:17:41 +02:00

test_cuda_indexer.cu

Optimize Spark attention and support native MXFP4 DSpark experts

2026-10-05 14:17:41 +02:00

test_cuda_mixed_batch.c

DeepSeek v4.1 Flash support for CUDA

2026-09-13 17:32:48 +02:00

test_cuda_mxfp4_split.c

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_pool.cu

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_cuda_q8_0_gemv.cu

Optimize Spark attention and support native MXFP4 DSpark experts

2026-10-05 14:17:41 +02:00

test_cuda_q8_pair.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_q8_rows.c

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_cuda_q8_scratch.cu

Use aligned Q8 vector kernels for small Spark batches

2026-09-06 00:42:59 +02:00

test_cuda_reductions.cu

DeepSeek v4.1 Flash support for CUDA

2026-09-13 17:32:48 +02:00

test_cuda_router_stream.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_session_batch.c

DeepSeek v4.1 Flash support for CUDA

2026-09-13 17:32:48 +02:00

test_cuda_shared_side.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_shared.cu

Balance V4.1 shared experts across CUDA TP peers

2026-10-08 13:12:38 +02:00

test_cuda_ssd_batch.c

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_cuda_ssd_cache.c

Overlap CUDA SSD expert reads with V4.1 prefill

2026-09-14 00:00:54 +02:00

test_cuda_ssd_repack.cu

DeepSeek v4.1 Flash support for CUDA

2026-09-13 17:32:48 +02:00

test_cuda_tokentile.c

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_cuda_topk_stream.cu

Optimize Spark attention and support native MXFP4 DSpark experts

2026-10-05 14:17:41 +02:00

test_cuda_tp_link.c

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_cuda_tp_repack.cu

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_cuda_tp.c

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_cuda_v41_fusion.cu

Fuse V4.1 CUDA normalization and HC BF16 stores

2026-10-08 12:43:22 +02:00

test_cuda_verify_head.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_deepseek4_vision_image.c

Add DeepSeek V4 Flash vision support

2026-09-01 00:11:58 +02:00

test_deepseek41_cli.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek41_conversion.py

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek41_dspark_conversion.py

Allow DSpark conversion from only the native draft shards

2026-10-09 13:18:45 +02:00

test_deepseek41_dspark.c

Enable DeepSeek V4.1 DSpark over Metal and CUDA TP

2026-10-10 13:10:32 +02:00

test_deepseek41_gguf.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek41_graph.c

Merge remote-tracking branch 'origin/main' into ds41-perf

2026-10-20 09:57:24 +11:00

test_deepseek41_http.py

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek41_manifest.py

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek41_metal.c

Speed up the V4.1 decode top-512

2026-10-21 15:50:18 -04:00

test_deepseek41_preencode.c

Encode the next V4.1 decode token while this one finishes

2026-10-21 15:50:18 -04:00

test_deepseek41_prefill.c

Process medium CUDA SSD appends in one layer sweep

2026-09-14 00:01:08 +02:00

test_deepseek41_prompt.py

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek41_rope.py

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek41_vision_conversion.py

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek41_vision_image.py

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_deepseek_indexer.c

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_dspark_confidence_input.py

Optimize Spark attention and support native MXFP4 DSpark experts

2026-10-05 14:17:41 +02:00

test_dspark_eos_contract.py

Avoid full context rebuilds at DSpark response boundaries

2026-10-07 21:43:05 +02:00

test_dspark_rewind.py

Avoid full context rebuilds at DSpark response boundaries

2026-10-07 21:43:05 +02:00

test_dspark_window.py

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_engine_correctness.c

Add CUDA tensor parallelism and session batching

2026-07-18 18:50:03 +02:00

test_engine_mgpu_placement.c

Add optimized CUDA inference for Qwen3.8 Flash Next

2026-09-15 15:00:11 +02:00

test_engine_mgpu_refusal.c

Make release builds warning-free

2026-07-20 11:09:28 +02:00

test_engine_mgpu_runtime.c

Add CUDA tensor parallelism and session batching

2026-07-18 18:50:03 +02:00

test_engram.c

Overlap V4.1 disk n-gram reads with CUDA decode

2026-10-08 13:12:15 +02:00

test_glm53_kda.c

Merge GLM 5.3 SSD streaming and full checkpoint support

2026-08-31 13:25:05 +02:00

test_glm53_vision_engine.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_glm53_vision_prompt.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_glm_attention.c

Test GLM pooled decode padding against an independent reference

2026-09-18 09:48:57 +02:00

test_glm_reference_render.py

Add long GLM FP8 references with reasoning-aware scoring

2026-09-05 20:26:22 +02:00

test_glm_rope_prefill.py

Preserve RoPE in full GLM compact prefill

2026-09-06 12:00:03 +02:00

test_gpu_args_cli.sh

Extend ds4-eval with hard capability suite

2026-09-03 15:28:24 -07:00

test_gpu_args.c

Add CUDA tensor parallelism and session batching

2026-07-18 18:50:03 +02:00

test_gpu_lookup_cache_strict.c

Add CUDA tensor parallelism and session batching

2026-07-18 18:50:03 +02:00

test_gpu_model_cache.c

Add CUDA tensor parallelism and session batching

2026-07-18 18:50:03 +02:00

test_gpu_xdev.c

Add CUDA tensor parallelism and session batching

2026-07-18 18:50:03 +02:00

test_image_decode.c

Make JPEG regression failure diagnostics safe

2026-09-18 09:54:34 +02:00

test_jpeg_decode.py

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00

test_layer_pack.c

Add CUDA tensor parallelism and session batching

2026-07-18 18:50:03 +02:00

test_linux_memory.c

Budget ROCm allocations against usable pinned memory

2026-09-05 19:41:28 +02:00

test_metal_command_memory.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_metal_dense_mpp.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_metal_moe_prefill.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_metal_session_batch.c

Harden Qwen Metal batch integration and quality checks

2026-09-16 14:14:36 +02:00

test_metal_ssd_experts.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_metal_tp_bulk.c

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_metal_tp_cancel.c

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_metal_tp_spec.c

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_mmvq.cu

Repair DSpark and ROCm regression tests

2026-10-06 17:48:04 +02:00

test_model_download.py

Document and download self-contained Qwen BF16 n-gram releases

2026-09-14 13:46:19 +02:00

test_mxfp4_cuda.cu

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_mxfp4_dot.c

Add MXFP4 GGUF format and scalar reference

2026-08-01 09:50:29 +02:00

test_mxfp4_metal.c

metal: make exact fast paths automatic and trim experiments

2026-08-09 17:48:01 +02:00

test_mxfp4_rocm.c

MXFP4 prefill MoE occupancy variants (env-gated, bit-exact, hw-verified)

2026-08-20 18:45:13 -07:00

test_mxfp4_staged_cuda.cu

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_prompt_prefix.c

Add conversation prefix conditioning

2026-09-03 01:25:25 +02:00

test_q4k_dot.c

Keep Q4K regression test warning-free

2026-08-04 13:57:21 +02:00

test_q8_prefill_variants.c

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00

test_quality_api.c

Score API top choices independently of alternative order

2026-10-06 18:35:04 +02:00

test_qwen4_checkpoint_replay.py

Read Qwen native BF16 n-grams directly from the GGUF

2026-09-14 13:32:48 +02:00

test_qwen4_cli_vision.py

Read Qwen native BF16 n-grams directly from the GGUF

2026-09-14 13:32:48 +02:00

test_qwen4_conv_parallel.c

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00

test_qwen4_kernels.c

Keep Metal-only Qwen batch tests out of the CUDA suite

2026-09-19 20:23:37 +02:00

test_qwen4_logit_dump.py

Add optimized CUDA inference for Qwen3.8 Flash Next

2026-09-15 15:00:11 +02:00

test_qwen4_moe_mm_specialize.c

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00

test_qwen4_mtp_limits.py

Read Qwen native BF16 n-grams directly from the GGUF

2026-09-14 13:32:48 +02:00

test_qwen4_ngram_state.c

Serve Qwen batched MTP with safe session state and output limits

2026-09-16 15:17:25 +02:00

test_qwen4_ngrams.c

Add optimized CUDA inference for Qwen3.8 Flash Next

2026-09-15 15:00:11 +02:00

test_qwen4_prefill.c

Add optimized CUDA inference for Qwen3.8 Flash Next

2026-09-15 15:00:11 +02:00

test_qwen4_steering.py

Honor steering in standalone greedy Qwen generation

2026-09-14 13:41:49 +02:00

test_qwen4_vision_ref.py

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00

test_qwen4_vision.c

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00

test_rocm_memory.cu

Budget ROCm allocations against usable pinned memory

2026-09-05 19:41:28 +02:00

test_sampling.c

dspark: support exact stochastic speculative decoding

2026-08-23 15:05:13 +02:00

test_serve_concurrency_bench.py

Reject incomplete concurrency benchmarks and oversized requests

2026-09-16 15:02:09 +02:00

test_server_batching.py

Serve Qwen batched MTP with safe session state and output limits

2026-09-16 15:17:25 +02:00

test_server_speculative_stop.py

Avoid full context rebuilds at DSpark response boundaries

2026-10-07 21:43:05 +02:00

test_server_story.py

Add optimized CUDA inference for Qwen3.8 Flash Next

2026-09-15 15:00:11 +02:00

test_server_vision_agent.py

DeepSeek v4.1 Flash support for Metal

2026-09-12 11:47:54 +02:00

test_server_vision_cache.py

Preserve live vision context across server tool calls

2026-09-08 13:57:09 +02:00

test_server_vision_live_state.py

Validate image positions before token-prefix slot reuse

2026-09-18 23:27:51 +02:00

test_session_state.c

Distribute and accelerate Flash speculative decoding on Spark

2026-10-05 14:17:41 +02:00

test_ssd_cache.c

Use more of the safe Metal working set for DeepSeek expert caching

2026-09-06 12:19:01 +02:00

test_tp_commands.c

Avoid full context rebuilds at DSpark response boundaries

2026-10-07 21:43:05 +02:00

test_tp_link.c

Enable DeepSeek V4.1 DSpark over Metal and CUDA TP

2026-10-10 13:10:32 +02:00

test_tp_rdma.c

Stop tensor-parallel inference after a transport failure

2026-10-05 14:17:41 +02:00

test_tp_tcp.c

Add two-host CUDA tensor parallelism for DeepSeek V4 Flash

2026-10-05 14:17:41 +02:00

test_web_recovery.c

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00

test_web_recovery.py

Add Qwen3.8 Flash Next model support

2026-09-13 14:57:29 +02:00