On one RTX PRO 6000, Qwen3.8-Flash-Next NVFP4 on SGLang beats Qwen3.8-27B NVFP4 without a drafter in every scenario I ran. Against the 27B with DFlash2 (8 draft tokens), the setup I run day to day, it is about half as fast at chat, ties at 40k, and wins only from 80k: 1.9x at 120k. It takes the whole card for that, 88 GB against 44 GB. Squeezed into 44 GB with llama.cpp, it runs at a fifth to a quarter of the 27B’s speed.

Results

Whole card, output tok/s, median time to first token (TTFT) in parentheses, one run per config. Bold is the fastest.

The 27B numbers are the runs from the spec-decode post, not rerun.

At the 27B’s 44 GB budget (llama.cpp, Q4 GGUF, experts of 30 of 48 layers in host RAM):

This run is partial: 80k failed (see caveats) and 120k didn’t run.

Takeaways

• Flash-Next wins on prefill, the 27B on decode. Flash-Next prefills about 2x faster at 80k and 120k (6,471 and 7,481 prompt tok/s vs 4,453 and 3,859), and its TTFT is lower at every prompt length. Its decode is flat at 10.4–10.7 ms per output token. That’s faster than the 27B without a drafter (13.9–16.5 ms) but slower than the 27B with DFlash2 (5.4–6.5 ms).
• So where it wins depends on the answer length. These requests are 512 output tokens. From the medians, the 27B with DFlash2 catches up after about 1,600 output tokens at 80k and about 4,100 at 120k. That’s my estimate (TTFT difference over per-token difference), not a measurement.
• It needs the whole card. 87.7 GB of GPU memory plus about 70 GiB of host RAM, most of it the 51B-parameter n-gram embedding table (~48 GiB in FP8) that SGLang keeps in pinned host RAM. The 27B with DFlash2 leaves half the card free.
• SGLang can’t run it in 44 GB. The flag that keeps the n-gram table in host RAM (--ple-offload-embedding) is rejected together with SGLang’s weight offload flags (--cpu-offload-gb, --offload-group-size), and KTransformers’ expert offload doesn’t support the model yet (ktransformers#2179, open on 2026-10-29). With llama.cpp, the experts in host RAM cost 3x in decode (32–33 ms per token) and about 4x in prefill (756 vs 2,835 prompt tok/s at 20k).

I’m keeping the 27B with DFlash2 as the daily driver. Most of my requests are short to medium prompts, and the model that wins only past 40k costs the other half of the card.

Setup

Hardware: same machine as the NVFP4 vs FP8 post. RTX PRO 6000 Blackwell 96 GB at a 400 W power limit, PCIe Gen 4 x16, Ryzen 9 7950X, 250 GiB DDR5, Ubuntu 24.04.2, driver 580.178.04, rootless Docker. Full list in hardware.md.

Model: Qwen/Qwen3.8-Flash-Next, 125B parameters with 6B active per token, plus a 51B n-gram embedding and a 4B MTP head. 48 layers: 36 Gated DeltaNet, 12 Qwen Sparse Attention.

Flash-Next, whole card:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261029-c10e0e17, with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1.
• Checkpoint: RadixArk/Qwen3.8-Flash-Next-NVFP4 @ 7b719225.
• No drafter. KV pool 611,520 tokens, 12 mamba slots.

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static 0.88 --max-running-requests 4 --max-mamba-cache-size 12
  --context-length 262144 --dtype bfloat16 --quantization modelopt_fp4 --tp 1
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --cuda-graph-backend-prefill disabled
  --ple-offload-embedding --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-track-interval 64
  --chunked-prefill-size 2048 --page-size 64 --kv-cache-dtype fp8_e4m3 --sleep-on-idle

Prefill chunks of 2048 and no prefill CUDA graph: with 4096-token chunks, the first 20k-token prefill ran out of GPU memory at --mem-fraction-static 0.92–0.96.

Flash-Next, 44 GB:

• Image: ghcr.io/ggml-org/llama.cpp@sha256:0283744a (b11977).
• Checkpoint: unsloth/Qwen3.8-Flash-Next-GGUF @ 766911a6, UD-Q4_K_XL (103.7 GiB).
• --n-cpu-moe 30 sets the GPU memory: 44.3 GB after load, 44.5 GB peak. Host RAM 47–64 GiB.

llama-server -m <UD-Q4_K_XL shard 1>
  -ngl 999 -ot per_layer_token_embd=CPU --n-cpu-moe 30 -fit off
  -c 153600 -kvu --parallel 4 -fa on --jinja -b 4096 -ub 4096 -t 16 -lm none

27B: RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c, with and without z-lab/Qwen3.8-27B-DFlash2 (8 draft tokens), on lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48, --mem-fraction-static 0.41, context 153,600, 64 GiB HiCache in host RAM. Flags in the spec-decode post.

Serving: python3 -m sglang.benchmark.serving --warmup-requests 2 --seed 1, reasoning_effort=medium. Backend sglang-oai-chat with --flush-cache on SGLang; vllm-chat on llama.cpp, with cache_prompt: false on every request because llama.cpp has no cache flush endpoint.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: random 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.
• 40k, 80k, 120k prompt: random input, 512 output tokens, 4 prompts at 1 user.

Caveats

• One run per config. The long scenarios have 4 prompts each, so a few percent either way is noise; the 20k, 4 users tie (134.2 vs 137.5) is a tie.
• The long prompts use SGLang’s random dataset, which repeats one ShareGPT prompt to length. It’s the same input for every config, but it isn’t a real long document.
• The 27B and Flash-Next runs used SGLang nightlies 19 days apart (2026-10-10 and 2026-10-29), and Flash-Next has no HiCache. I didn’t rerun the 27B on the newer image.
• No drafter on Flash-Next. Native MTP and incoai/Qwen3.8-Flash-Next-DFlash2 both loaded at --mem-fraction-static 0.92–0.96 and ran out of GPU memory on the first 20k prefill. I didn’t try them at 0.88. Stock SGLang also caps the Flash-Next draft block at 4 tokens. A drafter would raise the whole-card chat numbers; it can’t fix the memory.
• No quality comparison. The BFCL subset from the earlier posts scored 136/205 for Flash-Next vs 171–176 for the 27B, almost all of it in parallel (4/20) and parallel_multiple (10/20). The responses really do make one call where two are needed, but BFCL’s Qwen/Qwen3-32B-FC handler builds its own prompt instead of using the model’s chat template, so I don’t trust that as a model comparison. The scores and every response are in the repo.
• The llama.cpp run is partial. At 80k the unified KV pool (153,600 tokens) ran out because the first warmup request was still in slot 0 when the second started, and the client failed on a non-JSON stream chunk. Two 78k requests in a row on a fresh server worked. 120k and BFCL didn’t run.
• 400 W power limit, not the default 600 W. Output lengths are fixed (ignore_eos), so this measures speed only.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. The 27B results are in the spec-decode folder.
• results/fn-nospec/ (whole card) and results/fn-gguf-q4/ (44 GB): bench-serving.jsonl (one line per scenario, server args included), server.log, resources.json (GPU and host memory, startup time), and bfcl/ with every model response.

On one RTX PRO 6000 with SGLang, DFlash2 with 8 draft tokens is the fastest drafter for stock Qwen3.8-27B NVFP4 in every scenario I ran, from short chat to a 120k-token prompt. It beats the model’s own MTP head at 3 and 7 steps, DFlash2 with 4 draft tokens, and RadixArk’s DSpark. Tool calling (BFCL) doesn’t move.

The end-to-end gain drops from 2.5x on chat to 1.16x at 120k. That’s prefill, not the drafter: time per output token stays about 2.5x faster at every length.

Results

Output tok/s, one run per config. Bold is the fastest.

Mean accept length (tokens per verify step, 1 user):

Takeaways

• Acceptance doesn’t fall with context. No drafter shows a trend from chat to 120k, only swings of 0.1–0.3. Median time per output token for DFlash2 8 is 5.4–6.5 ms against 13.9–16.5 ms without a drafter, i.e. 2.5–2.7x faster at every length.
• So why does the gain shrink? Because these requests are 512 output tokens behind a long prompt. At 120k, prefill takes 27.8 s and the 512 tokens take about 3.3 s. The drafter only speeds up the 3.3 s. Output tok/s is total output over wall time, so prefill eats the gain. With longer answers the end-to-end number would sit closer to the decode number.
• More accepted tokens isn’t faster. MTP 7 accepts more per step than DFlash2 8 at 40k and 80k (3.65 and 3.79 vs 3.28 and 3.43) and is still slower everywhere, because each MTP step costs more (6.9–8.3 ms per token at 1 user vs 5.4–6.5).
• DSpark doesn’t beat DFlash2 4 here. It accepts about as many tokens as MTP 3. It also has the slowest BFCL calls of all six configs, slower than no drafter (3.24 s mean vs 2.53 s; DFlash2 8: 0.91 s). I haven’t looked into why.

DFlash2 with 8 draft tokens is what I run day to day, and nothing here changes that.

The drafter costs KV cache

Same --mem-fraction-static 0.41 for every config. The draft model and verify buffers come out of the same budget:

DFlash2 8 gives up 26% of the KV pool and 76% of the mamba slots (max_total_num_tokens and mamba_cache_size from each server.log). With 4 running requests and a 153,600-token context that’s still enough for every scenario above. I haven’t measured what it costs in multi-turn prefix cache hits.

Is the low accept length a thinking or sampling artifact?

The DFlash2 card reports 4.10 on MT-Bench with 7 draft tokens (H200, xhigh reasoning). I get 3.43 on ShareGPT at medium. So I reran DFlash2 8 with thinking off and with temperature 0:

Neither one closes the gap. The prompts (ShareGPT vs MT-Bench) and the hardware differ, so the card’s number isn’t a target I should expect to hit.

Setup

Hardware: same machine as the NVFP4 vs FP8 post. RTX PRO 6000 Blackwell 96 GB at a 400 W power limit, Ryzen 9 7950X, 250 GiB DDR5, Ubuntu 24.04.2, driver 580.178.04, rootless Docker. Full list in hardware.md.

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48 for every config (it has both DFLASH and DSPARK).
• Target: RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c.
• Drafters: z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c, RadixArk/Qwen3.8-27B-DSpark @ b9a5dbdf (DSpark v2). MTP uses the head built into the target checkpoint.
• Requests use reasoning_effort=medium.

Shared flags, the same as the first post:

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static 0.41 --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Then the speculative flags, appended after the shared ones:

Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: random 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.
• 40k, 80k, 120k prompt: random input, 512 output tokens, 4 prompts at 1 user.

BFCL: bfcl-eval 2026.3.23, the Qwen/Qwen3-32B-FC handler, the same fixed 205-case subset as the first post (case ids).

Caveats

• One run per config. The 40k–120k scenarios have 4 prompts each, so their rates carry a few percent of noise. BFCL categories have 10–25 cases, so 171 vs 176 is noise.
• DSpark runs on the shared flags so only the drafter changes. Its model card serves it with float32 mamba state, --mamba-radix-cache-strategy extra_buffer, --mamba-full-memory-ratio 11.93 and 32k prefill chunks. With those it might do better than here.
• 400 W power limit, not the default 600 W.
• Output lengths are fixed (ignore_eos), so this measures speed only.
• Deep-AI-Evo found MTP slower than no MTP at long context on the same card: 18.2 vs 43.7 tok/s decode at ~177k tokens, NVFP4, MTP k=2 (repo). I don’t see that up to 120k: MTP 3 keeps decode at 8.5 ms per token vs 16.5 without a drafter. Their stack is vLLM (0.21 and 0.26) with unsloth/Qwen3.8-27B-NVFP4, and I stopped at 120k, so this doesn’t contradict them.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh reruns every config, about two hours.
• results/<label>/ per config: bench-serving.jsonl (one line per scenario, full server args included), server.log, and bfcl/ with every model response.

On one RTX PRO 6000 with SGLang and DFlash2 speculative decoding, NVFP4 is 32–37% faster than FP8 for stock Qwen3.8-27B, on short chat and on 20k-token prompts. BFCL tool calling stays within noise (174 vs 167 of 205).

Results

Stock Qwen3.8-27B: Qwen/Qwen3.8-27B-FP8 vs RadixArk/Qwen3.8-27B-NVFP4.

Takeaways

• The gain holds with a 20k-token prompt. Time to first token drops 27–32%, so prefill gets faster as well as decode.
• DFlash2 acceptance barely moves (accept length 3.49 vs 3.41), so the gain comes from the target model, not the drafter. Median time per output token in chat at 1 user drops from 7.30 ms to 5.59 ms.
• BFCL shows no measurable change. Most of the +7 is irrelevance (12 → 18) and multi_turn_base (14 → 16), categories of 20 cases each.

BFCL by category

Setup

Hardware and system:

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB. PCIe Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum). Power limit 400 W, set at boot (default 600 W). Idle during runs, one config at a time.
• CPU: AMD Ryzen 9 7950X (16 cores, 32 threads).
• RAM: DDR5, 250 GiB usable.
• Board: Gigabyte B650 AORUS ELITE AX.
• Models on a Crucial P3 Plus 4 TB NVMe (ext4 on LUKS).
• OS: Ubuntu 24.04.2, kernel 6.8.0-147, NVIDIA driver 580.178.04 (CUDA 13.0), rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1.

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: Qwen/Qwen3.8-27B-FP8 @ 017b9c7a, RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c.
• Speculative decoding: DFlash2 (z-lab/Qwen3.8-27B-DFlash2), 8 draft tokens, for every run.
• --mem-fraction-static 0.50 for FP8, 0.41 for NVFP4, so both get about the same KV pool (240,512 vs 236,864 tokens, 28 mamba slots each).
• Requests use reasoning_effort=medium.

Launch flags:

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static <0.50|0.41> --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --speculative-algorithm DFLASH --speculative-draft-model-path <DFlash2>
  --speculative-num-draft-tokens 8
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: random 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.

BFCL: bfcl-eval 2026.3.23, the Qwen/Qwen3-32B-FC handler, a fixed 205-case subset across 10 categories (case ids).

Caveats

• One run per config. Categories have 10–25 cases, so a few points either way is noise.
• The GPU ran at a 400 W power limit, not the default 600 W. I haven’t measured either format at 600 W.
• Output lengths are fixed (ignore_eos), so tok/s compares speed only, not how much each model thinks.
• BFCL uses its own Qwen3 FC prompt, not the model’s chat template. Compare these scores with each other, not with the public leaderboard.
• Deep-AI-Evo measured a smaller gap on the same card with vLLM 0.21 and unsloth/Qwen3.8-27B-NVFP4: 50.9 → 56.4 tok/s without MTP, 115.8 → 132.3 with MTP k=2, single stream (repo). The engine, checkpoint and speculative decoding all differ from mine, and I haven’t tested which of them explains the difference.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns both configs.
• Serving results (one JSON line per scenario, including full server args): FP8, NVFP4.
• BFCL scores and every model response: results/<label>/bfcl/ in the same folder.