On one RTX PRO 6000, Qwen3.8-Flash-Next NVFP4 on SGLang beats Qwen3.8-27B NVFP4 without a drafter in every scenario I ran. Against the 27B with DFlash2 (8 draft tokens), the setup I run day to day, it is about half as fast at chat, ties at 40k, and wins only from 80k: 1.9x at 120k. It takes the whole card for that, 88 GB against 44 GB. Squeezed into 44 GB with llama.cpp, it runs at a fifth to a quarter of the 27B’s speed.

Results

Whole card, output tok/s, median time to first token (TTFT) in parentheses, one run per config. Bold is the fastest.

The 27B numbers are the runs from the spec-decode post, not rerun.

At the 27B’s 44 GB budget (llama.cpp, Q4 GGUF, experts of 30 of 48 layers in host RAM):

This run is partial: 80k failed (see caveats) and 120k didn’t run.

Takeaways

• Flash-Next wins on prefill, the 27B on decode. Flash-Next prefills about 2x faster at 80k and 120k (6,471 and 7,481 prompt tok/s vs 4,453 and 3,859), and its TTFT is lower at every prompt length. Its decode is flat at 10.4–10.7 ms per output token. That’s faster than the 27B without a drafter (13.9–16.5 ms) but slower than the 27B with DFlash2 (5.4–6.5 ms).
• So where it wins depends on the answer length. These requests are 512 output tokens. From the medians, the 27B with DFlash2 catches up after about 1,600 output tokens at 80k and about 4,100 at 120k. That’s my estimate (TTFT difference over per-token difference), not a measurement.
• It needs the whole card. 87.7 GB of GPU memory plus about 70 GiB of host RAM, most of it the 51B-parameter n-gram embedding table (~48 GiB in FP8) that SGLang keeps in pinned host RAM. The 27B with DFlash2 leaves half the card free.
• SGLang can’t run it in 44 GB. The flag that keeps the n-gram table in host RAM (--ple-offload-embedding) is rejected together with SGLang’s weight offload flags (--cpu-offload-gb, --offload-group-size), and KTransformers’ expert offload doesn’t support the model yet (ktransformers#2179, open on 2026-10-29). With llama.cpp, the experts in host RAM cost 3x in decode (32–33 ms per token) and about 4x in prefill (756 vs 2,835 prompt tok/s at 20k).

I’m keeping the 27B with DFlash2 as the daily driver. Most of my requests are short to medium prompts, and the model that wins only past 40k costs the other half of the card.

Setup

Hardware: same machine as the NVFP4 vs FP8 post. RTX PRO 6000 Blackwell 96 GB at a 400 W power limit, PCIe Gen 4 x16, Ryzen 9 7950X, 250 GiB DDR5, Ubuntu 24.04.2, driver 580.178.04, rootless Docker. Full list in hardware.md.

Model: Qwen/Qwen3.8-Flash-Next, 125B parameters with 6B active per token, plus a 51B n-gram embedding and a 4B MTP head. 48 layers: 36 Gated DeltaNet, 12 Qwen Sparse Attention.

Flash-Next, whole card:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261029-c10e0e17, with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1.
• Checkpoint: RadixArk/Qwen3.8-Flash-Next-NVFP4 @ 7b719225.
• No drafter. KV pool 611,520 tokens, 12 mamba slots.

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static 0.88 --max-running-requests 4 --max-mamba-cache-size 12
  --context-length 262144 --dtype bfloat16 --quantization modelopt_fp4 --tp 1
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --cuda-graph-backend-prefill disabled
  --ple-offload-embedding --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-track-interval 64
  --chunked-prefill-size 2048 --page-size 64 --kv-cache-dtype fp8_e4m3 --sleep-on-idle

Prefill chunks of 2048 and no prefill CUDA graph: with 4096-token chunks, the first 20k-token prefill ran out of GPU memory at --mem-fraction-static 0.92–0.96.

Flash-Next, 44 GB:

• Image: ghcr.io/ggml-org/llama.cpp@sha256:0283744a (b11977).
• Checkpoint: unsloth/Qwen3.8-Flash-Next-GGUF @ 766911a6, UD-Q4_K_XL (103.7 GiB).
• --n-cpu-moe 30 sets the GPU memory: 44.3 GB after load, 44.5 GB peak. Host RAM 47–64 GiB.

llama-server -m <UD-Q4_K_XL shard 1>
  -ngl 999 -ot per_layer_token_embd=CPU --n-cpu-moe 30 -fit off
  -c 153600 -kvu --parallel 4 -fa on --jinja -b 4096 -ub 4096 -t 16 -lm none

27B: RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c, with and without z-lab/Qwen3.8-27B-DFlash2 (8 draft tokens), on lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48, --mem-fraction-static 0.41, context 153,600, 64 GiB HiCache in host RAM. Flags in the spec-decode post.

Serving: python3 -m sglang.benchmark.serving --warmup-requests 2 --seed 1, reasoning_effort=medium. Backend sglang-oai-chat with --flush-cache on SGLang; vllm-chat on llama.cpp, with cache_prompt: false on every request because llama.cpp has no cache flush endpoint.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: random 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.
• 40k, 80k, 120k prompt: random input, 512 output tokens, 4 prompts at 1 user.

Caveats

• One run per config. The long scenarios have 4 prompts each, so a few percent either way is noise; the 20k, 4 users tie (134.2 vs 137.5) is a tie.
• The long prompts use SGLang’s random dataset, which repeats one ShareGPT prompt to length. It’s the same input for every config, but it isn’t a real long document.
• The 27B and Flash-Next runs used SGLang nightlies 19 days apart (2026-10-10 and 2026-10-29), and Flash-Next has no HiCache. I didn’t rerun the 27B on the newer image.
• No drafter on Flash-Next. Native MTP and incoai/Qwen3.8-Flash-Next-DFlash2 both loaded at --mem-fraction-static 0.92–0.96 and ran out of GPU memory on the first 20k prefill. I didn’t try them at 0.88. Stock SGLang also caps the Flash-Next draft block at 4 tokens. A drafter would raise the whole-card chat numbers; it can’t fix the memory.
• No quality comparison. The BFCL subset from the earlier posts scored 136/205 for Flash-Next vs 171–176 for the 27B, almost all of it in parallel (4/20) and parallel_multiple (10/20). The responses really do make one call where two are needed, but BFCL’s Qwen/Qwen3-32B-FC handler builds its own prompt instead of using the model’s chat template, so I don’t trust that as a model comparison. The scores and every response are in the repo.
• The llama.cpp run is partial. At 80k the unified KV pool (153,600 tokens) ran out because the first warmup request was still in slot 0 when the second started, and the client failed on a non-JSON stream chunk. Two 78k requests in a row on a fresh server worked. 120k and BFCL didn’t run.
• 400 W power limit, not the default 600 W. Output lengths are fixed (ignore_eos), so this measures speed only.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. The 27B results are in the spec-decode folder.
• results/fn-nospec/ (whole card) and results/fn-gguf-q4/ (44 GB): bench-serving.jsonl (one line per scenario, server args included), server.log, resources.json (GPU and host memory, startup time), and bfcl/ with every model response.