NVFP4 is 32–37% faster than FP8 on Qwen3.8-27B

On one RTX PRO 6000 with SGLang and DFlash2 speculative decoding, NVFP4 is 32–37% faster than FP8 for stock Qwen3.8-27B, on short chat and on 20k-token prompts. BFCL tool calling stays within noise (174 vs 167 of 205).

Results

Stock Qwen3.8-27B: Qwen/Qwen3.8-27B-FP8 vs RadixArk/Qwen3.8-27B-NVFP4.

Scenario	FP8	NVFP4	Δ
Chat, 1 user, out tok/s	133.8	176.3	+32%
Chat, 4 users, out tok/s	449.9	596.5	+33%
20k prompt, 1 user, out tok/s	73.3	98.5	+34%
20k prompt, 4 users, out tok/s	104.1	142.3	+37%
20k prompt, 1 user, TTFT p50	3.20 s	2.34 s	−27%
20k prompt, 4 users, TTFT p50	8.48 s	5.78 s	−32%
DFlash2 accept length, chat 1 user	3.49	3.41	−2%
BFCL subset, correct	167/205	174/205	+7
BFCL mean latency per call	1.22 s	0.92 s	−25%

NVFP4 output tok/s gain over FP8, per scenario

Takeaways

The gain holds with a 20k-token prompt. Time to first token drops 27–32%, so prefill gets faster as well as decode.
DFlash2 acceptance barely moves (accept length 3.49 vs 3.41), so the gain comes from the target model, not the drafter. Median time per output token in chat at 1 user drops from 7.30 ms to 5.59 ms.
BFCL shows no measurable change. Most of the +7 is irrelevance (12 → 18) and multi_turn_base (14 → 16), categories of 20 cases each.

BFCL by category

Category	FP8	NVFP4
simple_python	25/25	25/25
multiple	24/25	23/25
parallel	18/20	18/20
parallel_multiple	19/20	19/20
live_simple	18/20	18/20
live_multiple	20/25	19/25
multi_turn_base	14/20	16/20
multi_turn_miss_param	3/10	4/10
irrelevance	12/20	18/20
live_irrelevance	14/20	14/20
Total	167	174

Setup

Hardware and system:

GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB. PCIe Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum). Power limit 400 W, set at boot (default 600 W). Idle during runs, one config at a time.
CPU: AMD Ryzen 9 7950X (16 cores, 32 threads).
RAM: DDR5, 250 GiB usable.
Board: Gigabyte B650 AORUS ELITE AX.
Models on a Crucial P3 Plus 4 TB NVMe (ext4 on LUKS).
OS: Ubuntu 24.04.2, kernel 6.8.0-147, NVIDIA driver 580.178.04 (CUDA 13.0), rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1.

Serving stack:

Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
Checkpoints: Qwen/Qwen3.8-27B-FP8 @ 017b9c7a, RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c.
Speculative decoding: DFlash2 (z-lab/Qwen3.8-27B-DFlash2), 8 draft tokens, for every run.
--mem-fraction-static 0.50 for FP8, 0.41 for NVFP4, so both get about the same KV pool (240,512 vs 236,864 tokens, 28 mamba slots each).
Requests use reasoning_effort=medium.

Launch flags:

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static <0.50|0.41> --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --speculative-algorithm DFLASH --speculative-draft-model-path <DFlash2>
  --speculative-num-draft-tokens 8
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1.

Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
20k prompt: random 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.

BFCL: bfcl-eval 2026.3.23, the Qwen/Qwen3-32B-FC handler, a fixed 205-case subset across 10 categories (case ids).

Caveats

One run per config. Categories have 10–25 cases, so a few points either way is noise.
The GPU ran at a 400 W power limit, not the default 600 W. I haven’t measured either format at 600 W.
Output lengths are fixed (ignore_eos), so tok/s compares speed only, not how much each model thinks.
BFCL uses its own Qwen3 FC prompt, not the model’s chat template. Compare these scores with each other, not with the public leaderboard.
Deep-AI-Evo measured a smaller gap on the same card with vLLM 0.21 and unsloth/Qwen3.8-27B-NVFP4: 50.9 → 56.4 tok/s without MTP, 115.8 → 132.3 with MTP k=2, single stream (repo). The engine, checkpoint and speculative decoding all differ from mine, and I haven’t tested which of them explains the difference.

Raw data

Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns both configs.
Serving results (one JSON line per scenario, including full server args): FP8, NVFP4.
BFCL scores and every model response: results/<label>/bfcl/ in the same folder.