On one RTX PRO 6000 with SGLang and DFlash2 speculative decoding, NVFP4 is 32–37% faster than FP8 for stock Qwen3.8-27B, on short chat and on 20k-token prompts. BFCL tool calling stays within noise (174 vs 167 of 205).

Results

Stock Qwen3.8-27B: Qwen/Qwen3.8-27B-FP8 vs RadixArk/Qwen3.8-27B-NVFP4.

Takeaways

• The gain holds with a 20k-token prompt. Time to first token drops 27–32%, so prefill gets faster as well as decode.
• DFlash2 acceptance barely moves (accept length 3.49 vs 3.41), so the gain comes from the target model, not the drafter. Median time per output token in chat at 1 user drops from 7.30 ms to 5.59 ms.
• BFCL shows no measurable change. Most of the +7 is irrelevance (12 → 18) and multi_turn_base (14 → 16), categories of 20 cases each.

BFCL by category

Setup

Hardware and system:

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB. PCIe Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum). Power limit 400 W, set at boot (default 600 W). Idle during runs, one config at a time.
• CPU: AMD Ryzen 9 7950X (16 cores, 32 threads).
• RAM: DDR5, 250 GiB usable.
• Board: Gigabyte B650 AORUS ELITE AX.
• Models on a Crucial P3 Plus 4 TB NVMe (ext4 on LUKS).
• OS: Ubuntu 24.04.2, kernel 6.8.0-147, NVIDIA driver 580.178.04 (CUDA 13.0), rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1.

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: Qwen/Qwen3.8-27B-FP8 @ 017b9c7a, RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c.
• Speculative decoding: DFlash2 (z-lab/Qwen3.8-27B-DFlash2), 8 draft tokens, for every run.
• --mem-fraction-static 0.50 for FP8, 0.41 for NVFP4, so both get about the same KV pool (240,512 vs 236,864 tokens, 28 mamba slots each).
• Requests use reasoning_effort=medium.

Launch flags:

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static <0.50|0.41> --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --speculative-algorithm DFLASH --speculative-draft-model-path <DFlash2>
  --speculative-num-draft-tokens 8
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: random 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.

BFCL: bfcl-eval 2026.3.23, the Qwen/Qwen3-32B-FC handler, a fixed 205-case subset across 10 categories (case ids).

Caveats

• One run per config. Categories have 10–25 cases, so a few points either way is noise.
• The GPU ran at a 400 W power limit, not the default 600 W. I haven’t measured either format at 600 W.
• Output lengths are fixed (ignore_eos), so tok/s compares speed only, not how much each model thinks.
• BFCL uses its own Qwen3 FC prompt, not the model’s chat template. Compare these scores with each other, not with the public leaderboard.
• Deep-AI-Evo measured a smaller gap on the same card with vLLM 0.21 and unsloth/Qwen3.8-27B-NVFP4: 50.9 → 56.4 tok/s without MTP, 115.8 → 132.3 with MTP k=2, single stream (repo). The engine, checkpoint and speculative decoding all differ from mine, and I haven’t tested which of them explains the difference.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns both configs.
• Serving results (one JSON line per scenario, including full server args): FP8, NVFP4.
• BFCL scores and every model response: results/<label>/bfcl/ in the same folder.