On one RTX PRO 6000 with SGLang and DFlash2 speculative decoding, the RadixArk NVFP4 checkpoint of Qwen3.8-27B produced 32–37% more output tokens per benchmark second than Qwen’s FP8 checkpoint, on short chat and on 20k-token prompts. BFCL (Berkeley Function Calling Leaderboard) function-call graders passed 174 of 205 subset cases on NVFP4 and 167 on FP8, one run each; that gap doesn’t establish a gain or equivalence. This compares these two checkpoints and their recipes, not every NVFP4 against every FP8. Both ran with an FP8 E4M3 KV cache.

Results

Checkpoints compared: Qwen/Qwen3.8-27B-FP8 vs RadixArk/Qwen3.8-27B-NVFP4. Different quantization recipes from different publishers; both use an FP8 E4M3 KV cache.

Takeaways

• At 1 user, time to first token on a 20k prompt drops 27% (3.20 → 2.34 s), consistent with faster prefill, though this is median TTFT over four requests, not a separately timed prefill. At 4 users it drops 32%, but four 20k prompts prefill at the same time, so that number includes waiting on the other three.
• On the 20k rows, tok/s is output tokens over the whole request, prefill included. Median time per output token drops 31% at 1 user (7.77 → 5.38 ms) and 14% at 4 users (17.07 → 14.65 ms), so the 37% at 4 users is partly prefill.
• DFlash2 accept length barely moves: within 3% in all four scenarios (chat, 1 user: 3.49 vs 3.41). The gain isn’t from better drafting. Median time per output token in chat at 1 user drops from 7.30 ms to 5.59 ms.
• BFCL is 167/205 vs 174/205 on this subset in one run. A seven-case gap, from one run each, is not enough to establish a gain or equivalence. The +7 is irrelevance (12 → 18) and multi_turn_base (14 → 16), 20 cases each. Everything else nets −1.
• BFCL mean latency was 1.22 s per call on FP8 and 0.92 s on NVFP4. The runs took different multi-turn paths (544 vs 529 model calls; 2.50M vs 2.37M input tokens), so this is not a controlled speed comparison. One NVFP4 case (multi_turn_base_60) hit an inference error and counts as a failure.

BFCL by category

Setup

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: Qwen/Qwen3.8-27B-FP8 @ 017b9c7a, RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c.
• Speculative decoding: DFlash2 (z-lab/Qwen3.8-27B-DFlash2), 8 draft tokens, for every run.
• --mem-fraction-static 0.50 for FP8, 0.41 for NVFP4, so both get about the same KV pool (240,512 vs 236,864 tokens, 28 mamba slots each).
• Speed requests use reasoning_effort=medium. BFCL does not: it runs the handler’s default effort at temperature 0.001.
• KV cache: --kv-cache-dtype fp8_e4m3 for both.

Launch flags:

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static <0.50|0.41> --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --speculative-algorithm DFLASH --speculative-draft-model-path <DFlash2>
  --speculative-num-draft-tokens 8
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: SGLang random dataset (one ShareGPT prompt repeated to length), 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.

BFCL (Berkeley Function Calling Leaderboard V4): bfcl-eval 2026.3.23, the Qwen/Qwen3-32B-FC handler, default effort, temperature 0.001, scored by BFCL’s own graders (scores are grader passes, not human judgments), a fixed 205-case subset across 10 categories (case ids).

Caveats

• One run per config, with 4–32 prompts per speed scenario and no variance reported. The gain is consistent across all four scenarios, but I can’t put an error bar on it. Later posts reran the NVFP4 config: spec-decode and Swift vs stock are within 4% of these tok/s rows, which is a rough size for run-to-run noise. They also scored 176 and 168 on the BFCL subset against 174 here, so the +7 over FP8 is smaller than the spread between NVFP4 runs. BFCL categories have 10–25 cases, so a few points either way is noise.
• The GPU ran at a 400 W power limit, not the default 600 W. I haven’t measured either format at 600 W.
• Speed scenarios use fixed output lengths (ignore_eos), so tok/s compares speed only, not how much each model thinks.
• BFCL uses its own Qwen3 FC prompt, not the model’s chat template. Compare these scores with each other, not with the public leaderboard.
• Deep-AI-Evo measured a smaller gap on the same card with vLLM 0.21 and unsloth/Qwen3.8-27B-NVFP4: 50.9 → 56.4 tok/s without MTP (+11%), 115.8 → 132.3 with MTP k=2 (+14%), single-stream decode on a 35-token prompt that asks the model to count from 1 to 500 (repo). Their ~1.4K-token prompt gives 52.6 → 58.6 without MTP and 95.6 → 100.2 with MTP. The engine, checkpoint, drafter, task and metric all differ from mine, so this is context, not a replication, and I haven’t tested which of them explains the difference.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns both configs.
• The scripts on main keep changing. To rerun with the harness as committed with these results, check out d4d7869.
• Serving results (one JSON line per scenario, including full server args): FP8, NVFP4.
• BFCL scores and every model response: results/<label>/bfcl/ in the same folder.

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.