Can Qwen3.8-27B, served for chat through SGLang, double as a decision model? Does that hold for Swift 1.5’s fine-tune, and at NVFP4?

All seven configs ran on Typed Decisions: 400 test cases from four workflows (agent traces, customer service, invoices, security incidents), five questions each. Its answers are the average of three samples from a teacher model that the dataset card doesn’t name beyond “roughly 4B-class”, so its scores measure agreement with that teacher, not correctness.

Three more datasets have answers that don’t come from a model. Typed Decision Bench uses labels from 21 public datasets and one generator. Procedural Typed Decisions computes answers from rules stated in the prompt. Calibrated Typed Decisions has human labels, plus a PubMedQA split with expert labels. Only the two NVFP4 configs ran on these three, so they compare Swift with stock, not the quantizations.

On the two NVFP4 configs, both models come out of the server overconfident: every fitted temperature is above 1. One temperature per question type, fitted per dataset and applied in the client, improves the aggregate probability scores on all four datasets without changing a top answer. On Calibrated Typed Decisions’ human labels, log loss drops from 0.60–0.67 to 0.44–0.46 and ECE from 0.095–0.099 to 0.010–0.011; the same fit lowers log loss on the unseen PubMedQA task from 0.90–0.95 to 0.68–0.69. On Typed Decisions, NVFP4 and BF16 agree with the teacher at similar rates, though about 9–10% of their top answers differ. Observed Swift–stock gaps are under 2 points, in both directions.

Results

SGLang answers a decision from a chat model by reading the probabilities of the answer labels, with thinking off (docs). Typed Decisions test: 400 cases, 2,000 decisions, one request per case through /v1/systemone, the dataset’s own scorer.

Agreement means the model’s top answer matches the gold’s top answer. Swift is the more confident one; it doesn’t agree much more.

Calibration on the two NVFP4 configs, fitted on train (1,200 cases) and scored once on test:

Datasets with real answers

The Typed Decisions gold above is a teacher’s opinion; these three don’t come from a model (second column). Same calibration method, fitted separately for each dataset; no item is scored with a fit that used its answer.

DecisionScore and proper score are 100 × (1 − normalised Brier), averaged per item, so they reward probabilities, not just the top answer. Lower log loss is better.

Takeaways

• Per-type temperatures improved the aggregate probability scores on all four datasets. One temperature per question type: two on Calibrated Typed Decisions, three elsewhere. Each dataset gets its own fit. Typed Decision Bench uses five folds, each scored with a fit on the other four. No item is scored with parameters fitted on its own answer. Every fitted temperature is above 1, so both models are too sure of their answers. On Calibrated Typed Decisions test (human labels), ECE drops from 0.095 to 0.010 (stock) and 0.099 to 0.011 (Swift), log loss from 0.60 to 0.46 and 0.67 to 0.44. That is the aggregate: log loss improves in 10 of its 12 source and type groups for each model, gets worse on CLINC and Banking77 choice questions (CLINC 0.06 → 0.10), and GoEmotions stays poorly calibrated (ECE 0.18–0.23 after). Typed Decision Bench gains 3.1 (stock) and 4.0 (Swift) points of score, procedural 2.2 and 3.0 (all intervals above 0). A temperature can’t change the top answer, so accuracy doesn’t move.
• Temperatures helped on the one unseen task, PubMedQA. The fit never saw it, so no prior applies and accuracy can’t change. PubMedQA is all choice questions, so only the Calibrated Typed Decisions choice temperature acts on it. It still cuts log loss from 0.90 to 0.69 (stock) and 0.95 to 0.68 (Swift), and ECE from 0.15 to 0.06. The temperatures fitted together with priors do a little worse there (0.72 and 0.70). It is one task, so this is one data point for transfer.
• Calibration runs in the client. /v1/decisions accepts a per-request temperature; /v1/systemone refuses it, and neither has a bias field. So both calibrations run on the probabilities the client gets back, and chat on the same server is untouched.
• On Typed Decisions, the temperatures cut KL from the teacher by about two-thirds: Swift 0.676 → 0.198, stock 0.542 → 0.203. Fitted there, they are 2.6–3.3 for stock and 2.8–5.0 for Swift, highest on Swift’s yes/no questions (4.95).
• Priors add accuracy where labeled examples exist for the same questions (one log-prior per workflow, task or source, question and option, temperatures refit alongside): +5.5 (stock) and +3.6 (Swift) points on Calibrated Typed Decisions, and another 2.9–3.7 points of score on top of the temperatures on the other two. On Typed Decisions they lift agreement by 0.077 (stock) and 0.065 (Swift), to 0.743 and 0.748. Both are just above the teacher self-agreement of 0.735 reported on the Typed Decisions card, which reads scores well above it as learning the teacher’s quirks, so this is agreement with that teacher, not getting more right. New questions need labeled examples before priors apply.
• Observed Swift–stock gaps are under 2 points, in both directions (points of accuracy, teacher agreement or score); some intervals reach past 2. Uncalibrated, Swift leads on human and expert labels (+1.4 points of accuracy on Calibrated Typed Decisions test, interval [+0.6, +2.3]; +1.5 on PubMedQA, [+0.1, +2.9]) and on agreement with the Typed Decisions teacher (+1.3 at BF16 and +1.75 at NVFP4, intervals above 0; +0.55 at FP8, includes 0). It trails by 0.55 points of score on Typed Decision Bench ([−0.86, −0.25]) and is level on procedural (+0.10, [−0.63, +0.84]). After temperatures and priors, only procedural (+0.66, [+0.04, +1.28]) and PubMedQA, which calibration doesn’t change, exclude 0. That is about ten comparisons with no correction for multiple testing, so the gaps that barely exclude 0 are weak evidence. Swift is also more confident: its Typed Decisions KL is 0.66–0.71, stock’s 0.54–0.59.
• Quantization doesn’t measurably change the agreement rate with the Typed Decisions teacher. Every BF16 → FP8 and BF16 → NVFP4 gap is under 0.01, and every 95% interval includes 0 (widest bound +0.021; paired bootstrap over cases). That is the rate, not the same answers: against BF16, FP8 changes 3.1–3.2% of the 2,000 top answers and NVFP4 8.8–9.9%, gaining about as many matches with the teacher as it loses. KL is 0.03–0.05 lower at NVFP4 than at BF16 (no interval on that). NVFP4 also answers in less than half the time of BF16 (166 vs 386 ms per case). The real-answer datasets ran at NVFP4 only, so they don’t test quantization.
• Drafter and API don’t change the answers with a cold prefix cache: at most 3 of 2,000 top answers move, the same as a repeat of the same pass. A decision is prefill only, so DFlash2 has nothing to speed up. A warm cache does change them; see Caveats.
• For scale: on the Typed Decisions card’s zero-shot table, decision fine-tunes of the same base score 0.728 (Cloudflare/clef) and 0.753 (perplexity-ai/pplx-decider-v1.1-27b); both list Qwen/Qwen3.8-27B as base in the Hugging Face API, checked 2026-10-30. Raw Swift NVFP4’s 0.683 would be 27th of the table’s 39 rows (roughly; see Caveats). The calibrated rows are fitted on train, so they belong in the Typed Decisions card’s second table, where Amos 1 large’s priors go from 0.761 to 0.8025 (+0.0415). A similar fit on Swift (priors plus temperatures) is +0.065.
• On the Typed Decision Bench leaderboard, scoring its published answer files with the same scorer reproduces the published DecisionScores to two decimals. Uncalibrated, both NVFP4 models land between OpenJev 0.1 (75.56) and decider-2b-v8 (77.56), 4.5–5.1 points under the top entry, Jev 1.13.0 (81.70). Calibrated they score 2.3–2.5 above Jev, but that fit comes from the test items themselves (the set has no other split; 5 folds, each scored by a fit on the other four), so it isn’t a like-for-like row.

Setup

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: Qwen/Qwen3.8-27B @ 1d4bf0f2, Qwen/Qwen3.8-27B-FP8 @ 017b9c7a, RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c, ukisai/Swift-1.5-Qwen3.8-27b @ b4c84d42, ukisai/Swift-1.5-Qwen3.8-27b-NVFP4 @ 25482027. Swift FP8 is my own block-FP8 conversion of Swift BF16 (128×128 blocks, stock FP8’s layout); realderpz/Swift-1.5-Qwen3.8-27B-FP8 is a public equivalent (llmcompressor FP8_BLOCK).
• Drafter on the DFlash2 row only: z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c, 8 draft tokens.
• Same flags as the earlier posts, no speculative decoding except the DFlash2 row. --mem-fraction-static 0.76 for BF16, 0.50 for FP8, 0.41 for NVFP4.

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static <0.76|0.50|0.41> --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Benchmark:

• LocalLLaMA/typed-decisions @ 1415d375, all config. Four workflows (agent trace observability, customer service, invoice processing, security incidents), five questions per case, three types: yes/no (noul), one of N labels (choice), ordered levels (score). test has 400 cases; train has 1,200 cases from a separate generation run.
• One request per case (about 800 input tokens, all five questions), 1 in flight. Two passes on /v1/systemone and one on native /v1/decisions. The headline numbers are pass 1. All three passes ran with an almost empty prefix cache (about 5% of prompt tokens cached, server log).
• Real-answer datasets, one /v1/systemone pass: Typed Decision Bench v0.3.0 @ 79ec2b9b, all 5,387 items, the 359 it withholds rebuilt with its own builder (blobfishai/jevfish @ 80262b5) and checked against its SHA256SUMS; its DecisionScore and accuracy. Procedural Typed Decisions all @ 05ca1d03, a fixed sample of up to 10 states per task and level (874 test, 880 validation for the fit); it has no scorer, so Typed Decision Bench’s per-item proper score. Calibrated Typed Decisions v2.1 @ 34189779, its human labels (not its soft teacher labels), validation for the fit; accuracy, log loss, Brier, ECE. Typed Decision Bench is calibrated by 5-fold cross-fitting within its items.
• Typed Decisions scoring: its scorer, unchanged. Agreement of the top answers, KL(gold ‖ prediction), Brier.
• Typed Decisions calibration: calibrate.py fits by soft cross-entropy against its train gold. “T per type” scales each distribution as p^1/T, with one T for each question type. “T + priors” also multiplies each option by e^b, with one b per workflow, question and option (L2-penalised), and the three temperatures are refit with them (stock 2.00/2.05/2.43, Swift 2.25/3.30/2.71 for choice/noul/score). It is scored once on test.

Caveats

• The Typed Decisions gold is a teacher, not ground truth. Per the Typed Decisions card, it is the mean of three samples from a model of roughly 4B-class capability; it doesn’t name the model. Its references: label frequencies alone score 0.470, and a fresh teacher sample scores 0.735 against the rest. “Calibrated” here means calibrated to that teacher. The card reads scores well above 0.735 as learning the teacher’s quirks; my 0.743 and 0.748 are only about 0.01 above, so I read them as fitting the teacher, not as more correct.
• The Typed Decisions intervals come from 400 cases, resampled with all five decisions of a case kept together. The real-answer intervals resample cases within task or source groups. Swift’s lead over stock is small: 0.013 at BF16, 0.0175 at NVFP4 (0.683 vs 0.666 in the table are rounded), with a NVFP4 interval of [+0.003, +0.033]. At FP8 the interval includes 0.
• The Typed Decisions leaderboard rows are only roughly comparable. Most were scored on vLLM or a vendor API. Mine are SGLang on a chat model. The comparison is fair between my configs (same prompts, same scorer), not against the leaderboard’s absolute numbers.
• I left ECE out of the Typed Decisions results. Scored against its own top answers, that dataset’s gold has an ECE of 0.341 on test. This ECE compares confidence with agreement on the teacher’s top label, not with its full distribution, so matching the teacher exactly doesn’t minimise it. KL and Brier are the calibration numbers there. ECE for every row is in the repo.
• I fitted calibration for the two NVFP4 configs only, per dataset. The fitted temperatures differ by dataset (temperatures alone: 1.3–2.0 for choice and 2.9–4.6 for noul on procedural and Calibrated Typed Decisions), so a fit from one dataset isn’t shown to carry to another. The one transfer test is PubMedQA, a single task the fit never saw, scored with the Calibrated Typed Decisions fit.
• A warm prefix cache changes answers. The Typed Decisions noise checks ran cold: the server keeps 117 Mamba states, too few for 400 cases, so pass 2 found its prefixes evicted (my reading of the log, not tested). Rerun on 50 cases with the cache warm, 68–76% of prompt tokens cached: 25 (stock) and 34 (Swift) of 250 top answers change between a cold and a warm pass of the same prompt, up to 0.58 in probability; two warm passes through the two APIs differ on 8 and 10. Agreement over the 50 cases moves by under 0.01. The Typed Decisions tables are mostly cold-cache numbers. The real-answer runs were not: about 30% of Typed Decision Bench and 16% of procedural prompt tokens came from the cache. A single served decision depends on what is in the cache. A follow-up on Swift NVFP4 varied the cache, precision and deterministic-inference settings. In the variants tested, answers changed only with the hierarchical cache on; without it, 0.9% of prompt tokens came from the cache on every pass and repeats were identical. BF16 KV still changes 30 of 250; deterministic inference with Triton attention halves the changes (34 to 17) but doesn’t remove them, though SGLang means that mode to make cached and uncached prefill identical. SGLang’s own cached-vs-uncached check (test_deterministic --test-mode radix_cache) fails on these flags, with logprobs 0.49 apart (0.20 with deterministic inference, Triton attention and BF16 KV). Open SGLang PRs #43110 and #46707 fix a hierarchical-cache Mamba restore that leaves an active request’s state stale; whether that is the cause here isn’t tested. Details in the repo README, “Prefix cache ablation”.
• The real-answer runs are one pass on two configs. Procedural is a sample of 874 of 18,000 states. On Typed Decision Bench, the leaderboard systems are scored without any fit.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns all seven configs in about an hour. The README has per-type tables, the noise checks, the real-answer and cache tables with paired intervals, and the rerun commands.
• The scripts on main keep changing. To rerun with them exactly as these runs had them, check out d445be4 for Typed Decisions, 4a91a4c for the real-answer and cache runs, and e8e2a51 for the cache ablation.
• results/<config>/decisions/: every raw answer (pass1.jsonl, pass2.jsonl, native1.jsonl) and summary.json. results-train/ holds the train-split answers for the two NVFP4 configs.
• results/{stock,swift}-nvfp4/calibration.json: the fitted temperatures and priors, and the test metrics. Code: scripts/decisions/calibrate.py.
• paired_ci.py and results/paired-agreement.txt: the paired gaps and bootstrap intervals.
• results-real/: raw answers per dataset and split, fitted calibrations, and tables.md (every row with paired intervals, from real_answers.py). results-cache/: the 50-case warm-cache run; results-cache-ablation/: the cache, precision and determinism follow-up (cache-ablation.sh, cache_check.py). Dataset adapters and scorers: scripts/decisions/sets.py.

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.

On one RTX PRO 6000 with SGLang and DFlash2 (8 draft tokens), ukisai’s Swift 1.5 NVFP4 scores the same as stock Qwen3.8-27B NVFP4 on GSM8K and a 205-case BFCL subset (97.04% vs 96.82%, 168 vs 168) and decodes at about the same speed per token. At reasoning_effort=medium it spends 14–19% fewer thinking tokens on GSM8K and SPEED-Bench, so a response is 8–10% shorter and finishes about 10% sooner.

Results

Stock RadixArk/Qwen3.8-27B-NVFP4 vs Swift ukisai/Swift-1.5-Qwen3.8-27b-NVFP4, both with DFlash2 8, reasoning_effort=medium.

¹ The 125 of 142 turns where both models stopped on their own; the output-token row counts all 142.

Accept length on the five sets of the DFlash2 model card, with its protocol (xhigh, temperature 1.0, top-p 0.95, top-k 20, max 4096 new tokens), 32 prompts per set, 1 user. “8 draft tokens” is SGLang’s --speculative-num-draft-tokens 8, DFlash2’s block size 8 (7 drafted per step):

Mean thinking tokens per answer on the same sets, on the prompts where both models stopped on their own. At medium that’s 30–32 of 32 prompts; at xhigh 31 (MATH-500), 26 (MBPP), 24 (MT-Bench) and 21 (HumanEval), with GSM8K at 32.

Mean tokens per answer, thinking plus answer, all 32 prompts (answers cut off at 4096 count as 4096):

Takeaways

• Same speed per token; the time saved is the shorter answers. Median time per output token on SPEED-Bench is 4.49 vs 4.47 ms. Swift finishes the 110 prompts in 913 s instead of 1,016 s because it writes 10% fewer tokens. Both runs hit the 4,096-token cap on some turns (11% of stock’s, 9% of Swift’s). Cut-off answers count as 4,096, so the real gap is probably a bit larger.
• Chat at 1 user is the exception: 4% slower, accept length 3.27 vs 3.44. One run on 12 prompts, so I can’t say whether it’s real. The other serving rows in the repo (4 users, 20k, 80k, 120k) are within 1%; 40k at 1 user is +4% for Swift (4 prompts).
• The saving is mostly thinking. At medium Swift thinks 5–38% less on the DFlash2 card’s sets and 14–19% less on GSM8K and SPEED-Bench; at xhigh, 17–47% less. Total minus thinking (an estimate: completion_tokens against re-tokenized reasoning) is about the same on full GSM8K (134 vs 136) and at xhigh on GSM8K and MATH-500, but it moves elsewhere: MT-Bench at xhigh −29%, HumanEval at xhigh +26%, and MBPP at medium saves 82 tokens of which only 36 are thinking. Swift’s model card claims 24.8% fewer thinking tokens at medium and 41.9% at xhigh (card @ b4c84d42, checked 2026-10-30). That is BF16 on vLLM; the xhigh figure is its GPQA-Diamond row and the table doesn’t name a benchmark for medium.
• On three of five sets xhigh costs Swift about what medium does. Whole responses at xhigh vs medium: GSM8K 296 vs 368 tokens, MATH-500 864 vs 833, MBPP 899 vs 895. MT-Bench (1,163 vs 1,002) and HumanEval (1,454 vs 1,131) cost 16–29% more. Stock pays more for xhigh on four of five (HumanEval 2,042 vs 1,196). I didn’t measure accuracy at xhigh, and the 4,096 cap hides the longest answers.
• Stock accept length holds up on Swift. It is within 2 standard errors of stock on every set and SPEED-Bench domain, and slightly higher on all five DFlash2 card sets at xhigh. So the fine-tune didn’t hurt the stock drafter. I didn’t test a drafter tuned on Swift.
• The DFlash2 model card’s numbers reproduce here. On its prompts and protocol, stock NVFP4 lands within 0.18 of the DFlash2 card on every set, and output tok/s at 1 user is within 7% of the H200 (180–240 vs 184–236). The 3.4 in the spec-decode post is ShareGPT chat at medium. Prompts, sampling, effort and max length all differ from the DFlash2 card’s runs, so the gap is most likely the workload.
• BFCL ties at 168. Stock NVFP4 scored 174 and 176 on this config in the earlier posts, so the same config already moves by 8 cases between runs. By category, Swift gains multi_turn_base (14 → 16 of 20) and loses irrelevance (15 → 11); with 20 cases each, neither is outside noise.
• Quality here is two benchmarks. GSM8K is near ceiling and BFCL is a 205-case subset, so “same” means no difference I could see. Swift’s model card shows 82.22% vs 84.14% for the base at medium, on a benchmark it doesn’t name. I ran nothing harder.

Setup

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c, ukisai/Swift-1.5-Qwen3.8-27b-NVFP4 @ 25482027 (ukisai’s own NVFP4 build).
• Drafter: z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c, 8 draft tokens, on every run.
• Flags: the launch command from the NVFP4 vs FP8 post (scripts/args/qwen38-27b.sh) with --mem-fraction-static 0.41. Both get a 236,864-token KV pool and 43.8 GB of GPU memory after load.

Benchmarks, all at reasoning_effort=medium unless noted:

• GSM8K: the full test split, sgl-eval’s prompt and grader (last \boxed{}), the model’s own sampling, max 32,768 tokens, 32 requests in flight. No answer was cut off.
• BFCL: bfcl-eval 2026.3.23, the Qwen/Qwen3-32B-FC handler, the same fixed 205-case subset as the earlier posts (case ids).
• DFlash2 card sets: GSM8K, MATH-500, HumanEval, MBPP, MT-Bench (first turn), prompt formats and shuffle from z-lab/dflash at 07ebd93, 32 prompts per set, 1 user. Once with the DFlash2 card’s protocol (xhigh), once at medium with the same sampling.
• SPEED-Bench: the qualitative split, 10 prompts per domain (110 prompts, 142 turns), max 4096 tokens, 1 user, server-default sampling.
• Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1, the same chat and 20k scenarios as the first post.

Caveats

• One run per config, 32 prompts per DFlash2 card set and 10–26 turns per SPEED-Bench domain. Gaps under about 2 standard errors can’t be told from noise, and every Swift-vs-stock accept-length gap is under that. Chat at 1 user is 12 prompts; the 20k–120k rows are 4 prompts each (8 at 4 users).
• “Output tokens” and “tokens per answer” are completion_tokens, i.e. thinking plus answer; answers cut off at 4096 count as 4096. “Thinking tokens” is the saved reasoning text of each response, tokenized with the model’s tokenizer, counted only on prompts where both models stopped on their own (thinking_tokens.py in the repo).
• The DFlash2 model card doesn’t say how many prompts it ran or at what precision; its launch example serves BF16 Qwen/Qwen3.8-27B. Mine is NVFP4 on an RTX PRO 6000 at 400 W.
• The serving rows use fixed output lengths (ignore_eos), so they measure speed only and can’t show the token savings. The 20k prompt is SGLang’s random dataset (one ShareGPT prompt repeated to length).
• BFCL uses its own Qwen3 FC prompt, not the model’s chat template. Compare these scores with each other, not with the public leaderboard.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns all four configs, about two and a half hours.
• The scripts on main keep changing. To rerun with them exactly as these runs had them, check out 50fa2ce.
• results/stock-nvfp4-quality/ and results/swift-nvfp4-quality/: GSM8K and BFCL, every response included.
• results/dflash8/ (stock) and results/swift-dflash8/ (Swift): accept-sets/, speed-bench/ (one line per request, generated text included), bench-serving.jsonl and server.log.
• thinking_tokens.py: the thinking-token tables, from the saved responses.

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.

On one RTX PRO 6000, Qwen3.8-Flash-Next NVFP4 on SGLang beats Qwen3.8-27B NVFP4 without a drafter in every scenario I ran. Against the 27B with DFlash2 (8 draft tokens), it is about half as fast at chat, ties at 40k, and wins only from 80k: 1.9x at 120k. It takes the whole card for that, 88 GB against 44 GB. Capped at 44 GB with llama.cpp, it runs at 6–24% of that 27B’s speed (17% at chat with 1 user).

Results

Whole card, output tok/s, median time to first token (TTFT) in parentheses, one run per config. Bold is the fastest; at 40k the two bold cells are a tie.

The 27B numbers are the runs from the spec-decode post, not rerun. Its GPU memory row (43.8 GB) comes from resources.json of the same server config in the Swift vs stock data.

At the 27B’s 44 GB budget (llama.cpp, Q4 GGUF, experts of 30 of 48 layers in host RAM):

This run is partial: 80k failed (see caveats) and 120k didn’t run.

Takeaways

• Flash-Next wins on prefill, the 27B on decode. At 1 user, TTFT is 1.4x lower at 20k (1.69 vs 2.35 s), 2.1x at 80k (6.98 vs 14.8 s) and 2.6x at 120k (10.6 vs 27.8 s). At chat length it’s higher (81 vs 48 ms).
• Decode at 1 user is flat at 10.4–10.7 ms per output token. That beats the 27B without a drafter (13.9–16.5 ms) and loses to the 27B with DFlash2 (5.4–6.5 ms).
• Where it wins depends on the answer length. These requests are 512 output tokens. From the medians, the 27B with DFlash2 catches up after about 1,600 output tokens at 80k and about 4,100 at 120k. That’s my estimate (TTFT difference over per-token difference), not a measurement.
• It needs the whole card. 87.7 GB of GPU memory, against 43.8 GB for the 27B, which leaves half the card free.
• Host RAM is about 70 GiB on top, mostly the 51B-parameter n-gram embedding table (~48 GiB in FP8), which SGLang keeps in pinned memory.
• I couldn’t get SGLang under 44 GB. The image rejects --ple-offload-embedding together with --cpu-offload-gb and --offload-group-size. KTransformers’ expert offload doesn’t support the model yet (ktransformers#2179, open on 2026-10-29).
• llama.cpp does fit in 44 GB, with the experts in host RAM. At 1 user, decode costs 3x the whole-card run (32–33 ms per token) and TTFT at 20k is 5.8x (9.81 vs 1.69 s).

Flash-Next only wins past 40k, and it takes the whole card to do it.

Setup

Model: Qwen/Qwen3.8-Flash-Next, 125B parameters with 6B active per token, plus a 51B n-gram embedding and a 4B MTP head. 48 layers: 36 Gated DeltaNet, 12 Qwen Sparse Attention.

Flash-Next, whole card:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261029-c10e0e17, with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1.
• Checkpoint: RadixArk/Qwen3.8-Flash-Next-NVFP4 @ 7b719225.
• No drafter. KV pool 611,520 tokens, 12 mamba slots.

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static 0.88 --max-running-requests 4 --max-mamba-cache-size 12
  --context-length 262144 --dtype bfloat16 --quantization modelopt_fp4 --tp 1
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --cuda-graph-backend-prefill disabled
  --ple-offload-embedding --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-track-interval 64
  --chunked-prefill-size 2048 --page-size 64 --kv-cache-dtype fp8_e4m3 --sleep-on-idle

Chunks of 2048 and no prefill CUDA graph came after a first attempt (native MTP, --mem-fraction-static 0.96, 4096-token chunks) ran out of GPU memory on the first 20k-token prefill. I didn’t test the two changes separately.

Flash-Next, 44 GB:

• Image: ghcr.io/ggml-org/llama.cpp@sha256:0283744a (b11977).
• Checkpoint: unsloth/Qwen3.8-Flash-Next-GGUF @ 766911a6, UD-Q4_K_XL (103.7 GiB).
• --n-cpu-moe 30 sets the GPU memory: 44.3 GB after load, 44.5 GB peak. Host RAM 47–64 GiB.

llama-server -m <UD-Q4_K_XL shard 1>
  -ngl 999 -ot per_layer_token_embd=CPU --n-cpu-moe 30 -fit off
  -c 153600 -kvu --parallel 4 -fa on --jinja -b 4096 -ub 4096 -t 16 -lm none

27B: stock RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c, with and without z-lab/Qwen3.8-27B-DFlash2 (8 draft tokens), on lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48, --mem-fraction-static 0.41, context 153,600, 64 GiB HiCache in host RAM. Flags in the spec-decode post.

Serving: python3 -m sglang.benchmark.serving --warmup-requests 2 --seed 1, reasoning_effort=medium. Backend sglang-oai-chat with --flush-cache on SGLang; vllm-chat on llama.cpp, with cache_prompt: false on every request because llama.cpp has no cache flush endpoint.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: SGLang random dataset (one ShareGPT prompt repeated to 20,000 tokens), 512 output tokens; 4 prompts at 1 user, 8 at 4 users.
• 40k, 80k, 120k prompt: same dataset, 512 output tokens, 4 prompts at 1 user.

Caveats

• One run per config. The long scenarios have 4 prompts each, so a few percent either way is noise; 134.2 vs 137.5 at 20k with 4 users is a tie.
• The long prompts use SGLang’s random dataset, which repeats one ShareGPT prompt to length. It’s the same input for every config, but it isn’t a real long document.
• The 27B and Flash-Next runs used SGLang nightlies 19 days apart (2026-10-10 and 2026-10-29), and Flash-Next has no HiCache. I didn’t rerun the 27B on the newer image.
• No drafter on Flash-Next. Native MTP loaded at --mem-fraction-static 0.96 and ran out of GPU memory on the first 20k prefill (chunk 4096). I didn’t try it at 0.88, and I didn’t run incoai/Qwen3.8-Flash-Next-DFlash2. Stock SGLang also caps the Flash-Next draft block at 4 tokens. A drafter might raise the whole-card chat numbers; I have no measurement either way.
• No quality comparison. The BFCL subset from the earlier posts scored 136/205 for Flash-Next vs 168–176 for the 27B NVFP4 runs, over half of the gap in parallel (4/20 vs 18–19 for the 27B) and parallel_multiple (10/20 vs 18–19). None of the 16 failed parallel responses has two tool calls: the model makes one call or answers in prose. But BFCL’s Qwen/Qwen3-32B-FC handler builds its own prompt instead of using the model’s chat template, so I don’t trust that as a model comparison. The scores and every response are in the repo.
• The llama.cpp run is partial. At 80k the unified KV pool (153,600 tokens) ran out because the first warmup request was still in slot 0 when the second started, and the client failed on a non-JSON stream chunk. Two 78k requests in a row on a fresh server worked. 120k and BFCL didn’t run.
• 400 W power limit, not the default 600 W. Output lengths are fixed (ignore_eos), so this measures speed only.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. The 27B results are in the spec-decode folder.
• The scripts on main keep changing. To rerun with them exactly as these runs had them, check out 45d1fd2.
• results/fn-nospec/ (whole card) and results/fn-gguf-q4/ (44 GB): bench-serving.jsonl (one line per scenario, server args included), server.log, resources.json (GPU and host memory, startup time), and bfcl/ with every model response.

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.

On one RTX PRO 6000 with SGLang, DFlash2 with 8 draft tokens is the fastest drafter for stock Qwen3.8-27B NVFP4 in every scenario I ran, from short chat to a 120k-token prompt. It beats the model’s own MTP head at 3 and 7 steps, DFlash2 with 4 draft tokens, and RadixArk’s DSpark. Tool calling (BFCL) stays within noise.

The end-to-end gain drops from 2.5x on chat to 1.16x at 120k. That’s prefill, not the drafter: at 1 user, time per output token stays 2.5–2.7x faster at every length. From 40k up the lead over the nearest drafter is only 3–4%, which is inside the noise (see Caveats).

Results

Output tok/s, one run per config. Bold is the fastest.

Mean accept length (tokens per verify step, 1 user):

Takeaways

• Acceptance held roughly flat with context, on a repeated ShareGPT prompt. Swings from chat to 120k are 0.15–0.43 per drafter. Only DFlash2 4 drifts down (2.81 to 2.66); DFlash2 8 doesn’t. The 20k–120k rows are one prompt repeated, and real prompts are only checked up to 32k (see the update below).
• At 1 user the drafter speeds up decode 2.5–2.7x at every length. Median time per output token for DFlash2 8 is 5.4–6.5 ms against 13.9–16.5 ms without a drafter. At 20k with 4 users it’s 15.3 vs 20.5 ms, only 1.34x.
• So why does the end-to-end gain shrink? These requests are 512 output tokens behind a long prompt. At 120k, prefill takes 27.8 s and the 512 tokens take about 3.3 s. The drafter only speeds up the 3.3 s.
• More accepted tokens isn’t faster. MTP 7 accepts more per step than DFlash2 8 at 40k, 80k and 120k (3.65, 3.79 and 3.55 vs 3.28, 3.43 and 3.46) and is still slower everywhere, because its time per token is higher (6.9–8.3 ms at 1 user vs 5.4–6.5).
• DSpark is 1–5% ahead of DFlash2 4 in five of seven rows and level in the other two. Its accept length is about MTP 3’s. It also has the slowest BFCL calls of all six configs, slower than no drafter (3.24 s mean vs 2.53 s; DFlash2 8: 0.91 s). I haven’t looked into why.

The drafter costs KV cache

Same --mem-fraction-static 0.41 for every config. The draft model and verify buffers come out of the same budget:

DFlash2 8 gives up 26% of the KV pool and 76% of the mamba slots (max_total_num_tokens and mamba_cache_size from each server.log). The largest scenarios above (4 × 20k, 1 × 120k) fit in the 237k-token pool. I haven’t measured what it costs in multi-turn prefix cache hits.

Is the 3.4 accept length a thinking or sampling artifact?

The DFlash2 model card reports 4.10 on MT-Bench (H200, xhigh reasoning, block size 8 with 7 draft tokens per step, the setting I call DFlash2 8). I get 3.43 on ShareGPT at medium. So I reran DFlash2 8 with thinking off and with temperature 0:

Neither one closes the gap. The workload does (see the update below).

Update (2026-10-30): On the DFlash2 card’s own five sets and protocol, the same stock NVFP4 setup lands within 0.18 of its accept length on all five sets (MT-Bench 4.13 vs 4.10), so the gap here is most likely the workload, not the hardware or the quant. SPEED-Bench’s real prompts from 1k to 32k tokens accept 3.9–4.3 (12 prompts per length, standard error 0.2–0.3), so acceptance staying roughly flat with context holds on real prompts up to 32k too. The 20k–120k rows above repeat one ShareGPT prompt to length, so their accept lengths aren’t representative. The 80k and 120k rows, including the ~2.5x decode speedup there, are still repeated-prompt measurements; I haven’t rerun them on real prompts. Details: Swift 1.5 vs stock.

Setup

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48 for every config (it has both DFLASH and DSPARK).
• Target: RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c.
• Drafters: z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c, RadixArk/Qwen3.8-27B-DSpark @ b9a5dbdf. MTP uses the head built into the target checkpoint.
• Requests use reasoning_effort=medium.

Shared flags, the same as the first post:

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static 0.41 --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Then the speculative flags, appended after the shared ones. The number in a name is the verify width: --speculative-num-draft-tokens for DFlash2 (4, 8) and steps for MTP (3, 7), so MTP 3 and DFlash2 4 both verify 4 tokens per step, and MTP 7 and DFlash2 8 both verify 8.

Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: SGLang random dataset (one ShareGPT prompt repeated to length), 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.
• 40k, 80k, 120k prompt: same random dataset at that length, 512 output tokens, 4 prompts at 1 user.

BFCL: bfcl-eval 2026.3.23, the Qwen/Qwen3-32B-FC handler, the same fixed 205-case subset as the first post (case ids).

Caveats

• One run per config. The 40k–120k scenarios have 4 prompts each, so their rates carry a few percent of noise, and DFlash2 8’s lead over the nearest drafter at 40k (+3.2%), 80k (+3.6%) and 120k (+3.8%) is inside it. The same config reran in the other two posts: 20k at 4 users came out 142.3 and 138.7 against 137.5 here, and BFCL 174 and 168 against 176. BFCL categories have 10–25 cases, so 171 vs 176 is noise.
• DSpark runs on the shared flags so only the drafter changes. Its model card serves it with float32 mamba state, --mamba-radix-cache-strategy extra_buffer, --mamba-full-memory-ratio 11.93 and 32k prefill chunks. With those it might do better than here.
• 400 W power limit, not the default 600 W.
• Output lengths are fixed (ignore_eos), so this measures speed only.
• Deep-AI-Evo found MTP slower than no MTP at long context on the same card: 18.2 vs 43.7 tok/s decode at ~177k tokens, NVFP4, MTP k=2 (repo). I don’t see that up to 120k: MTP 3 keeps decode at 8.5 ms per token vs 16.5 without a drafter. They ran vLLM (0.21 and 0.26) with unsloth/Qwen3.8-27B-NVFP4 and went to ~177k.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh reruns every config, about two hours.
• The scripts on main keep changing. To rerun with them exactly as these runs had them, check out 4b5cf81.
• results/<label>/ per config: bench-serving.jsonl (one line per scenario, full server args included), server.log, and bfcl/ with every model response.

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.

On one RTX PRO 6000 with SGLang and DFlash2 speculative decoding, NVFP4 is 32–37% faster than FP8 for stock Qwen3.8-27B, on short chat and on 20k-token prompts. BFCL tool calling stays within noise (174 vs 167 of 205).

Results

Stock Qwen3.8-27B: Qwen/Qwen3.8-27B-FP8 vs RadixArk/Qwen3.8-27B-NVFP4.

Takeaways

• At 1 user, time to first token on a 20k prompt drops 27% (3.20 → 2.34 s), so prefill is faster too. At 4 users it drops 32%, but four 20k prompts prefill at the same time, so that number includes waiting on the other three.
• On the 20k rows, tok/s is output tokens over the whole request, prefill included. Median time per output token drops 31% at 1 user (7.77 → 5.38 ms) and 14% at 4 users (17.07 → 14.65 ms), so the 37% at 4 users is partly prefill.
• DFlash2 accept length barely moves: within 3% in all four scenarios (chat, 1 user: 3.49 vs 3.41). The gain isn’t from better drafting. Median time per output token in chat at 1 user drops from 7.30 ms to 5.59 ms.
• BFCL shows no measurable change. The +7 is irrelevance (12 → 18) and multi_turn_base (14 → 16), 20 cases each. Everything else nets −1.
• Mean output per call is the same (146.6 vs 146.5 tokens), so the 25% BFCL latency drop is speed.

BFCL by category

Setup

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: Qwen/Qwen3.8-27B-FP8 @ 017b9c7a, RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c.
• Speculative decoding: DFlash2 (z-lab/Qwen3.8-27B-DFlash2), 8 draft tokens, for every run.
• --mem-fraction-static 0.50 for FP8, 0.41 for NVFP4, so both get about the same KV pool (240,512 vs 236,864 tokens, 28 mamba slots each).
• Requests use reasoning_effort=medium.

Launch flags:

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static <0.50|0.41> --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --speculative-algorithm DFLASH --speculative-draft-model-path <DFlash2>
  --speculative-num-draft-tokens 8
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1.

• Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
• 20k prompt: SGLang random dataset (one ShareGPT prompt repeated to length), 20,000-token input, 512 output tokens; 4 prompts at 1 user, 8 at 4 users.

BFCL: bfcl-eval 2026.3.23, the Qwen/Qwen3-32B-FC handler, a fixed 205-case subset across 10 categories (case ids).

Caveats

• One run per config, with 4–32 prompts per speed scenario and no variance reported. The gain is consistent across all four scenarios, but I can’t put an error bar on it. Later posts reran the NVFP4 config: spec-decode and Swift vs stock are within 4% of these tok/s rows, which is a rough size for run-to-run noise. They also scored 176 and 168 on the BFCL subset against 174 here, so the +7 over FP8 is inside that spread. BFCL categories have 10–25 cases, so a few points either way is noise.
• The GPU ran at a 400 W power limit, not the default 600 W. I haven’t measured either format at 600 W.
• Speed scenarios use fixed output lengths (ignore_eos), so tok/s compares speed only, not how much each model thinks.
• BFCL uses its own Qwen3 FC prompt, not the model’s chat template. Compare these scores with each other, not with the public leaderboard.
• Deep-AI-Evo measured a smaller gap on the same card with vLLM 0.21 and unsloth/Qwen3.8-27B-NVFP4: 50.9 → 56.4 tok/s without MTP (+11%), 115.8 → 132.3 with MTP k=2 (+14%), single-stream decode on a ~1.4K-token prompt (repo). The engine, checkpoint and speculative decoding all differ from mine, and I haven’t tested which of them explains the difference.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns both configs.
• The scripts on main keep changing. To rerun with them exactly as these runs had them, check out d4d7869.
• Serving results (one JSON line per scenario, including full server args): FP8, NVFP4.
• BFCL scores and every model response: results/<label>/bfcl/ in the same folder.

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.