On Qwen3.8-27B NVFP4 with DFlash2 8 (seven drafted tokens plus one target token per verify step) on SGLang, 76–82% of the rejects on 880 SPEED-Bench qualitative prompts (prompts only) were at a token where the target’s own top-1 probability was below 0.9 (77–82% when each request’s cut-off last step is left out). That held for stock and Swift 1.5 checkpoints, greedy and sampled.
Treating each reject at a confident token (top-1 ≥ 0.9) as an accepted position, with the recorded tokens unchanged, raises replayed tokens per step by 12–16%. That is an optimistic replay of one intervention, not a bound on a better drafter and not a measured speedup.
Results
A verify step holds up to 8 tokens. A non-final step with fewer than 8 tokens ended at a reject: the target did not accept the next draft. A request’s last step can also be short because the server cut it at a stop token or the 512-token cap; the tables below count those as rejects too (see Caveats for the numbers without them). “p1” is the target’s top-1 probability at the reject position. “Headroom” is tokens per step in the replay with every reject at p1 ≥ 0.9 (or ≥ 0.5) removed (how). “Tokens/step” pools tokens over all verify steps, leaving out each request’s first token; the earlier posts report accept length as a mean of per-request values. Four configs, 880 prompts each, one request at a time:
- Stock:
RadixArk/Qwen3.8-27B-NVFP4. - Swift:
ukisai/Swift-1.5-Qwen3.8-27b-NVFP4. - Drafter for both:
z-lab/Qwen3.8-27B-DFlash2(the DFlash2 card). The DFlash2 card namesQwen/Qwen3.8-27Bas its target and does not mention Swift.
| Config (880 prompts) | Tokens/step [95% CI] | Reject rate | Rejects with p1 < 0.9 [95% CI] | Rejects with p1 < 0.5 | Headroom, p1 ≥ 0.9 fixed | Headroom, p1 ≥ 0.5 fixed | Out tok/s |
|---|---|---|---|---|---|---|---|
| Stock, greedy, thinking off | 3.88 [3.81, 3.97] | 81.5% | 80.2% [79.5, 80.9] | 33.0% | 4.35 (+12%) | 6.13 (+58%) | 193.9 |
| Swift, greedy, thinking off | 3.92 [3.84, 3.99] | 81.1% | 77.9% [77.2, 78.7] | 29.9% | 4.45 (+14%) | 6.29 (+60%) | 196.6 |
| Stock, sampled, thinking on | 3.60 [3.56, 3.65] | 86.0% | 81.5% [81.0, 82.0] | 38.9% | 4.05 (+12%) | 5.66 (+57%) | 179.4 |
| Swift, sampled, thinking on | 3.66 [3.62, 3.71] | 85.2% | 76.4% [75.8, 76.9] | 30.6% | 4.26 (+16%) | 6.12 (+67%) | 183.2 |
The greedy pair runs with thinking off and the sampled pair with thinking on, so greedy against sampled in this table is also thinking off against on. The 88-prompt table below has the pairs that differ in one thing only.
Where the drafter loses a step, in one greedy run (stock, thinking off, request 80 of the 880, picked for the figure; verify steps 1–14). Each box is a token; a thick bar ends a verify step. The last token of each step is where the draft failed. Blue: the target’s p1 was below 0.9. Red: it was at least 0.9.
14 steps, 47 tokens, 14 rejects. Three were at confident tokens: inflation (target p1 ≥ 0.995), three (≥ 0.995) and economic (0.96). The other eleven were at tokens like relationship (0.81 against a runner-up), Gross (0.26) and fundamental (0.56). The colour-coded text for 2 requests per category of every run is results/<label>/rejects.html in the data folder.
The 88-prompt set is the first 8 prompts per category of the 880. It has the greedy thinking-on runs and the pairs that change one setting; its prompts are drawn from the 880, and the four matching rows’ p1 < 0.9 shares differ from the 880 rows by 0.1–1.4 points (other metrics can differ more):
| Config (88 prompts) | Tokens/step | Reject rate | Rejects with p1 < 0.9 | Headroom, p1 ≥ 0.9 fixed | Out tok/s |
|---|---|---|---|---|---|
| Stock, greedy, thinking off | 3.94 | 80.2% | 79.4% | 4.46 (+13%) | 199.3 |
| Swift, greedy, thinking off | 4.02 | 79.5% | 76.5% | 4.60 (+14%) | 205.1 |
| Stock, greedy, thinking on | 3.80 | 82.4% | 86.6% | 4.11 (+8%) | n/a |
| Swift, greedy, thinking on | 3.83 | 82.5% | 83.8% | 4.20 (+10%) | 195.7 |
| Stock, sampled, thinking on | 3.63 | 85.5% | 81.4% | 4.09 (+13%) | 182.6 |
| Swift, sampled, thinking on | 3.75 | 83.7% | 75.1% | 4.40 (+17%) | 189.2 |
The first stock thinking-on run was not timed. The stock thinking-off row is the rerun with timing; the first run of that config gave 3.94, 80.2% and 79.5%.
What the drafter proposed at a reject
The tables above see only the target’s side. To see the drafter’s, I patched the server to log, at each reject, the drafter’s token and where the target ranks it. The patch only reads tensors. The hook runs were greedy, thinking off, 88 prompts. For non-final verify steps, the hook log matched the same run’s stream in accept length and committed tokens on all 7,321 stock and 6,947 Swift steps; each request’s final step matched as a prefix. Hook-only trailing steps were not compared. Hooked stock runs measured 3.97 and 3.94 tokens per step; three unhooked reruns measured 3.94, 3.94 and 3.95. Separate runs are not token-identical, so this comparison does not establish that the hook leaves acceptance unchanged. It costs speed (about 5–7% of tok/s), so it was used for this analysis only.
| 88 prompts, greedy, thinking off | Stock | Swift |
|---|---|---|
| Rejects | 5,847 | 5,539 |
| Drafter’s token was the target’s #2 | 42.9% | 40.7% |
| … #3 | 15.3% | 14.8% |
| … #4 / #5 | 7.7% / 4.8% | 8.4% / 4.9% |
| Drafter’s token in the target’s top 3 | 60.0% | 56.9% |
| … #6 or lower | 27.5% | 29.8% |
| Target’s probability of the drafter’s token, median | 0.060 | 0.044 |
| … below 0.01 | 31.4% | 35.5% |
| Both tokens punctuation or whitespace | 6.3% | 6.3% |
| One token is a prefix of the other | 5.6% | 5.7% |
| Token strings equal after stripping whitespace and lowercasing (includes whitespace-only tokens) | 1.2% | 1.5% |
The most common pair, the where the target took a, is 34 of 5,847 rejects on stock (32 of 5,539 on Swift), under 1%. Rejects are a long tail of word choices, plus articles and punctuation swapping places. That string-equality category is 1.2–1.5% of rejects and prefix pairs 5.6–5.7%; it is not a spelling-variant count, and other spelling near misses were not classified. About 1.4–1.8% of rejects have the drafter’s token ranked #1 by the hook’s recomputation; I read those as ties with the target’s chosen token in the logged logits: in all 104 stock and 78 Swift cases the two logprobs are equal and the chosen token has the lower id, consistent with a greedy argmax tie-break [INFERENCE, not checked in the server code].
Takeaways
- Most rejects are at tokens the target is itself unsure of. 76–82% of rejects have p1 below 0.9 on all four configs (880 prompts), and 30–39% have p1 below 0.5. This describes the target’s confidence at the rejects; it does not say which rejects another drafter could fix. The thinking-on greedy runs, only at 88 prompts, are higher: 86.6% stock, 83.8% Swift.
- Treating the confident-token rejects as accepted in the replay adds 12–16% tokens per step (880 prompts; +8% to +17% across the 88-prompt runs). Treating every reject at p1 ≥ 0.5 that way instead adds 57–67%. The replay assumes those rejects stay fixed and later tokens keep being accepted as recorded, so both are optimistic. How many of the 0.5–0.9 rejects a drafter could learn is not measured here.
- Swift’s rejects have a larger share at confident tokens than stock’s. The share with p1 below 0.9 is 2.3 points lower on Swift greedy (77.9% vs 80.2%) and 5.1 points lower sampled (76.4% vs 81.5%); the request-resampled intervals don’t overlap, and each cell is one run. Swift’s target also has a different confidence profile overall, so this does not isolate how well the drafter fits Swift. Tokens per step are 0.04 (greedy) and 0.06 (sampled) higher on Swift on these matched prompts; each configuration ran once, and the per-run intervals overlap but do not measure run-to-run variation.
- On the 88 matched thinking-on prompts, the sampled run had 3.63 tokens per step against 3.80 for greedy on stock, and 3.75 against 3.83 on Swift; each config ran once.
- The rejected draft is not necessarily a close alternative. It ranks #2 under the target in 41–43% of rejects and in the top 3 in 57–60%. Its median target probability is only 0.04–0.06, and in 31–36% of rejects the target gave it under 1%. Rank alone does not say whether it was a close alternative.
- Category averages span more than the checkpoint gap. In the stock greedy thinking-off run, tokens per step by category range from 2.53 (roleplay) to 5.98 (coding), against 0.04 between stock and Swift overall. Those are averages for this prompt mix, not isolated task effects. Across the four 880-prompt runs, 21–23% of steps accept no draft and 14–19% accept all 7.
Headroom: the replay
headroom.py takes the recorded output of a run, keeps each reject whose p1 is below the threshold, treats the others as accepted positions, and re-chunks the token stream into verify steps of at most 8 tokens. A different drafter would propose different later tokens, so treat the numbers as one optimistic intervention.
Setup
Serving stack:
- Image:
lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48(package0.0.0.dev1+g961bcf481), the image of the spec-decode and Swift vs stock posts. The hook runs bind-mount a patched copy of the image’s own Python package over it (hook/); all other runs are unpatched. - Target checkpoints:
RadixArk/Qwen3.8-27B-NVFP4@319f741c,ukisai/Swift-1.5-Qwen3.8-27b-NVFP4@25482027. Drafter:z-lab/Qwen3.8-27B-DFlash2@50307d4c, the same for both. The DFlash2 card describes it as a block-diffusion drafter that predicts a block of tokens in one pass, keeps the top candidates at every position and picks one path with a selector; it says decoding is lossless (greedy output matches the target, sampling preserves its distribution). - Speculative decoding: DFlash2,
--speculative-num-draft-tokens 8(7 drafts plus one target token per step).
Launch flags:
python3 -m sglang.launch_server --model-path <checkpoint>
--mem-fraction-static 0.80 --max-running-requests 4 --context-length 153600
--dtype auto --attention-backend flashinfer
--reasoning-parser qwen3 --tool-call-parser qwen3_coder
--speculative-algorithm DFLASH --speculative-draft-model-path <DFlash2>
--speculative-num-draft-tokens 8
--mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
--mamba-radix-cache-strategy extra_buffer_lazy
--chunked-prefill-size 2048 --page-size 64
--enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
--kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
--sleep-on-idle --trust-remote-code
--mem-fraction-static is 0.80 here and 0.41 in the earlier posts; every other flag is the same. I did not test whether it changes tokens per step.
Protocol:
- Prompts: SPEED-Bench
qualitativesplit at revisionfbd6dc64(NVIDIA Research; 880 prompts, 80 in each of 11 categories: coding, math, humanities, STEM, writing, summarization, roleplay, RAG, multilingual, reasoning, QA; aggregated from 18 public sources and selected by NVIDIA for semantic diversity). It holds prompts only, no reference answers. 494 of the 880 released rows are masked placeholders; I resolved them from the public sources with NVIDIA’s own resolver (NVIDIA/Model-Optimizer@2802302b). First turn of each row only. The 88-prompt set is the first 8 per category in file order. The Swift vs stock post ran SPEED-Benchqualitativetoo (110 prompts, all turns, a 4,096-token output limit, reasoning effort medium, server-default sampling) and measured 4.35 (stock) and 4.36 (Swift) tokens per step. The 3.6–3.9 here is lower. Part of that is the definition: the earlier post’s per-request definition (mean of completion tokens over verify steps, first token included) gives 4.26 (stock) and 4.30 (Swift) on the greedy 880-prompt runs here, against 4.35 and 4.36 there and the pooled 3.88 and 3.92. The remaining 0.09 and 0.05 come with the setup differences above, and I did not isolate their causes. - Each prompt goes through the checkpoint’s chat template with
enable_thinkingfalse or true, one request at a time, 512 output tokens. - Counting steps: with one request in flight, SGLang’s
/generatestream sends one chunk per scheduler step (the first is the prefill’s token, each later one a verify step). Every request in every run satisfiedchunks == spec_verify_ct + 1, so no chunks were merged.return_logprobwithtop_logprobs_num 2gives the target’s top-2 log probabilities at each emitted token. - Greedy:
temperature 0. At a reject the step’s last token is the target’s argmax. - Sampled:
temperature 1.0, top_k 20, top_p 0.95, read from the target’sgeneration_config.json(identical in these fields for both checkpoints) and sent by the client, withsampling_seed = 1234 + ifor request i. DFlash2’s sampling verify path decides acceptance; the step’s last token is drawn from the target’s residual distribution, so a sampled “reject” is a verify reject: the draft failed the accept test. p1 is the target’s top-1 probability at that position, whichever token was drawn. The server’s logprobs there arelog_softmax(logits / temperature), before top-k and top-p; at temperature 1.0 that is the untruncated distribution, while acceptance uses the truncated one. - tok/s: all output tokens of the run divided by the summed per-request wall time at the client (prefill, HTTP and streaming included). With one request at a time, this is an end-to-end single-request output rate, not multi-user throughput.
- 95% intervals resample requests (1,000 bootstrap draws).
Caveats
- One run per config. Sampled runs are one draw; the sampling verify step in this image draws its accept coins from the global torch RNG [INFERENCE from
dflash_utils.py, not tested], so rerunning the seeds is not guaranteed to reproduce the text. Greedy reruns of the same config differ in the text of 5–16 of 88 requests (source not isolated); tokens per step moved by at most 0.01 between identical reruns, and tok/s by about 0.5 (199.3 vs 199.9). The intervals above cover prompt-to-prompt spread, not that run-to-run noise. - The 880-prompt greedy and sampled pairs differ in thinking as well as sampling. The thinking-on greedy pair was not scaled to 880.
- 512 output tokens. In the thinking-on runs only 37–51% of requests emit
</think>within the cap (36.8% stock and 45.0% Swift in the 880-prompt sampled runs), so most requests are cut off inside the thinking. - p1 is a proxy. A reject at p1 = 0.7 can still be a drafter that could have picked the target’s argmax; p1 below 0.9 means the target was not sure, not that no drafter could get it. For sampled runs acceptance depends on the proposed token and the truncated distribution, so p1 is not the acceptance probability.
- The hook costs about 5–7% of tok/s (stock: 189.2 and 188.1 tok/s with it against 199.3 and 199.9 without; Swift: 191.1 against 205.1; greedy thinking off, 88 prompts). The rank table is from 88 prompts, greedy, thinking off, and only for those two checkpoints.
- This is a mechanism study on SPEED-Bench prompts (a fixed mix of 11 task types). One drafter, NVFP4 targets, one request at a time.
- A request’s last step is cut short by the server at a stop token or the 512-token cap, so it can look like a reject. The 880-prompt tables count it, at most one per request (827–843 of 60,697–91,529 rejects). Without those steps the p1 < 0.9 shares are 80.9 / 78.6 / 81.8 / 76.8% (stock greedy, Swift greedy, stock sampled, Swift sampled), 77–82% instead of 76–82%. The hook table leaves them out.
Raw data
- Scripts, configs and raw results: adcostanza/benchmarks.
run.shreruns a config;analyze.py,table.pyandbootstrap.pyproduce the numbers above. - The scripts on
mainkeep changing. To rerun with the harness as committed with these results, check outb984d34. - The folder does not contain the SPEED-Bench prompts: SPEED-Bench’s license (NVIDIA Evaluation Dataset License) does not allow redistributing it.
scripts/speed/prepare.shandmake_prompts.pyrebuild them (README, Method). - Per-run
steps.jsonl(token ids per verify step plus the target’s top-2 log probabilities),summary.json,rejects.htmlandconfig.jsonare inresults/<label>/; the hook runs addhook.jsonl.
Test system
- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
- PCIe: Gen 4 x16 (the card supports Gen 5;
nvidia-smireports Gen 4 as this system’s maximum). - CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
- RAM: 250 GiB DDR5.
- OS: Ubuntu 24.04.2, kernel 6.8.0-147.
- NVIDIA: driver 580.178.04, CUDA 13.0.
- Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
- Full list: hardware.md at the commit these runs used.