•
10 min read
Inkling 2 Mini reaches 2.2x Qwen3.8-27B's output rate on a 120k prompt in 41.5 GiB, 1.2x at chat, and scores lower on GSM8K

On one RTX PRO 6000 with SGLang, Inkling 2 Mini with its bundled DSpark drafter (SGLang cookbook recipe, here at --mem-fraction-static 0.34), in 41.5 GiB of GPU memory, produced 1.2x the aggregate output tokens per second of Qwen3.8-27B NVFP4 + DFlash2 8 (42.8 GiB) at chat with one user, rising to 2.2x on a 120k-token prompt. Qwen has the higher rate at chat with four users (592.5 vs 545.2 tok/s). One run per config, on synthetic serving scenarios; the Qwen runs are from October on another SGLang image with FP8 KV, Mini’s were run on November 21 with bf16 KV.

On GSM8K (grade-school math with human-written reference answers), Mini gets 1,241 of 1,319 against Qwen’s 1,277: Qwen leads by 36 answers (2.7 points), more than twice the standard error of the gap, one run each. On a 205-case BFCL subset (function calls checked by BFCL’s automatic graders) Mini passes 156 against 168, but the two models ran through different BFCL handlers, so that gap is not a model comparison. GSM8K and BFCL ran on Mini’s separate no-drafter Balanced recipe at the default memory fraction, not the 41.5 GiB speed config. No real-world or coding tasks were run.

Results

Mini at --mem-fraction-static 0.34 with bf16 KV and the flags of the Qwen server (spec-decode post: 153,600 context, hierarchical cache, lpm scheduling, session radix cache) against that Qwen server at 0.41. Output tok/s is aggregate output tokens over the benchmark’s wall time, prefill included; median time to first token (TTFT) in parentheses.

ScenarioQwen3.8-27B + DFlash2 8Inkling 2 Mini + DSparkΔ
Chat, 1 user178.0 (48 ms)215.9 (51 ms)+21%
Chat, 4 users592.5 (73 ms)545.2 (89 ms)−8%
20k prompt, 1 user98.4 (2.35 s)128.7 (1.28 s)+31%
20k prompt, 4 users137.5 (5.73 s)213.3 (3.17 s)+55%
40k prompt, 1 user58.7 (5.57 s)92.1 (2.76 s)+57%
80k prompt, 1 user28.5 (14.8 s)54.7 (6.41 s)+92%
120k prompt, 1 user16.5 (27.8 s)35.9 (11.0 s)+118%
GPU memory after load42.8 GiB41.5 GiB−1.3 GiB
Drafter accept length, chat 1 user3.432.71
GSM8K, correct of 1,319 (Mini: no-drafter recipe)1,277 (96.8%)1,241 (94.1%)−36
BFCL subset, correct of 205 (Mini: no-drafter recipe; different handlers)168156−12

Inkling 2 Mini (41.5 GiB) aggregate output tok/s as a multiple of Qwen3.8-27B + DFlash2 8 (42.8 GiB), one user, one run each

The Qwen speed numbers are the runs from the spec-decode post, and the GSM8K and BFCL numbers are from the Swift vs stock post, same server config; its 42.8 GiB (43,849 MiB) is gpu_mem_after_load_mib in that folder’s results/dflash8/resources.json. Neither was rerun for this post. Mini’s GSM8K and BFCL come from the SGLang cookbook’s no-drafter recipe at SGLang’s default memory fraction (mini-balanced), not from the 0.34 config.

41.5 vs 42.8 GiB is GPU memory after load at different memory fractions and KV dtypes. It isn’t equal KV capacity (Mini’s full-attention pool holds 199,424 tokens, Qwen’s 236,864) and it leaves out host memory.

Takeaways

  • At 80k and 120k, most of each request’s time comes before the first token (120k: TTFT 11.0 vs 27.8 s, median time per output token 6.09 vs 6.53 ms). These runs don’t isolate the models’ prefill speed, because the two images and their chunked-prefill sizes differ.
  • At the same --mem-fraction-static 0.41, Mini takes 47.3 GiB, 4.5 GiB more than Qwen: the fraction covers weights and KV, and CUDA graphs and workspaces sit outside it. At 0.34 it drops to 41.5 GiB with a 199,424-token full-attention KV pool. That is more than one 153,600-token context, not four at once, and I didn’t test full-length requests. Speed at 0.34 and 0.41 differs by under 1% in all seven scenarios.
  • FP8 KV, which the Qwen config uses, fails at startup on this image with the recipe’s fa4 attention backend: AssertionError: q.dtype torch.bfloat16 != k.dtype torch.float8_e4m3fn while capturing the first prefill CUDA graph. Mini runs bf16 KV here.
  • DSpark adds 45% at chat with one user over Mini without a drafter (231.3 vs 159.5 tok/s with the cookbook recipes), but accepts 2.3–2.7 tokens per step against DFlash2’s 3.0–3.5 on Qwen. Qwen has the higher aggregate output rate at chat with four users (592.5 vs 545.2); these runs don’t show why.
  • GSM8K is 94.09% ± 0.65 vs 96.82% ± 0.48, both at reasoning_effort=medium with the same question text and answer extraction, each through its own chat template. Mini used fewer tokens (276 vs 348 mean output).
  • BFCL: 156 vs 168 of 205. Qwen ran BFCL’s Qwen/Qwen3-32B-FC raw-prompt handler at its default effort; Mini has no raw-prompt handler in BFCL, so it ran over chat completions with tools at its template’s default effort. Two Mini cases ended in client timeouts and count as failures. The same Qwen server scored 176 in another run, so even the Qwen side moves by 8 cases between runs.

Mini configs

Config--mem-fraction-staticGPU memoryChat, 1 user120k, 1 user
Cookbook DSpark recipe (mini-dspark)default79.6 GiB231.336.0
Cookbook Balanced recipe, no drafter (mini-balanced)default87.8 GiB159.533.2
DSpark + Qwen config flags, bf16 KV (mini-dspark-daily-bf16kv)0.4147.3 GiB215.935.9
Same at 0.34 (mini-dspark-034-bf16kv)0.3441.5 GiB215.935.9

The configs with the Qwen server’s flags are within 1% of the cookbook DSpark recipe in all seven scenarios except chat with one user (215.9 vs 231.3). They differ from it in context length, hierarchical cache, scheduling and session radix cache; one run doesn’t establish whether those flags account for the roughly 7% chat difference. The SGLang cookbook reports 94.24% on GSM8K for its Balanced recipe on this card and 93.86% for its DSpark recipe, with temperature 0, an 8,192-token cap and no effort override (benchmark record), not this run’s protocol.

Setup

Serving stack:

  • Mini: thinkingmachines/Inkling-2-Mini @ ed40e7da (26.4B-parameter looped MoE, 3.4B active per pass and about 5.4B effective active per token across three passes, native NVFP4 + MXFP8, per the Inkling 2 Mini model card; the DSpark drafter ships in the repo as mtp.safetensors). Image lmsysorg/sglang:inkling-2-mini (sha256:efac853b46811ccb760545b7b29e7119bdf907537ccf6e863fac3011fe8b98de).
  • Qwen: RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c with z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c, 8 draft tokens, FP8 E4M3 KV, image lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48. Flags in the spec-decode post.
  • Requests use reasoning_effort=medium (Mini’s chat template renders it as effort 0.7) for serving and GSM8K.

Mini launch flags for the 0.34 config (the SGLang cookbook’s RTX PRO 6000 DSpark recipe, then the Qwen server’s flags where they apply):

python3 -m sglang.launch_server --model-path <Inkling-2-Mini>
  --tp 1 --ep 1 --attention-backend fa4 --dtype bfloat16 --page-size 128
  --speculative-algorithm DSPARK --speculative-num-draft-tokens 8
  --max-running-requests 4
  --grammar-backend xgrammar --enable-multimodal
  --reasoning-parser inkling --tool-call-parser inkling2
  --mem-fraction-static 0.34 --context-length 153600
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --schedule-policy lpm --enable-session-radix-cache --sleep-on-idle

with four environment variables from the DSpark recipe (SGLANG_OPT_INKLING2_VERIFY_REL_BIAS_KERNEL=1, SGLANG_OPT_DSV41_DSPARK_MID_FOLD=1, SGLANG_OPT_DSV41_DSPARK_PREPARE_INPUTS_ONCE=1, SGLANG_OPT_DSV41_DSPARK_SKIP_PREFILLED_INPUTS=1). Not carried over from the Qwen server: its Qwen-specific mamba flags (SGLang allocated no SSM state for Mini in this run), page size 64 and chunked prefill 2048.

Serving: python3 -m sglang.benchmark.serving --backend sglang-oai-chat --flush-cache --warmup-requests 2 --seed 1.

  • Chat: ShareGPT prompts, 1024 output tokens; 12 prompts at 1 user, 32 at 4 users.
  • 20k–120k prompt: SGLang random dataset (one ShareGPT prompt repeated to length), 512 output tokens; 4 prompts at 1 user, 8 at 4 users (20k only).

GSM8K @ 740312ad: all 1,319 test questions, zero-shot, answer in \boxed{}, graded against the dataset’s human-written reference answers with sgl-eval’s math_equal.

BFCL (Berkeley Function Calling Leaderboard V4): a fixed 205-case subset across 10 categories (case ids), temperature 0.001, scored by BFCL’s own graders (grader passes, not human judgments). Handlers differ per model, above.

Caveats

  • One run per config, with 4–32 prompts per speed scenario and no variance reported. Each scenario runs two warmup requests, then flushes the cache before the measured ones; the measured 20k–120k requests show no cached prompt tokens in the server logs. They still repeat one ShareGPT prompt to length, so their accept lengths, throughput and TTFT are not results for distinct, real long documents.
  • The Qwen runs are from October on another SGLang image, not rerun beside Mini.
  • GSM8K and BFCL are not coding or agent tasks, and I ran no real-world tasks. GSM8K quality was measured on the cookbook’s no-drafter recipe, not the 0.34 config.
  • Mini runs bf16 KV and Qwen FP8 KV, because FP8 KV fails on Mini’s recipe (above). I didn’t try another attention backend.
  • Both servers have a 64 GB hierarchical cache in host RAM; machine-wide host memory peaked at 95.3 GiB during Mini’s 0.34 run. Mini’s largest host buffer failed mlock under this run’s memlock limit, and I didn’t measure the host cache under memory pressure.
  • The GPU ran at a 400 W power limit, not the default 600 W.

Raw data

  • Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns every Mini config; the folder README has the full tables.
  • The scripts on main keep changing. To rerun with the harness as committed with these results, check out 53cb9b9.
  • Serving results (one JSON line per scenario, including full server args): results/<label>/bench-serving.jsonl. GSM8K and BFCL answers and scores: results/mini-balanced/gsm8k/ and results/mini-balanced/bfcl/. The failed FP8 KV start: results/mini-dspark-daily/server.log.

Test system

  • GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
  • PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
  • CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
  • RAM: 250 GiB DDR5.
  • OS: Ubuntu 24.04.2, kernel 6.8.0-147.
  • NVIDIA: driver 580.178.04, CUDA 13.0.
  • Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
  • Full list: hardware.md at the commit these runs used.