Can Qwen3.8-27B, served for chat through SGLang, double as a decision model? Does that hold for Swift 1.5’s fine-tune, and at NVFP4?

All seven configs ran on Typed Decisions: 400 test cases from four workflows (agent traces, customer service, invoices, security incidents), five questions each. Its answers are the average of three samples from a teacher model that the dataset card doesn’t name beyond “roughly 4B-class”, so its scores measure agreement with that teacher, not correctness.

Three more datasets have answers that don’t come from a model. Typed Decision Bench uses labels from 21 public datasets and one generator. Procedural Typed Decisions computes answers from rules stated in the prompt. Calibrated Typed Decisions has human labels, plus a PubMedQA split with expert labels. Only the two NVFP4 configs ran on these three, so they compare Swift with stock, not the quantizations.

On the two NVFP4 configs, both models come out of the server overconfident: every fitted temperature is above 1. One temperature per question type, fitted per dataset and applied in the client, improves the aggregate probability scores on all four datasets without changing a top answer. On Calibrated Typed Decisions’ human labels, log loss drops from 0.60–0.67 to 0.44–0.46 and ECE from 0.095–0.099 to 0.010–0.011; the same fit lowers log loss on the unseen PubMedQA task from 0.90–0.95 to 0.68–0.69. On Typed Decisions, NVFP4 and BF16 agree with the teacher at similar rates, though about 9–10% of their top answers differ. Observed Swift–stock gaps are under 2 points, in both directions.

Results

SGLang answers a decision from a chat model by reading the probabilities of the answer labels, with thinking off (docs). Typed Decisions test: 400 cases, 2,000 decisions, one request per case through /v1/systemone, the dataset’s own scorer.

Agreement means the model’s top answer matches the gold’s top answer. Swift is the more confident one; it doesn’t agree much more.

Calibration on the two NVFP4 configs, fitted on train (1,200 cases) and scored once on test:

Datasets with real answers

The Typed Decisions gold above is a teacher’s opinion; these three don’t come from a model (second column). Same calibration method, fitted separately for each dataset; no item is scored with a fit that used its answer.

DecisionScore and proper score are 100 × (1 − normalised Brier), averaged per item, so they reward probabilities, not just the top answer. Lower log loss is better.

Takeaways

• Per-type temperatures improved the aggregate probability scores on all four datasets. One temperature per question type: two on Calibrated Typed Decisions, three elsewhere. Each dataset gets its own fit. Typed Decision Bench uses five folds, each scored with a fit on the other four. No item is scored with parameters fitted on its own answer. Every fitted temperature is above 1, so both models are too sure of their answers. On Calibrated Typed Decisions test (human labels), ECE drops from 0.095 to 0.010 (stock) and 0.099 to 0.011 (Swift), log loss from 0.60 to 0.46 and 0.67 to 0.44. That is the aggregate: log loss improves in 10 of its 12 source and type groups for each model, gets worse on CLINC and Banking77 choice questions (CLINC 0.06 → 0.10), and GoEmotions stays poorly calibrated (ECE 0.18–0.23 after). Typed Decision Bench gains 3.1 (stock) and 4.0 (Swift) points of score, procedural 2.2 and 3.0 (all intervals above 0). A temperature can’t change the top answer, so accuracy doesn’t move.
• Temperatures helped on the one unseen task, PubMedQA. The fit never saw it, so no prior applies and accuracy can’t change. PubMedQA is all choice questions, so only the Calibrated Typed Decisions choice temperature acts on it. It still cuts log loss from 0.90 to 0.69 (stock) and 0.95 to 0.68 (Swift), and ECE from 0.15 to 0.06. The temperatures fitted together with priors do a little worse there (0.72 and 0.70). It is one task, so this is one data point for transfer.
• Calibration runs in the client. /v1/decisions accepts a per-request temperature; /v1/systemone refuses it, and neither has a bias field. So both calibrations run on the probabilities the client gets back, and chat on the same server is untouched.
• On Typed Decisions, the temperatures cut KL from the teacher by about two-thirds: Swift 0.676 → 0.198, stock 0.542 → 0.203. Fitted there, they are 2.6–3.3 for stock and 2.8–5.0 for Swift, highest on Swift’s yes/no questions (4.95).
• Priors add accuracy where labeled examples exist for the same questions (one log-prior per workflow, task or source, question and option, temperatures refit alongside): +5.5 (stock) and +3.6 (Swift) points on Calibrated Typed Decisions, and another 2.9–3.7 points of score on top of the temperatures on the other two. On Typed Decisions they lift agreement by 0.077 (stock) and 0.065 (Swift), to 0.743 and 0.748. Both are just above the teacher self-agreement of 0.735 reported on the Typed Decisions card, which reads scores well above it as learning the teacher’s quirks, so this is agreement with that teacher, not getting more right. New questions need labeled examples before priors apply.
• Observed Swift–stock gaps are under 2 points, in both directions (points of accuracy, teacher agreement or score); some intervals reach past 2. Uncalibrated, Swift leads on human and expert labels (+1.4 points of accuracy on Calibrated Typed Decisions test, interval [+0.6, +2.3]; +1.5 on PubMedQA, [+0.1, +2.9]) and on agreement with the Typed Decisions teacher (+1.3 at BF16 and +1.75 at NVFP4, intervals above 0; +0.55 at FP8, includes 0). It trails by 0.55 points of score on Typed Decision Bench ([−0.86, −0.25]) and is level on procedural (+0.10, [−0.63, +0.84]). After temperatures and priors, only procedural (+0.66, [+0.04, +1.28]) and PubMedQA, which calibration doesn’t change, exclude 0. That is about ten comparisons with no correction for multiple testing, so the gaps that barely exclude 0 are weak evidence. Swift is also more confident: its Typed Decisions KL is 0.66–0.71, stock’s 0.54–0.59.
• Quantization doesn’t measurably change the agreement rate with the Typed Decisions teacher. Every BF16 → FP8 and BF16 → NVFP4 gap is under 0.01, and every 95% interval includes 0 (widest bound +0.021; paired bootstrap over cases). That is the rate, not the same answers: against BF16, FP8 changes 3.1–3.2% of the 2,000 top answers and NVFP4 8.8–9.9%, gaining about as many matches with the teacher as it loses. KL is 0.03–0.05 lower at NVFP4 than at BF16 (no interval on that). NVFP4 also answers in less than half the time of BF16 (166 vs 386 ms per case). The real-answer datasets ran at NVFP4 only, so they don’t test quantization.
• Drafter and API don’t change the answers with a cold prefix cache: at most 3 of 2,000 top answers move, the same as a repeat of the same pass. A decision is prefill only, so DFlash2 has nothing to speed up. A warm cache does change them; see Caveats.
• For scale: on the Typed Decisions card’s zero-shot table, decision fine-tunes of the same base score 0.728 (Cloudflare/clef) and 0.753 (perplexity-ai/pplx-decider-v1.1-27b); both list Qwen/Qwen3.8-27B as base in the Hugging Face API, checked 2026-10-30. Raw Swift NVFP4’s 0.683 would be 27th of the table’s 39 rows (roughly; see Caveats). The calibrated rows are fitted on train, so they belong in the Typed Decisions card’s second table, where Amos 1 large’s priors go from 0.761 to 0.8025 (+0.0415). A similar fit on Swift (priors plus temperatures) is +0.065.
• On the Typed Decision Bench leaderboard, scoring its published answer files with the same scorer reproduces the published DecisionScores to two decimals. Uncalibrated, both NVFP4 models land between OpenJev 0.1 (75.56) and decider-2b-v8 (77.56), 4.5–5.1 points under the top entry, Jev 1.13.0 (81.70). Calibrated they score 2.3–2.5 above Jev, but that fit comes from the test items themselves (the set has no other split; 5 folds, each scored by a fit on the other four), so it isn’t a like-for-like row.

Setup

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: Qwen/Qwen3.8-27B @ 1d4bf0f2, Qwen/Qwen3.8-27B-FP8 @ 017b9c7a, RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c, ukisai/Swift-1.5-Qwen3.8-27b @ b4c84d42, ukisai/Swift-1.5-Qwen3.8-27b-NVFP4 @ 25482027. Swift FP8 is my own block-FP8 conversion of Swift BF16 (128×128 blocks, stock FP8’s layout); realderpz/Swift-1.5-Qwen3.8-27B-FP8 is a public equivalent (llmcompressor FP8_BLOCK).
• Drafter on the DFlash2 row only: z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c, 8 draft tokens.
• Same flags as the earlier posts, no speculative decoding except the DFlash2 row. --mem-fraction-static 0.76 for BF16, 0.50 for FP8, 0.41 for NVFP4.

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static <0.76|0.50|0.41> --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Benchmark:

• LocalLLaMA/typed-decisions @ 1415d375, all config. Four workflows (agent trace observability, customer service, invoice processing, security incidents), five questions per case, three types: yes/no (noul), one of N labels (choice), ordered levels (score). test has 400 cases; train has 1,200 cases from a separate generation run.
• One request per case (about 800 input tokens, all five questions), 1 in flight. Two passes on /v1/systemone and one on native /v1/decisions. The headline numbers are pass 1. All three passes ran with an almost empty prefix cache (about 5% of prompt tokens cached, server log).
• Real-answer datasets, one /v1/systemone pass: Typed Decision Bench v0.3.0 @ 79ec2b9b, all 5,387 items, the 359 it withholds rebuilt with its own builder (blobfishai/jevfish @ 80262b5) and checked against its SHA256SUMS; its DecisionScore and accuracy. Procedural Typed Decisions all @ 05ca1d03, a fixed sample of up to 10 states per task and level (874 test, 880 validation for the fit); it has no scorer, so Typed Decision Bench’s per-item proper score. Calibrated Typed Decisions v2.1 @ 34189779, its human labels (not its soft teacher labels), validation for the fit; accuracy, log loss, Brier, ECE. Typed Decision Bench is calibrated by 5-fold cross-fitting within its items.
• Typed Decisions scoring: its scorer, unchanged. Agreement of the top answers, KL(gold ‖ prediction), Brier.
• Typed Decisions calibration: calibrate.py fits by soft cross-entropy against its train gold. “T per type” scales each distribution as p^1/T, with one T for each question type. “T + priors” also multiplies each option by e^b, with one b per workflow, question and option (L2-penalised), and the three temperatures are refit with them (stock 2.00/2.05/2.43, Swift 2.25/3.30/2.71 for choice/noul/score). It is scored once on test.

Caveats

• The Typed Decisions gold is a teacher, not ground truth. Per the Typed Decisions card, it is the mean of three samples from a model of roughly 4B-class capability; it doesn’t name the model. Its references: label frequencies alone score 0.470, and a fresh teacher sample scores 0.735 against the rest. “Calibrated” here means calibrated to that teacher. The card reads scores well above 0.735 as learning the teacher’s quirks; my 0.743 and 0.748 are only about 0.01 above, so I read them as fitting the teacher, not as more correct.
• The Typed Decisions intervals come from 400 cases, resampled with all five decisions of a case kept together. The real-answer intervals resample cases within task or source groups. Swift’s lead over stock is small: 0.013 at BF16, 0.0175 at NVFP4 (0.683 vs 0.666 in the table are rounded), with a NVFP4 interval of [+0.003, +0.033]. At FP8 the interval includes 0.
• The Typed Decisions leaderboard rows are only roughly comparable. Most were scored on vLLM or a vendor API. Mine are SGLang on a chat model. The comparison is fair between my configs (same prompts, same scorer), not against the leaderboard’s absolute numbers.
• I left ECE out of the Typed Decisions results. Scored against its own top answers, that dataset’s gold has an ECE of 0.341 on test. This ECE compares confidence with agreement on the teacher’s top label, not with its full distribution, so matching the teacher exactly doesn’t minimise it. KL and Brier are the calibration numbers there. ECE for every row is in the repo.
• I fitted calibration for the two NVFP4 configs only, per dataset. The fitted temperatures differ by dataset (temperatures alone: 1.3–2.0 for choice and 2.9–4.6 for noul on procedural and Calibrated Typed Decisions), so a fit from one dataset isn’t shown to carry to another. The one transfer test is PubMedQA, a single task the fit never saw, scored with the Calibrated Typed Decisions fit.
• A warm prefix cache changes answers. The Typed Decisions noise checks ran cold: the server keeps 117 Mamba states, too few for 400 cases, so pass 2 found its prefixes evicted (my reading of the log, not tested). Rerun on 50 cases with the cache warm, 68–76% of prompt tokens cached: 25 (stock) and 34 (Swift) of 250 top answers change between a cold and a warm pass of the same prompt, up to 0.58 in probability; two warm passes through the two APIs differ on 8 and 10. Agreement over the 50 cases moves by under 0.01. The Typed Decisions tables are mostly cold-cache numbers. The real-answer runs were not: about 30% of Typed Decision Bench and 16% of procedural prompt tokens came from the cache. A single served decision depends on what is in the cache. A follow-up on Swift NVFP4 varied the cache, precision and deterministic-inference settings. In the variants tested, answers changed only with the hierarchical cache on; without it, 0.9% of prompt tokens came from the cache on every pass and repeats were identical. BF16 KV still changes 30 of 250; deterministic inference with Triton attention halves the changes (34 to 17) but doesn’t remove them, though SGLang means that mode to make cached and uncached prefill identical. SGLang’s own cached-vs-uncached check (test_deterministic --test-mode radix_cache) fails on these flags, with logprobs 0.49 apart (0.20 with deterministic inference, Triton attention and BF16 KV). Open SGLang PRs #43110 and #46707 fix a hierarchical-cache Mamba restore that leaves an active request’s state stale; whether that is the cause here isn’t tested. Details in the repo README, “Prefix cache ablation”.
• The real-answer runs are one pass on two configs. Procedural is a sample of 874 of 18,000 states. On Typed Decision Bench, the leaderboard systems are scored without any fit.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. run.sh there reruns all seven configs in about an hour. The README has per-type tables, the noise checks, the real-answer and cache tables with paired intervals, and the rerun commands.
• The scripts on main keep changing. To rerun with them exactly as these runs had them, check out d445be4 for Typed Decisions, 4a91a4c for the real-answer and cache runs, and e8e2a51 for the cache ablation.
• results/<config>/decisions/: every raw answer (pass1.jsonl, pass2.jsonl, native1.jsonl) and summary.json. results-train/ holds the train-split answers for the two NVFP4 configs.
• results/{stock,swift}-nvfp4/calibration.json: the fitted temperatures and priors, and the test metrics. Code: scripts/decisions/calibrate.py.
• paired_ci.py and results/paired-agreement.txt: the paired gaps and bootstrap intervals.
• results-real/: raw answers per dataset and split, fitted calibrations, and tables.md (every row with paired intervals, from real_answers.py). results-cache/: the 50-case warm-cache run; results-cache-ablation/: the cache, precision and determinism follow-up (cache-ablation.sh, cache_check.py). Dataset adapters and scorers: scripts/decisions/sets.py.

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.