Does a temperature fitted on one dataset work on another, for Qwen3.8-27B and Swift 1.5 NVFP4 served through SGLang’s decision API? Mostly yes on text. A fit pooled over the other text datasets improves the probability scores on all seven text targets for both models and keeps 49–97% of the in-domain gain (the gain from fitting on the target’s own labeled cases). A fit from one source can be worse than none: ChaosNLI, used as the source, is worse than raw on procedural and TD Bench (Typed Decision Bench) for both models.

This follows the earlier decisions post, which fitted one temperature per question type separately for each dataset. It doesn’t repeat that setup. Here I score transferred fits on other datasets alongside each dataset’s own fit where one exists (PubMedQA is scored only as an unseen target), use three passes per run to show the serving spread, and vary fit size in a learning curve. This post covers text datasets only.

Datasets and where their answers come from

Six text source datasets and seven scored text targets (Calibrated Typed Decisions’ separate PubMedQA split is the seventh target). The answers behind each score:

• Typed Decisions: the average of three samples from a teacher model that the Typed Decisions card doesn’t name beyond “roughly 4B-class”. Its scores are agreement with that teacher, not correctness.
• Procedural Typed Decisions: answers computed from rules stated in the prompt.
• Calibrated Typed Decisions: human labels. Its heldout PubMedQA split (expert labels) is a target only.
• Typed Decision Bench (“TD Bench” below): labels from 21 public datasets (human annotations and automated verifier outcomes) and one rule-derived generator.
• ChaosNLI: the share of 100 human votes per item on SNLI, MNLI and αNLI.
• LeWiDi: the share of annotators’ votes on sarcasm, offensiveness and Brexit hate speech.

Links are pinned to the measured revisions; Setup has the splits.

Results

Each decision has a question type: choice (one of N labels), noul (yes/no), score (ordered levels). A temperature T rescales each decision’s probabilities as p_i^1/T / Σ_j p_j^1/T. This is standard temperature scaling with a separate T for each of the three question types; it is not a per-class temperature, and I did not compare it with a single global T. T = 1 is the server’s output (“raw”). All numbers are stock-nvfp4 and swift-nvfp4, the mean of three passes per dataset (sessions and cache: Setup). Fits minimise mean cross-entropy to the dataset’s answer distributions (one-hot for hard labels), with T > 0. Scores use each dataset’s own metric: KL from the gold distribution (lower is better) for Typed Decisions, ChaosNLI and LeWiDi; log loss for Calibrated Typed Decisions and PubMedQA; proper score and DecisionScore (higher is better; both are 100 × (1 − normalised Brier), averaged per item) for procedural and TD Bench. I report no ECE or per-class calibration. A temperature changes confidence, not which answer ranks first, so accuracy is unchanged.

Columns: in-domain is a fit on the target’s own fit split (for TD Bench, a 5-fold cross-fit). Pooled text is one fit over every text source except the target. The two last columns are the best and worst fit from a single other text source, with the share of the in-domain gain each keeps, (x − raw) / (in-domain − raw), computed from the means in the table. Negative means worse than raw. The share compares against each target’s own measured in-domain result, not a matched-training-budget baseline: the training sets differ in size and, for TD Bench, in kind (cross-fit vs separate fit split).

Stock NVFP4:

Swift 1.5 NVFP4:

Fitted temperatures per source (choice / noul / score; - = the source has no question of that type):

Takeaways

• The pooled text fit improved all seven text targets for both models, and keeps 49–97% of the in-domain gain. It keeps 88% or more on Typed Decisions, Calibrated TD, TD Bench and ChaosNLI, 78–79% on procedural, and 49% (stock) and 65% (Swift) on LeWiDi. Each target gets its own pooled fit, which excludes that target.
• LeWiDi and procedural transfer worst, and their own temperatures are the outliers. LeWiDi’s own noul temperatures (1.58 stock, 2.05 Swift) are lower than every other text source’s (2.92–3.83, 4.08–5.08), and procedural’s own choice ones (1.35, 1.47) are the lowest of the text sources. A fit from sources that need more softening may over-soften these two; I didn’t test it.
• A single source can hurt, and ChaosNLI did. Of 33 text-to-text source-target cells per model, 3 (stock) and 2 (Swift) are worse than raw in at least one pass. Four of those five use the ChaosNLI fit (procedural and TD Bench, both models); the fifth is Calibrated TD on LeWiDi, stock, 0.397 vs raw 0.394. ChaosNLI is the only text source with choice questions alone, and its choice temperature is the highest of any source (3.11 stock, 3.71 Swift, against 1.35–2.58 and 1.47–2.79 for the other text sources).
• Pooling avoids picking a source without target labels. It improved all seven targets here, though the retrospectively best single source beats it on five of the six targets with an in-domain fit, for both models. That choice uses the target’s scores, so it isn’t something you can do on a new target. The pooled fit is never more than 11 percentage points of retained gain behind the best single source, and ahead of it on Typed Decisions, for both models. Pooling weights decisions within each question type. With all six sources in, TD Bench supplies about 60% of noul decisions but 34% of choice and 10% of score; the mix changes when the target is left out.
• On PubMedQA, a task no fit saw, every text source with choice questions beats raw (LeWiDi has none and leaves it unchanged), and the pooled fit is close to the best. Log loss 0.902 → 0.678 (stock) and 0.948 → 0.657 (Swift); the best single source is Typed Decisions at 0.676 and 0.652, the worst is procedural at 0.765 and 0.756. PubMedQA is a split of Calibrated Typed Decisions, so its fit source is related to it.
• Across three passes within one server session, the scored text metrics varied by at most 0.07 score points or 0.003 KL/log loss, and a fitted temperature by at most 0.05 (stock procedural noul, 3.35–3.40). Fit-split runs varied more (procedural validation, Swift: 0.105 points). These ranges don’t measure between-launch or dataset-sampling uncertainty, so small gaps between fits stay unresolved. Details below.
• In these five text datasets, the worst of 20 random draws kept at least 0.79 of the full-fit gain at 100 fit cases and 0.88 at 200. At 50 or fewer, a draw can be worse than none. Learning curve below.

Serving noise across passes

Each run is three passes through /v1/systemone. Each dataset’s passes ran on one server session; Typed Decisions train was collected in a separate session from the test datasets. Pass 1 precedes that dataset’s repeated passes, while passes 2 and 3 can reuse its cached prefixes. The table counts top answers that changed between pass 1 and pass 2 (pass_drift in each summary.json), stock / Swift:

ChaosNLI and Calibrated TD have identical probabilities in passes 1 and 2 (mean and max difference 0.0). The largest mean absolute probability difference is 0.0033 (stock TD Bench), but single decisions move more: the largest is 0.82 (stock TD Bench, 0.817). The transfer cells move much less than single decisions (ranges in Takeaways), but those ranges come from one session, not from separate launches or resampled datasets. A gap under them is not resolved: the Calibrated TD fit on LeWiDi (stock, 0.397 vs 0.394 raw) is the closest cell I discuss.

How many labeled cases

For each dataset with a fit split separate from its scored split, T per type was fitted on n cases drawn at random from the fit split (20 draws per n, pass 1) and scored on the same scored split as above. Cells: median share of the full-split fit’s gain over raw, with the worst of the 20 draws in parentheses. Below 0 is worse than raw.

• At n = 25 the median is already 0.73 or more on all five sets, both models. The risk is the worst draw: a Calibrated TD draw of 25 cases lands at −5.00 (stock), five times the full fit’s gain in the wrong direction, and a LeWiDi draw at −31.83. LeWiDi’s full-split gain is small (KL 0.394 → 0.327), so its share is volatile.
• At n = 100 the worst draw on every text set is at least 0.79; at n = 200, at least 0.88.
• Procedural, Calibrated TD and LeWiDi need more than the rest. Medians at n = 100 are 0.97 (procedural, stock) and 0.94 (Calibrated TD, stock); procedural reaches 1.00 at n = 200.
• A case can hold several questions, and n counts cases.

Fitting temperatures to your own data

What the data supports for a model like these two:

• Start with 100–200 labeled cases and fit T per question type. In the five text datasets above that kept at least 0.79–0.88 of the full-fit gain in the worst draw. It is a starting size, not a guarantee for a new workload. Keep separate labeled cases for evaluation and add fit cases if the held-out score calls for it.
• Score on cases the fit didn’t use. Fit on one split, check on another. At 50 cases or fewer a draw can be worse than no temperature, and you can only see that on held-out cases.
• Apply it in the client. /v1/decisions accepts a per-request temperature and /v1/systemone refuses it, per the SGLang decision docs. Scaling the returned probabilities as p^1/T and renormalising does the same either way, and chat on the same server is unaffected.
• With no target labels but labeled source datasets, a fit pooled over several sources is an option to test. It improved every text target here and avoids choosing one source. A single source worse than raw is a real possibility (ChaosNLI), and the pooled fit leaves part of the gain on the table (49–97% kept). Validate it once target labels exist.
• These are two related Qwen3.8-27B NVFP4 checkpoints, six text source datasets and seven scored text targets. The seven are not seven independent replications (PubMedQA is a split of Calibrated Typed Decisions). The evidence says nothing yet about other model families, other precisions, or question types beyond choice, noul and score.

Setup

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c (“stock NVFP4”) and ukisai/Swift-1.5-Qwen3.8-27b-NVFP4 @ 25482027 (“Swift 1.5 NVFP4”; ModelOpt mixed, no KV scales).
• Same flags as the earlier decisions post, --mem-fraction-static 0.41, no speculative decoding.

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static 0.41 --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Protocol:

• /v1/systemone, one request in flight, three passes per dataset. Each dataset’s passes share one server session; Typed Decisions train ran in a separate session from the test datasets, which ran one after another in a single session per config. Pass 1 precedes the repeated passes; passes 2 and 3 can reuse cached prefixes. Thinking off, as the decision API does.
• Datasets, all text:
  • LocalLLaMA/typed-decisions @ 1415d375, all config: train (1,200 cases) is the fit split, test (400 cases) the scored split. KL from gold.
  • SamuelChien821/typed-decision-bench v0.3.0 @ 79ec2b9b, all 5,387 items (359 rebuilt with the dataset’s own builder). It has no other split, so in-domain is a 5-fold cross-fit (stratified by task), and as a source it is fitted on all of it. DecisionScore.
  • tasksource/procedural-typed-decisions all @ 05ca1d03: a fixed sample of up to 10 states per task and level, 880 validation cases to fit and 874 test to score. Proper score (TD Bench’s per-item score, since it has no scorer).
  • m-lagnajit/calibrated-typed-decisions v2.1 @ 34189779, human labels: validation (1,799) fits, test (1,797) scores, heldout PubMedQA (1,000) scores only. Log loss.
  • earino/chaosnli @ bf8d66a1 (a mirror of ChaosNLI v1.0): one upstream split, halved by sha256 of the item id into validation (2,334 cases) and test (2,311). KL.
  • tasksource/lewidi @ 9ba2e83b: validation 1,976 to fit, test 3,929 to score. Sarcasm is score (6 levels); offensiveness and Brexit hate speech are noul. KL.
• Transfer: each source is fitted on its fit split, with one T per question type it has; a type it lacks keeps T = 1, and a cell is - where the source has none of the target’s types. Fit by transfer.py per pass, and the cells are the mean over three passes. No target is scored with temperatures fitted on its own answers. “Pooled text” is one fit over all text sources except the target, with decisions pooled within each question type.
• Learning curve: learning_curve.py, pass 1, 20 random draws per n from the fit split.

Caveats

• ChaosNLI is the worst source here, and I haven’t tested why. It has only choice questions and the highest choice temperature (3.11 stock, 3.71 Swift). Its transferred scores are worse on procedural and TD Bench. Whether over-softening their choice decisions explains those aggregate declines was not tested.
• The spread across passes comes from repeated passes within a session. It is the range over three passes of a dataset (pass 1 first, 2 and 3 able to reuse cached prefixes), not sampling error of the scored split. I computed no confidence intervals for the transfer cells, so small gaps (a few thousandths of KL, tenths of a score point) aren’t resolved.
• Typed Decisions gold is a teacher. Its cells are agreement with an unnamed ~4B-class teacher, and a fit “in-domain” there fits the teacher.
• TD Bench’s in-domain value is a 5-fold cross-fit on its own items; the other in-domain values are fits on a separate split, and pooled fits use every other source item. “Share kept” is therefore a descriptive ratio, not a matched-budget comparison.
• Pretraining overlap. SNLI and MNLI (the sources behind ChaosNLI), and several public datasets inside TD Bench and Calibrated Typed Decisions, are probably in the models’ training data [unverified]. I didn’t check.
• Learning-curve draws are from fit splits of 880–2,334 cases, so n = 200 covers 9–23% of a fit split and the curve doesn’t say what happens with a fit split of different make-up. Percentiles are over draws and say nothing about the scored split’s own sampling error.
• Two NVFP4 configs only, one image and one set of flags. BF16, FP8 and other models weren’t run here.

Raw data

• Scripts, configs and raw results: adcostanza/benchmarks. The README sections “Temperature transfer across datasets” and “How many labeled cases” hold the question, rerun commands and caveats.
• To rerun with the scripts as these runs had them, check out 2e3e934.
• results-repeat/test/<config>/<set>/ and results-repeat/train/<config>/decisions/: every raw answer (pass1.jsonl, pass2.jsonl, pass3.jsonl) and summary.json (with pass_drift).
• results-repeat/transfer.md: the full matrix, every source row against every target column with [min, max] over the three passes, and the fitted temperatures. results-repeat/learning_curve.md: every n for every set.
• Code: transfer.py and learning_curve.py.

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.