RadixArk’s NVFP4 Qwen3.8-27B on SGLang scores 66.09 on the public half of the Decision Index vision score, with all 12,210 questions answered. Scores are chance-corrected agreement with each source benchmark’s own answer key, not an independent truth. Among the board’s 62 entrants, 29 score higher and 33 lower. All 20 entrants I classed as tuned 27B-class models score 68.13 to 77.93 (median 74.91), and the board’s Qwen3.5-397B reference scores 68.52. Swift 1.5 NVFP4 scores 65.80; with one pass each, the 0.29-point gap sits inside a paired bootstrap interval of −0.82 to +1.52.

The tuned entries’ lead is small on KIE (median +1.4) and large on CharXiv (+14.7) and Hateful Memes (+14.3). On seven of the 11 benchmarks, between 1 and 3 of the 20 score below stock. Part of the gap may be quantization and engine rather than tuning: the board lists a Qwen3.8-27B base at 68.54 for one entry (BF16, its own code).

There is no Full vision score: the maintainers’ private half wasn’t run. The run departs from the kit’s reference setup in the endpoint, the Mind2Web font and the R-Bench-M and MMMU-Pro prompt, and ran with the prefix cache on. Details are under Caveats.

The suite and where its answers come from

The kit’s docs/vision.md (kit commit 25c6ec7b, vision edition 0.3) rebuilds eleven public benchmarks into one letter-choice question per row, over one or two images (no row has more). The answer key is each source benchmark’s own label, converted by the kit’s scripts; its conversion list uses no model to label anything. The kit’s table doesn’t say who annotated each source’s labels, and I didn’t check the source papers. Four sets get distractor options built by rule from the same data (CharXiv, InfographicVQA, KIE, Mind2Web); the rest keep their native options.

Per benchmark, skill = (accuracy − chance) / (1 − chance) × 100, where chance is the mean of 1 / (number of options) over the benchmark’s rows. The public index is the weighted mean of the skills (each clipped to 0 to 1) times 100, so on the two-option sets (Hateful Memes, Winoground) skill is twice the distance above 50% accuracy: 65.05% is a skill of 30.09. R-Bench-M and MMMU-Pro are headroom sets with low weight: of the 9.75 total weight they hold 5% and 3%.

Results

Board numbers are the public columns of data/vision.json in the Hugging Face Space multimodalart/jev-decision-index, revision c130faf6 (last modified 2026-10-29, board edition 0.4; fetched 2026-11-02). Columns:

• Qwen3.5-397B: the board’s stock reference Qwen/Qwen3.5-397B-A17B, via API, read from its generated letter rather than letter probabilities (per the board). Qwen3-VL-8B: the board’s small stock reference, Qwen/Qwen3-VL-8B-Instruct. Both are separate from the 62 entrants.
• 20 tuned 27B: the board entrants with 26 to 28.5 B parameters, no active-parameter tag, autoregressive and not listed as an inference technique. That grouping is mine, from the board file’s params, kind and technique fields. The kind field calls each a fine-tune, LoRA or adapter; only one (JEV-27B-VL) states its base. Their private scores and the Full vision score are not used.

Skill points. Stock and Swift: one pass each via /v1/systemone. The other columns are the board’s published public scores, from other serving paths:

Among the board’s 62 entrants, 29 score above stock on the public index and 33 below (the board ranks vision by Full score, so this is not its ranking). Bars in the chart start at 0; the lowest tuned MMMU-Pro vision entry is −3.03. The data folder’s README has the same table for the six named fine-tunes and the raw accuracies.

Takeaways

• Stock sits mid-board and below every tuned 27B-class entry. The index gaps run from 2.0 (68.13) to 11.8 (77.93) points; stock is 2.4 under Qwen3.5-397B and 3.8 over Qwen3-VL-8B. The board states a base for only one of the 20, so this is stock against a group, not a measured tuning gain. It also compares one quantized SGLang run with other people’s runs through their own engines and precisions.
• Two stock baselines give different gaps. The board lists JEV-27B-VL’s base, Qwen/Qwen3.8-27B, at 68.54 on the public half (BF16, through that entrant’s own code) against 72.80 for the entry, +4.26. The board’s own caveat says that base was read through a different, head-free path than the tuned entry, so +4.26 is not an isolated tuning effect. Against my stock NVFP4 run (66.09) the entry’s gap is +6.71. The extra 2.45 points can’t be assigned to engine, NVFP4, FP8 KV cache, route, prompt or resolution handling: I tested none of them.
• The lead depends on the benchmark. The median tuned entry beats stock by 1.4 on KIE, 4.6 on CV-Bench and 4.7 on BLINK, and by 14.7 on CharXiv, 14.3 on Hateful Memes and 11.9 on R-Bench-M. The table’s last column has the counts of tuned entries below stock. Stock’s CharXiv score of 64.91 is 7.3 under the lowest tuned entry and 2.8 over Qwen3.5-397B.
• Swift did not raise the index in these single passes. 65.80 against 66.09, stock − Swift +0.29 (−0.82 to +1.52). The interval is a paired bootstrap over rows (2,000 resamples within each benchmark, rows treated as independent, so it is optimistic and says nothing about run-to-run variation); it doesn’t show the two are equivalent.
• Per benchmark, Swift is ahead on CharXiv by 4.0 and Hateful Memes by 6.9, and stock on Winoground by 5.0 and RealWorldQA by 3.9. Only Hateful Memes and Winoground have intervals that exclude zero (uncertainty.md), and 11 benchmarks were compared without correction. The two configs gave the same letter on 10,220 of 12,210 rows (83.7%), so the net 0.29 hides a lot of row-level churn.
• MMMU-Pro vision, R-Bench-M and Hateful Memes have the widest ranges among the tuned entries: −3.03 to 48.55, 17.16 to 56.14 and 23.17 to 58.29. Stock’s Hateful Memes score of 30.09 (65.05% accuracy on a yes/no task) is above two tuned entries; Swift’s 37.02 is above three. The board’s 0.38 for Qwen3.5-397B on R-Bench-M is unexplained. MMMU-Pro and R-Bench-M carry weights of 0.25 and 0.5 in the index.
• Latency was well under the board’s limit in this run. With --workers 1 over all 12,210 rows the kit’s per-request wall time was a median of 126 ms, a mean of 212 ms and a p80 of 362 ms for stock (Swift 127, 214 and 365), with a maximum of 1,087 ms (Swift 1,150). The rule is a median, mean and p80 each at most 1,000 ms, measured by the maintainers on a private sample, so mine is a guide. Per-benchmark figures are in latency.txt.

Setup

Serving stack:

• Image: lmsysorg/sglang:nightly-dev-cu13-20261010-961bcf48.
• Checkpoints: RadixArk/Qwen3.8-27B-NVFP4 @ 319f741c (“stock”; 960,500 downloads in the 30 days before 2026-11-02) and ukisai/Swift-1.5-Qwen3.8-27b-NVFP4 @ 25482027 (“Swift 1.5 NVFP4”; 7,765 in the same window). Both resolved through the Hugging Face API on 2026-11-02.
• Flags as in the image decisions post: --mem-fraction-static 0.41, no speculative decoding, FP8 E4M3 KV cache.

python3 -m sglang.launch_server --model-path <checkpoint>
  --mem-fraction-static 0.41 --max-running-requests 4 --context-length 153600
  --dtype auto --attention-backend flashinfer
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
  --mamba-full-memory-ratio 0.9 --mamba-ssm-dtype bfloat16
  --mamba-radix-cache-strategy extra_buffer_lazy
  --chunked-prefill-size 2048 --page-size 64
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through
  --kv-cache-dtype fp8_e4m3 --schedule-policy lpm --enable-session-radix-cache
  --sleep-on-idle --trust-remote-code

Protocol:

• The kit, pinned at commit 25c6ec7b, ran in a container of the SGLang image: vision pipeline with the decide engine, protocol=decisions, path=/v1/systemone, --workers 1, images as data URLs capped at 1,638,400 pixels (the kit’s 1.6 MP cap; the checkpoints’ processors allow 16.7 MP, so no author cap sits below it). The largest image sent was 1,638,400 pixels. Each row is one request; the answer is the choice the endpoint returns (the highest-probability letter on all 12,210 rows). Thinking is off on this route.
• The suite was rebuilt on the CPU from the pinned sources with suite rebuild --vision; Winoground was included after the account accepted its gate. One pass per config, 12,210 of 12,210 rows with status ok. Wall time including about 3 minutes of server load: 46 min 25 s (stock) and 47 min 7 s (Swift).
• A 99-row smoke run (9 per benchmark) preceded the full runs.

Caveats

• Engine, precision and prompt differ from many board rows. This is NVFP4 weights with an FP8 KV cache through SGLang, reading letter probabilities, against BF16 and other engines on the board. Tuned entries also use their own image budgets (one of the 20 ran at 1.05 MP) and mostly don’t state their base. Gaps to board rows are not controlled comparisons.
• R-Bench-M and MMMU-Pro prompt. The suite marks R-Bench-M’s 665 rows and MMMU-Pro vision’s 1,730 rows with a readout: the reference vLLM engine continues the assistant turn with The answer is (. The kit’s decide engine drops the whole prompt object, so these 2,395 rows ran without it, and MMMU-Pro’s 1,730 rows also without options_in_image (which tells the reference engine not to repeat the image’s options as text); I didn’t inspect the resulting server-side prompt (I found this in the kit’s engines.py during review, not before the run). I haven’t checked which board entrants used that prefix, and I didn’t measure the effect on either skill or the index. R-Bench-M (20.59) and MMMU-Pro (33.24) are two of stock’s three lowest scores (Hateful Memes is 30.09); the missing prefix may or may not be why, since I didn’t test it.
• Endpoint. The kit’s protocol=decisions body is {model, state, questions, images}. This SGLang image’s /v1/decisions returned HTTP 400 for it (it wants input and a question list), so the engine points at /v1/systemone, which accepted the same body. The image decisions post ran its main passes there; its /v1/decisions route check used the other body shape, which that endpoint accepted. The first smoke attempt on /v1/decisions failed all 9 rows; nothing from it is in the results. A submission would need to give the maintainers this endpoint and its settings.
• Mind2Web font. The kit draws the box letters in Arial Bold, which this machine doesn’t have. I used Liberation Sans Bold (metric-compatible with Arial; sha256 3973aa50…1582), so vision verify fails the Mind2Web images and payloads and only those; run.sh passes --no-verify. Mind2Web scores are on slightly different pixels than the board’s. The suite was first built with Pillow 13.0.0, which the kit doesn’t list; InfographicVQA’s resized images failed their digest, so I rebuilt it with 12.3.0 and it verifies.
• Prefix cache. The server ran with its prefix cache on, with a 13.9% token hit rate in both runs (1.77 M of 12.7 M tokens over 12,608 prefill batches, from server.log). An earlier post found, on text prompts with these flags, that a warm cache (68–76% of tokens cached) changed 25 (stock) and 34 (Swift) of 250 top answers between a cold and a warm pass. That test didn’t cover images. I did not rerun with the cache off, so the effect on these scores is unmeasured. The board sets no cache requirement.
• One pass per config, no seeds. The bootstrap covers row sampling only and treats rows as independent, which they are not (CharXiv figures with several questions, Winoground pairs, BLINK items).
• Scores are agreement with the sources’ answer keys. I didn’t audit the keys. The “27B-class” grouping is mine.
• Public half only. No private half, no Full vision score, no ranking by the board’s own metric. Nothing here was submitted.
• Two NVFP4 checkpoints, one pass each, one suite edition. Rows hold one or two images. Nothing here covers BF16, other quants or other prompts.

Raw data

• Scripts, configs and results: adcostanza/benchmarks. run.sh there reruns both configs; the README has the suite rebuild commands.
• To rerun with the harness as committed with these results, check out 96c5e22.
• results/<config>/run/vision-scores.json has rows, answered count, accuracy, chance and skill per benchmark; rows-slim.jsonl has one line per row with run id, status, latency and (for rows with more than two options, 6,972 of 12,210; on two-option rows correctness would reveal the answer key) the chosen letter and whether it was right. The full results.jsonl with the kit’s request payloads stays out: it holds the benchmarks’ question text and options. The suite itself isn’t redistributed (source licences).
• uncertainty.md: the paired bootstrap and per-benchmark disagreement counts from analysis.py, which also needs the withheld results.jsonl and the rebuilt suite. board-table.md: every board column. board_range.py (in the folder) made this post’s table and chart from the board file. The board file itself isn’t copied: fetch data/vision.json from the pinned Space revision (sha256 1f870dff…9b90).

Test system

• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, power limit 400 W (default 600 W). Nothing else on the GPU, one config at a time.
• PCIe: Gen 4 x16 (the card supports Gen 5; nvidia-smi reports Gen 4 as this system’s maximum).
• CPU: AMD Ryzen 9 7950X, 16 cores, 32 threads.
• RAM: 250 GiB DDR5.
• OS: Ubuntu 24.04.2, kernel 6.8.0-147.
• NVIDIA: driver 580.178.04, CUDA 13.0.
• Containers: rootless Docker 28.0.4, NVIDIA Container Toolkit 1.18.1, GPU through CDI.
• Full list: hardware.md at the commit these runs used.