Reported, not reproduced
This site did not rerun these suites. The table preserves Qwen's published values and compares the same rows in the same official model card.
Qwen3.8 27B benchmark
Qwen3.8 27B benchmark scores need their test conditions.
These are selected results reported by Qwen for the original checkpoint. Use them to form a test plan—not as a guarantee for your quant, runtime, prompt, or hardware.
Selected official results
Scores are reproduced from the model card available on August 22, 2026.
| Benchmark | Qwen3.8 27B | Qwen3.6 27B | Difference | What it probes |
|---|---|---|---|---|
| TerminalBench 2.1 | 73.0 | 63.4 | +9.6 | Agentic terminal coding |
| SWE-bench Pro | 61.7 | 53.5 | +8.2 | Software engineering |
| LiveCodeBench v6 | 90.3 | 83.9 | +6.4 | Competitive coding |
| OSWorld | 84.3 | 63.9 | +20.4 | Visual computer use |
| WebArena | 64.8 | 48.8 | +16.0 | Web agent tasks |
Source: Official Qwen3.8 27B model card and evaluation settings
Harnesses matter
The card reports SWE-bench Pro with a Claude Code harness, temperature 1.0, top-p 0.95, and a 256K context window. A different agent loop can move the result.
Some sets are in-house
QwenSWEBench, CoWorkBench, and RecreationBench are described as in-house evaluations. Treat their methodology and corrected labels as part of the claim.
Read scores safely
Four questions before you compare.
- 01
Was thinking enabled?
Reasoning effort and token budget can materially change both score and cost.
- 02
Was the original checkpoint used?
FP8, GGUF, AWQ, and other conversions may behave differently from the published evaluation.
- 03
Were tools and scaffolds identical?
Agent benchmarks depend on tool permissions, environment images, timeouts, and retry rules.
- 04
Does the benchmark match your task?
A coding score does not validate your support workflow, extraction schema, or domain language.
From score to decision
Use the Qwen3.8 27B benchmark as a shortlist, not a verdict.
The published improvements are meaningful enough to justify a trial, especially for coding and computer-use workloads. They do not tell you whether a quantized local build will follow your schema, call your tools correctly, or finish within your latency budget.
Match the scaffold
Agent scores include more than model weights. Record the system prompt, tool definitions, retry policy, timeout, sandbox permissions, and context management. A stronger scaffold can rescue a weaker model—and hide a brittle one.
Match the precision
The official results describe the source checkpoint. GGUF, FP8, AWQ, and provider-hosted versions can use different kernels, templates, sampling defaults, or reasoning controls. Label every result with the exact artifact.
Measure end-to-end cost
A high task score can still be expensive if the model spends many tokens thinking, retries tool calls, or needs a large context. Track successful tasks per dollar and per minute, not tokens per second alone.
Inspect the misses
Average scores conceal failure clusters. Separate wrong answers, invalid tool arguments, environment errors, refusals, timeouts, and judge disagreements so one fix does not create a different regression.
Your own evaluation
A small, honest test beats a large borrowed table.
Build 30–100 representative cases, freeze the prompt and runtime, record latency and failure categories, and review the same outputs across every model candidate. Keep the test set private if it mirrors production data.
Quality
Task success, factuality, instruction following, tool-call validity.
Efficiency
Time to first token, completion latency, input and output tokens.
Reliability
Timeouts, malformed output, repetition, unsafe or ungrounded answers.
Keep a minimum record for every benchmark run.
Save the model identifier and artifact hash, runtime version, hardware, quantization, context limit, system prompt, sampling values, reasoning mode, tool schema, timeout, and harness commit. Record how many attempts each task received and whether a judge was automatic or human. Without that record, a score cannot be reproduced and a later regression cannot be explained.
Also separate first-pass success from pass-after-retry. An agent that reaches the same final score only after repeated tool calls may cost more, take longer, and create more side effects in production than the headline number suggests.