MMLU and SWE-bench Explained: What They Actually Test

MMLU tests whether a model can answer multiple-choice questions across 57 academic and professional subjects. SWE-bench tests whether a model can write a code patch that closes a real, previously filed GitHub issue. Both are useful for narrowing a shortlist. Neither is a verdict on which model you should ship.

I watch the leaderboard shuffle every time a lab drops a new model, because these two numbers are the ones every announcement leads with. MMLU is the standard citation for how much a model knows. SWE-bench, and its more curated sibling SWE-bench Verified, is the standard citation for whether a model can actually write code that works. Both are older than they feel, both have documented flaws serious enough that their own creators have had to walk parts of them back, and both still get treated by buyers as a final answer instead of a first filter.

Key takeaways

If you read nothing else, read these.

• MMLU and SWE-bench measure different things. MMLU is a knowledge-recall exam across 57 subjects; SWE-bench checks whether a generated patch resolves a real GitHub issue against the repo's own tests. Neither substitutes for the other.
• MMLU has a measured error rate. A 2024 re-annotation, MMLU-Redux, found 6.49% of questions wrong on average, and as high as 57% wrong in the Virology subset.
• SWE-bench Verified is a 500-task, human-screened subset of the original 2,294-task SWE-bench, not the same test, and a model's score on one does not predict its score on the other.
• OpenAI stopped reporting SWE-bench Verified in early 2026, after its own audit found most remaining "failures" were broken test cases, plus contamination in every frontier model it tested.
• A leaderboard rank is a filter for a shortlist, not a selection criterion. The only number that predicts your production result is a small eval built on your own task.

What MMLU actually tests

MMLU stands for Massive Multitask Language Understanding. Hendrycks and colleagues built it in 2020 as a four-choice exam spanning 57 subjects: high school and college STEM, law, medicine, history, ethics, accounting, and more. A model reads a question and four options and picks one. Nothing about the format involves writing, planning, or using a tool. It is a closed-book knowledge and reasoning test, structured the way a standardized exam is structured.

That format is exactly why MMLU became the default citation. It is cheap to run, produces a single comparable number, and covers enough breadth that a low score is a real red flag. If a model scores poorly on MMLU, something is wrong with its knowledge base or its instruction-following, and that is worth knowing before you spend a day building a task-specific eval.

What MMLU does not test matters just as much. It does not test multi-step agentic work, tool use, code that runs, or knowledge past the exam's own cutoff. A model can ace MMLU and still fail your actual task, because reciting the right answer to a law exam question and drafting a contract clause your legal team will accept are only loosely related skills.

MMLU's known flaws: label errors and a shrinking ceiling

The exam itself has mistakes. A 2024 paper, "Are We Done with MMLU?", built MMLU-Redux by manually re-annotating 5,700 questions, 100 sampled from each of the 57 subjects, with domain experts checking every answer key. The overall error rate came out to 6.49%. Some subjects were nearly clean. Virology was not: 57% of the analyzed questions in that subset contained an error, whether a wrong answer key, an ambiguous question, or more than one defensible correct choice.

Correcting those errors moved the scoreboard. Re-evaluating models on MMLU-Redux shifted rankings by up to 10 to 15 percentage points in the worst-affected subsets. That is not a rounding error. That is the difference between "this model leads the category" and "this model is mid-pack," depending on which version of the answer key you trust.

Now put that error rate next to how tightly frontier models cluster at the top of MMLU today. When two models are separated by two or three points and the exam itself carries a measured 6.49% error rate, you are reading noise, not signal. MMLU did its job for years as a knowledge floor check. At the top of the leaderboard, it has stopped being a ranking tool.

What SWE-bench actually tests

SWE-bench is a different kind of test entirely. Built from 2,294 real software engineering problems pulled from actual GitHub issues and their corresponding pull requests across 12 popular Python repositories, it hands a model an issue description and a snapshot of the codebase, then checks the resulting patch against the repository's own test suite. There is no multiple choice. There is a working patch, or there is not.

The task was hard by design. At launch, the best-performing model tested, Claude 2, resolved a mere 1.96% of the issues. That number is worth sitting with. This was not a toy benchmark humbling a weak model; it was a real production bug report humbling the frontier. That is exactly why SWE-bench stuck as the reference point every lab now cites: it correlates with something an engineer recognizes on sight, a real issue and a real fix.

What it still does not cover is your repository. The original task set is entirely Python, drawn from a narrow slice of popular open-source projects. Your polyglot monorepo, your internal conventions, and your team's review bar are not in that distribution. I go deeper on this gap, and on SWE-bench's siblings LiveCodeBench and Terminal-Bench, in what code generation benchmarks actually measure.

SWE-bench Verified vs. the original, and why OpenAI retired it

SWE-bench Verified is not the same test as SWE-bench. It is a 500-task subset that OpenAI built with the SWE-bench authors, hand-screened by professional software developers to remove tasks with under-specified issue descriptions or unit tests so strict they rejected a correct fix. Since its 2024 release, Verified became the number every model announcement leads with, because it is smaller, faster to run, and less noisy than the original 2,294-task set.

It did not stay clean. In February 2026, OpenAI published its own audit of the benchmark it had built, and the results were bad enough that OpenAI stopped reporting the score entirely. Auditors reviewed the subset of problems frontier models were still failing and found that at least 59.4% of those had flawed test cases that rejected functionally correct submissions. Models were not failing the coding task. The benchmark's own grading was failing them.

The second finding was worse for anyone using the score to compare models. Every frontier model OpenAI tested, across labs, could reproduce the exact, verbatim, human-written ground-truth patch for problems it was supposedly solving from scratch. That is not something you get by reasoning your way to the same fix independently. It is the signature of contamination: the problem and its solution were already in the training data.

None of this makes the scores from before the audit useless as a historical marker. Claude Opus 4.5 posted 80.9% on SWE-bench Verified in late 2025, the first model past the 80% line, ahead of GPT-5.1 at 76.3% and Gemini 3 Pro at 76.2%. That was a real, comparable jump at the time. It also landed right before the benchmark's own maintainers concluded the remaining gap was measuring broken tests and memorization as much as capability. OpenAI now points teams to SWE-bench Pro, a newer, larger set built with private and held-out splits specifically to resist the contamination that sank Verified.

How to read a benchmark leaderboard like an engineer, not a shopper

Use MMLU and SWE-bench rank the way you would use a resume screen: to cut a long list to a short one, not to make the hire. A model that scores poorly on both is worth dropping from consideration immediately. A model that leads on both earns a closer look, not an automatic contract.

Before you trust any number, check three things. Which exact variant is being cited, since the original SWE-bench, Verified, and Pro measure different tasks under different grading. When the score was published, since a number from before a contamination audit ages fast. And whether the benchmark's own maintainers still stand behind it, which is now a real, live question for SWE-bench Verified.

Then build a small eval from your own task before the final call. This is the step a leaderboard glance skips, and the step that actually predicts production behavior. I lay out the five-factor framework I use for this, and where public benchmarks fit into it as a first filter, in how to choose an LLM. For the wider landscape of benchmark families, and what each type is actually built to measure, see the companion piece on what benchmarks measure.

Name the trade-off honestly: building that eval costs you a day or two you do not spend reading a leaderboard. It is worth it anyway. A model picked off stale Verified rank alone, then swapped six weeks later once a contamination story breaks, costs far more in re-integration and lost trust than the eval would have cost up front. That is an engineering decision with a line item on somebody's budget, not just a technical preference.

There is a pattern underneath all of this that I write about at length in my book The Bitter Lesson, Revisited: curated, handcrafted tests keep losing to scale faster than their curators can patch them. MMLU took years to saturate. SWE-bench Verified took about 18 months. The benchmarks built to replace them, MMLU-Redux and SWE-bench Pro, will likely follow the same arc. Betting your model selection on any single leaderboard is betting against a trend that has not reversed yet.

What's the difference between the original SWE-bench and SWE-bench Verified?

The original is 2,294 tasks pulled directly from GitHub issues across 12 Python repositories, with no human screening of the grading. Verified is a 500-task subset that OpenAI and the SWE-bench authors hand-checked to remove tasks with broken or overly strict unit tests. They are related but not interchangeable, and a model's score on one does not predict its score on the other.

Is SWE-bench Verified still a reliable way to compare coding models?

Not on its own, as of OpenAI's 2026 audit. The benchmark's own creator found flawed test cases behind most remaining failures and contamination across every frontier model tested, and now recommends SWE-bench Pro instead. Treat any Verified score from before that audit as a historical marker, not a current ranking.

Why does MMLU still get cited if it has a known error rate?

Because a 6.49% error rate is still low enough to make MMLU useful as a coarse knowledge floor check, and it remains the cheapest, broadest single number available. The problem is not that MMLU is useless. The problem is using it to rank models that are separated by less than its own margin of error.

How should I actually use MMLU and SWE-bench when picking a model?

Use them to eliminate weak candidates from a long list, not to choose the winner. Confirm which exact benchmark variant and date you are looking at, then build a small eval from your own task and data before you commit. The public score gets you to a shortlist of three to five models. Your own eval gets you to a decision.

If you are staring down a model shortlist and want a second set of eyes on the build-vs-buy call before you commit budget, that is a conversation worth having early. ViitorCloud's technology consulting practice helps teams turn a benchmark shortlist into an eval-backed decision, with the harness built around your codebase and your data, not someone else's leaderboard.