Multimodal Model Selection Is a Different Eval Problem

Multimodal model selection is not text-model selection with a camera and a microphone bolted on. Vision reasoning, document OCR, and real-time voice are three separate capabilities that don't predict each other, and a model that tops a text leaderboard can still misread a chart, trust a caption over a photo, or add most of a second to a voice call. Evaluate each modality on its own terms, with your own images and audio, before you trust a single "best multimodal model" ranking.

I watched a team pick a frontier model for a document-processing pipeline off its text benchmark scores alone. It reasoned beautifully over the extracted text. It also misread three of the twelve line-item totals on a scanned invoice, because nobody had tested the OCR step at all. The text model was excellent. The vision model attached to it was never evaluated. That gap is where most multimodal projects lose money, and it's avoidable with a harness built for the job.

Key takeaways

If you read nothing else, read these.

• Multimodal model selection needs a per-modality eval, not one score. OCR accuracy, image reasoning, and voice latency are distinct capabilities that must each be tested on your own data.
• Harder vision benchmarks expose the gap public leaderboards hide. MMMU-Pro strips out the text-only shortcuts that inflate scores, and accuracy drops 16.8 to 26.9 percentage points versus standard MMMU once it does.
• Audio tokens cost far more than text tokens on the same model. On OpenAI's own pricing, gpt-realtime-2.1 charges $32 per million tokens for audio input versus $4 for text input, an 8x gap.
• Image token cost depends on tokenization method, not just the sticker price. The same phone photo costs $0.012 on GPT-5.5, $0.033 on Claude Opus 4.7, and $0.012 on Gemini 3.1 Pro.
• Captioning and image reasoning are different skills. A model can describe a photo accurately and still fail to extract, compare, or compute from what's in it. Test the actual task.

Why multimodal model selection isn't LLM selection with extra steps

My framework for choosing an LLM ranks candidates on task accuracy, cost per successful call, latency, context window, and deployment fit. All five factors still apply here. What changes is that "task accuracy" now has to be measured separately for every modality the product touches, because vision and voice capability don't travel with language capability.

A model can reason brilliantly over dense legal text and still fail to read a low-contrast scanned signature. A model can hold a coherent multi-turn conversation in text and still add unacceptable latency the moment you switch it into native audio mode. Treating "multimodal" as one line item on a scorecard hides exactly the failures that show up in production.

The practical fix is to split your eval into as many tracks as your product uses modalities, and to score each one against its own pass condition. A model only clears the bar for a given feature if it clears the bar on that feature's own evidence, not on its overall reputation.

OCR and document accuracy: read the text, don't just describe it

Document OCR is a narrower, harder-to-fake task than general image description. The model has to extract exact characters, digits, and layout structure from a scanned invoice, a handwritten form, or a low-resolution screenshot, with zero tolerance for a transposed digit or a dropped decimal point. A vision model that writes an eloquent summary of a page can still get the account number wrong.

Benchmarks like DocVQA, ChartQA, and OCRBench exist because this is a distinct skill from general visual question answering, and scores on one don't transfer cleanly to the others. A model strong on clean, born-digital PDFs can degrade sharply on handwriting, skewed phone photos, or dense multilingual tables, because those are different failure surfaces entirely.

Test on your own document types before you commit. If your pipeline processes invoices, test invoices, not a public receipts dataset that happens to look similar. If you're feeding extracted document text into a retrieval pipeline, the accuracy of that extraction step sets a ceiling nothing downstream can fix. I cover the retrieval side of that problem in my guide to RAG chatbot architecture, but the ceiling gets set here, at the OCR step, before retrieval ever runs.

Image reasoning versus captioning are two different skills

Captioning answers "what is in this image." Image reasoning answers a harder question: using what's in the image to compare, count, calculate, or cross-reference against other information. A model can nail the first and still fail the second, and most general vision benchmarks reward captioning far more than they test reasoning.

MMMU-Pro exists to close that gap. It removes the questions a model can answer from the text alone, without looking at the image, and it expands multiple-choice options from four to ten to cut down on lucky guessing. When researchers ran that stricter version, model accuracy dropped 16.8 to 26.9 percentage points across the board compared to the easier, original MMMU benchmark.

That drop is the whole point. It's the gap between a model that can describe a chart and one that can actually read the axis labels and tell you which quarter's bar is taller. On Artificial Analysis's live MMMU-Pro leaderboard, frontier models in 2026 cluster in the low-to-mid 80s, well under the 90%-plus scores the same model families post on saturated text benchmarks. Check the live leaderboard before you shortlist, because a snapshot in an article goes stale within weeks and the underlying models don't stop shipping updates.

The same forces that keep bigger, more general text models winning over narrow, cleverly-tuned ones are showing up in vision too. Raw scale is closing the image-reasoning gap faster than specialized captioning tricks, which is the core argument in my book The Bitter Lesson, Revisited. That's a reason to weight general frontier models higher for reasoning-heavy visual tasks, and weight narrow, purpose-built OCR models higher for extraction tasks where scale buys you less.

If your use case is closer to reasoning than description (comparing two product photos, verifying a chart against a claim, auditing a form against a policy) test that exact reasoning task. A high captioning score tells you almost nothing about whether the model will get the comparison right.

Real-time voice: latency is the constraint no accuracy benchmark shows you

Voice products live or die on time-to-first-audio, the gap between when a user stops talking and when they hear a reply start. Independent 2026 benchmarking of real-time voice APIs found real spread here: OpenAI's GPT-Realtime-1.5 measured 0.81 seconds, Google's Gemini 3.1 Flash Live in minimal-reasoning mode measured 0.96 seconds, and specialty low-latency stacks measured as low as 0.44 seconds.

None of that shows up on a text accuracy leaderboard. A model can reason perfectly and still feel broken in a live conversation if the pause before it answers crosses a few hundred milliseconds, because human conversational turn-taking has almost no tolerance for dead air. Measure this at your own traffic's p95, under your own network conditions, the same discipline I recommend for text latency in AI agent architecture generally, just with a tighter budget.

Name the trade-off honestly: the models with the strongest reasoning under voice conditions are frequently not the fastest ones, and pushing a model toward higher reasoning effort tends to add latency in exchange for accuracy. Decide which side of that trade your product needs before you pick a default, because a support bot and a translation earpiece have opposite tolerances for the same half-second.

The token economics change once vision or voice joins the call

Image and audio tokens are not priced like text tokens, and the gap is large enough to change a build decision. Image tokenization differs by provider: GPT-5.5 uses 32x32-pixel patch tiles at $5 per million input tokens, Claude Opus 4.7 uses an area-based formula (width times height divided by 750) at the same $5 rate, and Gemini 3.1 Pro uses a flat 258-token minimum per 768x768 tile at $2 per million, the lowest posted rate among the three.

Audio is a bigger jump. On OpenAI's own Realtime API pricing, gpt-realtime-2.1 charges $32 per million tokens for audio input versus $4 per million for text input on that same model, an 8x gap, and $64 per million for audio output versus $24 per million for text output, a 2.7x gap. A voice product's real per-conversation cost is dominated by the audio token rate, not the text pricing you'll find advertised first.

This is where the engineering decision becomes a P&L decision. A support product that processes a thousand hours of voice a month is pricing audio tokens, not text tokens, and that number belongs in the build-or-buy math before launch, not in the first invoice after it.

Build a cross-modal eval set, not a leaderboard screenshot

The same discipline that works for text selection works here, applied per modality. Pull 30 to 50 real examples for each modality your product touches: scanned documents from your own pipeline, photos taken in the conditions your users will actually use (not studio lighting), and real customer call recordings or realistic transcripts if you don't have call data yet.

Write a pass condition for each example that a person can grade without ambiguity, the same way I describe in my guide to LLM evaluation. Run every candidate model against the frozen set and score each modality separately. Don't average a strong text score with a weak vision score into one blended number; that number hides exactly the weakness you built the eval to find.

Name the cost honestly: a genuinely representative multimodal eval set takes longer to build than a text-only one, because you need real images and real audio, not just written prompts. Budget a few days, not an afternoon. It's still cheaper than shipping the wrong model and finding out in production.

The failure mode: models that ace text benchmarks and miss the picture

The failure mode that costs real money isn't a blurry image. It's a confident, wrong answer from a model that trusted the wrong signal. A 2026 arXiv study on text-overlay-induced hallucination tested vision-language models on images and video where on-screen text contradicted what was actually shown, and found the models systematically believed the text over the pixels.

Qwen3-VL-8B resisted the misleading overlay text only 27.8% of the time. The strongest model tested, Gemini-3.1-Pro, still resisted only 72.8% of the time, meaning it took the false caption at face value more than one time in four. These are frontier-class systems, not toy models, and they still get fooled by a mislabeled chart axis or a doctored timestamp baked into a screenshot.

If your use case touches anything adversarial-adjacent (content moderation, invoice fraud detection, identity document verification) this is exactly where a purely text-strong model quietly fails. No model tested is immune to it yet. If visual ground truth is safety- or revenue-critical, keep a verification step in the loop instead of trusting one model's read of one image.

A short checklist before you commit to a multimodal model

• Write the job per modality. "Extract the total from a scanned invoice" and "answer a spoken question in under a second" are two different pass conditions, not one.
• Test OCR and image reasoning separately. A high captioning score doesn't predict accuracy on extraction, comparison, or math over an image.
• Measure real-time voice latency at your own p95. A vendor's demo number is not your network, your traffic, or your reasoning-effort setting.
• Price image and audio tokens on your real volume. Tokenization method changes cost more than the advertised per-token rate suggests.
• Keep a verification step for adversarial or high-stakes visual tasks. Current models still trust misleading on-screen text a meaningful share of the time.

Frequently asked questions

What's the difference between a vision-capable model and a truly multimodal model?

A vision-capable model can accept an image as input and describe or answer questions about it. A genuinely useful multimodal model also reasons over that image (comparing, counting, extracting structured data) which is a harder, separately-tested skill from description alone.

How do I evaluate a model for real-time voice specifically?

Measure time-to-first-audio at your own traffic's p95, under real network conditions, and at the reasoning-effort setting you plan to ship. A model's text latency or its offline transcription accuracy tells you very little about how it feels in a live, full-duplex conversation.

Are image and audio tokens really that much more expensive than text tokens?

Yes, on current 2026 provider pricing. OpenAI's own Realtime API pricing puts audio input at roughly 8x the cost of text input on the same model, and image tokenization methods vary enough across providers that the same photo can cost more than double depending on which model processes it.

Can I trust a public benchmark like MMMU to pick a multimodal model?

Use it to narrow a long list to a short one, not to make the final call. MMMU-Pro was built specifically because the original MMMU let models guess correctly without truly using the image, and accuracy drops sharply once that shortcut is removed. Your own eval set, built from your real images and audio, is what should make the final decision.

Multimodal model selection rewards the same discipline as text-only selection: a small eval set on your own data, scored per modality, before you sign a contract. If you're making this call under a real deadline and want the eval harness built alongside the model shortlist, that's the kind of work ViitorCloud's technology consulting practice does for teams shipping vision and voice products right now.