Structured Outputs vs. JSON Mode: The Real Difference

Structured outputs and JSON mode solve different problems, and conflating them is what breaks production pipelines. JSON mode guarantees the model returns syntactically valid JSON: balanced braces, quoted keys, no trailing comma. It says nothing about which fields show up, what type they are, or whether a required field exists at all. Structured outputs use constrained decoding to make the response provably match your JSON Schema, every time, because the model is mechanically blocked from generating a token that would violate it.

I learned the difference the slow way, building an extraction pipeline before OpenAI shipped strict mode. JSON mode kept the output parseable. It did not keep the output complete. A meaningful share of responses came back with a missing field, a renamed key, or a number returned as a string, not malformed JSON, just JSON that didn't match what the downstream code expected. Every one of those was a silent failure until something further down the pipeline broke on it.

Key takeaways

• JSON mode and structured outputs are not interchangeable. JSON mode guarantees valid syntax; structured outputs guarantee your schema, every time.
• The mechanism is constrained decoding, not a better prompt. The model's token sampler is masked at each step so it can only emit tokens that keep the output on a legal path through your schema.
• The reliability jump is measured, not marketing. OpenAI's own evals went from roughly 36% schema match with prompting alone, to about 80% with legacy JSON mode, to 100% with strict mode on gpt-4o-2024-08-06.
• Rigid schemas can tax reasoning. Research on format-restricted generation shows accuracy drops on math and multi-hop tasks when a model is forced into a final shape before it has reasoned freely.
• Function calling and structured outputs answer different questions. One takes an action. The other shapes an answer. Use the wrong one and you either under-constrain a decision or over-engineer a classification.

What structured outputs guarantee that JSON mode doesn't

Structured outputs guarantee that a language model's response matches a JSON Schema you define, exactly, on every call. JSON mode only guarantees the response is syntactically valid JSON. It has no concept of your schema: it doesn't know which keys are required, what type a field should be, or whether an array should hold one item or ten. That gap between "syntactically valid" and "matches my schema" is wider in practice than most teams expect, right up until a required field silently disappears in production.

The reliability numbers, and what that spread costs a pipeline

Before OpenAI shipped Structured Outputs, prompting a model to hit a target schema through instructions alone was reliable about 35.9% of the time in OpenAI's own evals (OpenAI, 2024). Legacy JSON mode improved that meaningfully, syntax stopped breaking, but schema match still landed around 80%: four calls out of five came back shaped correctly, and the fifth needed a retry, a repair pass, or a human to notice. With strict: true on gpt-4o-2024-08-06, schema match hit 100% in the same evals.

That gap between "mostly works" and "always works" is not a rounding error once you're running a pipeline instead of a demo. At 80% reliability, one call in five needs handling: a retry that costs latency and tokens, a regex-and-try/except repair layer someone has to maintain, or a data-quality problem that surfaces three weeks later in a report nobody double-checked. At scale, that retry tax is a real line on somebody's infrastructure bill, and it belongs in a business case, not a footnote in a demo.

If you're weighing whether to build that retry-and-repair layer yourself or fix the pipeline once with schema enforcement, that's a conversation worth having before anyone writes the glue code. That is the kind of build ViitorCloud takes on directly in technology consulting engagements, where the deliverable is the pipeline that never needed the retry layer in the first place.

How constrained decoding works under the hood

The mechanism behind the guarantee is constrained decoding, sometimes called grammar-constrained generation. Your JSON Schema gets compiled into a grammar, effectively a map of every legal next token at every position in the output. At each decoding step, the model's sampler masks out every token that would break that grammar before it samples anything. The model chooses from a legal subset of the vocabulary, not the whole vocabulary with a polite request attached.

That is mechanically different from prompting a model to "please output JSON," or even pasting your schema into the system prompt. A prompt is an instruction the model can, and under enough pressure will, ignore, even a well-built one that borrows solid prompt engineering techniques like prefilling the response's opening brace or spelling out an explicit output contract. A grammar mask isn't an instruction. It's a constraint applied at the token-sampling layer, after the model has decided what it wants to say and before that choice becomes an emitted token. If a required key named confidence comes next in the grammar, the sampler will not accept a token that starts a different key name, no matter how the model was leaning.

The trade is that the model can still be wrong about the content. Constrained decoding enforces shape, not truth. A schema-perfect response can still carry a hallucinated value in a well-formed field. Structured outputs fix "the JSON parses and has the right keys." They do not fix "the number in the amount field is correct." That's still your job, and it's why evaluation doesn't disappear once you add a schema. It moves downstream, from catching parse errors to judging content.

OpenAI's strict mode vs. Anthropic's structured outputs in 2026

Both major providers ship this now, with different plumbing. OpenAI's version lives on the response_format parameter: pass a JSON Schema with strict: true, either for a direct answer or on a function definition for a tool call, and gpt-4o-2024-08-06 and later models will match it exactly. Anthropic shipped native structured outputs later, in public beta for Claude Sonnet 4.5 and Opus 4.1 in late 2025, and it reached general availability on the Claude Developer Platform and AWS Bedrock in early 2026, since extended to Haiku 4.5 (Anthropic, Claude Docs). Anthropic's version runs through an output_format parameter with two modes: a JSON mode for a direct schema-matched answer, and a strict tool-use mode for schema-guaranteed function arguments.

Functionally, the guarantee is the same on both platforms: exact schema match, every call. The decision that matters more than which vendor's parameter name you memorize is the one below, whether you need the model to take an action, or just to shape an answer.

Function calling vs. structured outputs: when each is the right tool

Function calling and structured outputs share the same plumbing, constrained decoding against a schema, which is exactly why people conflate them. The distinction that matters is intent, not mechanism. Function calling exists to let the model take an action: call a tool that does something in the world, moves a ticket, sends an email, queries a database. Structured outputs exist to shape an answer: no execution, no side effect, just a guarantee that the response has the fields you need.

Picture a support-ticket workflow with two separate jobs. The first routes an incoming ticket to a queue, that's function calling: the model decides to invoke route_ticket(queue, priority), and something downstream moves the ticket. The second classifies the same ticket's sentiment for a dashboard, that's structured output: the model returns something like {"sentiment": "negative", "confidence": 0.82}, nothing executes, the response just needs to be shaped correctly for the chart reading it.

Mixing the two costs you in both directions. Route every classification through a tool call and you're paying orchestration overhead, a tool invocation, a round trip, for something that was always just an answer. Skip a proper tool definition for something that genuinely takes an action, and you've built a system that can produce a plausible-looking action it never executes. If you're unsure which one you're building, ask whether anything downstream does something different because the model responded. If yes, that's function calling. If the response is the product, that's structured output.

The failure mode nobody puts in the docs: schema constraints tax reasoning

Every vendor page for structured outputs reads like a pure upgrade: more reliability, no downside. The research says otherwise. A 2024 EMNLP Industry Track paper from Appier AI Research and National Taiwan University, "Let Me Speak Freely?", tested format-restricted generation against free-form generation across reasoning and classification tasks (Tam et al., 2024). Forcing a rigid output format measurably hurts performance on math and multi-hop reasoning tasks. The same restriction measurably helps classification-style tasks like slot-filling and intent detection.

The mechanism is intuitive once you see it. A model reasoning freely can think in prose before committing to an answer: try an approach, notice it's wrong, backtrack, try another. A schema that puts the final answer field first, or leaves no room to think before the answer, forecloses that path. The model has to commit to structure close to the first token, which means committing to an answer before it has had the chance to reason its way to one.

The honest trade-off, and the fix, are the same thing: don't force the final shape onto the first token. Put a reasoning or scratchpad field ahead of the final-answer field in your schema, so the model reasons in text before it has to commit to structure. Or split it into two calls: let the model think freely in the first, then constrain only the second, which turns a free-form answer into your schema. Either approach costs a few hundred extra tokens. Both are cheaper than a schema-perfect wrong answer nobody caught because the JSON parsed fine.

A practical decision checklist

Here is the version I use when a team asks which mode to reach for.

• Extracting or classifying structured data from unstructured text? Use structured outputs with a strict schema. This is the classification-style case where the research says format restriction helps, not hurts.
• Taking an action, calling an API, moving state somewhere? Use function calling, with strict mode on the tool definition once you're past prototyping. The guarantee you want is that the arguments are shaped correctly before anything executes.
• Doing math, multi-hop reasoning, or open-ended analysis that needs a structured final answer? Let the model reason first. Add a scratchpad field ahead of the answer field, or split it into two calls. Don't force the shape onto the first token.
• Building a multi-step agent where the right schema might change based on what happened three steps ago? That's usually not a structured-output problem at all. It's a context engineering problem, what the model can see at this step, not how its next answer gets shaped.

Structured outputs and JSON mode both live at the boundary between a model's answer and the code that has to trust it. Get that boundary right and evaluation moves to where it belongs: judging whether the content is correct, not writing a parser to guess at what the model meant.

FAQ

Is JSON mode the same as structured outputs?

No. JSON mode guarantees the response is syntactically valid JSON, balanced braces, quoted keys, nothing more. Structured outputs use constrained decoding to guarantee the response matches your exact JSON Schema, every field, every type, every required key, on every call.

Does structured output hurt LLM reasoning quality?

It can, on tasks that need free-form reasoning before a final answer. Research on format-restricted generation (the 2024 "Let Me Speak Freely?" study) found accuracy drops on math and multi-hop reasoning tasks under rigid format constraints, while the same constraints help classification-style tasks. The fix is to let the model reason in a scratchpad field before the answer field, not to avoid structured outputs entirely.

Should I use function calling or structured outputs for data extraction?

Structured outputs, in most cases. Data extraction is answer-shaping, not action-taking: you want the response formatted correctly, and nothing downstream needs to execute because the model responded. Reach for function calling only when the model's response should trigger something to happen.

How does Claude's structured outputs compare to OpenAI's strict mode?

Functionally, they guarantee the same thing: exact schema match on every call. OpenAI ships it through response_format with strict: true. Anthropic ships it through an output_format parameter with a JSON mode and a strict tool-use mode. Anthropic's version reached general availability on the Claude Developer Platform and AWS Bedrock in early 2026, after a public beta on Claude Sonnet 4.5 and Opus 4.1.

If you want the fuller discipline this sits inside, versioning the schemas themselves, handling the long tail of malformed inputs a schema alone won't catch, building the observability that flags a schema-perfect response that's still wrong, I wrote From Prompt to Pipeline on exactly that gap. And if the split between what you ask a model and what it can see is still fuzzy, I go deeper on that line in context engineering vs. prompt engineering.