Structured Outputs: JSON You Can Actually Parse

Every system that consumes model output has the same moment of truth: the answer arrives as text, and the system needs it as structure. A JSON object, a list of fields, a value that fits an enum. When the text parses, the pipeline hums. When it does not — a trailing comma, a prose preamble, a field renamed by the model's mood — the pipeline stops, and the failure is yours to handle.

This is not a rare edge. It is the single most common integration failure in production LLM systems, and it is expensive in a way the pricing page never shows: every malformed answer is either a retry (paid twice) or a dropped request (a user lost).

The fix is a discipline, not a hope: constrain the model, validate the result, and treat the parse gate as part of the product, not part of the prompt.

Because they are trained to be helpful in prose, and helpfulness includes helpful extras. Ask for JSON and the model may wrap it in a sentence, add a comment field it thinks you would appreciate, or rename a key to something it considers clearer. None of this is misbehavior from the model's perspective; it is the model doing what it was trained to do, which is to produce text that helps.

Temperature makes it probabilistic: the same prompt produces the schema today and a near-miss tomorrow, and a near-miss is still a miss to a parser. Long outputs make it likelier: the more fields, the more chances for one to drift.

And the prompt is not the contract. A prompt that says "respond in JSON" is a request, not a constraint — the model can and will decline the request in small ways. The constraint has to come from the API, not the prose.

The fix has two strengths, and they are worth distinguishing. JSON mode constrains the output to be valid JSON — no preamble, no trailing prose, no markdown fences. That alone eliminates the most common failure class. Schema enforcement goes further: the output must conform to a schema you supply, with the right fields, the right types, and nothing else.

Where the platform supports schema enforcement, use it for anything a parser consumes. It converts "the model might" into "the API guarantees", and a guarantee is what a pipeline needs.

Where only JSON mode exists, pair it with a validation gate on your side: parse, validate against the schema, and on failure retry with the error message appended. The resilience post covers the retry mechanics; the point here is that the gate is the contract, and the retry is the enforcement.

Even with schema enforcement, validate. The schema guarantees shape, not sense: a field can be the right type and the wrong value, an enum can be respected and still be the wrong choice. The parse gate from the resilience post — "the response is not done until it parses and validates" — is the minimum, and business validation is the next layer.

Validation also decides the retry policy. A response that fails the schema is worth retrying with the error attached; a response that passes the schema but fails a business rule is a different failure, and retrying it blindly is pure spend. Classify before you retry.

And log the failure rate. Schema violations per hundred calls is a metric that should sit on the dashboard next to latency — it is the earliest symptom of prompt drift, a model change, or a temperature setting that stopped being cute.

More than the retry. The failed attempt billed in full — the prompt tokens, the generated tokens, the parse error. The retry billed again, usually longer because the error message rode along. The latency of two round trips where one was promised. And, in the worst case, the user who left between attempt one and attempt two.

The cost-per-task framework is built for exactly this: the all-in cost of an accepted answer, retries included, is the number that matters, and a schema that fails often is a hidden multiplier on that number. Teams that measure it usually find the parse failure rate is the cheapest thing they can fix in the whole pipeline.

Which is the optimistic reading: this is a failure class that responds to engineering. A schema, a gate, and a retry policy convert a probabilistic annoyance into a deterministic pipeline.

When a human reads the output and nothing parses it. Drafting, summarization, chat — the places where prose is the product. Forcing JSON onto a drafting task buys nothing and costs the model's best quality, because the constraint narrows the answer.

The mistake is the middle ground: output that is "mostly JSON" consumed by a parser that is "mostly strict". Either the consumer is a parser — then constrain and validate — or the consumer is a person — then let the model write. The middle ground is where the retries live.