LLM Structured Output: JSON Mode Across Every Major API

When you wire an LLM into a production system, the first thing that breaks usually isn't the model's reasoning — it's the output format. The model returns a blob of text with Markdown fences, an apology, and maybe some commentary, and your json.loads() throws an exception that takes down the whole pipeline.

LLM structured output (JSON mode) fixes this at the API layer. Instead of hoping the model produces valid JSON, you pass a schema and the API enforces it — in most cases down to the token level. Downstream code gets a clean dict, no parsing gymnastics required.

This guide covers three things: how to use the schema constraint mechanism for each major API, how to write schemas that actually work, and what to do when things still go wrong.

How Each API Enforces JSON Structure

OpenAI (GPT-5.5 / GPT-5.4 / GPT-5.4-mini)

OpenAI's Structured Outputs use the response_format field with a json_schema object and strict: true. In strict mode, output is constrained by a finite state machine at the token level — the model will produce exactly the fields you declared, no more, no less.

response = client.chat.completions.create(
    model="gpt-5.4-mini",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "product_review",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
                    "score": {"type": "number", "minimum": 0, "maximum": 10},
                    "summary": {"type": "string"}
                },
                "required": ["sentiment", "score", "summary"],
                "additionalProperties": False
            }
        }
    },
    messages=[{"role": "user", "content": "Review this product: ..."}]
)

One gotcha: additionalProperties: false is required in strict mode. Leave it out and you'll get a 400 error.

Anthropic (Claude Opus 4.8 / Sonnet 4.6 / Haiku 4.5)

Anthropic doesn't have a standalone JSON mode. Instead, structured output comes through Tool Use — you define a tool with only an input_schema, force the model to call it, and the tool's input arguments are your JSON payload.

response = client.messages.create(
    model="claude-sonnet-4-6",
    tools=[{
        "name": "extract_review",
        "description": "Extract structured review data",
        "input_schema": {
            "type": "object",
            "properties": {
                "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
                "score": {"type": "number"},
                "summary": {"type": "string"}
            },
            "required": ["sentiment", "score", "summary"]
        }
    }],
    tool_choice={"type": "tool", "name": "extract_review"},
    messages=[{"role": "user", "content": "Review this product: ..."}]
)
result = response.content[0].input  # already a dict

Setting tool_choice to a specific tool name prevents the model from ignoring the tool and just answering in plain text.

Google (Gemini 3 series)

Gemini uses generation_config with response_mime_type: "application/json" and response_schema. It's also token-level constrained under the hood.

response = client.generate_content(
    contents="Review this product: ...",
    generation_config={
        "response_mime_type": "application/json",
        "response_schema": {
            "type": "OBJECT",
            "properties": {
                "sentiment": {"type": "STRING"},
                "score": {"type": "NUMBER"},
                "summary": {"type": "STRING"}
            },
            "required": ["sentiment", "score", "summary"]
        }
    }
)

The most common trap here: Gemini's schema type names are uppercase (STRING, NUMBER, OBJECT, ARRAY), unlike standard JSON Schema's lowercase. Mix them up and the schema silently fails — no error, just degraded output.

Quick Comparison

Feature	OpenAI GPT-5.x	Claude 4.x	Gemini 3
Constraint mechanism	json_schema + strict	Tool Use input_schema	response_schema
Token-level enforcement	✅ strict mode	✅	✅
Native enum support	✅	✅	✅
Nested objects	✅	✅	✅
Result location	message.content	content[0].input	text (needs parse)

Writing Schemas That Actually Work

Even when the API doesn't error, a poorly written schema produces poor output. A few things that matter in practice:

Write description for every field. Descriptions aren't just documentation — the model reads them during generation to understand what you want. A field named score with no description leaves the model guessing whether you mean 1–10 or 0–100. Adding "description": "Rating from 0 to 10, higher is better" meaningfully improves accuracy.

Use enum instead of string-plus-instructions. If a field has a fixed set of valid values, list them in enum. Writing "type": "string", "description": "must be positive, neutral, or negative" has no enforcement power outside of strict mode.

Always specify items on arrays. {"type": "array"} without items causes some models to collapse the array into a string. Use {"type": "array", "items": {"type": "string"}}.

Keep nesting shallow. Beyond four levels deep, models start losing track of where they are in the structure. Testing shows Haiku 4.5 and GPT-5.4-mini compliance drops around 15% at five or more levels of nesting. If your data genuinely requires complex structure, split the extraction into two separate calls.

Fallback Strategies for When Things Go Wrong

Schema constraints are not a guarantee. Production systems will hit three failure modes: safety filters returning empty output, finish_reason: length producing truncated JSON, and degraded output from APIs that don't support strict mode.

First line: check finish_reason

choice = response.choices[0]
if choice.finish_reason != "stop":
    raise ValueError(f"Unexpected stop reason: {choice.finish_reason}")

A length finish means the output was cut off by your token limit. Truncated JSON is not recoverable — retry with a higher max_tokens value, don't try to patch the broken string.

Second line: JSON repair

For APIs without strict enforcement — domestic Chinese models like DeepSeek V3, Qwen 3, or GLM, but also any older endpoint — common issues include trailing explanation text, numbers wrapped in quotes, and Python-style single quotes. The json-repair library handles most of these:

from json_repair import repair_json
import json

raw = response.choices[0].message.content
try:
    result = json.loads(raw)
except json.JSONDecodeError:
    result = json.loads(repair_json(raw))

This fixes roughly 80% of minor formatting issues. Don't rely on it for semantic problems like misspelled field names or out-of-range enum values — those need to be caught differently.

Third line: schema validation with self-correction

Valid JSON doesn't mean schema-compliant JSON. Use jsonschema to validate after parsing, and if it fails, feed the error back to the model:

import jsonschema

def validate_or_retry(client, messages, schema, max_retries=2):
    for attempt in range(max_retries + 1):
        response = call_llm(client, messages)
        try:
            data = json.loads(response)
            jsonschema.validate(data, schema)
            return data
        except (json.JSONDecodeError, jsonschema.ValidationError) as e:
            if attempt == max_retries:
                raise
            messages.append({"role": "assistant", "content": response})
            messages.append({"role": "user", "content": f"Output didn't match the required schema: {e}. Please regenerate."})

Cap retries at two. If you're still failing after that, the problem is in your prompt or schema design, not the model's execution — fix the source, not the loop.

Models Without Strict Mode

DeepSeek V3, GLM-5.2, and Qwen 3 all support response_format: {"type": "json_object"}, but none of them support OpenAI-style json_schema strict constraints. The model is told to output JSON, but field structure has no token-level enforcement — missing fields, extra fields, and type mismatches all happen regularly.

The most reliable approach with these models: put a complete JSON example object in the system prompt (not a description of the schema, an actual example). Pair that with json-repair plus jsonschema validation and you can hold compliance above 90%. That's below OpenAI strict mode's 99%+, but it's workable for most non-realtime applications.

---

If you're routing across multiple providers for cost optimization or A/B testing — which is common once you start taking these structured output patterns seriously — maintaining separate auth credentials and schema adaptation logic for each API gets tedious fast. I use XycAi to handle this: one OpenAI-compatible endpoint that routes to GPT-5.5, Claude Opus 4.8, Gemini 3, DeepSeek V3, and 200+ other models, with the schema format differences abstracted away on their side. GPT and Claude official models start at 14% of list price, which makes multi-model experimentation a lot more practical without the per-provider integration overhead.