Structured Output With LLMs: JSON Schemas, Validation, Retries, and Error Handling

Overview

Language models generate text, while most production systems need data with a predictable shape. A workflow may require a list of extracted entities, a classification label, a set of database fields, or an action plan that another service can consume. Asking for “valid JSON” in a prompt is useful, but it is not the same as enforcing a contract.

A dependable structured-output design has several layers:

• Instruction: Tell the model what the fields mean and how they should be used.
• Schema: Define required properties, allowed types, enumerations, limits, and nesting.
• Generation control: Use an API’s structured-output or constrained-generation capability when available.
• Validation: Parse the response and validate it against the schema before business logic runs.
• Recovery: Repair, retry, or route the request to a fallback path when validation fails.
• Observation: Record failure categories, latency, token use, model version, and prompt version without storing unnecessary sensitive data.

This separation matters because syntactically valid JSON can still be wrong. A response may contain a string where a number is expected, omit a required field, use an unsupported category, or include an empty value that downstream code cannot interpret. JSON validation in an AI workflow should therefore check both the shape and the meaning of the result.

For broader workflow design, pair this pattern with a monitored pipeline approach such as the AI workflow automation blueprint. If the output depends on retrieved documents, the same contract should be evaluated alongside retrieval quality in a reliable RAG application.

Template structure

Start with a small, explicit contract. The following example describes an email-triage result. The exact schema syntax depends on your validation library, but the design principles are portable across languages and LLM APIs.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["category", "priority", "summary", "needs_reply"],
  "properties": {
    "category": {
      "type": "string",
      "enum": ["billing", "technical", "sales", "other"]
    },
    "priority": {
      "type": "string",
      "enum": ["low", "normal", "high"]
    },
    "summary": {
      "type": "string",
      "minLength": 1,
      "maxLength": 500
    },
    "needs_reply": {
      "type": "boolean"
    }
  }
}

A matching system prompt should explain the contract without duplicating every implementation detail:

You classify inbound support messages for a ticketing system.
Return only the fields defined by the supplied response schema.
Use "other" when no category is a safe match.
Do not infer account details that are absent from the message.
Keep the summary factual and concise.

Use the API’s native schema or structured-output mode when it supports the fields you need. Treat that feature as a first line of defense, not a replacement for application-side validation. Native constraints may reduce malformed responses, but your application still needs to handle refusals, truncation, transport errors, unsupported schema features, and semantically unsuitable values.

A robust request handler follows a predictable order:

1. Send the input with the schema and versioned prompt.
2. Check the transport response and any provider-level refusal or incomplete-status field.
3. Parse the returned text as JSON if necessary.
4. Validate the parsed object with a trusted library.
5. Run domain checks that the schema cannot express, such as date relationships or permitted business transitions.
6. Return a typed result, or classify the failure for recovery.

How to customize

Make the schema reflect the smallest useful unit of work. Broad schemas encourage vague responses and make failures difficult to diagnose. Prefer an enum when downstream code supports a fixed set of values. Use nullable fields only when “unknown,” “not applicable,” and “missing” have a defined meaning. Otherwise, the model may use null as a general escape hatch.

Keep field descriptions operational. “Priority of the request” is less useful than “high means the user reports a service interruption or a time-sensitive security concern.” Examples can clarify borderline cases, but they should not conflict with the schema. If an output contains free text, set a practical length limit and state whether it should quote, summarize, or transform the source.

Separate validation into two stages:

• Structural validation checks types, required fields, lengths, patterns, and allowed values.
• Domain validation checks conditions such as whether an identifier exists, whether a selected action is authorized, or whether a date range is logically ordered.

Do not automatically retry every failure. Retry transient transport problems and selected generation failures, but treat a deterministic schema error differently. A bounded retry can use the same model with a concise correction message, or a second attempt can use a lower-temperature or more constrained configuration where supported. Set a maximum attempt count and preserve the original failure category.

Repair is appropriate only for limited, low-risk formatting problems, such as a trailing comma or a response wrapped in a code fence. A repair prompt should receive the invalid output and the schema, then return corrected JSON without changing the underlying values. Never use repair to silently invent missing business data. For high-impact actions, send the case to a human or a deterministic fallback instead.

For implementation comparisons, assess developer tools by their validation behavior, observability, schema support, and failure handling rather than by prompt convenience alone. The AI developer tools evaluation checklist offers a useful framework for that broader decision.

Examples

Extraction workflow

For invoice extraction, define fields such as supplier name, invoice number, currency, total, and line items. Require a clear policy for missing values. For example, a missing invoice number might be null, while an unreadable total should produce a validation failure or a review status rather than an invented number. Add domain checks that the line-item sum and total are compatible within an explicitly chosen tolerance.

Classification workflow

For sentiment or support routing, use a closed set of labels and include a confidence or evidence field only if your team has a defined use for it. A confidence number is not automatically a calibrated probability. If the result controls routing, test it against labeled fixtures and monitor disagreement rates by category.

Tool-calling workflow

For an assistant that chooses an API action, constrain the action name and arguments separately. Validate arguments before execution, check authorization outside the model, and make execution idempotent where possible. The model may propose an action, but application code should decide whether that action is permitted.

Test fixtures

Maintain fixtures for ordinary inputs, empty inputs, ambiguous requests, long inputs, malformed source data, unsupported categories, refusals, truncated responses, and provider errors. Assert both acceptance and rejection. A useful fixture records the input, expected structured result or error class, schema version, prompt version, and any required human-review outcome.

Use these fixtures in a prompt testing framework before changing a system prompt, schema, model, or parsing library. The prompt testing frameworks guide and LLM prompt testing framework cover versioning and evaluation practices that complement structured-output validation.

When to update

Revisit the contract whenever the downstream consumer changes, a new category or field is introduced, source documents change, or an observed failure reveals ambiguity. Update the schema, prompt, fixtures, and operational documentation together. A schema change without updated fixtures can create false confidence; a prompt change without a version marker makes regressions difficult to trace.

Review the implementation after changing the model, provider integration, decoding settings, context strategy, or retrieval pipeline. Context-window and latency tradeoffs can affect truncation and incomplete responses, so monitor them with the same discipline as validation failures. Cost and token behavior should also be reviewed as workflows evolve; see the guide to monitoring token usage and controlling AI API costs.

As a practical next step, choose one production response, write its smallest useful schema, and add ten representative fixtures. Implement parse, validate, classify, and bounded-retry stages before expanding the contract. Then measure failure categories for a defined review period. This creates a durable baseline for improving prompt engineering and LLM app architecture without treating valid JSON as proof that the answer is correct.