Stop Regex-Parsing LLM Output: Structured Outputs, JSON Schema and Constrained Decoding

Everyone who shipped an LLM feature before structured outputs existed wrote the same function. Prompt the model with "respond with JSON only, no markdown". Strip the code fences it added anyway. Find the first { and the last }. Try json.loads. On failure, retry with a "your last response was invalid JSON" message. Give up after three attempts and log an error nobody reads.

That function worked about 95 percent of the time, which sounds fine until you process 40,000 documents a month and 2,000 of them silently fall into an exception branch. Constrained decoding removes that entire class of failure, and understanding how it does so tells you exactly which problems it solves and which it very much does not.

Three levels of "give me JSON"

Level one: ask nicely. Put the schema in the prompt and hope. Failure rate depends on model, temperature, prompt length and luck. Unbounded tail risk.

Level two: tool calling. Define a tool whose input schema is your desired shape and force the model to call it. Much better, because the model has been trained on this shape, but the arguments are still generated freely unless the provider enforces them.

Level three: constrained decoding. The sampler itself is restricted so that invalid output is not merely discouraged, it is unreachable. This is the level worth understanding.

How constrained decoding actually works

A language model produces, at each step, a probability distribution over its whole vocabulary. Sampling picks a token from that distribution. Constrained decoding inserts one step in between: before sampling, mask to negative infinity every token that could not legally continue the output under your grammar. The model then samples only from tokens that keep the string a valid prefix of something the grammar accepts.

The machinery differs by what you are constraining:

• Regular expressions and enums compile to a finite state machine. Cheap, fast, and enough for classification and extraction of fixed formats.
• JSON Schema with nesting needs a context-free grammar and a pushdown automaton, because matching brackets is not a regular language. llama.cpp's GBNF (a Backus-Naur variant) and XGrammar both do this.

The naive implementation is slow: checking every token in a 150,000-token vocabulary at every decode step would dominate your latency. XGrammar's contribution was to split the vocabulary into context-independent tokens, which can be checked once and cached, and a small set of context-dependent tokens checked at runtime against the automaton stack - and to overlap that check with the GPU's forward pass. The result is near-zero wall-clock overhead, which is why it became the default grammar backend in both vLLM and SGLang.

The practical consequence: schema-valid output is now a guarantee from the decoder, not a probabilistic property of the prompt. Your parse step cannot fail on malformed syntax. It can still fail on wrong content, and we will come back to that.

Using it on the Claude API

Structured outputs go in output_config.format. The SDK's messages.parse() helper validates the response against your schema for you:

import anthropic

TRANSACTION_SCHEMA = {
    "type": "object",
    "properties": {
        "evidence":     {"type": "string"},
        "reference":    {"type": ["string", "null"]},
        "amount_kes":   {"type": ["number", "null"]},
        "counterparty": {"type": ["string", "null"]},
        "direction": {
            "type": "string",
            "enum": ["received", "sent", "paybill", "withdrawal", "unknown"],
        },
        "balance_after_kes": {"type": ["number", "null"]},
        "confidence": {"type": "string", "enum": ["high", "medium", "low"]},
    },
    "required": ["evidence", "reference", "amount_kes", "counterparty",
                 "direction", "balance_after_kes", "confidence"],
    "additionalProperties": False,
}

client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    system=(
        "Extract a single transaction from the SMS. Quote the exact substring "
        "you relied on in 'evidence' first. If a field is genuinely absent, use "
        "null or 'unknown' rather than inventing it, and lower the confidence."
    ),
    output_config={"format": {"type": "json_schema", "schema": TRANSACTION_SCHEMA}},
    messages=[{"role": "user", "content": sms_body}],
)

if resp.stop_reason == "refusal":
    handle_refusal(resp)          # schema is NOT guaranteed on a refusal
elif resp.stop_reason == "max_tokens":
    handle_truncation(resp)       # a valid prefix of a valid object is invalid JSON
else:
    txn = json.loads(resp.content[0].text)

In TypeScript the equivalent uses Zod:

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";

const Ticket = z.object({
  category: z.enum(["billing", "delivery", "technical", "account", "other"]),
  severity: z.enum(["p1", "p2", "p3"]),
  summary: z.string(),
  customer_reference: z.string().nullable(),
  needs_human: z.boolean(),
});

const client = new Anthropic();

const response = await client.messages.parse({
  model: "claude-haiku-4-5",
  max_tokens: 1024,
  messages: [{ role: "user", content: rawTicketText }],
  output_config: { format: zodOutputFormat(Ticket) },
});

const ticket = response.parsed_output;   // null if the model refused

For tool arguments rather than final responses, set strict: true on the tool definition alongside additionalProperties: false and a complete required list. That guarantees the arguments your handler receives validate exactly, which means you can delete the defensive if "x" not in args checks.

Schema limits worth knowing before you design

The supported subset is real but not the whole of JSON Schema. Supported: object, array, string, integer, number, boolean, null, plus enum, const, anyOf, allOf, $ref, and the common string formats (date, date-time, email, uri, uuid, ipv4). Every object needs additionalProperties: false.

Not supported: recursive schemas, numeric constraints such as minimum and multipleOf, string length constraints, and complex array constraints. The Python and TypeScript SDKs handle this gracefully by stripping unsupported constraints from what they send and validating them client-side, so you get an error rather than silent non-enforcement.

Two operational notes: a new schema pays a one-time compilation cost on first use and is then cached for about 24 hours, so do not generate schemas dynamically per request if latency matters. And structured outputs are incompatible with citations and with assistant prefills.

Doing the same thing locally

vLLM exposes guided decoding through its OpenAI-compatible server, so the same client code works against a self-hosted model:

curl http://localhost:8000/v1/chat/completions \
  -H "content-type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Classify: parcel never arrived"}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "ticket",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "category": {"type": "string",
                         "enum": ["billing", "delivery", "technical", "other"]},
            "severity": {"type": "string", "enum": ["p1", "p2", "p3"]}
          },
          "required": ["category", "severity"],
          "additionalProperties": false
        }
      }
    }
  }'

llama.cpp takes a JSON Schema and converts it to GBNF internally, or you can hand it a grammar file directly. Either way the guarantee is the same, because the mechanism is the same.

The traps that constrained decoding does not fix

This is the part that separates people who have shipped this from people who have read about it.

Well-formed is not correct. The grammar guarantees the shape of the answer, not its truth. A schema-perfect extraction with the wrong amount in it is worse than a parse error, because the parse error was loud. Your evaluation still has to measure field-level accuracy.

The model will fill required fields it cannot support. This is the single biggest trap. If your schema requires invoice_number as a non-nullable string and the document has no invoice number, the decoder will not let the model omit it, so the model produces something. Always give an escape hatch: nullable types, an "unknown" enum member, or a boolean found flag. A schema without an escape hatch is a hallucination generator.

Field order changes quality. Generation is left to right, and every token is conditioned on the ones before it. Put a short reasoning or evidence string field before the fields that depend on judgement, and accuracy improves measurably, because the model has committed its reasoning to the context before it must commit to a label. Put the label first and you get an unreflective guess that the rest of the object then rationalises.

Over-constraining hurts. Forcing a deeply nested twelve-field object out of a small model in one pass often produces worse content than two simpler calls. If quality drops after you add the schema, split the task.

Refusals bypass the schema. If the model declines for safety reasons, the output may not match your schema at all. Check stop_reason before you trust parsed_output, and handle a None parse without crashing.

Truncation looks like corruption. If you hit max_tokens mid-object you get a valid prefix of a valid object, which is to say invalid JSON. Check for stop_reason == "max_tokens" explicitly rather than blaming the schema.

Why this changed the engineering, not just the ergonomics

Before constrained decoding, an LLM was a component whose output contract was "usually a string that usually looks like what you asked for". You could not put that in the middle of a pipeline without a validation-and-retry layer around it, and that layer added latency, cost and a failure mode of its own.

Now the output contract is a type. The model becomes a function with a signature, and the rest of your system can treat it like any other unreliable-but-typed dependency: validate the semantics, handle the error path, and move on. That is a much smaller thing to reason about, and it is the reason LLM features graduated from demos to production plumbing.

Just do not let the type lull you. The compiler checks the shape. Only your eval set checks whether the answer is right.