Constrained Decoding: Making Invalid Output Unreachable

Ask-and-retry works most of the time, which is a poor guarantee for anything feeding a downstream system. Constraint changes the guarantee rather than improving the odds.

Retry loops are not a strategy

The standard way to get structured output from a model is to ask for JSON, parse it, and retry when parsing fails. It works most of the time, which is the problem — most of the time is a poor guarantee for anything that feeds a downstream system.

Constrained decoding replaces the guarantee. Rather than checking the output afterwards, the sampler is restricted at each step to tokens that keep the output valid under a schema or grammar. Invalid output is not rejected; it is unreachable.

What it guarantees, and what it does not

This is the distinction that matters and the one most often blurred. Constrained decoding guarantees form. It says nothing about content.

{
  "file": "src/does_not_exist.py",
  "line": 999999,
  "severity": "critical",
  "message": "Undefined behaviour in the quantum module"
}

Every field is the right type. The file does not exist, the line is past the end, and the module is imaginary. A schema cannot catch any of that. Teams that adopt constrained decoding and stop validating semantics have swapped one class of failure for a quieter one.

Designing a schema the model can satisfy

A schema is not only a validator; it is also an instruction. Its structure shapes what the model produces, and small design choices change output quality noticeably.

{
  "type": "object",
  "properties": {
    "reasoning": {
      "type": "string",
      "description": "Why this is a problem. Write this BEFORE deciding severity."
    },
    "severity": { "enum": ["info", "warning", "error"] },
    "file":     { "type": "string" },
    "line":     { "type": "integer", "minimum": 1 },
    "fix": {
      "type": "object",
      "properties": {
        "description": { "type": "string" },
        "confident":   { "type": "boolean" }
      },
      "required": ["description", "confident"]
    }
  },
  "required": ["reasoning", "severity", "file", "line", "fix"],
  "additionalProperties": false
}

Three deliberate choices. reasoning comes first, so the model generates its justification before committing to a severity rather than after — ordering in a schema is ordering in generation. Every categorical field is an enum, which makes an out-of-range value impossible rather than merely unlikely. And confident gives the model somewhere to express uncertainty, which is much better than it expressing uncertainty by inventing a hedge in the description field.

Two failure modes of over-constraint

Before reaching for a schema everywhere, two things go wrong when the structure is too tight.

Required fields the model cannot fill. A schema requiring line for every finding forces a number even when the issue is file-level. The model produces one, and it is wrong. Make fields optional when the answer might legitimately be "not applicable".

No room to decline. A schema with no way to express "I found nothing" or "I am not sure" guarantees a confident answer, because the structure has no other shape available. Include an explicit findings: [] path and a confidence field, or you have designed a system that cannot report an absence.

Grammars, for output that is not JSON

JSON Schema covers structured data. When the output is a language — a query, a DSL, a config format — a context-free grammar constrains it the same way.

root        ::= "SELECT " columns " FROM " table where? ";"
columns     ::= "*" | ident ("," ws ident)*
table       ::= ident
where       ::= " WHERE " condition (" AND " condition)*
condition   ::= ident ws op ws value
op          ::= "=" | "!=" | "<" | ">" | "<=" | ">="
value       ::= number | "'" [^']* "'"
ident       ::= [a-zA-Z_] [a-zA-Z0-9_]*
number      ::= [0-9]+
ws          ::= " "?

A model decoding under that grammar cannot emit a DROP, cannot emit a subquery, and cannot emit anything that fails to parse. For a tool that turns a natural-language question into a query against a fixed schema, this is a stronger safety property than any amount of instruction — the dangerous statements are not in the language.

Grammar authoring is genuinely fiddly, and this is a good task to hand an agent: give it examples of valid and invalid output and ask for a grammar that accepts the first set and rejects the second. Then test it against both sets, because a grammar that is too permissive fails silently.

Where to apply it

Row four is worth stating plainly, because it is a tempting idea. Models rarely produce syntactically invalid code, and the compiler catches it when they do. The failures in code generation are semantic, and no grammar addresses those.

Where the constraint lives

Support varies by where you run inference, and the difference decides what is available to you.

The practical consequence: if arbitrary grammars matter to your use case — a DSL, a query language, a config format — that is an argument for local inference on its own, independent of any privacy or cost consideration. It is one of the few genuinely capability-shaped reasons to run your own.

Check what your provider supports before designing around it. "JSON mode" and "constrained to this schema" are different guarantees, and the first is closer to a strong hint than to a constraint.

The cost nobody mentions

Heavy constraint degrades quality. A model forced through a narrow structure has less room to arrive at a good answer, and over-specified schemas produce output that satisfies the shape while saying less.

The practical rule: constrain the parts a program consumes, and leave the parts a human reads free. A schema with a tightly-constrained severity enum and an unconstrained reasoning string gets both properties. One that also enumerates permissible reasoning phrasings gets neither.