Structured Output and Evaluating Prompts Like Real Software

Your prompt works perfectly in the demo. You ship it. Then at two in the morning your parser crashes, because the model politely added the words, Sure, here is your JSON, before the actual JSON. This is the most common way language model applications fail, and it has nothing to do with intelligence. Today we fix it, properly.

• Works in demo
• Crashes in production

In the previous lesson we looked at chain of thought and reasoning models, which improve how a model thinks. This lesson is about how it answers, and how you know the answer is good. Part A is structured output: getting valid JSON every single time. Part B is evaluation: testing prompts the way engineers test software, instead of trusting vibes.

• Part A: reliable structure
• Part B: measured quality

By the end you will understand four ways to get structure out of a model, including how constrained decoding physically makes invalid output impossible. You will write a Pydantic validate and repair loop, and a small evaluation harness that prints a results table. This is the last lesson of the prompting module, so we will make it count.

• Pydantic validate and retry
• Small evaluation harness

A program expects an exact format. A language model, as you know from the sampling lesson, samples the next token from a probability distribution, and it is trained to produce likely text, not valid text. Most of the time likely and valid coincide. The rest of the time, your pipeline pays for the difference.

• Programs need exact formats
• Models produce likely text

Failures come in a few flavors. A chatty preamble before the brace. Markdown code fences around the JSON. A trailing comma or single quotes that break the parser. A key renamed from name to full name. Or the right key with the wrong type, like the string twenty instead of the number twenty. Each one is a different crash.

• Chatty preamble
• Markdown code fences
• Trailing commas, bad quotes
• Missing or renamed keys
• Wrong types

Here is why this matters at scale. Suppose each model call returns valid output ninety nine percent of the time. That sounds great. But chain ten such calls in an agent or a workflow, and the success rate is zero point nine nine to the tenth power, about ninety percent. One run in ten fails. At a thousand requests a day, that is a hundred broken requests.

• 99% per step
• 10 steps: about 90%

Option one: ask nicely. Say return only JSON with these keys, and include one example. This works surprisingly often, and modern models are good at it. But it is a request, not a guarantee. The model can still wrap the answer in prose, and you will only find out after the parser fails. Fine for prototypes, risky for production.

• Describe format in the prompt
• Add an example
• No guarantee

Option two: JSON mode, offered by many providers. The API guarantees the output parses as JSON, so no more chatty preambles or broken quotes. But notice what is not guaranteed: the shape. The keys can be missing, renamed, or the wrong type. You get valid syntax, not a valid contract. You still need validation on your side.

• Output is syntactically valid JSON
• Schema not enforced

Option three: structured output APIs. You send a JSON Schema describing exactly the keys, types, and allowed enum values, and the provider guarantees the response conforms to it. This is the strongest option you can use with a hosted model, and the guarantee is enforced during generation, not checked afterward. The next question is how that is even possible.

• You send a JSON Schema
• Output must conform

Option four is the engine underneath: constrained decoding. Recall the generation loop from the sampling lesson. The model produces logits, one score per vocabulary token, then softmax, then we sample. Constrained decoding inserts one step before softmax. At every position, we ask: which tokens are legal here? Every illegal token gets its logit set to minus infinity.

• Mask invalid tokens
• Every step, before sampling

A tiny example. The grammar says the first character must be an opening brace. The model's logits are: Sure, three point five. Opening brace, two. Hello, one. Closing brace, zero point five. Unconstrained, Sure wins. With the mask, Sure, hello, and the closing brace become minus infinity. Softmax over what remains gives the opening brace probability one. The model had no choice.

• Expect an opening brace
• Sure scores highest, but is illegal

Now a position where a number is expected. Logits: the token seven scores two, three scores one, and cat scores three. Cat is illegal, so it is masked. What remains: e squared is about seven point four, e to the one is about two point seven, total ten point one. So seven gets about seventy three percent and three gets twenty seven percent. The model still chooses among valid options, using its own preferences.

• Digit or minus sign allowed
• Logits 2 and 1 for 7 and 3
• Probabilities 0.73 and 0.27

How does the system know what is legal? The schema is compiled into a grammar, often a finite state machine. After each token, the machine moves to a new state, and each state knows which tokens may come next. After an opening brace, expect a quote starting a key. After a key, expect a colon. After an integer value, expect a comma or a closing brace. Libraries for open models implement exactly this.

• Grammar compiled to a finite-state machine
• State decides the legal tokens

Constrained decoding guarantees form, never truth. A perfectly valid object can still say the age is seven for someone who is forty. Also, forcing a model straight to a tiny answer field can reduce quality on hard questions, because it loses room to think. A useful trick is to put a short reasoning string field before the answer field in your schema, since generation goes in order.

• Valid does not mean correct
• Tight schemas can hurt quality
• Allow a reasoning field

Even with guarantees, you should validate in your own code, because the schema is your contract with the rest of the system. Pydantic is perfect for this. Define a class with typed fields: a category limited to billing, bug, or other; a priority integer between one and five; and a summary string with a maximum length. The same class can generate the JSON Schema you send to the API.

• Pydantic model = schema + validator
• Types, ranges, enums

Here is the loop. Call the model. Parse the text as JSON. Validate it with Pydantic. If it passes, return a typed object. If it fails, take the exact validation error, such as priority must be less than or equal to five, append it to the conversation, and ask the model to fix it. Cap the attempts at three. Errors are specific, and models are good at repairing specific errors.

• Parse the text
• Validate against the model
• On error: feed the error back

In code: build a messages list with the task. Loop up to max tries. Call your model function and get the raw text. Inside a try block, call Ticket dot model validate json on that text and return the result. In the except block for ValidationError, append the model's bad reply and a user message containing the error string, then continue. After the loop ends, raise an error, so failure is loud, never silent.

• try model_validate_json
• except ValidationError
• retry with error text

If each attempt is ninety percent valid and attempts were independent, three tries give one minus zero point one cubed, or ninety nine point nine percent. But attempts are not independent. If the input is truly confusing, the model fails the same way again. So retries help with random slips, not systematic problems. Log every failure, because those logs become tomorrow's test cases.

• 90% valid: 3 tries gives 99.9%
• Failures are correlated
• Log every failure

Now Part B. Most prompt changes are judged like this: try five examples, they look better, ship it. That is not evidence. Five examples cannot represent the thousands of inputs real users will send, and your eye is drawn to the cases you happen to remember. Software engineers solved this long ago with tests. Prompts deserve the same discipline.

• Tried it on five examples
• Looks better to me

The foundation is a golden dataset: inputs paired with the expected outputs, verified by a human. Sample from real traffic, and deliberately add edge cases: empty input, another language, ambiguous requests, and attempts to trick the model. Fifty to two hundred good examples already beat any vibe check. Add every production failure you find, so the set grows with your product.

• Input plus expected output
• Real traffic, edge cases
• Start with 50 to 200

Now metrics. The simplest is exact match: the output equals the expected output, score one, otherwise zero. It is cheap, deterministic, and great for classification or short answers. But for structured objects it is harsh. If one field out of four is wrong, the whole example scores zero, and you cannot see where the prompt is failing.

• Whole output must equal expected
• Strict, simple, cheap

Field-level accuracy fixes that. Compare each field separately. If three of four fields are correct, that example scores zero point seven five. Then average per field across the dataset. Now you might see category is right ninety six percent of the time, but priority only seventy percent. That tells you exactly where to improve the prompt. Use tolerant comparison for free text, like normalized strings or numeric tolerance.

• Score each field separately
• 3 of 4 correct = 0.75

Some outputs have no single right answer, like summaries or explanations. Here you write a rubric: a short list of criteria, each scored on a small scale. Is it faithful to the source? Does it cover the key points? Is it concise? Clear criteria matter more than a vague question like, is this good. Humans can apply a rubric, and so can another model.

• Summaries have no single right answer
• Score against criteria

Human grading is slow, so a common approach is LLM as judge: give a strong model the input, the answer, and the rubric, and ask for scores in structured output. That is where Part A pays off again. It scales to thousands of cases overnight. But the judge is a model, with its own quirks, and treating its scores as ground truth is a mistake.

• A model grades the outputs
• Scales to thousands of cases

One known bias is position bias. When a judge compares two answers, it can prefer whichever comes first, regardless of quality. The fix is simple: run every comparison twice with the order swapped. If the verdict flips, count it as a tie, or as an unreliable judgment. This cheap trick removes a surprising amount of noise.

• Judges can favor the first answer
• Swap order and rerun

Another is verbosity bias: judges tend to reward longer answers, even when the extra words add nothing. Judges may also favor outputs written in their own style. Mitigate by putting length limits in the rubric, asking for evidence for each score, and, most importantly, calibrating the judge. Hand-label thirty examples yourself and check how often the judge agrees with you.

• Longer often scores higher
• Models may favor their own style

Calibration is concrete. Take thirty outputs, label them pass or fail yourself, and let the judge label them too. Suppose they agree on twenty five of thirty. That is eighty three percent agreement. Is that enough? It depends on the stakes. If the judge disagrees mostly in one direction, you learn its bias and can fix the rubric. Never trust a judge you have not checked.

• Compare with human labels
• Agreement rate before trust

Once you have a dataset and metrics, you get regression tests. Every time you edit a prompt, swap a model, or upgrade a library, run the full golden set. Set thresholds, for example field accuracy at least ninety percent, and fail the build if the score drops. A fix for one case often breaks three others, and only a suite will tell you.

• Run suite on every change
• Fail the build below threshold

A minimal harness is short. Load the dataset. For each prompt version and each example, call the model, validate the output, and score each field. Collect the rows and print a table: prompt version down the side; exact match, per-field accuracy, and valid JSON rate across the top. Seeing version two beat version one on priority, with everything else equal, is far more useful than a feeling.

• Run each prompt on each example
• Report per-field scores

Now a trap. Prompt A gets forty two of fifty right, eighty four percent. Prompt B gets forty five, ninety percent. B wins? Check the noise. For B, the standard error is the square root of point nine times point one over fifty, about four point two points. A rough ninety five percent interval is plus or minus eight points. The two intervals overlap heavily. Six points on fifty examples may be luck.

• A: 42 of 50 = 84%
• B: 45 of 50 = 90%

Better: compare on the same examples, in pairs. Examples where both prompts succeed or both fail tell you nothing. Only disagreements matter. Say B fixes five examples that A got wrong, while A fixes two that B got wrong. Five to two, out of seven. That leans toward B, but seven disagreements is thin evidence. To be confident, add more examples, or require a larger, consistent gap.

• Same examples for both prompts
• Only disagreements matter

There is a second source of noise: the model itself. With temperature above zero, the same prompt scores differently on different runs. Evaluate with low temperature where you can, and for noisy tasks repeat runs and look at the spread. A difference smaller than the run to run spread is not a difference. Report ranges, not single numbers.

• Sampling randomness
• Set temperature low
• Repeat runs for noisy tasks

The classic mistake is tuning on the test set. You run the suite, see failures, tweak the prompt to fix exactly those, and rerun. The score climbs, but you have just memorized your own examples, the same overfitting you know from the ML course. Split your data: a dev set for iterating, and a locked test set you check rarely, before decisions.

• You edit prompts after seeing failures
• Score inflates, generalization does not

A few more traps. Tiny datasets that cannot detect real differences. Datasets with only easy, average cases. Uncalibrated judges. And ignoring cost and latency: a prompt that gains one point of accuracy but triples tokens may be a bad trade. Always put accuracy, valid output rate, tokens, and response time in the same results table.

• Tiny datasets
• Only average cases
• Trusting judge blindly
• Ignoring cost and latency

Treat prompts as code. Store them in version control, not in a dashboard text box that anyone can edit. For each version, record the model name, the settings like temperature, the schema, and the eval scores. When production quality drops, you can see exactly what changed, and roll back in minutes. A prompt without a version history is a bug you cannot reproduce.

• Prompts live in version control
• Record model, settings, dataset
• Link version to scores

The best evaluation sets come from real failures. Log invalid outputs, retries, and user complaints. Label them, and add them to the golden set. Every bug becomes a permanent test. Over time, your suite encodes everything you have learned about how your system breaks, and each prompt change is checked against that history.

• Log failures
• Turn them into test cases
• Dataset keeps growing

Let us recap. Free text breaks pipelines, and small error rates compound. You have four tools: ask nicely, JSON mode, schema constrained APIs, and grammar constrained decoding, which masks illegal token logits to minus infinity at every step. Valid form is not correct content, so keep Pydantic validation and a capped retry loop that feeds errors back.

• Ask, JSON mode, schema, grammar
• Mask logits to enforce legality
• Validate and retry anyway

For evaluation: build a golden dataset with edge cases. Score with exact match, field level accuracy, and rubrics. Use an LLM judge, but swap positions, watch for verbosity bias, and calibrate against your own labels. Compare prompts on the same examples, respect the noise, keep a locked test set, and version everything. That closes our prompting module.

• Golden dataset with edge cases
• Field-level and rubric metrics
• Judge with care
• Dev set and locked test set

Next time we start a new module: embeddings and vector search. How do you turn a sentence into a point in space, so that similar meanings land close together? We will cover sentence embeddings and contrastive training, the foundation for retrieval augmented generation.

• From prompting to retrieval
• Meaning as vectors