Structured Output with Pydantic: Validate, Retry, Ground
Hands-on lab · IDE in your browser

Structured Output with Pydantic: Validate, Retry, Ground

Turn messy supplier emails into validated Python objects: declare the order shape with Pydantic types and constraints, add an ISBN check-digit validator and a computed total, hand the model the JSON schema as response_format, retry with the validation errors when a reply is wrong, and reject valid-looking output whose values are not in the source email.

Time
55 min
Checked steps
5
Level
Beginner
Setup
None
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

Lab cockpit55 min · 5 stepsSession running
2 / 5 steps passingHand the model the schema · step 3 of 5
orders.py▶ Run✓ Check
def extract(email, client=None):    """One attempt: return a validated BookOrder (raises ValidationError / ValueError if the reply is bad)."""    client = client or CLIENT       
TerminalOutput

The job

Brightline Books orders stock from three suppliers, and each confirms orders by email in its own layout: a tidy block, a chatty paragraph, a table with shipping costs mixed in. The stock system needs the same validated fields from all of them. You define the order as a Pydantic model, extract it with a language model guided by the model's JSON schema, retry when the output breaks a rule, and make sure no value reaches the stock system unless it can be found in the email. One email contains a real supplier typo that no amount of retrying should hide.

5 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Describe the shape of an order

    Brightline's suppliers confirm orders by email, each in its own layout (open the emails/ folder).

  2. 2

    Add the rules a type cannot express

    Types catch the wrong *kind* of value.

  3. 3

    Hand the model the schema

    BookOrder.model_json_schema() turns your class into a JSON schema: the standard, machine-readable description of the fields, their types and constraints.

    You writeextract()
  4. 4

    Retry with the errors

    When a reply fails validation, the cheapest fix is usually to show the model what was wrong and ask again.

    You writeextract_with_retry()
  5. 5

    Valid is not the same as true

    A validator proves the output has the right *shape*.

    You writeungrounded()process()

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Describe the shape of an order

Brightline's suppliers confirm orders by email, each in its own layout (open the emails/ folder). The stock system needs the same fields from every one: supplier, order reference, currency, delivery date and the lines ordered. In the Files and JSON lab you checked each field by hand with if statements. With more fields that gets long and easy to get wrong. Pydantic lets you declare the shape once and checks it for you.

A Pydantic model is a class whose attributes are fields with types:

class Line(BaseModel):
    quantity: int = Field(gt=0)      # an integer greater than 0

Line.model_validate(some_dict) returns a Line if the data fits, and raises a ValidationError listing every problem if it does not. The same types will become the JSON schema you show the model in Step 3.

Do this

1. Fill in Line: isbn: str, title: str with Field(min_length=1), quantity: int with Field(gt=0), unit_price: float with Field(ge=0).

2. Fill in BookOrder:

  • supplier: str = Field(min_length=1)
  • order_ref: str = Field(pattern=r"^[A-Z]{2}-\d{5}$"): two capitals, a hyphen, five digits
  • currency: Literal["GBP", "EUR", "USD"]: only those three values
  • delivery_date: Optional[date] = None: a date, or None when the email has none
  • lines: list[Line] = Field(min_length=1): at least one Line

3. Run. Four dictionaries go through BookOrder.model_validate. Read the errors: each names the field and the rule. And notice the last one: the quantity "4" arrives as text and is accepted as the number 4. Pydantic converts obvious cases by default (strict=True turns that off).

orders.py, the file you edit89 lines
"""Supplier emails in, validated BookOrder objects out."""
import json
import re
from datetime import date
from pathlib import Path
from typing import Literal, Optional

from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError, computed_field, field_validator

CLIENT = OpenAI()
MODEL = "meta/llama-3.1-8b-instruct"


# ---------- Step 1 + 2: the shape of an order, and the rules it must obey ----------
class Line(BaseModel):
    # TODO (Step 1): four fields with types and constraints:
    #   isbn: str
    #   title: str, at least 1 character          -> Field(min_length=1)
    #   quantity: int, greater than 0              -> Field(gt=0)
    #   unit_price: float, 0 or more               -> Field(ge=0)
    pass



class BookOrder(BaseModel):
    # TODO (Step 1):
    #   supplier: str, at least 1 character
    #   order_ref: str matching two capital letters, a hyphen, five digits  -> Field(pattern=r"^[A-Z]{2}-\d{5}$")
    #   currency: one of "GBP", "EUR", "USD"                                 -> Literal["GBP", "EUR", "USD"]
    #   delivery_date: a date, or None when unknown (default None)          -> Optional[date] = None
    #   lines: a list of Line with at least one item                         -> list[Line] = Field(min_length=1)
    pass



# ---------- Step 3: ask for JSON that matches the schema ----------
SYSTEM = ("You extract purchase orders from supplier emails. Reply with one JSON object that matches this JSON "
          "schema, and nothing else. Use null for a delivery date the email does not give. Copy ISBNs, "
          "references and prices exactly as written.\n\nSchema:\n")


def json_block(text):
    """The outermost {...} in a reply (models sometimes wrap JSON in fences or chatter)."""
    start, end = text.find("{"), text.rfind("}")
    if start == -1 or end < start:
        raise ValueError(f"no JSON object in the reply: {text[:80]!r}")
    return text[start:end + 1]


def extraction_messages(email):
    """The two messages that start an extraction: the system prompt with the schema, then the email."""
    return [
        {"role": "system", "content": SYSTEM + json.dumps(BookOrder.model_json_schema())},
        {"role": "user", "content": email},
    ]


def extract(email, client=None):
    """One attempt: return a validated BookOrder (raises ValidationError / ValueError if the reply is bad)."""
    client = client or CLIENT
    raise NotImplementedError("extract() arrives in Step 3")


# ---------- Step 4: when validation fails, show the model the errors ----------
def extract_with_retry(email, attempts=3, client=None):
    """Return (order, attempts_used). After a failed attempt, send the model its own reply and the
    validation errors, and ask for corrected JSON. Raise the last error if every attempt fails."""
    client = client or CLIENT
    raise NotImplementedError("extract_with_retry() arrives in Step 4")


def describe_error(err):
    """One readable line for any extraction error."""
    if isinstance(err, ValidationError):
        return "; ".join(f"{'.'.join(map(str, e['loc']))}: {e['msg']}" for e in err.errors())
    return str(err)


# ---------- Step 5: a valid object can still be wrong ----------
def ungrounded(order, email):
    """Values in the order that do not appear in the email: each ISBN (digits only) and the order_ref.
    Return a list of short problem descriptions; an empty list means everything was found."""
    raise NotImplementedError("ungrounded() arrives in Step 5")


def process(folder, out_path, client=None):
    """Extract every email in folder; write a JSON list of {"file", "ok", "attempts", "order" | "error"}."""
    raise NotImplementedError("process() arrives in Step 5")
Provided for you:emails/01-harbour.txtemails/02-pelican.txtemails/03-atlas.txtemails/04-harbour.txtemails/05-pelican.txtemails/06-atlas.txtemails/07-harbour.txtemails/08-harbour.txttry_it.py

Frequently asked questions

If the provider supports structured outputs, why validate?

Schema modes enforce types and required fields on providers that support them, and some providers ignore the parameter. Business rules such as check digits or cross-field totals are outside the schema. Validating every reply with Pydantic catches both.

How does retrying with validation errors work?

The failed reply is sent back as an assistant message, followed by a user message with Pydantic's error text, which names the field and the rule. The model corrects its own output. This is the loop that libraries such as Instructor run for you.

What is grounding in this context?

Checking that extracted values actually appear in the source. A valid ISBN that is not in the email is invented. The lab rejects such orders, including when the model 'repairs' a typo into a different, valid ISBN.

Which Pydantic version does the lab use?

Pydantic 2, with Field constraints, Literal, field_validator, computed_field, model_validate_json and model_json_schema.

Why structured output needs validation, retries and grounding

Asking a language model for JSON gets JSON most of the time. Production systems need it every time, with the right fields, types and values, and they need to know when it went wrong. Pydantic is the standard Python tool for that: a class declares the shape, and validation reports every problem by field. In this lab you build an extraction pipeline for purchase orders. You declare fields with types, patterns, ranges and allowed values, add a custom validator for ISBN-13 check digits and a computed total, generate a JSON schema for the prompt and for the provider's structured output mode, write the validate-and-retry loop that tools such as Instructor automate, and finish with a grounding check, because a retry loop can push a model into inventing a value that passes validation.