JSON Schema Design for Structured Web Data Extraction

Web scraping has changed its center of gravity. The dominant extraction pattern used to be CSS selectors: scripts that located data by its position in the page's HTML. That pattern has given way to schema-driven LLM pipelines, and the engineering effort that used to go into maintaining selectors now goes into designing schemas.

The reason for the shift is structural. A selector-based scraper finds a price by walking to a specific div, a specific class name, a specific nesting path. Changing the page layout breaks the path. Extraction logic survives changes that would have required a full script rewrite under the old approach.

Picture the extraction pipeline as four stages: fetch, render, clean, extract. The language model only enters at the last stage, after the page has already been pulled down, rendered, and stripped of noise. That last stage is where accuracy gets decided, and the schema is what shapes it. A weak schema wastes a clean, well-rendered page. A strong one makes full use of it.

Constrained decoding, under names like JSON Mode and Strict Mode, is now standard across the major model providers. That means the machinery for forcing a model's output into valid JSON is already built and already reliable. The open question in production extraction is no longer whether the output will be valid JSON. It's whether that valid JSON holds the right values. Schema quality is where the remaining work lives.

What a schema controls in an LLM extraction pipeline

It's the instruction set the model reasons against. It's the validation contract that the output gets checked against. And it's the interface between raw web content and whatever system consumes the output next, a database, a search index, an agent's tool call.

Schema-based extraction tools accept a schema written in JSON Schema format, and the model uses it to find fields by meaning rather than by position in the HTML, which is the mechanism behind the resilience described above: a field described as "the product's current sale price, not the original list price" survives a redesign because the description still points to the right concept even when the DOM path changes.

Each of the three jobs works differently. As an instruction, the field names, types, and descriptions function as the closest thing to a prompt the model receives for that individual field. As a downstream contract, the schema defines the exact shape that lands in a database table, a retrieval index, or an agent's function call, and changing that shape later breaks every consumer built against the old one.

JSON Schema wasn't built for this job originally. It was designed as a contract between human developers and static systems, a way to validate that a configuration file or an API payload had the right shape. Using it to instruct a reasoning model is a repurposing, not its original design intent, and that mismatch is where most extraction failures begin. A schema can be perfectly valid as a JSON Schema document and still give the model almost nothing to reason against.

The five failure modes that survive schema validation

Valid JSON is necessary for a working pipeline, but it doesn't guarantee the values inside are correct. A model can satisfy every type constraint, every required field, every enum restriction, and still return a price that's wrong, a date that contradicts another date in the same object, or a field fabricated out of nothing on the source page.

A structured output benchmark covering 21 models found that every model tested exceeded an 84% JSON pass rate, while none, at the time of initial publication, surpassed 80.4% value accuracy. The leaderboard behind that benchmark has since shown models crossing that figure, but the structural point stands regardless of where the exact ceiling sits: JSON validity and value accuracy move independently of each other. That gap, between a schema that validates and a value that's true, is the entire territory that schema design and semantic validation exist to cover.

This isn't theoretical. Semantic parameter errors, where the structure is valid but the values violate the business logic they're supposed to represent, have been documented in production LLM API calls, and schema validators have no way to catch them because nothing about the error breaks the schema's rules.

Five failure modes account for most of what goes wrong after validation passes, and each one points to a specific fix in schema design:

Enum hallucination happens when a model selects a valid enum member that carries the wrong meaning for the data in front of it. The fix is building enums whose members are semantically unambiguous, not just syntactically distinct from each other.

Confident fabrication happens when a free-text field gets filled with a plausible-sounding value that doesn't actually appear anywhere in the source.

Cross-field contradiction happens when every field passes its own individual check but the values together describe something impossible, an end date before a start date, a subtotal that doesn't match the line items.

Distributional collapse happens when a model, faced with ambiguity, converges on a safe, generic default.

Array hallucination happens when a model invents entries for an array instead of leaving it empty, filling it with fabricated items. The fix is an explicit instruction in the schema that an empty array is the correct output when no items are present.

Enforcement itself carries a cost worth factoring in. On any task where real reasoning is required to get the value right, schema enforcement isn't free. The schema's job is to reduce the ambiguity the model has to resolve, so less of its reasoning budget gets spent fighting the format and more of it goes toward getting the value right.

Field-level schema design: how descriptions, types, and enums instruct the model

The description property on each field is the single most reliable lever available in schema design. It functions as the model's primary instruction for that field, and its quality is what separates a model that extracts the right value from one that infers or fabricates a plausible one.

A strong field description does four things. It states what the value actually is, not just the field name restated in sentence form. It states where on the page to look, for example: "the price shown in the product header, not the shipping estimate." It states the format expected, for example "ISO 8601 date string" or "two-letter ISO country code." And it states what to return when the value is absent, for example "return null if not present on the page; do not infer."

Compare a weak version of a field description against a tight one. The second gives it a target, a location, a format, and a fallback.

Type selection carries weight beyond syntax. A mismatched type produces a coercion that the schema validator will happily accept and that downstream systems will then reject or mishandle.

Enum design has two rules that matter more than the rest. And every enum needs an explicit catch-all, an "other" or "unknown" value, so the model is never forced into choosing the least-wrong option from a list where nothing actually fits.

Required and optional field designation deserves the same care. Reserve the required designation for fields that are definitionally present on every instance of that page type.

Schema structure decisions: nesting, arrays, and depth thresholds

Individual field design only solves part of the problem. The overall shape of the schema, how deep it nests, how many keys it carries, determines how far a pipeline scales before failure rates climb sharply. A schema that performs well at a moderate size can fail abruptly once it crosses a size threshold, rather than degrading in a smooth, predictable curve.

Nesting carries a direct accuracy cost. Nest fields together only when they're always extracted as a group and always present together on the page. Flatten fields that are independent of each other or that might be conditionally absent.

A schema that needs heavy nesting to represent its target data is often telling the designer something: that the extraction job should be split into separate calls. Extract the parent object in one pass, then pass that result back into a second call that extracts the child fields, rather than asking one schema to resolve the whole structure at once.

Arrays need the same attention as any other field, applied per item. The item schema inside an array gets applied independently to every element the model decides to include, so a vague item schema produces hallucinated entries the same way a vague top-level field produces fabricated values. Avoid unbounded arrays on pages where the count of items is unknown and could be large: an unbounded array paired with a vague item schema is close to the highest-risk configuration a schema can have.

As a working rule, treat more than two to three levels of nesting as a signal, not a target to design toward. When a schema needs that much depth to represent the data, decompose the extraction into multiple passes.

Decomposed extraction: when to split one schema into sequential passes

A single schema asked to resolve everything about a complex page at once is the wrong tool for that job. Splitting the work into sequential, narrower passes produces more accurate results than one large schema carrying the full weight of the page.

The pattern starts with identifying what kind of page or document is in front of the model, a narrow classification extraction, before running the field-level schema that matches that type. This is the production version of what research on agentic extraction describes as a multi-step process: classify first, extract second.

A few cases show where this earns its cost. An authenticated portal benefits from a first pass that confirms the navigation actually landed on the expected page state before a second pass runs the real data schema against it.

Crawl4AI's handling of messy, freeform pages illustrates the same principle from the tooling side: a JSON schema describing the target data, paired with a natural-language instruction, gets passed along with the page content to whichever LLM provider is configured, and the framework returns structured JSON back. The decomposition is built into the workflow itself.

Grounding schemas against the live web: freshness, crawl cadence, and change detection

A schema built for a page's current structure becomes a liability the moment that structure changes, so the pipeline around it has to watch for that change. Schema design and monitoring cadence aren't separate concerns in production: they depend on each other.

Two monitoring strategies interact with schema design in different ways. In that second approach, the schema is what defines "moved" in the first place: a narrow, well-scoped schema detects changes that matter, while a broad, loosely scoped one generates noise from changes that don't.

The page itself matters as much as the schema pointed at it. Research comparing a human-oriented baseline website to one designed for agent readability found that structural clarity, action cues, evidence signals, and temporal validity indicators substantially improved the reliability of AI browser agents, with the agent-ready site reaching a strict success rate of 89.3% and requiring fewer steps to complete the same tasks. A schema aimed at a well-structured page has a more stable target than one aimed at a page built only with human readers in mind.

Semantic validation: catching the failures schema enforcement misses

Schema validation is the first gate a pipeline needs, not the last one. Checking that extracted values are coherent, grounded in the source, and consistent with each other is what separates a pipeline that produces valid-looking output from one that produces output safe to act on.

If validation stops at the schema, those are exactly the failures that reach whatever system consumes the output next.

A few techniques close that gap. Confidence thresholds let a pipeline treat uncertain fields differently from confident ones: a low-confidence field gets returned as null and flagged for review rather than accepting a value that merely sounds plausible.

The STAGE approach to generating training data offers a template for this at the data-construction stage: it verifies generated JSON values directly against the underlying source document rather than trusting the model's own output, and that same verification logic applies just as well at inference time inside a live extraction pipeline. Across the structured output benchmark discussed earlier, value accuracy lagged well behind JSON pass rate for every model tested, and that lag is the exact scope of work semantic validation exists to cover.

Schema design for specific extraction domains: products, brand data, and documents

The principles above apply differently depending on what's being extracted, and the differences are predictable.

Product data calls for tight enums around availability and condition, strict price formatting rules given how many ways a price can be displayed on a page, and explicit null handling for fields like sale price or shipping cost that aren't present on every listing.

Enum use here tends to be lighter, since brand attributes resist the kind of clean categorical boundaries that product conditions or shipping statuses allow.

Document extraction, reports, filings, long-form pages, is where the decomposition pattern matters most. Section structure should get extracted before field-level data, given how much a single monolithic schema struggles against a long, unevenly structured document. Cross-field consistency rules carry the most weight here too, since financial and legal documents are full of values that must agree with each other, a total that must match its components, a date that must fall within a stated range, and those are exactly the contradictions that schema validation alone cannot see.