← All Posts

The Contract Comes Before the Pipeline

Work

In Data Products for Robots I argued that artificial intelligence (AI) agents are now first-class consumers of your data, and that the documentation has to change to suit a reader that cannot ask a follow-up question. The artifact that makes that argument real is the data contract. I have written several, for consumers ranging from a downstream application to an agent building a pipeline, and the shape converged fast. This is the shape.

What a Contract Is Not

It is not a schema dump, and it is not a wiki page. A schema dump tells you what the columns are called. A wiki page tells you what someone hoped the data would be. A contract is short by design and every line in it is one of two things: a fact about the data, or a rule a machine can check. If a sentence is neither, it does not belong.

It is also not the pipeline's documentation. The contract comes first. It is what the consumer signs off on before anyone builds, and what the producer is held to after. When the order is reversed, the contract becomes a description of whatever got built, which is a different and much less useful document.

Every Section Prevents a Specific Failure

SectionThe failure it prevents
PurposeA dispute about priority with no shared fact to settle it
Grain and keysTwo teams joining on different ideas of what one row is
Schema, with a rule per fieldA field that is valid in the producer's head and invalid in the consumer's job
Data-quality rulesQuality as a feeling, argued about after the fact
AvailabilityLate data discovered by the consumer's users instead of the consumer
Change policyA rename shipped on a Friday that breaks a consumer on a Monday
ExamplesA pipeline built to the schema that fails on the first real row

The examples section is the one people skip and the one an agent needs most. One happy row, one edge row with a sentence on what makes it an edge, one error row with what the consumer should see and do. They double as test fixtures, which means the contract ships with its own tests.

Two Rules for Filling It In

Every field traces to a source. The schema table has a Source column, and it is not documentation, it is evidence: the system, table, and column the value comes from, or the derivation if it is computed. If you cannot name where a field comes from, you are not ready to promise it. This is evidence over inference applied to a column.

Every rule is a query. "Between zero and one," "matches this pattern," "present when status is active": each of these can be run. "Should be valid" cannot. Write the query, run it against real data before the contract goes active, and record what it returned. A rule that cannot be run is a hope with a table around it.

Availability Is a Number

The section that separates a contract from a data dictionary is availability. It names where the data lands, the cutoff by which each period is complete, with a time zone, and a service level agreement (SLA) target stated as a number: available by the cutoff on some percentage of business days per month. And it names what measures the number, which must be a job or a dashboard, never a person. A target measured by whoever remembers to check is not a target.

The change policy at the end classifies changes as breaking, additive, or fix, and sets the notice and sign-off each requires. Deciding which class a change falls into should not be a judgment call either. That is next in this track.

Take It

The template, with every section and the reasoning, is data-contract-template.md. Fill every section. A blank one is a decision the consumer will make for you, later, at the worst time.

This is part of a track on agent tooling and spec-driven development that goes deeper than a field report, with something you can download and run in each one. Free for now while the track grows; the deepest installments become a paid tier later.

0 Comments