A spec for data you already have¶
The other tutorials start from a spec you write. This one starts from files you already have: three tables from a small shop. You profile them into specs, with the keys between them found from the values; edit what the profile got too exact; validate the files against the specs; watch a later month for drift; and make a fake tenth of the data for your tests.
Download the three files into a folder called shop, and run everything
from the folder that holds it:
customers.csv: 150 customers, with anid, anemail, acountryand the date theysigned_up.orders.csv: 600 orders, each with acustomer_id, arefsuch asORD-100037, the date it wasplaced_onand astatus.lines.csv: 1,500 order lines, each with anorder_id, askusuch asSKU-9881, aqtyand aprice.
Profile the files¶
One command profiles all three:
It prints a line saying it inferred 3 specs, with 2 foreign keys between them.
shop.yaml is a registry: one spec per file, named after it
(Customers, Orders, Lines). Each column has the claims its values
support:
- Its dtype and extremes:
qtyis anInt64from 1 to 5, andplaced_onaDatewithin 2025. - Its values, where there are few:
statusis anEnumof four values, and--weightsrecords how often each occurs. - Its shape, where it has one:
emailnames theemailformat, andrefandskuget the pattern their values share,^ORD-[0-9]{6}$and^SKU-[0-9]{4}$. - Its key, where it is one: every
Orders.customer_idis aCustomers.id, and its name says so, so it becomes a foreign key. The same goes forLines.order_idintoOrders.id.
The same thing in Python is profile_all, over frames you have already read:
shop = {
name: pl.read_csv(f"shop/{name.lower()}.csv", try_parse_dates=True)
for name in ("Customers", "Orders", "Lines")
}
inferred = polspec.profile_all(shop)
for spec in inferred.specs:
for key in spec.foreign_keys:
print(f"{spec.name}.{key.columns[0]} -> {key.references}.{key.ref_columns[0]}")
A key is inferred only where the evidence is strong. See how keys are found, and how to declare one that was missed.
Read the registry¶
shop.yaml is meant to be read, and committed beside the pipeline that
writes these files. Load it to look at a column:
registry = Registry.from_yaml("shop.yaml")
print(registry["Orders"]["status"])
print(registry["Orders"]["ref"])
ColSpec(Enum, choices={'NEW': 0.085, 'PAID': 0.27, 'RETURNED': 0.055, 'SHIPPED': 0.59})
ColSpec(String, string_length=(10, 10), pattern='^ORD-[0-9]{6}$')
Edit what is too exact¶
A profile says what this data held, not what the data may hold. The
command line rounds bounds outward to two significant figures — so
Customers.id, from 1 to 150, is written [0, 150] — where
polspec.profile_all in Python writes them exactly; the command describes
data for a person to read, and the function imitates it. A few claims
still need a person to decide. Here, an order line may hold up to
20 of an item, and a price has no upper limit:
lines = registry["Lines"]
lines = lines.with_columns({
"qty": lines["qty"].replace(bounds=(1, 20)),
"price": lines["price"].replace(bounds=(0.0, None)),
})
registry = registry.replace(lines)
registry.to_yaml("shop.yaml")
Editing the YAML by hand does the same. -o shop.py writes the registry as
Python instead: a FrameSpec class per file, to edit and import like any
other.
Validate the files¶
Validate each file against its spec, and the keys between them:
It exits 1 when a file breaks its spec, so a pipeline can run it as a gate
on its own output. In Python, registry.validate_all(shop) does the same.
Watch for drift¶
Validation asks whether the data breaks a claim. Drift asks whether it has moved from what the spec recorded, while still passing. Say next month's orders come back with a quarter of them returned:
Path("next").mkdir()
for name in ("customers", "lines"):
shop[name.title()].write_csv(f"next/{name}.csv")
returned = pl.when(pl.col("id") % 4 == 0).then(pl.lit("RETURNED")).otherwise("status")
shop["Orders"].with_columns(status=returned).write_csv("next/orders.csv")
The first command reports that the frequencies of status moved, with
RETURNED at 29% where the spec recorded 5.5%, and exits 0: the change is
compatible, since every value is still one the spec allows. A breaking
change, which validation would reject, fails the run on its own.
--fail-on any fails on any change, for a CI job that should stop and ask.
--markdown writes the report as a pull-request comment.
Make a tenth for your tests¶
Tests need data shaped like the real thing, without the real rows. Make a fake set a tenth of the size, with every key held by a fake parent row:
polspec synthesize shop/customers.csv shop/orders.csv shop/lines.csv -o fake --scale 0.1 --seed 1
polspec validate --all shop.yaml fake
fake/ holds 15 customers, 60 orders and 150 lines, and validates against
the same registry as the real files. No source row is in it, and no email,
ref or sku either: those are drawn from their format or pattern. Values
of a column that became an Enum, such as country, are carried. Name a
column in --replace to keep its values out.
In Python, synthesize_all returns the frames:
fake = polspec.synthesize_all(shop, scale=0.1, seed=1)
print({name: frame.height for name, frame in fake.items()})
registry.validate_all(fake)
Commit fake/ as test fixtures, or generate frames from shop.yaml in the
tests themselves, as Testing pipelines shows.
Where next¶
- Infer a spec from data: every profiling option, and data read as text.
- Fake data from real data: how close the fake data is, and related tables.
- Schema and data drift: what drift measures, and its thresholds.
- Multiple specs: a registry, and the keys between them.