Constraints¶
Beyond the shape of a single value, a spec can assert relationships between values, rows and tables. Most are kept by generation as well as checked by validation; a few are only checked:
| Generated | Validated | |
|---|---|---|
ColRule |
yes | yes |
ForeignKey |
yes, when given parent data | yes |
unique=True, __unique_together__ |
yes | yes |
__order_by__ |
yes | yes |
__hierarchy__ — see Hierarchies |
yes | yes |
__rows__ |
no: generate() makes the rows asked for |
yes |
ColSpec.validators, __checks__ |
with checks=True |
yes |
Validators and checks wrap arbitrary expressions, and nothing can draw
data that satisfies an arbitrary predicate. So generation leaves them to
validation unless asked: generate(..., checks=True) keeps the rows
that pass them, and reads on for more — see
Keeping checks.
Writing conditions — col()¶
Rules, validators and checks all take a condition. Write it with col(),
which builds a small predicate tree rather than a Polars expression:
import polars as pl
from polspec import ColSpec, FrameSpec, col
col("total") >= col("subtotal")
col("email").str.contains("@")
col("status").is_in(["NEW", "PAID"]) & (col("qty") > 0)
col("shipped").is_null() | (col("shipped") >= col("placed"))
Supported: comparisons (== != < <= > >=), arithmetic (+ - * /),
&, |, ~, is_in, is_null, is_not_null, is_between, and the
string operations str.contains (a literal substring), str.starts_with,
str.ends_with, str.matches (a regular expression) and str.len_chars.
Scalars, dates and datetimes are fine as operands.
A predicate evaluates exactly as the Polars expression it stands for, and
unlike one it can be written to a spec file and read back, so rules, checks
and validators written this way survive to_yaml and to_python. A raw
pl.Expr is still accepted everywhere a predicate is, for anything the
predicate language cannot say; it just cannot be persisted.
Comparison operators build predicates, as they do on pl.Expr, so a
predicate has no truth value. Compare two structurally with Pred.equals.
Conditional values — ColRule¶
A rule overwrites a column on the rows where its condition matches.
from polspec import ColRule, col
class Shipments(FrameSpec):
region = ColSpec(pl.Enum(["UK", "US", "EU"]))
carrier = ColSpec(
pl.Enum(["RoyalMail", "UPS", "DHL"]),
rules=[
ColRule(when=col("region") == "UK", choices=["RoyalMail"]),
ColRule(when=col("region").is_in(["US", "EU"]), choices=["UPS", "DHL"]),
],
)
Multiple rules on one column are tried in declaration order, first match wins,
like a SQL CASE. choices may be weighted exactly as on a ColSpec:
when is a predicate built with col(), not an
arbitrary Polars expression, so that every rule can round-trip through a spec
file. Conditions compose:
Rules see the frame as it stands
Every when is evaluated against the values the column actually holds
when the rule runs, and the passes run in dependency order: a rule keyed
on a column that another rule or a foreign key rewrites reads the
rewritten values — the same ones validate() will check the rule
against. So rules chain:
class Orders(FrameSpec):
region = ColSpec(pl.Enum(["UK", "US"]))
carrier = ColSpec(
pl.Enum(["RoyalMail", "UPS"]),
rules=[ColRule(when=col("region") == "UK", choices=["RoyalMail"])],
)
tracked = ColSpec( # keyed on a column that carries rules of its own
pl.Enum(["yes", "no"]),
rules=[ColRule(when=col("carrier") == "RoyalMail", choices=["yes"])],
)
Two columns whose rules each read what the other writes have no such
order, and are refused at declaration with SpecError.
A rule also overwrites nulls on matching rows, so a nullable column with a rule
ends up with fewer nulls than null_probability suggests.
Single-column predicates — validators¶
A validator is a Polars expression that each row must satisfy, referencing only its own column:
A raw pl.Expr works too, for anything col() cannot say, but it cannot be
written to a spec file.
Wrap one in a Check to name it, describe it, or change null handling:
from polspec import Check
ColSpec(
pl.Float64,
validators=[
Check(
col("score") <= 100,
name="score_ceiling",
description="Scores are a percentage",
)
],
)
Referencing another column is rejected at declaration time — use __checks__
for that.
Derived columns — derive¶
A column computed from others is declared by its expression, written with
col(). Generation computes it, and validation
holds the column to it:
class Lines(FrameSpec):
qty = ColSpec(pl.Int64, bounds=(1, 20))
price = ColSpec(pl.Float64, bounds=(0.5, 100))
discount = ColSpec(pl.Float64, bounds=(0, 0.3), nullable=True)
total = ColSpec(pl.Float64, derive=col("qty") * col("price"))
net = ColSpec(pl.Float64, derive=col("total") * (1 - col("discount")), nullable=True)
large = ColSpec(pl.Boolean, derive=col("total") > 1_000)
lines = Lines.generate(1_000, seed=1)
Lines.validate(lines)
Added in 1.4.
- Generation computes a derived column after every column it reads:
rules, foreign keys and
fixed=pins included, and derived columns too, asnetreadstotal. It casts the result to the column's dtype, and refuses a value the dtype cannot hold. - Validation checks each value against the expression, as a
derivedfinding. Integers, decimals, text and flags must be equal; a float may be off by a relative 1e-9, so a pipeline that multiplies in another order, or in SQL, still passes. A null is right exactly where the expression is null, and NaN where it is NaN.derived=Falseswitches the check off. - The expression decides the values, so a derived column declares no
bounds,choices,rules,distribution,format,pattern,string_length,uniqueordefault. It isnullablewhen an input is, asnetis, since its value is null where theirs is. - Nothing else writes it:
fixed=cannot pin it, and it cannot be a foreign key's column, part of a composite key, or a hierarchy column. Two derived columns that read each other are refused.
A rule a generator can follow is a ColRule or a derived column; any
other is a check.
Multi-column invariants — __checks__¶
class Invoices(FrameSpec):
subtotal = ColSpec(pl.Float64, bounds=(0.0, 1000.0))
total = ColSpec(pl.Float64, bounds=(0.0, 2000.0))
__checks__ = [
Check(col("total") >= col("subtotal"), name="total_covers_subtotal"),
]
By default a row whose check evaluates to null passes, matching SQL CHECK
semantics. Check(..., ignore_nulls=False) treats null as a failure.
Checks are inherited: a subclass collects its bases' checks as well as its own, de-duplicated. Two different checks sharing a name is an error, since the name is what an error message points at.
A check as a function — @check¶
A check can be written as a function in the class body instead, named after the function. Added in 1.7.
from polspec import check
class Billed(FrameSpec):
subtotal = ColSpec(pl.Float64, bounds=(0.0, 1000.0))
total = ColSpec(pl.Float64, bounds=(0.0, 2000.0))
@check(description="tax never makes a total smaller")
def total_covers_subtotal():
return col("total") >= col("subtotal")
[c.name for c in Billed.spec.checks] # ['total_covers_subtotal']
The function takes no arguments and returns the condition: a col()
predicate, which a spec file can hold, or a Polars expression. It is
called once, when the class is built, and its check joins
__checks__ after any listed there. @check takes name=,
description= and ignore_nulls= as Check does; bare, the name is
the function's. The function does not stay on the class.
Composite uniqueness — __unique_together__¶
A composite key declares that a combination of columns is distinct, even where each column on its own repeats.
class Assignments(FrameSpec):
employee_id = ColSpec(pl.Int64, bounds=(1, 500))
project_id = ColSpec(pl.Int64, bounds=(1, 200))
__unique_together__ = [["employee_id", "project_id"]]
generate() satisfies it by resampling the rows that repeat a combination an
earlier row already used. Only the repeats move, so on a roomy domain almost
every row keeps the value it was generated with, along with whatever weights
or bounds shaped it. Rows where any member is null are exempt, matching how
the key is validated.
A group whose columns cannot take enough distinct combinations between them is refused, naming the group:
class TooTight(FrameSpec):
a = ColSpec(pl.Enum(["x", "y"]))
b = ColSpec(pl.Enum(["p", "q"]))
__unique_together__ = [["a", "b"]]
TooTight.generate(300, seed=1)
# GenerationError: Composite unique key ['a', 'b'] cannot be satisfied: the
# columns take 4 distinct combination(s) between them and 300 row(s) need one.
# Widen one of the columns, or generate fewer rows.
A member column may not also carry rules: a rule assigns from a fixed set,
which is how two rows come to share a combination, and the repair would
overwrite what the rule put there. Declare one or the other.
A foreign-keyed member is never resampled — that would break the key — so the repair works with the other members. If every member is foreign-keyed there is nothing it can move, and generation says so rather than returning data that fails its own validation.
Row order — __order_by__¶
An event log, a time series, a table a window function or join_asof reads
— each comes in an order. __order_by__ declares it, as
DataFrame.sort(by) takes it: the first column is the major key.
import datetime as dt
class Events(FrameSpec):
__order_by__ = ["user_id", "ts"] # each user's events, in time order
user_id = ColSpec(pl.Int64, bounds=(1, 500))
ts = ColSpec(pl.Datetime("us"),
bounds=(dt.datetime(2024, 1, 1), dt.datetime(2024, 7, 1)))
kind = ColSpec(pl.Enum(["view", "click", "buy"]))
events = Events.generate(10_000, seed=1)
Events.validate(events)
One column, ["ts"], is one order across the frame; listing a key first, as
here, is an order within each key. __descending__ turns a column high to
low — one bool for every column, or one per column:
class Latest(FrameSpec):
__order_by__ = ["user_id", "ts"]
__descending__ = [False, True] # each user's newest first
user_id = ColSpec(pl.Int64, bounds=(1, 500))
ts = ColSpec(pl.Datetime("us"))
Nulls come last, either way.
generate() sorts whole rows, after everything else: each value keeps the
row it was generated in, so every rule, check, composite key, foreign key
and hierarchy the rows satisfy, they still satisfy. The sort is stable, so
the same seed gives the same frame. validate() reports each row that
belongs before the one above it, under the finding code order; skip it
with order=False, or --skip order on the command line.
An order is a property of the whole frame, so the streaming verbs refuse an
ordered spec, as they refuse a hierarchy — write generate()'s frame out
instead:
Events.scan(10_000)
# SpecError: Events declares order_by ['user_id', 'ts'], which scan cannot
# produce: the rows' order is a property of the whole frame, and each batch
# is generated on its own. Use generate() for an ordered frame, and write it out
# yourself.
profile() declares an order it finds: the first column of single values,
with at least 20 distinct, whose rows come sorted one way or the other. An
order within each key is yours to declare.
A sequence: 1, 2, …, n¶
An auto-increment ID needs nothing new. A unique column holds each value of
its bounds once, so a unique column bounded (1, n) in a frame of n rows
holds every value from 1 to n. Ordered by it, that's the sequence:
class Orders(FrameSpec):
__order_by__ = ["order_id"]
order_id = ColSpec(pl.Int64, unique=True, bounds=(1, 1_000))
total = ColSpec(pl.Float64, bounds=(0, 500))
orders = Orders.generate(1_000, seed=7)
assert orders["order_id"].to_list() == list(range(1, 1_001))
Fewer rows than the bounds hold give distinct IDs in order, with gaps. More
rows are refused, since the bounds don't hold enough values. Leave out
__order_by__ for the same IDs in a shuffled order.
Row count — __rows__¶
An empty file passes every claim about its rows, as there are none to break.
__rows__ says how many a table may hold, either end open.
Added in 1.1.
class Orders(FrameSpec):
__rows__ = (1, None) # must not be empty
order_id = ColSpec(pl.Int64, unique=True)
Outside it, validate() reports a row_count finding, on a LazyFrame too.
It judges the whole frame, so passing_rows() refuses a report that holds one,
and refuses too when fewer rows pass than the table needs. rows=False and
--skip rows turn it off. A subclass inherits it; __rows__ = None clears it.
On a TableSpec, it is rows=, and with_rows() makes a copy.
generate() makes the rows it is asked for, whatever __rows__ says. The count
describes the real table, and a small generated frame is usually what you want,
so validate one with rows=False. profile() declares no count, as the size of
one file is not a rule about the next.
Referential integrity — ForeignKey¶
from polspec import ForeignKey
class Customers(FrameSpec):
id = ColSpec(pl.Int64, bounds=(1, 10_000))
class Orders(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
__foreign_keys__ = [
ForeignKey("customer_id", references=Customers, ref_columns="id"),
]
Rows where any key column is null are exempt — a null foreign key means "no reference", not an invalid one.
Generating consistent data¶
Pass the parent frame and the child's key values are sampled from it, so the result is referentially consistent by construction:
customers = Customers.generate(1_000, seed=1)
orders = Orders.generate(10_000, seed=2, references={Customers: customers})
Orders.validate(orders, references={Customers: customers})
Without references, generation leaves the column freely generated — but
validate() then raises, because it has nothing to check against. Supply the
parent to both calls, or disable the check with
validate(..., foreign_keys=False).
Composite keys are sampled as one joint pick per row, so multi-column keys stay internally consistent.
With several related specs, a Registry does the walk:
Registry(Customers, Orders, OrderLines).generate_all(1_000, seed=1) generates
parents first and threads each into its children, and validate_all checks
the whole set.
Self-references¶
class Employees(FrameSpec):
id = ColSpec(pl.Int64, bounds=(1, 500))
manager_id = ColSpec(pl.Int64, bounds=(1, 500), nullable=True)
__foreign_keys__ = [
ForeignKey("manager_id", references="self", ref_columns="id"),
]
"self" resolves to whichever spec the key ends up declared on, and needs no
references entry in either call.
The parent's domain has to fit inside the column's own
A foreign key overwrites its column with values from the parent, so the
parent's bounds or choices have to be ones the column itself declares
it can hold. Declaring bounds=(1, 50) on a column referencing keys in
100..200 would produce data that fails its own validation, so it is
refused when you declare it:
class Orders(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 50))
__foreign_keys__ = [
ForeignKey("customer_id", references=Customers, ref_columns="id")
]
# SpecError: ... column 'customer_id' is declared bounds [1, 50], but the
# key fills it with values from 'id' on 'Customers', where bounds
# [1, 10000] do not fit inside [1, 50].
A column that declares no bounds or choices accepts anything, so the
check only fires on a genuine contradiction. Widen or drop the child's
declaration, or narrow the parent's.
The check needs both specs, so a key naming its target as a string is
checked when a Registry resolves it, not before.