Skip to content

Constraints

Beyond the shape of a single value, a spec can assert relationships between values, rows and tables. Most are kept by generation as well as checked by validation; a few are only checked:

Generated Validated
ColRule yes yes
ForeignKey yes, when given parent data yes
unique=True, __unique_together__ yes yes
__order_by__ yes yes
__hierarchy__ — see Hierarchies yes yes
__rows__ no: generate() makes the rows asked for yes
ColSpec.validators, __checks__ with checks=True yes

Validators and checks wrap arbitrary expressions, and nothing can draw data that satisfies an arbitrary predicate. So generation leaves them to validation unless asked: generate(..., checks=True) keeps the rows that pass them, and reads on for more — see Keeping checks.

Writing conditions — col()

Rules, validators and checks all take a condition. Write it with col(), which builds a small predicate tree rather than a Polars expression:

import polars as pl
from polspec import ColSpec, FrameSpec, col

col("total") >= col("subtotal")
col("email").str.contains("@")
col("status").is_in(["NEW", "PAID"]) & (col("qty") > 0)
col("shipped").is_null() | (col("shipped") >= col("placed"))

Supported: comparisons (== != < <= > >=), arithmetic (+ - * /), &, |, ~, is_in, is_null, is_not_null, is_between, and the string operations str.contains (a literal substring), str.starts_with, str.ends_with, str.matches (a regular expression) and str.len_chars. Scalars, dates and datetimes are fine as operands.

A predicate evaluates exactly as the Polars expression it stands for, and unlike one it can be written to a spec file and read back, so rules, checks and validators written this way survive to_yaml and to_python. A raw pl.Expr is still accepted everywhere a predicate is, for anything the predicate language cannot say; it just cannot be persisted.

Comparison operators build predicates, as they do on pl.Expr, so a predicate has no truth value. Compare two structurally with Pred.equals.

Conditional values — ColRule

A rule overwrites a column on the rows where its condition matches.

from polspec import ColRule, col

class Shipments(FrameSpec):
    region = ColSpec(pl.Enum(["UK", "US", "EU"]))
    carrier = ColSpec(
        pl.Enum(["RoyalMail", "UPS", "DHL"]),
        rules=[
            ColRule(when=col("region") == "UK", choices=["RoyalMail"]),
            ColRule(when=col("region").is_in(["US", "EU"]), choices=["UPS", "DHL"]),
        ],
    )

Multiple rules on one column are tried in declaration order, first match wins, like a SQL CASE. choices may be weighted exactly as on a ColSpec:

ColRule(when=col("region") == "US", choices={"UPS": 3.0, "DHL": 1.0})

when is a predicate built with col(), not an arbitrary Polars expression, so that every rule can round-trip through a spec file. Conditions compose:

ColRule(
    when=col("region").is_in(["US", "EU"]) & (col("weight_kg") > 10),
    choices=["DHL"],
)

Rules see the frame as it stands

Every when is evaluated against the values the column actually holds when the rule runs, and the passes run in dependency order: a rule keyed on a column that another rule or a foreign key rewrites reads the rewritten values — the same ones validate() will check the rule against. So rules chain:

class Orders(FrameSpec):
    region  = ColSpec(pl.Enum(["UK", "US"]))
    carrier = ColSpec(
        pl.Enum(["RoyalMail", "UPS"]),
        rules=[ColRule(when=col("region") == "UK", choices=["RoyalMail"])],
    )
    tracked = ColSpec(  # keyed on a column that carries rules of its own
        pl.Enum(["yes", "no"]),
        rules=[ColRule(when=col("carrier") == "RoyalMail", choices=["yes"])],
    )

Two columns whose rules each read what the other writes have no such order, and are refused at declaration with SpecError.

A rule also overwrites nulls on matching rows, so a nullable column with a rule ends up with fewer nulls than null_probability suggests.

Single-column predicates — validators

A validator is a Polars expression that each row must satisfy, referencing only its own column:

class Accounts(FrameSpec):
    email = ColSpec(pl.String, validators=[col("email").str.contains("@")])

A raw pl.Expr works too, for anything col() cannot say, but it cannot be written to a spec file.

Wrap one in a Check to name it, describe it, or change null handling:

from polspec import Check

ColSpec(
    pl.Float64,
    validators=[
        Check(
            col("score") <= 100,
            name="score_ceiling",
            description="Scores are a percentage",
        )
    ],
)

Referencing another column is rejected at declaration time — use __checks__ for that.

Derived columns — derive

A column computed from others is declared by its expression, written with col(). Generation computes it, and validation holds the column to it:

class Lines(FrameSpec):
    qty      = ColSpec(pl.Int64, bounds=(1, 20))
    price    = ColSpec(pl.Float64, bounds=(0.5, 100))
    discount = ColSpec(pl.Float64, bounds=(0, 0.3), nullable=True)
    total    = ColSpec(pl.Float64, derive=col("qty") * col("price"))
    net      = ColSpec(pl.Float64, derive=col("total") * (1 - col("discount")), nullable=True)
    large    = ColSpec(pl.Boolean, derive=col("total") > 1_000)

lines = Lines.generate(1_000, seed=1)
Lines.validate(lines)

Added in 1.4.

  • Generation computes a derived column after every column it reads: rules, foreign keys and fixed= pins included, and derived columns too, as net reads total. It casts the result to the column's dtype, and refuses a value the dtype cannot hold.
  • Validation checks each value against the expression, as a derived finding. Integers, decimals, text and flags must be equal; a float may be off by a relative 1e-9, so a pipeline that multiplies in another order, or in SQL, still passes. A null is right exactly where the expression is null, and NaN where it is NaN. derived=False switches the check off.
  • The expression decides the values, so a derived column declares no bounds, choices, rules, distribution, format, pattern, string_length, unique or default. It is nullable when an input is, as net is, since its value is null where theirs is.
  • Nothing else writes it: fixed= cannot pin it, and it cannot be a foreign key's column, part of a composite key, or a hierarchy column. Two derived columns that read each other are refused.

A rule a generator can follow is a ColRule or a derived column; any other is a check.

Multi-column invariants — __checks__

class Invoices(FrameSpec):
    subtotal = ColSpec(pl.Float64, bounds=(0.0, 1000.0))
    total    = ColSpec(pl.Float64, bounds=(0.0, 2000.0))

    __checks__ = [
        Check(col("total") >= col("subtotal"), name="total_covers_subtotal"),
    ]

By default a row whose check evaluates to null passes, matching SQL CHECK semantics. Check(..., ignore_nulls=False) treats null as a failure.

Checks are inherited: a subclass collects its bases' checks as well as its own, de-duplicated. Two different checks sharing a name is an error, since the name is what an error message points at.

A check as a function — @check

A check can be written as a function in the class body instead, named after the function. Added in 1.7.

from polspec import check


class Billed(FrameSpec):
    subtotal = ColSpec(pl.Float64, bounds=(0.0, 1000.0))
    total    = ColSpec(pl.Float64, bounds=(0.0, 2000.0))

    @check(description="tax never makes a total smaller")
    def total_covers_subtotal():
        return col("total") >= col("subtotal")


[c.name for c in Billed.spec.checks]   # ['total_covers_subtotal']

The function takes no arguments and returns the condition: a col() predicate, which a spec file can hold, or a Polars expression. It is called once, when the class is built, and its check joins __checks__ after any listed there. @check takes name=, description= and ignore_nulls= as Check does; bare, the name is the function's. The function does not stay on the class.

Composite uniqueness — __unique_together__

A composite key declares that a combination of columns is distinct, even where each column on its own repeats.

class Assignments(FrameSpec):
    employee_id = ColSpec(pl.Int64, bounds=(1, 500))
    project_id  = ColSpec(pl.Int64, bounds=(1, 200))

    __unique_together__ = [["employee_id", "project_id"]]

generate() satisfies it by resampling the rows that repeat a combination an earlier row already used. Only the repeats move, so on a roomy domain almost every row keeps the value it was generated with, along with whatever weights or bounds shaped it. Rows where any member is null are exempt, matching how the key is validated.

A group whose columns cannot take enough distinct combinations between them is refused, naming the group:

class TooTight(FrameSpec):
    a = ColSpec(pl.Enum(["x", "y"]))
    b = ColSpec(pl.Enum(["p", "q"]))
    __unique_together__ = [["a", "b"]]

TooTight.generate(300, seed=1)
# GenerationError: Composite unique key ['a', 'b'] cannot be satisfied: the
# columns take 4 distinct combination(s) between them and 300 row(s) need one.
# Widen one of the columns, or generate fewer rows.

A member column may not also carry rules: a rule assigns from a fixed set, which is how two rows come to share a combination, and the repair would overwrite what the rule put there. Declare one or the other.

A foreign-keyed member is never resampled — that would break the key — so the repair works with the other members. If every member is foreign-keyed there is nothing it can move, and generation says so rather than returning data that fails its own validation.

Row order — __order_by__

An event log, a time series, a table a window function or join_asof reads — each comes in an order. __order_by__ declares it, as DataFrame.sort(by) takes it: the first column is the major key.

import datetime as dt


class Events(FrameSpec):
    __order_by__ = ["user_id", "ts"]   # each user's events, in time order
    user_id = ColSpec(pl.Int64, bounds=(1, 500))
    ts      = ColSpec(pl.Datetime("us"),
                      bounds=(dt.datetime(2024, 1, 1), dt.datetime(2024, 7, 1)))
    kind    = ColSpec(pl.Enum(["view", "click", "buy"]))

events = Events.generate(10_000, seed=1)
Events.validate(events)

One column, ["ts"], is one order across the frame; listing a key first, as here, is an order within each key. __descending__ turns a column high to low — one bool for every column, or one per column:

class Latest(FrameSpec):
    __order_by__ = ["user_id", "ts"]
    __descending__ = [False, True]     # each user's newest first
    user_id = ColSpec(pl.Int64, bounds=(1, 500))
    ts      = ColSpec(pl.Datetime("us"))

Nulls come last, either way.

generate() sorts whole rows, after everything else: each value keeps the row it was generated in, so every rule, check, composite key, foreign key and hierarchy the rows satisfy, they still satisfy. The sort is stable, so the same seed gives the same frame. validate() reports each row that belongs before the one above it, under the finding code order; skip it with order=False, or --skip order on the command line.

An order is a property of the whole frame, so the streaming verbs refuse an ordered spec, as they refuse a hierarchy — write generate()'s frame out instead:

Events.scan(10_000)
# SpecError: Events declares order_by ['user_id', 'ts'], which scan cannot
# produce: the rows' order is a property of the whole frame, and each batch
# is generated on its own. Use generate() for an ordered frame, and write it out
# yourself.

profile() declares an order it finds: the first column of single values, with at least 20 distinct, whose rows come sorted one way or the other. An order within each key is yours to declare.

A sequence: 1, 2, …, n

An auto-increment ID needs nothing new. A unique column holds each value of its bounds once, so a unique column bounded (1, n) in a frame of n rows holds every value from 1 to n. Ordered by it, that's the sequence:

class Orders(FrameSpec):
    __order_by__ = ["order_id"]
    order_id = ColSpec(pl.Int64, unique=True, bounds=(1, 1_000))
    total    = ColSpec(pl.Float64, bounds=(0, 500))

orders = Orders.generate(1_000, seed=7)
assert orders["order_id"].to_list() == list(range(1, 1_001))

Fewer rows than the bounds hold give distinct IDs in order, with gaps. More rows are refused, since the bounds don't hold enough values. Leave out __order_by__ for the same IDs in a shuffled order.

Row count — __rows__

An empty file passes every claim about its rows, as there are none to break. __rows__ says how many a table may hold, either end open.

Added in 1.1.

class Orders(FrameSpec):
    __rows__ = (1, None)   # must not be empty

    order_id = ColSpec(pl.Int64, unique=True)

Outside it, validate() reports a row_count finding, on a LazyFrame too. It judges the whole frame, so passing_rows() refuses a report that holds one, and refuses too when fewer rows pass than the table needs. rows=False and --skip rows turn it off. A subclass inherits it; __rows__ = None clears it. On a TableSpec, it is rows=, and with_rows() makes a copy.

generate() makes the rows it is asked for, whatever __rows__ says. The count describes the real table, and a small generated frame is usually what you want, so validate one with rows=False. profile() declares no count, as the size of one file is not a rule about the next.

Referential integrity — ForeignKey

from polspec import ForeignKey

class Customers(FrameSpec):
    id = ColSpec(pl.Int64, bounds=(1, 10_000))

class Orders(FrameSpec):
    customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))

    __foreign_keys__ = [
        ForeignKey("customer_id", references=Customers, ref_columns="id"),
    ]

Rows where any key column is null are exempt — a null foreign key means "no reference", not an invalid one.

Generating consistent data

Pass the parent frame and the child's key values are sampled from it, so the result is referentially consistent by construction:

customers = Customers.generate(1_000, seed=1)
orders    = Orders.generate(10_000, seed=2, references={Customers: customers})

Orders.validate(orders, references={Customers: customers})

Without references, generation leaves the column freely generated — but validate() then raises, because it has nothing to check against. Supply the parent to both calls, or disable the check with validate(..., foreign_keys=False).

Composite keys are sampled as one joint pick per row, so multi-column keys stay internally consistent.

With several related specs, a Registry does the walk: Registry(Customers, Orders, OrderLines).generate_all(1_000, seed=1) generates parents first and threads each into its children, and validate_all checks the whole set.

Self-references

class Employees(FrameSpec):
    id         = ColSpec(pl.Int64, bounds=(1, 500))
    manager_id = ColSpec(pl.Int64, bounds=(1, 500), nullable=True)

    __foreign_keys__ = [
        ForeignKey("manager_id", references="self", ref_columns="id"),
    ]

"self" resolves to whichever spec the key ends up declared on, and needs no references entry in either call.

The parent's domain has to fit inside the column's own

A foreign key overwrites its column with values from the parent, so the parent's bounds or choices have to be ones the column itself declares it can hold. Declaring bounds=(1, 50) on a column referencing keys in 100..200 would produce data that fails its own validation, so it is refused when you declare it:

class Orders(FrameSpec):
    customer_id = ColSpec(pl.Int64, bounds=(1, 50))
    __foreign_keys__ = [
        ForeignKey("customer_id", references=Customers, ref_columns="id")
    ]
# SpecError: ... column 'customer_id' is declared bounds [1, 50], but the
# key fills it with values from 'id' on 'Customers', where bounds
# [1, 10000] do not fit inside [1, 50].

A column that declares no bounds or choices accepts anything, so the check only fires on a genuine contradiction. Widen or drop the child's declaration, or narrow the parent's.

The check needs both specs, so a key naming its target as a string is checked when a Registry resolves it, not before.