# polspec documentation
> Declare a Polars schema once, then generate data from it and validate data against it.
Every page of https://maxwellb13.github.io/polspec/, in the order the documentation presents them.
---
# Home
Source: https://maxwellb13.github.io/polspec/
# polspec
Declare a Polars schema once. Generate data that matches it, and validate data
against it — from the same declaration.
```python
import polars as pl
from polspec import ColSpec, FrameSpec
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, None))
status = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]))
total = ColSpec(pl.Float64, bounds=(0.0, None))
placed = ColSpec(pl.Date, nullable=True)
df = Orders.generate(1_000_000, seed=42) # a million rows in well under a second
Orders.validate(df) # raises ValidationError on any breach
```
The generator is written in Rust and runs the columns in parallel, so a spec
that describes a realistic table produces millions of rows in the time it takes
to describe one.
## Why two directions from one declaration
Most schema tools do one or the other. A validation library tells you when
production data drifted; a fixture library gives you something to test against.
Keeping both behind one declaration means the fixtures and the contract cannot
disagree — and where they might, polspec has a test suite whose whole job is to
catch it (see [Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/)).
That is the practical payoff: the data in your tests is data your validator
already accepts, so a test that passes locally is not passing on a shape
production will reject.
## What you can declare
- **Types and shape**
Every dtype, generated and validated — integers, floats, `Decimal`,
booleans, strings, binary, all four temporal types, `Enum`,
`Categorical`, and a `List`, `Array` or `Struct` of any of them, nested
to any depth — plus nullability, bounds, string lengths, value domains
and named string formats such as `uuid4` and `email`.
[Declaring columns](https://maxwellb13.github.io/polspec/how-to/columns/) ·
[String formats](https://maxwellb13.github.io/polspec/how-to/formats/)
- **Rules and invariants**
Conditional values, single-column validators, multi-column checks,
composite uniqueness and foreign keys between specs.
[Constraints](https://maxwellb13.github.io/polspec/how-to/constraints/)
- **Data on demand**
Random or coverage-guaranteeing generation, reproducible seeds, and a
`LazyFrame` that generates only the columns and rows a plan asks for —
streaming straight to Parquet, CSV, Arrow IPC or NDJSON.
[Generating data](https://maxwellb13.github.io/polspec/how-to/generating/)
- **Specs from elsewhere**
Infer a spec by profiling an existing DataFrame, or load one from YAML so
non-Python tooling can read it too.
[YAML specs](https://maxwellb13.github.io/polspec/how-to/files/)
- **What changed**
Diff two versions of a spec, or a spec against data, into a report that
says what moved and whether a frame that passed before could fail now.
[Drift](https://maxwellb13.github.io/polspec/how-to/drift/)
- **From the command line**
`polspec generate` writes a data file from a spec, `polspec validate`
checks one, `polspec diff` and `polspec drift` gate a pull request on
what changed, and `polspec test` turns a schema into a pytest round-trip.
[Command line](https://maxwellb13.github.io/polspec/how-to/cli/)
## Install
```bash
uv add polspec # preferred
pip install polspec # alternative
```
The generator is a compiled Rust extension, but wheels are published for Linux
(x86_64, aarch64), macOS (Intel and Apple silicon) and Windows (x86_64), so
installing needs no Rust toolchain.
Nothing beyond Polars is needed at runtime.
Building from a checkout — which does need Rust and
[maturin](https://www.maturin.rs) — is covered in
[CONTRIBUTING.md](https://github.com/MaxwellB13/polspec/blob/main/CONTRIBUTING.md).
## Where to go next
Start with [Getting started](https://maxwellb13.github.io/polspec/tutorial/getting-started/) for the full loop — declare,
generate, validate — in about five minutes. If you're weighing polspec
against a hand-rolled fixture, Faker, or a data-quality framework, see
[Comparison to other approaches](https://maxwellb13.github.io/polspec/explanation/comparison/) for where each one fits and
the benchmark numbers behind the speed claim.
## For language models
The documentation is published in the [llms.txt](https://llmstxt.org) format:
[`/llms.txt`](https://maxwellb13.github.io/polspec/llms.txt) indexes every
page, and [`/llms-full.txt`](https://maxwellb13.github.io/polspec/llms-full.txt)
carries the full text of all of them -- including the API reference, expanded
to signatures and docstrings -- in one file.
---
# Getting started
Source: https://maxwellb13.github.io/polspec/tutorial/getting-started/
# Getting started
## Declare a spec
A spec is a class. Subclass `FrameSpec` and assign a `ColSpec` per column, in
the order the columns should appear.
```python
from datetime import date
import polars as pl
from polspec import ColSpec, FrameSpec
class Customers(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 100_000))
name = ColSpec(pl.String, string_length=(4, 20))
tier = ColSpec(pl.Enum(["free", "pro", "enterprise"]))
signed_up = ColSpec(pl.Date, bounds=(date(2020, 1, 1), None))
churned = ColSpec(pl.Boolean, nullable=True, null_probability=0.3)
```
Nothing runs at declaration time except validation of the declaration itself.
A contradictory spec fails here, at the line that caused it, rather than
thousands of rows later:
```python
ColSpec(pl.Int8, bounds=(0, 1_000))
# ValueError: ColSpec.bounds max (1000) is outside the range Int8 can represent [-128, 127]
```
## Generate data
```python
df = Customers.generate(10_000, seed=42)
```
`seed` makes the result reproducible across processes and machines. Omit it and
each call differs.
```python
Customers.generate(500, seed=7).equals(Customers.generate(500, seed=7)) # True
```
## Validate data
`validate()` checks a DataFrame or LazyFrame against the same declaration and
returns it, so it drops into a pipeline:
```python
clean = Customers.validate(raw_df, cast=True)
```
Every breach is collected before anything is raised, so one call tells you
everything that is wrong rather than only the first thing:
```python
from polspec import ValidationError
broken_df = pl.DataFrame(
{
"customer_id": [100_050, 150_000, 200_000], # all past the upper bound
"name": ["Adam", None, "Alan"], # one null in a non-nullable column
"tier": ["trial", "trial", "pro"], # "trial" is not a tier
"signed_up": [date(2021, 5, 1)] * 3,
"churned": [None, True, False],
}
)
try:
Customers.validate(broken_df)
except ValidationError as err:
for problem in err.errors:
print(problem)
```
```text
Column 'customer_id': found 3 value(s) out of bounds [1, 100000] (min found: 100050, max found: 200000). Out of bounds samples: [100050, 150000, 200000]
Column 'name': non-nullable column contains 1 null value(s)
Column 'tier': found 2 invalid value(s) not in allowed choices/categories ['free', 'pro', 'enterprise']. Invalid samples: ['trial']
```
!!! tip "One pass, not one per column"
Every check across every column is compiled into a single Polars
aggregation and evaluated in one scan. Validating a wide table costs about
the same as validating a narrow one.
## Handle data that nearly fits
Real input rarely arrives in exactly the declared shape. `validate()` takes
policies for the two structural mismatches:
```python
Customers.validate(
df,
extra_cols="drop", # "raise" (default) | "drop" | "allow"
missing_cols="raise", # "raise" (default) | "add" | "allow"
strict_dtypes=False, # allow Int32 where Int64 was declared, String for an Enum
cast=True, # cast surviving columns to the declared dtype
)
```
By default a String column arriving where an `Enum` was declared is accepted —
that is how data comes back from CSV and JSON. `strict_dtypes=True` demands the
exact dtype. For a file someone else hands you, [Checking a file you were
given](https://maxwellb13.github.io/polspec/how-to/validating/#checking-a-file-you-were-given) walks through
the whole flow, dates included.
## Infer a spec instead of writing one
Pointed at an existing DataFrame, polspec writes the spec for you:
```python
existing_df = pl.DataFrame(
{
"customer_id": [1, 2, 3, 4],
"tier": ["free", "free", "pro", None],
}
)
Profiled = FrameSpec.from_dataframe(existing_df, weights=True)
print(Profiled.to_markdown())
```
It infers nullability and observed null rates, narrows low-cardinality strings
to `Enum`, and — with `weights=True` — records how often each category actually
occurred, so regenerated data keeps the observed mix rather than a uniform one.
Treat the result as a first draft to edit, not a finished contract: it
describes the sample it saw, which may be narrower than the rule you actually
mean.
## Next
- [Declaring columns](https://maxwellb13.github.io/polspec/how-to/columns/) — everything a `ColSpec` accepts
- [Constraints](https://maxwellb13.github.io/polspec/how-to/constraints/) — rules, checks, uniqueness, foreign keys
- [Generating data](https://maxwellb13.github.io/polspec/how-to/generating/) — coverage, batching, writing to files
---
# Related tables
Source: https://maxwellb13.github.io/polspec/tutorial/related-tables/
# Related tables
[Getting started](https://maxwellb13.github.io/polspec/tutorial/getting-started/) covers one spec on its own. Real
schemas come in sets, with keys between them, and the interesting question is
how you generate a *consistent* set: orders whose `customer_id` values are
customers that exist.
This tutorial builds three related specs and generates all of them in one
call. The complete version, with more columns and a spec loaded from YAML,
lives in [`examples/related_specs.py`][example] in the repository and runs in
CI, so it cannot go stale.
[example]: https://github.com/MaxwellB13/polspec/blob/main/examples/related_specs.py
## A parent
Nothing new here — a spec like any other. The `unique=True` on `id` matters
for what follows: it is what makes this a table other tables can point at.
```python
import datetime as dt
import polars as pl
from polspec import ColSpec, ForeignKey, FrameSpec, Registry
class Customers(FrameSpec):
id = ColSpec(pl.Int64, bounds=(1, 10_000_000), unique=True)
name = ColSpec(pl.String, string_length=(3, 40))
country = ColSpec(pl.Enum(["UK", "US", "DE"]))
signed_up = ColSpec(pl.Date, bounds=(dt.date(2020, 1, 1), dt.date(2026, 1, 1)))
```
## A child
`__foreign_keys__` declares that `customer_id` only ever holds values that
exist in `Customers.id`.
```python
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 100_000_000), unique=True)
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000_000))
total = ColSpec(pl.Float64, bounds=(0.0, 6_000.0))
__foreign_keys__ = [
ForeignKey("customer_id", references=Customers, ref_columns="id"),
]
```
!!! note "The two `bounds` have to agree"
`customer_id` is declared `(1, 10_000_000)` — the same range as
`Customers.id`. That is not decoration. A key fills its column from the
parent, so the parent's domain has to fit inside the child's; declaring
`bounds=(1, 50)` here would be a contradiction, and polspec refuses it
when you write the class rather than when you run it.
## A composite key
`OrderLines` points at `Orders`, and declares that no order has two lines
with the same number.
```python
class OrderLines(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 100_000_000))
line_no = ColSpec(pl.Int32, bounds=(1, 1_000_000))
quantity = ColSpec(pl.UInt16, bounds=(1, 500))
__unique_together__ = [["order_id", "line_no"]]
__foreign_keys__ = [
ForeignKey("order_id", references=Orders, ref_columns="order_id"),
]
```
## Generating the set
A `Registry` holds the specs that belong together. `resolve()` binds every key
to its target and checks the set is coherent; `order()` is the parents-first
order the keys imply.
```python
registry = Registry(Customers, Orders, OrderLines).resolve()
print(registry.order())
# ('Customers', 'Orders', 'OrderLines')
frames = registry.generate_all(1_000, seed=1)
```
`generate_all` walks that order and threads each parent frame into its
children, so every key is satisfied by construction:
```python
orders = frames["Orders"]
customers = frames["Customers"]
assert set(orders["customer_id"]) <= set(customers["id"])
registry.validate_all(frames) # passes
```
Ask for different row counts per table by passing a mapping:
```python
frames = registry.generate_all(
{"Customers": 1_000, "Orders": 5_000, "OrderLines": 20_000}, seed=1
)
```
Each spec's seed is derived from the registry seed and the spec's *name*, so
adding a fourth table does not reshuffle the three you already had.
## Drawing the result
`to_mermaid()` renders the set as an entity-relationship diagram — one entity
per spec, one line per key:
```python
print(registry.to_mermaid())
```
```mermaid
erDiagram
Customers {
Int64 id PK "bounds: [1, 10000000]"
String name "len: [3, 40]"
Enum country
Date signed_up "bounds: [2020-01-01, 2026-01-01]"
}
Orders {
Int64 order_id PK "bounds: [1, 100000000]"
Int64 customer_id FK "bounds: [1, 10000000]"
Float64 total "bounds: [0.0, 6000.0]"
}
OrderLines {
Int64 order_id UK "bounds: [1, 100000000]"
Int32 line_no UK "bounds: [1, 1000000]"
UInt16 quantity "bounds: [1, 500]"
}
Customers ||--o{ Orders : "fk_customer_id__Customers"
Orders ||--o{ OrderLines : "fk_order_id__Orders"
```
## Where to go next
- [Multiple specs](https://maxwellb13.github.io/polspec/how-to/registry/) — the rest of what `Registry` does:
discovery from a directory, one file for the whole set, shared categories.
- [Constraints](https://maxwellb13.github.io/polspec/how-to/constraints/) — foreign keys in detail, including
self-references and composite keys.
- [Specs as files](https://maxwellb13.github.io/polspec/how-to/files/) — moving a spec out of Python entirely,
which the full worked example does for one of its tables.
---
# Declare columns
Source: https://maxwellb13.github.io/polspec/how-to/columns/
# Declaring columns
A `ColSpec` describes one column. Only `dtype` is required.
```python
ColSpec(
dtype,
col_name=None,
seed_name=None,
nullable=False,
bounds=None,
tags=(),
unique=False,
null_probability=0.1,
string_length=None,
list_length=None,
fields=None,
format=None,
pattern=None,
distribution=None,
distribution_params=None,
choices=None,
weights=None,
rules=(),
validators=(),
)
```
## Types
polspec generates every dtype below. A dtype passed as a class is instantiated
for you, so `pl.Int64` and `pl.Int64()` mean the same thing.
| Family | Types |
|:--|:--|
| Integer | `Int8` `Int16` `Int32` `Int64` `UInt8` `UInt16` `UInt32` `UInt64` |
| Float | `Float32` `Float64` |
| Decimal | `Decimal(precision, scale)` |
| Boolean | `Boolean` |
| Text | `String` |
| Bytes | `Binary` |
| Temporal | `Date` `Time` `Datetime` `Duration` |
| Categorical | `Enum` `Categorical` |
| Nested | `List(inner)` `Array(inner, width)` `Struct({name: dtype})` — nested to any depth; see [Nested columns](#nested-columns) |
That is every dtype: since 0.9.0 there is none polspec declares and cannot
generate.
## Nullability
`nullable=False` (the default) means validation rejects any null. When
`nullable=True`, `null_probability` sets how often generation emits one.
```python
ColSpec(pl.Int64, nullable=True, null_probability=0.25) # about a quarter null
```
`null_probability` is ignored when `nullable=False`, so switching nullability
off does not silently leave a stale rate behind.
Writing a rate of your own *without* `nullable=True` warns, though, because
that is the other way round — it reads as asking for nulls, and the column
generates none:
```python
ColSpec(pl.Int64, null_probability=0.25) # warns: no nulls
ColSpec(pl.Int64, nullable=True, null_probability=0.25) # about a quarter null
ColSpec(pl.Int64) # no nulls, no warning
```
Only a rate that cannot be a leftover warns: the default and an explicit
`0.0` both already agree with `nullable=False`.
## Bounds
`bounds` is an inclusive `[min, max]` for numeric and temporal columns. Pass a
tuple, a list, or a `Bound`:
```python
ColSpec(pl.Int64, bounds=(-100, 100))
ColSpec(pl.Float64, bounds=[0.0, 1.0])
ColSpec(pl.Date, bounds=(date(2020, 1, 1), date(2024, 12, 31)))
```
Temporal bounds accept real `date`, `datetime`, `time` and `timedelta` objects,
or the physical integer the dtype stores.
### Decimal bounds
A `Decimal(precision, scale)` column takes its bounds as an `int`, a
`decimal.Decimal`, or a string read exactly -- the form a spec file writes.
A float is read through its `repr`, so `0.1` is `0.1`. An endpoint with more
decimal places than the scale keeps is refused rather than rounded, and one
the precision cannot hold is refused like any other out-of-range bound:
```python
from decimal import Decimal
ColSpec(pl.Decimal(10, 2), bounds=(0, "99.99"))
ColSpec(pl.Decimal(10, 2), bounds=(Decimal("0.50"), None))
```
```python
ColSpec(pl.Decimal(10, 2), bounds=(0, "1.005"))
# SpecError: ColSpec.bounds max ('1.005') has more decimal places than Decimal(precision=10, scale=2) keeps (scale 2); ...
```
Generation draws a Decimal as the integer it physically is and scales it
back, so the default range with no bounds is the float default
(`±1,000,000`) or the widest the precision allows, whichever is narrower.
The draw is 64-bit: bounds needing more than eighteen significant digits
are refused at `generate()`, and only there -- validation checks the full
precision.
### Open-ended bounds
Either endpoint may be `None`, leaving that side unconstrained:
```python
ColSpec(pl.Int64, bounds=(0, None)) # non-negative
ColSpec(pl.Int64, bounds=(None, 0)) # non-positive
```
!!! warning "An open end means different things to generation and validation"
`validate()` treats it as genuinely unconstrained. `generate()` cannot
sample an unbounded range, so it falls back to the same default it would
use with no bounds at all.
```python
class S(FrameSpec):
n = ColSpec(pl.Int64, bounds=(0, None))
S.generate(1000, seed=1)["n"].max() # ~1_000_000, the Int64 default
S.validate(pl.DataFrame({"n": [10**15]})) # accepted — no upper limit
```
This mirrors how `bounds=None` already behaves rather than adding a third
rule.
For "always positive", note that bounds are *inclusive*: use `(1, None)` for
integers, and either a small floor like `(1e-9, None)` for floats or an
unsigned dtype, which cannot represent a negative at all.
Bounds outside what the dtype can hold are rejected when you declare them:
```python
ColSpec(pl.Float32, bounds=(-1e40, 1e40))
# ValueError: ColSpec.bounds min (-1e+40) is outside the range Float32 can represent
```
## Value domains
`choices` restricts a column to a fixed set:
```python
ColSpec(pl.String, choices=["GBP", "USD", "EUR"])
```
`weights` biases the draw. Supply them positionally, or as a `{choice: weight}`
mapping — never both:
```python
ColSpec(pl.String, choices=["a", "b", "c"], weights=[10.0, 5.0, 1.0])
ColSpec(pl.String, choices={"a": 10.0, "b": 5.0, "c": 1.0}) # same thing
```
Weights need a domain to apply to, so they require `choices`, an `Enum` dtype,
or `Boolean` (where they read `[p_false, p_true]`):
```python
ColSpec(pl.Enum(["x", "y", "z"]), weights=[1.0, 2.0, 7.0])
ColSpec(pl.Boolean, weights=[0.9, 0.1]) # 10% true
```
Choices are held in the column's own dtype, so a `datetime` choice on a
`Datetime` column or a `bytes` choice on a `Binary` column stays what it is.
They must be distinct once cast to that dtype -- `1` and `"1"` on a `String`
column are one value:
```python
ColSpec(pl.String, choices=[1, "1"])
# ValueError: ColSpec.choices contains values that are the same once cast to
# String: ['1']
```
## String and binary length
`string_length` is an inclusive `[min, max]` on characters (String) or bytes
(Binary). Unlike `bounds`, both endpoints are required.
```python
ColSpec(pl.String, string_length=(8, 8)) # fixed width
ColSpec(pl.Binary, string_length=(16, 64))
```
## Nested columns
A `List` or `Array` column is described by the same fields as a scalar one,
read as claims about **each element**: `bounds`, `choices`, `weights`,
`format`, `pattern`, `string_length` and `distribution` all apply to the
values inside the list. One field describes the list itself:
```python
ColSpec(pl.List(pl.Int64), bounds=(0, 10), list_length=(1, 5)) # 1 to 5 ints, each 0..10
ColSpec(pl.List(pl.String), format="email") # 0 to 5 addresses
ColSpec(pl.List(pl.Enum(["a", "b", "c"])), choices=["a", "b"]) # from a subset of the Enum
ColSpec(pl.Array(pl.Float64, 3), bounds=(0.0, 1.0)) # exactly three, from the dtype
```
`list_length` is the inclusive range of elements a value holds, both ends
required; without it generation makes 0 to 5. An `Array` takes its length
from the dtype and refuses `list_length`.
`nullable` and `null_probability` describe the list: a null cell, never a
null element. Generation never puts a null inside a list, and validation
reports one under `nullability` like a null in a non-nullable column.
Validation runs every element claim inside the list, and a list fails where
*any* element does — the finding's samples and `rows()` are the offending
lists. `list_length` gets its own finding code.
Generation draws the lengths and the elements as two columns of the inner
dtype and wraps one by the other, so an element is made by the same code
that would make a scalar of its dtype, and a `List` column keeps its data
across a rename through `seed_name` like any other.
What a nested column cannot carry: `unique` (a list is not drawn without
replacement) and `rules` (a rule's choices are values, and a list value
would be a list of lists). `validators` and `__checks__` work as on any
column — they are expressions.
### `fields`: what a struct's values are
A `Struct` column's dtype is its schema — every field's name and type comes
from it — and `fields` says what is *claimed* about the values in it:
```python
ColSpec(
pl.Struct({"lat": pl.Float64, "lon": pl.Float64, "label": pl.String}),
fields={
"lat": ColSpec(pl.Float64, bounds=(-90, 90)),
"lon": ColSpec(pl.Float64, bounds=(-180, 180)),
},
nullable=True,
)
```
Each value is a `ColSpec`, so a field is described exactly as a column of
the same dtype would be. `fields` is **partial**: a struct of twenty fields
where one needs bounds spells one field, and the rest are generated from
their dtypes alone. A name the dtype does not declare, or a field spec
whose dtype disagrees with the struct's, is refused where it is written.
A field is a value, not a column, so `unique`, `rules`, `validators`,
`seed_name` and `col_name` are refused on one — each is a claim about a
column among columns. (A validator about a field is written on the struct
column instead: `pl.col("point").struct.field("lat") != 0`.) As with a
list, the column's `nullable` describes the *cell* — a null struct — and a
field's own `nullable` says whether it may be null inside a struct that is
present. It defaults to `False`, like any column's.
A `List` of a `Struct` takes `fields` too, describing its element, and a
field may itself be a struct or a list, so a declaration nests as deeply as
the dtype does:
```python
ColSpec(pl.Struct({"xs": pl.List(pl.Int64)}),
fields={"xs": ColSpec(pl.List(pl.Int64), bounds=(0, 9), list_length=(2, 2))})
```
Generation makes one column per field and gathers them, so a field is drawn
by the code that draws a column of its dtype — and a struct column is
seeded by name like any other: renaming it with `seed_name` keeps every
field, and adding a field beside one moves nothing.
Validation checks each field's claims in place, and a finding names the
field it is about — its key is `point.lat__bounds`, its message says
`Column 'point.lat'` — while its `columns` stay `("point",)`, so
`report.rows(finding)` returns the rows of the frame that hold the
offending structs. Inside a list the samples are the offending lists, as
for any list. A struct in the data matches a declared one by field name,
not order, each field compatible by the usual rules; a field missing or
added is a `dtype` finding.
## String formats
`format=` names the shape a `String` column's values take, and both sides
read it: generation fills the column from that format's sampler and
validation checks every value against it.
```python
ColSpec(pl.String, format="uuid4", unique=True)
ColSpec(pl.String, format="email", nullable=True)
```
The set is `uuid4`, `email`, `ipv4`, `ipv6`, `mac`, `hostname`,
`iso_country` and `iso_currency`. A format owns the column's domain, so it
cannot sit beside `choices` or `string_length`, and only a `String` column
can carry one. See [String formats](https://maxwellb13.github.io/polspec/how-to/formats/) for what each generates,
what each accepts, and what none of them promises.
## String patterns
`pattern=` is a regular expression every value must match -- **checked by
validation only**. Generation does not read it: a column with a pattern is
filled with ordinary random text, so the round trip holds only with
`validate_pattern=False`, exactly as for `validators`. That is the honest
half of the split `format=` makes: polspec can check any regex, and can
generate a curated set.
```python
ColSpec(pl.String, pattern=r"^[A-Z]{3}-\d{4}$") # a SKU shape polspec cannot generate
```
Reach for `format` when the shape is one polspec has; reach for `pattern`
when it is not. The two cannot be combined -- a format already *is* a
pattern with a sampler. A pattern is compiled by Polars' regex engine at
declaration, so one Polars cannot run (look-around, for instance) is refused
with the engine's own message rather than failing on the first validation.
## Distributions
Numeric and temporal columns can be drawn from a shape other than uniform:
| Distribution | Parameters (aliases accepted) |
|:--|:--|
| `uniform` | — |
| `normal` | `mean`/`mu`/`loc`, `std`/`sigma`/`scale` |
| `lognormal` | `mean`/`mu`/`meanlog`, `std`/`sigma`/`sdlog` |
| `exponential` (`exp`) | `rate`/`lambda`/`lambda_`, or `scale` |
| `poisson` | `lambda`/`lambda_`/`rate`/`mean` |
| `gamma` | `shape`/`alpha`/`k`, `scale`/`beta`/`theta` |
| `beta` | `alpha`/`a`/`shape1`, `beta`/`b`/`shape2` |
```python
ColSpec(
pl.Float64,
bounds=(0.0, 500.0),
distribution="lognormal",
distribution_params={"mean": 2.0, "std": 0.6},
)
```
!!! warning "Bounds clamp, they do not resample"
A draw outside the bounds lands *on* the boundary rather than being drawn
again. A `normal` centred at 0 squeezed into `(0, 50)` puts roughly half
the column on the floor as one repeated value.
When you want a positive-skewed shape, reach for a distribution that is
already non-negative — `lognormal`, `exponential`, `gamma` — instead of
clamping a symmetric one.
## Uniqueness
`unique=True` declares that values must be distinct. `generate()` draws the
column without replacement, so the data it produces satisfies it.
Nulls are exempt, as they are for foreign keys: a null means "no value", so a
nullable unique column may repeat nulls and nothing else.
A domain too small to cover the row count is refused, naming the column:
```python
class Narrow(FrameSpec):
id = ColSpec(pl.Int8, unique=True)
Narrow.generate(300, seed=1)
# GenerationError: Column 'id' is unique, but its domain holds only 256
# distinct value(s) and 300 are needed. Widen its bounds or choices, or
# generate fewer rows.
```
`unique=True` cannot be combined with `weights`, a non-uniform `distribution`,
or `rules`: the first two describe how often a value recurs, which a draw
without replacement has no room for, and a rule would reintroduce the
duplicates. Each is refused at declaration rather than quietly ignored.
## Tags
Tags group columns for later selection. They carry no generation or validation
meaning.
```python
class Events(FrameSpec):
user_id = ColSpec(pl.Int64, tags=["pii", "key"])
email = ColSpec(pl.String, tags="pii")
duration = ColSpec(pl.Int64, tags="metric")
Events.tag("pii") # ['user_id', 'email']
Events.tag("pii", "key", match="all") # ['user_id']
```
## Column names that are not identifiers
A column declared as a class attribute takes the attribute's name, and an
attribute name has to be a valid Python identifier. Real data is not so
polite. There are two ways out, for two different situations.
### `col_name`: the data's name has spaces or punctuation
Keep a clean attribute name and tell the `ColSpec` what the column is really
called:
```python
class Sales(FrameSpec):
unit_price = ColSpec(pl.Float64, col_name="Unit Price", bounds=(0, None))
region = ColSpec(pl.Enum(["UK", "US"]), col_name="Sales Region")
Sales.schema() # Schema({'Unit Price': Float64, 'Sales Region': Enum(...)})
Sales.generate(3).columns # ['Unit Price', 'Sales Region']
```
`col_name` is the column's name everywhere the spec is used: in the generated
frame, in `validate()`, in a `ColRule` condition built with `col()`, in
`__unique_together__`, in `ForeignKey` columns and in `tag()` results. The
attribute name exists only in the class body. Two attributes that resolve to
the same `col_name` are rejected at declaration, and overriding an attribute
on a subclass removes the column it named, whatever `col_name` it carried.
`to_yaml()` and `to_python()` write the real column name as the key, so a
spec that came from a file never needs `col_name`.
A third name, `seed_name`, is not about what the column is called but about
what it *generates*: a renamed column declared with `seed_name="old"` keeps
producing the data it did under the old name. See
[Renaming a column without changing its data](https://maxwellb13.github.io/polspec/how-to/generating/#renaming-a-column-without-changing-its-data).
### `__columns__`: the name is an identifier but cannot be an attribute
A leading underscore is skipped by the class-body scan, so a column called
`_id` needs the explicit mapping. A name that matches one of `FrameSpec`'s
methods (`schema`, `tag`, …) is fine either way: the method keeps working and
the column is reachable as `Spec.col("schema")` -- see
[Specs as values](https://maxwellb13.github.io/polspec/how-to/tablespec/#column-names-and-method-names).
```python
class Raw(FrameSpec):
__columns__ = {
"_id": ColSpec(pl.Int64),
"schema": ColSpec(pl.String),
}
```
`__columns__` is never looked up as an attribute, so both the column and the
method survive. The dict key already is the column name, so a `col_name` that
disagrees with its key is rejected. `from_dataframe`, `from_yaml` and
`to_python` all declare columns this way, since their names come from data
rather than from someone's class body.
---
# String formats
Source: https://maxwellb13.github.io/polspec/how-to/formats/
# String formats
A `String` column can say how long its values are. `format=` lets it say what
they look like, and the two sides of the spec read that one word: generation
fills the column from the format's own sampler, and validation checks every
value against the format's own test.
```python
class Users(FrameSpec):
user_id = ColSpec(pl.String, format="uuid4", unique=True)
email = ColSpec(pl.String, format="email")
host = ColSpec(pl.String, format="ipv4", nullable=True)
df = Users.generate(10_000, seed=42)
Users.validate(df) # passes -- which is the point
```
Without a format, a column carrying a validator as ordinary as
`col("email").str.contains("@")` could not be generated to satisfy its own
spec. With one it can, because the sampler and the validator are one
declaration in `polspec.formats`, reviewed in one diff and pinned by one
round-trip test per format.
## The formats
| `format` | Generates | Validates |
|:--|:--|:--|
| `uuid4` | 32 hex digits in 8-4-4-4-12 groups, version nibble `4`, variant `8`–`b` | the same shape, either case |
| `email` | a local part, `@`, a domain and a TLD from a small pool | one `@`, non-empty both sides, a dot in the domain |
| `ipv4` | four dotted octets | four dotted decimals, each 0–255 |
| `ipv6` | eight colon-separated four-digit hextets | eight colon-separated hextets of 1–4 hex digits, uncompressed |
| `mac` | six colon-separated hex pairs | the same shape, either case |
| `hostname` | two or three labels ending in a TLD | dot-separated RFC 1123 labels, at most 253 characters |
| `iso_country` | one of the 249 ISO 3166-1 alpha-2 codes | membership in that list |
| `iso_currency` | one of the ISO 4217 alpha-3 codes of circulating currencies | membership in that list |
A validator is deliberately wider than its sampler where real data is: an
uppercase UUID from another system validates, even though polspec generates
lowercase ones. It is never wider than the standard it names.
The set is closed. A named format is a twenty-line sampler with an
unambiguous test; generating from an arbitrary regex would need a
regex-to-sampler compiler and has no answer for `.*`. Validation of an
arbitrary regex is what [`pattern=`](https://maxwellb13.github.io/polspec/how-to/columns/#string-patterns) is for --
checked like a format, generated like nothing at all.
## What a format promises
Syntax. `format="email"` generates a well-formed address and validates a
well-formed address; nothing is looked up, so `nobody@example.invalid` is
accepted and no generated address is deliverable. `hostname` accepts
`localhost`; `ipv4` accepts `0.0.0.0`. See
[Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/).
## What a format combines with
A format owns the column's whole domain, so it refuses anything that says
what the values are a second time:
```python
ColSpec(pl.String, format="email", choices=["a@b.co"])
# SpecError: ColSpec cannot carry both format='email' and choices: each is a
# complete description of the column's domain, and they cannot both hold.
ColSpec(pl.String, format="uuid4", string_length=(36, 36))
# SpecError: ... the format already fixes how long a value is.
ColSpec(pl.Int64, format="uuid4")
# SpecError: ColSpec.format is only supported for pl.String, got Int64.
```
Everything that says how the values are *distributed* still applies:
- `nullable` and `null_probability` work as on any column.
- `unique=True` draws without replacement. `uuid4` has room for any frame;
`iso_country` runs out at 249 rows and says so by name, the way a small
`choices` list does.
- `validators` run alongside the format check. A column can carry
`format="email"` and a validator on the domain it should come from.
- `tags` and `col_name` are untouched.
## Foreign keys
A format is part of the column's domain, so a key into a formatted parent is
checked at declaration like every other domain narrowing. A child column with
`format="uuid4"` accepts a parent with the same format and refuses one with
another format or none; a plain `String` child accepts any parent. A parent
whose `choices` are listed is checked value by value.
```python
class Sessions(FrameSpec):
user_id = ColSpec(pl.String, format="uuid4")
__foreign_keys__ = [ForeignKey("user_id", references=Users)]
users = Users.generate(100, seed=1)
sessions = Sessions.generate(1_000, seed=2, references={Users: users})
Sessions.validate(sessions, references={Users: users})
```
## Reports and files
The `format` finding reports values that do not match, with the format named
and samples listed:
```
Column 'host': found 3 value(s) that are not ipv4 (four dotted decimal
octets, each 0-255). Invalid samples: ['300.1.1.1', 'abc']
```
A format is one scalar key in a spec file -- `format: email` -- and one
keyword in generated Python. Neither changes the file format version.
`from_dataframe()` does not infer formats: guessing `email` from a sample is a
guess, and the profiler's contract is to describe what is there.
---
# Specs as values
Source: https://maxwellb13.github.io/polspec/how-to/tablespec/
# Specs as values
A `FrameSpec` class body is the convenient way to *write* a spec. What it
builds is a `TableSpec`: an immutable value holding the columns, checks,
composite keys and foreign keys, reachable as `.spec` on the class.
```python
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, unique=True)
total = ColSpec(pl.Float64, bounds=(0.0, None))
internal_note = ColSpec(pl.String, nullable=True)
Orders.spec # TableSpec(name='Orders', columns={...}, ...)
Orders.spec.name # 'Orders'
list(Orders.spec) # ['order_id', 'total', 'internal_note']
Orders.spec["total"] # the ColSpec
Orders.spec.schema() # the same pl.Schema as Orders.schema()
```
Every verb the library offers is a function over a `TableSpec`; the
classmethods on `FrameSpec` are one-line forwards that pass `cls.spec`. So a
`TableSpec` is the thing being operated on either way:
```python
import polspec
Orders.generate(1_000, seed=1) # the class
polspec.generate(Orders.spec, 1_000, seed=1) # the function, over the value
```
Both reach the same code. The functions are exported from `polspec` itself:
```python
from polspec import generate, generate_batches, inspect, validate
from polspec import sink_csv, sink_ipc, sink_ndjson, sink_parquet
df = generate(Orders.spec, 1_000, seed=1)
report = inspect(Orders.spec, df)
validate(Orders.spec, df)
```
Each takes a `TableSpec` as its first argument, and each has a `FrameSpec`
classmethod that forwards to it with `cls.spec`. Use whichever suits the code:
the classmethods read better when a class body declared the spec, the
functions when the spec is a value that was built, loaded or derived.
## Building one directly
A `TableSpec` can be constructed without a class body, which is how
`from_yaml` and `from_dataframe` work internally:
```python
from polspec import TableSpec
spec = TableSpec(
"Orders",
{"order_id": ColSpec(pl.Int64, unique=True), "total": ColSpec(pl.Float64)},
unique_together=[["order_id"]],
)
```
Everything a class body validates at declaration is validated here too. A
`TableSpec` that constructs is one that can be used.
To get the class-shaped API back, wrap it:
```python
Rebuilt = FrameSpec.from_spec(spec) # a subclass named Orders
Renamed = FrameSpec.from_spec(spec, name="Orders2026")
```
## Deriving one spec from another
Each operation returns a new `TableSpec`; the original is never changed.
| Operation | Effect |
|:--|:--|
| `with_columns({...}, **cols)` | Add columns, or replace existing ones in place |
| `drop(*names)` | Remove columns, and any composite or foreign key that used them |
| `select(*names)` | Keep only the named columns, in that order |
| `rename({old: new})` | Rename columns, rewriting rules, composite keys and foreign keys |
| `with_checks(*checks)`, `with_foreign_keys(*fks)`, `with_unique_together(*groups)` | Append constraints |
| `with_name(name)` | Change the name |
| `with_catspec(registry)` | Re-type columns against a `CatSpec`; see [Shared categories](https://maxwellb13.github.io/polspec/how-to/categories/) |
```python
staging = Orders.spec.drop("internal_note").rename({"total": "amount"})
Staging = FrameSpec.from_spec(staging, name="StagingOrders")
```
Two deliberate limits. `drop` leaves a rule on a surviving column that points
at a dropped one for validation to reject, since silently dropping a rule
would change what the surviving column generates. `rename` refuses a column
carrying `validators`, because a validator is a Polars expression naming the
column, and rewriting expressions is not something this library does.
## Column names and method names
Because the class body's `ColSpec` attributes are taken out of the namespace
before the class exists, a column may share a name with a method. The method
wins on attribute access; the column is reachable by name:
```python
class Raw(FrameSpec):
schema = ColSpec(pl.String)
tag = ColSpec(pl.String)
Raw.schema() # the method: Schema({'schema': String, 'tag': String})
Raw.col("schema") # the column
Raw.spec["tag"] # also the column
```
An ordinary column is still an attribute (`Orders.order_id`), through a
fallback that runs only when normal lookup fails.
## Foreign keys point at names
`ForeignKey.references` is stored as the target spec's *name*. Declaring
`references=Customers` binds the target for declaration-time checks and
stores `"Customers"`; a key can also be declared against a bare name, which
nothing checks until a spec of that name is supplied:
```python
ForeignKey("customer_id", references=Customers, ref_columns="id") # checked now
ForeignKey("customer_id", references="Customers", ref_columns="id") # checked later
```
`generate(references={...})` and `validate(references={...})` accept the
parent frame keyed by the class, the `TableSpec`, or the name. A
[`Registry`](https://maxwellb13.github.io/polspec/how-to/registry/) holding both specs binds the name and runs the
checks the class form would have run at declaration:
```python
class Shipments(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
__foreign_keys__ = [
ForeignKey("customer_id", references="Customers", ref_columns="id")
]
Shipments.spec.foreign_keys[0].target # None -- nothing to check against
bound = Registry(Customers, Shipments).resolve()
bound["Shipments"].foreign_keys[0].target # Customers.spec
```
---
# Constraints
Source: https://maxwellb13.github.io/polspec/how-to/constraints/
# Constraints
Beyond the shape of a single value, a spec can assert relationships. They fall
into two groups worth keeping straight:
| | Generated | Validated |
|:--|:--:|:--:|
| `ColRule` | yes | yes |
| `ForeignKey` | yes, when given parent data | yes |
| `unique=True`, `__unique_together__` | yes | yes |
| `ColSpec.validators`, `__checks__` | no | yes |
The last row is validation-only by design: both wrap arbitrary Polars
expressions, and nothing can produce data satisfying an arbitrary predicate.
Generation makes no attempt, and that boundary is pinned down by tests.
## Writing conditions — `col()`
Rules, validators and checks all take a condition. Write it with `col()`,
which builds a small predicate tree rather than a Polars expression:
```python
from polspec import col
col("total") >= col("subtotal")
col("email").str.contains("@")
col("status").is_in(["NEW", "PAID"]) & (col("qty") > 0)
col("shipped").is_null() | (col("shipped") >= col("placed"))
```
Supported: comparisons (`== != < <= > >=`), arithmetic (`+ - * /`),
`&`, `|`, `~`, `is_in`, `is_null`, `is_not_null`, `is_between`, and the
string operations `str.contains` (a literal substring), `str.starts_with`,
`str.ends_with`, `str.matches` (a regular expression) and `str.len_chars`.
Scalars, dates and datetimes are fine as operands.
A predicate evaluates exactly as the Polars expression it stands for, and
unlike one it can be written to a spec file and read back, so rules, checks
and validators written this way survive `to_yaml` and `to_python`. A raw
`pl.Expr` is still accepted everywhere a predicate is, for anything the
predicate language cannot say; it just cannot be persisted.
Comparison operators build predicates, as they do on `pl.Expr`, so a
predicate has no truth value. Compare two structurally with `Pred.equals`.
## Conditional values — `ColRule`
A rule overwrites a column on the rows where its condition matches.
```python
from polspec import ColRule, col
class Shipments(FrameSpec):
region = ColSpec(pl.Enum(["UK", "US", "EU"]))
carrier = ColSpec(
pl.Enum(["RoyalMail", "UPS", "DHL"]),
rules=[
ColRule(when=col("region") == "UK", choices=["RoyalMail"]),
ColRule(when=col("region").is_in(["US", "EU"]), choices=["UPS", "DHL"]),
],
)
```
Multiple rules on one column are tried in declaration order, first match wins,
like a SQL `CASE`. `choices` may be weighted exactly as on a `ColSpec`:
```python
ColRule(when=col("region") == "US", choices={"UPS": 3.0, "DHL": 1.0})
```
`when` is a predicate built with [`col()`](#writing-conditions-col), not an
arbitrary Polars expression, so that every rule can round-trip through a spec
file. Conditions compose:
```python
ColRule(
when=col("region").is_in(["US", "EU"]) & (col("weight_kg") > 10),
choices=["DHL"],
)
```
!!! note "Rules see the frame as it stands"
Every `when` is evaluated against the values the column actually holds
when the rule runs, and the passes run in dependency order: a rule keyed
on a column that another rule or a foreign key rewrites reads the
rewritten values — the same ones `validate()` will check the rule
against. So rules chain:
```python
class Orders(FrameSpec):
region = ColSpec(pl.Enum(["UK", "US"]))
carrier = ColSpec(
pl.Enum(["RoyalMail", "UPS"]),
rules=[ColRule(when=col("region") == "UK", choices=["RoyalMail"])],
)
tracked = ColSpec( # keyed on a column that carries rules of its own
pl.Enum(["yes", "no"]),
rules=[ColRule(when=col("carrier") == "RoyalMail", choices=["yes"])],
)
```
Two columns whose rules each read what the other writes have no such
order, and are refused at declaration with `SpecError`.
A rule also overwrites nulls on matching rows, so a nullable column with a rule
ends up with fewer nulls than `null_probability` suggests.
## Single-column predicates — `validators`
A validator is a Polars expression that each row must satisfy, referencing only
its own column:
```python
class Accounts(FrameSpec):
email = ColSpec(pl.String, validators=[pl.col("email").str.contains("@")])
```
Wrap one in a `Check` to name it, describe it, or change null handling:
```python
from polspec import Check
ColSpec(
pl.Float64,
validators=[
Check(
pl.col("score") <= 100,
name="score_ceiling",
description="Scores are a percentage",
)
],
)
```
Referencing another column is rejected at declaration time — use `__checks__`
for that.
## Multi-column invariants — `__checks__`
```python
class Invoices(FrameSpec):
subtotal = ColSpec(pl.Float64, bounds=(0.0, 1000.0))
total = ColSpec(pl.Float64, bounds=(0.0, 2000.0))
__checks__ = [
Check(pl.col("total") >= pl.col("subtotal"), name="total_covers_subtotal"),
]
```
By default a row whose check evaluates to null passes, matching SQL `CHECK`
semantics. `Check(..., ignore_nulls=False)` treats null as a failure.
Checks are inherited: a subclass collects its bases' checks as well as its own,
de-duplicated. Two *different* checks sharing a name is an error, since the name
is what an error message points at.
## Composite uniqueness — `__unique_together__`
A composite key declares that a *combination* of columns is distinct, even
where each column on its own repeats.
```python
class Assignments(FrameSpec):
employee_id = ColSpec(pl.Int64, bounds=(1, 500))
project_id = ColSpec(pl.Int64, bounds=(1, 200))
__unique_together__ = [["employee_id", "project_id"]]
```
`generate()` satisfies it by resampling the rows that repeat a combination an
earlier row already used. Only the repeats move, so on a roomy domain almost
every row keeps the value it was generated with, along with whatever weights
or bounds shaped it. Rows where any member is null are exempt, matching how
the key is validated.
A group whose columns cannot take enough distinct combinations between them is
refused, naming the group:
```python
class TooTight(FrameSpec):
a = ColSpec(pl.Enum(["x", "y"]))
b = ColSpec(pl.Enum(["p", "q"]))
__unique_together__ = [["a", "b"]]
TooTight.generate(300, seed=1)
# GenerationError: Composite unique key ['a', 'b'] cannot be satisfied: the
# columns take 4 distinct combination(s) between them and 300 row(s) need one.
```
A member column may not also carry `rules`: a rule assigns from a fixed set,
which is how two rows come to share a combination, and the repair would
overwrite what the rule put there. Declare one or the other.
A foreign-keyed member is never resampled — that would break the key — so the
repair works with the other members. If *every* member is foreign-keyed there
is nothing it can move, and generation says so rather than returning data that
fails its own validation.
## Referential integrity — `ForeignKey`
```python
from polspec import ForeignKey
class Customers(FrameSpec):
id = ColSpec(pl.Int64, bounds=(1, 10_000))
class Orders(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
__foreign_keys__ = [
ForeignKey("customer_id", references=Customers, ref_columns="id"),
]
```
Rows where any key column is null are exempt — a null foreign key means "no
reference", not an invalid one.
### Generating consistent data
Pass the parent frame and the child's key values are sampled from it, so the
result is referentially consistent by construction:
```python
customers = Customers.generate(1_000, seed=1)
orders = Orders.generate(10_000, seed=2, references={Customers: customers})
Orders.validate(orders, references={Customers: customers})
```
Without `references`, generation leaves the column freely generated — but
`validate()` then *raises*, because it has nothing to check against. Supply the
parent to both calls, or disable the check with
`validate(..., validate_foreign_keys=False)`.
Composite keys are sampled as one joint pick per row, so multi-column keys stay
internally consistent.
With several related specs, a [`Registry`](https://maxwellb13.github.io/polspec/how-to/registry/) does the walk:
`Registry(Customers, Orders, OrderLines).generate_all(1_000, seed=1)` generates
parents first and threads each into its children, and `validate_all` checks
the whole set.
### Self-references
```python
class Employees(FrameSpec):
id = ColSpec(pl.Int64, bounds=(1, 500))
manager_id = ColSpec(pl.Int64, bounds=(1, 500), nullable=True)
__foreign_keys__ = [
ForeignKey("manager_id", references="self", ref_columns="id"),
]
```
`"self"` resolves to whichever spec the key ends up declared on, and needs no
`references` entry in either call.
!!! warning "The parent's domain has to fit inside the column's own"
A foreign key overwrites its column with values from the parent, so the
parent's `bounds` or `choices` have to be ones the column itself declares
it can hold. Declaring `bounds=(1, 50)` on a column referencing keys in
`100..200` would produce data that fails its own validation, so it is
refused when you declare it:
```python
class Orders(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 50))
__foreign_keys__ = [
ForeignKey("customer_id", references=Customers, ref_columns="id")
]
# SpecError: ... column 'customer_id' is declared bounds [1, 50], but the
# key fills it with values from 'id' on 'Customers', where bounds
# [1, 10000] do not fit inside [1, 50].
```
A column that declares no `bounds` or `choices` accepts anything, so the
check only fires on a genuine contradiction. Widen or drop the child's
declaration, or narrow the parent's.
The check needs both specs, so a key naming its target as a string is
checked when a [`Registry`](https://maxwellb13.github.io/polspec/how-to/registry/) resolves it, not before.
---
# Generate data
Source: https://maxwellb13.github.io/polspec/how-to/generating/
# Generating data
```python
df = Orders.generate(1_000_000, seed=42)
```
Columns are generated independently and in parallel by the Rust extension, then
cast to their declared dtypes in one Polars pass.
## Reproducibility
A `seed` fixes the result across processes, machines and thread counts. Each
column derives its own seed from the frame seed and its *name*, and each
65,536-row chunk from its index, so the same seed gives the same frame
regardless of how many threads did the work, and adding or reordering columns
never changes the values of the others. The passes that run after the columns
are filled -- rules, the hierarchy, foreign keys, composite uniqueness -- are
seeded the same way, from the frame seed and a key naming what the pass is
for, so a column added beside a ruled or foreign-keyed one leaves it alone
too.
```python
Orders.generate(500, seed=7).equals(Orders.generate(500, seed=7)) # True
```
Omit `seed` and generation is seeded from the clock.
### Renaming a column without changing its data
Because a column's seed comes from its name, renaming a column changes the
values it produces -- which matters when generated frames are snapshots that
other things are compared against. `seed_name` is the name the seed is
derived from when it is not the column's own, so a renamed column keeps
producing the data it did:
```python
class Before(FrameSpec):
id = ColSpec(pl.Int64, unique=True)
status = ColSpec(pl.String, nullable=True)
class After(FrameSpec):
id = ColSpec(pl.Int64, unique=True)
state = ColSpec(pl.String, nullable=True, seed_name="status") # renamed
before = Before.generate(100, seed=1)
after = After.generate(100, seed=1)
after["state"].equals(before["status"]) # True
```
A rename is declared, never guessed: `TableSpec.rename()` does not set
`seed_name`, and [`diff(renames=)`](https://maxwellb13.github.io/polspec/how-to/drift/) is told about the same rename
on the validation side. Two columns of one spec cannot share a seed name, and
one cannot borrow another column's name -- they would draw identical values,
which is what a [rule](https://maxwellb13.github.io/polspec/how-to/constraints/) is for.
`seed_name` covers the column's passes as well as its values: a ruled
column renamed with one keeps its rule's draw. What no seed name holds is
the frame across polspec *versions* -- see
[Roadmap and stability](https://maxwellb13.github.io/polspec/explanation/roadmap/#yaml-format-and-generated-values-may-change).
## Lazy output — `scan()`
`scan()` returns a `pl.LazyFrame` that has not been generated. Nothing is
drawn until the plan is collected, and then only the columns and rows the
plan asks for:
```python
lf = Orders.scan(50_000_000, seed=1)
lf.sink_parquet("orders.parquet") # streams; bounded memory
lf.select("total").head(5).collect() # five rows of one column
lf.filter(pl.col("status") == "PAID").collect() # every row drawn, matching kept
```
**Projecting cannot change what a column holds.** Every column is seeded by
its name and every pass by what it is for, so dropping a column's
neighbours leaves it alone — `lf.select(cols).collect()` is always
`lf.collect().select(cols)`. Where a column depends on others (a rule reads
the columns its `when` names, a composite key is repaired as a group), those
are generated too and dropped again on the way out.
A predicate filters rows that were drawn; it never narrows the draw. `n`
rows are generated and the matching ones kept, because drawing only matching
rows would quietly change what a `null_probability` or a `unique=True`
column means.
Rows arrive in batches, so a scan carries the terms
[batching](#batching) does: a spec declaring a `__hierarchy__` is refused,
and uniqueness holds within a batch. Leaving `batch_size` unset lets polars
ask for the size it wants; setting it pins the size.
`Registry.scan_all()` is the same for a set of specs: parents are generated
eagerly — a foreign key needs the whole parent column to sample from — and
the children are lazy.
!!! note
`generate()` builds the whole frame before it returns, so `.lazy()` on
its result is a handle on memory already spent. `scan()` is the frame
that has not been built. (`generate(lazy=True)` did the former while
looking like the latter, and was removed in 0.9.0.)
## Coverage — `method="cartesian"`
The default `method="random"` draws each column independently, so a rare enum
value may not appear at all. `method="cartesian"` guarantees it will:
```python
df = Orders.generate(500, method="cartesian", seed=1)
```
It builds the cross-product of every finite domain — each `Enum`'s categories,
both booleans, and the negative / zero / positive / null partitions of every
bounded numeric column — so every combination is present at least once.
Columns with no finite domain (String, bare `Categorical`) are filled in
randomly alongside.
!!! warning "`n` is a minimum here, not a count"
If the coverage set is smaller than `n` it is padded with random rows. If
it is **larger**, all of it is kept and `n` is exceeded. Two ten-category
enums produce 100 rows however small `n` was.
A safety cap refuses to build more than 50 million coverage rows, naming
each dimension's cardinality so you can see which one exploded.
## Batching
For volumes that should not be held in memory at once:
```python
for batch in Orders.generate_batches(10_000_000, batch_size=250_000, seed=1):
process(batch)
```
Each batch is a **window onto the one frame the seed describes**: a column
no pass rewrites holds, batch by batch, exactly the rows
`Orders.generate(n, seed=1)` would, whatever `batch_size` is -- so a stream
written at one batch size and re-read at another is the same data, and the
third batch can be checked against `generate(n).slice(...)`. What is drawn
per batch instead, deterministic but not row for row the whole frame's, is
a column with rules, a foreign key, a composite key, and a `List` column's
elements (its lengths are a window). Uniqueness holds only *within* a batch.
A batch smaller than 65,536 rows -- the engine's chunk -- costs up to one
chunk of extra draws per batch, because a window that starts mid-chunk fills
the chunk from its start and slices the head off. Batches of a chunk or
more cost nothing extra.
## Writing straight to a file
Four sinks stream batches to disk without materialising the whole frame:
```python
Orders.sink_parquet("orders.parquet", 50_000_000, compression="zstd")
Orders.sink_csv("orders.csv", 1_000_000)
Orders.sink_ipc("orders.arrow", 1_000_000, compression="zstd")
Orders.sink_ndjson("orders.ndjson", 1_000_000)
```
Each is [`scan()`](#lazy-output-scan) handed to the matching
`LazyFrame.sink_*`, so what a sink writes is what collecting the scan gives.
All four take `batch_size`, `method`, `seed` and `references`, create the
parent directory if needed, and pass extra keyword arguments through to
polars' own sink. Nothing beyond Polars is needed for any of them.
With `n=0`, Parquet, IPC and CSV still write a valid schema-bearing file.
A CSV cannot hold a `Duration`, `List` or `Struct` column, so `sink_csv`
refuses a spec with one; Parquet and IPC keep every dtype.
A sink is the shorthand; `Orders.scan(n, seed=1).sink_parquet(path)` is the
same write with the rest of a lazy plan available — a `filter`, a `select`,
a `sort` — before it reaches the file.
## Memory
`generate()` allocates the whole frame before it returns, so it is worth
knowing what that is before asking for it:
```python
Orders.estimated_size(50_000_000) / 1024**3 # gibibytes
```
Read off the declaration — the width of each dtype, the lengths the spec
declares — so it costs nothing and needs no data. Past four gibibytes
`generate()` says so in a warning naming the estimate; `max_bytes=` makes
it a refusal instead, for a CI job that should fail rather than swap, and
`max_bytes=0` silences both.
What each column costs per row:
| Declared | Bytes per row |
|:--|--:|
| `Int8`/`UInt8` … `Int64`/`Float64` | 1 … 8 |
| `Boolean` | ⅛ |
| `Date` | 4; `Time`, `Datetime`, `Duration` | 8 |
| `Decimal` | 16 |
| `Enum` | 1, 2 or 4 — the narrowest that holds the categories |
| `Categorical` | its registry's physical width, 4 by default |
| `String`, `Binary` | **16**, plus the length of any value past 12 bytes, on the rows that are not null |
| `List(inner)` | 8, plus the mean `list_length` × the element's cost, on the rows that are not null |
| `Array(inner, w)` | `w` × the element's cost |
| `Struct` | the sum of its fields, each costed as a column |
| `nullable=True` | + ⅛ |
The sixteen bytes a text value costs before any content is the lever worth
knowing: a low-cardinality string column declared as `pl.Enum([...])` costs
**one** byte per row instead of twenty, and one drawn from `choices` costs
the sixteen but not the content, because the values are gathered from one
shared buffer.
The estimate is the frame, not the process. Generation holds working
buffers on top — most visibly for `Decimal` and `List`, assembled in Polars
rather than filled by the engine — so a peak is higher. For a frame of
scalar columns the two agree within a percent.
[`scan()`](#lazy-output-scan) and [batching](#batching) are the way out of
the question entirely: both hold a batch at a time rather than the frame.
## Foreign keys
`references` maps a parent spec to its data, and makes generated keys
referentially consistent. See [Constraints](https://maxwellb13.github.io/polspec/how-to/constraints/#referential-integrity-foreignkey).
```python
orders = Orders.generate(10_000, seed=2, references={Customers: customers})
```
## What generation does not enforce
Generation satisfies dtypes, nullability, bounds, string lengths, value
domains, `format`s, weights, distributions, `unique=True`,
`__unique_together__`, `ColRule`s, hierarchies and — when given parent data —
foreign keys.
It does **not** attempt `ColSpec.validators`, `__checks__` or
`ColSpec.pattern`, by design: the first two hold arbitrary expressions and the
third an arbitrary regex, and nothing can generate data to satisfy either in
general. See [Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/).
---
# Validate data
Source: https://maxwellb13.github.io/polspec/how-to/validating/
# Validating data
```python
clean = Orders.validate(df)
```
`validate()` accepts a `DataFrame` or a `LazyFrame` and returns the same kind,
so it drops into a pipeline. On success the returned frame has its declared
columns first, in declaration order.
## Collecting every problem at once
All checks across all columns are compiled into one Polars aggregation and
evaluated in a single scan. Every breach is gathered before anything is raised:
```python
from polspec import ValidationError
try:
Orders.validate(df)
except ValidationError as err:
print(len(err.errors), "problems")
for problem in err.errors:
print(problem)
```
`ValidationError` is a `PolspecError` (and still a `ValueError`); see
[Errors](https://maxwellb13.github.io/polspec/reference/errors/). `err.errors` is the list of
individual messages; `str(err)` is the same list formatted as a report, and
`err.report` is the `ValidationReport` behind both.
## Findings as data — `inspect()`
An exception is the right shape for someone reading a traceback. For code
that wants to *act* on what was found — quarantine the offending rows, count
problems per column, write a report — use `inspect()`, which returns the same
findings as a `ValidationReport` and never raises for a bad frame:
```python
suspect = df.with_columns(pl.col("total") * -1) # every total now negative
report = Orders.inspect(suspect)
report.passed # False
for finding in report:
finding.code # "bounds", "check", "foreign_key", ...
finding.key # "total__bounds", "check:total_covers_subtotal",
# "point.lat__bounds" for a struct's field
finding.columns # ("total",)
finding.count # rows violating it (None for structural findings)
finding.samples # up to five offending values
finding.details # {"bounds": [0.0, None], "min_found": -3.0, ...}
finding.message # the same text validate() would have raised
report.by_column()["total"] # every finding involving one column
report.by_code("foreign_key") # every finding of one kind
report.to_json() # everything above, JSON-safe
```
The offending rows are reachable lazily, so nothing is materialised until you
ask:
```python
bad = report.by_code("bounds")[0]
report.rows(bad).collect() # just the rows violating that one claim
report.failing_rows().collect() # every violating row, with a `__polspec_finding`
# column naming the claim (a row violating two
# claims appears twice)
```
The column `failing_rows()` adds is named by `polspec.validation.FINDING_COLUMN`
rather than spelled out, so grouping by it does not hard-code the name:
```python
from polspec.validation import FINDING_COLUMN
quarantined = report.failing_rows().collect()
quarantined.group_by(FINDING_COLUMN).len() # how many rows each claim caught
```
Structural findings (`extra_columns`, `missing_columns`, `dtype`,
`foreign_key_unresolved`) describe the frame's shape rather than its rows and
have no rows to return. `inspect()` takes exactly the options `validate()`
does; `validate()` is `inspect()` followed by `report.raise_if_failed()` and
the structural transformations below. The full list of codes is in
[Errors](https://maxwellb13.github.io/polspec/reference/errors/#finding-codes), and `polspec validate` on the
[command line](https://maxwellb13.github.io/polspec/how-to/cli/#validate-check-data-against-a-schema) prints the same
report.
## Checking a file you were given
The usual way in: someone hands you a file, and there is a spec it should
meet. Read it loosely, let `inspect()` say everything that is wrong, decide
whether the file or the spec is at fault, and only then ask for the typed
frame.
```python
Path("customers.csv").write_text( # the file you were handed
"id,name,country,signed_up\n"
"1,Ada Lovelace,UK,2021-03-04\n"
"2,Grace Hopper,US,2019-12-31\n"
"2,Al,FR,2022-01-01\n"
)
```
**1. Read it without forcing the spec's types.**
```python
given = pl.read_csv("customers.csv", try_parse_dates=True)
```
`try_parse_dates=True` matters: a CSV has no date type, so without it
`signed_up` arrives as `String`, the report says only *expected Date, got
String*, and none of the column's own checks run. The other gaps between a
CSV and a spec take care of themselves -- a `String` column is accepted
where an `Enum` or `Categorical` is declared, an integer where a float or
`Decimal` is, and the values are checked either way.
Reading with the spec's schema is the tempting alternative, and the wrong
first step: Polars stops at the first value that does not fit, so one
error replaces the whole report.
```python
pl.read_csv("customers.csv", schema_overrides=Customers.schema())
# ComputeError: could not parse `FR` as dtype `enum` at column 'country'
```
**2. Ask what is wrong -- all of it.**
```python
report = Customers.inspect(given)
print(report)
```
```
Validation failed for DataFrame against 'Customers' (4 error(s) found):
- Column 'id': unique column contains 2 duplicate value(s). Duplicate samples: [2]
- Column 'name': found 1 value(s) with string length outside [3, 20]. Invalid samples: ['Al']
- Column 'country': found 1 invalid value(s) not in allowed choices/categories ['UK', 'US', 'DE']. Invalid samples: ['FR']
- Column 'signed_up': found 1 value(s) out of bounds [2020-01-01, 2026-01-01] (min found: 2019-12-31, max found: 2022-01-01). Out of bounds samples: [datetime.date(2019, 12, 31)]
```
`report.rows(finding)` is the offending rows, and `report.to_json()` is
something to send back to whoever sent the file:
```python
report.rows(report.by_code("choices")[0]).collect() # the row with country "FR"
```
**3. Decide which is wrong: the file, or the spec.** A bad row is fixed or
filtered at the source. A spec that has fallen behind -- `FR` is a real
country now -- is changed, and [`diff`](https://maxwellb13.github.io/polspec/how-to/drift/#two-declarations) says
whether the change is breaking for anything already validated against it:
```python
Widened = Customers.spec.with_columns(country=ColSpec(pl.Enum(["UK", "US", "DE", "FR"])))
Customers.diff(Widened).breaking # () -- widening a domain breaks nothing
```
While you are still finding out what the ranges really are,
`validate_bounds=False` checks everything but the bounds, and
[`from_dataframe`](https://maxwellb13.github.io/polspec/tutorial/getting-started/#infer-a-spec-instead-of-writing-one)
describes what the file actually holds, to compare against what the spec
says it should.
**4. Then take the typed frame.** Once it passes, `cast=True` returns each
column as its declared dtype -- the `Enum` an `Enum`, not the `String` the
CSV held -- so the code downstream reads the types the spec promises:
```python
corrected = given.filter(pl.col("name") == "Ada Lovelace") # the file, fixed
clean = Customers.validate(corrected, cast=True)
clean.schema["country"] # Enum(categories=['UK', 'US', 'DE'])
```
From the shell, `polspec validate customers.py customers.csv` does steps 1
and 2 in one go, parsing the columns the spec declares as dates and times;
see [the CLI](https://maxwellb13.github.io/polspec/how-to/cli/#validate-check-data-against-a-schema).
!!! note "What a CSV cannot hold"
A CSV has no way to write a `Duration`, a `List` or a `Struct`, so a
spec with one of those cannot be met by a CSV however it is read.
Parquet or Arrow IPC keeps every dtype.
## Options
```python
Orders.validate(
df,
extra_cols="raise", # "raise" | "drop" | "allow"
missing_cols="raise", # "raise" | "add" | "allow"
strict_dtypes=False,
cast=False,
streaming=False,
references=None,
validate_rules=True,
validate_validators=True,
validate_unique=True,
validate_checks=True,
validate_foreign_keys=True,
validate_hierarchy=True,
validate_pattern=True,
validate_bounds=True,
)
```
Every one of these is a field of `ValidationOptions`, which is what a report
carries as `report.options` — so a report says what it was asked to check, not
only what it found:
```python
report = Orders.inspect(df, validate_checks=False)
report.options.checks # False
report.options.extra_cols # "raise"
```
The `validate_*` switches are named for what they switch, so
`validate_checks` is `options.checks`. An option name polspec does not accept
is a `TypeError` naming the closest one it does.
You can also pass the whole set as one value, which is the shape to reach for
when the same settings go through several calls:
```python
from polspec import ValidationOptions
lenient = ValidationOptions(extra_cols="drop", checks=False)
for frame in (df, df.head(10)):
Orders.validate(frame, options=lenient)
```
`options=` and the individual keywords are alternatives, not a base and an
override — passing both raises rather than quietly picking one.
### Structural mismatches
`extra_cols` decides what happens to columns the spec does not declare —
refuse, drop them from the result, or keep them (appended after the declared
ones).
`missing_cols` decides what happens to declared columns the frame lacks —
refuse, add them as all-null, or ignore them.
!!! warning "`missing_cols="add"` can produce a frame that fails re-validation"
Columns are added *after* validation has run, including for columns
declared `nullable=False`. Feed the result straight back into `validate()`
and it will object to the nulls it just inserted.
### Dtype strictness
By default polspec accepts what a real pipeline delivers: any integer width for
a declared integer, an integer or float for a declared float, any temporal for
a temporal, and `String`/`Categorical` for a declared `Enum`. `strict_dtypes=True`
requires the exact dtype, treating only `String` and `Utf8` as interchangeable.
### Casting
`cast=True` casts each column to its declared dtype *after* validation passes,
so a String column that holds only valid enum members comes back as the `Enum`.
### Streaming
`streaming=True` evaluates with the Polars streaming engine, for frames larger
than memory.
### Turning checks off
Each `validate_*` switch disables a whole category of check. Most exist for
what generation cannot satisfy yet -- `__checks__`, validators and `pattern`
are validated but not generated:
```python
Orders.validate(Orders.generate(1_000, seed=1), validate_checks=False)
```
`validate_bounds=False` is the other way round: generation always stays in
bounds, so it is for real data -- a file whose ranges you want to look at
before holding it to them, while every other claim is still checked. It
covers every `bounds`, including a `List`'s elements and a struct's fields;
`string_length` and `list_length` have codes of their own and stay on.
```python
Orders.validate(df, validate_bounds=False)
```
To loosen one column rather than all of them, validate against a spec with
that column's bounds removed:
```python
import dataclasses
from polspec import validate
loose = Orders.spec.with_columns(
total=dataclasses.replace(Orders.col("total"), bounds=None)
)
validate(loose, df)
```
## What gets checked
| Check | From |
|:--|:--|
| Column present / not extra | the spec's column set |
| Dtype compatible | `ColSpec.dtype` |
| No unexpected nulls | `nullable` |
| Value in domain | `choices`, `Enum` categories |
| Value within range | `bounds` |
| Length within range | `string_length` |
| List has the declared number of elements | `list_length` |
| Value has the declared shape | `format` |
| Value matches the regex | `pattern` |
| Conditional values hold | `rules` |
| Single-column predicates | `validators` |
| Values distinct | `unique` |
| Composite key distinct | `__unique_together__` |
| Multi-column invariants | `__checks__` |
| Referential integrity | `__foreign_keys__` |
Bounds, lengths, rules, validators and uniqueness are skipped for a column
whose dtype is already wrong — comparing values of the wrong type would bury
the dtype error under noise.
## Foreign keys need their parent
A key referencing another spec needs that spec's data:
```python
Orders.validate(orders, references={Customers: customers})
```
Without it, the key is reported as a `foreign_key_unresolved` finding naming
the spec it needed, so `validate()` raises and `inspect()` lists it alongside
everything else. `references` may be keyed by the class, its `TableSpec`, or
the spec's name. Self-referencing keys are checked against the frame itself
and need nothing.
Each foreign key is an anti-join against the parent, so these run separately
from the single-pass aggregation above.
---
# Schema and data drift
Source: https://maxwellb13.github.io/polspec/how-to/drift/
# Schema and data drift
`validate()` says whether data meets its spec. Drift says *what moved* --
between two versions of a declaration, or between a declaration and the
data it is supposed to describe -- and whether each move would break
validation.
```python
from polspec.drift import diff, drift
report = diff(OldOrders, NewOrders) # two declarations
report = drift(Orders, df) # a declaration and a frame
```
Both are also methods on a spec: `OldOrders.diff(NewOrders)` and
`Orders.drift(df)`. Either way the result is a `DriftReport`.
## One rule for severity
Every finding is `breaking` or `compatible`, and the line between them is
mechanical: a finding is **breaking when a frame that satisfied the old side
could fail the new one**. For `drift()`, that means this frame fails this
spec on that column. Nothing else is a judgement call:
| Change | Severity | Because |
|:--|:--|:--|
| a bound or domain narrowed | breaking | values that validated before now fail |
| a bound or domain widened | compatible | nothing that passed now fails |
| a column added | breaking | a frame without it fails `missing_cols="raise"` |
| a column removed | breaking | a frame carrying it fails `extra_cols="raise"` |
| a constraint added (`unique`, a validator, a check, a key) | breaking | rows that passed may fail |
| a constraint removed | compatible | |
| `nullable` turned off | breaking | nulls that validated before now fail |
| a null rate that moved within a nullable column | compatible | the data still validates; the declaration describes it less well |
A column added is breaking under this rule, which surprises people in a pull
request. It is the honest answer -- validation of the old data would fail --
and the CLI's `--fail-on` is how a team decides what to gate on.
## Two declarations
```python
import datetime as dt
class OrdersV1(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, None), unique=True)
status = ColSpec(pl.Enum(["NEW", "PAID"]))
total = ColSpec(pl.Float64, bounds=(0.0, 1_000.0))
placed = ColSpec(pl.Date, bounds=(dt.date(2024, 1, 1), dt.date(2025, 1, 1)))
class OrdersV2(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, None), unique=True)
status = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"])) # widened
total = ColSpec(pl.Float64, bounds=(0.0, 500.0)) # narrowed
placed = ColSpec(pl.Date) # widened
channel = ColSpec(pl.Enum(["web", "store"])) # added
report = OrdersV1.diff(OrdersV2)
print(report)
```
```
Drift: 2 breaking, 3 compatible, 'OrdersV1' and 'OrdersV2'
- [breaking] Column 'channel' added; a frame without it fails missing_cols='raise'
- [breaking] Column 'total': domain narrowed from bounds [0.0, 1000.0] to bounds [0.0, 500.0]; values that validated before now fail
- [compatible] Column 'status': dtype changed from Enum(categories=['NEW', 'PAID']) to Enum(categories=['NEW', 'PAID', 'SHIPPED'])
- [compatible] Column 'status': domain widened from one of ['NEW', 'PAID'] to one of ['NEW', 'PAID', 'SHIPPED']; values that failed before are now accepted
- [compatible] Column 'placed': domain widened from bounds [2024-01-01, 2025-01-01] to any Date; values that failed before are now accepted
```
Widened and narrowed are decided by the same `Domain` comparison a foreign
key uses at declaration, run in both directions. A change that is neither --
`choices=["A", "B"]` to `["B", "C"]`, or `format="email"` to `"uuid4"` -- is
`domain_changed`, and breaking.
A rename is not guessed from similar names. Say it, and it is reported as
one `column_renamed` finding instead of a column removed and another added:
```python
class Renamed(FrameSpec):
order_ref = ColSpec(pl.Int64, bounds=(1, None), unique=True)
status = ColSpec(pl.Enum(["NEW", "PAID"]))
total = ColSpec(pl.Float64, bounds=(0.0, 1_000.0))
placed = ColSpec(pl.Date, bounds=(dt.date(2024, 1, 1), dt.date(2025, 1, 1)))
assert [f.code for f in OrdersV1.diff(Renamed, renames={"order_id": "order_ref"})] == [
"column_renamed"
]
```
## A declaration and data
`drift()` asks the frame the questions the spec makes answerable: is
anything outside the declared bounds, and by how much; which values are
outside the declared `choices`, `Enum` or `format`; which declared values
never appear; has the null rate moved.
```python
good = OrdersV1.generate(2_000, seed=1)
assert OrdersV1.drift(good).unchanged
moved = good.with_columns(
total=pl.col("total") * 3,
placed=pl.col("placed") + pl.duration(days=200),
status=pl.lit("PAID").cast(pl.String),
)
print(OrdersV1.drift(moved))
```
```
Drift: 2 breaking, 2 compatible, DataFrame against 'OrdersV1'
- [breaking] Column 'total': values escape bounds [0.0, 1000.0]: max found 2999.36861541796 by 1999.36861541796 above. Widen the bounds, or fix the source
- [breaking] Column 'placed': values escape bounds [2024-01-01, 2025-01-01]: max found 2025-07-20 by 200 days above. Widen the bounds, or fix the source
- [compatible] Column 'status': holds String, declared Enum(categories=['NEW', 'PAID'])
- [compatible] Column 'status': 1 of 2 declared value(s) never appear: ['NEW']
```
The generated frame drifts by nothing, and that is pinned by a test: what
`generate()` produces never drifts breakingly from the spec that produced
it, the same round trip validation is held to. The other direction is pinned
too -- every breaking finding is a column `validate()` would report -- so
`breaking` never means more than "validation fails here".
A struct column is compared field by field, in both directions: each
field its `fields` describes -- or its dtype alone, where `fields` says
nothing -- goes through every comparison a column does, and a finding names
it by path. Describing `point.lat` with bounds is `domain_narrowed` on
`point.lat`, breaking for the same reason it is on a column; data whose
`lat` escapes those bounds is `bounds_exceeded` on `point.lat`. A field's
null rate is measured inside the structs that are present, which is what
its `null_probability` claims. The finding's `columns` stay `("point",)`.
What `drift()` does **not** measure is what validation already does:
uniqueness, composite keys, foreign keys and checks are pass/fail claims
about rows, not summaries that move. `validate()` is still the verdict.
### Options
```python
from polspec import DriftOptions
lenient = DriftOptions(null_rate_tolerance=0.2, unseen_values=False)
OrdersV1.drift(good, options=lenient)
OrdersV1.drift(good, null_rate_tolerance=0.2) # or one keyword at a time, not both
```
| Option | Default | Meaning |
|:--|:--|:--|
| `null_rate_tolerance` | `0.05` | how far the observed null rate may sit from `null_probability` before `null_rate_moved` is reported; absolute, not relative |
| `unseen_values` | `True` | report declared values the data never holds (`cardinality_moved`) |
| `strict_dtypes` | `False` | the same switch as validation's, decided by the same function: whether a `dtype_changed` is breaking |
| `max_samples` | `10` | how many offending values a finding's `details` carry |
## The report
A `DriftReport` is data first. `report.breaking` and `report.compatible`
are the two halves; `by_column()` and `by_code()` slice it; `to_dict()` and
`to_json()` serialise it; `bool(report)` is `report.unchanged`, the way
`bool(ValidationReport)` is `passed`.
`to_markdown()` renders the shape of a pull-request comment, breaking
findings first:
```python
text = OrdersV1.diff(OrdersV2).to_markdown()
assert text.index("## Breaking") < text.index("## Compatible")
```
The finding codes are listed with the validation codes in
[Errors and findings](https://maxwellb13.github.io/polspec/reference/errors/#drift-codes).
---
# Shared categories
Source: https://maxwellb13.github.io/polspec/how-to/categories/
# Shared categories
A `CatSpec` is a registry of `Enum` and `Categorical` definitions shared across
specs, so several tables agree on a domain instead of each restating it.
## Declaring one by hand
Subclass `CatSpec`, one line per entry, in the same vocabulary `ColSpec.dtype`
already accepts:
```python
import polars as pl
from polspec import CatSpec, ColSpec, FrameSpec
class Categories(CatSpec):
STATUS = pl.Enum(["NEW", "PAID", "SHIPPED"])
CURRENCY = pl.Categorical(pl.Categories("CURRENCY", physical=pl.UInt8))
```
Naming an entry gives back its dtype, so it plugs straight into a `ColSpec`:
```python
class Orders(FrameSpec):
status = ColSpec(Categories.STATUS)
currency = ColSpec(Categories.CURRENCY)
```
This is deliberately not a dict. `CatSpec(enums={...}, categoricals={...},
choices={...})` puts one name across up to three parallel mappings that all
have to stay in step; a class body puts each entry on its own line, in the
declaration order that also documents it.
The entries are lifted out of the class body before the class exists — the same
thing `FrameSpec` does with `ColSpec` columns — so an entry may be named
anything, including a name `CatSpec` already uses:
```python
class TrickyNames(CatSpec):
get = pl.Enum(["A", "B"]) # an entry, not a collision
TrickyNames.get # still the method
TrickyNames.spec.get("get") # pl.Enum(["A", "B"])
```
An unnamed `pl.Categorical()` is rejected outright, since a registry entry with
no name can't act as a shared key.
`Categories.spec` is the `CatSpec` value the class body declares. Anywhere a
registry is expected — `with_catspec`, `Registry(categories=...)` — the class
and the value are interchangeable.
## The dict constructor
The form `CatSpec.infer()`, `from_dataframe()` and `from_yaml()` build
programmatically, since their entry names come from data at runtime rather
than from a class body someone writes by hand:
```python
categories = CatSpec(
enums={"STATUS": ["NEW", "PAID", "SHIPPED"]},
categoricals={"CURRENCY": pl.Categories("CURRENCY", physical=pl.UInt8)},
)
```
The two forms compose rather than compete: a class-body subclass's entries
become the defaults, and an explicit `enums=`/`categoricals=`/`choices=`
argument at construction time can still add to or override them per key.
```python
extended = Categories(enums={"REASON": ["FRAUD", "DUPLICATE"]})
extended.get_enum("STATUS") # ["NEW", "PAID", "SHIPPED"] -- inherited
extended.get_enum("REASON") # ["FRAUD", "DUPLICATE"] -- added
```
They also compare equal when they say the same thing, so a registry loaded from
a file can be checked against the one a class body declares:
```python
Categories.spec == CatSpec.from_yaml("categories.yaml")
```
## Using a registry
Whichever form built it, the accessors are the same, and naming an entry always
means the same thing: the dtype.
```python
categories.STATUS # -> pl.Enum([...])
categories.CURRENCY # -> pl.Categorical(...)
categories["STATUS"] # -> the same dtype
categories.get("STATUS") # -> the same dtype, or None
```
Ask for the pieces underneath when you want them rather than the dtype:
```python
categories.get_enum("STATUS") # -> list[str]
categories.get_categorical("CURRENCY") # -> pl.Categories
categories.get_choices("CURRENCY") # -> the domain pool, or None
```
And name the kind when you want the lookup to insist on it — these refuse an
entry of the other kind instead of quietly returning it:
```python
categories.enum.STATUS # -> pl.Enum
categories.enum["STATUS"] # item access
categories.enum("STATUS") # callable
categories.categorical.CURRENCY # -> pl.Categorical
```
Lookup falls back to case variants, so a column named `status` finds a registry
entry named `STATUS`. Convenient, but worth knowing about if you have entries
differing only in case.
## Why a shared `Categories` matters
A named `pl.Categories()` registry gives two columns the same physical codes,
so frames can be joined on the code rather than the string. polspec preserves
that identity through generation and through a YAML round-trip.
Choosing a narrow physical dtype is a real memory saving on wide tables:
| Physical | Distinct categories |
|:--|:--|
| `UInt8` | 255 |
| `UInt16` | 65,535 |
| `UInt32` (default) | ~4 billion |
polspec respects the ceiling: a `Categorical` on a `UInt8` registry generates
from a pool sized to the registry rather than overflowing it. Where the
registry is *named*, that pool is derived from the registry's own identity, so
two specs sharing it draw from the same domain.
## Building a registry from what you have
```python
CatSpec.from_dataframe(df) # existing Enum/Categorical columns
CatSpec.from_framespec(Orders) # a spec's declared columns
```
## Inferring one
`infer` picks a representation per column by cardinality:
```python
categories = CatSpec.infer(df, max_enum_cardinality=30)
```
- at most `max_enum_cardinality` distinct values → `Enum`
- otherwise, up to `max_categorical_cardinality` and either a low unique ratio
or under 256 values → `Categorical` with the narrowest physical dtype that fits
- otherwise, left as `String`
Identifier-shaped names are skipped by default, since they are high-cardinality
by nature: `*_id`, `*_uuid`, `*_hash`, `*_url`, `*_key`. Override with
`exclude_patterns`, or force specific columns with `include_columns`.
## Re-typing a spec
`with_catspec` returns a new spec with matching columns re-pointed at the
registry's types:
```python
Optimized = Orders.with_catspec(categories)
Optimized = Orders.with_catspec(CatSpec.infer(df)) # infer, then apply
```
Re-typing changes the dtype and nothing else a column declared — `unique`,
`string_length`, `nullable`, tags, rules and validators all carry over. The two
fields a dtype change can genuinely invalidate are dropped with a warning:
`weights`, which is positional over a domain that just resized, and `choices`,
when the new dtype has no category for them.
## Persisting a registry
```python
categories.to_yaml("categories.yaml")
categories = CatSpec.from_yaml("categories.yaml")
```
```yaml
enums:
STATUS: [NEW, PAID, SHIPPED]
categoricals:
CURRENCY:
name: CURRENCY
physical: UInt8
categories: [GBP, USD, EUR]
```
A spec's YAML can point at a registry file, and `FrameSpec.from_yaml` resolves
it automatically — see [YAML specs](https://maxwellb13.github.io/polspec/how-to/files/).
Also available: `to_markdown()` for a documentation table, and `to_mermaid()`
for a class diagram of the registry.
---
# Specs as files
Source: https://maxwellb13.github.io/polspec/how-to/files/
# YAML specs
A spec can live in a file instead of a class body, so tooling outside Python
can read it and so it can be reviewed as a document.
```python
Orders.to_yaml("orders.yaml")
Loaded = FrameSpec.from_yaml("orders.yaml")
Loaded.generate(1_000, seed=1)
```
The output is plain, readable YAML — defaults are omitted so the file shows
only what you actually declared:
```yaml
version: 3
name: Orders
columns:
order_id:
dtype: Int64
bounds: [1, 100000]
unique: true
status:
dtype:
Enum: [NEW, PAID, SHIPPED]
total:
dtype: Float64
bounds: [0.0, null]
placed:
dtype: Date
nullable: true
unique_together:
- [order_id, status]
```
An open-ended bound writes as `null` and reads back unchanged.
`version:` records the file format that wrote the file. A file from an
earlier version is migrated on the way in; a file from a later polspec is
refused with a message saying so. A key the reader does not know is an
error naming the closest known key, since silently reading a misspelt option
as its default is the worst outcome -- pass `strict=False` to
`from_yaml` to downgrade that to a warning.
## What survives a round-trip
| | Round-trips |
|:--|:--:|
| dtypes, including parametrized `Enum` / `Datetime` / `Duration` / named `Categorical` | yes |
| `nullable`, `null_probability`, `bounds`, `string_length`, `list_length`, `fields`, `format`, `pattern`, `seed_name`, `unique`, `tags` | yes |
| `choices`, `weights`, `distribution`, `distribution_params` | yes |
| `rules` (`ColRule`) | yes |
| `__unique_together__` | yes |
| `__foreign_keys__`, self-referencing or to another spec (by name) | yes |
| `__checks__` and `ColSpec.validators` written with `col()` | yes |
| `__checks__` and `ColSpec.validators` over a raw `pl.Expr` | **no** |
A foreign key to another spec is written as that spec's *name*; nothing
checks it until a spec of that name is supplied, through `references=` on
`generate`/`validate` or a registry. What cannot be written is an arbitrary
`polars.Expr`, which Polars cannot serialize stably. `to_yaml()` warns about
each, naming exactly what will be lost:
```text
UserWarning: Orders declares 1 __checks__ ('total_covers_subtotal') that cannot
be represented in YAML (a Check wraps an arbitrary polars.Expr) and will NOT be
written to orders.yaml. They will be lost on FrameSpec.from_yaml() unless
re-declared on a subclass of the loaded spec.
```
The suggested recovery is to subclass what you loaded:
```python
Loaded = FrameSpec.from_yaml("orders.yaml")
class Orders(Loaded):
__checks__ = [Check(pl.col("total") >= pl.col("subtotal"), name="total_covers_subtotal")]
```
Columns, rules, unique keys, foreign keys, and any check or validator
written with `col()` come from the file; only raw-expression parts need
re-declaring in Python.
A check in YAML is its predicate in data form:
```yaml
checks:
- expr:
ge:
- col: total
- col: subtotal
name: total_covers_subtotal
```
## Sharing categories between files
A spec file can reference a `CatSpec` registry by path, resolved relative to
the spec file:
```yaml
name: Orders
categories: categories.yaml
columns:
status:
dtype:
Enum: STATUS
currency:
dtype:
Categorical: CURRENCY
```
`$categories.STATUS` and `categories.STATUS` are accepted as prefixed forms of
the same reference.
Or pass a registry explicitly, which wins over anything the file names:
```python
FrameSpec.from_yaml("orders.yaml", categories=CatSpec.from_yaml("categories.yaml"))
FrameSpec.from_yaml("orders.yaml", categories="categories.yaml")
```
A spec can also emit the registry its own columns imply:
```python
Orders.catspec().to_yaml("categories.yaml")
```
## Python instead of YAML
The same spec can be written as an importable Python module. It is the right
choice when the spec will be edited by hand from now on, or when it needs the
parts YAML cannot hold:
```python
Orders.to_python("orders_spec.py")
```
```python
"""Declares the Orders schema."""
import polars as pl
from polspec import ColSpec, FrameSpec
class Orders(FrameSpec):
__columns__ = {
'order_id': ColSpec(pl.Int64, bounds=(1, 100000), unique=True),
'status': ColSpec(pl.Enum(['NEW', 'PAID', 'SHIPPED'])),
'total': ColSpec(pl.Float64, bounds=(0.0, None)),
'placed': ColSpec(pl.Date, nullable=True),
}
__unique_together__ = [['order_id', 'status']]
```
Columns are declared through `__columns__` because a name straight from data
is not always a valid identifier. What survives is exactly the
[round-trip table](#what-survives-a-round-trip) above: `__checks__`,
cross-spec `ForeignKey`s and `ColSpec.validators` warn and are dropped, and
the file is where you then add them back by hand. `polspec schema infer` uses
this path when its output ends in `.py`; see
[Command line](https://maxwellb13.github.io/polspec/how-to/cli/).
## Column names from data
`from_yaml` and `to_python` declare columns through `__columns__`, so names
that could not be class attributes — a leading underscore, a collision with a
method name like `schema`, or a name with spaces — load correctly. The YAML
key is the column's real name; a `col_name` set in a class body is not
written, because the key already carries it. See
[Column names that are not identifiers](https://maxwellb13.github.io/polspec/how-to/columns/#column-names-that-are-not-identifiers).
## Several specs in one file
A [`Registry`](https://maxwellb13.github.io/polspec/how-to/registry/) writes every spec it holds, and the categories it
was declared with, to one file keyed by spec name, and reads it back with the
same version and strictness rules:
```python
Registry(Customers, Orders, OrderLines, categories=categories).to_yaml("specs.yaml")
registry = Registry.from_yaml("specs.yaml").resolve()
```
---
# Multiple specs
Source: https://maxwellb13.github.io/polspec/how-to/registry/
# Multiple specs
A `ForeignKey` names the spec it points at, and a single spec knows nothing
beyond that name. A `Registry` is the declared set of specs that belong
together: it resolves every cross-spec key, orders parents before children,
generates or validates the whole set in one call, and draws the relationships
between them.
```python
from polspec import Registry
registry = Registry(Customers, Orders, OrderLines)
registry.names # ('Customers', 'Orders', 'OrderLines')
registry["Orders"] # the TableSpec, by name, class or spec
registry.order() # parents first: ('Customers', 'Orders', 'OrderLines')
```
A registry is declared, not global. Two test modules may each define an
`Orders`, and neither sees the other's. Two *different* specs with one name in
the same registry is an error.
## Resolving names
A key declared against a class is checked at declaration: the referenced
columns exist and their dtypes are compatible. A key declared against a bare
name, or read from a file, is not — nothing knows what `"Orders"` is yet.
`resolve()` binds every such key to the spec of that name and runs the same
checks, returning a new registry:
```python
resolved = registry.resolve()
resolved["OrderLines"].foreign_keys[0].target # the Orders TableSpec
```
It raises `RegistryError` for a key whose target is not in the registry, for
a reference to a column the target lacks or cannot hold, for a cycle between
specs, and for a column disagreeing with the shared categories described
below.
## Generating a related set
`generate_all` walks the foreign-key graph, generates each parent before its
children, and threads every frame into `references=` for you:
```python
frames = registry.generate_all(1_000, seed=1)
frames["Orders"]["customer_id"] # every value exists in frames["Customers"]["id"]
```
`n` is one count for every spec or a mapping with each spec's own:
```python
frames = registry.generate_all(
{Customers: 1_000, Orders: 10_000, OrderLines: 30_000}, seed=1
)
```
Each spec's seed is derived from `seed` and the spec's name, so adding a
table to the registry never changes the rows another table produces. A frame
passed in `references=` is used as-is instead of being generated — real
customers under synthetic orders — and also stands in for a parent that is
not in the registry at all.
`generate_related(Orders, n)` is the same walk restricted to one spec and
everything it depends on.
## Validating a related set
```python
reports = registry.inspect_all(frames) # {name: ValidationReport}
registry.validate_all(frames) # raises once, listing every spec's findings
```
Every frame is a possible parent for every other, so no `references=` is
needed for keys inside the set; pass one for parents that live outside it.
Both take the options `validate()` does, and `validate_all` returns the frames
with the same structural transformations applied. A frame for a spec not in
the registry is a `RegistryError`.
## Shared categories
Declaring the registry with a `CatSpec` says which `Enum` and `Categorical`
definitions the specs are expected to share; `resolve()` then refuses a column
whose declaration disagrees with it:
```python
registry = Registry(Orders, Products, categories=categories)
registry.resolve() # RegistryError if Orders.status and categories.STATUS differ
```
Without one, `catspec()` derives a registry from the specs' own columns and
refuses two specs that define the same name differently — the disagreement
[Shared categories](https://maxwellb13.github.io/polspec/how-to/categories/) warns about, now noticed:
```python
registry.catspec() # CatSpec merged from every Enum/Categorical column
```
## One file for the set
A registry writes to a single YAML file: the format version, the declared
categories, and every spec keyed by name. Foreign keys are written as names
and bound again by `resolve()` on the way back:
```yaml
version: 3
categories:
enums:
STATUS: [NEW, PAID, SHIPPED]
specs:
Customers:
columns:
id: {dtype: Int64, unique: true}
Orders:
columns:
customer_id: {dtype: Int64}
status: {dtype: {Enum: STATUS}}
foreign_keys:
- {columns: [customer_id], references: Customers, ref_columns: [id]}
```
```python
registry.to_yaml("specs.yaml")
registry = Registry.from_yaml("specs.yaml").resolve()
```
`categories:` may also be a path to a `CatSpec` file, relative to the registry
file. Everything [YAML specs](https://maxwellb13.github.io/polspec/how-to/files/) says about what survives a round-trip
applies to each spec in the file.
## Finding specs
`Registry.discover()` builds one from files and directories. A `.py` file is
imported and every `FrameSpec` subclass or `TableSpec` bound in it is taken;
a `.yaml` file is a spec, or a whole registry when it has a `specs:` key; a
directory is walked for both, skipping names starting with `_` or `test_`:
```python
registry = Registry.discover("specs/")
registry = Registry.from_module(my_project.specs)
```
Importing a Python file runs it, so point `discover` only at files you would
import anyway.
## The whole picture
`to_mermaid()` draws every spec and every key between them in one
entity-relationship diagram — the relationships a single spec's
[`to_mermaid`](https://maxwellb13.github.io/polspec/how-to/documenting/#entity-relationship-diagram) cannot see:
```python
registry.to_mermaid("docs/schema.mmd")
```
```mermaid
erDiagram
Customers {
Int64 id PK
}
Orders {
Int64 order_id PK
Int64 customer_id FK
}
OrderLines {
Int64 order_id UK
Int32 line_no UK
}
Customers ||--o{ Orders : "fk_customer_id__Customers"
Orders ||--o{ OrderLines : "fk_order_id__Orders"
```
---
# Hierarchies
Source: https://maxwellb13.github.io/polspec/how-to/hierarchies/
# Hierarchies and link tables
A link table is an edge list: one column holds a reference, the other holds
the reference it points at, and both draw on the same set of values. A child
pointing at its parent, and that parent pointing at its own parent, are the
same row shape — which is why one table holds both.
`Hierarchy` declares that shape, so `generate()` produces a real tree and
`validate()` can say when data is not one.
## Declaring one
```python
class Links(FrameSpec):
PARENT_REF = ColSpec(pl.String)
CHILD_REF = ColSpec(pl.String)
__hierarchy__ = Hierarchy(
child="CHILD_REF",
parent="PARENT_REF",
max_depth=5,
)
df = Links.generate(1_000_000, seed=7)
```
`child` is the column doing the pointing and `parent` the column being
pointed at. A row is an edge, so `n` is the number of rows. Ultimate parents
have nothing to point at and so have no row of their own.
What that gives you:
- **One parent per reference.** Every value in `CHILD_REF` appears exactly
once, so walking upward from any row reaches exactly one ultimate parent.
- **A known depth.** No chain is longer than `max_depth` hops, and at least one
chain is exactly that long — so a test of "resolve to the ultimate parent"
always exercises the boundary rather than whatever the draw happened to give.
- **No cycles**, unless you ask for them.
`branching` controls the shape — the mean number of children a reference has,
and so how many ultimate parents `n` rows imply. `roots` is the same dial from
the other end: give an exact number of ultimate parents instead, and the
branching follows. They are mutually exclusive, and the default is
`branching=3.0`.
```python
Hierarchy(child="CHILD_REF", parent="PARENT_REF", max_depth=5, roots=40)
```
## Generating data that is deliberately broken
Code that walks a hierarchy has to cope with data that is not one. A loop in
the parent chain is the case that matters; because a resolver written without
a visited set does not fail on it — it runs forever.
```python
Links.generate(1_000_000, seed=7, cycles=10)
Links.generate(1_000_000, seed=7, self_references=5)
```
`cycles=n` closes that many chains into loops, each a few hops long and none
overlapping another. `self_references=n` points that many rows straight at
themselves, which is the degenerate loop your resolver most likely tests for
and the one a `child == parent` guard already catches.
Neither is declared on the spec, because the spec still says the data *should*
be an acyclic forest. That is what makes them useful: `validate()` reports what
was injected, so a test can assert its own resolver and polspec agree.
```python
broken = Links.generate(10_000, seed=7, cycles=3)
report = Links.inspect(broken)
report.passed # False
[f.code for f in report.findings] # ['hierarchy_cycle']
report.rows(report.findings[0]).collect().height # the rows that never terminate
```
## What validation checks
Three findings, on top of everything a column declares for itself:
| code | meaning |
|:--|:--|
| `hierarchy_multi_parent` | a reference appears as a child in more than one row, so it has no single ultimate parent |
| `hierarchy_cycle` | a row's chain of parents never reaches an ultimate parent |
| `hierarchy_depth` | a chain is longer than the declared `max_depth`, without being a loop |
A row inside a loop is reported as a cycle rather than as a depth violation:
it is both, and the loop is the useful half. `hierarchy_cycle` counts every
row that never terminates, which includes rows hanging *below* a loop — they
do not reach an ultimate parent either.
Both checks are bounded. Depth costs `max_depth` steps; cycle detection walks
by pointer doubling, so it covers a chain of a million rows in about twenty.
Validating cyclic data terminates — which matters, because the whole point of
generating it was to have something that breaks a naive walk.
## Resolving the ultimate parent
The reason to generate this data is to test code that collapses it. The same
bounded walk works in Polars:
```python
links = Links.generate(50_000, seed=7)
resolved = links.select(
node=pl.col("CHILD_REF"), root=pl.col("PARENT_REF")
)
lookup = links.select(
pl.col("CHILD_REF").alias("root"), pl.col("PARENT_REF").alias("next")
)
for _ in range(5): # max_depth
resolved = (
resolved.join(lookup, on="root", how="left")
.with_columns(root=pl.coalesce("next", "root"))
.drop("next")
)
```
After `max_depth` rounds every `root` is an ultimate parent, because the spec
promised no chain is longer than that. Run the same loop over a frame from
`cycles=10` and it silently returns a value that is not a root — which is
exactly the bug worth having a fixture for.
## What it will not do
- **`generate_batches` and the `sink_*` functions refuse a spec with a
hierarchy.** Each batch is generated independently, so a batched hierarchy
would be a pile of unrelated fragments. Use `generate()` and write the frame
out yourself.
- **The pass owns both columns.** A `null_probability`, `distribution` or
`weights` on either is not what you get: the references have to come from one
pool for the two columns to join at all.
- **One parent, not many.** A reference with two parents is a finding, not a
supported shape.
---
# Generated documentation
Source: https://maxwellb13.github.io/polspec/how-to/documenting/
# Generated documentation
A spec already holds everything a data dictionary needs, so polspec renders one
rather than asking you to keep a second copy in step.
## Markdown data dictionary
```python
Orders.to_markdown("docs/orders.md") # writes and returns
markdown = Orders.to_markdown() # just returns
```
The document has three parts: an overview, a table of every column, and — when
the spec declares any — a constraints section covering composite keys, checks,
foreign keys, conditional rules and column validators.
```markdown
# Orders
## Overview
- **Schema:** `Orders`
- **Total Columns:** 4
- **Composite Unique Keys:** `['order_id', 'status']`
- **Foreign Keys:** 1 key(s)
## Columns
| Column | Type | Nullable | Bounds | Domain / Choices | String Length | Tags | Rules | Unique |
|:---|:---|:---|:---|:---|:---|:---|:---|:---|
| `order_id` | `Int64` | No | [1, 100000] | - | - | - | - | Yes |
| `status` | `Enum(['NEW', 'PAID', 'SHIPPED'])` | No | - | - | - | - | - | No |
| `total` | `Float64` | No | >= 0.0 | - | - | - | - | No |
```
Long category and choice lists are elided rather than blowing out the table,
and an open-ended bound reads as `>= 0.0` rather than `[0.0, None]`.
Pass `title=` to override the heading, which otherwise uses the class name.
## Entity-relationship diagram
```python
Orders.to_mermaid("docs/orders.mmd")
```
```mermaid
erDiagram
Orders {
Int64 order_id PK
Enum status
Float64 total "bounds: >= 0.0"
Date placed "nullable"
}
Customers ||--o{ Orders : "fk_customer_id__Customers"
```
Columns are annotated with what the spec declares — nullability, bounds or
choices, tags, string lengths — and keyed as `PK` (a `unique` column), `UK` (a
member of a composite key) or `FK`.
Mermaid renders in GitHub, GitLab and most documentation sites, including this
one, so the diagram stays live rather than becoming a stale screenshot.
A lone `unique=True` column is the entity's `PK`; when several columns are
unique each is marked `UK`, as is every member of a `__unique_together__`
group, and a foreign-keyed column `FK`.
## Several specs in one diagram
A single spec's diagram can only name the entity a key points at. A
[`Registry`](https://maxwellb13.github.io/polspec/how-to/registry/) draws every spec and every key between them:
```python
Registry(Customers, Orders, OrderLines).to_mermaid("docs/schema.mmd")
```
## Documenting a category registry
`CatSpec` renders the same two ways:
```python
categories.to_markdown("docs/categories.md")
categories.to_mermaid("docs/categories.mmd")
```
The Markdown lists enums with their variants and categoricals with their
physical dtype, namespace and domain pool. The Mermaid output is a class
diagram, with each enum as an `<>`.
## Keeping generated docs current
Both renderers are pure functions of the spec, so wiring them into a build or a
pre-commit hook keeps the documentation honest:
```python
from pathlib import Path
for spec in (Customers, Orders, Shipments):
spec.to_markdown(Path("docs/schemas") / f"{spec.__name__.lower()}.md")
```
---
# Test pipelines
Source: https://maxwellb13.github.io/polspec/how-to/testing/
# Testing pipelines with polspec
A spec is a schema and a data source at once, which makes it a natural fit
for the tests around a data pipeline: declare what a stage of the pipeline
expects, generate data that matches, and validate what it produces.
## Why this fits hermetic tests
A hermetic test doesn't reach outside itself — no network call, no shared
fixture file that drifts, no "works on my machine" because someone's local
`sample_data.csv` is newer than the one in CI. `FrameSpec.generate(n, seed=...)`
is a pure function of its arguments: the same seed produces the same frame on
any machine, in any process, with any number of threads. There's no file to
check into the repo, and no file to go stale.
```python
class Customers(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
tier = ColSpec(pl.Enum(["free", "pro", "enterprise"]))
signed_up = ColSpec(pl.Date, bounds=(date(2020, 1, 1), None))
def test_pipeline_handles_all_tiers():
df = Customers.generate(500, seed=42)
assert set(df["tier"].unique()) <= {"free", "pro", "enterprise"}
```
The spec is the fixture. When the pipeline's input schema changes, the type
error is in the `ColSpec` declaration, not in a `.parquet` file nobody
remembers generating.
## Testing a full pipeline
Declare the shape of each stage — including the *output* — and validate the
real function against it. This catches two different kinds of drift: the
pipeline producing the wrong shape, and the test's own expectations going
stale.
```python
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, None))
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
amount = ColSpec(pl.Float64, bounds=(0.0, 500.0))
__foreign_keys__ = [
ForeignKey("customer_id", references=Customers, ref_columns="customer_id")
]
class CustomerSpend(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
tier = ColSpec(pl.Enum(["free", "pro", "enterprise"]))
total_spend = ColSpec(pl.Float64, bounds=(0.0, None))
order_count = ColSpec(pl.UInt32)
def summarize_spend(customers: pl.DataFrame, orders: pl.DataFrame) -> pl.DataFrame:
"""The pipeline under test."""
return (
orders.group_by("customer_id")
.agg(
total_spend=pl.col("amount").sum(),
order_count=pl.len().cast(pl.UInt32),
)
.join(customers.select("customer_id", "tier"), on="customer_id", how="inner")
.select("customer_id", "tier", "total_spend", "order_count")
)
def test_summarize_spend_matches_declared_output_shape():
customers = Customers.generate(200, seed=1)
orders = Orders.generate(2_000, seed=2, references={Customers: customers})
Orders.validate(orders, references={Customers: customers})
result = summarize_spend(customers, orders)
CustomerSpend.validate(result, extra_cols="allow", missing_cols="allow")
```
`references={Customers: customers}` makes `orders.customer_id` referentially
consistent with the generated `customers` frame, so the join in
`summarize_spend` isn't silently testing against orphaned rows. Validating
the *input* and the *output* against separate specs means a pipeline bug that
drops a column, or a schema change nobody updated the test for, both surface
as a specific, readable `ValidationError` rather than a downstream assertion
failure three functions later.
For a pipeline with more than two stages — raw events into a bronze table,
bronze into a cleaned silver table, silver into an aggregated gold table — the
same pattern repeats at each boundary: a `FrameSpec` per stage, a
`ForeignKey` where one stage's identity flows into the next, `validate()`
between every pair of stages the tests actually exercise.
## Large dataframes and files
Generating a realistic volume of data for a load or performance test doesn't
need a large fixture file checked into version control. `generate_batches`
streams rows without holding all of them in memory:
```python
def test_pipeline_handles_a_million_rows_without_holding_them_all():
total = 0
for batch in Customers.generate_batches(1_000_000, batch_size=100_000, seed=1):
total += process(batch).height
assert total == 1_000_000
```
For a pipeline stage that specifically reads from a file — a `scan_parquet`
step, an ingestion job watching a directory — `sink_*` writes a large file to
a `tmp_path`, which pytest cleans up automatically:
```python
def test_pipeline_reads_a_large_parquet_file(tmp_path):
path = tmp_path / "customers.parquet"
Customers.sink_parquet(path, 2_000_000, batch_size=200_000)
result = pl.scan_parquet(path).select(pl.len()).collect().item()
assert result == 2_000_000
```
Nothing here is committed to the repository, nothing needs cleaning up by
hand, and the file is exactly as large as the test needs — a different test
asking for 50,000,000 rows costs nothing to write.
## Edge-case testing
### Guaranteed coverage with `method="cartesian"`
Random generation might never happen to produce a negative amount paired with
a particular payment method in 50 rows. `method="cartesian"` guarantees every
combination of each `Enum`/`Boolean` value with the negative/zero/positive/null
partitions of every bounded numeric column appears at least once:
```python
class Payment(FrameSpec):
method = ColSpec(pl.Enum(["card", "wire", "cash"]))
amount = ColSpec(pl.Int64, bounds=(-1000, 1000), nullable=True)
def refund_flag(df: pl.DataFrame) -> pl.DataFrame:
"""The pipeline under test: refunds are negative amounts."""
return df.with_columns(is_refund=pl.col("amount") < 0)
def test_refund_flag_handles_every_sign_and_method_combination():
edge_cases = Payment.generate(50, method="cartesian", seed=1)
result = refund_flag(edge_cases)
assert result.filter(pl.col("amount") < 0)["is_refund"].all()
assert not result.filter(pl.col("amount") >= 0)["is_refund"].any()
```
Every method now appears alongside a negative amount, a zero amount, a
positive amount, and a null — the sign/null boundary a naive `amount < 0`
check is actually at risk of getting wrong — without hand-writing sixteen
rows.
### Forcing a specific case with `ColRule`
Cartesian coverage guarantees signs and combinations exist somewhere in the
frame; it doesn't put a specific value on a specific row. When a test needs an
exact scenario — "a wire transfer of exactly zero, paired with this other
column's exact value" — a `ColRule` pins it deterministically instead of
filtering generated rows and hoping one matches:
```python
from polspec import ColRule, col
class PaymentWithForcedCase(FrameSpec):
method = ColSpec(pl.Enum(["card", "wire", "cash"]))
amount = ColSpec(
pl.Int64,
bounds=(-1000, 1000),
rules=[ColRule(when=col("method") == "wire", choices=[0])],
)
def test_zero_amount_wire_transfer_is_not_a_refund():
df = PaymentWithForcedCase.generate(20, seed=1)
result = refund_flag(df)
assert not result.filter(pl.col("method") == "wire")["is_refund"].any()
```
Every `wire` row is forced to `amount = 0`, while `card` and `cash` still vary
normally — useful for a boundary the pipeline treats specially and cartesian
coverage alone wouldn't reliably isolate.
## Generating the test boilerplate
The [`polspec test`](https://maxwellb13.github.io/polspec/how-to/cli/) command builds the round-trip skeleton for a
schema automatically:
```bash
polspec test orders.yaml -o test_orders.py
```
Point it at a spec written by hand, or one produced by `polspec schema infer`
against a sample of real production data — a fast way to turn "here's what our
data actually looks like" into a schema you can generate more of.
## A caveat, not a footnote
polspec is early alpha — see [Roadmap and stability](https://maxwellb13.github.io/polspec/explanation/roadmap/).
Tests built on it today are exercising real, useful properties (shape,
referential integrity, boundary coverage), but the exact values a given seed
produces are not guaranteed to survive a polspec upgrade. Pin a seed for
*reproducibility within a test run*, not as an assertion baked into a snapshot
that expects byte-identical output after you bump the version.
---
# Command line
Source: https://maxwellb13.github.io/polspec/how-to/cli/
# Command line
`polspec` does at the shell what a spec does in Python: create one from data,
turn one into a test, generate data from one, check data against one, and
say what moved. Every verb is a thin wrapper over a `FrameSpec` method that
already exists — `from_dataframe`, `to_yaml`, `generate`, `validate`, `diff`,
`drift` — so the CLI is argument parsing and templating, not new behaviour.
```bash
polspec schema infer orders.parquet -o orders.yaml
polspec schema new Orders -o orders.py
polspec test orders.yaml -o test_orders.py
polspec generate orders.yaml -n 1000 -o orders.parquet --seed 1
polspec validate orders.yaml orders.parquet
polspec diff orders_v1.yaml orders_v2.yaml --markdown
polspec drift orders.yaml orders.parquet
```
## `schema infer` — profile data into a spec
```bash
polspec schema infer SOURCE -o OUTPUT.yaml [options]
polspec schema infer SOURCE -o OUTPUT.py [options]
```
`SOURCE` is a `.csv`, `.tsv`, `.parquet`, `.ndjson`/`.jsonl`, `.json`, or Arrow
IPC (`.arrow`/`.ipc`/`.feather`) file. It is read with the matching Polars
reader and profiled with `FrameSpec.from_dataframe`, the same function behind
[Getting started](https://maxwellb13.github.io/polspec/tutorial/getting-started/#infer-a-spec-instead-of-writing-one).
`OUTPUT`'s extension picks the format: `.yaml`/`.yml` writes a YAML spec via
`FrameSpec.to_yaml`; `.py` writes a `FrameSpec` subclass via
`FrameSpec.to_python` — a starting-point module you can edit like any other
source file, rather than a data file `from_yaml` re-parses.
```console
$ polspec schema infer orders.parquet -o orders.yaml --weights
Inferred 3 column(s) from 12,483 row(s) of orders.parquet -> orders.yaml
$ polspec schema infer orders.parquet -o orders.py --weights
Inferred 3 column(s) from 12,483 row(s) of orders.parquet -> orders.py
```
```python
"""Declares the Orders schema."""
import polars as pl
from polspec import ColSpec, FrameSpec
class Orders(FrameSpec):
__columns__ = {
"order_id": ColSpec(pl.Int64, bounds=(1, 12483)),
"status": ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]), weights=[0.4, 0.3, 0.3]),
"total": ColSpec(pl.Float64, bounds=(10.0, 500.0)),
}
```
Columns are declared through `__columns__` rather than as class attributes,
same as `from_yaml` — see [Column names that are not
identifiers](https://maxwellb13.github.io/polspec/how-to/columns/#column-names-that-are-not-identifiers). The `.py`
output is passed through `ruff format` when it's on `PATH`, same as `schema
new`.
| Option | Effect |
|:--|:--|
| `--name NAME` | Class name (default: derived from the file name) |
| `--weights` | Record each category's observed frequency |
| `--max-unique-enum N` | Max distinct values for a string column to become an `Enum` (default 50) |
| `--no-bounds` | Skip computing numeric/temporal bounds and string lengths |
| `--sample N` | Profile only the first N rows |
Treat the output as a draft. It describes the sample it saw — edit bounds,
add rules, tighten a domain — before trusting it as a contract.
## `schema new` — start from nothing
```bash
polspec schema new NAME -o OUTPUT.py
```
Writes a blank `FrameSpec` with the two imports it will need and a few
commented `ColSpec` examples, for the case where there's no data yet to
profile.
## `test` — a round-trip test from a schema
```bash
polspec test SOURCE -o OUTPUT_test.py [options]
```
`SOURCE` is a `.yaml`/`.yml` spec (from `schema infer`, or written by
`FrameSpec.to_yaml`) or a `.py` file defining one or more `FrameSpec`
subclasses (from `schema new`, filled in). The generated file asserts the
property this project's own test suite is built around:
```python
def test_orders_roundtrip():
df = Orders.generate(500, seed=42)
Orders.validate(df)
def test_orders_cartesian_coverage():
df = Orders.generate(500, method="cartesian", seed=42)
Orders.validate(df)
```
| Option | Effect |
|:--|:--|
| `--rows N` | Rows to generate (default 500) |
| `--seed N` | Generation seed (default 42) |
| `--no-cartesian` | Skip the coverage-guaranteeing test |
| `--class NAME` | Generate a test for only this class, when the source defines several |
### It will not hand you a test that fails on the spot
`generate()` does not attempt everything `validate()` checks — see
[Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/). A spec using `__checks__` or
`ColSpec.validators` would otherwise generate a test that fails the moment it
runs, because both wrap arbitrary expressions nothing can be generated to
satisfy. The generator checks for each and disables the corresponding
`validate()` flag, with a comment explaining why:
```python
def test_invoices_roundtrip():
# __checks__ wraps arbitrary expressions that generation cannot be made
# to satisfy
df = Invoices.generate(500, seed=42)
Invoices.validate(df, validate_checks=False)
```
`unique=True` and `__unique_together__` used to be on that list. They are
generated now, so the generated test validates them like anything else.
A spec with a foreign key referencing *another* spec needs that spec's data
via `references=`, which the CLI cannot supply on its own — that test is
marked `@pytest.mark.skip` with a reason, rather than guessed at:
```python
@pytest.mark.skip(
reason=(
"Child has foreign key(s) 'fk_parent_id__Parent' referencing another "
"FrameSpec. generate()/validate() need a parent DataFrame via "
"references={OtherSpec: parent_df} -- see "
"docs/how-to/constraints.md#referential-integrity-foreignkey."
)
)
def test_child_roundtrip():
pass
```
Similarly, the cartesian test is only emitted when the spec actually has
something for `method="cartesian"` to build coverage from — an `Enum`,
`Boolean`, or bounded numeric column. A spec of only unbounded strings gets a
comment instead of a test that would raise `ValueError` on the first run.
### Regenerating
The generated file names the command that made it:
```python
"""Generated by `polspec test orders.yaml`.
Regenerate with:
polspec test orders.yaml -o test_orders.py
This file is only overwritten by running that command again -- edit freely.
"""
```
It is a plain file, not managed state — add assertions, rename the functions,
delete the parts you don't want. Nothing re-reads it.
## `generate` — data from a schema
```bash
polspec generate orders.yaml -n 1000 -o orders.parquet --seed 1
polspec generate specs.py --class Orders -n 500 -o orders.csv --references Customers=customers.parquet
polspec generate orders.yaml -n 50 -o edge_cases.ndjson --method cartesian
```
Generates `-n` rows and writes one file; the extension picks the format
(`.parquet`/`.pq`, `.csv`, `.tsv`, `.ndjson`/`.jsonl`, `.json`,
`.arrow`/`.ipc`/`.feather` — the same set `validate` and `drift` read).
`--seed` makes the file reproducible; `--method cartesian` guarantees
coverage the way [`generate()`](https://maxwellb13.github.io/polspec/how-to/generating/#coverage-methodcartesian)
does; `--references NAME=PATH` supplies parent data for a foreign key, as
for `validate`.
The frame is built in memory and written once. For a file too large to hold,
the streaming [`sink_*`](https://maxwellb13.github.io/polspec/how-to/generating/#writing-straight-to-a-file) functions
are a Python surface. A CSV or TSV cannot hold a `Duration`, `List` or
`Struct` column; Polars refuses to write one, so use Parquet or Arrow IPC
for a spec that has any.
### `--all` — every spec in a directory
```bash
polspec generate --all specs/ -n 1000 -o data/ --seed 1
polspec generate --all specs/ -n 1000 -o data/ --format csv
```
With `--all`, `SPEC` is a directory (or any file `Registry.discover`
accepts) and `-o` is a directory: every spec found is generated, **parents
first with their keys threaded into their children** as
[`Registry.generate_all`](https://maxwellb13.github.io/polspec/how-to/registry/#generating-a-related-set) does, and
written as `.` (Parquet by default). No `--references` are
needed for keys between the discovered specs; supply them for a parent
outside the directory.
## `validate` — check data against a schema
```bash
polspec validate orders.yaml orders.parquet
polspec validate specs.py orders.parquet --class Orders --references Customers=customers.parquet
polspec validate orders.yaml orders.csv --json > report.json
```
Reads a data file (CSV, Parquet, NDJSON or Arrow IPC), runs
[`inspect()`](https://maxwellb13.github.io/polspec/how-to/validating/#findings-as-data-inspect) against the spec, and
prints the report: the same text `validate()` would raise, or the full
structured report with `--json`. The exit status is 0 when the data passes
and 1 when it does not, so a spec can gate a pipeline step in CI with no
Python at all.
`--references NAME=PATH` supplies parent data for a foreign key to another
spec, by that spec's name; repeat it for several. `--allow-extra` and
`--allow-missing` relax the structural checks; `--strict-dtypes` tightens the
dtype check. `--skip CHECK` turns off one kind of check, as the matching
`validate_*=False` does in Python -- `--skip bounds --skip checks` -- and
takes any of `rules`, `validators`, `unique`, `checks`, `foreign_keys`,
`hierarchy`, `pattern` and `bounds`.
A CSV, TSV or JSON file has no date type, so a date arrives as text. The CLI
reads each column the spec declares as a `Date`, `Datetime` or `Time` as
that type when every value in it parses; a column holding a value that does
not stays text, and is reported as a `dtype` finding rather than turned into
a null. A `String` column of date-shaped text is left alone. `drift` reads
the same way, and `schema infer`, with no spec to go by, recognises a CSV's
dates itself.
### `--all` — every spec against the file named after it
```bash
polspec validate --all specs/ data/
polspec validate --all specs/ data/ --json > reports.json
```
With `--all`, `SPEC` is a directory of specs and `DATA` a directory of data
files named after them (`Orders.parquet` for `Orders`, in any format the
CLI reads). Every spec with a file is checked, **each seeing the others'
files as its parents**, as [`Registry.inspect_all`](https://maxwellb13.github.io/polspec/how-to/registry/#validating-a-related-set)
does; specs with no file are listed and skipped. The exit status is 1 when
any report fails; `--json` prints one report per spec, keyed by name.
## `diff` and `drift` — what moved
```bash
polspec diff orders_v1.yaml orders_v2.yaml # two schemas
polspec diff specs_v1.py specs_v2.py --class Orders --rename id=order_id
polspec drift orders.yaml last_night.parquet # a schema and data
polspec drift orders.yaml last_night.parquet --markdown > drift.md
```
`diff` runs [`diff()`](https://maxwellb13.github.io/polspec/how-to/drift/#two-declarations) between two spec files;
`drift` runs [`drift()`](https://maxwellb13.github.io/polspec/how-to/drift/#a-declaration-and-data) between a spec and
a data file. Both print the report as text, `--json`, or `--markdown` (the
shape of a pull-request comment, breaking findings first).
The exit status is decided by `--fail-on`: `breaking` (the default) exits
`1` when any finding would break validation; `any` exits `1` on any finding
at all, a widened bound included; `none` always exits `0`, for posting a
report without gating on it. So a schema change in a pull request, or a
nightly load, can be gated with no Python:
```bash
polspec diff main/orders.yaml pr/orders.yaml --markdown --fail-on breaking
```
`--strict-dtypes` makes any dtype change breaking, as it does for
`validate`. `drift` also takes `--null-rate-tolerance`, `--no-unseen`,
`--max-samples` and `--sample N` — see
[`DriftOptions`](https://maxwellb13.github.io/polspec/how-to/drift/#options).
### `drift --all` — every spec against the file named after it
```bash
polspec drift --all specs/ nightly/
polspec drift --all specs/ nightly/ --json > drift.json
```
As for `validate --all`: `SPEC` is a directory of specs and `DATA` a
directory of files named after them. Every spec with a file is measured,
specs without one are listed and skipped, and `--fail-on` decides the exit
status across all of them. `diff` compares two declarations rather than a
declaration and data, so it has no `--all`.
## Exit codes and errors
Every subcommand returns `0` on success and `1` on a reported error, printed
as `error: ...` on stderr rather than a traceback — a missing file, an
unreadable format, an invalid class name. `validate`, `diff` and `drift`
also return `1` when the report itself fails, so a `1` means "look at the
output", whichever kind of problem it was.
```console
$ polspec schema infer nope.csv -o out.yaml
error: no such file: nope.csv
```
---
# Overview
Source: https://maxwellb13.github.io/polspec/reference/api/
# API reference
Every name `polspec` exports, rendered from its own docstrings. A page here
cannot describe a signature the code does not have.
| Name | Page |
| --- | --- |
| [`ColSpec`][polspec.ColSpec] | [Columns](https://maxwellb13.github.io/polspec/reference/api/columns/) |
| [`Bound`][polspec.Bound] | [Columns](https://maxwellb13.github.io/polspec/reference/api/columns/) |
| [`ColRule`][polspec.ColRule] | [Columns](https://maxwellb13.github.io/polspec/reference/api/columns/) |
| [`Check`][polspec.Check] | [Columns](https://maxwellb13.github.io/polspec/reference/api/columns/) |
| [`col`][polspec.col] | [Predicates](https://maxwellb13.github.io/polspec/reference/api/predicates/) |
| [`Pred`][polspec.Pred] | [Predicates](https://maxwellb13.github.io/polspec/reference/api/predicates/) |
| [`TableSpec`][polspec.TableSpec] | [Specs](https://maxwellb13.github.io/polspec/reference/api/specs/) |
| [`FrameSpec`][polspec.FrameSpec] | [Specs](https://maxwellb13.github.io/polspec/reference/api/specs/) |
| [`ForeignKey`][polspec.ForeignKey] | [Specs](https://maxwellb13.github.io/polspec/reference/api/specs/) |
| [`Hierarchy`][polspec.Hierarchy] | [Specs](https://maxwellb13.github.io/polspec/reference/api/specs/) |
| [`Registry`][polspec.Registry] | [Registry](https://maxwellb13.github.io/polspec/reference/api/registry/) |
| [`CatSpec`][polspec.CatSpec] | [Registry](https://maxwellb13.github.io/polspec/reference/api/registry/) |
| [`generate`][polspec.generate] | [Generation](https://maxwellb13.github.io/polspec/reference/api/generation/) |
| [`generate_batches`][polspec.generate_batches] | [Generation](https://maxwellb13.github.io/polspec/reference/api/generation/) |
| [`scan`][polspec.scan] | [Generation](https://maxwellb13.github.io/polspec/reference/api/generation/) |
| [`sink_parquet`][polspec.sink_parquet] | [Generation](https://maxwellb13.github.io/polspec/reference/api/generation/) |
| [`sink_ipc`][polspec.sink_ipc] | [Generation](https://maxwellb13.github.io/polspec/reference/api/generation/) |
| [`sink_csv`][polspec.sink_csv] | [Generation](https://maxwellb13.github.io/polspec/reference/api/generation/) |
| [`sink_ndjson`][polspec.sink_ndjson] | [Generation](https://maxwellb13.github.io/polspec/reference/api/generation/) |
| [`inspect`][polspec.inspect] | [Validation](https://maxwellb13.github.io/polspec/reference/api/validation/) |
| [`validate`][polspec.validate] | [Validation](https://maxwellb13.github.io/polspec/reference/api/validation/) |
| [`ValidationOptions`][polspec.ValidationOptions] | [Validation](https://maxwellb13.github.io/polspec/reference/api/validation/) |
| [`ValidationReport`][polspec.ValidationReport] | [Validation](https://maxwellb13.github.io/polspec/reference/api/validation/) |
| [`Finding`][polspec.Finding] | [Validation](https://maxwellb13.github.io/polspec/reference/api/validation/) |
| [`DriftOptions`][polspec.DriftOptions] | [Drift](https://maxwellb13.github.io/polspec/reference/api/drift/) |
| [`DriftReport`][polspec.DriftReport] | [Drift](https://maxwellb13.github.io/polspec/reference/api/drift/) |
| [`DriftFinding`][polspec.DriftFinding] | [Drift](https://maxwellb13.github.io/polspec/reference/api/drift/) |
| [`profile_dataframe`][polspec.profile_dataframe] | [Profiling](https://maxwellb13.github.io/polspec/reference/api/profiling/) |
| [`PolspecError`][polspec.PolspecError] | [Errors](https://maxwellb13.github.io/polspec/reference/api/errors/) |
| [`SpecError`][polspec.SpecError] | [Errors](https://maxwellb13.github.io/polspec/reference/api/errors/) |
| [`GenerationError`][polspec.GenerationError] | [Errors](https://maxwellb13.github.io/polspec/reference/api/errors/) |
| [`ValidationError`][polspec.ValidationError] | [Errors](https://maxwellb13.github.io/polspec/reference/api/errors/) |
| [`MultiValidationError`][polspec.MultiValidationError] | [Errors](https://maxwellb13.github.io/polspec/reference/api/errors/) |
| [`SerializationError`][polspec.SerializationError] | [Errors](https://maxwellb13.github.io/polspec/reference/api/errors/) |
| [`RegistryError`][polspec.RegistryError] | [Errors](https://maxwellb13.github.io/polspec/reference/api/errors/) |
| [`CliError`][polspec.CliError] | [Errors](https://maxwellb13.github.io/polspec/reference/api/errors/) |
Anything not listed here is internal: it can change in a patch release without
a changelog entry.
---
# Columns
Source: https://maxwellb13.github.io/polspec/reference/api/columns/
# Columns
What one column declares, and the pieces that make up a declaration.
## ColSpec
### ColSpec(dtype: pl.DataType | type[pl.DataType], col_name: str | None=None, seed_name: str | None=None, nullable: bool=False, bounds: Bound[Any] | tuple[Any, Any] | list[Any] | None=None, tags: str | Sequence[str] | None=(), unique: bool=False, null_probability: float=_DEFAULT_NULL_PROBABILITY, string_length: Bound[int] | tuple[int, int] | list[int] | None=None, list_length: Bound[int] | tuple[int, int] | list[int] | None=None, fields: Mapping[str, ColSpec] | None=None, format: str | None=None, pattern: str | None=None, distribution: str | None=None, distribution_params: dict[str, float] | None=None, choices: Sequence[Any] | dict[Any, float] | None=None, weights: Sequence[float] | None=None, rules: Sequence[ColRule]=(), validators: Check | pl.Expr | Pred | Sequence[Check | pl.Expr | Pred] | None=())
One column's declaration: its type, and every claim made about its values.
A `ColSpec` is what `generate()` samples from and what `validate()` checks
against, so each field below is a claim both sides read.
Parameters
----------
dtype : pl.DataType | type[pl.DataType]
The data type of the column.
col_name : str | None, optional
Overrides the column's name in the generated/validated DataFrame.
Declaring columns as class attributes on a `FrameSpec` requires a
valid Python identifier, which cannot contain spaces or other special
characters -- `col_name` lets the attribute keep a clean Python name
(`unit_price`) while the actual column is named whatever the data uses
(`"Unit Price"`). Everything else that refers to this column by name --
`ColRule`, `unique_together`, tags lookups, `validate()` -- uses
`col_name`, not the attribute name.
seed_name : str | None, optional
The name the column's seed is derived from, when it is not the
column's own. Generation seeds each column from the frame seed and
the column *name*, so renaming a column changes the values it
produces; a column declared with `seed_name="old"` keeps producing
the data it did as `"old"`, its rules included. Nothing is promised
across polspec versions. Two columns of one spec cannot share a
seed name, nor may one name another column: they would draw
identical values.
nullable : bool, optional
Whether the column allows null values.
bounds : Bound | tuple | list | None, optional
The inclusive range of values allowed in the column, as a `Bound` or a
2-sequence. Only supported for numeric and temporal data types. Either
endpoint may be None to leave that side unconstrained --
`bounds=(0, None)` for a non-negative column, `bounds=(None, 0)` for a
non-positive one.
An open end means different things to the two consumers of this field,
deliberately. `validate()` treats it as genuinely unconstrained and
omits that half of the check. `generate()` cannot sample an unbounded
range, so it falls back to the same default it would use with no bounds
at all -- `bounds=(0, None)` on Int64 generates 0..1,000,000 while
validating any value >= 0. This mirrors how `bounds=None` already
behaves rather than adding a third rule.
tags : str | Sequence[str], optional
Tag or tags classifying the column, for later selection.
unique : bool, optional
Whether values in the column must be distinct. Generation draws the
column without replacement; nulls are exempt. Cannot be combined with
`weights`, a non-uniform `distribution`, or `rules`, none of which
survive a draw without replacement.
null_probability : float, optional
Probability of a value being null. Must be between 0 and 1, and only
has effect alongside `nullable=True` -- so that turning nullability
off does not also require deleting the rate beside it. Declaring a
rate of your own without `nullable=True` warns, since that reads as
asking for nulls rather than as a leftover.
string_length : Bound | tuple[int, int] | list[int] | None, optional
The inclusive range of string lengths, where that applies.
list_length : Bound | tuple[int, int] | list[int] | None, optional
For a `List` column, the inclusive range of elements a value holds;
generation defaults to 0..5. On a `List` or `Array` column every
other field that describes a value -- `bounds`, `choices`, `weights`,
`format`, `pattern`, `string_length`, `distribution` -- describes
each *element*; `nullable` and `null_probability` describe the list
itself, and generation never puts a null inside one. An `Array`
takes its length from the dtype and refuses `list_length`.
fields : Mapping[str, ColSpec] | None, optional
For a `Struct` column, what is claimed about each field's values:
a `ColSpec` per field, keyed by name. The dtype is the schema --
every field's name and type comes from it -- and `fields` is what
is claimed about the values within it, so a struct of twenty
fields where one needs bounds spells one field. A field the
mapping omits is generated from its dtype alone.
A field is a value, not a column: `unique`, `rules`, `validators`,
`seed_name` and `col_name` are refused on one, since each is a
statement about a column among columns. Everything that describes a
value is allowed -- `nullable` included, which says whether the
field may be null inside a struct that is present -- and `fields`
too, so a struct may nest to any depth. A finding about a field
names it: `point.lat`.
A `List` or `Array` of a `Struct` takes `fields` too: it describes
the element, as `bounds` and `format` already do.
format : str | None, optional
The shape a `String` column's values take, by name: `"uuid4"`,
`"email"`, `"ipv4"`, `"ipv6"`, `"mac"`, `"hostname"`,
`"iso_country"` or `"iso_currency"`. Generation fills the column
from that format's own sampler and validation checks every value
against it, so a column carrying `format="email"` is generated to
satisfy its own spec. A format owns the column's whole domain:
it cannot be combined with `choices` or `string_length`, and only a
`String` column can carry one. What it promises is syntax -- an
address that is well-formed, not one that is deliverable.
pattern : str | None, optional
A regular expression every value of a `String` column must match,
checked by validation only. Generation does not read it: a `String`
column with a pattern is filled with ordinary random text, and the
round trip that holds for every other field does not hold here --
the same boundary as `validators`. Prefer `format` for a shape
polspec can generate; use `pattern` for a shape it cannot. Cannot
be combined with `format`, which already is a pattern with a
sampler. Polars regex syntax; a pattern that does not compile is
refused at declaration.
distribution : str | None, optional
The name of the probability distribution for the column's values
(e.g. `"uniform"`, `"normal"`).
distribution_params : dict[str, float] | None, optional
Parameters specific to the chosen distribution.
choices : tuple | list | dict | None, optional
A finite set of allowed values. A dict maps each choice to its weight.
weights : tuple[float, ...] | list[float] | None, optional
Weights associated with `choices`, biasing selection probabilities.
rules : tuple[ColRule, ...], optional
Rules (`ColRule`) that overwrite the column's values on the rows their
condition matches.
validators : Check | pl.Expr | Pred | Sequence[...] | None, optional
A single-column business rule, or several: each either a `pl.Expr`
boolean predicate (referencing only this column) or a `Check` (for a
custom name, description or null handling). Unlike
`FrameSpec.__checks__`, these travel with the column's own declaration.
Examples
--------
>>> ColSpec(pl.Int64, bounds=(1, 100), nullable=True, null_probability=0.1)
>>> ColSpec(pl.String, choices=["NEW", "PAID"], weights=[3.0, 1.0])
## Bound
### Bound(min: 'T | None', max: 'T | None') -> None
An inclusive [min, max] range, used for numeric bounds, temporal ranges, and string lengths.
Either endpoint may be None, meaning that side is unconstrained -- see
`ColSpec.bounds`, the only field that accepts an open end.
- `Bound.closed(self) -> 'tuple[T, T]'` -- Both endpoints of a bound that has both, as `string_length` does.
## ColRule
### ColRule(when: 'Pred', choices: 'tuple', weights: 'tuple[float, ...] | None' = None) -> None
Restricts a column's generated values on rows where `when` matches.
Applied as a pass over the generated frame: rows where `when` matches get
a value resampled uniformly (or according to `weights`) from `choices`
instead of whatever was freely generated for them. Multiple rules on the
*same* column are checked in declaration order, first match wins (like
SQL CASE/WHEN).
`when` is evaluated against the frame as it stands when the rule runs,
and the passes run in dependency order: a rule keyed on a column that
another rule or a foreign key rewrites sees the rewritten values -- the
same values validation checks the rule against. Two columns whose rules
each read what the other writes have no such order and are rejected at
declaration.
`when` is a predicate built with `polspec.col()`, not an arbitrary polars
expression, so that every rule can round-trip through a spec file:
ColRule(when=col("region") == "UK", choices=["RoyalMail"])
ColRule(when=col("region").is_in(["US", "EU"]) & (col("qty") > 10), choices=["UPS"])
## Check
### Check(expr: pl.Expr | Pred, name: str | None=None, description: str | None=None, ignore_nulls: bool=True, pred: Pred | None=None)
A declarative multi-column validation constraint evaluated as a Polars boolean expression.
Parameters
----------
expr : pl.Expr | Pred
The boolean condition each row must satisfy: a Polars expression, or a
predicate built with `polspec.col()`. A predicate can be written to a
YAML or Python spec file; a raw expression cannot.
name : str | None, optional
A human-readable identifier for the check constraint (e.g. 'total_gte_subtotal').
If omitted, defaults to the string representation of the expression.
description : str | None, optional
An optional description detailing the business logic or rationale for this check.
ignore_nulls : bool, default True
Whether rows evaluating to null in the check condition are considered valid
(standard SQL CHECK constraint semantics). If False, null results are treated as failures.
Examples
--------
>>> check = Check(pl.col("total") >= pl.col("subtotal"), name="total_gte_subtotal")
---
# Predicates
Source: https://maxwellb13.github.io/polspec/reference/api/predicates/
# Predicates
`col()` builds the conditions a `ColRule` or a `Check` carries. Unlike a raw
`pl.Expr`, a predicate built this way survives a round trip through a spec
file -- see [Specs as files](https://maxwellb13.github.io/polspec/how-to/files/).
## col
### col(name: 'str') -> 'Col'
A reference to a column, the starting point of every predicate.
## Pred
### Pred() -> None
Base class of every predicate node. Build one with `col()`.
- `Pred.children(self) -> 'tuple[Pred, ...]'` -- The operand nodes this one is built from, in written order.
- `Pred.equals(self, other: 'object') -> 'bool'` -- Structural equality, since `==` builds a predicate.
- `Pred.is_between(self, lower: 'Any', upper: 'Any') -> 'Between'` -- A predicate true where this value falls within `[lower, upper]`.
- `Pred.is_in(self, values: 'Sequence[Any]') -> 'IsIn'` -- A predicate true where this value is one of `values`.
- `Pred.is_not_null(self) -> 'Not'` -- A predicate true where this value is present.
- `Pred.is_null(self) -> 'IsNull'` -- A predicate true where this value is null.
- `Pred.literals(self) -> 'list[Any]'` -- Every constant this predicate compares against.
- `Pred.rebuild(self, children: 'tuple[Pred, ...]') -> 'Pred'` -- This node with `children` in place of its own, same order.
- `Pred.rename(self, mapping: 'Mapping[builtins.str, builtins.str]') -> 'Pred'` -- The same predicate with its columns renamed by `mapping`.
- `Pred.root_names(self) -> 'set[builtins.str]'` -- Every column name this predicate reads.
- `Pred.to_data(self) -> 'Any'` -- This predicate as plain data, for writing to a spec file.
- `Pred.to_expr(self) -> 'pl.Expr'` -- This predicate as the Polars expression that evaluates it.
- `Pred.to_source(self) -> 'builtins.str'` -- This predicate as the `col(...)` Python that would rebuild it.
---
# Specs
Source: https://maxwellb13.github.io/polspec/reference/api/specs/
# Specs
A spec is a `TableSpec`: an immutable record of columns and constraints.
`FrameSpec` is the class syntax that builds one and forwards every verb to it.
## TableSpec
### TableSpec(name: 'str', columns: 'Mapping[str, ColSpec]' = , checks: 'Sequence[Check]' = (), unique_together: 'Sequence[Sequence[str]]' = (), foreign_keys: 'Sequence[ForeignKey]' = (), hierarchy: 'Hierarchy | None' = None) -> None
The columns and constraints of one table, as an immutable value.
Parameters
----------
name : str
What the table is called: the class name for a `FrameSpec`, the
`name:` key for a file. Foreign keys refer to a spec by this name.
columns : Mapping[str, ColSpec]
Column name to declaration, in the order columns should appear.
checks : Sequence[Check]
Multi-column invariants; see `FrameSpec.__checks__`.
unique_together : Sequence[Sequence[str]]
Composite unique keys. A single group may be given as a flat list.
foreign_keys : Sequence[ForeignKey]
Referential-integrity constraints.
hierarchy : Hierarchy | None
Declares two columns as a parent/child edge list; see `Hierarchy`.
Notes
-----
Everything a `FrameSpec` class body validates at declaration is validated
here, so a `TableSpec` that constructs is one that can be used.
- `TableSpec.drop(self, *names: 'str') -> 'TableSpec'` -- Removes columns, and any composite or foreign key that used them.
- `TableSpec.estimated_size(self, n: 'int') -> 'int'` -- Bytes a frame of `n` generated rows is expected to hold.
- `TableSpec.rename(self, mapping: 'Mapping[str, str]') -> 'TableSpec'` -- Renames columns, rewriting every constraint that names them.
- `TableSpec.resolve_target(self, fk: 'ForeignKey') -> 'TableSpec | None'` -- The spec a foreign key points at.
- `TableSpec.schema(self) -> 'pl.Schema'` -- The Polars schema this spec declares: column name to dtype.
- `TableSpec.select(self, *names: 'str') -> 'TableSpec'` -- Keeps only the named columns, in the order given.
- `TableSpec.tag(self, *tags: 'str | Sequence[str]', match: "Literal['any', 'all']" = 'any') -> 'list[str]'` -- Column names carrying any (or all) of the tags, in declaration order.
- `TableSpec.with_catspec(self, catspec: 'CatSpec | type[CatSpec]') -> 'TableSpec'` -- Re-points columns at the registry's Enum and Categorical types.
- `TableSpec.with_checks(self, *checks: 'Check') -> 'TableSpec'` -- A copy of this spec with `checks` added to the ones it has.
- `TableSpec.with_columns(self, mapping: 'Mapping[str, ColSpec] | None' = None, /, **columns: 'ColSpec') -> 'TableSpec'` -- Adds columns, or replaces existing ones of the same name in place.
- `TableSpec.with_foreign_keys(self, *foreign_keys: 'ForeignKey') -> 'TableSpec'` -- A copy of this spec with `foreign_keys` added to the ones it has.
- `TableSpec.with_hierarchy(self, hierarchy: 'Hierarchy | None') -> 'TableSpec'` -- A copy of this spec declaring (or, with None, dropping) a hierarchy.
- `TableSpec.with_name(self, name: 'str') -> 'TableSpec'` -- A copy of this spec under a different name.
- `TableSpec.with_unique_together(self, *groups: 'Sequence[str]') -> 'TableSpec'` -- A copy of this spec with `groups` added as composite unique keys.
## FrameSpec
### FrameSpec()
Base class for declaring a DataFrame/LazyFrame specification.
Subclass it and assign a `ColSpec` per column, in the order columns should
appear:
class DataSource(FrameSpec):
string_1 = ColSpec(pl.String)
enum_1 = ColSpec(pl.Enum(["mammal", "reptile"]), nullable=True)
int_1 = ColSpec(pl.Int64, bounds=(-100, 100), nullable=True)
df = DataSource.generate(1_000_000, seed=42)
The class body builds `DataSource.spec`, a `TableSpec`; every classmethod
here forwards to a function over it. A column may take any name: one that
collides with a method (`schema`, `tag`, ...) is reachable as
`DataSource.col("schema")` while the method keeps working. Names that
cannot be attributes at all -- a leading underscore, or one straight from
data -- go through `__columns__`:
class Raw(FrameSpec):
__columns__ = {"_id": ColSpec(pl.Int64), "Unit Price": ColSpec(pl.Float64)}
- `FrameSpec.catspec(cls) -> 'CatSpec'` -- The CatSpec registry this spec's Enum and Categorical columns imply.
- `FrameSpec.checks(cls) -> 'tuple[Check, ...]'` -- The Check constraints defined on this FrameSpec.
- `FrameSpec.col(cls, name: 'str') -> 'ColSpec'` -- The declaration of one column, whatever it is called.
- `FrameSpec.diff(cls, other: 'TableSpec | type[FrameSpec]', *, renames: 'Mapping[str, str] | None' = None, options: 'DriftOptions | None' = None) -> 'DriftReport'` -- What changed from this spec to `other`, as a `DriftReport`.
- `FrameSpec.drift(cls, df: 'pl.DataFrame | pl.LazyFrame', *, options: 'DriftOptions | None' = None, null_rate_tolerance: 'float | None' = None, unseen_values: 'bool | None' = None, strict_dtypes: 'bool | None' = None, max_samples: 'int | None' = None) -> 'DriftReport'` -- How `df` has moved relative to this spec, as a `DriftReport`.
- `FrameSpec.estimated_size(cls, n: 'int') -> 'int'` -- Bytes a frame of `n` generated rows is expected to hold.
- `FrameSpec.foreign_keys(cls) -> 'tuple[ForeignKey, ...]'` -- The ForeignKey constraints defined on this FrameSpec.
- `FrameSpec.from_dataframe(cls, df: 'pl.DataFrame', *, name: 'str' = 'ProfiledFrameSpec', weights: 'bool' = False, max_unique_enum: 'int' = 50, calculate_bounds: 'bool' = True) -> 'type[FrameSpec]'` -- Infers a spec by profiling an existing DataFrame.
- `FrameSpec.from_spec(cls, spec: 'TableSpec', *, name: 'str | None' = None) -> 'type[FrameSpec]'` -- A `FrameSpec` subclass wrapping an existing `TableSpec`.
- `FrameSpec.from_yaml(cls, source: 'str | Path', *, categories: 'CatSpec | type[CatSpec] | str | Path | None' = None, strict: 'bool' = True) -> 'type[FrameSpec]'` -- Builds a new FrameSpec subclass from a YAML file written by `to_yaml`.
- `FrameSpec.generate(cls, n: 'int', *, method: "Literal['random', 'cartesian']" = 'random', seed: 'int | None' = None, references: 'References' = None, cycles: 'int' = 0, self_references: 'int' = 0, max_bytes: 'int | None' = None) -> 'pl.DataFrame'` -- Generates a DataFrame matching this spec.
- `FrameSpec.generate_batches(cls, n: 'int', *, batch_size: 'int' = 100000, method: "Literal['random', 'cartesian']" = 'random', seed: 'int | None' = None, references: 'References' = None) -> 'Iterator[pl.DataFrame]'` -- Yields chunks of generated rows without holding all `n` in memory.
- `FrameSpec.inspect(cls, df: 'pl.DataFrame | pl.LazyFrame', *, options: 'ValidationOptions | None' = None, references: 'References' = None, extra_cols: "Literal['drop', 'allow', 'raise'] | None" = None, missing_cols: "Literal['add', 'allow', 'raise'] | None" = None, strict_dtypes: 'bool | None' = None, validate_rules: 'bool | None' = None, validate_validators: 'bool | None' = None, validate_unique: 'bool | None' = None, validate_checks: 'bool | None' = None, validate_foreign_keys: 'bool | None' = None, validate_hierarchy: 'bool | None' = None, validate_pattern: 'bool | None' = None, validate_bounds: 'bool | None' = None, cast: 'bool | None' = None, streaming: 'bool | None' = None) -> 'validation.ValidationReport'` -- Everything this spec has to say about `df`, as a `ValidationReport`.
- `FrameSpec.scan(cls, n: 'int', *, seed: 'int | None' = None, batch_size: 'int | None' = None, method: "Literal['random', 'cartesian']" = 'random', references: 'References' = None) -> 'pl.LazyFrame'` -- A `LazyFrame` of `n` rows, generated as they are collected.
- `FrameSpec.schema(cls) -> 'pl.Schema'` -- The Polars schema this spec declares: column name to dtype.
- `FrameSpec.sink_csv(cls, path: 'str | Path', n: 'int', *, batch_size: 'int' = 100000, include_header: 'bool' = True, method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, **kwargs: 'Any') -> 'None'` -- Generates `n` rows and streams them to a CSV file in batches.
- `FrameSpec.sink_ipc(cls, path: 'str | Path', n: 'int', *, batch_size: 'int' = 100000, compression: 'IpcCompression | None' = 'zstd', method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, **kwargs: 'Any') -> 'None'` -- Generates `n` rows and streams them to an Arrow IPC file in batches.
- `FrameSpec.sink_ndjson(cls, path: 'str | Path', n: 'int', *, batch_size: 'int' = 100000, method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, **kwargs: 'Any') -> 'None'` -- Generates `n` rows and streams them to an NDJSON file in batches.
- `FrameSpec.sink_parquet(cls, path: 'str | Path', n: 'int', *, batch_size: 'int' = 100000, compression: 'ParquetCompression' = 'zstd', method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, **kwargs: 'Any') -> 'None'` -- Generates `n` rows and streams them to a Parquet file in batches.
- `FrameSpec.tag(cls, *tags: 'str | Sequence[str]', match: "Literal['any', 'all']" = 'any') -> 'list[str]'` -- Column names carrying any (or all) of the tags, in declaration order.
- `FrameSpec.to_markdown(cls, path: 'str | Path | None' = None, *, title: 'str | None' = None) -> 'str'` -- A Markdown data dictionary for this spec, written to `path` if given.
- `FrameSpec.to_mermaid(cls, path: 'str | Path | None' = None, *, title: 'str | None' = None) -> 'str'` -- A Mermaid entity-relationship diagram for this spec.
- `FrameSpec.to_python(cls, source: 'str | Path') -> 'None'` -- Writes this spec as an importable Python module defining a subclass.
- `FrameSpec.to_yaml(cls, source: 'str | Path') -> 'None'` -- Writes this spec to a human-readable YAML file at `source`.
- `FrameSpec.unique_together(cls) -> 'tuple[tuple[str, ...], ...]'` -- The composite unique column groups defined on this FrameSpec.
- `FrameSpec.validate(cls, df: 'pl.DataFrame | pl.LazyFrame', *, options: 'ValidationOptions | None' = None, references: 'References' = None, extra_cols: "Literal['drop', 'allow', 'raise'] | None" = None, missing_cols: "Literal['add', 'allow', 'raise'] | None" = None, strict_dtypes: 'bool | None' = None, validate_rules: 'bool | None' = None, validate_validators: 'bool | None' = None, validate_unique: 'bool | None' = None, validate_checks: 'bool | None' = None, validate_foreign_keys: 'bool | None' = None, validate_hierarchy: 'bool | None' = None, validate_pattern: 'bool | None' = None, validate_bounds: 'bool | None' = None, cast: 'bool | None' = None, streaming: 'bool | None' = None) -> 'pl.DataFrame | pl.LazyFrame'` -- Validates a DataFrame or LazyFrame against this spec.
- `FrameSpec.with_catspec(cls, catspec: 'CatSpec | type[CatSpec]', *, name: 'str | None' = None) -> 'type[FrameSpec]'` -- A new FrameSpec subclass with columns re-typed against `catspec`.
## ForeignKey
### ForeignKey(columns: str | Sequence[str], references: type[FrameSpec] | TableSpec | str, ref_columns: str | Sequence[str] | None=None, name: str | None=None, target: TableSpec | None=None)
Declares referential integrity: one or more columns must only contain
values that exist in another FrameSpec's (or this same FrameSpec's) columns.
Parameters
----------
columns : str | Sequence[str]
The local column(s) that must reference existing parent values.
references : type[FrameSpec] | TableSpec | str
The spec this key references -- a `FrameSpec` subclass, a
`TableSpec`, or a spec's *name* -- or the literal string "self" for a
self-referencing key (an `employee.manager_id` pointing back at
`employee.id`). "self" always resolves to whichever spec the key ends
up declared or inherited on, not the class it was first written in.
After construction `references` is always a string: the target's
name. When a spec object was given, it is kept as `target`, so its
columns can be checked at declaration; a bare name has no `target`
until a registry resolves it.
ref_columns : str | Sequence[str] | None, optional
The referenced column(s) on the target, in the same order as
`columns`. Defaults to `columns` (same names on both sides).
name : str | None, optional
A human-readable identifier. Defaults to a name derived from the
columns and target.
Notes
-----
Rows where any of `columns` is null are exempt (standard FK semantics --
a null foreign key means "no reference", not "an invalid one").
Examples
--------
>>> class OrderSpec(FrameSpec):
... customer_id = ColSpec(pl.Int64)
... __foreign_keys__ = [
... ForeignKey("customer_id", references=CustomerSpec, ref_columns="id"),
... ]
>>> class EmployeeSpec(FrameSpec):
... id = ColSpec(pl.Int64, unique=True)
... manager_id = ColSpec(pl.Int64, nullable=True)
... __foreign_keys__ = [
... ForeignKey("manager_id", references="self", ref_columns="id"),
... ]
## Hierarchy
### Hierarchy(child: 'str', parent: 'str', max_depth: 'int' = 1, branching: 'float | None' = None, roots: 'int | None' = None) -> None
Declares that two columns of a spec form a parent/child edge list.
Parameters
----------
child : str
The column holding the lower reference -- the one doing the pointing.
Each value appears in exactly one row, which is what gives every
reference a single ultimate parent.
parent : str
The column holding the reference being pointed at.
max_depth : int
Hops from an ultimate parent to the furthest child. Generation
guarantees at least one chain of exactly this length and none longer,
so a walk that stops early and one that runs away are both caught.
branching : float | None
Mean children per reference, which is what decides how many ultimate
parents `n` rows imply. Mutually exclusive with `roots`; one of the two
must be given, and `branching=3.0` is the default when neither is.
roots : int | None
An exact number of ultimate parents, as an alternative to `branching`.
Notes
-----
A row is an edge, so `generate(n)` produces `n` rows. Ultimate parents have
no row of their own -- nothing to point at -- so `n` edges over `R` roots
need `n + R` distinct references, drawn from the child column's own
declaration.
Examples
--------
>>> class Links(FrameSpec):
... PARENT_REF = ColSpec(pl.String)
... CHILD_REF = ColSpec(pl.String)
... __hierarchy__ = Hierarchy(
... child="CHILD_REF", parent="PARENT_REF", max_depth=5
... )
---
# Registry and categories
Source: https://maxwellb13.github.io/polspec/reference/api/registry/
# Registry and categories
A declared set of specs, and the shared category domains they draw on.
## Registry
### Registry(*specs: 'TableSpec | type', categories: 'CatSpec | type[CatSpec] | None' = None) -> 'None'
A declared set of specs, with everything that needs more than one.
Parameters
----------
*specs : TableSpec | type[FrameSpec]
The specs, in any order. Each is stored under its name; two different
specs with one name are an error.
categories : CatSpec | None
A shared category registry the specs are expected to agree with.
When given, `resolve()` checks every `Enum`/`Categorical` column that
binds to one of its entries against it, and the registry file carries
it. When omitted, `catspec()` derives one from the specs themselves.
- `Registry.add(self, spec: 'TableSpec | type') -> 'Registry'` -- Adds a spec, returning the registry so calls chain.
- `Registry.ancestors(self, key: 'Any') -> 'tuple[str, ...]'` -- Every spec a spec depends on, directly or through other specs.
- `Registry.catspec(self) -> 'CatSpec'` -- The categories these specs share: the one declared, or one merged from every spec's Enum and Categorical columns.
- `Registry.discover(cls, *paths: 'str | Path', categories: 'CatSpec | type[CatSpec] | None' = None, strict: 'bool' = True) -> 'Registry'` -- Every spec found under the given files and directories.
- `Registry.from_dict(cls, data: 'Mapping[str, Any]', *, strict: 'bool' = True) -> 'Registry'` -- A registry read from the data form `to_dict` writes.
- `Registry.from_module(cls, module: 'ModuleType', *, own_only: 'bool' = False, categories: 'CatSpec | type[CatSpec] | None' = None) -> 'Registry'` -- Every `FrameSpec` subclass and `TableSpec` bound in a module.
- `Registry.from_yaml(cls, source: 'str | Path', *, strict: 'bool' = True) -> 'Registry'` -- A registry read from one YAML file written by `to_yaml`.
- `Registry.generate_all(self, n: 'int | Mapping[Any, int]', *, seed: 'int | None' = None, method: "Literal['random', 'cartesian']" = 'random', references: 'Frames | None' = None) -> 'dict[str, pl.DataFrame]'` -- One frame per spec, parents generated first and threaded into their children, so every foreign key is satisfied by construction.
- `Registry.generate_related(self, key: 'Any', n: 'int | Mapping[Any, int]', *, seed: 'int | None' = None, method: "Literal['random', 'cartesian']" = 'random', references: 'Frames | None' = None) -> 'dict[str, pl.DataFrame]'` -- `generate_all` restricted to one spec and everything it depends on.
- `Registry.inspect_all(self, frames: 'Frames', *, references: 'Frames | None' = None, **options: 'Any') -> 'dict[str, ValidationReport]'` -- A `ValidationReport` per frame, each spec seeing every other frame as a possible parent. Takes the options `validate()` does.
- `Registry.order(self) -> 'tuple[str, ...]'` -- Every spec name, parents before children.
- `Registry.parents(self, key: 'Any') -> 'tuple[str, ...]'` -- Names of the specs one spec's foreign keys point at, self excluded.
- `Registry.resolve(self) -> 'Registry'` -- A registry whose every cross-spec key is bound to its target.
- `Registry.scan_all(self, n: 'int | Mapping[Any, int]', *, seed: 'int | None' = None, batch_size: 'int | None' = None, references: 'Frames | None' = None) -> 'dict[str, pl.LazyFrame]'` -- One `LazyFrame` per spec, each generating as it is collected.
- `Registry.to_dict(self) -> 'dict[str, Any]'` -- This registry as plain data: every spec, plus shared categories.
- `Registry.to_mermaid(self, path: 'str | Path | None' = None) -> 'str'` -- One entity-relationship diagram with every spec and every key.
- `Registry.to_yaml(self, source: 'str | Path') -> 'None'` -- Writes every spec, and the declared categories, to one file.
- `Registry.validate_all(self, frames: 'Frames', *, references: 'Frames | None' = None, **options: 'Any') -> 'dict[str, pl.DataFrame | pl.LazyFrame]'` -- Validates every frame, or returns them with the structural transformations `validate()` applies.
## CatSpec
### CatSpec(*, enums: 'Mapping[str, Sequence[str]] | None' = None, categoricals: 'Mapping[str, pl.Categories | dict[str, Any] | str | pl.DataType] | None' = None, choices: 'Mapping[str, Sequence[Any]] | None' = None) -> 'None'
A set of shared `Enum` and `Categorical` domains, as a value.
Parameters
----------
enums : Mapping[str, Sequence[str]], optional
Entry name to its ordered category list.
categoricals : Mapping[str, pl.Categories | dict | str | pl.DataType], optional
Entry name to its shared `pl.Categories`. A physical dtype (`pl.UInt8`)
or its name (`"UInt8"`) is shorthand for a registry of that name; a
mapping is the file form, and its `categories` key becomes `choices`.
choices : Mapping[str, Sequence[Any]], optional
The pool of values an entry draws from when generating. An `Enum`'s
categories already are its pool; this is for `Categorical` entries,
whose registry names the domain without listing it.
Notes
-----
Naming an entry -- `cats.STATUS`, `cats["STATUS"]`, `cats.get("STATUS")` --
gives back the dtype, ready to hand to `ColSpec`. Lookup is
case-insensitive, so a column named `status` finds `STATUS`, and an `Enum`
wins over a `Categorical` of the same name.
A subclass of `CatSpec` declares its entries in the class body; see the
module docstring. Instantiating one takes those entries as defaults, so
`Categories(enums={"REASON": [...]})` extends rather than replaces.
Examples
--------
>>> cats = CatSpec(enums={"STATUS": ["PENDING", "COMPLETED"]})
>>> cats.STATUS
Enum(categories=['PENDING', 'COMPLETED'])
>>> cats.get_enum("status")
['PENDING', 'COMPLETED']
- `CatSpec.dtype_of(self, name: 'str') -> 'pl.DataType | None'` -- The dtype registered under `name`, or None if nothing is.
- `CatSpec.from_dataframe(cls, df: 'pl.DataFrame | pl.LazyFrame') -> 'CatSpec'` -- The `Enum` and `Categorical` columns a frame already declares.
- `CatSpec.from_dict(cls, data: 'dict[str, Any]', *, strict: 'bool' = True) -> 'CatSpec'` -- A registry read from the data form `to_dict` writes.
- `CatSpec.from_framespec(cls, spec: 'TableSpec | type[FrameSpec]') -> 'CatSpec'` -- The `Enum` and `Categorical` columns a spec already declares.
- `CatSpec.from_yaml(cls, source: 'str | Path', *, strict: 'bool' = True) -> 'CatSpec'` -- A registry read from a YAML file written by `to_yaml`.
- `CatSpec.get(self, name: 'str', default: 'Any' = None) -> 'Any'` -- The dtype registered under `name`, or `default` if nothing is.
- `CatSpec.get_categorical(self, name: 'str') -> 'pl.Categories'` -- The shared `pl.Categories` of a `Categorical` entry.
- `CatSpec.get_choices(self, name: 'str') -> 'list[Any] | None'` -- The pool of values an entry draws from, if it has one.
- `CatSpec.get_enum(self, name: 'str') -> 'list[str]'` -- The category list of an `Enum` entry.
- `CatSpec.infer(cls, target: 'pl.DataFrame | pl.LazyFrame | TableSpec | type[FrameSpec]', *, max_enum_cardinality: 'int' = 30, max_categorical_cardinality: 'int' = 10000, max_categorical_ratio: 'float' = 0.2, include_columns: 'Sequence[str] | None' = None, exclude_patterns: 'Sequence[str] | None' = ('(?:^|.*_)id$', '(?:^|.*_)uuid$', '(?:^|.*_)hash$', '(?:^|.*_)url$', '(?:^|.*_)key$'), default_physical: 'pl.DataType | None' = None) -> 'CatSpec'` -- A registry of the domains `target` looks like it has.
- `CatSpec.resolve_key(self, name: 'str') -> 'tuple[Kind, str] | None'` -- Which entry a name binds to, if any.
- `CatSpec.to_dict(self) -> 'dict[str, Any]'` -- This registry as plain data, without the file's `version` key.
- `CatSpec.to_markdown(self, path: 'str | Path | None' = None, *, title: 'str | None' = None) -> 'str'` -- A Markdown table of every entry; written to `path` when given.
- `CatSpec.to_mermaid(self, path: 'str | Path | None' = None, *, title: 'str | None' = None) -> 'str'` -- A Mermaid class diagram of every entry; written to `path` when given.
- `CatSpec.to_yaml(self, source: 'str | Path | None' = None) -> 'str | None'` -- Writes this registry as YAML to `source`, or returns the text.
---
# Generation
Source: https://maxwellb13.github.io/polspec/reference/api/generation/
# Generation
Every function here takes a `TableSpec` as its first argument, and every one
has a `FrameSpec` classmethod that forwards to it with `cls.spec` -- see
[Generating data](https://maxwellb13.github.io/polspec/how-to/generating/) for what the options mean and
[Specs as values](https://maxwellb13.github.io/polspec/how-to/tablespec/) for when to reach for which.
## generate
### generate(spec: 'TableSpec', n: 'int', *, method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, cycles: 'int' = 0, self_references: 'int' = 0, max_bytes: 'int | None' = None) -> 'pl.DataFrame'
Generates a DataFrame matching `spec`.
method="random" (default): `n` rows, each column drawn independently.
method="cartesian": guarantees a minimum level of coverage. Builds the
cartesian product of every Enum/Boolean column's full set of values,
crossed with the negative/zero/positive/null partitions of every bounded
numeric column, so every enum combination appears alongside every numeric
sign/null case. `n` is then a *minimum*: if that coverage set has fewer
than `n` rows it is padded with random rows; if it has more, all of it is
kept.
`ColSpec.rules` and any `ForeignKey` the spec declares are then applied as
vectorised passes over the generated frame, regardless of method. Each
pass sees the frame the passes before it produced, and they run in the
order their reads and writes imply -- a rule keyed on a foreign-keyed
column reads the parent's values, not the freely generated ones they
replaced -- so the result satisfies the same declarations `validate`
checks it against.
A foreign key is only made referentially consistent where data for its
target is available: self-referencing keys always are, sampled from this
same frame; a key referencing another spec only is if `references`
carries an entry for it, keyed by the spec, its class, or its name --
otherwise that column is left exactly as freely generated. Composite keys
are sampled as one joint pick per row; a single-column key whose ColSpec
is `unique=True` samples without replacement when the parent has enough
distinct rows to cover `n`.
A `unique=True` column is drawn without replacement by the engine itself,
and a `__unique_together__` group is separated afterwards by resampling
the rows that repeat a combination. Either refuses, naming the column or
the group, when the domain is too small to cover `n`.
A spec declaring a `Hierarchy` has its two link columns rewritten as a
forest of the declared depth. `cycles` and `self_references` then damage it
on purpose -- closing that many chains into loops, and pointing that many
rows at themselves -- which is how a graph walk gets something to fail
against. Both default to zero, and `validate()` reports whatever they
injected.
Before anything is allocated the frame's size is estimated from the
declaration. Past four gibibytes that is a warning naming the estimate;
`max_bytes=` makes it a refusal instead, and `max_bytes=0` silences both.
The whole frame is built before this returns. `scan()` is the lazy
verb: it generates as the plan is collected, so only the columns and
rows a plan asks for are made.
## generate_batches
### generate_batches(spec: 'TableSpec', n: 'int', *, batch_size: 'int' = 100000, method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None) -> 'Iterator[pl.DataFrame]'
Yields chunks of generated rows without holding all `n` in memory.
Each batch is a window onto the one frame `seed` describes: a column no
pass rewrites holds, batch by batch, exactly the rows `generate(n, seed=seed)`
would, whatever `batch_size` is. What is drawn per batch instead --
deterministic, but not row for row the whole frame's -- is a column with
rules, a foreign key, a composite key, and a List column's elements.
Uniqueness only holds *within* a batch, not across the whole `n`: that
applies to a `unique=True` column, a `__unique_together__` group, and a
foreign-key column sampled without replacement alike.
## scan
### scan(spec: 'TableSpec', n: 'int', *, seed: 'int | None' = None, batch_size: 'int | None' = None, method: 'Method' = 'random', references: 'References' = None) -> 'pl.LazyFrame'
A `LazyFrame` of `n` generated rows, produced as they are collected.
`batch_size` left unset lets polars ask for the size it would like, so a
sink gets the batches it writes best; setting it pins the size whatever
polars asks.
## estimated_size
### estimated_size(spec: 'TableSpec', n: 'int') -> 'int'
Bytes `generate(spec, n)` is expected to hold, as whole bytes.
See the module docstring for what this does and does not count.
## sink_parquet
### sink_parquet(spec: 'TableSpec', path: 'str | Path', n: 'int', *, batch_size: 'int' = 100000, compression: 'ParquetCompression' = 'zstd', method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, **kwargs: 'Any') -> 'None'
Generates `n` rows and streams them to a Parquet file in batches.
Extra keyword arguments go to `pl.LazyFrame.sink_parquet`.
## sink_ipc
### sink_ipc(spec: 'TableSpec', path: 'str | Path', n: 'int', *, batch_size: 'int' = 100000, compression: 'IpcCompression | None' = 'zstd', method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, **kwargs: 'Any') -> 'None'
Generates `n` rows and streams them to an Arrow IPC / Feather file in batches.
Extra keyword arguments go to `pl.LazyFrame.sink_ipc`.
## sink_csv
### sink_csv(spec: 'TableSpec', path: 'str | Path', n: 'int', *, batch_size: 'int' = 100000, include_header: 'bool' = True, method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, **kwargs: 'Any') -> 'None'
Generates `n` rows and streams them to a CSV file in batches.
Extra keyword arguments go to `pl.LazyFrame.sink_csv`.
## sink_ndjson
### sink_ndjson(spec: 'TableSpec', path: 'str | Path', n: 'int', *, batch_size: 'int' = 100000, method: 'Method' = 'random', seed: 'int | None' = None, references: 'References' = None, **kwargs: 'Any') -> 'None'
Generates `n` rows and streams them to a newline-delimited JSON file in batches.
Extra keyword arguments go to `pl.LazyFrame.sink_ndjson`.
---
# Validation
Source: https://maxwellb13.github.io/polspec/reference/api/validation/
# Validation
`inspect()` returns a report; `validate()` raises one. Both carry the same
findings as data -- see [Validating data](https://maxwellb13.github.io/polspec/how-to/validating/).
## inspect
### inspect(spec: 'TableSpec', df: 'pl.DataFrame | pl.LazyFrame', *, options: 'ValidationOptions | None' = None, references: 'References' = None, **option_kwargs: 'Any') -> 'ValidationReport'
Everything `spec` has to say about `df`, as a `ValidationReport`.
Never raises for a frame that fails: every violation is a `Finding` on
the report, with `report.rows(finding)` and `report.failing_rows()`
giving the offending rows back lazily. See `validate` for the options.
## validate
### validate(spec: 'TableSpec', df: 'pl.DataFrame | pl.LazyFrame', *, options: 'ValidationOptions | None' = None, references: 'References' = None, **option_kwargs: 'Any') -> 'pl.DataFrame | pl.LazyFrame'
Validates a DataFrame or LazyFrame against `spec`.
Parameters
----------
df : pl.DataFrame | pl.LazyFrame
The frame to validate. A LazyFrame comes back as a LazyFrame.
options : ValidationOptions, optional
Every option at once, as a value -- useful for passing one setting
through several calls. Cannot be combined with the keywords below.
references : mapping
Parent frames for foreign keys that reference another spec, keyed by
that spec, its FrameSpec class, or its name. A key with no entry is
reported as a `foreign_key_unresolved` finding.
**option_kwargs
The fields of `ValidationOptions`, one at a time, with the check
switches spelled `validate_rules`, `validate_validators`,
`validate_unique`, `validate_checks`, `validate_foreign_keys`,
`validate_hierarchy`, `validate_pattern` and `validate_bounds`. See
`ValidationOptions` for what each means and what it defaults to; an
unknown name raises `TypeError` naming the closest match.
Returns
-------
The validated, optionally transformed frame -- a LazyFrame if `df` was one.
Raises
------
ValidationError
Carrying a `ValidationReport` of every violation.
TypeError
For an option name this does not accept.
ValueError
For an accepted option given a value outside its choices.
## ValidationOptions
### ValidationOptions(extra_cols: "Literal['drop', 'allow', 'raise']" = 'raise', missing_cols: "Literal['add', 'allow', 'raise']" = 'raise', strict_dtypes: 'bool' = False, rules: 'bool' = True, validators: 'bool' = True, unique: 'bool' = True, checks: 'bool' = True, foreign_keys: 'bool' = True, hierarchy: 'bool' = True, pattern: 'bool' = True, bounds: 'bool' = True, cast: 'bool' = False, streaming: 'bool' = False) -> None
Which checks to run and what to do about structural mismatches.
Grouped rather than passed as ten separate arguments, so adding a check
does not widen every signature between here and `FrameSpec.validate`.
## ValidationReport
### ValidationReport(spec_name: 'str', findings: 'tuple[Finding, ...]', frame: 'pl.LazyFrame', options: 'ValidationOptions' = None) -> None
Every finding for one frame against one spec.
- `ValidationReport.by_code(self, code: 'FindingCode') -> 'tuple[Finding, ...]'` -- Every finding of one kind, such as `"bounds"` or `"foreign_key"`.
- `ValidationReport.by_column(self) -> 'dict[str, tuple[Finding, ...]]'` -- Findings grouped by column; structural findings under `""`.
- `ValidationReport.failing_rows(self) -> 'pl.LazyFrame'` -- Every row that violates a row-level finding, lazily.
- `ValidationReport.raise_if_failed(self) -> 'None'` -- Raises `ValidationError` carrying this report, if anything was found.
- `ValidationReport.rows(self, finding: 'Finding') -> 'pl.LazyFrame'` -- The rows violating one finding, lazily.
- `ValidationReport.to_dict(self) -> 'dict[str, Any]'` -- This report as JSON-ready data: the spec, the verdict, the findings.
- `ValidationReport.to_json(self, *, indent: 'int | None' = 2) -> 'str'` -- This report as a JSON string. `indent=None` for one line.
## Finding
### Finding(code: 'FindingCode', key: 'str', message: 'str', columns: 'tuple[str, ...]' = (), count: 'int | None' = None, samples: 'tuple[Any, ...]' = (), details: 'Mapping[str, Any]' = , _locate: 'Callable[[pl.LazyFrame], pl.LazyFrame] | None' = None) -> None
One violation of one claim the spec makes.
Attributes
----------
code : FindingCode
Which kind of claim was violated.
key : str
A stable identifier for the claim within its spec, such as
`"total__bounds"` or `"check:total_covers_subtotal"`.
message : str
The human-readable description of what was violated.
columns : tuple[str, ...]
The columns involved; empty for structural findings.
count : int | None
How many rows violate the claim; `None` for structural findings.
samples : tuple
Up to five offending values (or structs of values, for multi-column
claims).
details : Mapping
Code-specific facts: the expected and actual dtype, the observed
extremes, the foreign key's target.
- `Finding.rows(self, frame: 'pl.LazyFrame') -> 'pl.LazyFrame'` -- The rows of `frame` that violate this claim, lazily.
- `Finding.to_dict(self) -> 'dict[str, Any]'` -- This finding as JSON-ready data.
---
# Drift
Source: https://maxwellb13.github.io/polspec/reference/api/drift/
# Drift
`polspec.drift.diff()` compares two declarations; `polspec.drift.drift()`
compares a declaration to a frame. Both return a `DriftReport`, and both are
reachable from a spec as `Orders.diff(...)` and `Orders.drift(df)` -- see
[Schema and data drift](https://maxwellb13.github.io/polspec/how-to/drift/).
## diff
### diff(old: 'TableSpec | type', new: 'TableSpec | type', *, renames: 'Mapping[str, str] | None' = None, options: 'DriftOptions | None' = None) -> 'DriftReport'
What changed between two declarations, as a `DriftReport`.
Parameters
----------
old, new : TableSpec | FrameSpec class
The two declarations. A spec's name is not compared: the report is
about what the two say, not what they are called.
renames : mapping, optional
Columns renamed between the two, `{old_name: new_name}`. Applied to
`old` first, through `TableSpec.rename`, so a rename is reported
as `column_renamed` rather than as a column removed and another
added. The caller asserts the rename; nothing is guessed from
similar names.
options : DriftOptions, optional
Only `strict_dtypes` is read when diffing two specs.
Notes
-----
A finding is *breaking* when a frame that satisfied `old` could fail
`new`: a narrowed domain, a dropped nullability, a column added (a frame
that lacks it fails `missing_cols="raise"`), a constraint added. A
widened domain or a removed constraint is *compatible*.
## drift
### drift(spec: 'TableSpec | type', df: 'Frame', *, options: 'DriftOptions | None' = None, **option_kwargs: 'Any') -> 'DriftReport'
How `df` has moved relative to what `spec` declares, as a `DriftReport`.
Parameters
----------
spec : TableSpec | FrameSpec class
The declaration.
df : pl.DataFrame | pl.LazyFrame
The frame. A LazyFrame is collected: every measurement here is a
summary of the whole column.
options : DriftOptions, optional
Every option at once. Cannot be combined with the keywords.
**option_kwargs
The fields of `DriftOptions`, one at a time.
Notes
-----
A finding is *breaking* when this frame fails this spec on that column:
values outside the domain, a bound exceeded, nulls where none are
allowed, a format not matched. A null rate that moved within a nullable
column, or declared values the data never holds, is *compatible* -- the
data still validates; the declaration has stopped describing it well.
Uniqueness, composite keys, foreign keys and checks are not measured
here; they are pass/fail claims that `validate()` already reports.
## DriftOptions
### DriftOptions(null_rate_tolerance: 'float' = 0.05, unseen_values: 'bool' = True, strict_dtypes: 'bool' = False, max_samples: 'int' = 10) -> None
What counts as drift, said once.
Parameters
----------
null_rate_tolerance : float, default 0.05
How far the observed null rate may sit from a nullable column's
`null_probability` before `null_rate_moved` is reported. Absolute,
not relative: a relative tolerance is unstable near zero.
unseen_values : bool, default True
Whether to report declared `choices` or `Enum` categories the data
never holds (`cardinality_moved`).
strict_dtypes : bool, default False
The same switch as `ValidationOptions.strict_dtypes`, and decided by
the same function: whether a `dtype_changed` is breaking.
max_samples : int, default 10
How many offending values a finding's `details` carry.
## DriftReport
### DriftReport(kind: 'Kind', old: 'str', new: 'str', findings: 'tuple[DriftFinding, ...]', options: 'DriftOptions | None' = None) -> None
Every difference between two specs, or between a spec and a frame.
`kind` says which: `"diff"` compares `old` to `new`, both declarations;
`"drift"` compares the declaration `old` to data, and `new` is what the
data was called. `bool(report)` is `report.unchanged`, the way
`bool(ValidationReport)` is `passed`.
- `DriftReport.by_code(self, code: 'DriftCode') -> 'tuple[DriftFinding, ...]'` -- Every finding of one kind, such as `"domain_narrowed"`.
- `DriftReport.by_column(self) -> 'dict[str, tuple[DriftFinding, ...]]'` -- Findings grouped by column; table-level findings under `""`.
- `DriftReport.to_dict(self) -> 'dict[str, Any]'` -- This report as JSON-ready data.
- `DriftReport.to_json(self, *, indent: 'int | None' = 2) -> 'str'` -- This report as a JSON string. `indent=None` for one line.
- `DriftReport.to_markdown(self, path: 'str | Path | None' = None) -> 'str'` -- This report as Markdown, written to `path` if given.
## DriftFinding
### DriftFinding(code: 'DriftCode', severity: 'Severity', key: 'str', message: 'str', columns: 'tuple[str, ...]' = (), details: 'Mapping[str, Any]' = ) -> None
One difference between two declarations, or a declaration and data.
Attributes
----------
code : DriftCode
Which kind of difference.
severity : "breaking" | "compatible"
Breaking when data that satisfied the old side could fail the new
one; compatible otherwise.
key : str
A stable identifier within the report, such as `"total__bounds"` or
`"check:total_covers_subtotal"`.
message : str
What changed, naming the column, and what to do about it.
columns : tuple[str, ...]
The columns involved; empty for table-level differences.
details : Mapping
Code-specific facts: the old and new value, how far a bound was
exceeded, which values were new.
- `DriftFinding.to_dict(self) -> 'dict[str, Any]'` -- This finding as JSON-ready data.
---
# Profiling
Source: https://maxwellb13.github.io/polspec/reference/api/profiling/
# Profiling
Inferring a spec from data you already have.
## profile_dataframe
### profile_dataframe(df: 'pl.DataFrame', *, weights: 'bool' = False, max_unique_enum: 'int' = 50, calculate_bounds: 'bool' = True) -> 'dict[str, ColSpec]'
Infers ColSpec column definitions by profiling an existing DataFrame.
---
# Exceptions
Source: https://maxwellb13.github.io/polspec/reference/api/errors/
# Exceptions
Every error polspec raises descends from `PolspecError`, so one `except`
clause catches the lot. Most also descend from the built-in they replaced, so
existing `except ValueError` handlers keep working. The
[Errors and findings](https://maxwellb13.github.io/polspec/reference/errors/) page explains when each is raised.
## PolspecError
### PolspecError
Base class for every error polspec raises on its own behalf.
## SpecError
### SpecError
A declaration that cannot mean anything.
Raised while a `ColSpec`, `ColRule`, `Check`, `ForeignKey`, `FrameSpec`
or `CatSpec` is being built: bounds a dtype cannot hold, a rule pointing
at a column that does not exist, two columns resolving to one name.
Inherits both `ValueError` and `TypeError` because it replaces both.
## GenerationError
### GenerationError
A spec that declares fine cannot be turned into data as asked.
A dtype the engine cannot fill, a cartesian coverage set past the size
cap, a foreign key with an empty parent, a `unique` domain smaller than
the row count. Errors raised inside the Rust extension surface as this.
## ValidationError
### ValidationError(report: 'Any', errors: 'list[str] | None' = None) -> 'None'
Data does not meet its spec.
Carries the `ValidationReport` of every violation found as `report`.
`errors` is the same findings as a plain list of messages, for the common
case of printing them.
## MultiValidationError
### MultiValidationError(reports: 'Any') -> 'None'
Several frames failed validation together, as one registry call.
`reports` holds the `ValidationReport` of every spec that failed, keyed by
spec name, so `failing_rows()`, `by_code()` and the rest are reachable for
each of them. `report` is None: there is no single report here, and
picking one of several arbitrarily would be worse than saying so. `str()`
and `errors` read as they always have -- every failing spec's findings,
one after another.
## SerializationError
### SerializationError
A spec file cannot be written or read.
A dtype with no file representation, a key the reader does not know, a
file written by a newer format version.
## RegistryError
### RegistryError
A collection of specs is inconsistent.
An unknown or duplicated spec name, a cycle in the foreign-key graph, two
specs disagreeing about a shared category.
## CliError
### CliError
An expected failure on the command line, reported without a traceback.
---
# Errors and findings
Source: https://maxwellb13.github.io/polspec/reference/errors/
# Errors
Everything polspec raises on its own behalf derives from one base class, so
a caller can separate "polspec objected" from "something else went wrong"
with a single clause:
```python
from polspec import PolspecError
try:
Orders.validate(df)
except PolspecError as exc:
log.warning("rejected: %s", exc)
```
| Exception | Raised when | Also a |
|:--|:--|:--|
| `PolspecError` | Base class; never raised directly | `Exception` |
| `SpecError` | A declaration cannot mean anything: bounds a dtype cannot hold, a rule naming a column that does not exist, two attributes resolving to one column name | `ValueError`, `TypeError` |
| `ValidationError` | Data does not meet its spec. `err.report` is the `ValidationReport`; `err.errors` lists its messages | `ValueError` |
| `GenerationError` | A spec that declares fine cannot be turned into data as asked: no column for `method="cartesian"` to cover, a coverage set past the size cap, a foreign key with an empty parent. Errors from the Rust engine surface as this | `ValueError` |
| `SerializationError` | A spec file cannot be written or read: a dtype with no file representation, an unrecognised dtype name, a category reference the registry does not hold | `ValueError` |
| `RegistryError` | A `Registry` is inconsistent: an unknown or duplicated spec name, a key whose target is not in it, a cycle, two specs disagreeing about a shared category | `LookupError` |
Each subclass keeps the built-in type it replaced, so `except ValueError`
written against an earlier version still catches it.
Ordinary argument misuse is not a `PolspecError`. A negative row count, an
unknown `method=`, a `batch_size` of zero, or the wrong object passed where a
DataFrame was expected raise the plain `ValueError` or `TypeError` any Python
API would.
The command line prints a `PolspecError` as a one-line `error: ...` and exits
with status 1; anything else is a bug and keeps its traceback.
## Finding codes
Every violation `inspect()` reports, and `validate()` raises, is a `Finding`
with one of these codes. Row-level findings can return the offending rows
through `report.rows(finding)`; structural ones describe the frame's shape.
| Code | Kind | Raised when |
|:--|:--|:--|
| `extra_columns` | structural | the frame has columns the spec does not declare (`extra_cols="raise"`) |
| `missing_columns` | structural | the frame lacks declared columns (`missing_cols="raise"`) |
| `dtype` | structural | a column's dtype is not compatible with its declaration |
| `foreign_key_unresolved` | structural | a key references another spec and `references=` had no entry for it |
| `nullability` | row-level | a non-nullable column holds nulls |
| `choices` | row-level | a value is outside `choices` or the `Enum` categories |
| `bounds` | row-level | a value is outside `bounds`; `details` carry the extremes found |
| `string_length` | row-level | a string or binary value's length is outside `string_length` |
| `list_length` | row-level | a list holds a number of elements outside `list_length` |
| `format` | row-level | a string value does not have the declared `format`; `details` name the format |
| `pattern` | row-level | a string value does not match the declared `pattern`; `details` carry it |
| `rule` | row-level | a row matched a `ColRule` but holds a value outside its choices |
| `validator` | row-level | a `ColSpec.validators` predicate is false |
| `unique` | row-level | a `unique=True` column holds duplicates |
| `unique_together` | row-level | a composite key holds duplicate combinations |
| `check` | row-level | a `__checks__` predicate is false |
| `foreign_key` | row-level | a key value has no matching parent row (also structural when the parent lacks the referenced columns) |
## Drift codes
A `DriftReport` (from `diff()` or `drift()`) carries `DriftFinding`s with one
of these codes. Each also carries a `severity`: **breaking** when a frame
that satisfied the old side could fail the new one, **compatible**
otherwise. See [Schema and data drift](https://maxwellb13.github.io/polspec/how-to/drift/).
| Code | From | Severity | Raised when |
|:--|:--|:--|:--|
| `column_added` | both | breaking | the new spec declares a column the old lacks, or the data has an undeclared column |
| `column_removed` | both | breaking | the old spec declares a column the new lacks, or the data lacks a declared column |
| `column_renamed` | `diff` | compatible | a rename given through `renames=` |
| `dtype_changed` | both | as validation decides | the dtypes differ; compatible when the old values would still validate |
| `nullability_changed` | both | off → on compatible; on → off breaking; nulls in a non-nullable column breaking | |
| `domain_widened` | `diff` | compatible | bounds, choices, format or string length accept more than before |
| `domain_narrowed` | `diff` | breaking | they accept less |
| `domain_changed` | `diff` | breaking | neither is inside the other |
| `bounds_exceeded` | `drift` | breaking | values escape `bounds`, `string_length` or `list_length`; `details` say which and by how much |
| `new_values` | `drift` | breaking | values outside `choices`, an `Enum` or a finite format |
| `format_violated` | `drift` | breaking | values that do not match the declared `format` |
| `null_rate_moved` | `drift` | compatible | the null rate sits further from `null_probability` than the tolerance |
| `cardinality_moved` | `drift` | compatible | declared values the data never holds |
| `constraint_added` | `diff` | breaking | `unique`, a validator, rule, check, composite key, foreign key or hierarchy present only in the new spec |
| `constraint_removed` | `diff` | compatible | the reverse |
| `field_changed` | `diff` | compatible | `tags`, `weights`, `distribution`, `distribution_params`, `null_probability`, `pattern` or `seed_name` differ |
A struct's fields are compared by every row above, as columns are, and a
finding about one is keyed by its path -- `point.lat__domain` -- with
`columns` naming the struct column.
---
# Architecture
Source: https://maxwellb13.github.io/polspec/explanation/architecture/
# Architecture
polspec is a small Python package over a Rust extension. The Python side owns
the vocabulary — what a column can declare and what that means; the Rust side
owns only the inner loop that fills arrays with values.
## Modules
| Module | Responsibility |
|:--|:--|
| `bound` | An inclusive `[min, max]`, either end optionally open |
| `check` | A named boolean expression, with SQL-style null handling |
| `constants` | Default generation ranges |
| `dtypes` | What each dtype can actually hold |
| `distributions` | The distributions available, and each one's parameter aliases |
| `spec` | `ColSpec` — one column's declaration, and everything it validates about itself |
| `rules` | `ColRule` — conditional values, and the pass that applies them |
| `foreign_key` | `ForeignKey` — declaration, and the pass that makes generated keys consistent |
| `engine` | Turning a spec into the `ColumnPlan` the Rust extension takes, and finishing the result: gathering typed choices, casting temporal columns back |
| `_ffi` | The only module that imports the Rust extension (lazily), building plans and re-raising its errors as `GenerationError` |
| `errors` | The `PolspecError` hierarchy |
| `constraints` | What both sides read: `Domain` (the values a column may hold) and `Pass`/`order` (which rewrite of a generated frame runs first) |
| `validation` | `inspect` and `validate` over a `TableSpec`: every claim becomes a `_Constraint` that produces a `Finding`. The `constraints/` package holds them by kind: `_values` (one value's domain, bounds, length, format, pattern, lifted over a List's elements), `_rules`, `_table` (composite keys, checks), `_relations` (foreign keys, hierarchy); `report.py` holds `Finding` and `ValidationReport` |
| `tablespec` | `TableSpec` — a spec as an immutable value, with its declaration-time checks and structural operations |
| `framespec` | `FrameSpec` — the metaclass that builds a `TableSpec` from a class body, and the facade forwarding every verb to it |
| `generation` | `generate`, `generate_batches` and the file sinks, as functions over a `TableSpec`; `composite.py` separates a `__unique_together__` group; `seeds.py` keys every pass's seed by name, as the engine keys columns |
| `catspec` | `CatSpec` — a shared registry of enums and categoricals, as a value, plus the metaclass that builds one from a class body (the same split as `tablespec`/`framespec`) |
| `registry` | `Registry` — a declared set of specs: resolving cross-spec keys, ordering parents before children, `generate_all`/`validate_all`, one file and one diagram for the set |
| `serialization` | Spec files: a field registry (`fields.py`) that YAML, generated Python and the `import datetime` decision all derive from; the dtype codec table (`dtypes.py`); format versions and migrations (`migrations.py`) |
| `profiler` | Inferring a spec from an existing DataFrame |
| `report` | Rendering a spec, or a registry of them, as Markdown or Mermaid |
| `cli` | The `polspec` command, one module per verb: `_schema` (infer, new), `_data` (validate, generate), `_drift` (diff, drift), `_test`; `_io` holds the readers, writers and spec loaders they share |
The dependency direction is one-way: `spec` and `tablespec` know nothing about `framespec`,
and `report` is not reachable from either the generation or validation path.
## Generating
```mermaid
flowchart LR
A["FrameSpec.generate(n, seed)"] --> B["_generate_random
or _generate_cartesian"]
B --> C["_plan_column: one
ColumnPlan per column"]
C --> D["Rust: generate_dataframe
columns in parallel"]
D --> E["_finish: gather typed choices,
cast temporal columns back"]
E --> F["order the passes
by reads and writes"]
F --> G["each pass: rules, foreign keys,
composite-key repair"]
G --> H[DataFrame]
```
Each column becomes a `ColumnPlan` — kind, nullability, exact bounds, domain
size and weights, lengths, distribution — crossing into Rust once. Rust fills the
columns in parallel, and within a column in 65,536-row chunks whose seeds come
from the chunk index, so output is identical regardless of thread count.
For a fixed-width column a chunk is a unit of work, not a unit of storage. The
values buffer and the validity bitmap are each allocated once at the column's
full length, and a chunk fills its own disjoint slice of both — which is why
the chunk size is a multiple of 8, so the bitmap divides on a byte boundary and
no two threads touch the same byte. The column reaches Polars as a single
chunk, so nothing downstream — the gather behind a `choices` domain, the cast
behind a temporal dtype, a `sink_*` write — pays for a column split into
hundreds of pieces.
String columns are the exception: a row's width is not known until it is drawn,
and Polars backs them with view arrays, which merge by copying sixteen bytes of
view per row. That costs more than the split it would remove, so a long string
column stays chunked.
Rules and foreign keys are applied afterwards as vectorised passes over the
finished frame, not row by row. Each pass declares the columns it reads and
the ones it writes, and `constraints.order` runs them so no pass reads a
column a later one rewrites: a rule keyed on a foreign-keyed column sees the
parent's values, and a self-referencing key drawing from a foreign-keyed
column draws from values that are actually there. That ordering is what makes
generated data satisfy the same claims validation checks it against — which
is why a spec whose passes cannot be ordered is refused at declaration rather
than generated and then failed by its own spec.
Seeds are drawn per pass in declaration order, so which order they end up
running in does not change the values any one of them samples.
A `unique=True` column never reaches a pass: the engine draws it without
replacement in the first place (`src/unique.rs`), shuffling a materialised
domain when the domain is barely bigger than the frame and rejecting against a
set when it is roomy. A `__unique_together__` group is a pass, because
distinctness across columns can only be judged once they all exist: it
resamples the rows repeating a combination, and reads every member so it runs
after the rules and keys that settle them.
## Validating
```mermaid
flowchart LR
A["FrameSpec.inspect(df) / validate(df)"] --> B[Structural checks]
B --> C["Build one _Constraint
per declared claim"]
C --> D["One Polars aggregation
over the whole frame"]
D --> E["Each constraint turns its
result into a Finding"]
E --> F["ValidationReport
(what inspect returns)"]
F -->|validate: findings| G[ValidationError carrying the report]
F -->|validate: none| H["Drop / add / cast / reorder"]
```
Every claim a spec makes becomes a `_Constraint` that contributes aggregation
expressions and turns the results back into a `Finding`: a code, a count,
samples, code-specific details, and a lazy filter that locates the rows. They are collected
first and evaluated together, so validating a wide table costs one scan rather
than one per column. Foreign keys are the exception: each needs its own
anti-join against a parent frame.
Adding a new kind of check means adding a class, not editing two distant loops.
## The Python / Rust boundary
Python builds one `ColumnPlan` per column -- a `#[pyclass]` in `src/plan.rs`
that validates itself at construction, so an unknown kind, a weight vector of
the wrong length or a distribution parameter out of range is refused with a
message naming the column before any sampling starts. `polspec/_ffi.py` is the
only module that imports the extension, lazily: validation, spec files and the
registry work without a built extension, and only generation asks for one.
Rust knows about *kinds*, not about polspec's vocabulary: `int8` .. `uint64`,
`float32`/`float64`, `bool`, `string`, and `index`. `Date` crosses as an
`int32` day count and `Datetime`/`Duration`/`Time` as an `int64` in their own
unit. Anything with a finite domain -- `choices`, an `Enum`, a
capacity-limited `Categorical` -- crosses as `index` with the domain's size
and weights; Rust returns `UInt32` indices and Python gathers the typed values
back, so a `datetime` or a `bytes` choice never passes through a string.
Bounds cross as a `Limit`: an `i64`, a `u64` or an `f64`, whichever holds the
Python value exactly, so `Int64` and `UInt64` bounds keep every bit.
Distribution parameter *aliases* live only in `polspec/distributions.py` and
are resolved when a column is declared; `src/dist.rs` reads canonical keys and
exports its table as `distribution_params()`, which a test compares with the
Python one. Each column's seed is derived from the frame seed and the column
*name* (`sample.rs`), so inserting a column never reshuffles its neighbours.
`src/sample.rs` and `src/unique.rs` have no Python types and carry the unit
tests `cargo test` runs; `python/polspec/_polspec.pyi` is the stub, and a test asserts its names
match the module.
## Tests
| File | Covers |
|:--|:--|
| `test_roundtrip.py` | The property tying the two directions together: anything `generate()` produces, `validate()` accepts |
| `test_declaration.py` | Declaration-time contracts that never reach generated data |
| `test_generation.py` | Random and cartesian generation, dtype coverage, distributions, weights |
| `test_rules.py` | `ColRule`: what a rule may declare and which rows it touches |
| `test_serialization.py` | `to_yaml`/`from_yaml` and `to_python`, and what they warn about and drop |
| `test_profiler.py` | `from_dataframe` inference |
| `test_framespec.py` | The class body: inheritance, tags, `__checks__`, `__unique_together__`, validators |
| `test_report.py` | Markdown data dictionaries and Mermaid diagrams |
| `test_foreign_key.py` | `ForeignKey` declaration, persistence and generation |
| `test_validation.py` | Validation behaviour and error reporting |
| `test_inspect.py` | `inspect()`: findings as data, lazy failing rows, JSON |
| `test_tablespec.py` | `TableSpec` as a value: construction, structural operations, the metaclass |
| `test_expr.py` | The `col()` predicate language and its data form |
| `test_serialization_format.py` | The field registry, format versions, migrations, unknown keys |
| `test_registry.py` | `Registry`: resolution, ordering, `generate_all`/`validate_all`, files, discovery |
| `test_errors.py` | The exception hierarchy |
| `test_catspec.py` | Shared category registries: both declaration forms, and that they agree |
| `test_streaming.py` | Batching and the file sinks |
| `test_cli.py` | The command line, including running a generated test file under pytest |
| `test_engine.py` | The Python / Rust boundary: typed plans, exact bounds, typed choices, per-column seeds, the stub |
| `test_constraints.py` | What both sides share: `Domain`, and the pass ordering that lets rules and keys see each other's work |
| `test_docs.py` | That the documentation points at things that exist: every exported name, link and nav entry, and the generated `llms.txt` |
| `test_doc_examples.py` | That the documentation's Python examples run |
The round-trip file carries `xfail(strict=True)` markers for known gaps, so a
fix turns the marker into a failure rather than passing unnoticed. See
[Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/).
---
# Generation and validation
Source: https://maxwellb13.github.io/polspec/explanation/two-sides/
# Generation and validation
polspec does two things with one declaration: it makes data that matches a
spec, and it checks whether data matches a spec. That sounds like one job read
in two directions, but the two directions are not symmetric, and most of the
library's design follows from where they differ.
## Why one declaration is harder than two
A validator only has to *recognise* a violation. A generator has to *avoid*
one. Recognising is easy for almost any claim you can write down: a Polars
expression over the frame gives you the answer. Avoiding is only easy for
claims with a shape a sampler can exploit.
`bounds=(1, 100)` has that shape — draw uniformly from the range and no value
can be out of bounds. `pl.col("email").str.contains("@")` does not. Both are
perfectly good validators; only one is a usable generator.
So the honest position is that the two sides cover different amounts of
ground, and the interesting engineering is in narrowing the gap without
pretending it isn't there.
## The failure mode: two implementations
The dangerous version of this is implementing each claim twice — once in the
sampler, once in the checker — and hoping they agree. They drift. A bound is
inclusive on one side and exclusive on the other; a choice is compared as a
string here and as a typed value there; a rule is evaluated against the
freely-generated frame while validation reads the final one.
Every one of those was a real bug in polspec, and each produced the same
symptom: `Spec.validate(Spec.generate(n))` raising. Data the library made,
rejected by the library that made it.
That property has a name here — the *round-trip* — and it is asserted
directly, in `tests/test_roundtrip.py`:
```python
SpecCls.validate(SpecCls.generate(n, seed=...)) # must not raise
```
## What is shared, and what is only tested
The structural answer is to give both sides one definition to read. That is
what `polspec.constraints` holds:
- **`Domain`** — the values a column may hold: its `choices`, an `Enum`'s
categories, its `bounds`. Generation samples from it, validation checks
against it, and a foreign key asks whether a parent's domain fits inside a
child's. One definition, three readers.
- **`Pass` and `order`** — which rewrite of a generated frame runs first,
derived from the columns each pass reads and writes. This is what lets a
rule keyed on a foreign-keyed column see the parent's values, which is the
same thing validation will check the rule against.
What is not shared is held in step by the round-trip test instead. That is a
weaker guarantee than a shared definition, and the difference is deliberate:
sharing costs an abstraction, and it is only worth paying where the two sides
genuinely say the same thing.
## Where the gap remains
Three claims are validated and not generated, and all three are permanent:
`__checks__` and `ColSpec.validators` wrap arbitrary Polars expressions, and
`ColSpec.pattern` is an arbitrary regex. Nothing can generate data satisfying
an arbitrary predicate — that is a statement about predicates, not about
polspec — so generation makes no attempt, and each boundary is pinned by its
own test rather than papered over.
The way to close *that* gap is not a cleverer generator. It is a richer
vocabulary for describing values, and `format="email"` is what that looks
like: it says the same thing as `col("email").str.contains("@")`, in a shape
a sampler can use. `pattern=` is the same vocabulary without the sampler —
any regex can be checked, the curated set can be generated. See
[String formats](https://maxwellb13.github.io/polspec/how-to/formats/) and the [roadmap](https://maxwellb13.github.io/polspec/explanation/roadmap/).
Everything else on the list has been closed rather than documented away:
uniqueness by drawing without replacement, rule and foreign-key dependencies
by ordering the passes. The current list is in
[Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/), and each entry there is backed by a test
that fails the moment the entry stops being true.
## What this means when you declare a spec
Two practical consequences.
**A contradiction is refused when you write it, not when you run it.** A
foreign key whose parent domain cannot fit inside its own column's, two
columns whose rules each depend on the other, `unique=True` alongside
`weights` — none of these have a coherent reading, so they raise `SpecError`
at declaration rather than producing data that fails its own spec.
**A validation-only claim is still worth declaring.** A validator that
generation cannot satisfy is not wasted: it still guards real data on the way
in. Generate with `validate_validators=False` when you need synthetic rows,
and keep the claim for the data that matters.
---
# Known limitations
Source: https://maxwellb13.github.io/polspec/explanation/limitations/
# Known limitations
polspec generates data and validates it from one declaration. Where both
sides read the same definition they cannot drift: what values a column may
hold, and the order the passes that rewrite a generated frame run in, both
live in `polspec.constraints`. What is left below is what generation does not
attempt at all, plus a few edges worth knowing about.
Each is pinned by a test in `tests/test_roundtrip.py`. A gap meant to close
one day carries `xfail(strict=True)`: the suite stays green while it exists,
and the moment it is fixed pytest turns the XPASS into a failure. A boundary
that is deliberate is pinned by an ordinary passing test instead. Either way
this page cannot quietly go stale — changing what polspec does forces the test
to be updated.
## Generation does not enforce these
### `__checks__`, `ColSpec.validators` and `ColSpec.pattern` are validation-only
This one is by design, not a defect: checks and validators wrap arbitrary
Polars expressions, and a pattern is an arbitrary regex; nothing can generate
data satisfying an arbitrary predicate, and generating from an arbitrary
regex has no answer for `.*`. Validate generated data with
`validate_checks=False` / `validate_validators=False` /
`validate_pattern=False`, or construct the rows those invariants describe
yourself. For a shape polspec *can* generate, declare a
[`format`](https://maxwellb13.github.io/polspec/how-to/formats/) instead of a pattern.
### A self-referencing foreign key is referential, not acyclic
A `ForeignKey(..., references="self")` guarantees exactly what it says: every
non-null value in the child column is a value that exists in the referenced
column of the same frame. It does **not** guarantee the result is a tree.
Parents are sampled from the frame as it stands, which builds a random
functional graph — so a row can be its own parent, and two rows can be each
other's. This is not rare:
```python
class Node(FrameSpec):
Reference = ColSpec(pl.String, unique=True)
Parent = ColSpec(pl.String, nullable=True, null_probability=0.2)
__foreign_keys__ = [
ForeignKey("Parent", references="self", ref_columns="Reference")
]
```
At 20 rows that typically leaves a handful of rows inside a cycle and one or
two pointing at themselves; at 20,000 it is a fraction of a percent. Rare is
not the same as safe — a cycle is exactly what makes a recursive CTE or a
hierarchy walk fail to terminate, and `validate()` will not report one, because
nothing in a spec can currently say "acyclic".
Where you need a genuine hierarchy, declare one. `Hierarchy` is the same two
columns with the shape written down — one parent per reference, a bounded
depth, no cycles — and generation satisfies it rather than leaving it to the
draw:
```python
class Node(FrameSpec):
Reference = ColSpec(pl.String)
Parent = ColSpec(pl.String)
__hierarchy__ = Hierarchy(child="Reference", parent="Parent", max_depth=5)
```
See [Hierarchies and link tables](https://maxwellb13.github.io/polspec/how-to/hierarchies/), including how to
ask for the cycles back when they are what you are testing against.
## Cartesian generation
### `n` is a minimum, not a count
Under `method="cartesian"`, if the coverage set is larger than `n` all of it is
kept. `generate_batches` and every `sink_*` inherit this, so asking for 5 rows
from two ten-category enums yields 100.
## A `format` promises syntax, not existence
`format="email"` generates a well-formed address, not a deliverable one, and
validates the shape, not whether anything answers. Nothing is looked up on
either side, so `nobody@example.invalid` passes, `hostname` accepts
`localhost`, and `ipv4` accepts `0.0.0.0`. A column that has to hold *real*
identifiers is a `choices` list or a foreign key into the table that owns
them. See [String formats](https://maxwellb13.github.io/polspec/how-to/formats/).
## Smaller sharp edges
- **A list's elements are never null.** Generation fills a `List` column's
cells with non-null elements, and no field can ask otherwise; validation
reports a null element as a `nullability` finding.
- **A `Decimal` is drawn through 64 bits.** Generation fills a Decimal as
its physical integer, so bounds needing more than eighteen significant
digits are refused at `generate()`. Validation checks the full precision.
- **`missing_cols="add"` can produce a frame that fails re-validation**, since
columns are added after validation runs, including for non-nullable columns.
- **A `Hierarchy` cannot be batched or streamed.** `generate_batches` and
every `sink_*` refuse a spec that declares one, because each batch is
generated independently and a forest is a property of the whole frame.
- **A `Hierarchy` owns both its columns.** A `null_probability`,
`distribution` or `weights` declared on either is not what you get: the
references have to come from one pool for the two columns to join at all.
- **Uniqueness holds within a batch, not across one.** A batch of
`generate_batches` or a `sink_*` is a window onto one frame for a column
no pass rewrites, but a `unique=True` column, a `__unique_together__`
group, a rule and a foreign key are drawn per batch, so distinctness is
only within each batch.
- **A `unique` column ignores `weights` and a non-uniform `distribution`** --
both are refused at declaration rather than silently dropped, since neither
has anything to say about a draw without replacement.
- **A foreign key still overwrites its column's distribution.** The parent's
domain has to fit inside the column's own — a contradiction is refused at
declaration — but within it, values come from the parent, so a declared
`distribution` or `weights` on a foreign-keyed column is not what you get.
---
# Comparison
Source: https://maxwellb13.github.io/polspec/explanation/comparison/
# Comparison to other approaches
polspec sits at the intersection of two things usually solved by separate
tools: generating test data, and validating that data against a schema. This
page is about that intersection — what generating *and* validating from one
declaration buys you that the alternatives don't, where those alternatives
are still the better tool, and the actual numbers behind the speed claim.
## Benchmarks
`benchmarks/bench.py compare` generates the same four-column frame — a
non-nullable string, a nullable enum, a nullable bounded int, a nullable
bounded float — three ways: polspec's Rust generator, a hand-vectorized NumPy
implementation, and a pure-Python loop using `random`. All three produce an
equivalent `pl.DataFrame`, so the comparison is "how fast can each approach
hand back a usable frame," not raw loop speed in isolation.
| n_rows | polspec (Rust) | NumPy | Python |
|-----------:|---------------:|----------:|----------:|
| 1,000 | 0.0001s | 0.0006s | 0.0014s |
| 10,000 | 0.0003s | 0.0049s | skipped |
| 100,000 | 0.0020s | 0.0472s | 0.1454s |
| 1,000,000 | 0.0069s | 0.4797s | 1.4837s |
| 5,000,000 | 0.0286s | 2.4156s | skipped |
| 20,000,000 | 0.1127s | 9.7757s | skipped |
Measured 2026-09-08 on an Intel 13900K with 64GB DDR5; yours will differ, and
the shape matters more than the absolute numbers. Three things worth reading
off it:
- **The gap widens with size, not just the ratio.** At 1,000 rows all three
are fast enough that the difference doesn't matter to a test suite. At
20,000,000, pure Python is impractical (skipped past a 5-second cutoff at
a much smaller size) and NumPy's ~2 million rows/second becomes a real wait
in a CI loop, while polspec is still around a tenth of a second.
- **NumPy's implementation is the hard-won version.** Its string column uses
a fixed-width byte-array trick because NumPy has no efficient way to
vectorize *ragged* per-row lengths — the other two implementations generate
strings 5–15 characters long; NumPy's are fixed at 15 and decoded back down.
That's not a knock on NumPy — it's the actual cost of writing this by hand:
the fast version needs a specific trick per dtype, and someone has to know
it.
- **A benchmark is a measurement of a machine, not only of code.** Every
number here is the fastest of several runs, each in a process of its own so
that one case cannot leave the allocator warm for the next, and the run
records the CPU, the thread count, the Polars version and the cargo profile
beside the timings. That last one matters more than it sounds: building the
extension as a single codegen unit moves the `unique` path by a factor of two
on its own.
Reproduce it yourself:
```bash
uv run --group bench python benchmarks/bench.py compare
```
The same harness guards against regressions. `record` writes a baseline for
the machine you are on, and `check` re-measures every case — each column kind,
both branches of the unique draw, the cartesian and rule and foreign-key
passes, the sinks — and exits non-zero if one has regressed:
```bash
uv run --group bench python benchmarks/bench.py record # before a change
uv run --group bench python benchmarks/bench.py check # after it
```
Generation speed is only half the story — [validation](https://maxwellb13.github.io/polspec/how-to/validating/)
compiles every check across every column into a single Polars aggregation,
so validating a fifty-column table costs about the same as validating a
five-column one. That isn't benchmarked here, since there's no equivalent
"validate this by hand" baseline to compare it against.
## Compared to hand-written fixtures
The common alternative is a Python dict or list literal, copy-pasted between
test files and edited by hand when the shape needs to change:
```python
def make_customer_row(customer_id=1, tier="free"):
return {"customer_id": customer_id, "tier": tier, "signed_up": "2023-01-01"}
```
This works, and for a handful of fixed cases it's often the right amount of
machinery. It stops working as the schema grows: the dtype lives nowhere —
`tier` being one of three strings is enforced by nobody until something
downstream breaks — and every edge case (a null, a boundary value, a specific
combination of two columns) is a row someone remembered to write by hand.
There's also nothing stopping the fixture and the real schema from drifting
apart; the dict doesn't know the pipeline added a column last month.
A `ColSpec` declaration is both the definition and the generator: the dtype,
the bound, and the domain are enforced the same way whether you're generating
data or checking it, and [`method="cartesian"`](https://maxwellb13.github.io/polspec/how-to/generating/#coverage-methodcartesian)
covers the boundary/null cases that hand-written fixtures tend to under-cover
because nobody thought to write them.
## Compared to Faker and similar
[Faker](https://faker.readthedocs.io/) and libraries built on it are the
right tool for *semantically realistic* values — names that look like names,
addresses that parse like addresses, emails with plausible domains. polspec
doesn't try to compete there: its strings are bounded-length ASCII, not
locale-aware people or places, because it's solving a different problem —
statistically-shaped data that respects a schema, not human-plausible data
that respects cultural conventions.
The two combine rather than compete. A Faker-generated pool of realistic
values becomes a `ColSpec.choices` list; polspec supplies the bounds,
nullability, cross-column rules, and cross-table referential integrity that
sit around it:
```python
import polars as pl
from faker import Faker
from polspec import ColSpec, FrameSpec
fake = Faker()
first_names = list({fake.first_name() for _ in range(500)}) # choices must be distinct
class Customers(FrameSpec):
name = ColSpec(pl.String, choices=first_names)
signup_bonus = ColSpec(pl.Float64, bounds=(0.0, 50.0))
```
What Faker doesn't do on its own is hand back a typed `pl.DataFrame`, enforce
a bound, or keep a foreign key consistent across two generated tables — those
are the parts of the problem polspec is actually for.
## Compared to NumPy or a bespoke script
The benchmark above *is* this comparison: a hand-written NumPy implementation
is faster than pure Python and can be made fast enough for most purposes, but
someone has to write it, and it has to be rewritten — bounds, nullability,
dtype casts — for every new column and every schema change. There's also
nothing left over afterward: the script that generated the data has no
relationship to a validator that checks it, because there was never a shared
declaration for the two to share.
polspec's Rust generator is faster than a hand-written NumPy version because
it doesn't pay Python's per-call overhead and fills columns in parallel — but
the bigger difference for day-to-day use is that the declaration doesn't have
to be rewritten by hand for each column, and the same one both generates and
validates.
## Compared to property-based testing (Hypothesis)
[Hypothesis](https://hypothesis.readthedocs.io/) solves a genuinely different
problem well: given a strategy for producing values, explore the space of
possible inputs, and when one fails, *shrink* it to the smallest failing
case. polspec has no shrinking and makes no attempt at exhaustive space
exploration — `method="cartesian"` is a fixed, deterministic set of
known-important combinations (every enum value, every numeric sign, null),
not an open-ended search.
These are complementary rather than competing: a `FrameSpec.generate(n,
seed=...)` call is a perfectly good data source *inside* a Hypothesis
strategy or a `@given` test, if what you need is Hypothesis's shrinking on
top of polspec's schema-shaped, Polars-native output.
## Compared to data-quality frameworks (Great Expectations, pandera, ...)
These frameworks are built around a different center of gravity: validating,
profiling, and monitoring data that already exists — often production
tables, with drift detection and reporting as first-class concerns. That's a
larger and more operational surface than polspec's `validate()`, which is
schema-shaped correctness checking, not statistical monitoring.
The distinguishing feature runs the other way, too: most validation-first
tools don't generate matching synthetic data for you. `FrameSpec` is meant to
be small enough to declare once and use for both jobs in a test suite, not to
replace a data-quality platform watching a production warehouse.
## What polspec doesn't try to be
Worth being direct about, in the same spirit as the
[known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/) and
[roadmap](https://maxwellb13.github.io/polspec/explanation/roadmap/) pages:
- **Not a realistic-fake-data library.** No locales, no plausible names or
addresses out of the box — pair it with Faker for that.
- **Not a data-quality or monitoring platform.** No drift detection, no
profiling dashboards, no anomaly scoring.
- **Not a property-based shrinking engine.** No search, no shrinking —
`method="cartesian"` is a fixed set of known-important cases, not an
open-ended exploration.
- **Not everything that validates generates.**
`__checks__`/`ColSpec.validators`/`ColSpec.pattern` are validated but not
generated, by design — see [Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/).
## Where it fits
Polars-native pipelines that need fast, schema-shaped synthetic data and
matching validation from one declaration — especially across several related
tables via `ForeignKey`, at volumes where a pure-Python or pandas generator
starts to cost real CI time, in tests that need to stay hermetic. See
[Testing pipelines](https://maxwellb13.github.io/polspec/how-to/testing/) for that in practice.
---
# Roadmap and stability
Source: https://maxwellb13.github.io/polspec/explanation/roadmap/
# Roadmap and stability
!!! warning "Early alpha"
polspec is early. The sections below are the honest version of "what's
next" — not a promise of when, just where the rough edges are and which
direction they're likely to move. Treat everything here, and everything
the library produces, as breakable between versions until it says
otherwise.
## Dtype coverage is complete
polspec generates every Polars dtype it accepts — integers, floats,
`Decimal`, booleans, strings, binary, `Date`/`Time`/`Datetime`/`Duration`,
`Enum`, `Categorical`, and `List`, `Array` and `Struct` of any of them,
nested to any depth. A `Datetime` carrying a `time_zone` is included: the
physical value is an offset from the naive epoch whatever the zone, so the
zone rides along and `generate()` hands back a column of the dtype you
declared. A `Decimal` is the same idea: an integer and a scale, drawn as
the integer and scaled back.
What a nested column *claims* is one declaration per value: a `List`'s
elements take the fields a scalar column takes, and a `Struct`'s take a
`ColSpec` each under
[`fields=`](https://maxwellb13.github.io/polspec/how-to/columns/#fields-what-a-structs-values-are). The
question this answered — whether a struct's fields are a declaration of
their own or a second, smaller thing — is settled the first way: a value
is described the same way wherever it sits, which is why the same change
that generated a struct generated a list of lists.
## Generation is getting more guardrails, not fewer
Two different kinds of "limit" are in scope here, and they're worth telling
apart:
**Safety limits that already exist and will grow.** `method="cartesian"`
refuses to build a coverage set past 50 million rows, naming the dimension
that caused it, rather than silently trying to allocate one, and
`generate()` says how large a frame will be before it allocates it — a
warning by default, a refusal with `max_bytes=`. That's the shape future
guardrails will take elsewhere in generation — an explicit, named refusal
before a runaway allocation, not a mysterious hang. Expect more of these as
generation is asked to handle larger and stranger specs: sanity limits on
distribution parameters, on cartesian dimensionality, on batch sizing.
**Constraints `generate()` doesn't enforce**, which is a different, more
interesting problem. What is left is `__checks__`, `ColSpec.validators` and
`ColSpec.pattern`, and that is by design: the first two wrap arbitrary Polars
expressions, the third is an arbitrary regex, and nothing can generate data
satisfying an arbitrary predicate.
Everything else on this list has been worked through — rule and foreign-key
dependencies by ordering the passes rather than asserting the dependencies
don't exist, and uniqueness by drawing without replacement instead of hoping a
wide domain would do. What remains is narrowing the gap from the other end:
letting a column *describe* its values well enough that a validator becomes
generatable.
**Domains generation can only partly express.** A `String` column can
say what its values look like through a named
[`format`](https://maxwellb13.github.io/polspec/how-to/formats/) — `uuid4`, `email`, `ipv4` and five
others — and is generated to satisfy it, so the validator
`col("email").str.contains("@")` no longer has to fail its own spec. The set
is closed, and [`pattern=`](https://maxwellb13.github.io/polspec/how-to/columns/#string-patterns) is the
honest other half: any regex can be *validated*, and only the curated set
can be generated. Generating from an arbitrary pattern is a much larger
piece of work with no answer for `.*`, and is not planned.
Both directions are active. Neither has a fixed shape yet, so the specific
options `generate()` accepts may well change under you.
## Specs know about each other through a `Registry`, and only there
A `ForeignKey` names the spec it points at; nothing above a single spec knows
which specs exist unless they are put in a
[`Registry`](https://maxwellb13.github.io/polspec/how-to/registry/). That is deliberate — two test modules may
each define an `Orders` — but it leaves edges:
- **`drift` takes one spec.** `polspec generate --all` and `validate --all`
run over every spec in a directory with the keys between them bound;
`drift` still takes one spec and its parents as `--references NAME=PATH`.
- **Discovery imports code.** `Registry.discover("specs/")` runs every `.py`
file it finds. A declared `Registry(...)` in a module of your own is the
safer shape, and `discover` is a convenience over it.
- **Shared categories are checked only when declared.** `resolve()` compares
columns against the `CatSpec` a registry was given; without one,
`catspec()` merges what the specs declare and refuses a disagreement, but
nothing checks unless asked.
- **A single spec's `to_mermaid()` still draws one entity.** The whole
picture is `registry.to_mermaid()`.
## YAML format and generated values may change
Two things this project has made no compatibility promise about yet:
- **The YAML spec format.** The keys `to_yaml()` writes and `from_yaml()`
reads are what today's `ColSpec`/`FrameSpec`/`CatSpec` happen to need. A
new field, a renamed key, or a different nesting for something like
distribution parameters could all still happen as the underlying Python API
settles.
- **The exact values `generate()` produces for a given seed.** Determinism
*within* a version is a hard guarantee — the same seed on the same version
always produces the same frame, and that's load-bearing for the round-trip
tests this project is built around. Determinism *across* versions is not
guaranteed yet: a bug fix to a distribution, a change to how a chunk's seed
is derived, or a fix to one of the [known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/) can
all legitimately change what a given seed produces. 0.7.0 was such a
release: every spec with rules, a foreign key, a hierarchy or a composite
key produces different values for the same seed than 0.6 did, once, so
that inserting a column never changes its neighbours again. 0.8.0 was
another, for batched output only: `generate_batches` and the sinks now
produce windows onto the frame `generate` would, whatever the batch size,
so every seeded batched or sunk output changed, once.
The first of those is easier to live with than it sounds, because a spec file
now says which format wrote it. Every file `to_yaml()` writes carries
`version: 3`; a file with no `version:` key is read as version 1 and migrated
on load, and one written by a newer polspec than the reader is refused by name
rather than misread. So a format change is a migration to write, not a class of
file that silently stops loading — which is what makes the rest of this section
a smaller promise than it looks.
What is still not promised is that a *given key* survives a minor release. A
renamed key needs a migration, and migrations are written when the rename
happens, not before.
Neither of these is likely to move for the sake of moving — but until this
page says otherwise, don't build something that depends on today's YAML
surviving a version bump byte-for-byte, or on a specific seed producing the
same values after an upgrade.
## Directions, not commitments
Lower confidence than everything above: opportunities noticed rather than gaps
being actively closed. They are here because the machinery each would need
already exists, not because any of them is started.
**Synthetic look-alike data.** `from_dataframe()` profiles real data into a
spec and `generate()` turns a spec back into data, so the trip from a real
table to a statistically similar fake one is already two calls. Making it one —
with `tags` marking which columns should be replaced outright rather than
imitated — would serve the share-realistic-data-without-sharing-real-data case
directly.
**A profiled spec that names a `format`.** `from_dataframe()` reads a
string column as a `String` with a length range; it does not notice that
every value is an email address. Inference is a decision about how sure to be
before naming a format, and a wrong guess is a spec that rejects real data,
so it has not been made. [Drift](https://maxwellb13.github.io/polspec/how-to/drift/) is the reason to want
it: a `format_violated` finding on a column the profiler named would have
been the drift that mattered.
**Test-framework integration.** A pytest fixture or plugin, or a Hypothesis
strategy built from a spec, are the natural adjacent surfaces for a library
whose whole pitch is that fixtures and contracts stay in step. Adjacent,
though — not core.
## Deferred on purpose
**`ColSpec` holding a `Domain` instead of `bounds`/`choices`/`format`.**
`polspec.constraints.Domain` is already the one definition generation,
validation, foreign keys and drift read; `ColSpec` still stores the three
fields it is derived from, and every module re-derives it. Folding the fields
into the domain would remove that repetition and nothing a user can see, at
the cost of touching every attribute read in the library. It has been
considered and set aside at each of the last three releases. The condition
it waited on — nested dtypes forcing a richer domain model — was tested by
0.9.0 and did not hold: a struct's fields are described by `ColSpec`s of
their own, so each field has an ordinary domain and `Domain` needed no
struct case. It stays deferred until something else asks for it.
---
# Changelog
Source: https://maxwellb13.github.io/polspec/changelog/
# Changelog
All notable changes to polspec are recorded here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Until 1.0, minor
versions may break the Python API, the YAML format, and the values a given
seed produces; see
[Roadmap and stability](https://maxwellb13.github.io/polspec/explanation/roadmap/).
## [Unreleased]
## [0.9.1] - 2026-09-24
A patch release around one question: *someone hands you a CSV, and there
is a spec it should meet -- what do you do?* The CLI now reads a text
file's dates the way the spec declares them, where before every CSV with
a date column failed `polspec validate`; the docs walk the flow end to
end; and bounds can be set aside while you find out what the ranges
really are. No seeded output changes, and no spec file needs migrating.
### Added
- **Checking a file you were given**, a section of
[Validating data](https://maxwellb13.github.io/polspec/how-to/validating/#checking-a-file-you-were-given):
read the file with `try_parse_dates=True` rather than the spec's schema
(which stops at the first bad value), `inspect()` it for every problem at
once, decide whether the file or the spec is wrong, then
`validate(cast=True)` for the typed frame. The example runs in the suite.
- **`polspec validate --skip CHECK`**, repeatable, for any of the
`validate_*` switches -- `--skip bounds --skip checks`. Its choices come
from the switch list itself, so a switch added later reaches the CLI.
- **`polspec.__version__`**, the installed version.
- **`validate_bounds=False`** turns off the bounds checks, beside the
other `validate_*` switches and as `ValidationOptions(bounds=False)`. The
other switches cover what generation cannot yet satisfy; this one is for
real data, whose ranges you may want to look at before holding it to
them while every other claim is still checked. It covers every `bounds`,
a `List`'s elements and a struct's fields included; `string_length` and
`list_length` stay on.
### Fixed
- **The CLI reads a text file's dates in the spec's terms.** A CSV or JSON
file has no date type, so a `Date` column arrived as text and validation
reported its dtype and checked nothing else -- every CSV with a date
column failed `polspec validate`, including one `polspec generate` had
just written. `validate`, `drift` and both `--all` modes now parse each
column the spec declares as a `Date`, `Datetime` or `Time` that arrived
as text, keeping the parse only when every value parses: a column with
a bad value stays text, so its `dtype` finding stays true, and a
`String` column of date-shaped text is left alone. `schema infer`, with
no spec to go by, reads a CSV with `try_parse_dates` and declares a date
column as a `Date`.
- **`estimated_size` no longer charges a null row for content it does not
hold.** A nullable text or `List` column was estimated as if every row
held a value -- 23% high for a string column at a 30% null rate, 32-38%
for a list. A string's bytes past its view and a list's elements now
scale by the share of rows expected to be present; views, offsets,
fixed-width values, an `Array`'s slots and a struct's fields are paid
on a null row too and are unchanged.
- **The API reference for `FrameSpec.validate`, `diff` and `drift`**
rendered prose after their parameter lists as parameters named `A`,
`that` and `Uniqueness`. It is a *Notes* section now, and a test parses
every docstring in the package as the reference does, failing on a
warning.
## [0.9.0] - 2026-09-24
Two things are **removed**, as 0.8.0 announced: `generate(lazy=True)`,
which is now a `TypeError` -- `scan()` is the lazy frame -- and the
`polspec[arrow]` extra, which no longer resolves. Both are under
*Removed* below. Everything else in the release is additive.
The headline is `Struct`, the last dtype polspec declared and could not
generate. `ColSpec(fields={name: ColSpec(...)})` says what is claimed about
each field's values, so a field is described exactly as a column of its
dtype would be, and every part of polspec reads it: generation, validation,
the spec file, `diff` and `drift`, `from_dataframe`, the data dictionary
and `estimated_size`. The same recursion generates a `List(Struct)`, a
`List(List)` and an `Array(Struct)`, so every dtype now generates, nested
to any depth. No seeded output of an existing spec changes, and no spec
file needs migrating.
### Added
- **Every dtype generates.** A `Struct` column is generated from its
[`fields`](https://maxwellb13.github.io/polspec/how-to/columns/#fields-what-a-structs-values-are)
-- one column per field, gathered -- and the same recursion generates a
`List(Struct)`, a `List(List)`, an `Array(Struct)` and a struct of
either, nested as deeply as the dtype goes. There is no dtype left that
polspec declares and cannot generate, and `test_roundtrip.py` pins that
claim rather than the list of exceptions it used to hold.
A field is drawn by the code that draws a column of its dtype, and is
seeded under its parent by name: renaming a struct column with
`seed_name` keeps every field, adding a field beside one moves nothing,
and a struct column is a window under `generate_batches` and `scan()`
like any other. The column's `nullable` is the *cell* -- a null struct
-- and a field's own `nullable` says whether it may be null inside a
struct that is present.
Validation checks each field's claims in place, and a finding names the
field: key `point.lat__bounds`, message `Column 'point.lat'`, while its
`columns` stay `("point",)` so `report.rows()` returns the offending
rows. Inside a list the samples are the offending lists, and a list of
lists tells its levels apart (`c__element_null`, `c[]__element_null`).
A struct in the data matches a declared one by field name rather than
order, each field compatible by the usual rules -- an `Int32` field
stands in for a declared `Int64` outside `strict_dtypes`.
- **`ColSpec(fields=…)`: what is claimed about a struct's field values.**
A `Struct` column's dtype is its schema -- every field's name and type
comes from it -- and `fields` is a `ColSpec` per field saying what its
values are, so a field is described exactly as a column of the same
dtype would be:
```python
ColSpec(
pl.Struct({"lat": pl.Float64, "lon": pl.Float64}),
fields={"lat": ColSpec(pl.Float64, bounds=(-90, 90))},
nullable=True,
)
```
It is **partial**: a struct of twenty fields where one needs bounds
spells one field. A name the dtype does not declare, or a field spec
whose dtype disagrees with the struct's, is refused where it is written,
as are `unique`, `rules`, `validators`, `seed_name` and `col_name` --
each a claim about a column among columns, which a value inside a
struct is not. A `List` of a `Struct` takes `fields` too, and a field
may itself be a struct, so a declaration nests as deeply as the dtype
does.
A `Struct` dtype now has a spec-file form (`{Struct: {name: }}`),
so a struct column round-trips through YAML and `to_python` -- and so do
`List(Struct)` and `List(List)`, which had no written form before. The
format version stays at 3: `fields:` is an added optional key.
- **The rest of polspec reads a struct too.** `diff` and `drift` compare
a struct's fields as they compare columns -- every comparator, keyed by
path, so describing `point.lat` with bounds is `domain_narrowed` on
`point.lat` (breaking, as on a column) and data escaping them is
`bounds_exceeded` there; a field's null rate is measured inside the
structs that are present. `from_dataframe` re-declares a struct column by
its fields, a `List` of structs by its element's, and a `List` of lists
by its outer length, where it used to record the dtype alone. The data
dictionary gives each field a row (`point.lat`) under its column, and
`estimated_size` costs a struct as the sum of its fields.
### Removed
- **`generate(lazy=True)`**, deprecated in 0.8.0. It built the whole frame
and called `.lazy()` on it, so it looked like the lazy verb while being
the eager one. `scan()` is the frame that has not been built; `.lazy()`
on a generated frame is a handle on one that has. Passing `lazy=` is now
a `TypeError`.
- **The `polspec[arrow]` extra**, redundant since 0.8.0 made the sinks
Polars' own. Nothing in polspec needs PyArrow, so `polspec[arrow]` no
longer resolves -- install `polspec` and, if you want PyArrow for your
own code, PyArrow.
## [0.8.0] - 2026-09-22
The headline is `scan()`: a `LazyFrame` that generates rows as they are
collected, so a plan that wants five rows of one column out of fifty
million generates five rows of one column. Projection is exact rather than
approximate, because 0.7.0 keyed every column and every pass by name.
Around it, a batch becomes a window onto one frame -- the same rows
whatever the batch size -- the sinks become `scan()` written out and stop
needing PyArrow, `estimated_size()` says how large a frame will be before
it is allocated, and `generate(lazy=True)` is deprecated in favour of the
verb that means it. Every seeded *batched* or *sunk* output changes, once;
`generate()` is unchanged, and no spec file needs migrating.
### Added
- **`estimated_size(n)`, and a word before a large frame is allocated.**
`Orders.estimated_size(50_000_000)` reads the answer off the declaration
-- the width of each dtype, the lengths the spec declares -- so it costs
nothing and needs no data. `generate()` now estimates before it
allocates: past four gibibytes it warns, naming the figure and pointing
at `scan()` and `generate_batches()`; `max_bytes=` makes it a refusal
instead, for a CI job that should fail rather than swap, and
`max_bytes=0` silences both. `polspec generate` prints the same note.
The estimate is the frame, not the process peak -- generation holds
working buffers on top, most visibly for `Decimal` and `List` -- and for
a frame of scalar columns it is within a percent of measured.
- **`scan()`: a `LazyFrame` that generates rows as they are collected.**
`Orders.scan(50_000_000, seed=1)` builds nothing; the plan decides what
is drawn. `.sink_parquet(...)` streams in bounded memory,
`.select("total").head(5).collect()` generates five rows of one column,
and a predicate filters rows that were drawn rather than narrowing the
draw. Projection is exact: every column is seeded by its name and every
pass by what it is for, so `lf.select(cols).collect()` is always
`lf.collect().select(cols)` -- where a column depends on others (a rule's
`when` columns, a composite key's members) the closure is generated and
dropped again on the way out. A scan is batched, so it carries
`generate_batches`' terms: a `__hierarchy__` is refused, and uniqueness
holds within a batch. `Registry.scan_all()` does the same for a set of
specs, generating parents eagerly -- a foreign key needs the whole parent
column -- and the children lazily. Built on polars' `register_io_source`,
which polars marks unstable; the tests pin behaviour rather than the
API surface.
- **`polspec drift --all specs/ data/`**, completing the `--all` trio:
every spec under a directory measured against the data file named after
it, specs without one listed and skipped, `--fail-on` deciding the exit
status across all of them, and `--json` printing one report per spec.
`diff` compares two declarations rather than a declaration and data, so
it takes no `--all`.
### Changed
- **The four `sink_*` functions are `scan()` written out.** Each is now
`scan(...)` handed to the matching `LazyFrame.sink_*`, so **PyArrow is no
longer needed for the Parquet and IPC sinks** -- nothing beyond Polars is
needed at runtime, and the `arrow` extra is redundant. Signatures,
defaults and the `n=0` behaviour are unchanged; what a sink's `**kwargs`
reach is now polars' own sink rather than `pyarrow.parquet.ParquetWriter`
or `pyarrow.ipc.new_file`, so a call passing a PyArrow-only writer option
needs the polars spelling instead. `compression=` is typed as the literal
polars accepts rather than `str`, so a misspelling is caught where it is
written. The `polspec[arrow]` extra still resolves and is removed in 0.9.
- **`generate(lazy=True)` is deprecated**, and removed in 0.9. It builds
the whole frame and calls `.lazy()` on it, so the memory is already
spent: use `scan()` for a frame that generates as it is collected, or
`.lazy()` on the result for a handle on an eager one.
- **A batch is a window onto one frame.** `generate_batches` and the
`sink_*` functions used to seed batch *k* from the *k*-th draw of the
caller's seed, so the same seed with a different `batch_size` was a
different frame and no batch could be made without the ones before it.
The engine now takes a row offset and numbers its chunks from it, so for
a column no pass rewrites `pl.concat(generate_batches(n, batch_size=b,
seed=s))` equals `generate(n, seed=s)` for every `b`; a `List` column's
lengths are a window too. Rules, foreign keys, composite keys and List
elements are still drawn per batch -- deterministic, keyed by the
batch's offset, but not row for row the whole frame's -- and uniqueness
still holds within a batch. **Every seeded batched or sunk output
changes**, once; `generate()` is unchanged. A batch smaller than the
engine's 65,536-row chunk costs up to one chunk of extra draws.
### Internal
- **A link to a heading that does not exist fails the test suite**, not
just `zensical build --strict`. `tests/test_docs.py` checked that a
linked *page* existed but never its anchor, so a wrong slug reached CI
before anyone saw it.
- `polspec.validation.constraints` is a package, one module per kind of
claim: `_values` (what one value must be, lifted over a List's elements),
`_rules`, `_table` (composite keys, checks) and `_relations` (foreign
keys, the hierarchy), with the `_Constraint` base and the dtype rule in
`_base`. No import outside the package changed.
- A generated column is dropped as it is finished, so a temporal, Binary
or gathered column never exists twice while the frame is assembled.
## [0.7.0] - 2026-09-22
The headline is nested and decimal dtypes: `List`, `Array` and `Decimal`
columns generate, validate and round-trip, described by the same fields a
scalar column takes. Around it, one change to what a seed produces -- the
passes that run after the columns are filled are now keyed by name, so
inserting a column never changes its neighbours -- and, alongside it, the
type checker moves to `ty` with nothing suppressed, the CLI gains
`--all`, and four long-standing limitations close, the last strict `xfail`
among them. Specs with rules, foreign keys, a hierarchy or a composite key
produce different values for the same seed than 0.6 did, once; see
*Changed*. No spec file needs migrating.
### Added
- **`pl.List` and `pl.Array` columns generate, validate, and round-trip.**
A nested column is described by the same fields as a scalar one, read as
claims about each element: `ColSpec(pl.List(pl.Int64), bounds=(0, 10),
list_length=(1, 5))`. `bounds`, `choices`, `weights`, `format`, `pattern`,
`string_length` and `distribution` describe the values inside the list;
the new `list_length` describes the list (an `Array` takes its length
from the dtype); `nullable` describes the cell, and an element is never
null. Generation draws the lengths and the elements as two columns of the
inner dtype and wraps one by the other, so an element is made by the same
code that makes a scalar of its dtype -- every generatable dtype, `Enum`,
`Decimal` and a `format` included -- and a `List` column keeps its data
across a rename through `seed_name`. Validation runs every element claim
inside the list and fails a list where any element does; `list_length`
is a new finding code and a null element a `nullability` one. `diff`
compares `list_length` like `string_length`; `drift` measures a List
column as its elements and its lengths; `from_dataframe` declares one by
its elements; `{List: }` and `{Array: {inner: , width: N}}`
are the spec-file forms. `unique` and `rules` are refused on a nested
column, and a `List` of a `List` or `Struct` declares and validates by
dtype only. What generation cannot fill is now `Struct` and those.
- **`pl.Decimal` columns generate.** A `Decimal(precision, scale)` is an
integer and a scale, so it is drawn as the integer through the engine's
64-bit kind and scaled back to the declared type -- with `bounds`, a
`distribution`, `choices`, nullability and cartesian coverage like any
other numeric column, and a default range of the float default or the
widest the precision allows. Bounds are held exactly (`int`,
`decimal.Decimal`, or a string read exactly, which is how a spec file
writes one); an endpoint finer than the scale is refused rather than
rounded. Validation compares values whatever the frame holds them as
(a CSV hands a Decimal back as a float), drift measures the extent in
the column's own type, `from_dataframe` declares one with its extent,
and `{Decimal: {precision: P, scale: S}}` is its spec-file form. The
ungeneratable dtypes are now the three nested ones.
- `ColSpec(seed_name=...)`: rename a column without changing the data it
generates. Each column is seeded from the frame seed and its *name*, so a
rename had always changed a column's values -- which matters when
generated frames are snapshots other things are compared against. A
column declared with `seed_name="old"` is seeded as `"old"` and keeps
producing what it did. Declared, not guessed: `rename()` does not set it,
and two columns of one spec cannot share one (they would draw identical
values). It holds across a rename and nothing else: the passes that run
after the columns are filled draw their seeds in declaration order, so an
inserted rules column still reshuffles the ones after it, and that
boundary is pinned and stated in *Known limitations*.
### Documentation
- *Generating data* said each column derives its seed "from its position".
It is from its name -- the reason `seed_name` is needed at all.
- **`polspec generate --all` and `polspec validate --all`.** With `--all`,
`SPEC` is a directory of specs: `generate` writes one `.`
per spec into `-o DIR`, parents first with their keys threaded into
their children, as `Registry.generate_all` does; `validate` checks each
spec against `DATA/.`, every spec seeing the others' files
as parents, as `Registry.inspect_all` does, exiting 1 when any fails and
printing one report per spec with `--json`.
### Changed
- **A rule leaves a null a null.** A nullable column with rules used to
lose nulls on the rows a rule matched, ending up below its declared
`null_probability`. The column's nullability is decided when it is drawn;
a rule says what a *value* on a matched row is, which is also all
validation checks it against.
- **`to_mermaid` marks one primary key.** A lone `unique=True` column is
the entity's `PK`; when several columns are unique each is a `UK`, as a
`__unique_together__` member already was.
- **A `CatSpec` refuses two entries differing only in case** (`STATUS` and
`status`), naming both: lookup is case-insensitive, so they were one
name with two answers.
- **Pass seeds are keyed by name, not drawn in order.** The passes that
run after the columns are filled -- rules, the hierarchy, foreign keys,
composite uniqueness, a bounded categorical's pool, cartesian coverage
representatives -- used to take their seeds from one `random.Random` in
declaration order, so inserting a rules column shifted every later pass
and `seed_name` could not hold a ruled column's data. Each now mixes the
frame seed with a key naming what it is for, by the same construction
the engine uses for columns. **A spec with any of those produces
different values for the same seed than 0.6 did**, once; a spec with
plain columns is unchanged, and so is every column's own draw. The
`limitations.md` bullet that described the old behaviour is gone.
### Fixed
- `generate(0, method="cartesian")` returns the empty typed frame
`generate(0)` and the sinks do, rather than the whole coverage set. The
last strict `xfail` in `tests/test_roundtrip.py` is gone with it.
### Internal
- **`ty` replaces mypy, and nothing is suppressed.** The 23-module ignore
list was one pattern: `ColSpec`, `Check` and `ForeignKey` annotated their
fields with what the constructor *accepts* (`pl.Int64` or `pl.Int64()`, a
tuple or a `Bound`, one validator or several) while `__post_init__`
narrows every one to what the instance *holds*. The fields now say what
they hold, and each class declares the accepted signature in an
`__init__` that exists only for type checkers, held to the fields by a
test. Every `spec.dtype.is_integer()` in the library type-checks; the
`cast()` accessors drift needed are gone; the API reference and
`llms-full.txt` show the accepted signature. CI runs `uv run ty check`.
- `polspec.cli` is a package, one module per verb, with the readers,
writers and spec loaders every verb shares in `_io`.
- `validate()`/`inspect()` and `diff()`/`drift()` shared 25 lines of
options handling twice; it is one `polspec._options.options_from`.
- `Bound.closed()`: both endpoints of a bound that has both.
## [0.6.0] - 2026-09-14
The headline is drift: a report of what changed between two specs, or
between a spec and the data it describes, with a severity per finding under
one mechanical rule -- and three CLI verbs so that report can gate a pull
request or a nightly load with no Python. Alongside it, `pattern=` completes
the split `format=` began. Nothing in the engine changes; no seed produces a
different frame, and no spec file needs migrating.
### Added
- **Drift as a report.** `polspec.drift.diff(old, new)` says what changed
between two declarations; `polspec.drift.drift(spec, df)` says how data
has moved relative to what its spec declares -- values outside the
domain, a bound exceeded and by how much, a format no longer matched, a
null rate that moved, declared values never seen. Both return one
`DriftReport`, also reachable as `Orders.diff(Other)` and
`Orders.drift(df)`:
```python
report = OrdersV1.diff(OrdersV2)
report.breaking # a narrowed bound, an added column, a new constraint
report.compatible # a widened domain, a removed check
report.to_markdown() # the shape of a pull-request comment
```
Every finding carries a `severity` under one mechanical rule: *breaking*
when a frame that satisfied the old side could fail the new one --
decided, for dtypes, by the same function validation uses. Two tests hold
the two sides to it: what `generate()` produces never drifts breakingly
from its own spec, and every breaking finding against data is a column
`validate()` reports. Sixteen closed finding codes, a `DriftOptions`
with explicit tolerances, and a comparator per `ColSpec` and `TableSpec`
field with a parity test, so a field added to a declaration is one entry
or one red test. `DriftReport`, `DriftFinding` and `DriftOptions` are
exported from `polspec`.
- `ColSpec(pattern=...)`: a regular expression every value of a `String`
column must match, **checked by validation only**. Generation does not
read it -- the column is filled with ordinary random text, and the round
trip holds only with `validate_pattern=False`, exactly as for
`validators`. That is the honest half of the split `format=` made: any
regex can be checked, the curated set can be generated. A `pattern`
finding code, a `validate_pattern` switch on `ValidationOptions`, one
`pattern:` key in spec files (the format version stays at 3: an added
optional key is not a new version, and `migrations.py` now says so).
Cannot be combined with `format`; compiled by Polars' own engine at
declaration, so a pattern Polars cannot run is refused with its message.
- **Three CLI verbs.** `polspec generate SPEC -n N -o FILE` writes generated
rows to any format the CLI reads (`--seed`, `--method`, `--references`
as for `validate`); `polspec diff OLD NEW` and `polspec drift SPEC DATA`
print a drift report as text, `--json` or `--markdown`, and gate on it
with `--fail-on breaking|any|none` -- so a schema change in a pull
request, or a nightly load, can fail CI with no Python:
```bash
polspec generate orders.yaml -n 1000 -o orders.parquet --seed 1
polspec drift orders.yaml orders.parquet
polspec diff main/orders.yaml pr/orders.yaml --markdown --fail-on breaking
```
### Documentation
- Five claims the docs had stopped being true about are fixed, and one of
them is pinned: `unique=True` and `__unique_together__` had been listed as
work generation does not attempt (generated since 0.2.0); `generate()` on
an unsupported dtype raises `SpecError`, not `TypeError`; every yaml
example carried `version: 2` at format version 3, and a test now holds
each `version:` in the docs to `FORMAT_VERSION`.
- The [roadmap](https://maxwellb13.github.io/polspec/explanation/roadmap/)
gains a *Deferred on purpose* section, starting with the `ColSpec` ->
`Domain` restructure that two release plans had set aside without saying
so on the page, and the condition under which it would be revisited.
## [0.5.0] - 2026-09-11
The headline is `format=`: a `String` column that says what its values look
like, and is generated to satisfy its own validator. Around it, the release
closes five silent edges, settles the validation option surface, spells out
the `FrameSpec` signatures, and puts a type checker in CI. One narrow
breaking change, under *Changed*.
### Added
- `ColSpec(format=...)`: a `String` column that says what its values look
like, and is generated to satisfy its own validator. Eight formats --
`uuid4`, `email`, `ipv4`, `ipv6`, `mac`, `hostname`, `iso_country`,
`iso_currency` -- each with a sampler in the engine and a check in
validation declared side by side in `polspec.formats`, and pinned by a
round trip per format:
```python
class Users(FrameSpec):
user_id = ColSpec(pl.String, format="uuid4", unique=True)
email = ColSpec(pl.String, format="email")
Users.validate(Users.generate(1_000_000, seed=42)) # passes
```
Validation reports a new `format` finding. A format is one `format:` key in
a spec file and takes part in the domain check a foreign key runs at
declaration. It cannot be combined with `choices` or `string_length`, and
only a `String` column can carry one; each is refused with a message
saying which to drop. What a format promises is syntax: see *Known
limitations*.
- `ValidationOptions` is exported from `polspec`, and `validate()` and
`inspect()` take it as `options=`. Every switch as one value, for when the
same settings go through several calls:
```python
lenient = ValidationOptions(extra_cols="drop", checks=False)
Orders.validate(df, options=lenient)
Customers.validate(other_df, options=lenient)
```
`options=` and the individual keywords are alternatives rather than a base
and an override, so passing both raises instead of quietly picking one.
### Changed
- `FrameSpec.validate`, `inspect` and the four `sink_*` classmethods spell
their options out instead of forwarding `**kwargs`. Editors complete
them, a type checker sees a typo, and a mistyped option is a `TypeError`
from the classmethod's own signature rather than from a function several
frames away. The validation keywords default to `None`, meaning "the
`ValidationOptions` default", so the defaults are still defined once. The
sinks keep their trailing `**kwargs`: that is the documented passthrough
to the underlying writer, not a gap.
- A misspelled spec name in `references={...}` now warns instead of passing
silently. A key nothing points at was skipped without a word, so the column
was generated freely while the caller believed the parent had been used --
and `validate()` then reported the key as unresolved, which is the round
trip failing with only its second half audible. The warning names the key
that went unfilled and suggests the one supplied:
```
Orders: references={...} supplied ['Custmers'] that no foreign key points
at, while 'Customers' (supplied 'Custmers'?) went unfilled.
```
It takes both halves to warn -- something supplied that went unused *and*
something unused that went unfilled -- so supplying no parent at all stays
silent, as documented, and a `Registry` handing every spec the whole set of
frames says nothing either. `generate_batches` warns once per call rather
than once per batch.
- `ColSpec(pl.Int64, null_probability=0.9)` now warns. `nullable=False` still
wins and the rate is still ignored -- turning nullability off should not
also require deleting the rate beside it -- but the same silence covered
asking for nulls and forgetting `nullable=True`, where the column generates
none and nothing says why. Only a rate that cannot be a leftover warns: the
default and an explicit `0.0` already agree with `nullable=False`.
- `references=` given something that is not a mapping raises `SpecError`
naming the three key forms it accepts, rather than an `AttributeError` from
inside `resolve_references`. Every collection argument elsewhere in the API
is a sequence, so passing one here was an easy mistake with an unhelpful
answer, and it was the one complaint polspec made that was not a
`PolspecError`.
- An unknown validation option names the option you meant. It used to surface
as `ValidationOptions.__init__() got an unexpected keyword argument
'validate_uniqe'`, naming a private class that is not exported and not the
option intended; it is now `Unknown validation option(s): 'validate_uniqe'
(did you mean 'validate_unique'?)`, with the accepted list.
- **Breaking, narrowly: the check switches have one spelling.**
`inspect(spec, df, unique=False)` used to work alongside
`validate_unique=False`, while `validate(spec, df, unique=False)` raised --
a second public spelling reachable through half the API, from a rename map
that was meant to be internal. Only the `validate_*` form is accepted now,
by both verbs. The bare names live on as the fields of `ValidationOptions`,
which is where they were always meant to be.
### Documentation
- `Decimal` joins `List`, `Struct` and `Array` on the list of dtypes that
declare and validate but cannot be generated. It had been missing from both
[Dtype coverage](https://maxwellb13.github.io/polspec/explanation/roadmap/)
and [Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/),
and it is the one people miss, being the only one of the four that is not a
nested type.
- A `Datetime` carrying a `time_zone` generates, which the docs had never said
either way and readers assumed meant no. Both claims are now pinned by tests
in `tests/test_roundtrip.py`, so neither page can go stale.
### Internal
- The four copies of "declares no ColSpec columns" are one
`tablespec.require_columns`.
- **mypy runs in CI.** The package ships `py.typed` and a stub for the Rust
extension, so every consumer's type checker trusts these signatures, and
nothing was checking them. The first run found 247 errors, of which 87 trace
to two declarations: `ColSpec.dtype` and `ForeignKey.references` are both
annotated with what the constructor accepts rather than what the instance
ends up holding. Narrowing either is a design decision on a public field, so
the modules carrying that backlog are listed in `pyproject.toml` and
everything else is enforced -- the list can only shrink.
- The predicate nodes in `expr.py` no longer each carry their own
`root_names`, `literals` and `rename`. A node reports its operands through
`children()` and the three traversals are derived from that on `Pred`, which
turns thirty-six implementations into twelve. `rename` is the one that
mattered: the base implementation returned `self`, so a node that forgot to
override it left a renamed spec pointing at a column that no longer existed,
with nothing raised. Forgetting `children()` now raises.
- `References` and `Method` were declared identically in four and two modules;
`_collect`/`_to_lazy` in three. They are one `polspec.frames`.
- The four `sink_*` functions built the same six-argument batch-stream call
each. `_prepare` now returns the checked call as one value. The public
signatures stay spelled out, which is what makes a typo in one of them fail
at the call site.
- `registry.py` imported `serialization` lazily in five methods and `report` in
a sixth, while importing `generation` and `validation` at module level. There
was no cycle; all six are hoisted.
## [0.4.1] - 2026-09-09
The hierarchy release. A self-referencing `ForeignKey` says that every parent
value exists somewhere in the frame, and nothing more. This adds the
declaration that says the rest of it -- one parent per reference, a bounded
depth, no cycles -- and makes `generate()` satisfy it rather than leaving it to
the draw.
### Added
- **`Hierarchy`**, declaring that two columns of a spec are a parent/child
edge list drawn on one pool of references: a child pointing at its parent,
and that parent pointing at its own parent, are the same row shape.
```python
class Links(FrameSpec):
PARENT_REF = ColSpec(pl.String)
CHILD_REF = ColSpec(pl.String)
__hierarchy__ = Hierarchy(
child="CHILD_REF", parent="PARENT_REF", max_depth=5
)
```
`generate()` produces a real forest: one parent per reference, so every row
resolves to a single ultimate parent, and no chain longer than `max_depth`
with at least one reaching it exactly -- so a test of a graph walk exercises
the boundary rather than whatever the draw happened to give. `branching`
shapes the tree, or `roots` pins the number of ultimate parents.
`generate(n, cycles=10, self_references=5)` then breaks it on purpose, which
is the other half of testing a graph walk: a resolver written without a
visited set does not fail on a loop, it runs forever. The spec still says the
data should be acyclic, so `validate()` reports what was injected --
`hierarchy_cycle`, `hierarchy_depth` and `hierarchy_multi_parent` are new
finding codes -- and a test can assert that its own resolver and polspec
agree about what is broken.
Both checks are bounded, so validating deliberately cyclic data terminates:
depth costs `max_depth` steps and cycle detection walks by pointer doubling,
covering a million-row chain in about twenty. A validator that walked until
it reached a root would hang on the fixtures this feature exists to make.
`generate_batches` and the `sink_*` functions refuse a spec carrying one: a
forest spans the whole frame, and batches are generated independently. See
[Hierarchies and link tables](https://maxwellb13.github.io/polspec/how-to/hierarchies/).
### Documentation
- The spec file format is version 3, which adds the `hierarchy:` key. A
version 2 file loads unchanged.
- [Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/)
now points a self-referencing `ForeignKey` at `Hierarchy` for the case that
wants a real tree, rather than carrying a recipe of its own. Two tests pin
the two apart.
## [0.4.0] - 2026-09-08
The internals release. 0.2.0 and 0.3.0 settled the vocabulary; this one goes
underneath it, to the Rust generator and the places where the same table was
being maintained in two or three languages.
Nothing about how a spec is written changes. One thing does break, and it is
the same thing the roadmap has always reserved: **the values a given seed
produces are different**. Any test asserting on specific generated values
needs re-baselining; a test asserting on their *properties* -- bounds,
distinctness, null share, distribution shape -- does not. polspec's own suite
needed no changes, which is the shape of test this library is built to support.
Generation got faster, by between a tenth and a third depending on the column.
Measured A/B against v0.3.0 on one machine, same build profile, twenty million
rows: the four-column frame in `benchmarks/bench.py` 1.30x, a
`unique=True` Int64 column 1.21x, a bounded nullable Int64 column 1.17x, an
Enum column 1.25x, a String column unchanged. Treat the ratios rather than the
absolute numbers as the claim.
### Added
- `generate`, `generate_batches`, `inspect`, `validate`, `sink_parquet`,
`sink_ipc`, `sink_csv` and `sink_ndjson` are exported from `polspec` itself.
Each takes a `TableSpec` as its first argument and each is what the matching
`FrameSpec` classmethod already called -- but they lived in
`polspec.generation` / `polspec.validation`, which the API reference calls
internal and free to change in a patch release. So the `TableSpec`-first
half of the library had no stable import path; now it does, and both halves
appear in [the API reference](https://maxwellb13.github.io/polspec/reference/api/).
### Changed
- **Breaking: the values a given seed produces have changed.** polspec now
builds on `rand` 0.10 (from 0.8), whose samplers draw differently. Same seed,
same version, same frame -- as before; across this version boundary, not.
- A generated numeric, boolean, temporal or categorical column arrives as
**one chunk** rather than one per 65,536 rows. The values buffer and the
validity bitmap are each allocated once at full length and filled in parallel
through disjoint slices, instead of being built per chunk and appended
together. Nothing downstream now pays for a column split into hundreds of
pieces -- the gather behind a `choices` domain, the cast behind a temporal
dtype, every `sink_*` write.
String columns are the exception and stay chunked, because Polars backs them
with view arrays: merging those copies no string bytes, but it does copy
sixteen bytes of view per row, which costs more than the split it removes.
They still gain the other half of the change -- the chunks are collected in
one go rather than appended one at a time, and an append rescanned both sides
for their first and last non-null value to maintain a sorted flag that random
strings will not have set anyway.
- Drawing a `unique=True` column no longer materialises its domain. A domain
only a little wider than the row count used to be built in full and partially
shuffled, which allocates in proportion to the domain rather than to the
output: ten million distinct values from a range of eighty million reserved
1.4 GB before writing anything. That branch is now Floyd's algorithm, which
holds only the values it has chosen -- the same case now peaks at 491 MB.
Roomier domains keep drawing and rejecting, which is faster there and was
never the memory problem.
- Every character of the generated-string alphabet is now exactly equally
likely. Six random bits give 64 values for a 62-character alphabet, and the
two spare ones fell back on `% 62` over a fresh 32-bit draw, which is biased
by about one part in 70 million -- far too little to see, but free to remove:
the fallback now rejects properly instead.
- `rand` 0.8 was compiled alongside the `rand` 0.10 that Polars already links,
so the extension carried two copies of `rand`, `rand_core` and their chacha
backends. There is now one of each.
- The release profile builds the crate as a single codegen unit. Measured on
one machine against otherwise identical v0.3.0 code, that alone is worth
2.1x on the `unique=True` path, for about twenty seconds of build time. Fat
LTO on top of it was tried and dropped: a further 5% for eight more minutes
per build.
### Documentation
- A self-referencing `ForeignKey` guarantees that every parent value exists,
and nothing more -- in particular not that the result is a tree. Parents are
sampled from the whole frame, so some rows end up in a cycle or pointing at
themselves, which is what makes a recursive query fail to terminate, and
`validate()` does not report it because no part of a spec can say "acyclic".
[Known limitations](https://maxwellb13.github.io/polspec/explanation/limitations/)
now says so, with a recipe for a genuine hierarchy, and the roadmap carries
what closing the gap would need. Two tests pin the behaviour.
### Fixed
- `Registry.validate_all` applies each report's structural transformations
using the same bound spec the report was produced against, rather than the
unbound copy.
### Internal
- `ColumnPlan::build` takes a `PlanArgs` struct instead of thirteen positional
arguments, so a call site names what it sets and leaves the rest to
`Default`. Four `#[allow(clippy::too_many_arguments)]` and a great many
`None`s went with it.
- `Kind`'s three parallel lists -- the names, the parse, the reverse lookup --
are generated from one declaration, so a new column kind cannot be added to
two of them and forgotten in the third.
- The fixed-width integer ranges were written three times: once as polspec's
default generation range, once as the limits a declared bound may not exceed,
and once as the Rust samplers' defaults. The first is now read from the
second.
- The four `sink_*` functions share a typed batch-stream helper rather than
forwarding `**kwargs`.
- `benchmarks/bench_generate.py` is replaced by `benchmarks/bench.py`, which
measures the same comparison and adds a regression mode. The old harness
timed one run per case, in a process shared with the implementations it was
comparing against, and recorded nothing about the machine -- so its numbers
varied by around 25% between runs and could not be compared across days. It
also measured exactly one column shape, which is how a change that made the
`unique` path half as fast came within an afternoon of being released as a
speed-up. The new one takes the fastest of several runs, gives every
measurement its own process, repeats a short case until its floor settles,
records the CPU, thread count, Polars version and cargo profile, and covers
each column kind, both branches of the unique draw, the cartesian, rule,
foreign-key and composite-key passes, and a sink. `record` writes a local
baseline and `check` exits non-zero when a case regresses past a tolerance;
repeated measurements now agree to within about 2%.
- The benchmark table in the comparison guide is re-measured. It had been
recorded on 2026-09-03, which is before both 0.2.0 and 0.3.0, so it had been
describing an engine two releases old: the four-column frame at twenty
million rows was published as 0.0827s, measures 0.1409s on v0.3.0, and
0.1127s here. Some of that gap is still unaccounted for and is worth
chasing. The NumPy and pure-Python columns re-measure to within 1% of what
was published, which is what says the difference is polspec's and not the
machine's.
## [0.3.0] - 2026-09-06
The refactor 0.2.0 started, finished. `CatSpec` was the one declarative
surface left doing everything in one class; it is now a value with a
metaclass facade, like `TableSpec` and `FrameSpec`. Behind it came the fixes
that were waiting for a release allowed to break something.
One thing breaks, and it is worth reading before you upgrade: naming a
`CatSpec` entry now always gives back the dtype, whichever form declared the
registry. `pl.Enum(cats.STATUS)` becomes `cats.STATUS`, and
`cats.CURRENCY.physical()` becomes
`cats.get_categorical("CURRENCY").physical()`. Nothing else in the public API
changed shape.
The documentation gained a test: every Python example in `docs/` is executed
by the suite, which found five broken examples that had been shipping.
### Changed
- **Breaking.** `CatSpec` is a value, and the class body that declares one is
read by a metaclass rather than left in the namespace -- the same split
`TableSpec` and `FrameSpec` already had. What follows:
- **Naming an entry always gives back the dtype.** It used to depend on how
the registry was built: a class-body entry was a real class attribute and
returned the dtype, while a dict-built registry's `.STATUS` returned the
raw category list, and only one of the two could be handed to a `ColSpec`.
Both now return the dtype, so `ColSpec(cats.STATUS)` and
`ColSpec(Categories.STATUS)` mean the same thing. `cats["STATUS"]` and
`cats.get("STATUS")` follow the same rule. Replace `pl.Enum(cats.STATUS)`
with `cats.STATUS`, and `cats.CURRENCY.physical()` with
`cats.get_categorical("CURRENCY").physical()`.
- **An entry may share a name with a method.** Entries are removed from the
class body before the class exists, so declaring one called `get` no longer
warns and no longer costs you `CatSpec.get`. The entry is reached through
the registry (`Categories.spec.get("get")`).
- **`CatSpec.infer_from_dataframe` and `CatSpec.infer_from_framespec` are
removed**; `CatSpec.infer(target, ...)` dispatches on what it is given, as
it already did. `from_dataframe` and `from_framespec` are unchanged --
those read what is declared rather than inferring what could be.
- **`Categories.spec`** is the `CatSpec` a class body declares. Anywhere a
registry is expected -- `with_catspec`, `Registry(categories=...)`,
`FrameSpec.from_yaml(categories=...)` -- the class and the value are now
interchangeable.
- **`CatSpec` has value semantics.** Two registries that say the same thing
compare equal and hash equal, so one loaded from a file can be checked
against one a class body declares.
- **`CatSpec.dtype_of(name)`** is the one lookup everything else is built on:
the dtype an entry names, or None. `resolve_key` still says which kind of
entry a name binds to.
- `enums`, `categoricals` and `choices` are read-only mappings rather than
fresh dicts. `dict(cats.enums)` if you need a mutable copy.
### Added
- `polspec.MultiValidationError`, raised by `Registry.validate_all` when
several frames fail at once. It is a `ValidationError`, so an existing
`except` clause still catches it, and it carries every failing spec's
`ValidationReport` as `reports`, keyed by spec name -- previously
`validate_all` raised with a joined string and the reports were lost, so
`failing_rows()`, `by_code()` and `to_json()` were unreachable from the
registry path.
- `polspec.CliError` is exported, so `except polspec.CliError` works. It was
the one exception in the hierarchy reachable only from `polspec.errors`.
- Every Python example in the documentation is executed by
`tests/test_doc_examples.py`. A block that cannot run standalone says so in
an HTML comment (``), and one that demonstrates an error
is checked to still raise (``). This found four broken
examples, fixed here: a `drop()` of a column the page never declared, a
`TableSpec` example rebinding the name a later block used, and two blocks
naming frames (`broken_df`, `existing_df`) that were never built.
### Fixed
- `ColSpec(tags={...})` is reproducible. A `set` was kept in its own iteration
order, which Python salts per process, so `to_yaml` wrote a different `tags:`
line on every run and two identically-written specs compared unequal across
processes. A set is now sorted; a list or tuple keeps the order it was
written in.
- `Registry.generate_all`, `generate_related` and `inspect_all` bind their
cross-spec foreign keys before doing anything, so a key whose dtypes do not
match is a `RegistryError` naming both columns rather than a Polars cast
error from inside generation. Only `resolve()` used to run that check, and
nothing said it had to be called first. A key whose target is supplied
through `references=` rather than held by the registry is still accepted, as
it was.
- A `ColSpec` carrying the same validator twice keeps it once, so it produces
one finding rather than two identical ones. `TableSpec` already collapsed
identical checks and foreign keys.
- `generate_batches` and the `sink_*` functions resolve `references` once per
call rather than once per batch. A `LazyFrame` parent was collected inside
every batch, so a scan-backed parent was re-read as many times as there were
batches.
- `inspect()` no longer raises a raw Polars error for a column whose dtype is
wrong *and* whose spec declares `choices` or an `Enum`. The domain check was
built before the dtype check could bail out, and comparing values against
choices of another type is not something Polars will compile at all, so the
frame most likely to arrive -- a column read back from CSV or JSON as the
wrong type -- crashed instead of reporting a `dtype` finding.
- A `ColRule` whose condition is null on a row no longer excuses every later
rule on that row. Generation folds a null `when` to `False` before testing it
and before accumulating it into the claimed mask; validation did neither, so
the null propagated through Kleene logic and left rows that generation *had*
rewritten unchecked.
- `TableSpec` is hashable, so `references={Orders.spec: df}` works. It is one
of the three forms `generate()` and `validate()` document, and the only one
that could not be put in a dict: the dataclass's generated `__hash__` cannot
hash a mapping of columns, nor a `ColSpec` carrying `distribution_params`.
### Documentation
- The install sections of the README and the documentation home said polspec
was not published to PyPI, directly below a `pip install polspec` block. Both
now describe the published wheels, and point at `CONTRIBUTING.md` for
building from a checkout.
- The documentation workflow runs on changes to `python/**` and
`scripts/generate_llms_txt.py`. The API reference is `:::` directives filled
in by mkdocstrings from the live docstrings, so a docstring that breaks
`--strict` used to pass its own pull request and fail the next one to touch
`docs/`.
- The roadmap's "YAML format may change" section described the missing format
version key that 0.2.0 shipped, and said an unsupported dtype raises
`TypeError` rather than `SpecError`.
- `FINDING_COLUMN` and `ValidationOptions` are documented in
the validation guide; both are exported and appeared nowhere.
- `how-to/tablespec.md` taught `polspec.generation.generate(spec, ...)` while
the API reference says anything unlisted may change in a patch. The page now
says which of the two it is.
- `CONTRIBUTING.md` gives the runnable form of the Windows `cargo test`
workaround, and names the `STATUS_DLL_NOT_FOUND` failure it fixes.
## [0.2.0] - 2026-09-05
The architecture release. Specs became data, constraints gained one definition
each, and the generator learned to satisfy claims it used to only check.
This release breaks a lot. Every incompatible change below is marked
**Breaking** and says what to do instead. Three are worth knowing before you
upgrade: `Spec._columns` and friends are now `Spec.spec.columns`; the values a
given seed produces have changed, so any test asserting on generated values
needs re-baselining; and several declarations that used to be accepted and
quietly misbehave are now refused at declaration time.
### Added
- `docs/llms.txt` and `docs/llms-full.txt`, published at the documentation
site root in the [llms.txt](https://llmstxt.org) format: an index of every
page, and the full text of all of them in one file. Generated by
`scripts/generate_llms_txt.py` from the nav, the pages, and -- for the API
reference, whose source is `:::` directives -- the live docstrings, so a
language model reads signatures rather than an empty page. A test fails if
either file is stale.
- `polspec.constraints`: the definitions generation and validation both read,
so they cannot drift. `Domain` is what a column may hold (its `choices`, an
`Enum`'s categories, its `bounds`); `Pass` and `order` decide which rewrite
of a generated frame runs first, from the columns each one reads and writes.
- `unique=True` is generated, not just validated. The engine draws the column
without replacement (`src/unique.rs`): it shuffles a materialised domain
when the domain is barely larger than the frame, and rejects against a set
when it is roomy. Every dtype is covered, nulls are exempt (a nullable
unique column may repeat nulls and nothing else), and a domain too small to
cover the row count is refused by name instead of quietly producing
duplicates. `method="cartesian"` holds unique columns out of the coverage
product and draws them once over the finished frame.
- `__unique_together__` is generated. A pass resamples the rows repeating a
combination an earlier row already used, so only the repeats move and the
rest keep the values their own columns' weights and bounds gave them. Rows
with a null member are exempt, matching validation. A group whose columns
cannot take enough distinct combinations is refused, naming the group; a
foreign-keyed member is never resampled, since that would break its key.
- The Rust boundary is typed. Python builds one `ColumnPlan` per column (a
`#[pyclass]` validated at construction, with errors naming the column)
instead of a positional tuple. Bounds cross as an `i64`, `u64` or `f64`,
so `Int64`/`UInt64` bounds beyond 2^53 are exact and the generation clamp
for an unbounded distribution reaches the dtype's true limits. Columns
with a finite domain (`choices`, `Enum`) receive indices back and the typed
values are gathered on the Python side, so a `datetime`, `bytes` or `True`
choice never passes through a string; choices need only be distinct in the
column's dtype, not as strings. `python/polspec/_polspec.pyi` is a stub for
the extension; `src/` is split into `plan.rs`, `dist.rs` and `sample.rs`
with unit tests under `cargo test`; a test compares the distribution
parameter tables on both sides.
- `import polspec` works without the Rust extension: validation, spec files,
the registry and the report renderers need no build. Only generation
imports it, and raises one actionable `ImportError` when it is missing.
- `Registry`: a declared set of specs. `Registry(Customers, Orders, ...)`
resolves foreign keys declared against names (`resolve()`, running the
checks a class-bound key gets at declaration), orders parents before
children (`order()`), generates the whole set with every key satisfied
(`generate_all`, with a per-spec seed so adding a table changes no other;
`generate_related` for one spec and its ancestors), validates it in one
call (`inspect_all`, `validate_all`), merges or checks shared categories
(`catspec()`, `categories=`), writes and reads one file for the set
(`to_yaml`/`from_yaml`, a `specs:` mapping plus `categories:`), collects
specs from modules and directories (`from_module`, `discover`), and draws
one entity-relationship diagram (`to_mermaid`). See the new *Multiple
specs* guide.
- `inspect()`: validation results as data. `FrameSpec.inspect(df)` (and
`polspec.validation.inspect(spec, df)`) returns a `ValidationReport` of
`Finding` records -- each with a code, a stable key, the columns involved,
a count, samples and code-specific details -- and never raises for a bad
frame. `report.rows(finding)` and `report.failing_rows()` return the
offending rows lazily; `by_column()`, `by_code()` and `to_json()` slice
and export them. Checks and validators now carry samples too.
- `ValidationError.report` carries the same `ValidationReport`; `.errors` is
still the list of messages.
- `polspec validate SPEC DATA [--references NAME=PATH] [--json]` on the
command line, exiting 1 on findings, so a spec can gate a pipeline in CI.
- A foreign key whose parent was not supplied is a `foreign_key_unresolved`
finding rather than a `ValueError`, matching how `generate()` already
treats it; a parent lacking the referenced columns is a `foreign_key`
finding.
- Spec files carry a `version:` (now 2). Files from version 1 are migrated
on read; a file from a newer polspec is refused with a clear message. A
key the reader does not know is an error naming the closest known key;
`from_yaml(..., strict=False)` downgrades it to a warning.
- Foreign keys to other specs are written to YAML and Python as the target's
name and read back unresolved, instead of being dropped with a warning.
- `polspec.serialization` is a package driven by one field registry
(`fields.py`): YAML in both directions, generated Python, and the
`import datetime` decision all derive from it, and a test asserts every
dataclass field has an entry. `to_dict`/`from_dict` are public.
- `CatSpec` files keep choices recorded for plain string columns.
- `polspec.col()`, a small predicate language for rules, validators and
checks: `col("total") >= col("subtotal")`, `col("email").str.contains("@")`,
`is_in`, `is_between`, `is_null`, `&`/`|`/`~`, arithmetic, and string
operations. A predicate evaluates like the Polars expression it stands
for and, unlike one, is written to and read from YAML and generated
Python. `__checks__` and `ColSpec.validators` written with `col()` now
round-trip through `to_yaml`/`from_yaml` and `to_python`. Raw `pl.Expr`
is still accepted and still warns on export.
- `ColRule.when` accepts a predicate, so a rule may depend on several
columns. The one-column dict form is still accepted and converted.
- `TableSpec`: the spec as an immutable value. A `FrameSpec` class body now
builds one, reachable as `Spec.spec`, and every verb (`generate`,
`validate`, `to_yaml`, `to_markdown`, ...) is a function over it in
`polspec.generation`, `polspec.validation`, `polspec.serialization` and
`polspec.report`. `TableSpec` offers `with_columns`, `drop`, `select`,
`rename`, `with_checks`, `with_foreign_keys`, `with_unique_together`,
`with_name` and `with_catspec`; `FrameSpec.from_spec` wraps one in a class.
See the new *Specs as values* guide.
- `FrameSpec.col(name)` reaches a column whatever it is called.
- `ForeignKey.references` may be a spec's name, for keys whose target is not
importable where the key is declared.
- An exception hierarchy under `PolspecError`: `SpecError` for declarations
that cannot mean anything, `ValidationError` for data that fails its spec,
`GenerationError` when a spec cannot be turned into data (including every
error raised inside the Rust engine), `SerializationError` for files that
cannot be written or read, and `RegistryError`, reserved for the spec
registry. All are exported from `polspec`; see the new *Errors* reference
page.
### Changed
- **Breaking.** `ColRule.when` no longer accepts the one-column dict
(`{"column": "region", "equals": "UK"}`). `col()` is the only spelling:
write `col("region") == "UK"`. A spec file written by an earlier version
still loads -- its conditions are converted as the file migrates -- but a
file declaring the current version must carry the predicate form. The error
names the column and says what to write.
- **Breaking.** The `le` and `ge` condition keys are gone; they were
undocumented duplicates of `lte` and `gte`. (`le`/`ge` remain the canonical
operator names in a predicate's *data* form, which is unrelated.)
- **Breaking.** `polspec.serialization` no longer re-exports the names of the
pre-package module layout: `_YAML_DTYPES`, `_YAML_NAME_TO_DTYPE`,
`_dtype_to_yaml`, `_dtype_from_yaml`, `_dtype_to_python`, `_colspec_to_yaml`,
`_colspec_from_yaml` and `_colspec_to_python`. Use the names in
`polspec.serialization.fields` and `polspec.serialization.dtypes`.
- **Breaking.** `ColRule.when` is evaluated against the frame as it stands
when the rule runs, not against the freely generated values. Rules and
foreign keys are applied in dependency order, so a rule keyed on a column
that another rule or a foreign key rewrites now reads the rewritten values
-- the ones `validate()` checks it against. Chained rules, chained foreign
keys, and a rule keyed on a foreign-keyed column all round-trip; the values
a given seed produces for such a spec change.
- **Breaking.** Two columns whose rules each read what the other writes have
no order that satisfies both, and are now refused at declaration with a
`SpecError` naming them.
- **Breaking.** A `ForeignKey` whose parent's declared domain does not fit
inside its own column's is refused at declaration (or when a `Registry`
resolves a key that names its target as a string). A key overwrites its
column with the parent's values, so `bounds=(1, 50)` on a column
referencing keys in `100..200` could only ever generate data that fails its
own validation. A column declaring no `bounds` or `choices` still accepts
anything.
- **Breaking.** `unique=True` can no longer be combined with `weights`, a
non-uniform `distribution`, or `rules`. The first two describe how often a
value recurs, which a draw without replacement has no room for; a rule
assigns from a fixed set, which is how duplicates would get back in. Each
is refused at declaration rather than silently ignored.
- **Breaking.** A column carrying `rules` may not also be part of a
`__unique_together__` group, for the same reason: the repair that separates
repeated combinations would overwrite what the rule put there.
- **Breaking.** A `ForeignKey` filling a `unique=True` column now refuses when
the parent holds fewer distinct values than there are rows, instead of
falling back to sampling with replacement and producing the duplicates the
column forbids.
- `polspec test` no longer emits `validate_unique=False` in generated tests.
Uniqueness is generated now, so the generated test asserts it.
- A foreign key spanning textual dtypes -- a `String` column referencing an
`Enum` key, which declaration has always allowed and generation has always
handled -- now validates instead of raising `SchemaError` from the
anti-join. The parent's keys are cast to the local dtype for the join, so
`ValidationReport.rows()` still returns the frame's own rows unchanged.
- **Breaking.** Each column's generation seed is derived from the frame
seed and the column's *name*, not its position, so inserting a column no
longer reshuffles the columns after it. The values a given seed produces
change from previous versions.
- `ColRule` application samples only as many values as there are matched
rows and scatters them into place, instead of filling the whole column per
rule.
- `polspec.validation` is a package (`report.py`, `constraints.py`); foreign
key anti-joins are collected together with `pl.collect_all` instead of one
`collect` per key.
- **Breaking.** `ColSpec.distribution` and `distribution_params` are stored
in canonical form (`"exp"` becomes `"exponential"`, `mu`/`sigma` become
`mean`/`std`, and so on), so spec files are canonical. Every alias is
still accepted when declaring.
- **Breaking.** An unrecognised physical dtype in a `CatSpec` entry is now a
`SerializationError` instead of silently becoming `UInt32`.
- **Breaking.** `ColRule.when` is a predicate after construction rather
than a dict (`rule.when.root_names()` lists the columns it reads); rules
in YAML are written in the predicate data form, and the old dict form is
still read.
- **Breaking.** A column may now share a name with a `FrameSpec` method:
the metaclass takes `ColSpec` attributes out of the class namespace, so
`schema`, `tag` and friends no longer shadow anything and no longer warn.
The private `_columns`, `_checks`, `_unique_together` and `_foreign_keys`
class attributes are gone; read `Spec.spec.columns` and friends instead.
- **Breaking.** `ForeignKey.references` is the target's *name* after
construction (the bound spec is available as `ForeignKey.target`), and
`references={...}` on `generate`/`validate` accepts the class, the
`TableSpec` or the name as key.
- **Breaking.** Removed: the `FrameSchema` alias; `FrameSpec.generate_catspec`,
`write_catspec`, `infer_catspec` and `with_inferred_catspec` (use
`catspec()`, `catspec().to_yaml()`, `CatSpec.infer(...)` and
`with_catspec(CatSpec.infer(...))`); the `max_unique` and `bounds` alias
keyword arguments of `from_dataframe` (use `max_unique_enum` and
`calculate_bounds`).
- `to_yaml` and `to_python` share one set of warnings about what a file
cannot hold.
- **Breaking, mildly.** Errors that were bare `ValueError` or `TypeError`
are now the subclass above. Each keeps the built-in type it replaced, so
`except ValueError` still catches it; only code matching on the exact type
(`type(exc) is ValueError`) sees a difference. Plain argument misuse
(`n < 0`, an unknown `method=`) is unchanged.
- The command line prints any `PolspecError` as a one-line `error: ...`
instead of a `TypeName: message` line.
### Fixed
- Foreign-key sampling during generation drew parent keys from an unordered
`unique()`, so the same seed could give different child rows between runs.
The parent's distinct keys now keep their order and generation is
reproducible.
## [0.1.5] - 2026-09-03
### Added
- `python/polspec/py.typed`, so type checkers use the package's annotations.
- `CONTRIBUTING.md`, this changelog, and a `.python-version` file.
- `examples/related_specs.py`: a worked example of four related specs
(foreign keys, shared categories, rules, checks, a YAML-declared spec). It
runs in CI as a smoke test.
- A release-workflow job that refuses a `vX.Y.Z` tag whose version does not
match `pyproject.toml`.
- CI now runs `ruff check` with a wider rule set, `ruff format --check`, `cargo fmt --check`,
`cargo clippy -D warnings` and `cargo test`, tests on macOS as well as
Linux and Windows, and tests against the newest Polars release inside the
declared bound. The docs build runs strictly on pull requests.
- Release builds now produce wheels for Linux aarch64 and macOS (x86_64 and
arm64) alongside Linux and Windows x86_64, plus an sdist, and only publish
when the test workflow is green.
- An optional `.pre-commit-config.yaml` with ruff and cargo fmt hooks.
- `tests/test_colspec.py` (2,000 lines, unsectioned) is split into
`test_generation.py`, `test_rules.py`, `test_serialization.py`,
`test_profiler.py`, `test_framespec.py`, `test_report.py` and
`test_foreign_key.py`, each with a docstring saying what it covers.
### Changed
- The crate version in `Cargo.toml` is a placeholder; `pyproject.toml` is the
only place the version is set, so `uv version --bump` works.
- The `parquet`, `ipc` and `all` extras (all identical) are replaced by a
single `arrow` extra. Install with `polspec[arrow]` for the Parquet and
Arrow IPC sinks.
- `polars` is bounded to `<2`; the Rust extension is coupled to a Polars
release line.
- The abi3 floor is now Python 3.12, matching `requires-python`.
- `cargo test` links again (`extension-module` is no longer an unconditional
crate feature; maturin enables it).
### Fixed
- Repository URL in package metadata pointed at the repository's old name.
- README claimed the license was unspecified; it is MIT.
- Documentation: `ColSpec(col_name=...)` is now described in *Declaring
columns*, `FrameSpec.to_python()` in *YAML specs*, the getting-started
example imports `date`, and the architecture page lists the `cli` module
and its tests.
## [0.1.4] - 2026-09-02
### Added
- `polspec schema infer --output spec.py` and `FrameSpec.to_python()`, which
write a spec as an editable Python module rather than YAML.
## [0.1.3] - 2026-09-01
### Added
- `ColSpec(col_name=...)`, so a column's name in data may differ from the
attribute name used to declare it.
### Changed
- Roadmap expanded with detailed plans for a spec registry, structured
validation results, and generation guardrails.
## [0.1.2] - 2026-08-31
Version bump only; no user-facing change.
## [0.1.1] - 2026-08-31
### Added
- Test workflow on GitHub Actions (Linux and Windows, Python 3.12 to 3.14).
### Fixed
- `ColSpec.dtype` accepts a dtype class as well as an instance.
## [0.1.0] - 2026-08-31
First tagged release.
- `ColSpec` and `FrameSpec`: declare a Polars schema with nullability,
bounds, string lengths, choices and weights, distributions, tags, and
conditional `ColRule`s.
- `generate()` backed by a parallel Rust extension, `method="cartesian"` for
coverage sets, batched generation and Parquet/CSV/IPC/NDJSON sinks.
- `validate()` collecting every violation in one Polars aggregation, with
column validators, multi-column `Check`s, composite uniqueness and
`ForeignKey`s.
- `CatSpec` registries for shared `Enum`/`Categorical` domains.
- YAML round-trip, `from_dataframe()` profiling, Markdown and Mermaid output.
- CLI: `polspec schema infer`, `polspec schema new`, `polspec test`.
- Documentation site, comparison guide, and release automation.
[Unreleased]: https://github.com/MaxwellB13/polspec/compare/v0.9.1...HEAD
[0.9.1]: https://github.com/MaxwellB13/polspec/compare/v0.9.0...v0.9.1
[0.9.0]: https://github.com/MaxwellB13/polspec/compare/v0.8.0...v0.9.0
[0.8.0]: https://github.com/MaxwellB13/polspec/compare/v0.7.0...v0.8.0
[0.7.0]: https://github.com/MaxwellB13/polspec/compare/v0.6.0...v0.7.0
[0.6.0]: https://github.com/MaxwellB13/polspec/compare/v0.5.0...v0.6.0
[0.5.0]: https://github.com/MaxwellB13/polspec/compare/v0.4.1...v0.5.0
[0.4.1]: https://github.com/MaxwellB13/polspec/compare/v0.4.0...v0.4.1
[0.4.0]: https://github.com/MaxwellB13/polspec/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/MaxwellB13/polspec/compare/v0.2.0...v0.3.0
[0.2.0]: https://github.com/MaxwellB13/polspec/compare/v0.1.5...v0.2.0
[0.1.5]: https://github.com/MaxwellB13/polspec/compare/v0.1.4...v0.1.5
[0.1.4]: https://github.com/MaxwellB13/polspec/compare/v0.1.3...v0.1.4
[0.1.3]: https://github.com/MaxwellB13/polspec/compare/v0.1.2...v0.1.3
[0.1.2]: https://github.com/MaxwellB13/polspec/compare/v0.1.1...v0.1.2
[0.1.1]: https://github.com/MaxwellB13/polspec/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/MaxwellB13/polspec/releases/tag/v0.1.0