polspec¶
Declare a Polars schema once. Generate data that matches it, and validate data against it — from the same declaration.
import polars as pl
from polspec import ColSpec, FrameSpec
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, None))
status = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]))
total = ColSpec(pl.Float64, bounds=(0.0, None))
placed = ColSpec(pl.Date, nullable=True)
df = Orders.generate(1_000_000, seed=42) # a million rows in milliseconds
Orders.validate(df) # raises ValidationError on any breach
The generator is written in Rust and runs the columns in parallel, so a spec that describes a realistic table produces millions of rows in the time it takes to describe one.
Why two directions from one declaration¶
Most schema tools do one or the other. A validation library tells you when production data drifted; a fixture library gives you something to test against. Keeping both behind one declaration means the fixtures and the contract cannot disagree — and where they might, polspec has a test suite whose whole job is to catch it (see Known limitations).
That is the practical payoff: the data in your tests is data your validator already accepts, so a test that passes locally is not passing on a shape production will reject.
What you can declare¶
-
Types and shape
Every dtype, generated and validated — integers, floats,
Decimal, booleans, strings, binary, all four temporal types,Enum,Categorical, and aList,ArrayorStructof any of them, nested to any depth, and on Polars 2 aMap— plus nullability, bounds, string lengths, value domains, named string formats such asuuid4andemail, and regex patterns, which are generated from. -
Rules and invariants
Conditional values, columns derived from others, single-column validators, multi-column checks, composite uniqueness, the order rows come in, how many rows a table holds, foreign keys between specs, and parent/child hierarchies with a bounded depth.
-
Checking data
validate()raises on any breach;inspect()returns every finding as data, with the offending rows a filter away.read()loads a data file in the spec's terms first, parsing dates, decimals and durations that arrived as text, and declared defaults can fill what is missing. -
Data on demand
Random or coverage-guaranteeing generation, reproducible seeds, columns pinned to the values a test needs, and a
LazyFramethat generates only the columns and rows a plan asks for — streaming straight to Parquet, CSV, Arrow IPC or NDJSON. -
Specs from elsewhere
Infer a spec by profiling existing data, load one from YAML so non-Python tooling can read it too, derive one by joining two, and keep a set of related specs in one
Registrythat generates and validates them together.Infer a spec · Specs as files · Multiple specs · A spec for data you already have
-
Fake data from real data
polspec.synthesizeturns a real dataset — a frame, a lazy scan or a file — into a fake one with the same schema, shapes, frequencies and keys, and none of its rows or sensitive values. -
Tests that say what broke
assert_validfails a test with the findings, the failing rows and the seed to rerun it with; a pytest fixture gives each test frames from a seed of its own;strategyhands a spec to Hypothesis. -
What changed
Diff two versions of a spec, or a spec against data, into a report that says what moved and whether a frame that passed before could fail now.
-
From the command line
polspec generatewrites a data file from a spec,polspec validatechecks one,polspec diffandpolspec driftgate a pull request on what changed, andpolspec testturns a schema into a pytest round-trip.
Install¶
The generator is a compiled Rust extension, but wheels are published for Linux (x86_64, aarch64), macOS (Intel and Apple silicon) and Windows (x86_64), so installing needs no Rust toolchain. At runtime polspec needs:
- Polars, any release from 1.39, Polars 2 included (Python 3.15 needs Polars 2). The same seed generates the same data on each, and on every operating system.
- PyYAML.
- tzdata, on Windows only, for time-zone data.
Building from a checkout — which does need Rust and maturin — is covered in CONTRIBUTING.md.
Where to go next¶
Start with Getting started for the full loop — declare, generate, validate — in about five minutes. If you have the data and not yet a spec, A spec for data you already have profiles a set of files into one and makes a fake copy for tests. If you're weighing polspec against a hand-rolled fixture, Faker, or a data-quality framework, see Comparison to other approaches for where each one fits and the benchmark numbers behind the speed claim.
For language models¶
The documentation is published in the llms.txt format:
/llms.txt indexes every
page, and /llms-full.txt
carries the full text of all of them — including the API reference, expanded
to signatures and docstrings — in one file.