Command line¶
polspec does at the shell what a spec does in Python: create one from data,
turn one into a test, generate data from one, check data against one, and
say what moved. Every verb is a thin wrapper over a FrameSpec method that
already exists — from_dataframe, to_yaml, generate, validate, diff,
drift — so the CLI is argument parsing and templating, not new behaviour.
This page walks through what each command is for; the
command-line reference lists every argument.
polspec schema infer orders.parquet -o orders.yaml
polspec schema new Orders -o orders.py
polspec test orders.yaml -o test_orders.py
polspec generate orders.yaml -n 1000 -o orders.parquet --seed 1
polspec validate orders.yaml orders.parquet
polspec diff orders_v1.yaml orders_v2.yaml --markdown
polspec drift orders.yaml orders.parquet
polspec synthesize orders.parquet -o fake_orders.parquet --replace customer_name
schema infer — profile data into a spec¶
polspec schema infer SOURCE -o OUTPUT.yaml [options]
polspec schema infer SOURCE -o OUTPUT.py [options]
polspec schema infer SOURCE -o - [options]
SOURCE is a .csv, .tsv, .parquet, .ndjson/.jsonl, .json, or Arrow
IPC (.arrow/.ipc/.feather) file. It is read with the matching Polars
reader and profiled with FrameSpec.from_dataframe, rounding what it writes;
Infer a spec from data covers the options from Python.
OUTPUT's extension picks the format: .yaml/.yml writes a YAML spec via
FrameSpec.to_yaml; .py writes a FrameSpec subclass via
FrameSpec.to_python — a starting-point module you can edit like any other
source file, rather than a data file from_yaml re-parses.
-o - prints the YAML instead of writing a file, to read the spec before
keeping it or to pipe it on. The summary line and any notes go to stderr,
so redirecting stdout captures only the spec. Added in 1.3.
polspec schema infer orders.parquet -o - | less
polspec schema infer orders.parquet -o - > orders.yaml
$ polspec schema infer orders.parquet -o orders.yaml --weights
Inferred 3 column(s) from 12,483 row(s) of orders.parquet -> orders.yaml
$ polspec schema infer orders.parquet -o orders.py --weights
Inferred 3 column(s) from 12,483 row(s) of orders.parquet -> orders.py
"""Declares the Orders schema."""
import polars as pl
from polspec import ColSpec, FrameSpec
class Orders(FrameSpec):
__columns__ = {
"order_id": ColSpec(pl.Int64, bounds=(0, 13000)),
"status": ColSpec(pl.Enum, choices={"NEW": 0.4, "PAID": 0.3, "SHIPPED": 0.3}),
"total": ColSpec(pl.Float64, bounds=(10.0, 500.0)),
}
order_id ran from 1 to 12,483, and status was 40.2% NEW: bounds are
written rounded outward and frequencies to two significant figures, as the
options below describe. A column with no nulls says nothing about them, as
one written by hand would not.
These are not polspec.profile()'s defaults, on purpose. schema infer
describes data, as FrameSpec.from_dataframe does, for a person to read
and edit: no frequencies, keys or row order unless asked (--weights,
--keys, --order), and rounded numbers, so ids from 1 to 150 are written
[0, 150]. profile() imitates data, for generating a stand-in, and
records everything, exactly. --round-bounds exact --round-rates exact
writes what was observed. See How the pieces
fit.
Columns are declared through __columns__ rather than as class attributes,
same as from_yaml — see Column names that are not
identifiers. The .py
output is passed through ruff format when it's on PATH, same as schema
new.
Every option, with its default, is in the
command-line reference: what to
record (--weights, --shape, --keys, --order), what to leave out
(--no-bounds, --no-formats, --replace), and how finely to write it
(--round-bounds, --round-rates). Infer a spec from data
explains each from Python.
Treat the output as a draft. It describes the sample it saw — edit bounds, add rules, tighten a domain — before trusting it as a contract.
Several files — a registry¶
Given several files, schema infer profiles them together and writes one
registry: a spec per file, named after it (customers.csv is Customers),
with the foreign keys between them found from their values. .yaml writes
a registry file; .py writes a class per file, parents first, and a
registry = Registry(...) after them; -o - prints the YAML.
polspec schema infer customers.csv orders.csv lines.csv -o shop.yaml
polspec schema infer customers.csv orders.csv -o shop.py --foreign-key Orders.buyer=Customers.id
--foreign-key CHILD.COLUMN=PARENT.COLUMN declares a key the values do
not show, and is checked against them; --no-foreign-keys finds none.
--name names one spec, so it is refused with several files. How a key is
found is in Related tables; the
tutorial on data you already have walks
through it. Added in 1.5.
--no-patterns leaves out the pattern a code-like text column's values
share (^ORD-[0-9]{6}$), and --parse-text reads text as the numbers,
flags and dates it holds, in the format it is written in, saying why a
column that nearly parses stays text. See
Data read as text.
The draft is written as a person would write it: bounds rounded outward and
rates rounded up, to two significant figures — amount: [3.17, 9871.23]
becomes [0, 9900] and a 2.37% null rate 0.024 — so the file reads as a
choice rather than a measurement. Rounding outward and up keeps the data
valid against it. --round-bounds 1k --round-rates 0.01 picks the
precision, and exact writes what was observed:
schema new — start from nothing¶
Writes a blank FrameSpec with the two imports it will need and a few
commented ColSpec examples, for the case where there's no data yet to
profile.
test — a round-trip test from a schema¶
SOURCE is a .yaml/.yml spec (from schema infer, or written by
FrameSpec.to_yaml) or a .py file defining one or more FrameSpec
subclasses (from schema new, filled in). The generated file asserts the
property this project's own test suite is built around:
def test_orders_roundtrip(polspec_frames):
frame = polspec_frames(Orders, 500)
assert_valid(frame, Orders)
def test_orders_cartesian_coverage(polspec_frames):
frame = polspec_frames(Orders, 500, method="cartesian")
assert_valid(frame, Orders)
The frames come from the polspec_frames fixture,
seeded per test from --polspec-seed, and a failure says how to rerun it.
--seed N pins every frame to one seed instead. The options — --rows,
--seed, --no-cartesian, --class — are in the
command-line reference. Changed in 1.3:
the tests used to call generate() and validate() with seed 42.
It will not hand you a test that fails on the spot¶
generate() does not attempt everything validate() checks — see
Known limitations. A spec using __checks__
or ColSpec.validators, or a pattern the generator cannot follow, would
otherwise get a test that fails the moment it runs, since generation cannot
satisfy them, and one declaring __rows__ a test of the wrong size. The generator checks for each and turns
off the matching validate() switch, with a comment explaining why:
def test_invoices_roundtrip(polspec_frames):
# __checks__ wraps arbitrary expressions that generation cannot be made
# to satisfy
frame = polspec_frames(Invoices, 500)
assert_valid(frame, Invoices, checks=False)
A spec with a foreign key into another spec declared in the same file generates that spec first, and its parents in turn:
def test_lines_roundtrip(polspec_frames):
parents = {}
parents[Customers] = polspec_frames(Customers, 500)
parents[Orders] = polspec_frames(Orders, 500, references=parents)
frame = polspec_frames(Lines, 500, references=parents)
assert_valid(frame, Lines, references=parents)
One whose parent is declared in another file — or left out by --class —
is marked
@pytest.mark.skip with a reason, rather than guessed at:
@pytest.mark.skip(
reason=(
"Child has foreign key(s) into Parent, which this file does not load. "
"Generate the test from a file declaring it too, or pass "
"its frame as references={Spec: frame} -- see docs/how-to/testing.md."
)
)
def test_child_roundtrip():
pass
Similarly, the cartesian test is only emitted when the spec actually has
something for method="cartesian" to build coverage from — an Enum,
Boolean, or bounded numeric column. A spec of only unbounded strings gets a
comment instead of a test that would raise ValueError on the first run.
Regenerating¶
The generated file names the command that made it:
"""Generated by `polspec test orders.yaml`.
Regenerate with:
polspec test orders.yaml -o test_orders.py
This file is only overwritten by running that command again -- edit freely.
"""
It is a plain file, not managed state — add assertions, rename the functions, delete the parts you don't want. Nothing re-reads it.
generate — data from a schema¶
polspec generate orders.yaml -n 1000 -o orders.parquet --seed 1
polspec generate specs.py --class Orders -n 500 -o orders.csv --references Customers=customers.parquet
polspec generate orders.yaml -n 50 -o edge_cases.ndjson --method cartesian
polspec generate orders.yaml -n 1000 -o paid.parquet --fixed status=PAID --fixed qty=1
Generates -n rows and writes one file; the extension picks the format
(.parquet/.pq, .csv, .tsv, .ndjson/.jsonl, .json,
.arrow/.ipc/.feather — the same set validate and drift read).
--seed makes the file reproducible; --method cartesian guarantees
coverage the way generate()
does; --references NAME=PATH supplies parent data for a foreign key, as
for validate. --fixed COLUMN=VALUE pins a column to one value in every
row, as fixed= does: the value is
read as the column's dtype, as validate reads text (3, true,
2026-01-02), and empty text is a null. --fixed added in 1.4.
The frame is built in memory and written once. For a file too large to hold,
the streaming sink_* functions
are a Python surface. A CSV or TSV cannot hold a Duration, List or
Struct column; Polars refuses to write one, so use Parquet or Arrow IPC
for a spec that has any.
--all — every spec in a directory¶
polspec generate --all specs/ -n 1000 -o data/ --seed 1
polspec generate --all specs/ -n 1000 -o data/ --format csv
With --all, SPEC is a directory (or any file Registry.discover
accepts) and -o is a directory: every spec found is generated, parents
first with their keys threaded into their children as
Registry.generate_all does, and
written as <name>.<format> (Parquet by default). No --references are
needed for keys between the discovered specs; supply them for a parent
outside the directory. A pin names its spec: --fixed Orders.status=PAID.
synthesize — fake data from real data¶
polspec synthesize customers.parquet -o fake_customers.parquet -n 1000000 --seed 1 --replace name email --spec fake_customers.yaml
Profiles the file — shapes, keys, category frequencies, null rates — and
writes a fake one with the same schema, -n rows (as many as the source,
by default), or --scale times as many: --scale 0.1 is a tenth.
--replace names the columns whose values must not be carried over;
--sample N profiles a random sample of N rows; --spec writes the
profiled spec beside the data, to review or to regenerate from. See
Fake data from real data for what carries over.
Given several files, it writes a fake set into a directory, one file per source, every key between them held by a fake parent row:
polspec synthesize customers.csv orders.csv lines.csv -o fake/ --scale 0.1 --seed 1
polspec synthesize customers.csv orders.csv -o fake/ --format parquet --spec fake/shop.yaml
Each file keeps its own format unless --format names one for all, and
--spec writes the registry. --foreign-key and --no-foreign-keys work
as for schema infer. With --parse-text, a date read in a format, such
as 30 Sep 2031, is written back in it to a CSV or NDJSON file.
Several files added in 1.5.
validate — check data against a schema¶
polspec validate orders.yaml orders.parquet
polspec validate specs.py orders.parquet --class Orders --references Customers=customers.parquet
polspec validate orders.yaml orders.csv --json > report.json
Reads a data file, in any of the formats generate writes, runs
inspect() against the spec, and
prints the report: the same text validate() would raise, or the full
structured report with --json. The exit status is 0 when the data passes
and 1 when it does not, so a spec can gate a pipeline step in CI with no
Python at all.
The JSON is ValidationReport.to_dict(), led by a schema_version:
{
"schema_version": 1,
"spec": "Orders",
"passed": false,
"findings": [
{"code": "bounds", "key": "total__bounds", "message": "Column 'total': ...",
"columns": ["total"], "count": 1, "samples": [-1.0],
"details": {"bounds": [0, 10], "min_found": -1.0, "max_found": 1.0}}
]
}
The keys, and the types of their values, are what
Stability promises; a key
may be added, and schema_version rises if one is ever renamed, removed or
changes type.
--output PATH writes the rows that passed, typed as the spec declares —
they are validated again with cast=True, so the file is one the spec
accepts — and --failing PATH writes the rows that did not, each once, in
the file's order, with a __polspec_finding column naming every claim it
broke, comma-separated. The two files split the input exactly: their rows
add up to the file's. (In Python, report.failing_rows(once=True) is the
same view; plain failing_rows() lists a row once per claim, for grouping
by claim.) Either or both; the
extension picks the format, as for generate, and the exit status still
says whether the whole file passed, so a pipeline can quarantine the bad
rows and fail the step. A structural finding (a missing column, a wrong
dtype) judges the whole file, so neither is written:
--references NAME=PATH supplies parent data for a foreign key to another
spec, by that spec's name; repeat it for several. --allow-extra and
--allow-missing relax the structural checks; --strict-dtypes tightens the
dtype check. --fill-defaults fills each defaulted column's nulls, and adds
it when absent, before checking, as
fill_defaults=True does. --skip CHECK
turns off one kind of check, as CHECK=False does in Python —
--skip bounds --skip checks — and takes any of rules, derived,
validators, unique, checks, foreign_keys, hierarchy, order,
rows, pattern and bounds.
--tolerate [KEY=]SHARE accepts findings up to a share of the rows, as
tolerate= does: a
bare share for every finding, or one by finding code, column, or
column:code, repeated for several. A share is 0.001 or 0.1%. A
tolerated finding is printed, marked, and the exit status is 0 if nothing
else failed. Added in 1.7.
A CSV, TSV or JSON file has no date type, so a date arrives as text — and
JSON holds a decimal or a duration as text too. The CLI reads each column
the spec declares as a Date, Datetime, Time, Decimal or Duration
as that type when every value in it parses exactly, as
read() does; a column holding a value that does not stays
text, and is reported as a dtype finding rather than turned into a null. A String column of date-shaped text is left alone. drift reads
the same way, and schema infer, with no spec to go by, recognises a CSV's
dates itself.
--all — every spec against the file named after it¶
With --all, SPEC is a directory of specs and DATA a directory of data
files named after them (Orders.parquet for Orders, in any format the
CLI reads). Every spec with a file is checked, each seeing the others'
files as its parents, as Registry.inspect_all
does; specs with no file are listed and skipped. The exit status is 1 when
any report fails; --json prints one document with every report keyed by
spec name under reports — {"schema_version": 1, "reports": {"Orders":
{...}}} — since a spec may be called anything.
diff and drift — what moved¶
polspec diff orders_v1.yaml orders_v2.yaml # two schemas
polspec diff specs_v1.py specs_v2.py --class Orders --rename id=order_id
polspec drift orders.yaml last_night.parquet # a schema and data
polspec drift orders.yaml last_night.parquet --markdown > drift.md
diff runs diff() between two spec files;
drift runs drift() between a spec and
a data file. Both print the report as text, --json, or --markdown (the
shape of a pull-request comment, breaking findings first).
The exit status is decided by --fail-on: breaking (the default) exits
1 when any finding would break validation; any exits 1 on any finding
at all, a widened bound included; none always exits 0, for posting a
report without gating on it. So a schema change in a pull request, or a
nightly load, can be gated with no Python:
--strict-dtypes makes any dtype change breaking, as it does for
validate. drift also takes a flag for every other
DriftOptions field — --significance,
--null-rate-tolerance, --frequency-tolerance,
--distribution-tolerance, --no-unseen, --max-samples — each
defaulting to the option's own default, and --sample N, a seeded random
sample of N rows.
drift --all — every spec against the file named after it¶
As for validate --all: SPEC is a directory of specs and DATA a
directory of files named after them. Every spec with a file is measured,
specs without one are listed and skipped, and --fail-on decides the exit
status across all of them; --json prints the reports under reports, as
validate --all does. diff compares two declarations rather than a
declaration and data, so it has no --all.
Exit codes and errors¶
Every subcommand returns 0 on success and 1 on a reported error, printed
as error: ... on stderr rather than a traceback — a missing file, an
unreadable format, an invalid class name. validate, diff and drift
also return 1 when the report itself fails, so a 1 means "look at the
output", whichever kind of problem it was. A command line that is itself
wrong — an unknown verb, flag or choice — returns 2, with a usage message.
Stability says which of
these are promised.