Skip to content

Command line

polspec does at the shell what a spec does in Python: create one from data, turn one into a test, generate data from one, check data against one, and say what moved. Every verb is a thin wrapper over a FrameSpec method that already exists — from_dataframe, to_yaml, generate, validate, diff, drift — so the CLI is argument parsing and templating, not new behaviour. This page walks through what each command is for; the command-line reference lists every argument.

polspec schema infer orders.parquet -o orders.yaml
polspec schema new Orders -o orders.py
polspec test orders.yaml -o test_orders.py
polspec generate orders.yaml -n 1000 -o orders.parquet --seed 1
polspec validate orders.yaml orders.parquet
polspec diff orders_v1.yaml orders_v2.yaml --markdown
polspec drift orders.yaml orders.parquet
polspec synthesize orders.parquet -o fake_orders.parquet --replace customer_name

schema infer — profile data into a spec

polspec schema infer SOURCE -o OUTPUT.yaml [options]
polspec schema infer SOURCE -o OUTPUT.py [options]
polspec schema infer SOURCE -o - [options]

SOURCE is a .csv, .tsv, .parquet, .ndjson/.jsonl, .json, or Arrow IPC (.arrow/.ipc/.feather) file. It is read with the matching Polars reader and profiled with FrameSpec.from_dataframe, rounding what it writes; Infer a spec from data covers the options from Python.

OUTPUT's extension picks the format: .yaml/.yml writes a YAML spec via FrameSpec.to_yaml; .py writes a FrameSpec subclass via FrameSpec.to_python — a starting-point module you can edit like any other source file, rather than a data file from_yaml re-parses.

-o - prints the YAML instead of writing a file, to read the spec before keeping it or to pipe it on. The summary line and any notes go to stderr, so redirecting stdout captures only the spec. Added in 1.3.

polspec schema infer orders.parquet -o - | less
polspec schema infer orders.parquet -o - > orders.yaml
$ polspec schema infer orders.parquet -o orders.yaml --weights
Inferred 3 column(s) from 12,483 row(s) of orders.parquet -> orders.yaml

$ polspec schema infer orders.parquet -o orders.py --weights
Inferred 3 column(s) from 12,483 row(s) of orders.parquet -> orders.py
"""Declares the Orders schema."""

import polars as pl
from polspec import ColSpec, FrameSpec


class Orders(FrameSpec):
    __columns__ = {
        "order_id": ColSpec(pl.Int64, bounds=(0, 13000)),
        "status": ColSpec(pl.Enum, choices={"NEW": 0.4, "PAID": 0.3, "SHIPPED": 0.3}),
        "total": ColSpec(pl.Float64, bounds=(10.0, 500.0)),
    }

order_id ran from 1 to 12,483, and status was 40.2% NEW: bounds are written rounded outward and frequencies to two significant figures, as the options below describe. A column with no nulls says nothing about them, as one written by hand would not.

These are not polspec.profile()'s defaults, on purpose. schema infer describes data, as FrameSpec.from_dataframe does, for a person to read and edit: no frequencies, keys or row order unless asked (--weights, --keys, --order), and rounded numbers, so ids from 1 to 150 are written [0, 150]. profile() imitates data, for generating a stand-in, and records everything, exactly. --round-bounds exact --round-rates exact writes what was observed. See How the pieces fit.

Columns are declared through __columns__ rather than as class attributes, same as from_yaml — see Column names that are not identifiers. The .py output is passed through ruff format when it's on PATH, same as schema new.

Every option, with its default, is in the command-line reference: what to record (--weights, --shape, --keys, --order), what to leave out (--no-bounds, --no-formats, --replace), and how finely to write it (--round-bounds, --round-rates). Infer a spec from data explains each from Python.

Treat the output as a draft. It describes the sample it saw — edit bounds, add rules, tighten a domain — before trusting it as a contract.

Several files — a registry

Given several files, schema infer profiles them together and writes one registry: a spec per file, named after it (customers.csv is Customers), with the foreign keys between them found from their values. .yaml writes a registry file; .py writes a class per file, parents first, and a registry = Registry(...) after them; -o - prints the YAML.

polspec schema infer customers.csv orders.csv lines.csv -o shop.yaml
polspec schema infer customers.csv orders.csv -o shop.py --foreign-key Orders.buyer=Customers.id

--foreign-key CHILD.COLUMN=PARENT.COLUMN declares a key the values do not show, and is checked against them; --no-foreign-keys finds none. --name names one spec, so it is refused with several files. How a key is found is in Related tables; the tutorial on data you already have walks through it. Added in 1.5.

--no-patterns leaves out the pattern a code-like text column's values share (^ORD-[0-9]{6}$), and --parse-text reads text as the numbers, flags and dates it holds, in the format it is written in, saying why a column that nearly parses stays text. See Data read as text.

The draft is written as a person would write it: bounds rounded outward and rates rounded up, to two significant figures — amount: [3.17, 9871.23] becomes [0, 9900] and a 2.37% null rate 0.024 — so the file reads as a choice rather than a measurement. Rounding outward and up keeps the data valid against it. --round-bounds 1k --round-rates 0.01 picks the precision, and exact writes what was observed:

polspec schema infer orders.parquet -o orders.yaml --round-bounds 10k --round-rates 0.01

schema new — start from nothing

polspec schema new NAME -o OUTPUT.py

Writes a blank FrameSpec with the two imports it will need and a few commented ColSpec examples, for the case where there's no data yet to profile.

test — a round-trip test from a schema

polspec test SOURCE -o OUTPUT_test.py [options]

SOURCE is a .yaml/.yml spec (from schema infer, or written by FrameSpec.to_yaml) or a .py file defining one or more FrameSpec subclasses (from schema new, filled in). The generated file asserts the property this project's own test suite is built around:

def test_orders_roundtrip(polspec_frames):
    frame = polspec_frames(Orders, 500)
    assert_valid(frame, Orders)


def test_orders_cartesian_coverage(polspec_frames):
    frame = polspec_frames(Orders, 500, method="cartesian")
    assert_valid(frame, Orders)

The frames come from the polspec_frames fixture, seeded per test from --polspec-seed, and a failure says how to rerun it. --seed N pins every frame to one seed instead. The options — --rows, --seed, --no-cartesian, --class — are in the command-line reference. Changed in 1.3: the tests used to call generate() and validate() with seed 42.

It will not hand you a test that fails on the spot

generate() does not attempt everything validate() checks — see Known limitations. A spec using __checks__ or ColSpec.validators, or a pattern the generator cannot follow, would otherwise get a test that fails the moment it runs, since generation cannot satisfy them, and one declaring __rows__ a test of the wrong size. The generator checks for each and turns off the matching validate() switch, with a comment explaining why:

def test_invoices_roundtrip(polspec_frames):
    # __checks__ wraps arbitrary expressions that generation cannot be made
    # to satisfy
    frame = polspec_frames(Invoices, 500)
    assert_valid(frame, Invoices, checks=False)

A spec with a foreign key into another spec declared in the same file generates that spec first, and its parents in turn:

def test_lines_roundtrip(polspec_frames):
    parents = {}
    parents[Customers] = polspec_frames(Customers, 500)
    parents[Orders] = polspec_frames(Orders, 500, references=parents)
    frame = polspec_frames(Lines, 500, references=parents)
    assert_valid(frame, Lines, references=parents)

One whose parent is declared in another file — or left out by --class — is marked @pytest.mark.skip with a reason, rather than guessed at:

@pytest.mark.skip(
    reason=(
        "Child has foreign key(s) into Parent, which this file does not load. "
        "Generate the test from a file declaring it too, or pass "
        "its frame as references={Spec: frame} -- see docs/how-to/testing.md."
    )
)
def test_child_roundtrip():
    pass

Similarly, the cartesian test is only emitted when the spec actually has something for method="cartesian" to build coverage from — an Enum, Boolean, or bounded numeric column. A spec of only unbounded strings gets a comment instead of a test that would raise ValueError on the first run.

Regenerating

The generated file names the command that made it:

"""Generated by `polspec test orders.yaml`.

Regenerate with:

    polspec test orders.yaml -o test_orders.py

This file is only overwritten by running that command again -- edit freely.
"""

It is a plain file, not managed state — add assertions, rename the functions, delete the parts you don't want. Nothing re-reads it.

generate — data from a schema

polspec generate orders.yaml -n 1000 -o orders.parquet --seed 1
polspec generate specs.py --class Orders -n 500 -o orders.csv --references Customers=customers.parquet
polspec generate orders.yaml -n 50 -o edge_cases.ndjson --method cartesian
polspec generate orders.yaml -n 1000 -o paid.parquet --fixed status=PAID --fixed qty=1

Generates -n rows and writes one file; the extension picks the format (.parquet/.pq, .csv, .tsv, .ndjson/.jsonl, .json, .arrow/.ipc/.feather — the same set validate and drift read). --seed makes the file reproducible; --method cartesian guarantees coverage the way generate() does; --references NAME=PATH supplies parent data for a foreign key, as for validate. --fixed COLUMN=VALUE pins a column to one value in every row, as fixed= does: the value is read as the column's dtype, as validate reads text (3, true, 2026-01-02), and empty text is a null. --fixed added in 1.4.

The frame is built in memory and written once. For a file too large to hold, the streaming sink_* functions are a Python surface. A CSV or TSV cannot hold a Duration, List or Struct column; Polars refuses to write one, so use Parquet or Arrow IPC for a spec that has any.

--all — every spec in a directory

polspec generate --all specs/ -n 1000 -o data/ --seed 1
polspec generate --all specs/ -n 1000 -o data/ --format csv

With --all, SPEC is a directory (or any file Registry.discover accepts) and -o is a directory: every spec found is generated, parents first with their keys threaded into their children as Registry.generate_all does, and written as <name>.<format> (Parquet by default). No --references are needed for keys between the discovered specs; supply them for a parent outside the directory. A pin names its spec: --fixed Orders.status=PAID.

synthesize — fake data from real data

polspec synthesize customers.parquet -o fake_customers.parquet -n 1000000 --seed 1 --replace name email --spec fake_customers.yaml

Profiles the file — shapes, keys, category frequencies, null rates — and writes a fake one with the same schema, -n rows (as many as the source, by default), or --scale times as many: --scale 0.1 is a tenth. --replace names the columns whose values must not be carried over; --sample N profiles a random sample of N rows; --spec writes the profiled spec beside the data, to review or to regenerate from. See Fake data from real data for what carries over.

Given several files, it writes a fake set into a directory, one file per source, every key between them held by a fake parent row:

polspec synthesize customers.csv orders.csv lines.csv -o fake/ --scale 0.1 --seed 1
polspec synthesize customers.csv orders.csv -o fake/ --format parquet --spec fake/shop.yaml

Each file keeps its own format unless --format names one for all, and --spec writes the registry. --foreign-key and --no-foreign-keys work as for schema infer. With --parse-text, a date read in a format, such as 30 Sep 2031, is written back in it to a CSV or NDJSON file. Several files added in 1.5.

validate — check data against a schema

polspec validate orders.yaml orders.parquet
polspec validate specs.py orders.parquet --class Orders --references Customers=customers.parquet
polspec validate orders.yaml orders.csv --json > report.json

Reads a data file, in any of the formats generate writes, runs inspect() against the spec, and prints the report: the same text validate() would raise, or the full structured report with --json. The exit status is 0 when the data passes and 1 when it does not, so a spec can gate a pipeline step in CI with no Python at all.

The JSON is ValidationReport.to_dict(), led by a schema_version:

{
  "schema_version": 1,
  "spec": "Orders",
  "passed": false,
  "findings": [
    {"code": "bounds", "key": "total__bounds", "message": "Column 'total': ...",
     "columns": ["total"], "count": 1, "samples": [-1.0],
     "details": {"bounds": [0, 10], "min_found": -1.0, "max_found": 1.0}}
  ]
}

The keys, and the types of their values, are what Stability promises; a key may be added, and schema_version rises if one is ever renamed, removed or changes type.

--output PATH writes the rows that passed, typed as the spec declares — they are validated again with cast=True, so the file is one the spec accepts — and --failing PATH writes the rows that did not, each once, in the file's order, with a __polspec_finding column naming every claim it broke, comma-separated. The two files split the input exactly: their rows add up to the file's. (In Python, report.failing_rows(once=True) is the same view; plain failing_rows() lists a row once per claim, for grouping by claim.) Either or both; the extension picks the format, as for generate, and the exit status still says whether the whole file passed, so a pipeline can quarantine the bad rows and fail the step. A structural finding (a missing column, a wrong dtype) judges the whole file, so neither is written:

polspec validate orders.py orders.csv --output clean.parquet --failing rejected.csv

--references NAME=PATH supplies parent data for a foreign key to another spec, by that spec's name; repeat it for several. --allow-extra and --allow-missing relax the structural checks; --strict-dtypes tightens the dtype check. --fill-defaults fills each defaulted column's nulls, and adds it when absent, before checking, as fill_defaults=True does. --skip CHECK turns off one kind of check, as CHECK=False does in Python — --skip bounds --skip checks — and takes any of rules, derived, validators, unique, checks, foreign_keys, hierarchy, order, rows, pattern and bounds.

--tolerate [KEY=]SHARE accepts findings up to a share of the rows, as tolerate= does: a bare share for every finding, or one by finding code, column, or column:code, repeated for several. A share is 0.001 or 0.1%. A tolerated finding is printed, marked, and the exit status is 0 if nothing else failed. Added in 1.7.

polspec validate orders.py orders.csv --tolerate nullability=1% --tolerate total:bounds=0.1%

A CSV, TSV or JSON file has no date type, so a date arrives as text — and JSON holds a decimal or a duration as text too. The CLI reads each column the spec declares as a Date, Datetime, Time, Decimal or Duration as that type when every value in it parses exactly, as read() does; a column holding a value that does not stays text, and is reported as a dtype finding rather than turned into a null. A String column of date-shaped text is left alone. drift reads the same way, and schema infer, with no spec to go by, recognises a CSV's dates itself.

--all — every spec against the file named after it

polspec validate --all specs/ data/
polspec validate --all specs/ data/ --json > reports.json

With --all, SPEC is a directory of specs and DATA a directory of data files named after them (Orders.parquet for Orders, in any format the CLI reads). Every spec with a file is checked, each seeing the others' files as its parents, as Registry.inspect_all does; specs with no file are listed and skipped. The exit status is 1 when any report fails; --json prints one document with every report keyed by spec name under reports — {"schema_version": 1, "reports": {"Orders": {...}}} — since a spec may be called anything.

diff and drift — what moved

polspec diff orders_v1.yaml orders_v2.yaml                 # two schemas
polspec diff specs_v1.py specs_v2.py --class Orders --rename id=order_id
polspec drift orders.yaml last_night.parquet               # a schema and data
polspec drift orders.yaml last_night.parquet --markdown > drift.md

diff runs diff() between two spec files; drift runs drift() between a spec and a data file. Both print the report as text, --json, or --markdown (the shape of a pull-request comment, breaking findings first).

The exit status is decided by --fail-on: breaking (the default) exits 1 when any finding would break validation; any exits 1 on any finding at all, a widened bound included; none always exits 0, for posting a report without gating on it. So a schema change in a pull request, or a nightly load, can be gated with no Python:

polspec diff main/orders.yaml pr/orders.yaml --markdown --fail-on breaking

--strict-dtypes makes any dtype change breaking, as it does for validate. drift also takes a flag for every other DriftOptions field — --significance, --null-rate-tolerance, --frequency-tolerance, --distribution-tolerance, --no-unseen, --max-samples — each defaulting to the option's own default, and --sample N, a seeded random sample of N rows.

drift --all — every spec against the file named after it

polspec drift --all specs/ nightly/
polspec drift --all specs/ nightly/ --json > drift.json

As for validate --all: SPEC is a directory of specs and DATA a directory of files named after them. Every spec with a file is measured, specs without one are listed and skipped, and --fail-on decides the exit status across all of them; --json prints the reports under reports, as validate --all does. diff compares two declarations rather than a declaration and data, so it has no --all.

Exit codes and errors

Every subcommand returns 0 on success and 1 on a reported error, printed as error: ... on stderr rather than a traceback — a missing file, an unreadable format, an invalid class name. validate, diff and drift also return 1 when the report itself fails, so a 1 means "look at the output", whichever kind of problem it was. A command line that is itself wrong — an unknown verb, flag or choice — returns 2, with a usage message. Stability says which of these are promised.

$ polspec schema infer nope.csv -o out.yaml
error: no such file: nope.csv