Changelog¶
All notable changes to polspec are recorded here. The format follows Keep a Changelog. From 1.0, the public API, the spec-file format, seeded output, the command line and the finding and drift codes change only as the Stability page promises; before 1.0, a minor version could break any of them.
Unreleased¶
1.7.0 - 2026-11-17¶
Generated data that keeps its checks, and validation that tolerates a
little. generate(checks=True) keeps only the rows a spec's checks pass,
so fake data no longer fails its own spec; @check declares a check as a
function. tolerate= lets a finding cover a share of the rows and still
pass, from Python or polspec validate --tolerate. polspec.formats()
lists every string format, synthesize can return the spec it generated
from, and a TableSpec asked for a verb names the function to call. Text
of whole numbers followed by decimals now reads as Float64. The docs gain
Performance and How the pieces fit.
Everything is additive under the 1.0 promises. No seeded output moves:
checks=True is off by default, and a frame whose rows all pass is the
frame 1.6 generated. Profiles change where a column of numbers written as
text is now read as Float64, which Stability leaves open. The JSON
report gains a tolerated key on each finding. Nothing is deprecated. The
spec-file format stays at version 4, and 1.6 reads 1.7's files.
Added¶
checks=Truekeeps a spec's checks in the data it generates.__checks__and column validators were validation-only, so a generated frame could fail its own spec. Withchecks=True,generate,generate_batches,scan, everysink_*and a registry'sgenerate_all,generate_relatedandscan_allkeep the first rows of the seed's frame that pass every check, reading on for more when some fail. Auniquecolumn stays unique, across batches too, and a frame whose rows all pass is the framechecks=Falsegives.method="cartesian", a hierarchy, a key into the spec's own rows and a pin of allnvalues are refused, and a check that rejects nearly every row is aGenerationErrornaming it. Off by default, so no seeded output moves.@checkdeclares a check as a function on aFrameSpecclass, named after the function: it returns acol()predicate or a Polars expression, and joins__checks__.tolerate=accepts a little bad data.validate,inspectand their facades take a share of the rows that a finding may cover and still pass: one for every finding, or by finding code, column, or(column, code). A tolerated finding stays in the report, markedFinding.tolerated;ValidationReport.failuresand.toleratedsplit them, andpassedcounts failures only.polspec validate --tolerate [KEY=]SHAREtakes the same, with0.1%for a share. The JSON report gains atoleratedkey on each finding.polspec.formats()lists every string format as a frame: its name, a summary, whether it is a finite list, an example value, and a finite format's values.synthesize_all(return_registry=True)andsynthesize(return_spec=True)return what the fake data was generated from. A larger fake set widens its key ranges, soprofile_all's registry refused it on bounds; the one returned accepts it.
Changed¶
- A
TableSpecasked for a verb names the function.polspec.profile(df) .validate(df)raised a bareAttributeError; a spec is a value and its verbs are functions over it, so it now says to callpolspec.validate(spec, ...), or wrap it withFrameSpec.from_spec. - Docs: Performance, a new how-to page with what each choice costs — generating, streaming, keeping checks, validating a lazy frame, profiling a sample — measured. Generate data and Validate data show what their examples print, and Multiple specs declares the key its example relied on.
- Docs: How the pieces fit, a new explanation page with one diagram:
the declarations, verbs as functions, what generation keeps, why
schema inferandprofile()have different defaults, and seeds. schema inferandsynthesizedescribe their shared flags alike.--max-unique-enum,--sampleand--replaceread the same on both commands'--help, andsynthesizelists them inschema infer's order. What each flag does is unchanged.
Fixed¶
- Text of whole numbers then decimals reads as
Float64. Text is tried on its first 1,000 values first, and a column whose first values were whole and whose decimals came later stayed text, saying its values were too large for an Int64 and quoting the decimals. Every later type is now tried on the whole column, and that reason is given only for whole numbers an Int64 cannot hold.
1.6.1 - 2026-11-16¶
Every choice and its share, in the data dictionary. to_markdown can
now document a long list of choices whole, in an appendix or in its cell,
for a table of many Enum columns whose weights the summary cut short.
Additive under the 1.0 promises: the default writes exactly what 1.6.0 wrote, no seeded output moves, and the spec-file format is unchanged.
Added¶
to_markdown(choices=)shows every choice and its share. A column of more than four choices still shows its first three and its count;choices="appendix"adds a Choices section listing every value and share of each such column, in one table, andchoices="full"lists them all in the cell. The default,"summary", writes what 1.6.0 wrote.FrameSpec.to_markdowntakes it too.
1.6.0 - 2026-11-16¶
Faster where it was slow, and dates as they are written. Small batches,
scans and sinks stop paying for a whole engine chunk each: a million rows in
batches of 1,000 take 12 ms, not 897 ms. List, Map and Decimal
columns are built in the engine, a unique parent is read as it stands,
validation skips a check an Enum cannot fail, and profiling runs its
columns at once. profile(parse_text=True) reads dates written as
30 Sep 2031 or 30/09/2031, keeps the format as text_format, and
writes fake files in it; a column that nearly parses says why it stays text.
to_markdown shows an Enum's choices and shares, and Registry.replace
swaps one spec.
Everything is additive under the 1.0 promises. No seeded output moves:
batched and generated frames hold exactly the values they held in 1.5,
checked case by case across chunk boundaries. Profiles change where a
column of dates written as text is now read, or now says why it stays text,
which Stability leaves open, as it does a repr and a rendered document.
Nothing is deprecated. The spec-file format stays at version 4: a
text_format key is written only when used, and 1.5 refuses it by name.
Added¶
- Dates, datetimes and times written as text are read in the form they
are written.
profile(parse_text=True)reads30 Sep 2031,30 September 2031,30-Sep-2031,Sep 30, 2031,30/09/2031,09/30/2031,2031/09/30and30.09.2031, unpadded too, any of them with a time, and a time alone (14:05,2:05 PM), where only ISO was read before. A format fits when every value parses with it and writes back as itself, so30 Feb 2031fits nothing. When two formats fit and read the values differently, as day-first and month-first do for01/02/2031, the column stays text and says which: polspec does not guess. ColSpec(text_format=...): how aDate,DatetimeorTimecolumn is written as text, as a strftime string ("%d %b %Y"). Profiling writes it for a column read that way;validate(parse_text=True)andread()read the column with it, and a value written otherwise is adtypefinding that names it; a CSV or NDJSON file polspec writes holds the column in it, from thesink_*verbs and frompolspec generate,synthesizeandvalidate --output. In memory, and in Parquet and IPC, a date stays a date. A directive the column cannot be written with is refused where it is declared. A new optional key on spec-file version 4, written only when set; 1.5 refuses a file that uses it by name.- A column that nearly parses says why it stays text. With
parse_text=True, a column of nearly numbers or nearly dates is described asStays text: ...: the values that do not parse ('1,500,000.00','N/A'), a value written with a leading zero, a whole number past anInt64, or the two date formats that both fit.schema inferprints the same line. Withoutparse_text, a column that would parse says in which format, and takes no shape pattern. Numbers with thousands separators or decimal commas are left to the caller to clean, and the docs show how. Registry.replace(spec): a new registry with one spec swapped for the one of its name, in the same place and with the same categories. Editing a profiled registry no longer means rebuilding it by hand.
Changed¶
to_markdownwrites anEnumas itsreprdoes:Enumin the Type column and its categories in Domain / Choices, each beside its share when weighted (SEN60%,SUB40%); weighted choices on any column the same. A nullable column shows its null rate,Yes (5%). AnEnuminside aListorStructkeeps the dtype's own form.- A
ForeignKey'sreprnames its target rather than printing it. It printed the whole parent spec, over 600 characters for one key of the shop tutorial's registry. TableSpec.selectanddroptake a list, as Polars' do, as well as names one by one; a list raisedTypeError: unhashable typebefore.-
from_yamlgiven YAML text says so.to_yaml()with no path returns text, and passing it back raised a rawOSError;from_yaml,Registry.from_yamlandCatSpec.from_yamlnow raise aSerializationErrornaming the call that reads text. -
Batches, scans and sinks are faster where they were slow. The engine fills a column in 65,536-row chunks, so a batch that started inside one filled it from its start and dropped the head, and a batch of one chunk filled it on one thread. A stream now draws its columns many chunks at a time, up to about 64 MiB, in parallel, and slices each batch from them. A million rows of four columns in batches of 10,000 take 7 ms, not 118 ms; in batches of 1,000, 12 ms, not 897 ms.
sink_parquetis a third faster, and batches with a foreign key five times. Seeded output is unchanged: a batch holds the rows it held. AList,ArrayorMapcolumn is still drawn batch by batch. List,Array,MapandDecimalcolumns generate several times faster. A list's cells are built in the engine from each cell's length and the flat elements, where a group-by over every element built them before: a million rows ofList(Int64)take 5 ms, not 105 ms, and ofList(String)20 ms, not 140 ms. ADecimalis built from its physical integers, not divided into: 5 ms a million, not 20 ms. A unique column's permutation runs in 64-bit arithmetic where its domain fits, a tenth faster. Seeded output is unchanged.- A foreign key reads a unique parent without deduplicating it. A parent's key is usually unique, so its keys are the column as it stands; one count of distinct values says so. A 1,000-row child of a 1M-row parent takes 3 ms, not 33 ms, and a self-referencing key a quarter of the time. Seeded output is unchanged.
- Validation leaves out a check that cannot fail. Values that arrive in
the very
Enuma column declares, with nochoicesnarrowing it, are all its categories; testing each against them took most of the time of validating a frame with anEnumin it. A four-column frame of a million rows validates in 1 ms, not 20 ms. - Profiling is faster. A frame of 50,000 rows or more has its columns
profiled at once, on a pool of threads; a column every value of which has
a format (an
email, auuid4) is checked in one pass, not counted value by value. An 11-column frame of a million rows profiles in 204 ms, not 509 ms; a UUID column in 96 ms, not 182 ms. Every profile is the same as before, warnings and their order included.
Documentation¶
- The README and Command line catch up with 1.5: related tables
profiled and faked together, several files on the command line,
--foreign-key,--no-foreign-keys,--scale,--formatand--no-patterns. The README's timings are re-measured. - A weighted column is taught as
choices={...}, the spelling itsreprprints andprofilereports;weights=is shown as the same spec given apart, and for aBoolean. - Dates written as text, and why a column stays text, have their own sections in Infer a spec and Validate data, a recipe, and rows in ColSpec fields and Spec files. Generate data says small batches cost little now.
- Comparison is re-measured on 1.6: generation, last measured on 0.21.0, and validation, where polspec is 1.4 to 2.1 times as fast as dataframely and pandera.
1.5.0 - 2026-11-15¶
Real data in. profile_all profiles a set of related tables into a
Registry, with the foreign keys between them found from their values,
and synthesize_all makes a fake copy of the set, a tenth of the size or
three times it, with every key held by a fake parent. The command line
takes several files for both. profile writes a code's shape as its
pattern, so fake order references and SKUs look like the real ones, and
a weighted Enum prints as ColSpec(Enum, choices={'A': 0.5, ...}).
Validation is benchmarked beside dataframely and pandera, and a third
tutorial starts from data you already have.
Everything is additive under the 1.0 promises. Seeded generation is
unchanged. Profiles change where a code now gets its pattern, which
Stability leaves open, as it does a repr. Nothing is deprecated. The
spec-file format stays at version 4.
Added¶
polspec.profile_all(frames)profiles a set of related tables into aRegistry, with the foreign keys between them found from their values. A column becomes a key into a unique integer, text or date column -- of another table or its own -- when every value it holds is one of the parent's, in all the rows, and either the names link the two (customer_idorcustomerintoCustomers.id,parent_idinto its own table'sid) or it is text with at least 20 distinct values. A child with orphans, or with two parents to choose between, gets no key; the second warns. The key's column takes its parent's claims and keeps its own null rate, so the registry validates its sources and generates frames that pass.keys=declares keys instead, checked against the data, andkeys=Falseinfers none.polspec.synthesize_all(frames, n=, scale=)isgenerate_all()overprofile_all(): a fake frame for each table, in its source's dtypes, every key between them held by a fake parent.scale=0.1takes a tenth of each table, keeping the ratios between them;nis a count for every table or a mapping. A parent's key range widens for a larger set, and its children with it.polspec schema inferandpolspec synthesizetake several files.schema infer a.csv b.csv -o shop.yamlwrites a registry (.yaml, or.pywith aRegistryafter the classes), andsynthesize a.csv b.csv -o fake/writes one fake file per source into a directory, in its own format or--format's. Each spec is named after its file.--foreign-key Orders.buyer=Customers.iddeclares a key and--no-foreign-keysfinds none.synthesize --scale Xsizes the output as a multiple of the source's, for one file too.profilewrites a code's shape as itspattern. A text column of at least 20 distinct values that no format fits and that is not anEnum-- an order reference, a SKU, a postcode -- gets the pattern its values share when they have at most three shapes: each run of digits, capitals or small letters becomes its class with the lengths seen, and a part every value shares stays as written (^ORD-[0-9]{6}$). Every source value matches, and the fake codes have the shape. Not for areplaced column, for prose, or for numbers and dates written as text. A column of distinct values gets one only when it holds ten times as many.patterns=False, or--no-patterns, turns it off.
Changed¶
- A column's
reprwrites an Enum's categories, and weighted choices, aschoices:ColSpec(Enum, choices={'A': 0.5, 'B': 0.3, 'C': 0.2})rather thanColSpec(Enum(categories=['A', 'B', 'C']), weights=(0.5, 0.3, 0.2)), each category beside its share. It is a spellingColSpectakes. An Enum inside aListorStructkeeps the dtype's own. Spec files are unchanged.
Fixed¶
--allfinds a data file by the spec name its stem gives.validate --all,drift --alland the other--allverbs looked for<Spec>.<suffix>and left the match to the file system, soorders.csvwas found forOrderson Windows and macOS but not on Linux. A file matches when its stem is the spec's name or gives it, as several-sourceschema infernames a spec after its file (order_lines.csvforOrderLines), on every platform. Two files for one spec are refused, as before.
Documentation¶
- Validation is benchmarked.
benchmarks/validation.pyvalidates the same million orders and their customers with polspec, dataframely and pandera, each written as its documentation writes it, eager and lazy. Comparison has the numbers: polspec is 1.3 to 2 times as fast, the gap between one aggregation and several passes. dataframely and pandera join thebenchdependency group only. - A third tutorial, A spec for data you already have: three small
CSV files of a shop, in
docs/data/shop/, profiled into a registry with its keys, edited, validated, watched for drift and synthesized at a tenth for tests. Its commands run in the docs tests, as its Python does: a shell block marked<!-- docs: run -->runs eachpolspecline through the CLI. - Fake data from real data gains Related tables:
synthesize_all, its sizes, how keys are found and how to declare them. Infer a spec from data gains Several tables. Known limitations says what survives between tables, and Recipes gains a fake copy of related tables and a key the profile missed.
1.4.0 - 2026-11-15¶
Generate what you validate. A pattern column is generated from its
pattern, through the automaton Polars' own regex engine builds, so a spec
with order references, postcodes or SKUs passes its own round trip with
every check on; unique=True works on one. A column can be derived from
others (derive=col("qty") * col("price")), computed by generation and
held to its expression by validation. fixed= reaches every way of
generating and the command line. schema_only=True checks a lazy frame's
shape without running it, each finding carries its share of the rows, and
uuid7, ulid, url and iso_country3 join the formats.
Everything is additive under the 1.0 promises, but for one change to
seeded output the Stability page allows: a pattern column whose plain text
broke its own pattern now draws from the pattern. A pattern every plain
string passes, and one the generator cannot follow, keep their output
exactly. Nothing is deprecated. The spec-file format stays at version 4: a
derive: key is written only when used, and 1.3 refuses it by name.
Added¶
- Strings from a
pattern. AStringcolumn with apatternis generated from it: each value is a full match, drawn by a walk through the DFA thatregex-automata-- the engine under theregexcrate Polars validates with -- builds for the pattern, so every value passes validation. ASCII wherever the pattern allows it; withinstring_lengthwhen declared; an unbounded repeat runs to 16 characters past the shortest match. A list of strings with a pattern generates its elements the same way.unique=Truedraws each of the pattern's values once: they are counted at each length, so an index decodes to one value, and the column is a keyed permutation of them like any other unique column. A million values take about 8 ms. - Derived columns:
ColSpec(pl.Float64, derive=col("qty") * col("price")). A column computed from others with thecol()expressions rules and checks use. Generation computes it after every column it reads -- rules, foreign keys,fixed=pins and other derived columns -- and casts it to the column's dtype; validation holds the column to it, as a newderivedfinding: equal, or for a float within a relative 1e-9, and null exactly where the expression is.derived=False(andpolspec validate --skip derived) switches the check off. The expression decides the values, so a derived column declares no bounds, choices, rules, distribution, format, pattern, length,uniqueordefault, is nullable when an input is, and is written by nothing else; a cycle of derived columns is refused. Spec files write it as aderive:key, which 1.3 refuses by name;diffreports one added or changed as breaking and one removed as compatible;renamerewrites it, andjoindrops it, as it drops rules. The data dictionary and the ER diagram show the expression. fixed=everywhere:generate_batches,scan, everysink_*, theirFrameSpecforms, and a registry'sgenerate_all,generate_relatedandscan_all, which take pins spec by spec ({Customers: {"tier": "pro"}}). The rules aregenerate()'s. A batch is a window onto the pinned frame: a scalar fills every batch, a sequence of allnvalues gives each batch its stretch, and the batches are row for row whatgenerate()makes. A scan checks its pins when it is built.polspec generate --fixed COLUMN=VALUEpins from the shell, each value read as its column's dtype (--fixed SPEC.COLUMN=VALUEwith--all).inspect(frame, schema_only=True)-- andvalidateandassert_valid-- checks what the schema says (missing, extra and optional columns, each column's dtype) and reads no rows: aLazyFrame's query is planned, never run. It does not combine withparse_text=True.Finding.share: a counted finding'scountas a fraction of the frame's rows,Nonewherecountis. The text report adds(2.5% of rows)to each such finding, the Markdown report shows it beside the count, and the JSON report has asharekey (schema_versionstays 1: a key added, none changed). The rows are counted in the aggregation that already runs.- Formats
uuid7,ulid,urlandiso_country3. A version-7 UUID and a ULID by their syntax (their timestamps are not read); anhttporhttpsURL with a hostname, optional port, path, query and fragment; and the ISO 3166-1 alpha-3 codes of the same 249 countries asiso_country. Each is generated, validated and recognised byprofile(). A profile of data holding them names the format where it named none before; three codes (BTN,MKD,SLE) are both countries and currencies, too few for either format to be taken for the other. - A warning for a pattern the generator cannot follow -- a Unicode word
boundary, an automaton over its size limit, no match within the declared
string_length-- naming the column and the reason. The column is filled with plain random text, as before. polspec testchecks patterns. A generated test no longer passespattern=False, except for a pattern the generator cannot follow, with a comment naming it and why.
Changed¶
- Seeded output, for pattern columns only. A pattern column used to be
filled with random alphanumeric text that its own pattern refused, and
a minor release may change output that broke its own spec: those
columns now draw from their pattern. A pattern that every such string
already passes (
^[A-Za-z0-9]+$) is decided on its automaton and keeps exactly the output it had. So does a pattern the generator cannot follow: a Unicode word boundary, an automaton over its size limit, no match within the declared lengths. Stability lists the change with the pre-1.0 ones, and the seeded-output catalogue now holds a pattern column, a unique one, one that keeps its plain text, a derived column and the four new formats, so their output is promised from here on. - Docs. Nested types (
List,Array,Struct,Map) have a page of their own, Nested columns, out of Declaring columns; patterns moved to String formats. Recipes gains an order reference or postcode, a computed total, and a lazy pipeline's shape checked without running it. Pages that called patterns validation-only no longer do.
1.3.0 - 2026-11-15¶
Tests that say what broke. assert_valid checks a frame at the end of a
test and, when it fails, names each finding, shows the failing rows and the
seed to rerun with. check_pipeline tests a whole pipeline in one call. A
pytest plugin gives each test frames from a seed of its own, so adding a
test changes no other test's data, and strategy hands a spec to
Hypothesis. polspec test writes its tests with them. The docs gain
Recipes and a page for readers coming from dataframely, pandera or
Pointblank. Everything is additive under the 1.0 promises, and no seeded
output changes; nothing is deprecated.
The spec-file format stays at version 4, and 1.3 writes nothing 1.2 cannot
read. The plugin loads whenever pytest runs with polspec installed;
-p no:polspec turns it off.
Added¶
assert_valid(df, spec, **options):validate()as an assertion, for the end of a test. It returns the validated frame, so it chains, and on a failure raisesAssertionError-- a failed test in any runner, not an error -- whose message names the spec, each finding by its key, and up to ten failing rows as a table.check_pipeline(fn, inputs, output, *, n=, seed=): a pipeline test in one call. Each input is a spec to generate or a frame to pass; the specs are generated together, so a foreign key between inputs lines up, from a seed that defaults to 0. It callsfnwith them, checks the result againstoutputwithassert_valid, and returns the result and the inputs.- A pytest plugin with a
polspec_framesfixture. It loads whenever pytest runs with polspec installed (-p no:polspecturns it off).polspec_frames(Orders, 500)isgenerate()with a seed of the test's own, derived from the session seed and the test's id, so adding, removing or reordering tests changes no other test's frames.--polspec-seed=Nsets the session seed, 0 by default, and--polspec-seed=randomdraws one; the report header names it, and a failing test that polspec took part in says how to rerun it.assert_validnames the seed in its message, andcheck_pipelinegenerates from the test's seed under pytest. strategy(spec, *, min_rows=, max_rows=): frames a spec accepts, as a Hypothesis strategy, for property tests. Each isgenerate()on a drawn row count and seed, so it meets the spec by construction and follows its distributions; a failure shrinks to the fewest rows that fail. Hypothesis is not a dependency: it is imported when a strategy is asked for.polspec schema infer SOURCE -o -prints the YAML spec instead of writing a file; the summary and any notes go to stderr, so redirecting stdout captures only the spec.- Docs: Recipes, short answers to "how do I…" (a primary key, ids
1..n, sorted timestamps, an empty frame, quarantining bad rows, a drift gate in CI, a nullable foreign key, a CSV read as text), each linking to the page that covers it. - Docs: Coming from dataframely, pandera or Pointblank, a feature table, a concept map and one schema translated from each, checked against their documentation on 2026-11-15. Comparison names the three, and says how polspec and Hypothesis now combine.
Changed¶
- CI waits less. A pull request runs the oldest and newest Python on
each OS, with the floor and newest-Polars jobs: eight test jobs, where
it ran sixteen.
mainand a weekly run keep every Python on every OS, and coverage is measured on Python 3.12, a leg every pull request runs. A release no longer runs the matrix again: it waits for the Test run onmainfor the tagged commit and builds only if it passed. A test job compiles the Rust extension only when its sources changed: the built module is cached on them, one per OS, saving 5 to 14 minutes a job. polspec testwrites its tests with the new helpers. Each test draws from thepolspec_framesfixture and checks withassert_valid, so a failure shows the rows and the seed to rerun it with. A spec whose foreign keys reach other specs in the same file generates those first, and its test runs where it used to be skipped; one into a spec the file does not declare is still skipped, saying which.--seednow pins every frame to one seed, and is unset by default, where it was 42.- CI measures coverage with
coverage run -m pytest, notpytest --cov. pytest loads polspec's own plugin at start-up, which imports the package before pytest-cov begins, so every module's top-level lines counted as missed and coverage read 86% instead of 94%.
1.2.0 - 2026-11-14¶
Quicker to write, quicker to run. Data read as text profiles and
validates as the types it holds. A spec is shorter to write: a bare dtype is
a column, an Enum's weights sit beside its categories, dates in bounds can
be text, and a class's docstring is its description. Declarations that never
worked now fail where they are written, saying why. Specs and reports
pickle, a spec reads as an Arrow schema, and every public name's reference
shows a runnable example. Batched foreign keys, profiling and import
polspec are 1.7× to 10× faster. Everything is additive under the 1.0
promises, and no seeded output changes; nothing is deprecated.
The spec-file format stays at version 4, and polspec writes nothing 1.1
cannot read. A spec whose class has a docstring now writes it as a
description: key, which 1.0 refuses by name, as it does any 1.1 key.
Added¶
profile(parse_text=True)reads a text column as the type its values hold, for a frame read with its types switched off (pl.read_csv(..., infer_schema=False)):Boolean,Int64,Float64,DateorDatetime, the first every value parses as exactly. Empty text is a null; a value written with a leading zero, or an integer pastInt64, keeps its column text.synthesize,FrameSpec.from_dataframeandProfileOptionstake it, andpolspec schema inferandpolspec synthesizetake--parse-text. Off by default: a text column that would parse stays text, and its description says what it would parse as, which the two commands also print.validate(parse_text=True)reads each column that arrived as text, where the spec declares a type text can hold, as that type before checking anything, so a text frame validates against its ownprofile(parse_text=True)and comes back typed.inspect,ValidationOptions,validate_allandinspect_alltake it. A column with a value that does not parse stays text, and itsdtypefinding now counts those values and gives some as samples;polspec validatealways reports them so.- A bare dtype is a column.
opened = pl.Datein a class body, a dtype in__columns__or inTableSpec(columns=...), andopened: Datein a spec file each meanColSpecof that dtype. Spec files are still written in the long form, which 1.0 and 1.1 read. -
Dates and times in bounds may be ISO text:
bounds=("2020-01-01", None)on aDate,DatetimeorTimecolumn is read as the date it spells, where it is declared. -
A bare
pl.Enumtakes its categories fromchoices.ColSpec(pl.Enum, choices={"A": 0.5, "B": 0.3, "C": 0.2})declarespl.Enum(["A", "B", "C"])with those weights, and is the same spec as that spelling -- equal, the same seeded frame, the same file.to_pythonwrites a weightedEnumthis way, each category beside its share; spec files keep theEnum: [...]andweights:form 1.0 and 1.1 read, and also read a bareEnumwithchoices. A barepl.Enumwith no choices is refused, saying to list its categories. TableSpec.from_schema(schema, name=): a spec of apl.Schema, a mapping of column name to dtype, or an Arrow schema, each column declaring only its dtype.ColSpec.replace(**changes), for a column with a field changed.Orders.names.total: each column's name by the attribute it is declared as, honouringcol_name, so code can name a column without spelling it as text; onTableSpectoo.- Reports show as Markdown in a notebook, through
_repr_markdown_onValidationReportandDriftReport. - Specs read as Arrow schemas. A
TableSpecand aFrameSpecclass implement__arrow_c_schema__, sopl.Schema(Orders),pyarrow.schema(Orders)and any other reader of the Arrow PyCapsule interface take a spec where they take a schema.
Changed¶
- Faster where it was slow, measured with
benchmarks/bench.pyagainst 1.1.0 on the same machine, with every other case unchanged: - Batches, scans and sinks with a foreign key into a large parent: a million rows in 10k batches from a 1M-row parent, 3.72 s to 0.36 s. Each parent's keys are read once per call, not once per batch. Seeded output is unchanged.
profile(): 235 ms to 93 ms on a million rows, 411 ms to 243 ms on the same rows as text. A column is no longer hashed again for each thing asked of it. Every profile is the same as before.import polspec: 200 ms to under 60 ms on top of Polars. Generating, validating, drift, profiling, spec files and rendering load on first use.- A class's docstring is its description. A
FrameSpecwith a docstring and no__description__now has that docstring as itsdescription, whichto_markdownshows andto_yamlwrites as adescription:key -- one 1.1 reads, and 1.0 refuses by name.__description__wins when a class has both. Not inherited. - A spec prints as it was declared. A
ColSpecshows its dtype and the fields that differ from their defaults,ColSpec(Int64, bounds=(0, 10)), and aTableSpecone column to a line, where both used to print every field. - Choices that are an
Enum's categories, in its order, are dropped at declaration, sopl.Enum(cats)withchoices=the same is the specweights=alone makes. They generated the same frame already. read()types numbers and booleans too. A column the spec declares as aBoolean, integer or float that arrived as text is parsed, as dates, decimals and durations already were, soread(spec, path, infer_schema=False)hands back a typed frame. A column of its own is stripped first and empty text read as a null. A column with a value that does not parse stays text, as before.polspec validatereads files the same way.
Fixed¶
Four declarations that never worked now fail where they are written, with a
message saying why. A class or spec that hits one did not declare what it
said, so each now raises SpecError at import where before it failed later,
or silently declared less:
- A dtype that is not a Polars one.
ColSpec(int)orColSpec("Int64")failed on first use with an unrelatedAttributeError. It is refused with the Polars spelling:use pl.Int64. - Text in the bounds of a date, datetime or time column was kept as text,
and validating against it raised on Polars 2, which no longer casts text
to a date. It is read as the value it spells, and text that is not ISO,
or text on any other column but a
Decimal, is refused. Adatetimeon aDatecolumn is refused as one, not as outside the range aDateholds. - Class-body slips. A trailing comma after
ColSpec(...), aCheck,ForeignKey,HierarchyorColRuleunder a name polspec does not read, and a misspelt declaration dunder (__unique_togther__,__check__) each built a spec without what they declared. Each is refused, a misspelt dunder with the one meant. - A range that is not a pair.
bounds=(5,)raisedValueError: not enough values to unpack;bounds,string_lengthandlist_lengthnow say they must be a(min, max)pair.
An unknown dtype name in a spec file names the closest known one, or lists them.
-
Specs and reports could not be pickled. A
TableSpec, a struct column, a format'sextra_values, aRegistry, aValidationReportand aDriftReporteach raisedcannot pickle 'mappingproxy' object(or failed on a closure), so none could go to another process or be deep-copied. Each now pickles and deep-copies, and a report still names its rows afterwards. Their read-only mappings print as dicts. -
Validating a number too large for a
Decimalwith choices crashed. Asking whether a float or integer standing in for aDecimal(p, s)was one of itschoices, or a rule's, cast it to theDecimal; Polars raises on a cast past the precision, soinspect()andvalidate()raisedComputeErrorwhere they should have reported the value. It is now thedtypefinding it is without choices. profile()could refuse its own hostname column. A dot is what makes a column hostnames to the profiler, so a dotless value such aslocalhostor0was kept as anextra_valuesstand-in. But the hostname format accepts those, andColSpecrefuses a stand-in its format already accepts, soprofile(),synthesize()andpolspec schema inferraisedSpecErroron the column. Stand-ins are now only values the format refuses.profile()took decimals held as text for hostnames.0.45passes the hostname check, as host0in domain45, so a numeric column read as text (a CSV read withinfer_schema=False) was profiled withformat: hostname. A value ending in an all-digit label is no longer evidence of a hostname, since no real one ends that way; validating a declaredformat="hostname"is unchanged.
1.1.0 - 2026-11-08¶
A spec says more. A table or column can say what it means, how many rows a table holds, what fills a missing value and which columns may be absent; a generated frame can be pinned to the values a test needs; and a join of two specs is a spec. Everything is additive: code that runs on 1.0 runs unchanged, no seeded output changes, and nothing is deprecated.
The spec-file format stays at version 4. A file that uses none of the new
keys reads in 1.0 as before; one that uses description, rows, default
or optional is refused by 1.0, which names the key it does not know.
Added¶
description=onColSpecandTableSpec,__description__on aFrameSpec, andTableSpec.with_description(): what a table or column means.to_markdownshows it, spec files andto_pythonkeep it, anddiffreports a change ascompatible. Nothing else reads it.rows=onTableSpec,__rows__on aFrameSpec, andTableSpec.with_rows(): how many rows a table may hold, such as(1, None)for a table that must not be empty. Outside it, validation reports the new structural findingrow_count;rows=Falseand--skip rowsturn it off.generate()still makes the rows it is asked for.polspec testskips the check in the tests it writes.default=andoptional=onColSpec, andfill_defaults=True(--fill-defaults) on validation. A default is a value its column accepts, checked where it is declared;fill_defaultsfills a defaulted column's nulls, and adds it when absent, before any claim is checked. An optional column may be absent without amissing_columnsfinding.missing_cols="add"adds a column holding its default.diffreports a column made optional, or an optional column added, ascompatible; made required,breaking.generate()reads neither.generate(fixed=)andFrameSpec.generate(fixed=): pin columns to given values, a scalar or exactlynof them, and generate the rest. A rule keyed on a pinned column follows it, and no other column's values change. A pin its column would refuse, or one on a column a pass writes, is aGenerationErrornaming the column. Not yet taken bygenerate_batches,scan, the sinks or the command line.TableSpec.join(): the spec of a join of two frames, for everyhow,suffixandcoalescePolars takes, its columns named as the installed Polars names them. Claims about values are kept, a side that can go unmatched becomes nullable, and claims about rows are dropped, except auniquethe other side's unique key preserves.semiandantikeep the left spec but for its row count and order.TableSpec.prefix()andTableSpec.suffix(): every column renamed, asrenamerenames.- The finding code
row_count, added in a minor release as the Stability page allows.
Changed¶
- The documentation was reviewed page by page: sample outputs match the specs each page declares, the new fields are covered where a reader looks for them, and the site renders a dash as a dash.
1.0.0 - 2026-10-22¶
Stable. The public API, the spec-file format, seeded output, the command line and the finding and drift codes now change only in a major release, after a deprecation: the Stability page states each promise and the test that holds it. 1.0 removes what 0.x deprecated, below; its other changes were made in 0.23, so code that runs on 0.23 without deprecation warnings runs on 1.0. No seeded output changes.
Removed¶
What 0.x deprecated, each of which warned for at least one minor release:
| Removed | Deprecated in | Use instead |
|---|---|---|
polspec.profile_dataframe(df, ...) |
0.18.0 | polspec.profile(df, weights=False, shape=False, detect_unique=False, detect_order=False).columns |
polspec.serialization.COLSPEC_FIELDS, TABLESPEC_FIELDS, COLRULE_FIELDS, CHECK_FIELDS, FK_FIELDS |
0.18.0 | the keys the Spec files reference lists |
FrameSpec.checks() |
0.21.0 | FrameSpec.spec.checks |
FrameSpec.foreign_keys() |
0.21.0 | FrameSpec.spec.foreign_keys |
FrameSpec.unique_together() |
0.21.0 | FrameSpec.spec.unique_together |
A column may now be called checks, foreign_keys or unique_together
without shadowing anything. From 1.0, a deprecated name is removed only at
the next major release.
Documentation¶
- The site is slimmer: the changelog (on GitHub), roadmap and
generation-and-validation pages are gone -- the seeded-output history now
sits under Stability's promise, and the registry's edges and the
two-sides argument in Known limitations -- and the architecture page is
ARCHITECTURE.md, for contributors. Inferring a spec sits beside declaring one, and the command-line guide links to its generated reference rather than copying its flag tables.
0.23.0 - 2026-10-22¶
The surface 1.0 freezes, released first. A review before 1.0 read every
public name and signature, since 1.0 makes them permanent, and this release
makes every change it found: one namespace, polspec; constructors that
take their options by keyword; one spelling for each option and parameter;
typed options on every entry point, FrameSpec.drift among them, which
took four of its seven; and the internals out of the API reference. They
are breaking, made outright with no deprecation period, and listed below
with the replacement for each. No seeded output changes.
Breaking¶
0.23 makes every change to the public surface before 1.0 freezes it, so 1.0 freezes one that has been released. Each is made outright, without a deprecation period; this list, with its replacements, is the migration guide.
- One namespace: everything public is imported from
polspec.polspec.drift,polspec.validationandpolspec.serializationare internal now, as every other module is.
| Before | Now |
|---|---|
from polspec.drift import drift, diff |
from polspec import drift, diff (or polspec.drift(spec, df)) |
polspec.drift.DriftCode, polspec.drift.Severity |
polspec.DriftCode, polspec.Severity |
polspec.validation.FINDING_COLUMN, polspec.validation.FindingCode |
polspec.FINDING_COLUMN, polspec.FindingCode |
polspec.serialization.to_dict, from_dict, FORMAT_VERSION |
polspec.to_dict, polspec.from_dict, polspec.FORMAT_VERSION |
polspec.serialization.catspec_to_dict(cats), catspec_to_yaml(cats, path) |
cats.to_dict(), cats.to_yaml(path) |
polspec.serialization.catspec_from_dict(data), catspec_from_yaml(path) |
CatSpec.from_dict(data), CatSpec.from_yaml(path) |
polspec.serialization.registry_to_dict(r), registry_to_yaml(r, path) |
r.to_dict(), r.to_yaml(path) |
polspec.serialization.registry_from_dict(data), registry_from_yaml(path) |
Registry.from_dict(data), Registry.from_yaml(path) |
Every name already at the top level stays where it was.
- CatSpec reads an entry three ways, and its parts from three
mappings. cats["X"], cats.X and cats.get("X") give the dtype, matching
a name's case as before; cats.enums, cats.categoricals and
cats.choices hold what is underneath, keyed exactly.
| Before | Now |
|---|---|
cats.dtype_of("X") |
cats.get("X") |
cats.enum.X, cats.enum["X"], cats.enum("X") |
cats.X or cats["X"] |
cats.categorical.X (and its item and call forms) |
cats.X or cats["X"] |
cats.get_enum("X") |
cats.enums["X"], or cats["x"].categories.to_list() for any case |
cats.get_categorical("X") |
cats.categoricals["X"], or cats["x"].categories for any case |
cats.get_choices("X") |
cats.choices.get("X") |
- Constructors take everything past what identifies the thing by
keyword.
ColSpec(dtype, *, ...),TableSpec(name, columns, *, ...),Check(expr, *, ...)andForeignKey(columns, *, references=..., ...);ColRule,Hierarchy,ValidationOptions,DriftOptions,ProfileOptions,FindingandDriftFindingby keyword throughout.ColSpec(pl.Int64, "x")used to setcol_namewithout saying so.Bound(min, max)stays positional.
| Before | Now |
|---|---|
ColSpec(pl.Int64, None, None, True) |
ColSpec(pl.Int64, nullable=True) |
ForeignKey("customer_id", Customers) |
ForeignKey("customer_id", references=Customers) |
Hierarchy("node", "parent") |
Hierarchy(child="node", parent="parent") |
ValidationOptions("drop") |
ValidationOptions(extra_cols="drop") |
- One spelling for each validation switch: the field's name.
validate(df, checks=False), asValidationOptions(checks=False)and--skip checksalready said it. The old spelling is refused, naming the new one ('validate_checks' (did you mean 'checks'?)).
| Before | Now |
|---|---|
validate_rules=, validate_validators=, validate_unique=, validate_checks=, validate_foreign_keys=, validate_hierarchy=, validate_order=, validate_pattern=, validate_bounds= |
rules=, validators=, unique=, checks=, foreign_keys=, hierarchy=, order=, pattern=, bounds= |
- Writers take
path; a reader'ssourcestays.to_yaml,to_python,FrameSpec.to_yaml/to_python,CatSpec.to_yamlandRegistry.to_yamlcall their destinationpath, asto_markdown,to_mermaidand everysink_*already did.from_dict's label for its errors iswhere.
| Before | Now |
|---|---|
to_yaml(spec, source="orders.yaml") (and the other writers) |
to_yaml(spec, path="orders.yaml"), or positionally as before |
from_dict(data, source="orders.yaml") |
from_dict(data, where="orders.yaml") |
- Every drift and validation entry point takes
options=or typed keywords.FrameSpec.drift(df, ...)takes all sevenDriftOptionsfields -- it named four, sosignificance=was aTypeErrorthere and worked inpolspec.drift()-- anddrift(),diff()andFrameSpec.diff()type them (DriftKeywords).Registry.inspect_allandvalidate_alltakeoptions=or the validation keywords, typed, where they took untyped**options. FrameSpec.from_dataframe()names its spec"Profiled", asprofile()does, not"ProfiledFrameSpec".- Internals out of the public API.
Predis what builds, evaluates, writes and renames a predicate:to_expr,to_data,to_source,rename,equals, and the operators. Its tree walks --children(),rebuild(),root_names(),literals()-- are internal, as isTableSpec.resolve_target().CheckandForeignKeyno longer takepred=andtarget=, which they fill in themselves;check.predandfk.targetare still there to read.
| Before | Now |
|---|---|
pred.root_names() |
set(pred.to_expr().meta.root_names()) |
spec.resolve_target(fk) |
fk.target (or the spec itself, for references="self") |
TableSpec.estimated_size(n)ispolspec.estimated_size(spec, n);FrameSpec.estimated_size(n)stays.Finding.rows(frame)is internal;report.rows(finding)is how a report's rows are read.CliErroris no longer exported. The command line never let one reach a caller:main()turns every polspec error into an exit status.
0.22.0 - 2026-10-22¶
Beta: the contract, written down and tested. The
Stability
page states what polspec promises now and what 1.0 will -- the public API,
spec files, seeded output, the command line, the finding and drift codes --
and a test holds each promise to a golden copy, so a release cannot break
one without a test failing first. What 1.0 removes is listed there and
warns today. Two small changes come with it: every --json document leads
with a schema_version, and the --all documents key their reports under
reports; and schema infer no longer writes a null rate on a column with
no nulls, and rounds category frequencies. No seeded output changes.
Changed¶
-
Beta. The
Development Statusclassifier says so, and the API is frozen pending 1.0: no public name changes in place. What 1.0 removes is listed on the Stability page, Removed at 1.0, and each one warns today:profile_dataframe, serialization's five*_FIELDStables, andFrameSpec.checks(),.foreign_keys()and.unique_together(). -
Every
--jsondocument carries aschema_version, and so doesValidationReport.to_dict()andDriftReport.to_dict(). Its keys and their types are the promise, and the version rises if one is ever renamed, removed or changes type. validate --all --jsonanddrift --all --jsonkey their reports underreports:{"schema_version": 1, "reports": {"Orders": {...}}}. They were keyed by spec name at the top level, where a spec calledschema_versionwould have collided. A script readingdoc["Orders"]readsdoc["reports"]["Orders"]now.schema inferwrites what a person would. A column with no nulls carries no null rate -- it is not nullable, so the rate meant nothing, and every such column used to saynull_probability: 0.0. Under--round-rates(on by default) category frequencies are written at the rates' precision too, each to its nearest step and never to 0:weights=[0.4, 0.3, 0.3], not[0.4023872466554514, ...]. The same holds forprofile()andFrameSpec.from_dataframe()givenround_rates. What a spec generates and accepts is unchanged.
Documentation¶
- The finding codes table gains the three hierarchy codes it lacked:
hierarchy_multi_parent,hierarchy_cycleandhierarchy_depth. - Errors said an exception other than a
PolspecErrorkeeps its traceback on the command line; it is printed as one line, with its class, and exits 1. - Stability, a new page: what polspec promises now, in the beta, and what 1.0 promises -- the public API, spec files, seeded output, the command line and its exit codes, finding and drift codes -- with the test that holds each, what is not promised, and the supported Python, Polars and operating systems. The roadmap keeps where polspec is going, and the list of releases that changed seeded output.
- The command-line guide says a command line that is itself wrong exits
2.
0.21.1 - 2026-10-22¶
The documentation, current and usable. No code changes.
Documentation¶
- Infer a spec from data, a new how-to:
profileandFrameSpec.from_dataframeside by side -- one imitates the data, one describes it -- with every option,ProfileOptions, rounding, and theschema inferequivalent. Until now this was split across three pages, and rounding from Python was only in the API reference. - Every page declares what its examples use. Nine pages used specs only
the docs tests declared, and eleven used names they never imported, so
copying an example could fail with a
NameError. Two foreign-key examples passedreferences=to a spec with no foreign key; they now use one that has. - Testing pipelines gains a fixture per spec, a stage that must reject
bad rows (asserted against
inspect()andpassing_rows()), and fixtures that look like production. - Corrected:
schema infer's example output, which predated rounding; the cartesian example's row count (twelve, not sixteen); three error comments namingValueErrorforSpecError; four error messages shown shorter than they are; "early alpha" on two pages, against the README's "pre-1.0". - Roadmap and stability: which releases changed seeded output is a table, current to 0.21. The benchmark table is re-measured on 0.21.0.
- The home page lists row order among what a spec can declare, and links to inferring a spec. Known limitations no longer opens on internal module names; the architecture page says it is for contributors.
- The docs tests now check what a page shows, not only that its code runs: each error comment against the error raised, and each output block against what the block prints.
0.21.0 - 2026-10-22¶
A consolidation release: what 0.20 added, made sound. passing_rows()
keeps its promise under order_by and a key into the frame's own rows,
profile() is fast again, and an error in a spec file says where it is.
Profiling takes one options object, ProfileOptions, as validation takes
ValidationOptions, and synthesize takes all of it. The rest is
groundwork for the 0.22 contract: three FrameSpec accessors deprecated in
favour of .spec, a weekly deep run of the properties, and a test suite
that tests through the public API only.
Added¶
ProfileOptions: one definition of what profiling takes.profile,FrameSpec.from_dataframeandsynthesizetake the options as anoptions=object or as the keywords themselves -- asvalidatetakesValidationOptions-- and the keywords are typed (ProfileKeywords), so an editor completes and checks them.synthesizetakes every one now, not four of them: it can be told not to keep the source's order, not to fit shapes, or to round.FrameSpec.from_dataframetakessampletoo.
Changed¶
- The validation keywords are one typed list,
ValidationKeywords.validate,inspectand theFrameSpecfacades take**keywordstyped by it rather than spelling every option out four times, so an editor still sees each one. A misspelt option is refused asvalidaterefuses it, naming the option meant ('validate_uniqe' (did you mean 'validate_unique'?)), where the facades said only "unexpected keyword argument". Still aTypeError.
Deprecated¶
FrameSpec.checks(),.foreign_keys()and.unique_together(), removed in 1.0. Each call warns, and still works. Use.spec.checks,.spec.foreign_keysand.spec.unique_together: everything a declaration says is on.spec, which no column can shadow, and an order or a hierarchy needs no accessor of its own.
Documentation¶
- Known limitations says that an ordered spec can't be batched or streamed, as a hierarchy can't.
- The drift how-to lists row order among the pass/fail claims that
drift()leaves tovalidate(). - The constraints how-to gains a recipe for a sequence,
1, 2, …, n: a unique column bounded(1, n)and ordered by itself.
Fixed¶
passing_rows()could keep rows the spec rejects. Taking the failing rows away could break a claim relating rows to each other in the rows left: withorder_by,[3, 1, 2, 5, 4, 6]passed[3, 2, 5, 6], still out of order; with a key into the frame's own rows, a row whose parent failed passed as an orphan. What is left is now validated again until nothing more fails, so the passing rows are always a frame the spec accepts, andfailing_rows(once=True)names the rows taken later by the claim they broke -- the two still split the frame exactly. A spec with no such claim validates once, as before.profile()was about 20 times slower in 0.20, finding the order a frame comes in by ranking every column whole, both ways. It now reads each column's first rows first, which rules out nearly every unsorted column, and checks the rest whole: on 1,000,000 rows by 20 columns, order detection takes 8 ms, not 1.2 s. It finds the same order.- An error in a spec file says where it is. A declaration refused while
reading a file --
bounds: [5, 1], anorder_bynaming a missing column -- named neither the file nor the column. It now does (orders.yaml: columns.total: Bound min (5) must be <= max (1)), and a registry file's names the spec too (specs.orders: ...). The error keeps its type.
0.20.0 - 2026-10-21¶
Rows that come in order. A spec can declare the order its rows come in
-- __order_by__ = ["user_id", "ts"] for each user's events in time
order -- which event logs, time series, window functions and join_asof
all depend on. generate() sorts whole rows, so every other claim still
holds; validate() reports rows out of order; profile() finds an order
a frame already has, so synthesize() keeps it.
Inferred specs that read like written ones. polspec schema infer
now writes bounds rounded outward and rates rounded up -- [0, 9900], not
[3.17, 9871.23] -- at a precision you can choose, always keeping the
data valid; dates as dates rather than day counts; and an unbounded unique
ID column no longer stops at two million rows. A spec also comes back as
text from to_yaml() and to_python() without a path.
Python 3.15 is supported, with Polars 2, and tested in CI on every OS.
No seeded output changes: every catalogue case generates what 0.19.0
does, checked against the published 0.19.0 across 463 cases. No spec file
needs migrating: order_by and descending are new optional keys.
schema infer's files change by default (--round-bounds exact
--round-rates exact writes what 0.19.0 wrote), and so do profile()'s
temporal bounds and the order of a sampled frame's rows.
Added¶
- Python 3.15. The wheels are built against Python's stable ABI, so the
same wheel runs on 3.12 to 3.15; CI now tests 3.15 on Linux, macOS and
Windows. On 3.15, polspec requires Polars 2: Polars 1 does not run on
Python 3.15 -- some of its
Seriesmethods returnNonethere, and the process can crash after -- so pip installs Polars 2 on 3.15, and Polars 1.39 or later on 3.12 to 3.14 as before. PyYAML publishes no 3.15 wheels yet, so on 3.15 pip installs it from source -- its pure-Python version, with no compiler needed. order_by: rows that come sorted. A spec declares the columns its rows are sorted by, asDataFrame.sorttakes them --__order_by__ = ["user_id", "ts"], with__descending__for the direction -- for event logs, time series and anything a window function,join_asofor a keep-the-latest dedup reads in order.generate()sorts whole rows last, so every rule, check, composite key, foreign key and hierarchy still holds;validate()reports each row that belongs before the one above it (finding codeorder, skipped withvalidate_order=Falseor--skip order). Nulls come last. A spec file carries it asorder_by:anddescending:,to_pythonas the class attributes, and drift reports an order added as breaking. The streaming verbs --generate_batches,scan()and the sinks -- refuse an ordered spec, naming why, as they refuse a hierarchy: an order is a property of the whole frame. The Markdown render names the order.profile()declares the order a frame comes in: the first column of single values -- at least 20 of them distinct, so not by chance -- whose rows come sorted either way, nulls last. Sosynthesize()'s fake rows come in the source's order.detect_order=Falseturns it off;FrameSpec.from_dataframe(detect_order=True)andpolspec schema infer --orderturn it on, asdetect_uniqueand--keysare. An order within each key is the user's to declare.- Inferences at a person's precision.
polspec schema inferwrites each numeric column's bounds rounded outward and each null and NaN rate rounded up --[3.17, 9871.23]as[0, 9900],0.023651as0.024-- so a draft spec reads like a written one.--round-boundsand--round-ratestake a step (1000,1k,1M,0.01),auto(the default: two significant figures, and whole seconds for a timestamp) orexact. Outward and up, within each dtype's range, so the data still passes the spec it was inferred from; a rate of 0 stays 0.profile()andFrameSpec.from_dataframe()takeround_bounds=andround_rates=too, and write exactly what they observed unless asked. - A spec as text.
to_yaml(spec)andto_python(spec)-- andregistry_to_yaml, and theFrameSpecandRegistrymethods -- return the text when given no path, asto_markdownandcatspec_to_yamlalready did.
Changed¶
polspec schema infer's bounds and rates are rounded by default (auto, above).--round-bounds exact --round-rates exactwrites what 0.19.0 wrote.- A profiled date, time, datetime or duration column's bounds are dates
and times, as a person writes them --
[2024-01-01, 2024-12-31], not[19723, 20088]-- in the column's time zone. A nanosecond column's bounds round outward to the microsecond, Python's limit, so every value stays inside; a bound Python cannot hold stays the integer. The data generated is the same. -
A
unique=TrueInt64orUInt64(or 128-bit) column with no bounds makes room for the frame. Its default range, +-1,000,000, refused a frame of more than 2,000,001 rows; it now widens upward to fit. Only where it refused before, so nothing that generated changes. -
A sample keeps its source's row order.
profile(sample=...)and the command line's--sampledraw the same rows as before, for the same seed, but in the order the file has them, not shuffled: a sorted file's sample is sorted too, and reads in file order.
0.19.0 - 2026-10-13¶
The same seed, the same data, on every operating system. A column drawn from a non-uniform distribution could differ between Windows, Linux and macOS, because its sampler used each platform's math library. 0.19.0 samples with maths of its own, the same bits everywhere, and CI now checks every seeded case against one digest on all three. The Known limitation is gone, ahead of 1.0's promise that a seed's output is stable.
Seeded output changes, once, for some distribution columns -- on
Windows, a lognormal column and a batched beta column; everything
else draws what 0.18.0 drew. No spec file needs migrating, and no API
changes.
And faster: distribution and uniform columns generate as fast as 0.18.0 or faster, all but one case.
Changed¶
- A seed draws the same data on every operating system. A column drawn
from a non-uniform distribution (
normal,lognormal,exponential,gamma,beta,poisson) could differ between Windows, Linux and macOS: its sampler tookexp,lnandpowfrom each platform's math library, which round the last bit differently on a few inputs. The samplers are now polspec's own:rand_distr's algorithms, ported draw for draw, withexp,lnandpowfrom a port of Arm's optimized-routines -- the algorithms glibc uses, kept to plain IEEE arithmetic -- which round the same way everywhere and are as fast as a platform's own; alognormalcolumn draws faster than before. Seeded output changes for such columns wherever the platform's library and the port disagree. On Windows that is alognormalcolumn and a batchedbetacolumn only:normal,exponential,gammaandpoissoncolumns draw what 0.18.0 drew. On Linux and macOS, where 0.18.0 kept no record to compare against, the same kinds of column may change. A sampler draws the same random numbers as before, so where a value changes, it changes from the first draw the libraries disagree on. - Distribution columns generate faster. A column's values are drawn by
a loop specialized to its distribution, rather than one loop deciding
per value, so each holds only the code it needs. Against 0.18.0 on
Windows, 20 million values each:
lognormal8% faster,gamma5-9%,beta6%,exponential12%, a uniform column 6-12%;normalthe same, and apoissonwith a small lambda 5% slower. profile()rounds a fitted distribution's parameters to 12 significant digits --mean: 1.08778398481, not1.0877839848116615. They are measured with the platform's float maths (a log-normal's throughln), which may differ in the last bit between operating systems; rounded, a profiled spec, and whatsynthesize()draws from it, is the same on each in practice. A spec file reads more cleanly too. A spec declared or loaded from a file is unaffected.
0.18.0 - 2026-10-09¶
What you can import is what polspec promises. The public API is now
written down: polspec's __all__, which gains the seven names that
lived only in submodules, plus three public submodules,
polspec.drift, polspec.validation and polspec.serialization. Every
other module says it is internal, and a test holds the docs, the
docstrings and the command line to that line. There is one profiler,
profile(), and a public quarantine view, failing_rows(once=True).
Nothing that worked breaks. This is polspec's first release with
deprecations: profile_dataframe and serialization's field tables still
work, warn, and go at 1.0. No seeded output changes: every catalogue
case generates what 0.17.0 does, checked against the published 0.17.0
across 463 cases. No spec file needs migrating.
Added¶
- The public names that lived only in submodules are at the top level:
to_yaml,from_yaml,to_python,to_markdown,to_mermaid-- each taking aTableSpecor aFrameSpecclass, as every verb does, and each the function of the same nameFrameSpechas a method for --lit, besidecol, andestimated_size. The submodule paths still work: they are the same functions. The API reference has a Files and documents page. ValidationReport.failing_rows(once=True): each failing row once, naming every claim it broke -- the quarantine view--failingwrites, which withpassing_rows()splits a frame exactly. The default is unchanged.profile()takes the switchesprofile_dataframehad:weights,shape,detect_uniqueandcalculate_bounds, each on by default -- whatprofile()already did -- so one profiler covers both uses.- The public API is written down, and tested. It is
polspec's__all__, and the__all__ofpolspec.drift,polspec.validationandpolspec.serialization. Every other module says in its docstring that it is internal, and may change in any release (Roadmap and stability, The public API is whatpolspecexports). A test holds the line: the docs import only public names, every public class and function has a docstring, and no module takes another's private names -- the command line included, which now uses only whatreading,validationandcatspecexpose for it. The serialization functions for aCatSpecand aRegistrygained the docstrings they lacked.
Deprecated¶
profile_dataframe, removed in 1.0. Each call warns, and still works. The same columns areprofile(df, weights=False, shape=False, detect_unique=False).columns. This is polspec's first deprecation: a public name now warns for at least one minor release before it goes, rather than changing in place (CONTRIBUTING, Deprecations).polspec.serialization's field tables (COLSPEC_FIELDS,TABLESPEC_FIELDS,COLRULE_FIELDS,CHECK_FIELDS,FK_FIELDS), removed in 1.0. They describe polspec's own codec and were never documented; the Spec files reference lists the keys a file holds. They are out of the module's__all__, and each still works, and warns.
0.17.0 - 2026-10-08¶
Every Polars from 1.39 to 2.x. polspec declared polars>=1.44.1,<2;
it declares >=1.39,<3 now, and makes it true. Below 1.43 it had 24
failing tests: Polars features newer than 1.39, and Polars bugs fixed
since, each worked around in the one place it touched. CI tests the floor,
the newest 1.x and the newest 2.x on every pull request, and the same seed
generates the same data on all of them -- every seeded catalogue case is
compared against a stored digest of its values, on each. Comparing them
across operating systems too found one thing that never held: values drawn
from a non-uniform distribution may differ between Windows, Linux and macOS
(Known limitations). They are the same on every Polars on one OS.
No seeded output changes. Every spec that constructs under 0.16.0 generates what it did -- checked against the published 0.16.0 across 463 cases, and on every Polars from 1.39.0 to 2.0.0 across 356 -- and no spec file needs migrating.
Performance is flat on Polars 1.44 and 2.0: validation, drift,
profiling, synthesis and cartesian generation time the same as 0.16.0,
interleaved run for run. On 1.43 and later, an Array's elements are
checked exactly as before; below it, through a list, which older Polars
needs.
Fixed¶
- polspec works on Polars 1.39 to 1.43. Every test passes on the latest patch of each, from 1.39.0, where 24 failed before. Each cause was a Polars feature newer than 1.39, or a Polars bug fixed since:
- cartesian generation joined its columns with
concat(how="horizontal_extend"), which Polars has from 1.42 -- while 1.42 deprecateshow="horizontal"; it usesDataFrame.hstacknow; - an
Array's elements were checked witharr.eval, which before 1.42 refuses anis_inor a filter and, before 1.43, panicked filtering anArrayofDecimalorDuration; on those versions they are checked as a list's are, and from 1.43 exactly as before; - Polars before 1.43 writes an
Arraywith null rows to JSON wrongly -- a null row as[null, null], the rows after it shifted -- sosink_ndjsonand the command line write anArrayas the list it is in JSON; read()rebuilt an array's null row as an array of nulls on Polars before 1.43; it stays null.
Changed¶
- polspec supports Polars 1.39 to 2.x:
polars>=1.39,<3, where it declared>=1.44.1,<2. CI tests the floor (1.39.0), the newest 1.x (the lock) and the newest 2.x on every pull request. A seed generates the same data on every one: the seeded catalogue's values are compared, case for case, against a stored reference on each -- a column drawn from a non-uniform distribution on the operating system that recorded it. On Polars 1.39 alone, a nullable struct with no fields holds no nulls (Known limitations).
Internal¶
tests/contracts/test_seeded.pyholds the seeded output to a reference,seeded.json: a digest of the values each catalogue case generates -- every column case whole, batched and scanned, every dtype, the table-level features. It digests the values rather thanhash_rows, whose hash differs between Polars majors.POLSPEC_WRITE_SEEDED=1rewrites it, for a deliberate change.- CI: the newest-polars job resolves Polars 2.x and is the required Polars 2 run; the pinned 2.0.0 job and the pre-release job are gone.
0.16.0 - 2026-10-07¶
Any column name. The review of 0.15.1, on the day Polars 2.0.0 was
released, found one class of defect: a column polspec could not reach. A
column named * or ^…$ is a pattern to Polars; a column named __row,
__polspec_row or count collided with a helper column of polspec's or
Polars' own; and failing_rows() wrote over a column named
__polspec_finding. Every name now works, and that one is refused by name.
The property tests draw all of them, and a test keeps a bare pl.col of a
column name out of the package.
Polars 2.0.0. The suite passes on it unchanged, and CI runs it on every
pull request. A panic in Polars 2.0.0's streaming engine, on validation's
aggregation over a list column, is avoided by validating on the in-memory
engine. polspec still declares polars<2; 0.17.0 widens that to <3, and
lowers the floor to 1.39.
Changed, on purpose: col("*") in a check means the column named *.
No seeded output changes. Every spec that constructs under 0.15.1 generates what it did -- checked against the published 0.15.1 across 463 cases -- and no spec file needs migrating.
Performance is flat. Validation, drift, profiling and synthesis time the same as 0.15.1, interleaved run for run on a million rows; an ordinary column name compiles to exactly the expressions it did.
Fixed¶
- A column named
*, or^…$, works like any other. Polars readspl.col("*")as every column andpl.col("^id$")as a regular expression, and so doselect,group_by,pl.struct,dropand a join's keys given the name as a string. Every place polspec names a column now refers to it exactly -- generation, validation, foreign keys, profiling, drift,read()-- through one helper,frames.column, which is the samepl.colas before for every other name. Many PolarsSeriesmethods also select their Series by its own name, so a Series named^c$, a pattern that does not match itself, failsfill_null,is_null,uniqueorexplodein Polars 1 and 2. polspec measures such a column under a plain name, and draws it under a plain name seeded by its own, so its values are what any name would draw. Known limitation since 0.15.1; the test that pinned it now passes. - polspec's helper columns no longer take a name the data has. A list
column named
__rowcould not be generated; a column named__polspec_rowmadepassing_rows(), and--outputwith--failing, raiseDuplicateError; a column namedcountmadeprofile()raise, since Polars'value_countsnames its own column that. Each helper column now takes a name absent from the frame, or polspec measures the data under names of its own. failing_rows()no longer writes over a column named__polspec_finding. It wrote the finding keys over the data's own column of that name, silently; it raisesValueErrornow, naming the column, andpolspec validate --failingrefuses before writing either file.- Validating a list column no longer panics on Polars 2.0.0. Polars
2.0.0 runs a lazy query on its streaming engine by default, and that
engine panics on validation's single aggregation over a list column --
list.evalbeside afilter-- for some frames: one concatenated from batches, whose columns are chunked differently, and a two-row frame of a list of categories beside a column of nulls. Found by the batched and arrival properties, about one draw in 25,000, and reproduced without polspec. Validation runs on the in-memory engine unlessstreaming=Trueis asked for -- the engine a lazy query already ran on in Polars 1, and faster on 2.0.0 too (26 ms against 34 ms for a million rows).
Changed¶
col("*")in a check means the column named*, ascol()means a column everywhere else -- not every column. Likewisecol("^id$").
Internal¶
- The lock moves to Polars 1.44.2, and the required Polars 2 CI job from the 2.0.0rc2 release candidate to 2.0.0.
POLSPEC_DEEP_EXAMPLES=3000 uv run pytest tests/contracts/test_properties.pyruns every property 3,000 times; CONTRIBUTING has a Deep runs section. A new property holds generated data to 0.14.0's promise: a spec does not drift from its own output.scripts/deep/render/parses drawn specs' ER diagrams with Mermaid 11 and 10 and their data dictionaries with GitHub-flavoured Markdown rules;scripts/deep/polars2_eval_chunks_repro.pyreproduces the Polars 2.0.0 panic without polspec, for the upstream report.
0.15.1 - 2026-10-05¶
What polspec writes, it reads back -- and a person can open. A patch
release from the 0.15.0 review, about the edges rather than the core: the
documents, diagrams and spec files polspec writes for people, the NDJSON it
writes for itself, and the quarantine file --failing writes. Each fix
makes something that failed work, or refuses it by name, and is pinned by a
test that fails on 0.15.0. The property tests now draw awkward names --
pipes, spaces, keywords, the empty string -- and found two more defects of
their own.
Changed, on purpose: --failing writes each failing row once, naming
every claim it broke. A Binary column is refused when writing JSON, and a
column name holding a NUL when the spec is declared; both panicked inside
Polars before.
No seeded output changes. Every spec that constructs under 0.15.0 generates what it did -- checked against the published 0.15.0 across 463 cases -- and no spec file needs migrating.
Fixed¶
to_markdown()keeps its tables whole, whatever the names. A|in a column name, category, choice, tag or pattern split its cell -- inside backticks too -- and a newline ended the row; both are escaped now, in the data dictionary, the drift report and the category registry. A name holding a backtick is fenced so its code span holds, and an empty name is an empty code span rather than a stray "``".to_mermaid()draws a diagram Mermaid can parse, whatever the names. An attribute is drawn under a name Mermaid reads as one word (ASCII letters, digits,_,-, not a leading digit, notPK,FKorUK), with its real name first in the note when that changed it; two that would draw alike are told apart by a suffix. An entity name no longer starts with a digit, holds a non-ASCII letter Mermaid 10 refuses, or is a word Mermaid reserves (class,end,one, ...). A~in a note -- a choice, a pattern, a name -- is drawn as the look-alike∼, since Mermaid 10's parser hung on it. Names that already read are drawn as before.to_pythonrefuses a spec name it cannot declare, instead of writing a module that does not parse. A spec's name is the class name, and aTableSpecmay be named anything:Odd Name,class, or a name Python would read as another (fileasfile) wroteclass Odd Name(FrameSpec):. It raisesSerializationErrornow, before anything is written, naming aspec.with_name(...)that works andto_yaml(), which keeps any name.- What
sink_ndjsonwrites,read()reads back. Polars writes aDecimalto JSON as text ("1.25"), aDurationas ISO 8601 seconds ("PT3600S"), a fixed-sizeArrayas a list, a date inside a list or struct as text, and a column of nothing but nulls with no type -- so a spec's own NDJSON output failed its own validation.read(), and the command line through it, now reads each back as the spec declares it, when every value reads exactly: a decimal from its digits and never rounded, a duration with integer arithmetic to the nanosecond, an array only when every list has its width. Anything else stays as it arrived and is reported. Every case in the test catalogue JSON can hold now reads back equal to what was generated; NaN, which JSON writes asnull, cannot. - Writing JSON no longer panics on categories. A list of a few hundred
EnumorCategoricalvalues panicked Polars' JSON writer (1.x and 2.0 alike), insink_ndjsonand inpolspec generate,--outputand--failingto.ndjson,.jsonlor.json. Categories are written as their text -- the same file -- and read back as declared. - A list or array column named
""generates. It raised aSpecErrorabout its element'sseed_name; its values are what a column of any other name would draw.
Changed¶
polspec validate --failingwrites each failing row once. A row breaking several claims was written once per claim, so--outputand--failingheld more rows than the file and a count of rejected rows overcounted. Each row is written once now, in the file's order, its__polspec_findingnaming every claim it broke, comma-separated: the two files split the input exactly.ValidationReport.failing_rows()is unchanged -- once per claim, for grouping by claim.- A
Binarycolumn is refused by name when writing JSON, which has no bytes, with aGenerationErrorfromsink_ndjsonand an error from the command line, before anything is written. It panicked inside Polars' writer before -- an exceptionexcept Exceptiondoes not catch. - A column name holding a NUL character is refused when the spec is
declared, with a
SpecError. It constructed before, then panicked in Arrow's C interface the first time anything generated.
Internal¶
- The property tests draw awkward names -- punctuation Markdown and Mermaid read, whitespace, keywords, leading digits, non-ASCII, the empty string and arbitrary text -- and hold generation, validation, YAML, the data dictionary and the ER diagram to them. They found the two defects above.
- The README's example runs in the test suite, on its own, as someone copying it would run it.
- Every case in the test catalogue is written with
sink_ndjsonand read back withread(), and must equal what was generated. - A newly known limitation, found by the same properties: a column named
*or^…$cannot be generated or validated, since Polars readspl.col()of either as a pattern. It is listed in Known limitations and pinned by a strict xfail, so the day it works the suite says so.
0.15.0 - 2026-10-04¶
A profile never contradicts its own data. 0.14.0's promise -- a spec
does not drift from data it describes -- broke in two places: a country or
currency column profiled from a file drifted from that file, and a
hierarchy's nulls and a rule's NaN drifted from their own generated output.
Both are fixed, each pinned by a test that fails on 0.14.0, and a realistic
source is now checked against its own profile. A few repeated country codes
also keep their frequencies through synthesize again, which 0.14.0 lost to
the format.
Changed, on purpose: a few repeated country or currency codes profile as
an Enum again; --sample N is a random sample of the file rather than its
first N rows; a composite key's column cannot declare nan_probability.
No seeded output changes. Every spec that constructs under 0.14.0 generates what it did -- checked against the published 0.14.0 across 463 cases -- and no spec file needs migrating.
Performance is unchanged -- generation, validation, drift, profiling and synthesis measured against 0.14.0 on one build of the extension, all within noise.
Fixed¶
- A spec drifted from data it describes, three ways. A column with a
finite
format--iso_country,iso_currency-- counted every code as a declared value, so a profiled country column drifted from its own source (cardinality_moved: 224 of 249 never appear); a format promises a valid code, not every code, and onlychoices,Enumcategories andextra_valuescount now. A nullableHierarchycolumn drifted from its own output on its null rate, and a column a rule rewrites on its NaN share: drift now skips, on each column a pass writes, exactly the claims that pass overrides -- measured per pass, so a foreign key's null rate is still watched. polspec driftcan set every drift option. 0.14.0 addedsignificance,frequency_toleranceanddistribution_tolerancetoDriftOptionsand no flags for them;--significance,--frequency-toleranceand--distribution-tolerancenow exist, every flag's default is the option's own -- said once, onDriftOptions-- and a test holds the command to the options. An option out of range is anerror:line, not a traceback.
Changed¶
- A few repeated country or currency codes profile as an
Enumagain. 0.14.0 named theiso_countryoriso_currencyformat before a column could narrow, so a column nine tenthsGBprofiled as every country evenly, andsynthesizespread it over 249. A finite format is now named only where the column would not narrow to anEnum-- more distinct codes thanmax_unique_enum, or areplaced column -- and anEnumkeeps the codes' frequencies. Template formats (email, UUID, IP, MAC, hostname) are named first, as in 0.14.0. --sample Nis a random sample.polspec schema inferandpolspec driftread a file's first N rows, which for a file sorted by date or by amount is not a sample of it: bounds that reject the rest, a distribution that has "moved". Both now draw N rows at random from the whole file, seeded so the same file gives the same result, aspolspec synthesizealready did; a sample of zero or fewer rows is refused.- A composite unique key's column cannot declare
nan_probability, as aunique=Trueone cannot: no NaN equals another. The repair that separates repeated combinations hashed NaN as one value and generated none of a declared share.
Internal¶
docs/llms.txtanddocs/llms-full.txtare built by the docs workflow, not committed. Every page or docstring edit regenerated 400 KB of tracked text -- 78 of the repository's first 148 commits touched it. The site serves the same files, built from the same source. So that the build can import polspec without installing it,polspec.__version__is"0+unknown"in a checkout that was never installed, where it used to raise.
0.14.0 - 2026-09-29¶
A profiled spec says what the data is, and drift notices when that stops
being true. A float declares how often it is NaN (nan_probability); the
profiler records that share, names a text column's format -- emails,
UUIDs, IPs, country codes -- and makes a text column an Enum only when its
values repeat; and drift compares weighted frequencies and declared
distributions, reporting a move only when it is significant at the rows
seen and large enough to matter.
Breaking, on purpose:
- a NaN in a float column that does not declare one is a nan finding;
- spec files are format version 4, and polspec 0.13 and earlier refuse
every file 0.14.0 writes;
- from_dataframe() and schema infer infer differently: formats named,
Enums only of repeated values;
- drift reports fewer, better-founded findings: no small-sample rate
alarms, no dtype that validates, and new codes for frequencies and
distributions.
No seeded output changes. Every spec that constructs under 0.13.1 generates what it did -- checked against the published 0.13.1 across 435 cases -- and a version 3 spec file reads as before.
Generation and validation run as fast as they did -- measured against
0.13.1 on one build of the extension. Drift and profiling do more than they
did, and take longer for it: on a million rows, drift with two declared
distributions 44 ms to 79 ms, profiling with a text column checked for
formats 0.10 s to 0.22 s (formats=False skips that).
Added¶
ColSpec.nan_probability: NaN in a float value, declared. How often a present value is NaN, from 0 to 1, on a float column, a list's float elements or a struct's float field. Generation draws NaN on that share of the present rows -- whole and batched alike, from a draw of its own, so a column that declares none generates exactly what it did -- and spec files carry it asnan_probability.diffcalls allowing NaN compatible and forbidding it breaking;driftreports a NaN share that has moved, significantly and by at leastnull_rate_tolerance, asnan_rate_moved. Refused on any other dtype and withunique=True.- Profiling records a float's NaN share.
profile(),from_dataframe(),synthesize()and both commands record how often a float column -- or a list's float elements, or a struct's float field -- is NaN as itsnan_probability, so a source holding NaN validates against its own profile (pinned as a strict xfail since 0.13.1, now passing) and its fake data holds NaN on the same share. The 0.13.1 warning now names infinities alone, which stay a matter of bounds. - Profiling names a text column's format.
profile(),from_dataframe(),synthesize(),polspec schema inferandpolspec synthesizename theformata text column's values have --uuid4,ipv6,ipv4,mac,iso_country,iso_currency,email,hostname, tried in that order -- when at least twenty distinct values have it and every value does, or 99% of rows do and the rest are at most five repeated stand-ins, which becomeextra_valuesat their observed shares. A hostname has to hold a dot. A synthesized email column is email-shaped now, where 0.13.0 made text of an email's length. On by default;formats=Falseor--no-formatsturns it off. Areplaced column is named a format only when every value has it. - Drift on frequencies and distributions.
frequencies_movedcompares a weighted column -- anEnum, aBoolean,choices-- with itsweights;distribution_movedcompares a column that declares adistributionwith a sample drawn from the declaration itself, so the clamping at its bounds and an integer's rounding are what generation does.DriftOptionsgainssignificance,frequency_toleranceanddistribution_tolerance. - A foreign key from a parent that holds NaN is refused unless its column declares NaN too -- the parent's NaN keys would fill it, and fail its own claim.
- The data dictionary shows a NaN share in the Nullable column
(
Yes; NaN 5%), and the diagram as a comment.
Changed¶
- A NaN a float column does not declare is a finding. The default,
nan_probability=0, claims a column holds numbers, so validation now reports a NaN in it as a row-levelnanfinding, anddriftasnew_values-- where before a NaN in a column with noboundspassed unremarked. And NaN is no longer judged byboundsorchoices: in a bounded column it used to be aboundsfinding, since it compares greater than every number; it is now thenanfinding alone, and nothing at all where it is declared. - Spec files are format version 4.
nan_probabilityis the reason: a float that does not declare it now claims to hold no NaN, which an older reader would not check, so the version has a 0.13 reader refuse the file by name rather than read it more loosely than it was written. Every file 0.14.0 writes carriesversion: 4-- polspec 0.13 and earlier refuse all of them, whether or not they use the key. Version 3 files read as before, through a migration that changes nothing. - A profiled text column becomes an
Enumonly when its values repeat.from_dataframe(),profile(),synthesize(),polspec schema inferandpolspec synthesizenarrowed any text column with at mostmax_unique_enum(50) distinct values to anEnumof them -- so forty distinct names in a forty-row sample became anEnumof those forty, and the spec rejected every other name. The values now have to repeat, each at least twice on average (polspec.profiler.ENUM_MIN_REPEATS); a column of distinct values staysString, with its lengths, and aCategoricalstaysCategorical. A spec inferred before is unchanged unless it came from such a column; the same holds for a list's elements and a struct's fields. - Drift reports a move only when it is both significant and large. A
null rate, an element null rate or a NaN share is reported as moved only
when it is at least
null_rate_tolerancefrom the declared rate and unlikely -- atsignificance, 0.001 by default -- to be sampling noise over the rows seen: fifty rows of a column declared 30% null holding 22% used to be a finding, and a spec's own output could drift from it. The same rule holds for the new frequency and distribution findings, and a declared value missing from the data iscardinality_movedonly when, at its declared share, missing it is itself unlikely. A column a pass rewrites -- a rule, a foreign key, a hierarchy, a composite key -- is not held to its weights, distribution or unseen values, none of which its values are drawn by. A spec's own seeded output does not drift from it at 50, 5,000 or 5,000,000 rows. - A dtype that validates is no drift finding.
Stringfor anEnum,Int64for anInt32,Datetimefor aDate-- what a CSV hands back -- used to be a compatibledtype_changedagainst data thatvalidate()accepts; it is now reported only understrict_dtypes=True, where it is breaking.diffbetween two declarations still reports every change.
Fixed¶
- A test that runs
pyteston a generated file could fail now and then on Windows: left to find its own root, the subprocess scanned directories up into the system temp folder, where another program's file vanishing mid-scan failed collection. It is now rooted in the file's own folder.
0.13.1 - 2026-09-28¶
Fixes, for data nobody generated. 0.13.0 pointed polspec at real
tables; this release makes it answer for them. inspect() promises never to
raise for a frame that fails, and three ways a frame arriving in another
dtype broke that are closed -- most found by a new property test that
carries generated frames through the dtypes real data arrives as. Profiling
survives NaN and infinities, a Null column generates, and every function
that takes a spec takes a FrameSpec class.
Validation is stricter in two places, on purpose: a value the declared
dtype cannot hold (an Int8 column arriving as Int64 with 1000 in it) is
now a dtype finding, and a naive Datetime no longer stands in for a
zoned one, or a Time or Duration for a Date. Each was a frame that was
not what its spec declared, and could end in a raw Polars error one step
later; a frame with no findings now always survives validate(cast=True).
No seeded output changes. Every spec that constructs under 0.13.0 generates what it did -- checked against the published 0.13.0 across 435 cases -- and no spec file needs migrating.
Fixed¶
- A temporal stands in only for its own kind. Validation's default,
permissive dtype check accepted any temporal for any declared temporal,
so a
Timeor aDurationpassed for aDatecolumn with no bounds, and madeinspect()raise Polars'InvalidOperationError-- andpolspec validateend in a traceback -- for one with bounds. ADateand aDatetimestill stand in for each other, as text formats need; aTimeonly for aTimeand aDurationonly for aDuration, any unit. Anything else is adtypefinding. Stricter: a naive and a zonedDatetimeno longer stand in for each other -- one is a wall-clock reading, the other an instant; two zones still do. Drift'sdtype_changedseverity follows, decided by the same function. - A value the declared dtype cannot hold is a finding. A wider dtype
stands in for a declared integer or
Decimal, and its values were never asked to fit: anInt8column arriving asInt64with 1000 in it, aUInt8with -5, passed -- andvalidate(cast=True), which only runs on a frame that passed, then raised Polars' conversion error. Stricter: such a value is now a row-leveldtypefinding keyed<column>__dtype_range, with samples and rows, in list elements, struct fields and map values too. A float or integer too large for aDecimal's integer digits, after rounding to its scale, is one as well. A declared bound has to fit its dtype, so on a side a checked bound closes, such a value is out of bounds too and reported there alone -- which keeps the check free for the usual case, a CSV'sInt64for a boundedInt32. A frame with no findings now always survivescast=True. - A claim over a column that arrived as another dtype no longer takes
inspect()down. Found by a new property test that carries generated frames through the dtypes data really arrives as. A rule or check reading a column of the wrong dtype -- aBooleanflag read as text -- was compiled anyway and Polars refused it; it is now skipped, as one reading a missing column is, the column being adtypefinding already. A check'sis_inover dates refused aDatetimecolumn standing in for aDate. And a column's or rule'schoicesagainst a dtype standing in for the declared one -- datetimes against aDate-- are now asked of the column ascast=Truewould hand it back, so a clean report stays clean once cast. Temporal bounds are measured the same way: a bound a microsecond past midnight, on aDatetimecolumn arriving as aDate, rounded to the day and passed a value the cast then put outside it. - A float column holding NaN or an infinity can be profiled. An
infinity made
profile(),synthesize(),from_dataframe(),polspec schema inferandpolspec synthesizefail -- a bound must be finite -- and a NaN was left outside the inferred bounds without a word, so the spec rejected the file it came from. The bounds are now the finite values', and aUserWarning(awarning:line on the command line) names each column holding either, with how many. A spec still cannot declare NaN, so validating the source against its profile reports them; see Known limitations. - A
Nullcolumn generates. A column of nothing but nulls -- what an empty column is in Parquet -- could be profiled but not generated, sosynthesizefailed on it, and it could not be written to a spec file.ColSpec(pl.Null, nullable=True)now generates all nulls, writes asNull, and validates any dtype holding only nulls, a value in it being a row-leveldtypefinding. It must be declared nullable, and its null rate is 1. - A column named
""came back from generation namedcolumn_0, so a spec -- or a synthesized file -- with an empty column name failed. - Every function taking a spec takes a
FrameSpecclass.polspec.generate(Orders, 10)-- andgenerate_batches,scan, the foursink_*functions,inspect,validateandpolspec.sizing.estimated_size-- failed withAttributeError: type object 'Orders' has no attribute 'columns'; each now reads the class as its.spec, asdrift,readand the renderers already did. A class that is not a spec is refused naming the class.
0.13.0 - 2026-09-27¶
Two features: a fake dataset from a real one, and a format with values of
your own beside it. polspec.synthesize turns a DataFrame, a
LazyFrame or a data file into a stand-in with the same schema, shapes,
frequencies and keys, none of its rows, and no value of a column named in
replace=; extra_values widens a format -- ISO countries plus UK (ISO),
IPv4 or NOT AVAILABLE -- for validation and generation alike.
Three breaking changes, each a refusal of something that already did
not work: a choice its dtype cannot hold, and a check naming a column the
spec does not declare, are refused at declaration -- both used to fail at
generation or validation -- and TableSpec == compares columns in order.
No seeded output changes. Every spec that constructs under 0.12.0 and still constructs generates what it did -- checked against the published 0.12.0 across 519 cases -- and no spec file needs migrating.
Alongside them, a round of fixes found by fuzzing specs nobody wrote down:
Time, Duration and Decimal values in spec files, checks Polars
refused to evaluate, a skipped rule's claims, and to_python imports.
Added¶
-
polspec.synthesize: a fake dataset from a real one. From aDataFrame, aLazyFrameor a data file, a frame with the same schema whose columns follow the source's shapes, category frequencies and null rates, whose keys stay distinct, and which holds none of its rows.replace=names columns whose values must not be carried over; text in them is generated from its lengths, never as anEnumof the values the source holds.sample=profiles a random sample of a large source.polspec synthesizedoes the same from the shell, and--specwrites the spec the data was generated from. Columns are profiled one at a time, so relationships between them are not kept -- see Fake data from real data. -
ColSpec.extra_values: a format with values of your own beside it.ColSpec(pl.String, format="iso_country", extra_values=["UK (ISO)", "UK (ISLANDS)"]), orformat="ipv4"withextra_values={"NOT AVAILABLE": 0.05, "INVALID": 0.01}. Validation accepts a value that has the format or is an extra; generation draws each extra on its share of the present rows -- 1% each for a list, the given share for a mapping -- for a finite format and a template one alike, whole and batched. It is written to spec files (a list, or a mapping when a share is not the default), drifts as a domain (an extra removed is breaking), and takes part in a foreign key's domain check.formatbesidechoicesis still refused, and the message now points here. -
polspec.profile, the half ofsynthesizethat describes the source as aTableSpec, to review, edit or keep before generating. -
Profiling can fit a shape, find keys, and leave values behind.
from_dataframe()andprofile_dataframe()takeshape=True-- each numeric or temporal column's distribution, fitted against every one the engine draws and kept if it beats drawing evenly (polspec.shape) --detect_unique=True, andreplace=;polspec schema infertakes--shape,--keysand--replace. All three are off by default: a profile with none of them is exactly what it was.
Changed¶
-
A choice its dtype cannot hold is refused at declaration.
ColSpec(pl.Int8, choices=[1000])constructed, and generation gathered the choice through a lenient cast: it came out as a null, in a column that need not allow one, and the spec failed its own validation.1.5on anInt8came out as1, a domain nobody declared, and text on aDatecolumn failed inside Polars at generation. A column's choices and each rule's are now held to the column's dtype and refused, naming them, when it cannot hold them as written. Text on aStringcolumn, an integral float on an integer one, a date on aDatetimeand the nearest float on a float column are all held, as before. -
A check naming a column the spec does not declare is refused. It constructed, and failed only when validated.
pl.all()and a regex selector name no column, and pass. -
TableSpec ==compares columns in order.columnswas compared as a mapping, so two specs holding the same columns in a different order were equal though theirschema()s -- and every frame generated or validated against them -- differ. They are now two specs.
Fixed¶
-
Validating a frame that lacks a column a check needs no longer crashes. The check's mask was built regardless, and Polars raised
ColumnNotFoundErrorout ofinspect()andvalidate()-- and out ofpolspec validateas a traceback. The check is now skipped, as a rule or composite key already was: the absent column is themissing_columnsfinding, or was allowed to be absent. -
drop()andselect()remove the checks they strand, as they remove a composite or foreign key that loses a member. -
A rule that cannot be evaluated no longer hands its rows to the next. Rules are first-match-wins; with an earlier rule's column absent (
missing_cols="allow"), a later rule was checked against rows the earlier one had rewritten, and reported them as failing. Once a rule cannot be evaluated, the rules after it on that column are not checked either. -
TimeandDurationvalues can be written to a spec file. YAML has no form for a time of day or a duration, soto_yaml()crashed with PyYAML'sRepresenterErroron aTimeorDurationbound, a choice, a rule's choice, or a check's literal -- a spec could be declared in Python that no file could hold. Both are now written in a tagged form,{time: "12:30:00"}and{duration: {days: 1, seconds: 0, microseconds: 0}}, and read back as themselves. Not a new format version: a file without them reads as before. -
Decimalchoices can be written to a spec file. A Decimal bound was always written as the exact string it prints as; a Decimal choice was written raw, andto_yaml()crashed on it. Choices now follow bounds, and a text choice on aDecimalcolumn is read back as aDecimal. -
to_python()imports what it writes. Its imports were guessed from the bounds alone, so aDecimalchoice wroteDecimal('0.5')into a module that never imported it. They now follow from the source written. -
Anything else YAML cannot hold is a
SerializationError, naming where in the file it sits (columns.price.choices[0]), and raised before the file is opened, so no half-written file is left behind. -
A check naming a column twice no longer crashes validation. A range check --
(col("a") > 0) & (col("a") < 10)-- namesatwice, and the struct its samples were drawn into had two fields of one name: Polars raisedDuplicateError. Each column is now named once. -
is_inover datetimes, times and durations works on a coarser unit. Python holds them to the microsecond, and Polars'is_inrefuses a list finer than the column's unit:col("d").is_in([timedelta(0)])on aDuration("ms")column raised, in a check and in a rule's condition -- and so atgenerate(). Such a membership is now evaluated as the equalities it stands for.
Internal¶
- The Hypothesis round trip draws what it did not: choices from every
dtype's own value space,
TimeandDurationbounds, a rule on a column, and checks whose literals have the column's type -- including a range check that names its column twice. Its first runs found the last three fixes above. A second property draws declarations that cannot mean anything and requires each to be refused as aSpecError.
Documentation¶
-
Specs as files showed a check comparing against
subtotal, a column itsOrdersdoes not declare -- the new refusal found it -- and quoted the unrepresentable-check warning as polspec worded it beforecol(). -
Roadmap and stability no longer says
drifttakes one spec:polspec drift --allruns over a directory, asgenerateandvalidatedo.
0.12.0 - 2026-09-26¶
One seeded change, once, and a fix rather than a choice: the elements of
a List, Array or Map column in every batch after the first of
generate_batches, scan() or a sink. Every batch used to draw them from
the first batch's seed, so a stream repeated its opening elements over and
over; each batch now draws its own. generate(), and a stream's first
batch, produce exactly what 0.11.0 did, and no spec file needs migrating.
One breaking change: a report is falsy when it is empty. if report:
now means something was found; write report.passed or
report.unchanged for the opposite.
Alongside them, three more fixes -- a projected cartesian scan, a chunked
Array on Polars 1, TableSpec.rename and checks -- reference pages for
ColSpec's fields, the spec-file format and the command line, and CI that
tests Map, the lowest dependency versions and every release wheel.
Changed¶
- A report is falsy when it is empty.
ValidationReportandDriftReportdefinedbool()as "passed" or "unchanged" whilelen()counted their findings, so an empty report was truthy -- the reverse of every other collection.__bool__is gone: a report is truthy exactly when it holds findings, soif report:now means something was found. Code that wroteif report:for "passed" must writeif report.passed:(orreport.unchanged), which has always worked.
Fixed¶
-
A nested column no longer repeats itself batch after batch. The elements of a
List,ArrayorMapcolumn were drawn from the same seed in every batch ofgenerate_batches,scan()and the sinks, so a stream of any size repeated its first batch's elements over and over. Each batch after the first now draws them from a seed of its own. This changes the seeded output of those batches;generate(), and the first batch, are unchanged. -
Projecting a
method="cartesian"scan no longer changes its values. A scan generated only the columns a projection asked for, but a coverage set is the product of every column, so dropping one changed the rows of the others:scan(..., method="cartesian").select("c")was not the full scan'sc. A cartesian scan now generates every column and projects afterwards. -
Validating an
Arraycolumn in several chunks no longer panics on Polars 1. Taking distinct samples of anArraycolumn that arrived in more than one chunk -- concatenated, or read from a file in row groups -- panicked inside Polars 1 whenever no row failed, which is every frame that passes. Samples of anArrayare now taken as the lists they are; they come back as the same values. Polars 2 was not affected. -
TableSpec.renamerenames checks and validators too. It rewrote rules, keys and the hierarchy but leftchecksalone, so a check naming a renamed column kept naming the old one, and validating the renamed spec failed inside Polars with aColumnNotFoundError. A check or validator written withcol()is now renamed with its column -- its name following if it was the default -- and one written as a rawpl.Exprthat names a renamed column is refused atrename()with aSpecErrorsaying why. A validator written withcol()no longer stops its column being renamed.
Documentation¶
- Three reference pages, each held to the code by a test: ColSpec fields (every field, what it applies to, and what generation and validation do with it), Spec files (the YAML format key by key, the dtype forms, the predicate operations and the format versions), and Command line (every command and argument, generated from the parser).
- The how-to guides are grouped -- declaring a spec, using one, specs as values and files, the command line -- rather than listed as fourteen.
- The home page names
Map, patterns, hierarchies, theRegistryandread(), and no longer says Polars is the only runtime dependency. The README points into the docs instead of copying part of their nav, and CONTRIBUTING's checks match what CI runs.
Internal¶
- The round-trip contract covers the streaming verbs and drawn specs:
tests/contracts/test_streaming.pyruns the case catalogue (tests/cases.py) throughgenerate_batchesandscan, andtests/contracts/test_properties.pyholds specs Hypothesis draws to the round trip, whole, batched and through a spec file. - Spec files from every older format version are pinned as golden fixtures
under
tests/serialization/fixtures/, each read against the spec it has to become. - CI: a required job on a pinned Polars 2 release candidate, so
Mapis tested where it gates a merge; a job on the lowest version of every direct dependency; a coverage floor of 90% (branch) on one leg of the matrix; and each release wheel installed alone and smoke-tested before it is published. The docs rebuild whenCHANGELOG.mdchanges. - The
devgroup'spyarrowfloor rises to 16, the first that imports beside numpy 2. - The package is laid out by side. Declarations (
ColRule,ForeignKey,Hierarchy) no longer carry the generation passes that apply them, which live inpolspec.generation.passes; the engine, the pipeline and the passes are all underpolspec.generation.polspec.constraintsispolspec.domainandpolspec.pass_order, andpolspec.report-- rendering -- ispolspec.render, so no two modules share a name with a different meaning. Helpers used across modules lose their leading underscore, and a handful of duplicates are one copy. None of these modules is public API, and no seeded output changes. - An import-linter contract (
lint-imports, in CI) holds the layering: declarations import no verb, file format or engine; generation and validation import nothing of each other; only generation reaches the Rust extension.
0.11.0 - 2026-09-26¶
One seeded change, once: a unique=True column filled by a foreign key
generates different parent keys for the same seed -- in generate() too,
not only when batched -- because it is now a permutation of the parent's
keys. That is what makes it unique across a whole frame however it is
produced: until now, generate_batches, scan() and the sinks drew it
afresh in every batch, so two batches could hand out the same parent key.
Nothing else's seeded output changes, and no spec file needs migrating.
Alongside it, Polars 2's Map(key, value) generates, so every Polars
dtype that holds data now does.
Changed¶
-
A unique foreign key is a permutation of its parent's keys, so it is unique across the whole frame. Row
rtakes the parent key atπ(r), for a permutation keyed by the frame's seed -- the one behind every other unique column since 0.10.0 -- so a batch takes the keys the whole frame takes there. This covers a key into another spec and a unique key into the spec's own rows; a many-to-one key is still drawn with replacement, and a__unique_together__group is still distinct within a batch only. -
A parent too small for a unique foreign key is refused before the first batch.
generate_batches,scan()and the sinks check that the parent holds a distinct key for every one of thenrows up front. Before, each batch needed only a batch's worth, so a frame larger than its parent was produced -- with keys repeating across batches -- rather than refused.
Added¶
Mapcolumns generate, on Polars 2. AMap(K, V)is the list of{key, value}entries Polars stores it as, and polspec declares it as one:list_lengthis how many entries a map holds, andfieldssays what itskeyandvalueare. Every map is drawn with its keys present and distinct -- a key that repeats within its map is redrawn before the cast, which would otherwise fold the two entries into one. With nolist_length, a map holds up to five entries, or as many as its keys can take: two, forBooleankeys.
A nullable key, a declared list_length longer than the keys can fill,
and choices, weights or element_null_probability on the map itself
are refused at declaration.
- A
Mapvalidates, drifts, profiles and is sized as its entries. A finding about a key or a value is namedm.keyorm.value, the lengthm__list_len, and a map holding one key twice -- which a map read from Arrow can -- ism.key__unique. A map whose key or value is wider than declared stands in for it unlessstrict_dtypes=True.from_dataframere-declares a map by its key and value, and the generated docs list its entries.
Internal¶
- The keyed permutation is reachable from Python as
_ffi.permuted_indices(domain, seed, start, n): a window of it is the whole range's slice. - The dtype census covers
Mapwhere Polars has one, and its list of dtypes not generated yet is empty. - A docs example marked
<!-- docs: polars2 -->runs only where Polars 2's dtypes exist.
0.10.1 - 2026-09-25¶
Ready for Polars 2. polspec passes its whole suite on the Polars 2.0
release candidates, and CI now runs it there on every change; the bound
stays polars>=1.44.1,<2 until 2.0 itself is released and the same suite
is green against it. Along the way, testing against Polars 2 turned up two
bugs that Polars 1 users hit today, both fixed here. No seeded output
changes, and no spec file needs migrating.
Fixed¶
choiceson aDecimal, or aDatetimenot in microseconds. Drift failed on aDecimalcolumn withchoices; validation raised an error on aDatetime("ms")column with them, and on aDatetime("ns")column rejected the column's own generated data. The choices reached Polars'is_inas a Python list, which arrives at full precision or in microseconds. Validation, a rule's choices and drift now compare against the choices as a Series of the column's own dtype -- which is also what Polars 2 requires of an integer choice on a float column.
Added¶
-
scan()names itself inexplain()on Polars 2. A scan's node readsPYTHON[polspec: Orders] SCAN, with anINFO:line giving the rows, the seed and the batches. Whether to pass the labels is read offregister_io_source's signature, so on Polars 1 the call is unchanged and the header staysPYTHON SCAN. -
Polars 2's
Mapdtype has a spec-file form,{Map: {key: <dtype>, value: <dtype>}}, so a profiled map column can be saved --polspec schema inferon a file with one no longer fails. AMapcolumn declares, validates, profiles and drifts by its dtype;generate()refuses it by name until drawing its values arrives.
Internal¶
- A
polars-nextCI job runs the suite on the newest Polars 2 pre-release, informational until 2.0 is final.
0.10.0 - 2026-09-25¶
One seeded change, once: every unique=True column generates different
values for the same seed -- in generate() too, not only when batched --
because a unique column is now a permutation of its value space. That is
what makes it unique across a whole frame however it is produced: until
now, generate_batches, scan() and the sinks drew a unique column afresh
in every batch and the same way each time, so a key repeated every batch
(0.9.2 warned; this release fixes it). Nothing else's seeded output
changes, and no spec file needs migrating.
Around it: Orders.read(path) reads a file in the spec's terms,
report.passing_rows() and polspec validate --output / --failing split
a file into what passed and what did not, a list's elements can be null,
and the test suite is organised by what each file covers.
Changed¶
- A unique column is a permutation, so it is unique across the whole
frame. Its value at row
ris theπ(r)-th value of its space, whereπis a keyed permutation -- a Feistel network walking its cycle back into range -- so distinct rows get distinct values and a row's value depends only on the row and the seed. Every unique kind: integers, booleans, choices andEnum(an offset into the domain), floats (an evenly spaced grid over the bounds, four ulps apart at the widest magnitude so a cast cannot merge two points), dates,Decimal,Int128,Float16, and strings and formats, whose values decode from an index one digit per template part. A batch ofgenerate_batches, ascan()or a sink is now the whole frame's slice for a unique column as for any other, and the 0.9.2 warning is gone with what it warned about.__unique_together__and a foreign key drawn without replacement are still distinct only within a batch.
Two consequences to know. A unique string is drawn uniformly from every
string its lengths allow, so it favours its longest length -- 98% of a
string_length=(5, 15) column is fifteen characters. And a null row
still takes its place in the permutation, so a nullable unique column's
domain has to hold every row, nulls included.
- Much faster, and bounded in memory. The unique draw no longer holds a set of the values it has drawn, and fills chunk by chunk in parallel like any column: 10 million rows from a crowded integer domain take 0.016s where they took 1.42s, and a million unique strings 0.014s where they took 0.185s.
Added¶
-
Orders.read(path)(andpolspec.read(spec, path)): a data file in the spec's terms. The extension picks the reader -- the setpolspec validatereads -- Polars' options pass through (separator=";"), and each column the spec declares as aDate,DatetimeorTimethat arrived as text is parsed when every value parses, leaving date-shaped text in aStringcolumn alone. Nothing else is cast: the frame goes toinspect()orvalidate(cast=True). "Checking a file you were given" starts with it, and the CLI reads through the same code. -
report.passing_rows(): every row no finding touched, lazily and in order -- the complement offailing_rows(), so the two split a frame into what to keep and what to send back. A structural finding judges the whole frame, so it refuses until that is fixed. -
polspec validate --output PATH --failing PATH: the passing rows, validated again withcast=Trueso the file is one the spec accepts and typed as it declares, and the failing rows with the claim each broke. The exit status is unchanged, so a pipeline can quarantine the bad rows and still fail the step; notes go to stderr so--jsonstays one document. -
element_null_probability: how often an element inside a list is null. 0 by default -- elements are never null, as before -- and any rate up to 1 accepts them, on aListor anArray, a struct or list element null as a whole. Generation, validation, the spec file (an added optional key),diffanddrift(which now measures element nulls), the profiler, the data dictionary andestimated_sizeall read it.
Internal¶
- The test suite in folders, by what each file covers:
contracts/(the round trip, and every "each field has an entry" test in onetest_parity.py),declaration/,generation/,validation/,drift/,serialization/,features/(one file per feature across every verb),cli/anddocs/. The same 1135 tests collected before and after the move; a test now fails on a test file at the top level or without a docstring.
0.9.2 - 2026-09-24¶
The fixes from a review of 0.9.1, and the three numeric dtypes 0.9.0's
"every dtype generates" missed: Int128, UInt128 and Float16 now
generate, and a census test fails the day Polars adds one that does not.
The one to know about: a unique=True column repeats its values in every
batch of generate_batches, scan() and the sinks -- that has been true
since batching arrived, and the docs understated it. This release warns
when it happens and says so plainly; 0.10.0 makes the draw batch-stable.
No seeded output of an existing spec changes, and no spec file needs
migrating.
Added¶
Int128,UInt128andFloat16generate. They declared and validated before, butgenerate()failed with an engine error andto_yamlcould not write them. Each is drawn through a type the engine has and cast back, as aDecimalis: a 128-bit integer through 64 bits -- bounds past that are refused atgenerate(), naming the column, while validation checks the full range -- and a half as a single, rounded to the nearest half. The draw stays between the halves nearest each bound from inside, so rounding never carries a value past one; bounds too close together to hold any half are refused where they are written, and a unique half is drawn from the finite set of halves between its bounds. YAML,to_python,estimated_sizeandfrom_dataframeknow all three.
"Every dtype" now reads every dtype that holds data: Null, Object,
Unknown and extension types declare and validate by dtype, and
generate() refuses each by name. A census test walks every dtype class
Polars exports and fails on one that is neither.
Changed¶
- A unique column that repeats across batches says so. Under
generate_batches,scan()and everysink_*, aunique=Truecolumn's draw starts over in each batch the same way, so every batch holds the same values -- the docs called it "drawn per batch", which read as independent draws. polspec now warns once, as the second batch is drawn, naming the columns and pointing atgenerate()for a frame that must be unique throughout; a run that fits in one batch, or a scan that never collects the column, stays silent. The limitations and generating pages describe what happens.
Fixed¶
-
A finding on a zoned
Datetimeno longer panics on Windows. A finding's samples are Python values, and a zoned datetime needs the IANA database to become one; Windows ships none, so Polars panicked -- past anyexcept Exception-- on the first finding on such a column.tzdatais now a dependency on Windows. -
ColSpec(pl.Struct({}))generates aStruct({})column, where it generated an untyped null one. -
Docs:
comparison.mdno longer says polspec has no drift detection, andarchitecture.md's module table gains the seven modules it had fallen behind on, pinned by a test.
Internal¶
- Warnings are errors in the test suite, so a warning is asserted where it
is expected -- with
pytest.warns, or the doc runner's new<!-- docs: warns -->marker -- and a failure everywhere else.
0.9.1 - 2026-09-24¶
A patch release around one question: someone hands you a CSV, and there
is a spec it should meet -- what do you do? The CLI now reads a text
file's dates the way the spec declares them, where before every CSV with
a date column failed polspec validate; the docs walk the flow end to
end; and bounds can be set aside while you find out what the ranges
really are. No seeded output changes, and no spec file needs migrating.
Added¶
-
Checking a file you were given, a section of Validating data: read the file with
try_parse_dates=Truerather than the spec's schema (which stops at the first bad value),inspect()it for every problem at once, decide whether the file or the spec is wrong, thenvalidate(cast=True)for the typed frame. The example runs in the suite. -
polspec validate --skip CHECK, repeatable, for any of thevalidate_*switches ----skip bounds --skip checks. Its choices come from the switch list itself, so a switch added later reaches the CLI. -
polspec.__version__, the installed version. -
validate_bounds=Falseturns off the bounds checks, beside the othervalidate_*switches and asValidationOptions(bounds=False). The other switches cover what generation cannot yet satisfy; this one is for real data, whose ranges you may want to look at before holding it to them while every other claim is still checked. It covers everybounds, aList's elements and a struct's fields included;string_lengthandlist_lengthstay on.
Fixed¶
-
The CLI reads a text file's dates in the spec's terms. A CSV or JSON file has no date type, so a
Datecolumn arrived as text and validation reported its dtype and checked nothing else -- every CSV with a date column failedpolspec validate, including onepolspec generatehad just written.validate,driftand both--allmodes now parse each column the spec declares as aDate,DatetimeorTimethat arrived as text, keeping the parse only when every value parses: a column with a bad value stays text, so itsdtypefinding stays true, and aStringcolumn of date-shaped text is left alone.schema infer, with no spec to go by, reads a CSV withtry_parse_datesand declares a date column as aDate. -
estimated_sizeno longer charges a null row for content it does not hold. A nullable text orListcolumn was estimated as if every row held a value -- 23% high for a string column at a 30% null rate, 32-38% for a list. A string's bytes past its view and a list's elements now scale by the share of rows expected to be present; views, offsets, fixed-width values, anArray's slots and a struct's fields are paid on a null row too and are unchanged. -
The API reference for
FrameSpec.validate,diffanddriftrendered prose after their parameter lists as parameters namedA,thatandUniqueness. It is a Notes section now, and a test parses every docstring in the package as the reference does, failing on a warning.
0.9.0 - 2026-09-24¶
Two things are removed, as 0.8.0 announced: generate(lazy=True),
which is now a TypeError -- scan() is the lazy frame -- and the
polspec[arrow] extra, which no longer resolves. Both are under
Removed below. Everything else in the release is additive.
The headline is Struct, the last dtype polspec declared and could not
generate. ColSpec(fields={name: ColSpec(...)}) says what is claimed about
each field's values, so a field is described exactly as a column of its
dtype would be, and every part of polspec reads it: generation, validation,
the spec file, diff and drift, from_dataframe, the data dictionary
and estimated_size. The same recursion generates a List(Struct), a
List(List) and an Array(Struct), so every dtype now generates, nested
to any depth. No seeded output of an existing spec changes, and no spec
file needs migrating.
Added¶
- Every dtype generates. A
Structcolumn is generated from itsfields-- one column per field, gathered -- and the same recursion generates aList(Struct), aList(List), anArray(Struct)and a struct of either, nested as deeply as the dtype goes. There is no dtype left that polspec declares and cannot generate, andtest_roundtrip.pypins that claim rather than the list of exceptions it used to hold.
A field is drawn by the code that draws a column of its dtype, and is
seeded under its parent by name: renaming a struct column with
seed_name keeps every field, adding a field beside one moves nothing,
and a struct column is a window under generate_batches and scan()
like any other. The column's nullable is the cell -- a null struct
-- and a field's own nullable says whether it may be null inside a
struct that is present.
Validation checks each field's claims in place, and a finding names the
field: key point.lat__bounds, message Column 'point.lat', while its
columns stay ("point",) so report.rows() returns the offending
rows. Inside a list the samples are the offending lists, and a list of
lists tells its levels apart (c__element_null, c[]__element_null).
A struct in the data matches a declared one by field name rather than
order, each field compatible by the usual rules -- an Int32 field
stands in for a declared Int64 outside strict_dtypes.
ColSpec(fields=…): what is claimed about a struct's field values. AStructcolumn's dtype is its schema -- every field's name and type comes from it -- andfieldsis aColSpecper field saying what its values are, so a field is described exactly as a column of the same dtype would be:
ColSpec(
pl.Struct({"lat": pl.Float64, "lon": pl.Float64}),
fields={"lat": ColSpec(pl.Float64, bounds=(-90, 90))},
nullable=True,
)
It is partial: a struct of twenty fields where one needs bounds
spells one field. A name the dtype does not declare, or a field spec
whose dtype disagrees with the struct's, is refused where it is written,
as are unique, rules, validators, seed_name and col_name --
each a claim about a column among columns, which a value inside a
struct is not. A List of a Struct takes fields too, and a field
may itself be a struct, so a declaration nests as deeply as the dtype
does.
A Struct dtype now has a spec-file form ({Struct: {name: <dtype>}}),
so a struct column round-trips through YAML and to_python -- and so do
List(Struct) and List(List), which had no written form before. The
format version stays at 3: fields: is an added optional key.
- The rest of polspec reads a struct too.
diffanddriftcompare a struct's fields as they compare columns -- every comparator, keyed by path, so describingpoint.latwith bounds isdomain_narrowedonpoint.lat(breaking, as on a column) and data escaping them isbounds_exceededthere; a field's null rate is measured inside the structs that are present.from_dataframere-declares a struct column by its fields, aListof structs by its element's, and aListof lists by its outer length, where it used to record the dtype alone. The data dictionary gives each field a row (point.lat) under its column, andestimated_sizecosts a struct as the sum of its fields.
Removed¶
generate(lazy=True), deprecated in 0.8.0. It built the whole frame and called.lazy()on it, so it looked like the lazy verb while being the eager one.scan()is the frame that has not been built;.lazy()on a generated frame is a handle on one that has. Passinglazy=is now aTypeError.- The
polspec[arrow]extra, redundant since 0.8.0 made the sinks Polars' own. Nothing in polspec needs PyArrow, sopolspec[arrow]no longer resolves -- installpolspecand, if you want PyArrow for your own code, PyArrow.
0.8.0 - 2026-09-22¶
The headline is scan(): a LazyFrame that generates rows as they are
collected, so a plan that wants five rows of one column out of fifty
million generates five rows of one column. Projection is exact rather than
approximate, because 0.7.0 keyed every column and every pass by name.
Around it, a batch becomes a window onto one frame -- the same rows
whatever the batch size -- the sinks become scan() written out and stop
needing PyArrow, estimated_size() says how large a frame will be before
it is allocated, and generate(lazy=True) is deprecated in favour of the
verb that means it. Every seeded batched or sunk output changes, once;
generate() is unchanged, and no spec file needs migrating.
Added¶
-
estimated_size(n), and a word before a large frame is allocated.Orders.estimated_size(50_000_000)reads the answer off the declaration -- the width of each dtype, the lengths the spec declares -- so it costs nothing and needs no data.generate()now estimates before it allocates: past four gibibytes it warns, naming the figure and pointing atscan()andgenerate_batches();max_bytes=makes it a refusal instead, for a CI job that should fail rather than swap, andmax_bytes=0silences both.polspec generateprints the same note. The estimate is the frame, not the process peak -- generation holds working buffers on top, most visibly forDecimalandList-- and for a frame of scalar columns it is within a percent of measured. -
scan(): aLazyFramethat generates rows as they are collected.Orders.scan(50_000_000, seed=1)builds nothing; the plan decides what is drawn..sink_parquet(...)streams in bounded memory,.select("total").head(5).collect()generates five rows of one column, and a predicate filters rows that were drawn rather than narrowing the draw. Projection is exact: every column is seeded by its name and every pass by what it is for, solf.select(cols).collect()is alwayslf.collect().select(cols)-- where a column depends on others (a rule'swhencolumns, a composite key's members) the closure is generated and dropped again on the way out. A scan is batched, so it carriesgenerate_batches' terms: a__hierarchy__is refused, and uniqueness holds within a batch.Registry.scan_all()does the same for a set of specs, generating parents eagerly -- a foreign key needs the whole parent column -- and the children lazily. Built on polars'register_io_source, which polars marks unstable; the tests pin behaviour rather than the API surface. -
polspec drift --all specs/ data/, completing the--alltrio: every spec under a directory measured against the data file named after it, specs without one listed and skipped,--fail-ondeciding the exit status across all of them, and--jsonprinting one report per spec.diffcompares two declarations rather than a declaration and data, so it takes no--all.
Changed¶
- The four
sink_*functions arescan()written out. Each is nowscan(...)handed to the matchingLazyFrame.sink_*, so PyArrow is no longer needed for the Parquet and IPC sinks -- nothing beyond Polars is needed at runtime, and thearrowextra is redundant. Signatures, defaults and then=0behaviour are unchanged; what a sink's**kwargsreach is now polars' own sink rather thanpyarrow.parquet.ParquetWriterorpyarrow.ipc.new_file, so a call passing a PyArrow-only writer option needs the polars spelling instead.compression=is typed as the literal polars accepts rather thanstr, so a misspelling is caught where it is written. Thepolspec[arrow]extra still resolves and is removed in 0.9. generate(lazy=True)is deprecated, and removed in 0.9. It builds the whole frame and calls.lazy()on it, so the memory is already spent: usescan()for a frame that generates as it is collected, or.lazy()on the result for a handle on an eager one.- A batch is a window onto one frame.
generate_batchesand thesink_*functions used to seed batch k from the k-th draw of the caller's seed, so the same seed with a differentbatch_sizewas a different frame and no batch could be made without the ones before it. The engine now takes a row offset and numbers its chunks from it, so for a column no pass rewritespl.concat(generate_batches(n, batch_size=b, seed=s))equalsgenerate(n, seed=s)for everyb; aListcolumn's lengths are a window too. Rules, foreign keys, composite keys and List elements are still drawn per batch -- deterministic, keyed by the batch's offset, but not row for row the whole frame's -- and uniqueness still holds within a batch. Every seeded batched or sunk output changes, once;generate()is unchanged. A batch smaller than the engine's 65,536-row chunk costs up to one chunk of extra draws.
Internal¶
- A link to a heading that does not exist fails the test suite, not
just
zensical build --strict.tests/test_docs.pychecked that a linked page existed but never its anchor, so a wrong slug reached CI before anyone saw it. polspec.validation.constraintsis a package, one module per kind of claim:_values(what one value must be, lifted over a List's elements),_rules,_table(composite keys, checks) and_relations(foreign keys, the hierarchy), with the_Constraintbase and the dtype rule in_base. No import outside the package changed.- A generated column is dropped as it is finished, so a temporal, Binary or gathered column never exists twice while the frame is assembled.
0.7.0 - 2026-09-22¶
The headline is nested and decimal dtypes: List, Array and Decimal
columns generate, validate and round-trip, described by the same fields a
scalar column takes. Around it, one change to what a seed produces -- the
passes that run after the columns are filled are now keyed by name, so
inserting a column never changes its neighbours -- and, alongside it, the
type checker moves to ty with nothing suppressed, the CLI gains
--all, and four long-standing limitations close, the last strict xfail
among them. Specs with rules, foreign keys, a hierarchy or a composite key
produce different values for the same seed than 0.6 did, once; see
Changed. No spec file needs migrating.
Added¶
pl.Listandpl.Arraycolumns generate, validate, and round-trip. A nested column is described by the same fields as a scalar one, read as claims about each element:ColSpec(pl.List(pl.Int64), bounds=(0, 10), list_length=(1, 5)).bounds,choices,weights,format,pattern,string_lengthanddistributiondescribe the values inside the list; the newlist_lengthdescribes the list (anArraytakes its length from the dtype);nullabledescribes the cell, and an element is never null. Generation draws the lengths and the elements as two columns of the inner dtype and wraps one by the other, so an element is made by the same code that makes a scalar of its dtype -- every generatable dtype,Enum,Decimaland aformatincluded -- and aListcolumn keeps its data across a rename throughseed_name. Validation runs every element claim inside the list and fails a list where any element does;list_lengthis a new finding code and a null element anullabilityone.diffcompareslist_lengthlikestring_length;driftmeasures a List column as its elements and its lengths;from_dataframedeclares one by its elements;{List: <dtype>}and{Array: {inner: <dtype>, width: N}}are the spec-file forms.uniqueandrulesare refused on a nested column, and aListof aListorStructdeclares and validates by dtype only. What generation cannot fill is nowStructand those.pl.Decimalcolumns generate. ADecimal(precision, scale)is an integer and a scale, so it is drawn as the integer through the engine's 64-bit kind and scaled back to the declared type -- withbounds, adistribution,choices, nullability and cartesian coverage like any other numeric column, and a default range of the float default or the widest the precision allows. Bounds are held exactly (int,decimal.Decimal, or a string read exactly, which is how a spec file writes one); an endpoint finer than the scale is refused rather than rounded. Validation compares values whatever the frame holds them as (a CSV hands a Decimal back as a float), drift measures the extent in the column's own type,from_dataframedeclares one with its extent, and{Decimal: {precision: P, scale: S}}is its spec-file form. The ungeneratable dtypes are now the three nested ones.ColSpec(seed_name=...): rename a column without changing the data it generates. Each column is seeded from the frame seed and its name, so a rename had always changed a column's values -- which matters when generated frames are snapshots other things are compared against. A column declared withseed_name="old"is seeded as"old"and keeps producing what it did. Declared, not guessed:rename()does not set it, and two columns of one spec cannot share one (they would draw identical values). It holds across a rename and nothing else: the passes that run after the columns are filled draw their seeds in declaration order, so an inserted rules column still reshuffles the ones after it, and that boundary is pinned and stated in Known limitations.
Documentation¶
-
Generating data said each column derives its seed "from its position". It is from its name -- the reason
seed_nameis needed at all. -
polspec generate --allandpolspec validate --all. With--all,SPECis a directory of specs:generatewrites one<name>.<format>per spec into-o DIR, parents first with their keys threaded into their children, asRegistry.generate_alldoes;validatechecks each spec againstDATA/<name>.<suffix>, every spec seeing the others' files as parents, asRegistry.inspect_alldoes, exiting 1 when any fails and printing one report per spec with--json.
Changed¶
- A rule leaves a null a null. A nullable column with rules used to
lose nulls on the rows a rule matched, ending up below its declared
null_probability. The column's nullability is decided when it is drawn; a rule says what a value on a matched row is, which is also all validation checks it against. to_mermaidmarks one primary key. A loneunique=Truecolumn is the entity'sPK; when several columns are unique each is aUK, as a__unique_together__member already was.- A
CatSpecrefuses two entries differing only in case (STATUSandstatus), naming both: lookup is case-insensitive, so they were one name with two answers. - Pass seeds are keyed by name, not drawn in order. The passes that
run after the columns are filled -- rules, the hierarchy, foreign keys,
composite uniqueness, a bounded categorical's pool, cartesian coverage
representatives -- used to take their seeds from one
random.Randomin declaration order, so inserting a rules column shifted every later pass andseed_namecould not hold a ruled column's data. Each now mixes the frame seed with a key naming what it is for, by the same construction the engine uses for columns. A spec with any of those produces different values for the same seed than 0.6 did, once; a spec with plain columns is unchanged, and so is every column's own draw. Thelimitations.mdbullet that described the old behaviour is gone.
Fixed¶
generate(0, method="cartesian")returns the empty typed framegenerate(0)and the sinks do, rather than the whole coverage set. The last strictxfailintests/test_roundtrip.pyis gone with it.
Internal¶
tyreplaces mypy, and nothing is suppressed. The 23-module ignore list was one pattern:ColSpec,CheckandForeignKeyannotated their fields with what the constructor accepts (pl.Int64orpl.Int64(), a tuple or aBound, one validator or several) while__post_init__narrows every one to what the instance holds. The fields now say what they hold, and each class declares the accepted signature in an__init__that exists only for type checkers, held to the fields by a test. Everyspec.dtype.is_integer()in the library type-checks; thecast()accessors drift needed are gone; the API reference andllms-full.txtshow the accepted signature. CI runsuv run ty check.polspec.cliis a package, one module per verb, with the readers, writers and spec loaders every verb shares in_io.validate()/inspect()anddiff()/drift()shared 25 lines of options handling twice; it is onepolspec._options.options_from.Bound.closed(): both endpoints of a bound that has both.
0.6.0 - 2026-09-14¶
The headline is drift: a report of what changed between two specs, or
between a spec and the data it describes, with a severity per finding under
one mechanical rule -- and three CLI verbs so that report can gate a pull
request or a nightly load with no Python. Alongside it, pattern= completes
the split format= began. Nothing in the engine changes; no seed produces a
different frame, and no spec file needs migrating.
Added¶
- Drift as a report.
polspec.drift.diff(old, new)says what changed between two declarations;polspec.drift.drift(spec, df)says how data has moved relative to what its spec declares -- values outside the domain, a bound exceeded and by how much, a format no longer matched, a null rate that moved, declared values never seen. Both return oneDriftReport, also reachable asOrders.diff(Other)andOrders.drift(df):
report = OrdersV1.diff(OrdersV2)
report.breaking # a narrowed bound, an added column, a new constraint
report.compatible # a widened domain, a removed check
report.to_markdown() # the shape of a pull-request comment
Every finding carries a severity under one mechanical rule: breaking
when a frame that satisfied the old side could fail the new one --
decided, for dtypes, by the same function validation uses. Two tests hold
the two sides to it: what generate() produces never drifts breakingly
from its own spec, and every breaking finding against data is a column
validate() reports. Sixteen closed finding codes, a DriftOptions
with explicit tolerances, and a comparator per ColSpec and TableSpec
field with a parity test, so a field added to a declaration is one entry
or one red test. DriftReport, DriftFinding and DriftOptions are
exported from polspec.
-
ColSpec(pattern=...): a regular expression every value of aStringcolumn must match, checked by validation only. Generation does not read it -- the column is filled with ordinary random text, and the round trip holds only withvalidate_pattern=False, exactly as forvalidators. That is the honest half of the splitformat=made: any regex can be checked, the curated set can be generated. Apatternfinding code, avalidate_patternswitch onValidationOptions, onepattern:key in spec files (the format version stays at 3: an added optional key is not a new version, andmigrations.pynow says so). Cannot be combined withformat; compiled by Polars' own engine at declaration, so a pattern Polars cannot run is refused with its message. -
Three CLI verbs.
polspec generate SPEC -n N -o FILEwrites generated rows to any format the CLI reads (--seed,--method,--referencesas forvalidate);polspec diff OLD NEWandpolspec drift SPEC DATAprint a drift report as text,--jsonor--markdown, and gate on it with--fail-on breaking|any|none-- so a schema change in a pull request, or a nightly load, can fail CI with no Python:
polspec generate orders.yaml -n 1000 -o orders.parquet --seed 1
polspec drift orders.yaml orders.parquet
polspec diff main/orders.yaml pr/orders.yaml --markdown --fail-on breaking
Documentation¶
- Five claims the docs had stopped being true about are fixed, and one of
them is pinned:
unique=Trueand__unique_together__had been listed as work generation does not attempt (generated since 0.2.0);generate()on an unsupported dtype raisesSpecError, notTypeError; every yaml example carriedversion: 2at format version 3, and a test now holds eachversion:in the docs toFORMAT_VERSION. - The roadmap
gains a Deferred on purpose section, starting with the
ColSpec->Domainrestructure that two release plans had set aside without saying so on the page, and the condition under which it would be revisited.
0.5.0 - 2026-09-11¶
The headline is format=: a String column that says what its values look
like, and is generated to satisfy its own validator. Around it, the release
closes five silent edges, settles the validation option surface, spells out
the FrameSpec signatures, and puts a type checker in CI. One narrow
breaking change, under Changed.
Added¶
ColSpec(format=...): aStringcolumn that says what its values look like, and is generated to satisfy its own validator. Eight formats --uuid4,email,ipv4,ipv6,mac,hostname,iso_country,iso_currency-- each with a sampler in the engine and a check in validation declared side by side inpolspec.formats, and pinned by a round trip per format:
class Users(FrameSpec):
user_id = ColSpec(pl.String, format="uuid4", unique=True)
email = ColSpec(pl.String, format="email")
Users.validate(Users.generate(1_000_000, seed=42)) # passes
Validation reports a new format finding. A format is one format: key in
a spec file and takes part in the domain check a foreign key runs at
declaration. It cannot be combined with choices or string_length, and
only a String column can carry one; each is refused with a message
saying which to drop. What a format promises is syntax: see Known
limitations.
ValidationOptionsis exported frompolspec, andvalidate()andinspect()take it asoptions=. Every switch as one value, for when the same settings go through several calls:
lenient = ValidationOptions(extra_cols="drop", checks=False)
Orders.validate(df, options=lenient)
Customers.validate(other_df, options=lenient)
options= and the individual keywords are alternatives rather than a base
and an override, so passing both raises instead of quietly picking one.
Changed¶
-
FrameSpec.validate,inspectand the foursink_*classmethods spell their options out instead of forwarding**kwargs. Editors complete them, a type checker sees a typo, and a mistyped option is aTypeErrorfrom the classmethod's own signature rather than from a function several frames away. The validation keywords default toNone, meaning "theValidationOptionsdefault", so the defaults are still defined once. The sinks keep their trailing**kwargs: that is the documented passthrough to the underlying writer, not a gap. -
A misspelled spec name in
references={...}now warns instead of passing silently. A key nothing points at was skipped without a word, so the column was generated freely while the caller believed the parent had been used -- andvalidate()then reported the key as unresolved, which is the round trip failing with only its second half audible. The warning names the key that went unfilled and suggests the one supplied:
Orders: references={...} supplied ['Custmers'] that no foreign key points
at, while 'Customers' (supplied 'Custmers'?) went unfilled.
It takes both halves to warn -- something supplied that went unused and
something unused that went unfilled -- so supplying no parent at all stays
silent, as documented, and a Registry handing every spec the whole set of
frames says nothing either. generate_batches warns once per call rather
than once per batch.
- ColSpec(pl.Int64, null_probability=0.9) now warns. nullable=False still
wins and the rate is still ignored -- turning nullability off should not
also require deleting the rate beside it -- but the same silence covered
asking for nulls and forgetting nullable=True, where the column generates
none and nothing says why. Only a rate that cannot be a leftover warns: the
default and an explicit 0.0 already agree with nullable=False.
- references= given something that is not a mapping raises SpecError
naming the three key forms it accepts, rather than an AttributeError from
inside resolve_references. Every collection argument elsewhere in the API
is a sequence, so passing one here was an easy mistake with an unhelpful
answer, and it was the one complaint polspec made that was not a
PolspecError.
- An unknown validation option names the option you meant. It used to surface
as ValidationOptions.__init__() got an unexpected keyword argument
'validate_uniqe', naming a private class that is not exported and not the
option intended; it is now Unknown validation option(s): 'validate_uniqe'
(did you mean 'validate_unique'?), with the accepted list.
- Breaking, narrowly: the check switches have one spelling.
inspect(spec, df, unique=False)used to work alongsidevalidate_unique=False, whilevalidate(spec, df, unique=False)raised -- a second public spelling reachable through half the API, from a rename map that was meant to be internal. Only thevalidate_*form is accepted now, by both verbs. The bare names live on as the fields ofValidationOptions, which is where they were always meant to be.
Documentation¶
DecimaljoinsList,StructandArrayon the list of dtypes that declare and validate but cannot be generated. It had been missing from both Dtype coverage and Known limitations, and it is the one people miss, being the only one of the four that is not a nested type.- A
Datetimecarrying atime_zonegenerates, which the docs had never said either way and readers assumed meant no. Both claims are now pinned by tests intests/test_roundtrip.py, so neither page can go stale.
Internal¶
- The four copies of "declares no ColSpec columns" are one
tablespec.require_columns. - mypy runs in CI. The package ships
py.typedand a stub for the Rust extension, so every consumer's type checker trusts these signatures, and nothing was checking them. The first run found 247 errors, of which 87 trace to two declarations:ColSpec.dtypeandForeignKey.referencesare both annotated with what the constructor accepts rather than what the instance ends up holding. Narrowing either is a design decision on a public field, so the modules carrying that backlog are listed inpyproject.tomland everything else is enforced -- the list can only shrink. - The predicate nodes in
expr.pyno longer each carry their ownroot_names,literalsandrename. A node reports its operands throughchildren()and the three traversals are derived from that onPred, which turns thirty-six implementations into twelve.renameis the one that mattered: the base implementation returnedself, so a node that forgot to override it left a renamed spec pointing at a column that no longer existed, with nothing raised. Forgettingchildren()now raises. ReferencesandMethodwere declared identically in four and two modules;_collect/_to_lazyin three. They are onepolspec.frames.- The four
sink_*functions built the same six-argument batch-stream call each._preparenow returns the checked call as one value. The public signatures stay spelled out, which is what makes a typo in one of them fail at the call site. registry.pyimportedserializationlazily in five methods andreportin a sixth, while importinggenerationandvalidationat module level. There was no cycle; all six are hoisted.
0.4.1 - 2026-09-09¶
The hierarchy release. A self-referencing ForeignKey says that every parent
value exists somewhere in the frame, and nothing more. This adds the
declaration that says the rest of it -- one parent per reference, a bounded
depth, no cycles -- and makes generate() satisfy it rather than leaving it to
the draw.
Added¶
Hierarchy, declaring that two columns of a spec are a parent/child edge list drawn on one pool of references: a child pointing at its parent, and that parent pointing at its own parent, are the same row shape.
class Links(FrameSpec):
PARENT_REF = ColSpec(pl.String)
CHILD_REF = ColSpec(pl.String)
__hierarchy__ = Hierarchy(
child="CHILD_REF", parent="PARENT_REF", max_depth=5
)
generate() produces a real forest: one parent per reference, so every row
resolves to a single ultimate parent, and no chain longer than max_depth
with at least one reaching it exactly -- so a test of a graph walk exercises
the boundary rather than whatever the draw happened to give. branching
shapes the tree, or roots pins the number of ultimate parents.
generate(n, cycles=10, self_references=5) then breaks it on purpose, which
is the other half of testing a graph walk: a resolver written without a
visited set does not fail on a loop, it runs forever. The spec still says the
data should be acyclic, so validate() reports what was injected --
hierarchy_cycle, hierarchy_depth and hierarchy_multi_parent are new
finding codes -- and a test can assert that its own resolver and polspec
agree about what is broken.
Both checks are bounded, so validating deliberately cyclic data terminates:
depth costs max_depth steps and cycle detection walks by pointer doubling,
covering a million-row chain in about twenty. A validator that walked until
it reached a root would hang on the fixtures this feature exists to make.
generate_batches and the sink_* functions refuse a spec carrying one: a
forest spans the whole frame, and batches are generated independently. See
Hierarchies and link tables.
Documentation¶
- The spec file format is version 3, which adds the
hierarchy:key. A version 2 file loads unchanged. - Known limitations
now points a self-referencing
ForeignKeyatHierarchyfor the case that wants a real tree, rather than carrying a recipe of its own. Two tests pin the two apart.
0.4.0 - 2026-09-08¶
The internals release. 0.2.0 and 0.3.0 settled the vocabulary; this one goes underneath it, to the Rust generator and the places where the same table was being maintained in two or three languages.
Nothing about how a spec is written changes. One thing does break, and it is the same thing the roadmap has always reserved: the values a given seed produces are different. Any test asserting on specific generated values needs re-baselining; a test asserting on their properties -- bounds, distinctness, null share, distribution shape -- does not. polspec's own suite needed no changes, which is the shape of test this library is built to support.
Generation got faster, by between a tenth and a third depending on the column.
Measured A/B against v0.3.0 on one machine, same build profile, twenty million
rows: the four-column frame in benchmarks/bench.py 1.30x, a
unique=True Int64 column 1.21x, a bounded nullable Int64 column 1.17x, an
Enum column 1.25x, a String column unchanged. Treat the ratios rather than the
absolute numbers as the claim.
Added¶
generate,generate_batches,inspect,validate,sink_parquet,sink_ipc,sink_csvandsink_ndjsonare exported frompolspecitself. Each takes aTableSpecas its first argument and each is what the matchingFrameSpecclassmethod already called -- but they lived inpolspec.generation/polspec.validation, which the API reference calls internal and free to change in a patch release. So theTableSpec-first half of the library had no stable import path; now it does, and both halves appear in the API reference.
Changed¶
- Breaking: the values a given seed produces have changed. polspec now
builds on
rand0.10 (from 0.8), whose samplers draw differently. Same seed, same version, same frame -- as before; across this version boundary, not. - A generated numeric, boolean, temporal or categorical column arrives as
one chunk rather than one per 65,536 rows. The values buffer and the
validity bitmap are each allocated once at full length and filled in parallel
through disjoint slices, instead of being built per chunk and appended
together. Nothing downstream now pays for a column split into hundreds of
pieces -- the gather behind a
choicesdomain, the cast behind a temporal dtype, everysink_*write.
String columns are the exception and stay chunked, because Polars backs them
with view arrays: merging those copies no string bytes, but it does copy
sixteen bytes of view per row, which costs more than the split it removes.
They still gain the other half of the change -- the chunks are collected in
one go rather than appended one at a time, and an append rescanned both sides
for their first and last non-null value to maintain a sorted flag that random
strings will not have set anyway.
- Drawing a unique=True column no longer materialises its domain. A domain
only a little wider than the row count used to be built in full and partially
shuffled, which allocates in proportion to the domain rather than to the
output: ten million distinct values from a range of eighty million reserved
1.4 GB before writing anything. That branch is now Floyd's algorithm, which
holds only the values it has chosen -- the same case now peaks at 491 MB.
Roomier domains keep drawing and rejecting, which is faster there and was
never the memory problem.
- Every character of the generated-string alphabet is now exactly equally
likely. Six random bits give 64 values for a 62-character alphabet, and the
two spare ones fell back on % 62 over a fresh 32-bit draw, which is biased
by about one part in 70 million -- far too little to see, but free to remove:
the fallback now rejects properly instead.
- rand 0.8 was compiled alongside the rand 0.10 that Polars already links,
so the extension carried two copies of rand, rand_core and their chacha
backends. There is now one of each.
- The release profile builds the crate as a single codegen unit. Measured on
one machine against otherwise identical v0.3.0 code, that alone is worth
2.1x on the unique=True path, for about twenty seconds of build time. Fat
LTO on top of it was tried and dropped: a further 5% for eight more minutes
per build.
Documentation¶
- A self-referencing
ForeignKeyguarantees that every parent value exists, and nothing more -- in particular not that the result is a tree. Parents are sampled from the whole frame, so some rows end up in a cycle or pointing at themselves, which is what makes a recursive query fail to terminate, andvalidate()does not report it because no part of a spec can say "acyclic". Known limitations now says so, with a recipe for a genuine hierarchy, and the roadmap carries what closing the gap would need. Two tests pin the behaviour.
Fixed¶
Registry.validate_allapplies each report's structural transformations using the same bound spec the report was produced against, rather than the unbound copy.
Internal¶
ColumnPlan::buildtakes aPlanArgsstruct instead of thirteen positional arguments, so a call site names what it sets and leaves the rest toDefault. Four#[allow(clippy::too_many_arguments)]and a great manyNones went with it.Kind's three parallel lists -- the names, the parse, the reverse lookup -- are generated from one declaration, so a new column kind cannot be added to two of them and forgotten in the third.- The fixed-width integer ranges were written three times: once as polspec's default generation range, once as the limits a declared bound may not exceed, and once as the Rust samplers' defaults. The first is now read from the second.
- The four
sink_*functions share a typed batch-stream helper rather than forwarding**kwargs. benchmarks/bench_generate.pyis replaced bybenchmarks/bench.py, which measures the same comparison and adds a regression mode. The old harness timed one run per case, in a process shared with the implementations it was comparing against, and recorded nothing about the machine -- so its numbers varied by around 25% between runs and could not be compared across days. It also measured exactly one column shape, which is how a change that made theuniquepath half as fast came within an afternoon of being released as a speed-up. The new one takes the fastest of several runs, gives every measurement its own process, repeats a short case until its floor settles, records the CPU, thread count, Polars version and cargo profile, and covers each column kind, both branches of the unique draw, the cartesian, rule, foreign-key and composite-key passes, and a sink.recordwrites a local baseline andcheckexits non-zero when a case regresses past a tolerance; repeated measurements now agree to within about 2%.- The benchmark table in the comparison guide is re-measured. It had been recorded on 2026-09-03, which is before both 0.2.0 and 0.3.0, so it had been describing an engine two releases old: the four-column frame at twenty million rows was published as 0.0827s, measures 0.1409s on v0.3.0, and 0.1127s here. Some of that gap is still unaccounted for and is worth chasing. The NumPy and pure-Python columns re-measure to within 1% of what was published, which is what says the difference is polspec's and not the machine's.
0.3.0 - 2026-09-06¶
The refactor 0.2.0 started, finished. CatSpec was the one declarative
surface left doing everything in one class; it is now a value with a
metaclass facade, like TableSpec and FrameSpec. Behind it came the fixes
that were waiting for a release allowed to break something.
One thing breaks, and it is worth reading before you upgrade: naming a
CatSpec entry now always gives back the dtype, whichever form declared the
registry. pl.Enum(cats.STATUS) becomes cats.STATUS, and
cats.CURRENCY.physical() becomes
cats.get_categorical("CURRENCY").physical(). Nothing else in the public API
changed shape.
The documentation gained a test: every Python example in docs/ is executed
by the suite, which found five broken examples that had been shipping.
Changed¶
-
Breaking.
CatSpecis a value, and the class body that declares one is read by a metaclass rather than left in the namespace -- the same splitTableSpecandFrameSpecalready had. What follows: -
Naming an entry always gives back the dtype. It used to depend on how the registry was built: a class-body entry was a real class attribute and returned the dtype, while a dict-built registry's
.STATUSreturned the raw category list, and only one of the two could be handed to aColSpec. Both now return the dtype, soColSpec(cats.STATUS)andColSpec(Categories.STATUS)mean the same thing.cats["STATUS"]andcats.get("STATUS")follow the same rule. Replacepl.Enum(cats.STATUS)withcats.STATUS, andcats.CURRENCY.physical()withcats.get_categorical("CURRENCY").physical(). - An entry may share a name with a method. Entries are removed from the
class body before the class exists, so declaring one called
getno longer warns and no longer costs youCatSpec.get. The entry is reached through the registry (Categories.spec.get("get")). CatSpec.infer_from_dataframeandCatSpec.infer_from_framespecare removed;CatSpec.infer(target, ...)dispatches on what it is given, as it already did.from_dataframeandfrom_framespecare unchanged -- those read what is declared rather than inferring what could be.Categories.specis theCatSpeca class body declares. Anywhere a registry is expected --with_catspec,Registry(categories=...),FrameSpec.from_yaml(categories=...)-- the class and the value are now interchangeable.CatSpechas value semantics. Two registries that say the same thing compare equal and hash equal, so one loaded from a file can be checked against one a class body declares.CatSpec.dtype_of(name)is the one lookup everything else is built on: the dtype an entry names, or None.resolve_keystill says which kind of entry a name binds to.enums,categoricalsandchoicesare read-only mappings rather than fresh dicts.dict(cats.enums)if you need a mutable copy.
Added¶
polspec.MultiValidationError, raised byRegistry.validate_allwhen several frames fail at once. It is aValidationError, so an existingexceptclause still catches it, and it carries every failing spec'sValidationReportasreports, keyed by spec name -- previouslyvalidate_allraised with a joined string and the reports were lost, sofailing_rows(),by_code()andto_json()were unreachable from the registry path.polspec.CliErroris exported, soexcept polspec.CliErrorworks. It was the one exception in the hierarchy reachable only frompolspec.errors.- Every Python example in the documentation is executed by
tests/test_doc_examples.py. A block that cannot run standalone says so in an HTML comment (<!-- docs: skip -->), and one that demonstrates an error is checked to still raise (<!-- docs: raises -->). This found four broken examples, fixed here: adrop()of a column the page never declared, aTableSpecexample rebinding the name a later block used, and two blocks naming frames (broken_df,existing_df) that were never built.
Fixed¶
ColSpec(tags={...})is reproducible. Asetwas kept in its own iteration order, which Python salts per process, soto_yamlwrote a differenttags:line on every run and two identically-written specs compared unequal across processes. A set is now sorted; a list or tuple keeps the order it was written in.Registry.generate_all,generate_relatedandinspect_allbind their cross-spec foreign keys before doing anything, so a key whose dtypes do not match is aRegistryErrornaming both columns rather than a Polars cast error from inside generation. Onlyresolve()used to run that check, and nothing said it had to be called first. A key whose target is supplied throughreferences=rather than held by the registry is still accepted, as it was.- A
ColSpeccarrying the same validator twice keeps it once, so it produces one finding rather than two identical ones.TableSpecalready collapsed identical checks and foreign keys. generate_batchesand thesink_*functions resolvereferencesonce per call rather than once per batch. ALazyFrameparent was collected inside every batch, so a scan-backed parent was re-read as many times as there were batches.inspect()no longer raises a raw Polars error for a column whose dtype is wrong and whose spec declareschoicesor anEnum. The domain check was built before the dtype check could bail out, and comparing values against choices of another type is not something Polars will compile at all, so the frame most likely to arrive -- a column read back from CSV or JSON as the wrong type -- crashed instead of reporting adtypefinding.- A
ColRulewhose condition is null on a row no longer excuses every later rule on that row. Generation folds a nullwhentoFalsebefore testing it and before accumulating it into the claimed mask; validation did neither, so the null propagated through Kleene logic and left rows that generation had rewritten unchecked. TableSpecis hashable, soreferences={Orders.spec: df}works. It is one of the three formsgenerate()andvalidate()document, and the only one that could not be put in a dict: the dataclass's generated__hash__cannot hash a mapping of columns, nor aColSpeccarryingdistribution_params.
Documentation¶
- The install sections of the README and the documentation home said polspec
was not published to PyPI, directly below a
pip install polspecblock. Both now describe the published wheels, and point atCONTRIBUTING.mdfor building from a checkout. - The documentation workflow runs on changes to
python/**andscripts/generate_llms_txt.py. The API reference is:::directives filled in by mkdocstrings from the live docstrings, so a docstring that breaks--strictused to pass its own pull request and fail the next one to touchdocs/. - The roadmap's "YAML format may change" section described the missing format
version key that 0.2.0 shipped, and said an unsupported dtype raises
TypeErrorrather thanSpecError. FINDING_COLUMNandValidationOptionsare documented in the validation guide; both are exported and appeared nowhere.how-to/tablespec.mdtaughtpolspec.generation.generate(spec, ...)while the API reference says anything unlisted may change in a patch. The page now says which of the two it is.CONTRIBUTING.mdgives the runnable form of the Windowscargo testworkaround, and names theSTATUS_DLL_NOT_FOUNDfailure it fixes.
0.2.0 - 2026-09-05¶
The architecture release. Specs became data, constraints gained one definition each, and the generator learned to satisfy claims it used to only check.
This release breaks a lot. Every incompatible change below is marked
Breaking and says what to do instead. Three are worth knowing before you
upgrade: Spec._columns and friends are now Spec.spec.columns; the values a
given seed produces have changed, so any test asserting on generated values
needs re-baselining; and several declarations that used to be accepted and
quietly misbehave are now refused at declaration time.
Added¶
docs/llms.txtanddocs/llms-full.txt, published at the documentation site root in the llms.txt format: an index of every page, and the full text of all of them in one file. Generated byscripts/generate_llms_txt.pyfrom the nav, the pages, and -- for the API reference, whose source is:::directives -- the live docstrings, so a language model reads signatures rather than an empty page. A test fails if either file is stale.polspec.constraints: the definitions generation and validation both read, so they cannot drift.Domainis what a column may hold (itschoices, anEnum's categories, itsbounds);Passandorderdecide which rewrite of a generated frame runs first, from the columns each one reads and writes.unique=Trueis generated, not just validated. The engine draws the column without replacement (src/unique.rs): it shuffles a materialised domain when the domain is barely larger than the frame, and rejects against a set when it is roomy. Every dtype is covered, nulls are exempt (a nullable unique column may repeat nulls and nothing else), and a domain too small to cover the row count is refused by name instead of quietly producing duplicates.method="cartesian"holds unique columns out of the coverage product and draws them once over the finished frame.-
__unique_together__is generated. A pass resamples the rows repeating a combination an earlier row already used, so only the repeats move and the rest keep the values their own columns' weights and bounds gave them. Rows with a null member are exempt, matching validation. A group whose columns cannot take enough distinct combinations is refused, naming the group; a foreign-keyed member is never resampled, since that would break its key. -
The Rust boundary is typed. Python builds one
ColumnPlanper column (a#[pyclass]validated at construction, with errors naming the column) instead of a positional tuple. Bounds cross as ani64,u64orf64, soInt64/UInt64bounds beyond 2^53 are exact and the generation clamp for an unbounded distribution reaches the dtype's true limits. Columns with a finite domain (choices,Enum) receive indices back and the typed values are gathered on the Python side, so adatetime,bytesorTruechoice never passes through a string; choices need only be distinct in the column's dtype, not as strings.python/polspec/_polspec.pyiis a stub for the extension;src/is split intoplan.rs,dist.rsandsample.rswith unit tests undercargo test; a test compares the distribution parameter tables on both sides. import polspecworks without the Rust extension: validation, spec files, the registry and the report renderers need no build. Only generation imports it, and raises one actionableImportErrorwhen it is missing.Registry: a declared set of specs.Registry(Customers, Orders, ...)resolves foreign keys declared against names (resolve(), running the checks a class-bound key gets at declaration), orders parents before children (order()), generates the whole set with every key satisfied (generate_all, with a per-spec seed so adding a table changes no other;generate_relatedfor one spec and its ancestors), validates it in one call (inspect_all,validate_all), merges or checks shared categories (catspec(),categories=), writes and reads one file for the set (to_yaml/from_yaml, aspecs:mapping pluscategories:), collects specs from modules and directories (from_module,discover), and draws one entity-relationship diagram (to_mermaid). See the new Multiple specs guide.inspect(): validation results as data.FrameSpec.inspect(df)(andpolspec.validation.inspect(spec, df)) returns aValidationReportofFindingrecords -- each with a code, a stable key, the columns involved, a count, samples and code-specific details -- and never raises for a bad frame.report.rows(finding)andreport.failing_rows()return the offending rows lazily;by_column(),by_code()andto_json()slice and export them. Checks and validators now carry samples too.ValidationError.reportcarries the sameValidationReport;.errorsis still the list of messages.polspec validate SPEC DATA [--references NAME=PATH] [--json]on the command line, exiting 1 on findings, so a spec can gate a pipeline in CI.- A foreign key whose parent was not supplied is a
foreign_key_unresolvedfinding rather than aValueError, matching howgenerate()already treats it; a parent lacking the referenced columns is aforeign_keyfinding. - Spec files carry a
version:(now 2). Files from version 1 are migrated on read; a file from a newer polspec is refused with a clear message. A key the reader does not know is an error naming the closest known key;from_yaml(..., strict=False)downgrades it to a warning. - Foreign keys to other specs are written to YAML and Python as the target's name and read back unresolved, instead of being dropped with a warning.
polspec.serializationis a package driven by one field registry (fields.py): YAML in both directions, generated Python, and theimport datetimedecision all derive from it, and a test asserts every dataclass field has an entry.to_dict/from_dictare public.CatSpecfiles keep choices recorded for plain string columns.polspec.col(), a small predicate language for rules, validators and checks:col("total") >= col("subtotal"),col("email").str.contains("@"),is_in,is_between,is_null,&/|/~, arithmetic, and string operations. A predicate evaluates like the Polars expression it stands for and, unlike one, is written to and read from YAML and generated Python.__checks__andColSpec.validatorswritten withcol()now round-trip throughto_yaml/from_yamlandto_python. Rawpl.Expris still accepted and still warns on export.ColRule.whenaccepts a predicate, so a rule may depend on several columns. The one-column dict form is still accepted and converted.TableSpec: the spec as an immutable value. AFrameSpecclass body now builds one, reachable asSpec.spec, and every verb (generate,validate,to_yaml,to_markdown, ...) is a function over it inpolspec.generation,polspec.validation,polspec.serializationandpolspec.report.TableSpecofferswith_columns,drop,select,rename,with_checks,with_foreign_keys,with_unique_together,with_nameandwith_catspec;FrameSpec.from_specwraps one in a class. See the new Specs as values guide.FrameSpec.col(name)reaches a column whatever it is called.ForeignKey.referencesmay be a spec's name, for keys whose target is not importable where the key is declared.- An exception hierarchy under
PolspecError:SpecErrorfor declarations that cannot mean anything,ValidationErrorfor data that fails its spec,GenerationErrorwhen a spec cannot be turned into data (including every error raised inside the Rust engine),SerializationErrorfor files that cannot be written or read, andRegistryError, reserved for the spec registry. All are exported frompolspec; see the new Errors reference page.
Changed¶
- Breaking.
ColRule.whenno longer accepts the one-column dict ({"column": "region", "equals": "UK"}).col()is the only spelling: writecol("region") == "UK". A spec file written by an earlier version still loads -- its conditions are converted as the file migrates -- but a file declaring the current version must carry the predicate form. The error names the column and says what to write. - Breaking. The
leandgecondition keys are gone; they were undocumented duplicates oflteandgte. (le/geremain the canonical operator names in a predicate's data form, which is unrelated.) - Breaking.
polspec.serializationno longer re-exports the names of the pre-package module layout:_YAML_DTYPES,_YAML_NAME_TO_DTYPE,_dtype_to_yaml,_dtype_from_yaml,_dtype_to_python,_colspec_to_yaml,_colspec_from_yamland_colspec_to_python. Use the names inpolspec.serialization.fieldsandpolspec.serialization.dtypes. - Breaking.
ColRule.whenis evaluated against the frame as it stands when the rule runs, not against the freely generated values. Rules and foreign keys are applied in dependency order, so a rule keyed on a column that another rule or a foreign key rewrites now reads the rewritten values -- the onesvalidate()checks it against. Chained rules, chained foreign keys, and a rule keyed on a foreign-keyed column all round-trip; the values a given seed produces for such a spec change. - Breaking. Two columns whose rules each read what the other writes have
no order that satisfies both, and are now refused at declaration with a
SpecErrornaming them. - Breaking. A
ForeignKeywhose parent's declared domain does not fit inside its own column's is refused at declaration (or when aRegistryresolves a key that names its target as a string). A key overwrites its column with the parent's values, sobounds=(1, 50)on a column referencing keys in100..200could only ever generate data that fails its own validation. A column declaring noboundsorchoicesstill accepts anything. - Breaking.
unique=Truecan no longer be combined withweights, a non-uniformdistribution, orrules. The first two describe how often a value recurs, which a draw without replacement has no room for; a rule assigns from a fixed set, which is how duplicates would get back in. Each is refused at declaration rather than silently ignored. - Breaking. A column carrying
rulesmay not also be part of a__unique_together__group, for the same reason: the repair that separates repeated combinations would overwrite what the rule put there. - Breaking. A
ForeignKeyfilling aunique=Truecolumn now refuses when the parent holds fewer distinct values than there are rows, instead of falling back to sampling with replacement and producing the duplicates the column forbids. polspec testno longer emitsvalidate_unique=Falsein generated tests. Uniqueness is generated now, so the generated test asserts it.-
A foreign key spanning textual dtypes -- a
Stringcolumn referencing anEnumkey, which declaration has always allowed and generation has always handled -- now validates instead of raisingSchemaErrorfrom the anti-join. The parent's keys are cast to the local dtype for the join, soValidationReport.rows()still returns the frame's own rows unchanged. -
Breaking. Each column's generation seed is derived from the frame seed and the column's name, not its position, so inserting a column no longer reshuffles the columns after it. The values a given seed produces change from previous versions.
ColRuleapplication samples only as many values as there are matched rows and scatters them into place, instead of filling the whole column per rule.polspec.validationis a package (report.py,constraints.py); foreign key anti-joins are collected together withpl.collect_allinstead of onecollectper key.- Breaking.
ColSpec.distributionanddistribution_paramsare stored in canonical form ("exp"becomes"exponential",mu/sigmabecomemean/std, and so on), so spec files are canonical. Every alias is still accepted when declaring. - Breaking. An unrecognised physical dtype in a
CatSpecentry is now aSerializationErrorinstead of silently becomingUInt32. - Breaking.
ColRule.whenis a predicate after construction rather than a dict (rule.when.root_names()lists the columns it reads); rules in YAML are written in the predicate data form, and the old dict form is still read. - Breaking. A column may now share a name with a
FrameSpecmethod: the metaclass takesColSpecattributes out of the class namespace, soschema,tagand friends no longer shadow anything and no longer warn. The private_columns,_checks,_unique_togetherand_foreign_keysclass attributes are gone; readSpec.spec.columnsand friends instead. - Breaking.
ForeignKey.referencesis the target's name after construction (the bound spec is available asForeignKey.target), andreferences={...}ongenerate/validateaccepts the class, theTableSpecor the name as key. - Breaking. Removed: the
FrameSchemaalias;FrameSpec.generate_catspec,write_catspec,infer_catspecandwith_inferred_catspec(usecatspec(),catspec().to_yaml(),CatSpec.infer(...)andwith_catspec(CatSpec.infer(...))); themax_uniqueandboundsalias keyword arguments offrom_dataframe(usemax_unique_enumandcalculate_bounds). to_yamlandto_pythonshare one set of warnings about what a file cannot hold.- Breaking, mildly. Errors that were bare
ValueErrororTypeErrorare now the subclass above. Each keeps the built-in type it replaced, soexcept ValueErrorstill catches it; only code matching on the exact type (type(exc) is ValueError) sees a difference. Plain argument misuse (n < 0, an unknownmethod=) is unchanged. - The command line prints any
PolspecErroras a one-lineerror: ...instead of aTypeName: messageline.
Fixed¶
- Foreign-key sampling during generation drew parent keys from an unordered
unique(), so the same seed could give different child rows between runs. The parent's distinct keys now keep their order and generation is reproducible.
0.1.5 - 2026-09-03¶
Added¶
python/polspec/py.typed, so type checkers use the package's annotations.CONTRIBUTING.md, this changelog, and a.python-versionfile.examples/related_specs.py: a worked example of four related specs (foreign keys, shared categories, rules, checks, a YAML-declared spec). It runs in CI as a smoke test.- A release-workflow job that refuses a
vX.Y.Ztag whose version does not matchpyproject.toml. - CI now runs
ruff checkwith a wider rule set,ruff format --check,cargo fmt --check,cargo clippy -D warningsandcargo test, tests on macOS as well as Linux and Windows, and tests against the newest Polars release inside the declared bound. The docs build runs strictly on pull requests. - Release builds now produce wheels for Linux aarch64 and macOS (x86_64 and arm64) alongside Linux and Windows x86_64, plus an sdist, and only publish when the test workflow is green.
- An optional
.pre-commit-config.yamlwith ruff and cargo fmt hooks. tests/test_colspec.py(2,000 lines, unsectioned) is split intotest_generation.py,test_rules.py,test_serialization.py,test_profiler.py,test_framespec.py,test_report.pyandtest_foreign_key.py, each with a docstring saying what it covers.
Changed¶
- The crate version in
Cargo.tomlis a placeholder;pyproject.tomlis the only place the version is set, souv version --bumpworks. - The
parquet,ipcandallextras (all identical) are replaced by a singlearrowextra. Install withpolspec[arrow]for the Parquet and Arrow IPC sinks. polarsis bounded to<2; the Rust extension is coupled to a Polars release line.- The abi3 floor is now Python 3.12, matching
requires-python. cargo testlinks again (extension-moduleis no longer an unconditional crate feature; maturin enables it).
Fixed¶
- Repository URL in package metadata pointed at the repository's old name.
- README claimed the license was unspecified; it is MIT.
- Documentation:
ColSpec(col_name=...)is now described in Declaring columns,FrameSpec.to_python()in YAML specs, the getting-started example importsdate, and the architecture page lists theclimodule and its tests.
0.1.4 - 2026-09-02¶
Added¶
polspec schema infer --output spec.pyandFrameSpec.to_python(), which write a spec as an editable Python module rather than YAML.
0.1.3 - 2026-09-01¶
Added¶
ColSpec(col_name=...), so a column's name in data may differ from the attribute name used to declare it.
Changed¶
- Roadmap expanded with detailed plans for a spec registry, structured validation results, and generation guardrails.
0.1.2 - 2026-08-31¶
Version bump only; no user-facing change.
0.1.1 - 2026-08-31¶
Added¶
- Test workflow on GitHub Actions (Linux and Windows, Python 3.12 to 3.14).
Fixed¶
ColSpec.dtypeaccepts a dtype class as well as an instance.
0.1.0 - 2026-08-31¶
First tagged release.
ColSpecandFrameSpec: declare a Polars schema with nullability, bounds, string lengths, choices and weights, distributions, tags, and conditionalColRules.generate()backed by a parallel Rust extension,method="cartesian"for coverage sets, batched generation and Parquet/CSV/IPC/NDJSON sinks.validate()collecting every violation in one Polars aggregation, with column validators, multi-columnChecks, composite uniqueness andForeignKeys.CatSpecregistries for sharedEnum/Categoricaldomains.- YAML round-trip,
from_dataframe()profiling, Markdown and Mermaid output. - CLI:
polspec schema infer,polspec schema new,polspec test. - Documentation site, comparison guide, and release automation.