Validating data¶
The examples on this page check data against these specs:
from datetime import date
from pathlib import Path
import polars as pl
from polspec import ColSpec, ForeignKey, FrameSpec
class Customers(FrameSpec):
id = ColSpec(pl.Int64, bounds=(1, 10_000), unique=True)
name = ColSpec(pl.String, string_length=(3, 20), tags="pii")
country = ColSpec(pl.Enum(["UK", "US", "DE"]))
signed_up = ColSpec(pl.Date, bounds=(date(2020, 1, 1), date(2026, 1, 1)))
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 1_000_000), unique=True)
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
status = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]))
total = ColSpec(pl.Float64, bounds=(0.0, 5_000.0))
internal_note = ColSpec(pl.String, string_length=(0, 40), nullable=True)
class OrderLines(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 1_000_000))
line_no = ColSpec(pl.Int32, bounds=(1, 50))
quantity = ColSpec(pl.UInt16, bounds=(1, 500))
__unique_together__ = [["order_id", "line_no"]]
__foreign_keys__ = [ForeignKey("order_id", references=Orders)]
df = Orders.generate(500, seed=2)
validate() accepts a DataFrame or a LazyFrame and returns the same kind,
so it drops into a pipeline — df.pipe(Orders.validate) works as written. On
success the returned frame has its declared columns first, in declaration
order.
A LazyFrame is validated by running its query, and collecting the frame
validate() hands back runs it again; report.failing_rows() and
passing_rows() run it once more each. That is the price of handing back a
lazy frame. When the query is expensive to run — a large scan, a slow join —
collect it once and validate the DataFrame.
Collecting every problem at once¶
All checks across all columns are compiled into one Polars aggregation and evaluated in a single scan. Every breach is gathered before anything is raised:
from polspec import ValidationError
row = pl.int_range(pl.len())
broken = df.with_columns(
# Three totals made negative, and the tenth order given the first's id.
total=pl.when(row < 3).then(-pl.col("total")).otherwise("total"),
order_id=pl.when(row == 9).then(pl.col("order_id").first()).otherwise("order_id"),
)
try:
Orders.validate(broken)
except ValidationError as err:
print(len(err.errors), "problems")
for problem in err.errors:
print(problem)
2 problems
Column 'order_id': unique column contains 2 duplicate value(s). Duplicate samples: [633030]
Column 'total': found 3 value(s) out of bounds [0.0, 5000.0] (min found: -4626.908973514467, max found: 4952.446695621952). Out of bounds samples: [-2506.9942460024845, -1514.4631123193444, -4626.908973514467]
ValidationError is a PolspecError (and still a ValueError); see
Errors. err.errors is the list of
individual messages; str(err) is the same list formatted as a report, and
err.report is the ValidationReport behind both.
Findings as data — inspect()¶
An exception is the right shape for someone reading a traceback. For code
that wants to act on what was found — quarantine the offending rows, count
problems per column, write a report — use inspect(), which returns the same
findings as a ValidationReport and never raises for a bad frame:
suspect = df.with_columns(pl.col("total") * -1) # every total now negative
report = Orders.inspect(suspect)
report.passed # False
for finding in report:
finding.code # "bounds", "check", "foreign_key", ...
finding.key # "total__bounds", "check:total_covers_subtotal",
# "point.lat__bounds" for a struct's field
finding.columns # ("total",)
finding.count # rows violating it (None for structural findings)
finding.share # count as a fraction of the rows, 0 to 1 (added in 1.4)
finding.samples # up to five offending values
finding.details # {"bounds": [0.0, None], "min_found": -3.0, ...}
finding.message # the same text validate() would have raised
report.by_column()["total"] # every finding involving one column
report.by_code("foreign_key") # every finding of one kind
report.to_json() # everything above, JSON-safe
The offending rows are reachable lazily, so nothing is materialised until you ask:
bad = report.by_code("bounds")[0]
report.rows(bad).collect() # just the rows violating that one claim
report.failing_rows().collect() # every violating row, with a `__polspec_finding`
# column naming the claim (a row violating two
# claims appears twice)
report.failing_rows(once=True).collect() # each violating row once, naming
# every claim it broke
For the broken frame above, the quarantine view names both problems of
the one row that has two:
quarantine = Orders.inspect(broken).failing_rows(once=True).collect()
print(quarantine.select("order_id", "total", "__polspec_finding"))
shape: (4, 3)
┌──────────┬──────────────┬────────────────────────────────┐
│ order_id ┆ total ┆ __polspec_finding │
│ --- ┆ --- ┆ --- │
│ i64 ┆ f64 ┆ str │
╞══════════╪══════════════╪════════════════════════════════╡
│ 633030 ┆ -2506.994246 ┆ order_id__unique,total__bounds │
│ 562818 ┆ -1514.463112 ┆ total__bounds │
│ 509480 ┆ -4626.908974 ┆ total__bounds │
│ 633030 ┆ 1231.613496 ┆ order_id__unique │
└──────────┴──────────────┴────────────────────────────────┘
once=True is the quarantine view: with passing_rows() it splits a frame
exactly, every row in one or the other, and it is what
polspec validate --failing writes.
The column failing_rows() adds is named by polspec.FINDING_COLUMN
rather than spelled out, so grouping by it does not hard-code the name:
from polspec import FINDING_COLUMN
quarantined = report.failing_rows().collect()
quarantined.group_by(FINDING_COLUMN).len() # how many rows each claim caught
It is the one name polspec adds to your rows, so a frame that already has a
column called __polspec_finding is refused by name — rename it — rather
than losing that column to the finding keys. Every other name works: the
helper columns polspec uses along the way take names the frame does not
have.
passing_rows() is the other half: the rows that pass, in the frame's
order, so the two split a frame into what to keep and what to send back. A
duplicate key fails both of its rows, and an orphaned foreign key its row.
Some claims relate rows to each other — an order_by, a key into the
frame's own rows — and taking rows away can break them in the rows left: a
row whose parent was taken, rows no longer in order. So what is left is
validated again, and anything that now fails goes too, until nothing more
does; failing_rows(once=True) names those rows by the claim they broke.
What passes is a frame the spec accepts:
Some findings are structural: they judge the frame as a whole, not
particular rows — an extra or missing column, a wrong dtype, a foreign key
with no parent supplied, a row count outside __rows__. They have no rows
to return, and passing_rows() refuses to answer until they are fixed. (A
dtype finding about values the declared dtype cannot hold, keyed
<column>__dtype_range, is the exception: it has rows.) The full list of
codes, and which kind each is, is in
Errors and findings.
inspect() takes exactly the options validate() does: validate() is
inspect(), then report.raise_if_failed(), then the structural
transformations described below. polspec validate on the
command line prints the same
report.
Tolerating a few bad rows — tolerate=¶
A pipeline often accepts a little bad data and fails only past a
threshold: a few nulls in a feed, a handful of totals out of range.
tolerate= says how large a share of the rows is acceptable. Added in
1.7.
from polspec import TableSpec
feed = TableSpec("Feed", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
frame = pl.DataFrame({"total": [250.0] + [10.0] * 999}) # 0.1% out of bounds
polspec.inspect(feed, frame).passed # False
polspec.inspect(feed, frame, tolerate=0.001).passed # True
One number tolerates every finding up to that share. A mapping tolerates some, by finding code, by column, or by both, and the most specific key wins:
polspec.validate(feed, frame, tolerate={"bounds": 0.01}) # by code
polspec.validate(feed, frame, tolerate={"total": 0.001}) # by column
polspec.validate(feed, frame, tolerate={("total", "bounds"): 0.01}) # by both
A share is inclusive: 0.001 accepts up to one row in a thousand. A
tolerated finding stays in the report, marked, so nothing is hidden:
report = polspec.inspect(feed, frame, tolerate=0.001)
report.passed # True
[f.code for f in report.tolerated] # ['bounds']
report.failures # () -- what fails the report
report.findings[0].tolerated # True
validate() raises only for a finding no tolerance covers, and
passing_rows() still leaves out every row a finding names, tolerated or
not. A finding with no share — a missing or extra column, a wrong dtype, an
unresolved foreign key — judges the whole frame and is never tolerated. A
key naming neither a finding code nor a column of the spec is refused,
naming the nearest; so is a key that is both, such as a column called
order, which ("order", "order") or the code's own column then
disambiguates. polspec validate --tolerate takes the same on the
command line.
Checking a file you were given¶
The usual way in: someone hands you a file, and there is a spec it should
meet. Read it loosely, let inspect() say everything that is wrong, decide
whether the file or the spec is at fault, and only then ask for the typed
frame.
Path("customers.csv").write_text( # the file you were handed
"id,name,country,signed_up\n"
"1,Ada Lovelace,UK,2021-03-04\n"
"2,Grace Hopper,US,2019-12-31\n"
"2,Al,FR,2022-01-01\n"
)
1. Read it without forcing the spec's types.
read() picks the reader from the extension (.csv, .tsv, .parquet,
.ndjson, .json, .arrow — the set polspec validate reads, with any
Polars reader option passed through, such as separator=";") and does one
thing in the spec's name. A CSV has no date type, so signed_up arrives as
text; read() parses each column the spec declares as a date or time, when
every value in it parses. Without that, the report would say only expected
Date, got String, and none of the column's own checks would run. The other
gaps between a CSV and a spec take care of themselves — a String column
is accepted where an Enum or Categorical is declared, an integer where a
float or Decimal is, and the values are checked either way.
A file too unclean to type on reading can be read with its types switched
off, and read() types it in the spec's terms instead:
Customers.read("customers.csv", infer_schema=False) hands back each
column the spec declares as a number, a flag or a date parsed as one, when
every value in it parses. Text is stripped and empty text is a null there,
as in Data read as text. Added in 1.2.
JSON has a few more gaps, and read() closes each the same way, only for
what the spec declares and only when every value reads exactly. Polars
writes a Decimal as text ("1.25"), read back from its digits — never
rounded, so "1.255" in a two-place column stays text and is reported. It
writes a Duration as ISO 8601 seconds ("PT3600S", "-PT0.5S", "P0D"),
a fixed-size Array as a list, dates inside a list or struct as text, and a
column of nothing but nulls with no type at all; each is read back as
declared. What JSON cannot hold at all no reader brings back: NaN is
written as null.
In plain Polars, pl.read_csv("customers.csv", try_parse_dates=True) comes
close; the difference is that Polars parses every column that looks like a
date, where read() parses only the ones the spec says are dates.
Reading with the spec's schema is the tempting alternative, and the wrong first step: Polars stops at the first value that does not fit, so one error replaces the whole report.
pl.read_csv("customers.csv", schema_overrides=Customers.schema())
# ComputeError: could not parse `FR` as dtype `enum` at column 'country' ...
2. Ask what is wrong — all of it.
Validation failed for DataFrame against 'Customers' (4 error(s) found):
- Column 'id': unique column contains 2 duplicate value(s). Duplicate samples: [2] (66.7% of rows)
- Column 'name': found 1 value(s) with string length outside [3, 20]. Invalid samples: ['Al'] (33.3% of rows)
- Column 'country': found 1 invalid value(s) not in allowed choices/categories ['UK', 'US', 'DE']. Invalid samples: ['FR'] (33.3% of rows)
- Column 'signed_up': found 1 value(s) out of bounds [2020-01-01, 2026-01-01] (min found: 2019-12-31, max found: 2022-01-01). Out of bounds samples: [datetime.date(2019, 12, 31)] (33.3% of rows)
report.rows(finding) is the offending rows, and report.to_json() is
something to send back to whoever sent the file:
3. Decide which is wrong: the file, or the spec. A bad row is fixed or
filtered at the source. A spec that has fallen behind — FR is a real
country now — is changed, and diff says
whether the change is breaking for anything already validated against it:
Widened = Customers.spec.with_columns(country=ColSpec(pl.Enum(["UK", "US", "DE", "FR"])))
Customers.diff(Widened).breaking # () -- widening a domain breaks nothing
While you are still finding out what the ranges really are,
bounds=False checks everything but the bounds, and
from_dataframe
describes what the file actually holds, to compare against what the spec
says it should.
If the file is mostly right and the rest can wait, keep what passed and
send back what did not: report.passing_rows() and report.failing_rows()
split it (see Findings as data), and
polspec validate --output --failing does the same from the shell.
4. Then take the typed frame. Once it passes, cast=True returns each
column as its declared dtype — the Enum an Enum, not the String the
CSV held — so the code downstream reads the types the spec promises:
corrected = given.filter(pl.col("name") == "Ada Lovelace") # the file, fixed
clean = Customers.validate(corrected, cast=True)
clean.schema["country"] # Enum(categories=['UK', 'US', 'DE'])
From the shell, polspec validate customers.py customers.csv does steps 1
and 2 in one go, parsing the columns the spec declares as dates and times;
see the CLI.
What a CSV cannot hold
A CSV has no way to write a Duration, a List or a Struct, so a
spec with one of those cannot be met by a CSV however it is read.
Parquet or Arrow IPC keeps every dtype.
Options¶
Orders.validate(
df,
extra_cols="raise", # "raise" | "drop" | "allow"
missing_cols="raise", # "raise" | "add" | "allow"
strict_dtypes=False,
cast=False,
fill_defaults=False,
parse_text=False,
streaming=False,
references=None,
rules=True,
validators=True,
unique=True,
checks=True,
foreign_keys=True,
hierarchy=True,
order=True,
rows=True,
pattern=True,
bounds=True,
)
Every one of these is a field of ValidationOptions, which is what a report
carries as report.options — so a report says what it was asked to check, not
only what it found:
report = Orders.inspect(df, checks=False)
report.options.checks # False
report.options.extra_cols # "raise"
A keyword is the field it sets: checks=False is options.checks, as
--skip checks is on the command line. An option name polspec does not
accept is a TypeError naming the closest one it does.
You can also pass the whole set as one value, which is the shape to reach for when the same settings go through several calls:
from polspec import ValidationOptions
lenient = ValidationOptions(extra_cols="drop", checks=False)
for frame in (df, df.head(10)):
Orders.validate(frame, options=lenient)
options= and the individual keywords are alternatives, not a base and an
override — passing both raises rather than quietly picking one.
Structural mismatches¶
extra_cols decides what happens to columns the spec does not declare —
refuse, drop them from the result, or keep them (appended after the declared
ones).
missing_cols decides what happens to declared columns the frame lacks —
refuse, add them, or ignore them. An added column holds its default, or
nulls when it has none. A column declared optional=True may be absent
whatever this says; present, it is checked like any other.
missing_cols="add" can produce a frame that fails re-validation
Columns are added after validation has run, including for columns
declared nullable=False. A column with no default is added as nulls,
and fed straight back into validate() it will object to them.
Filling defaults¶
fill_defaults=True (--fill-defaults) fills each column's nulls with its
default, and adds an absent column holding it, before anything is
checked — so a null in a column that allows none passes once filled, and the
frame validate returns, report.frame and passing_rows() are all the
filled one. A NaN is not filled: it is a value.
class Tickets(FrameSpec):
status = ColSpec(pl.Enum(["open", "closed"]), default="open")
channel = ColSpec(pl.String, optional=True)
Tickets.validate(pl.DataFrame({"status": [None, "closed"]}), fill_defaults=True)
# status: ["open", "closed"]; channel may be absent
A default must be a value its column accepts — its dtype, bounds, choices, format, pattern, lengths and validators — and one that is not is refused where it is declared. Filling is opt-in: a spec that gains a default changes nothing until asked, and without it a null is still a finding.
Data read as text¶
parse_text=True (polspec validate always asks it) reads each column that
arrived as text, where the spec declares a type text can hold, as that type
before anything is checked: a Boolean from true or false in any case,
an integer, a float, a Date, Datetime or Time, a Decimal, a
Duration. It is for a frame read with its types switched off — the frame
a profile with parse_text=True was made
from validates against it this way. Added in 1.2.
import polspec
Loans = polspec.TableSpec("Loans", {"LGD": ColSpec(pl.Float64, nullable=True)})
raw = pl.DataFrame({"LGD": ["0.45", "", "0.2"]}) # read as text
polspec.validate(Loans, raw, parse_text=True) # LGD as Float64: [0.45, None, 0.2]
Text is stripped and empty text is a null, so fill_defaults=True fills
it. A column is parsed only when every present value parses. Otherwise it
stays text, and its dtype finding counts the values that did not and
gives some as its samples:
Column 'LGD': expected dtype Float64, got String: 1 value(s) do not parse as Float64, such as ['N/A']
A Date, Datetime or Time column with a text_format is read with
it, and a value written another way is reported by name:
Bonds = polspec.TableSpec(
"Bonds", {"maturity": ColSpec(pl.Date, text_format="%d %b %Y")}
)
report = polspec.inspect(
Bonds, pl.DataFrame({"maturity": ["30 Sep 2031", "2031-09-30"]}), parse_text=True
)
print(report.findings[0].message)
Column 'maturity': expected dtype Date, got String: 1 value(s) do not parse as Date written as '%d %b %Y', such as ['2031-09-30']
Without a text_format, a date, datetime or time is read in whatever form
Polars recognises. text_format added in 1.6.
An integer is read as an Int64, so a value past a narrower declared width
is reported as any wider integer standing in is: see the next section. The
frame comes back with the parsed columns; add cast=True for the declared
widths.
Dtype strictness¶
By default polspec accepts what a real pipeline delivers: any integer width for
a declared integer, an integer or float for a declared float, and
String/Categorical for a declared Enum. A temporal is held to its kind:
a Date and a Datetime stand in for each other, since text formats hand
one back as the other, while a Time stands in only for a Time and a
Duration only for a Duration, of any unit. A zoned Datetime and a
naive one do not stand in for each other — one is an instant, the other a
wall-clock reading — but two zones do, being the same instants shown
differently. strict_dtypes=True requires the exact dtype, treating only
String and Utf8 as interchangeable.
A wider dtype standing in still has to hold only values the declared one
can. An Int64 column where an Int8 is declared passes with values from
-128 to 127, and a value outside that is a dtype finding with its rows and
samples; so is a float or integer too large for a declared Decimal's
integer digits, after rounding to its scale. A declared bound has to fit its
dtype, so on a side a bound closes such a value is out of bounds as well,
and reported only as that — unless bounds=False. That is what makes a frame
with no findings safe to hand to cast=True. Floats are left alone: a
Float64 too large for a declared Float32 casts to an infinity rather
than failing.
Casting¶
cast=True casts each column to its declared dtype after validation passes,
so a String column that holds only valid enum members comes back as the Enum.
Streaming¶
streaming=True evaluates with the Polars streaming engine, for frames larger
than memory.
The schema alone¶
schema_only=True checks what a frame's schema says — missing, extra and
optional columns, and each column's dtype — and reads no rows. On a
LazyFrame the query is planned, never run, so a test can hold a lazy
pipeline's output shape to its spec for free:
source = pl.LazyFrame(schema=Orders.spec.schema()) # pl.scan_parquet(...) in practice
pipeline = source.with_columns(pl.col("total") * 2)
Orders.validate(pipeline, schema_only=True) # planned, never collected
Added in 1.4. It does not combine with parse_text=True, which reads the
values to decide their type.
Turning checks off¶
Each switch disables a whole category of check. Most exist for what
generation cannot satisfy — __checks__, validators and pattern are
validated but not generated, and __rows__ is not a limit on generate():
bounds=False is the other way round: generation always stays in
bounds, so it is for real data — a file whose ranges you want to look at
before holding it to them, while every other claim is still checked. It
covers every bounds, including a List's elements and a struct's fields;
string_length and list_length have codes of their own and stay on.
To loosen one column rather than all of them, validate against a spec with that column's bounds removed:
from polspec import validate
loose = Orders.spec.with_columns(total=Orders.col("total").replace(bounds=None))
validate(loose, df)
What gets checked¶
| Check | From |
|---|---|
| Column present / not extra | the spec's column set |
| Dtype compatible | ColSpec.dtype |
| No unexpected nulls | nullable |
| Value in domain | choices, Enum categories |
| Value within range | bounds |
| Length within range | string_length |
| List has the declared number of elements | list_length |
| No NaN where none is declared | nan_probability |
| Value has the declared shape | format |
| Value matches the regex | pattern |
| Conditional values hold | rules |
| Single-column predicates | validators |
| Values distinct | unique |
| Composite key distinct | __unique_together__ |
| Multi-column invariants | __checks__ |
| Rows in the declared order | __order_by__ |
| Number of rows within range | __rows__ |
| Referential integrity | __foreign_keys__ |
| One parent, bounded depth, no cycles | __hierarchy__ |
Bounds, lengths, rules, validators and uniqueness are skipped for a column whose dtype is already wrong — comparing values of the wrong type would bury the dtype error under noise.
Foreign keys need their parent¶
A key referencing another spec needs that spec's data:
orders = Orders.generate(500, seed=2)
order_lines = OrderLines.generate(800, seed=3, references={Orders: orders})
OrderLines.validate(order_lines, references={Orders: orders})
Without it, the key is reported as a foreign_key_unresolved finding naming
the spec it needed, so validate() raises and inspect() lists it alongside
everything else. references may be keyed by the class, its TableSpec, or
the spec's name. Self-referencing keys are checked against the frame itself
and need nothing.
Each foreign key is an anti-join against the parent, so these run separately from the single-pass aggregation above.