Skip to content

Validating data

The examples on this page check data against these specs:

from datetime import date
from pathlib import Path

import polars as pl
from polspec import ColSpec, ForeignKey, FrameSpec


class Customers(FrameSpec):
    id        = ColSpec(pl.Int64, bounds=(1, 10_000), unique=True)
    name      = ColSpec(pl.String, string_length=(3, 20), tags="pii")
    country   = ColSpec(pl.Enum(["UK", "US", "DE"]))
    signed_up = ColSpec(pl.Date, bounds=(date(2020, 1, 1), date(2026, 1, 1)))


class Orders(FrameSpec):
    order_id      = ColSpec(pl.Int64, bounds=(1, 1_000_000), unique=True)
    customer_id   = ColSpec(pl.Int64, bounds=(1, 10_000))
    status        = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]))
    total         = ColSpec(pl.Float64, bounds=(0.0, 5_000.0))
    internal_note = ColSpec(pl.String, string_length=(0, 40), nullable=True)


class OrderLines(FrameSpec):
    order_id = ColSpec(pl.Int64, bounds=(1, 1_000_000))
    line_no  = ColSpec(pl.Int32, bounds=(1, 50))
    quantity = ColSpec(pl.UInt16, bounds=(1, 500))
    __unique_together__ = [["order_id", "line_no"]]
    __foreign_keys__ = [ForeignKey("order_id", references=Orders)]


df = Orders.generate(500, seed=2)
clean = Orders.validate(df)

validate() accepts a DataFrame or a LazyFrame and returns the same kind, so it drops into a pipeline — df.pipe(Orders.validate) works as written. On success the returned frame has its declared columns first, in declaration order.

A LazyFrame is validated by running its query, and collecting the frame validate() hands back runs it again; report.failing_rows() and passing_rows() run it once more each. That is the price of handing back a lazy frame. When the query is expensive to run — a large scan, a slow join — collect it once and validate the DataFrame.

Collecting every problem at once

All checks across all columns are compiled into one Polars aggregation and evaluated in a single scan. Every breach is gathered before anything is raised:

from polspec import ValidationError

row = pl.int_range(pl.len())
broken = df.with_columns(
    # Three totals made negative, and the tenth order given the first's id.
    total=pl.when(row < 3).then(-pl.col("total")).otherwise("total"),
    order_id=pl.when(row == 9).then(pl.col("order_id").first()).otherwise("order_id"),
)

try:
    Orders.validate(broken)
except ValidationError as err:
    print(len(err.errors), "problems")
    for problem in err.errors:
        print(problem)
2 problems
Column 'order_id': unique column contains 2 duplicate value(s). Duplicate samples: [633030]
Column 'total': found 3 value(s) out of bounds [0.0, 5000.0] (min found: -4626.908973514467, max found: 4952.446695621952). Out of bounds samples: [-2506.9942460024845, -1514.4631123193444, -4626.908973514467]

ValidationError is a PolspecError (and still a ValueError); see Errors. err.errors is the list of individual messages; str(err) is the same list formatted as a report, and err.report is the ValidationReport behind both.

Findings as data — inspect()

An exception is the right shape for someone reading a traceback. For code that wants to act on what was found — quarantine the offending rows, count problems per column, write a report — use inspect(), which returns the same findings as a ValidationReport and never raises for a bad frame:

suspect = df.with_columns(pl.col("total") * -1)   # every total now negative
report = Orders.inspect(suspect)

report.passed                 # False
for finding in report:
    finding.code              # "bounds", "check", "foreign_key", ...
    finding.key               # "total__bounds", "check:total_covers_subtotal",
                              # "point.lat__bounds" for a struct's field
    finding.columns           # ("total",)
    finding.count             # rows violating it (None for structural findings)
    finding.share             # count as a fraction of the rows, 0 to 1 (added in 1.4)
    finding.samples           # up to five offending values
    finding.details           # {"bounds": [0.0, None], "min_found": -3.0, ...}
    finding.message           # the same text validate() would have raised

report.by_column()["total"]   # every finding involving one column
report.by_code("foreign_key") # every finding of one kind
report.to_json()              # everything above, JSON-safe

The offending rows are reachable lazily, so nothing is materialised until you ask:

bad = report.by_code("bounds")[0]
report.rows(bad).collect()        # just the rows violating that one claim
report.failing_rows().collect()   # every violating row, with a `__polspec_finding`
                                  # column naming the claim (a row violating two
                                  # claims appears twice)
report.failing_rows(once=True).collect()  # each violating row once, naming
                                          # every claim it broke

For the broken frame above, the quarantine view names both problems of the one row that has two:

quarantine = Orders.inspect(broken).failing_rows(once=True).collect()
print(quarantine.select("order_id", "total", "__polspec_finding"))
shape: (4, 3)
┌──────────┬──────────────┬────────────────────────────────┐
│ order_id ┆ total        ┆ __polspec_finding              │
│ ---      ┆ ---          ┆ ---                            │
│ i64      ┆ f64          ┆ str                            │
╞══════════╪══════════════╪════════════════════════════════╡
│ 633030   ┆ -2506.994246 ┆ order_id__unique,total__bounds │
│ 562818   ┆ -1514.463112 ┆ total__bounds                  │
│ 509480   ┆ -4626.908974 ┆ total__bounds                  │
│ 633030   ┆ 1231.613496  ┆ order_id__unique               │
└──────────┴──────────────┴────────────────────────────────┘

once=True is the quarantine view: with passing_rows() it splits a frame exactly, every row in one or the other, and it is what polspec validate --failing writes.

The column failing_rows() adds is named by polspec.FINDING_COLUMN rather than spelled out, so grouping by it does not hard-code the name:

from polspec import FINDING_COLUMN

quarantined = report.failing_rows().collect()
quarantined.group_by(FINDING_COLUMN).len()   # how many rows each claim caught

It is the one name polspec adds to your rows, so a frame that already has a column called __polspec_finding is refused by name — rename it — rather than losing that column to the finding keys. Every other name works: the helper columns polspec uses along the way take names the frame does not have.

passing_rows() is the other half: the rows that pass, in the frame's order, so the two split a frame into what to keep and what to send back. A duplicate key fails both of its rows, and an orphaned foreign key its row. Some claims relate rows to each other — an order_by, a key into the frame's own rows — and taking rows away can break them in the rows left: a row whose parent was taken, rows no longer in order. So what is left is validated again, and anything that now fails goes too, until nothing more does; failing_rows(once=True) names those rows by the claim they broke. What passes is a frame the spec accepts:

kept = report.passing_rows().collect()
Orders.validate(kept)                 # passes

Some findings are structural: they judge the frame as a whole, not particular rows — an extra or missing column, a wrong dtype, a foreign key with no parent supplied, a row count outside __rows__. They have no rows to return, and passing_rows() refuses to answer until they are fixed. (A dtype finding about values the declared dtype cannot hold, keyed <column>__dtype_range, is the exception: it has rows.) The full list of codes, and which kind each is, is in Errors and findings.

inspect() takes exactly the options validate() does: validate() is inspect(), then report.raise_if_failed(), then the structural transformations described below. polspec validate on the command line prints the same report.

Tolerating a few bad rows — tolerate=

A pipeline often accepts a little bad data and fails only past a threshold: a few nulls in a feed, a handful of totals out of range. tolerate= says how large a share of the rows is acceptable. Added in 1.7.

from polspec import TableSpec

feed = TableSpec("Feed", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
frame = pl.DataFrame({"total": [250.0] + [10.0] * 999})   # 0.1% out of bounds

polspec.inspect(feed, frame).passed                        # False
polspec.inspect(feed, frame, tolerate=0.001).passed        # True

One number tolerates every finding up to that share. A mapping tolerates some, by finding code, by column, or by both, and the most specific key wins:

polspec.validate(feed, frame, tolerate={"bounds": 0.01})            # by code
polspec.validate(feed, frame, tolerate={"total": 0.001})            # by column
polspec.validate(feed, frame, tolerate={("total", "bounds"): 0.01}) # by both

A share is inclusive: 0.001 accepts up to one row in a thousand. A tolerated finding stays in the report, marked, so nothing is hidden:

report = polspec.inspect(feed, frame, tolerate=0.001)
report.passed                                  # True
[f.code for f in report.tolerated]             # ['bounds']
report.failures                                # () -- what fails the report
report.findings[0].tolerated                   # True

validate() raises only for a finding no tolerance covers, and passing_rows() still leaves out every row a finding names, tolerated or not. A finding with no share — a missing or extra column, a wrong dtype, an unresolved foreign key — judges the whole frame and is never tolerated. A key naming neither a finding code nor a column of the spec is refused, naming the nearest; so is a key that is both, such as a column called order, which ("order", "order") or the code's own column then disambiguates. polspec validate --tolerate takes the same on the command line.

Checking a file you were given

The usual way in: someone hands you a file, and there is a spec it should meet. Read it loosely, let inspect() say everything that is wrong, decide whether the file or the spec is at fault, and only then ask for the typed frame.

Path("customers.csv").write_text(          # the file you were handed
    "id,name,country,signed_up\n"
    "1,Ada Lovelace,UK,2021-03-04\n"
    "2,Grace Hopper,US,2019-12-31\n"
    "2,Al,FR,2022-01-01\n"
)

1. Read it without forcing the spec's types.

given = Customers.read("customers.csv")

read() picks the reader from the extension (.csv, .tsv, .parquet, .ndjson, .json, .arrow — the set polspec validate reads, with any Polars reader option passed through, such as separator=";") and does one thing in the spec's name. A CSV has no date type, so signed_up arrives as text; read() parses each column the spec declares as a date or time, when every value in it parses. Without that, the report would say only expected Date, got String, and none of the column's own checks would run. The other gaps between a CSV and a spec take care of themselves — a String column is accepted where an Enum or Categorical is declared, an integer where a float or Decimal is, and the values are checked either way.

A file too unclean to type on reading can be read with its types switched off, and read() types it in the spec's terms instead: Customers.read("customers.csv", infer_schema=False) hands back each column the spec declares as a number, a flag or a date parsed as one, when every value in it parses. Text is stripped and empty text is a null there, as in Data read as text. Added in 1.2.

JSON has a few more gaps, and read() closes each the same way, only for what the spec declares and only when every value reads exactly. Polars writes a Decimal as text ("1.25"), read back from its digits — never rounded, so "1.255" in a two-place column stays text and is reported. It writes a Duration as ISO 8601 seconds ("PT3600S", "-PT0.5S", "P0D"), a fixed-size Array as a list, dates inside a list or struct as text, and a column of nothing but nulls with no type at all; each is read back as declared. What JSON cannot hold at all no reader brings back: NaN is written as null.

In plain Polars, pl.read_csv("customers.csv", try_parse_dates=True) comes close; the difference is that Polars parses every column that looks like a date, where read() parses only the ones the spec says are dates.

Reading with the spec's schema is the tempting alternative, and the wrong first step: Polars stops at the first value that does not fit, so one error replaces the whole report.

pl.read_csv("customers.csv", schema_overrides=Customers.schema())
# ComputeError: could not parse `FR` as dtype `enum` at column 'country' ...

2. Ask what is wrong — all of it.

report = Customers.inspect(given)
print(report)
Validation failed for DataFrame against 'Customers' (4 error(s) found):
  - Column 'id': unique column contains 2 duplicate value(s). Duplicate samples: [2] (66.7% of rows)
  - Column 'name': found 1 value(s) with string length outside [3, 20]. Invalid samples: ['Al'] (33.3% of rows)
  - Column 'country': found 1 invalid value(s) not in allowed choices/categories ['UK', 'US', 'DE']. Invalid samples: ['FR'] (33.3% of rows)
  - Column 'signed_up': found 1 value(s) out of bounds [2020-01-01, 2026-01-01] (min found: 2019-12-31, max found: 2022-01-01). Out of bounds samples: [datetime.date(2019, 12, 31)] (33.3% of rows)

report.rows(finding) is the offending rows, and report.to_json() is something to send back to whoever sent the file:

report.rows(report.by_code("choices")[0]).collect()   # the row with country "FR"

3. Decide which is wrong: the file, or the spec. A bad row is fixed or filtered at the source. A spec that has fallen behind — FR is a real country now — is changed, and diff says whether the change is breaking for anything already validated against it:

Widened = Customers.spec.with_columns(country=ColSpec(pl.Enum(["UK", "US", "DE", "FR"])))
Customers.diff(Widened).breaking      # () -- widening a domain breaks nothing

While you are still finding out what the ranges really are, bounds=False checks everything but the bounds, and from_dataframe describes what the file actually holds, to compare against what the spec says it should.

If the file is mostly right and the rest can wait, keep what passed and send back what did not: report.passing_rows() and report.failing_rows() split it (see Findings as data), and polspec validate --output --failing does the same from the shell.

4. Then take the typed frame. Once it passes, cast=True returns each column as its declared dtype — the Enum an Enum, not the String the CSV held — so the code downstream reads the types the spec promises:

corrected = given.filter(pl.col("name") == "Ada Lovelace")   # the file, fixed
clean = Customers.validate(corrected, cast=True)
clean.schema["country"]       # Enum(categories=['UK', 'US', 'DE'])

From the shell, polspec validate customers.py customers.csv does steps 1 and 2 in one go, parsing the columns the spec declares as dates and times; see the CLI.

What a CSV cannot hold

A CSV has no way to write a Duration, a List or a Struct, so a spec with one of those cannot be met by a CSV however it is read. Parquet or Arrow IPC keeps every dtype.

Options

Orders.validate(
    df,
    extra_cols="raise",        # "raise" | "drop" | "allow"
    missing_cols="raise",      # "raise" | "add" | "allow"
    strict_dtypes=False,
    cast=False,
    fill_defaults=False,
    parse_text=False,
    streaming=False,
    references=None,
    rules=True,
    validators=True,
    unique=True,
    checks=True,
    foreign_keys=True,
    hierarchy=True,
    order=True,
    rows=True,
    pattern=True,
    bounds=True,
)

Every one of these is a field of ValidationOptions, which is what a report carries as report.options — so a report says what it was asked to check, not only what it found:

report = Orders.inspect(df, checks=False)
report.options.checks        # False
report.options.extra_cols    # "raise"

A keyword is the field it sets: checks=False is options.checks, as --skip checks is on the command line. An option name polspec does not accept is a TypeError naming the closest one it does.

You can also pass the whole set as one value, which is the shape to reach for when the same settings go through several calls:

from polspec import ValidationOptions

lenient = ValidationOptions(extra_cols="drop", checks=False)

for frame in (df, df.head(10)):
    Orders.validate(frame, options=lenient)

options= and the individual keywords are alternatives, not a base and an override — passing both raises rather than quietly picking one.

Structural mismatches

extra_cols decides what happens to columns the spec does not declare — refuse, drop them from the result, or keep them (appended after the declared ones).

missing_cols decides what happens to declared columns the frame lacks — refuse, add them, or ignore them. An added column holds its default, or nulls when it has none. A column declared optional=True may be absent whatever this says; present, it is checked like any other.

missing_cols="add" can produce a frame that fails re-validation

Columns are added after validation has run, including for columns declared nullable=False. A column with no default is added as nulls, and fed straight back into validate() it will object to them.

Filling defaults

fill_defaults=True (--fill-defaults) fills each column's nulls with its default, and adds an absent column holding it, before anything is checked — so a null in a column that allows none passes once filled, and the frame validate returns, report.frame and passing_rows() are all the filled one. A NaN is not filled: it is a value.

class Tickets(FrameSpec):
    status = ColSpec(pl.Enum(["open", "closed"]), default="open")
    channel = ColSpec(pl.String, optional=True)

Tickets.validate(pl.DataFrame({"status": [None, "closed"]}), fill_defaults=True)
# status: ["open", "closed"]; channel may be absent

A default must be a value its column accepts — its dtype, bounds, choices, format, pattern, lengths and validators — and one that is not is refused where it is declared. Filling is opt-in: a spec that gains a default changes nothing until asked, and without it a null is still a finding.

Data read as text

parse_text=True (polspec validate always asks it) reads each column that arrived as text, where the spec declares a type text can hold, as that type before anything is checked: a Boolean from true or false in any case, an integer, a float, a Date, Datetime or Time, a Decimal, a Duration. It is for a frame read with its types switched off — the frame a profile with parse_text=True was made from validates against it this way. Added in 1.2.

import polspec

Loans = polspec.TableSpec("Loans", {"LGD": ColSpec(pl.Float64, nullable=True)})
raw = pl.DataFrame({"LGD": ["0.45", "", "0.2"]})   # read as text

polspec.validate(Loans, raw, parse_text=True)       # LGD as Float64: [0.45, None, 0.2]

Text is stripped and empty text is a null, so fill_defaults=True fills it. A column is parsed only when every present value parses. Otherwise it stays text, and its dtype finding counts the values that did not and gives some as its samples:

Column 'LGD': expected dtype Float64, got String: 1 value(s) do not parse as Float64, such as ['N/A']

A Date, Datetime or Time column with a text_format is read with it, and a value written another way is reported by name:

Bonds = polspec.TableSpec(
    "Bonds", {"maturity": ColSpec(pl.Date, text_format="%d %b %Y")}
)
report = polspec.inspect(
    Bonds, pl.DataFrame({"maturity": ["30 Sep 2031", "2031-09-30"]}), parse_text=True
)
print(report.findings[0].message)
Column 'maturity': expected dtype Date, got String: 1 value(s) do not parse as Date written as '%d %b %Y', such as ['2031-09-30']

Without a text_format, a date, datetime or time is read in whatever form Polars recognises. text_format added in 1.6.

An integer is read as an Int64, so a value past a narrower declared width is reported as any wider integer standing in is: see the next section. The frame comes back with the parsed columns; add cast=True for the declared widths.

Dtype strictness

By default polspec accepts what a real pipeline delivers: any integer width for a declared integer, an integer or float for a declared float, and String/Categorical for a declared Enum. A temporal is held to its kind: a Date and a Datetime stand in for each other, since text formats hand one back as the other, while a Time stands in only for a Time and a Duration only for a Duration, of any unit. A zoned Datetime and a naive one do not stand in for each other — one is an instant, the other a wall-clock reading — but two zones do, being the same instants shown differently. strict_dtypes=True requires the exact dtype, treating only String and Utf8 as interchangeable.

A wider dtype standing in still has to hold only values the declared one can. An Int64 column where an Int8 is declared passes with values from -128 to 127, and a value outside that is a dtype finding with its rows and samples; so is a float or integer too large for a declared Decimal's integer digits, after rounding to its scale. A declared bound has to fit its dtype, so on a side a bound closes such a value is out of bounds as well, and reported only as that — unless bounds=False. That is what makes a frame with no findings safe to hand to cast=True. Floats are left alone: a Float64 too large for a declared Float32 casts to an infinity rather than failing.

Casting

cast=True casts each column to its declared dtype after validation passes, so a String column that holds only valid enum members comes back as the Enum.

Streaming

streaming=True evaluates with the Polars streaming engine, for frames larger than memory.

The schema alone

schema_only=True checks what a frame's schema says — missing, extra and optional columns, and each column's dtype — and reads no rows. On a LazyFrame the query is planned, never run, so a test can hold a lazy pipeline's output shape to its spec for free:

source = pl.LazyFrame(schema=Orders.spec.schema())   # pl.scan_parquet(...) in practice
pipeline = source.with_columns(pl.col("total") * 2)
Orders.validate(pipeline, schema_only=True)   # planned, never collected

Added in 1.4. It does not combine with parse_text=True, which reads the values to decide their type.

Turning checks off

Each switch disables a whole category of check. Most exist for what generation cannot satisfy — __checks__, validators and pattern are validated but not generated, and __rows__ is not a limit on generate():

Orders.validate(Orders.generate(1_000, seed=1), checks=False)

bounds=False is the other way round: generation always stays in bounds, so it is for real data — a file whose ranges you want to look at before holding it to them, while every other claim is still checked. It covers every bounds, including a List's elements and a struct's fields; string_length and list_length have codes of their own and stay on.

Orders.validate(df, bounds=False)

To loosen one column rather than all of them, validate against a spec with that column's bounds removed:

from polspec import validate

loose = Orders.spec.with_columns(total=Orders.col("total").replace(bounds=None))
validate(loose, df)

What gets checked

Check From
Column present / not extra the spec's column set
Dtype compatible ColSpec.dtype
No unexpected nulls nullable
Value in domain choices, Enum categories
Value within range bounds
Length within range string_length
List has the declared number of elements list_length
No NaN where none is declared nan_probability
Value has the declared shape format
Value matches the regex pattern
Conditional values hold rules
Single-column predicates validators
Values distinct unique
Composite key distinct __unique_together__
Multi-column invariants __checks__
Rows in the declared order __order_by__
Number of rows within range __rows__
Referential integrity __foreign_keys__
One parent, bounded depth, no cycles __hierarchy__

Bounds, lengths, rules, validators and uniqueness are skipped for a column whose dtype is already wrong — comparing values of the wrong type would bury the dtype error under noise.

Foreign keys need their parent

A key referencing another spec needs that spec's data:

orders = Orders.generate(500, seed=2)
order_lines = OrderLines.generate(800, seed=3, references={Orders: orders})

OrderLines.validate(order_lines, references={Orders: orders})

Without it, the key is reported as a foreign_key_unresolved finding naming the spec it needed, so validate() raises and inspect() lists it alongside everything else. references may be keyed by the class, its TableSpec, or the spec's name. Self-referencing keys are checked against the frame itself and need nothing.

Each foreign key is an anti-join against the parent, so these run separately from the single-pass aggregation above.