Skip to content

Validation

read() brings a file in, in a spec's terms; inspect() returns a report on it and validate() raises one. Both carry the same findings as data — see Validating data. Each takes a TableSpec or a FrameSpec class as its spec.

read

read

read(spec: TableSpec | type, path: str | PathLike[str], **reader_options: Any) -> DataFrame

A data file as a frame, in the terms spec declares -- read, not validated.

Parameters:

Name Type Description Default
spec TableSpec | FrameSpec class

The declaration the file should meet.

required
path str | PathLike

The file. Its extension picks the reader: .csv, .tsv, .parquet/.pq, .ndjson/.jsonl, .json or .arrow/.ipc/.feather -- the set polspec validate reads.

required
**reader_options Any

Passed to the Polars reader, e.g. separator=";" for a CSV.

{}

Returns:

Type Description
DataFrame

The file as Polars reads it, with one change: each column the spec declares as a Boolean, integer, float, Date, Datetime, Time, Decimal or Duration that arrived as text -- itself, or as the elements of a list or the fields of a struct -- is parsed as what it declares, when every value in it parses exactly. A column of its own is stripped first, with empty text a null, so a file read with infer_schema=False comes back typed; an integer comes back as an Int64, and a float as a Float64. A Decimal is parsed from its digits, with no more decimal places than its scale; a Duration from the ISO 8601 text Polars writes ("PT3600S", "-PT0.5S", "P0D"). A column holding a value that does not parse stays text, so validation reports its dtype rather than a null -- or a rounded value -- that was never in the file. Nothing else is cast -- pass the frame to inspect() for every problem at once, or to validate(cast=True) for the typed frame.

Raises:

Type Description
ValueError

For an extension polspec does not know.

Examples:

>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> _ = (folder / "orders.csv").write_text("id,placed\n1,2024-01-31\n2,2024-02-01\n")
>>> spec = TableSpec("Orders", {"id": pl.Int64, "placed": pl.Date})
>>> read(spec, folder / "orders.csv").schema  # the date parsed, as declared
Schema({'id': Int64, 'placed': Date})

inspect

inspect

inspect(spec: SpecLike, df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **option_kwargs: Unpack[ValidationKeywords]) -> ValidationReport

Everything spec has to say about df, as a ValidationReport.

Never raises for a frame that fails: every violation is a Finding on the report, with report.rows(finding) and report.failing_rows() giving the offending rows back lazily. See validate for the options.

Examples:

>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> report = inspect(spec, pl.DataFrame({"total": [10.0, 250.0, 30.0]}))
>>> report.passed
False
>>> [(finding.code, finding.count) for finding in report.findings]
[('bounds', 1)]
>>> report.failing_rows().collect()["total"].to_list()
[250.0]
>>> report.passing_rows().collect()["total"].to_list()
[10.0, 30.0]

validate

validate

validate(spec: SpecLike, df: DataFrame, *, options: ValidationOptions | None = None, references: References = None, **option_kwargs: Unpack[ValidationKeywords]) -> DataFrame
validate(spec: SpecLike, df: LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **option_kwargs: Unpack[ValidationKeywords]) -> LazyFrame
validate(spec: SpecLike, df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **option_kwargs: Unpack[ValidationKeywords]) -> DataFrame | LazyFrame

Validates a DataFrame or LazyFrame against spec.

Parameters:

Name Type Description Default
df DataFrame | LazyFrame

The frame to validate. A LazyFrame comes back as a LazyFrame.

required
options ValidationOptions

Every option at once, as a value -- useful for passing one setting through several calls. Cannot be combined with the keywords below.

None
references mapping

Parent frames for foreign keys that reference another spec, keyed by that spec, its FrameSpec class, or its name. A key with no entry is reported as a foreign_key_unresolved finding.

None
**option_kwargs Unpack[ValidationKeywords]

The fields of ValidationOptions, one at a time, by their names -- checks=False, unique=False. See ValidationOptions for what each means and what it defaults to; an unknown name raises TypeError naming the closest match.

{}

Returns:

Type Description
The validated, optionally transformed frame -- a LazyFrame if `df` was one.

Raises:

Type Description
ValidationError

Carrying a ValidationReport of every violation.

TypeError

For an option name this does not accept.

ValueError

For an accepted option given a value outside its choices.

Examples:

>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> validate(spec, pl.DataFrame({"total": [10.0, 20.0]})).height
2
>>> try:
...     validate(spec, pl.DataFrame({"total": [250.0]}))
... except ValidationError as error:
...     print([finding.code for finding in error.report.findings])
...
['bounds']
>>> validate(spec, pl.DataFrame({"total": ["10.5"]}), parse_text=True).schema
Schema({'total': Float64})

ValidationOptions

ValidationOptions dataclass

ValidationOptions(*, extra_cols: Literal['drop', 'allow', 'raise'] = 'raise', missing_cols: Literal['add', 'allow', 'raise'] = 'raise', strict_dtypes: bool = False, rules: bool = True, derived: bool = True, validators: bool = True, unique: bool = True, checks: bool = True, foreign_keys: bool = True, hierarchy: bool = True, order: bool = True, rows: bool = True, pattern: bool = True, bounds: bool = True, cast: bool = False, fill_defaults: bool = False, parse_text: bool = False, streaming: bool = False, schema_only: bool = False, tolerate: Tolerance = None)

Which checks to run and what to do about structural mismatches.

Grouped rather than passed as ten separate arguments, so adding a check does not widen every signature between here and FrameSpec.validate.

tolerate accepts findings up to a share of the rows, so a pipeline can fail only above, say, 0.1% bad rows: one share for every finding (0.001), or a mapping by finding code ({"nullability": 0.01}), by column ({"total": 0.001}) or by both ({("total", "bounds"): 0.01}). The most specific key wins. A tolerated finding stays in the report, marked tolerated, and does not fail it; a finding with no share, such as a missing column or a wrong dtype, is never tolerated. A key that names neither a code nor a column of the spec is refused, as is one that names both. Added in 1.7.

Examples:

>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> lenient = ValidationOptions(extra_cols="drop", bounds=False)
>>> validate(spec, pl.DataFrame({"total": [250.0], "note": ["x"]}), options=lenient).columns
['total']
>>> frame = pl.DataFrame({"total": [250.0] + [10.0] * 999})
>>> inspect(spec, frame, tolerate={"bounds": 0.001}).passed
True

ValidationReport

ValidationReport dataclass

ValidationReport(spec_name: str, findings: tuple[Finding, ...], frame: LazyFrame, options: ValidationOptions = FILLED_IN, _recheck: Callable[[LazyFrame], ValidationReport] | None = None, _min_rows: int = 0, _settled: dict[str, dict[str, DataFrame]] = dict())

Every finding for one frame against one spec.

A collection of findings: len() counts them, iterating yields them, and like any collection it is falsy when empty -- so if report: means "something was found". report.passed says the opposite in words.

Examples:

>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> report = inspect(spec, pl.DataFrame({"total": [10.0, 250.0]}))
>>> report.by_code("bounds")[0].count
1
>>> report.to_dict()["passed"]
False

passed property

passed: bool

True when nothing was found, or only findings tolerate= accepts.

failures property

failures: tuple[Finding, ...]

The findings that fail the report: every one tolerate= does not accept. Added in 1.7.

tolerated property

tolerated: tuple[Finding, ...]

The findings tolerate= accepts, which stay in the report and do not fail it. Added in 1.7.

raise_if_failed

raise_if_failed() -> None

Raises ValidationError carrying this report, if it failed.

by_column

by_column() -> dict[str, tuple[Finding, ...]]

Findings grouped by column; structural findings under "".

by_code

by_code(code: FindingCode) -> tuple[Finding, ...]

Every finding of one kind, such as "bounds" or "foreign_key".

rows

rows(finding: Finding) -> LazyFrame

The rows violating one finding, lazily.

failing_rows

failing_rows(*, once: bool = False) -> LazyFrame

Every row that violates a row-level finding, lazily, with a __polspec_finding column (FINDING_COLUMN) naming what it broke.

Parameters:

Name Type Description Default
once bool

By default a row violating several claims appears once per claim, its finding column naming one -- the shape to group or filter by claim. With once=True each row appears once, in the frame's order, its finding column naming every claim it broke, comma-separated: the quarantine view, which with passing_rows() splits a frame exactly. polspec validate --failing writes this view.

False

Raises:

Type Description
ValueError

When the frame already has a column named FINDING_COLUMN, which these rows would lose.

passing_rows

passing_rows() -> LazyFrame

The rows that pass: a frame the spec accepts, lazily and in the frame's order.

First every row no finding touched. Some claims are about rows in relation to other rows -- an order, a key into the frame's own rows -- and taking the found rows away can break them in the rows that are left: a row whose parent was removed, rows no longer in order. So what is left is validated again, and any row that now fails is taken away too, until nothing more is found. With failing_rows(once=True) -- which names those later rows by the claim they broke -- the two split a frame exactly.

Raises ValueError when a structural finding (a wrong dtype, missing_columns, extra_columns, foreign_key_unresolved, row_count) is in the report: that is a verdict on the whole frame, not on rows, so there is no row-level answer to give. A dtype finding about values the declared dtype cannot hold has rows, and does not raise. It raises too when fewer rows pass than the spec's rows allow, as no frame of what passed would be accepted.

to_dict

to_dict() -> dict[str, Any]

This report as JSON-ready data: its schema_version, the spec, the verdict, the findings.

to_json

to_json(*, indent: int | None = 2) -> str

This report as a JSON string. indent=None for one line.

Finding

Finding dataclass

Finding(*, code: FindingCode, key: str, message: str, columns: tuple[str, ...] = (), count: int | None = None, share: float | None = None, samples: tuple[Any, ...] = (), details: Mapping[str, Any] = dict(), tolerated: bool = False, _locate: Callable[[LazyFrame], LazyFrame] | None = None)

One violation of one claim the spec makes.

Attributes:

Name Type Description
code FindingCode

Which kind of claim was violated.

key str

A stable identifier for the claim within its spec, such as "total__bounds" or "check:total_covers_subtotal".

message str

The human-readable description of what was violated.

columns tuple[str, ...]

The columns involved; empty for structural findings.

count int | None

How many rows violate the claim; None for structural findings.

share float | None

count as a fraction of the frame's rows, 0 to 1; None where count is. Added in 1.4.

samples tuple

Up to five offending values (or structs of values, for multi-column claims).

details Mapping

Code-specific facts: the expected and actual dtype, the observed extremes, the foreign key's target.

tolerated bool

Whether tolerate= accepts this finding: its share is at or under the tolerance given for it, so it does not fail the report. Added in 1.7.

Examples:

>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> finding = inspect(spec, pl.DataFrame({"total": [250.0]})).findings[0]
>>> finding.code, finding.columns, finding.count, finding.samples
('bounds', ('total',), 1, (250.0,))
>>> finding.share
1.0

row_level property

row_level: bool

Whether this finding can name the rows that violate it.

to_dict

to_dict() -> dict[str, Any]

This finding as JSON-ready data.

FindingCode

FindingCode module-attribute

FindingCode = Literal['extra_columns', 'missing_columns', 'dtype', 'nullability', 'nan', 'choices', 'bounds', 'string_length', 'list_length', 'format', 'pattern', 'rule', 'derived', 'validator', 'unique', 'unique_together', 'check', 'foreign_key', 'foreign_key_unresolved', 'hierarchy_multi_parent', 'hierarchy_cycle', 'hierarchy_depth', 'order', 'row_count']

The code of a Finding: what kind of claim it found broken. The full list, with what raises each, is on the Errors and findings page.

FINDING_COLUMN

FINDING_COLUMN module-attribute

FINDING_COLUMN = '__polspec_finding'

The column failing_rows() adds, naming the claim each row broke.