Validation¶
read() brings a file in, in a spec's terms; inspect() returns a
report on it and validate() raises one. Both carry the same findings as
data — see Validating data. Each takes a
TableSpec or a FrameSpec class as its spec.
read¶
read
¶
read(spec: TableSpec | type, path: str | PathLike[str], **reader_options: Any) -> DataFrame
A data file as a frame, in the terms spec declares -- read, not
validated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
TableSpec | FrameSpec class
|
The declaration the file should meet. |
required |
path
|
str | PathLike
|
The file. Its extension picks the reader: |
required |
**reader_options
|
Any
|
Passed to the Polars reader, e.g. |
{}
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The file as Polars reads it, with one change: each column the spec
declares as a |
Raises:
| Type | Description |
|---|---|
ValueError
|
For an extension polspec does not know. |
Examples:
>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> _ = (folder / "orders.csv").write_text("id,placed\n1,2024-01-31\n2,2024-02-01\n")
>>> spec = TableSpec("Orders", {"id": pl.Int64, "placed": pl.Date})
>>> read(spec, folder / "orders.csv").schema # the date parsed, as declared
Schema({'id': Int64, 'placed': Date})
inspect¶
inspect
¶
inspect(spec: SpecLike, df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **option_kwargs: Unpack[ValidationKeywords]) -> ValidationReport
Everything spec has to say about df, as a ValidationReport.
Never raises for a frame that fails: every violation is a Finding on
the report, with report.rows(finding) and report.failing_rows()
giving the offending rows back lazily. See validate for the options.
Examples:
>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> report = inspect(spec, pl.DataFrame({"total": [10.0, 250.0, 30.0]}))
>>> report.passed
False
>>> [(finding.code, finding.count) for finding in report.findings]
[('bounds', 1)]
>>> report.failing_rows().collect()["total"].to_list()
[250.0]
>>> report.passing_rows().collect()["total"].to_list()
[10.0, 30.0]
validate¶
validate
¶
validate(spec: SpecLike, df: DataFrame, *, options: ValidationOptions | None = None, references: References = None, **option_kwargs: Unpack[ValidationKeywords]) -> DataFrame
validate(spec: SpecLike, df: LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **option_kwargs: Unpack[ValidationKeywords]) -> LazyFrame
validate(spec: SpecLike, df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **option_kwargs: Unpack[ValidationKeywords]) -> DataFrame | LazyFrame
Validates a DataFrame or LazyFrame against spec.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame | LazyFrame
|
The frame to validate. A LazyFrame comes back as a LazyFrame. |
required |
options
|
ValidationOptions
|
Every option at once, as a value -- useful for passing one setting through several calls. Cannot be combined with the keywords below. |
None
|
references
|
mapping
|
Parent frames for foreign keys that reference another spec, keyed by
that spec, its FrameSpec class, or its name. A key with no entry is
reported as a |
None
|
**option_kwargs
|
Unpack[ValidationKeywords]
|
The fields of |
{}
|
Returns:
| Type | Description |
|---|---|
The validated, optionally transformed frame -- a LazyFrame if `df` was one.
|
|
Raises:
| Type | Description |
|---|---|
ValidationError
|
Carrying a |
TypeError
|
For an option name this does not accept. |
ValueError
|
For an accepted option given a value outside its choices. |
Examples:
>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> validate(spec, pl.DataFrame({"total": [10.0, 20.0]})).height
2
>>> try:
... validate(spec, pl.DataFrame({"total": [250.0]}))
... except ValidationError as error:
... print([finding.code for finding in error.report.findings])
...
['bounds']
>>> validate(spec, pl.DataFrame({"total": ["10.5"]}), parse_text=True).schema
Schema({'total': Float64})
ValidationOptions¶
ValidationOptions
dataclass
¶
ValidationOptions(*, extra_cols: Literal['drop', 'allow', 'raise'] = 'raise', missing_cols: Literal['add', 'allow', 'raise'] = 'raise', strict_dtypes: bool = False, rules: bool = True, derived: bool = True, validators: bool = True, unique: bool = True, checks: bool = True, foreign_keys: bool = True, hierarchy: bool = True, order: bool = True, rows: bool = True, pattern: bool = True, bounds: bool = True, cast: bool = False, fill_defaults: bool = False, parse_text: bool = False, streaming: bool = False, schema_only: bool = False, tolerate: Tolerance = None)
Which checks to run and what to do about structural mismatches.
Grouped rather than passed as ten separate arguments, so adding a check
does not widen every signature between here and FrameSpec.validate.
tolerate accepts findings up to a share of the rows, so a pipeline
can fail only above, say, 0.1% bad rows: one share for every finding
(0.001), or a mapping by finding code ({"nullability": 0.01}), by column
({"total": 0.001}) or by both ({("total", "bounds"): 0.01}). The
most specific key wins. A tolerated finding stays in the report, marked
tolerated, and does not fail it; a finding with no share, such as a
missing column or a wrong dtype, is never tolerated. A key that names
neither a code nor a column of the spec is refused, as is one that
names both. Added in 1.7.
Examples:
>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> lenient = ValidationOptions(extra_cols="drop", bounds=False)
>>> validate(spec, pl.DataFrame({"total": [250.0], "note": ["x"]}), options=lenient).columns
['total']
>>> frame = pl.DataFrame({"total": [250.0] + [10.0] * 999})
>>> inspect(spec, frame, tolerate={"bounds": 0.001}).passed
True
ValidationReport¶
ValidationReport
dataclass
¶
ValidationReport(spec_name: str, findings: tuple[Finding, ...], frame: LazyFrame, options: ValidationOptions = FILLED_IN, _recheck: Callable[[LazyFrame], ValidationReport] | None = None, _min_rows: int = 0, _settled: dict[str, dict[str, DataFrame]] = dict())
Every finding for one frame against one spec.
A collection of findings: len() counts them, iterating yields them, and
like any collection it is falsy when empty -- so if report: means
"something was found". report.passed says the opposite in words.
Examples:
>>> spec = TableSpec("Orders", {"total": ColSpec(pl.Float64, bounds=(0, 100))})
>>> report = inspect(spec, pl.DataFrame({"total": [10.0, 250.0]}))
>>> report.by_code("bounds")[0].count
1
>>> report.to_dict()["passed"]
False
failures
property
¶
failures: tuple[Finding, ...]
The findings that fail the report: every one tolerate= does not
accept. Added in 1.7.
tolerated
property
¶
tolerated: tuple[Finding, ...]
The findings tolerate= accepts, which stay in the report and do
not fail it. Added in 1.7.
raise_if_failed
¶
Raises ValidationError carrying this report, if it failed.
by_column
¶
by_column() -> dict[str, tuple[Finding, ...]]
Findings grouped by column; structural findings under "".
by_code
¶
by_code(code: FindingCode) -> tuple[Finding, ...]
Every finding of one kind, such as "bounds" or "foreign_key".
failing_rows
¶
Every row that violates a row-level finding, lazily, with a
__polspec_finding column (FINDING_COLUMN) naming what it broke.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
once
|
bool
|
By default a row violating several claims appears once per
claim, its finding column naming one -- the shape to group or
filter by claim. With |
False
|
Raises:
| Type | Description |
|---|---|
ValueError
|
When the frame already has a column named |
passing_rows
¶
The rows that pass: a frame the spec accepts, lazily and in the frame's order.
First every row no finding touched. Some claims are about rows in
relation to other rows -- an order, a key into the frame's own rows
-- and taking the found rows away can break them in the rows that
are left: a row whose parent was removed, rows no longer in order.
So what is left is validated again, and any row that now fails is
taken away too, until nothing more is found. With
failing_rows(once=True) -- which names those later rows by the
claim they broke -- the two split a frame exactly.
Raises ValueError when a structural finding (a wrong dtype,
missing_columns, extra_columns, foreign_key_unresolved,
row_count) is in the report: that is a verdict on the whole frame,
not on rows, so there is no row-level answer to give. A dtype
finding about values the declared dtype cannot hold has rows, and
does not raise. It raises too when fewer rows pass than the spec's
rows allow, as no frame of what passed would be accepted.
to_dict
¶
This report as JSON-ready data: its schema_version, the spec, the
verdict, the findings.
to_json
¶
This report as a JSON string. indent=None for one line.
Finding¶
Finding
dataclass
¶
Finding(*, code: FindingCode, key: str, message: str, columns: tuple[str, ...] = (), count: int | None = None, share: float | None = None, samples: tuple[Any, ...] = (), details: Mapping[str, Any] = dict(), tolerated: bool = False, _locate: Callable[[LazyFrame], LazyFrame] | None = None)
One violation of one claim the spec makes.
Attributes:
| Name | Type | Description |
|---|---|---|
code |
FindingCode
|
Which kind of claim was violated. |
key |
str
|
A stable identifier for the claim within its spec, such as
|
message |
str
|
The human-readable description of what was violated. |
columns |
tuple[str, ...]
|
The columns involved; empty for structural findings. |
count |
int | None
|
How many rows violate the claim; |
share |
float | None
|
|
samples |
tuple
|
Up to five offending values (or structs of values, for multi-column claims). |
details |
Mapping
|
Code-specific facts: the expected and actual dtype, the observed extremes, the foreign key's target. |
tolerated |
bool
|
Whether |
Examples:
FindingCode¶
FindingCode
module-attribute
¶
FindingCode = Literal['extra_columns', 'missing_columns', 'dtype', 'nullability', 'nan', 'choices', 'bounds', 'string_length', 'list_length', 'format', 'pattern', 'rule', 'derived', 'validator', 'unique', 'unique_together', 'check', 'foreign_key', 'foreign_key_unresolved', 'hierarchy_multi_parent', 'hierarchy_cycle', 'hierarchy_depth', 'order', 'row_count']
The code of a Finding: what kind of claim it found broken. The full list,
with what raises each, is on the Errors and findings page.
FINDING_COLUMN¶
FINDING_COLUMN
module-attribute
¶
The column failing_rows() adds, naming the claim each row broke.