Skip to content

Drift

polspec.drift.diff() compares two declarations; polspec.drift.drift() compares a declaration to a frame. Both return a DriftReport, and both are reachable from a spec as Orders.diff(...) and Orders.drift(df) -- see Schema and data drift.

diff

diff 

diff(old: TableSpec | type, new: TableSpec | type, *, renames: Mapping[str, str] | None = None, options: DriftOptions | None = None) -> DriftReport

What changed between two declarations, as a DriftReport.

Parameters:

Name Type Description Default
old TableSpec | FrameSpec class

The two declarations. A spec's name is not compared: the report is about what the two say, not what they are called.

 required
new TableSpec | FrameSpec class

The two declarations. A spec's name is not compared: the report is about what the two say, not what they are called.

 required
renames mapping

Columns renamed between the two, {old_name: new_name}. Applied to old first, through TableSpec.rename, so a rename is reported as column_renamed rather than as a column removed and another added. The caller asserts the rename; nothing is guessed from similar names.

 None
options DriftOptions

Only strict_dtypes is read when diffing two specs.

 None
Notes

A finding is breaking when a frame that satisfied old could fail new: a narrowed domain, a dropped nullability, a column added (a frame that lacks it fails missing_cols="raise"), a constraint added. A widened domain or a removed constraint is compatible.

drift

drift 

drift(spec: TableSpec | type, df: Frame, *, options: DriftOptions | None = None, **option_kwargs: Any) -> DriftReport

How df has moved relative to what spec declares, as a DriftReport.

Parameters:

Name Type Description Default
spec TableSpec | FrameSpec class

The declaration.

 required
df DataFrame | LazyFrame

The frame. A LazyFrame is collected: every measurement here is a summary of the whole column.

 required
options DriftOptions

Every option at once. Cannot be combined with the keywords.

 None
**option_kwargs Any

The fields of DriftOptions, one at a time.

 {}
Notes

A finding is breaking when this frame fails this spec on that column: values outside the domain, a bound exceeded, nulls where none are allowed, a format not matched. A null rate that moved within a nullable column, or declared values the data never holds, is compatible -- the data still validates; the declaration has stopped describing it well.

Uniqueness, composite keys, foreign keys and checks are not measured here; they are pass/fail claims that validate() already reports.

DriftOptions

DriftOptions dataclass 

DriftOptions(null_rate_tolerance: float = 0.05, unseen_values: bool = True, strict_dtypes: bool = False, max_samples: int = 10)

What counts as drift, said once.

Parameters:

Name Type Description Default
null_rate_tolerance float

How far the observed null rate may sit from a nullable column's null_probability before null_rate_moved is reported. Absolute, not relative: a relative tolerance is unstable near zero.

 0.05
unseen_values bool

Whether to report declared choices or Enum categories the data never holds (cardinality_moved).

 True
strict_dtypes bool

The same switch as ValidationOptions.strict_dtypes, and decided by the same function: whether a dtype_changed is breaking.

 False
max_samples int

How many offending values a finding's details carry.

 10

DriftReport

DriftReport dataclass 

DriftReport(kind: Kind, old: str, new: str, findings: tuple[DriftFinding, ...], options: DriftOptions | None = None)

Every difference between two specs, or between a spec and a frame.

kind says which: "diff" compares old to new, both declarations; "drift" compares the declaration old to data, and new is what the data was called. bool(report) is report.unchanged, the way bool(ValidationReport) is passed.

unchanged property 

unchanged: bool

True when nothing differs.

breaking property 

breaking: tuple[DriftFinding, ...]

The findings a CI gate should fail on.

by_column 

by_column() -> dict[str, tuple[DriftFinding, ...]]

Findings grouped by column; table-level findings under "".

by_code 

by_code(code: DriftCode) -> tuple[DriftFinding, ...]

Every finding of one kind, such as "domain_narrowed".

to_dict 

to_dict() -> dict[str, Any]

This report as JSON-ready data.

to_json 

to_json(*, indent: int | None = 2) -> str

This report as a JSON string. indent=None for one line.

to_markdown 

to_markdown(path: str | Path | None = None) -> str

This report as Markdown, written to path if given.

DriftFinding

DriftFinding dataclass 

DriftFinding(code: DriftCode, severity: Severity, key: str, message: str, columns: tuple[str, ...] = (), details: Mapping[str, Any] = dict())

One difference between two declarations, or a declaration and data.

Attributes:

Name Type Description
code DriftCode

Which kind of difference.
severity 'breaking' | 'compatible'

Breaking when data that satisfied the old side could fail the new one; compatible otherwise.
key str

A stable identifier within the report, such as "total__bounds" or "check:total_covers_subtotal".
message str

What changed, naming the column, and what to do about it.
columns tuple[str, ...]

The columns involved; empty for table-level differences.
details Mapping

Code-specific facts: the old and new value, how far a bound was exceeded, which values were new.

to_dict 

to_dict() -> dict[str, Any]

This finding as JSON-ready data.