Specs¶

A spec is a TableSpec: an immutable record of columns and constraints. FrameSpec is the class syntax that builds one and forwards every verb to it.

TableSpec¶

TableSpec `dataclass` ¶

TableSpec(name: str, columns: Mapping[str, ColSpec] = dict(), checks: Sequence[Check] = (), unique_together: Sequence[Sequence[str]] = (), foreign_keys: Sequence[ForeignKey] = (), hierarchy: Hierarchy | None = None)

The columns and constraints of one table, as an immutable value.

Parameters:

Name	Type	Description	Default
`name`	`str`	What the table is called: the class name for a `FrameSpec`, the `name:` key for a file. Foreign keys refer to a spec by this name.	required
`columns`	`Mapping[str, ColSpec]`	Column name to declaration, in the order columns should appear.	`dict()`
`checks`	`Sequence[Check]`	Multi-column invariants; see `FrameSpec.__checks__`.	`()`
`unique_together`	`Sequence[Sequence[str]]`	Composite unique keys. A single group may be given as a flat list.	`()`
`foreign_keys`	`Sequence[ForeignKey]`	Referential-integrity constraints.	`()`
`hierarchy`	`Hierarchy \| None`	Declares two columns as a parent/child edge list; see `Hierarchy`.	`None`

Notes

Everything a FrameSpec class body validates at declaration is validated here, so a TableSpec that constructs is one that can be used.

resolve_target ¶

resolve_target(fk: ForeignKey) -> TableSpec | None

The spec a foreign key points at.

This spec for "self", the bound target when the key was declared against a spec object, None for a bare name nothing has resolved.

schema ¶

schema() -> Schema

The Polars schema this spec declares: column name to dtype.

estimated_size ¶

estimated_size(n: int) -> int

Bytes a frame of n generated rows is expected to hold.

Read off the declaration -- the width of each dtype, the lengths the spec declares -- so it costs nothing and needs no data. It measures the frame: generation holds working buffers on top, most visibly for Decimal and List, so a process peak is higher.

Orders.spec.estimated_size(1_000_000) / 1024**2 # doctest: +SKIP 27.5

tag ¶

tag(*tags: str | Sequence[str], match: Literal['any', 'all'] = 'any') -> list[str]

Column names carrying any (or all) of the tags, in declaration order.

with_name ¶

with_name(name: str) -> TableSpec

A copy of this spec under a different name.

with_columns ¶

with_columns(mapping: Mapping[str, ColSpec] | None = None, /, **columns: ColSpec) -> TableSpec

Adds columns, or replaces existing ones of the same name in place.

with_checks ¶

with_checks(*checks: Check) -> TableSpec

A copy of this spec with checks added to the ones it has.

with_foreign_keys ¶

with_foreign_keys(*foreign_keys: ForeignKey) -> TableSpec

A copy of this spec with foreign_keys added to the ones it has.

with_hierarchy ¶

with_hierarchy(hierarchy: Hierarchy | None) -> TableSpec

A copy of this spec declaring (or, with None, dropping) a hierarchy.

with_unique_together ¶

with_unique_together(*groups: Sequence[str]) -> TableSpec

A copy of this spec with groups added as composite unique keys.

drop ¶

drop(*names: str) -> TableSpec

Removes columns, and any composite or foreign key that used them.

select ¶

select(*names: str) -> TableSpec

Keeps only the named columns, in the order given.

rename ¶

rename(mapping: Mapping[str, str]) -> TableSpec

Renames columns, rewriting every constraint that names them.

Rules, composite keys and foreign keys are rewritten. A column carrying validators cannot be renamed: a validator is a Polars expression that names the column, and rewriting expressions is not something this library does.

with_catspec ¶

with_catspec(catspec: CatSpec | type[CatSpec]) -> TableSpec

Re-points columns at the registry's Enum and Categorical types.

A column whose name resolves in the registry (exactly, or by case) takes the registry's dtype; everything else it declared carries over. choices a new Enum cannot hold, and weights over a domain that changed size, are dropped with a warning.

Takes the registry as a value or as the class body that declares it.

FrameSpec¶

FrameSpec ¶

Base class for declaring a DataFrame/LazyFrame specification.

Subclass it and assign a ColSpec per column, in the order columns should appear:

class DataSource(FrameSpec):
    string_1 = ColSpec(pl.String)
    enum_1 = ColSpec(pl.Enum(["mammal", "reptile"]), nullable=True)
    int_1 = ColSpec(pl.Int64, bounds=(-100, 100), nullable=True)

df = DataSource.generate(1_000_000, seed=42)

The class body builds DataSource.spec, a TableSpec; every classmethod here forwards to a function over it. A column may take any name: one that collides with a method (schema, tag, ...) is reachable as DataSource.col("schema") while the method keeps working. Names that cannot be attributes at all -- a leading underscore, or one straight from data -- go through __columns__:

class Raw(FrameSpec):
    __columns__ = {"_id": ColSpec(pl.Int64), "Unit Price": ColSpec(pl.Float64)}

from_spec `classmethod` ¶

from_spec(spec: TableSpec, *, name: str | None = None) -> type[FrameSpec]

A FrameSpec subclass wrapping an existing TableSpec.

from_yaml `classmethod` ¶

from_yaml(source: str | Path, *, categories: CatSpec | type[CatSpec] | str | Path | None = None, strict: bool = True) -> type[FrameSpec]

Builds a new FrameSpec subclass from a YAML file written by to_yaml.

categories is a CatSpec registry, or a path to one, used to resolve shared Enums and Categoricals; when omitted, a categories: key in the file is loaded automatically. An unknown key in the file is an error unless strict=False, which downgrades it to a warning.

from_dataframe `classmethod` ¶

from_dataframe(df: DataFrame, *, name: str = 'ProfiledFrameSpec', weights: bool = False, max_unique_enum: int = 50, calculate_bounds: bool = True) -> type[FrameSpec]

Infers a spec by profiling an existing DataFrame.

weights=True records empirical frequencies for categorical, enum and boolean columns. A string or categorical column with at most max_unique_enum distinct values becomes an Enum. calculate_bounds records observed (min, max) for numeric and temporal columns and (min_len, max_len) for strings and binary.

col `classmethod` ¶

col(name: str) -> ColSpec

The declaration of one column, whatever it is called.

schema `classmethod` ¶

schema() -> Schema

The Polars schema this spec declares: column name to dtype.

tag `classmethod` ¶

tag(*tags: str | Sequence[str], match: Literal['any', 'all'] = 'any') -> list[str]

Column names carrying any (or all) of the tags, in declaration order.

checks `classmethod` ¶

checks() -> tuple[Check, ...]

The Check constraints defined on this FrameSpec.

foreign_keys `classmethod` ¶

foreign_keys() -> tuple[ForeignKey, ...]

The ForeignKey constraints defined on this FrameSpec.

unique_together `classmethod` ¶

unique_together() -> tuple[tuple[str, ...], ...]

The composite unique column groups defined on this FrameSpec.

catspec `classmethod` ¶

catspec() -> CatSpec

The CatSpec registry this spec's Enum and Categorical columns imply.

with_catspec `classmethod` ¶

with_catspec(catspec: CatSpec | type[CatSpec], *, name: str | None = None) -> type[FrameSpec]

A new FrameSpec subclass with columns re-typed against catspec.

to_yaml `classmethod` ¶

to_yaml(source: str | Path) -> None

Writes this spec to a human-readable YAML file at source.

to_python `classmethod` ¶

to_python(source: str | Path) -> None

Writes this spec as an importable Python module defining a subclass.

generate `classmethod` ¶

generate(n: int, *, method: Literal['random', 'cartesian'] = 'random', seed: int | None = None, references: References = None, cycles: int = 0, self_references: int = 0, max_bytes: int | None = None) -> DataFrame

Generates a DataFrame matching this spec.

cycles and self_references apply only to a spec declaring a __hierarchy__, and deliberately violate it. See polspec.generation.generate for the full contract. scan() is the lazy verb.

estimated_size `classmethod` ¶

estimated_size(n: int) -> int

Bytes a frame of n generated rows is expected to hold.

See TableSpec.estimated_size. generate() warns when the estimate passes four gibibytes, and takes max_bytes= to refuse instead.

scan `classmethod` ¶

scan(n: int, *, seed: int | None = None, batch_size: int | None = None, method: Literal['random', 'cartesian'] = 'random', references: References = None) -> LazyFrame

A LazyFrame of n rows, generated as they are collected.

Unlike generate(), which builds the whole frame before it returns, nothing is generated until the plan is collected -- and then only the columns and rows the plan asks for:

Orders.scan(50_000_000, seed=1).sink_parquet("orders.parquet")
Orders.scan(50_000_000, seed=1).select("total").head(5).collect()

The first streams in bounded memory; the second generates five rows of one column. A projected column holds the values the whole frame would: every column is seeded by its name, so dropping its neighbours cannot move it.

Rows come in batches, so a scan carries generate_batches' terms: a spec declaring a __hierarchy__ is refused, and a unique=True column or a __unique_together__ group is distinct within a batch rather than across n. Leaving batch_size unset lets polars ask for the size it wants; setting it pins the size.

generate_batches `classmethod` ¶

generate_batches(n: int, *, batch_size: int = 100000, method: Literal['random', 'cartesian'] = 'random', seed: int | None = None, references: References = None) -> Iterator[DataFrame]

Yields chunks of generated rows without holding all n in memory.

Each batch is a window onto the one frame seed describes: a column no pass rewrites holds, batch by batch, exactly the rows generate(n, seed=seed) would, whatever batch_size is.

sink_parquet `classmethod` ¶

sink_parquet(path: str | Path, n: int, *, batch_size: int = 100000, compression: ParquetCompression = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, **kwargs: Any) -> None

Generates n rows and streams them to a Parquet file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_parquet.

sink_csv `classmethod` ¶

sink_csv(path: str | Path, n: int, *, batch_size: int = 100000, include_header: bool = True, method: Method = 'random', seed: int | None = None, references: References = None, **kwargs: Any) -> None

Generates n rows and streams them to a CSV file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_csv.

sink_ipc `classmethod` ¶

sink_ipc(path: str | Path, n: int, *, batch_size: int = 100000, compression: IpcCompression | None = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, **kwargs: Any) -> None

Generates n rows and streams them to an Arrow IPC file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_ipc.

sink_ndjson `classmethod` ¶

sink_ndjson(path: str | Path, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None, **kwargs: Any) -> None

Generates n rows and streams them to an NDJSON file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_ndjson.

diff `classmethod` ¶

diff(other: TableSpec | type[FrameSpec], *, renames: Mapping[str, str] | None = None, options: DriftOptions | None = None) -> DriftReport

What changed from this spec to other, as a DriftReport.

This spec is the old side. See polspec.drift.diff.

drift `classmethod` ¶

drift(df: DataFrame | LazyFrame, *, options: DriftOptions | None = None, null_rate_tolerance: float | None = None, unseen_values: bool | None = None, strict_dtypes: bool | None = None, max_samples: int | None = None) -> DriftReport

How df has moved relative to this spec, as a DriftReport.

Each keyword left as None takes the DriftOptions default. See polspec.drift.drift.

to_markdown `classmethod` ¶

to_markdown(path: str | Path | None = None, *, title: str | None = None) -> str

A Markdown data dictionary for this spec, written to path if given.

to_mermaid `classmethod` ¶

to_mermaid(path: str | Path | None = None, *, title: str | None = None) -> str

A Mermaid entity-relationship diagram for this spec.

inspect `classmethod` ¶

inspect(df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, extra_cols: Literal['drop', 'allow', 'raise'] | None = None, missing_cols: Literal['add', 'allow', 'raise'] | None = None, strict_dtypes: bool | None = None, validate_rules: bool | None = None, validate_validators: bool | None = None, validate_unique: bool | None = None, validate_checks: bool | None = None, validate_foreign_keys: bool | None = None, validate_hierarchy: bool | None = None, validate_pattern: bool | None = None, validate_bounds: bool | None = None, cast: bool | None = None, streaming: bool | None = None) -> ValidationReport

Everything this spec has to say about df, as a ValidationReport.

Never raises for a frame that fails: each violation is a Finding with a code, a count, samples and, for row-level findings, the offending rows reachable lazily through report.rows(finding) or report.failing_rows(). Takes the same options as validate.

validate `classmethod` ¶

validate(df: DataFrame, *, options: ValidationOptions | None = None, references: References = None, extra_cols: Literal['drop', 'allow', 'raise'] | None = None, missing_cols: Literal['add', 'allow', 'raise'] | None = None, strict_dtypes: bool | None = None, validate_rules: bool | None = None, validate_validators: bool | None = None, validate_unique: bool | None = None, validate_checks: bool | None = None, validate_foreign_keys: bool | None = None, validate_hierarchy: bool | None = None, validate_pattern: bool | None = None, validate_bounds: bool | None = None, cast: bool | None = None, streaming: bool | None = None) -> DataFrame

validate(df: LazyFrame, *, options: ValidationOptions | None = None, references: References = None, extra_cols: Literal['drop', 'allow', 'raise'] | None = None, missing_cols: Literal['add', 'allow', 'raise'] | None = None, strict_dtypes: bool | None = None, validate_rules: bool | None = None, validate_validators: bool | None = None, validate_unique: bool | None = None, validate_checks: bool | None = None, validate_foreign_keys: bool | None = None, validate_hierarchy: bool | None = None, validate_pattern: bool | None = None, validate_bounds: bool | None = None, cast: bool | None = None, streaming: bool | None = None) -> LazyFrame

validate(df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, extra_cols: Literal['drop', 'allow', 'raise'] | None = None, missing_cols: Literal['add', 'allow', 'raise'] | None = None, strict_dtypes: bool | None = None, validate_rules: bool | None = None, validate_validators: bool | None = None, validate_unique: bool | None = None, validate_checks: bool | None = None, validate_foreign_keys: bool | None = None, validate_hierarchy: bool | None = None, validate_pattern: bool | None = None, validate_bounds: bool | None = None, cast: bool | None = None, streaming: bool | None = None) -> DataFrame | LazyFrame

Validates a DataFrame or LazyFrame against this spec.

Raises ValidationError carrying a ValidationReport of every violation, or returns the (optionally transformed) frame -- a LazyFrame if df was one. Use inspect for the report without the exception.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame \| LazyFrame`	The frame to validate.	required
`options`	`ValidationOptions`	Every option at once, as a value. Cannot be combined with the keywords below.	`None`
`references`	`mapping`	Parent frames for foreign keys into other specs, keyed by that spec, its FrameSpec class, or its name.	`None`
`extra_cols`	`Literal['drop', 'allow', 'raise'] \| None`	The structural options, named as on `ValidationOptions`.	`None`
`missing_cols`	`Literal['drop', 'allow', 'raise'] \| None`	The structural options, named as on `ValidationOptions`.	`None`
`strict_dtypes`	`Literal['drop', 'allow', 'raise'] \| None`	The structural options, named as on `ValidationOptions`.	`None`
`cast`	`Literal['drop', 'allow', 'raise'] \| None`	The structural options, named as on `ValidationOptions`.	`None`
`streaming`	`Literal['drop', 'allow', 'raise'] \| None`	The structural options, named as on `ValidationOptions`.	`None`
`validate_rules`	`bool \| None`	The check switches: `validate_` in front of the field they set.	`None`
`validate_validators`	`bool \| None`	The check switches: `validate_` in front of the field they set.	`None`
`validate_unique`	`bool \| None`	The check switches: `validate_` in front of the field they set.	`None`
`validate_checks`	`bool \| None`	The check switches: `validate_` in front of the field they set.	`None`
`validate_foreign_keys`	`bool \| None`	The check switches: `validate_` in front of the field they set.	`None`
`validate_hierarchy`	`bool \| None`	The check switches: `validate_` in front of the field they set.	`None`
`validate_pattern`	`bool \| None`	The check switches: `validate_` in front of the field they set.	`None`
`validate_bounds`	`bool \| None`	The check switches: `validate_` in front of the field they set.	`None`

Notes

A keyword left as None takes the ValidationOptions default; see that class for what every option means.

ForeignKey¶

ForeignKey `dataclass` ¶

ForeignKey(columns: str | Sequence[str], references: type[FrameSpec] | TableSpec | str, ref_columns: str | Sequence[str] | None = None, name: str | None = None, target: TableSpec | None = None)

Declares referential integrity: one or more columns must only contain values that exist in another FrameSpec's (or this same FrameSpec's) columns.

Parameters:

Name	Type	Description	Default
`columns`	`str \| Sequence[str]`	The local column(s) that must reference existing parent values.	required
`references`	`type[FrameSpec] \| TableSpec \| str`	The spec this key references -- a `FrameSpec` subclass, a `TableSpec`, or a spec's name -- or the literal string "self" for a self-referencing key (an `employee.manager_id` pointing back at `employee.id`). "self" always resolves to whichever spec the key ends up declared or inherited on, not the class it was first written in. After construction `references` is always a string: the target's name. When a spec object was given, it is kept as `target`, so its columns can be checked at declaration; a bare name has no `target` until a registry resolves it.	required
`ref_columns`	`str \| Sequence[str] \| None`	The referenced column(s) on the target, in the same order as `columns`. Defaults to `columns` (same names on both sides).	`None`
`name`	`str \| None`	A human-readable identifier. Defaults to a name derived from the columns and target.	`None`

Notes

Rows where any of columns is null are exempt (standard FK semantics -- a null foreign key means "no reference", not "an invalid one").

Examples:

>>> class OrderSpec(FrameSpec):
...     customer_id = ColSpec(pl.Int64)
...     __foreign_keys__ = [
...         ForeignKey("customer_id", references=CustomerSpec, ref_columns="id"),
...     ]
>>> class EmployeeSpec(FrameSpec):
...     id = ColSpec(pl.Int64, unique=True)
...     manager_id = ColSpec(pl.Int64, nullable=True)
...     __foreign_keys__ = [
...         ForeignKey("manager_id", references="self", ref_columns="id"),
...     ]

Hierarchy¶

Hierarchy `dataclass` ¶

Hierarchy(child: str, parent: str, max_depth: int = 1, branching: float | None = None, roots: int | None = None)

Declares that two columns of a spec form a parent/child edge list.

Parameters:

Name	Type	Description	Default
`child`	`str`	The column holding the lower reference -- the one doing the pointing. Each value appears in exactly one row, which is what gives every reference a single ultimate parent.	required
`parent`	`str`	The column holding the reference being pointed at.	required
`max_depth`	`int`	Hops from an ultimate parent to the furthest child. Generation guarantees at least one chain of exactly this length and none longer, so a walk that stops early and one that runs away are both caught.	`1`
`branching`	`float \| None`	Mean children per reference, which is what decides how many ultimate parents `n` rows imply. Mutually exclusive with `roots`; one of the two must be given, and `branching=3.0` is the default when neither is.	`None`
`roots`	`int \| None`	An exact number of ultimate parents, as an alternative to `branching`.	`None`

Notes

A row is an edge, so generate(n) produces n rows. Ultimate parents have no row of their own -- nothing to point at -- so n edges over R roots need n + R distinct references, drawn from the child column's own declaration.

Examples:

>>> class Links(FrameSpec):
...     PARENT_REF = ColSpec(pl.String)
...     CHILD_REF = ColSpec(pl.String)
...     __hierarchy__ = Hierarchy(
...         child="CHILD_REF", parent="PARENT_REF", max_depth=5
...     )

columns `property` ¶

columns: tuple[str, str]

The two columns this declaration writes, child first.

Specs¶

TableSpec¶

TableSpec dataclass ¶

resolve_target ¶

schema ¶

estimated_size ¶

tag ¶

with_name ¶

with_columns ¶

with_checks ¶

with_foreign_keys ¶

with_hierarchy ¶

with_unique_together ¶

drop ¶

select ¶

rename ¶

with_catspec ¶

FrameSpec¶

FrameSpec ¶

from_spec classmethod ¶

from_yaml classmethod ¶

from_dataframe classmethod ¶

col classmethod ¶

schema classmethod ¶

tag classmethod ¶

checks classmethod ¶

foreign_keys classmethod ¶

unique_together classmethod ¶

catspec classmethod ¶

with_catspec classmethod ¶

to_yaml classmethod ¶

to_python classmethod ¶

generate classmethod ¶

estimated_size classmethod ¶

scan classmethod ¶

generate_batches classmethod ¶

sink_parquet classmethod ¶

sink_csv classmethod ¶

sink_ipc classmethod ¶

sink_ndjson classmethod ¶

diff classmethod ¶

drift classmethod ¶

to_markdown classmethod ¶

to_mermaid classmethod ¶

inspect classmethod ¶

validate classmethod ¶

ForeignKey¶

ForeignKey dataclass ¶

Hierarchy¶

Hierarchy dataclass ¶

columns property ¶

TableSpec `dataclass` ¶

from_spec `classmethod` ¶

from_yaml `classmethod` ¶

from_dataframe `classmethod` ¶

col `classmethod` ¶

schema `classmethod` ¶

tag `classmethod` ¶

checks `classmethod` ¶

foreign_keys `classmethod` ¶

unique_together `classmethod` ¶

catspec `classmethod` ¶

with_catspec `classmethod` ¶

to_yaml `classmethod` ¶

to_python `classmethod` ¶

generate `classmethod` ¶

estimated_size `classmethod` ¶

scan `classmethod` ¶

generate_batches `classmethod` ¶

sink_parquet `classmethod` ¶

sink_csv `classmethod` ¶

sink_ipc `classmethod` ¶

sink_ndjson `classmethod` ¶

diff `classmethod` ¶

drift `classmethod` ¶

to_markdown `classmethod` ¶

to_mermaid `classmethod` ¶

inspect `classmethod` ¶

validate `classmethod` ¶

ForeignKey `dataclass` ¶

Hierarchy `dataclass` ¶

columns `property` ¶