Skip to content

Specs

A spec is a TableSpec: an immutable record of columns and constraints. FrameSpec is the class syntax that builds one and forwards every verb to it.

TableSpec

TableSpec dataclass

TableSpec(name: str, columns: Mapping[str, ColSpec | DataType | type[DataType]] = FrozenDict(), *, description: str | None = None, rows: Bound[int] | tuple[int | None, int | None] | None = None, checks: Sequence[Check] = (), unique_together: Sequence[Sequence[str]] = (), foreign_keys: Sequence[ForeignKey] = (), hierarchy: Hierarchy | None = None, order_by: str | Sequence[str] = (), descending: bool | Sequence[bool] = False)

The columns and constraints of one table, as an immutable value.

Parameters:

Name Type Description Default
name str

What the table is called: the class name for a FrameSpec, the name: key for a file. Foreign keys refer to a spec by this name.

required
columns Mapping[str, ColSpec | DataType]

Column name to declaration, in the order columns should appear. A bare dtype is a column of it: {"id": pl.Int64} is {"id": ColSpec(pl.Int64)}.

FrozenDict()
description str | None

What the table holds, for a reader: to_markdown shows it, and neither generation nor validation reads it.

None
rows Bound[int] | tuple[int | None, int | None] | None

How many rows the table may hold, either end open: (1, None) for a table that must not be empty. validate() reports a row_count finding outside it. generate() makes the rows it is asked for, so a smaller frame than rows allows is validated with rows=False.

None
checks Sequence[Check]

Multi-column invariants; see FrameSpec.__checks__.

()
unique_together Sequence[Sequence[str]]

Composite unique keys. A single group may be given as a flat list.

()
foreign_keys Sequence[ForeignKey]

Referential-integrity constraints.

()
hierarchy Hierarchy | None

Declares two columns as a parent/child edge list; see Hierarchy.

None
order_by Sequence[str]

Columns the rows come sorted by, the first the major key, as DataFrame.sort(by) takes them: ["ts"] for one order across the frame, ["user_id", "ts"] for an order within each user. Nulls come last. generate() sorts its rows so, and validate() checks them.

()
descending bool | Sequence[bool]

Whether each order_by column runs high to low: one bool for all of them, or one per column.

False
Notes

Everything a FrameSpec class body validates at declaration is validated here, so a TableSpec that constructs is one that can be used.

Examples:

>>> spec = TableSpec("Orders", {
...     "id": ColSpec(pl.Int64, unique=True),
...     "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> spec
TableSpec('Orders', {
    'id': ColSpec(Int64, unique=True),
    'total': ColSpec(Float64, bounds=(0, 100)),
})
>>> spec.select("total").schema()
Schema({'total': Float64})
>>> spec.names.total
'total'

names property

names: ColumnNames

Each column's name, as an attribute: spec.names.total is "total", so code can name a column without spelling it as text.

from_schema classmethod

from_schema(schema: Any, *, name: str = 'Schema') -> TableSpec

A spec of schema's columns, each declaring nothing but its dtype.

schema is a pl.Schema, a mapping of column name to dtype, or anything with an Arrow schema (__arrow_c_schema__): a pyarrow schema, another spec. Each column is ColSpec(dtype) -- not nullable, so a frame with nulls fails until a column says it may hold them. For a spec that says more, profile data: profile().

schema

schema() -> Schema

The Polars schema this spec declares: column name to dtype.

tag

tag(*tags: str | Sequence[str], match: Literal['any', 'all'] = 'any') -> list[str]

Column names carrying any (or all) of the tags, in declaration order.

with_name

with_name(name: str) -> TableSpec

A copy of this spec under a different name.

with_rows

with_rows(rows: Bound[int] | tuple[int | None, int | None] | None) -> TableSpec

A copy of this spec holding rows rows; None for any number.

with_description

with_description(description: str | None) -> TableSpec

A copy of this spec with description in place of its own.

with_columns

with_columns(mapping: Mapping[str, ColSpec] | None = None, /, **columns: ColSpec) -> TableSpec

Adds columns, or replaces existing ones of the same name in place.

with_checks

with_checks(*checks: Check) -> TableSpec

A copy of this spec with checks added to the ones it has.

with_foreign_keys

with_foreign_keys(*foreign_keys: ForeignKey) -> TableSpec

A copy of this spec with foreign_keys added to the ones it has.

with_hierarchy

with_hierarchy(hierarchy: Hierarchy | None) -> TableSpec

A copy of this spec declaring (or, with None, dropping) a hierarchy.

with_unique_together

with_unique_together(*groups: Sequence[str]) -> TableSpec

A copy of this spec with groups added as composite unique keys.

with_order_by

with_order_by(*names: str, descending: bool | Sequence[bool] = False) -> TableSpec

A copy of this spec whose rows come sorted by names; with none, a copy with no order.

drop

drop(*names: str | Sequence[str]) -> TableSpec

Removes columns, and any composite key, foreign key or check that used them. The names may be given one by one or as a list, as Polars' own drop takes them.

select

select(*names: str | Sequence[str]) -> TableSpec

Keeps only the named columns, in the order given: one by one or as a list, as Polars' own select takes them.

rename

rename(mapping: Mapping[str, str]) -> TableSpec

Renames columns, rewriting every constraint that names them.

Rules, validators, checks, composite keys, foreign keys, the hierarchy and the order are rewritten. A validator or check written as a raw pl.Expr cannot be: rewriting Polars expressions is not something this library does, so renaming a column one names is refused. Write it with col(), whose predicates rename like everything else.

prefix

prefix(text: str) -> TableSpec

Every column renamed with text in front, as rename renames: every claim follows, and a raw pl.Expr naming a column is refused.

suffix

suffix(text: str) -> TableSpec

Every column renamed with text after it; see prefix.

join

join(other: SpecLike, on: str | Sequence[str] | None = None, *, left_on: str | Sequence[str] | None = None, right_on: str | Sequence[str] | None = None, how: JoinHow = 'inner', suffix: str = '_right', coalesce: bool | None = None, name: str | None = None) -> TableSpec

The spec of frame.join(other_frame, ...), for frames of this spec and other, with the arguments LazyFrame.join takes.

The columns, their order and their names are those the installed Polars gives the join, read from a join of two empty frames, so every how, suffix and coalesce behaves as it does on data. Each column then carries the claims about its values from the side it came from: dtype, bounds, choices, weights, format, pattern, lengths, tags, description, validators and default. A side that can go unmatched -- the right of left, the left of right, both of full -- becomes nullable.

A claim about rows is not carried: checks, rules, foreign keys, composite keys, the hierarchy, the order and rows are dropped, each to be added back with its with_* if it still holds. A column's unique survives only when the other side's join key is declared unique -- a single unique column, or a composite key -- since then no row of this side is repeated. semi and anti keep this spec as it is, but for rows and the order: Polars does not promise a join keeps its rows' order.

The name defaults to Left_Right. Join keys of different dtypes are refused, as Polars refuses them.

with_catspec

with_catspec(catspec: CatSpec | type[CatSpec]) -> TableSpec

Re-points columns at the registry's Enum and Categorical types.

A column whose name resolves in the registry (exactly, or by case) takes the registry's dtype; everything else it declared carries over. choices a new Enum cannot hold, and weights over a domain that changed size, are dropped with a warning.

Takes the registry as a value or as the class body that declares it.

FrameSpec

FrameSpec

Base class for declaring a DataFrame/LazyFrame specification.

Subclass it and assign a ColSpec per column, in the order columns should appear:

class DataSource(FrameSpec):
    string_1 = ColSpec(pl.String)
    enum_1 = ColSpec(pl.Enum(["mammal", "reptile"]), nullable=True)
    int_1 = ColSpec(pl.Int64, bounds=(-100, 100), nullable=True)

df = DataSource.generate(1_000_000, seed=42)

The class body builds DataSource.spec, a TableSpec; every classmethod here forwards to a function over it. A column may take any name: one that collides with a method (schema, tag, ...) is reachable as DataSource.col("schema") while the method keeps working. Names that cannot be attributes at all -- a leading underscore, or one straight from data -- go through __columns__:

class Raw(FrameSpec):
    __columns__ = {"_id": ColSpec(pl.Int64), "Unit Price": ColSpec(pl.Float64)}

Examples:

>>> class Orders(FrameSpec):
...     "One row per order."
...
...     id = ColSpec(pl.Int64, unique=True)
...     status = ColSpec(pl.Enum, choices={"NEW": 0.7, "PAID": 0.3})
...     total = ColSpec(pl.Float64, bounds=(0, 100))
...
>>> frame = Orders.generate(1_000, seed=1)
>>> Orders.validate(frame).height
1000
>>> Orders.spec.description
'One row per order.'
>>> Orders.names.status
'status'

from_spec classmethod

from_spec(spec: TableSpec, *, name: str | None = None) -> type[FrameSpec]

A FrameSpec subclass wrapping an existing TableSpec.

from_yaml classmethod

from_yaml(source: str | Path, *, categories: CatSpec | type[CatSpec] | str | Path | None = None, strict: bool = True) -> type[FrameSpec]

Builds a new FrameSpec subclass from a YAML file written by to_yaml.

categories is a CatSpec registry, or a path to one, used to resolve shared Enums and Categoricals; when omitted, a categories: key in the file is loaded automatically. An unknown key in the file is an error unless strict=False, which downgrades it to a warning.

from_dataframe classmethod

from_dataframe(df: DataFrame, *, name: str = 'Profiled', options: ProfileOptions | None = None, **keywords: Unpack[ProfileKeywords]) -> type[FrameSpec]

Infers a spec by profiling an existing DataFrame.

polspec.profile with less recorded by default -- weights, shape, detect_unique and detect_order off -- so it describes what each column is rather than a stand-in for the data. A string or categorical column with at most max_unique_enum distinct values, repeating, becomes an Enum; one whose values have a format -- an email, a UUID, a country code -- names it, unless formats=False; one of codes writes the pattern they share, unless patterns=False. calculate_bounds records observed (min, max) for numeric and temporal columns and (min_len, max_len) for strings and binary.

options is a ProfileOptions, or pass its fields as keywords: round_bounds and round_rates write bounds and rates at a person's precision, as polspec schema infer does by default.

col classmethod

col(name: str) -> ColSpec

The declaration of one column, whatever it is called.

schema classmethod

schema() -> Schema

The Polars schema this spec declares: column name to dtype.

tag classmethod

tag(*tags: str | Sequence[str], match: Literal['any', 'all'] = 'any') -> list[str]

Column names carrying any (or all) of the tags, in declaration order.

catspec classmethod

catspec() -> CatSpec

The CatSpec registry this spec's Enum and Categorical columns imply.

with_catspec classmethod

with_catspec(catspec: CatSpec | type[CatSpec], *, name: str | None = None) -> type[FrameSpec]

A new FrameSpec subclass with columns re-typed against catspec.

to_yaml classmethod

to_yaml(path: str | Path | None = None) -> str | None

This spec as human-readable YAML: written to the file path, or, with no path, returned as text.

to_python classmethod

to_python(path: str | Path | None = None) -> str | None

This spec as an importable Python module defining a subclass: written to the file path, or, with no path, returned as text.

generate classmethod

generate(n: int, *, method: Literal['random', 'cartesian'] = 'random', seed: int | None = None, references: References = None, cycles: int = 0, self_references: int = 0, max_bytes: int | None = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> DataFrame

Generates a DataFrame matching this spec.

cycles and self_references apply only to a spec declaring a __hierarchy__, and deliberately violate it. See polspec.generation.generate for the full contract. scan() is the lazy verb.

estimated_size classmethod

estimated_size(n: int) -> int

Bytes a frame of n generated rows is expected to hold.

See polspec.estimated_size. generate() warns when the estimate passes four gibibytes, and takes max_bytes= to refuse instead.

scan classmethod

scan(n: int, *, seed: int | None = None, batch_size: int | None = None, method: Literal['random', 'cartesian'] = 'random', references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> LazyFrame

A LazyFrame of n rows, generated as they are collected.

Unlike generate(), which builds the whole frame before it returns, nothing is generated until the plan is collected -- and then only the columns and rows the plan asks for:

Orders.scan(50_000_000, seed=1).sink_parquet("orders.parquet")
Orders.scan(50_000_000, seed=1).select("total").head(5).collect()

The first streams in bounded memory; the second generates five rows of one column. A projected column holds the values the whole frame would: every column is seeded by its name, so dropping its neighbours cannot move it.

Rows come in batches, so a scan carries generate_batches' terms: a spec declaring a __hierarchy__ is refused, and a unique=True column or a __unique_together__ group is distinct within a batch rather than across n. Leaving batch_size unset lets polars ask for the size it wants; setting it pins the size.

generate_batches classmethod

generate_batches(n: int, *, batch_size: int = 100000, method: Literal['random', 'cartesian'] = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> Iterator[DataFrame]

Yields chunks of generated rows without holding all n in memory.

Each batch is a window onto the one frame seed describes: a column no pass rewrites holds, batch by batch, exactly the rows generate(n, seed=seed) would, whatever batch_size is.

sink_parquet classmethod

sink_parquet(path: str | Path, n: int, *, batch_size: int = 100000, compression: ParquetCompression = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None

Generates n rows and streams them to a Parquet file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_parquet.

sink_csv classmethod

sink_csv(path: str | Path, n: int, *, batch_size: int = 100000, include_header: bool = True, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None

Generates n rows and streams them to a CSV file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_csv.

sink_ipc classmethod

sink_ipc(path: str | Path, n: int, *, batch_size: int = 100000, compression: IpcCompression | None = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None

Generates n rows and streams them to an Arrow IPC file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_ipc.

sink_ndjson classmethod

sink_ndjson(path: str | Path, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None

Generates n rows and streams them to an NDJSON file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_ndjson.

diff classmethod

diff(other: TableSpec | type[FrameSpec], *, renames: Mapping[str, str] | None = None, options: DriftOptions | None = None, **keywords: Unpack[DriftKeywords]) -> DriftReport

What changed from this spec to other, as a DriftReport.

This spec is the old side. See polspec.diff.

drift classmethod

drift(df: DataFrame | LazyFrame, *, options: DriftOptions | None = None, **keywords: Unpack[DriftKeywords]) -> DriftReport

How df has moved relative to this spec, as a DriftReport: polspec.drift, which takes every DriftOptions field as a keyword or options= as an object.

to_markdown classmethod

to_markdown(path: str | Path | None = None, *, title: str | None = None, choices: Literal['summary', 'appendix', 'full'] = 'summary') -> str

A Markdown data dictionary for this spec, written to path if given; choices as polspec.to_markdown takes it.

to_mermaid classmethod

to_mermaid(path: str | Path | None = None, *, title: str | None = None) -> str

A Mermaid entity-relationship diagram for this spec.

read classmethod

read(path: str | PathLike[str], **reader_options: Any) -> DataFrame

A data file as a frame, in the terms this spec declares -- read, not validated. See polspec.read: the extension picks the reader, a declared date or time that arrived as text is parsed, and nothing else is cast.

inspect classmethod

inspect(df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **keywords: Unpack[ValidationKeywords]) -> ValidationReport

Everything this spec has to say about df, as a ValidationReport.

Never raises for a frame that fails: each violation is a Finding with a code, a count, samples and, for row-level findings, the offending rows reachable lazily through report.rows(finding) or report.failing_rows(). Takes the same options as validate.

validate classmethod

validate(df: DataFrame, *, options: ValidationOptions | None = None, references: References = None, **keywords: Unpack[ValidationKeywords]) -> DataFrame
validate(df: LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **keywords: Unpack[ValidationKeywords]) -> LazyFrame
validate(df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **keywords: Unpack[ValidationKeywords]) -> DataFrame | LazyFrame

Validates a DataFrame or LazyFrame against this spec.

Raises ValidationError carrying a ValidationReport of every violation, or returns the (optionally transformed) frame -- a LazyFrame if df was one. Use inspect for the report without the exception.

Parameters:

Name Type Description Default
df DataFrame | LazyFrame

The frame to validate.

required
options ValidationOptions

Every option at once, as a value. Cannot be combined with the keywords below.

None
references mapping

Parent frames for foreign keys into other specs, keyed by that spec, its FrameSpec class, or its name.

None
**keywords Unpack[ValidationKeywords]

The options one at a time, as ValidationKeywords lists them: extra_cols, missing_cols, strict_dtypes, cast, fill_defaults, parse_text and streaming, and each check switch by the ValidationOptions field it sets (unique, order, ...).

{}
Notes

A keyword left out, or given as None, takes the ValidationOptions default; see that class for what every option means.

ForeignKey

ForeignKey dataclass

ForeignKey(columns: str | Sequence[str], *, references: type[FrameSpec] | TableSpec | str, ref_columns: str | Sequence[str] | None = None, name: str | None = None)

Declares referential integrity: one or more columns must only contain values that exist in another FrameSpec's (or this same FrameSpec's) columns.

Parameters:

Name Type Description Default
columns str | Sequence[str]

The local column(s) that must reference existing parent values.

required
references type[FrameSpec] | TableSpec | str

The spec this key references -- a FrameSpec subclass, a TableSpec, or a spec's name -- or the literal string "self" for a self-referencing key (an employee.manager_id pointing back at employee.id). "self" always resolves to whichever spec the key ends up declared or inherited on, not the class it was first written in.

After construction references is always a string: the target's name. When a spec object was given, it is kept as target, so its columns can be checked at declaration; a bare name has no target until a registry resolves it.

required
ref_columns str | Sequence[str] | None

The referenced column(s) on the target, in the same order as columns. Defaults to columns (same names on both sides).

None
name str | None

A human-readable identifier. Defaults to a name derived from the columns and target.

None
Notes

Rows where any of columns is null are exempt (standard FK semantics -- a null foreign key means "no reference", not "an invalid one").

Examples:

>>> class CustomerSpec(FrameSpec):
...     id = ColSpec(pl.Int64, bounds=(1, 1_000), unique=True)
>>> class OrderSpec(FrameSpec):
...     customer_id = ColSpec(pl.Int64, bounds=(1, 1_000))
...     __foreign_keys__ = [
...         ForeignKey("customer_id", references=CustomerSpec, ref_columns="id"),
...     ]
>>> class EmployeeSpec(FrameSpec):
...     id = ColSpec(pl.Int64, unique=True)
...     manager_id = ColSpec(pl.Int64, nullable=True)
...     __foreign_keys__ = [
...         ForeignKey("manager_id", references="self", ref_columns="id"),
...     ]

Hierarchy

Hierarchy dataclass

Hierarchy(*, child: str, parent: str, max_depth: int = 1, branching: float | None = None, roots: int | None = None)

Declares that two columns of a spec form a parent/child edge list.

Parameters:

Name Type Description Default
child str

The column holding the lower reference -- the one doing the pointing. Each value appears in exactly one row, which is what gives every reference a single ultimate parent.

required
parent str

The column holding the reference being pointed at.

required
max_depth int

Hops from an ultimate parent to the furthest child. Generation guarantees at least one chain of exactly this length and none longer, so a walk that stops early and one that runs away are both caught.

1
branching float | None

Mean children per reference, which is what decides how many ultimate parents n rows imply. Mutually exclusive with roots; one of the two must be given, and branching=3.0 is the default when neither is.

None
roots int | None

An exact number of ultimate parents, as an alternative to branching.

None
Notes

A row is an edge, so generate(n) produces n rows. Ultimate parents have no row of their own -- nothing to point at -- so n edges over R roots need n + R distinct references, drawn from the child column's own declaration.

Examples:

>>> class Links(FrameSpec):
...     PARENT_REF = ColSpec(pl.String)
...     CHILD_REF = ColSpec(pl.String)
...     __hierarchy__ = Hierarchy(
...         child="CHILD_REF", parent="PARENT_REF", max_depth=5
...     )

columns property

columns: tuple[str, str]

The two columns this declaration writes, child first.