Skip to content

Columns

What one column declares, and the pieces that make up a declaration.

ColSpec

ColSpec dataclass

ColSpec(dtype: DataType | type[DataType], *, col_name: str | None = None, seed_name: str | None = None, nullable: bool = False, optional: bool = False, default: Any = None, bounds: Bound[Any] | tuple[Any, Any] | list[Any] | None = None, tags: str | Sequence[str] | None = (), description: str | None = None, unique: bool = False, null_probability: float = DEFAULT_NULL_PROBABILITY, nan_probability: float = 0.0, string_length: Bound[int] | tuple[int, int] | list[int] | None = None, list_length: Bound[int] | tuple[int, int] | list[int] | None = None, element_null_probability: float = 0.0, fields: Mapping[str, ColSpec] | None = None, format: str | None = None, extra_values: Sequence[str] | Mapping[str, float] | None = None, pattern: str | None = None, distribution: str | None = None, distribution_params: dict[str, float] | None = None, choices: Sequence[Any] | dict[Any, float] | None = None, weights: Sequence[float] | None = None, rules: Sequence[ColRule] = (), validators: Check | Expr | Pred | Sequence[Check | Expr | Pred] | None = (), derive: Pred | None = None, text_format: str | None = None)

One column's declaration: its type, and every claim made about its values.

A ColSpec is what generate() samples from and what validate() checks against, so each field below is a claim both sides read.

Parameters:

Name Type Description Default
dtype DataType | type[DataType]

The data type of the column. A bare pl.Enum takes its categories from choices, in the order written: ColSpec(pl.Enum, choices={"A": 0.5, "B": 0.5}) is pl.Enum(["A", "B"]) with those weights.

required
col_name str | None

Overrides the column's name in the generated/validated DataFrame. Declaring columns as class attributes on a FrameSpec requires a valid Python identifier, which cannot contain spaces or other special characters -- col_name lets the attribute keep a clean Python name (unit_price) while the actual column is named whatever the data uses ("Unit Price"). Everything else that refers to this column by name -- ColRule, unique_together, tags lookups, validate() -- uses col_name, not the attribute name.

None
seed_name str | None

The name the column's seed is derived from, when it is not the column's own. Generation seeds each column from the frame seed and the column name, so renaming a column changes the values it produces; a column declared with seed_name="old" keeps producing the data it did as "old", its rules included. Nothing is promised across polspec versions. Two columns of one spec cannot share a seed name, nor may one name another column: they would draw identical values.

None
nullable bool

Whether the column allows null values.

False
optional bool

Whether the column may be absent from a frame. An absent optional column is no missing_columns finding, and none of its other claims is checked; present, it is held to every one. Generation makes it like any other column.

False
default Any

The value that stands for a missing one: validate(fill_defaults=True) fills the column's nulls with it, and adds the column holding it when it is absent; missing_cols="add" adds it holding it too. It must be a value the column's own claims accept -- its dtype, bounds, choices, format, pattern, lengths and validators -- and is refused at declaration if not. None, the default, is no default. A List, Array, Map or Struct column takes none. Generation does not read it.

None
bounds Bound | tuple | list | None

The inclusive range of values allowed in the column, as a Bound or a 2-sequence. Only supported for numeric and temporal data types. Either endpoint may be None to leave that side unconstrained -- bounds=(0, None) for a non-negative column, bounds=(None, 0) for a non-positive one.

An open end means different things to the two consumers of this field, deliberately. validate() treats it as genuinely unconstrained and omits that half of the check. generate() cannot sample an unbounded range, so it falls back to the same default it would use with no bounds at all -- bounds=(0, None) on Int64 generates 0..1,000,000 while validating any value >= 0. This mirrors how bounds=None already behaves rather than adding a third rule.

None
tags str | Sequence[str]

Tag or tags classifying the column, for later selection.

()
description str

What the column means, for a reader: to_markdown shows it, and neither generation nor validation reads it.

None
unique bool

Whether values in the column must be distinct. Generation draws the column without replacement; nulls are exempt. Cannot be combined with weights, a non-uniform distribution, or rules, none of which survive a draw without replacement.

False
null_probability float

Probability of a value being null. Must be between 0 and 1, and only has effect alongside nullable=True -- so that turning nullability off does not also require deleting the rate beside it. Declaring a rate of your own without nullable=True warns, since that reads as asking for nulls rather than as a leftover.

DEFAULT_NULL_PROBABILITY
nan_probability float

For a float column -- or a list's float elements, or a struct's float field -- how often a present value is NaN, from 0 (the default: a NaN is never generated, and validation reports one) to 1. NaN is not a value in any other sense: bounds and choices say nothing about it. Refused with unique=True, since no NaN equals another.

0.0
string_length Bound | tuple[int, int] | list[int] | None

The inclusive range of string lengths, where that applies.

None
list_length Bound | tuple[int, int] | list[int] | None

For a List column, the inclusive range of elements a value holds; generation defaults to 0..5. On a List or Array column every other field that describes a value -- bounds, choices, weights, format, pattern, string_length, distribution -- describes each element; nullable and null_probability describe the list itself. An Array takes its length from the dtype and refuses list_length. For a Map, it is the range of entries a map holds; undeclared, 0..5 or as many as its keys can take, if fewer.

None
element_null_probability float

For a List or Array column, how often an element is null, from 0 (the default: elements are never null, and validation reports one) to 1. The list's own nullable is a null cell; this is a null inside a present one.

0.0
fields Mapping[str, ColSpec] | None

For a Struct column, what is claimed about each field's values: a ColSpec per field, keyed by name. The dtype is the schema -- every field's name and type comes from it -- and fields is what is claimed about the values within it, so a struct of twenty fields where one needs bounds spells one field. A field the mapping omits is generated from its dtype alone.

A field is a value, not a column: unique, rules, validators, seed_name and col_name are refused on one, since each is a statement about a column among columns. Everything that describes a value is allowed -- nullable included, which says whether the field may be null inside a struct that is present -- and fields too, so a struct may nest to any depth. A finding about a field names it: point.lat.

A List or Array of a Struct takes fields too: it describes the element, as bounds and format already do. So does a Map, which is a list of {key, value} entries: fields may say what its key and its value are. A key is never null and never repeats within a map, so a nullable key is refused, and so is a list_length longer than the keys can fill.

None
format str | None

The shape a String column's values take, by name: "uuid4", "email", "ipv4", "ipv6", "mac", "hostname", "iso_country" or "iso_currency". Generation fills the column from that format's own sampler and validation checks every value against it, so a column carrying format="email" is generated to satisfy its own spec. A format owns the column's whole domain: it cannot be combined with choices or string_length, and only a String column can carry one. What it promises is syntax -- an address that is well-formed, not one that is deliverable.

None
extra_values Sequence[str] | Mapping[str, float] | None

Values a format column holds beside the format's own: ISO countries plus "UK (ISO)", IPv4 addresses or "NOT AVAILABLE". Validation accepts a value that has the format or is one of these. Generation draws each on its share of the present rows -- a list gives each 1%, a mapping sets each share -- and the format's own values on the rest. Only a column with a format takes them; a value the format already accepts, a repeated one, and shares summing past 1 are refused, and so is unique=True.

None
pattern str | None

A regular expression every value of a String column must match. Generation fills the column with full matches of it, drawn through the automaton Polars' own regex engine builds, so the values pass validation; ASCII wherever the pattern allows, within string_length when it is declared, and an unbounded repeat runs to 16 characters past the pattern's shortest match. A pattern that every plain random string already passes keeps those strings, and one the generator cannot follow (a Unicode word boundary, an automaton too large) is filled with them, as before 1.4. Cannot be combined with format, which names a shape polspec knows. Polars regex syntax; a pattern that does not compile is refused at declaration. Generated from since 1.4.

None
distribution str | None

The name of the probability distribution for the column's values (e.g. "uniform", "normal").

None
distribution_params dict[str, float] | None

Parameters specific to the chosen distribution.

None
choices tuple | list | dict | None

A finite set of allowed values. A dict maps each choice to its weight.

None
weights tuple[float, ...] | list[float] | None

Weights associated with choices, biasing selection probabilities.

None
rules tuple[ColRule, ...]

Rules (ColRule) that overwrite the column's values on the rows their condition matches.

()
validators Check | Expr | Pred | Sequence[...] | None

A single-column business rule, or several: each either a pl.Expr boolean predicate (referencing only this column) or a Check (for a custom name, description or null handling). Unlike FrameSpec.__checks__, these travel with the column's own declaration.

()
derive Pred | None

A col() expression computing the column from others, such as col("qty") * col("price"). Generation computes it after every column it reads, and casts it to the column's dtype; validation holds the column to it -- equal, or for a float within a relative 1e-9 -- as a derived finding. A derived value is null exactly where the expression is. The expression decides the values, so a derived column declares none of bounds, choices, rules, distribution, format, pattern, string_length, unique or default, and is nullable when an input is. Added in 1.4.

None

Examples:

>>> ColSpec(pl.Int64, bounds=(1, 100), nullable=True, null_probability=0.2)
ColSpec(Int64, nullable=True, bounds=(1, 100), null_probability=0.2)
>>> ColSpec(pl.String, choices={"NEW": 0.75, "PAID": 0.25})
ColSpec(String, choices={'NEW': 0.75, 'PAID': 0.25})
>>> ColSpec(pl.Enum, choices={"NEW": 0.75, "PAID": 0.25}).dtype
Enum(categories=['NEW', 'PAID'])

value_dtype property

value_dtype: DataType

The dtype each value has: the element's for a List or Array column, the column's own otherwise. Every field that describes a value is checked against this.

replace

replace(**changes: Any) -> ColSpec

This column with changes made, checked as a new declaration is: Orders.col("total").replace(bounds=None).

Bound

Bound dataclass

Bound(min: T | None, max: T | None)

An inclusive [min, max] range, used for numeric bounds, temporal ranges, and string lengths.

Either endpoint may be None, meaning that side is unconstrained -- see ColSpec.bounds, the only field that accepts an open end.

Examples:

>>> ColSpec(pl.Int64, bounds=Bound(0, None)).bounds  # no upper end
Bound(min=0, max=None)
>>> ColSpec(pl.Int64, bounds=(0, None)).bounds == Bound(0, None)
True

is_open property

is_open: bool

True when either endpoint is unconstrained.

is_open_both property

is_open_both: bool

True when neither endpoint constrains anything.

closed

closed() -> tuple[T, T]

Both endpoints of a bound that has both, as string_length does.

ColRule

ColRule dataclass

ColRule(*, when: Pred, choices: tuple, weights: tuple[float, ...] | None = None)

Restricts a column's generated values on rows where when matches.

Applied as a pass over the generated frame: rows where when matches get a value resampled uniformly (or according to weights) from choices instead of whatever was freely generated for them. Multiple rules on the same column are checked in declaration order, first match wins (like SQL CASE/WHEN).

when is evaluated against the frame as it stands when the rule runs, and the passes run in dependency order: a rule keyed on a column that another rule or a foreign key rewrites sees the rewritten values -- the same values validation checks the rule against. Two columns whose rules each read what the other writes have no such order and are rejected at declaration.

when is a predicate built with polspec.col(), not an arbitrary polars expression, so that every rule can round-trip through a spec file:

ColRule(when=col("region") == "UK", choices=["RoyalMail"])
ColRule(when=col("region").is_in(["US", "EU"]) & (col("qty") > 10), choices=["UPS"])

Examples:

>>> spec = TableSpec("Shipments", {
...     "region": ColSpec(pl.Enum(["UK", "US"])),
...     "carrier": ColSpec(pl.Enum(["RoyalMail", "UPS"]), rules=[
...         ColRule(when=col("region") == "UK", choices=["RoyalMail"]),
...     ]),
... })
>>> frame = generate(spec, 1_000, seed=1)
>>> frame.filter(pl.col("region") == "UK")["carrier"].unique().to_list()
['RoyalMail']

Check

Check dataclass

Check(expr: Expr | Pred, *, name: str | None = None, description: str | None = None, ignore_nulls: bool = True)

A declarative multi-column validation constraint evaluated as a Polars boolean expression.

Parameters:

Name Type Description Default
expr Expr | Pred

The boolean condition each row must satisfy: a Polars expression, or a predicate built with polspec.col(). A predicate can be written to a YAML or Python spec file; a raw expression cannot.

required
name str | None

A human-readable identifier for the check constraint (e.g. 'total_gte_subtotal'). If omitted, defaults to the string representation of the expression.

None
description str | None

An optional description detailing the business logic or rationale for this check.

None
ignore_nulls bool

Whether rows evaluating to null in the check condition are considered valid (standard SQL CHECK constraint semantics). If False, null results are treated as failures.

True

Examples:

>>> check = Check(pl.col("total") >= pl.col("subtotal"), name="total_gte_subtotal")

check

check

check(function: _CheckFunction) -> DeclaredCheck
check(*, name: str | None = None, description: str | None = None, ignore_nulls: bool = True) -> Callable[[_CheckFunction], DeclaredCheck]
check(function: _CheckFunction | None = None, /, *, name: str | None = None, description: str | None = None, ignore_nulls: bool = True) -> DeclaredCheck | Callable[[_CheckFunction], DeclaredCheck]

Declares a function in a FrameSpec class body as one of its checks, named after the function. Added in 1.7.

The function takes no arguments and returns the condition each row must satisfy: a Polars expression, or a predicate built with polspec.col(), which a spec file can hold. It is called once, when the class is built, and the check joins __checks__ after any listed there, in the order the functions are declared. The function's name does not stay on the class: the check is Orders.spec.checks.

Parameters:

Name Type Description Default
function callable

The function, when @check is used bare.

None
name str

The check's name, when it should not be the function's.

None
description str

What the check means, for a reader: to_markdown shows it.

None
ignore_nulls bool

As Check takes it: a row whose condition is null passes.

True

Examples:

>>> class Orders(FrameSpec):
...     subtotal = ColSpec(pl.Float64, bounds=(0, 1000))
...     total = ColSpec(pl.Float64, bounds=(0, 1100))
...
...     @check(description="tax never makes a total smaller")
...     def total_gte_subtotal():
...         return pl.col("total") >= pl.col("subtotal")
>>> [c.name for c in Orders.spec.checks]
['total_gte_subtotal']

formats

formats

formats() -> DataFrame

Every format ColSpec(format=...) accepts, one row each, in the order they are documented. Added in 1.7.

Returns:

Type Description
DataFrame

name, what format= takes; summary, what a value of it looks like; finite, whether it is a list of values (a country, a currency) rather than a shape (an email, a UUID); example, one value of it; and values, a finite format's every value, null for a shape.

Examples:

>>> listed = formats()
>>> listed.filter(pl.col("name") == "email").select("name", "example").row(0)
('email', 'alex42@example.com')
>>> listed.filter(pl.col("finite"))["name"].to_list()
['iso_country', 'iso_country3', 'iso_currency']