Skip to content

Declaring columns

A ColSpec describes one column. Only dtype is required. This page shows the fields in use; ColSpec fields is the table of all of them — what each applies to, and what generation and validation do with it.

Every other field is a keyword, so a declaration reads as a list of claims:

import polars as pl
from polspec import ColSpec

ColSpec(pl.Int64, bounds=(1, None), unique=True, description="The order number.")

Types

polspec generates every dtype below. A dtype passed as a class is instantiated for you, so pl.Int64 and pl.Int64() mean the same thing. It has to be a Polars dtype: ColSpec(int) or ColSpec("Int64") is refused, with the Polars spelling (pl.Int64) in the message.

A column that claims nothing beyond its dtype can be declared as the dtype alone, in a class body, in __columns__ or in a TableSpec; it is a ColSpec of that dtype. A spec file reads id: Int64 the same way, and is still written in the long form, which earlier releases read too. Added in 1.2.

class Accounts(FrameSpec):
    account_id = ColSpec(pl.Int64, unique=True)
    holder     = pl.String                          # ColSpec(pl.String)
    opened     = pl.Date

Since a dtype in a class body is a column, a class constant meant to hold a dtype has to live outside the class.

Family Types
Integer Int8 Int16 Int32 Int64 Int128 UInt8 UInt16 UInt32 UInt64 UInt128
Float Float16 Float32 Float64
Decimal Decimal(precision, scale)
Boolean Boolean
Text String
Bytes Binary
Temporal Date Time Datetime Duration
Categorical Enum Categorical
Nested List(inner) Array(inner, width) Struct({name: dtype}) — nested to any depth; see Nested columns
Map (Polars 2) Map(key, value) — see Maps

That is every dtype that holds data. Null, Object, Unknown and extension types declare and validate by dtype, but hold nothing polspec can draw, so generate() refuses each by name. A 128-bit integer is drawn through 64 bits and a Float16 as a Float32 rounded to the nearest half, so bounds past 64 bits are refused at generate() and bounds too close to hold a half at declaration; see Known limitations.

Nullability

nullable=False (the default) means validation rejects any null. When nullable=True, null_probability sets how often generation emits one.

from datetime import date

import polars as pl
from polspec import ColSpec, FrameSpec

ColSpec(pl.Int64, nullable=True, null_probability=0.25)   # about a quarter null

The rate only matters while nullable=True, so you can turn nullability off without deleting the rate beside it. Setting a rate on a column that is not nullable warns, though, since it reads as asking for nulls the column will never hold:

ColSpec(pl.Int64, null_probability=0.25)                   # warns: no nulls
ColSpec(pl.Int64, nullable=True, null_probability=0.25)    # about a quarter null
ColSpec(pl.Int64)                                          # no nulls, no warning

The default rate and an explicit 0.0 never warn.

NaN

A float value can be present and still not a number. nan_probability says how often a present value is NaN — nulls are counted first, and NaN is a share of what is left:

ColSpec(pl.Float64, nullable=True, null_probability=0.1, nan_probability=0.05)

The default, 0, claims a column holds numbers: validation reports a NaN in it as a nan finding. NaN is not a value in any other sense — it sits in no range and equals nothing — so bounds and choices never judge one; the NaN claim does, once. It applies wherever a float value sits: a column, a list's elements, a struct's field. A unique=True column cannot declare it, since no NaN equals another, and a column of any other dtype cannot hold one — a missing value there is a null.

Bounds

bounds is an inclusive [min, max] for numeric and temporal columns. Pass a tuple, a list, or a Bound:

ColSpec(pl.Int64, bounds=(-100, 100))
ColSpec(pl.Float64, bounds=[0.0, 1.0])
ColSpec(pl.Date, bounds=(date(2020, 1, 1), date(2024, 12, 31)))

Temporal bounds accept real date, datetime, time and timedelta objects, or the physical integer the dtype stores. A date, datetime or time may also be written as ISO text, read where it is declared, as a hand-written spec file holds it: bounds=("2020-01-01", None). Text that is not one is refused there, as is text on any other column but a Decimal. Added in 1.2.

Decimal bounds

A Decimal(precision, scale) column takes its bounds as an int, a decimal.Decimal, or a string read exactly — the form a spec file writes. A float is read through its repr, so 0.1 is 0.1. An endpoint with more decimal places than the scale keeps is refused rather than rounded, and one the precision cannot hold is refused like any other out-of-range bound:

from decimal import Decimal

ColSpec(pl.Decimal(10, 2), bounds=(0, "99.99"))
ColSpec(pl.Decimal(10, 2), bounds=(Decimal("0.50"), None))
ColSpec(pl.Decimal(10, 2), bounds=(0, "1.005"))
# SpecError: ColSpec.bounds max ('1.005') has more decimal places than Decimal(precision=10, scale=2) keeps (scale 2); ...

Generation draws a Decimal as the integer it physically is and scales it back, so the default range with no bounds is the float default (±1,000,000) or the widest the precision allows, whichever is narrower. The draw is 64-bit: bounds needing more than eighteen significant digits are refused at generate(), and only there — validation checks the full precision.

Open-ended bounds

Either endpoint may be None, leaving that side unconstrained:

ColSpec(pl.Int64, bounds=(0, None))    # non-negative
ColSpec(pl.Int64, bounds=(None, 0))    # non-positive

An open end means different things to generation and validation

validate() treats it as genuinely unconstrained. generate() cannot sample an unbounded range, so it falls back to the same default it would use with no bounds at all.

class S(FrameSpec):
    n = ColSpec(pl.Int64, bounds=(0, None))

S.generate(1000, seed=1)["n"].max()     # ~1_000_000, the Int64 default
S.validate(pl.DataFrame({"n": [10**15]}))   # accepted — no upper limit

A unique=True 64- or 128-bit integer with no bounds at all is the one exception: when the frame needs more distinct values than the default range holds, the range widens upward to fit it, rather than refusing an ID column its dtype could hold.

For "always positive", note that bounds are inclusive: use (1, None) for integers, and either a small floor like (1e-9, None) for floats or an unsigned dtype, which cannot represent a negative at all.

Bounds outside what the dtype can hold are rejected when you declare them:

ColSpec(pl.Float32, bounds=(-1e40, 1e40))
# SpecError: ColSpec.bounds min (-1e+40) is outside the range Float32 can represent ...

Value domains

choices restricts a column to a fixed set:

ColSpec(pl.String, choices=["GBP", "USD", "EUR"])

Choices that are not equally likely are written as a mapping, each choice beside its weight. A weight is relative, so {"a": 10, "b": 5, "c": 1} draws a ten times as often as c:

ColSpec(pl.String, choices={"a": 10.0, "b": 5.0, "c": 1.0})

A bare pl.Enum takes its categories from choices, in the order written, each beside its share. It is how a column's repr and to_python write a weighted Enum, and how profile reports one:

print(ColSpec(pl.Enum, choices={"x": 0.1, "y": 0.2, "z": 0.7}))
ColSpec(Enum, choices={'x': 0.1, 'y': 0.2, 'z': 0.7})

The same weights can be given apart, as weights=, in the order of the choices or of an Enum's categories; the two spellings are the same spec. A Boolean has no choices to map, so it takes them that way, as [p_false, p_true]:

ColSpec(pl.Enum(["x", "y", "z"]), weights=[0.1, 0.2, 0.7])   # the Enum above
ColSpec(pl.Boolean, weights=[0.9, 0.1])                        # 10% true

choices as a mapping on a bare Enum added in 1.2.

Choices are held in the column's own dtype, so a datetime choice on a Datetime column or a bytes choice on a Binary column stays what it is. They must be distinct once cast to that dtype — 1 and "1" on a String column are one value:

ColSpec(pl.String, choices=[1, "1"])
# SpecError: ColSpec.choices contains values that are the same once cast to
# String: ['1']

String and binary length

string_length is an inclusive [min, max] on characters (String) or bytes (Binary). Unlike bounds, both endpoints are required.

ColSpec(pl.String, string_length=(8, 8))    # fixed width
ColSpec(pl.Binary, string_length=(16, 64))

Nested columns

A List or Array column takes the scalar fields as claims about each element, plus list_length; a Struct says what its fields hold with fields; a Map is a list of {key, value} entries. See Nested columns.

String formats

format= names the shape a String column's values take, and both sides read it: generation fills the column from that format's sampler and validation checks every value against it.

ColSpec(pl.String, format="uuid4", unique=True)
ColSpec(pl.String, format="email", nullable=True)

The set is uuid4, uuid7, ulid, email, ipv4, ipv6, mac, hostname, url, iso_country, iso_country3 and iso_currency. A format owns the column's domain, so it cannot sit beside choices or string_length, and only a String column can carry one. See String formats for what each generates, what each accepts, and what none of them promises.

A format can be widened with the values a column holds beside it — ISO countries plus UK (ISO), IPv4 or NOT AVAILABLE — with extra_values.

String patterns

pattern= is a regular expression every value must match, and generation fills the column with full matches of it. See Patterns. Generated from since 1.4.

Distributions

Numeric and temporal columns can be drawn from a shape other than uniform:

Distribution Parameters (aliases accepted)
uniform —
normal mean/mu/loc, std/sigma/scale
lognormal mean/mu/meanlog, std/sigma/sdlog
exponential (exp) rate/lambda/lambda_, or scale
poisson lambda/lambda_/rate/mean
gamma shape/alpha/k, scale/beta/theta
beta alpha/a/shape1, beta/b/shape2
ColSpec(
    pl.Float64,
    bounds=(0.0, 500.0),
    distribution="lognormal",
    distribution_params={"mean": 2.0, "std": 0.6},
)

Bounds clamp, they do not resample

A draw outside the bounds lands on the boundary rather than being drawn again. A normal centred at 0 squeezed into (0, 50) puts roughly half the column on the floor as one repeated value.

When you want a positive-skewed shape, reach for a distribution that is already non-negative — lognormal, exponential, gamma — instead of clamping a symmetric one.

Uniqueness

unique=True declares that values must be distinct. generate() draws the column as a permutation of everything it could hold — the integers in its bounds, its choices, a grid over a float range, every string its lengths or format allow — so the data it produces satisfies it, and a value depends only on its row and the seed. That makes a unique column unique across a whole frame however it is produced: generate_batches, scan() and the sinks give the same values generate() does.

Drawn uniformly from every string it could hold, a unique string favours its longer lengths, since they hold most of the strings: with string_length=(5, 15) nearly every value is fifteen characters. Narrow the lengths if their spread matters.

Nulls are exempt, as they are for foreign keys: a null means "no value", so a nullable unique column may repeat nulls and nothing else. A null row still takes its place in the permutation, so the column's domain has to hold every row, nulls included.

A domain too small to cover the row count is refused, naming the column:

class Narrow(FrameSpec):
    id = ColSpec(pl.Int8, unique=True)

Narrow.generate(300, seed=1)
# GenerationError: Column 'id' is unique, but its domain holds only 256
# distinct value(s) and 300 are needed. Widen its bounds or choices, or
# generate fewer rows.

unique=True cannot be combined with weights, a non-uniform distribution, or rules: the first two describe how often a value recurs, which a draw without replacement has no room for, and a rule would reintroduce the duplicates. Each is refused at declaration rather than quietly ignored.

Describing a column

Added in 1.1.

description= says what a column means, for a reader. to_markdown() shows it, spec files keep it, and neither generation nor validation reads it:

ColSpec(pl.Int64, unique=True, description="The order's number, as printed on the invoice.")

A table's own description is __description__ on a FrameSpec, or description= on a TableSpec; see Generated documentation.

Defaults and optional columns

Two fields say what to do when a value, or a whole column, is missing.

Added in 1.1.

default= is the value that stands in for a missing one. It is used only when you ask: validate(fill_defaults=True) fills the column's nulls with it before checking, and missing_cols="add" adds an absent column holding it. Generation does not read it.

ColSpec(pl.Enum(["open", "closed"]), default="open")

A default has to be a value the column itself accepts — its dtype, bounds, choices, format, pattern, lengths and validators — and one that is not is refused where it is declared. A List, Array, Map or Struct column takes no default.

optional=True means the column may be absent. A frame without it gives no missing_columns finding; a frame with it is checked against every claim as usual. Generation makes the column like any other.

ColSpec(pl.String, optional=True)

Filling defaults shows both at work.

Tags

Tags group columns for later selection. They carry no generation or validation meaning.

class Events(FrameSpec):
    user_id  = ColSpec(pl.Int64, tags=["pii", "key"])
    email    = ColSpec(pl.String, tags="pii")
    duration = ColSpec(pl.Int64, tags="metric")

Events.tag("pii")                      # ['user_id', 'email']
Events.tag("pii", "key", match="all")  # ['user_id']

Column names that are not identifiers

A column declared as a class attribute takes the attribute's name, and an attribute name has to be a valid Python identifier. Real data is not so polite. There are two ways out, for two different situations.

col_name: the data's name has spaces or punctuation

Keep a clean attribute name and tell the ColSpec what the column is really called:

class Sales(FrameSpec):
    unit_price = ColSpec(pl.Float64, col_name="Unit Price", bounds=(0, None))
    region     = ColSpec(pl.Enum(["UK", "US"]), col_name="Sales Region")

Sales.schema()              # Schema({'Unit Price': Float64, 'Sales Region': Enum(...)})
Sales.generate(3).columns   # ['Unit Price', 'Sales Region']

col_name is the column's name everywhere the spec is used: in the generated frame, in validate(), in a ColRule condition built with col(), in __unique_together__, in ForeignKey columns and in tag() results. The attribute name exists only in the class body. Two attributes that resolve to the same col_name are rejected at declaration, and overriding an attribute on a subclass removes the column it named, whatever col_name it carried.

to_yaml() and to_python() write the real column name as the key, so a spec that came from a file never needs col_name.

A third name, seed_name, is not about what the column is called but about what it generates: a renamed column declared with seed_name="old" keeps producing the data it did under the old name. See Renaming a column without changing its data.

__columns__: the name is an identifier but cannot be an attribute

A leading underscore is skipped by the class-body scan, so a column called _id needs the explicit mapping. A name that matches one of FrameSpec's methods (schema, tag, …) is fine either way: the method keeps working and the column is reachable as Spec.col("schema") — see Specs as values.

class Raw(FrameSpec):
    __columns__ = {
        "_id": ColSpec(pl.Int64),
        "schema": ColSpec(pl.String),
    }

__columns__ is never looked up as an attribute, so both the column and the method survive. The dict key already is the column name, so a col_name that disagrees with its key is rejected. from_dataframe, from_yaml and to_python all declare columns this way, since their names come from data rather than from someone's class body.