Skip to content

Specs as files

A spec can live in a YAML file instead of a class body, so tooling outside Python can read it and so it can be reviewed as a document. It can also be written out as a Python module, to edit by hand from then on. Spec files is the format, key by key.

The examples on this page write and read these specs:

from datetime import date

import polars as pl
from polspec import Check, ColSpec, ForeignKey, FrameSpec


class Customers(FrameSpec):
    id        = ColSpec(pl.Int64, bounds=(1, 10_000), unique=True)
    name      = ColSpec(pl.String, string_length=(3, 20), tags="pii")
    country   = ColSpec(pl.Enum(["UK", "US", "DE"]))
    signed_up = ColSpec(pl.Date, bounds=(date(2020, 1, 1), date(2026, 1, 1)))


class Orders(FrameSpec):
    order_id      = ColSpec(pl.Int64, bounds=(1, 1_000_000), unique=True)
    customer_id   = ColSpec(pl.Int64, bounds=(1, 10_000))
    status        = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]))
    total         = ColSpec(pl.Float64, bounds=(0.0, 5_000.0))
    internal_note = ColSpec(pl.String, string_length=(0, 40), nullable=True)


class OrderLines(FrameSpec):
    order_id = ColSpec(pl.Int64, bounds=(1, 1_000_000))
    line_no  = ColSpec(pl.Int32, bounds=(1, 50))
    quantity = ColSpec(pl.UInt16, bounds=(1, 500))
    __unique_together__ = [["order_id", "line_no"]]
    __foreign_keys__ = [ForeignKey("order_id", references=Orders)]
Orders.to_yaml("orders.yaml")
Loaded = FrameSpec.from_yaml("orders.yaml")

Loaded.generate(1_000, seed=1)

With no path, to_yaml() and to_python() return the text instead of writing it — for a test, a notebook or a review. polspec.to_dict(spec) and polspec.from_dict(data) are the same file as plain Python data, for storing a spec somewhere other than a file.

The output is plain, readable YAML — defaults are omitted so the file shows only what you actually declared:

version: 4
name: Orders
columns:
  order_id:
    dtype: Int64
    bounds:
    - 1
    - 1000000
    unique: true
  customer_id:
    dtype: Int64
    bounds:
    - 1
    - 10000
  status:
    dtype:
      Enum:
      - NEW
      - PAID
      - SHIPPED
  total:
    dtype: Float64
    bounds:
    - 0.0
    - 5000.0
  internal_note:
    dtype: String
    nullable: true
    string_length:
    - 0
    - 40

An open-ended bound, such as bounds=(0, None), writes its open end as null and reads back unchanged. A file you write by hand may use either YAML list style: bounds: [1, 1000000] reads the same.

version: records the file format that wrote the file. A file from an earlier version is migrated on the way in; a file from a later polspec is refused with a message saying so. A key the reader does not know is an error naming the closest known key, since silently reading a misspelt option as its default is the worst outcome — pass strict=False to from_yaml to downgrade that to a warning.

What survives a round-trip

Round-trips
dtypes, including parametrized Enum / Datetime / Duration / named Categorical yes
nullable, optional, default, null_probability, nan_probability, bounds, string_length, list_length, element_null_probability, fields yes
format, extra_values, pattern, seed_name, unique, tags, description yes
choices, weights, distribution, distribution_params yes
rules (ColRule) yes
__unique_together__, __order_by__, __descending__, __rows__, __hierarchy__, __description__ yes
__foreign_keys__, self-referencing or to another spec (by name) yes
__checks__ and ColSpec.validators written with col() yes
__checks__ and ColSpec.validators over a raw pl.Expr no

A foreign key to another spec is written as that spec's name; nothing checks it until a spec of that name is supplied, through references= on generate/validate or a registry. What cannot be written is an arbitrary polars.Expr, which Polars cannot serialize stably. to_yaml() warns about each, naming exactly what will be lost:

UserWarning: Orders declares 1 __checks__ ('under_the_cap') that cannot be
represented in YAML (a Check over a raw polars.Expr; write it with
polspec.col() to persist it) and will NOT be written to orders.yaml. They will
be lost on FrameSpec.from_yaml() unless re-declared on a subclass of the
loaded spec.

The suggested recovery is to subclass what you loaded:

Loaded = FrameSpec.from_yaml("orders.yaml")

class Orders(Loaded):
    __checks__ = [Check(pl.col("total") < 5_000, name="under_the_cap")]

Columns, rules, unique keys, foreign keys, and any check or validator written with col() come from the file; only raw-expression parts need re-declaring in Python. A check in YAML is its predicate in data form:

checks:
- expr:
    lt:
    - col: total
    - 5000
  name: under_the_cap

Sharing categories between files

A spec file can reference a CatSpec registry by path, resolved relative to the spec file:

name: Orders
categories: categories.yaml
columns:
  status:
    dtype:
      Enum: STATUS
  currency:
    dtype:
      Categorical: CURRENCY

$categories.STATUS and categories.STATUS are accepted as prefixed forms of the same reference.

Or pass a registry explicitly, which wins over anything the file names:

FrameSpec.from_yaml("orders.yaml", categories=CatSpec.from_yaml("categories.yaml"))
FrameSpec.from_yaml("orders.yaml", categories="categories.yaml")

A spec can also emit the registry its own columns imply:

Orders.catspec().to_yaml("categories.yaml")

Python instead of YAML

The same spec can be written as an importable Python module. It is the right choice when the spec will be edited by hand from now on — including to add back the raw-expression checks a file cannot hold. The Orders subclass above carries one, so writing it warns:

Orders.to_python("orders_spec.py")   # warns: its raw-expression check is dropped
"""Declares the Orders schema."""

import polars as pl
from polspec import ColSpec, FrameSpec


class Orders(FrameSpec):
    __columns__ = {
        'order_id': ColSpec(pl.Int64, bounds=(1, 1000000), unique=True),
        'customer_id': ColSpec(pl.Int64, bounds=(1, 10000)),
        'status': ColSpec(pl.Enum(['NEW', 'PAID', 'SHIPPED'])),
        'total': ColSpec(pl.Float64, bounds=(0.0, 5000.0)),
        'internal_note': ColSpec(pl.String, nullable=True, string_length=(0, 40)),
    }

Columns are declared through __columns__ because a name straight from data is not always a valid identifier. What survives is exactly the round-trip table above: a check or validator over a raw pl.Expr warns and is dropped, and the module is where you add it back by hand. A foreign key to another spec is kept, by the spec's name. polspec schema infer uses this path when its output ends in .py; see Command line.

The spec's own name is the class name, so it has to be one Python can declare. A TableSpec may be named anything — by profile(name="daily orders"), or a YAML file's name: — and one that is not an identifier, or is a keyword like class, raises SerializationError before anything is written. Rename it, or keep it in YAML, which holds any name:

from polspec import TableSpec
from polspec import to_python

spec = TableSpec("daily orders", {"id": ColSpec(pl.Int64)})
to_python(spec.with_name("DailyOrders"), "daily_orders.py")

Column names from data

from_yaml and to_python declare columns through __columns__, so names that could not be class attributes — a leading underscore, a collision with a method name like schema, or a name with spaces — load correctly. The YAML key is the column's real name; a col_name set in a class body is not written, because the key already carries it. See Column names that are not identifiers.

Several specs in one file

A Registry writes every spec it holds, and the categories it was declared with, to one file keyed by spec name, and reads it back with the same version and strictness rules:

from polspec import CatSpec, Registry

categories = CatSpec(
    enums={"STATUS": ["NEW", "PAID", "SHIPPED"]},
    categoricals={"CURRENCY": pl.Categories("CURRENCY", physical=pl.UInt8)},
)

Registry(Customers, Orders, OrderLines, categories=categories).to_yaml("specs.yaml")
registry = Registry.from_yaml("specs.yaml").resolve()