Specs as files¶
A spec can live in a YAML file instead of a class body, so tooling outside Python can read it and so it can be reviewed as a document. It can also be written out as a Python module, to edit by hand from then on. Spec files is the format, key by key.
The examples on this page write and read these specs:
from datetime import date
import polars as pl
from polspec import Check, ColSpec, ForeignKey, FrameSpec
class Customers(FrameSpec):
id = ColSpec(pl.Int64, bounds=(1, 10_000), unique=True)
name = ColSpec(pl.String, string_length=(3, 20), tags="pii")
country = ColSpec(pl.Enum(["UK", "US", "DE"]))
signed_up = ColSpec(pl.Date, bounds=(date(2020, 1, 1), date(2026, 1, 1)))
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 1_000_000), unique=True)
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
status = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]))
total = ColSpec(pl.Float64, bounds=(0.0, 5_000.0))
internal_note = ColSpec(pl.String, string_length=(0, 40), nullable=True)
class OrderLines(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 1_000_000))
line_no = ColSpec(pl.Int32, bounds=(1, 50))
quantity = ColSpec(pl.UInt16, bounds=(1, 500))
__unique_together__ = [["order_id", "line_no"]]
__foreign_keys__ = [ForeignKey("order_id", references=Orders)]
Orders.to_yaml("orders.yaml")
Loaded = FrameSpec.from_yaml("orders.yaml")
Loaded.generate(1_000, seed=1)
With no path, to_yaml() and to_python() return the text instead of
writing it — for a test, a notebook or a review. polspec.to_dict(spec)
and polspec.from_dict(data) are the same file as plain Python data, for
storing a spec somewhere other than a file.
The output is plain, readable YAML — defaults are omitted so the file shows only what you actually declared:
version: 4
name: Orders
columns:
order_id:
dtype: Int64
bounds:
- 1
- 1000000
unique: true
customer_id:
dtype: Int64
bounds:
- 1
- 10000
status:
dtype:
Enum:
- NEW
- PAID
- SHIPPED
total:
dtype: Float64
bounds:
- 0.0
- 5000.0
internal_note:
dtype: String
nullable: true
string_length:
- 0
- 40
An open-ended bound, such as bounds=(0, None), writes its open end as
null and reads back unchanged. A file you write by hand may use either
YAML list style: bounds: [1, 1000000] reads the same.
version: records the file format that wrote the file. A file from an
earlier version is migrated on the way in; a file from a later polspec is
refused with a message saying so. A key the reader does not know is an
error naming the closest known key, since silently reading a misspelt option
as its default is the worst outcome — pass strict=False to
from_yaml to downgrade that to a warning.
What survives a round-trip¶
| Round-trips | |
|---|---|
dtypes, including parametrized Enum / Datetime / Duration / named Categorical |
yes |
nullable, optional, default, null_probability, nan_probability, bounds, string_length, list_length, element_null_probability, fields |
yes |
format, extra_values, pattern, seed_name, unique, tags, description |
yes |
choices, weights, distribution, distribution_params |
yes |
rules (ColRule) |
yes |
__unique_together__, __order_by__, __descending__, __rows__, __hierarchy__, __description__ |
yes |
__foreign_keys__, self-referencing or to another spec (by name) |
yes |
__checks__ and ColSpec.validators written with col() |
yes |
__checks__ and ColSpec.validators over a raw pl.Expr |
no |
A foreign key to another spec is written as that spec's name; nothing
checks it until a spec of that name is supplied, through references= on
generate/validate or a registry. What cannot be written is an arbitrary
polars.Expr, which Polars cannot serialize stably. to_yaml() warns about
each, naming exactly what will be lost:
UserWarning: Orders declares 1 __checks__ ('under_the_cap') that cannot be
represented in YAML (a Check over a raw polars.Expr; write it with
polspec.col() to persist it) and will NOT be written to orders.yaml. They will
be lost on FrameSpec.from_yaml() unless re-declared on a subclass of the
loaded spec.
The suggested recovery is to subclass what you loaded:
Loaded = FrameSpec.from_yaml("orders.yaml")
class Orders(Loaded):
__checks__ = [Check(pl.col("total") < 5_000, name="under_the_cap")]
Columns, rules, unique keys, foreign keys, and any check or validator
written with col() come from the file; only raw-expression parts need
re-declaring in Python.
A check in YAML is its predicate in data form:
Sharing categories between files¶
A spec file can reference a CatSpec registry by path, resolved relative to
the spec file:
name: Orders
categories: categories.yaml
columns:
status:
dtype:
Enum: STATUS
currency:
dtype:
Categorical: CURRENCY
$categories.STATUS and categories.STATUS are accepted as prefixed forms of
the same reference.
Or pass a registry explicitly, which wins over anything the file names:
FrameSpec.from_yaml("orders.yaml", categories=CatSpec.from_yaml("categories.yaml"))
FrameSpec.from_yaml("orders.yaml", categories="categories.yaml")
A spec can also emit the registry its own columns imply:
Python instead of YAML¶
The same spec can be written as an importable Python module. It is the right
choice when the spec will be edited by hand from now on — including to add
back the raw-expression checks a file cannot hold. The Orders subclass
above carries one, so writing it warns:
"""Declares the Orders schema."""
import polars as pl
from polspec import ColSpec, FrameSpec
class Orders(FrameSpec):
__columns__ = {
'order_id': ColSpec(pl.Int64, bounds=(1, 1000000), unique=True),
'customer_id': ColSpec(pl.Int64, bounds=(1, 10000)),
'status': ColSpec(pl.Enum(['NEW', 'PAID', 'SHIPPED'])),
'total': ColSpec(pl.Float64, bounds=(0.0, 5000.0)),
'internal_note': ColSpec(pl.String, nullable=True, string_length=(0, 40)),
}
Columns are declared through __columns__ because a name straight from data
is not always a valid identifier. What survives is exactly the
round-trip table above: a check or validator
over a raw pl.Expr warns and is dropped, and the module is where you add
it back by hand. A foreign key to another spec is kept, by the spec's name. polspec schema infer uses
this path when its output ends in .py; see
Command line.
The spec's own name is the class name, so it has to be one Python can
declare. A TableSpec may be named anything — by
profile(name="daily orders"), or a YAML file's name: — and one that is
not an identifier, or
is a keyword like class, raises SerializationError before anything is
written. Rename it, or keep it in YAML, which holds any name:
from polspec import TableSpec
from polspec import to_python
spec = TableSpec("daily orders", {"id": ColSpec(pl.Int64)})
to_python(spec.with_name("DailyOrders"), "daily_orders.py")
Column names from data¶
from_yaml and to_python declare columns through __columns__, so names
that could not be class attributes — a leading underscore, a collision with a
method name like schema, or a name with spaces — load correctly. The YAML
key is the column's real name; a col_name set in a class body is not
written, because the key already carries it. See
Column names that are not identifiers.
Several specs in one file¶
A Registry writes every spec it holds, and the categories it
was declared with, to one file keyed by spec name, and reads it back with the
same version and strictness rules:
from polspec import CatSpec, Registry
categories = CatSpec(
enums={"STATUS": ["NEW", "PAID", "SHIPPED"]},
categoricals={"CURRENCY": pl.Categories("CURRENCY", physical=pl.UInt8)},
)
Registry(Customers, Orders, OrderLines, categories=categories).to_yaml("specs.yaml")
registry = Registry.from_yaml("specs.yaml").resolve()