YAML specs¶

A spec can live in a file instead of a class body, so tooling outside Python can read it and so it can be reviewed as a document.

Orders.to_yaml("orders.yaml")
Loaded = FrameSpec.from_yaml("orders.yaml")

Loaded.generate(1_000, seed=1)

The output is plain, readable YAML — defaults are omitted so the file shows only what you actually declared:

version: 3
name: Orders
columns:
  order_id:
    dtype: Int64
    bounds: [1, 100000]
    unique: true
  status:
    dtype:
      Enum: [NEW, PAID, SHIPPED]
  total:
    dtype: Float64
    bounds: [0.0, null]
  placed:
    dtype: Date
    nullable: true
unique_together:
  - [order_id, status]

An open-ended bound writes as null and reads back unchanged.

version: records the file format that wrote the file. A file from an earlier version is migrated on the way in; a file from a later polspec is refused with a message saying so. A key the reader does not know is an error naming the closest known key, since silently reading a misspelt option as its default is the worst outcome -- pass strict=False to from_yaml to downgrade that to a warning.

What survives a round-trip¶

A foreign key to another spec is written as that spec's name; nothing checks it until a spec of that name is supplied, through references= on generate/validate or a registry. What cannot be written is an arbitrary polars.Expr, which Polars cannot serialize stably. to_yaml() warns about each, naming exactly what will be lost:

UserWarning: Orders declares 1 __checks__ ('total_covers_subtotal') that cannot
be represented in YAML (a Check wraps an arbitrary polars.Expr) and will NOT be
written to orders.yaml. They will be lost on FrameSpec.from_yaml() unless
re-declared on a subclass of the loaded spec.

The suggested recovery is to subclass what you loaded:

Loaded = FrameSpec.from_yaml("orders.yaml")

class Orders(Loaded):
    __checks__ = [Check(pl.col("total") >= pl.col("subtotal"), name="total_covers_subtotal")]

Columns, rules, unique keys, foreign keys, and any check or validator written with col() come from the file; only raw-expression parts need re-declaring in Python. A check in YAML is its predicate in data form:

checks:
- expr:
    ge:
    - col: total
    - col: subtotal
  name: total_covers_subtotal

Sharing categories between files¶

A spec file can reference a CatSpec registry by path, resolved relative to the spec file:

name: Orders
categories: categories.yaml
columns:
  status:
    dtype:
      Enum: STATUS
  currency:
    dtype:
      Categorical: CURRENCY

$categories.STATUS and categories.STATUS are accepted as prefixed forms of the same reference.

Or pass a registry explicitly, which wins over anything the file names:

FrameSpec.from_yaml("orders.yaml", categories=CatSpec.from_yaml("categories.yaml"))
FrameSpec.from_yaml("orders.yaml", categories="categories.yaml")

A spec can also emit the registry its own columns imply:

Orders.catspec().to_yaml("categories.yaml")

Python instead of YAML¶

The same spec can be written as an importable Python module. It is the right choice when the spec will be edited by hand from now on, or when it needs the parts YAML cannot hold:

Orders.to_python("orders_spec.py")

"""Declares the Orders schema."""

import polars as pl
from polspec import ColSpec, FrameSpec


class Orders(FrameSpec):
    __columns__ = {
        'order_id': ColSpec(pl.Int64, bounds=(1, 100000), unique=True),
        'status': ColSpec(pl.Enum(['NEW', 'PAID', 'SHIPPED'])),
        'total': ColSpec(pl.Float64, bounds=(0.0, None)),
        'placed': ColSpec(pl.Date, nullable=True),
    }
    __unique_together__ = [['order_id', 'status']]

Columns are declared through __columns__ because a name straight from data is not always a valid identifier. What survives is exactly the round-trip table above: __checks__, cross-spec ForeignKeys and ColSpec.validators warn and are dropped, and the file is where you then add them back by hand. polspec schema infer uses this path when its output ends in .py; see Command line.

Column names from data¶

from_yaml and to_python declare columns through __columns__, so names that could not be class attributes — a leading underscore, a collision with a method name like schema, or a name with spaces — load correctly. The YAML key is the column's real name; a col_name set in a class body is not written, because the key already carries it. See Column names that are not identifiers.

Several specs in one file¶

A Registry writes every spec it holds, and the categories it was declared with, to one file keyed by spec name, and reads it back with the same version and strictness rules:

Registry(Customers, Orders, OrderLines, categories=categories).to_yaml("specs.yaml")
registry = Registry.from_yaml("specs.yaml").resolve()