Specs as values¶
A FrameSpec class body is the convenient way to write a spec. What it
builds is a TableSpec: an immutable value holding the columns, checks,
composite keys and foreign keys, reachable as .spec on the class.
import polars as pl
from polspec import ColSpec, ForeignKey, FrameSpec, Registry
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, unique=True)
total = ColSpec(pl.Float64, bounds=(0.0, None))
internal_note = ColSpec(pl.String, nullable=True)
Orders.spec # TableSpec(name='Orders', columns={...}, ...)
Orders.spec.name # 'Orders'
list(Orders.spec) # ['order_id', 'total', 'internal_note']
Orders.spec["total"] # the ColSpec
Orders.spec.schema() # the same pl.Schema as Orders.schema()
Every verb the library offers is a function over a TableSpec; the
classmethods on FrameSpec are one-line forwards that pass cls.spec. So a
TableSpec is the thing being operated on either way:
import polspec
Orders.generate(1_000, seed=1) # the class
polspec.generate(Orders.spec, 1_000, seed=1) # the function, over the value
Both reach the same code. The functions are exported from polspec itself:
from polspec import generate, generate_batches, inspect, validate
from polspec import sink_csv, sink_ipc, sink_ndjson, sink_parquet
df = generate(Orders.spec, 1_000, seed=1)
report = inspect(Orders.spec, df)
validate(Orders.spec, df)
Each takes a TableSpec as its first argument, and each has a FrameSpec
classmethod that forwards to it with cls.spec. Use whichever suits the code:
the classmethods read better when a class body declared the spec, the
functions when the spec is a value that was built, loaded or derived.
A TableSpec itself has no verb methods, so a spec stays data that every
verb reads. Asked for one, it names the function: spec.validate raises an
AttributeError saying to call polspec.validate(spec, ...), or to wrap
the spec with FrameSpec.from_spec(spec). Since 1.7. See How the pieces
fit.
Building one directly¶
A TableSpec can be constructed without a class body, which is how
from_yaml and from_dataframe work internally:
from polspec import TableSpec
spec = TableSpec(
"Orders",
{"order_id": ColSpec(pl.Int64, unique=True), "total": ColSpec(pl.Float64)},
unique_together=[["order_id"]],
)
Everything a class body validates at declaration is validated here too. A
TableSpec that constructs is one that can be used.
A spec of nothing but a schema's dtypes — the columns of a frame you
already have, empty or not, or a pyarrow schema — comes from
TableSpec.from_schema. Each column declares its dtype alone, so it is a
starting point to add claims to, not a profile of data:
starter = TableSpec.from_schema(pl.Schema({"order_id": pl.Int64, "total": pl.Float64}), name="Orders")
starter # each column a ColSpec of its dtype, and nothing more
Added in 1.2.
A spec prints as it was declared: each column on its own line, showing only
what it claims beyond its dtype — ColSpec(Int64, unique=True) — and only
the table claims that are not the default.
To get the class-shaped API back, wrap it:
Rebuilt = FrameSpec.from_spec(spec) # a subclass named Orders
Renamed = FrameSpec.from_spec(spec, name="Orders2026")
Deriving one spec from another¶
Each operation returns a new TableSpec; the original is never changed.
| Operation | Effect |
|---|---|
with_columns({...}, **cols) |
Add columns, or replace existing ones in place |
drop(*names) |
Remove columns, and any composite key, foreign key or check that used them |
select(*names) |
Keep only the named columns, in that order, and the constraints that still apply |
rename({old: new}) |
Rename columns, rewriting rules, validators, checks, composite keys, foreign keys and the hierarchy |
prefix(text), suffix(text) |
Rename every column, as rename does |
join(other, on=..., how=...) |
The spec of a join of two frames; see below |
with_checks(*checks), with_foreign_keys(*fks), with_unique_together(*groups) |
Append constraints |
with_name(name) |
Change the name |
with_catspec(registry) |
Re-type columns against a CatSpec; see Shared categories |
staging = Orders.spec.drop("internal_note").rename({"total": "amount"})
Staging = FrameSpec.from_spec(staging, name="StagingOrders")
Two deliberate limits. drop leaves a rule on a surviving column that points
at a dropped one for validation to reject, since silently dropping a rule
would change what the surviving column generates. rename rewrites a
validator or check written with col()
like everything else, but refuses to rename a column that one written as a
raw pl.Expr names: rewriting Polars expressions is not something this
library does, and leaving it would keep a check naming a column the spec no
longer has.
Joining two specs¶
join takes the arguments LazyFrame.join takes — on, or left_on and
right_on, how, suffix, coalesce — and returns the spec of the frame
that join makes. Its columns, their order and their names are the ones the
installed Polars gives: polspec runs the join on two empty frames and reads
the result, so a suffixed clash or a coalesced key is named as it will be.
Added in 1.1.
class Customers(FrameSpec):
customer_id = ColSpec(pl.Int64, unique=True)
tier = ColSpec(pl.Enum(["gold", "silver"]))
class Sales(FrameSpec):
sale_id = ColSpec(pl.Int64, unique=True)
customer_id = ColSpec(pl.Int64)
enriched = Sales.spec.join(Customers, on="customer_id", how="left")
enriched.name # 'Sales_Customers'
enriched["tier"].nullable # True: a sale may meet no customer
enriched["sale_id"].unique # True: each sale meets one customer at most
Each column keeps what it claims about its values — dtype, bounds,
choices, weights, format, pattern, lengths, tags, description, validators
and default — from the side it came from. A side that can go unmatched
becomes nullable: the right of left, the left of right, both of full.
What a spec says about its rows is not carried, since a join changes
which rows there are: checks, rules, foreign keys, composite keys, the
hierarchy, the order and rows are dropped; add back any that still holds
with its with_*. unique stays on a side's columns only when the other
side's join key is declared unique — a unique column, or a composite key —
because then no row of that side is repeated. semi and anti keep the
left spec, but for rows and the order: Polars does not promise a join
keeps its rows' order. Join keys of different dtypes are refused, as Polars
refuses them.
Naming a column from code¶
names gives each column's name as an attribute, so code that reads a
column — a Polars expression, a rule, a pipeline — need not spell it as
text, and a typo is an AttributeError naming the closest column:
class Invoices(FrameSpec):
unit_price = ColSpec(pl.Float64, col_name="Unit Price")
quantity = pl.Int64
Invoices.names.unit_price # "Unit Price" -- the name the data uses
Invoices.names.quantity # "quantity"
list(Invoices.names) # ["Unit Price", "quantity"]
pl.col(Invoices.names.unit_price) * pl.col(Invoices.names.quantity)
A TableSpec has names too, each column under its own name. A column
called names is reached the way any column named like a method is, with
Invoices.col("names"). Added in 1.2.
To change one field of a column, replace makes a new one, checked as a
new declaration is: Invoices.col("quantity").replace(nullable=True).
Column names and method names¶
Because the class body's ColSpec attributes are taken out of the namespace
before the class exists, a column may share a name with a method. The method
wins on attribute access; the column is reachable by name:
class Raw(FrameSpec):
schema = ColSpec(pl.String)
tag = ColSpec(pl.String)
Raw.schema() # the method: Schema({'schema': String, 'tag': String})
Raw.col("schema") # the column
Raw.spec["tag"] # also the column
An ordinary column is still an attribute (Orders.order_id), through a
fallback that runs only when normal lookup fails.
Foreign keys point at names¶
ForeignKey.references is stored as the target spec's name. Declaring
references=Customers binds the target for declaration-time checks and
stores "Customers"; a key can also be declared against a bare name, which
nothing checks until a spec of that name is supplied:
class Customers(FrameSpec):
id = ColSpec(pl.Int64, bounds=(1, 10_000), unique=True)
ForeignKey("customer_id", references=Customers, ref_columns="id") # checked now
ForeignKey("customer_id", references="Customers", ref_columns="id") # checked later
generate(references={...}) and validate(references={...}) accept the
parent frame keyed by the class, the TableSpec, or the name. A
Registry holding both specs binds the name and runs the
checks the class form would have run at declaration:
class Shipments(FrameSpec):
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
__foreign_keys__ = [
ForeignKey("customer_id", references="Customers", ref_columns="id")
]
Shipments.spec.foreign_keys[0].target # None -- nothing to check against
bound = Registry(Customers, Shipments).resolve()
bound["Shipments"].foreign_keys[0].target # Customers.spec
Handing a spec to Polars, Arrow or another process¶
A spec, or its class, reads as an Arrow schema, so anything that takes a schema through the Arrow interface takes it:
pl.Schema(Orders) # the same as Orders.schema()
pl.DataFrame(schema=pl.Schema(Orders)) # an empty frame with the spec's columns
pyarrow.schema(Orders) gives the Arrow schema in the same way.
A spec, a registry, a report and a drift report each pickle and deep-copy,
so they can go to another process — a multiprocessing pool, joblib,
Dask, Ray. A report keeps the rows of each finding, and its
passing_rows(), on the far side. A FrameSpec class pickles by
reference, as any class does, so it has to be importable where it is
unpickled; for a class built at runtime — from_yaml, from_spec —
pickle its .spec. Added in 1.2.