Specs¶
A spec is a TableSpec: an immutable record of columns and constraints.
FrameSpec is the class syntax that builds one and forwards every verb to it.
TableSpec¶
TableSpec
dataclass
¶
TableSpec(name: str, columns: Mapping[str, ColSpec | DataType | type[DataType]] = FrozenDict(), *, description: str | None = None, rows: Bound[int] | tuple[int | None, int | None] | None = None, checks: Sequence[Check] = (), unique_together: Sequence[Sequence[str]] = (), foreign_keys: Sequence[ForeignKey] = (), hierarchy: Hierarchy | None = None, order_by: str | Sequence[str] = (), descending: bool | Sequence[bool] = False)
The columns and constraints of one table, as an immutable value.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
What the table is called: the class name for a |
required |
columns
|
Mapping[str, ColSpec | DataType]
|
Column name to declaration, in the order columns should appear. A
bare dtype is a column of it: |
FrozenDict()
|
description
|
str | None
|
What the table holds, for a reader: |
None
|
rows
|
Bound[int] | tuple[int | None, int | None] | None
|
How many rows the table may hold, either end open: |
None
|
checks
|
Sequence[Check]
|
Multi-column invariants; see |
()
|
unique_together
|
Sequence[Sequence[str]]
|
Composite unique keys. A single group may be given as a flat list. |
()
|
foreign_keys
|
Sequence[ForeignKey]
|
Referential-integrity constraints. |
()
|
hierarchy
|
Hierarchy | None
|
Declares two columns as a parent/child edge list; see |
None
|
order_by
|
Sequence[str]
|
Columns the rows come sorted by, the first the major key, as
|
()
|
descending
|
bool | Sequence[bool]
|
Whether each |
False
|
Notes
Everything a FrameSpec class body validates at declaration is validated
here, so a TableSpec that constructs is one that can be used.
Examples:
>>> spec = TableSpec("Orders", {
... "id": ColSpec(pl.Int64, unique=True),
... "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> spec
TableSpec('Orders', {
'id': ColSpec(Int64, unique=True),
'total': ColSpec(Float64, bounds=(0, 100)),
})
>>> spec.select("total").schema()
Schema({'total': Float64})
>>> spec.names.total
'total'
names
property
¶
Each column's name, as an attribute: spec.names.total is
"total", so code can name a column without spelling it as text.
from_schema
classmethod
¶
from_schema(schema: Any, *, name: str = 'Schema') -> TableSpec
A spec of schema's columns, each declaring nothing but its dtype.
schema is a pl.Schema, a mapping of column name to dtype, or
anything with an Arrow schema (__arrow_c_schema__): a pyarrow
schema, another spec. Each column is ColSpec(dtype) -- not
nullable, so a frame with nulls fails until a column says it may
hold them. For a spec that says more, profile data: profile().
tag
¶
Column names carrying any (or all) of the tags, in declaration order.
with_rows
¶
A copy of this spec holding rows rows; None for any number.
with_description
¶
with_description(description: str | None) -> TableSpec
A copy of this spec with description in place of its own.
with_columns
¶
Adds columns, or replaces existing ones of the same name in place.
with_checks
¶
A copy of this spec with checks added to the ones it has.
with_foreign_keys
¶
with_foreign_keys(*foreign_keys: ForeignKey) -> TableSpec
A copy of this spec with foreign_keys added to the ones it has.
with_hierarchy
¶
A copy of this spec declaring (or, with None, dropping) a hierarchy.
with_unique_together
¶
with_unique_together(*groups: Sequence[str]) -> TableSpec
A copy of this spec with groups added as composite unique keys.
with_order_by
¶
with_order_by(*names: str, descending: bool | Sequence[bool] = False) -> TableSpec
A copy of this spec whose rows come sorted by names; with none,
a copy with no order.
drop
¶
drop(*names: str | Sequence[str]) -> TableSpec
Removes columns, and any composite key, foreign key or check that
used them. The names may be given one by one or as a list, as
Polars' own drop takes them.
select
¶
select(*names: str | Sequence[str]) -> TableSpec
Keeps only the named columns, in the order given: one by one or
as a list, as Polars' own select takes them.
rename
¶
rename(mapping: Mapping[str, str]) -> TableSpec
Renames columns, rewriting every constraint that names them.
Rules, validators, checks, composite keys, foreign keys, the
hierarchy and the order are rewritten. A validator or check written as a raw
pl.Expr cannot be: rewriting Polars expressions is not something
this library does, so renaming a column one names is refused. Write it
with col(), whose predicates rename like everything else.
prefix
¶
prefix(text: str) -> TableSpec
Every column renamed with text in front, as rename renames:
every claim follows, and a raw pl.Expr naming a column is refused.
join
¶
join(other: SpecLike, on: str | Sequence[str] | None = None, *, left_on: str | Sequence[str] | None = None, right_on: str | Sequence[str] | None = None, how: JoinHow = 'inner', suffix: str = '_right', coalesce: bool | None = None, name: str | None = None) -> TableSpec
The spec of frame.join(other_frame, ...), for frames of this spec
and other, with the arguments LazyFrame.join takes.
The columns, their order and their names are those the installed
Polars gives the join, read from a join of two empty frames, so every
how, suffix and coalesce behaves as it does on data. Each
column then carries the claims about its values from the side it
came from: dtype, bounds, choices, weights, format, pattern,
lengths, tags, description, validators and default. A side that can
go unmatched -- the right of left, the left of right, both of
full -- becomes nullable.
A claim about rows is not carried: checks, rules, foreign keys,
composite keys, the hierarchy, the order and rows are dropped,
each to be added back with its with_* if it still holds. A
column's unique survives only when the other side's join key is
declared unique -- a single unique column, or a composite key -- since
then no row of this side is repeated. semi and anti keep this
spec as it is, but for rows and the order: Polars does not promise
a join keeps its rows' order.
The name defaults to Left_Right. Join keys of different dtypes
are refused, as Polars refuses them.
with_catspec
¶
Re-points columns at the registry's Enum and Categorical types.
A column whose name resolves in the registry (exactly, or by case)
takes the registry's dtype; everything else it declared carries over.
choices a new Enum cannot hold, and weights over a domain that
changed size, are dropped with a warning.
Takes the registry as a value or as the class body that declares it.
FrameSpec¶
FrameSpec
¶
Base class for declaring a DataFrame/LazyFrame specification.
Subclass it and assign a ColSpec per column, in the order columns should
appear:
class DataSource(FrameSpec):
string_1 = ColSpec(pl.String)
enum_1 = ColSpec(pl.Enum(["mammal", "reptile"]), nullable=True)
int_1 = ColSpec(pl.Int64, bounds=(-100, 100), nullable=True)
df = DataSource.generate(1_000_000, seed=42)
The class body builds DataSource.spec, a TableSpec; every classmethod
here forwards to a function over it. A column may take any name: one that
collides with a method (schema, tag, ...) is reachable as
DataSource.col("schema") while the method keeps working. Names that
cannot be attributes at all -- a leading underscore, or one straight from
data -- go through __columns__:
class Raw(FrameSpec):
__columns__ = {"_id": ColSpec(pl.Int64), "Unit Price": ColSpec(pl.Float64)}
Examples:
>>> class Orders(FrameSpec):
... "One row per order."
...
... id = ColSpec(pl.Int64, unique=True)
... status = ColSpec(pl.Enum, choices={"NEW": 0.7, "PAID": 0.3})
... total = ColSpec(pl.Float64, bounds=(0, 100))
...
>>> frame = Orders.generate(1_000, seed=1)
>>> Orders.validate(frame).height
1000
>>> Orders.spec.description
'One row per order.'
>>> Orders.names.status
'status'
from_spec
classmethod
¶
A FrameSpec subclass wrapping an existing TableSpec.
from_yaml
classmethod
¶
from_yaml(source: str | Path, *, categories: CatSpec | type[CatSpec] | str | Path | None = None, strict: bool = True) -> type[FrameSpec]
Builds a new FrameSpec subclass from a YAML file written by to_yaml.
categories is a CatSpec registry, or a path to one, used to resolve
shared Enums and Categoricals; when omitted, a categories: key in
the file is loaded automatically. An unknown key in the file is an
error unless strict=False, which downgrades it to a warning.
from_dataframe
classmethod
¶
from_dataframe(df: DataFrame, *, name: str = 'Profiled', options: ProfileOptions | None = None, **keywords: Unpack[ProfileKeywords]) -> type[FrameSpec]
Infers a spec by profiling an existing DataFrame.
polspec.profile with less recorded by default -- weights,
shape, detect_unique and detect_order off -- so it describes
what each column is rather than a stand-in for the data. A string or
categorical column with at most max_unique_enum distinct values,
repeating, becomes an Enum; one whose values have a format -- an
email, a UUID, a country code -- names it, unless formats=False;
one of codes writes the pattern they share, unless
patterns=False.
calculate_bounds records observed (min, max) for numeric and
temporal columns and (min_len, max_len) for strings and binary.
options is a ProfileOptions, or pass its fields as keywords:
round_bounds and round_rates write bounds and rates at a
person's precision, as polspec schema infer does by default.
tag
classmethod
¶
Column names carrying any (or all) of the tags, in declaration order.
catspec
classmethod
¶
catspec() -> CatSpec
The CatSpec registry this spec's Enum and Categorical columns imply.
with_catspec
classmethod
¶
A new FrameSpec subclass with columns re-typed against catspec.
to_yaml
classmethod
¶
This spec as human-readable YAML: written to the file path, or,
with no path, returned as text.
to_python
classmethod
¶
This spec as an importable Python module defining a subclass:
written to the file path, or, with no path, returned as
text.
generate
classmethod
¶
generate(n: int, *, method: Literal['random', 'cartesian'] = 'random', seed: int | None = None, references: References = None, cycles: int = 0, self_references: int = 0, max_bytes: int | None = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> DataFrame
Generates a DataFrame matching this spec.
cycles and self_references apply only to a spec declaring a
__hierarchy__, and deliberately violate it. See
polspec.generation.generate for the full contract. scan() is
the lazy verb.
estimated_size
classmethod
¶
Bytes a frame of n generated rows is expected to hold.
See polspec.estimated_size. generate() warns when the estimate
passes four gibibytes, and takes max_bytes= to refuse instead.
scan
classmethod
¶
scan(n: int, *, seed: int | None = None, batch_size: int | None = None, method: Literal['random', 'cartesian'] = 'random', references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> LazyFrame
A LazyFrame of n rows, generated as they are collected.
Unlike generate(), which builds the whole frame before it
returns, nothing is generated until the plan is collected -- and
then only the columns and rows the plan asks for:
Orders.scan(50_000_000, seed=1).sink_parquet("orders.parquet")
Orders.scan(50_000_000, seed=1).select("total").head(5).collect()
The first streams in bounded memory; the second generates five rows of one column. A projected column holds the values the whole frame would: every column is seeded by its name, so dropping its neighbours cannot move it.
Rows come in batches, so a scan carries generate_batches' terms:
a spec declaring a __hierarchy__ is refused, and a unique=True
column or a __unique_together__ group is distinct within a batch
rather than across n. Leaving batch_size unset lets polars ask
for the size it wants; setting it pins the size.
generate_batches
classmethod
¶
generate_batches(n: int, *, batch_size: int = 100000, method: Literal['random', 'cartesian'] = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> Iterator[DataFrame]
Yields chunks of generated rows without holding all n in memory.
Each batch is a window onto the one frame seed describes: a column
no pass rewrites holds, batch by batch, exactly the rows
generate(n, seed=seed) would, whatever batch_size is.
sink_parquet
classmethod
¶
sink_parquet(path: str | Path, n: int, *, batch_size: int = 100000, compression: ParquetCompression = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None
Generates n rows and streams them to a Parquet file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_parquet.
sink_csv
classmethod
¶
sink_csv(path: str | Path, n: int, *, batch_size: int = 100000, include_header: bool = True, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None
Generates n rows and streams them to a CSV file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_csv.
sink_ipc
classmethod
¶
sink_ipc(path: str | Path, n: int, *, batch_size: int = 100000, compression: IpcCompression | None = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None
Generates n rows and streams them to an Arrow IPC file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_ipc.
sink_ndjson
classmethod
¶
sink_ndjson(path: str | Path, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None
Generates n rows and streams them to an NDJSON file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_ndjson.
diff
classmethod
¶
diff(other: TableSpec | type[FrameSpec], *, renames: Mapping[str, str] | None = None, options: DriftOptions | None = None, **keywords: Unpack[DriftKeywords]) -> DriftReport
What changed from this spec to other, as a DriftReport.
This spec is the old side. See polspec.diff.
drift
classmethod
¶
drift(df: DataFrame | LazyFrame, *, options: DriftOptions | None = None, **keywords: Unpack[DriftKeywords]) -> DriftReport
How df has moved relative to this spec, as a DriftReport:
polspec.drift, which takes every DriftOptions field as a keyword
or options= as an object.
to_markdown
classmethod
¶
to_markdown(path: str | Path | None = None, *, title: str | None = None, choices: Literal['summary', 'appendix', 'full'] = 'summary') -> str
A Markdown data dictionary for this spec, written to path if
given; choices as polspec.to_markdown takes it.
to_mermaid
classmethod
¶
A Mermaid entity-relationship diagram for this spec.
read
classmethod
¶
A data file as a frame, in the terms this spec declares -- read, not
validated. See polspec.read: the extension picks the reader, a
declared date or time that arrived as text is parsed, and nothing else
is cast.
inspect
classmethod
¶
inspect(df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **keywords: Unpack[ValidationKeywords]) -> ValidationReport
Everything this spec has to say about df, as a ValidationReport.
Never raises for a frame that fails: each violation is a Finding
with a code, a count, samples and, for row-level findings, the
offending rows reachable lazily through report.rows(finding) or
report.failing_rows(). Takes the same options as validate.
validate
classmethod
¶
validate(df: DataFrame, *, options: ValidationOptions | None = None, references: References = None, **keywords: Unpack[ValidationKeywords]) -> DataFrame
validate(df: LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **keywords: Unpack[ValidationKeywords]) -> LazyFrame
validate(df: DataFrame | LazyFrame, *, options: ValidationOptions | None = None, references: References = None, **keywords: Unpack[ValidationKeywords]) -> DataFrame | LazyFrame
Validates a DataFrame or LazyFrame against this spec.
Raises ValidationError carrying a ValidationReport of every
violation, or returns the (optionally transformed) frame -- a
LazyFrame if df was one. Use inspect for the report without the
exception.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame | LazyFrame
|
The frame to validate. |
required |
options
|
ValidationOptions
|
Every option at once, as a value. Cannot be combined with the keywords below. |
None
|
references
|
mapping
|
Parent frames for foreign keys into other specs, keyed by that spec, its FrameSpec class, or its name. |
None
|
**keywords
|
Unpack[ValidationKeywords]
|
The options one at a time, as |
{}
|
Notes
A keyword left out, or given as None, takes the
ValidationOptions default; see that class for what every option
means.
ForeignKey¶
ForeignKey
dataclass
¶
ForeignKey(columns: str | Sequence[str], *, references: type[FrameSpec] | TableSpec | str, ref_columns: str | Sequence[str] | None = None, name: str | None = None)
Declares referential integrity: one or more columns must only contain values that exist in another FrameSpec's (or this same FrameSpec's) columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
columns
|
str | Sequence[str]
|
The local column(s) that must reference existing parent values. |
required |
references
|
type[FrameSpec] | TableSpec | str
|
The spec this key references -- a After construction |
required |
ref_columns
|
str | Sequence[str] | None
|
The referenced column(s) on the target, in the same order as
|
None
|
name
|
str | None
|
A human-readable identifier. Defaults to a name derived from the columns and target. |
None
|
Notes
Rows where any of columns is null are exempt (standard FK semantics --
a null foreign key means "no reference", not "an invalid one").
Examples:
>>> class CustomerSpec(FrameSpec):
... id = ColSpec(pl.Int64, bounds=(1, 1_000), unique=True)
>>> class OrderSpec(FrameSpec):
... customer_id = ColSpec(pl.Int64, bounds=(1, 1_000))
... __foreign_keys__ = [
... ForeignKey("customer_id", references=CustomerSpec, ref_columns="id"),
... ]
>>> class EmployeeSpec(FrameSpec):
... id = ColSpec(pl.Int64, unique=True)
... manager_id = ColSpec(pl.Int64, nullable=True)
... __foreign_keys__ = [
... ForeignKey("manager_id", references="self", ref_columns="id"),
... ]
Hierarchy¶
Hierarchy
dataclass
¶
Hierarchy(*, child: str, parent: str, max_depth: int = 1, branching: float | None = None, roots: int | None = None)
Declares that two columns of a spec form a parent/child edge list.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
child
|
str
|
The column holding the lower reference -- the one doing the pointing. Each value appears in exactly one row, which is what gives every reference a single ultimate parent. |
required |
parent
|
str
|
The column holding the reference being pointed at. |
required |
max_depth
|
int
|
Hops from an ultimate parent to the furthest child. Generation guarantees at least one chain of exactly this length and none longer, so a walk that stops early and one that runs away are both caught. |
1
|
branching
|
float | None
|
Mean children per reference, which is what decides how many ultimate
parents |
None
|
roots
|
int | None
|
An exact number of ultimate parents, as an alternative to |
None
|
Notes
A row is an edge, so generate(n) produces n rows. Ultimate parents have
no row of their own -- nothing to point at -- so n edges over R roots
need n + R distinct references, drawn from the child column's own
declaration.
Examples:
>>> class Links(FrameSpec):
... PARENT_REF = ColSpec(pl.String)
... CHILD_REF = ColSpec(pl.String)
... __hierarchy__ = Hierarchy(
... child="CHILD_REF", parent="PARENT_REF", max_depth=5
... )