Skip to content

Registry and categories

A declared set of specs, and the shared category domains they draw on.

Registry

Registry

Registry(*specs: TableSpec | type, categories: CatSpec | type[CatSpec] | None = None)

A declared set of specs, with everything that needs more than one.

Parameters:

Name Type Description Default
*specs TableSpec | type[FrameSpec]

The specs, in any order. Each is stored under its name; two different specs with one name are an error.

()
categories CatSpec | None

A shared category registry the specs are expected to agree with. When given, resolve() checks every Enum/Categorical column that binds to one of its entries against it, and the registry file carries it. When omitted, catspec() derives one from the specs themselves.

None

Examples:

>>> customers = TableSpec("Customers", {"id": ColSpec(pl.Int64, bounds=(1, 1_000), unique=True)})
>>> orders = TableSpec("Orders", {"customer_id": ColSpec(pl.Int64, bounds=(1, 1_000))},
...     foreign_keys=[ForeignKey("customer_id", references="Customers", ref_columns="id")])
>>> registry = Registry(customers, orders)
>>> frames = registry.generate_all({"Customers": 100, "Orders": 500}, seed=1)
>>> {name: frame.height for name, frame in frames.items()}
{'Customers': 100, 'Orders': 500}
>>> sorted(registry.validate_all(frames))  # every frame passed
['Customers', 'Orders']

names property

names: tuple[str, ...]

Spec names in the order they were added.

categories property

categories: CatSpec | None

The shared category registry this one was declared with, if any.

add

add(spec: TableSpec | type) -> Registry

Adds a spec, returning the registry so calls chain.

replace

replace(spec: TableSpec | type) -> Registry

A new registry with spec in place of the one of its name, in the same place and with the same categories. This one is left as it was. A name the registry does not hold is a RegistryError.

Examples:

>>> registry = Registry(TableSpec("Orders", {"qty": ColSpec(pl.Int64, bounds=(1, 5))}))
>>> wider = registry["Orders"].with_columns({"qty": ColSpec(pl.Int64, bounds=(1, 20))})
>>> registry.replace(wider)["Orders"]["qty"].bounds
Bound(min=1, max=20)

Added in 1.6.

from_module classmethod

from_module(module: ModuleType, *, own_only: bool = False, categories: CatSpec | type[CatSpec] | None = None) -> Registry

Every FrameSpec subclass and TableSpec bound in a module.

own_only=True keeps only classes the module itself defines, leaving out ones it imported.

discover classmethod

discover(*paths: str | Path, categories: CatSpec | type[CatSpec] | None = None, strict: bool = True) -> Registry

Every spec found under the given files and directories.

A .py file is imported and searched like from_module; a .yaml file is read as a spec, or as a registry file when it has a specs: key; a directory is walked for both. Importing a Python file runs it, so point this only at files you would import anyway.

resolve

resolve() -> Registry

A registry whose every cross-spec key is bound to its target.

Binding runs the checks a key declared against a class gets at declaration -- the referenced columns exist and are dtype-compatible -- for keys that were declared against a bare name or read from a file. Also refuses a key whose target is not in the registry, a cycle between specs, and a column disagreeing with categories.

Not a prerequisite for anything: generate_all and inspect_all run the same binding themselves. Call it to check a registry, or to hold on to the bound specs.

parents

parents(key: Any) -> tuple[str, ...]

Names of the specs one spec's foreign keys point at, self excluded.

order

order() -> tuple[str, ...]

Every spec name, parents before children.

Specs with no dependency between them keep the order they were added in. A target outside the registry imposes no order; a cycle is an error.

ancestors

ancestors(key: Any) -> tuple[str, ...]

Every spec a spec depends on, directly or through other specs.

scan_all

scan_all(n: int | Mapping[Any, int], *, seed: int | None = None, batch_size: int | None = None, references: Frames | None = None, fixed: Mapping[Any, Mapping[str, Any]] | None = None, checks: bool = False) -> dict[str, LazyFrame]

One LazyFrame per spec, each generating as it is collected.

Parents are generated eagerly and threaded into their children, as generate_all does -- a foreign key needs the whole parent column to sample from, so only the children are lazy. A spec with no children is lazy either way.

Each spec's seed is derived from seed and its name, as in generate_all, so the frames agree with the eager ones. A scan carries generate_batches' terms: see FrameSpec.scan.

generate_all

generate_all(n: int | Mapping[Any, int], *, seed: int | None = None, method: Literal['random', 'cartesian'] = 'random', references: Frames | None = None, fixed: Mapping[Any, Mapping[str, Any]] | None = None, checks: bool = False) -> dict[str, DataFrame]

One frame per spec, parents generated first and threaded into their children, so every foreign key is satisfied by construction.

n is a row count for every spec, or a mapping from spec (or name) to its own count. Each spec's seed is derived from seed and its name, so adding a spec to the registry never changes the rows another one produces. A frame in references is used as-is in place of generating that spec, and also serves parents outside the registry.

fixed pins columns spec by spec, {Orders: {"status": "PAID"}}, each as generate() takes it; a child's foreign key draws from its pinned parent's rows. Added in 1.4.

checks=True keeps each spec's checks, as generate() does; a child's foreign key draws from its parent's kept rows. Added in 1.7.

generate_related(key: Any, n: int | Mapping[Any, int], *, seed: int | None = None, method: Literal['random', 'cartesian'] = 'random', references: Frames | None = None, fixed: Mapping[Any, Mapping[str, Any]] | None = None, checks: bool = False) -> dict[str, DataFrame]

generate_all restricted to one spec and everything it depends on.

inspect_all

inspect_all(frames: Frames, *, options: ValidationOptions | None = None, references: Frames | None = None, **keywords: Unpack[ValidationKeywords]) -> dict[str, ValidationReport]

A ValidationReport per frame, each spec seeing every other frame as a possible parent. Takes the options validate() does: an options= object, or its fields as keywords.

validate_all

validate_all(frames: Frames, *, options: ValidationOptions | None = None, references: Frames | None = None, **keywords: Unpack[ValidationKeywords]) -> dict[str, DataFrame | LazyFrame]

Validates every frame, or returns them with the structural transformations validate() applies.

Raises MultiValidationError -- a ValidationError, so one except still catches both -- carrying every failing spec's ValidationReport as reports, keyed by spec name.

catspec

catspec() -> CatSpec

The categories these specs share: the one declared, or one merged from every spec's Enum and Categorical columns.

Merging refuses two specs that define the same name differently; pass categories= to the registry to settle which is right.

to_dict

to_dict() -> dict[str, Any]

This registry as plain data: every spec, plus shared categories.

from_dict classmethod

from_dict(data: Mapping[str, Any], *, strict: bool = True) -> Registry

A registry read from the data form to_dict writes.

strict=False downgrades an unknown key from an error to a warning.

to_yaml

to_yaml(path: str | Path | None = None) -> str | None

Every spec, and the declared categories, as one YAML document: written to the file path, or, with no path, returned as text.

from_yaml classmethod

from_yaml(source: str | Path, *, strict: bool = True) -> Registry

A registry read from one YAML file written by to_yaml.

strict=False downgrades an unknown key from an error to a warning.

to_mermaid

to_mermaid(path: str | Path | None = None) -> str

One entity-relationship diagram with every spec and every key.

CatSpec

CatSpec

CatSpec(*, enums: Mapping[str, Sequence[str]] | None = None, categoricals: Mapping[str, Categories | dict[str, Any] | str | DataType] | None = None, choices: Mapping[str, Sequence[Any]] | None = None)

A set of shared Enum and Categorical domains, as a value.

Parameters:

Name Type Description Default
enums Mapping[str, Sequence[str]]

Entry name to its ordered category list.

None
categoricals Mapping[str, Categories | dict | str | DataType]

Entry name to its shared pl.Categories. A physical dtype (pl.UInt8) or its name ("UInt8") is shorthand for a registry of that name; a mapping is the file form, and its categories key becomes choices.

None
choices Mapping[str, Sequence[Any]]

The pool of values an entry draws from when generating. An Enum's categories already are its pool; this is for Categorical entries, whose registry names the domain without listing it.

None
Notes

Naming an entry -- cats.STATUS, cats["STATUS"], cats.get("STATUS") -- gives back the dtype, ready to hand to ColSpec. Lookup is case-insensitive, so a column named status finds STATUS, and an Enum wins over a Categorical of the same name.

A subclass of CatSpec declares its entries in the class body; see the module docstring. Instantiating one takes those entries as defaults, so Categories(enums={"REASON": [...]}) extends rather than replaces.

Examples:

>>> cats = CatSpec(enums={"STATUS": ["PENDING", "COMPLETED"]})
>>> cats.STATUS
Enum(categories=['PENDING', 'COMPLETED'])
>>> cats["status"]
Enum(categories=['PENDING', 'COMPLETED'])
>>> cats.enums["STATUS"]
['PENDING', 'COMPLETED']

enums property

enums: Mapping[str, list[str]]

Every Enum entry: name to its ordered category list.

categoricals property

categoricals: Mapping[str, Categories]

Every Categorical entry: name to its shared pl.Categories.

choices property

choices: Mapping[str, list[Any]]

Every recorded domain pool, keyed by entry name.

get

get(name: str, default: Any = None) -> Any

The dtype registered under name, or default if nothing is.

from_dataframe classmethod

from_dataframe(df: DataFrame | LazyFrame) -> CatSpec

The Enum and Categorical columns a frame already declares.

Reads what is there; infer is what looks at String columns and decides what could be one.

from_framespec classmethod

from_framespec(spec: TableSpec | type[FrameSpec]) -> CatSpec

The Enum and Categorical columns a spec already declares.

infer classmethod

infer(target: DataFrame | LazyFrame | TableSpec | type[FrameSpec], *, max_enum_cardinality: int = 30, max_categorical_cardinality: int = 10000, max_categorical_ratio: float = 0.2, include_columns: Sequence[str] | None = None, exclude_patterns: Sequence[str] | None = DEFAULT_EXCLUDE_PATTERNS, default_physical: DataType | None = None) -> CatSpec

A registry of the domains target looks like it has.

Existing Enum and Categorical columns are kept as declared. A String column becomes an Enum when it holds few enough distinct values, or a Categorical when it holds many but repeats them; one that looks like an identifier is skipped.

Parameters:

Name Type Description Default
target DataFrame | LazyFrame | TableSpec | type[FrameSpec]

Data to measure, or a spec to read declarations from.

required
max_enum_cardinality int

At most this many distinct values makes a column an Enum.

30
max_categorical_cardinality int

Beyond this many, a column is left as String.

10000
max_categorical_ratio float

Distinct values as a fraction of rows, above which a column is too close to unique to be worth a category registry. Frames only -- a spec has no row count to measure against.

0.2
include_columns Sequence[str]

Consider only these columns, exempting them from exclude_patterns.

None
exclude_patterns Sequence[str]

Regexes for names to skip; defaults to identifier-shaped names (*_id, *_uuid, *_hash, *_url, *_key).

DEFAULT_EXCLUDE_PATTERNS
default_physical DataType

Physical dtype for new Categorical registries, instead of the narrowest one that fits.

None

from_dict classmethod

from_dict(data: dict[str, Any], *, strict: bool = True) -> CatSpec

A registry read from the data form to_dict writes.

from_yaml classmethod

from_yaml(source: str | Path, *, strict: bool = True) -> CatSpec

A registry read from a YAML file written by to_yaml.

to_dict

to_dict() -> dict[str, Any]

This registry as plain data, without the file's version key.

to_yaml

to_yaml(path: str | Path | None = None) -> str | None

Writes this registry as YAML to path, or returns the text.

to_markdown

to_markdown(path: str | Path | None = None, *, title: str | None = None) -> str

A Markdown table of every entry; written to path when given.

to_mermaid

to_mermaid(path: str | Path | None = None, *, title: str | None = None) -> str

A Mermaid class diagram of every entry; written to path when given.