Skip to content

Registry and categories

A declared set of specs, and the shared category domains they draw on.

Registry

Registry 

Registry(*specs: TableSpec | type, categories: CatSpec | type[CatSpec] | None = None)

A declared set of specs, with everything that needs more than one.

Parameters:

Name Type Description Default
*specs TableSpec | type[FrameSpec]

The specs, in any order. Each is stored under its name; two different specs with one name are an error.

 ()
categories CatSpec | None

A shared category registry the specs are expected to agree with. When given, resolve() checks every Enum/Categorical column that binds to one of its entries against it, and the registry file carries it. When omitted, catspec() derives one from the specs themselves.

 None

names property 

names: tuple[str, ...]

Spec names in the order they were added.

categories property 

categories: CatSpec | None

The shared category registry this one was declared with, if any.

add 

add(spec: TableSpec | type) -> Registry

Adds a spec, returning the registry so calls chain.

from_module classmethod 

from_module(module: ModuleType, *, own_only: bool = False, categories: CatSpec | type[CatSpec] | None = None) -> Registry

Every FrameSpec subclass and TableSpec bound in a module.

own_only=True keeps only classes the module itself defines, leaving out ones it imported.

discover classmethod 

discover(*paths: str | Path, categories: CatSpec | type[CatSpec] | None = None, strict: bool = True) -> Registry

Every spec found under the given files and directories.

A .py file is imported and searched like from_module; a .yaml file is read as a spec, or as a registry file when it has a specs: key; a directory is walked for both. Importing a Python file runs it, so point this only at files you would import anyway.

resolve 

resolve() -> Registry

A registry whose every cross-spec key is bound to its target.

Binding runs the checks a key declared against a class gets at declaration -- the referenced columns exist and are dtype-compatible -- for keys that were declared against a bare name or read from a file. Also refuses a key whose target is not in the registry, a cycle between specs, and a column disagreeing with categories.

Not a prerequisite for anything: generate_all and inspect_all run the same binding themselves. Call it to check a registry, or to hold on to the bound specs.

parents 

parents(key: Any) -> tuple[str, ...]

Names of the specs one spec's foreign keys point at, self excluded.

order 

order() -> tuple[str, ...]

Every spec name, parents before children.

Specs with no dependency between them keep the order they were added in. A target outside the registry imposes no order; a cycle is an error.

ancestors 

ancestors(key: Any) -> tuple[str, ...]

Every spec a spec depends on, directly or through other specs.

scan_all 

scan_all(n: int | Mapping[Any, int], *, seed: int | None = None, batch_size: int | None = None, references: Frames | None = None) -> dict[str, LazyFrame]

One LazyFrame per spec, each generating as it is collected.

Parents are generated eagerly and threaded into their children, as generate_all does -- a foreign key needs the whole parent column to sample from, so only the children are lazy. A spec with no children is lazy either way.

Each spec's seed is derived from seed and its name, as in generate_all, so the frames agree with the eager ones. A scan carries generate_batches' terms: see FrameSpec.scan.

generate_all 

generate_all(n: int | Mapping[Any, int], *, seed: int | None = None, method: Literal['random', 'cartesian'] = 'random', references: Frames | None = None) -> dict[str, DataFrame]

One frame per spec, parents generated first and threaded into their children, so every foreign key is satisfied by construction.

n is a row count for every spec, or a mapping from spec (or name) to its own count. Each spec's seed is derived from seed and its name, so adding a spec to the registry never changes the rows another one produces. A frame in references is used as-is in place of generating that spec, and also serves parents outside the registry.

generate_related(key: Any, n: int | Mapping[Any, int], *, seed: int | None = None, method: Literal['random', 'cartesian'] = 'random', references: Frames | None = None) -> dict[str, DataFrame]

generate_all restricted to one spec and everything it depends on.

inspect_all 

inspect_all(frames: Frames, *, references: Frames | None = None, **options: Any) -> dict[str, ValidationReport]

A ValidationReport per frame, each spec seeing every other frame as a possible parent. Takes the options validate() does.

validate_all 

validate_all(frames: Frames, *, references: Frames | None = None, **options: Any) -> dict[str, DataFrame | LazyFrame]

Validates every frame, or returns them with the structural transformations validate() applies.

Raises MultiValidationError -- a ValidationError, so one except still catches both -- carrying every failing spec's ValidationReport as reports, keyed by spec name.

catspec 

catspec() -> CatSpec

The categories these specs share: the one declared, or one merged from every spec's Enum and Categorical columns.

Merging refuses two specs that define the same name differently; pass categories= to the registry to settle which is right.

to_dict 

to_dict() -> dict[str, Any]

This registry as plain data: every spec, plus shared categories.

from_dict classmethod 

from_dict(data: Mapping[str, Any], *, strict: bool = True) -> Registry

A registry read from the data form to_dict writes.

strict=False downgrades an unknown key from an error to a warning.

to_yaml 

to_yaml(source: str | Path) -> None

Writes every spec, and the declared categories, to one file.

from_yaml classmethod 

from_yaml(source: str | Path, *, strict: bool = True) -> Registry

A registry read from one YAML file written by to_yaml.

strict=False downgrades an unknown key from an error to a warning.

to_mermaid 

to_mermaid(path: str | Path | None = None) -> str

One entity-relationship diagram with every spec and every key.

CatSpec

CatSpec 

CatSpec(*, enums: Mapping[str, Sequence[str]] | None = None, categoricals: Mapping[str, Categories | dict[str, Any] | str | DataType] | None = None, choices: Mapping[str, Sequence[Any]] | None = None)

A set of shared Enum and Categorical domains, as a value.

Parameters:

Name Type Description Default
enums Mapping[str, Sequence[str]]

Entry name to its ordered category list.

 None
categoricals Mapping[str, Categories | dict | str | DataType]

Entry name to its shared pl.Categories. A physical dtype (pl.UInt8) or its name ("UInt8") is shorthand for a registry of that name; a mapping is the file form, and its categories key becomes choices.

 None
choices Mapping[str, Sequence[Any]]

The pool of values an entry draws from when generating. An Enum's categories already are its pool; this is for Categorical entries, whose registry names the domain without listing it.

 None
Notes

Naming an entry -- cats.STATUS, cats["STATUS"], cats.get("STATUS") -- gives back the dtype, ready to hand to ColSpec. Lookup is case-insensitive, so a column named status finds STATUS, and an Enum wins over a Categorical of the same name.

A subclass of CatSpec declares its entries in the class body; see the module docstring. Instantiating one takes those entries as defaults, so Categories(enums={"REASON": [...]}) extends rather than replaces.

Examples:

>>> cats = CatSpec(enums={"STATUS": ["PENDING", "COMPLETED"]})
>>> cats.STATUS
Enum(categories=['PENDING', 'COMPLETED'])
>>> cats.get_enum("status")
['PENDING', 'COMPLETED']

enums property 

enums: Mapping[str, list[str]]

Every Enum entry: name to its ordered category list.

categoricals property 

categoricals: Mapping[str, Categories]

Every Categorical entry: name to its shared pl.Categories.

choices property 

choices: Mapping[str, list[Any]]

Every recorded domain pool, keyed by entry name.

enum property 

enum: _Accessor

Entries as pl.Enum dtypes: cats.enum.STATUS.

categorical property 

categorical: _Accessor

Entries as pl.Categorical dtypes: cats.categorical.CURRENCY.

resolve_key 

resolve_key(name: str) -> tuple[Kind, str] | None

Which entry a name binds to, if any.

Exact match first, then the upper- and lower-case forms, so a column named status finds STATUS. An Enum wins over a Categorical of the same name.

dtype_of 

dtype_of(name: str) -> DataType | None

The dtype registered under name, or None if nothing is.

get_enum 

get_enum(name: str) -> list[str]

The category list of an Enum entry.

get_categorical 

get_categorical(name: str) -> Categories

The shared pl.Categories of a Categorical entry.

get_choices 

get_choices(name: str) -> list[Any] | None

The pool of values an entry draws from, if it has one.

An Enum's categories are its pool. A Categorical's comes from the choices it was registered with, since a pl.Categories names a shared domain without listing what is in it.

get 

get(name: str, default: Any = None) -> Any

The dtype registered under name, or default if nothing is.

from_dataframe classmethod 

from_dataframe(df: DataFrame | LazyFrame) -> CatSpec

The Enum and Categorical columns a frame already declares.

Reads what is there; infer is what looks at String columns and decides what could be one.

from_framespec classmethod 

from_framespec(spec: TableSpec | type[FrameSpec]) -> CatSpec

The Enum and Categorical columns a spec already declares.

infer classmethod 

infer(target: DataFrame | LazyFrame | TableSpec | type[FrameSpec], *, max_enum_cardinality: int = 30, max_categorical_cardinality: int = 10000, max_categorical_ratio: float = 0.2, include_columns: Sequence[str] | None = None, exclude_patterns: Sequence[str] | None = DEFAULT_EXCLUDE_PATTERNS, default_physical: DataType | None = None) -> CatSpec

A registry of the domains target looks like it has.

Existing Enum and Categorical columns are kept as declared. A String column becomes an Enum when it holds few enough distinct values, or a Categorical when it holds many but repeats them; one that looks like an identifier is skipped.

Parameters:

Name Type Description Default
target DataFrame | LazyFrame | TableSpec | type[FrameSpec]

Data to measure, or a spec to read declarations from.

 required
max_enum_cardinality int

At most this many distinct values makes a column an Enum.

 30
max_categorical_cardinality int

Beyond this many, a column is left as String.

 10000
max_categorical_ratio float

Distinct values as a fraction of rows, above which a column is too close to unique to be worth a category registry. Frames only -- a spec has no row count to measure against.

 0.2
include_columns Sequence[str]

Consider only these columns, exempting them from exclude_patterns.

 None
exclude_patterns Sequence[str]

Regexes for names to skip; defaults to identifier-shaped names (*_id, *_uuid, *_hash, *_url, *_key).

 DEFAULT_EXCLUDE_PATTERNS
default_physical DataType

Physical dtype for new Categorical registries, instead of the narrowest one that fits.

 None

from_dict classmethod 

from_dict(data: dict[str, Any], *, strict: bool = True) -> CatSpec

A registry read from the data form to_dict writes.

from_yaml classmethod 

from_yaml(source: str | Path, *, strict: bool = True) -> CatSpec

A registry read from a YAML file written by to_yaml.

to_dict 

to_dict() -> dict[str, Any]

This registry as plain data, without the file's version key.

to_yaml 

to_yaml(source: str | Path | None = None) -> str | None

Writes this registry as YAML to source, or returns the text.

to_markdown 

to_markdown(path: str | Path | None = None, *, title: str | None = None) -> str

A Markdown table of every entry; written to path when given.

to_mermaid 

to_mermaid(path: str | Path | None = None, *, title: str | None = None) -> str

A Mermaid class diagram of every entry; written to path when given.