Skip to content

Profiling

Inferring a spec from data you already have, and generating a stand-in for it. See Fake data from real data for the workflow, and what carries over from the source.

synthesize

synthesize

synthesize(source: Source, n: int | None = None, *, seed: int | None = None, return_spec: Literal[False] = False, options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> DataFrame
synthesize(source: Source, n: int | None = None, *, seed: int | None = None, return_spec: Literal[True], options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> tuple[DataFrame, TableSpec]
synthesize(source: Source, n: int | None = None, *, seed: int | None = None, return_spec: bool = False, options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> DataFrame | tuple[DataFrame, TableSpec]

A fake frame that looks like source: generate() over profile().

The result has source's schema, and n rows -- as many as source has, by default. Each column follows the profiled column's null rate, extremes, category frequencies and fitted distribution; a key column stays distinct, and rows that came sorted come sorted. Columns are generated independently, so a relationship between two columns in source is not kept.

options -- or its fields as keywords -- is how source is profiled: every ProfileOptions switch, replace and sample among them. See profile. With parse_text, a text column read as a number, flag or date is generated as one, so the result's dtype is the parsed one there.

seed seeds the generation, and the sample and fitting behind it, so the same source and seed give the same frame.

return_spec=True returns the frame and the spec it was generated from, which the frame validates against. It is the profile, except that a key column is widened when n is more rows than the source has keys for, so the profile itself would refuse the larger frame on bounds. To edit the spec before generating, call profile and generate separately. Added in 1.7.

Examples:

>>> frame = pl.DataFrame({"id": range(1, 201), "status": ["NEW", "PAID"] * 100})
>>> fake = synthesize(frame, 50, seed=1)
>>> fake.shape, fake.schema == frame.schema
((50, 2), True)
>>> bigger, spec = synthesize(frame, 1_000, seed=1, return_spec=True)
>>> spec["id"].bounds
Bound(min=1, max=1000)

profile

profile

profile(source: Source, *, name: str = 'Profiled', options: ProfileOptions | None = None, **keywords: Unpack[ProfileKeywords]) -> TableSpec

A spec describing source well enough to generate a stand-in for it.

Parameters:

Name Type Description Default
source DataFrame | LazyFrame | str | PathLike

The data: a frame, a lazy frame (any scan_* -- a file, a glob, cloud storage), or the path of a CSV, TSV, Parquet, NDJSON, JSON or Arrow IPC file.

required
name str

The spec's name.

"Profiled"
options ProfileOptions

What to infer and how finely to write it -- or pass its fields as keywords instead: replace, sample, seed, max_unique_enum, formats, weights, shape, detect_unique, calculate_bounds, detect_order, round_bounds, round_rates, parse_text. See ProfileOptions.

None

Returns:

Type Description
TableSpec

Everything FrameSpec.from_dataframe records, plus each column's category frequencies, fitted distribution and uniqueness.

Examples:

>>> frame = pl.DataFrame({"id": range(1, 201), "status": ["NEW", "PAID"] * 100})
>>> spec = profile(frame, name="Orders")
>>> spec.schema()
Schema({'id': Int64, 'status': Enum(categories=['NEW', 'PAID'])})
>>> spec["id"].bounds, spec["id"].unique
(Bound(min=1, max=200), True)
>>> profile(pl.DataFrame({"lgd": ["0.45", "", "0.2"]}), parse_text=True).schema()
Schema({'lgd': Float64})

synthesize_all

synthesize_all

synthesize_all(frames: Mapping[str, Source], n: int | Mapping[str, int] | None = None, *, scale: float | None = None, seed: int | None = None, keys: Keys | Literal[False] | None = None, return_registry: Literal[False] = False, options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> dict[str, DataFrame]
synthesize_all(frames: Mapping[str, Source], n: int | Mapping[str, int] | None = None, *, scale: float | None = None, seed: int | None = None, keys: Keys | Literal[False] | None = None, return_registry: Literal[True], options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> tuple[dict[str, DataFrame], Registry]
synthesize_all(frames: Mapping[str, Source], n: int | Mapping[str, int] | None = None, *, scale: float | None = None, seed: int | None = None, keys: Keys | Literal[False] | None = None, return_registry: bool = False, options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> dict[str, DataFrame] | tuple[dict[str, DataFrame], Registry]

Fake frames that look like a set of related tables, with the keys between them intact: generate_all() over profile_all().

Each frame has its source's schema, and each column follows the source's as synthesize makes it. Every foreign key profile_all finds, or keys declares, holds in the fake set: a child row points at a fake parent row, never at a real one. Within a table, columns are generated independently, as synthesize generates them, and parents are drawn uniformly, so how many children each parent has is not kept.

Parameters:

Name Type Description Default
frames Mapping[str, DataFrame | LazyFrame | str | PathLike]

Each table, by name.

required
n int | Mapping[str, int]

Rows for every table, or for those a mapping names. By default, each source's own height.

None
scale float

Each table's height times scale instead -- 0.1 for a tenth of each, keeping the ratios between them -- and at least one row. Not with n.

None
seed int

Seeds the generation, and the sample and fitting behind it, so the same sources and seed give the same frames.

None
keys Mapping[str, Sequence[ForeignKey]] | False

Keys to declare rather than infer, as profile_all takes them.

None
return_registry bool

Also return the registry the frames were generated from, which they validate against. It is profile_all's, except that a key column is widened when a table gets more rows than its source has keys for -- at a scale above 1, say -- so profile_all's own registry would refuse the fake set on bounds. Added in 1.7.

False
options ProfileOptions

How every table is profiled -- or pass its fields as keywords.

None

Returns:

Type Description
dict[str, DataFrame]

A fake frame per table, in the order of frames; with return_registry=True, that and the Registry.

Examples:

>>> customers = pl.DataFrame({"id": range(1, 101)})
>>> orders = pl.DataFrame({"id": range(1, 301), "customer_id": [1, 2, 3] * 100})
>>> real = {"Customers": customers, "Orders": orders}
>>> fake = synthesize_all(real, scale=0.5, seed=1)
>>> {name: frame.height for name, frame in fake.items()}
{'Customers': 50, 'Orders': 150}
>>> fake["Orders"]["customer_id"].is_in(fake["Customers"]["id"].implode()).all()
True
>>> bigger, registry = synthesize_all(real, scale=2, seed=1, return_registry=True)
>>> all(report.passed for report in registry.inspect_all(bigger).values())
True

Added in 1.5.

profile_all

profile_all

profile_all(frames: Mapping[str, Source], *, keys: Keys | Literal[False] | None = None, options: ProfileOptions | None = None, **keywords: Unpack[ProfileKeywords]) -> Registry

A Registry describing a set of related tables, with the foreign keys between them found from their values.

Each source is profiled as profile profiles it, with the same options. Then a column becomes a foreign key into a column of another table, or of its own, when:

  • the parent column is an integer, text or date column with no nulls and no value twice, in all of its rows;
  • every non-null value of the child is one of the parent's, again in all of its rows, whatever sample profiled;
  • and either the names link them -- customer_id or customer into Customers.id, customer_id into Customers.customer_id, parent_id into its own table's id -- or the child is text with at least 20 distinct values. An integer column with no name link is never a key: a quantity of 1 to 20 is contained in every id.

A child that could point at more than one parent, with nothing to choose between them, gets no key, and a warning names the candidates; so does a key that would close a cycle between tables. The child column of a key takes its parent's dtype and value claims, which are what generation fills it with, and keeps its own null rate and uniqueness; its description says the key was inferred.

Parameters:

Name Type Description Default
frames Mapping[str, DataFrame | LazyFrame | str | PathLike]

Each table, by the name its spec takes.

required
keys Mapping[str, Sequence[ForeignKey]] | False

Keys to declare rather than infer, by the name of the table that holds them: {"Orders": [ForeignKey("buyer", references="Customers", ref_columns="id")]}. A declared key replaces any key inferred on its columns, and one the data does not satisfy raises SpecError. False infers none.

None
options ProfileOptions

How every table is profiled -- or pass its fields as keywords. A column in replace need only be in one of the tables.

None

Returns:

Type Description
Registry

A spec per table, in the order of frames, resolved.

Examples:

>>> customers = pl.DataFrame({"id": range(1, 101)})
>>> orders = pl.DataFrame({"id": range(1, 301), "customer_id": [1, 2, 3] * 100})
>>> registry = profile_all({"Customers": customers, "Orders": orders})
>>> registry["Orders"].foreign_keys[0].references
'Customers'
>>> registry["Orders"]["customer_id"].bounds
Bound(min=1, max=100)

Added in 1.5.

ProfileOptions

ProfileOptions dataclass

ProfileOptions(*, replace: Sequence[str] = (), sample: int | None = None, seed: int = 0, max_unique_enum: int = 50, formats: bool = True, patterns: bool = True, weights: bool = True, shape: bool = True, detect_unique: bool = True, calculate_bounds: bool = True, detect_order: bool = True, round_bounds: Rounding = None, round_rates: Rounding = None, parse_text: bool = False)

What profile() infers from data, and how finely it writes it.

profile, FrameSpec.from_dataframe and synthesize take these as an options= object or as the keywords themselves -- one or the other, as validate takes ValidationOptions. FrameSpec.from_dataframe records less by default: weights, shape, detect_unique and detect_order off.

Parameters:

Name Type Description Default
replace sequence of str

Columns whose values must not be carried into the spec. A text column named here is described by its lengths, never as an Enum of the values it holds, and carries no weights.

()
sample int

Profile a random sample of this many rows rather than every row.

None
seed int

Seeds the sample and the shape fitting, so a profile is the same every run.

0
max_unique_enum int

A text column with at most this many distinct values, repeating -- and not in replace -- becomes an Enum of them.

50
formats bool

Name the format a text column's values have -- an email, a UUID, an IP, a country code -- so its fake values have it too: when at least 20 distinct values have one of email, uuid4, ipv4, ipv6, mac, hostname, iso_country or iso_currency, and every value does -- or 99% do, and the rest are at most five repeated stand-ins, which become its extra_values.

True
patterns bool

Write the pattern a code-like text column's values share -- an order reference, a SKU, a postcode -- so its fake values have that shape: each run of digits, capitals or small letters a class of the lengths seen (ORD-004211, ORD-918200 give ^ORD-[0-9]{6}$), and a part every value shares kept as written. Only for a column of at least 20 distinct values that no format fits and that would not be an Enum, when every value has one of at most three shapes; for one of distinct values, which may be a key, only when the pattern holds ten times as many. Not for a column in replace, whose shared parts are real text, nor for numbers, dates or booleans written as text.

True
weights bool

Record each category's observed frequency, for categorical, enum and boolean columns.

True
shape bool

Fit the distribution each numeric or temporal column follows, so it generates that shape rather than uniformly between its extremes. Needs calculate_bounds.

True
detect_unique bool

Declare an integer or text column unique=True when every one of at least a hundred values is distinct.

True
calculate_bounds bool

Record the observed extremes of numeric and temporal columns, and the length range of text and binary ones.

True
detect_order bool

Declare order_by when the rows come sorted by a column of single values -- either way, nulls last, with at least 20 distinct values, so not by chance -- so a stand-in comes sorted too. The first such column, in column order. A sample keeps the source's row order, so a sorted file's sample shows it.

True
round_bounds (number, 'auto' or None)

Write each numeric column's bounds rounded outward to this step -- 1000, 0.01 -- the minimum down and the maximum up, within what its dtype holds, so the data stays inside them. "auto" keeps two significant figures of the column's span ([3.17, 9871.23] becomes [0, 9900]) and rounds a datetime, time or duration outward to the second. None writes them exactly as observed.

None
round_rates (number, 'auto' or None)

Write each null, NaN and list-element null rate rounded up to this step, never past 1; "auto" keeps two significant figures. A rate of 0 stays 0. Category frequencies (weights) are written at the same precision, each to its nearest step and never to 0. None writes them exactly as observed.

None
parse_text bool

Read a text column as the type its values hold, for a frame whose types were switched off (pl.read_csv(..., infer_schema=False)): Boolean, Int64, Float64, Date or Datetime, the first that every present value parses as exactly. Text is stripped and empty text is a null. A column with a value written with a leading zero (007) stays text, being a code, as does one in replace. Left False, such a column stays text and its description says what it would parse as. Validate the same frame with parse_text=True too.

False

Examples:

>>> house = ProfileOptions(round_bounds="auto", replace=["name"])
>>> frame = pl.DataFrame({"name": ["Ada", "Grace"] * 50, "total": [3.17, 9871.23] * 50})
>>> profile(frame, options=house)["total"].bounds
Bound(min=0.0, max=9900.0)