Profiling¶
Inferring a spec from data you already have, and generating a stand-in for it. See Fake data from real data for the workflow, and what carries over from the source.
synthesize¶
synthesize
¶
synthesize(source: Source, n: int | None = None, *, seed: int | None = None, return_spec: Literal[False] = False, options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> DataFrame
synthesize(source: Source, n: int | None = None, *, seed: int | None = None, return_spec: Literal[True], options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> tuple[DataFrame, TableSpec]
synthesize(source: Source, n: int | None = None, *, seed: int | None = None, return_spec: bool = False, options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> DataFrame | tuple[DataFrame, TableSpec]
A fake frame that looks like source: generate() over profile().
The result has source's schema, and n rows -- as many as source
has, by default. Each column follows the profiled column's null rate,
extremes, category frequencies and fitted distribution; a key column
stays distinct, and rows that came sorted come sorted. Columns are
generated independently, so a relationship between two columns in
source is not kept.
options -- or its fields as keywords -- is how source is profiled:
every ProfileOptions switch, replace and sample among them. See
profile. With parse_text, a text column read as a number, flag or
date is generated as one, so the result's dtype is the parsed one there.
seed seeds the generation, and the sample and fitting behind it, so the
same source and seed give the same frame.
return_spec=True returns the frame and the spec it was generated from,
which the frame validates against. It is the profile, except that a key
column is widened when n is more rows than the source has keys for,
so the profile itself would refuse the larger frame on bounds. To edit
the spec before generating, call profile and generate separately.
Added in 1.7.
Examples:
profile¶
profile
¶
profile(source: Source, *, name: str = 'Profiled', options: ProfileOptions | None = None, **keywords: Unpack[ProfileKeywords]) -> TableSpec
A spec describing source well enough to generate a stand-in for it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
DataFrame | LazyFrame | str | PathLike
|
The data: a frame, a lazy frame (any |
required |
name
|
str
|
The spec's name. |
"Profiled"
|
options
|
ProfileOptions
|
What to infer and how finely to write it -- or pass its fields as
keywords instead: |
None
|
Returns:
| Type | Description |
|---|---|
TableSpec
|
Everything |
Examples:
>>> frame = pl.DataFrame({"id": range(1, 201), "status": ["NEW", "PAID"] * 100})
>>> spec = profile(frame, name="Orders")
>>> spec.schema()
Schema({'id': Int64, 'status': Enum(categories=['NEW', 'PAID'])})
>>> spec["id"].bounds, spec["id"].unique
(Bound(min=1, max=200), True)
>>> profile(pl.DataFrame({"lgd": ["0.45", "", "0.2"]}), parse_text=True).schema()
Schema({'lgd': Float64})
synthesize_all¶
synthesize_all
¶
synthesize_all(frames: Mapping[str, Source], n: int | Mapping[str, int] | None = None, *, scale: float | None = None, seed: int | None = None, keys: Keys | Literal[False] | None = None, return_registry: Literal[False] = False, options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> dict[str, DataFrame]
synthesize_all(frames: Mapping[str, Source], n: int | Mapping[str, int] | None = None, *, scale: float | None = None, seed: int | None = None, keys: Keys | Literal[False] | None = None, return_registry: Literal[True], options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> tuple[dict[str, DataFrame], Registry]
synthesize_all(frames: Mapping[str, Source], n: int | Mapping[str, int] | None = None, *, scale: float | None = None, seed: int | None = None, keys: Keys | Literal[False] | None = None, return_registry: bool = False, options: ProfileOptions | None = None, **keywords: Unpack[SynthesizeKeywords]) -> dict[str, DataFrame] | tuple[dict[str, DataFrame], Registry]
Fake frames that look like a set of related tables, with the keys
between them intact: generate_all() over profile_all().
Each frame has its source's schema, and each column follows the
source's as synthesize makes it. Every foreign key profile_all
finds, or keys declares, holds in the fake set: a child row points
at a fake parent row, never at a real one. Within a table, columns are
generated independently, as synthesize generates them, and parents
are drawn uniformly, so how many children each parent has is not kept.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frames
|
Mapping[str, DataFrame | LazyFrame | str | PathLike]
|
Each table, by name. |
required |
n
|
int | Mapping[str, int]
|
Rows for every table, or for those a mapping names. By default, each source's own height. |
None
|
scale
|
float
|
Each table's height times |
None
|
seed
|
int
|
Seeds the generation, and the sample and fitting behind it, so the same sources and seed give the same frames. |
None
|
keys
|
Mapping[str, Sequence[ForeignKey]] | False
|
Keys to declare rather than infer, as |
None
|
return_registry
|
bool
|
Also return the registry the frames were generated from, which they
validate against. It is |
False
|
options
|
ProfileOptions
|
How every table is profiled -- or pass its fields as keywords. |
None
|
Returns:
| Type | Description |
|---|---|
dict[str, DataFrame]
|
A fake frame per table, in the order of |
Examples:
>>> customers = pl.DataFrame({"id": range(1, 101)})
>>> orders = pl.DataFrame({"id": range(1, 301), "customer_id": [1, 2, 3] * 100})
>>> real = {"Customers": customers, "Orders": orders}
>>> fake = synthesize_all(real, scale=0.5, seed=1)
>>> {name: frame.height for name, frame in fake.items()}
{'Customers': 50, 'Orders': 150}
>>> fake["Orders"]["customer_id"].is_in(fake["Customers"]["id"].implode()).all()
True
>>> bigger, registry = synthesize_all(real, scale=2, seed=1, return_registry=True)
>>> all(report.passed for report in registry.inspect_all(bigger).values())
True
Added in 1.5.
profile_all¶
profile_all
¶
profile_all(frames: Mapping[str, Source], *, keys: Keys | Literal[False] | None = None, options: ProfileOptions | None = None, **keywords: Unpack[ProfileKeywords]) -> Registry
A Registry describing a set of related tables, with the foreign
keys between them found from their values.
Each source is profiled as profile profiles it, with the same
options. Then a column becomes a foreign key into a column of another
table, or of its own, when:
- the parent column is an integer, text or date column with no nulls and no value twice, in all of its rows;
- every non-null value of the child is one of the parent's, again in
all of its rows, whatever
sampleprofiled; - and either the names link them --
customer_idorcustomerintoCustomers.id,customer_idintoCustomers.customer_id,parent_idinto its own table'sid-- or the child is text with at least 20 distinct values. An integer column with no name link is never a key: a quantity of 1 to 20 is contained in everyid.
A child that could point at more than one parent, with nothing to choose between them, gets no key, and a warning names the candidates; so does a key that would close a cycle between tables. The child column of a key takes its parent's dtype and value claims, which are what generation fills it with, and keeps its own null rate and uniqueness; its description says the key was inferred.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frames
|
Mapping[str, DataFrame | LazyFrame | str | PathLike]
|
Each table, by the name its spec takes. |
required |
keys
|
Mapping[str, Sequence[ForeignKey]] | False
|
Keys to declare rather than infer, by the name of the table that
holds them: |
None
|
options
|
ProfileOptions
|
How every table is profiled -- or pass its fields as keywords. A
column in |
None
|
Returns:
| Type | Description |
|---|---|
Registry
|
A spec per table, in the order of |
Examples:
>>> customers = pl.DataFrame({"id": range(1, 101)})
>>> orders = pl.DataFrame({"id": range(1, 301), "customer_id": [1, 2, 3] * 100})
>>> registry = profile_all({"Customers": customers, "Orders": orders})
>>> registry["Orders"].foreign_keys[0].references
'Customers'
>>> registry["Orders"]["customer_id"].bounds
Bound(min=1, max=100)
Added in 1.5.
ProfileOptions¶
ProfileOptions
dataclass
¶
ProfileOptions(*, replace: Sequence[str] = (), sample: int | None = None, seed: int = 0, max_unique_enum: int = 50, formats: bool = True, patterns: bool = True, weights: bool = True, shape: bool = True, detect_unique: bool = True, calculate_bounds: bool = True, detect_order: bool = True, round_bounds: Rounding = None, round_rates: Rounding = None, parse_text: bool = False)
What profile() infers from data, and how finely it writes it.
profile, FrameSpec.from_dataframe and synthesize take these as an
options= object or as the keywords themselves -- one or the other, as
validate takes ValidationOptions. FrameSpec.from_dataframe
records less by default: weights, shape, detect_unique and
detect_order off.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
replace
|
sequence of str
|
Columns whose values must not be carried into the spec. A text
column named here is described by its lengths, never as an |
()
|
sample
|
int
|
Profile a random sample of this many rows rather than every row. |
None
|
seed
|
int
|
Seeds the sample and the shape fitting, so a profile is the same every run. |
0
|
max_unique_enum
|
int
|
A text column with at most this many distinct values, repeating --
and not in |
50
|
formats
|
bool
|
Name the |
True
|
patterns
|
bool
|
Write the |
True
|
weights
|
bool
|
Record each category's observed frequency, for categorical, enum and boolean columns. |
True
|
shape
|
bool
|
Fit the distribution each numeric or temporal column follows, so it
generates that shape rather than uniformly between its extremes.
Needs |
True
|
detect_unique
|
bool
|
Declare an integer or text column |
True
|
calculate_bounds
|
bool
|
Record the observed extremes of numeric and temporal columns, and the length range of text and binary ones. |
True
|
detect_order
|
bool
|
Declare |
True
|
round_bounds
|
(number, 'auto' or None)
|
Write each numeric column's bounds rounded outward to this step --
|
None
|
round_rates
|
(number, 'auto' or None)
|
Write each null, NaN and list-element null rate rounded up to this
step, never past 1; "auto" keeps two significant figures. A rate of
0 stays 0. Category frequencies ( |
None
|
parse_text
|
bool
|
Read a text column as the type its values hold, for a frame whose
types were switched off ( |
False
|
Examples:
>>> house = ProfileOptions(round_bounds="auto", replace=["name"])
>>> frame = pl.DataFrame({"name": ["Ada", "Grace"] * 50, "total": [3.17, 9871.23] * 50})
>>> profile(frame, options=house)["total"].bounds
Bound(min=0.0, max=9900.0)