Generation¶
Every function here takes a spec as its first argument — a TableSpec,
or a FrameSpec class, which stands for its .spec — and every one has a
FrameSpec classmethod that forwards to it with cls.spec — see
Generating data for what the options mean and
Specs as values for when to reach for which.
generate¶
generate
¶
generate(spec: SpecLike, n: int, *, method: Method = 'random', seed: int | None = None, references: References = None, cycles: int = 0, self_references: int = 0, max_bytes: int | None = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> DataFrame
Generates a DataFrame matching spec.
method="random" (default): n rows, each column drawn independently.
method="cartesian": guarantees a minimum level of coverage. Builds the
cartesian product of every Enum/Boolean column's full set of values,
crossed with the negative/zero/positive/null partitions of every bounded
numeric column, so every enum combination appears alongside every numeric
sign/null case. n is then a minimum: if that coverage set has fewer
than n rows it is padded with random rows; if it has more, all of it is
kept.
ColSpec.rules and any ForeignKey the spec declares are then applied as
vectorised passes over the generated frame, regardless of method. Each
pass sees the frame the passes before it produced, and they run in the
order their reads and writes imply -- a rule keyed on a foreign-keyed
column reads the parent's values, not the freely generated ones they
replaced -- so the result satisfies the same declarations validate
checks it against.
A foreign key is only made referentially consistent where data for its
target is available: self-referencing keys always are, sampled from this
same frame; a key referencing another spec only is if references
carries an entry for it, keyed by the spec, its class, or its name --
otherwise that column is left exactly as freely generated. Composite keys
are sampled as one joint pick per row; a single-column key whose ColSpec
is unique=True samples without replacement when the parent has enough
distinct rows to cover n.
A unique=True column is drawn without replacement by the engine itself,
and a __unique_together__ group is separated afterwards by resampling
the rows that repeat a combination. Either refuses, naming the column or
the group, when the domain is too small to cover n.
A spec declaring a Hierarchy has its two link columns rewritten as a
forest of the declared depth. cycles and self_references then damage it
on purpose -- closing that many chains into loops, and pointing that many
rows at themselves -- which is how a graph walk gets something to fail
against. Both default to zero, and validate() reports whatever they
injected.
Before anything is allocated the frame's size is estimated from the
declaration. Past four gibibytes that is a warning naming the estimate;
max_bytes= makes it a refusal instead, and max_bytes=0 silences both.
A spec declaring order_by has its rows sorted by those columns last,
after every pass: each value keeps its row, so every claim the rows make
about their own columns still holds. Nulls come last.
fixed pins columns to values given: {"status": ["open", "closed"],
"region": "EU"}, a scalar for every row or a sequence or Series of
exactly n, cast strictly to the column's dtype. Every other column is
what it would have been, since each is seeded by its own name; a rule
reading a pinned column follows the pin. A pin must be values its
column accepts, and a column a pass writes -- a rule's, a foreign
key's, a hierarchy's, a composite key's -- cannot be pinned. With
order_by, pinned rows move with the sort like any other.
method="cartesian", whose row count is not n, takes no pins.
checks=True keeps the spec's __checks__ and column validators,
which are otherwise validation-only: the frame is drawn as always, the
rows every check passes are kept, and when fewer than n pass, a
larger frame is drawn from a seed of its own and its first n passing
rows taken. A spec whose rows all pass is the frame checks=False
gives. Rates a check reads -- a null rate, a weight -- become rates
among the rows that pass. A check that rejects nearly every row is
refused with a GenerationError naming it; a hierarchy, a key into the
spec's own rows, method="cartesian" and a pin of all n values
cannot be filtered, and are refused. Added in 1.7.
The whole frame is built before this returns. scan() is the lazy
verb: it generates as the plan is collected, so only the columns and
rows a plan asks for are made.
Examples:
generate_batches¶
generate_batches
¶
generate_batches(spec: SpecLike, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> Iterator[DataFrame]
Yields chunks of generated rows without holding all n in memory.
Each batch is a window onto the one frame seed describes: a column no
pass rewrites holds, batch by batch, exactly the rows generate(n, seed=seed)
would, whatever batch_size is. What is drawn per batch instead --
deterministic, but not row for row the whole frame's -- is a column with
rules, a foreign key, a composite key, and a List column's elements.
A unique=True column is a window too: its values are a permutation of
its value space -- or, filled by a foreign key, of its parent's keys --
unique across the whole n however it is batched; a parent too small
for all n is refused before the first batch. A __unique_together__
group and a unique key into the spec's own rows are drawn afresh per
batch, so they are distinct within a batch and collide across batches
only by chance.
fixed pins columns as generate() takes it: a scalar for every row, or
a sequence of all n values, which each batch takes its stretch of.
Added in 1.4.
checks=True keeps the spec's checks, as generate() keeps them:
each batch is the first rows that pass from where the last batch
stopped reading, so a unique=True column stays unique across n.
Batches whose rows all pass are the batches checks=False gives.
Added in 1.7.
Examples:
scan¶
scan
¶
A LazyFrame that generates rows on demand.
generate() builds the whole frame before it returns. scan() hands back
a frame that has not been built: polars asks for the columns and the rows
it actually needs, and only those are generated. So
Orders.scan(50_000_000, seed=1).sink_parquet("orders.parquet")
streams in bounded memory, and
Orders.scan(50_000_000, seed=1).select("total").head(5).collect()
generates five rows of one column.
Projecting cannot change what a column holds. Every column is seeded by its
name and every pass by what it is for, so dropping a column's neighbours
leaves it alone: scan(n, seed=s).select(cols).collect() is
scan(n, seed=s).collect().select(cols), for every subset. That is what
makes the pushdown free rather than a trade -- except under
method="cartesian", where the coverage set is the product of every column
and so every column is generated, then projected.
Rows come in batches, so a scan carries generate_batches' terms: a
Hierarchy is refused, uniqueness holds within a batch rather than across
n, and a column no pass rewrites holds the rows generate(n, seed=s)
would while a ruled or foreign-keyed one is drawn per batch.
Internal: not part of the public API.
needed_columns
¶
needed_columns(spec: TableSpec, wanted: frozenset[str]) -> list[str]
wanted plus every column its values depend on, in spec order.
A projection cannot simply drop the rest: a rules column's values depend
on the columns its when reads, a composite key's on every member of the
group, a foreign key's on the other columns of its key. passes_of
already knows what each pass reads and writes, so the closure is that
relation followed to a fixed point -- generate these, hand back only what
was asked for.
scan
¶
scan(spec: SpecLike, n: int, *, seed: int | None = None, batch_size: int | None = None, method: Method = 'random', references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> LazyFrame
A LazyFrame of n generated rows, produced as they are collected.
batch_size left unset lets polars ask for the size it would like, so a
sink gets the batches it writes best; setting it pins the size whatever
polars asks.
A scan is batched, so it has generate_batches' terms: a unique=True
column is unique across the whole scan -- a foreign-keyed one too -- and
a composite key only within each batch. fixed pins columns as
generate_batches takes it. fixed added in 1.4.
checks=True keeps the spec's checks batch by batch, as
generate_batches does. Which rows are kept depends on every column a
check reads, so those are generated whatever the plan projects, and a
head takes the first rows of the batches the whole scan would make.
Added in 1.7.
Examples:
estimated_size¶
estimated_size
¶
Bytes generate(spec, n) is expected to hold, as whole bytes.
See the module docstring for what this does and does not count.
Examples:
sink_parquet¶
sink_parquet
¶
sink_parquet(spec: SpecLike, path: str | Path, n: int, *, batch_size: int = 100000, compression: ParquetCompression = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None
Generates n rows and streams them to a Parquet file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_parquet.
Examples:
>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> spec = TableSpec("Orders", {
... "id": ColSpec(pl.Int64, unique=True),
... "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> sink_parquet(spec, folder / "orders.parquet", 250_000, seed=1)
>>> pl.scan_parquet(folder / "orders.parquet").select(pl.len()).collect().item()
250000
sink_ipc¶
sink_ipc
¶
sink_ipc(spec: SpecLike, path: str | Path, n: int, *, batch_size: int = 100000, compression: IpcCompression | None = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None
Generates n rows and streams them to an Arrow IPC / Feather file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_ipc.
Examples:
>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> spec = TableSpec("Orders", {
... "id": ColSpec(pl.Int64, unique=True),
... "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> sink_ipc(spec, folder / "orders.arrow", 1_000, seed=1)
>>> pl.read_ipc(folder / "orders.arrow").shape
(1000, 2)
sink_csv¶
sink_csv
¶
sink_csv(spec: SpecLike, path: str | Path, n: int, *, batch_size: int = 100000, include_header: bool = True, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None
Generates n rows and streams them to a CSV file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_csv.
Examples:
>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> spec = TableSpec("Orders", {
... "id": ColSpec(pl.Int64, unique=True),
... "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> sink_csv(spec, folder / "orders.csv", 1_000, seed=1)
>>> pl.read_csv(folder / "orders.csv").shape
(1000, 2)
sink_ndjson¶
sink_ndjson
¶
sink_ndjson(spec: SpecLike, path: str | Path, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None
Generates n rows and streams them to a newline-delimited JSON file in batches.
JSON holds a category as its text, so an Enum or Categorical column
is written as one -- which read() reads back as declared. JSON has no
bytes: a spec with a Binary column raises GenerationError before
anything is written, rather than panicking Polars' writer.
Extra keyword arguments go to pl.LazyFrame.sink_ndjson.
Examples:
>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> spec = TableSpec("Orders", {
... "id": ColSpec(pl.Int64, unique=True),
... "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> sink_ndjson(spec, folder / "orders.ndjson", 1_000, seed=1)
>>> pl.read_ndjson(folder / "orders.ndjson").shape
(1000, 2)