Generation¶
Every function here takes a TableSpec as its first argument, and every one
has a FrameSpec classmethod that forwards to it with cls.spec -- see
Generating data for what the options mean and
Specs as values for when to reach for which.
generate¶
generate(spec: TableSpec, n: int, *, method: Method = 'random', seed: int | None = None, references: References = None, cycles: int = 0, self_references: int = 0, max_bytes: int | None = None) -> DataFrame
Generates a DataFrame matching spec.
method="random" (default): n rows, each column drawn independently.
method="cartesian": guarantees a minimum level of coverage. Builds the
cartesian product of every Enum/Boolean column's full set of values,
crossed with the negative/zero/positive/null partitions of every bounded
numeric column, so every enum combination appears alongside every numeric
sign/null case. n is then a minimum: if that coverage set has fewer
than n rows it is padded with random rows; if it has more, all of it is
kept.
ColSpec.rules and any ForeignKey the spec declares are then applied as
vectorised passes over the generated frame, regardless of method. Each
pass sees the frame the passes before it produced, and they run in the
order their reads and writes imply -- a rule keyed on a foreign-keyed
column reads the parent's values, not the freely generated ones they
replaced -- so the result satisfies the same declarations validate
checks it against.
A foreign key is only made referentially consistent where data for its
target is available: self-referencing keys always are, sampled from this
same frame; a key referencing another spec only is if references
carries an entry for it, keyed by the spec, its class, or its name --
otherwise that column is left exactly as freely generated. Composite keys
are sampled as one joint pick per row; a single-column key whose ColSpec
is unique=True samples without replacement when the parent has enough
distinct rows to cover n.
A unique=True column is drawn without replacement by the engine itself,
and a __unique_together__ group is separated afterwards by resampling
the rows that repeat a combination. Either refuses, naming the column or
the group, when the domain is too small to cover n.
A spec declaring a Hierarchy has its two link columns rewritten as a
forest of the declared depth. cycles and self_references then damage it
on purpose -- closing that many chains into loops, and pointing that many
rows at themselves -- which is how a graph walk gets something to fail
against. Both default to zero, and validate() reports whatever they
injected.
Before anything is allocated the frame's size is estimated from the
declaration. Past four gibibytes that is a warning naming the estimate;
max_bytes= makes it a refusal instead, and max_bytes=0 silences both.
The whole frame is built before this returns. scan() is the lazy
verb: it generates as the plan is collected, so only the columns and
rows a plan asks for are made.
generate_batches¶
generate_batches(spec: TableSpec, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None) -> Iterator[DataFrame]
Yields chunks of generated rows without holding all n in memory.
Each batch is a window onto the one frame seed describes: a column no
pass rewrites holds, batch by batch, exactly the rows generate(n, seed=seed)
would, whatever batch_size is. What is drawn per batch instead --
deterministic, but not row for row the whole frame's -- is a column with
rules, a foreign key, a composite key, and a List column's elements.
Uniqueness only holds within a batch, not across the whole n: that
applies to a unique=True column, a __unique_together__ group, and a
foreign-key column sampled without replacement alike.
scan¶
A LazyFrame that generates rows on demand.
generate() builds the whole frame before it returns. scan() hands back
a frame that has not been built: polars asks for the columns and the rows
it actually needs, and only those are generated. So
Orders.scan(50_000_000, seed=1).sink_parquet("orders.parquet")
streams in bounded memory, and
Orders.scan(50_000_000, seed=1).select("total").head(5).collect()
generates five rows of one column.
Projecting cannot change what a column holds. Every column is seeded by its
name and every pass by what it is for, so dropping a column's neighbours
leaves it alone: scan(n, seed=s).select(cols).collect() is
scan(n, seed=s).collect().select(cols), for every subset. That is what
makes the pushdown free rather than a trade.
Rows come in batches, so a scan carries generate_batches' terms: a
Hierarchy is refused, uniqueness holds within a batch rather than across
n, and a column no pass rewrites holds the rows generate(n, seed=s)
would while a ruled or foreign-keyed one is drawn per batch.
needed_columns(spec: TableSpec, wanted: frozenset[str]) -> list[str]
wanted plus every column its values depend on, in spec order.
A projection cannot simply drop the rest: a rules column's values depend
on the columns its when reads, a composite key's on every member of the
group, a foreign key's on the other columns of its key. passes_of
already knows what each pass reads and writes, so the closure is that
relation followed to a fixed point -- generate these, hand back only what
was asked for.
scan(spec: TableSpec, n: int, *, seed: int | None = None, batch_size: int | None = None, method: Method = 'random', references: References = None) -> LazyFrame
A LazyFrame of n generated rows, produced as they are collected.
batch_size left unset lets polars ask for the size it would like, so a
sink gets the batches it writes best; setting it pins the size whatever
polars asks.
estimated_size¶
estimated_size(spec: TableSpec, n: int) -> int
Bytes generate(spec, n) is expected to hold, as whole bytes.
See the module docstring for what this does and does not count.
sink_parquet¶
sink_parquet(spec: TableSpec, path: str | Path, n: int, *, batch_size: int = 100000, compression: ParquetCompression = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, **kwargs: Any) -> None
Generates n rows and streams them to a Parquet file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_parquet.
sink_ipc¶
sink_ipc(spec: TableSpec, path: str | Path, n: int, *, batch_size: int = 100000, compression: IpcCompression | None = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, **kwargs: Any) -> None
Generates n rows and streams them to an Arrow IPC / Feather file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_ipc.
sink_csv¶
sink_csv(spec: TableSpec, path: str | Path, n: int, *, batch_size: int = 100000, include_header: bool = True, method: Method = 'random', seed: int | None = None, references: References = None, **kwargs: Any) -> None
Generates n rows and streams them to a CSV file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_csv.
sink_ndjson¶
sink_ndjson(spec: TableSpec, path: str | Path, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None, **kwargs: Any) -> None
Generates n rows and streams them to a newline-delimited JSON file in batches.
Extra keyword arguments go to pl.LazyFrame.sink_ndjson.