Skip to content

Generation

Every function here takes a spec as its first argument — a TableSpec, or a FrameSpec class, which stands for its .spec — and every one has a FrameSpec classmethod that forwards to it with cls.spec — see Generating data for what the options mean and Specs as values for when to reach for which.

generate

generate

generate(spec: SpecLike, n: int, *, method: Method = 'random', seed: int | None = None, references: References = None, cycles: int = 0, self_references: int = 0, max_bytes: int | None = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> DataFrame

Generates a DataFrame matching spec.

method="random" (default): n rows, each column drawn independently.

method="cartesian": guarantees a minimum level of coverage. Builds the cartesian product of every Enum/Boolean column's full set of values, crossed with the negative/zero/positive/null partitions of every bounded numeric column, so every enum combination appears alongside every numeric sign/null case. n is then a minimum: if that coverage set has fewer than n rows it is padded with random rows; if it has more, all of it is kept.

ColSpec.rules and any ForeignKey the spec declares are then applied as vectorised passes over the generated frame, regardless of method. Each pass sees the frame the passes before it produced, and they run in the order their reads and writes imply -- a rule keyed on a foreign-keyed column reads the parent's values, not the freely generated ones they replaced -- so the result satisfies the same declarations validate checks it against.

A foreign key is only made referentially consistent where data for its target is available: self-referencing keys always are, sampled from this same frame; a key referencing another spec only is if references carries an entry for it, keyed by the spec, its class, or its name -- otherwise that column is left exactly as freely generated. Composite keys are sampled as one joint pick per row; a single-column key whose ColSpec is unique=True samples without replacement when the parent has enough distinct rows to cover n.

A unique=True column is drawn without replacement by the engine itself, and a __unique_together__ group is separated afterwards by resampling the rows that repeat a combination. Either refuses, naming the column or the group, when the domain is too small to cover n.

A spec declaring a Hierarchy has its two link columns rewritten as a forest of the declared depth. cycles and self_references then damage it on purpose -- closing that many chains into loops, and pointing that many rows at themselves -- which is how a graph walk gets something to fail against. Both default to zero, and validate() reports whatever they injected.

Before anything is allocated the frame's size is estimated from the declaration. Past four gibibytes that is a warning naming the estimate; max_bytes= makes it a refusal instead, and max_bytes=0 silences both.

A spec declaring order_by has its rows sorted by those columns last, after every pass: each value keeps its row, so every claim the rows make about their own columns still holds. Nulls come last.

fixed pins columns to values given: {"status": ["open", "closed"], "region": "EU"}, a scalar for every row or a sequence or Series of exactly n, cast strictly to the column's dtype. Every other column is what it would have been, since each is seeded by its own name; a rule reading a pinned column follows the pin. A pin must be values its column accepts, and a column a pass writes -- a rule's, a foreign key's, a hierarchy's, a composite key's -- cannot be pinned. With order_by, pinned rows move with the sort like any other. method="cartesian", whose row count is not n, takes no pins.

checks=True keeps the spec's __checks__ and column validators, which are otherwise validation-only: the frame is drawn as always, the rows every check passes are kept, and when fewer than n pass, a larger frame is drawn from a seed of its own and its first n passing rows taken. A spec whose rows all pass is the frame checks=False gives. Rates a check reads -- a null rate, a weight -- become rates among the rows that pass. A check that rejects nearly every row is refused with a GenerationError naming it; a hierarchy, a key into the spec's own rows, method="cartesian" and a pin of all n values cannot be filtered, and are refused. Added in 1.7.

The whole frame is built before this returns. scan() is the lazy verb: it generates as the plan is collected, so only the columns and rows a plan asks for are made.

Examples:

>>> spec = TableSpec("Orders", {
...     "id": ColSpec(pl.Int64, unique=True),
...     "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> frame = generate(spec, 1_000, seed=1)
>>> frame.shape
(1000, 2)
>>> frame["id"].is_unique().all()
True

generate_batches

generate_batches

generate_batches(spec: SpecLike, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> Iterator[DataFrame]

Yields chunks of generated rows without holding all n in memory.

Each batch is a window onto the one frame seed describes: a column no pass rewrites holds, batch by batch, exactly the rows generate(n, seed=seed) would, whatever batch_size is. What is drawn per batch instead -- deterministic, but not row for row the whole frame's -- is a column with rules, a foreign key, a composite key, and a List column's elements.

A unique=True column is a window too: its values are a permutation of its value space -- or, filled by a foreign key, of its parent's keys -- unique across the whole n however it is batched; a parent too small for all n is refused before the first batch. A __unique_together__ group and a unique key into the spec's own rows are drawn afresh per batch, so they are distinct within a batch and collide across batches only by chance.

fixed pins columns as generate() takes it: a scalar for every row, or a sequence of all n values, which each batch takes its stretch of. Added in 1.4.

checks=True keeps the spec's checks, as generate() keeps them: each batch is the first rows that pass from where the last batch stopped reading, so a unique=True column stays unique across n. Batches whose rows all pass are the batches checks=False gives. Added in 1.7.

Examples:

>>> spec = TableSpec("Orders", {"id": ColSpec(pl.Int64, unique=True)})
>>> [batch.height for batch in generate_batches(spec, 25_000, batch_size=10_000, seed=1)]
[10000, 10000, 5000]

scan

scan

A LazyFrame that generates rows on demand.

generate() builds the whole frame before it returns. scan() hands back a frame that has not been built: polars asks for the columns and the rows it actually needs, and only those are generated. So

Orders.scan(50_000_000, seed=1).sink_parquet("orders.parquet")

streams in bounded memory, and

Orders.scan(50_000_000, seed=1).select("total").head(5).collect()

generates five rows of one column.

Projecting cannot change what a column holds. Every column is seeded by its name and every pass by what it is for, so dropping a column's neighbours leaves it alone: scan(n, seed=s).select(cols).collect() is scan(n, seed=s).collect().select(cols), for every subset. That is what makes the pushdown free rather than a trade -- except under method="cartesian", where the coverage set is the product of every column and so every column is generated, then projected.

Rows come in batches, so a scan carries generate_batches' terms: a Hierarchy is refused, uniqueness holds within a batch rather than across n, and a column no pass rewrites holds the rows generate(n, seed=s) would while a ruled or foreign-keyed one is drawn per batch.

Internal: not part of the public API.

needed_columns

needed_columns(spec: TableSpec, wanted: frozenset[str]) -> list[str]

wanted plus every column its values depend on, in spec order.

A projection cannot simply drop the rest: a rules column's values depend on the columns its when reads, a composite key's on every member of the group, a foreign key's on the other columns of its key. passes_of already knows what each pass reads and writes, so the closure is that relation followed to a fixed point -- generate these, hand back only what was asked for.

scan

scan(spec: SpecLike, n: int, *, seed: int | None = None, batch_size: int | None = None, method: Method = 'random', references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False) -> LazyFrame

A LazyFrame of n generated rows, produced as they are collected.

batch_size left unset lets polars ask for the size it would like, so a sink gets the batches it writes best; setting it pins the size whatever polars asks.

A scan is batched, so it has generate_batches' terms: a unique=True column is unique across the whole scan -- a foreign-keyed one too -- and a composite key only within each batch. fixed pins columns as generate_batches takes it. fixed added in 1.4.

checks=True keeps the spec's checks batch by batch, as generate_batches does. Which rows are kept depends on every column a check reads, so those are generated whatever the plan projects, and a head takes the first rows of the batches the whole scan would make. Added in 1.7.

Examples:

>>> spec = TableSpec("Orders", {
...     "id": ColSpec(pl.Int64, unique=True),
...     "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> lazy = scan(spec, 50_000_000, seed=1)  # nothing generated yet
>>> lazy.select("total").head(3).collect().shape  # three rows of one column
(3, 1)

estimated_size

estimated_size

estimated_size(spec: SpecLike, n: int) -> int

Bytes generate(spec, n) is expected to hold, as whole bytes.

See the module docstring for what this does and does not count.

Examples:

>>> spec = TableSpec("Orders", {
...     "id": ColSpec(pl.Int64, unique=True),
...     "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> estimated_size(spec, 1_000_000)
16000000

sink_parquet

sink_parquet

sink_parquet(spec: SpecLike, path: str | Path, n: int, *, batch_size: int = 100000, compression: ParquetCompression = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None

Generates n rows and streams them to a Parquet file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_parquet.

Examples:

>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> spec = TableSpec("Orders", {
...     "id": ColSpec(pl.Int64, unique=True),
...     "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> sink_parquet(spec, folder / "orders.parquet", 250_000, seed=1)
>>> pl.scan_parquet(folder / "orders.parquet").select(pl.len()).collect().item()
250000

sink_ipc

sink_ipc

sink_ipc(spec: SpecLike, path: str | Path, n: int, *, batch_size: int = 100000, compression: IpcCompression | None = 'zstd', method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None

Generates n rows and streams them to an Arrow IPC / Feather file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_ipc.

Examples:

>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> spec = TableSpec("Orders", {
...     "id": ColSpec(pl.Int64, unique=True),
...     "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> sink_ipc(spec, folder / "orders.arrow", 1_000, seed=1)
>>> pl.read_ipc(folder / "orders.arrow").shape
(1000, 2)

sink_csv

sink_csv

sink_csv(spec: SpecLike, path: str | Path, n: int, *, batch_size: int = 100000, include_header: bool = True, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None

Generates n rows and streams them to a CSV file in batches.

Extra keyword arguments go to pl.LazyFrame.sink_csv.

Examples:

>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> spec = TableSpec("Orders", {
...     "id": ColSpec(pl.Int64, unique=True),
...     "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> sink_csv(spec, folder / "orders.csv", 1_000, seed=1)
>>> pl.read_csv(folder / "orders.csv").shape
(1000, 2)

sink_ndjson

sink_ndjson

sink_ndjson(spec: SpecLike, path: str | Path, n: int, *, batch_size: int = 100000, method: Method = 'random', seed: int | None = None, references: References = None, fixed: Mapping[str, Any] | None = None, checks: bool = False, **kwargs: Any) -> None

Generates n rows and streams them to a newline-delimited JSON file in batches.

JSON holds a category as its text, so an Enum or Categorical column is written as one -- which read() reads back as declared. JSON has no bytes: a spec with a Binary column raises GenerationError before anything is written, rather than panicking Polars' writer.

Extra keyword arguments go to pl.LazyFrame.sink_ndjson.

Examples:

>>> import pathlib, tempfile
>>> folder = pathlib.Path(tempfile.mkdtemp())
>>> spec = TableSpec("Orders", {
...     "id": ColSpec(pl.Int64, unique=True),
...     "total": ColSpec(pl.Float64, bounds=(0, 100)),
... })
>>> sink_ndjson(spec, folder / "orders.ndjson", 1_000, seed=1)
>>> pl.read_ndjson(folder / "orders.ndjson").shape
(1000, 2)