Generating data¶

df = Orders.generate(1_000_000, seed=42)

Columns are generated independently and in parallel by the Rust extension, then cast to their declared dtypes in one Polars pass.

Reproducibility¶

A seed fixes the result across processes, machines and thread counts. Each column derives its own seed from the frame seed and its name, and each 65,536-row chunk from its index, so the same seed gives the same frame regardless of how many threads did the work, and adding or reordering columns never changes the values of the others. The passes that run after the columns are filled -- rules, the hierarchy, foreign keys, composite uniqueness -- are seeded the same way, from the frame seed and a key naming what the pass is for, so a column added beside a ruled or foreign-keyed one leaves it alone too.

Orders.generate(500, seed=7).equals(Orders.generate(500, seed=7))   # True

Omit seed and generation is seeded from the clock.

Renaming a column without changing its data¶

Because a column's seed comes from its name, renaming a column changes the values it produces -- which matters when generated frames are snapshots that other things are compared against. seed_name is the name the seed is derived from when it is not the column's own, so a renamed column keeps producing the data it did:

class Before(FrameSpec):
    id     = ColSpec(pl.Int64, unique=True)
    status = ColSpec(pl.String, nullable=True)

class After(FrameSpec):
    id    = ColSpec(pl.Int64, unique=True)
    state = ColSpec(pl.String, nullable=True, seed_name="status")   # renamed

before = Before.generate(100, seed=1)
after = After.generate(100, seed=1)
after["state"].equals(before["status"])   # True

A rename is declared, never guessed: TableSpec.rename() does not set seed_name, and diff(renames=) is told about the same rename on the validation side. Two columns of one spec cannot share a seed name, and one cannot borrow another column's name -- they would draw identical values, which is what a rule is for.

seed_name covers the column's passes as well as its values: a ruled column renamed with one keeps its rule's draw. What no seed name holds is the frame across polspec versions -- see Roadmap and stability.

Lazy output — scan()¶

scan() returns a pl.LazyFrame that has not been generated. Nothing is drawn until the plan is collected, and then only the columns and rows the plan asks for:

lf = Orders.scan(50_000_000, seed=1)

lf.sink_parquet("orders.parquet")               # streams; bounded memory
lf.select("total").head(5).collect()            # five rows of one column
lf.filter(pl.col("status") == "PAID").collect() # every row drawn, matching kept

Projecting cannot change what a column holds. Every column is seeded by its name and every pass by what it is for, so dropping a column's neighbours leaves it alone — lf.select(cols).collect() is always lf.collect().select(cols). Where a column depends on others (a rule reads the columns its when names, a composite key is repaired as a group), those are generated too and dropped again on the way out.

A predicate filters rows that were drawn; it never narrows the draw. n rows are generated and the matching ones kept, because drawing only matching rows would quietly change what a null_probability or a unique=True column means.

Rows arrive in batches, so a scan carries the terms batching does: a spec declaring a __hierarchy__ is refused, and uniqueness holds within a batch. Leaving batch_size unset lets polars ask for the size it wants; setting it pins the size.

Registry.scan_all() is the same for a set of specs: parents are generated eagerly — a foreign key needs the whole parent column to sample from — and the children are lazy.

Coverage — method="cartesian"¶

The default method="random" draws each column independently, so a rare enum value may not appear at all. method="cartesian" guarantees it will:

df = Orders.generate(500, method="cartesian", seed=1)

It builds the cross-product of every finite domain — each Enum's categories, both booleans, and the negative / zero / positive / null partitions of every bounded numeric column — so every combination is present at least once. Columns with no finite domain (String, bare Categorical) are filled in randomly alongside.

Batching¶

For volumes that should not be held in memory at once:

for batch in Orders.generate_batches(10_000_000, batch_size=250_000, seed=1):
    process(batch)

Each batch is a window onto the one frame the seed describes: a column no pass rewrites holds, batch by batch, exactly the rows Orders.generate(n, seed=1) would, whatever batch_size is -- so a stream written at one batch size and re-read at another is the same data, and the third batch can be checked against generate(n).slice(...). What is drawn per batch instead, deterministic but not row for row the whole frame's, is a column with rules, a foreign key, a composite key, and a List column's elements (its lengths are a window). Uniqueness holds only within a batch.

A batch smaller than 65,536 rows -- the engine's chunk -- costs up to one chunk of extra draws per batch, because a window that starts mid-chunk fills the chunk from its start and slices the head off. Batches of a chunk or more cost nothing extra.

Writing straight to a file¶

Four sinks stream batches to disk without materialising the whole frame:

Orders.sink_parquet("orders.parquet", 50_000_000, compression="zstd")
Orders.sink_csv("orders.csv", 1_000_000)
Orders.sink_ipc("orders.arrow", 1_000_000, compression="zstd")
Orders.sink_ndjson("orders.ndjson", 1_000_000)

Each is scan() handed to the matching LazyFrame.sink_*, so what a sink writes is what collecting the scan gives. All four take batch_size, method, seed and references, create the parent directory if needed, and pass extra keyword arguments through to polars' own sink. Nothing beyond Polars is needed for any of them.

With n=0, Parquet, IPC and CSV still write a valid schema-bearing file. A CSV cannot hold a Duration, List or Struct column, so sink_csv refuses a spec with one; Parquet and IPC keep every dtype.

A sink is the shorthand; Orders.scan(n, seed=1).sink_parquet(path) is the same write with the rest of a lazy plan available — a filter, a select, a sort — before it reaches the file.

Memory¶

generate() allocates the whole frame before it returns, so it is worth knowing what that is before asking for it:

Orders.estimated_size(50_000_000) / 1024**3     # gibibytes

Read off the declaration — the width of each dtype, the lengths the spec declares — so it costs nothing and needs no data. Past four gibibytes generate() says so in a warning naming the estimate; max_bytes= makes it a refusal instead, for a CI job that should fail rather than swap, and max_bytes=0 silences both.

What each column costs per row:

The sixteen bytes a text value costs before any content is the lever worth knowing: a low-cardinality string column declared as pl.Enum([...]) costs one byte per row instead of twenty, and one drawn from choices costs the sixteen but not the content, because the values are gathered from one shared buffer.

The estimate is the frame, not the process. Generation holds working buffers on top — most visibly for Decimal and List, assembled in Polars rather than filled by the engine — so a peak is higher. For a frame of scalar columns the two agree within a percent.

scan() and batching are the way out of the question entirely: both hold a batch at a time rather than the frame.

Foreign keys¶

references maps a parent spec to its data, and makes generated keys referentially consistent. See Constraints.

orders = Orders.generate(10_000, seed=2, references={Customers: customers})

What generation does not enforce¶

Generation satisfies dtypes, nullability, bounds, string lengths, value domains, formats, weights, distributions, unique=True, __unique_together__, ColRules, hierarchies and — when given parent data — foreign keys.

It does not attempt ColSpec.validators, __checks__ or ColSpec.pattern, by design: the first two hold arbitrary expressions and the third an arbitrary regex, and nothing can generate data to satisfy either in general. See Known limitations.