Generating data¶
The examples on this page generate data from these specs:
from datetime import date
import polars as pl
from polspec import ColSpec, ForeignKey, FrameSpec
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 1_000_000), unique=True)
customer_id = ColSpec(pl.Int64, bounds=(1, 10_000))
status = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]))
total = ColSpec(pl.Float64, bounds=(0.0, 5_000.0))
internal_note = ColSpec(pl.String, string_length=(0, 40), nullable=True)
class OrderLines(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 1_000_000))
line_no = ColSpec(pl.Int32, bounds=(1, 50))
quantity = ColSpec(pl.UInt16, bounds=(1, 500))
__unique_together__ = [["order_id", "line_no"]]
__foreign_keys__ = [ForeignKey("order_id", references=Orders)]
shape: (3, 5)
┌──────────┬─────────────┬────────┬─────────────┬─────────────────────────────┐
│ order_id ┆ customer_id ┆ status ┆ total ┆ internal_note │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ enum ┆ f64 ┆ str │
╞══════════╪═════════════╪════════╪═════════════╪═════════════════════════════╡
│ 935845 ┆ 597 ┆ PAID ┆ 199.269963 ┆ eUhZbMQR62f36osPhyxduKbiR │
│ 512715 ┆ 8167 ┆ PAID ┆ 744.84394 ┆ 1NrtfQRgkw7Vj8OuZZQ21dDWe9N │
│ 79860 ┆ 4558 ┆ PAID ┆ 4410.974884 ┆ lNbQawe │
└──────────┴─────────────┴────────┴─────────────┴─────────────────────────────┘
Columns are generated independently and in parallel by the Rust extension, then
cast to their declared dtypes in one Polars pass. Every value meets its
column's claims — order_id unique, total within its bounds — but a
column with no further claim, like internal_note, is only as realistic as
its declaration: text of the declared lengths. A
format or pattern says what a value
looks like.
Reproducibility¶
A seed fixes the result across processes, machines and thread counts. Each
column derives its own seed from the frame seed and its name, and each
65,536-row chunk from its index, so the same seed gives the same frame
regardless of how many threads did the work, and adding or reordering columns
never changes the values of the others. The passes that run after the columns
are filled — rules, the hierarchy, foreign keys, composite uniqueness — are
seeded the same way, from the frame seed and a key naming what the pass is
for, so a column added beside a ruled or foreign-keyed one leaves it alone
too.
Omit seed and generation is seeded from the clock.
Renaming a column without changing its data¶
Because a column's seed comes from its name, renaming a column changes the
values it produces — which matters when generated frames are snapshots that
other things are compared against. seed_name is the name the seed is
derived from when it is not the column's own, so a renamed column keeps
producing the data it did:
class Before(FrameSpec):
id = ColSpec(pl.Int64, unique=True)
status = ColSpec(pl.String, nullable=True)
class After(FrameSpec):
id = ColSpec(pl.Int64, unique=True)
state = ColSpec(pl.String, nullable=True, seed_name="status") # renamed
before = Before.generate(100, seed=1)
after = After.generate(100, seed=1)
after["state"].equals(before["status"]) # True
A rename is declared, never guessed: TableSpec.rename() does not set
seed_name, and diff(renames=) is told about the same rename
on the validation side. Two columns of one spec cannot share a seed name, and
one cannot borrow another column's name — they would draw identical values,
which is what a rule is for.
seed_name covers the column's passes as well as its values: a ruled
column renamed with one keeps its rule's draw. What no seed name holds is
the frame across polspec versions — see
Stability.
Pinning columns — fixed=¶
fixed= sets columns to values you give, and generates the rest: a scalar
for every row, or a sequence or Series of exactly n.
Added in 1.1.
df = Orders.generate(3, seed=1, fixed={"status": ["NEW", "PAID", "PAID"]})
df["status"].to_list() # ['NEW', 'PAID', 'PAID']
The pins replace their columns after the draw and before the
passes, so a ColRule keyed on a
pinned column follows it: pin country to "UK" and a currency rule on it
gives "GBP". No other column moves, since each is seeded by its own name.
A pin is held to the promise the rest of generate() makes, that its output
validates:
- Its values must be ones the column accepts — dtype, nullability, bounds,
choices, format, pattern, lengths,
uniqueand the column's validators. Values are cast strictly, so1.5on anInt64is refused rather than rounded. - A column a pass writes — a rule's, a foreign key's, a hierarchy's, a composite key's, a derived column — cannot be pinned; the error names the pass. A column a pass only reads can, and a derived column reading it follows the pin.
Each refusal is a GenerationError naming the column. With order_by,
pinned rows are sorted like any other. method="cartesian" takes no pins,
since its row count is not n.
Pins in batches, scans, sinks and registries¶
generate_batches, scan, every sink_*, and a registry's
generate_all, generate_related and scan_all take fixed= too, with
the same rules. Added in 1.4.
batches = Orders.generate_batches(30_000, batch_size=10_000, seed=1, fixed={"status": "PAID"})
lazy = Orders.scan(1_000_000, seed=1, fixed={"status": "PAID"})
A batch is a window onto the frame generate() makes, pins included: a
scalar fills every batch, and a sequence of all n values gives each batch
its own stretch of them. A scan checks its pins when it is built, before
anything is collected, and a pin on a column the plan never reads is
checked and then not drawn. A registry takes pins spec by spec —
{Customers: {"tier": "pro"}}, keyed by the spec, its class or its name —
and a child's foreign key draws from its pinned parent.
The command line takes --fixed COLUMN=VALUE, repeated for several pins,
each value read as its column's dtype: see
generate.
Keeping checks — checks=True¶
__checks__ and column validators hold arbitrary expressions, so a
free draw can break them. checks=True keeps them: the frame is read
as always, the rows every check passes are kept, and when fewer than
n pass, more of the same frame is read. Added in 1.7.
from polspec import Check, col
class Invoices(FrameSpec):
subtotal = ColSpec(pl.Float64, bounds=(0.0, 100.0))
total = ColSpec(pl.Float64, bounds=(0.0, 110.0))
__checks__ = [Check(col("total") >= col("subtotal"), name="total_covers_subtotal")]
invoices = Invoices.generate(1_000, seed=1, checks=True)
Invoices.inspect(invoices).passed # True
Every row kept is a row of the one frame the seed describes, so:
- A frame whose rows all pass is the frame
checks=Falsegives. - A
uniquecolumn stays unique and a foreign key still points at its parent.generate_batches,scan, the sinks and a registry'sgenerate_all,generate_relatedandscan_alltakechecks=Truetoo, and each batch reads on from where the last stopped, so uniqueness holds across batches. - A rate a check reads becomes a rate among the rows that pass: keep
total >= subtotaland fewer high subtotals are left.
Some things cannot be kept, and each is a SpecError naming why:
method="cartesian", whose coverage set filtering would break; a
__hierarchy__ or a key into the spec's own rows, which link rows to
rows that filtering removes; and a fixed= pin of all n values. A
single-value pin works. A check that rejects nearly every row, so that
more than fifty times n rows would be needed, is a GenerationError
naming each check and how many rows it rejected. A spec with no
checks is generated as it always is.
Lazy output — scan()¶
scan() returns a pl.LazyFrame that has not been generated. Nothing is
drawn until the plan is collected, and then only the columns and rows the
plan asks for:
lf = Orders.scan(50_000_000, seed=1)
lf.sink_parquet("orders.parquet") # streams; bounded memory
lf.select("total").head(5).collect() # five rows of one column
lf.filter(pl.col("status") == "PAID").collect() # every row drawn, matching kept
Projecting cannot change what a column holds. Every column is seeded by
its name and every pass by what it is for, so dropping a column's
neighbours leaves it alone — lf.select(cols).collect() is always
lf.collect().select(cols). Where a column depends on others (a rule reads
the columns its when names, a composite key is repaired as a group), those
are generated too and dropped again on the way out. Under
method="cartesian" every column is generated whatever the projection, since
the coverage set is the product of all of them.
A predicate filters rows that were drawn; it never narrows the draw. n
rows are generated and the matching ones kept, because drawing only matching
rows would quietly change what a null_probability or a unique=True
column means.
Rows arrive in batches, so a scan carries the terms
batching does: a spec declaring a __hierarchy__ is refused,
and a composite key is distinct within each batch rather than across the
scan; see Known limitations.
Leaving batch_size unset lets polars ask for the size it wants; setting it
pins the size.
Reading explain() for a scan. The generator appears in a query plan as
a Python scan, with the columns and the filter polars handed it underneath:
PROJECT 1/18 COLUMNS says one of the spec's eighteen columns is
generated; SELECTION is the predicate applied to the rows drawn; * in
place of a count means every column.
On Polars 2 the scan names itself, and says what it was asked for:
PYTHON[polspec: Orders] SCAN []
PROJECT 1/18 COLUMNS
INFO: 1,000,000 rows, seed=100, batches chosen by polars
Polars 1 gives a Python source nowhere to put a name, so there the header
stays PYTHON SCAN — polspec asks register_io_source what it accepts and
passes the labels only where they are taken.
Registry.scan_all() is the same for a set of specs: parents are generated
eagerly — a foreign key needs the whole parent column to sample from — and
the children are lazy.
Note
generate() builds the whole frame before it returns, so .lazy() on
its result is a handle on memory already spent. scan() is the frame
that has not been built. (generate(lazy=True) did the former while
looking like the latter, and was removed in 0.9.0.)
Coverage — method="cartesian"¶
The default method="random" draws each column independently, so a rare enum
value may not appear at all. method="cartesian" guarantees it will:
It builds the cross-product of every finite domain — each Enum's categories,
both booleans, and the negative / zero / positive / null partitions of every
bounded numeric column — so every combination is present at least once.
Columns with no finite domain (String, bare Categorical) are filled in
randomly alongside.
n is a minimum here, not a count
If the coverage set is smaller than n it is padded with random rows. If
it is larger, all of it is kept and n is exceeded. Two ten-category
enums produce 100 rows however small n was.
A safety cap refuses to build more than 50 million coverage rows, naming each dimension's cardinality so you can see which one exploded.
Batching¶
For volumes that should not be held in memory at once:
Each batch is a window onto the one frame the seed describes: a column
no pass rewrites holds, batch by batch, exactly the rows
Orders.generate(n, seed=1) would, whatever batch_size is — so a stream
written at one batch size and re-read at another is the same data, and the
third batch can be checked against generate(n).slice(...). What is drawn
per batch instead, deterministic but not row for row the whole frame's, is
a column with rules, a foreign key, a composite key, and the elements of a
List, Array or Map column (a List's lengths are a window) — each
batch drawing its elements from a seed of its own. A unique=True column is a window
too: its values are a permutation of its value space, unique across every
batch.
A small batch costs little. The columns a window draws are filled many
batches at a time, up to about 64 MiB, in parallel, and each batch is a
slice of that: a million rows in batches of 1,000 take about twice as long
as one generate() of them. A List, Array or Map column is the
exception, drawn batch by batch, so many small batches of a spec with one
cost more. Faster since 1.6.
Writing straight to a file¶
Four sinks stream batches to disk without materialising the whole frame:
Orders.sink_parquet("orders.parquet", 50_000_000, compression="zstd")
Orders.sink_csv("orders.csv", 1_000_000)
Orders.sink_ipc("orders.arrow", 1_000_000, compression="zstd")
Orders.sink_ndjson("orders.ndjson", 1_000_000)
Each is scan() handed to the matching
LazyFrame.sink_*, so what a sink writes is what collecting the scan gives.
All four take batch_size, method, seed and references, create the
parent directory if needed, and pass extra keyword arguments through to
polars' own sink. Nothing beyond Polars is needed for any of them.
With n=0, Parquet, IPC and CSV still write a valid schema-bearing file.
A CSV cannot hold a Duration, List or Struct column, so sink_csv
refuses a spec with one; Parquet and IPC keep every dtype. JSON holds
categories, decimals and durations as text, which
read() reads back as declared; it has no bytes, so
sink_ndjson refuses a Binary column, and no NaN, which it writes as
null.
A sink is the shorthand; Orders.scan(n, seed=1).sink_parquet(path) is the
same write with the rest of a lazy plan available — a filter, a select,
a sort — before it reaches the file.
Memory¶
Performance gathers what each way of generating costs, in time and memory.
generate() allocates the whole frame before it returns, so it is worth
knowing what that is before asking for it:
Read off the declaration — the width of each dtype, the lengths the spec
declares — so it costs nothing and needs no data. Past four gibibytes
generate() says so in a warning naming the estimate; max_bytes= makes
it a refusal instead, for a CI job that should fail rather than swap, and
max_bytes=0 silences both.
What each column costs per row:
| Declared | Bytes per row |
|---|---|
Int8/UInt8 … Int64/Float64 |
1 … 8 |
Boolean |
⅛ |
Date |
4; Time, Datetime, Duration |
Decimal |
16 |
Enum |
1, 2 or 4 — the narrowest that holds the categories |
Categorical |
its registry's physical width, 4 by default |
String, Binary |
16, plus the length of any value past 12 bytes, on the rows that are not null |
List(inner) |
8, plus the mean list_length × the element's cost, on the rows that are not null |
Array(inner, w) |
w × the element's cost |
Struct |
the sum of its fields, each costed as a column |
nullable=True |
+ ⅛ |
The sixteen bytes a text value costs before any content is the lever worth
knowing: a low-cardinality string column declared as pl.Enum([...]) costs
one byte per row instead of twenty, and one drawn from choices costs
the sixteen but not the content, because the values are gathered from one
shared buffer.
The estimate is the frame, not the process. Generation holds working
buffers on top — most visibly for Decimal and List, assembled in Polars
rather than filled by the engine — so a peak is higher. For a frame of
scalar columns the two agree within a percent.
scan() and batching are the way out of
the question entirely: both hold a batch at a time rather than the frame.
Foreign keys¶
references maps a parent spec to its data, and makes generated keys
referentially consistent. See Constraints.
orders = Orders.generate(10_000, seed=2)
order_lines = OrderLines.generate(30_000, seed=3, references={Orders: orders})
What generation does not enforce¶
Generation satisfies dtypes, nullability, bounds, string lengths, value
domains, formats, patterns, weights, distributions, unique=True,
__unique_together__, ColRules, derived columns, hierarchies and — when
given parent data — foreign keys.
It does not attempt ColSpec.validators or __checks__ unless asked:
they hold arbitrary expressions, and nothing can draw data to satisfy one
in general. checks=True keeps the rows
that pass them. A pattern the generator cannot follow is filled with
plain text, and says so. See Known limitations.