Performance¶
polspec is fast by default: columns are filled in Rust, in parallel, and validation is one Polars query. A few choices still decide whether a job takes milliseconds or minutes, and whether it fits in memory. This page gathers them, with what each costs.
The numbers are medians on an Intel 13900K with 64 GB of DDR5, Polars 1.44,
polspec 1.7, for a million rows unless a row says otherwise. Yours will
differ; the ratios matter more than the absolute times.
benchmarks/bench.py measures every case below.
Generating¶
| A million rows of | Time |
|---|---|
| a four-column frame (string, enum, int, float) | 7 ms |
| the same, 20 million rows | 115 ms |
| an eleven-column frame of every kind a real table has | 49 ms |
one unique integer key |
4 ms |
| one column with rules | 29 ms |
a composite unique key (__unique_together__) |
95 ms |
| a hierarchy | 60 ms |
Plain columns cost almost nothing; the claims that relate rows to other rows cost most. A composite key is resampled until its combinations are distinct, and holds several copies of the frame while it does — about 850 MB at a million rows — so it is the claim to leave out of a spec that does not need it.
Memory: generate, batches, scan or a sink¶
generate() builds the whole frame before it returns: the 20-million-row
frame above peaks near 1 GB. For more than fits, stream it:
| A million rows | Holds | Four columns | Eleven columns |
|---|---|---|---|
generate(n) |
the whole frame | 7 ms | 49 ms |
generate_batches(n, batch_size=10_000) |
one batch | 8 ms | |
generate_batches(n, batch_size=1_000) |
one batch | 12 ms | |
scan(n), collected |
what the plan asks for | 76 ms | |
sink_parquet(path, n), written |
one batch | 27 ms |
A small batch costs little: the columns are drawn many batches at a time
and sliced. scan() generates only the columns and rows a plan
asks for, so .select("total").head(5) makes five values of one column,
and a projection never changes what a column holds. See
Lazy output and
Batching. estimated_size(n) says what a frame
will hold before anything is allocated, and max_bytes= refuses one too
large; see Memory.
Keeping checks costs re-reads¶
checks=True keeps the rows every check passes and reads more of the
frame for the rest, so its cost grows with how many rows the checks
reject:
| A million rows of the eleven columns | Time |
|---|---|
generate(n) |
49 ms |
checks=True, a check rejecting 10% of rows |
143 ms |
checks=True, a check rejecting 50% of rows |
328 ms |
A longer read regenerates the rows it read before, which keeps the result the first rows of one seeded frame whatever the batch size. A check most rows pass is cheap to keep; for one that rejects most of them, narrow the columns it reads — bounds, choices, a rule — so fewer rows fail. See Keeping checks.
Validating¶
Every claim on every column is compiled into one Polars aggregation, so a frame is read once however many claims there are:
| A million rows | Time |
|---|---|
| the eleven-column frame, validated | 64 ms |
the same, as a LazyFrame validated and collected |
61 ms |
| one bounded column | 1.5 ms |
A LazyFrame is validated by running its query, and collecting what
validate() hands back runs it again; report.failing_rows() and
passing_rows() run it once more each. That costs nothing for a frame in
memory and a lot for a slow scan or join: collect once, then validate the
DataFrame.
frame = pl.scan_parquet("s3://bucket/orders/*.parquet").collect() # read once
report = Orders.inspect(frame)
clean = report.passing_rows().collect()
streaming=True runs validation on Polars' streaming engine, for a frame
larger than memory. See
Streaming.
Profiling¶
Profiling reads every value of every column: the eleven-column frame
takes 370 ms at a million rows. sample= profiles a random sample instead,
seeded so it is the same every run:
| A million rows of the eleven columns | Time |
|---|---|
profile(frame) |
370 ms |
profile(frame, sample=100_000) |
89 ms |
A sample sees most of what the whole column holds, and misses its rare
values and extremes: bounds come from the sample, so a few real values may
fall outside them. Profile the whole frame for a contract, and a sample to
get started on a large one. Columns are profiled in parallel from 50,000
rows. parse_text=True reads text as numbers and dates first: a frame
whose numbers and dates arrive as text takes about twice as long to
profile as the same frame typed.
Importing¶
import polspec takes about half a second cold, most of it Polars itself.
Generation, validation, drift and profiling load on first use, so a script
that only declares specs pays for none of them.