Architecture¶

polspec is a small Python package over a Rust extension. The Python side owns the vocabulary — what a column can declare and what that means; the Rust side owns only the inner loop that fills arrays with values.

Modules¶

The dependency direction is one-way: spec and tablespec know nothing about framespec, and report is not reachable from either the generation or validation path.

Generating¶

flowchart LR
    A["FrameSpec.generate(n, seed)"] --> B["_generate_random<br/>or _generate_cartesian"]
    B --> C["_plan_column: one<br/>ColumnPlan per column"]
    C --> D["Rust: generate_dataframe<br/>columns in parallel"]
    D --> E["_finish: gather typed choices,<br/>cast temporal columns back"]
    E --> F["order the passes<br/>by reads and writes"]
    F --> G["each pass: rules, foreign keys,<br/>composite-key repair"]
    G --> H[DataFrame]

Each column becomes a ColumnPlan — kind, nullability, exact bounds, domain size and weights, lengths, distribution — crossing into Rust once. Rust fills the columns in parallel, and within a column in 65,536-row chunks whose seeds come from the chunk index, so output is identical regardless of thread count.

For a fixed-width column a chunk is a unit of work, not a unit of storage. The values buffer and the validity bitmap are each allocated once at the column's full length, and a chunk fills its own disjoint slice of both — which is why the chunk size is a multiple of 8, so the bitmap divides on a byte boundary and no two threads touch the same byte. The column reaches Polars as a single chunk, so nothing downstream — the gather behind a choices domain, the cast behind a temporal dtype, a sink_* write — pays for a column split into hundreds of pieces.

String columns are the exception: a row's width is not known until it is drawn, and Polars backs them with view arrays, which merge by copying sixteen bytes of view per row. That costs more than the split it would remove, so a long string column stays chunked.

Rules and foreign keys are applied afterwards as vectorised passes over the finished frame, not row by row. Each pass declares the columns it reads and the ones it writes, and constraints.order runs them so no pass reads a column a later one rewrites: a rule keyed on a foreign-keyed column sees the parent's values, and a self-referencing key drawing from a foreign-keyed column draws from values that are actually there. That ordering is what makes generated data satisfy the same claims validation checks it against — which is why a spec whose passes cannot be ordered is refused at declaration rather than generated and then failed by its own spec.

Seeds are drawn per pass in declaration order, so which order they end up running in does not change the values any one of them samples.

A unique=True column never reaches a pass: the engine draws it without replacement in the first place (src/unique.rs), shuffling a materialised domain when the domain is barely bigger than the frame and rejecting against a set when it is roomy. A __unique_together__ group is a pass, because distinctness across columns can only be judged once they all exist: it resamples the rows repeating a combination, and reads every member so it runs after the rules and keys that settle them.

Validating¶

flowchart LR
    A["FrameSpec.inspect(df) / validate(df)"] --> B[Structural checks]
    B --> C["Build one _Constraint<br/>per declared claim"]
    C --> D["One Polars aggregation<br/>over the whole frame"]
    D --> E["Each constraint turns its<br/>result into a Finding"]
    E --> F["ValidationReport<br/>(what inspect returns)"]
    F -->|validate: findings| G[ValidationError carrying the report]
    F -->|validate: none| H["Drop / add / cast / reorder"]

Every claim a spec makes becomes a _Constraint that contributes aggregation expressions and turns the results back into a Finding: a code, a count, samples, code-specific details, and a lazy filter that locates the rows. They are collected first and evaluated together, so validating a wide table costs one scan rather than one per column. Foreign keys are the exception: each needs its own anti-join against a parent frame.

Adding a new kind of check means adding a class, not editing two distant loops.

The Python / Rust boundary¶

Python builds one ColumnPlan per column -- a #[pyclass] in src/plan.rs that validates itself at construction, so an unknown kind, a weight vector of the wrong length or a distribution parameter out of range is refused with a message naming the column before any sampling starts. polspec/_ffi.py is the only module that imports the extension, lazily: validation, spec files and the registry work without a built extension, and only generation asks for one.

Rust knows about kinds, not about polspec's vocabulary: int8 .. uint64, float32/float64, bool, string, and index. Date crosses as an int32 day count and Datetime/Duration/Time as an int64 in their own unit. Anything with a finite domain -- choices, an Enum, a capacity-limited Categorical -- crosses as index with the domain's size and weights; Rust returns UInt32 indices and Python gathers the typed values back, so a datetime or a bytes choice never passes through a string.

Bounds cross as a Limit: an i64, a u64 or an f64, whichever holds the Python value exactly, so Int64 and UInt64 bounds keep every bit.

Distribution parameter aliases live only in polspec/distributions.py and are resolved when a column is declared; src/dist.rs reads canonical keys and exports its table as distribution_params(), which a test compares with the Python one. Each column's seed is derived from the frame seed and the column name (sample.rs), so inserting a column never reshuffles its neighbours. src/sample.rs and src/unique.rs have no Python types and carry the unit tests cargo test runs; python/polspec/_polspec.pyi is the stub, and a test asserts its names match the module.

Tests¶

The round-trip file carries xfail(strict=True) markers for known gaps, so a fix turns the marker into a failure rather than passing unnoticed. See Known limitations.