String formats¶

A String column can say how long its values are. format= lets it say what they look like, and the two sides of the spec read that one word: generation fills the column from the format's own sampler, and validation checks every value against the format's own test.

class Users(FrameSpec):
    user_id = ColSpec(pl.String, format="uuid4", unique=True)
    email   = ColSpec(pl.String, format="email")
    host    = ColSpec(pl.String, format="ipv4", nullable=True)

df = Users.generate(10_000, seed=42)
Users.validate(df)   # passes -- which is the point

Without a format, a column carrying a validator as ordinary as col("email").str.contains("@") could not be generated to satisfy its own spec. With one it can, because the sampler and the validator are one declaration in polspec.formats, reviewed in one diff and pinned by one round-trip test per format.

The formats¶

A validator is deliberately wider than its sampler where real data is: an uppercase UUID from another system validates, even though polspec generates lowercase ones. It is never wider than the standard it names.

The set is closed. A named format is a twenty-line sampler with an unambiguous test; generating from an arbitrary regex would need a regex-to-sampler compiler and has no answer for .*. Validation of an arbitrary regex is what pattern= is for -- checked like a format, generated like nothing at all.

What a format promises¶

Syntax. format="email" generates a well-formed address and validates a well-formed address; nothing is looked up, so nobody@example.invalid is accepted and no generated address is deliverable. hostname accepts localhost; ipv4 accepts 0.0.0.0. See Known limitations.

What a format combines with¶

A format owns the column's whole domain, so it refuses anything that says what the values are a second time:

ColSpec(pl.String, format="email", choices=["a@b.co"])
# SpecError: ColSpec cannot carry both format='email' and choices: each is a
# complete description of the column's domain, and they cannot both hold.

ColSpec(pl.String, format="uuid4", string_length=(36, 36))
# SpecError: ... the format already fixes how long a value is.

ColSpec(pl.Int64, format="uuid4")
# SpecError: ColSpec.format is only supported for pl.String, got Int64.

Everything that says how the values are distributed still applies:

• nullable and null_probability work as on any column.
• unique=True draws without replacement. uuid4 has room for any frame; iso_country runs out at 249 rows and says so by name, the way a small choices list does.
• validators run alongside the format check. A column can carry format="email" and a validator on the domain it should come from.
• tags and col_name are untouched.

Foreign keys¶

A format is part of the column's domain, so a key into a formatted parent is checked at declaration like every other domain narrowing. A child column with format="uuid4" accepts a parent with the same format and refuses one with another format or none; a plain String child accepts any parent. A parent whose choices are listed is checked value by value.

class Sessions(FrameSpec):
    user_id = ColSpec(pl.String, format="uuid4")
    __foreign_keys__ = [ForeignKey("user_id", references=Users)]

users = Users.generate(100, seed=1)
sessions = Sessions.generate(1_000, seed=2, references={Users: users})
Sessions.validate(sessions, references={Users: users})

Reports and files¶

The format finding reports values that do not match, with the format named and samples listed:

Column 'host': found 3 value(s) that are not ipv4 (four dotted decimal
octets, each 0-255). Invalid samples: ['300.1.1.1', 'abc']

A format is one scalar key in a spec file -- format: email -- and one keyword in generated Python. Neither changes the file format version. from_dataframe() does not infer formats: guessing email from a sample is a guess, and the profiler's contract is to describe what is there.