Skip to content

polars

Tabular data backed by a polars DataFrame. The Asset envelope is payload-agnostic, so the same AssetKind.TABULAR slot can hold a polars frame, and sunstone.polars provides a lineage-tracking facade mirroring sunstone.pandas.

  • AssetKind: AssetKind.TABULAR
  • Payload: polars.DataFrame
  • Typed accessor: Asset.as_polars() -> polars.DataFrame (alongside Asset.as_pandas() for pandas).
  • Status: Supported (eager mode). Install with the [polars] extra: pip install 'sunstone-py[polars]'. Operation-level Activity tracking for chained polars ops is deferred — derived frames propagate their source lineage, and writing a derived frame emits a one-shot LineageWarning.

Why polars

Polars is faster for many group-by, join, and projection workloads than pandas, has lazy execution, and has a cleaner expression API. For new analyses that don't need pandas-ecosystem integrations, polars is often the better default.

Usage

from sunstone import polars as pl
import sunstone

sunstone.set_project_path(".")

df = pl.read_csv("inputs/schools.csv")            # -> sunstone.polars.DataFrame
has_students = df.filter(pl.col("students") > 0)  # chained polars ops
has_students.write_csv("outputs/has_students.csv", slug="schools-with-students", name="Schools with students")

pl.read_csv/read_parquet/read_json/read_dataset resolve a slug or registered path against datasets.yaml, build construction-time lineage (source attribution, a file-content hash, and field derivations) with engine="polars", and return a sunstone.polars.DataFrame. The facade composes an AssetKind.TABULAR Asset:

  • df.asset — the underlying Asset
  • df.data — the polars.DataFrame (df.asset.as_polars())
  • df.metadata — the Metadata container (df.asset.metadata)

Unknown attributes/methods delegate to the underlying polars frame. DataFrame-returning operations are re-wrapped via Asset.derive(...) so source lineage propagates; non-DataFrame results (Series, scalars, .shape) pass through unchanged.

df.set_field_metadata(col, unit=..., description=..., source=...) attaches column-level metadata that flows to datasets.yaml on write, the same as the pandas facade.

Relational operations

Multi-frame operations are first-class on the facade: df1.join(df2, on=...), join_asof, vstack, hstack, extend, and the top-level pl.concat([df1, df2]). They accept facade DataFrames directly (no need to pass .data), combine lineage so the result carries every parent's sources, and merge field metadata (first wins). Overlapping value columns are unit-checked, warning on incompatible units.

joined = customers.join(orders, on="customer_id")   # both are facades
stacked = pl.concat([q1, q2, q3])                    # combined lineage

Note: unlike raw polars, the facade extend is non-mutating — it leaves the caller untouched and returns a new facade DataFrame.

These relational ops live on the eager facade only. Once you call df.lazy(), you are operating on a raw polars LazyFrame (wrapped in a pass-through proxy), so its join/concat/etc. expect raw polars arguments — pass other.data, not the facade. Mixing a facade into a lazy op fails loudly with a TypeError rather than silently dropping lineage.

How it works

The roadmap items that unlocked first-class polars support, now in place:

  1. Polars-aware accessor. Asset.as_polars() -> pl.DataFrame alongside as_pandas(), raising TypeError on a pandas payload (convert explicitly with pl.from_pandas).
  2. Polars-returning format handlers. BuiltinFormatHandler takes an engine="polars" knob on read/write, parsing/writing CSV, JSON, Parquet, and TSV via polars' native readers/writers.
  3. DataFrame facade for polars. sunstone.polars mirrors sunstone.pandas, with lineage-aware read/write helpers. (Op-level Activity tracking for the expression API is deferred.)
  4. Field-metadata bridge. Polars dtypes are inferred into the engine-agnostic field schema at write time; explicit set_field_metadata takes precedence over inference.

Selecting polars at the I/O boundary

You can also select polars per-call at the generic I/O boundary, so the same datasets.yaml entry can be read into either engine:

import sunstone as ss

asset = ss.read("inputs/schools.csv", payload="polars")
df = asset.as_polars()

The payload= argument is the dispatch knob; the format handler remains the same. payload= names the returned Asset's payload type; the same selection is recorded internally as the engine field on lineage — same value ("polars"), different layer.

Lineage parity

Lineage tracking is at parity with the pandas path for reads and writes: source attribution, field derivations, and component metadata all flow through Metadata, which is engine-agnostic. The one gap is operation-level Activity records for chained polars ops — these are deferred, and the absence of an Activity chain is the honest signal that an op is untracked. Writing a detectably-derived frame (a different output slug than its source, or multiple sources) without an Activity chain emits a one-shot LineageWarning.

See also