Polars DataFrame AI Skill Guide

Overview & Engine Architecture

Polars is a Rust-backed DataFrame library with eager DataFrame and deferred LazyFrame plans. Expressions (pl.col, pl.when) describe column work; the optimizer pushes projections and predicates into scans. Agents default to lazy + collect() (or streaming) for multi-file Parquet pipelines and keep joins/filters as expressions, not Python loops.

scan_parquet / scan_csv
      -> LazyFrame plan
          -> filter / select / group_by / join
          -> collect() or sink_parquet()

When to use this skill

• Faster local ETL than pandas on medium/large files
• Columnar pipelines that stay on Parquet
• Expression-heavy transforms before export to DuckDB/Spark

Operational directives

1. Prefer scan_* + LazyFrame over read_* when files are large.
2. Put filters early so predicate pushdown can skip row groups.
3. Select only needed columns before wide joins.
4. Use collect(streaming=True) / sink_parquet for bigger-than-RAM paths (API varies by version).
5. Avoid .map_elements unless no expression alternative exists.

Lazy groupby example

import polars as pl

q = (
    pl.scan_parquet("data/orders/*.parquet")
    .filter(pl.col("created_at") >= pl.datetime(2025, 1, 1))
    .group_by("customer_id")
    .agg(
        pl.col("amount").sum().alias("revenue"),
        pl.len().alias("n_orders"),
    )
    .sort("revenue", descending=True)
)

top = q.collect()

Join with explicit coalesce

result = (
    orders_lf.join(customers_lf, on="customer_id", how="left")
    .select("order_id", "customer_id", "segment", "amount")
    .collect()
)

Common failures

Symptom	Cause	Fix
Schema mismatch on concat	differing dtypes/names	cast / align schemas
Slow collect	materializing too early	keep lazy until end
OOM	wide joins + full collect	project columns; streaming sink
Wrong null join	null keys	filter or coalesce intentionally

Best practices

• Write Parquet with snappy/zstd; partition by date when scans are date-filtered.
• Use explain(optimized=True) when plans surprise you.
• Prefer pl.Expr over UDFs for SIMD-friendly execution.
• Pin Polars version in lockfiles - expression APIs move quickly.

Limitations

• Not a distributed cluster engine (use @spark for that).
• Some pandas idioms (heavy index semantics) do not translate 1:1.
• Streaming support and collect flags differ across minor versions.

Related skills

• @pandas - ecosystem compatibility and small-frame UX
• @duckdb - SQL analytics over the same files
• @spark - multi-node scale-out