Apache Spark AI Skill Guide

Overview & Engine Architecture

Spark distributes DataFrame/Dataset work across executors. Transformations build a lineage; actions (count, write) trigger jobs split into stages at shuffle boundaries. Agents design partition keys, avoid wide shuffles, cache only reused hot datasets, and read Spark UI / explain plans before throwing more cores at a bad query.

Driver (SparkSession)
   -> jobs / stages / tasks
        -> executors (partitions)
             -> shuffle / reduce
             -> sink (Parquet, table)

When to use this skill

• Multi-node ETL beyond single-machine Polars/DuckDB
• Lakehouse table transforms (Parquet/Delta/Iceberg via connectors)
• Spark SQL shared with analytics engineers

Operational directives

1. Prefer DataFrame API / Spark SQL over RDDs for optimizer benefits.
2. Repartition or bucket on join/group keys intentionally; do not repartition(200) blindly.
3. Filter and project before joins.
4. Treat collect() to the driver as a bug on large results.
5. Size AQE / broadcast hints with measured data, not folklore.

PySpark transform example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("orders-mart").getOrCreate()

orders = spark.read.parquet("s3a://lake/orders/")
daily = (
    orders
    .where(F.col("created_at") >= "2025-01-01")
    .groupBy(F.to_date("created_at").alias("day"), "customer_id")
    .agg(F.sum("amount").alias("revenue"), F.count("*").alias("orders"))
)

daily.write.mode("overwrite").parquet("s3a://lake/marts/daily_revenue/")

Diagnose skew

daily.explain(extended=True)
# Spark UI: Stages -> look for long-tail tasks / spill

Common failures

Symptom	Cause	Fix
Shuffle spill	uneven keys / wide join	salt keys; broadcast small side
Driver OOM	.collect / huge broadcast	write out; sample
Tiny files	too many partitions on write	coalesce / maxRecordsPerFile
Slow job	UDF per row	built-in functions / Pandas UDFs carefully

Best practices

• Idempotent writes to dated paths or transactional tables.
• Persist (cache) only multi-action reuse; then unpersist.
• Align partition columns with filter patterns (dt=YYYY-MM-DD).
• Pin Spark + connector versions with the cluster runtime.

Limitations

• Local mode is for unit tests, not production scale claims.
• Streaming Structured Streaming needs separate checkpoint design.
• Cloud IAM/storage credentials are environment-specific.

Related skills

• @duckdb / @polars - single-node alternatives
• @airflow / @prefect - schedule Spark jobs
• @dbt - SQL models on Spark/Databricks SQL warehouses