Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
tests/ext_conformance/artifacts/agents-wshobson/data-engineering/skills/spark-optimization/SKILL.md(main)