skills sourced live from GitHub · all skills listed are free · each skill is provided under its author's license.
home / Data Analysis / Spark Optimization Skill by @wshobson
The skill is the plugins/data-engineering/skills/spark-optimization folder inside that repository. Copy it into the folder your agent loads skills from; check your agent's documentation for where that is.
Tuning guide for Apache Spark jobs: partitioning, caching, shuffle reduction and memory settings.
Explains the execution model and key performance factors, then shows optimised session setup, efficient reads and transformations. Aimed at engineers fighting slow stages or scaling a pipeline.
The original SKILL.md is organised into these sections:
Written and maintained by @wshobson. It is published under the MIT license, which allows reuse with attribution. See the author profile. SkillDirectory does not claim authorship: this page adds our own review and links back to the original, and the skill's text belongs to its author.
# Apache Spark Optimization
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
## When to Use This Skill
- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew
## Core Concepts
### 1. Spark Execution Model
```
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
```
### 2. Key Performance Factors
| Factor | Impact | Solution |
| ----------------- | --------------------- | ----------------------------- |
| **Shuffle** | Network I/O, disk I/O | Minimize wide transformations |
| **Data Skew** | Uneven task duration | Salting, broadcast joins |
…
Excerpt only, quoted from the author's repository. Open the repository for the complete file.