$ skilldirectory
— zsh — registry.skilldirectory.dev
skill library blog

home / Data Analysis / Spark Optimization Skill by @wshobson

Spark Optimization Skill by @wshobson

FREE
github stars★ 42,243updatedMay 2026repo@wshobson
$git clone https://github.com/wshobson/agents

The skill is the plugins/data-engineering/skills/spark-optimization folder inside that repository. Copy it into the folder your agent loads skills from; check your agent's documentation for where that is.

Read the opening of the original SKILL.md
# Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

## When to Use This Skill

- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew

## Core Concepts

### 1. Spark Execution Model

```
Driver Program
    ↓
Job (triggered by action)
    ↓
Stages (separated by shuffles)
    ↓
Tasks (one per partition)
```

### 2. Key Performance Factors

| Factor            | Impact                | Solution                      |
| ----------------- | --------------------- | ----------------------------- |
| **Shuffle**       | Network I/O, disk I/O | Minimize wide transformations |
| **Data Skew**     | Uneven task duration  | Salting, broadcast joins      |
…

Excerpt only, quoted from the author's repository. Open the repository for the complete file.