$ skilldirectory
— zsh — registry.skilldirectory.dev
skill library blog

home / AI Agents & MCP / LLM Evaluation Skill by @wshobson

LLM Evaluation Skill by @wshobson

FREE
github stars★ 42,243updatedJul 2026repo@wshobson
$git clone https://github.com/wshobson/agents

The skill is the plugins/llm-application-dev/skills/llm-evaluation folder inside that repository. Copy it into the folder your agent loads skills from; check your agent's documentation for where that is.

Read the opening of the original SKILL.md
# LLM Evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

## When to Use This Skill

- Measuring LLM application performance systematically
- Comparing different models or prompts
- Detecting performance regressions before deployment
- Validating improvements from prompt changes
- Building confidence in production systems
- Establishing baselines and tracking progress over time
- Debugging unexpected model behavior

## Core Evaluation Types

### 1. Automated Metrics

Fast, repeatable, scalable evaluation using computed scores.

**Text Generation:**

- **BLEU**: N-gram overlap (translation)
- **ROUGE**: Recall-oriented (summarization)
- **METEOR**: Semantic similarity
- **BERTScore**: Embedding-based similarity
- **Perplexity**: Language model confidence

**Classification:**

- **Accuracy**: Percentage correct
- **Precision/Recall/F1**: Class-specific performance
- **Confusion Matrix**: Error patterns
…

Excerpt only, quoted from the author's repository. Open the repository for the complete file.