skills sourced live from GitHub · all skills listed are free · each skill is provided under its author's license.
home / AI Agents & MCP / LLM Evaluation Skill by @wshobson
The skill is the plugins/llm-application-dev/skills/llm-evaluation folder inside that repository. Copy it into the folder your agent loads skills from; check your agent's documentation for where that is.
Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking.
Implements evaluation strategies for LLM applications: automated metrics, human feedback, LLM-as-judge and benchmarking. It starts from the main evaluation types, then gives a quick start so you can measure a prompt or model change before shipping it.
The original SKILL.md is organised into these sections:
Written and maintained by @wshobson. It is published under the MIT license, which allows reuse with attribution. See the author profile. SkillDirectory does not claim authorship: this page adds our own review and links back to the original, and the skill's text belongs to its author.
# LLM Evaluation Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing. ## When to Use This Skill - Measuring LLM application performance systematically - Comparing different models or prompts - Detecting performance regressions before deployment - Validating improvements from prompt changes - Building confidence in production systems - Establishing baselines and tracking progress over time - Debugging unexpected model behavior ## Core Evaluation Types ### 1. Automated Metrics Fast, repeatable, scalable evaluation using computed scores. **Text Generation:** - **BLEU**: N-gram overlap (translation) - **ROUGE**: Recall-oriented (summarization) - **METEOR**: Semantic similarity - **BERTScore**: Embedding-based similarity - **Perplexity**: Language model confidence **Classification:** - **Accuracy**: Percentage correct - **Precision/Recall/F1**: Class-specific performance - **Confusion Matrix**: Error patterns …
Excerpt only, quoted from the author's repository. Open the repository for the complete file.