skills sourced live from GitHub · all skills listed are free · each skill is provided under its author's license.
home / DevOps / Ai Pipeline Orchestration Skill by @BagelHole
The skill is the devops/ai/ai-pipeline-orchestration folder inside that repository. Copy it into the folder your agent loads skills from; check your agent's documentation for where that is.
Orchestrate AI/ML pipelines for data ingestion, model training, batch inference, and RAG indexing using Prefect, Airflow, or Dagster.
Shows how to orchestrate AI and ML pipelines with Prefect, Airflow or Dagster, with starter code for RAG ingestion, batch LLM inference and model-training workflows. Its tool-selection section helps you choose between the three before you commit.
The original SKILL.md is organised into these sections:
Written and maintained by @BagelHole. It is published under the MIT license, which allows reuse with attribution. See the author profile. SkillDirectory does not claim authorship: this page adds our own review and links back to the original, and the skill's text belongs to its author.
# AI Pipeline Orchestration Build reliable, observable AI workflows — from document ingestion to batch inference to model training pipelines. ## When to Use This Skill Use this skill when: - Scheduling recurring RAG document ingestion and re-indexing - Orchestrating multi-step batch LLM processing workflows - Running nightly model evaluation and fine-tuning jobs - Building ETL pipelines that feed into AI models - Managing dependencies between data preparation and model serving ## Tool Selection | Tool | Best For | Complexity | GPU Jobs | |------|----------|------------|----------| | **Prefect** | Modern Python-first; easy to adopt | Low | Good | | **Airflow** | Complex DAGs; large teams; existing usage | High | Good | | **Dagster** | Asset-centric; strong data lineage | Medium | Excellent | | **Temporal** | Long-running workflows; reliability-first | Medium | Good | ## Prefect — Quick Start ```bash pip install prefect prefect-kubernetes …
Excerpt only, quoted from the author's repository. Open the repository for the complete file.