skills sourced live from GitHub · all skills listed are free · each skill is provided under its author's license.
home / SEO / LLM Crawler Access Check Skill by @davepoon
The skill is the plugins/all-skills/skills/llm-crawler-access-check folder inside that repository. Copy it into the folder your agent loads skills from; check your agent's documentation for where that is.
Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot.
Checks whether a site's robots.txt allows the AI crawlers behind ChatGPT Search, Perplexity, Claude, Gemini and Copilot, and flags three common mistakes in how such checks are done.
The original SKILL.md is organised into these sections:
Written and maintained by @davepoon. It is published under the MIT license, which allows reuse with attribution. See the author profile. SkillDirectory does not claim authorship: this page adds our own review and links back to the original, and the skill's text belongs to its author.
# AI crawler access check One wrong line in `robots.txt` removes a site from AI answers completely, and no amount of content work can compensate. This check takes under a minute and should run before any other AI-visibility work. ## Scope Read `https://<domain>/robots.txt` and nothing else. Do not crawl the site, do not attempt to access disallowed paths, and do not bypass any access control. This is a read of one public file. ## Procedure ### 1. Fetch Fetch `https://<domain>/robots.txt`. - **404 or empty** - everything is allowed by default. Say so; that is a valid and often correct configuration. Stop and report. - **Non-200 other than 404, or unreachable** - report the status code and stop. Do not guess at contents. - **Served as HTML** (a soft 404 returning the site's error page) - flag it. Crawlers may parse it as garbage. This is itself a finding. ### 2. Resolve each agent For each agent below, apply standard robots.txt matching: the most specific …
Excerpt only, quoted from the author's repository. Open the repository for the complete file.