$ skilldirectory
— zsh — registry.skilldirectory.dev
skill library blog

home / SEO / LLM Crawler Access Check Skill by @davepoon

LLM Crawler Access Check Skill by @davepoon

FREE
github stars★ 3,728updatedAug 2026repo@davepoon
$git clone https://github.com/davepoon/buildwithclaude

The skill is the plugins/all-skills/skills/llm-crawler-access-check folder inside that repository. Copy it into the folder your agent loads skills from; check your agent's documentation for where that is.

Read the opening of the original SKILL.md
# AI crawler access check

One wrong line in `robots.txt` removes a site from AI answers completely, and no
amount of content work can compensate. This check takes under a minute and
should run before any other AI-visibility work.

## Scope

Read `https://<domain>/robots.txt` and nothing else. Do not crawl the site, do
not attempt to access disallowed paths, and do not bypass any access control.
This is a read of one public file.

## Procedure

### 1. Fetch

Fetch `https://<domain>/robots.txt`.

- **404 or empty** - everything is allowed by default. Say so; that is a valid
  and often correct configuration. Stop and report.
- **Non-200 other than 404, or unreachable** - report the status code and stop.
  Do not guess at contents.
- **Served as HTML** (a soft 404 returning the site's error page) - flag it.
  Crawlers may parse it as garbage. This is itself a finding.

### 2. Resolve each agent

For each agent below, apply standard robots.txt matching: the most specific
…

Excerpt only, quoted from the author's repository. Open the repository for the complete file.