Input
Routes through the ThrillSEO proxy. robots.txt is per host + protocol — fetching one origin doesn't tell you about another.
Summary
—
Test URLs vs. *
URL
Result
Winning rule
Group
Linter findings
AI-crawler panel — 2026
Modern best practice for many publishers: block training, allow retrieval — stay out of model training sets while remaining citable in ChatGPT, Claude and Perplexity answers. robots.txt is voluntary — not all bots obey it; treat this as visibility control, not security.
Training crawlers (block to keep content out of training)
Retrieval / on-demand (usually allow — these cite sources and send traffic)
Applies the RFC 9309 / Google interpretation (longest-match, Allow-wins-ties). Other crawlers may implement matching differently. robots.txt controls crawling, not indexing, and is obeyed voluntarily — treat it as visibility control, not security. robots.txt is per host and protocol; a missing or 4xx robots.txt means the site is fully crawlable. Everything above runs locally. Tool provided free by ThrillSEO.