Robots.txt tester
See how robots.txt controls Google, Bing, ChatGPT, Claude, Perplexity, and Gemini agents.
How to test robots.txt access
Check the robots.txt rules that apply to search crawlers and AI-use agents for one exact URL.
Enter the page you want each crawler to access.
Click Test crawl access to evaluate its website’s public robots.txt file.
Review each crawler separately before changing any intentional restrictions.
Continue checking your website
After reviewing crawl rules, use the Sitemap checker to inspect the URLs your site publishes in its XML sitemap.
To review crawler permissions alongside page signals and optional agent discovery, try the AI SEO checker on the same page.
Frequently asked questions
It evaluates Googlebot, bingbot, OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended for the exact URL you enter. Google-Extended is an AI-use control rather than a search crawler.
They are separate OpenAI user agents with different purposes. OAI-SearchBot is associated with search discovery, while GPTBot is associated with model training controls, so the tool reports them separately.
No. Allowed only means the robots.txt rules do not block that user agent from the tested URL. It does not prove a crawl, index entry, training use, citation, or ranking.
A normal 404 response means no robots.txt rules were found, so the tester reports the URL as allowed. Other unavailable responses are reported as unknown because crawler behavior can differ.
Yes. Allow and Disallow rules can target particular paths, so test the exact page you care about rather than only the homepage.
Skip 
Search ConsoleJust ask your AI
- Google Search Console
Google Analytics 4
SERP Analysis
Keyword Research- Backlink Analysis
- Technical SEO
Free to start · Takes 2 minutes