Robots.txt tester
See how robots.txt controls Google, Bing, ChatGPT, Claude, Perplexity, and Gemini agents.
How to test robots.txt access
Check the robots.txt rules that apply to search crawlers and AI-use agents for one exact URL.
Enter the page you want each crawler to access.
Click Test crawl access to evaluate its website’s public robots.txt file.
Review each crawler separately before changing any intentional restrictions.
Frequently asked questions
It evaluates Googlebot, bingbot, OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended for the exact URL you enter. Google-Extended is an AI-use control rather than a search crawler.
They are separate OpenAI user agents with different purposes. OAI-SearchBot is associated with search discovery, while GPTBot is associated with model training controls, so the tool reports them separately.
No. Allowed only means the robots.txt rules do not block that user agent from the tested URL. It does not prove a crawl, index entry, training use, citation, or ranking.
A normal 404 response means no robots.txt rules were found, so the tester reports the URL as allowed. Other unavailable responses are reported as unknown because crawler behavior can differ.
Yes. Allow and Disallow rules can target particular paths, so test the exact page you care about rather than only the homepage.
Your 
Search ConsoleWith AI superpowers
Turn your search data into clear next steps, right inside your favorite AI.
- Google Search Console
Google Analytics 4
SERP Analysis
Keyword Research- Backlink Analysis
- Technical SEO