Is your robots.txt blocking the wrong crawlers?

A plain-text file at the root of your site that tells crawlers which parts they may visit. It controls crawling, not indexing - a blocked page can still show up in Google if other sites link to it, just without a description.

Whether the file loads, whether Google and Bing can reach your homepage and the CSS and JavaScript it needs, whether it points to a working sitemap, which AI crawlers you allow or block, and which lines search engines ignore because of typos or unsupported rules.

Yes. Enter the full address of the page instead of just your domain, and the report shows whether Google and Bing may crawl it - and which line of your robots.txt decides that.

It depends on what you want. Blocking training crawlers like GPTBot or Google-Extended keeps your content out of model training and doesn’t affect Google Search. Blocking search crawlers like OAI-SearchBot or PerplexityBot means AI tools can’t cite your pages in their answers.

Google renders pages like a browser. If it can’t load your stylesheets and scripts, it may see a broken or empty page - which can hurt how your pages rank, especially on mobile.

Yes. We read your public robots.txt, your homepage and the sitemaps it lists - the same files any search engine reads. Nothing is stored and no login is needed.