Tell crawlers where they can go.
Each starts with / and * matches anything - separate with spaces.
Blocking training doesn't affect Google or Bing search.
Tells crawlers where to find every page - separate several with spaces.
User-agent: *
Disallow:
- Search engines can crawl your whole site.
- No Sitemap line - add your sitemap address so crawlers find every page.
What to do next
Robots.txt, explained.
Where does robots.txt go?
At the root of your site, so it opens at yoursite.com/robots.txt. Crawlers only look there - a file in a subfolder is ignored. Each subdomain needs its own.
What should I block?
Only areas with nothing worth finding in search: admin and login pages, carts and checkouts, internal search results. Never block the CSS and JavaScript your pages load - Google needs them to see the page the way visitors do.
Does blocking a page remove it from Google?
No. robots.txt stops crawling, not indexing - a blocked page can still appear in results if other sites link to it, just without a description. To keep a page out of Google, leave it crawlable and add a noindex tag.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers keeps your content out of AI model training without affecting Google or Bing. Blocking AI search crawlers also stops tools like ChatGPT search and Perplexity from citing your pages - which most businesses want to keep.
Why add a sitemap line?
It tells every crawler where your sitemap is, without you submitting it to each search engine. Use the full address, starting with https://.
My site runs on Shopify, Wix or Squarespace - can I use this?
Those platforms create robots.txt for you and limit how you edit it - Shopify through a theme template, Wix and Squarespace through their SEO settings. Use the file here as a reference for the rules to add there.
Is my data stored?
No. The file is built in your browser - nothing you enter is sent to us or saved.
Read more on this.
Why Does a Small Business Need Digital Marketing?
Learn why digital marketing for small business matters, from local SEO and online visibility to building trust, generating leads and reaching customers.
Growth Marketing Services: A Smarter Way to Scale
Explore Growth Marketing Services connecting digital marketing, lead generation, SEO and performance marketing to drive scalable business growth.
Being crawlable is only the first step.
We fix the technical SEO, content and site speed that decide whether search traffic turns into leads.