Can search engines find all your pages?

A file that lists the pages on your site you want search engines to find, with the date each one last changed. It doesn’t guarantee a page gets indexed, but it helps Google discover new and updated pages faster - especially on new sites or sites with many pages.

Whether search engines can find your sitemap through robots.txt, whether the file loads and follows the sitemap format, whether the URLs it lists are on your own site, use https and aren’t blocked by robots.txt, and whether a sample of those pages actually load and are open to indexing.

Where search engines do: the Sitemap lines in your robots.txt first, then the usual addresses - /sitemap.xml and /sitemap_index.xml. You can also paste the full address of a sitemap to check that file directly.

A sitemap tells Google “these are the pages I want in search”. Listing pages that redirect, return errors, carry noindex or name a different canonical page sends mixed signals - Google may trust your sitemap less and crawl your real pages less often.

We read up to 10 MB of each sitemap, open up to three sitemaps listed in robots.txt and five from a sitemap index, and load a spread-out sample of five pages. Every URL we read is checked against your robots.txt - the page sample is there to catch patterns, like a whole section redirecting.

Yes. We only read public files - your robots.txt, your sitemap and a handful of pages - the same way a search engine does. Nothing is stored and no login is needed.