Resources
Check to see if your site can be crawled
We created a simple, up-to-date tool that will help you check if your site can be crawled by common search engines and Google. You can input a domain, or a page to see what crawlers can and cannot crawl your site! Remember that this is just to check if it is possible to crawl these pages based on the robots.txt, not if they can be rendered by Google (if using server-side javascript).