Testing Your Setup
It’s extremely important to make sure you’re blocking and allowing the right visitors. A wrongly configured setup may create more problems than it solves. It is important that if you’re blocking SEO tools, for example Ahrefs, DO NOT test your site by explicitly requesting a crawl (using a site audit tool, for example). If your site is wrongly configured you will be exposing your content to undesired bots.
In this guide, we’ll be testing the responses for three cases:
- How google sees your site
- How a visitor with a specific user agent sees your site
- How a regular visitor sees your site
Testing how Google sees your site
If you’re filtering Google only by its user agent, skip to Test B. If you’re filtering Google by it’s (domain) IP, there’s no other way to verify what it sees than using its official “Fetch as Google” tool from the Google Console (Webmasters Tool). You can view detailed instructions here: https://www.google.com/webmasters/tools/googlebot-fetch
Testing how a visitor with a specific user agent sees your site
Most bots define a specific user agent that make them identify themselves when crawling the web. You can test this by either installing a browser extension that allows you to change the user agent on-the-fly or you can use an online service like this one.
Here’s an example of testing for an ahrefs user agent using the aforementioned service:
The user agent for ahrefs: Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
Testing how a regular visitor sees your site
If you want to test how a regular visitor sees your site, simply log out of your site and go to your site’s homepage. You can also do this in a private browser tab or window. If you’re the site’s admin and are logged in, the filters will not run in order to prevent you from accidentally blocking yourself from accessing your own site.