What to do if we are Unable to Retrieve Content

If you add a new website and see this message, it generally means that you have a Firewall or CDN (Content Distribution Network) blocking bots. You will need to give our bot access to the site before we can index it for you.

Here are a few common ways in which this happens, along with fixes. If you find a fix that is not mentioned, we would LOVE to hear this in the comments, so that our other users do not have the issue.

CloudFlare

This is probably the most common CDN that blocks bots. Go into Cloudflare and select the site that has the problem. Then select > Forewall > Tools. Put our IP address: 199.16.129.36 (Previously 85.236.156.10) in here and set it to “Allow”. A note is optional, but helpful in case another Cloudflare user wants to know what this rule is for.

Whitelisting an IP Address in Cloudfare

WordPress

If you are not blocking via a CDN, then it is likely to be a plugin if you are using WordPress. The most common plugin for stopping automated traffic is WordFence, which does not usually block us, but in case it does, you can fix it by whitelisting our IP address here:

Wordfence Whitelist

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *