If you’re running a website, you would have heard of Cloudflare, the US company that provides CDN as well as Internet Security and Firewall services to hundreds of thousands of sites. We had used Cloudflare internally at the business until early this year, while some sites still remained on the legacy platform as late as last week, we had been making a concerted effort to migrate all the websites across and here’s why.
Cloudflare has been causing significant issues for us with some clients, affecting our marketing greatly by preventing Google from crawling the site reliably. It would often drop random requests returning a 503 error code. Causing significant drops in Google ranking and revenue for our client sites.
Cloudflare was never aware of the issue, their platform seemed to be only rejecting requests from Google as normal users had no problem. However, that still had a huge effect on our rankings and dropped us from #1 to #7, impacting client businesses significantly.
Below, you’ll see Google’s crawl results before we removed Cloudflare, and you’ll see how some crawls were rejected and some were not. We did further testing and saw that some were “partial” where some resources were not being served to Google and returning error codes.
These issues made Google believe our sites were unreliable and resulted in drops on the ranking. Once the issue was fixed by removing Cloudflare, Google was able to crawl the website successfully without issues.
So, once we had removed Cloudflare the site began ranking higher and Google started crawling it faster, more often, and more than before. Another thing we noticed is total download time for this site dropped significantly, to 800ms from 2s.
If you’re on a rubbish host, Cloudflare may be a good option, however it is a band-aid solution and it’s no where near as good as high quality hosting from a reputable provider. If you’d like to be put in touch with a reputable web host, feel free to reach out in the comments below and we’ll put you in touch.