Edge Cases & SEO Consequences: Bad Behavior vs. Cloudflare

One of our consultant clients pointed this one out to me. It’s a good example of how vitally important it is for SEO consultants to understand how the search engine machine actually works.

While troubleshooting a potential Penguin issue for a client, she discovered that the client was using the “Bad Behavior” plugin for WordPress, which blocks access for bad bots, fake search engine bots, and the like. By itself, this is awesome.

She also learned that the client had recently started using Cloudflare as well.

Cloudflare is a pretty cool CDN & caching system which, in a lot of cases, will make your website faster and more secure. By itself, this too is pretty awesome. It’s not the only solution but it’s pretty awesome and it’s easy to set up.

A little digging by our friend led her to this TechNet post by Jeremy Clark. Like bleach and ammonia (never mix them!), we have in this case two useful solutions, which lead to disaster when combined.

Here’s what happens:

  1. A real live Googlebot (or even a Bing bot) shows up and requests a page.
  2. Because they’re using Cloudflare, that request gets passed through to WordPress from a Cloudflare IP address.
  3. Bad Behavior checks the IP address of this alleged Googlebot and discovers that it’s not coming from a Google IP address.
  4. Bad Behavior says “no content for you, bad bot!” and returns a 403 Forbidden response to Googlebot.
  5. Bing, Google, and other search engines can’t get any content from the site, rankings drop, bad things happen, etc.

This kind of problem can be especially difficult to diagnose if the timing lines up with a Google update. Like, say, Penguin – as it did in this case. Which is why we tell all of our clients that the date of a ranking/traffic drop is a hint, but not the final answer.

Since we live in a day and age when Google is dropping multiple significant change events every single month, the chances of a “false positive” are WAY too high to just assume “well, rankings dropped on April 19 or 20, must be that Panda data push.”

Think about it for a second here… in April alone, we had at least 3 significant change events at Google, and each one spans 2-3 days on the calendar, depending on what time zone you’re in. So, that’s potentially 9 different dates – out of 30 – in the month of April.

If something happened to your site that’s completely unrelated to one of those change events, there’s still a 30% chance that it happened on or near the same date.

Fortunately for this client, the solution was simple – just turn off the Bad Behavior plugin, because Cloudflare already does the job of blocking bad bots.

UPDATE: Thanks to Matthew Prince (https://twitter.com/#!/eastdakota) for pointing out another solution. Cloudflare offers an Apache mod that can allow Bad Behavior to function with Cloudflare (http://www.cloudflare.com/wiki/Log_Files).

Comments

  1. Hi Dan and Leslie,

    I have been a huge fan of both of you and your work for years now, even from before you teamed up to form the braintrust. You have both been great examples for the SEO and Internet marketing industry as to how to operate and educate.

    Keep up the great work!

    Regards,
    Lee

  2. Hi Dan and Leslie
    I have been following your recent Sales campaign and am thinking of buying your course before 2 Jun

    One thing I need to ask is that one of the videos you asked your potential clients to watch was showing the date “2009″.

    Your course has 40 videos in it.

    How many of the videos were done in 2009 and if there were quite a few how are you approaching the recent changes with Google Penguin (and Panda)

    All the best

    Sandy MacGregor

    • Sandy, none of the videos in the White Hat Black Belt course was produced in 2009. They were all produced between late 2011 and 2012. The material that is relevant to Penguin was all produced in the last month.

      We do use some supplemental videos that date back to 2009, 2010, and 2011, because they are still accurate and relevant to doing the job today.

      Penguin, and the anti-link-spam efforts that Google has been engaged in, both target schemes and techniques that we have never taught. I have been warning people against the blog networks, for example, since 2006 when they first started becoming “popular.”

      A lot of people who have been hit by the Penguin update will not “recover” no matter what they do to address any on-site issues, because they have also lost a very large number of links, that are never coming back.

  3. Dan, my rankings drop happened within a few days of adding some new plugins to my blog section of my site. Around March 6th. I was actually on the WordPress SEO seminar that you did with Joost and added a number of plugins that he recommended:

    Pinterest Pin It Button
    Use Google Libraries
    W3 Total Cache
    WP-PageNavi
    WP Smush.it

    It took a long time to disable them because I assumed it had something to do with either my on site SEO or some link building I had done in the (distant) past, but the date my rankings dropped didn’t seem to be around any of the Google updates, so I only deactivated the plugins on May 8th. It’s now May 29th, but my rankings haven’t bounced back.

    Do you know if there’s anything about any of these plugins that might be causing the issue? I’m especially thinking of W3 Total Cache. I remember that this had something about setting up a CDN?

    Thanks,

    • Leigh,

      When I look at the top referring domains for your site in MajesticSEO, I see a lot of (apparently paid) blogroll links. Often linking to these guys: http://internetconsultingnow.com/ – do you know them, or are they just buying links from the same place?

      Have you ever used “SE Nuke?” Have you ever done article spinning? Ever signed up for any blog networks?

      I only ask because of what I see on SERPs like this:
      http://www.google.com/search?q=“206-LM-ULT5-fb25″
      http://www.google.com/search?q=“No%3A1206-LM-ULT5-dc11″

      I see a lot of blog networks, scraper sites, etc. I can see multiple spins of the same article showing up on the same site, with titles like “strategies for picking and wearing jewelry,” “strategies for choosing and wearing jewelry” and on and on. A lot of the top referring domains for your site are de-indexed now, because a lot of them are complete garbage.

      Regardless of whether you put any of those links out there, Google has been removing a whole lot of that kind of stuff from the link graph. Links you were getting credit for before, are not counting any longer, and the pattern I see here suggests that it’s more than accidental. If you were using any networks, those tend to get wiped out (de-indexed) in one big shot.

  4. Thanks Dan, I very much appreciate you taking the time to look at this. I have used a number of link building services over the years (been online since 2003). I’m pretty sure that spinning was in the mix at some stage.

    Over the last 2 years I’ve been concentrating heavily on building relationships with other sites/blogs so that I can write articles which are placed on their blogs. These are real sites. I’ve even got an article onto Virgin, a very high authority site. So, even though there’s some not so good links to my site, I’ve done quite a bit of valuable link building.

    When I compare my back link profile to many of my competitors, it still seems that I have a lot more authority links than most, and many of them seem to be still doing reciprocal linking, so I’m wondering is Google just discounting the “bad” links they find, or are they penalizing me for having them there, and if I can get some of those links removed, could it make a difference?

  5. Dan, just wanted to see if you had an opinion on whether you believe they may be penalizing the site, or just ignoring those links. One thing to note: I never received an “unnatural linking” email from Google. Can an unnatural link penalty be in place even if I don’t have such a warning? Thanks.

    • Did you get your website in the Google webmaster? I received an unnatural warning 1 years ago and the website never get into the top 40 result s of Google. So check if your website is still ranking within 40 result then most likely it’s not the Google penalty for unnatural link

  6. Margaret says:

    Hi Leigh,
    I’m not anywhere near the expert Dan is, I’m very much a newbie and a dedicated student of Dan & Leslie’s. Here’s what I understand from their teachings, would be the best course of action:
    1) Get rid of as much crap / spun / bought links / any-other-trick-to-game-the-search-engines.
    2) Fix up internal links – internal site crazy linking and any-other-trick-to-game-the-search-engines is bad on-site as well as off
    3) Start investing in and pumping out quality content, daily if possible

    And most importantly, if you’re not already, become a member and get on the coaching calls if what you’ve done above isn’t giving good results. If you are a member, get on the calls!

  7. Wow, that was a pretty deep article. I use the bad behavior plugin on quite a few of my sites. I am definitely going to check into this. Are you aware if this particular plugin has been known to conflict with any others that could cause ranking issues?

  8. In my experience, cloudflare has been very damaging to my rankings. Just wanted to share. Fantastic app but not at the cost of rankings.