The RSS Blog

I'm amazed how many bots hit my websites. I'm not just talking Google bot, I'm talking all variety of bots. The fastest growing genre of bot is the referrer bot. This is a bot that simply goes to your website to get listed in your referral logs, so that someone looking at your referral logs will check them out. These referral bots are easily detected because you'll get a ton of referral hits from the same domain, but if you navigate back to the referral URL, you won't actually find a link to your website (examples include sitemeter.com).

But that's not the only case. Often website will put links to your website and create bots that go to their site and click thru. These bots are much more difficult to discover, but there's more tricks to find these. Some referral bots, navigate to your page, but never further. So pageviews per visitor is very close to one (examples include langocha.com). Other bots go further and actually pretend to navigate your site randomly (like a user) and thus increase pageviews per visitor to something more akin to human traffic. But even these referrers have one fatal flaw. They generally only click on HTML links and not javascript generated links.

Let's examine the referrer logs of sensibleerection.com. Hundred of hits, many even click HTML links to navigate to other pages on my website, but not once has anyone ever clicked on a javascript link. Are these real users? Other referrers like that include geocities.jp/net_style3050.

Anyhow, I decided to compile a list of sites that fall into this category. It's not a list of site I think are running bots, but rather sites where I get a lot of referrers but almost zero clicks on javascript links. There are many reasons why this data is so and I'll try to explain some of those reasons.

1. images.google.com (0.19%)
2. technorati.com (0.32%)
3. gada.de (0.00%)
4. googleblog.blogspot.com (0.10%)
5. tankride.com (0.00%)
6. sensibleerection.com (0.00%)
7. images.google.ca (0.16%)
8. youtube (0.27%)
9. langocha.com (0.00%)
10. sitemeter.com (0.00%)
11. eurekster.com (0.00%)

Google's image search wraps the referrer in a frame, thus reducing the likeliness of the user clicking on your site links altogether, which explains the low numbers there. Google blog, eurekster.com and tankride.com all link to pages that have few javascript links that I'm tracking. Both Technorati and YouTube have very low CTR on my tracked javascript links, but not zero. The interesting one is gada.de. You would think that out of the thousands of referrers I get monthly from them that just one user would accidently click on a javascript link. I wonder about gada.be referrers and I honestly wonder about YouTube and Technorati too. Now, don't get me wrong, clearly I must be getting some legitimate referrers from all three. I'm going to tighten up the reporting to try and find an alternate explanation. Does anybody have an alternate explanation?