Bot Hits - The RSS Blog
RSS, OPML and the XML platform.
Copyright 2003-5 Randy Charles Morin
The RSS Blog
<< Previous Main Next >>
Thu, 11 May 2006 15:47:10 GMT
Bot Hits

I'm amazed how many bots hit my websites. I'm not just talking Google bot, I'm talking all variety of bots. The fastest growing genre of bot is the referrer bot. This is a bot that simply goes to your website to get listed in your referral logs, so that someone looking at your referral logs will check them out. These referral bots are easily detected because you'll get a ton of referral hits from the same domain, but if you navigate back to the referral URL, you won't actually find a link to your website (examples include

But that's not the only case. Often website will put links to your website and create bots that go to their site and click thru. These bots are much more difficult to discover, but there's more tricks to find these. Some referral bots, navigate to your page, but never further. So pageviews per visitor is very close to one (examples include Other bots go further and actually pretend to navigate your site randomly (like a user) and thus increase pageviews per visitor to something more akin to human traffic. But even these referrers have one fatal flaw. They generally only click on HTML links and not javascript generated links.

Let's examine the referrer logs of Hundred of hits, many even click HTML links to navigate to other pages on my website, but not once has anyone ever clicked on a javascript link. Are these real users? Other referrers like that include

Anyhow, I decided to compile a list of sites that fall into this category. It's not a list of site I think are running bots, but rather sites where I get a lot of referrers but almost zero clicks on javascript links. There are many reasons why this data is so and I'll try to explain some of those reasons.

  1. (0.19%)
  2. (0.32%)
  3. (0.00%)
  4. (0.10%)
  5. (0.00%)
  6. (0.00%)
  7. (0.16%)
  8. youtube (0.27%)
  9. (0.00%)
  10. (0.00%)
  11. (0.00%)

Google's image search wraps the referrer in a frame, thus reducing the likeliness of the user clicking on your site links altogether, which explains the low numbers there. Google blog, and all link to pages that have few javascript links that I'm tracking. Both Technorati and YouTube have very low CTR on my tracked javascript links, but not zero. The interesting one is You would think that out of the thousands of referrers I get monthly from them that just one user would accidently click on a javascript link. I wonder about referrers and I honestly wonder about YouTube and Technorati too. Now, don't get me wrong, clearly I must be getting some legitimate referrers from all three. I'm going to tighten up the reporting to try and find an alternate explanation. Does anybody have an alternate explanation?

Reader Comments Subscribe

So you are getting hits from my site? Wish I could help you. But only reason I know about your site is the hits I started getting from you after you put my site up.

Chris Berg

I was reading earlier
 That when you get natural type in traffic the referrer will most likely say either unknown referrer, direct traffic, or no referrer and this is a good thing. On the other hand, when you pay for traffic you may notice that almost all the referrer info is the same meaning since the referrer can be changed to what ever then who really knows what's what. swindled halftack sale! We paid for it and it never materialized along with our monies paid.
Hi Hi Hiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.............................
Try analytics.
Type "339":
Top Articles
  1. Unblock MySpace
  2. MySpace
  3. FaceParty, the British MySpace
  4. and
  5. Blocking Facebook and MySpace
  1. Review of RSS Readers
  2. MySpace Layouts
  3. RSS Stock Ticker
  4. RSS Gets an Enema
  5. Google Reader rejects