Login to download the latest version of Mint and your favorite Pepper, purchase additional licenses, or post in the Forum. Don't have an account? Create one!

In Partnership with Media Temple

Mint Forum

Spider with JavaScript?

a.jaffe
Minted
Posted on Feb 06, '08 at 09:13 am

HI All,

I’ve been getting lots of hits (dozens per day, which is a significant fraction of all my traffic) from *.nat.svl.kavam.net. It appears to be come kind of spider or robot that, obviously, does use javascript so mint sees it [there is some evidence that at least some of the traffic is due to a robot called Charlotte by a group called SearchMe, but including the appropriate incantation in robots.txt does NOT get rid of the hits].

So - has anyone else seen lots of hits from these domains? And, more importantly, is there any way to filter these out? I thought I remembered there being some such mechanism but I admit I can’t find the posts on that topic!

Yours,

Andrew

Jeff N
Minted
Posted on Feb 06, '08 at 09:52 am

I’m having the exact same issue — an overload of cookied hits from this spider. And just like you, I’ve used their suggested robots.txt to no avail.

I’m currently trying this solution Shaun helped me with a while back haveamint.com/forum/troubleshooting/773 … post_5229, which uses advanced preferences (?preferences&advanced query command) to ignore IP addresses from a certain range (this particular spider is 208.111.154.* for me). I’ll let you know if it works.

a.jaffe
Minted
Posted on Feb 06, '08 at 02:39 pm

Hi,

Yes, that was the magic incantation I couldn’t find.

It seems to be working — thanks!

Andrew

You must be logged in to reply. Login above or create an account