Removing SPiders ... Extermination Problem

[MrDarkWolf]

This has been an ongoing problem for sometime. The famous Spider Infestation on Forums.

Lately I have noticed a large number of guests on the site again, all are ikotomi spiders (better known as yahoo searching for more email addresses to sell and spam). I have my settings set so that guests can no longer access the boards. They should only be able to see the standard error screen stating that they do not have permission to view the site.

So, how is it that spiders are accessing forum pages?

How is it that a Spider is actually using the Forum E-Mail services to E-Mail members on my site? (See attached screenshot)

Atleast now I know why I have suddenly been getting a large increase in the virus and spam traffic through our website email accounts.

If you have any suggestions... I welcome them ...

I am running VB3.0 RC3
Guests are blocked from viewing pages. Unless you are a registered member, you can not view the forums.

[Jake Bunce]

The image means they are viewing a "no permission" page. The spiders in that screenshot do not have access.

[Andy]

The image means they are viewing a "no permission" page. The spiders in that screenshot do not have access.

If the indicates they don't have access then why does the location say they are in an area that has been restricted?

[merk]

If the indicates they don't have access then why does the location say they are in an area that has been restricted?

Because the who's online system determins their location by the file they have accessed (memberlist.php) and/or by the variables passed into vBulletin when a page is requested (emailing a person).

URL links still exist in YAHOOs database, so they will still know the location of pages to peform actions.

[Andy]

Thank you Merk, now it makes sense.

[Jake Bunce]

If the indicates they don't have access then why does the location say they are in an area that has been restricted?

They found the link somewhere and followed it.

[I, Brian]

Note that Yahoo / Inktomi is not an e-mail harvester - but pages with e-mails listed by Yahoo/MSN, etc, may get harvested.

If you really see a problem with spiders, then use a robots.txt file to block them all, such as:

User-agent: *
Disallow: /forums/

where "/forum/" is your actual forums folder. If you forums are installed at the root directory, then simply use:

User-agent: *
Disallow: /

[Stadler]

I'd recommend putting the following into your robots.txt:

 User-agent: *
 Disallow: /forum/attachment.php
 Disallow: /forum/avatar.php
 Disallow: /forum/editpost.php
 Disallow: /forum/login.php
 Disallow: /forum/member.php
 Disallow: /forum/member2.php
 Disallow: /forum/misc.php
 Disallow: /forum/moderator.php
 Disallow: /forum/newreply.php
 Disallow: /forum/newthread.php
 Disallow: /forum/online.php
 Disallow: /forum/poll.php
 Disallow: /forum/postings.php
 Disallow: /forum/printthread.php
 Disallow: /forum/private.php
 Disallow: /forum/private2.php
 Disallow: /forum/profile.php
 Disallow: /forum/report.php
 Disallow: /forum/register.php
 Disallow: /forum/search.php
 Disallow: /forum/sendmessage.php
 Disallow: /forum/subscription.php
 Disallow: /forum/sendtofriend.php
 Disallow: /forum/threadrate.php
 Disallow: /forum/usercp.php
 Disallow: /forum/admincp/
 Disallow: /forum/modcp/
 Disallow: /forum/images/

With that, you'll save loads of bandwidth, since the spiders won't access any scripts they can't access as a guest. Thanks to Scott for the first version of that

[MrDarkWolf]

I do use the robots.txt file in our forums and other areas of the site...

and while what you say makes sense regarding lining to pages in its databases, these spiders are spending time visiting brand new threads, even e-mailing members who have only recently joined the site.

Yahoo is well known for various activities that have not been proven, such as selling e-mail addresses.

Currently, I have found 2 of the Ikotomi Spiders registering on the site, with real accounts being created at the time of their connection (no other guests showing up in the who's online).

I currently deny access to all pages to everyone but those that are registered members with activated accounts. Unfortunately, I have noticed a large increase in the number of "Members" registering on our Forums. These "Members" never seem to activate their accounts, but they are always trying to get online. Other Members that do activate their accounts are again found sitting in the members listings, etc.

So ... if as you say, the spiders are harmless, respond properly to the robots.txt restrictions, and are only able to access the restricted message... why are they still populating our forums and accessing pages and members that did not exist prior to the updated security settings?

And ... is there a cure to this spider infestation?

[Hendy]

inktomi2-ashf.dtvserver.ntli.net

Not sure which thread to put this in, noticed today that
inktomi2-ashf.dtvserver.ntli.net & inktomi1-ashf.dtvserver.ntli.net are appearing is this the new look for the robot? Extra bandwidth last month cost me just over £300 and I am deperate for it not to happen again.

Block each one as I see it but they keep on coming in this form and the more familiar look j3169.inktomisearch.com

Thank u for any help to get rid of the blighters.

[Floris]

You could also block the ip range those spiders use. They can (if they want to) ignore the robots.txt file, but they can't ignore getting blocked server side.