Announcement

Collapse
No announcement yet.

Is this normal: 123 guests/spiders?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Is this normal: 123 guests/spiders?

    I have about 7 members and 123 guests online, and many of them are lj2541.inktomisearch.com or similar. It seems this is the same spider always on there and I fear it's hogging bandwidth and slowing the site down. I don't mind a few, but six pages of this? Is this normal? And how do I stop it?
    Marc James
    Sports Central — Beyond the Scores
    Sports Central Message Boards

  • #2
    Looks like it's the Yahoo! "Slurp" spider and this is not uncommon. I am going to add this to robots.txt to slow it down:

    User-agent: Slurp
    Crawl-delay: 30

    Not sure what delay to use, any suggestions? Also, will putting robots.txt in my root directory still work for my forums in another directory?
    Marc James
    Sports Central — Beyond the Scores
    Sports Central Message Boards

    Comment

    widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
    Working...
    X