Announcement

Collapse
No announcement yet.

What does this mean/What is this?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • dynamite
    replied
    Originally posted by Faranth
    Thanks... I was looking for something that blocked certain things, but not the spider completely. Does anyone know how it would affect ranking if they were only able to spider the archive? That should get the text it needs right, but cut down on the bandwidth?

    Leave a comment:


  • Andrew111888
    replied
    If you temporarily turn IP resolving on, you can see if it's inktomi.

    Leave a comment:


  • Zachery
    replied
    http://www.vbulletin.com/forum/showt...ght=robots.txt

    may i suggest that post?

    Leave a comment:


  • dynamite
    replied
    Originally posted by Drile
    Google needs to start sending out checks to reimburse some webmasters for the amount of bandwidth they use up with their spiders.
    I have to agree on this one... 1.57 GB so far this month. I liked it better when they only came once a month. They keep spidering the same threads over and over, and half of them have probably never changed. Wouldn't be too bad if it were only one or two at a time, but the other day I had almost 60!!!

    Leave a comment:


  • Zachery
    replied
    thats what adsense is for ;D

    Leave a comment:


  • SaintDog
    replied
    Originally posted by Drile
    ....Google needs to start sending out checks to reimburse some webmasters for the amount of bandwidth they use up with their spiders.

    heh, that would be the icing on the cake right there

    Leave a comment:


  • Drile
    replied
    Originally posted by Scott MacVicar
    we can have up to 100 inktomi spiders here having a field day and when google starts at the start of the month the server actually lags a bit.
    In the end, it is definitely worth the server lag though for the results you get.

    Google needs to start sending out checks to reimburse some webmasters for the amount of bandwidth they use up with their spiders.

    I almost wish I had already upgraded with the amount of search engines hitting my sites the last few days.

    Leave a comment:


  • Scott MacVicar
    replied
    we can have up to 100 inktomi spiders here having a field day and when google starts at the start of the month the server actually lags a bit.

    Leave a comment:


  • f-a_org
    replied
    My board get's like 5 google spiders an hour, so it probably is a spider.

    Leave a comment:


  • Drile
    replied
    Ahhh, okay thanks.

    That clears up some real worries for me. That is also good news too. As soon as vB 3 goes to RC I am upgrading, which means I will get spidered quickly.

    Leave a comment:


  • UHN_ED
    replied
    It is a spider, pretty sure it is inktomi.

    Leave a comment:


  • Zachery
    replied
    there probally spiders, regardless if you want to belive it

    spiders cant still exsists even if your board isnt spider friendly, they at least try

    Leave a comment:


  • Drile
    started a topic What does this mean/What is this?

    What does this mean/What is this?

    Question.

    If I almost constantly have about ten unkown users on my forum that all share almost the same ip (changes occasionally, for now we will say it is 66.196.90.XX).

    What does this mean? I can't understand why there is almost always somewhere between 5-15 unknown users on with such similar ip's. And I don't think it could be a spider bot, as my forums are not search friendly right now, and there is NO way I am constantly being spidered.
Loading...
Working...
X