Announcement

Collapse
No announcement yet.

The inktomisearch spider thread

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • The inktomisearch spider thread

    And a right pain it is too. Last night I had 35 guests online and five members. Strange I thought and did a quick who's online. Well the 35 guests were inktomisearch spiders see image below. Seems it ignores the robots.txt file too. My members quote it being on my site for over 5 hours, and I just noted that these forums are also being hit right now by the sod.
    Attached Files

  • #2
    Not new, just annoying
    Running vB since 4-14-2002

    Comment


    • #3
      I think there is a html head tag you can add to turn away this spider. Search for one of the many other threads about this.

      Comment


      • #4
        What does he do anyways?
        Running vB since 4-14-2002

        Comment


        • #5
          She (?) indexes your site for the inktomi based search engines (http://www.inktomi.com/customers/). Inktomi was purchased by Yahoo! earlyer this year, I (along with others), expect yahoo to move from google to Ink by the end of this year. You really do not want to block this spider from your sites!

          My personal spider is this one:
          Attached Files
          Bike Forums.net

          Comment


          • #6
            Originally posted by Jakeman
            I think there is a html head tag you can add to turn away this spider. Search for one of the many other threads about this.
            Yes:
            HTML Code:
            <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
            But if it's ignoring robots.txt, it will probably ignore the meta tag too.

            Comment


            • #7
              From all the people that complain, does anybody actually know how much traffic the spider is pulling ?

              Comment


              • #8
                Or how much taffic it is costing us?
                Running vB since 4-14-2002

                Comment


                • #9
                  Originally posted by IDN
                  Or how much taffic it is costing us?
                  Well you have to weigh in how much bandwidth you have, how much you use (on average per month), and how much you're willing to set aside for that type of thing.

                  Comment


                  • #10
                    Sorry, meant, traffic.

                    bandwidth is the site of the pipe, traffic is how much goes through the pipe.

                    if it visites every page on the site with mutiple spiders, must be costing us alot
                    Running vB since 4-14-2002

                    Comment


                    • #11
                      My site is still to be hit by them lol...

                      When they come, I'm not worried

                      Satan

                      Comment


                      • #12
                        Search Engine Spiders...

                        Whats been everyones experience with them?? I have had the Inktomi spider on my side non stop for the last 2 days, it just doesnt give up. When ever a new thread is made its like its starts over again and goes through them all. Very wierd, so far the highest number of spiders has been 14 at one time. Just curious what everyone else has seen on their boards...

                        Comment


                        • #13
                          Its been on my site for 2 days now (Inktomi) i think this spider crawls really slow :/

                          Comment


                          • #14
                            It was on my site for two weeks, and my forums don't allow guests to access them. It must like "no permission" pages. It's a retarded spider.

                            Comment


                            • #15
                              I've had 45 google spiders on at once one day and I have loads of inkitomis come on, but they have been quiet over past couple of days.

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...
                              X