Announcement

Collapse
No announcement yet.

Yahoo Slurp Spiders...

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Yahoo Slurp Spiders...

    All of a sudden i have 60 of this visiting my forum at the same time.....normally i see about 10 a day.

    What would cause such a dramatic increase?

    And will it slaughter my bandwidth?

  • #2
    Do you have a robots.txt file? It enables you to control what SE bots can access.
    You're spending millions of dollars on a website?!

    Comment


    • #3
      no idea what a robots.txt is...or how to get one?

      Comment


      • #4
        A one-off mass visit won't hurt you much. If they keep doing it then yes, it might do. Google robots.txt for all the details and how to use them. I find spiders - not just Yahoo - generally arrive en masse once a month, and in an even bigger mass shortly after they discover your forum.

        Comment


        • #5
          I'm having the same problem with Yahoo. Fot the last few days I have had as many as 50 Yahoo robots indexing my forum.

          Interesting that Google appears to only use one robot at a time to index.

          I'm considering disallowing Yahoo robots.

          Comment


          • #6
            I've seen Google, MSN, Gigabot and Yahoo all make mass invasions. Yahoo seems to do it slightly more though, yes.

            Comment


            • #7
              I've also had the same in the past with Yahoo Slurp!

              A sudden invasion

              Comment


              • #8
                I currently have about 70 Yahoo! Slurp spiders on my site. Thy've been there for the last few days. How long can it take 70 spiders to crawl a site?
                My vBulletin: CADTutor - AutoCAD Forums

                Comment


                • #9
                  Originally posted by cfish View Post
                  I currently have about 70 Yahoo! Slurp spiders on my site. Thy've been there for the last few days. How long can it take 70 spiders to crawl a site?
                  Depends really, but it won't matter, before those spiders leave other Yahoo spider will come and start crawling..

                  Not a bad thing, if you want to get indexed on yahoo..
                  MCSE, MVP, CCIE
                  Microsoft Beta Team

                  Comment


                  • #10
                    I once had more than 300 of those spiders on my site.
                    selling kawaiiNation.com

                    Comment


                    • #11
                      Why would anyone want to keep yahoo off there site. You can't get search engine placement if they don't come and look at your site. If its eating up your bandwith you need to buy more.
                      Devilsown Water Injection
                      Grand prix forum

                      Comment


                      • #12
                        Originally posted by devilsown View Post
                        Why would anyone want to keep yahoo off there site. You can't get search engine placement if they don't come and look at your site. If its eating up your bandwith you need to buy more.
                        I'm not worried about the spiders being on my site, I'm just curious why there are so many of them and why they need to stick around for so long. I kind of imagined that these things were really efficient and could index a site in a few minutes. As I write, there are about 30 on my board and they've been there for most of the week!

                        It would be interesting to know a bit more about how these things operate.
                        My vBulletin: CADTutor - AutoCAD Forums

                        Comment


                        • #13
                          I see nothing wrong with blocking Yahoo. Nearly everybody visits my forum from Google.

                          I get about 35000 hits per day from Google search. Less than 3 per day from Yahoo. This is not counting spiders, this counts only visitors they send me.

                          If Yahoo spiders started hurting my bandwidth I would block them without a second thought.
                          OnlineConversion.com

                          Comment


                          • #14
                            I had 357 Yahoo spiders on my forum earlier!

                            Comment


                            • #15
                              I had over 500 recently, it's crazy. There doesn't seem to be an accepted reason other than it may be something to do with a change to Yahoo Overture (their PPC engine) or possibly looking at the fact the MSNBot has also been busy following the announcement that MSN Beta is now live and will be called Live and will phase out old MSN search.


                              Bob

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...
                              X