Announcement

Collapse
No announcement yet.

limit number of yahoo bots that can be on my site at one time?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • limit number of yahoo bots that can be on my site at one time?

    for some reason the yahoo bots are eating up my site. i don't obviously want to ban them all, but is there any way to limit them to like 2-3 bots at a time?
    Nick Humphrey
    My experimental music, art, poetry and pictures.
    My album Classic E B-sides can be purchased on both itunes and amazon.com.
    My phrasebook of world languages.
    My blog.

  • #2
    No, but they shouldn't be there for long, give thema day or two.

    Comment


    • #3
      they have been there for weeks without stop. they keep coming back ALL DAY LONG. i am just worried it is them that are causing my too many connections errors i seem to be getting. maybe it is just my host, not sure.
      Nick Humphrey
      My experimental music, art, poetry and pictures.
      My album Classic E B-sides can be purchased on both itunes and amazon.com.
      My phrasebook of world languages.
      My blog.

      Comment


      • #4
        Are they from the same ip address, or do they appear to be unique? And what is your cookie timeout setting? The default 900 or something custom?

        Comment


        • #5
          i just saw that i had 9 yahoo slurp bots on my site just now at one time. they are all unique. where is the default cookie timeout setting? in admincp?
          Last edited by Nickleus; Tue 11 Oct '05, 5:44am.
          Nick Humphrey
          My experimental music, art, poetry and pictures.
          My album Classic E B-sides can be purchased on both itunes and amazon.com.
          My phrasebook of world languages.
          My blog.

          Comment


          • #6
            If you don't know you probably didn't change it

            AdminCP > vB Options > vB Options > Cookie settings.

            Comment


            • #7
              Yahoo spiders

              I had the same problem with Yahoo spiders; several dozen on at any given time.
              I did two things:

              - Use "rel" tags to request spiders not follow certain links that you don't want them to visit, like user profile, etc). There are multiple threads about that here or at vbulletin.org. I don't remember where I got the details.

              - Modify your robots.txt to slow down the crawling like:
              User-agent: Slurp
              Crawl-delay: 30
              http://help.yahoo.com/help/us/ysearc.../slurp-03.html

              Comment


              • #8
                I have about 100 of these things and I'm pretty sure they are there all day long too. Anyone point me to a thread with a way to cut down these yahoo slurp bots?

                Comment


                • #9
                  Just came across this post.

                  Exactly what I was looking for!

                  We have sometimes 100+ Yahoo bots at any given time and it has been causing some trouble.

                  Thanks.

                  Comment


                  • #10
                    It would be nice if there was an ability to limit amount or which spiders can be on, also have them spider crawling for a certain time then close them out for the rest of the time.

                    Comment

                    widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                    Working...
                    X