Announcement

Collapse
No announcement yet.

Spambots

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • [Forum] Spambots

    Hi Guys, should I be concerned by this spambot?,

    Google Spider 10:41 PM Modifying Private Message Folders 66.249.71.144



    Thanks Ace

  • #2
    It's a google bot not a spam bot. They crawl your site to index it on Google. If you see a red icon next to the IP of the search bot it means they don't have permission to do whatever it says they're trying to do.
    They are harmless. However if you want to tighten up what they can and can't look at, check your robot.txt file. If you don't have one you can Google the term and get some tuts that will guide you.
    ...

    Comment


    • #3
      Ah Thank you.

      Comment


      • #4
        The basic content of a robots.txt for vb:
        User-agent: *
        Disallow: /forum/printthread.php
        Disallow: /forum/subscription.php
        Disallow: /forum/payments.php
        Disallow: /forum/profile.php
        Disallow: /forum/faq.php
        Disallow: /forum/calendar.php
        Disallow: /forum/private.php
        Disallow: /forum/sendmessage.php
        Disallow: /forum/sendmessage.php?do=
        Disallow: /forum/showgroups.php
        Disallow: /forum/reputation.php
        Disallow: /forum/report.php
        Disallow: /forum/threadrate.php
        Disallow: /forum/postings.php
        Disallow: /forum/newthread.php
        Disallow: /forum/search.php
        Disallow: /forum/newreply.php
        Disallow: /forum/register.php
        Disallow: /forum/login.php
        Disallow: /forum/faq.php
        Disallow: /forum/image.php
        Disallow: /forum/cron.php
        Disallow: /forum/joinrequests.php
        Disallow: /forum/usercp.php

        User-agent: Slurp
        Crawl-delay: 1500

        User-agent: *
        Disallow: /admin/
        Disallow: /contrib/
        Disallow: /doc/
        Disallow: /lib/
        Disallow: /modules/
        Disallow: /plugins/
        Disallow: /scripts/
        Disallow: /tmp/

        Comment


        • #5
          The robots.txt will stop Google and other well-behaved bots from indexing that content but it won't necessarily stop them from crawling the content looking for other links to follow.

          Basically, it's a non-issue that requires no intervention. Spiders will attempt to follow links - ALL links. Some ofn those are deadends with "you don't have permission..." messages. That's as it should be. vBulletin working as designed.
          Psychlinks Web Services Affordable Web Design & Site Management
          Specializing in Small Businesses and vBulletin/Xenforo Forums

          Comment


          • #6
            How many google bots you have? I have about 160 online right now. Is it normal if you have a big site?

            Comment


            • #7
              Originally posted by sebbo View Post
              How many google bots you have? I have about 160 online right now. Is it normal if you have a big site?
              Yes.

              Comment


              • #8
                Originally posted by sebbo View Post
                How many google bots you have? I have about 160 online right now. Is it normal if you have a big site?
                Are not they "Baidu" ?


                vB5 is unequivocally the best forum software, but not yet...

                Comment


                • #9
                  I gotta disagree that 160 bots all from the same outfit at the same time is 'normal'. This sounds like one of those mysterious 'guest attacks' we've seen intermittently. But when it happens to us they're all from different sites, so I guess not. Still, any one site really only needs one spider to crawl one site.

                  Right now our site is showing only two Google Adsense spiders, but 12 Bing spiders. One Yandex and one Baidu.

                  Comment


                  • #10
                    What is your session time out? I change it to 60 and now i have 21 spiders ( 3 x Google AdSense Spider and 18 x Google Spider). Those spiders have 20 different IP addresses. My normal session time out is 3600. With that i have 71 AdSense Spiders and 66 Google Spiders. Those come from 70 different IP´s.

                    It is a little bit embarrassing that i have many guest but almost all of those are spiders.

                    Comment


                    • #11
                      Me? Our timeout has been 900 (15 minutes) for ages. I can't imagine lowering it to 1 minute - nobody could post!?

                      Comment


                      • #12
                        This will help you control them and bring down the load...etc on your server http://www.vbulletin.org/forum/showthread.php?t=268208
                        Kind regards,
                        Simon
                        Microsoft Office Discussion

                        Comment

                        widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                        Working...
                        X