No announcement yet.

How do I block search engine spiders?

  • Filter
  • Time
  • Show
Clear All
new posts

  • How do I block search engine spiders?

    Right now on my forums, there are 31 "guests", all of which has the similar IP addresses. I am assuming that they are all search engine spiders or something like that. It's been getting worse and worse every day. This is the most so far.

    If I ban the IP address in the CP, won't that just ban that IP from registering? How do I ban access to my site from these annoying little buggers?

  • #2
    Use htaccess.

    .htaccess guide
    Steve Machol, former vBulletin Customer Support Manager (and NOT retired!)
    Change CKEditor Colors to Match Style (for 4.1.4 and above)

    Steve Machol Photography

    Mankind is the only creature smart enough to know its own history, and dumb enough to ignore it.


    • #3
      thanks Steve!


      • #4
        Is there a way to just deny access to all directories/files that no one needs to get to? Does anyone have a good example of a vB .htaccess file I can look at? You know...just to use as a template, until I know what I'm doing.
        The Grey Area - Tweaking Obsession


        • #5
          Put a file called robots.txt in your root directory, all the major search engines, will look for this file and abide by it.

          Put something like this in it.

          User-agent: *
          Disallow: attachment.php
          Disallow: avatar.php
          Disallow: editpost.php
          Disallow: member.php
          Disallow: member2.php
          Disallow: misc.php
          Disallow: moderator.php
          Disallow: newreply.php
          Disallow: newthread.php
          Disallow: online.php
          Disallow: poll.php
          Disallow: postings.php
          Disallow: printthread.php
          Disallow: private.php
          Disallow: private2.php
          Disallow: report.php
          Disallow: search.php
          Disallow: sendtofriend.php
          Disallow: threadrate.php
          Disallow: usercp.php
          Disallow: /admin/
          Disallow: /images/
          Disallow: /mod/
          Disallow: /attachment
          Disallow: /showthread


          widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.