Announcement

Collapse
No announcement yet.

Question about robots.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • [Forum] Question about robots.txt

    I have a robots.txt file in the root directory of my forum. Among other things it has this bit in it:

    User-agent: Baiduspider
    Disallow: /

    However, when I look at who's online for the forum I can see the Baidu spider apparently looking at things?

    Have I got something wrong somewhere?

    Thanks.

  • #2
    No.. Baidu is a bad spider and will take about a week to recognize your robots.txt file. Or after it indexes your site for the Chinese site.
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud demonstration site.
    vBulletin 5 API

    Comment


    • #3
      Originally posted by Wayne Luke View Post
      No.. Baidu is a bad spider and will take about a week to recognize your robots.txt file. Or after it indexes your site for the Chinese site.
      Bugger, any other way of blocking the thing? Does it use a standard IP range I can block, or something like that?

      Comment


      • #4
        I just blocked 180.76.*

        We'll see how that goes. I'm highly unlikely to get Chinese users on my forum so no problem with blocking that range.

        Comment


        • #5
          OK, looking good so far, no bad Baidu since the block.

          Comment

          widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
          Working...
          X