Announcement

Collapse
No announcement yet.

robots.txt not working

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • robots.txt not working

    Any other ways to block spiders?

    I've read that the no-follow tag doesn't work, either.

  • #2
    robots.txt works, only with honest search engine crawlers; same thing with the no-follow tag. There's nothing you can do to prevent those that doesn't follow your policies as they will always find their way in. If you do not wish to be index'ed for search, you should make it so that users are required to login in order to view your content.
    Best Regards,
    Andy Huang

    Comment


    • #3
      I was wondering why my robots.txt didn't work for a while.

      My robots.txt was:

      Code:
      User-agent: *
      Disallow: /admincp/
      Disallow: /cgi-bin/
      Disallow: /clientscript/
      ...etc
      when it should have been:

      Code:
      User-agent: *
      Disallow: /forum/admincp/
      Disallow: /cgi-bin/
      Disallow: /forum/clientscript/
      ....etc
      Hope this helps to someone.

      Jack Reacher

      Comment


      • #4
        Also remember that more likely than not search engines will only check these files every 2-3 weeks.

        Comment

        widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
        Working...
        X