Announcement

Collapse
No announcement yet.

help with robots.txt file - keeping spiders completely out?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • help with robots.txt file - keeping spiders completely out?

    My robots.txt file currently contains this:

    User-agent: *
    Disallow: /forums/

    which keeps spiders out of my main boards, posts and threads - but i find that other things like the memberlist, the calendar, the search etc, will show up when you search google. Doesn't using /forums/ mean EVERYTHING in the subdirectory forums should be hidden? Has the spider process changed? Do I need to list each php function file like memberlist.php and calendar.php now too?

    Any ideas?

  • #2
    User-agent: *
    Disallow: /

    Comment


    • #3
      yeah, but...

      Doesn't that disallow EVERYTHING?

      I have a main site that can stay in the search engines, but the message board is private, and I dont want any part of the message board showing up in the search engines...

      Comment


      • #4
        Originally posted by jilly
        I dont want any part of the message board showing up in the search engines...
        Your robots.txt file looks fine to me if that's what your wanting to do.

        Comment


        • #5
          Once your results are in Google, you've generally got very little control over things. The robots.txt you provided will only prevent spiders from indexing more pages.. On several of my sites, I've got results that haven't existed for years, and have been blocked by robots.txt for awhile.

          Matt Cutts, of google, recently quipped something to the effect of "eh, who cares if you've got spurious results showing up in google? You're getting traffic!"

          There's basically no way to get your results de-listed from google.

          Comment


          • #6
            ugh...

            thanks falter.. It's annoying because my board is a private one, and I put that robots.txt file up to make sure it WASNT in google.. dont want the boards in there at all.. Yet they partially are. However I have tightened up viewing so that guests can't view anything, calenday, arcade, anything on the navbar, etc. This will help in the future, i hope..

            Comment

            widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
            Working...
            X