Announcement

Collapse
No announcement yet.

robots.txt does not work

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • robots.txt does not work

    I created and added the robots.txt listing all files/directories to ignore. Well, I still get the msnbots/ yahoo slurp etc still visiting my site. I also banned their ip address and they still seem to say they are creating event. In whos online, i do see a stop sign icon.

    My website is a regional one and i do not want people worldwide registering

    Joe

  • #2
    See www.robotstxt.org, and make sure you implemented your robots.txt file correctly. I know the Googlebot only requests robots.txt once every 24 hours, so you have to let that time pass before it will obey the rules in robots.txt. Also, try validating your robots.txt file.
    ~~~~~

    Comment


    • #3
      Robots.txt is not a vBulletin feature. It is an optional protocol that bots can choose to follow. We cannot make it work for you. I do know it needs to be in the root directory of your site, not the forum directory. A Webmaster related site like SitePoint.com could probably provide more assistance in their forums.

      If you see a stop sign, then they are "seeing" a no permission screen.
      Translations provided by Google.

      Wayne Luke
      The Rabid Badger - a vBulletin Cloud demonstration site.
      vBulletin 5 API

      Comment


      • #4
        the robots.txt validator states its correct.

        when i ftp into my site, the following directories are listed:

        .cp
        .session_data
        .tmp
        logs
        public_html

        my VBulletin is installed in the public_html directory as is the robots.txt
        Is this not correct?

        Joe

        Comment


        • #5
          As Wayne states, this has nothing to do with vBulletin, per se. What is your URL?
          ~~~~~

          Comment


          • #6
            http://www.northernbuysell.com

            Comment


            • #7
              Looks like you're good to go. Remember not all robots obey it, and others only read the file periodically.
              ~~~~~

              Comment


              • #8
                ok thanks!

                Joe

                Comment

                Related Topics

                Collapse

                Working...
                X