Announcement

Collapse
No announcement yet.

robots.txt does not work

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • joesdomain
    replied
    ok thanks!

    Joe

    Leave a comment:


  • David Grove
    replied
    Looks like you're good to go. Remember not all robots obey it, and others only read the file periodically.

    Leave a comment:


  • joesdomain
    replied
    http://www.northernbuysell.com

    Leave a comment:


  • David Grove
    replied
    As Wayne states, this has nothing to do with vBulletin, per se. What is your URL?

    Leave a comment:


  • joesdomain
    replied
    the robots.txt validator states its correct.

    when i ftp into my site, the following directories are listed:

    .cp
    .session_data
    .tmp
    logs
    public_html

    my VBulletin is installed in the public_html directory as is the robots.txt
    Is this not correct?

    Joe

    Leave a comment:


  • Wayne Luke
    replied
    Robots.txt is not a vBulletin feature. It is an optional protocol that bots can choose to follow. We cannot make it work for you. I do know it needs to be in the root directory of your site, not the forum directory. A Webmaster related site like SitePoint.com could probably provide more assistance in their forums.

    If you see a stop sign, then they are "seeing" a no permission screen.

    Leave a comment:


  • David Grove
    replied
    See www.robotstxt.org, and make sure you implemented your robots.txt file correctly. I know the Googlebot only requests robots.txt once every 24 hours, so you have to let that time pass before it will obey the rules in robots.txt. Also, try validating your robots.txt file.

    Leave a comment:


  • joesdomain
    started a topic robots.txt does not work

    robots.txt does not work

    I created and added the robots.txt listing all files/directories to ignore. Well, I still get the msnbots/ yahoo slurp etc still visiting my site. I also banned their ip address and they still seem to say they are creating event. In whos online, i do see a stop sign icon.

    My website is a regional one and i do not want people worldwide registering

    Joe

Related Topics

Collapse

Working...
X