Announcement

Collapse
No announcement yet.

Robots.txt: Can You Give Us Disallow Updated List?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Alien
    replied
    Hey good idea... I'll take care of that.

    So with that done, I won't have to worry about my previous question then.

    Leave a comment:


  • David Grove
    replied
    Originally posted by Alien View Post
    I had a feeling.. So I guess I'd also need to duplicate content, for example..

    forums.sitename.com robots.txt = /posthistory.php
    www.sitename.com robots.txt = /forums/posthistory.php

    Is that right?
    You shouldn't have your forums available from both www.example.com/forums/ and forums.example.com Create a 301 permanent redirect from one to the other.

    Leave a comment:


  • Alien
    replied
    I had a feeling.. So I guess I'd also need to duplicate content, for example..

    forums.sitename.com robots.txt = /posthistory.php
    www.sitename.com robots.txt = /forums/posthistory.php

    Is that right?

    Leave a comment:


  • David Grove
    replied
    Originally posted by Alien View Post
    If I have a forum using the subdomain forums.website.com and a portal at www.website.com, do I have to use TWO robots.txt or do I still drop only one inside of public_html root?
    You need a robots.txt file in the root directory for each host name, this includes subdomains.

    Leave a comment:


  • Alien
    replied
    If I have a forum using the subdomain forums.website.com and a portal at www.website.com, do I have to use TWO robots.txt or do I still drop only one inside of public_html root?

    Leave a comment:


  • joesdomain
    replied
    majority are from out of the country. saudia arabia, netherlands, russia etc

    Joe

    Leave a comment:


  • Nick
    replied
    Originally posted by joesdomain View Post
    Hi:

    I was able to validate the file. However, the MSN Bot, Slurper etc are still visiting my site. I have also banned their IP address so i hope that fixes it. Its a full time job to keep these out.

    I have also banned alot of people also, on my site, i state that my site is for residents of northern ontario but i get people trying to register from all over.

    Joe
    How do you know that those from all over aren't wanting to research Ontario for whatever reason? Perhaps they are planning to move there?

    Leave a comment:


  • joesdomain
    replied
    Hi:

    I was able to validate the file. However, the MSN Bot, Slurper etc are still visiting my site. I have also banned their IP address so i hope that fixes it. Its a full time job to keep these out.

    I have also banned alot of people also, on my site, i state that my site is for residents of northern ontario but i get people trying to register from all over.

    Joe

    Leave a comment:


  • G3MM4
    replied
    Okay thanks, but it was kind of unnecessary to point it out to me, I shouldn't have to go to his website to look at it.

    Actually, he should read that thread because his robots.txt file looks okay to me. Hopefully he validated it, or he can disallow these bots individually.

    Leave a comment:


  • David Grove
    replied
    No, just pointing out how you can see the contents of his robots.txt file.

    Leave a comment:


  • G3MM4
    replied
    Thanks but it isn't me who has the issue.

    Leave a comment:


  • David Grove
    replied
    Originally posted by G3MM4 View Post
    Can you post the contents of your robots.txt here please? It might be a typo or something.
    See this: http://www.vbulletin.com/forum/showthread.php?p=1651382

    Leave a comment:


  • G3MM4
    replied
    Can you post the contents of your robots.txt here please? It might be a typo or something.

    Leave a comment:


  • joesdomain
    replied
    i uploaded it into my public_html directory and did name it robots.txt

    Joe

    Leave a comment:


  • G3MM4
    replied
    Where did you upload your robots.txt? And I presume you did name the file as robots.txt and not as robot.txt like some people do? It should take effect immediately as soon as a spider starts to crawl your site because one of the first things they look for is the robots.txt file - at least the legimate ones do. There are some bots/spiders that ignore the robots.txt.

    This might answer your questions: www.robotstxt.org

    Leave a comment:

Related Topics

Collapse

Working...
X