Announcement

**joesdomain** · Tue 11 Nov '08, 5:31pm

i uploaded it into my public_html directory and did name it robots.txt

Joe

**G3MM4** · Wed 12 Nov '08, 9:56am

Can you post the contents of your robots.txt here please? It might be a typo or something.

**David Grove** · Wed 12 Nov '08, 10:37am

Originally posted by G3MM4 View Post

Can you post the contents of your robots.txt here please? It might be a typo or something.

See this: http://www.vbulletin.com/forum/showthread.php?p=1651382

**G3MM4** · Wed 12 Nov '08, 4:17pm

Thanks but it isn't me who has the issue.

**David Grove** · Wed 12 Nov '08, 6:15pm

No, just pointing out how you can see the contents of his robots.txt file.

**G3MM4** · Wed 12 Nov '08, 6:31pm

Okay thanks, but it was kind of unnecessary to point it out to me, I shouldn't have to go to his website to look at it.

Actually, he should read that thread because his robots.txt file looks okay to me. Hopefully he validated it, or he can disallow these bots individually.

**joesdomain** · Fri 14 Nov '08, 2:23pm

Hi:

I was able to validate the file. However, the MSN Bot, Slurper etc are still visiting my site. I have also banned their IP address so i hope that fixes it. Its a full time job to keep these out.

I have also banned alot of people also, on my site, i state that my site is for residents of northern ontario but i get people trying to register from all over.

Joe

**Nick** · Fri 14 Nov '08, 2:40pm

Originally posted by joesdomain View Post

Hi:

I was able to validate the file. However, the MSN Bot, Slurper etc are still visiting my site. I have also banned their IP address so i hope that fixes it. Its a full time job to keep these out.

I have also banned alot of people also, on my site, i state that my site is for residents of northern ontario but i get people trying to register from all over.

Joe

How do you know that those from all over aren't wanting to research Ontario for whatever reason? Perhaps they are planning to move there?

**joesdomain** · Fri 14 Nov '08, 2:42pm

majority are from out of the country. saudia arabia, netherlands, russia etc

Joe

**Alien** · Tue 25 Nov '08, 2:43pm

If I have a forum using the subdomain forums.website.com and a portal at www.website.com, do I have to use TWO robots.txt or do I still drop only one inside of public_html root?

**David Grove** · Tue 25 Nov '08, 3:00pm

Originally posted by Alien View Post

If I have a forum using the subdomain forums.website.com and a portal at www.website.com, do I have to use TWO robots.txt or do I still drop only one inside of public_html root?

You need a robots.txt file in the root directory for each host name, this includes subdomains.

**Alien** · Tue 25 Nov '08, 3:08pm

I had a feeling.. So I guess I'd also need to duplicate content, for example..

forums.sitename.com robots.txt = /posthistory.php
www.sitename.com robots.txt = /forums/posthistory.php

Is that right?

**David Grove** · Tue 25 Nov '08, 3:11pm

Originally posted by Alien View Post

I had a feeling.. So I guess I'd also need to duplicate content, for example..

forums.sitename.com robots.txt = /posthistory.php
www.sitename.com robots.txt = /forums/posthistory.php

Is that right?

You shouldn't have your forums available from both www.example.com/forums/ and forums.example.com Create a 301 permanent redirect from one to the other.

**Alien** · Thu 27 Nov '08, 3:28pm

Hey good idea... I'll take care of that.

So with that done, I won't have to worry about my previous question then.

Announcement

Robots.txt: Can You Give Us Disallow Updated List?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Related Topics