Mod Rewrite Friendly URL's & robots.txt

[too_cool_3]

I just made my own robots.txt file from a combination of Wayne Luke (link) from vbulletin.com, Brandon Sheley (link) from vbulletinsetup.com, and Brian Cummiskey(link) from vbseo.com

I'm...

[jamunkala]

hello,

I have put this into robots.txt but I am not sure if it will work for seo friendly urls

User-agent: *
Disallow: /ajax.php
Disallow: /archive/
Disallow:

...

[AusPhotography]

G'Day
Assuming I've setup the XML sitemap do I need to do anything more than just put a line in robots.txt to get the bots to pick it up? ...

Sitemap: http://www.ausphotography.net

...

[StephenKay]

My site was suspended yesterday by my shared hosting provider for using too much resources.

So, I am attempting to stop all robots from indexing the site at least temporarily while I work...

[ap0llo_*]

Hi,

Here's an exemplar robots.txt file for a vB4 Forum installation. It has been designed to combat / limit high bandwidth using bots and has the correct files disallowed / allowed....

[cheefoo]

Hi,

May I know what is the best practices to place inside robots.txt to suit VB 4.0 ?
Are these enough for the new features such as CMS and Blog ? Please kindly advice.
...

[WiseOne38221]

Hi, with this new 3.7 I want to make sure I am disallowing everything I should. I found ONE more to block: /posthistory.php (search engines won't like that)

Here is what I have now:
...