robots.txt help please

[BarryoBrien]

What do I need to block in the robots.txt file example cgi.bin what else?

[Jose Amaral Rego]

http://www.robotstxt.org/wc/faq.html
http://www.searchengineworld.com/rob...s_tutorial.htm
http://help.yahoo.com/help/us/ysearc.../slurp-03.html

You can enter this in templates with <meta> tags, but not all robots read or care to follow <meta> tags or robot.txt. Banning those that do not follow is another option.

<meta name="robots" content="All">
<meta name="revisit-after" content="30 days">

User-agent: *
Disallow: /attachment.php
Disallow: /avatar.php
Disallow: /editpost.php
Disallow: /member.php
[strike_temp]Disallow: /member2.php[/strike_temp]
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /report.php
Disallow: /search.php
Disallow: /sendtofriend.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /admincp/
Disallow: /modcp/
Disallow: /images/
Disallow: /sendmessage.php
Disallow: /register.php
Disallow: /subscription.php

*This is not uptodate

[BarryoBrien]

thats great thanks

[BarryoBrien]

what does this block? /postings.php/ does this mean the posts are blocked?

[Jose Amaral Rego]

Just by looking into the file postings.php it had to do with some thread admin functions to post.

[MRGTB]

Does anybody know exactly what should and should not be blocked in your robots.txt file for vBulletin vB.3.6

To get the best SEO out of your site.

I mean every files and folder that should be dis-allowed.