Perfect VBulletin 4 robots.txt

[Zachery]

I wouldn't block image.php, or faq.php, or announcement.php. But thats me.

[Jetlag]

I wouldn't block image.php, or faq.php, or announcement.php. But thats me.

Not used for guess on my sites except faq.php so if you want guess/bots to see it remove them from robots.txt

[jmvb]

What does this do?

User-agent: Mediapartners-Google
Allow: /member.php
Allow: /private.php
Allow: /usercp.php

I have 3.8.x and I have adsense showing to guests and members below navbar.

Thanks.

[NickyDee]

This thread is a really good idea OP, thanks.

It would be great if vB had a built in robots.txt generator in the settings menu, it would be very straightforward to code and could use forum variables to automatically generate lists with the correct ./path/to/files - then the user would just need to copy paste the generated text.

Given that member profiles are headed in more of a social-media direction, i'm not sure I'd want to block the search engines from that - but it probably depends on the site. If people have filled out a profile with reasonable content then definately don't block it as its just adding to your backlinks.

[calvingarfield]

tags.php

I would block tags.php ..if you have many many tags..i think it disrupts the search engine listings.

[syrus.xl]

I would not recommend using this list. For a start the robots.txt file is exactly that 'a text file', anyone can read. You have told everyone where your admincp and modcp directories are located. I, for one rename mine - so no one knows the path to either of them. I certainly would not broadcast where my admincp or modcp directories are, some people even password protect these directories just to be that bit safer.

My robots.txt

User-agent: *
Disallow: /admincp/
Disallow: /arcade/
Disallow: /archive/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customgroupicons/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /includes/
Disallow: /modcp/
Disallow: /packages/
Disallow: /signaturepics/
Disallow: /vb/
Disallow: /ajax.php
Disallow: /album.php
Disallow: /announcement.php
Disallow: /asset.php
Disallow: /assetmanage.php
Disallow: /attachment.php
Disallow: /attachment_inlinemod.php
Disallow: /blog_ajax.php
Disallow: /blog_external.php
Disallow: /blog_inlinemod.php
Disallow: /blog_report.php
Disallow: /blog_search.php
Disallow: /blog_subscription.php
Disallow: /blog_usercp.php
Disallow: /calendar.php
Disallow: /clear.gif
Disallow: /converse.php
Disallow: /cron.php
Disallow: /editor.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /global.php
Disallow: /group_inlinemod.php
Disallow: /groupsubscription.php
Disallow: /image.php
Disallow: /infraction.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /list.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderation.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /picture.php
Disallow: /picture_inlinemod.php
Disallow: /picturecomment.php
Disallow: /posthistory.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /sendtofriend.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /threadtag.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /visitormessage.php
Disallow: /widget.php

[ampersand]

so are we any closer to the "ultimate" robot.txt file ?

[Zachery]

so are we any closer to the "ultimate" robot.txt file ?

There isn't one, develop your own based on your needs.

[cmmguy]

I have /forum/search.php in my robots.txt file and Googld adsense tells me it cant access the file during crawls. I guess it works but I also noticed that 95% of the time when my forum is using the search.php file(like looking for new posts, etc) no relevant ads will be shown, only PSAs or irrelevant ads. Wondering if blocking the google crawler hurts relevant ad display during search.

[JakeC]

There isn't one, develop your own based on your needs.

This might be true, but for many of us who don't live and breath vB, something like a 95% solution and some guidance would be VERY much appreciated. Some of us don't know what our needs are!

[webducknj]

i'm new to vBulletin but learning a lot every day. can someone explain the purpose of the file you're discussing in simple terms so i can attempt to wrap my pea brain around it?
many thanks,
bart

[webducknj]

This might be true, but for many of us who don't live and breath vB, something like a 95% solution and some guidance would be VERY much appreciated. Some of us don't know what our needs are!

Jake
Saw your website in your sig line. I just sold a Pitts S1-SS project to fund my current biz. I'll check out your site today.
Bart

[Loco.M]

It covers every aspect for a successful VBulletin forum. It has all the necessary VB4 files disallowed and combats against high bandwidth bots.

Building accurate robots.txt files is my profession, so I thought I would give something back to the VBulletin community for all the help I've received in the past.

well I think your missing a few lines, also I would suggest adding your sitemap in it as well.
Here is mine if you want to compare, enjoy!

[Lord of Mirrors]

i'm new to vBulletin but learning a lot every day. can someone explain the purpose of the file you're discussing in simple terms so i can attempt to wrap my pea brain around it?
many thanks,
bart

Essentially it is to control bots, where they go, what they look at, when they crawl your site.

http://en.wikipedia.org/wiki/Robots.txt

[stevectaylor]

any one know how to restrict yahoo paid spider?

everytime we set a PPC they send a spider to every term at once 1000+ in one go!!