Announcement
Collapse
No announcement yet.
Perfect VBulletin 4 robots.txt
Collapse
X
-
-
This thread is a really good idea OP, thanks.
It would be great if vB had a built in robots.txt generator in the settings menu, it would be very straightforward to code and could use forum variables to automatically generate lists with the correct ./path/to/files - then the user would just need to copy paste the generated text.
Given that member profiles are headed in more of a social-media direction, i'm not sure I'd want to block the search engines from that - but it probably depends on the site. If people have filled out a profile with reasonable content then definately don't block it as its just adding to your backlinks.
Comment
-
tags.php
I would block tags.php ..if you have many many tags..i think it disrupts the search engine listings.----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Visit India's technology discussion forum at TechCentral.in
Comment
-
I would not recommend using this list. For a start the robots.txt file is exactly that 'a text file', anyone can read. You have told everyone where your admincp and modcp directories are located. I, for one rename mine - so no one knows the path to either of them. I certainly would not broadcast where my admincp or modcp directories are, some people even password protect these directories just to be that bit safer.
Originally posted by Jetlag View PostMy robots.txt
Code:User-agent: * Disallow: /admincp/ Disallow: /arcade/ Disallow: /archive/ Disallow: /cgi-bin/ Disallow: /clientscript/ Disallow: /cpstyles/ Disallow: /customavatars/ Disallow: /customgroupicons/ Disallow: /customprofilepics/ Disallow: /images/ Disallow: /includes/ Disallow: /modcp/ Disallow: /packages/ Disallow: /signaturepics/ Disallow: /vb/ Disallow: /ajax.php Disallow: /album.php Disallow: /announcement.php Disallow: /asset.php Disallow: /assetmanage.php Disallow: /attachment.php Disallow: /attachment_inlinemod.php Disallow: /blog_ajax.php Disallow: /blog_external.php Disallow: /blog_inlinemod.php Disallow: /blog_report.php Disallow: /blog_search.php Disallow: /blog_subscription.php Disallow: /blog_usercp.php Disallow: /calendar.php Disallow: /clear.gif Disallow: /converse.php Disallow: /cron.php Disallow: /editor.php Disallow: /editpost.php Disallow: /external.php Disallow: /faq.php Disallow: /global.php Disallow: /group_inlinemod.php Disallow: /groupsubscription.php Disallow: /image.php Disallow: /infraction.php Disallow: /inlinemod.php Disallow: /joinrequests.php Disallow: /list.php Disallow: /login.php Disallow: /member.php Disallow: /memberlist.php Disallow: /misc.php Disallow: /moderation.php Disallow: /moderator.php Disallow: /newattachment.php Disallow: /newreply.php Disallow: /newthread.php Disallow: /online.php Disallow: /payment_gateway.php Disallow: /payments.php Disallow: /picture.php Disallow: /picture_inlinemod.php Disallow: /picturecomment.php Disallow: /posthistory.php Disallow: /postings.php Disallow: /printthread.php Disallow: /private.php Disallow: /profile.php Disallow: /register.php Disallow: /report.php Disallow: /reputation.php Disallow: /search.php Disallow: /sendmessage.php Disallow: /sendtofriend.php Disallow: /showgroups.php Disallow: /showpost.php Disallow: /subscription.php Disallow: /threadrate.php Disallow: /threadtag.php Disallow: /usercp.php Disallow: /usernote.php Disallow: /visitormessage.php Disallow: /widget.php
Comment
-
I have /forum/search.php in my robots.txt file and Googld adsense tells me it cant access the file during crawls. I guess it works but I also noticed that 95% of the time when my forum is using the search.php file(like looking for new posts, etc) no relevant ads will be shown, only PSAs or irrelevant ads. Wondering if blocking the google crawler hurts relevant ad display during search.vBulletin 3.8.2
Comment
-
There isn't one, develop your own based on your needs.Jake Crause
www.homebuiltairplanes.com
Comment
-
Originally posted by JakeC View PostThis might be true, but for many of us who don't live and breath vB, something like a 95% solution and some guidance would be VERY much appreciated. Some of us don't know what our needs are!
Saw your website in your sig line. I just sold a Pitts S1-SS project to fund my current biz. I'll check out your site today.
Bart
Comment
-
Originally posted by ap0llo_* View PostIt covers every aspect for a successful VBulletin forum. It has all the necessary VB4 files disallowed and combats against high bandwidth bots.
Building accurate robots.txt files is my profession, so I thought I would give something back to the VBulletin community for all the help I've received in the past.
Here is mine if you want to compare, enjoy!
Comment
-
Originally posted by webducknj View Posti'm new to vBulletin but learning a lot every day. can someone explain the purpose of the file you're discussing in simple terms so i can attempt to wrap my pea brain around it?
many thanks,
bart
http://en.wikipedia.org/wiki/Robots.txt
Comment
Related Topics
Collapse
-
by Maik StorckHello,
I think it would be very nice if a standard robots.txt file could be included with vbulletin.
In this file, the basic rules should be included to prohibit all non-essential links...-
Channel: vBulletin 5 Suggestions
Sun 10th Feb '13, 4:07am -
Comment