Announcement

Collapse
No announcement yet.

Perfect VBulletin 4 robots.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Salome
    replied
    I could really use some feedback on this subject. My robots.txt file had a disallow for sitemap and it also, at the end of the .txt, had a hard return and then Sitemap: http://bellydanceforums.net/sitemap_index.xml.gz I deleted both those things, was that the right thing to do?

    I am having a repeated problems with adsense being able to crawl and show relevant ads. One thing was, I had locked several of my forums from guest view because I was not aware the adsense crawler was not compatible with the VB login and so forth. So I unlocked all my forums. But after a month and a half, I am still getting messages from adsense that they are being blocked... So I am wondering if the robots.txt file below is ok?

    User-agent: Mediapartners-Google

    Disallow: /editpost.php
    Disallow: /misc.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /register.php
    Disallow: /report.php
    Disallow: /search.php
    Disallow: /showgroups.php
    Disallow: /online.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /poll.php
    Disallow: /attachment.php
    Disallow: /avatar.php
    Disallow: /usercp.php
    Disallow: /vbseocp.php

    Leave a comment:


  • Loco.M
    commented on 's reply
    Ah.. I would add a few more files that don't need indexed, also I wouldn't limit the showpost file
    I don't run vb4 anymore so I don't have my old robots.txt file handy to check on.

  • stevectaylor
    replied
    any one know how to restrict yahoo paid spider?

    everytime we set a PPC they send a spider to every term at once 1000+ in one go!!

    Leave a comment:


  • Lord of Mirrors
    replied
    Originally posted by webducknj View Post
    i'm new to vBulletin but learning a lot every day. can someone explain the purpose of the file you're discussing in simple terms so i can attempt to wrap my pea brain around it?
    many thanks,
    bart
    Essentially it is to control bots, where they go, what they look at, when they crawl your site.

    http://en.wikipedia.org/wiki/Robots.txt

    Leave a comment:


  • Loco.M
    replied
    Originally posted by ap0llo_* View Post
    It covers every aspect for a successful VBulletin forum. It has all the necessary VB4 files disallowed and combats against high bandwidth bots.

    Building accurate robots.txt files is my profession, so I thought I would give something back to the VBulletin community for all the help I've received in the past.
    well I think your missing a few lines, also I would suggest adding your sitemap in it as well.
    Here is mine if you want to compare, enjoy!

    Leave a comment:


  • webducknj
    replied
    Originally posted by JakeC View Post
    This might be true, but for many of us who don't live and breath vB, something like a 95% solution and some guidance would be VERY much appreciated. Some of us don't know what our needs are!
    Jake
    Saw your website in your sig line. I just sold a Pitts S1-SS project to fund my current biz. I'll check out your site today.
    Bart

    Leave a comment:


  • webducknj
    replied
    i'm new to vBulletin but learning a lot every day. can someone explain the purpose of the file you're discussing in simple terms so i can attempt to wrap my pea brain around it?
    many thanks,
    bart

    Leave a comment:


  • JakeC
    replied
    There isn't one, develop your own based on your needs.
    This might be true, but for many of us who don't live and breath vB, something like a 95% solution and some guidance would be VERY much appreciated. Some of us don't know what our needs are!

    Leave a comment:


  • cmmguy
    replied
    I have /forum/search.php in my robots.txt file and Googld adsense tells me it cant access the file during crawls. I guess it works but I also noticed that 95% of the time when my forum is using the search.php file(like looking for new posts, etc) no relevant ads will be shown, only PSAs or irrelevant ads. Wondering if blocking the google crawler hurts relevant ad display during search.

    Leave a comment:


  • Zachery
    replied
    Originally posted by ampersand View Post
    so are we any closer to the "ultimate" robot.txt file ?
    There isn't one, develop your own based on your needs.

    Leave a comment:


  • ampersand
    replied
    so are we any closer to the "ultimate" robot.txt file ?

    Leave a comment:


  • syrus.xl
    replied
    I would not recommend using this list. For a start the robots.txt file is exactly that 'a text file', anyone can read. You have told everyone where your admincp and modcp directories are located. I, for one rename mine - so no one knows the path to either of them. I certainly would not broadcast where my admincp or modcp directories are, some people even password protect these directories just to be that bit safer.

    Originally posted by Jetlag View Post
    My robots.txt
    Code:
    User-agent: *
    Disallow: /admincp/
    Disallow: /arcade/
    Disallow: /archive/
    Disallow: /cgi-bin/
    Disallow: /clientscript/
    Disallow: /cpstyles/
    Disallow: /customavatars/
    Disallow: /customgroupicons/
    Disallow: /customprofilepics/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /modcp/
    Disallow: /packages/
    Disallow: /signaturepics/
    Disallow: /vb/
    Disallow: /ajax.php
    Disallow: /album.php
    Disallow: /announcement.php
    Disallow: /asset.php
    Disallow: /assetmanage.php
    Disallow: /attachment.php
    Disallow: /attachment_inlinemod.php
    Disallow: /blog_ajax.php
    Disallow: /blog_external.php
    Disallow: /blog_inlinemod.php
    Disallow: /blog_report.php
    Disallow: /blog_search.php
    Disallow: /blog_subscription.php
    Disallow: /blog_usercp.php
    Disallow: /calendar.php
    Disallow: /clear.gif
    Disallow: /converse.php
    Disallow: /cron.php
    Disallow: /editor.php
    Disallow: /editpost.php
    Disallow: /external.php
    Disallow: /faq.php
    Disallow: /global.php
    Disallow: /group_inlinemod.php
    Disallow: /groupsubscription.php
    Disallow: /image.php
    Disallow: /infraction.php
    Disallow: /inlinemod.php
    Disallow: /joinrequests.php
    Disallow: /list.php
    Disallow: /login.php
    Disallow: /member.php
    Disallow: /memberlist.php
    Disallow: /misc.php
    Disallow: /moderation.php
    Disallow: /moderator.php
    Disallow: /newattachment.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /payment_gateway.php
    Disallow: /payments.php
    Disallow: /picture.php
    Disallow: /picture_inlinemod.php
    Disallow: /picturecomment.php
    Disallow: /posthistory.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /profile.php
    Disallow: /register.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /search.php
    Disallow: /sendmessage.php
    Disallow: /sendtofriend.php
    Disallow: /showgroups.php
    Disallow: /showpost.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /threadtag.php
    Disallow: /usercp.php
    Disallow: /usernote.php
    Disallow: /visitormessage.php
    Disallow: /widget.php

    Leave a comment:


  • calvingarfield
    replied
    tags.php

    I would block tags.php ..if you have many many tags..i think it disrupts the search engine listings.

    Leave a comment:


  • NickyDee
    replied
    This thread is a really good idea OP, thanks.

    It would be great if vB had a built in robots.txt generator in the settings menu, it would be very straightforward to code and could use forum variables to automatically generate lists with the correct ./path/to/files - then the user would just need to copy paste the generated text.

    Given that member profiles are headed in more of a social-media direction, i'm not sure I'd want to block the search engines from that - but it probably depends on the site. If people have filled out a profile with reasonable content then definately don't block it as its just adding to your backlinks.

    Leave a comment:


  • jmvb
    replied
    What does this do?

    User-agent: Mediapartners-Google
    Allow: /member.php
    Allow: /private.php
    Allow: /usercp.php

    I have 3.8.x and I have adsense showing to guests and members below navbar.

    Thanks.

    Leave a comment:

Related Topics

Collapse

  • Maik Storck
    Include robots.txt
    by Maik Storck
    Hello,
    I think it would be very nice if a standard robots.txt file could be included with vbulletin.
    In this file, the basic rules should be included to prohibit all non-essential links...
    Sun 10th Feb '13, 4:07am
Working...
X