Announcement

Collapse
No announcement yet.

Vbulletin archive SEO

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Vbulletin archive SEO

    Short and Simple:

    I don't want duplicate content penalties from the search engines and I want the best possible SERPs...

    Should I disallow all posts in robots.txt EXCEPT for the archives? Or is it okay to let the bots spider both the main forums and archives? Thanks.

  • #2
    Some spiders will index regular posts, some will not. For consistency, it would probably be better to direct all robots to the archive.

    Comment


    • #3
      how?

      Comment


      • #4
        The author of this thread is using a robots file. See this page for a tutorial (I pulled it from Google).

        Comment


        • #5
          already read that, doesn't say how to send all spiders to one directory

          Comment


          • #6
            Originally posted by Gary Bolton
            already read that, doesn't say how to send all spiders to one directory
            It's not a redirect or anything. You just disallow access to everything but the archive.

            Comment


            • #7
              so what do think of this bearing in mind I still want bots to index threads and also still allow google adsence on the site. Feel free to point out mistakes!!

              Code:
              User-agent: *
              Disallow: /cgi-bin/
              Disallow: /vb/admincp/
              Disallow: /vb/clientscript/
              Disallow: /vb/cpstyles/
              Disallow: /vb/customavatars/
              Disallow: /vb/images/
              Disallow: /vb/includes/
              Disallow: /vb/install/
              Disallow: /vb/modcp/
              Disallow: /vb/modules/
              Disallow: /vb/subscriptions/
              Disallow: /vb/calendar.php
              Disallow: /vb/clear.gif
              Disallow: /vb/cron.php
              Disallow: /vb/editpost.php
              Disallow: /vb/external.php
              Disallow: /vb/global.php
              Disallow: /vb/image.php
              Disallow: /vb/joinrequests.php
              Disallow: /vb/login.php
              Disallow: /vb/member.php
              Disallow: /vb/memberlist.php
              Disallow: /vb/misc.php
              Disallow: /vb/moderator.php
              Disallow: /vb/newattachment.php
              Disallow: /vb/newreply.php
              Disallow: /vb/online.php
              Disallow: /vb/private.php
              Disallow: /vb/profile.php
              Disallow: /vb/register.php
              Disallow: /vb/report.php
              Disallow: /vb/reputation.php
              Disallow: /vb/sendmessage.php
              Disallow: /vb/showgroups.php
              Disallow: /vb/subscription.php
              Disallow: /vb/subscriptions.php
              Disallow: /vb/threadrate.php
              Disallow: /vb/usercp.php
              Disallow: /vb/usernote.php

              Comment


              • #8
                It looks good at first glance. I have never used robot files though.

                Comment


                • #9
                  I ran it through a validator here and was fine, just wasn't sure about some of the files I have disallowed.

                  http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

                  Comment

                  widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                  Working...
                  X