Announcement

Collapse
No announcement yet.

Mod Rewrite Friendly URL's & robots.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • [Forum] Mod Rewrite Friendly URL's & robots.txt

    I just made my own robots.txt file from a combination of Wayne Luke (link) from vbulletin.com, Brandon Sheley (link) from vbulletinsetup.com, and Brian Cummiskey(link) from vbseo.com

    I'm using vB 4.1.7 installed in my root with Automatic Sitemap Generation, my Sitemap File Path stated in AdminCP is './store_sitemap' however, when I click to view my sitemap URL in AdminCP it shows me this link: http://tbmotoring.com/xmlsitemap.php

    1.) Is http://tbmotoring.com/xmlsitemap.php the correct sitemap link to put in my robots.txt file? And do I need to add '.gz' to the end (http://tbmotoring.com/xmlsitemap.php.gz)?

    2.) I have Mod Rewrite Friendly URL's enabled, does that mean I need to remove the all the '.php' extensions in my robots.txt? (ie. instead of Disallow: /ajax.php should I put Disallow: /ajax)

    PHP Code:
    User-agentia_archiver
    Allow
    : /
     
    User-agentBaiduspider
    Disallow
    : /
     
    User-agentBoardTracker
    Disallow
    : /
     
    User-agentGigabot
    Disallow
    : /
     
    User-agentTwiceler
    Disallow
    : /
     
    User-agentSlurp
    Crawl
    -delay2
     
    User
    -agentmsnbot
    Crawl
    -delay2
     
    User
    -agent: *
    Disallow: *.js
    Disallow
    : /ajax.php
    Disallow
    : /attachment.php
    Disallow
    : /calendar.php
    Disallow
    : /clientscript/
    Disallow: /cpstyles/
    Disallow: /cron.php
    Disallow
    : /customavatars/
    Disallow: /customprofilepics/
    Disallow: /editpost.php
    Disallow
    : /global.php
    Disallow
    : /includes/
    Disallow: /inlinemod.php
    Disallow
    : /install/
    Disallow: /joinrequests.php
    Disallow
    : /login.php
    Disallow
    : /member.php
    Disallow
    : /memberlist.php
    Disallow
    : /misc.php
    Disallow
    : /moderator.php
    Disallow
    : /newattachment.php
    Disallow
    : /newreply.php
    Disallow
    : /newthread.php
    Disallow
    : /online.php
    Disallow
    : /poll.php
    Disallow
    : /post.php
    Disallow
    : /postings.php
    Disallow
    : /printthread.php
    Disallow
    : /private.php
    Disallow
    : /profile.php
    Disallow
    : /register.php
    Disallow
    : /report.php
    Disallow
    : /reputation.php
    Disallow
    : /search.php
    Disallow
    : /sendmessage.php
    Disallow
    : /showgroups.php
    Disallow
    : /showpost.php
    Disallow
    : /subscription.php
    Disallow
    : /threadrate.php
    Disallow
    : /usercp.php
    Disallow
    : /usernote.php
     
    Sitemap
    http://tbmotoring.com/xmlsitemap.php 
    Feel free to give me your feedback on my robots.txt file - if you think anything needs to be added/removed/modified ?

    Thank you,
    -Marc
    Last edited by too_cool_3; Wed 2nd Nov '11, 7:09pm. Reason: clarified

  • #2
    1) You shouldn't like Robots.txt anywhere. Just put it in your root directory.

    2) Shouldn't matter. The Friendly URLs only redirect things you want to be indexed anyway.
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud customization and demonstration site.
    vBulletin 5 Documentation - Updated every Friday. Report issues here.
    vBulletin 5 API - Full / Mobile
    I am not currently available for vB Messenger Chats.

    Comment


    • #3
      Pardon me Wayne, I don't understand what you mean by your response for 1)

      Comment


      • #4
        sitemap does not have php at the end put what u see

        Comment


        • #5
          Originally posted by too_cool_3 View Post
          Pardon me Wayne, I don't understand what you mean by your response for 1)
          Robots.txt has nothing to do with any kind of sitemap.

          Robots.txt is not linked from anywhere in vBulletin. It is just a file that bots will request automatically. Just put the file in your root directory and forget about it.

          I don't know what the response above me is about because the xmlsitemap.php file does indeed have a php extension.
          Translations provided by Google.

          Wayne Luke
          The Rabid Badger - a vBulletin Cloud customization and demonstration site.
          vBulletin 5 Documentation - Updated every Friday. Report issues here.
          vBulletin 5 API - Full / Mobile
          I am not currently available for vB Messenger Chats.

          Comment


          • #6
            To clarify, Wayne, I read in other threads that you should put a link to your sitemap in your robots.txt file, is that proper procedure and is this: http://tbmotoring.com/xmlsitemap.php the proper link to put in my robots.txt?

            Previously, what I meant by removing .php extension is that if I am using mod rewrite friendly url's do I leave .php at the end of each path in robots.txt ( ie. instead of Disallow: /ajax.php should I put Disallow: /ajax)?

            Thanks for your patience.
            Last edited by too_cool_3; Wed 2nd Nov '11, 8:24pm.

            Comment


            • #7
              Don't see anything about a sitemap here: http://www.robotstxt.org/

              Leave the .php.
              Translations provided by Google.

              Wayne Luke
              The Rabid Badger - a vBulletin Cloud customization and demonstration site.
              vBulletin 5 Documentation - Updated every Friday. Report issues here.
              vBulletin 5 API - Full / Mobile
              I am not currently available for vB Messenger Chats.

              Comment


              • #8
                Thank you for the helpful link, I just finished reading the FAQ at http://www.robotstxt.org lot of good information. The only reason I put my sitemap in the robots.txt file is from seeing it done by Brian Cummiskey over at vbseo in this thread: http://www.vbseo.com/f2/tutorial-how...70/#post253465

                Thanks again Wayne for all the info!

                Comment

                widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                Working...
                X