Announcement

Collapse
No announcement yet.

Robots.txt for VB 5

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Robots.txt for VB 5

    Click image for larger version

Name:	Untitled.png
Views:	115
Size:	34.8 KB
ID:	4354696 Need this file Robots.txt for VB 5.2.4


    Click image for larger version  Name:	Untitled.png Views:	1 Size:	34.8 KB ID:	4354693
    Last edited by Alsuheimat; Tue 27th Sep '16, 3:27am.

  • #3
    The reason we do not supply a robots.txt is because the file is required to be in the root of the website. There is no guarantee that vBulletin will be in the root of the website. Many people actually do not install vBulletin in the root of their website.

    http://www.robotstxt.org/robotstxt.html
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud customization and demonstration site.
    vBulletin 5 Documentation - Updated every Friday. Report issues here.
    vBulletin 5 API - Full / Mobile
    I am not currently available for vB Messenger Chats.

    Comment


    • #4
      Alsuheimat,

      I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

      User-Agent:*

      and put it into root of my website with sufficent access to read. Google and Bing picked it up.

      Comment


      • #5
        Originally posted by SpaceStar View Post
        Alsuheimat,

        I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

        User-Agent:*

        and put it into root of my website with sufficent access to read. Google and Bing picked it up.
        Thanks
        I attach a file robots.txt to which you have

        Comment


        • #6
          Originally posted by SpaceStar View Post
          Alsuheimat,

          I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

          User-Agent:*

          and put it into root of my website with sufficent access to read. Google and Bing picked it up.
          I want the file that you have

          Comment


          • #7
            Here is an example of a part of my robots.txt


            Code:
            Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz
            
            User-agent: *
            Disallow: /admincp/
            Disallow: /usage/
            Disallow: /logs/
            Disallow: /rausda.php
            
            User-agent: SentiBot
            Disallow: /
            
            User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
            Disallow: /
            
            User-agent: MJ12bot
            Disallow: /
            
            User-agent: MegaIndex.ru/2.0
            Disallow: /

            It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
            läuft

            Comment


            • #8
              Originally posted by Alsuheimat View Post

              I want the file that you have
              It is just a blank file with only one entry anyone attached....

              Attached Files

              Comment


              • #9
                Originally posted by Madekozu View Post
                Here is an example of a part of my robots.txt


                Code:
                Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz
                
                User-agent: *
                Disallow: /admincp/
                Disallow: /usage/
                Disallow: /logs/
                Disallow: /rausda.php
                
                User-agent: SentiBot
                Disallow: /
                
                User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
                Disallow: /
                
                User-agent: MJ12bot
                Disallow: /
                
                User-agent: MegaIndex.ru/2.0
                Disallow: /

                It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
                thanks

                Comment


                • #10
                  Originally posted by SpaceStar View Post

                  It is just a blank file with only one entry anyone attached....
                  thanks

                  Comment


                  • #11
                    Originally posted by Madekozu View Post
                    Here is an example of a part of my robots.txt


                    Code:
                    Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz
                    
                    User-agent: *
                    Disallow: /admincp/
                    Disallow: /usage/
                    Disallow: /logs/
                    Disallow: /rausda.php
                    
                    User-agent: SentiBot
                    Disallow: /
                    
                    User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
                    Disallow: /
                    
                    User-agent: MJ12bot
                    Disallow: /
                    
                    User-agent: MegaIndex.ru/2.0
                    Disallow: /

                    It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
                    Good example there. I may also exclude Baidu ...

                    Comment

                  Related Topics

                  Collapse

                  Working...
                  X