Announcement

Collapse
No announcement yet.

Robots.txt for VB 5

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Madekozu
    commented on 's reply
    i have exclude many (bad) bots - not with robot.txt - i do this over the .htaccess
    -->> http://badbots.vps.tips/

  • SpaceStar
    replied
    Originally posted by Madekozu View Post
    Here is an example of a part of my robots.txt


    Code:
    Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz
    
    User-agent: *
    Disallow: /admincp/
    Disallow: /usage/
    Disallow: /logs/
    Disallow: /rausda.php
    
    User-agent: SentiBot
    Disallow: /
    
    User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
    Disallow: /
    
    User-agent: MJ12bot
    Disallow: /
    
    User-agent: MegaIndex.ru/2.0
    Disallow: /

    It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
    Good example there. I may also exclude Baidu ...

    Leave a comment:


  • Alsuheimat
    replied
    Originally posted by SpaceStar View Post

    It is just a blank file with only one entry anyone attached....
    thanks

    Leave a comment:


  • Alsuheimat
    replied
    Originally posted by Madekozu View Post
    Here is an example of a part of my robots.txt


    Code:
    Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz
    
    User-agent: *
    Disallow: /admincp/
    Disallow: /usage/
    Disallow: /logs/
    Disallow: /rausda.php
    
    User-agent: SentiBot
    Disallow: /
    
    User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
    Disallow: /
    
    User-agent: MJ12bot
    Disallow: /
    
    User-agent: MegaIndex.ru/2.0
    Disallow: /

    It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
    thanks

    Leave a comment:


  • SpaceStar
    replied
    Originally posted by Alsuheimat View Post

    I want the file that you have
    It is just a blank file with only one entry anyone attached....

    Attached Files

    Leave a comment:


  • Madekozu
    replied
    Here is an example of a part of my robots.txt


    Code:
    Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz
    
    User-agent: *
    Disallow: /admincp/
    Disallow: /usage/
    Disallow: /logs/
    Disallow: /rausda.php
    
    User-agent: SentiBot
    Disallow: /
    
    User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
    Disallow: /
    
    User-agent: MJ12bot
    Disallow: /
    
    User-agent: MegaIndex.ru/2.0
    Disallow: /

    It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!

    Leave a comment:


  • Alsuheimat
    replied
    Originally posted by SpaceStar View Post
    Alsuheimat,

    I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

    User-Agent:*

    and put it into root of my website with sufficent access to read. Google and Bing picked it up.
    I want the file that you have

    Leave a comment:


  • Alsuheimat
    replied
    Originally posted by SpaceStar View Post
    Alsuheimat,

    I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

    User-Agent:*

    and put it into root of my website with sufficent access to read. Google and Bing picked it up.
    Thanks
    I attach a file robots.txt to which you have

    Leave a comment:


  • SpaceStar
    replied
    Alsuheimat,

    I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

    User-Agent:*

    and put it into root of my website with sufficent access to read. Google and Bing picked it up.

    Leave a comment:


  • Wayne Luke
    replied
    The reason we do not supply a robots.txt is because the file is required to be in the root of the website. There is no guarantee that vBulletin will be in the root of the website. Many people actually do not install vBulletin in the root of their website.

    http://www.robotstxt.org/robotstxt.html

    Leave a comment:


  • Trevor Hannant
    replied
    Not something we supply, you will need to create your own.

    Leave a comment:


  • Alsuheimat
    started a topic Robots.txt for VB 5

    Robots.txt for VB 5

    Click image for larger version

Name:	Untitled.png
Views:	115
Size:	34.8 KB
ID:	4354696 Need this file Robots.txt for VB 5.2.4


    Click image for larger version  Name:	Untitled.png Views:	1 Size:	34.8 KB ID:	4354693
    Last edited by Alsuheimat; Tue 27th Sep '16, 3:27am.

Related Topics

Collapse

Working...
X