Robots.txt for VB 5

[Madekozu]

i have exclude many (bad) bots - not with robot.txt - i do this over the .htaccess
-->> http://badbots.vps.tips/

[SpaceStar]

Here is an example of a part of my robots.txt

Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz

User-agent: *
Disallow: /admincp/
Disallow: /usage/
Disallow: /logs/
Disallow: /rausda.php

User-agent: SentiBot
Disallow: /

User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: MegaIndex.ru/2.0
Disallow: /

It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!

Good example there. I may also exclude Baidu ...

[Alsuheimat]

It is just a blank file with only one entry anyone attached....

thanks

[Alsuheimat]

Here is an example of a part of my robots.txt

Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz

User-agent: *
Disallow: /admincp/
Disallow: /usage/
Disallow: /logs/
Disallow: /rausda.php

User-agent: SentiBot
Disallow: /

User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: MegaIndex.ru/2.0
Disallow: /

It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!

thanks

[SpaceStar]

I want the file that you have

It is just a blank file with only one entry anyone attached....

[Madekozu]

Here is an example of a part of my robots.txt

Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz

User-agent: *
Disallow: /admincp/
Disallow: /usage/
Disallow: /logs/
Disallow: /rausda.php

User-agent: SentiBot
Disallow: /

User-agent: SentiBot www.sentibot.eu (compatible with Googlebot)
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: MegaIndex.ru/2.0
Disallow: /

It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!

[Alsuheimat]

Alsuheimat,

I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

User-Agent:*

and put it into root of my website with sufficent access to read. Google and Bing picked it up.

I want the file that you have

[Alsuheimat]

Alsuheimat,

I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

User-Agent:*

and put it into root of my website with sufficent access to read. Google and Bing picked it up.

Thanks
I attach a file robots.txt to which you have

[SpaceStar]

Alsuheimat,

I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it

User-Agent:*

and put it into root of my website with sufficent access to read. Google and Bing picked it up.

[Wayne Luke]

The reason we do not supply a robots.txt is because the file is required to be in the root of the website. There is no guarantee that vBulletin will be in the root of the website. Many people actually do not install vBulletin in the root of their website.

http://www.robotstxt.org/robotstxt.html

[Trevor Hannant]

Not something we supply, you will need to create your own.

[Alsuheimat]

Need this file Robots.txt for VB 5.2.4