i have exclude many (bad) bots - not with robot.txt - i do this over the .htaccess
-->> http://badbots.vps.tips/
Announcement
Collapse
No announcement yet.
Robots.txt for VB 5
Collapse
X
-
Originally posted by Madekozu View PostHere is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
-
Originally posted by SpaceStar View Post
It is just a blank file with only one entry anyone attached....
Leave a comment:
-
Originally posted by Madekozu View PostHere is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
Leave a comment:
-
Originally posted by Alsuheimat View Post
I want the file that you have
Attached Files
Leave a comment:
-
Here is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
Leave a comment:
-
Originally posted by SpaceStar View PostAlsuheimat,
I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it
User-Agent:*
and put it into root of my website with sufficent access to read. Google and Bing picked it up.
Leave a comment:
-
Originally posted by SpaceStar View PostAlsuheimat,
I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it
User-Agent:*
and put it into root of my website with sufficent access to read. Google and Bing picked it up.
I attach a file robots.txt to which you have
Leave a comment:
-
Alsuheimat,
I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it
User-Agent:*
and put it into root of my website with sufficent access to read. Google and Bing picked it up.
- 1 like
Leave a comment:
-
The reason we do not supply a robots.txt is because the file is required to be in the root of the website. There is no guarantee that vBulletin will be in the root of the website. Many people actually do not install vBulletin in the root of their website.
http://www.robotstxt.org/robotstxt.html
Leave a comment:
-
Robots.txt for VB 5
Last edited by Alsuheimat; Tue 27 Sep '16, 3:27am.Tags: None
Related Topics
Collapse
-
by panoramiaDear Sirs,
I am trying to submit vBulletin 5 xml sitemap to Google. I am a bit confused with the instructions in vBulletin backend. It says that it will automatically submit sitemap to Google...2 Photos-
Channel: Support Issues & Questions
Thu 10 Nov '16, 4:26am -
-
by dtwsolutionsMy client's VB forum is running the latest version, but we can not get the XML sitemap to be loaded in Google. It says "Sitemap index has errors" then has:
This required tag is...-
Channel: Support Issues & Questions
Mon 16 Apr '18, 7:15pm -
-
by HIZIRBEYafter the creating my website it looks like that. what do I need to do to fix this ?1 Photo
-
Channel: vBulletin 5 Installs & Upgrades
Mon 11 Sep '17, 10:13am -
Leave a comment: