Announcement
Collapse
No announcement yet.
Robots.txt for VB 5
Collapse
X
-
The reason we do not supply a robots.txt is because the file is required to be in the root of the website. There is no guarantee that vBulletin will be in the root of the website. Many people actually do not install vBulletin in the root of their website.
http://www.robotstxt.org/robotstxt.htmlTranslations provided by Google.
Wayne Luke
The Rabid Badger - a vBulletin Cloud demonstration site.
vBulletin 5 API
Comment
-
Alsuheimat,
I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it
User-Agent:*
and put it into root of my website with sufficent access to read. Google and Bing picked it up.
- 1 like
Comment
-
Originally posted by SpaceStar View PostAlsuheimat,
I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it
User-Agent:*
and put it into root of my website with sufficent access to read. Google and Bing picked it up.
I attach a file robots.txt to which you have
Comment
-
Originally posted by SpaceStar View PostAlsuheimat,
I faced similar thing. Google said it cannot crawl because robots.txt does not exist. I created a dummy file with following text in it
User-Agent:*
and put it into root of my website with sufficent access to read. Google and Bing picked it up.
Comment
-
Here is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
Comment
-
Originally posted by Madekozu View PostHere is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
Comment
-
Originally posted by Madekozu View PostHere is an example of a part of my robots.txt
Code:Sitemap: http://madekozu.de/vbulletin_sitemap_index.xml.gz User-agent: * Disallow: /admincp/ Disallow: /usage/ Disallow: /logs/ Disallow: /rausda.php User-agent: SentiBot Disallow: / User-agent: SentiBot www.sentibot.eu (compatible with Googlebot) Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: /
It makes sense to use its own settings. Which Bot may search, which is not? You'll have to decide for your project!
Comment
-
i have exclude many (bad) bots - not with robot.txt - i do this over the .htaccess
-->> http://badbots.vps.tips/
-
Related Topics
Collapse
-
by panoramiaDear Sirs,
I am trying to submit vBulletin 5 xml sitemap to Google. I am a bit confused with the instructions in vBulletin backend. It says that it will automatically submit sitemap to Google...2 Photos-
Channel: Support Issues & Questions
Thu 10 Nov '16, 4:26am -
-
by dtwsolutionsMy client's VB forum is running the latest version, but we can not get the XML sitemap to be loaded in Google. It says "Sitemap index has errors" then has:
This required tag is...-
Channel: Support Issues & Questions
Mon 16 Apr '18, 7:15pm -
-
by HIZIRBEYafter the creating my website it looks like that. what do I need to do to fix this ?1 Photo
-
Channel: vBulletin 5 Installs & Upgrades
Mon 11 Sep '17, 10:13am -
Comment