Announcement

Collapse
No announcement yet.

Chinese web crawlers

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Chinese web crawlers

    Not sure id this information could be disclose or not but I was asked if sites on the vB Cloud are crawled -- or are allowed to be crawled -- by web crawlers from China like Baidu spider bot or not.

    We had some decent traffic from Baidu prior to the migration to the Cloud and at the moment we dont see them crawling the site.

    www.legaljunkies.com

  • #2
    vBulletin Cloud sites have this current robots.txt file:

    Code:
    Sitemap: /path/to/xmlsitemap.php
    
    User-agent: *
    crawl-delay: 1
    
    User-agent: *
    Disallow: /admincp/
    Disallow: /modcp/
    Disallow: /attachments/
    Disallow: /images/
    Disallow: /members/
    Disallow: /search/
    Disallow: /new-content/
    Disallow: /auth/
    Disallow: /register
    Disallow: /uploader/url
    
    User-agent: BaiDuSpider
    Disallow: /
    It is global to all sites. Baidu would be blocked by it. Baidu doesn't generally practice good indexing techniques and will eat all of your monthly bandwidth.
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud demonstration site.
    vBulletin 5 API - Full / Mobile
    Vote for your favorite feature requests and the bugs you want to see fixed.

    Comment


    • #3
      OK, thank you for that information.

      We do have some legitimate participants from China.

      If we want to allow Baidu to crawl just our website could an exception be made or is that not possible on the Cloud?
      www.legaljunkies.com

      Comment


      • #4
        As all Cloud sites share the same exact files, it is not possible to have a customized robot.txt file at this time. Do you have Chinese content on your site? AS far as I am aware, that is a legal requirement to be accepted in Baidu's search engine.
        Translations provided by Google.

        Wayne Luke
        The Rabid Badger - a vBulletin Cloud demonstration site.
        vBulletin 5 API - Full / Mobile
        Vote for your favorite feature requests and the bugs you want to see fixed.

        Comment


        • #5
          Thank you for the reply!

          Yes, we do have some Chinese content on our site but I understand the situation. It is not a critical issue at the moment.
          www.legaljunkies.com

          Comment

          Related Topics

          Collapse

          Working...
          X