Announcement

Collapse
No announcement yet.

Search Engine Spiders

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Search Engine Spiders

    I have just setup my website so that Guests cannot see the content of any posts on the site until they signup.

    They can see all the titles of the threads and how many have been posted in that forum etc... But not the content.

    My question is, do the search engine spiders get treated as Guests when they are crawling your website? And if so does this mean that they will not be able to index my site correctly?

    Thanks for any help

  • #2
    yes that is correct.

    Comment


    • #3
      Is there any way of altering the usergroup that search engine spiders belong to?

      Comment


      • #4
        Originally posted by HardGainer
        Is there any way of altering the usergroup that search engine spiders belong to?
        Submit a ticket, the responce is about 30mins.
        You know you can't live without vBulletin!

        Comment


        • #5
          Originally posted by HardGainer
          Is there any way of altering the usergroup that search engine spiders belong to?
          This would take modifying the code itself. You could ask at vBulletin.org.
          Translations provided by Google.

          Wayne Luke
          The Rabid Badger - a vBulletin Cloud demonstration site.
          vBulletin 5 API - Full / Mobile
          Vote for your favorite feature requests and the bugs you want to see fixed.

          Comment


          • #6
            Originally posted by Wayne Luke
            This would take modifying the code itself. You could ask at vBulletin.org.
            Thanks Wayne

            Comment


            • #7
              Most SE companys frown apon that heavily.

              Comment


              • #8
                Originally posted by Zachery
                Most SE companys frown apon that heavily.
                Yes I have heard that this is a no no.
                CommanderTalk.com --> www.commandertalk.com

                Comment


                • #9
                  do they not obey the robots.txt anymore?
                  I have a robots.txt which specifies that they don't enter the www.site.com/forum/ subdirectory. that worked perfectly for a while but they've started crawling again recently.

                  Also, I have the most current spiders_vbulletin.xml in /forum/includes/xml/ but the agent name does not show in "Who's Online." Any idea why?

                  Comment


                  • #10
                    Um you are aware that it can take up to 30 days to cache those files, and then they stay cached for 30 days.

                    Comment


                    • #11
                      but my robots.txt has been in my home directory for close to a year now and has not been changed. It used to keep them away, but recently my counter was spiked to an all time high.

                      Comment

                      widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                      Working...
                      X