Search engine spider recognition

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • yellow_spider
    Senior Member
    • Aug 2005
    • 139

    Search engine spider recognition

    Il admit im am more used to IPB to be honest and only have got back to VB recently...

    Anyway, how is search engine spider recognition handled in VB please ?

    With IPB you could add a list ... (within reason so as not to slow things down)

    example..

    Code:
    archive.org_bot=Archive.org
    archive_org=Archive.org
    ask jeeves=Ask Jeeves
    etc etc

    What does VB recognise as default ?

    And can more be added ?


    Thanks
  • Andy Huang
    Senior Member
    • Feb 2004
    • 4602

    #2
    This can be done in an XML file in vBulletin; these links will help you get started:

    Best Regards,
    Andy Huang

    Comment

    • yellow_spider
      Senior Member
      • Aug 2005
      • 139

      #3
      Many thanks

      I meant to ask, is there a way to make them show on the forum index page too ? (or would i be better asking this question at vb.org ?)

      Comment

      • Andy Huang
        Senior Member
        • Feb 2004
        • 4602

        #4
        They aren't listed on the who's online section because they don't always come from the same source, which makes replicates of them. If you look on who's online section of this forum right now, you will notice it is being bombarded by Yahoo Slurp... ("Suddenly, Yahoo Spiders everywhere, hundreds of them!!"?)

        I do believe there is a hack available on vborg that would tally them up and then list those that are currently on site, but I don't know where exactly it is. Sorry.
        Best Regards,
        Andy Huang

        Comment

        • yellow_spider
          Senior Member
          • Aug 2005
          • 139

          #5
          Cheers for detailed reply



          * Goes off to look at vb.org *

          Comment

          • DDIG
            Member
            • Oct 2007
            • 92
            • 3.8.x

            #6
            Originally posted by Andy Huang

            Is this XML file still beeing updated? I have just uploaded it to my server, it seems to recognize quite a few crawlers and spiders.

            I've noticed crawl-1.cuill.com isn't in it, but I also saw this: Last edited by Stadler; Tue 30th May '06 at 3:30pm. Reason: typo
            Wat You See Is What I Could Have Meant
            D-DIG.NL -
            Hèt trefpunt voor alles over digitale fotografie op internet... En meer!

            Comment

            • Andy Huang
              Senior Member
              • Feb 2004
              • 4602

              #7
              Probably not; but you can pick up and update from there if you want
              Best Regards,
              Andy Huang

              Comment

              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
              Working...
              😀
              😂
              🥰
              😘
              🤢
              😎
              😞
              😡
              👍
              👎