No announcement yet.

I want to ban Google, Yahoo - ALL of them!

  • Filter
  • Time
  • Show
Clear All
new posts

  • I want to ban Google, Yahoo - ALL of them!

    I am sick to the back teeth of spiders, bots, guests causing me problems!

    My site is tiny (perhaps 5 regular posters) yet my web providers are giving me warnings that I must upgrade because traffic volume is so HIGH!!!

    Any tips?
    Last edited by Jirina; Thu 16 Aug '07, 1:52pm.

  • #2
    Since posting I've managed to figure out how to create a robots.txt file and where to put it. So that's in place and in theory should block ALL search engines and I would have hoped all "guests" (I've also set unregistered visitors to be unable to view ANY forum). But still they get through, admittedly in smaller numbers, but still they come.
    Last edited by Jirina; Thu 16 Aug '07, 1:52pm.


    • #3
      User-agent: X
      Disallow: /

      You need to replace X with the spider name.

      From ...

      Where Do You Get the Name of the Robots?

      If you have a particular spider in mind which you want to block, you have to find out its name.
      To do this, the best way is to check out the website of the search engine. Respectable
      engines will usually have a page somewhere that gives you details on how you can prevent
      their spiders from accessing certain files or directories.
      Last edited by Michael2; Thu 16 Aug '07, 4:07am.
      HBO Forum - Discuss Game of Thrones, True Blood, Big Love, Boardwalk Empire, and more!
      Disney Forums - Disney theme parks, vacation, entertainment, and more!
      iLoveTalk - Our general discussion forum.
      VeggieBoards - The largest and most active vegetarian forum!


      • #4
        Thank you Michael. I thought with the most global ban it would automatically take in everything.

        You mean I have to list each and every spider and "guest" IP? That's quite a task.

        Even though I have set all unregistered guests not to see forums and have a global ban I still have up to seven guests in my forums at any one time and some of those are getting right through to threads - even though I and all the other members have to enter our usernames and passwords before being able to even SEE the forums. I don't get it and it's driving me mental.

        Thanks for your advice though, I really do appreciate your taking the time.
        Last edited by Jirina; Thu 16 Aug '07, 1:53pm.


        • #5
          Better look into getting another host.
          I buy 420 forums


          • #6
            I just got told today to sit tight, they're going to upgrade the spec at no cost next month. So it's not so bad.

            I'd almost forgotten trying to get the site listed in the search engines two years back, so this seems to be what's costing me now. Back then I wasn't at all aware of bots 'n spiders. Everyone should probably think carefully at the start of their site as to whether they want to stay small and cosy or not.

            Maybe they'd find you anyway though.

            At least my robots.txt file and/or my closing all forums to the unregistered means that the spiders only get the closed door to look at. Hopefully they'll eventually take the hint and take a hike.

            Anyways, I've done enough to not feel infested any longer.

            Thanks folks.


            widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.