Announcement

Collapse
No announcement yet.

Robots Question

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Scott MacVicar
    replied
    You have to remember that blocking the PHP Useragent also blocks vBulletin if someone tries to fetch an image from a URL for attachment, avatar etc

    Leave a comment:


  • Zachery
    replied
    I would have to say Mediapartners, but since you don't use adsense i don't see a problem with any of those that I am aware of.

    Leave a comment:


  • Shining Arcanine
    started a topic Robots Question

    Robots Question

    To cut down on bandwidth usage, today I through awstatus' unknown browsers list and created a list of robots for my .htaccess file. I then went through the raw access logs I had avaliable to me and removed the robots that hit robots.txt. My question is, are there any robots on my block list that most people would not want to be blocking?

    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^aipbot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^carleson [OR]
    RewriteCond %{HTTP_USER_AGENT} ^cfetch [OR]
    RewriteCond %{HTTP_USER_AGENT} ^CFNetwork [OR]
    RewriteCond %{HTTP_USER_AGENT} ^contype_WebWasher [OR]
    RewriteCond %{HTTP_USER_AGENT} ^CryptRetrieveObjectByUrl::InetSchemeProvider [OR]
    RewriteCond %{HTTP_USER_AGENT} ^DA_ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Democracy [OR]
    RewriteCond %{HTTP_USER_AGENT} ^edgeio-retriever [OR]
    RewriteCond %{HTTP_USER_AGENT} ^FDM_ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^feedfinder [OR]
    RewriteCond %{HTTP_USER_AGENT} ^GbPlugin [OR]
    RewriteCond %{HTTP_USER_AGENT} ^genieBot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Goldfire_Server [OR]
    RewriteCond %{HTTP_USER_AGENT} ^ICOO_Loader [OR]
    RewriteCond %{HTTP_USER_AGENT} ^IDA$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^IECheck$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^JoeDog [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Jyxobot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^larbin_ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^LWP::Simple [OR]
    RewriteCond %{HTTP_USER_AGENT} ^lwp-trivial [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mediapartners-Google [OR] #I do not use adsense
    RewriteCond %{HTTP_USER_AGENT} ^Microsoft_URL_Control [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mimetype_Getinfo$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^MVAClient$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^NaverBot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^nicebot$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^obot$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^OmniExplorer_Bot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^OSSProxy [OR]
    RewriteCond %{HTTP_USER_AGENT} ^PHP$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Python [OR]
    RewriteCond %{HTTP_USER_AGENT} ^SBD_Link_Tester [OR]
    RewriteCond %{HTTP_USER_AGENT} ^SURF$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^SurveyBot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^WordPress [OR]
    RewriteCond %{HTTP_USER_AGENT} ^[url]www.petitsage.fr_site_detector[/url]
    RewriteRule ^.* - [F]

Related Topics

Collapse

Working...
X