Tweet
Hello,
My board sometimes is heavily scanned by spiders. On my board is required to register to read/post. And sometimes (tonight being the most) There was 780 online users, but only near 200 were registered. So I think most of those unregistered users were spiders.
I don't want to block all spiders. Since I would still like allow spiders like Google, Yahoo, MSN, and all those big search engines. But some others I don't want them hitting that hard my server. If there would be an option, to block them only from my board folder (/foro/), but still allow the to spider the rest of the site would be great too.
Any suggestions welcome...
My board sometimes is heavily scanned by spiders. On my board is required to register to read/post. And sometimes (tonight being the most) There was 780 online users, but only near 200 were registered. So I think most of those unregistered users were spiders.
I don't want to block all spiders. Since I would still like allow spiders like Google, Yahoo, MSN, and all those big search engines. But some others I don't want them hitting that hard my server. If there would be an option, to block them only from my board folder (/foro/), but still allow the to spider the rest of the site would be great too.
Any suggestions welcome...
Comment