vBulletin and Sphider Search Engine

[bar338]

I'm using the php sphider search engine to index and search my website. vBulletin is just a part of the website. vBulletin still uses its internal search engine for local searches but I also need to be able to include vBulletin content in my site wide search. To do this i'm using the sphider search engine, however i'm having difficulties indexing the forums properly. When I index it searchs absolutly everything which includes a whole lot of meaningless content (for example it searches the calendars (which are empty) for what seems like forever). Are there any general rules for improving the searchability of the forums short of adding a hundred different rules to the search engine to prevent it from going places it shoudn't.

[Steve Machol]

You would have to configure your search program to ignore content from specific pages. This is not really a vB issue but one with that search program.

[chubba]

Sphider is really good here - it is showing you general spidering problems with your forum. If sphider follows these links then so will Googlebot. My suggestion is that you disallow the URLs you do not want Sphider to see in a robots.txt file. This way you can control what it will index and what it won't.

By using:

User-agent: *

You can stop all bots getting into the Calendar area. If you want Googlebot to see member profiles but you don't want sphider indexing them or vice-versa then you can set up rules in your robots.txt to do this too.

[linuxutut]

Thanks for the "Sphider" tip.
I never heard of it before.
Have it installed and in the process of running it now.
Will be most helpful in making a GOOD robots.txt and see what pages
( forums, and sub-forums are getting indexed. ).

Thanks,