[Forum] Can't prevent vBulletin content being ripped

[slinky]

I use RSS feeds internally on my site to provide "last 10 posts." I do not post RSS feeds publicly. Here's the problem:

- vBulletin doesn't limit the number of characters displayed in an RSS Feed

I'm noticing that certain sites are ripping my content, some of them entire posts, and storing them. I've noticed that others will include them in Javascript and my post doesn't even appear on Google pages. I recall seeing posts in Google linking to BoardReader and getting there and not even finding my content, shown in Google searches, even visible on their pages. Other sites like FeedFury will take a few hundred of my posts and link them as well. I can only wonder whether this impacts in negative SEO, especially since vBulletin allows them to take the full content of the post.

[Andy]

Do you use a robots.txt file?

Here's what I use and it appears the two sites you mentioned obey the robots.txt as my forum isn't listed in either one.

User-agent: Googlebot
Allow: /

User-agent: Yahoo! Slurp
Allow: /

User-agent: msnbot
Allow: /

User-agent: *
Disallow: /

[slinky]

Andy - yes, I use a robots.txt file. The problem with your solution is:

(a) I've only listed two site rippers. Most don't play by the rules.

(b) I need to control the amount of content for rss feeds if I ever do want to have my vBulletin RSS feeds shared. I don't want to use Feedburner for a variety of reasons.

[Andy]

- vBulletin doesn't limit the number of characters displayed in an RSS Feed

The rss2 feed output is limited on my forum to about one line of text after that I get the "more..." indication.

[slinky]

The rss2 feed output is limited on my forum to about one line of text after that I get the "more..." indication.

It might be the code that is pulling your feed limiting the feed to a number of characters and then adding "more..." -- that's what most good RSS scripts do.

[Ace]

The rss2 feed output is limited on my forum to about one line of text after that I get the "more..." indication.

No it isn't. The entire first post is displayed:

site.com/external.php?type=RSS2&forumids=5

(When you are pulling other sites' rss feeds, it's not your rss, which is what slinky is trying to control.)

[slinky]

No it isn't. The entire first post is displayed:

xxxxxx

(When you are pulling other sites' rss feeds, it's not your rss, which is what slinky is trying to control.)

Ace - he might be using Mozilla which appears to truncate some RSS feeds (at least on that URL.) I've noticed that if you pull the same feed in Chrome, you get to see the entire post as you've correctly confirmed.

[Ace]

Ace - he might be using Mozilla which appears to truncate some RSS feeds (at least on that URL.) I've noticed that if you pull the same feed in Chrome, you get to see the entire post as you've correctly confirmed.

Yeah, I never use FF for rss, it annoyed me the way it displayed. Chrome's bugging out on RSS, brings up a blank app store window, I saw that from IE.

(There's a big thread somewhere around about 'MOZ sites stealing content', you might want to take a look in there, might be some ideas?)

[slinky]

Thanks for the link reference - I've seen that one before. Doesn't stop this huge RSS hole from being plugged though. Limiting characters supplied to an RSS feed has been standard for a long time. Hopefully this will make it's way into vBulletin as soon as possible. Will see if it's in the JIRA.

[Andy]

It might be the code that is pulling your feed limiting the feed to a number of characters and then adding "more..." -- that's what most good RSS scripts do.

Sorry I'm not understanding the issue. Please provide an example of the problem.

[slinky]

If I use Chrome to pull the RSS feeds from your site as per the link Ace posted earlier and removed, your entire post is published. That's what most RSS readers will pull, NOT the one you'll see if you use Mozilla which seems to truncate posts to a length it finds acceptable.

So in short, your site is pushing the entire post to be published. I could subscribe to your sites RSS feed and if I decided to automatically place it in a Wordpress site, I would be publishing your entire post with a link. The only reason you're seeing a shorter version with an ellipsis is because some software you're using is abbreviating the post.

[Andy]

Thank you slinky, I now understand the issue regarding rss.

Question, how would truncating the post content in rss feeds help if a post itself shows the entire content? A person bent on ripping content doesn't need to use rss, they can simply rip the content directly from the thread/posts.

[slinky]

Question, how would truncating the post content in rss feeds help if a post itself shows the entire content? A person bent on ripping content doesn't need to use rss, they can simply rip the content directly from the thread/posts.

You're welcome. Here is the problem:

(1) Ripping a site for inclusion in another website takes skills far beyond the average person. It requires a programmer as well as periodical spidering to discover what content is new. RSS makes it simple so that anyone with Wordpress and a plugin can replicate your new posts in a matter of minutes and have "fresh" content when it's published on an ongoing basis via the feed (rolling last 5 or 10 posts.)

(2) I use the RSS feeds internally on my site. Once they are turned on, anyone can get the RSS feed and now rip my site because it's easy to determine what it is on a vBulletin forum. If I could limit the number of characters, it minimizes any damage of full posts being taken.

(3) As of right now, I have zero choice in providing RSS feeds to third parties to popularize my site. If I turn it on, I'm essentially giving them permission to take the whole post, aren't I? This is another reason why I haven't put up links to my forum RSS feeds.

[Andy]

Hi Slinky,

What you're saying makes complete sense. Let me look at the external.php code and see if it's a simple hack to truncate the feed content.

[Riasat]

Thanks for the link reference - I've seen that one before. Doesn't stop this huge RSS hole from being plugged though. Limiting characters supplied to an RSS feed has been standard for a long time. Hopefully this will make it's way into vBulletin as soon as possible. Will see if it's in the JIRA.

There is nothing standard about it. Also, as an end user, I hate sites that limit RSS with some char limit.


As for limiting it in vb, I dont think I saw anything like this in JIRA. But I'm sure you can do it easily with a plugin.