The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

rss - all the feed

This question probably belongs here...

RSS feeds only give you the most recent 10-20 posts. In most cases, that's all you need, but I would like to get all posts in certain circumstances. Is there a way to do that short of screenscraping?
all the feed
Tuesday, April 08, 2008
 
 
I don't think it belongs here, because you're not designing a RSS reader, you're just using it.

With Google Reader, if I just scroll down it keeps sucking in more and more articles. There doesn't seem to be a limit to it.
Daniel_DL Send private email
Tuesday, April 08, 2008
 
 
I don't use google reader so I didn't know that.
Thanks
all the feed
Tuesday, April 08, 2008
 
 
I've been googling for a while but I can't figure out how to do this. How does google reader do it? Anyone know?
all the feed
Tuesday, April 08, 2008
 
 
Google FeedApi :)
Masiosare Send private email
Tuesday, April 08, 2008
 
 
Google doesn't use its own api right? :)

Does RSS support this feature or did google do its own screenscraping?!
all the feed
Tuesday, April 08, 2008
 
 
I use the reader built in to safari, and if I don't check slash dot for a week or two I usually end up with a few thousand articles listed.  What you are talking about isn't a feature of RSS it is a limitation in the reader/site you have used. 

RSS is just an xml file that the reader checks periodically.
Brian
Tuesday, April 08, 2008
 
 
I am looking at Slashdot.org right now using Firefox (2.0.0.13). I clicked on the RSS icon next to the URL and a new XML-looking page loaded. It only has 15 entries, not the entire archive of Slashdot.org since the beginning of time.

So you are telling me Firefox is cutting of the actual feed at 15 entries?
all the feed
Tuesday, April 08, 2008
 
 
I am not sure what you are talking about Brian. A little more search revealed that RSS is limited to 15 entries in general. Maybe Safari is doing something more in the background?
all the feed
Tuesday, April 08, 2008
 
 
According to the RSS 1.0 spec

-- any object with a URI. There may be a minimum of one item per RSS document. While RSS 1.0 does not enforce an upper limit, for backward compatibility with RSS 0.9 and 0.91, a maximum of fifteen items is recommended.

According to the RSS 2.0 spec

There can be no more than 15 <items> in a 0.91 <channel>. There are no string-length or XML-level limits in RSS 0.92 and greater. Processors may impose their own limits, and generators may have preferences that say no more than a certain number of <item>s can appear in a channel, or that strings are limited in length.

So it depends on what version of RSS you are targeting 0.91 or later.
Brian
Tuesday, April 08, 2008
 
 
>> I am not sure what you are talking about Brian. A little more search revealed that RSS is limited to 15 entries in general. <<

An RSS feed supports however many posts the author of the feed specifies. On my techical blog, I specify 10. On one of my product blogs, I specify 20. It's entirely up to me.

Some RSS readers may impose additional limits, thereby reducing the limit imposed by the feed author. Some RSS readers may log historical entries for popular feeds, thus increasing the limit imposed by the feed author. But in general, the feed author is in control.
Mark Pearce Send private email
Tuesday, April 08, 2008
 
 
So I would need to do some kind of scraping then to get everything...
all the feed
Tuesday, April 08, 2008
 
 
According to the spec. the 15 element cap was apart of .91 spec all later versions are capless.
Brian
Tuesday, April 08, 2008
 
 
"So I would need to do some kind of scraping then to get everything..."

Well, of course RSS depends on the publisher. If I make a webpage with 200 articles and create a RSS feed which only includes the latest 10, you won't be able to get the other 190 at all.

Google Reader has no limit... other than the one imposed by the information source, of course.
Daniel_DL Send private email
Wednesday, April 09, 2008
 
 
> Google Reader has no limit... other than the one imposed
> by the information source, of course.

If you point Google Reader at the Slashdot RSS feed (rss.slashdot.org/Slashdot/slashdot), which shows only the 15 most recent posts, you are still able to see the earlier posts which are not in the RSS feed!

IOW, even though the RSS feed itself has only 15 entries, google reader is able to read earlier posts. Is it scraping the website? Is there a way to force the RSS feed to give more entries?
all the feed
Wednesday, April 09, 2008
 
 
>> If you point Google Reader at the Slashdot RSS feed (rss.slashdot.org/Slashdot/slashdot), which shows only the 15 most recent posts, you are still able to see the earlier posts which are not in the RSS feed! <<

I don't use Google Reader, but I suspect that you're seeing the benefit of somebody else having subscribed to the Slashdot feed earlier. Google Reader can cache blog entries from whenever the first person subscribed, and pass these on to new subscribers.
Mark Pearce Send private email
Wednesday, April 09, 2008
 
 
> pass these on to new subscribers.

Ah! Maybe...

On a side note, Google reader is very clunky compared to the desktop RSS readers. I am surprised it's all the rage.
all the feed
Wednesday, April 09, 2008
 
 
"On a side note, Google reader is very clunky compared to the desktop RSS readers. I am surprised it's all the rage."

Well, I like simple apps, I don't feel I need anything else, I don't care much about complex RSS functionality (I just want to know if there's something new in the blogs I follow). I haven't investigated a single desktop RSS reader. I imagine many people are in the same situation. We just don't investigate what we're not keenly interested in.

For example there may be a better Calendar than Google's but for my needs and interests I really don't care. I want simplicity.
Daniel_DL Send private email
Thursday, April 10, 2008
 
 
> I want simplicity.

That's not what I would call "simple". The UI design is awkward, the interface is slow, you can't even delete and get rid of a post for god's sake!

It's "thrown together" at best.
all the feed
Thursday, April 10, 2008
 
 
"The UI design is awkward"

It is very consistent with another Google app I'm used to: Gmail.

"the interface is slow"

Not to me by any means.

"you can't even delete and get rid of a post for god's sake!"

Well, _that_ is simple :) I don't want to delete anything. It's a reader. I suck in dozens of posts every day and I only read some. The others stay there and once in a while I mark them all as read just to have more peace of mind. But an aggregator's purpose is (to me) to see what's new in a number of places, not to do a complex anylisis of them. In a newspaper or at CNN.com, I don't delete the news I'm not interested in. They're just there. On TV, I don't delete the shows I don't like. They just happen.

That's the difference between communication and broadcasting. Sometimes I delete _my_ blog posts if I get tired of them. What other people post in their blogs, I just consume it as a spectator. It just happens. The world keeps spinning and I'm using an RSS Reader integrated with other basic web functionality I like from Google.

But go ahead and use what you like best :)
Daniel_DL Send private email
Thursday, April 10, 2008
 
 
> It is very consistent with another Google app I'm used
> to: Gmail.

Gmail, I use and "get". It has a very slick interface. I wish everyone else mimicked them.

Google reader on the other hand is just crappy. Example: Why do I have to switch to "Preferences" to delete a subscription?! I can add them from the left panel. Why can I not delete them from there too? I guess Google doesn't want me to delete subscriptions.  :)


> What other people post in their blogs, I just consume
> it as a spectator. It just happens.

This behavior is much different than mine, so I can see why you would like Google Reader.
all the feed
Thursday, April 10, 2008
 
 
"I can add them from the left panel. Why can I not delete them from there too? I guess Google doesn't want me to delete subscriptions.  :)"

Actually, it wants you to :)

1. Select one individual subscription on the left panel
2. Feed settings... menu button | Unsubscribe

"This behavior is much different than mine, so I can see why you would like Google Reader."

Different uses, different tools!
Daniel_DL Send private email
Thursday, April 10, 2008
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz