September 10, 2004

Scoble on RSS Aggregators Frequency of Polling

Robert looks at how often aggregators poll for new posts, figures out that we are heading for a mess of bandwidth issues what with all the aggregators constantly going looking for news posts and other content across blogs and websites. He asks what we might do to solve this and how often different aggregators update. Many of those listed in the comments look for new posts hourly. And many of the commenters suggest that sites should only serve changes or give a 403.

But, I was thinking, what about smarter aggregators that learned the posting patterns of the writers? How about aggregators that got a sense for the frequency of posting, and then polled at that rate? I tend to post once a day, often in the morning, because that when I have time to write. Robert posts practically in his sleep. Why can't my aggregator poll a blog like mine once or twice a day, and Robert's hourly? And Jay Rosen about once every 5 days or +5 days after his last post since he tends to do long essays about that often?

Rather than trying to engineer just based on changes to the website content, why not get smart about people's social patterns and then users can do a complete auto-refresh if something comes up and there's a conscious need....

This is social media after all, and it's not just about the tools, but the social interaction... we have the engineering part figured out somewhat as far as syndication and aggregation, but where is consideration for the social interaction built into the tools... if we are to be polite about our polling, we ought to think about how the other end works (writers - people) and react accordingly in the makeup of the tools.

Oh, and one other thought, what about using the pings put out by blogs with new content? Why not have my aggregator subscribe to ping-o-matic or whatever, so that when pings come into ping-o-matic, pings go out to subscribers like a dinner bell. Content that's good and good for ya. MMMmmm.

Posted by Mary Hodder at September 10, 2004 06:17 PM | TrackBack

I wonder if server-side aggregators like Bloglines help this... they could have one or two bots checking hourly no matter how many people subscribe to Bloglines.

Posted by: joe at September 11, 2004 10:49 AM