July 26, 2005

For the Vox Populi, Part II: A Comparison of How Some Blog Aggregation and RSS Search Tools Work for Keyword Search

This is part II continuing from a post on URL search I did yesterday. It follows a post on how URL search and link counts are done by Bloglines (as an information search tool, not a news reader tool), Blogpulse, Feedster, Pubsub and Technorati. The third will cover subscription search (watchlist) performance, the fourth will look at special services and the fifth will look at spam and controls for it. The sixth will summarize and make recommendations about how to best use the services.

Keyword search

Keyword search is very different than URL search, from the user perspective.

URL lookups are more straightforward in terms of user motivation: a user looks up a URL to see everything that links to it. No matter the search motivation, the user still wants to see all links, and all five blog aggregation tools that I reviewed in the last post, give results in reverse chronological order, with some kind of link count. So whether users are doing an ego surf (looking up their own URLs) or checking out who links to a client, their own company, the New York Times or the post of a blogger they are interested in checking influence or conversationalness, the results in that chronological form satisfy these different needs, together with the number of links, at least at first. It may be that a user then wants to sort the results to see more 'authoritative' posts first, from the results list, but that is more often a secondary need, and not currently offered by any of these five services. Since blogging and RSS search are very much about the 'aliveness' of the activity, serving results in reverse chron order does satisfy most user needs (this observation is based on user testing I did that looked at search results on URLs, keywords and topic browsing -- the last of which doesn't exist yet, though I built a front end system that did it, and tested it repeatedly with users).

Conversely, keyword search is harder to get right in terms of user expectations. First, user motivations for searching are different and spread more evenly across a more complicated set of goals than with URL search. Does the user want a quick taste of what is out there around a particular topic? Or do they want every instance of a keyword match, with an accurate count of those instances? Do they want to see only the most relevant posts that use the keywords, or the most recent? How each results is laid out is also an issue in satisfying user's search needs, because we have all been trained to map our information expectations, consciously or not, to Google keyword search results. They have us viewing keyword search results in very particular ways. And yet, the part of the web that is alive is very different than that which is static. So keyword search results for live data are naturally different, and yet we find ourselves first looking through our Google eyes that see relevance first, before we refocus on the live, and chronologically ordered results, that may be formated a little differently.

Below is an image with screenshots of Google, in order to compare it to Technorati, Feedster, Blogpulse and Bloglines (again, Pubsub does not have historical search, so they have no results to compare for this test). Each of the four blog search tools were captured with the first result for the term: napsterization. Google results give context about the number of results, but there is no indication of time. All four blog search results note the number of matches they see, but also include the time of a post. Bloglines has a "subscribe" button to pull the feed into its newsreader and Blogpulse and Feedster offer XML buttons for putting the feed into the user's newsreader. Blogpulse offers to track the conversation and provides a profile of the blogger for additional context.

But note the format of all of them. Everyone maps to a format that places a link at the top of the entry that corresponds to the except below with the match. Feedster and Technorati bold the search term as does Google. Bloglines and Blogpulse don't. Below the matching excerpt, they all place a link to the blog, and follow with context they find relevant. Google shows the size of the page, the others follow with dates and times of posting, and Technorati gives the number of links to the blog as an indicator of authority.

But the point is, blog search results are similar to web search results, with some additional information presented. The order of the results is different, though, in an attempt to meet most users' expectations and goals with the information, and more closely match the results with what is interesting about blog information. Google serves what they believe is the most relevant information, based on page rank. Blog search companies give what they see as most relevant, which are results are in reverse chronological order.

The picture below is simply a way to visually compare results:


Additionally, I've included another PDF chart comparing how the five blog search companies handle keyword search. Note that Pubsub does prospective search, so their value is in feeds of keyword searches an not historical information. I'll also update this post and chart if additional information comes out, as I've been doing with my first post in this series. Future posts will summarize and give more sense of the value of these search tools, but I'm focusing on how these things are done, and more long terms results of search quality here.

KeyWord Search Comparison Chart

Comparing search completeness, timeliness and cleanness

Right now, Doc Searls is running a short term look at blog search, including URLs and Keywords: The Sam Test and Your Findage May Vary. He is looking for the speed and inclusiveness of delivered results across several services. But comparing just the services I can review with historical search, I can see that Blogpulse, Feedster and Technorati seem to have very high, accurate results, with little duplication. Feedster has more results in the same period (48 hours) because they mix del.icio.us results into their search results instead of separating them as Technorati does. I'll be curious to see what the counts are, based on Doc's criteria. But my own counts show about 45 results talk about this in the past couple of days, and it appears that Blogpulse, Feedster and Technorati each have almost all the results, and Doc reports that Technorati was able to grab them the fastest.

Compare this to a very specific survey I did last fall that encompassed a week of results (not dependent on the speed posts were pulls into each search service but rather looking at duplication and coverage of the activity across posts). In that survey, Blogpulse won hands down. But since then, Technorati has redone keyword search, removed the 7 day result limit, and while results only go back to last October (the start of the redone database) their new keyword search is doing very well and shows dramatic improvement. In Doc's recent test, Technorati wins in the speed/coverage test. In my tests, which were based on completeness and cleanness, Technorati and Blogpulse both did well, but if it's true that Technorati was faster, I didn't notice because enough time had passed that Blogpulse had parity with similarly complete results and as clean of results. Also, Technorati has reduced duplication over the past 10 months, bringing it more in line with Blogpulse's cleaner result set from last year. But it appears from Doc's test that Technorati is now faster than Blogpulse, a few hours after an event, to pull in posts. Overall, Technorati's keyword service shows marked improvement and does very well.

The best interface, with most interesting options again has to go to Blogpulse, which has links to tracking the conversation (links to a post instead of just links to a blogger) and profiles for bloggers, as well as a clean easy-to-read style.

Additionally, in various test searches, Feedster still appears to mix blog posts, top down news and del.icio.us data together, though it did not appear to be as much of a problem as it was 10 months ago. Depending on your goals, this practics may or may not be helpful. And as the keyword survey below shows, it was not helpful for that particular use case, because repeated Associated Press articles cluttered the results so badly. However, Feedster did get the originating post in last year's survey, which was difficult, and no other service got.

In contrast to the current tests, below is the 10 month old survey of two searches for: kryptonite lock, and kryptonite bike lock, across 4 blog search services. Searches compared results to see which service provided the best user experience and results. The user task was to grok where this meme had started and to get a sense of what people were saying.

Search Comparison Summary dated 9/19/04

Kryptonite locks became a major story for bloggers this past week, when a bike rider (whose bike was stolen because his kryptonite lock was compromised with a bic pen), made a video of how to pick the lock. Bloggers picked up the story with speed by Tuesday 9/14/04, though things appear to have originated from a biking forum post from 9/12/04 that then was blogged on 9/13/04 (appears to be this one). That video appears on many blogs starting around Tuesday of this week. So keyword search for "kryptonite lock" and "kryptonite bike lock" (searches were NOT done with quotes) works across all services to find out what's going on with this story in the blogosphere. Also, by 9/17/04 (Thursday), the story appeared in newspapers across the country, both original stories by those news outlets, as well as by AP and Reuters stories.

All four searches returned results in reverse chronological order. Only Feedster got the originating blog post from Anarchocyclist. Blogpulse had the cleanest result set, with the fullest set of returns, though not complete. Many of the major blogger's posts were missing from the Blogpulse set, but they were also missing from Technorati's returns.

Blogpulse had by far the cleanest results sets (only three duplicate posts out of 160 and 241), cleanest and easiest to read presentation, no top down news posts (Feedster had around 70% of results from AP, Reuters and other news services and it was hard to distinguish them), so groking the blogosphere's take on this topic was easiest with them. Blogpulse had all results listed by blog post title, hyperlinked to the posts, that were spotchecked. Blogpulse gave the best overall experience and returned results data, despite missing some posts.

When blogs posts across the four services were isolated from the news stories that were listed in say, Feedster, as just another post, all four services did give a picture of what was happening across blogs regarding people's thoughts about the locks. However, the easiest and fastest way to do that was on Blogpulse as stated. Feedster's extreme mixing of top down news stories with blog posts may satisfy some searchers, because the order is most recent first and the news stories are more recent (followed by blog posts). But the actual experience was that the Feedster returns for this kind of search where results might produce both types of entries was not great, because the results pages resembled Yahoo news, and therefore getting the blogosphere take was much more difficult.

Technorati is giving a suboptimal experience, both because of the limited results returned due to forced phrase search, the extreme amount of duplication of posts, the lack of title hyperlinks that returned the correct post every single time, and the presentation.

Results in detail:

1. Technorati returns 41 results for -- kryptonite lock -- and 18 results for -- kryptonite bike lock --

Of the 41 results, 40 are actually shown (41 was somehow missing), 2 were exact repeats of nanopublishing.weblogs.inc and 12 were from badassgeek.com from "1 day 13 hours ago" and 9 were from "20 hours and 8 minutes ago." Those 21 "posts" were identical and obvious by skimming through them. The net result was that of the 40 shown posts, 19 were unique and 21 were repeats of 2 of the 19 unique posts. Because the search default is for phrases, these searches, not put into quotes, still return results in quotes and this severely limits the value of the search. As an example, the originating blog post that appears to have started this meme in the blogosphere uses the term: kryptonite U-locks, and Technorati's search would not therefore pick up this post for the phrase: kryptonite lock, though other search services did. Just to test, kryptonite U-locks did not return the Anarchocyclist post from 9/13, which it should have, but it did bring up 3 results that did not appear in the other two searches.

Earliest posts are from 2 or 3 days ago for the 'kryptonite bike lock' search and 4 to 5 days ago for 'kryptonite lock' search because of the phase search limitations. Posts do not show where the meme originated, and start well into when the blogosphere discovered this issue. Posts are shown 20 per page so paging through was not onerous.

2. Feedster returns 410 results for -- kryptonite bike lock - and 663 results for -- kryptonite lock

300 of the results for kryptonite bike lock were from Tuesday 9/14 forward and all posts before this were not about the bike locking picking issue. Feedster is also showing major news stories mixed with blog posts, and the only way to tell is either from the icon they use: a large blue square with a white "i" in the middle, or by reading each source. However, they use this for many informational blog posts as well, and so reading the source became a time consuming task to figure out what is coming from a blog and what comes from top down media. In fact, more than 70% of the "posts" were actually mainstream media articles, and if the goal was to find out what the blogosphere was saying, on an issue that appears in both places with lots of "press," Feedster was making this goal difficult. In fact the returned results looked very similar to a search on news.yahoo.com for the same terms. Percentages were similar for the 663 returned results on kryptonite lock. Feedster allows any instances of the searched words appearing in any order to be included in the returned results. Spotchecking post title links produced some posts that were not the ones Feedster returned on results pages.

Earliest posts go back months because they do not limit search to 7 days, but the older posts are not about the lock picking story. Posts were shown 10 per page, so paging through 30-50 pages was onerous to get to the beginning of the blog meme.

3. Bloglines returned 212 results for -- kryptonite lock and 129 for kryptonite bike lock

Numerous duplicates of posts were found. 4 of 1 post, 12 of another were duplicates -- because there were so many and so many pages of posts, I didn't do an exact count, but every page of results had numerous repeat posts, however, one post from Slashdot had over 40 entries as individual posts, and I would guess 40-50% of the entries were duplicates. Also, Bloglines has some AP and Reuters stories mixed into results, though not the 70% returned results that Feedster did. Bloglines had about 25% news stories, though they did not distinguish between these "posts" and blog posts, and it was even more difficult to tell them apart than the Feedster results. Bloglines allows any instances of the words appearing in any order to be counted as a result. Spot checking the title hyperlinks produced posts that were not the ones Bloglines listed.

Bloglines first story is from Eyebeam on Tuesday 9/14/05 for a blogpost on breaking the locks with a pen. This was not the first post, but was very early in the meme. Posts are shown 20 per page so paging through was not onerous.

4. Blogpulse returns 241 for kryptonite lock and 160 kryptonite bike lock Blogpulse had very few duplicate posts (maybe 3 which was amazing compared to the other three services), and no top down news stories. 143 of the 160 returned results for kryptonite bike lock, and 224 of the 241 posts for kryptonite lock were all blog posts, all on the bic pen topic, all matching their respective searches. Presentation was clean and easiest to read. Blogpulse had all results listed by blog post title, hyperlinked to the posts. All spotchecked post titles were correctly linked to the actual post returned in their results.

Blogpulse did not get the originating post for this meme, nor did the search for Kryptonite Bike Locks get the Engadget posts that kicked off the top down news and many of the hundred or so blog posts that followed this week. However the kryptonite lock search did pull the Engadget posts. Lots of Live Journal results; maybe 50% of entries are from Live Journal bloggers.

Earliest posts go back months before the bic pen results. Blogpulse returns 10 posts per page, so paging through 16 and 24 pages was a bit onerous.

Posted by Mary Hodder at July 26, 2005 11:36 PM | TrackBack

Your thoughtful analysis are much appreciated ...

Another great place to start with comparisons is MrSapo


Posted by: Jozef Imrich at July 29, 2005 11:29 PM