Live Web Search

I'm giving a talk late this afternoon on live web search in Marti Hearst's Search Engines: Technology, Society, and Business class at SIMS. I'll be talking about how the live web alters static web search, how live web search works and what sorts of things we would like to have to make search better in the future.

Update: Here are the slides (or in pdf) and link to the webcast.

The Woes of the Legacy Media Company in the 21st Century

Sony, which is selling CD's with a backdoor in software the CD's install semi-secretly, has also designed that software using code that's under LGPL license, but has not put the code back out to the community as the license requires.

This is according to a Dutch article (english here) and Slashdot.

Violating your privacy and the LGPL agreement. It's so hard navigating the modern world. Especially if you are a legacy company. Oy.

Tagging by Bloggers, a Small Study

Last February, I conducted a series of interviews with 20 bloggers. 10 were using what was then the 'new' Technorati Tag implementation and 10 were not. All were regular users of tags in other systems: del.icio.us or Flickr types of systems where a tag is applied by a user, and then the tags can be 'pivoted' on, or clicked on, to see all the other media that has the same tag. This pivot can occur either at the user level, to see everything that user has applied that tag to, or at a global level, to see everything where the entire community at that site has applied that tag. Additionally, these bloggers were beginning to use the Technorati tag pages to see how bloggers were tagging their posts or media, and how the Flickr or del.icio.us media with the same tags fit together on the Technorati page with those tagged blog posts.

I sent out email to these bloggers, essentially asking how they used tags in their posts, or how they wanted to use them, if they did not and yet wanted to, or if they didn't want to at all, why.

All of these bloggers were people I did not know before the interviews, but were semi-randomly chosen because they blogged with tags, or appeared to be aware of tags because they had links to sites with tags, though they did not tag their posts themselves. These interviews were done in email, and were conducted over a couple of days. I promised those interviewed that I would not pass on identifying information about them, or list them in any way. I also promised to aggregate the information before blogging or passing the survey on to others.

Of the ten who use the new rel="tag" format in their blog posts, most were dissatisfied to one degree or another. Sometimes, the Technorati system did not pick up their tags, even though their posts could be found in the search system in other pages besides tag pages, and they were using the correct format. Some also expressed similar desires to be able to do more with the tags and formats, as the non-tagging bloggers did. All expressed some desire to have more of something.

Those interviewed talked about the following that would help them tag or cause them to want to tag when they were not doing so now:
1. A desire to create tags in their blogging software in similar ways to how they create 'categories' -- meaning they wanted to use a pulldown menu or something with similar ease, to quickly tag a post. This included the desire to have tags be invisible on their blog pages, as some of them have invisible categories in their posts. Some of those with invisible categories at the post level still have category searches visible at the sidebar level of their blogs. They would be interested in showing tags at the sidebar location, if they choose. But all felt these choices of visibility and invisibility at various points in their blog posts and overall blogs should be left to them as it is now with categories, and those choices should not bar them from participating in Technorati's tag program.
2. These bloggers rarely added new categories to their blogs, and saw the value of having large buckets to categorize their posts, and didn't want to add new categories all the time. Partly this was due to how difficult the software make it to add categories, and partly this was due to seeing in practice that there was value to 'large bucket' categories, and 'little context' tags. These small tags were desirable because they could be applied to a post on just one time, but categories would come up at least every few days.
3. These bloggers all understood the meaning of a link in their posts. They knew the value of those links, and thought carefully about where they pointed in posts before doing so. They did not like being forced to put a link to something in their tags, if they were not so inclined. They would prefer to have the choice to make a link or not make a link, depending on the circumstances of the post.
4. If a link was placed in a tag, at their choosing, they wanted more flexibility to choose where the link went, beyond Technorati's tags pages, Flickr or del.icio.us. Many did not like that in order to make the tag, they had to place a link, and then because they wanted to make links that 'made sense' to their readers (the links would 'go somewhere'), they felt forced to repeatedly link to these same couple of sites. Some wanted to be able to easily make their 'own tag pages.' Some wanted to be able to link to other places besides tagging sites, that had some meaning to them. And some asked to be able to link somewhere, and tag the link, and have that be understood to their readers and the systems that would pick up those tags.
5. A little less than half of these bloggers asked to be able to tag a specific object in their blog posts. They regularly posted photos, either their own, or brought in some code from another site to repost the photo, with or without text around that photo. They wanted to be able to tag just the photo in their post, but tag the post at the bottom of the post, following Technorati's directions for tagging.
6. A couple of the respondents said they would like to be able to tag comments from readers of their blogs, and they might consider, if they have registered commenters, allowing those commenters to tag posts, objects or other comments as well.
7. All of them liked the idea that tagging would allow them to participate in a community, but they wanted to control that participation themselves, at the publishing level of a post.

Additionally, the ten who were not tagging were asked why. Answers ranged from,
"I can't figure out how to do it using Technorati's directions." (3)
"I don't want to until they change it so I don't have to link."
".. it's too hard."
"I want more choices and flexibility"
"I haven't had the time to figure out the system and it seems hard."
"I hate the link part. I don't want to do that."
"I want my tags to be invisible." (2)
"I don't want to give Technorati more links." (3)

My conclusion at the end of this was that we need more flexibility and better tools for tagging at the blog post level, including creating tags at several points: around an object, at the link level, at the post level, and possibly at the comment level. Additionally, users need easy ways to tag, and set their preferences around how their tags will appear, or be created. And they need flexibility for linking within the tags, as well as some different choices depending on what they were doing, about how the tags will function. It would be great if they had some easy way to make their own tag pages, as they now can do with categories that allow them to build pages of posts that all have a common category.

Note: afterwards, I did let them know that they could link to sites other than Technorati, if they were not aware of this, and I did help those who could not figure out how to do tagging. But it was clear to me that part of solving these issues is communicating to users better and making some simple tools so that users don't have to play around with html code, and wonder if they are getting it right.

Disclosure: I used to work at Technorati (well before this study) and be on their Advisory Board.

Kevin Mark's Brilliant Idea

Kevin suggests, in response to my earlier piece on the terminology of Users and Consumers, that we instead bring back the original meaning of 'amateurs.' Well, I love this:

We already have a word for people who create for the love of it, rather than being paid to, and it is 'amateurs'. As with many other pleasures, when we seek out opinions, we prefer those that flow from passion rather than from payment.

Now it may be argued that, given the decline in the teaching Latin and French, the loving root of 'amateur' is no longer perceived, so those who write pour l'amour ou pour le sport may see 'amateur' as a slight. In which case lets retranslate it to english and call it 'lovingly created media'.

Fantastic. Because it means we take back from the concept of 'professionals' the notion that 'good = professional." Instead we claim the aspects of our experience through creation that are so humanly, actively ours to own and enjoy, as unpaid creators. 'Amateur' has been derogatorily used to convey 'less than' status. Sometimes one or another works is less than, but it is not due to whether or not someone is paid for their work. A work should be judged lesser or greater because of its intrinsic qualities and value to those who apprehend it.

So, take the label 'amateur creator' as a point of pride. It means you create for love, and not for money.

Oh. Btw. Just in case you think there is confusion about the other term in my earlier post. 'Consumer' is still unacceptable in so many ways. Don't use it.

Full Disclosure

Over the last month, a lot of people asked about the fact that I'm an advisor to Sphere.

I have wanted to blog about this, for the sake of transparency, for this period, but out of consideration for others, had held off. But transparency is something I believe in and strongly believe needs to be done sooner rather than later.

During the 4 years I was at UC Berkeley, I spent all of it doing some sort of research and development on news and blog systems. In particular, I spent a couple of years working on the topic browsing of blog data, developing a front-end php-based system, 150 pages of usability, user and design research, preliminary algorithm design for determining topic communities through multiple metrics, scoping search up and down those communities, as well as weighting bloggers through other combinations of those same metrics (multiple metrics reduce the power law effects that a single metric can have on a community), as well as blogger profiling.

Additionally, before the topic browsing system, during my first couple of years at Berkeley, I also developed a news-blog system (interface, business plan, usability testing, and marketing research about online ad systems that could be developed to complement either the blog-news system or later, the topic browsing system).

When I was hired at Technorati, after graduating from Berkeley, I showed them the interface for the topic browsing and profiling systems. While they indicated they liked it, they also said the system was too hard to build, which I took to mean that it would take too many resources away from other commitments. We moved on to building other things there, and when I left Technorati later, I had a conversation with Dave Sifry that lasted hours. We discussed building products and companies, and at the end, I said I was still very committed to seeing my topic browsing research borne out. Dave told me that if I had any desire at all to build a company, I should absolutely go do it. I said I was thinking about, or I might find someone else to build it, but if any of that happened, I might want to license his data. He said we could talk about it then.

I was asked as I was leaving to be on Technorati's Advisory Board.

In the meantime, I realized I did not want to spend another four or five years continuing to work on topic browsing. Instead, I was committed to working on the video project I am developing now (I was a filmmaker for almost 7 years and love the idea of applying that knowledge to my technology development). But I was still committed to seeing my topic system research used, as it represents thousands of hours of work, and so I set about looking for someone else interested in building it.

I've talked with a couple of projects and companies over the past year, but each of them suggested that building topic browsing right was tough, and they had other commitments which most of their engineering staff were working toward. Sphere appeared this summer, wanting to build exactly what I'd done my research in at Berkeley. And so, I gave them all the work to use, and they made me an advisor.

Sphere has been working on blog search, though I said to them early on that I was just advising them on creating topic communities and so haven't yet looked their new search interface that debuted a month ago. In fact, they showed me early work and I told them I could not help with blog search because of my relationship with Technorati as an advisor.

I believe that blog search, at this point, is a baseline for any company in the space. You have to do it. But it's not so interesting to me, compared to making a leap forward toward something like topic browsing of communities or sophisticated weighting of bloggers. I'm less interested in 'yet another blog search' tool. The ones we had already were fine. But I've very interested in what Sphere is really here for: changing the ways we can view small topic communities and the bloggers within them in sophisticated ways that take us ahead of where we are now, which I equate to the place websearch was in in 1997, before Google.

In the interest of reducing confusion, however, I decided it made sense for me to resign from Technorati's Advisory Board. Many people have asked what happened, and I felt it was better to state it publicly, than not to alleviate confusion.

Technorati is still a new company and doing lots of interesting things. I wish them all the best. My decision to stick with advising Sphere is based on their heading down a path that's alongside my own.

So, now that this is out, I expect soon to post the rest of my blog search comparisons, as well as a couple of articles I've been working on around blogging and identity, rankism, etc.

Intelliseek: Using the Outdated Terminology... It's Users, Not Consumers

Updated: I've added this post on using the term, 'Amateur' to describe those who 'lovingly create media' instead of professional. Check it out as well. And it was Kevin Marks idea.

Yesterday, Pete Blackshaw and Intelliseek put out this press release:

Consumer-Generated Media Exceeds Traditional Advertising for Influencing Consumer Behavior, Finds Intelliseek Study

Where they said, "Consumers 50% more likely to be influenced by word-of-mouth behavior than radio/TV ads, says 2005 Intelliseek research of consumer behavior."

That's all very nice, and something we've all been seeing online, and participating in for years. But consumers?

No.

We are not creating our own media, writing blogs (Intelliseek owns Blogpulse, a blog search product), in some cases creating our own products, as 'consumers.' We are *users* with a proactive capital U.

Users are people who go out, find stuff they like, publish, remix and create a new. They are smart, they are proactive. They don't take being marketed to, but would rather either discover or get more real information from people they trust. Users have been operating digitally since the advent of the internet.

Consumers are those whose mouths are wide open, pointed toward the sky, so they can't see what's going on, like baby birds, helpless and clueless and waiting to be marketed to, while information or products are spoon-fed to them by marketers. Consumers are so 1980's.

It's Users. Get your ticket now on the cluetrain. If you keep talking about consumers, the users will pass you by as they take control of their own media, product interests and activities.

Jerry Reynolds, (former) Spam King (of Fargo, ND) Files Slapp Suits Trying to Squash Free Speech by People Exposing His Spammy Ways

He's apparently filed (complaint here) SLAPP suits against two people, David Ritz and Ed Falk, who found that in the 1990's, he was the largest spammer online (email wasn't so big then, but he had the largest porn spam operation on UUnet, with Sexzilla and Netzilla which were registered to Jerry Reynolds). And now he's using C&D's and these lawsuits to get whatever traces of the information that documents his spam and porn operation off of Google including search results and groups. He denies owning the site, btw, even though he was listed as administrative contact.

Reynolds has even gone so far as to subpoena Ritz' and Falk's computers and put a gag order on one, but the other one is out of jurisdiction (the suit is in North Dakota, though Falk lives and does business in CA).

Tomorrow, Ed Falk has to give a copy of all his computer harddrives to lawyers, who are still fighting over the jurisdiction issue. I think though that if this case went before a judge in CA, it would immediately be dismissed. All the evidence shows Reynolds was the SpamKing in 1999. Unfortunately, though, the case won't be resolved before Reynolds costs these guys thousands of dollars, and not before he has stopped at least Ritz from speaking out about the case.

The most recent lawsuits and C&D's have been filed by John Doe's or by Reynold's company, Sierra Corporate Designs.

The thing that is so despicable about this is not that he was once the biggest spammer and has moved on to other sorts of (legitimate?) business, but rather, that he would use the courts to have a few pieces of information about his old spam work removed, as if he were trying to rewrite history and squash people's rights to free speech. He was tracked as a spammer early in the 90's, but hit a peak in 1997 with porn spam, at least as far as UUNet / Usenet was concerned. He actually helped kill that community by turning it into a garbage heap for spam. And now he wants to evade responsibility for it. And in the process, cost Ritz and Falk a lot of money and time, defending the truth. Disgusting!

Read more about the lawsuit here.