November 10, 2005

Full Disclosure

Over the last month, a lot of people asked about the fact that I'm an advisor to Sphere.

I have wanted to blog about this, for the sake of transparency, for this period, but out of consideration for others, had held off. But transparency is something I believe in and strongly believe needs to be done sooner rather than later.

During the 4 years I was at UC Berkeley, I spent all of it doing some sort of research and development on news and blog systems. In particular, I spent a couple of years working on the topic browsing of blog data, developing a front-end php-based system, 150 pages of usability, user and design research, preliminary algorithm design for determining topic communities through multiple metrics, scoping search up and down those communities, as well as weighting bloggers through other combinations of those same metrics (multiple metrics reduce the power law effects that a single metric can have on a community), as well as blogger profiling.

Additionally, before the topic browsing system, during my first couple of years at Berkeley, I also developed a news-blog system (interface, business plan, usability testing, and marketing research about online ad systems that could be developed to complement either the blog-news system or later, the topic browsing system).

When I was hired at Technorati, after graduating from Berkeley, I showed them the interface for the topic browsing and profiling systems. While they indicated they liked it, they also said the system was too hard to build, which I took to mean that it would take too many resources away from other commitments. We moved on to building other things there, and when I left Technorati later, I had a conversation with Dave Sifry that lasted hours. We discussed building products and companies, and at the end, I said I was still very committed to seeing my topic browsing research borne out. Dave told me that if I had any desire at all to build a company, I should absolutely go do it. I said I was thinking about, or I might find someone else to build it, but if any of that happened, I might want to license his data. He said we could talk about it then.

I was asked as I was leaving to be on Technorati's Advisory Board.

In the meantime, I realized I did not want to spend another four or five years continuing to work on topic browsing. Instead, I was committed to working on the video project I am developing now (I was a filmmaker for almost 7 years and love the idea of applying that knowledge to my technology development). But I was still committed to seeing my topic system research used, as it represents thousands of hours of work, and so I set about looking for someone else interested in building it.

I've talked with a couple of projects and companies over the past year, but each of them suggested that building topic browsing right was tough, and they had other commitments which most of their engineering staff were working toward. Sphere appeared this summer, wanting to build exactly what I'd done my research in at Berkeley. And so, I gave them all the work to use, and they made me an advisor.

Sphere has been working on blog search, though I said to them early on that I was just advising them on creating topic communities and so haven't yet looked their new search interface that debuted a month ago. In fact, they showed me early work and I told them I could not help with blog search because of my relationship with Technorati as an advisor.

I believe that blog search, at this point, is a baseline for any company in the space. You have to do it. But it's not so interesting to me, compared to making a leap forward toward something like topic browsing of communities or sophisticated weighting of bloggers. I'm less interested in 'yet another blog search' tool. The ones we had already were fine. But I've very interested in what Sphere is really here for: changing the ways we can view small topic communities and the bloggers within them in sophisticated ways that take us ahead of where we are now, which I equate to the place websearch was in in 1997, before Google.

In the interest of reducing confusion, however, I decided it made sense for me to resign from Technorati's Advisory Board. Many people have asked what happened, and I felt it was better to state it publicly, than not to alleviate confusion.

Technorati is still a new company and doing lots of interesting things. I wish them all the best. My decision to stick with advising Sphere is based on their heading down a path that's alongside my own.

So, now that this is out, I expect soon to post the rest of my blog search comparisons, as well as a couple of articles I've been working on around blogging and identity, rankism, etc.

Posted by Mary Hodder at November 10, 2005 08:10 AM | TrackBack

Do you have a link to your original topic community paper?

Posted by: Dan at November 10, 2005 03:47 PM

No.. actually I haven't published it online and won't until the people building from it finish. I've also given it to two other companies who are doing things outside blog space. It turns out that research is valuable in several contexts.

After they are finished, I'll put it up. But thanks for asking.

Posted by: mary hodder at November 10, 2005 11:21 PM

Dan wrote: "Do you have a link to your original topic community paper?"

im intresting too.

greets from Berlin, Germany

Posted by: seekXL at November 14, 2005 06:45 AM

Hi, Mary -

I hope you don't mind a driveby comment. I was doing some Technorati research to try to figure out how rankism factors into the social media conversation, and it seems to me that you were the one who got the ball rolling on that. I then used Technorati's authority slider to get different views on the posts. According to Technorati, your blog has "a lot of authority". I thought you'd get a kick out of the irony of that.

I'm not sure if you're aware that Robert Fuller, the person who coined the idea of "rankism", lives in Berkeley. He doesn't wade that deeply into social media issues, but it's pretty easy to strike up a conversation with him if you're ever curious about his general ideas about overcoming rankism.

Posted by: Elisa at April 27, 2006 07:55 PM