Information Technology meets Medical: Why We Should All Be a Little Worried

Today I had what I would say was an anecdotal experience regarding data privacy.. calling my OBGYN to make my annual appointment. I ended up using their new website and giving various personal data, only to figure out that they have no privacy policy for data, that the data was going to a third party, and that in trying to make an online appointment, all I really got after sharing data was an email form to request an appointment.

So, here's the scoop.

In calling into the doctor's office, I got their voice system which has always required lots of number punching to finally get through to someone to make an appointment. It's better than 10 years ago where you could literally never talk to anyone in their offices and would just punch numbers endlessly until leaving them a message. That would be followed by a return call that you would invariably miss, having to start the process over, to get another call back.. all to just make an appointment.

Anyway, calling in today only requires two selections, before being told my call was in line to be picked up after approximately 6 minutes of estimated wait, OR I could use their online system. Whooppee! I could make an appointment using what I imagined was a calendar with available timeslots to book appointments? So here is Golden Gate Obstetrics (GGObgyn) big chance to show how they are using information technology to help people organize this process of getting an appointment better and faster!

Super cool!

Er... NOT. So. Fast.

Following the voice system at GGObgyn, I go to http://goldengateobgyn.medem.com/ which redirects me to http://www.ggobgyn.mymedfusion.com/:

The branding all over the site is "Golden Gate Obstetrics" so I'm thinking: okay, this is their site, even though it's got some other root domain name (mymedfusion.com).. in other words, Golden Gate Obstetrics is responsible for my health info, and I just need to get in to see their calendar and choose a time or something. So I go to "create an account" (Note below I've made screen shots of the *second* account I made, called 'testacct' to see what was going on a second time.. since the first time when I made an account for myself, it went by quickly and I wasn't suspicious until the end of the very end of the process):

I put in my name, SS # and DOB and email. After submitting, I was brought to this form (screenshots are in two parts as it was a longer page):

As you can see, there's enough data request there for someone to do some damage if they wanted to. At this point I was getting a little concerned about where this data was going, but keeping in mind GGObgyn's history where getting staff on the phone to make appointments is so difficult, I went ahead and submitted my data.

The screen instantly took me to a logged in state, saying "we are now your Health Record provider" which I found totally freaky. I don't want them to be my Health Record provider. I just want to schedule an appointment. All this, without requesting any sort of email verification or other checking... just gave me an account. At that point, I could go make an appointment:

To say the least, I was shocked. So I just put in all this personal information, dinked around with forms etc, to be given a glorified email form to request an appointment? With structured data about which day of the week I want the appointment? How about a calendar with available time slots? So I could just pick based upon my availability? No... it appears they are going to email me back or call me with times so we could go back and forth over schedules again, in email? Really? This is the promise of information technology for scheduling? I mean aside from the privacy issues, I really felt like I'd been had in terms of my time sink for their silly email form.

I notice there is no help or privacy statement on any of the pages in their system (and I clicked on all of them), and the "ask a question" page is all about medical stuff, not using the website. But I figure GGObgyn is responsible for this site. So I call them, and after a lengthy wait, get the appointment receptionist. And I ask, where did my data go? And she says she doesn't know, but they own the site, so therefore my data is safe.

This seemed reasonable given the interface on the GGObgyn website was so incomplete with so many important things missing (like a privacy statement as I entered in my SS # and DOB and address, etc. or even a privacy policy in the footer somewhere, or a help page, or real contact info), it had to have been done by people who don't normally develop websites.

I asked if the receptionist could give me the privacy policy, or tell me where my data had gone, and she said she would pass me to the "online manager" named Olivia. Olivia started off my telling me she sits on the system "all day long... as account requests from users to join their online system appear on my screen.. I look the patient up and put through the approval if the new user is in fact a patient."

ME: "Really? because my account approval seemed instantaneously to happen on my screen."
Olivia: "Oh yes.. I did that."
ME: "Wow.. you're fast."

Then Olivia reiterated to me that she's there literally every minute at work approving patient account requests.. because she manually approves all new accounts and also is there to pass along requests of appointments.. etc. And she was sure there was a privacy policy somewhere on the system. Her description of the account approval process sort of contradicts the fact that I could make an account called "testacct" and get right into their system without any approval but I didn't bother mentioning that. I just wanted to know where my data had gone from my first real account made with them.

After that, she could only talk about how to use the system from her perspective, not mine. In other words, Olivia had no idea what regular users face (ie, There is no privacy information, as I typed in my personal data, and no real idea other than from reading the URL in the address bar that maybe a third party was collecting my data, etc. Reading address bar URLs is something most users don't do.)

I told Olivia she literally wasn't getting the problem, because she just kept repeating to me how she uses the system (as an administrator over user accounts and for appointments where, I'm guessing, she has to be seeing an administrator version of the Medfusion system or some kind of much more powerful interface than the one regular users see when they log into the system). So she said she wanted to pass me to their office manager, Laura, who said, as she picked up the call:

"Mary, i've been listening to your call with Olivia" ... er.. okay.. no one disclosed to me that my call with Olivia was going to be monitored by others listening in. Unsettling. And possibly illegal. But whatever, that's really the least of my concerns here.

I told Laura there was no disclosure to me in advance of having a third party get my personal data.. and after Medfusion had it, I had no way of finding out what they are going to do with it.

I asked Laura about GGObgyn's ownership of Medfusion, but she replied that Golden Gate Obstetrics *did not* own Medfusion as the receptionist had told me. Instead, GGObgyn used them because they could not email "using Gmail or AOL" about appointments because that "wasn't safe." I was thinking really? Because having a website where my data just goes to third parties with no written privacy policy seems pretty unsafe.

So she explained that every page on their site (see all the screenshots and look hard for it!) have some sort of key symbol in yellow (it's not on any of the screen shots I took of the site, and I took shots of every page on their site), which if i click on the key, "will take me to their privacy policy." Okay.. so ignoring the obvious question of why they have a yellow key to signal a privacy policy (totally not intuitive from a user perspective), I look all over all the webpages that I can get to from the left side navigation, read them to Laura, and confirm that I cannot find the key.

Laura replied, "Well I can't help you anymore, because this is a waste of our time.. if you didn't want to put your information into MedFusion then you shouldn't have."

ME: "But your voice system told me to. And your name is on the website, and you aren't really disclosing that you are giving my data to a third party, MedFusion or telling me what they or you are going to do with it."

Laura: "Well, I can print the privacy policy and fax it to you."

ME: "But I don't have a fax machine. Can't you email it?"

Laura: "No.. maybe i could scan it and send it in email, but I'm not sure... and there isn't anything else I can do anyway." (It was clear she was trying to end the call.)

ME: "Er... Okay." (And then I hung up.)

A few hours later while writing this post, looking at the GGObgyn site, I noted that they added a privacy policy to the left side navigation, though that policy doesn't govern anything about what I entered into the GGObgyn site because it wasn't there when I gave my data. Medfusion and GGObgyn are under no obligation to keep my data safe or private, based on that policy.

No help or contact pages appeared afterward.

The privacy policy, which I read through, has a few issues. First, it starts off just saying "we" .. and my question is, We Who? I mean.. is it Medfusion? or GGObgyn? Me and GGObgyn together? Or someone else?

At the end of the privacy policy, it says under a section called OUR NOTICE OF PRIVACY PRACTICES:

By law, we must abide by the terms of this Notice of Privacy Practices. We reserve the right to change this notice at any time as allowed by law. If we change this Notice, the new privacy practices will apply to your health information that we already have as well as to such information that we may generate in the future. If we change our Notice of Privacy Practices, we will post the new notice in our Center, have copies available in our office and post it on our website.

So basically, they have to follow the policy, but can change their privacy policy at any time and it's retroactively applied to my old data and old terms? Well, I can see why GGObgyn wouldn't even bother having a privacy policy before because essentially, I have no rights over my data anyway.. because they can just change my rights whenever they want to suit themselves? I feel really good about my personal and medical information held by Golden Gate Obstetrics now.

And then, under COMPLAINTS:

If you think that we have not properly respected the privacy of your health information, you are free to complain to us or to the U.S. Department of Health and Human Services, Office for Civil Rights. We will not retaliate against you if you make a complaint. If you want to complain to us, send a written complaint to the contact person at the address shown at the beginning of this Notice. If you prefer, you can discuss your complaint in person or by phone.

So.. GGObgyn seriously expects me to complain to the USDoHHS? Why do we have to escalate this to a federal agency? Why can't they discuss it directly with their patients? I would rather just start by telling GGObgyn (which as you can see from the above dialog was incredibly successful, but they really ought to be open to hearing from their users about issues). In looking at the complaints section of the GGObgyn privacy policy, I note that I can contact the person listed "at the top of the privacy policy." Except, surprise! There is no one listed at the top of it. In fact, I don't even really know who "we" is in the policy language. So.. I guess I won't be contacting the "we" in this policy.

If I did want to complain about a privacy policy and questionable data usage problem, frankly I would use the Federal Trade Commission form because the FTC governs these things (see their most recent list of cases here where they go after companies that fail to protect user data and medical information, including the recent CVS case where they violated financial and medical data privacy rules). I have zero confidence that the Office of Civil Rights at the USDoHHS would even have a clue about privacy and my data on a website.

One thing.. after the GGObgyn privacy policy appeared, no one from GGObgn emailed me, or called me, to say that it was now up on their website. Of course, they have all this contact info and my name in their patient files and in their online system that Olivia who runs their website presumably could pull up very quickly and easily send me an email telling me to look at the policy.

I would also recommend that businesses like Golden Gate Obstetrics use the FTC page on Protecting their user's data and privacy which is very helpful when trying to figure out how to present privacy info on a website.

Frankly, I have no way to alert anyone at GGObgyn to this blog post, or to my thoughts on the subject, other than to call back, sit on hold, and talk with the three people I already discussed this with, who were ranged from unhelpful to hostile. Since GGObgyn doesn't seem open to discussing their websites problems and the fact that the cat is kind of out of the bag now with my data going God knows where into various company's hands, I'm posting this example of how companies, particularly *medical* entities, with no experience or understanding of information technology systems and websites need to use extreme care, and not assume that office staff trained to run a medical office has any idea what users need or will face with a website collecting personal or medical data.

I hope people at medical or other data collection companies will realize the importance of protecting user data and being straight with us about what's happening to personal and medical information. My experience is just one, but if this becomes representative of people's experience with their medical providers, we ought to be very worried.

Note: I took a look, when writing this post, at ratings for Dr. Wiggins, whom I really like and have enjoyed having as my doctor. You can see from the ratings at Health Grades that Dr. Wiggins is well liked by patients but the appointment system and her office staff.. not so much. I hope GGObrgn does an overhaul on all their office administration and website that interacts with patients before they venture further with information technology as tool for communications.

Celebrity Worship in the Post-Modern Internet

Doc Searls in a post on celebrity, black holes and productivity comments on celebrity as a form of coasting. Below is my reaction. I think there is more to it. Let me know what you think.

I've been talking about celebrity a lot since Thurs afternoon re: MJ, FF, EM, etc. Doc and I discussed this a month ago in depth too. I've been trying to figured out for the past year what the hyperdrive of microcelebrity is on Twitter that so many run after. And then the real celebrities hit twitter en masse and the hyperdrive of real celebrity is there as well. That drive diminishes at times one's ability to have a real conversation because some of those diving into conversations have agendas like trying to get the attention of the perceived AList (whatever that is.. oh yes.. the high follower counts, goosed by the twitter suggested follower list, provide us with a definitive answer... thank the gods).

But the triple-hit celebrity death match on Thursday drove me to my thought which is that most people need to follow, most people need something to worship, and most people have given up serious religion (of the type where you spend like 20 hours a week in church and the pope or the ayatollah or the supreme leader or whoever is your celebrity representative communicates with god for you and leads you and makes the decisions and you worship him to get to god because you can't talk to him yourself).

Michael Jackson and other celebs are the replacement for that sort of seriously time consuming difficult religion, because media and post-modernism make it easy (where premodern means god is above man, modern has everything equal: god, man, nature, and post-modern means nothing is more important than you). If nothing is more important than the individual, but he/she needs to follow something bigger than the self out of insecurity or whatever and there is very little ritual left post-old-style-religion to set people on their own course of confidence, productivity and humility in the world, and you have the media machine the past 100 years that now includes internet and self-publicity on things like twitter, well.. you have the perfect primordial soup to grow the MJ, etc worship replacing organized religion we see now. And of course, the celebrity version is so much easier and more fun, kind of like fast food.

Doc is right, celebrity worship is a tremendous form of distraction, but I would argue most don't have the confidence, discipline, or for that matter the interest in spending their time on more constructive things. While most are capable of much more, there is safety in worship. That's why the church/temple/mosque of old was so effective. It filled the rest of your time after work and set the order that god was first, then the supreme leader as the physical manifestation, then puny you, so you would worship up the hierarchy. Oh, and you were given a structure to think about life and death. Which is frightening to many. And there was a structure for work and discipline, however messed up these organized religions have been over the centuries.

In Post-Modernity, celebs fill the worship channel, effortlessly, where the celeb hierarchy is the order and the media connects you. Microcelebs are the long tail of this channel. Nothing going on with the top? Well.. there is always Guy Kawasaki or iJustine. And if you as the worshiper can get nearer to the celeb so much the better. People used to say: god is my savior. Now they say, "I remember exactly where I was when I heard MJ died." It's a way of placing yourself close to the worshiped thing.

Not to mention that you don't have to think about death if you go with the celebrity distraction mechanism, except when Farrah and Michael and Ed McMahon leave us, at which point people seem to just increase the worship but don't really have to face facts about their own lives.

It's utterly silly, and of course the internet and socialmedia send this tendency and need an order of magnitude higher than before. But I think it's a rat-brain need for the masses to worship something, and celebrity is the post-modern fast-food solution.

Opiates anyone?

Mobile Engineering: Why Coders with Old World Discipline Have the Advantage

A month or so ago, someone (I can't remember who) said to me that mobile engineering was hard for web engineers to do because it was so different. I've worked over the nine months on product development for several mobile applications at Apisphere, and more specifically the last couple of months seen coding for handsets up close. I can see why those who are great at coding the front end of websites that will go out to people with beefy computers might have trouble coding for tiny devices with limited memory, harddrives and processors. Even smart phones are no competition for the latest desk or laptop.

Working with engineers on Android, iPhone and Blackberry apps, where GPS data is involved, and each of these phones' quirks are being exposed, I've come to realize there is much more to this than just the difference between webcoding and mobile engineering. I started in tech in the 90's working on boxed software. Huge projects with 60 engineers making things for big machines of the time. Those kinds of projects required enormous specs, Market Research Docs (MRDs) and Product Research Docs (PRDs), etc. When I later switched and started writing algorithms for web apps, building little classification systems, and working closely with engineers on web apps creating the information architectures and meaning on sites, through interfaces and algorithms, I didn't think all that much about the differences between installed boxed software and web development, other than the specs I was writing were far smaller and we iterated a whole lot more on the web development in tighter cycles, and often the usability was built in a bit more from the beginning instead of bolted on at the end.

But now seeing development for mobile and creating mobile apps, I realize engineers who learned to code way back when have a huge advantage over web and large app engineers who've never been forced to economize. Those early coders know what it means to optimize for tiny amounts of ram and hard drive space, to create truly elegant code that is compact, efficient, and doesn't take over a device or machine.

In contrast, I find my Firefox usage often pushes my laptop out of control as javascripts go crazy on tabs in the background. Those pages were written by programmers unschooled in the art of system management, who may believe the system resources are unlimited or worse, dedicated *only* to the running of the browser+their webapp. They don't even seem to know they ought to be considering users and their resources based upon the pinwheel of death I regularly experience. I'm often climbing through FF tabs on pages open for work and play as I go through my day, trying desperately to locate that one tab that's going crazy, pushing FF to 125% according to Top. When I get it shut down, after massive frustration and system hangs, waiting to see if the next tab is it, I realize another tab is out of control. And so on until I get my machine back.

Building mobile apps, there is no way we can put that sort of strain on a smart phone, much less a little tiny phone. At this point after watching 9 months of mobile development, I'm realizing the preferred mobile developer is someone who has hardcore coding experience with languages like Java and C++/C#, who had to optimize for old computers with minimalist ram, hardrives and CPUs. People who code as if their program will be the only one open or up in a browser need not apply.

In fact, I would say that older coders with this sort of discipline will often have a distinct advantage over the young new web-only coders, and will be the ones who help us move mobile forward as a viable industry. Of course, those who embody all of these skills for all environments will have the best chances to work in mobile going forward, as I see mobile delivery of webpages as also key to this industry.

The Life of a Tweet

Twitter (and the ISchool -- or one of my poor brethern -- I have a masters from UCBerkeley's iSchool) seem to be in the tweetsphere over one ill-found tweet tossed off by a student and found by her summer internship employer likely via search.twitter.com. For background, you can see this: FattyCisco.com. The poor girl is likely humiliated and horrified over what she thought was an innocent and also, likely, a fleeting thought that didn't really reflect how she felt overall.

We've all had those momentary thoughts where when we are ambivalent, we toss something out of our mouths and once it's out there, we think, wow, that doesn't even ring true or, it did for a nanosecond, and now it's changed, or gee, that's about 5% of the way I actually feel about this. But out of mouth, truly ephemeral (unless recorded in some form) is different than written down and searchable in the grand database of the Googlezon and search at twitter. Or maybe it's just a joke.

This is one of the problems with online communities and specifically twitter:

You don't know who's listening, and because of search tools, you are findable beyond your follower list
or your "community" of known tweeters (ppl you @ with or read) unless your account is private.

I don't think we have at all sussed out what it means to tweet in the long term, or what the power of the tweet is, or where the tweet goes and what sort of life it has beyond the first few minutes or hours of it's life in the Twitter / client context.

This is another example of something that happened recently:

A PR exec going to Memphis to meet with a client, Fed Ex, insulted the client on the way to the meeting. The clients wrote a letter to the PR company and him, his bosses, and cc'd everyone at Fed Ex as well. Ooops.

The problem is, tweets go to those paying attention at the moment, those who may save tweets in clients (i leave my twitter client open and check it now and then as I have time -- right now I have 15k tweets from the past couple of days), those pivoting on a single user, those searching for key words, those looking a related conversations.

But when you tweet, in your head, you're often just thinking about those you expect to read it, like only a few your followers paying attention at the time. What happens with some tweets (some reading by some followers) is not what can happen with all tweets.

The interface and interaction at Twitter's website doesn't lead you to believe that what happens most often there will happen in incendiary examples. And different twitter clients (an android or Iphone app for example) don't lead you to understand the permanent nature of tweets, through use, that say, search.twitter.com might, as you see something you deleted appear there anyway.

It takes experience with all these different modalities to inform you because there is no advance disclosure or warning of the elasticity of a single tweet.

What is most interesting is this pushes me to think harder about what the interface of "aged information" online looks like (and I don't mean google search results that move from page 1 to page 3 over time).

And I have to ask myself what it would mean to have what Judith Donath discussed on the panel, Is Privacy Dead or Just Very Confused, moderated Saturday at SXSW by danah boyd. Judith discussed having some kind of a "mirror" for you of your digital self that would reflect all your online presentation and communications and expression... just so you might get a sense of what you show people and what you project at a moment in time. Right now it's really hard to gather that sense of yourself. Right now, you don't really see it in any sort of complete way. But others see pieces of you digitally represented at different times. It would be like re-disclosing for yourself what you've done, discovering how others view you, in slices or on the whole, in order to see the effect you have. It would probably be helpful to know what had reach and where, and what was for now at least, forgotten.

But frankly, the privacy implications of that are huge as well. So, I'm thinking. No answers on that one yet.

Getting The Orwellian Hazing of a Lifetime by Citibank and American Airlines Advantage Miles

Ok, let me just get this out there first. I hate Citibank. It wasn't always like this. I got my credit card through them 15 years ago, because I wanted American Airlines miles. But lately, the past couple of years, Citibank has just gotten worse and worse. It's like they actively subscribe to that newish thing corporations have been doing where they treat us all a little worse and a little worse, to make incrementally more money. And we all take it, because it's just a little and we don't have time to fix them by going away.

It started maybe five years ago, when they told me I couldn't walk in a check from my bank (not a Citibank check) to deposit a payment on the credit card any longer. Since I would sometimes forget to do the online billpay or mail a check in time for the deadline, I could walk in to the branch two blocks from my house and pay. Well, no longer. They simply wouldn't accept them in person unless I opened a Citibank account. Well, that seemed like the worst marketing ever to me (coercion), to get me to open a new account there.

After that I mostly used online billpay, and generally did it at least 5 days ahead of the deadline, because my bank takes 2 days to send them a notice that the money is waiting at my bank. Sort of an internal bank-to-bank email, that prompts Citibank to collect the money. They ask for it right away, and it's essentially a cash wire. Then Citibank credits my account right away. No problem.

Until recently.

I started getting calls from Citibank about 10 days or so after doing the online billpay, to, get this: "verify my payment." I literally said the first month they called, "You're joking, right?" I mean, they had a cash wire from my bank. Some online billpays are done via bank check and take five days, but not to major corporations and other banks like Citibank. Those take two days, and are very efficient. The minute I hit the "enter" button on my screen to send payment from my checking, money is withdrawn and held by my bank to make the payment to Citibank.

After the call, where I reiterated that Citibank had cash in hand from my bank, and we hung up. I assumed all was fine. No way. They "held" the funds, just to "make sure," for another 5 days, twice declining my card (i had just gone on vacation and was maxed out, thus, a large payment). These declines were for $10 and $24.95. I called and was told about the holds. They said something about how they usually allow small amounts through, but when I pointed out that $10 and $25 were pretty small and how small were they talking, anyway.. they dropped that one.

Then, a month later it all happens again. I get this call to "verify payment." Now I'm mad. Apparently they are only looking at the last six months of payments (the person I spoke with could not see back any further in the history) and since the payments had each been a little more than the last (by at least $200, so in other words, one was for $2500, the next for $2800, and the most recent for $3000), I'm now suspicious to Citibank. WTF. Why is that suspicious? Especially when I've had the same checking account at my other bank also for 15 years. So I'm told that paying more makes me risky. Nothing else matters. Wo.

So, basically, they hold the $3k again for 5 days just in case. And in my calls to them, they tell me that the department that does security sets the heuristics (my word, not theirs) for holds and Security told the woman I'm speaking with that there is nothing they can do. So while she understands Citibank is getting cash from my bank, directly, and that it's generally bad for them to decline purchases because they are losing money (they didn't the second month decline anything due to the hold but did it the prior month post vacay), they have to follow the security department's algorithms (again, my word, not hers). Great.

So then yesterday, I go to buy gas. And my card, which now has tons of room and almost no new charges, and is wide fucking open for a skyhigh-priced tank of gas, gets declined. The pump tells me to go inside, to the attendant. Great. Do so, even in bit of rush. And he swipes it, and it says on the screen: Declined. So I pay with ATM (I only carry one CC card, though I have more at home). Get gas. Call Citibank.

They explain that the charge hit a limit for the amount of gas I can buy (WTF!) in a month, and that I bought too much, or it could be that that station hasn't raised their limits what with all the new high gas prices per gallon to allow people to buy more in a month. Ok. So I buy gas twice a month. Costs around $90 to fill the tank. Your kidding me right? I can't spend $180 a month on gas if I want to? And I do.

So I reply, well, what is the answer? She tells me, you should tell that station to up their limits for CC charges per month. Again, WTF. Like I'm supposed to know about the backend heuristics and algorithms that Citibank and Union76 use to combat fraud? And do something about it? I haven't even bought gas at that exact station in over 2 months. And she verifies this in my records. But I bought at another Union76 station within the last month (my engine/mechanic asked me to get either Union76 or Chevron or Shell gas.. what can I say, I'm following order because I don't freaking understand car engines).

So basically, with gas prices rising, Citibank and the gas companies have some weird heuristics, that we don't even know about. And I got caught in one. So I respond, "You and Union76 are big companies and you must talk to each other because I'm sure a lot of people buy gas there with Citibank cards. So why don't YOU work out the raised limit for purchases instead of me?" To which she had no response and wanted to know if there was anything else she could help me with. OMG.. the possibilities are so great.

Anyway, I called Citibank back again to discuss more things about my card, as I got the next bill in the mail, and just decided I hate them too much to stay there, even if the only reason I have the card is to get AA miles. Which leads to the second clusterfuck going on here.

I have several hundred thousand miles generated through the use of this stupid card, over 15 years. And I've only once actually used the damn miles. Because every time I call to use them, they laugh at me because I only called say, in February for a June trip to Europe, or 2 months ahead for an upgrade. Once, once, an upgrade actually came through but not first without spending three months on the wait list to get the upgrade.

In other words, AA may be the gambit to get you to use the card, but you can't really use the stupid miles if you have a life and can't plan, oh, years in advance to get a plane ticket (i usually buy tix to europe a few days to a month ahead, and everywhere else, like days ahead, because I have a life, thanks).

Anyway, that's the story. So today I worked out getting rid of Citibank. I liked it because I have the number memorized for online purchases (probably the biggest reason I've put up with Citibank's crap). But I will memorize a new CC number. And I'm going to plan a trip like a year ahead so I can use up all the miles (I'll probably have to take about 8 people with me to someplace like Antartica) because if I don't have the card AA will probably cancel all the miles I've earned. And then this whole stupid corporate hazing I've been experiencing, with increasingly stupid rules, for good paying customers that make them a lot of fracking money! will have been for naught. So, where do all my readers want to go?

Alice In Wonderland Remix

Luv this remix (noted on Cartoon Brew) by Nick Bertke. He says 90% of the music is remixed from audio from the Disney (1951) film. You can download the mp3 here.

Data Sharing Summit Report

Last Friday and Saturday the Data Sharing Summit was held in SF. I attended a bit on Friday, but not Saturday. It looked like a lot got done by the participants, and so they did accomplish a lot!

Kaliya Hamlin has posted notes and goals for the next meeting in one month.

Here is an excerpt of the results:

Do-able Now
* Portable Identities (OpenID, LiveID, FB-ID)
* OAuth (sever to server) delegated auth.
* Contacts Portability (FOAF, XFN, Microformats, like MicroID)
* Sync (feed sync)
* Social Network Portability (Open Social FB platform)
* Social Application Portability

Do-able Soon
* Standard Schema for Profile
* Standard Schema for Address books
* Media portability + metadata + permissions
* Linking ID’s of different ecosystems?

Looking forward to the Data Sharing Summit 2 at the Computer History Museum in Mountain View on May 15th.

FCC Hearing at Stanford Today

I can't go, but I hope lots of folks out there who support and open and free internet do. Here's the schedule according to Save The Internet:

It is rare for all five members of the Federal Communications Commission to leave Washington, D.C., and they want to hear from you. There will be a public comment period - come speak up to save the Internet!

WHAT: Public Hearing on the Future of the Internet
WHEN: Thursday, April 17
TIME: 12:00 p.m. to 7:00 p.m.
WHERE: Dinkelspiel Auditorium, Stanford University
(471 Lagunita Drive, Palo Alto, CA) Map It!
For directions and travel information, visit: http://www.savetheinternet.com/=stanford_travel

FCC Public Hearing Agenda
12:00 p.m. - Welcome/Opening Remarks
12:45 p.m. - Panel 1: Network Management and Consumer Expectations
3:00 p.m. - Panel 2: Consumer Access to Emerging Internet Technologies and Applications
4:30 p.m. - Public Comment
6:30 p.m. - Closing Remarks
7:00 p.m. - Adjournment

Note also that Comcast is proposing a "P2P Bill of Rights and Responsibilities" according to ArsTechnica, who is skeptical. Don't see any users in that room, but if they don't invite us, I'd guess after Boston, we'd all get pretty mad and force them to include us. Either way, (FCC or voluntary code) I think it's going to be user centric in the end. We're just going to have to fight like hell.

Kevin Marks also makes a great point about Comcast: They are like The Producers who oversold their Broadway show, assuming it would fail, by getting 100 people to buy 10% of the who. Comcast, by overselling their network for internet access is doing the same, and then having secret levels above which they cut people off out of the blue, is pretty bad.

Data Sharing Events Coming Soon!

There are two new events coming up for the Data Sharing group (we met last August in great camp type open space event where many interesting things developed, came to light, got solved, etc.) I'm on the advisory group, and will definitely be there and would love to see anyone who cares about attention data, both the control aspects at a site, as well as ownership issues, get moved forward in a community oriented way there as well.

Also, Mitch Ratcliffe wrote a great post today on these issues which you should totally checkout.

Here is the write up from the Facebook group entry:

* A Data Sharing Workshop at the Downtown San Francisco State University campus on April 18th and 19th.

* Data Sharing Summit 2 at the Computer History Museum in Mountain View on May 15th. (This is immediately following the Internet Identity Workshop May 12-14).

Hopefully at the first event some more clarity will emerge about how to actually do and get adoption of data sharing technologies. The second event we can see progress (it being a month later) and may have more 'decision makers' considering data sharing implementations and vendors that have ways to do it.

The goal of these events is to work together to build consensus around and get adoption of emerging data sharing standards. As with the previous summit, the upcoming event will follow the open space (un)conference format. The agenda is created on the first day of the event, allowing everyone to participate in the discussion.

Although Marc Canter was a key organizer of the first Data Sharing Summit, he has stepped back and his involvement is just one of group of advisors:

* David Recordon, Six Apart
* Joseph Smarr, Plaxo
* Chris Saad, Faraday Media
* Mary Hodder, Dabble
* Luke Sontag, Vidoop
* Kevin Marks, Google
* Marc Canter, Broadband Mechanics

The events will be produced by Kaliya Hamlin and Laurie Rae, who are collaborating with the Data Portability community and the SFSU Institute for Next Generation Internet.

We would like to invite you to attend one or both of these events.
Please go to http://datasharingsummit.com or to go ahead and register right away to to our Eventbrite page to register. We will be charging admission to cover the costs required for organizing these events.

The Early Bird rates are as follows:

April 18-19 Workshop
* Regular, $110.00
* Independent/Startup/Non-Profit, $80.00
* Student, $50.00

Workshop One-Day Only:
* Regular, $65.00
* Independent/Startup/Non-Profit, $50.00

April 18-19 & May 15:
* Corporate, $200.00
* Independent/Startup/Non-Profit, $140.00

May 15th Summit Only:
* Corporate, $100.00
* Independent/Startup/Non-Profit, $70.00

The Early Bird cut-off dates are April 7, 2008 for the Workshop and May 7th, 2008 for the Summit. Prices will increase by $50.00 after the cut-off dates.

We can bring you this event at such a low admission fee because 1/2 our costs are paid by sponsors - both small ($200) to the large (several thousand). PLEASE contact Laurie Rae at laurierae@datasharingsummit.com if you would like to sponsor.

Please contact us if you have any questions identitywoman@datasharingsummit.com & laurierae@datasharingsummit.com

We look forward to seeing you in April and May.

Trashing Our Social Relationships (with Porn) to Get Your Numbers Up

Ok, there's a lot in that title. Let me explain (though I did blog about this earlier).

First, yesterday at the Supernova / Wharton event, in Jerry Michalski's session on business and social media (can't remember exact title) we spent much of the time talking privacy, online communication, games and social networks (er, social graphs but i really hate the fad where we make up a new word for something that is already working just so we can dink around with a new set of conferences, etc. but I digress. Though I would point out that one friend who attended SGP said the women at the event all seemed to get it, and then men all wanted to run calculations on our "social graphs" entirely missing the point. Oh well.)

At the end, Jeff Clavier, who apparently was at the Social Graphing conf/camp in San Diego earlier this week, gave a wrapping up of what happened. He mixed in a little of his perspective due to his investing in apps on Facebook, and threw in some perspective on the Stanford class that did some experiments on Facebook apps and their results. One example Jeff gave was about an app maker from the class had gotten 5 million people to click into his app (though they all immediately disappeared just after) in 5 weeks (correction, not 5 days -- as Jeff said to me later, correcting this, "damned French accent" because many of us heard "5 days").

I had to wonder, why would five million people do that? What's the benefit to them? Apparently the app maker, some young guy, is thrilled (and it sounded like Jeff might want to work with him or even invest). His experiment (with all of us, the greater Facebook community, as guinea pigs) worked for him, though I'd guess it wasted 5 million people's time, for a couple of minutes each.

I commented about the aspects of Facebook applications inadvertently trashing our relationships, at times, in order to get their numbers up, and using deceptive practices and features to do it, and said it thought it was really uncool. But there wasn't time to explain what I really thought, or the background of why I think this, and so, here we go:

Ok, imagine you get some sort of email message from a friend in Facebook. This is a real friend, someone you do business with and/or socialize with and maybe have known for a long time (as in, a lot longer than Mark Zuckerberg has been out of his teens and been (on paper) counted as a billionaire). Or maybe it's someone you work with (note that there's a lot of caselaw around sexual harassment.. so accidentally sending porn spam to people you work with or work for you, or you work for, doesn't seem like the greatest thing you would want to do either).

The message asks you to click into Facebook, at which point, you are asked to "install an app" (and, why? Just to read a message do I have to install an App? Oh yeah, this is about getting the applications numbers up ... so you do it, because you want to see your "real" friend's message). Then, once installed, you are taken to Slide's Fun Wall App, which shows you some porn, and says, "Click Foward to see what happen."

See this screen shot of the first round of porn spam I got (NSFW btw so be careful opening it).

I almost clicked "forward", but scrolled around past the fold. Turns out, if i'd clicked the "forward" button, Slide would have forwarded that spam to EVERYONE I KNOW in Facebook. All 500+ of them.

Now, let me explain who everyone is. Yes, of my 500 or so contacts, maybe 300 are in the tech community (and as such, expect early-adoptor screw ups and experimentation). But 200 are not. About 10 of these people are people I grew up with (we've been together as friends since nursery school). They don't know what the "tech community" is, much less care. Some of these people are religious and I would venture have never seen porn before or it's very rare in their lives. They aren't early adoptors. They expect that any communications are going to be real, and not some tech community experiment to figure out some thing that later promotes some business/VC investment, in order to see how the world of advertising can advance.

Or, take all the people I do business with, or the people who I work with, or work for me, or I work for. It would be just great to send them some porn spam. Or my brother and sister. They would luv to get porn spam from me. Not. Or how about my extended relatives in Europe? I think they are ripe for a little porn spam. No?

So.. I unchecked all 500+ contacts that Slide had checked, and wasn't able to view the message further (what was going to come next after I was asked to hit the "forward" button). So I figured out one profile I could link to who was a friend, and then forwarded the message there, "...to see what happen."

Well, guess what? Nothing "happen." Except that the message was forwarded to the one person I left checked. In other words. It's trick porn spam, features courtesy of Facebook and Slide.

So I sent in complaints to both companies (neither have contacted me back after a month -- guys, it's a social network, you know how to reach me.. give it a try!!)

After a while, I called people in each company that I knew through the tech comany. And was appalled at the responses I got. Now, these are people I know socially, and they gave me the real answers, but with the expectation that I would not attribute to them. However, I am confident that their answers reflect the culture and real value sets within these companies.

Facebook pointed the finger at Slide (the app maker in this case), and said, "There is nothing we can do. We have no control over the apps people make or the stuff they send." Oh, and if I wanted Facebook to change the rules for apps makers? I'd have to get say, 80k of my closest Facebook friends to sign on a petition or group, and then they might look at the way they have allowed porn spam to trick people into forwarding, but until then, there would be no feature review.

Slide said that they thought Facebook was the problem, because as the "governing" body, Facebook makes the rules and "Slide wouldn't be competitive if they changed what they do, and their competitors weren't forced to as well." In other words, Slides competitors use the same features to get more users (or trick more users as the case may be) and Slide didn't want to lose out on getting more users with similar features, regardless of the effect the features have on us and our relationships.

Also both companies told me that blogging doesn't affect them, because they don't read blogs. The only thing they pay attention to are Facebook groups. Because they don't look at problems that a single person discovers.

So in other words, a person with a legitimate complaint needs to have massive agreement and numbers in a Facebook group before these companies will even discuss a problem.

And, Slide and Facebook are willing to trash our relationships (real relationships) in order to get more numbers.

Now, note that many of the folks who sent the various porn spam (not just the ones in the photos above) sent very apologetic notes, because they were mortified that they had send their contacts porn spam.

Think about that. Your social networking / application software tricks you into doing something terribly socially embarrassing and you have to apologize? Wo. That's really messed up.

In other words, your social networking software / applications are, gasp, anti-social.

One guy in the Supernova / Wharton session yesterday asked how many people were in my Facebook list, and when I said 500, implied that most regular people have say, 50-100, and therefore it's not a bad problem. Well, I'd say each relationship is probably pretty important and this is an appalling justification for these applications and social network's feature sets and behaviors.

So I have to ask, if these young boys (Zuckerberg, the app makers in the class at Stanford, etc) are so clueless about relationships and social protocols, that they would build apps and a system that promotes bad behavior like this, where are their mentors? Where are their funders (who presumably have some input and sway into what's going on)? Why aren't Peter Thiel and Dave McClure or even Jeff Clavier (who sounded like he was trying to or has invested in some of the guys from the apps class at Stanford) advising these people that while they are experimenting, that these are real established relationships, and Facebook is now mainstream, and therefore the apps can't do this to people? I mean, it seems logical (and has happened in cultures around the world for millennia) that older, wiser men would advise young, clueless hormone driven boys how to act in the community. And what of Max Levechin? I mean, he's kind of in the middle, age wise, but shouldn't he know better than this?

Is the desperation for fame and money so great, that people would simply eschew social concerns in favor of ratings which then equal higher company valuations, and more billions on paper? Or do you want your claim to fame to be: "At least 15 million minutes wasted" from your experiments on Facebook (as I would imagine the Stanford student described above could claim)?

I guess the answer is yes, and so my response is, I can't trust Slide, or Facebook. Nor do I have respect for their founders if this is the way they handle themselves and their companies.

And where are the advertisers who might put pressure? The ones on the page I show above (not all the porn spam trickery I got, but the first batch) are Toyota and Gartner?

I deleted all my Slide apps after my last blog post, a month ago, and since heard from maybe 20 people in person that after reading my post, they'd done the same. But I guess we don't count, since we only have a few people concerned.

I hope the folks who attended the session yesterday at Wharton have a better idea as to why I find this upsetting, and upon hearing that more "experiments" with Facebook apps are happening, why I might get worried and distrust the process, the results and the motivations behind them.

Note: I am aware that Facebook did recently force apps makers to default turn "off" the checked names in forward (as far as I can tell from my own analysis of Facebook and via other blogs explanations). But I have yet to receive replies to my original support notes to these companies, and feel confused about an unspoken, barely there response. It's as though after barely changing one thing aspect of a feature, in order to mitigate the problem, they want to sweep it all under the rug. But I don't feel confident that these companies either care about the spam problem, the porn problem or the social abuse problems they are allowing.

For now, the answer for me is to use Facebook minimally and Slide not at all. Interestingly, at recent social gatherings I've mentioned these issues. At almost every one, people have said they are getting off Facebook and not going back, for precisely the reasons I mention above.

I guess that's the only way to make an impression on Facebook and Slide. Shut down your own use.

The NY Times on Girl Geeks: They are Fashion, Not Technology

The
NYTimes Stephanie Rosenblum has an article in today's *Fashion* section on Girls in Tech. Wo. Not in the *Technology* section. In Fashion.

Sorry, Boys, This Is Our Domain talks about how girls are coding up more content online: webpages, web art, blogs and podcasts.

And then they decorate it with an image of a girl at her laptop with a devilish tail. But instead of asking one of the girls they interviewed to make the artwork, they ask Adam Strange to do the art for the article:

So when they interview people like Doc Searls, Loic Le Meur or David Weinberger, all of whom are very smart about tech, those articles are in the tech section or business, but when they talk to girls, who for the record, are far more technical in this article than these three tech experts, girls are put in Fashion. I've never seen coverage with Doc or David or Loic in fashion. Maybe they should be there depending, but they aren't put there by the editors that I know of....

This is not about David or Loic or Doc (all extremely supportive of women in tech, btw), and certainly they don't choose the section the paper puts them in, but rather the way the editors and writers at the NYTimes see them, verses the girl geeks in this article.

My point is that the NYTimes puts men who talk tech and trends or social impact in tech/biz, and women who code web art / pages in fashion.

Can you tell I'm pissed? WTF?

However, the number of women in tech isn't great (Which is why we need more articles in the Tech section about this people!)

The article says that less "...than 15 percent of students who took the AP computer science exam in 2006, and there was a 70 percent decline in the number of incoming undergraduate women choosing to major in computer science from 2000 to 2005, according to the National Center for Women & Information Technology."

Chaos, In Pakistan and Silicon Valley

Today are the Pakistani elections. Why do I say chaos? And why also in Silicon Valley? And how are they at all connected?

As Amra Tareen, who is in Pakistan covering the event, says in her report about the elections:

As the day progressed more people started to show [to the polling stations], people were staying back home enjoying their morning off and due to concerns of violence. In the last 24 hours gunmen in Lahore and surrounding areas have killed 8 people and injured 40.

Check out this ballot from Pakistan, which Amra explains:

For example PML-Q (Musharraf's party) has the symbol cycle, PML-N (Nawaz's party) is represented by the symbol Tiger or a Lion and PPP (Benazir's party) is represented by the arrow. People caste their vote by placing a fingerprint and a seal over the symbol.

I've been helping Amra, a friend in the Silicon Valley for 4 years, with her company All Voices. Amra is from Pakistan, though she spent some of her educational years in Australia, including getting an engineering degree, and then went to Harvard for an MBA. She was also a VC in Silicon Valley for 6 years. Now she has founded All Voices with Erik Sundelof and the help of a great team of engineers and other folks.

I'm still working on Dabble, but I just find what is happening at All Voices so compelling, that I wanted to help her do this. She's raised VC money for a news and a conversation site that is meant to foster discussion from people around news events.

And how many Silicon Valley founders go to Pakistan to cover the elections, to kick off their companies?!?!

That's incredibly unusual, and to me, shows tremendous passion and guts about both the company, and her desire to see Americans and Middle Easterners talk about what goes on in their world. Anyone can talk, but she specifically wants to see these two groups getting to know each other on a more personal level, as opposed to say, an AP report.

So what is the chaos in Silicon Valley? Well, it's not on par with the Pakistani Elections, but the alpha All Voices is out, and people are commenting, talking about the election (finding a few bugs too!), making events, posting videos and photos, and it's the first big exposure the team has dealt with.

The site is pretty simple, really. The idea is that events happen in the world, and an event within AllVoices can then be assembled by pulling in news stories, photos and videos by news sources or blogs and say, creative common's licensed Flickr photos.

But you can also make an event, which is really more factual in nature, than opinion, about whatever has happened in their world. Then you might blog or add photos or videos to your own event, or you could add those elements to events made by others or the system. For example, Amra has put video here and here and here of the people in Pakistan talking about the election, onto the event she made noting election coverage. After you make your event, the system will match blog posts, articles, images and video to it, and more folks can come along and share eyewitness stories, comment, ask questions, etc.

All this gets put onto the map and front page which lists recent and active events.

So when others come to the site, they can find your stuff based on where it happens as well as by searching or by finding your list of activities via your profile. Amra's election reporting is, of course, located in Pakistan on the map.

The sites definitely is an alpha, where there are bugs and things. Her engineers have been working on this for about 6 months, and it's really great to see what they've done.

As I said, I've helped with a little consulting on the side. Normally, I wouldn't blog about things I offer consulting for, and normally I'm too busy with Dabble, but I think this site and Amra's work has the potential for so much social good, I'm breaking my own rule.

So take what I say with a grain of salt due to my work and bias. Go visit the site yourself and decide if you think it's worthwhile. But more than anything, I encourage you to get involved in supporting Pakistan as it hold its election. Pay attention, comment, blog, make a stink, but support democracy and the people of Pakistan as they stake a claim for their future!

TransitCamp Ideally: Promote Simplicity and Ubiquitousness

Tara Hunt has a post up about TransitCamp (a camp held in about a month at SocialText with the help of Heyward Robinson, Menlo Park city council, Adina Levin, Co Founder of Social Text and avid Menlo Park community activist, Margaret Okuzumi, from the Bay Rail Alliance and MTC).

I love the idea of transit camp to help people who work on those issues do better for all of us. I can't attend but I wanted to throw out a couple of ideas.

I rarely use public transit here in the Bay Area. I use it all the time on the east coast in Boston, NYC and Washington, as well as Amtrak linking the three. And in European cities like Paris, Barcelona, London, Amsterdam, Rome, as well as Euro-trains linking the continent.

The common thread across all those cities is that the interface, and the metaphor, is simplicity and instant access with little mental overhead. In other words, you don't have to know much to use them, other than to find an outlet and get in. Then after locating a map, you buy a pass and unless it's very late (after midnight or 2am depending) or very early (before 6am), you wait a few minutes and your train, bus or trolley shows up. And you're off. And it works from the airports too! Yipee!

The way Bay Area transit works is: you are a commuter, you already know the complicated and dysfunctional system section you use all the time and the rest is a byzantine mess of mismatched numerous connections, so therefore the regular user only will move say, one leg to get where you are going or maybe two at the most. Oh and you are doing this almost exclusively during commute hours in order to have any efficiency at all.

That's not really great.

I have tried to use public transit. And when I lived in SF, I had a flat rate card, took the bus, muni, BART (just within SF) and the trolleys. But even they really worked most efficiently during commute hours, with long waits before, during the midday, and after or weekend times. Though BART was nice if you were downtown and wanted to hit the mission or somewhere else along that specific line, because so many trains go from all over the BA through SF and out again. So there are a lot of options there for constant movement with little planning.

But from Berkeley, BART takes on a different metaphor, often requiring much more planning. For example, to take a train after hours or on a Sunday, you must change trains to get to SF. Recently, my car needed unexpected servicing in Mountain View, so I dropped it at a recommended shop there. The next day, after being back in Berkeley via a ride from a friend, I needed to retrieve my car.

I thought that it would be easy to go from Berkeley to Mt. View, after reviewing the various websites for about an hour or so to plan my trip. So I optimistically hoped on BART at 6pm. The online info said: "Cross-platform transfers at the Millbrae BART Station." So it turns out that my SF bound train from Berkeley didn't go to Millbrae. I had to change. Twice!

I arrived at Millbrae at 7:40pm, and saw a southbound Caltrain. W00t! I ran out, up the three flights of stairs to the overpass walkway, and just then watched the Caltrain pull out. Damn. They don't coordinate! Even as a bunch of people wanted to get on it. They just move out regardless of BART trains pulling in.
No problem, I thought, there have to be more. So I went downstairs to the Caltrain side, looked at the schedule. Which was difficult to read in interface, but said there was a train in 15 minutes. I bought my ticket to Mt. View, after which I waited, as the rain poured. Sideways too. Shelter was three stories up, so the rain and wind were going everywhere (Way to architect public spaces for people waiting for transit! But it looks pretty in the brochures.)

No train came, and I eventually made my way back to the totally unsheltered schedule. I had read the schedule, on paper, with no AM or PM specified, wrong. I had been looking at an AM schedule, so I looked all the way to the other side, where the PM side was. Next train, 75 minutes. So, I'd already waited 20. Wow. So now I really think Caltrain operators are jerks for pulling out when a BART train arrives from the North that might include riders on their train (there were 10 of us and they could have waited 30 seconds for us!).

I waited a while longer, and then called my friend in Mt. View to come get me as I was soaked and absolutely freezing, even with raincoat and umbrella and lots of scarves and things -- they don't protect you from sideways rain much, nor did the shelter 3 stories up do anything at all for us! I just couldn't see waiting another hour for the train, and then having a 25 min ride down there, and then walking in the rain to get my car (estimated arrival that way: 10pm). Actual arrival: 8:40pm.

It took him 20 minutes to drive. And that's my point.

If you plan for traffic, it's 40-45 minutes from Berkeley to Mt. View or Palo Alto, 20 minutes to SF, 25 minutes to San Bruno. I use Google maps with traffic turned on in my phone in the car, for instant planning after the general plans have been made.

My car, instead of hours on public transit, with many connections that are uncoordinated by the many agencies involved, and impossible to pay for in one lump.

I'd rather not drive, but how the heck do you manage my typical day of meetings like say, last Thursday?
AM: 9:30 meeting in the mission, 11am in Cole Valley, PM: 12pm in the outer Sunset, 2pm in San Bruno, 6:30pm in SF for dinner, 10pm in Emeryville, 11pm home. All of which I easily made in the car, but with transit in the BA, I'd have to plan two hours between each meeting and an hour home on the last leg. In NYC, each bridge to the next event would be 15-30 minutes (what I'd planned for driving between each thing I needed to get done).

Or for that matter yesterday: Mt View to San Bruno, then San Bruno to Oakland (on Mandella Parkway), then Berkeley, then Oakland, then Emeryville, then Berkeley, then Emeryville, then Berkeley. All between 1pm and 8pm. So you know, there isn't any option between Emeryville and Berkeley. And nothing between Mandella Parkway and Berkeley either that I know of... it would be a disaster if I didn't have a car.

The metaphor for transit in the BA: you use this already and if you don't, you're screwed. And you use transit during commute hours (like 7-9am and 4-6pm -- yeah.. that really works in the tech community and with all the events we all attend each evening). You have all the payments worked out in advance with monthly transit cards (not great for changing systems though some have recently connected better than in the past).

In fact, BART's own website acknowledges this:

Transit Connections to BART

Free Personalized Trip Planning Service!

We know that navigating public transit connections in the Bay Area can be difficult, that's why we're here to help: If you'd like an accurate, personalized trip plan that includes BART and connecting transit, call our Customer Service Department: it's fast, it's easy, and it's tailored just for you! (Emphasis mine)

In other words, it's so hard, they don't even bother to put the info online. They just have you call them to work out the byzantine system's details.

I'd like to see Transit Camp deal with the broken metaphor, the interface and execution (tickets and money, schedules and websites, mismatched transitions), and the assumption that this all happens during rush hour and otherwise there is no need.

Frankly, if you don't provide much after hours, people won't build it into their schedules. If you do, they will.

For me, Tara's picture in her post and at Flickr is more representative of what I see in the transit experience, where nothing quite works unless you live and operate in SF:

I'd like to see transit work holistically for the whole BA, where you just jump on and go where you need to go, up 'til say 2am. That would get me to leave my car behind. :)

The FAA TRACON Information Experience Live

Earlier today I had the delightful experience of touring the FAA's Northern California TRACON facility.

Basically, TRACON, which stands for terminal radar approach control, is the air traffic control center which, in this case, handles Northern California. TRACON handles traffic outside of each local control tower a plane might ultimately deal with as it lands. There are TRACONs all over the US for other regions. We weren't allowed to bring in cameras so I'll instead show you a news photo from SF gate that was representative of what we saw up on the wall of the facility. You get the idea there of what they are seeing on some of their screens.

Chronicle photo by Mark Costantini

This photo only shows traffic into SF, because it's a visualization from SFO traffic control, but just imagine more planes going into San Jose, Sacramento, and other smaller airports like Modesto. Also, these screens are synced between TRACON and the air traffic controllers who are local. And if anything happened to one TRACON, others would instantly fill in, as the system works somewhat like the internet in that sense.

TRACON is housed in a big, windowless building, extremely modern and cool with an air of serious importance about it (I always find that at say, buildings in Washington DC, and I kind of like it even if they do take whatever it is they do a bit too seriously sometimes). Our tour guide, a woman who is a trainer for other air traffic controllers, at one point said, "You have 10 seconds or so to make contract with a plane and move on. If you screw up, there are hundreds of lives on the line." That's pretty serious.

TRACON's building is basically an octopus design, where each leg has 20 or so terminals with about 10 people in each, manning a particular physical area (like planes coming into Sacramento) in order to follow planes as they enter the region first. All commercial flights must fly IFR -- Instrument Flight Rules -- which means they have to be in contact with TRACON, in case they can't see or there is bad weather, or there is simply a pile up of planes that need to be moderated into an airport. Planes that fly VFR -- Visual Flight Rules -- don't have to contact TRACON, but some do anyway for a variety of reasons. TRACON has longer range radar than the local air controllers, but the longer range radar updates more slowly. So that is the trade-off between regional (TRACON) and local control.

Once TRACON has the plane logged, they make a little block of data on their screens (a different type of screen than the one shown above) that shows the flight number, its altitude, and other information that will help them keep planes apart, on track and moderated as they reach the range of the local control towers who then take over moderating the planes.

In the cycle of life for a controller (who has to quit at age 56 and can not be considered after age 31 to start training), they typically have military training or attend a special school after college, and then are trained at the local site. Our host said that for the first few years (maybe up to 10) controllers are pretty tense on the job, but after 10 years they relax some. She said the most dangerous situations come when people are relaxed, and less is going on around them, rather than more. That's when mistakes are made.

Another thing our host said was that they have to keep the chit chat down, because if there is an accident, they don't want to have some controller chatting away on the transcript, just before it happens. They are pretty businesslike when talking to pilots. She talked pretty fast, she said, due to the edgy situation needed to quickly regulate the flow and placement of all the different planes they are watching, and that's how she trains people. I know from riding in a friend's plane frequently where I can listen to lots of this talk, that they are pretty succinct, and yet both pilots and controllers have a kind of cultural humor that is pretty funny, in those few words they exchange, and this allows some kind of personality to come through often. If you want to check out what happens, here are some example live sound feeds from a bunch of different air control areas.

So.. what were the information systems like? Well, I thought they were fascinating. The premise in building, training for and using them is very different than say, the web based systems I typically work on in my day to day life. In fact in many ways, they had the exact opposite goals and metaphors I use to build systems and interfaces. First, they train their people between 6 months and 5 years on these system -- but our guide said 2-5 years is typical.

Think about that. Training your user for 2 years. What would that mean to interface architecture and design? You could certainly do a lot different with it than what we do now on the web.

Their top menu, interestingly, is literally a series of very-1993 buttons, big squares, in rows, maybe 8 across and 12 down, though all those gorgeous 22 inch screens are touch screens. Each controller has two of them, not horizontally placed, but vertically, in the workspace. Some of those buttons go to pages that help track planes, but I did note one, placed furthest away from the user's sitting position, for that day's cafe menu. It appeared that all possible items were options at the top level. Nothing appeared to be pushed back to a lower level or made less important or secondary in the interface other than two items described below.

When you go into the main menu items, there is little to cue you back, and in fact many of the screens were missing back buttons. Some had them and some didn't. But with that much training before you can even get into a real working station, it doesn't seem to really matter. You know the system inside and out, as well as how and what to do with it and all the planes you have to manage (typically 10 - 20 at one time).

A lot of information is stored in the user's head, and as new plane info comes up, only the abbreviation or shorthand block code describing the plane is on the screen along with various map-based data to place the plane. This means that instead of giving lots of data on one plane on the screen, the data is offloaded to the user and the screen just has the shorthand.

That shorthand for a plane is shown in the middle screen (below the menu in the top screen), which has the map with blocks of data representing planes. Their systems look much like map systems we use online in a way but with way cooler visualizations because they have radar and more info about airspace restrictions and well.. I don't know any web service that has radar. Imagine "Google Radar" overlaid on Google maps? That would be a cool product launch.

So in other words, what the information systems metaphor seemed to be was the exact opposite of what we do in web systems: TRACON systems are built with high mental overhead -- you have to know a lot to use and understand both sets of systems before you start to navigate because nothing in those buttons really helps you know what is below, other than the word on top. During actual use, when you enter and track planes, you get that overhead in the years of training you do before you can operate the system in play. The information systems below those button also have little style that would take any one piece of information and make it more important than any other on the same screen. Information is chunked or grouped a little on those secondary pages, but that's it. So there is no expectation that anything is pushed back or pushed forward, other than the menu, where each little button represents a page/function, and each page has the function represented.

Instead of the software deciding what is most important at the moment of use, and emphasizing it in some formated way, the user just has all of it equally represented and therefore has to decide what's necessary or relevant. In some cases, there was a mini system below a secondary page via a link, to find backup documentation on a plane (if the controller asked the plane to do something, and the plane wasn't built for it, they could check the specs on the plane) or on a small airport (to get backup data on landing strips and landing directions). But these seemed to be relatively rare use cases that allowed the backup information to be lower down to a third level.

Our other tour guide, a man who'd checked us in, did an introduction presentation in power point to explain the basics, and then finished up at the end. He told a couple of stories like what happened on 9/11. He said they grounded every plane everywhere coming and going anywhere in the country. It was eerie, because all their screens (which we were seeing, depending on scope, with somewhere between 20 and thousands of planes) were almost completely empty. Black. With little white map lines showing various air, altitude or other restrictions and weather. They spent three days watching military jets fly around, and that was it. Nothing else.

My take on this sort of system was that it could stand visual and architectural improvement, but that without a lot of study and planning, it would be dangerous to change it. And, the users are so adept at the system they have now, and have so much responsibility and pressure to perform quickly, that changes would likely be unwelcome. Extensive study of user behavior and needs would have to happen, and then extensive testing would have to follow before anything could be put into practice. I can see why they maintain the same system (it's not from 1993 though.. it's much more recent), and just update it with new air space data and plane info, and don't do much to mess with a working system.

But it was still fascinating to see the TRACON information and understand the motivations for its construction and use. And comparing that to what we do building web systems? The best!

Fiber Optics in Sherborn Massachusetts

I'm visiting with some friends in Sherborn Massachusetts. They previously had dial up internet access, but sometime in the last two years, everyone (3,000) in this town, as well as more surrounding towns, got fiber optic lines put in by Verizon.

They have 5 mbs of downstream service for $35 a month, and if they pay $7 more per month, they can get 15mbs. It's rocket fast, so fast, as my host says, "it's too fast to take advantage of much besides video and VOIP because no one else has a fast connection to talk that fast with you." But it still rocks.

Everywhere I go in the Bay Area, work, home, friends offices, public places.. I wait for every website, video, voip connection, etc that I use. It's just amazing the contrast here. And every window I look through in my host's house has gorgeous forest and fall colors .. it's at least 100 yards to the next house., and all the houses here have that sort of spread. How do they do it when we can't get this in the denseness of Berkeley, San Francisco, Mountain View?

I'm sure the telcos that took $200 billion from the FCC and then didn't install fiber optic service have some excuse, but it's BS. They just need to install it since we paid for it, and then we can all move on.

James Cicconi of AT&T On Net Netrality

James Cicconi, Senior Executive VP Legislative and External Affairs for AT&T was at Esme Vos' Muniwireless conference yesterday, spewing what I would kindly call the greatest of spin, and unkindly as BS.

Net Neutrality is not about people telling network providers what to charge for tiered service. That's bull. Net Neutrality says that video packets, no matter where they come from, will get through at the same rates. Same with text or photos or VOIP or anything else. The network can't under Net Neutrality distinguish and discriminate because it doesn't like where something came from or the place the packet came from didn't pay the telco's any money to prioritize the packet.

To quote muniwireless (emphasis is mine):

It's Day 2 of the Muniwireless Silicon Valley Conference and they have an executive from AT&T talking about municipal wireless networks.

AT&T has not changed its tune. It is still against cities using public funds to compete with private enterprise and believes that communications should be left up to private firms like AT&T.

James Cicconi, Senior Executive VP Legislative and External Affairs for AT&T claims that there is no duopoly and there is enough competition in the market for telecommunications services, so cities should stay out.

What is AT&T's position on net neutrality?

Net neutrality is a challenge for all companies. You spend billions to deploy your assets and net neutrality means someone telling you what you can do with your assets - what you can charge, tiers of service, etc.

"All bits should be treated equal" is a problem for network engineers because one bit is porn another bit is heart surgery, another is email, yet another is voice, another is spam. That everything should be moved equally end to end is ludicrous. It's a more costly way to do things. It's not efficient, according to AT&T.

AT&T cannot build and maintain assets quickly enough to meet the demand. They are spending $19 billion this year. Some of the demand is driven by video. What happens when people start delivering high definition film? They can't build networks fast enough! What's the answer? Effective traffic management.

The antitrust laws can deal with the problems of net neutrality (side note: unfortunately these are not being enforced today). Why should AT&T want to degrade traffic? They will go to someone else (side note again: in a duopoly, you've got Comcast which has been blocking Bittorent traffic).

I don't know about you but where I live and work, we have two choices: AT&T for dsl or Comcast for cable internet access. They are both Mid-band services, and not great but better than dialup. And we pay exorbitantly for them compared to other countries.

So of course they want to take their AT&T/Comcast duopoly and spin Net Neutrality as being all about people interfering with their pricing models for tiered service when it's really all about prioritizing packets. They want to divert attention from the reality which is that they want to put their videos through first, their media, their VOIP or media/VOIP from people who've paid them off. Instead of letting users have what they want. The telco's want to own the pipes and the content.

It's wrong and we can't let the telcos win on this.

ESnipe Snaps Up Nothing

eSnipe is a service that a friend of mine at Yahoo told me about. He loved it, so I tried it. The way it works is you give them your Ebay login, and an auction you'd like them to bid on for you a few seconds before the end of the auction. They put through the bid and mostly, with them I've won, until yesterday.

I've gone from a happy user on the first few transactions to an unhappy user. Not a great trend on their part. I think Ebay may be causing trouble for Esnipe (via anticompetitive behavior), but the reality is, I can't have Esnipe causing problems. I can't trust them to bid for me. Details below on the situation.

First, eSnipe lets you make three successful Ebay bids for free. Then, without really explicitly telling you, you can do three more, but Esnipe charges you 1% of the auction, on credit. Example, if something cost $100 on Ebay, they charge you $1.00 for 100 points in their system. It was a surprise when they stopped me from making a 7th purchase because they wanted money for "points" which I didn't understand at all, and they didn't do much to make clear what points were. The surprise was mostly due to the lack of notice on previous transactions, which I think they should have given. And then at the time of the 7th bid, when I needed to make the bid and run off to a meeting in a hurry, they wanted money. So I paid through PayPal, $10, and then they said they needed 7 days to put through the transaction, even though I know that PayPal notifies them immediately of the payment from me and I can't take the payment back.

UGG. That was so uncool. So I made another Esnipe account, and made the bid, and went on my way. But not without significant frustration and lack on information. They weren't helpful and the interface doesn't give you a lot of information about what's going on.

But the latest disaster happened yesterday. Far worse than the stupid payment fiasco. I put through a bid, and eSnipe confirmed on the page and in email that all was well. Then, for something I've tracked for 10 days on Ebay, I sat and watched as the item went for 30% of my top bid with Esnipe, to SOMEONE ELSE. Esnipe didn't put through the bid effectively.

I went to my account at eSnipe, which said "eBay weekly bidding limit" under status. WHA? Are you kidding?

I've had this Ebay account for years, and the Paypal account associated with it. I'm not a big fan of Paypal, in fact I think some of their procedures and behaviors are pretty ridiculous, but I do have a solid track record with both companies and a good history over several years.

So I ask eSnipe, what happened? I'm pretty annoyed that they've done this, and now made me lose the auction.

They respond with the following message:

eBay rejected this bid because you have "exceeded your weekly spending limits". This is a new security measure that eBay has added to their site. You can read the details on this by going to these eBay pages:

http://pages.ebay.com/help/tp/activity-limits.html
http://pages.ebay.com/help/sell/add_verification.html

They will sometimes give this result if you are spending more than normal or your account has been inactive for a while and suddenly you are spending a lot, or have a fairly new account. Had you placed this bid directly, eBay would have prompted you to some verification process, but since Rovatron (eSnipe's system name.. like users even know what this is? I had to go read their FAQ to figure that one out...) can not answer questions for you, the bid was just rejected.

Um. Not true. I'm well within my normal Ebay use (about 10 a month, and this month so far, I've done 2 auctions, both of which I won and paid for on the spot). This is, according to the Ebay info, something that was instituted to keep new Ebay bidders in line, if they don't pay.

In the meantime, I use the Ebay interface to bid on another item a half our after the eSnipe bid fiasco, on something from the same seller. Which BTW is a seller I've purchase 5 things from over the past few months. Perfect transactions. And the bid was not at all rejected, and Ebay didn't ask for any verification.

So eSnipe writes back again:

eBay is the one who gave you this result of bidding limit and not eSnipe. I cannot explain to you why eBay is allowing bids to go in through their site directly and putting up the bidding limit notice on others. I can tell you that we are getting this notice which is why we are reporting it back to you.

http://pages.ebay.com/help/tp/activity-limits.html

Until you do the extra registration through eBay, your bids may continue to be rejected.

I reply: no way. I went to the link, which said these sorts of limits are put on new users (I'm in since 1998 as a regular customer of both EBay and Paypal), on users with unusually high bid rates out of sync with normal bid rates (I'm absolutely within my normal ranges), on users with low feedback scores (mine is quite high, with high recommendations), on users who need to submit a credit card to Paypal (PP has had credit card and checking info for me, consistently and unchanged -- other than expiration updates -- since 1998).

So what gives? I think Ebay is blocking randomly eSnipe transactions though I have no proof. And I think Esnipe probably knows or suspects this, but is taking bids anyway, with no notice to their users that they may fail, and causing their users to lose items (had they put through my bid on the items I tried for yesterday, there would have been no issue, I would have won well within my maximum bid, and won the auction. Damn!)

It may be anticompetitive, Ebay's actions, or something else. But the upshot is, I can't trust eSnipe anymore. I can't use them. Bummer, because the concept is great, and when it works it's great. But they don't have the interface, the documentation, the support emails or the bidding together. It's not just the bidding issue though that is significant. They can't tell me about their other issues and help me through them well at all. eSnipe has become high stress and unhelpful. And that's just not cool.

Update: a couple of people have suggested that I should share what I was buying: I had wanted a painting, and that's not something you can replace with something else, exactly. It happened to go for only 33 Euros, which is very reasonable, and so, you know, I was disappointed after getting my hopes up.

Warp Speed Internet, But Not For the US

See the WaPost story here: Japan's Warp-Speed Ride to Internet Future By Blaine Harden.

I've talked about this before.. years ago.. that we were falling behind here in the US and it would affect our ability to innovate. Well, it's now years later, our regulators and telco's have failed us, and we are way way way behind.

TOKYO -- Americans invented the Internet, but the Japanese are running away with it.

Broadband service here is eight to 30 times as fast as in the United States -- and considerably cheaper. Japan has the world's fastest Internet connections, delivering more data at a lower cost than anywhere else, recent studies show.

In a Tokyo demonstration using ultra-high-speed broadband, a life-size, high-definition image of a distant colleague is projected onto a screen. In a Tokyo demonstration using ultra-high-speed broadband, a life-size, high-definition image of a distant colleague is projected onto a screen.

Accelerating broadband speed in this country -- as well as in South Korea and much of Europe -- is pushing open doors to Internet innovation that are likely to remain closed for years to come in much of the United States.(emphasis mine)

The speed advantage allows the Japanese to watch broadcast-quality, full-screen television over the Internet, an experience that mocks the grainy, wallet-size images Americans endure.

Ultra-high-speed applications are being rolled out for low-cost, high-definition teleconferencing, for telemedicine -- which allows urban doctors to diagnose diseases from a distance -- and for advanced telecommuting to help Japan meet its goal of doubling the number of people who work from home by 2010.

The regulators and Congress gave American telcos $200 billion dollars to rewire for fiber. They didn't do it. They pocketed the money. And now we are in really big trouble.

Legacy Media Gloats About Fake Steve Jobs Outing

There's been a lot of gloating over the outing of the author of the blog, Fake Steve Jobs (see article below this link for gloating), which has been written by Dan Lyons, a Senior Editor of Forbes and where Brad Stone of the New York Times broke the story yesterday.

The interesting thing to me is that legacy media in NYC thinks that they are so clever for finding Lyons, or at least Lyons and his friends think this, when lots of bloggers on the west coast have spent months speculating and looking for FSJ. They even have pointed out how geeky the new media people are because they were doing things like checking the headers of email from FSJ, which appeared to point to a Boston location but that was all they got.

Brad Stone figured it out because he looked through Manhattan back channels to find a book agent was shopping a book proposal from FSJ (a few months ago, the book is due in October), and because the agent was telling publishers that the writer was a "the anonymous author was a published novelist and writer for a major business magazine," he could compare the FSJ writings with Lyons blog and figured it out.

Well, the point is that if you are part of legacy media, and more specifically, are in Manhattan, you probably have access through your network to other media entities in Manhattan. (Note that Brad Stone doesn't live in Manhattan, but as a member of the Times, I'd consider that very strong "local" access.) Of course if you get a clue through people you know that tell you about the FJS publishing package and mysterious writer status, you figure it out this way. No bloggers out in the hinterlands have that sort of access to be able to put this together.

It reveals both positives and negatives about legacy media, that bloggers have know for a long time and that legacy media has tried to sweep under the rug now and then. Bloggers over the past few years have pulled back the curtain on legacy media, and legacy media is now better for those conversations. But it's hard to break any story like this without both online and off-line access. You need both, and so it's not so much that Brad Stone of the NYTimes did the breaking, but rather Brad Stone, person with access Manhattanite and local connections to the publishing industry who broke the story. Could have been anyone with friends in publishing who figured it out.

But to say that it's "ironic" that a legacy media journalist was the FSJ and legacy media broke it may be true, but it's gloating at the same time. And it's that gloating attitude that got legacy media into trouble in the first place, and made the public so angry with them, after things like Jason Blair and Stephen Glass. So it may be a small victory, but often what legacy media doesn't understand is that there isn't a battle. Many conferences over the past few years have devolved into an either or battle where public demonstrations of these attitudes come out. But both sides need each other and if one went away, the other would be in deep trouble, at this point.

Bloggers drive a lot of traffic back to legacy media, as they discuss news print stories, and journalists get a lot of stories out of blogs (I first noticed this five or six years ago when my intellectual property stories started to get lifted a couple days after I published them.. by.. the New York Times. They added to them, but they took the arguments, the phrases, and never said a word about where it came from. I noticed this same thing with other IP bloggers like Law Meme and Copyfight as well.) News reporters report and bloggers opine. Bloggers opine and poke and reporters go looking for more.

We also need reporters who have access, not so much of the example above, but with access to important things, in order to get good information out. This is critical for the democracy and the reason reporters are given special dispensation in the Constitution. But reporters also need not abuse that power through gloating or arrogance.

This isn't about getting over on new media by the legacy guys. Many legacy guys have become new media guys.. Lyons and Stone each have blogs and write on the daily. The much more interesting story is how legacy media is using new media tools, changing their habits (like writing anonymous parody blogs !!!) and how everyone is interacting on and off-line to get better knowledge as well as entertainment. I have to admit, I have loved reading FSJ the past year. It's been totally entertaining.

Check this out: Fake Brad has come on the scene. Maybe a little less than FSJ-clever, but it is amusing. And ironically, it's amusing partly because it makes fun of legacy media arrogance. And as Geek.com notes, FSJ and FB may be the start of a whole new genre.

US Internet Speeds are Really Slow..

Via Dave Farber's IP list from Press Etc:

Average broadband download speed in the US is 1.9 Mbps. It is 61 Mbps in Japan, 45 Mbps in South Korea, 18 Mbps in Sweden, 17 Mpbs in France, and 7 Mbps in Canada.

I've talked about this before.

Americans are falling further and further behind, in socializing with technologies like high speed interent access as well as cell phone tools and service that are much more dynamic than the rest of the world has. This is due to terrible public policies around these technologies and selfish companies who provide the services in monopolisitic ways.

Two to four years after I first talked about this, we are further behind than ever. It's appalling but you can read about the $200 billion scam on the US by Verizon, QWest and the Bell companies here.

BOB is so ALIVE

So, secretly. Who is your favorite guilty pleasure read in the blogosphere? Mine is Bob Lefsetz. Actually, I've subscribed to his email list the past few months, which is easier because he only posts once or twice a day and I really want to read it the minute he puts it out. He's HILARIOUS.

I have been blogging about the music industry, IP, security and privacy, the napsterization of anything but in particular digital media, and how stupid legacy media is for about 5 years. So it's not like he's telling me anything I don't already know. But he's just so DAMN'D funny that I can't help it. He's so totally alive and passionate about music, the music business, the integrity of some people and the loss of control by others. And he podcasts about it too, like the Stubhub/Ticketmaster thing.

SoI love reading him, the minute he puts anything out, because he totally believes! It's great stuff.

Don't expect to see anything you haven't read on the music business before, but do expect to be completely and utterly entertained.

Thanks Hank for turning me onto Bob.

Putting Order to the Chaos of CES, Not to Mention Making New Media Proud

at the Bloghaus. By Podtech.

John and Linda Furrier and Maryam Scoble rock for putting it all together.

Wow. Made the chaotic CES scene fun and cool. And the video uploading.. 80 mbs up and down. It rocked!

An example of their work, vlogging the CES 2007 Keynote: Ed Zander, CEO Motorola:

Bloghaus is the best!

Dabble Playlist Contest: Take the Dabble Challenge

Dabble, my company, is running a playlist challenge contest.

Details are here, and today's challenge is for playlists on funny animals. There is a challenge each day where winners each get a tshirt and entry into the finals. On Dec 31, the finals made by the winners from the rest of the month will compete for $500 (grand prize) and a video iPod (runner up).

Go make your playlist or favorite the best ones!

The Future of Video

I'm moderating a session today at 10am on the Future of Video at the Web 2.0 Conference:

Future of Video: We Just Wanna Watch, Or Do We?
Mary Hodder, CEO, Dabble
Josh Felser, CEO, Grouper Networks
Mike Folgner, Jumpcut/Yahoo
Tod M. Sacerdoti, Founder and CEO, BrightRoll

YouTube has done a terrific job of leading the way for video 1.0 online, where watching is everything. But as people get more comfortable with watching video online, the old broadcast relationship they had to content changes, and they start to want more. What do users need for video 2.0 where watching is just a part of the story, where remixing and online editing, arranging, playlisting, searching, and most importantly, discovery through other sources are expected by users? What are the barriers and what is being done now to make regular users into power users, and give everyone more control and access than just watching in an on-demand style?

Come join us if you are at the conference. Should be a great discussion!

Update: Here is a Wired blog post on the session.

One Web Day! It's here!

Celebrate how the internet has changed our lives and made it better with people around the world!

Okay, that's a lot of explanation points, but the internet has made my life so different than it was before. And so much better.

First wave was email and research, in 1992, when I needed Supreme Court case law from the Cornell Law School website, or a news article from Dialog, or bulletin boards. Oh freedom from the law library for every little task!

Second wave was IM, more email with many more people and the web. Instant. Communication. Conversation. And all that primitively laid out info on the web. That was never so easy to get before.

Third wave was blogging (which has totally changed my life the most of all these waves) and lead me to research the live web, search algorithms based upon human behavior in many different types of circumstances and make my company. And introduced me to a whole huge circle of friends and colleagues.

Right now I'm staying with a friend in Amsterdam who I first new on the web. She's amazing. And her husband. Both of whom sustain themselves very nicely through their online blogs, which are entire businesses where the storefront, or office space, as it were, resides on these blogs. Partly our friendship bloomed out of respect for each other's work, visible online. And partly because our work on the web led us to meet in person and gave us a rich foundation to start our first conversation. About fashion. And online advertising and how we each hate marketing, are geeks, but wish the right shoe ads could show up in the right places, without violating our privacy.

Well.. it's One Web Day.

Tell your story on your blog, on the One Web Day wiki, or anywhere you like. But let people know how much richer your life is because you can communicate over the collapsed barriers of time and space the internet allows.

I'm going to be in London filming a proclamation from the Lord Mayor on One Web Day.

Throw up your own video at Blip.tv, tag it "onewebday" and it will end up in Dabble here: One Web Day video page.

Or throw your video up at whatever video hosting company you choose, tag it onewebday, and we'll do our best to get it posted to the One Web Day video page right away.

See you later today in London!

Thought Fashion: Are You In or Out?

It's true. I peeked.

Yes, I downloaded the AOL files. And I peeked. Why? Because I wanted to write this blog post and I wanted to see for myself what sort of gestures people were making as they searched for porn or socks or how to bury their pet birds or wives they'd just killed. I also needed to see the form the data was in. And I'm a voyeur just like everyone else in and around this story, and I wanted to rubberneck my way into other's private intellectual spaces.

But it's not right. The part where I and every news outlet, blogger, reader and looky-loo has been engaging in, judging people by their searches, making assumptions and behaving as if we ourselves have never made any searches or expressed any thoughts that would not look funny to someone else.

It's also not right because the data is personally identifying. Reporters have been tracking down people based upon their searches. It's not that hard, if you yourself are a good searcher.

What was it Bob Blakely said? About how "dragging all human behavior into the public is literally totalitarian." He is the chief security and privacy scientist for IBM's Tivoli Systems. "If you erode privacy, you erode liberty, because people don't tolerate things going on in front of them that they don't approve of." I was struck by how succinctly he answered the question that is always asked of people who object to the government or some other large and powerful entity as compared with you: What do you have to hide? If you're not doing anything wrong?

Every article on AOL's mess up that says something like AOL's Disturbing Glimpse Into Users' Lives is buying into this whether they know it or not. Thank you CNet for reaffirming our intolerance.

Let's get clear on the definition of "aggregated" data. For us geeks, we use this term often, as we reassure those whose data we work with that aggregation means we are removing anything personally identifying, and placing it with other user's data, so that it's just a pile of anonymized data that could never be distinguished by the person. An example might be the aggregation of all the searches on "dog," where who did them is removed but we know that 38 people searched on that term during a particular hour and day.

But users don't think that way. They hear the word, aggregated, and they think the data handlers are aggregating everything the system may know about just them, specifically and personally, and lumping it all together. Talk about miscommunication. And it terrifies the non-geeks.

What we really should be saying is that the data is "anonymized" and therefore you are safe. AOL's data was not safe because it was not anonymized, and for users, it was their definition of aggregated.

The AOL data which lumped each user's searches together with a user ID over three months, making profiling and finding them easy, meant that AOL provided enough data in some cases to indicate a lot about who the data related to very specifically. Leading to judgments by the rest of us. About the people who do or think things on the edges of society.

And why is this wrong? Because it hurts people. It makes them feel defensive about their own thoughts and ideas.

So, well, if you aren't doing anything wrong, what *do* you have to hide? Well, everyone has something they do or think about that would be an edge thought or that in one context would be in the middle, but in another, must be defended as it resides on the edge. And that would be disapproved by someone. Something the rest of society might not tolerate.

Intolerance leads to the totalitarian. We, the human race, have been intolerant since the beginning of time. What we are intolerant of is a moving target depending on the fashion of the day. In the 30's in some places it was fashionable to be intolerant of Jews and gays. In the 40's it was Germans and the Japanese, and in the 50's communists and socialists. In the 60's it was civil rights proponents and hippies and in the 70's liberals. In the 80's we were back to communists, and in the 90's it was Hispanics (remember all the state propositions outlawing them from medical care?). And what is it today? Islam? Are thoughts you think today and the cultural references associated with them that are in the middle going to fall to the edges in the next decade?

We have used the fear of all these intolerable people and their thoughts as excuses to hunt for more proof of their intolerableness by surveilling everyone in society and searching through all the detritus of our lives. With digital data more available, we think we can find the proof we need in these edge thoughts. And then we will persecute the people having them. And what better way to do it now that the internet, ISPs and heavily used search systems can provide one or another level of very personal, thought data. Search terms, or a database of intentions as John Battelle has talked about so much, are one slice of your data that tell a lot about you. And if we can get it in a neat little file, machine readable and searchable and quantifiable, then well, why not?

If you believe that sacrificing freedom to keep freedom is the way to go, then you probably don't see any problem with demonizing people who have thoughts you don't like. Especially if those thoughts are in the form of passing gestures such as search terms plugged into a browser.

But until we decide (or default) into a Minority Report society (and change our constitution), we are not yet convicting people for thinking things. Everyone has had the thought that they'd like to kill someone once or twice in their lives. But people, the vast vast majority, don't do it. The idea that we demonize someone for searching on this, which is a gesture I would put into the fleeting thought category for almost everyone, is taking an edge thought, which we all have from time to time, and putting it firmly under the scrutiny of the middle. I believe we really only want to find people who make serious plans to hurt others, or actually carry it out. That is what our law it based on, and the premise of our society. But to track everyone, their searches, their every digital gesture, and expose it in one or another ways is going to be troublesome. And it begs a question I've asked before: is your digital identity your personal intellectual property? Is your Google identity yours or someone else's? And by extension, is your clickstream a personal expression (carefully chosen and shaped by you)? In other words, can you copyright your clickstream and exert ownership?

There are at least two choices. One of them is to do what we are doing now: have ISPs and search services collect this data, and when asked by the government, have it turned over. But that means the data is still in many ways secret. Of course the companies don't want the data getting out because it is proprietary. And neither does the government, because they don't want anyone to know quite how much is out there about you, in case you are trying to cover your tracks or you want to defend yourself. But having all the data, the government has the upper hand. And secrets are powerful. How do you show, if you are being accused of something based upon your searches, that everyone else searches on those same things too? That it's actually a social norm? If you can only ask for your own searches to defend a case against you, and not everyone else's, in order to compare yourself to it, you won't be able to argue social norms which judges rely heavily on when making decisions.

But there is another choice. And that brings up the Attention Trust premise (I'm a Board Member) which is that people own a copy of their own data, no matter where they do things: Amazon.com purchases, Google searches, or AOL clickstreams, or anywhere else you might land in a browser on your computer. As a co-owner of your data, you can take it anywhere and do what you wish. There could be many business models built upon this data controlled and shared by users. Google takes all the data they collect and plugs it into AdSense. If lots of users took their own data and made it available voluntarily, a new and more 'open source' style AdSense could be created.

But much more importantly, something like Steve Gillmor's Gesture Bank, where users opt-in their clickstream information, in an anonymous form, exists to open up this kind of data. The Bank will make the aggregation of anonymized data available to anyone for any purpose. While this may lead to businesses working from this pool of searches and clicks, it also means that a growing pool of data is there to show the edge thoughts and potentially unpopular ideas people may exhibit. The pool can be used to defend against totalitarian efforts to single out in secret those who are out of fashion politically. Which may turn out to be you. Or someone who uses your computer.

That I think is far more important than an open source AdSense, though a business built upon this data would likely justify and make a better case for us to have a Gesture Bank of ideas and thoughts that support political freedom.

Seth Goldstein and Steve Gillmor already offer Root. net users the opportunity to put their data into the Gesture Bank if they wish, though any person can contribute to this anonymous pool of user data. And for that matter, attention streams can be sent to multiple services.

And, at the October 4 Attention Conference, Steve and Seth will announce Attention Soft. Stay tuned.

OpenID2 Developer Info Day Aug 10th Bay Area

From Kaliya Hamlin:

I am really pleased to announce that we have an OpenID Informational Evening for Developers August 10th 6-9 in Berkeley at 2029 University, Upstairs.

The Big news is the community has converged and figured out the authentication layer - OpenID…OpenID is just the authentication layer - but on top of this ad hoc standard lots of cool stuff can happen. The goal of the evening is not to geek out on identity but to connect with a developers working on applications that require users to login.

Find out more about what it is…how it works…how you can install. The incentives to learn are high with the $5000 bounty for having OpenID in Open Source projects.

Presenting and answering Questions

David Recordon formerly of Live Journal/Six Apart now of Verisign will be presenting a bit about the origins of OpenID but most importantly how it works…and how you install it.

Andy Dale from ooTao will talk a bit about i-names and how they work with OpenID2 and looking forward to what comes next after authentication - profile sharing. ooTao is also data sharing are running ibroker services.

Mary Hodder CEO of Dabble will talk about the work happening around the development of itags.

I am helping coordinate the evening please RSVP to me - kaliya (at) Mac (dot) com and feel free to ask me any questions.

If you know a developer - pass the word along.

ps. for all you Technorati guys who keep having questions, now is your chance to ask the guys who know.

UPDATE: Scott Keveton from JanRain will be there too. He just posted an OpenID walk through on his site.

UPDATE 2:Dick Hardt from Sxip will be in town and will also be joining us for the evening. Hopefully he will share some of the cool stuff sxip is doing with OpenID.

Dabble Launched, etc.

Some people know that I have been busy launching Dabble, the company I founded for searching, browsing and organizing media.

That was pretty much a month underwater, as we worked around the clock for 30 days in the most insane schedule. But we are lucky, as we have an amazing team and we were so happy with the response to the launch which was overwhelmingly good.

We are now working on fixing things for users and adding some new features they've asked us to do.

It's great, but also tiring. The best part is when someone tells us they are really enjoying using the site. It feels really nice.

If you have feedback, send us a message at feedback at dabble.com. We are looking for ways to improve and can use all the help we can get. Thanks!

Dabble Blog Goes Live

The Dabble Blog has long been inside our invited beta pages, and not accessible.

It's now public, as we move toward opening our site. We'll be putting all kinds of things on it including news about Dabble, development issues and interesting things we see people doing when they use Dabble.

But we'll also use it to point out cool media and users doing interesting things, and post videos (we aren't a hoster.. we link to hosters and act more like a guide to video, made by users, as well as straight search and browsing).

Check it out.. it's cool!

Core Values at Bloggercon

Mike Arrington is doing the next Core Values session at Bloggercon IV. I want to wish him well, and note the information from the Core Values session at Bloggercon III that I led, because I think that session was extraordinarily productive.

In fact afterward, many people kindly said that they felt it was most useful and interesting, because they left with something tangible (the list) and they really liked that rather than me telling them what the values are I see online, or that I feel are important myself, I just asked questions, and let them come to the conclusions about the values they share and the controls they felt should exist to support those values. They appreciated the light touch guiding them to find and develop their own conclusions.

Here the list the group of 80 made during the session, of what they feel are core values for the blogosphere:

Things we value:
Democracy
Non-exclusivity
Attribution
Transparency – disclosure
Innovation
Personalization
Accessibility
Honesty
Creativity
Knowing who people are
Editorial Independence
Connectedness
Anonymity

Things we devalue:
Power law economics
Lack of Attribution
Anonymity
Wuffie-hoarding
Links for money

Good luck Mike! I'm sure you'll take us to the next level. I hope the last sessions list is useful as a starting point.

Anti-Copyright and Anti-Fair Use: The Broadcast and Audio Flags

Broadcast and Audio Flags are provisions in Senate Bill 2686, up on Thursday. They are bad for users, bad for balanced copyright, bad for fair use, bad for innovation, and bad for new companies (including Dabble).

This is about incumbent media companies fearing the internet, much like the RIAA in 2001, and trying to get the government to protect them against digital media, instead of working with it to create new business models.

Call your Senator (there are some numbers below provided below in an except from an EFF email.

I just called Senator Boxer's office (212 number is below, or SF: 415-403-0100) to register my opposition, and I noted that Boxer's office takes phone comment anonymously. Interesting.

From EFF:

* Action Alert - Tell Your Senator To Take Out the Flags

The Communications, Consumers Choice, and Broadband
Deployment Act of 2006 is a monster name for a monster bill
-- in its latest form, it contains 159 pages of densely
plotted telecommunications reform. But while politicians
struggle with its major clauses, the RIAA and MPAA have
piggybacked their own agenda: the broadcast and audio flags,
which restrict innovation and legitimate use of recorded
digital radio and TV content. Your call today could force
the flags to find a home of their own.

The Committee markup of this bill is on Thursday, and your
Senator is on the Commerce Committee. One last push from
you could get Congress to remove the entertainment industry
mandates from the bill.

IF YOU HAVE FIVE MINUTES

Please call your Senator (numbers below). Here's a sample
script:

STAFFER:
Hello, Senator Lastname's office.

YOU:
Hi, I'm a constituent, and I'd like to let the Senator know
that I don't think the broadcast and audio flag provisions
belong in S. 2686, the Communications, Consumers Choice and
Broadband Deployment Act. These are anti-consumer
provisions, which would give the FCC far-reaching powers,
and give the entertainment industry a dangerous veto over
new technologies. I hope the Senator will insist on
excluding these provisions on Thursday.

STAFFER:
Okay, I'll let the Senator know. Thanks.

Chairman Ted Stevens (AK), (202) 224-3004
John McCain (AZ), (202) 224-2235
Conrad Burns (MT), Main: 202-224-2644
Trent Lott (MS), (202) 224-6253
Kay Bailey Hutchison (TX), (202) 224-5922
Gordon H. Smith (OR), (202) 224 3753
John Ensign (NV), (202) 224-6244
George Allen (VA), (202) 224-4024
John E. Sununu (NH), (202) 224-2841
Jim DeMint (SC), (202) 224-6121
David Vitter (LA),(202) 224-4623
Co-Chairman Daniel K. Inouye (HI), (202) 224-3934
John D. Rockefeller (WV), (202) 224-6472
John F. Kerry (MA), (202) 224-2742
Barbara Boxer (CA), (202) 224-3553
Bill Nelson (FL), (202) 224-5274
Maria Cantwell (WA), (202) 224-3441
Frank R. Lautenberg (NJ), (202) 224-3224
E. Benjamin Nelson (NE), (202) 224-6551
Mark Pryor (AR), (202) 224-2353

Respecting Open Space

Open space, in the camp conference style, requires some key elements to work well. I'm noticing after watching two different events develop that they may be missing what is important about Open Space.

A couple of months ago, I attended a conference on the east coast. The organizer told me he wanted to do an Open Space day, the day after his event. That day arrived, he bailed, and there were six of us who actually attended. And he insisted that I go, even though I really have a lot of other things to do. The Open Space day was meant to brainstorm ways to organize Net Neutrality support.

After the event, others who attended it suggested that Bar Camp a failure. Well.. I totally disagreed. They thought that somehow, calling an open day "Bar Camp" would make it happen.

They had a wiki with about 22 names on it, and a stellar group of people slated to attend. They had great space, in a lovely lawfirm with wifi, and everything we might need. What they didn't have was a leader to organize the space. Although one person who attended in the middle of the event suggested that if things weren't working, he could vote with his feet. And he would if he wanted to do so. Though considering there were six people in the room, it sounded more like a threat: you don't do what I want to today, or I'll leave. So everyone started doing what he wanted. The point of course of "the law of two feet" is that you don't stay somewhere where you aren't learning. But that applies to Open Space where there are maybe say 100 people, and multiple rooms where you can move and not be disruptive, not six people where you are an integral part of deciding what is happening. But with such a little group, that misapplication of that particular Open Space principle further caused the day to deteriorate as a camp. What little emerging leadership was happening was killed right there, though he wasn't wiling to lead. He just wanted everyone to do things his way.

Without a clear leader, supporting a basic framework for a day of sessions or some kind of plan, just didn't work. It wasn't the concept of Bar Camp that failed. It was a failure of the people proposing it and carrying it out.

Anyway, I'm wondering how Open Space is going to work at the Identity conference at Harvard, where today and tomorrow are regular top down conference days, where the broadcast model is followed. On Wednesday, there will be an Open Space day, led by Kaliya Hamlin and Jon Ramer. (I'm not attending this event, as I have too much work to do, but I'm noticing a trend here....)

I know the Open Space day is happening, more due to the Identity list I'm on, than the event web pages. After there was discussion on the list, I asked about the lack of information supporting the Open Space day on the website, and Paul Trevithick and John Clippinger did add a little information about the day on the session page, to let people know it was even happening. However, I had suggested on the email list that they make a page for the attendees to show that they were attending and add the speakers to the schedule and speaker's page. They did not.

The point I'm making is that I think people who do top down, broadcast style conferences are interested in what's happening with camps and Open Space, but they don't understand the dynamics of them or Open Space sensibility, and so in applying top down controls and information styles to their camps, they potentially harm the good that can come from the camp. And since the leaders of the camp are not traditional speakers, the organizers of the larger top down conference probably think they don't need to list the camp or Open Space leaders as speakers on the larger conference site because the camp facilitators aren't speaking in a traditional way. But this is not true. Listing them is critical to fostering the process of the day.

For example, we know from past successful camps that having a page where attendees say they are coming is key, because the agenda is made the day of the camp. Therefore, people choose to attend because other interesting people will be in the room, not based upon pre-arranged sessions. Secondly, the leaders of the day are key. They have to balance the right amount of support for the Open Space while leading just a little so that attendees make the agenda the morning of, and that things are pulled together at the end of the day. People choose to come, or not, based on who will be leading.

Currently, the leaders of the identity Open Space day are not on Harvard'sthe speaker list, nor does Harvard's the schedule note them, even though speakers the previous two days are listed on the schedule with their corresponding sessions.

I believe the Open Space day will go well due to Kaliya's and Jon's attention, because at least Kaliya has done this before (I don't know about Jon's work with Open Space) and understands well the dynamic needed to make this kind of day work. But the fact that the Open Space day at Harvard's Identity Conference has not been adequately supported with proper information at the event website shows the lack of respect for the dynamics of this kind of event. Since there is no sign up page, they will likely have a vastly diminished attendance compared to the broadcast conference days. A signup page might have actually brought in more people if they'd opened it up to more than just the attendees the first two days. In fact, bringing in new people to understand Identity in technology development is very important and this is a missed opportunity as well.

I do wish them good luck with it, but I wish that the Paul and John, with control of the conference website, understood better why what they have done with both the attendees of the open space day and the leaders may not help the day succeed as well as it should have. They can't blame the camp style for this, but rather themselves. If they day succeeds, it will be in spite of these problems, and due to Jon's and Kaliya's personal networking and leadership for the day.

idmashup06

Net Neutrality for the Little Guys

USA Today interviewed me and some other folks the other day. The article is here:

Internet Fast Lane Plan Worries Small Companies by Michelle Kessler.

Basically, it's that part of AT&T's and the other telco's new internet pricing plan, where they would charge the provider of the material to send their material through to subscribers, that is the problem.

As I've said before, we didn't make the internet to turn it back into cable tv.

We Didn't Build the Internet to Turn It Back Into Cable Tv

You know, the kind of cable TV where big entertainment companies pay off cable companies to get their channels on your set top box?

Congress didn't accept it, so net neutrality lost.

So we are keeping the system that started a year ago. It's the one that will make the internet like Cable TV.

It's critical to innovation, our companies (mine is Dabble.com) and to freedom of speech that we have a neutral net, where anything can move across it, where there is no fee to get some piece of information through to someone who wants to see it.

This isn't about tiered pricing. This is about who's packets paid the telco's fees.

This is about Hollywood keeping us from speaking, because if I'm watching my friend's video, I'm not watching Disney. Hollywood stands to benefit the most, after the telco's who charge the fees.

And Disney can afford to pay off the telcos to pass through their info, but my friends can't.

I'm going to Vloggercon today!

Hope to see you there, though you should know they are sold out!

It should be a great event. I'm speaking tomorrow on the Mashups and Remixing for Vloggers Dave Toole, Josh Leo. JD Lasica, David Dudas and Jan McLaughlin.

But we talked yesterday about making it an audience discussion, which I think is much better than a panel.

Josh Leo's We Are The Media, three times in one week!

I also suggested we play Josh Leo's video from the first Vloggercon because I think it's a great representation of user generated content, mashups, and for many other reasons, it's representative of the interesting and entertaining things going on online. In fact, I played it at the Culture, Commerce and Public Media conference during my session last Monday on User Generated Content with Kenyatta Cheese, Sam Klein, Dave Marvet. I also played the first two minutes of Kent Bye's Overview of his Echo Chamber Project, as an example of news and commentary video made by users, The Guinea Pig Dreaming video, and the Bush Blair Endless Love remix. I wanted to show the audience there (typically from archives of TV, PBS or other libraries of video projects) that users were doing an interesting variety of things.

That last one got a really big laugh.

I also played Josh's We Are the Media Video again yesterday at The Hyperlinked Society conference at UPenn and the Annenberg School of Communication. The point is, digital videos are a series of edits, and each edit, with an in and out point as hypertext, is like a video map, of links. Since it was a conference on links, I wanted to show a couple examples of linking that working in ways other than what everyone there was talking about.

I also showed some Attention Trust data, with a visualization of links a user might use to see where he goes day to day.

Anyway, if we play Josh's WATM video again tomorrow, that will be three times in a week. It's that great. You should check it out.

Haven't we been here before?

Digital Maoism vs. Voice

Isn't that much like the issues we've looked at over the past few years:

Wikipedia vs. Britannica
Bloggers vs. Journalists
Remix culture vs. TV
Flickr vs. Getty Images
Wiki's vs. Blogs

All the talk this past week about Jaron Lainer's essay, The Hazards of the New Online Collectivism is another one of those 'or' things that keeps coming up around the internet. (Or Internet with a capital 'I', if the NY Times is your style guide.) As an aside, Sam Klein of Wikipedia at the on Monday, asked everyone to please (smile) stop calling it "the wikipedia." It's just "wikipedia." Ok, back to Wikipedia. So Wikipedia is not a replacement for Encyclopedia Britiannica. Instead, you use one for some things (I use wikipedia for finding links because Google's search results for many kinds of items are too polluted and unhelpful) and a reference like Britannica (well, not Britannica, but I have lots of other traditional old style references) for things that those top down, traditional reference sources cover better.

We use reporting from professional journalists for reporting, access to places individuals can't get into, and some kinds of news, and blog posts for voice, commentary, and some kinds of news and reporting. We use remix video for humor, smaller stories and short form video, and TV for long form, high production video. We use Flickr for the stream of photo images that comes from our friends and for some kinds of reporting, and we use Getty for.. well.. they are hard to use. So we don't buy a lot from them. Wikis are used for the collection of information around a topic or event, blogs are used for voice and commentary. Collective tools are used for collective action, and voice tools are used for voice.

The thing is there are choices, based on purpose, goal, need, process and style. And the choices are based on nuances that the arguments above cannot reasonable reduce to an 'either or' situation. A single thing is not meant to work in all instances. And the beauty of the internet combined with information technology is that together they give us lots of choices. Both for production and consumption.

I think the rest of the folks who responded to Lanier's essay did a great job of discussing the subtler ideas and arguments, so I'll let those stand as they were terrific. There is no need to restate the idea that some of Lanier's criticisms do not necessarily apply to Wikipedia, or that some others do apply, in specific contexts, but that wikipedia is supposed to function the way that it does.

I just wanted to point out that online, as everywhere else in life, we make choices, and the idea is to choose the best thing for the circumstances, not to expect that all things will work in all circumstances. The internet is no exception.

Net Neutrality

First, watch the video. And then the other video. And this other video. Yah. It's worth it.

Then, check out David Isenberg's most terrific eTel talk about your freedom to connect.

And check out Save the Internet. They have tons of great information.

Then read below. Here's how I see this:

Another way of looking at this issue of net neutrality is... remember the old Highway system. Where El Camino Real on the peninsula in the Bay Area used to be a toll road, where you would not get mugged and the road was nice and fast, but it was expensive. And the Alameda (parallel to ECR) was the slow road, which wasn't taken care of, where you would likely be ambushed and was free?

Well, that's what the telcos would like us to see when they talk about two tiers. And think about what that kind of road system does to the economy of information? It's not very democratic is it? This isn't just a small or large bag of potato chips. Or dial up and broadband. It's about whether we support basic services for all people to get information. Cause if you are on dialup, you are missing much that is useful and interesting about the internet.

Secondly, the part that's different about the types of information that would be available in the slow cheap road verses the fast expensive road (dialup verses high-speed bandwidth) is that the packets would be treated differently.

The perverse part of the telco's proposal is that packets of certain types (VOIP and video, for example) that paid an additional toll, would get to you faster than those that didn't pay.

So it's not just the user who has to pay for the speed of their service, it's that the other side, the content maker, would also have to pay for you to get fast packets on a fast road. Disney and Viacom will pay their side of the tolls, but can PBS? Can little joe video blogger pay? Or will he get the same deal as the

What that means is is that the Hollywood and bit content producers would have the edge over the average person who wants to get a message out. So if you have a fast connection but joe blogger didn't pay, well, sorry, those packets won't get to you quickly. Instead, even though the user paid for faster service, they would not get all packets at the same speed. The content maker who didn't pay would have their packets come through slowly. And of course, the slow speed service buyer, who asked for a video from the content maker who didn't pay the toll would never see that video, it would be so slow.

AO Hollywood

I'm at Always On Hollywood, speaking on Thursday at 3:45pm. We're hearing Peter Hirshberg (one of Dabble's advisors and a great video story teller) talk about online video and things people are doing when they create and play online. People are eating this up -- it's a great time. I'll find the videos and bookmark them shortly.

Blog Spam from Netscape, and Netscape's Inability to Deal With It

I keep getting blogspam notifications (more than 500) after a Netscape blogger keeps trying to post what looks like automated blog comment spam with a link payload to my blog. They are here: http://mywebpage.netscape.com/ followed by pizdetcc/refinance.html (if you go to this link, it redirects to a search site for refinance and mortgages, but i don't want to publish the link, even with a nofollow).

I emailed Netscape at their policy2004@netscape.net privacy policy address. They have no abuse address, and their Terms Of Service doesn't say anything about how blog spam creation is against the TOS. So, my only option was to email the only address about any policy on their site, to let them know they are hosting spammers and not only do they not know, but it's not against the rules at Netscape.

Well... they wrote back. See below for the full correspondence, but they responded that I should contact MY HOSTER for MY BLOG. Wo.

Netscape is hosting blog spammers and this is their answer? Talk about not getting it.

Below is the original email, and their reply:

From: policy2004@netscape.net
Subject: Re: spam from your user at http://mywebpage.netscape.com/pizdetcc/
Date: April 21, 2006 6:57:11 AM PDT
To: mary@hodder.org

This mailbox is only able to address inquiries related to Netscape Network privacy. For assistance with your blog, please contact the hosting company directly.

Regards,
Netscape Privacy Team

-----Original Message-----
From: mary hodder
To: policy2004@netscape.net
Sent: Fri, 14 Apr 2006 09:14:53 -0700
Subject: spam from your user at http://mywebpage.netscape.com/pizdetcc/

TO: Mywebpage hosting

I have gotten hundreds of blog spam in comments and trackbacks
from one of your users.

NOTE: I read your TOS and there is no where in there to report abuse, or to tell users
that "blog spam" is against the TOS. This needs to be changed so that blog spam is made illegal by your TOS.

Below is a notification I received from my Blog's software (my blog is called Napsterization) where your user is spamming me. I have received hundreds of these attempts to leave comment spam, where the payload is a link to that uses commercial site.

Please block this user.

Thanks
mary hodder

....................
An unapproved comment has been posted on your blog Napsterization, for entry #291 (Blog Comment Spam - A New Low and So Bizarre). You need to approve this comment before it will appear on your site.

Approve this comment: =17465&blog_id=1>

IP Address: 196.40.43.74
Name: misty
Email Address: foloolk3@potran.gu
URL: http://mywebpage.netscape.com/ pizdetcc/refinance.html
Comments:

I like your website alot...its lots of fun... you have to help me out with mine...

--Powered by Movable Type
Version 3.2
http://www.sixapart.com/movabletype/

Tonight: "Email -- Should the Sender Pay?": EFF Fundraiser, Debate Between Esther Dyson and Danny O'Brien

Details below about this event at EFF.

At eTech, Esther and Annalee Newitz were talking about Goodmail, innovation in spam control for email and the controversy with EFF and others around this topic. I asked them both about stats. What I wanted to know was how much (number and percentages) email is spam, how much is non-profit email, how much is educational, and how much is political speech?

My feeling was that with those kinds of stats, and an agreement that we would let the IRS decide who should get free email if we instituted a pay for send system, we could give this a try. The issue with the IRS is this: they give tax exempt status to entities who are non-profits, some political organizations and others, and if an organization has that piece of paper from the IRS, we should exempt them from fees. The additional step for political organizations might be that we also use state and federal Fair Political Practice Commissions that also have organizations categorized. But with these kinds of certifications and exemptions from fees, we could try, innovate, experiment with different email systems that might help us solve some of the spam issues we currently have online.

One thing, when I was having this discussion with Esther and Annalee, I realized that I don't really get spam. This, even though my email address is on the front of my blog. I'm sure the spam is coming in like crazy, but because the ISP that hosts my hoster is clearing away some, and then my hoster clears more at the server level, after which the remaining batch has to go through the specific email system I have set up with my settings and training about what is spam on his servers and then I have more clearing going on at the email client level on my computer, I see about one spam email every week or so. It's rare, especially considering I get 1000 email a day. So I hadn't thought for a while about what a problem this is at the email level. In fact, I see far more spam blog, or splog, spam, via comments, trackbacks and in posts and through live web search, than I do in email. So my sense of the problem was really underwhelming for email and overwhelming for live web stuff.

Anyway, come to the debate tonight, to hear the arguments for and against!

Roxie Cinema
3117 16th St (Yahoo! Maps, Google Maps)
San Francisco, California

* "Email -- Should the Sender Pay?": EFF Fundraiser, Debate
Between Esther Dyson and Danny O'Brien

In light of AOL's adopting a "certified" email system, EFF
is hosting a debate on the future of email. With
distinguished entrepreneur Mitch Kapor moderating, EFF
Activist Coordinator Danny O'Brien and renowned tech expert
Esther Dyson will discuss the potential consequences if
people have to pay to send email. Would the Internet
deteriorate as a platform for free speech? Would spam or
phishing decline?

WHEN:
Thursday, April 20th, 2006
7:00 p.m. to 8:30 p.m.

WHAT:

"Email - Should the Sender Pay?"

WHO:

Danny O'Brien

Danny O'Brien is the Activist Coordinator for the EFF. His job is to help our membership in making their voice heard: in government and regulatory circles, in the marketplace, and with the wider public. Danny has documented and fought for digital rights in the UK for over a decade, where he also assisted in building tools of open democracy like Fax Your MP. He co-edits the award-winning NTK newsletter, has written and presented science and travel shows for the BBC, and has performed a solo show about the Net in the London's West End.

Esther Dyson

Esther Dyson is editor at large at CNET Networks, where she is responsible for its monthly newsletter, Release 1.0, and its PC Forum, the high-tech market's leading annual executive conference. As editor at large, she also contributes insight and content to CNET Networks' other properties. She sold her business, EDventure Holdings, to CNET Networks in early 2004. Previously, she had co-owned
EDventure and written/edited Release 1.0 since 1983. Recently, Esther authored a New York Times editorial called "You've Got Goodmail," defending a sender-pays model for the future of email.

Mitch Kapor

Mitchell Kapor is the President and Chair of the Open Source Applications Foundation, a non-profit organization he founded in 2001 to promote the development and acceptance of high-quality application software developed and distributed using open source methods and licenses. He is widely known
as the founder of Lotus Development Corporation and the designer of Lotus 1-2-3, the "killer application" which made the personal computer ubiquitous in the business world in the 1980's. In 1990, Kapor co-founded EFF.

WHERE:
Roxie Film Center
3117 16th Street, San Francisco
(between Valencia and Guerrero)
Tel: (415) 863-1087

See the link below for a map:
http://www.roxie.com/directions.cfm

Local Muni are the 22 and 53 (both at 16th & Valencia), 33
(18th & Valencia), 14 (16th & Mission), 49 (16th & Mission).
BART stops one block east at 16th & Mission.

Public Parking is available on Hoff Street, off of 16th
between Valencia and Mission at very reasonable rates.

This fundraiser is open to the general public. The suggested
donation is $20.
No one will be turned away for lack of funds.

Please RSVP to events@eff.org

Adaptive Path is the generous sponsor of this fundraising event. Founded in 2001, Adaptive Path is a leading user experience consulting, research, and training firm that has provided services to a range of clients, including Fortune 100 corporations, pure-Web startups, and established nonprofit organizations. The company is headquartered in San Francisco. To learn more about Adaptive Path, visit the company website at:

To learn more about the DearAOL campaign against AOL's planned system:

For Esther Dyson's editorial, "You've Got Goodmail".

The Conversational Middle: Maturing of the Blogosphere

On Saturday at Kinnernet, the last session I attended before leaving was led by an Israeli guy named Uri Baruchin who asserted that something had changed in the blogosphere, and we were starting to have a problem because a meme (my word, not his, and it's what I called it as I disagreed in the session) would not spread so fast in the blogosphere now that A-list bloggers were waning in link counts (a popularity scale because it uses a single digital social gesture, the link, and does not weigh at all the many other conversational gestures of a blog over time -- that would require multiple digital social gestures and a much more complex "algorithm" than just counting links). He was worried that the number of smaller discussions required to spread news would make the blogosphere as a whole less effective in broadcasting news, and somehow this meant there was some loss of power bloggers had been holding and was now waning.

I disagreed with his thesis, and gave some obvious statistics but also some ideas. First, I said that the blogosphere's purpose was not singular. This goes for both individual bloggers and on the whole, if there can be a "unified purpose," which I don't think there is because there are too many different kinds of blogging. Remember blogs are tools, and each person takes it and uses it in whatever way makes sense, which probably means there are 33+ million slightly different to extremely different variations of blogs now.

Anyway, back to disagreeing with the "unified purpose" idea. So, if you look at this from one view -- through the State of the Blogosphere reports put forth a in October 2004, where counts for the NY Times and MSNBC were in the neighborhood of 17 or 18 thousand links, and top 100 bloggers like Boing Boing and Instapundit had 6 or so thousand links.

At that point in time, Technorati tracked 4 million blogs and 400 million links, and now, a year and a half later, Technroati tracks 33+ million blogs and 2.2 billion links. In January, 2006, the NY Times has 55,000+ links and CNN has around 53,000+ but Boing Boing has 18,000+ and Instapundit has 5,600 links (the are no longer the number 2 blog, as that position is now held by Engadget at 15,600+ links) you can see that while the blogosphere has grown 7x and the links 5.5x, the inbound link counts of the top blogs and media has grown 3x.

Also, take into account that Technorati changed its methods of link counting last August, after several things occurred (Robert Scoble complained about its link counts in comparison to Bloglines which counts every link since it started counting, and I had reported on how link counting worked across 5 services and then other's reports of frustration with the Top 100 and A-list counts where sparce posters' links were favored over frequent posters' links). So Technorati changed from counting just links on the top pages of a blog (those posts that linked but dropped off the front pages were dropped from link counts) to any link that had occurred in the past 6 months. Technorati still counts one link only per blog, no matter it's location on the blog posts or blogroll, no matter how many links come from one blog, but all link types age out after 6 months. So these statistics for the later time frame are different and not exactly comparable, but let's do it for the sake of argument here.

So, what does this mean? Well, since there are 5.5 times more links in this 1.5 year time frame, I believe it means that there are more links made to non-A-list bloggers than bloggers further down the power law curve about , that are in what I call the "conversation middle of the power law curve" (the curve for specifically link counts), than those A-listers at the top are receiving. It means to me that while a year and a half ago, when I explained the conversational middle to people like Peter Hirshberg and Francis Piscani and thought it was far more interesting than what most people were discussing then (the top of the power law, or the existence of the power law curve at all), that now there is some evidence that as the blogosphere goes main stream, it is moving more to the middle, at least as link counts go, to more personal conversations rather than pointers to a few top media sites or the blogs that are act more like broadcasters. The broadcast model for links in blogs means that many more links went to a top blog, than they were able to link back to, because it was just physically impossible. Those top bloggers are 1-to-thousands in their distribution, and yet for inbound links counts, they have thousands of inbound links as opposed to far fewer outbound links to others.

From February 2003, here is one distribution curve showing bloggers inbound link counts in the Shirky article on power law curves:

But the conversational middle, then, and now, is both say, 12-to-12 for distribution and receiving links, or 50-to-50, or for larger blogs, maybe 500-to-200, depending on the size of the conversation verses those listening and linking back. And now the mainstreaming of the blogosphere supports this hypothesis more.

As modified from Chris Anderson's Long Tail article here is a representation of what I'm postulating:

The top of the power law curve has been referred to as the A-list, and certainly last summer, when Blogher discussions erupted at the last and well packed session of the day, and so many women expressed extreme anger and frustration at Technorati's link counting methods and particularly the there was a lot of interest in figuring out ways to reveal topic communities lower down that power law curve.

At Blogher, I suggested we work on something that would show not just a few bloggers in topic areas that were at the top, but rather sift the entire blogosphere, using as many as 22 different metrics, though some are not currently available but tool builders were welcome to build some of these out, to show "conversationalness" and "influence over time" instead of the "popularity" of link counts as the Top 100 or Top 500 that was subsequently built by Feedster, currently shows. Lots of people responded with lots of interesting ideas on the subject of how to approach this. A system like this could more accurately reveal the conversational middle, to make it much easier for more than just the participants in smaller communities of say 50, to find and expose their interests and conversations. This would also, for some of the women at Blogher, make them feel more validated or exposed as leaders in their topic areas (not politics or tech stuff), or if that was not their interest, make the Top 100 less validating and congratulatory of those who by virtue of being on the list seemed dominant, which clearly was their desire.

The idea that bloggers are not passing memes as effectively because there is less influence by broadcast style bloggers, even though more small conversations are experiencing more links going blogger-to-blogger, further down the power law curve, is silly. If people are genuinely interested in something, and have something to say, they'll blog it. These conversations occurring in the middle pass and discuss memes just as before, but the linking and diffusion of the conversation is evidence of a more mature, and interesting use of the internet, not less so. And now, if a meme crosses lots of blogs in the blogosphere, I believe it's a sign of far more interest by people, than under the earlier broadcast mode of meme spreading in the older, more early adopter blogosphere. And this certainly isn't a problem. The internet as a medium is more supportive of spread, edge conversation, than the amassing of top down broadcast distribution. The act of blogging is the act of subverting old broadcast methods of communication.

The maturing of the blogosphere with less broadcast distribution and more conversation between people spread far and wide is a welcome and more democratic way of bringing together people who want to discuss like interests. Certainly there are still bloggers who are more highly read with more inbound links that resemble broadcasters in some ways, and who PR people will continue to try to manipulate, but still, there is a shift to conversation with more symmetric linking, and that is positive overall.

Our next challenge now is how to see how small conversational communities and the attendant tools that sift these conversations can use more than one or two digital gestures, and create topic awareness of blogger groups with more than just the early adopter or blog-tool favoring metrics that post categories or tag indicators do now. These metrics too are subject to power law curves and the current uses of them one or two at at time only reflect top bloggers, or early adopters, just as link counts did before, emphasizing them over the conversational middle. When the tools of exposure change, the conversational middle will become accessible and apparent not just to those in and around a particular conversation but to those outside it.

Upgrading to Web 2.0

Yes.. folks, it's time for your upgrade for the internet.

So.. I met these very sweet folks from Dalla, Texas at SXSW at a party late Saturday night, and I asked what they did. They said, we're web designers, and right now we're working on upgrading all our clients from Web 1.0 to Web 2.0.

So I asked, what does that mean? And they said, well.. they have all these clients who haven't changed their websites in years and years, and now, with this concept of an upgrade, are open to improving and spending the money.

Well.. that just changed everything for me.

I thought Web 2.0 was some amorphous, meaningless, ridiculous term that no one could possibly take seriously except those VCs who write checks for fancy executive conferences. And a term that when used seriously, would tip you off to the fact that they didn't know it meant nothing and was silly.

But shoot. Now I get it. This term means something to IT consultants across the land, as they work with their clients to take them from the static web to the live web (my terminology, not theirs.. I don't think any of them will ever use those terms).

But it makes so much sense, and now I don't hate the term. I feel like well, if this is helping little mom and pop shops get a few people into better, more usable websites (we hope... they kept mentioning ajax over and over, plus blogs and wikis, and my highest hope for them is that they do it well, making things more usable for their client's users) then who can hate that? How can we begrudge them this terrific opportunity to explain the new social web to their clients, simply by putting it in terms of a software upgrade they can understand. I mean.. they all went from IE 5 to IE 6, yes? Well.. now it's Web 1.0 to Web 2.0.

So I now have complete respect for "web 2.0" in this context. Live long and prosper.

And now there is a certifier. How handy. (Note that 'humor' is one of the things that will get you certified by the Certifyr.) Too bad I didn't get their cards to send it along.

Attention or Eyeballs.. Attention or Intention.. Attention or Identity

"The eyes are the window to the soul." - Unknown
"If the eye is a window to the soul, then, the heart is the doorway to love." - Unknown.
"The world only exists in your eyes. You can make it as big or as small as you want." - F. Scott Fitzgerald

What's the difference between the static web and the live web?
Participation.

What's the difference between consumers and users/amateurs?
Participation.

What's the difference between attention and eyeballs?
Participation.

What is attention? Lot's of people have discussed it, including Nick Bradbury, Steve Gillmor and Seth Goldstein, all of the Attention Trust. I'm on the board too, but my interest in joining it was a little different, though I believe in the core idea just as they do. To quote Seth's blog: "Attention is the substance of focus." The idea for the Attention Trust is that "users own a copy of their data" or attention stream or intention stream or whatever you want to call it. I'm going to leave the intention debate to others because while I agree with John Battelle, that these kinds of recordings can form a sort of 'database of intention' it's not my interest in this post to pick that apart.

Caterina Fake also blogged about this idea of users owning their data.

Etech's theme this past week was attention, though I don't think anyone there except maybe Michael Goldhaber really got anywhere near the idea that the difference between the eyeballs of old (10 years ago) and the attention question is really about participation, at least as far as users collecting it on themselves and reusing it, or sharing it as they desire. Not to mention the digital social gestures that people can now make, and collect, through participatory media online that go much further than the simple mouse over or clicks that were all that could be collected before. Now the aggregate of both, clicks and gestures that are much more participatory in nature are richer and much more meaningful, and quite a bit different than "eyeballs".

And what is participation? As far as the AT, it's about user control and choice, and an absolute right to participate. Or not.

Surprisingly enough, since last August, when the AT was formed and announced, it's been just so easily accepted by anyone asked, from the top to the bottom of those "database of intention" makers, that you should own a copy of your data. They own one copy of course, but we really thought it would be much harder to gain acceptance of this ideal. And yet, here we are. Pretty much everyone has said, "...er, yes, users own a copy of their data." The hard part is, how, how much, when, in what way, will all these companies share a user's data with the users.

So the reason I joined the AT board was because I feel strongly that users should own a copy of their data. But I also feel strongly that users should be able to keep that data private, have complete control over their copy, and shared control over other copies depending on circumstances, and those users have the absolute, unequivocal right not to participate in the attention economy, at least as far as sharing their own data goes, if they are asked to by some vendor or company or other entity. No question.

If Visa wakes up one day and decides to tell me I must give them my attention stream or kiss my credit card good bye, well.. the AT would need to step into the middle of that one pronto. I cannot abide by that sort of coercion, and so, my real interest in the AT is making sure that it's as much an advocacy organization for user's privacy and security from coercion, as it is for making a place for people to come to learn about how to own and user their own data and possibly interact with entities that might trade them for it, or share the rewards of turning over leads for marketing.

Omidyar, the foundation established by Pierre Omidyar to fund both for profit and non-profit ideas, has given the Attention Trust its support to explore this idea of having a non-profit, independent group supporting user's rights.

I'm also going to work with EFF (and hopefully EPIC and Markel) to make sure the AT work and the recorder tools are the most user-friendly and affirmative of user-control, privacy and security as possible. I would also appreciate any help from people in covering this as well, so if you have thoughts, please send them to me in email to mary at hodder dot org .... or comment below.

Tonight there is a talk on attention, at SD Forum if you want to come check out some attention ideas. I encourage you to attend if you are interested and in the area.

Google, Fair Use and Open Media: Fair Use for All, or Just Google?

Updated: I've corrected the information below that said Google was signing exclusive contracts with libraries. Apparently after I left the book search panel at SXSW, this notion was corrected in the room (I had to leave about 10 minutes before it ended). Google is *not* signing exclusive contracts with libraries for the scanning of books.

I just watched Larry Lessig's talk about Google and Book Search and Fair use, which he's put into video online.

There's been a ton of bruhaha over Google and its scanning of books. Authors' guilds are upset because they don't consider Google's book search to be fair use, even if 9% of books are in print, 16% are in the public domain, and the remaining 75% are out of print, but still under copyright protection. Lessig talked about this at the NY Public Library and Susan Crawford discussed it just after with the Author's Guild. The Author's Guild does not see this as fair use at all, though Lessig defends it on his blog:

In December, novelist Susan Cheever, a member of the Authors Guild, published "Just Google 'thou shalt not steal,'" an article suggesting that there's some kind of official word limit, or percentage limit, to material you can copy in order for it to qualify as fair use. She writes:

"The Copyright Statute includes a 'fair use' clause' so that a few lines or phrases of a writer's work can be used as illustration by someone else. ...The amount of words that constitute fair use varies according to court case. At present, it is 400 words. Google cites 'fair use,' but it isn't using 400 words; it plans to digitize whole libraries and make them available piece by piece." (Emphasis added.)

Even this small quotation from Cheever's article fundamentally misstates copyright law and misleads readers about Google Book Search.

First, no such 400-word rule exists. Indeed, in some cases courts have ruled that copying and republishing the entire work is fair use. (You can read about one such court decision here.)

Second, Google does not show more than two or three sentences without the author's permission. And that's not all. If a copyright holder chooses not to participate in Google Book Search, not a single word from the book will appear in any searches.

Lessig also discusses it here in this video. Obviously, there is a lot of incorrect information and it's good to see lots of discussion in many places to help educate people on what fair use is, what technologies are being used and what copyright is about.

And I agree with Lessig, that showing a few sentences of books that have been scanned and made searchable, is good and helpful to society and our educational structure. But then he refers to the "googlization" of these books. Because there is an issue here where the fair use is not being passed along.

Google says that for books in the public domain (16%), it will grant full access. For books out of print but still in copyright (75%) it will grant snippet access, and the 9% in print will be given access based on what the publishers and authors allow and if no access is granted, then none is given. Fantastic!

But what about the rest of us? Do we have fair use rights over Google's fair use of public domain and out-of-print books? Or will they limit us technically from doing to them what they are doing to books in an analog way? They are copying those books, and asking us to back up this use of those copies. In other words, they have created a digital barrier to keep others away from fair use access to their fair use. Googlization indeed.

I think this is really a big problem. Google is saying that they want to prevail in the lawsuit, because even though they can afford to pay the author's guild, hardly anyone else can, and therefore, they must win to preserve everyone else's right to innovate. Well, that's true and a good argument. Except that they are one of the very few who can also afford to digitize books. And by putting books out digitally for search, they create a situation where everyone has to go to Google (or one or two others, inlcuding the Archive) for book search, because Google has paid for the scanning. For books that are not public domain, I can see limiting access, but for public domain books, there is a real problem with Google's communitarian sensibility.

I think if Google really cared about innovation and open markets, they would actually put the information out to be copied by others, and they would not ask our support unless they planned to play fair.

In the SXSW panel on Book Digitization and the Revenge of the Librarians, I brought up this issue, with the Google guy, Daniel Clancy , by asking about Google's activities around these issues. He and the other panelists engaged with the issue quite a bit, but I had to leave earlier than I wanted to so I missed the end of it. However, Daniel did say that Google is working on better search and community integration for the book scanning program and that in a few months, there would APIs or other tools to get closer to the material but they have to balance that with their costs of scanning. I'm skeptical of this. I think we really have to wait to see what they do, and of course the longer they make us wait, the less trusting I am of the promise to open the doors.

I loved Liz Lawley's suggestion that we all might take a book and scan it, instead of leaving the enormity of the scanning project to big companies and efforts like the Open Content Alliance. While I realize spending an hour scanning your favorite book may not be the most fun thing you can think of, making book search distributed is a very interesting idea. And could help create distributed multi-copy book search across the web. That's pretty cool.

Update: Note also Kevin Mark's rewriting of Cheever's words here.

Dabble: It's Balpha!

Om Malik mentioned us late last night, and was very complementary. Thanks Om!

Om mentioned that we are video bookmarking site, but we have some other features as well, for people to search, browse and ask for video and make playlists.

We have lots of folks to thank, as we head towards a closed balpha launch next week (we'll invite folks in order from the email submissions.. because the engineers want to see how things go with the service, and test a few things, before we open up to the public). We'll be demoing Dabble on Wednesday night at an IDG thing. The engineers have been working very hard, and deserve kudos for their great work making the site.

Big thanks to investors like Hank Barry, Evan Williams, Steve Schlenker and Mark Pincus. Their support goes way beyond the financial. They have been invaluable to me with their advice, support, introductions and help. The Dabble Team could not have done this without them.

And our advisors too have been really great: Kent Bye, John Borthwick, Peter Hirshberg, Ross Mayfield, Glenn Reid, Sally Rosenthal, Doc Searls, and Jeff Ubois.

Thanks to all, and if you are interested in an invite, add your email to Dabble, and if you have questions, email me at mary at dabble.com.

Notes from James Surowiecki's Talk at Intelligent Television

Intelligent Television conference info here.

1. Openness bridges all of these mechanisms: open source, p2p, shared work.
2. Intelligence is distributed rather than centralized: the knowledge is spread out in many locations
3. Bottom up works better than command and control mechanisms
-- people are better at understanding their own needs than the top
4. We are better off casting wide rather than narrow
-- don't know where the info is much of the time
5. Open access to creativity, knowledge -- benefits are greater the more people are involved
-- when people learn more, we learn more.. it's anti-rivalrous
6. Be very hesitant to filter who belongs to community
-- don't keep people out
7. People act better the more info they have

8. The internet allows us to become technically able to do so much more
-- distributed info and aggregation are so much more powerful
-- possibilities are immense

9. Different ways to tap into open systems
--obviously people using open systems to make money
-- Innocentive.. people go to register as a 'solver' where 10. You then get access to a problem set
----- if you solve a problem, you get a prize, but he company owns your solution

11. Systems that allow people to give ideas and innovation a piece at a time are interesting, because lots of people contribute. Prediction markets and prices work this way.

12. Can profit from an open content system.. leave everything open and free and then make money from talking about this stuff..

13. People find pleasure from the value of competition
-- from contributing to the growth of the pool of knowledge

14. Many of these systems are inefficient, because in a strict sense, they are redundant..
but the point is that even though this is the case, if we expand our ideas of efficiency, it's tremendously efficient.

15. What are the challenges to these systems?

Internal
-- problem with model in that a network or self organized model, it's difficult for individuals to contribute due to echo chamber effects...army ants .. work in ways where they do just what the ant is doing ahead of them.. if they start walking in a circle.. they actually die.. worry that if humans imitate others.. we will degrade because nothing new happens.. group loses collective intelligence.. drawing knowledge from just a few
-- challenge is to keep the ties in the networks loose.. and open and flowing

External
-- profound counter to our most deep seated ideas around authority, knowledge and expertise -- people have a fundamental desire to pick "the expert"
-- traditional need to develop a product, and then show it after it's out.. instead of working with people all along..
-- traditional needs to develop IP are challenged

16. Arthur Miller in the Harvard Law Review just wrote an article saying that what we need now is 'common law' for ideas.

17. Tom Bergeron -- host of dancing with the stars on why people like this.. because it is about
"wholeheartedly uniting our skills is the basis for all human interaction"

18. Collective systems may work better when there is an answer people think they can find, verses when a lead user or expert may be better at finding the right thing.

19.. Our imagining of the 'genius' is the failure to see that works of art are actually based on others ideas ... works of art always borrow from other works of art.

Microformats for Media, Part II

Figuring out the Microformats process has come down to this: once you know they will accept having a particular format, you then have to provide bottom up examples of use. After that, you brainstorm from the examples, and after that, some kind of schema is worked up. I'm not a schema writer, but I do read them. There are others working on this project, like Josh Kinberg and Lisa Rein who can contribute to the schema better than me, as well as all the folks who expressed interest over the past 7 months. Though as noted before, the old format page went off in the direction of figuring out what top down metadata schemas existed already. What I've done today is an attempt to look at this bottom up, as the Microformats people want.

In keeping with Tantek Çelik's request to have microformats examples here, at Media-Info-Examples, I've posted examples of media publishing by users. Each example is followed by what elements the user put in, and then an estimate of the numbers of uses that are similar. These numbers are based on what I'm seeing as we work on Dabble, but others may have higher amounts. We are seeing photos, audio and video, all with titles, creator, license, and associated URLs. But very often, we see tags, quotes (into the middle of video or audio or region annotation of photos), duration if it's audio or video, and descriptions or summaries.

Also, I've added to the Media Brainstorming page to note these elements that come up often in users publishing of media, and also note the items that rarely come up but that might want to be included at the bottom of a format somewhere, if desired.

Mash Up Camp Progress

So, Mashup Camp is moving along. To recap...

Dec 15, David Berlind said the idea in a meeting we were in, and later Doc Searls and I agreed to help.

Dec 22 David wrote about it on his blog, announcing it. Lot's of folks blogged about it and very very interested.

Jan 9 The domain was purchased and website set up with lots of help from Ross Mayfield.

Jan 13 After much discussion and coordination with very helpful people like Ross Mayfield and Lauren Gellman, of Stanford's Center for Internet and Society, I managed to get the Computer History Museum (thanks to Peter Hirshberg too!). To date, without a venue or time, we had 245 people out of 250 spots signed up.

On Jan 16, we "sold out" and started a wait list.

I think this is the fastest organized event I've seen, with this breadth of people and concerns. There is a fantastic group of people, including all the mashup developers and folks from big API provider companies like Apple, Google, Yahoo, Microsoft, Intel, Amazon, etc.

Now David and Doug are working on getting lunches and the evening event paid for, the site details and costs worked out (big thanks to the Museum for donating the site at a non profit rate, since this is a community event and costs $0 to attend -- consider joining the Museum here). Others are helping to get some hotels lined up for out of towners, and people are thinking about sessions they might lead and dinners.

Microformats and Media

Last night I attended a sort of meet up for people Tara Hunt had invited me to, to talk about microformats and media. She had wanted to start with photos, I think because of Riya, but it became clear after talking a bit that similar elements apply to rich media whether the piece being discussed was a photo or a video or an audio piece. The group started out mostly on computers trying to do a group chat, but I didn't have a computer, so I tried typing notes on Josh Kinberg's computer, but the software wasn't recording everyone's comments and it wasn't all that constructive.

I pulled out my notebook (I hadn't brought my laptop) and started writing a short list of elements that are common across all media types, in terms of what elements users publish over and over either on services like Flickr (and other photo sites) or Blip.tv (or other video sites) or audio sites like iTunes. At this point, everyone put away their laptops (funny how the paper can trump the computer once in a while, and while I don't really do paper, except for my notebook, it works for me at times like this). We centered around the notebook and the common document we were discussing, which consisted of a growing list of my notes:

If you want to know who attended, there are photos on Flickr. But the interesting part for me was realizing what we could make with this microformat, for users to publish with, for the publishing tools like Structured Blogging, which takes microformats and makes them into something bloggers can publish through plugins or through other tools that will be built later.

Microformats, as Tantek explained, need to have a page on the MF wiki that shows use cases that cover 80% of what users do now (as a rule of thumb) though arguments can be made for less, if they are really useful (like tags which are much lower across all users). On the Microformats list, the way Tantek and Ryan run it, it's been hard to tell what they meant by examples. When they would make these requests for examples, and I would then look at what people post for the examples, it didn't make any sense to me. But after talking, I think I understand what they want.

It's like the difference between taxonomy and folksonomy. Microformats come out of bottom up user generated use cases. Where as media metadata formats like SMIL and MPEG come out of top down committees. Not that they are bad, we are using those top down formats too in my other work. But as with taxonomy and folksonomy, so with microformats and top down metadata. They both have value and they each come from very different use cases and points of view.

We agreed that the Media metadata page had examples, and yet, it was overgrown, needed pruning, focused on metadata from the top down, instead of examples of what users do now. So last night Tantek explained what they meant by examples specifically. For example, we need to literally cut and paste a blog post from a user that can be used as an 80% use case, to show something as an example. Fair enough. So now, we need to add these examples in a constructive way, in order to argue the media format elements and microformat need for media publishing. We can think about a short list of elements that users use most of the time, when putting some media online, whether it's a photo at a service, or on their own blogs, or a video or audio piece.

Those elements (from my notes last night) are in the first list, becuase they reflect what I see online, though I will go find stats and use cases to back these up, or argue that the 20% useage of something enriches the whole community and so how far that argument goes -- tags are an example of that.

Base elements:
* Title
* Html URL
* Media URL
* Tags
* Description or quotes (subsets of the object: a video quote and tags/description associated with it, a region annotation note for a photo, or the quote of a podcast and tags/description -- the detail for these subsets exists in the 'more info' section below)
* Creator
* License (defaults to copyright, if none exists, but it's there, by US law, and many other areas of the world)

and for audio and visual:
* Duration

Other info:
(This is not the same for all types of media, and is published by users in very limited ways in practice, or is captured from the device or service or in some way, invisible to the user, and therefore often depends on a service to pick it up.)

The second piece is figuring out the elements and schema that lie around those 80% use cases.

I don't think this is so hard now, despite how chaotic and crazy media metadata can be, where some of that is reflected on the media metadata page. Though that page is a very good attempt to organize the chaos. But I now have a picture of how to make this happen in my mind, that is simple, and gets us to a place where we reflect what users do in practice, bottom up. So, based on my notes last night, I'm going to try to fulfill Tantek's requirements, and see how far I get with it. Will update here with pages as they happen.

Mash - Mash

David Berlind and I have been chatting for a couple of weeks about his terrific idea for Mash Up Camp. The idea is similar to Tag Camp and Bar Camp, where anyone, developers to start up founders to big company enthusiasts, will convene (probably about 200 people) to share Mash up ideas and projects, and develop some new and interesting ones.

It's sprouted from conversations about the anti-conference, and how much more fun and meaningful the camp thing can be.

Date is not yet set. But stay tuned. Should be announced soon with a date in the near future.

Meanwhile, Chris Messina just announced another Mash up camp, for about 12 people (maybe it's a Mash up Meetup as it's so small), which sounds really fun, for January 17. Sign up here. Looks like the goal is to develop a couple of mashups over the course of the day. Maybe Chris et al can share those at the big Camp a month or so later.

I can't go to the Mash up Mini, because I'll most likely be busy working, but if you have an interest, sign up now. There are 9 spots left as of this morning.

Also on the evening of the 17th, the Stanford Venture Laboratory is having a thing on User Generated Content and the Future of Media.

It's been so fantastic having this week and last be tech-event-free. We've been working in the office late late late every night, and yet there is still time to workout and clean out closets last weekend. I love this two week period because I feel like I can get so much done, plus get really healthy at the gym. I actually feel like I'm far ahead of where I was two weeks ago.

But it looks like that's going to change a bit come January.

We should hold a two week quite period on events every quarter just so we can all catch up on stuff.

Doc Searls on Corpuscles and Hearts, Among Other Things

Doc, as interviewed by Irina Slutsky at GETV.

"The Granddaddy of us all...." It's funny. Check it out. It was done right after his talk at Syndicate.

My Favorite Punster: Burpa.licio.us

Doc: Burp! Yahoo eats del.icio.us.

Developer Workshop For Those Interested in Using Identity in Their Services and Tools

The Internet Identity Workshop presents an
Informational Morning for Developers
Hosted by Doc Searls, Mary Hodder and Kaliya Hamlin.

Monday, December 12, 2005 9-12 noon, with lunch from 12-1
Canton Dim Sum @ 655 Folsom St in San Francisco.
Cost $20 for lunch (PLEASE RSVP HERE as the Canton Restaurant has been kind enough to give us the space if we all have lunch there, but we need an accurate count by Sunday at noon).

If you are a developer working on a application that has folks login - this is a morning for you.

Doc Searls will begin the day giving an overview of the identity landscape. He and others will answer the question:

* Why do identity systems matter when building new systems and tools?

We are bringing together a spectrum of folks who have been working on developing identity systems and tools. Identity Developers will share their work, basics and best practices to date to get started exploring integrating identity into these applications. These include YADIS, LID, Open ID, i-names/XRI, SXIP, among others.

Developers of applications who have included identity into their services and tools will share briefly how they've done it. Application developers will hear from and meet with identity developers to ask questions.

Event Info
Detailed Agenda
and
RSVP here.

Live Web Search

I'm giving a talk late this afternoon on live web search in Marti Hearst's Search Engines: Technology, Society, and Business class at SIMS. I'll be talking about how the live web alters static web search, how live web search works and what sorts of things we would like to have to make search better in the future.

Update: Here are the slides (or in pdf) and link to the webcast.

Tagging by Bloggers, a Small Study

Last February, I conducted a series of interviews with 20 bloggers. 10 were using what was then the 'new' Technorati Tag implementation and 10 were not. All were regular users of tags in other systems: del.icio.us or Flickr types of systems where a tag is applied by a user, and then the tags can be 'pivoted' on, or clicked on, to see all the other media that has the same tag. This pivot can occur either at the user level, to see everything that user has applied that tag to, or at a global level, to see everything where the entire community at that site has applied that tag. Additionally, these bloggers were beginning to use the Technorati tag pages to see how bloggers were tagging their posts or media, and how the Flickr or del.icio.us media with the same tags fit together on the Technorati page with those tagged blog posts.

I sent out email to these bloggers, essentially asking how they used tags in their posts, or how they wanted to use them, if they did not and yet wanted to, or if they didn't want to at all, why.

All of these bloggers were people I did not know before the interviews, but were semi-randomly chosen because they blogged with tags, or appeared to be aware of tags because they had links to sites with tags, though they did not tag their posts themselves. These interviews were done in email, and were conducted over a couple of days. I promised those interviewed that I would not pass on identifying information about them, or list them in any way. I also promised to aggregate the information before blogging or passing the survey on to others.

Of the ten who use the new rel="tag" format in their blog posts, most were dissatisfied to one degree or another. Sometimes, the Technorati system did not pick up their tags, even though their posts could be found in the search system in other pages besides tag pages, and they were using the correct format. Some also expressed similar desires to be able to do more with the tags and formats, as the non-tagging bloggers did. All expressed some desire to have more of something.

Those interviewed talked about the following that would help them tag or cause them to want to tag when they were not doing so now:
1. A desire to create tags in their blogging software in similar ways to how they create 'categories' -- meaning they wanted to use a pulldown menu or something with similar ease, to quickly tag a post. This included the desire to have tags be invisible on their blog pages, as some of them have invisible categories in their posts. Some of those with invisible categories at the post level still have category searches visible at the sidebar level of their blogs. They would be interested in showing tags at the sidebar location, if they choose. But all felt these choices of visibility and invisibility at various points in their blog posts and overall blogs should be left to them as it is now with categories, and those choices should not bar them from participating in Technorati's tag program.
2. These bloggers rarely added new categories to their blogs, and saw the value of having large buckets to categorize their posts, and didn't want to add new categories all the time. Partly this was due to how difficult the software make it to add categories, and partly this was due to seeing in practice that there was value to 'large bucket' categories, and 'little context' tags. These small tags were desirable because they could be applied to a post on just one time, but categories would come up at least every few days.
3. These bloggers all understood the meaning of a link in their posts. They knew the value of those links, and thought carefully about where they pointed in posts before doing so. They did not like being forced to put a link to something in their tags, if they were not so inclined. They would prefer to have the choice to make a link or not make a link, depending on the circumstances of the post.
4. If a link was placed in a tag, at their choosing, they wanted more flexibility to choose where the link went, beyond Technorati's tags pages, Flickr or del.icio.us. Many did not like that in order to make the tag, they had to place a link, and then because they wanted to make links that 'made sense' to their readers (the links would 'go somewhere'), they felt forced to repeatedly link to these same couple of sites. Some wanted to be able to easily make their 'own tag pages.' Some wanted to be able to link to other places besides tagging sites, that had some meaning to them. And some asked to be able to link somewhere, and tag the link, and have that be understood to their readers and the systems that would pick up those tags.
5. A little less than half of these bloggers asked to be able to tag a specific object in their blog posts. They regularly posted photos, either their own, or brought in some code from another site to repost the photo, with or without text around that photo. They wanted to be able to tag just the photo in their post, but tag the post at the bottom of the post, following Technorati's directions for tagging.
6. A couple of the respondents said they would like to be able to tag comments from readers of their blogs, and they might consider, if they have registered commenters, allowing those commenters to tag posts, objects or other comments as well.
7. All of them liked the idea that tagging would allow them to participate in a community, but they wanted to control that participation themselves, at the publishing level of a post.

Additionally, the ten who were not tagging were asked why. Answers ranged from,
"I can't figure out how to do it using Technorati's directions." (3)
"I don't want to until they change it so I don't have to link."
".. it's too hard."
"I want more choices and flexibility"
"I haven't had the time to figure out the system and it seems hard."
"I hate the link part. I don't want to do that."
"I want my tags to be invisible." (2)
"I don't want to give Technorati more links." (3)

My conclusion at the end of this was that we need more flexibility and better tools for tagging at the blog post level, including creating tags at several points: around an object, at the link level, at the post level, and possibly at the comment level. Additionally, users need easy ways to tag, and set their preferences around how their tags will appear, or be created. And they need flexibility for linking within the tags, as well as some different choices depending on what they were doing, about how the tags will function. It would be great if they had some easy way to make their own tag pages, as they now can do with categories that allow them to build pages of posts that all have a common category.

Note: afterwards, I did let them know that they could link to sites other than Technorati, if they were not aware of this, and I did help those who could not figure out how to do tagging. But it was clear to me that part of solving these issues is communicating to users better and making some simple tools so that users don't have to play around with html code, and wonder if they are getting it right.

Disclosure: I used to work at Technorati (well before this study) and be on their Advisory Board.

Kevin Mark's Brilliant Idea

Kevin suggests, in response to my earlier piece on the terminology of Users and Consumers, that we instead bring back the original meaning of 'amateurs.' Well, I love this:

We already have a word for people who create for the love of it, rather than being paid to, and it is 'amateurs'. As with many other pleasures, when we seek out opinions, we prefer those that flow from passion rather than from payment.

Now it may be argued that, given the decline in the teaching Latin and French, the loving root of 'amateur' is no longer perceived, so those who write pour l'amour ou pour le sport may see 'amateur' as a slight. In which case lets retranslate it to english and call it 'lovingly created media'.

Fantastic. Because it means we take back from the concept of 'professionals' the notion that 'good = professional." Instead we claim the aspects of our experience through creation that are so humanly, actively ours to own and enjoy, as unpaid creators. 'Amateur' has been derogatorily used to convey 'less than' status. Sometimes one or another works is less than, but it is not due to whether or not someone is paid for their work. A work should be judged lesser or greater because of its intrinsic qualities and value to those who apprehend it.

So, take the label 'amateur creator' as a point of pride. It means you create for love, and not for money.

Oh. Btw. Just in case you think there is confusion about the other term in my earlier post. 'Consumer' is still unacceptable in so many ways. Don't use it.

Full Disclosure

Over the last month, a lot of people asked about the fact that I'm an advisor to Sphere.

I have wanted to blog about this, for the sake of transparency, for this period, but out of consideration for others, had held off. But transparency is something I believe in and strongly believe needs to be done sooner rather than later.

During the 4 years I was at UC Berkeley, I spent all of it doing some sort of research and development on news and blog systems. In particular, I spent a couple of years working on the topic browsing of blog data, developing a front-end php-based system, 150 pages of usability, user and design research, preliminary algorithm design for determining topic communities through multiple metrics, scoping search up and down those communities, as well as weighting bloggers through other combinations of those same metrics (multiple metrics reduce the power law effects that a single metric can have on a community), as well as blogger profiling.

Additionally, before the topic browsing system, during my first couple of years at Berkeley, I also developed a news-blog system (interface, business plan, usability testing, and marketing research about online ad systems that could be developed to complement either the blog-news system or later, the topic browsing system).

When I was hired at Technorati, after graduating from Berkeley, I showed them the interface for the topic browsing and profiling systems. While they indicated they liked it, they also said the system was too hard to build, which I took to mean that it would take too many resources away from other commitments. We moved on to building other things there, and when I left Technorati later, I had a conversation with Dave Sifry that lasted hours. We discussed building products and companies, and at the end, I said I was still very committed to seeing my topic browsing research borne out. Dave told me that if I had any desire at all to build a company, I should absolutely go do it. I said I was thinking about, or I might find someone else to build it, but if any of that happened, I might want to license his data. He said we could talk about it then.

I was asked as I was leaving to be on Technorati's Advisory Board.

In the meantime, I realized I did not want to spend another four or five years continuing to work on topic browsing. Instead, I was committed to working on the video project I am developing now (I was a filmmaker for almost 7 years and love the idea of applying that knowledge to my technology development). But I was still committed to seeing my topic system research used, as it represents thousands of hours of work, and so I set about looking for someone else interested in building it.

I've talked with a couple of projects and companies over the past year, but each of them suggested that building topic browsing right was tough, and they had other commitments which most of their engineering staff were working toward. Sphere appeared this summer, wanting to build exactly what I'd done my research in at Berkeley. And so, I gave them all the work to use, and they made me an advisor.

Sphere has been working on blog search, though I said to them early on that I was just advising them on creating topic communities and so haven't yet looked their new search interface that debuted a month ago. In fact, they showed me early work and I told them I could not help with blog search because of my relationship with Technorati as an advisor.

I believe that blog search, at this point, is a baseline for any company in the space. You have to do it. But it's not so interesting to me, compared to making a leap forward toward something like topic browsing of communities or sophisticated weighting of bloggers. I'm less interested in 'yet another blog search' tool. The ones we had already were fine. But I've very interested in what Sphere is really here for: changing the ways we can view small topic communities and the bloggers within them in sophisticated ways that take us ahead of where we are now, which I equate to the place websearch was in in 1997, before Google.

In the interest of reducing confusion, however, I decided it made sense for me to resign from Technorati's Advisory Board. Many people have asked what happened, and I felt it was better to state it publicly, than not to alleviate confusion.

Technorati is still a new company and doing lots of interesting things. I wish them all the best. My decision to stick with advising Sphere is based on their heading down a path that's alongside my own.

So, now that this is out, I expect soon to post the rest of my blog search comparisons, as well as a couple of articles I've been working on around blogging and identity, rankism, etc.

Tag Camp

It's tonight. And at the last minute, even though I have something else this weekend, Anita, danah and Kaliya convinced me to go. So.. I'm going. Can't wait.

Been working on a tagging solution today involving Identity, Creative Common's licensing, and Tags for two different types of tagging in personal publishing (blogging). Drummond Reed and I have been working on this periodically for about 6 months, and finally figured out the security and privacy aspects at our meetings this past week, though we'd figured out the syntax and usefulness of it last April and June. Kaliya Hamlin has also been helping a bit. Looking forward to talking with folks about it to see if it makes sense. It's called iTags because it includes identity in the tags.

The idea is that a user could tag an object (photo, video, sound file, text or an entire blog post), where the tag, and the object, would then go out through the RSS feed or be spidered, with some additional information that doesn't now exist in tags. That tag and object would include the user's identity, the licensing for that object (presumably people would use this more for rich media objects than for just a blog post, as most blogs already have licensing generally for text on the blog) if needed, and the tag. It would remove the requirement for a tag to be coupled with the originating URL (blog post URL) because identity would be inside it. It would allow individual CC licensing which rich media producers want to do sometimes, in order to differentiate the rich media object from the rest of their blog, which may have different licensing.

This solution also addresses these user desires:
1. create tags without links that are still visible in the blog post
2. create tags that are not visible but still trusted (like categories which are not visible but still included in systems like Technorati's tag system, where of the 100% of tags, 95% are actually category entries and only 5% are actual tags).

It still allows users to create tags that do have explicit links if the user wants them, as is already being done. It's backwards compatible with Technorati, but allows additional functionality to meet these other goals.

Wiki for this stuff is here. Please feel free to add to the wiki or comment here about this. Would love to know what you think.

Symposium on Social Architecture

Update: I changed the name of the post to reflect the actual title of the event and Stowe's comments below.

Next month (in three weeks actually) the Symposium on Social Architecture will take place at Harvard. Stowe Boyd and David Weinberger have been working hard on this event, which is intended to be a fairly high level discussion about social spaces online. Terrific folks are coming, and the program includes Kevin Marks and me talking about Engines of Meaning: How Will We Scale Our Understanding:

"Ultimately no human brain, no planet full of human brains, can possibly catalog the dark, expanding ocean of data we spew. In a future of information auto-organized by folksonomy, we may not even have words for the kinds of sorting that will be going on; like mathematical proofs with 30,000 steps, they may be beyond comprehension. But they'll enable searches that are vast and eerily powerful. We won't be surfing with search engines any more. We'll be trawling with engines of meaning." – Bruce Sterling

How will we keep up with the "dark, expanding ocean of data we spew"? Algorithms? Social filters? Faster memex-like gadgets? Do we need open algorithms in future search, so that each person can tweak their own preferences? Will we become dependent on social networks to filter the world for us, and if so, are the current representations of social relations too coarse? Will we be spending more and more time creating explicit metadata, like tags, in order to help channel the "expanding ocean"? What does it mean to be smart, today?

Kevin and I are planning to work from an brief framing of the issues with a short outline for a led discussion. If you have any ideas, please comment below. Would love to hear your ideas.

Also, if you are interested in attending, there is a registration form here.

symposium+on+social+architecture

Metrics for Weighing Blogs

Last week I spoke at Bill Flitter's eBig monthly meeting on Blogs and RSS. My talk was about metrics and weighing blogs. Shel Holtz recorded my talk (thanks!) which is here (warning, giant mp3 follows that link) or see it here at the Hobson and Holtz Report.

Google Launches Blog Search

I just checked out the new Google Blog Search. I've been hearing about this for months, and now it's finally here. Of course, I checked searches I'm familiar with, to see how it does. One for my blog name (as key word) turns up many matches but also is missing a number of references I know exist. Not sure how they determine the relevance matches but on that one, for my own blog name, I come up the most. I would say that page after page of me referencing myself is not so useful, and I can't believe I'm more relevant than many of the people who link to me. But what that means is that they are likely matching terms as well as linkrank. It would be helpful if they filtered out the blog with the same name, or relevance searches for any blog that matches will just be useless lists of page after page of that blog's posts.

URL lookups are similar.. with many matches but many missing items as well. However, since I rarely link to myself, under a relevance view, I get lots of other people's posts linking to me. But wait, they are all strangely in reverse chron order, even under relevance. Date view shows the same exact list of results in the same order. Blog search is still in Beta.

The advanced search, under data range, appears to start March 1, 2005. Still, it's really useful to be able to isolate any period we want, and once there is significant historical information in Google's database, this will become far more useful.

I did about 20 searches on terms and links that I regularly monitor, and found that references created by a blog or blogger or concept associated with a blog show up as the top result(s), for those same searches. I'm not so sure that seeing blogs referencing themselves is so useful, and maybe these should be a third view, after relevance and date, where they provide *all results* including those kinds of references. Again, I'm seeing page after page of results that show the blog associated with my search term, instead of everyone else talking about that concept, blog or url. Not terribly useful as I can just go to that blog to see all its own posts. I want to see what others are saying about it. These searches are ones I have watched for years in my aggregator using 5 different feed search services, and I'm noting there are a lot of recent missing entries at Google as well.

I'll keep playing with it, but I think they have a ways to go in understanding blog search and getting the result sets right. And they need to pull in the rest of the results I know are there for all the searches I did. However, the search result layout is clean, and organized. It's a good start. And is very fast.

One more thing, Google Blog Search's FAQ says that it is searching RSS and Atom feeds only. It doesn't spider blogs. So it will only pick up what goes through a feed. Blogs that send through partial posts, will only have that part of the post in the RSS feed included in search results. This is a drawback, because the possibilities for finding topic communities are lost when a service doesn't spider blogs, because the full information is going to be necessary in order to figure out that problem.

Anil Dash has a good write up on the overall issues with this new search.

All things mechanical..

...seemed to fail or their failures became apparent yesterday.

I got a beep on my phone indicating voice mail, but there was no ring. So I got into VM and found there were 50 messages from the past two weeks. Took me 20 minutes to listen to them. Thank you Cingular.

And if you called me in the past two weeks, I'm sorry, the calls were apprently not all ringing through and for those who left messages, neither were they getting to me quickly. And now they magically are again. Thank goodness for small favors. I guess it's a favor when you pay for service and you actually get it?

And the DSL, from SBC Global.. spotty to non-existant for most of the day yesterday.

And my car, at the shop for a regular checkup.. they replaced some sort of dual oxygen censor.. and now it appears one of the new censors was bad. So I have to go in again. But the 'check engine' light that came on yesterday while on the bridge was not fun.

Lastly, Firefox and moveable type 3.2 are not cooperating. Third time this has come up with the new version.. I otherwise love the new 3.2, it's organized, easy, and thank god they made it possible for me to customize the new entry windows.. so that it can exist at a reasonable size. But they have no info yet on keywords (what are they, how do they work, are they like categories, can i display them.. so i'm not using them til i can figure out the value). Hopefully, after three attempts, I'm still hoping to get this and my other post on Andrew Rasiej up there.. after all.. the election is tomorrow!

By the end of this day, I wished I'd stayed in bed.

Flickr and Yahoo and Identity Management

(Note, updated below, 8:30am 9/5/05) I've been thinking about this identity issue since it came up Monday (okay, yes.. I'm really trying hard to take this week off from blogging but here I am, blogging).

Anyway, I had a thought Tuesday, and figured I would toss it out there. So, Yahoo could take it's hundreds of millions of IDs (that all authenticate in Yahoo's system through the front section of their email accounts) like joeblow468 which is really joeblow468@yahoo.com, and change those to iNames. Yahoo could become an iBroker for all of it's IDs, which are unique, turning them into:

=joeblow468

or better yet

=joeblow468.yahoo

These IDs could then easily be interchangeable with Flickr IDs:

=photojoe.flickr

... I think what Yahoo is really looking for is something simple to integrate IDs between their company and those they acquire, as well as ways to make themselves more open. I don't think they intend to freak everybody out, or make them paranoid because of the necessary integration. If Yahoo used an iName system, all ID's no matter where they come from could be made into iNames, with Yahoo as iBroker, where they could then integrate additional ID's into this system. ID's from newly acquired systems could remain essentially the same.

Also, it seems like what really matters for Yahoo is not the hundreds of millions of IDs (well, yes they do for now..) but rather Yahoo wants to be thought of as forward thinking, innovative and useful. And they want to *be* heavily used. Creating user ease and control for authentication, and being new and innovative with identity management could accomplish that. Having great services like Flickr that users love makes them want to come back, not lock-in.

The point here though is that people are very sensitive about their identities. It may not be rational or in proportion to the issues at hand, but user's reaction to identity integration is one that you can expect to get from people if you build systems that don't take into account how emotionally and personally people take their identities, and digital representations of our identities.

(Update 9/6/058:11am)
Yahoo did not reset cookies last week, per Stewart's comment below. Instead, it just appears to have happened coincidently for a group of users I was with, and then someone confirmed this to me a Yahoo. But actually, it's not true. Instead my cookie (and about 10 others) were just randomly triggered to ask for a login, where we were asked to reenter our login IDs.(End of update).

I was presented with a login screen which had two entry points: one for my Flickr ID and one for my Yahoo ID, with a note underneath the Yahoo entry saying that once I used the Yahoo ID, I would need to always login with the it (this has since changed to explain the tie in of ID's with either type of login, but doesn't warn users that once a Yahoo ID is used, there is no going back). This was a bit jarring. I could not go back to my Flickr ID (the backend authentication ID, though my front end ID stays the same). I love Flickr. I use it everyday. It's my photos, my emotional representation of how I spend time, who my friends are, who I see now and then, what I care about. And they want me to integrate with my Yahoo ID, not something I feel is the least bit cool or fun or that I have emotional attachment to compared to Flickr.

I had to think twice about whether I wanted all my Flickr photos to be associated with my Yahoo ID, which represents 9 years of random email, plus some old website data (once upon a time, hodder.org was hosted there for 6 months or something), and various yahoo groups, plus other random stuff. I don't think I do, because information online doesn't age very well. So I'm going to make a different Yahoo ID, when the time comes where we have to change over, so that my Flickr stuff can stay separate. But for now, I'm mourning a little bit the coming loss of my Flickr ID (for the login part of things). In a way, it actually represents both me, and my community interactions there.

(Note and lengthy update added here:) When I say the loss of my Flickr ID, what I mean is, the authentication ID that I use, which is an email address that I like. I don't mean the front end ID, which is slightly different, and what the world sees when they go to my photos. Nathan Arnold pointed out below in a comment that the front end ID isn't going away. And he's right. But from a user perspective, what I login with is a backend, or functional ID (my email address which is mary at hodder blah blah) and what appears in the front end is my Flickr ID name, or Mary Hodder, and for the user, these are functionally tied together. I put in the "mary at hodder" login email to get to the "Mary Hodder" account.

Which brings up another point. When I was making the example above, I used joeblow468 because to me it looks so much like what so many Yahoo and Hotmail and MSN and AOL emails look like: an address the user tried to get, that actually represented themselves, like joeblow, which was taken, so they wound up with some crazy kludge at the back of their ID or namespace, usually in the form of numbers.

My own Yahoo ID is not what I wanted, even from 9 years ago. It's got an extra "o" on the end of it, because at the time, the name I wanted (an old nickname from my boyfriend at the time) was gone. So since it had the extra "o" on the end, which I've never really felt good about. If it had some random number on the end, which at the time I was getting it, Yahoo offered me as a choice, I would feel even more random about my ID/email at Yahoo. It's one of the reasons I made that email sort of a dumping ground for (frankly) garbage. I couldn't get the name space I wanted, either for login (backend) , or as an email (front end and to be shared with others) that I cared about or for things that I want to keep well, like my Flickr account, which is perfect and pristine. So, by asking me to tie my perfect and accurately representative Flickr name space ID on their frontend, to my bastardized Yahoo namespace on the backend (for authentication), it feels bad. And I suspect this may also be what's behind some of the discomfort people feel when Yahoo asks us to enter a Yahoo ID to login, in order to come to a Flickr ID on the front end.

What iNames might do for Yahoo is allow them to also give many more ID names that people care about (maryhodder, for example, sans numbers) with something attached (like .flickr or .yahoo or .yahooligans or whatever they want to give after the initial namespace). That way, we could have the ID in the first section of the namespace that we care about, but have a different second name after the dot, which means that there could be lots of maryhodder's without a large number at the end). My guess is, we would start caring much more about things like our Yahoo accounts if we had a name ID that reflected what we wanted, and how we view ourselves.

No one views themselves as joeblow468... they view themselves as joeblow, and the idea that we have to put a number at the back serves to tell us that we are not unique, we don't really own our name (someone else does), and therefore I suspect that many feel like me about making it the place we don't care about or keep well. When all the good name spaces somewhere are taken, what's left is often treated as garbage. Others have told me they use these kinds of accounts as their "spam" accounts, but it would be interesting to do some research to see if many users are experiencing what I have just described. (End of update).

If Yahoo used iNames to manage identity, this wouldn't really matter. They could just take an iName into their system as a unique ID, whether it was =joeblow468.yahoo or photojoe.flickr or =mary.hodder. Then they could show themselves to be really cool, by being open as an iBroker to allowing IDs to flow in and out of the system as well, as users choose to move their iNames about from broker to broker.

It would be an awesome thing for identity management if I could use my Yahoo ID to log into the NYTimes, or my bank, or whatever, or use my =mary.hodder to login to Yahoo or the NYT or my bank, with single signon where I control who gets what information from my iBroker and I can have a couple of IDs to quell nervousness if I feel too much is held in one place. But the fact is, having something simple that works everywhere would be really cool.

One of the people who works for me showed me a database on Monday, while we were discussing the Flickhoo flap, that she's been maintaining for the past 10 years of all of her logins all over the internet. She has 249 different logins at that many sites. Solving this problem, so that she could just use one or two or three logins everywhere, makes a lot of sense. And Yahoo could really take the lead on Identity Management by adopting a system that would create simplicity for users, and simplicity for themselves. And turn down the public relations flap a notch when they acquire companies and have to integrate users and ID's into the company.

Google Talk: It's great but..

My work group is loving this, because we can use it talk and send files across lots of people, with different IM systems. But there is only one problem. Some of us are on macs, in fact the ones who tried this out first. In order to get all the windows people on it, we've had to go back to a windows person with Talk installed as an application, which has the only way to invite people who are not yet on Google Talk. If you are a mac user, you have to use a mac client with jabber support, but then you have no way to invite more people to use Google Talk, that we can find, and the Talk help section points to the windows Talk application in order to invite more people.

Very frustrating.

Important Questions About What Matters as we make an Ad Hoc Community Algorithm to Describe Blog Communities and Weight Bloggers In Them...

J. LeRoy on Convergence and Procreation talking about ad hoc groups in the blogosphere:

To tie this into my recent posts about rankings and tracking of expertise on the net, I want to note the transience of thought on the individual, the right to multiple associations, and the healthy aspects of not being an expert.

Right on! Dynamically generated communities will be key to this process, because day-to-day, these communities shift, and yet, it is also important to see conversation over time. The blogosphere is often self correcting, and bad actors get modded down over time, with useful work modded up. So we must achieve a balance in this dichotomy.

These groups and communities on the net are well formed. And, yes, the conversation is well formed and on-going, but we should be wary of rankings that build up expert or superblogger status on given individuals.

Inherently:

1 Ad Hoc Groups are created to solve problems

2 Blogs' subject matter is transient

3 Community is fluid

This leads me to wonder how we would establish relevance of blog posters by community indicators when the communities themselves are in flux by design. Communities defined by a given area of interest will tend to highlight those who are perhaps overly focused on those areas of interest. It may yield a search of those who are tunnel visioned and not those who are innovative.

So what are the metrics that help balance tunnel vision with newness and interestingness in a community and the weighting of blogs, to discover innovation? Or do we use the metrics we have now, but tune them to balance these concerns?

Julie Leung in Now serving: Blogher Bouillabaisse:

Why do metrics matter? From dialogue I've read, it seems metrics matter because they matter to the media. The Top Whatever blogs are the ones that will be referenced and used to represent blogging to the majority of people in the world who don't blog. They in a sense become who we are. Yet as Staci Kramer pointed out in the discussion, not all journalists care about the Top lists either.

Why should metrics matter to a blogger like me? I've been taking a bit of my own Blogging 101 advice these past couple weeks and considering what it is I am trying to do here. Why am I blogging? Sure, I use Technorati. In fact on the Blogher survey before the conference, I checked that I do care about traffic. I've even cried over my Technorati ranking. But that was mostly because I wanted to be involved in conversations. I wanted to know that others were reading and responding. And I also had mistaken ideas about what my Technorati rating should be, after seeing my husband's statistics. I've now realized I'll never have the links and traffic he does. We are blogging for different - but overlapping - communities and in separate niches.

All I want to do is write well and have good conversations. As far as finding good blogs, rankings only reveal what lots of people who link like to link. They are not necessarily indicators of good writing or good blogging or even blogs I want to read. I use Technorati, Feedster and PubSub to know who is linking and talking to me. But as for my ranking, I don't need to know it in order to blog or to sense I am blogging successfully.

Asking ourselves why we blog is very important in this discussion. It can lead us to uncover some implicit motivations and activities we engage in, that might help us with this effort. On reason I blog is to create a knowledge management system for myself. Another is to point to things I'm interested in, sharing that with others. And I like having an opinion exposed now and then. Digging deeper will help expose the answer to the broader issue at hand: finding out who does what, and what is done by whom, and what and who matter.

Mathemagenic in Link love: lists, clouds and action points asks from 'what to how?" and wonders.. where to get the data:

But it starts with the data. And the data is not public.

I can not speak for others, but I can talk about problems we have with the data needed for our research (which addresses some of the "link love" aspects). What we need to develop algorithms and tools are pretty simple: blog content in "full-text RSS quality" via APIs...

We tried many of the current blog indexing tools: no luck (those that are pretty close to what we need, BlogPulse, Technorati and Bloglines are either consider the data they collect commercial or do not have APIs to access it). As a results Anjo is working on weblog spider instead of community discovery algorithm.

Data is an issue. However, we will ask companies collecting data to help us. Three have firmly committed to running a community algorithm against their data, along with a startup in stealth mode, that also has a complete database. We'd love to have as many blog search companies as possible on board for this ad hoc community experiment. I've seen this experiment as something of a partnership between companies who create blog search for readers, bloggers, PR people, advertisers, and marketers, and the blogging community that is talking about this effort. If others see problems with this, please tell us.

Interesting Ideas Afloat

...by people thinking about how to better understand the blogosphere, through topic clouds and weighting of bloggers. People are asking for things they are interested in seeing, expressing concerns and making great observations about the overall problem. This continues a discussion I started there, and continued there, there, there and there.

The del.icio.us tag for this meme is linklove.

Ericka Menchen Trevino/technology and the social on Blog Rankings:

It’s not surprising that there is no single way to determine ‘the best’ blogs, and any attempt to sort the mass of blogs will be threatened by SPAM and the preconceptions of the rank creators. Alternative metrics do need to be developed - the more the merrier, but what will be more informative is studying how real people find the blogs they read.

So should we have different metrics for different communities? Or is that too much? Maybe we think about including ratios of different metrics of participation, so that many kinds of social gestures can be included depending on styles of participation. A blog would still be weighted for some participation in a community? In other words, if someone comments a lot, and posts now and then, they might be as conversational as someone who comments little and posts a lot.

Ed Vielmetti in Yi-Tan: Jerry Michalski and Mary Hodder on "Link Love" (a post based on today's Yi-Tan call where I was a guest host talking about the problem of links as ranking mechanisms, and how we might find topic communities and weight things we value, like conversationalness):

The observation is that all of the sites that keep track of "top 100" lists or "top hits" scores for searches or weblogs have opaque algorithms for determining same, and what results as the lists are not very accurate once you exit the mass public Internet and get into more specialized fields or subfields.

...

...one really interesting salient point about the metrics used to compute influence and activity within the sadly named "blogosphere". (ugh) Most of the blog ranking tools use links as their proxy for love. In my experience, however, it's the good and useful comments and discussions that are a lot better reflection of whether someone really cares enough to click through and make a difference...

Comment discussions are important and if we can solve the comment spam issue, might be a great metric for folding into measures of conversation.

Ed also suggests we look at energy as a measure:

...people are differently influential in a network by the amount of energy they bring to others in their sphere. Should we be recasting our measures and metrics as "link energy" rather than "link love"?

J. LeRoy on Are You Really Atrios?

So the issue here is how do we invent metrics to judge the relative social worth of one blog post over another. The analysis needs to take into account the fact that gamers of this sytem are flooding the internet with content-free blog posts and web pages that contain key words and copious links. These sites are primarily aimed to get eyeballs to make money for the people putting up these sites. They are web spammers. Web spammers are very good at gaming rankings like these and defeating their purpose.

Community is measured differently by different people. Some measure it by the amount of participation in discussions. Some measure it by linking. Some measure it by blogrolling. But all these are, at best, indirect measures.

Direct measures may be no better. If our solution is merely to tag links by their relative importance, those tags are easy to spot and easy to fabricate for web spammers.

Peter Kaminski talks about the Network Map vs. Cocktail Party

This reminds me of Flickr's new interestingness feature, which can rank large sets of pictures by, well, interestingness, as demonstrated by more or less social cues. Social cues are key, because they illuminate what real people find interesting, instead of just statisticians (or marketers).

...

...there is lots of interesting stuff in any long tail. So while you may find a (biased) ranking scheme you like, and be happy with the top 100 most of the time, you (or the system you're using) also need to make sure you randomly see some of the long tail once in a while, to discover the interesting things your favored ranking system hasn't.

...

...a network graph that shows nodes and maybe does fancy node sizing based on rank and all that is cool, and fun to play with if it's interactive (in the same way sodaplay is fun to play with), but it doesn't tell me anything. It's hard for my monkey mind to parse the map.

A better metaphor, I think, is a cocktail party. I want to see faces, who's clustering with whom, and how the clusters evolve. Make me an animated picture of a cocktail party (or a tribal gathering, or a barn-raising), and that I'll get -- I've been evolved over thousnads of years to intuitively grok those kinds of social situations.

That's a very interesting idea.. what that would look like I don't know, and many bloggers are pseudominious and don't really want their pictures all online (not all of us are exhibitionists, in fact few of us are across 14m blogs).

Blog Business Summit: Metric Talk Slides

I was asked at the last minute to cover blog metrics at the Blog Business Summit on Friday, in place of Anil Dash, who had jury duty.

Here are slides from the presentation (ppt). The first 34 are Sally Falkow's, and the rest are my additions.

I did a basic presentation about metrics for describing and weighing blogs. These metrics are, as I've said before, very very alpha. Getting link counts or subscription numbers doesn't really show a blog's conversationalness in it's topic community. If a person is trying to get a sense of where an unfamiliar blogger is situated, and how involved they are in talking, those metrics just don't do much for you. They are measures of popularity.

Hopefully, the efforts to make a system of community clouds around topics, and weight participation, will help this problem of finding and contextualizing unknown bloggers.

Adina Levin on Conversation Clouds... And Mitch Ratcliffe responds with Cloudmakers R Us

Awesome stuff by Adina Levin on Conversation Clouds which I'm just going to repost:

The cloud would be a picture of a conversation surrounding a person or a topic. The picture would show the relationships between the participants in a conversation. The densest areas would represent people who frequently cross-reference each other over time.

You can start with a participant (the url of a person's weblog), or a search term (a word or tag) Nodes are clustered based on closeness, measured by number of links and reverse links over a period of time (comments, too, if you can measure them).

If the picture starts with a link, then that link is at the center of the picture. The picture shows the links between the first node and the other nodes, and between other nodes that are connected to each other.

If the picture starts with a word, topic, or tag search, then the cloud contains a cluster of blogs that include the term or tag in the last time period. The picture shows lines between blogs that link to each other. Unlinked blogs are thrown out.

The cloud is built from a data set over a time period; the user should be able to scale the time (conversation over a week, a month, six months) The conversation cloud would need to provide ways to navigate through conversation space. If you click on a blog, perhaps you re-center around that blog's conversations. If you click on a tag or topic, you search based on that. You'd need to experiment with several ways of allowing browsing out from the first cloud.

This type of picture would not measure rank. Instead, it would illustrate the connections within subcommunities.

Cloud-browsing represents a pattern of blogsurfing. A reader might start with Mary Hodder's post on blog metrics, and then traverse to Dina Mehta, danah boyd, Stowe Boyd, Ross Mayfield.

The cloud would show in graphical form what a Technorati or Blogpulse search would -- who linked to the post. And it would also illustrate the repeated links and cross-links as people reply. If you zoomed out the time horizon, you'd see some relationships become more obviously dense, with repeated patterns of links and counterlinks.

I think this sort of presentation would get more of what we're looking for -- a picture of the relationships in a community that reveals participants, both loud and quiet. The ability to browse the conversation.

The results would be more interesting than a diagram of an email thread -- where participants already know who's talking to whom. It woudn't be particularly rankist, since webwide popularity isn't relevant to the picture. It would let you browse to related people, or related ideas that the same people are talking about.

The next step is to test this idea, maybe with a manually drawn picture, and then with a dataset and a toolkit like TouchGraph. This seems like a good experiment to me. It could be somebody's done this already. Or somebody's tried this and proved that it doesn't work. Please share if you know.

p.s. Zawodny talks about the need for content discovery. I don't know about you, but a lot of the content that I discover comes from browsing through a conversation and finding voices that I want to keep hearing.

And Mitch Ratcliffe, who has a company called Persuadio which visualizes relationships between data, responds with Cloudmakers R Us:

I've been following this discussion, mostly holding my tongue because it may look self-serving to respond with "here are the pictures you want of blog relationships" by pointing at the MyDensity site we've put together to show off the social analysis tools built by Persuadio. I also realize it would be incorrect, as we've focused on the big picture to the detriment of a small world.

Simply put, like many of the indexers, we've tried to capture the role of any blog or Web site in any conversation (being about more than blogs has been important to us from the very beginning). Meanwhile, it would have been simple all along to provide what Adina calls a "conversational cloud" that shows the relationships around a single posting or Web page. And, frankly, it took someone asking for that simple solution to realize it was the first thing we should have offered instead of trying to solve the really huge problems we're wrestling.

The current MyDensity maps show all the relationships around a blog, rather than the links to a single page (which we can do, but just hadn't).

Unfortunately, we hear often from customers that they want a "top" this or "top" that list and had decided to focus on that. With limited resources and real money coming from these people, we paid attention. It is what they are ready for.

The desire to see the big picture is endemic in a changing market. Top 10, Top 20 or Top 500 lists make a certain amount of sense if you are trying to aim for plain old low cost-per-thousand (CPM) or cost-per-impression advertising deals. Most advertising and marketing people aren't prepared to think outside the CPM box, and if they do, they think about relatively ineffective cost-per-click (CPC) ads.

The contours of this market are very poorly understood. ComScore, the Reston, Va.-based research firm, in an August 2005 report describes visitor traffic to the top blog hosting sites in aggregate even though the blogs hosted by those services, their authors and readers share few demographic or behavioral characteristics. For marketing and advertising purposes they are separate publications, not a monolith that can be compared to the traffic of the New York Times—however, ComScore does make that spurious comparison. Yes, more readers (ComScore does not distinguish between readers and bloggers visiting BlogSpot to author their own sites, confounding any attempt to characterize audience size) may visit BlogSpot in a month, but the information they are consuming and commenting on there is disorganized; by contrast, the editorially coherent sections of the New York Times create viable venues for addressing audiences with specific interests.

Marketers are stuck between that familiar composed environment of the Times, with all its shortcomings, and the apparent anarchy—from their perspective—of the blogosphere with all the opportunities it represents. Every discussion of a "top" list is predicated on mapping the reach of a site to the community around a blog or group of blogs. There's a hunger for something recognizable to grab hold of, which is why I keep harping on the question of how to get today's content owners to start across a bridge to content sharing.

If we can solve all these problems by laying out the flow of influence, the role of trust and conflict in discussions, magical things will happen to the marketplace of ideas.

When it comes to conversations about specific topics or just conversations between people, though, there are multiple dimensions of value, some personal—the kind of information in the clouds around a single posting—and some profoundly economic: If you can target advertising based on behavioral characteristics, the value of an ad can soar. If you know what people are talking about, you can guess why and position a contextually relevant and high-value CPC ad alongside the content of the page.

If the marketer were really radical, the ads would go away and the message, with all necessary disclaimers so that it would not pollute the content, would come through as part of the conversation.

When it comes to blogs, the content is so personal and bloggers so interested in understanding the intellectual currents around their writing, audio or video, that the first responsibility of a company that wants to be of service to the market is to be of service to the bloggers. So far, Persuadio has been of service to a couple customers, but if we cannot get more information to bloggers we'll forever be outside the market we most want to serve. For most of us bloggers, it really is about the neighborhood (Ross Mayfield's discussion of the Rule of 150 play well, even years later) we're talking with than our rank in the whole blogosphere (though such ranking is a guilty pleasure the honest blogger will cop to).

That said, as we map blogs we also map the rest of the Web and the relationships between all information, individuals and organizations we are often confused as primarily a mapping service rather than an analytics service. We want to offer information about who is talking, their relationships (even the hidden ones) so that everyone can judge ideas and movements based on the fullest information. We've been aiming at that, but thinking like an old-style analytics company, so we're going to change, but I hope you'll remember that there is a lot of social measurement going on in the background that have both social and economic value.

We'll have link clouds for you very shortly. Allow us a bit more time and we'll let you configure the variables of the map, so that you choose to include current or archival links in the calculation of influence, as well. We're awake to this, now.

Nice! Go look at Mitch's diagrams.. but you get the gist. It's just so cool.. I figured why rewrite, just put up their words!

More comments on...

a community based algorithm and the attendant issues...

Michael Frasse on Information authority and ranking:

Hodder says, rightly, that the metric for assessing weight in the blogosphere should be open, not closed. “Bloggers should have input about the importance of one social gesture over another,� she writes. “One metric over another, and know what it is that is included because it will be used to describe them.�

Kevin Burton has published an excellent response to Hodder that really advances the discussion. Burton has been working in this area since its inception (what, five years ago?) and has clearly been thinking about these issues. His main point seems to be that transparency in a blogosphere ranking and reputation system is problematic because of technological complexities and intellectual property barriers.

And Kevin may be right, but I think we should at least consciously talk about why something ought to be secret, and choose it, rather than just falling unconsciously into having a secret algorithm. I'd like to hear more about this, and discuss the reasons in the community, explicitly.

Barb Dybwad/The Social Software Weblog on Small is good, too: on quantifying connections:

There’s all this fixation on getting links, getting traffic, getting on the lists. Why? That’s about monetization, that’s about fame/celebritism, that’s about gaming a structure that uses heuristics to measure what is supposedly value. And distantly, it’s about having your voice heard — I get that. But it gets blown way out of proportion. We forget that getting on the Technorati 100, in the end, is supposed to be about getting your voice heard by the people you want to hear it — instead it just becomes yet another pissing contest.

But that’s an old model — it’s a vestige of the economics of scarcity, the analog world. The fact is, not everything I write needs to have a mass appeal. That’s precisely the beauty of the long tail.

It's a great post.. you should just read the whole thing. I only quibble with one statement:

If you want someone to pay attention to you, if you want to make a new connection, just ask.

Well.. if it means asking for a link from another blogger when they wouldn't have otherwise given that attention, it to me sounds like gaming the system, to get a higher link count to rise up the power curve, rather than putting our efforts to measuring conversation within topic communities. If it means really conversing with someone, and getting their attention that way, then I think it's great.

She finishes with this, which I very much agree with:

We don’t need to form new companies just so we can make new lists, in order to take action. We need to make new connections, and foster the connections that are honest, authentic and strong, as opposed to merely proliferating a bunch of opportunistic weak links.

I think making a representation of what we want to see measured and giving it to existing companies with existing databases in a community-company partnership is a great way to go.

Kevin Burton on Indexing and Blog Post Popularity:

My thinking is that we should write up how an ideal metric would recommend posts. The algorithm implementation problem is another discussion as this can be implemented in many different ways. The devil is in the details I'm afraid.

Kevin actually did a comprehensive response to my original post, much of which I'd thought of, but it was great to see his take on the same issues. I think that what is important right now it that the community discuss what it sees as important to weight in an algorithm, or system, to participate in the process. Oh, and Kevin, sorry I didn't get any of your response into my first post about what people were saying, as you were a really early responder. My mistake!

PC4Media on Blog Social Network Analysis:

Q: But, what happens when the scale gets bigger?

A: (Yes, I am interviewing myself.) Of the 250 odd people that read my blog and the 300 or so that have linked to it, how many of them are actually people that have a meaningful relationship with me? Or even just a two way relationship with me? How many of them know someone I know? Or 5 people that I know? Do those people have meaningful relationships with each other?

I have no freaking clue. But, damnit! I would love to know.

Jay Rosen, in the comments on Burningbird (where there is a fantastic discussion going on about this stuff):

what a blogroll “says� in my understanding of how to blog well, although I must add that people can and do use their blogroll for any damn thing they like. My point is it makes perfect sense to link to the post that is already in your conversational field (blogroll.) The idea that there is something basically corrupt about it is a fantastic suggestion, yet perfectly normal for this discussion.

But it is also worthwhile, I think, to reflect on what would seem to be “natural� linking practices, because they’re one way a blogger can get stuck in ruts, or become narrow minded without realizing it.

An excellent point.. because I think it's not just that we are not aware sometimes of our habitual behaviors in ourselves, but also, that any system has to take into account in some way the many 'kluges' many of us have developed, as early adopters as well as newbies, linking styles and 'technological workarounds' because blogging software just isn't that old. It's very early and we are still experimenting with lots of things. What appears from a blogger to be a social choice may be due to some funky piece of technology. It's yet another reason why I think having a public discussion where people talk about their habits and reasons for doing things is so useful, and will add to the solution.

Sarah Allen on The Value of Conversation

There was a time when the web was a small community. When I started working on web software in 1995, a search on Yahoo! would list a small number of results. I don't know whether the whole web was indexed, but I think they kept up with the majority of it. I could view most of the web pages on a topic that interested me and make my own choices about what was most interesting and credible. When the Shockwave player was released, we could view every Shockwave movie that was created as it came on-line.

As the web grew larger, it became more challenging to keep up. Thanks to the innovations of the search engines, we can do a quick search and find something relevant to our area of interest most of the time. However, with the creation of these search and ranking algorithms certain voices are omitted. I rarely look at more than a few pages of search results. Is it possible that the pages that would be most relevant to me might be later in the list? Or even worse, because of some artifact of gender use patterns could it be that pages more interesting to me will often be toward the end of the list?

I like the idea of tracking conversations instead of purely inbound links. If we had an alternate index that tracked only links from posts (rather than blogrolls and other collections), then we might see where conversations are happening. Adina Levin describes this as a cloud presentation.

More comments on a community based algorithm and the attendant issues...

...with Linkrank, the Technorati 100, A listers, and creating a dialog to move beyond it.

Tish G at Love and Hope and Sex and Dreams talks about trading links to create a vibrant community:

I love the community of voices on my blog, the different kinds of people, from men to mommybloggers to compulsive knitters. Linking back to them shows that in my own way I'm maintaining a vibrant community of interesting people who don't need to be in the same niche. It's more about Life than about one particular aspect or thing in life. Then again...that's just me being ecclectic and non-committal I guess ;-)

This may be true, that by linking on blogrolls, as we visit each other's blogs, we read those rolls to see the visible community. But additionally, Halley brings this up (and Tish responds) as she has been thinking about how to make more bloggers visible with inbound link counts by asking A-Listers to link to others. With this, she tries to define Noblesse Oblige:

There is a notion of NOBLESSE OBLIGE -- here's the definition -- it is the responsibility of those of high rank, power or nobility to be generous -- literally from the French "noblesse" nobility (being the queen) "oblige" obligates one. Nobility and rank obligates you to do selfless, generous acts.

Is it not generous of those of "high rank" in the blogosphere to blogroll someone?

Two things. One is that for now, a link is a link is a link. There is no differnce in Technorati's link count system of whether a linking blog is an A-lister or not, so really by this methodology, asking for and getting a link anywhere would help.

However, it concerns me that the reason we throw this solution out is to change the ranks of the less powerful. But in fact it doesn't change anything at all except for a few because in the end we are gaming the flawed system of *only* counting inbound links to the exclusion of all other metrics of conversation.

Maybe a better way to approach this issue is to define Robert Fuller's ideas around dignitarianism, where those in power don't abuse their power and the community creates solutions to problems as they arise. It seems to me that the solution to our rankist issues with inbound links is much more appropriately solved by changing the system, than by asking those who benefit from rankism to share some power with a few (there is no way A Listers have the time or the blogspace to do all the linking we would need to see to correct the problems of rankism with linking.)

Jonathan Carson at Buzzmetrics Mouthpiece:

One thing which I think might be interesting to add to the discourse, would be something around topicality. i.e. "influential on what?" Because BuzzMetrics is typically answering questions of influence within a commercial setting, we are rarely looking for "top bloggers." We are looking for "top influencers amongst wireless application developers who happen to be positively inclined towards the Linux platform."

This makes the problem much more complicated to solve, but would make it far more useful from a commercial standpoint. That may not be the goal of the open source ranking system that Mary recommends - but certainly that system would then be leveraged for such commercial purposes if it were to be built.

He is absolutely right that topics play a large role in figuring out communities of interest in order to find conversation, if in fact that proves to be the best way to go about this. However, I would dispute that the community proposal is a 'ranking system' but rather a community algorithm, based on topic communities. But I want to see this community algorithm, as both a description of communities, by topic, as well as community generated. So I'm looking for expressions of what communities mean to people, how they want to see those communities, and Jonathan's example community: 'wireless application developers who happen to be positively inclined towards the Linux platform' is great because it gives indications of his needs and desires for community, and in looking for influencers, linkrank would not tell him. Rather other factors will show this.

Christina's LIS Rant:

This seems a bit egocentric or even whiney at first glance, but keep in mind that 1) many bloggers are actually trying to help people and need to have readership to do it 2) many other bloggers are trying to sell ads.

Dina Meta in Blog Ranking and Popularity:

For instance, I have no interest in what my ranking on Technorati is, but I do visit it daily to see who is linking to me and how they might have progressed a thought. Yet, I'm not so happy when these get transformed into lists, ratings and rankings. Are you merely well-known, or well-read?

How would you define and measure popularity in the blogworld? Can there be a robust quantitative measure? Does the blog software you use matter? Are links and comments and page visits and clicks good measures? How is stickiness measured?

I'd rather look at more qualitative measures (disclaimer : I am a practitioner of qualitative research) like relevance, integrity and credibility when you engage readers in your areas of interest, empathy generated, stretch in teasing boundaries, intimacy with your audience. A combination of respect and amicability. There was a good discussion about some of these issues at the opening session at BlogHer.

So I want to know what people feel are the measures of relevance, integrity and credibility? Interest and empathy generated, stretch in teasing boundaries, intimacy with your audience? Respect and amicability? How do we fold those in?

And Dina Meta again with Blog Ranking and Popularity (2):

I like this measure - "i enjoy their company" - maybe someone should use that as some form of index? There are some bloggers who come up with really 'popular' posts which get linked to heavily - they may be 'popular' in a mechanized sense, but it isn't always the case that they make for relevant reads most of the time. There's value in what Alok says as it may lend itself to a more holistic approach - if someone loves hanging out at your blog, enjoys your company through conversations there, that's the best measure for me. It is what builds my network and community in ways that are far more compelling than from just links I may generate.

I like it too, but what is it that bring us to an understanding of 'enjoying your company' if that is a standard we want to include?

Somewhat Frank suggests that blogs be divided into three categories in order to help solve the problem:

• Primary: General consumer blogs, such as teen blogs, family blogs, and other personal blogs (i.e. My Space, etc.).

• Secondary: Business blogs...purely-for-business blogs. They exist to sell products and services and are usually tied to companies or organizations.

• Tertiary: Serious bloggers, or as John describes them those "who talk endlessly about "the blogosphere". Recognize them? They are serious bloggers, info-providers, probloggers, A, B, and C-listers, people who use blogs to sell themselves and their ideas ~ what H.G. Wells called "the originative intellectual workers".

Mitch at Ratcliffe blog asks this question in regard to news, but links to the post, so I assume he means it to apply to both news and blogs:

I reiterate my question: How do you create a measurable connection between producer and customer, one with sufficient transparency and give-and-take that enables a debate that would actually improve the value, truth and usefulness of the news today?

And Halley again, recounting Doc's notion that the internet is intrinsically feminine (Britt Blaser was the first to actually tell me this a year or so ago and I think it's pretty insightful):

Even the word "LINK" is not neutral. It describes the way the brain synapses work. It describes the way the Net was built. It describes an attitude of distributed SHARED intelligence. It's funny to remember what Doc concluded -- it's so FEMALE, the Net is so feminine, as is the notion of sharing power in a networked structure. (Ever heard of a "family?")

Wondiring on Blog Social Network Analysis:

Blogging is only ostensibly publishing and syndicating thoughts to an unknown audience, and really more about pinging your friends (via links, trackbacks, etc) about your thoughts on what they're thinking or to get their thoughts on what you're thinking. I blog knowing that I probably only have a couple regular readers (who I in turn read each day) and I'm really just talking to them, and anyone else who wants to stop by I suppose. In that sense, it's a very organic SN.

To me that sounds like a nod to conversation, but I may be mistaken.

Adina Levin at Book Blog notes that clouds of blogs by topic, with some sort influence weight, might solve this problem:

Time would be an interesting factor. Perhaps one could view the cloud by week, month, or year. See how participation ebbs and flows over time. A longer time frame would be interesting -- I wonder whether other bloggers are "bursty" in their topics of interest. A long time frame would catch people who come and go.

And Library Clips notes:

We have to be mindful that incoming links score high with what is important to our contemporary culture…if war, gadgets, politics, etc…are important or topical within our culture, then these types of posts will get you high on the list.

We need to help the emergence of important blog content that is absorbed in the long tail (just because of lack of incoming links, and the specificity of the topic)…how do we expose these types of blogs….I think top blogs by topic, by comments, etc…is a start.

Sour Duck notes the value of the image from Paris in making the subject more interesting. Thanks much!

And thanks to Jonathan at Buzzmetrics Mouthpiece for the little plug, noted on his blog recommending 'HodderNot' as the name for this effort .. and PC4MEdia's buying of "amihoddernot.com" ... cute! Not sure that's right for this whole thing (it shouldn't be about me) but I love your spirit.

I take this as a vote of confidence and support in the effort!

Lotta Linkin Going On.. Or Not

I wanted to summarize some of the very interesting things people have been saying about making a community based algorithm for understanding topic communities in my post Saturday, Link Love Lost.

Elisa Camahort at Worker Bees Blog:

... all of this talk and tempest around some relatively new companies and their tools makes me wonder why people don't get as up in arms about discussing the algorithms behind general Internet search tools. Oh yes, people occasionally compare their positions on Google vs. MSN vs. Yahoo search, but there's little accusation associated with those comparisons.

Why is that?

...

3. Most important: they don't provide an accounting of the links they are using to calculate their rankings.

Why is this last one important...because it removes the incentive to link to sites simply to get their attention and potential links back. Look, from a publisher's perspective it's great to see who's linking to you. Understood. But it also encourages people to link to the "top" blogs because a) they hope to be noticed by said blogs and b) when someone looks at the top blogs, they can see who's linking to them, and the linker will be on that list...and therefore might hope to be noticed that way too, even if said top blog doesn't link back to them.

The point of counting links is to calculate relevancy. But the results are distorted because the very transparency of the web of linking in the blog community encourages dishonest linking patterns....

In blog search, the point of counting link *sources* (which is what Technorati does to make link rank.. and the top 100 of those end up on the Top 100 list) is about what Technorati calls "authority." This is different than relevancy. It may or may not get a searcher anywhere near relevancy as they search for a term or URL, and relevancy is far harder and more complex than counting link sources that link to a blog.

I know it's heretical to make any argument for opacity in the blog environment, but a little more of it might lead to more authenticity in our blogging, and in the search tools for blogs.

I think it's perfectly reasonable to talk about every aspect of this, including creating an opaque standard, if that's the best solution.

Eric at CollabuTech:

There are many implications for the corporate blogosphere. How do you measure the worth of contributions? How do you help people find "blogmates" who have affinity for and knowledge of similar topics? Do you encourage a particular pattern of linking? What norms do you establish?

Is worth something we want to establish, or do we want to let readers retain responsibility for judging what they read?

Shelly at Burningbird:

Yesterday, Mary released her new effort to identity alternate algorithms based on a dinner she had with Ross, Doc Searls, Halley Suitt, and others in Paris a couple of months ago. It’s a very detailed and thoughtful post, and I respect the amount of work she put in it, but it seems to me that no matter how much the community is involved in this effort, it’s just propagating the same problems, because the issue isn’t about technology, it’s about people and how we behave.

...

think Mary should stop with …I hate rankism. I understand the motivations behind this work, but ultimately, whatever algorithm is derived will eventually end up replicating the existing patterns of ‘authority’ rather than replacing them. This pattern repeated itself within the links to Jay Rosen’s post; it repeated itself within the speaker list that Mary started for women (�where are the women speakers�), but had its first man within a few hours, and whose purpose was redefined within a day to include both men and women.

Rankings are based on competition. Those who seek to compete will always dominate within a ranking, no matter how carefully we try to ‘route’ around their own particular form of ‘damage’. What we need to challenge is the pattern, not the tools, or the tool results.

She's right.. whenever there is a measurement, a power law develops where those at the top sit and the rest bend their behaviors toward them, trying to attain top status they don't have. Link counts mean people change their behavior to get more links. It's not the spammers I fear, but us.

But if we get rid of rankings, and instead see topic based communities with long standing conversation, can we get out of some of that power law dynamic? I'm not sure. Maybe not. Maybe we must simply refuse the metrics all together. I think it's an open question.

Stowe Boyd at Get Real:

Instead of developing a single, open source, mega algorithm for determining blog value, how about developing a simple standard for publishing blog metrics so that individuals or groups could easily collate various sorts of interesting metrics about blogs into meta-indices?

For example, imagine that I were to create an online solution, let's call it Blognetter, that would discover the centrality of any given blog in the implicit social network that the blog is part of (this would be a very useful tool, by the way). Pointing Blognetter at Get Real would discover links from Get Real to Mary's, Doc's, and Ross' blogs, and vice versa. Using various parameters, it would rapidly determine a network that defines a community, of some number of hops via links away from Get Real. Blognetter would calculate that Get Real is connected to and from a specific number of those other blogs. That service could then provide that data in an agreed upon XML format.

...

...a collating service, let's call it RankOut, could aggregate these various feeds related to Get Real, and any RankOut user could override the default weighting built into RankOut. RankOut may "know" what the feeds "mean" in a sense -- the builders of RankOut may be aware of the point of Blognetter, for example....

And lastly, specific rating services -- the Robert Parkers of the blogosphere, if you will -- could then publish their ratings, based on what they deem to be most important....

These are interesting suggestions.. I'd like to see topic communities, as I proposed in my post, with a conversation weighting.. that would show conversationalness over time. But as Shelly points out, people will behaviorally lean toward any system, no matter what it is. As for ranking people, I have real trouble with that. Even Robert Parker acknowledges that people just look at the wine scores (90+ means they buy the wine without reading the review). I think rating each other would produce crazy results, where just like junior high, we were all so concerned about being popular. For me, the value lies in understanding cohesive communities.

Rachel at License to Roam:

The Technorati Popularity list - you can ignore it, love it or hate it for lots of reasons. It's the equivalent of the All-time greatest hits chart, looking at total number of links over time. But just because Elvis or The Beatles would always be on top of the charts looking at total sales, does not mean they would be on the chart if there was a smaller timescale.

...

(In response to Jason Calcanis' bounty of $50k for a better ranking tool) I'd add another requirement - the ability to slice and dice by category/metadata. That of course would need the categorising data to be collected from the blogs or when blogs are registered with the search services, but I can see the need to be able to assess 'popularity' with a niche, ie movie blogs, music blogs etc. But that's a longer term desire.

But is it popularity within a niche? Or do we have metrics to show conversation, collaboration, interest? I totally agree with wanting to see topic communities.

In putting this challenge up, you could argue that Jason is acting in the 'old model', or, more likely the 'male model'. There's a problem, here's a solution, throw money at it and get it fixed my way. This is in contrast to the more collaborative, discussion based way I see Mary Hodder's proposal developing. So is Jason just perpetuating the male domination of the space by making more lists based on popularity? I don't think so; he's trying to make what we have (a subjective, measurable analysis) better and is prepared to encourage it.

I don't see the link count and the corresponding rank as necessarily male, but rather as a legacy media model. However, legacy media measurements are were developed at a time where men completely ran that business. So they naturally reflect that point of view. Now that digital media allow us to measure things easily in many more ways, and we have many more styles of blogging than just those that fit legacy media paradigms, why not figure out better ways to discover interesting communities and discussions?

danah boyd at M2M and Apophenia on the biases of links:

There are a few things that we know in social networks. First, our social networks are frequently split by gender (from childhood on). Second, men tend to have large numbers of weak ties and women tend to have fewer, but stronger ties. This means that in traditional social networks, men tend to know far more people but not nearly as intimately as those women know. (This is a huge advantage for men in professional spheres but tends to wreak havoc when social support becomes more necessary and is often attributed to depression later in life.)

And yet, in all of these systems, a link is a link is a link.. with no distinction for type, or network ties, or styles of linking.. or God forbid, types of links (as in no-vote, + or - in the rel tags -- who knows what a blogger means when they use those tags).

While blog linking tends to be gender-dependent, the number of links seems to be primarily correlated with content type and service. Of course, since content type and service are correlated by gender, gender is likely a secondary effect.

Interestingly, there are distinct clusters of norms with linking in blogging, not a coherent and consistent one. The search engines (and the Technorati 100 and PubSub’s Daily 100 Top Links) are validating one of those clusters, regardless of whether or not that is what searchers are looking for. The Top 100 is a list of blogs who either fit into those norms or have adopted those norms in their patterns (most commonly the companies).

...

These services are definitely measuring something but what they’re measuring is what their algorithms are designed to do, not necessarily influence or prestige or anything else. They’re very effectively measuring the available link structure. The difficulty is that there is nothing consistent whatsoever with that link structure. There are disparate norms, varied uses of links and linking artifacts controlled by external sources (like the hosting company). There is power in defining the norms, but one should question whether or companies or collectives should define them. By squishing everyone into the same rule set so that something can be measured, the people behind an algorithm are exerting authority and power, not of the collective, but of their biased view of what should be. This is inherently why there’s nothing neutral about an algorithm.

Very interesting stuff. Keeping this in mind as we discuss what we make will be key to gaining something we consciously want to describe ourselves.

Assaf at LabNotes:

Start with this blog, use it as context and search for the keyword ‘blog’. First observation, there’s a lot of links coming out of this blog. Most are links to sources I find interesting, relevant, authoritative. Others may disagree, but in this particular context, my outbound links rule. Anything these sources have to say about ‘blog’ should be ranked highest.

One problem with outbound links are that they are extraordinarily susceptible to spam. However, we have to deal with that anyway, so thinking about how we weight outbound links is valuable.

Second observation, those blogs link to other blogs, which they find interesting, relevant, authoritative, etc. So that’s a second hop that increases the sphere of relevance. Repeat enough times and you’ll spider the entire Web, something to do with six degrees of separation. But now we’re just duplicating Google.

There may be a way to derive who uses whom for filters, but this may be reflected in RSS subscriptions and reading habits. However, there are serious identity, privacy and data ownership issues (users should, in my opinion, own their data) to figure out first before we can think about using this kind of information.

Third observation, limit the number of hops to a small set (say six), and decrease relevance in proportion to distance. So a blog four degrees of separation ranks less than a blog two degrees of separation. Interesting patterns start to emerge.

Link decay is a very interesting idea. PubSub does it now, but it's not clear to me yet what the effects are. However, I plan to discuss it with them so that I can understand it better. Bob Wyman explains more in this post.

The Vision Thing with Enough with the Lists:

...Mary Hodder's post about "better algorithms" (sorry to generalize, but my eyes glazed over and I have yet to read the whole thing), and in a nutshell, I’m really sick to death of "lists." If you've seen one Top N list, you've seen them all. Wake me when there's a list that actually conveys something interesting.

This is not about making a single list. This is about making a metric that takes several factors into consideration, to find topic groups who consistently talk about something. At least, that's what I first proposed in my eye glazing post (sorry about that). However, that may not be what we end up with, as I believe the community should decide what it wants. If something else is better, let's try it.

Link Love Lost or How Social Gestures within Topic Groups are More Interesting Than Link Counts

A discussion about creating a new metric for understanding blogs is something I think the community should have the chance to participate in to find a different way of perceiving a blog, or the ripples a blog makes. Partly I believe this because of the frustration people express about Google's secret algorithm for pagerank, where they feel something this powerful should not be secret (update: the algorithm is not secret but the ordering of the search results is secret). And partly because I see that blogging is a opportunity for people to talk transparently, so why shouldn't the algorithm used to express our weight in the blogosphere also be open. Bloggers should have input about the importance of one social gesture over another, one metric over another, and know what it is that is included because it will be used to describe them. And also, I cannot assume that the ways I read blogs is the same as everyone else, so I'd rather have a community algorithm in the sense that the community has commented on the weight of some metrics over others within the algorithm, and not just assume that the ways I or others weight these gestures in our blog search are correct for everyone.

A closed algorithm is purported to be a kind of spam control, as opposed to an algorithm that is open. But a community based standard means the community can help police those that try to game it, if we put in place mechanisms to flag those who abuse the system. Transparency as it exists in open source software, and as it should exist here, is the opposite of security by obscurity. But creating this is also an experiment, and help is needed in order to make creating a community based algorithm possible.

Currently, blogs are measured in systems like Technorati or ranked in PubSub by links or by number of subscribers to a feed in Feedster. In particular, these are the not very interesting, subtle or telling measures used to make indexes like the Technorati Top 100 or the PubSub 100 or the Feedster 100. In Particular, the Technorati Top 100 is based purely on inbound links. All of these lists tend to favor those who blog in more general, popular topic areas, and not those who are specialists in an area.

For many bloggers the relevant sphere of influence is not overall popularity, as those indexes express. It's influence and connection within a community. And the relevant measure of connection isn't the number of connections -- it's the depth and impact of those connections. This is about celebrating the niche, and measuring engagement over time.

Links alone are not a good metric for authority. There are several reasons for this. But the most important, I think is that as consequence for the blogosphere, it harms the way people see blogging. People know some bloggers want influence; many bloggers know they want it too, though many others don't want it at all. Counting links is very much like counting subscriptions to magazines in order to sell ads, as far as comparing it to a number not reflective of what is actually going on with the media it's meant to reflect. Link counts alone are an analog media model, but online media is dynamic, and what is digital often has the possibility of getting much closer to finding smaller, more granular, and more interesting ways of perceiving things, that are much more interesting, and orthogonal to legacy media models counting eyeballs.

I should also be clear that this effort, and the discussions I've been a part of this topic do not have the goal to make another list to replace the Technorati Top 100 or the Pubsub Link Rank 100 or the Feedster list, or any other top (insert #) list. Rather, this is about going beyond lists and links, to understand that the social relationships of expression between and across blogs is really about searching for a "metric for identity" or "metric for affiliation", "metric for community", or "metric for influence".

So a couple of months ago, at a dinner at Les Blogs, a group of us (including Ross Mayfield, Stowe Boyd, Doc Searls and Halley Suitt among others) talked about what it would mean to make an index that could give a clearer sense of a blogger's reach and influence, that might upend the inbound link counts to give some clarity to what is now opaque and hard for us to see blogs we are unfamiliar with but want to find context. Actually, the service was taking a while, and with 30 or so bloggers in the room, eventually things turn to blogging. We started talking about the issue of inbound links and how, counted up and reported as a kind of "attention index," as a show of interest or attention or conversation, they weren't very interesting or telling on their own, partly because they lump together all types of links, no matter when the links were made or where they are from (blogrolls or posts).

Part of what we want is a rich user generated ontology resulting in topic groups that is constantly adjusting to find what's delightful, useful, interesting across blogs. And a more complex metric for understanding those topic groups and individual users as they blog memes and interact with each other, with some context around those bloggers, would help quite a bit.

This issue came up again in force at Blogher where the opening discussion talked about how to play, or not play, the link game. Much of the room was, to one degree or another, very frustrated with using inbound link counts as an expression of attention, and how the derivation of "A list" bloggers comes from that, ignoring the many blogs that are very influential or conversational in their topic areas.

To automate this process, or create a score, is to judge the stone by the ripples in the pond. Right now, the Technorati Top 100 list is obtuse enough that we can all agree that it's not useful for judging 14 million blogs, because blogs are as different as their authors and those who would make a link rank for a person in one's topic community. As for lists of bloggers based on the number of subscribers, like the Feedster Top 100, we know that in this instance, the list is a count only those users of Feedster, so it reflects a small percentage of overall readers.

So I hear people dismiss the current indexes all the time. By doing so, we let the opacity of inbound links counts be a barrier to rankism or scoring that we don't really want to make more precise. The obtuseness is useful because it's can't be relied upon, and therefore the confusion as to the value of a blog is left to be determined by readers through their own methods, by those who look on their own for the ripples across blogs, combined with some reading of the blogs. And this may make many people happy. For me, I would rather have people do their own assessment of my blog because they read it or participate in discussions I am in, seeing what the activity is around it, to judge it, verses relying on a score or count of inbound links.

However, I'm beginning to see many reports prepared by PR people, communications consultants etc. that make assessments of 'influential bloggers' for particular clients. These reports 'score' bloggers by some random number based on something: maybe inbound links or the number of bloglines subscribers or some such single figure called out next to each blog's name. The bloglines measure in particular is not a great one on it's own, because RSS aggregator users are reported to be only approximately 20% of the blog readers, though I believe it's really half that, because my own user studies show that many who are asked if they use an RSS aggregator say yes, when in fact they don't know what it is (they just think they should know, so they answer yes to the question of whether they are users). And of those RSS aggregator users (I think it's 10% of blog readers), and of those, 50% supposedly use Bloglines. But my own assessment of Bloglines is that maybe 60% of their accounts are probably used regularly (not abandoned or very rarely checked), so if they have 35% of the RSS reader market, a Bloglines score might only reflects 3.5% of the total blog reading market -- a very low sample to judge the readership of a blog generally. Using the Bloglines count only counts users of that aggregation service. As a point of comparison, Bloglines shows 20k subscribers of BoingBoing, but Feedburner has 1.2 million subscribers to the BoingBoing feed itself, because they produce the feed, though those counts are only discoverable to the blog owner currently.

And these kinds of counts may or may not reflect the actual readership because users may not necessarily open the feed or posts. On the other hand, I think you do have to weight RSS users a little more heavily because right now, as that user base tends to be early adoptors, influencers, and a market that also tends to be the blog writer set. However, this won't always be the case. And I'm not confident that these PR/Communication agencies understand how to read this kind of information, and while it's one thing to gage the influence of a blogger who writes about their clients by reading the post, it's another to make decisions to send sponsorship or advertising based upon these kinds of measurements.

So the tension is, do we in the blogosphere figure out a more sophisticated, open standard based metric that reflects the way we see blogs, within and across communities, in order to score blogs? And do we do this within topic areas? Or does using a more sophisticated algorithm across all blogs make more sense? Or do we allow this all to be done for us, possibly in an opaque way by some of the blog search engines or by people who are trying to figure out blogger influence and communities for their clients, or do we write off those efforts because we know they cannot possibly understand us anyway?

I have to say, I've resisted this for the past year, even though many people have asked me to work on something like this, because I hate rankism. I think scoring, even a more sophisticated version of it, akin to page-rank, is problematic and takes what is delightful about the blogosphere away, namely the fun of discovering a new writer or media creator on their terms, not others. What I love is that people who read blogs are assessing them over time to see how to take a blogger and their work. But more recently, as I said, I'm seeing these poorly done reports floating around by PR people, communications companies, journalists, advertising entities and others trying to score or weight blogs. And after hearing the degree to which people are upset by the obtuseness of the top counts, and because they do want to monetize their blogs or be included into influencer ranks, I'm at the point where I'd like to consider making something that we agree to, not some secretly held metric that is foisted upon us.

If we are going to do this, I think the algorithm has to be open source, at least as far as the weighting of social gestures and what gestures are to be included. Many people are upset that page rank is secret, and that something so powerful online is not open to scrutiny by the community it ranks. So this is an attempt to have the community determine the social weighting as it goes into algorithm, and have it be transparent to the community.

At the Les Blogs dinner, a group of us made a list of things we might include in this algorithm. This list is an attempt to figure out what things we look at when we're trying to figure out where a blog is at, in terms of interest, conversation and value:

I think a newly made blogroll link now, in the age of 14,000,000+ blogs is far more telling of community and interest, than a blogroll link made five years ago when there were 100,000 blogs (in other words, few choices about where to link). And of course, links made in posts, which are more indicative of conversation or immediate attention about specific topics, are lumped in as well, with the same weight as a blogroll link, for the indexes we have now.

So.. below is a list based on the earliest discussion, but it really needs refinement and input on what is important, how it could be expressed and I'm looking for feedback to help define these issues better to help get the best set of social gestures weighted in the ways we see them across blogs for a community based algorithm.

A new metric could balance links with these other representations of activity (not all are available, but if we want them, we should ask tool builders and data aggregators to get these kinds of information for us). Note that many of these are subject to spam, and spam controls for them are implemented by the companies that track this stuff. Using a metric that incorporates those will require additional spam controls.

So then.. we talked about how important those kinds of information are us as we evaluate a blog or post, and then whether or not there was a number associated with that particular information or a ratio between two sorts of information that might be interesting, and whether it's information we have, and what we might do with it.

A new metric could balance links with the following items in this chart:

We wanted to see these measures used in an algorithm that balanced the weight of each social gesture, put against large data sets to see whether the resulting score or characterization felt right against what we know about blogs as readers and writers. One thing to consider is that some data sets are made up of spidered data (including blogrolls), while others are made up of RSS feed information (some partial and some whole posts, but there are no blogrolls in RSS feeds) and some are a blend. So we would want to adjust the algorithm for different types of data sets.

So this is my first post think about making an open source algorithm. And I'm wondering, is this a useful approach? I think it could be worthwhile, done right, and I put it out there to the blogging community to determine what is best here. As I said, after seeing what people who want to work with smaller topic communities are doing, it may be in blogger's interest to think about how this might be done so that is it more in keeping with the desires and views of the blogosphere.

Blogher: Women who want to fund, build & sell things

I'm leading a session at Blogher called Women who want to fund, build & sell things. I will be there with Denise Howell and Patricia Nakache. We will discuss with the audience questions about support and information they need to fund their own businesses. We want to answer questions about the venture market (what's getting funded, what's not), the venture process (both lifecycle of funding as well as in-the-meeting pointers), funding strategies (e.g., angel vs institutional), and typical investment decision criteria, and see where the discussion leads us. Please join us if you are thinking about starting a business, have started a company and are navigating the funding process or would like to contribute what you've learned in your work. We are looking forward to a rich discussion.

Tag: blogher

July Event: Planetwork FOCUS on DIGITAL IDENTITY TOOLS

Planetwork (I'm dying to say planetwerk...) is doing this event next week on identity systems:

Thursday, July 28th doors at 6, program at 7
CIIS, Namaste Hall,3rd Floor
1453 Mission St. San Francisco (2 blocks from Civic Center BART)

Kaliya Hamlin of Identity Woman has curated this line up that provides a great opportunity to learn more about some of the latest tools for next generation digital identity.

Light Weight Identity - LID
Johannes Ernst http://netmesh.info/jernst
NetMesh Inc. http://lid.netmesh.org/ .
Light-Weight Identity(tm)-- LID(tm)-- a new and very simple digital identity protocol that puts users in control of their own digital identities, without reliance on a centralized party and without approval from an "identity provider".

OpenID
Brad Fitzpatrick http://bradfitz.com/
Six Apart, Ltd. http://www.sixapart.com/
OpenID, a decentralized identity system, but one that's actually decentralized and doesn't entirely crumble if one company turns evil or goes out of business. An OpenID identity is just a URL.

Sun Single Sign On
Pat Patterson http://blogs.sun.com/superpat/
Sun Microsystems http://opensso.dev.java.net/
Sun is announcing the intention to open source web single sign-on. This project, called Open Web Single Sign-On, or OpenSSO, gives developers access to the source code to these basic identity services allows them to focus on innovations that solve more urgent problems, such as securely connecting partner networks, ensuring user privacy, and proving compliance.

Opinity, Inc
Ted Cho http://www.opinity.com
Opinity provides what might be called open reputation for end users. It is a young start up offering free online reputation management related services so that individuals can authenticate, aggregate, and mobilize their website (eBay, Amazon, etc.) reputations. Opinity also offers reputation management tools so that individuals can monitor, build, and work to enhance their own reputation going forward. Individuals can also review other individuals at the Opinity website.
_______
Planetwork has been hosting monthly networking forums in the Bay Area for the last 3 years. We are a unique network sitting at the nexus of technology use for social and environmental good. To support the monthly forums we invite voluntary donations (in a basket on the food table).

If you would like to join our mailing list to get more information about upcoming events please go to this page and get a planetwork i-name http://www.planetwork.net/community/index.html

Did EVERYBODY Get Up on the Wrong Side of the Bed This Past Week?

Okay, so I go to Chicago for a few days last week and don't have much internet access, get a little backed up on my aggregator, and return to a lot of work in CA, only to find out that a bunch of people are really saying some very, very strange things.

It started with Silicon Valley Watcher who reported Peter Hirsberg's remarks at an event on 7/8/05. Peter apparently mentioned that Technorati would sell filtered blog data to companies interested in tracking themselves and their competitors. Of course, anyone can make free watchlists now, but these would be more sophisticated versions of those filters.

And then Jeremy Zawodny asked whether Technorati was going to share any dough with bloggers. Huh?

Um, my understanding is that Yahoo 'sells' search and filtering. And it makes money doing this. On its website, Yahoo search results showing freely available webpages are valuable because of the filter/search service that Yahoo provides, and those search filters allow them, in exchange for this service, to place ads next to their search results, thereby make them money. They don't share the profits with the makers of the matched sites. Or at least, Yahoo has never sent me a check for serving my content. Cause they ARE NOT selling content. They are selling a service for searching and filtering, and that IS salable.

You can't sell content online. If you do as a publisher, your content becomes unlinkable (see the Wall Street Journal for their *tremendous* online reach and participation in the conversation). You can't sell data like blog information because it's free already. You can sell services that help manage data and content, including filtering, search and aggregation. Service providers like Yahoo, Google, Technorati, Feedster, Pubsub, Bloglines, and many more, who offer free online services with less sophisticated search and filtering, ARE NOT selling data. They are selling convenience and management of data. And they don't owe us creators of free blog posts and websites for selling convenience by filtering our data. More sophisticated filtering and search, that is often highly customized, is a charge service. And providers of those services don't owe us a cut of that either. Because they ARE NOT selling data. It's the filtering and search services that matter.

Please read Information Rules again, where they explain why and how YOU CAN'T SELL CONTENT in the digital world, except in highly unusual circumstances but YOU CAN SELL THE MANAGEMENT of information.

Oh, and let me disclose here: I used to work for Technorati, and I am on the advisory board. I've also been critical of Technorati at times, but this time, I think they were the first to push this conversation out into the view of people, and were unfairly singled out for the 'selling of the blogosphere' which is a complete misnomer and totally inaccurate. But the reality is, lots of other companies before them have given away free services online with lower level customization, put ads next to them, and also sold highly customized services for special purposes to individuals. Technorati is no different, and is actually doing the right thing for the company and their ability to provide the free service in the long term. It is often the case in the digital world that free services are supported by the selling of premium services, and in this case paid filtering can support free filtering offered on the Technorati website.

Other folks that wrote about this include: Those linking to SVW and Doc links to a bunch here. Additionally, lots of other posting about the quality of Technorati's service appeared around the same time, which is a totally different issue, but somehow was conflated with the selling of filtering. I understand that folks think Technorati should make a good service before they start selling it, but it seems to me they are working on that, the service online is free, and no one is forced to use it. There are lots of other choices for finding similar data, and everyone is free to go to whatever site they wish. My own view is that I hold multiple search feeds from all the services, because they return different data on the same searches, based on their data models and databases. But that's a different issue as well, and maybe I should do a post on that, explaining who covers what, and how the results compare, as I've monitored all the services for the past 18 months.

Honor Tags

Dan Gillmor and Bayosphere have worked up an interesting tagging system, to differentiate the types of blog posts people are making, if they choose to self-tag their posts. They plan to pull those tagged items into their site to reflect back activities in certain categories. This has some advantages but also presents some problems, though I think there are community solutions that can moderate the problems.

Benefits of Honor Tags include:
- more fine grained searching, based on the ability to pivot on a couple of tags... where you can find the intersection of a type of post and a topic to see just those that have both kinds of tags
- community affiliation, closeness and participation due to special tag understanding and use
- people will declare their intentions
- there is potential for advertising based on these tags or aggregated groups of posts with tags
- we might find there is better tracking of reputations than we did before
- it might allow users of the tags to have a better shot at legal protection for self proclaimed journalism makers, as they make a kind of journalism
- users of these Honor tags can self-tag one post at a time, so one tag can be about one thing, and another can be another.. it's on a post-by-post basis that we understand these designations instead of by blog or by blogger (in other words, it's not self-tagging as a journalist, but rather as self-tagging a journalism-categorized post).

Problems that might creep up with this tag system:
- people are often either not honest about themselves, or simply don't classify themselves well because they are not very self-aware, or understand the definitions of the classifications differently, so they may state something different that what the community perceives them to be
- some outside the Bayosphere community may feel it's elitist
- as the benefits/results of self-tagging one kind of post become more visible either in search results or other aggregation pages, some people will game the system, especially if there are advertising dollars at stake
- people will game reputations of themselves and others for variety of reasons
- people will game the links to be included in search or display systems in malicious ways just because they can

I think the community should moderate the use of these tags, to solve some of the problems that may arise from Honor Tag use. Some things that might help the community do this include:
- Users could make tags that are about self describing an action, at one point in time, in one post (so it's 'journalism' for a post, not 'journalist' as one's status)
- Users of the tags should make it clear that this Honor Tag system is for a particular community, and specifically for certain acts, not defining people, but that anyone can use the tags and is welcome to participate in this community through their blogs and use of these tags
- Users and the community as a whole could help make it clear that people can use any and all tags on a post by post basis.. meaning.. one post is journalism, and another is advocacy, depending on what's in the post
- Users and Bayosphere together could create some community moderation for the tag use so that if the community sees a bad actor, they can report it, and if there is a dispute, allow the community to decide what to do about it, and even how to handle it.

I'm very interested in seeing how people use these tags, and what the results are on Bayosphere aggregation pages (which I hear are coming soon) and through services like Technorati. One thing I'm already noticing is that I'm having trouble deciding which tag or tags to use for this post. I want to use one of the tags, but is this advocacy? or reporting? or personal? It's kind of all three, including the personal since right at this point in the post, I'm discussing my own tagging and classification issue. Humans are messy and we have trouble saying what we are, and sticking to one thing at a time.

I'm also really enjoying the creative ways people are coming up with to tag things, and I hope that Bayosphere and Honor Tags will keep tinkering with the classifications, tag structure and the UI and information meaning of the aggregation pages that collect the tags. These systems and tagging generally are very early stage and need a lot of work, but I'd definitely encourage people to try out Honor Tags and see what happens as their posts get pulled into other sites. I'm sure that the community around Bayosphere will have lots of feedback as they play with tagging. I think it's fantastic that Dan & crew are taking the plunge on this to try to figure out something interesting.

Good Luck!

HonorTagJournalist
HonorTagAdvocate
HonorTagPersonal

Mobile Tracking

Nathan Eagle, of MIT, talked at Where 2.0 today, about social activity and tracking. Below is the first part, explaining the Reality Mining, his project that used 100 Nokia phones, tracking these phone users, to see who they know and what they did together or apart, and where, on the video I shot below. Also, he noted that with this study, he could empirically show that a week has 7 days as part of his phd work. It's nice to know that in world where it feels like phds mostly think about commercial opportunities as they pursue their doctorate, or if they pursue teaching, that they still are thinking about how many contracts and commercial grants they can bring into the academic institution the work for, there are still one or two who can joke about non-commercial academic pursuits while still doing something useful. I know this isn't the fault of academics, but rather market forces on academia, but still, I liked him a lot for mentioning this.

The video is the first couple minutes of his talk..

Grokster Loses, and so do we on that and broadband

I'm sure you all know this by now, but Grokster has lost it's latest.. the ruling says: "P2P software manufacturers can be held liable for the infringing activities of people who use their software."

Ernie Miller has excellent information, including info on the press conferences with the RIAA and MPAA today as well as analysis of the case, or look at the CNN info on grokster.

Also, the Supreme Court overturned a decision that required cable operators to lease their lines to competitors... meaning that now they get to provide the physical cable connection *and* the internet access as a monopoly. According to the article, the FCC considers this a victory, along with large cable companies, and the losers are small ISPs, consumers and local governments.

So much for our freedom to connect!

Welcome to The Vlogosphere

It is different here. I'm dipping in my toe, have been since December.. and really, I've only made one vlog post myself. I learn both from watching others and doing this myself. The read/write nature of video is very very different than text or the genre of blogging.

What's interesting to me is how I'm now discovering the vlogosphere as I once did with the blogosphere about four years ago.. back when there were maybe 100k blogs... I had no idea what I was looking at because it was all mysterious then: the format, the linking, blogrolls, and the people, online trust and references. There were nuggets of magic, people who came through asynchronously to share and converse both information and points of view that were personal, passionate, deeply held and often far more expert and full of breadth than legacy media. I was taken, I knew there was something there.. but I couldn't figure it out until I started blogging at bIPlog and realized the linking was creating many trails of conversation; it was writers following those links, extending the conversation still further, that was making something totally new and exciting and relevant. Yes there were and are diarists, essayists, as well as others who put out bad information, and so I'm speaking here of those who blog about topics in a conversational way only. A blog is a tool as we've said a million times.. so let's not go back to that old skirmish. The point is, there are some kinds of blogs that create a conversation in blogging, through discussion and links and comments and still more posts, that are compelling, and give free speech a big push over the old analog world. Fast forward through four years of arguing the stupidity of blogging verses journalism because we don't need to go through that again either. We know they are complimentary and different, and need each other to survive.

But now.. vlogging as a low-transaction cost production medium, with reasonable bandwidth and storage costs, and vloggers with time and interest are creating a new kind of story telling that is very different than the text blog entries I can search, skim and remix aggregated by various services like Technorati, Feedster, Pubsub and Blogpulse. Vlog-posts are little movies, or a post wrapped around a little movie. One cannot link from within a movie, but one can reference, remix, explore. I know at last count there was a directory of vloggers that listed about 200 of them. So it's small now, but considering the power of video and the time it takes to make vlog-posts.. it's a pretty good start. I also thing there are probably many more folks online making video.. that aren't included there.

The ways we determine conversation in vlogs will be more along the lines of visual and aural references. Even if we had a transcript to search them, we would not get context or what is shown visually or in the sound beyond the words, nor would we get the references from one piece to the next, as we can now mouse over a text blog's links to see intended references. Vlog references must be viewed in order to see them. So conversation in media, just like in the analog world, for now, will not be tracked by counting hypertext links or key words. It will be different, and I wonder how we will show those vlog-posts conversing or remixing media in meaningful ways.

As I discover vloggers, get to know their work, see what they are thinking about as they explore and forge ahead with their vlogging work, I find myself presented with similar sensations of discovery and mystery as I did when I first was discovering blogs. And yet, because it is a video medium, the experience is different, I'm making the references between their pieces and the referenced subjects in my mind, I'm taken into a story that is not skimmable but rather gives me sound and visual narrative as a complete picture, where I see clips that may quote from others, but are no different in presentations from any of the other clips that may not be quoted. This kind of recognition was something I did in my early days of blog reading, making connections. But it was easy for all those aggregation services to make the connections for me, as they counted up links and made searchable key words in the texts. But who will be the Technorati or Pubsub of vlogging? What will we do with this medium to transform it from an industrial art that cannot be recognized computationally except by humans?

It's a whole other kind of media literacy, of understanding digital sharing of borrowed work, of seeing what remix and re-expression is about. This is true both for us, as viewers and makers of video, and for the computers we want to aid us in searching and discovering video and video conversation.

I also wonder, will broadcast and narrative legacy video producers claim that vloggers aren't 'real' in the same ways journalists have about bloggers? Or will we have learned enough to get past that to the much more interesting question of where the relationships between the top down and bottom up content with lie and how they might get on .. whether and when it will be complementary or contradictory?

The Real Killer App

It's not some new web service for video or whatever. No.. the real killer app is transmitting smell and taste over a web page. Man, if last night's dinner came through over my firefox browser, every chain restaurant would be out of biz. Kiss Olive Garden good bye. It was five hours of delicious beautiful food at Union Square Cafe.

Could someone please start working on this? Because that meal was a rare and perfect experience... and you just want to have it a little more often.

(One more small thing, much as that was lovely, I've got to get out of NY. People here don't work as hard as people in CA I know, and they party way too much. Being in NY is actually kind of like visiting a European city, sophisticated, fun, quaint, but you've gotta get some work done and they're not motivating to be around, with all that langorous strolling and dining and hanging out. Aren't NYers supposed to walk fast? Everyone is walking so slowly down the street. Time to get back to CA and the real work!)

How do you build systems that are not paternalistic but still protect consumers.. but give users control

That's the question on the table at the panel Marc Canter is moderating, with Bob Wyman, Matt McAlister and George Kane.

Bob Wyman thinks we shouldn't collect Attention XML data until we have good privacy methods in place, Marc doesn't care if people know what he reads, and I asked what to build to answer my question:

How do you build systems that are not paternalistic but still protect consumers.. but give users control?

They didn't have a clear answer. But I think it's an important question for all of us to address.

Comments on Bandwidth Post

Sorry.. will get comments fixed asap now that I'm back.

In the meantime, here are couple from email:

Jeff Clavier:

What puts me off even more is that my parents, who live in Tours (the average size town from central France I was born in) can get 15Mb down for 29 EUR ($40 - what I pay for 1.5Mb in Palo Alto).

15Mb!

And I have just seen a promotion in the subway (from Cegetel): 20Mb for 15 EUR. The fine print says that the speed actually varies from 512Kb to 20Mb depending on one's location (DSL's constraints) but we're talking roughly 1$/Mb people.

Esme Vos:

How about this in Hongkong? 1 Gbps symmetrical for $216/mo from the Hongkong Broadband Network. HKBN's entry level broadband service is 10 Mbps symmetrical for $16/mo and its mass market service is symmetric 100 Mbps for $34 per month. I just posted this on Muniwireless.

And note Esme's excellent post: US Increases Its Lead In Dialup Internet Service. Now we're talking trashy, but good!

100mb down and 20mb up for 26 euros (approx) in S. Korea

Yes. You heard that right. Yat Siu of Outblaze in China is comparing internet access in Asia compared to France (France averages 2mb up and down for 14 euros).

We are so totally freaking lame in the US. That's half the price I pay (approx $35 dollars for 100mb down/20 mb up in Asia verses my DSL from SBC for $60 a month for 768k down/ 384 up in Berkeley). And of course, we, the US, aren't even on chart up on the conference wall. And why should we be. As far as broadband access goes, we're a third world country paying ghetto rates (you know the deal.. where the supermarket charges less than ghetto corner markets cause the folks in the ghetto can't afford to drive out to get reasonably priced food).

Update: there has been a big discussion by some who attended the conference, who also blogged this information and were later challenged on it. Subsequently, Yat Sui, whose talk addressed a comparison between broadband service in Hong Kong to that of Paris, sent this article on 100mb DSL service for Korea. An English article mentions the goal to make this service available all over in Korea.

Last year, the Korean government forked out more than $2 billion of the $10 billion needed to build the world's fastest integrated network. This "broadband convergence network" (BcN) will provide connection speeds of between 50 to 100 megabits per second by the end of next year. The fastest connection speed in the U.S. is just 3 megabits right now. "BcN will be a core platform for the creation of an advanced communications market," says Minister Chin.

And lastly, Yat sends this:

I know of quite a number of people who have 100Mbps service, whether they GET a full 100Mbps is a different question alltogether I suppose. Korean Broadband providers are not thinking they will compete on price and size alone so they are leading the charge in value added content services (like digital TV). Most don't compete like HKBN, they call it "Super High Speed Mega Ultra whatever" and place the xDSL or vDSL "speeds". In many cases the bandwidth increases are being treated almost as "unlimited" for the pro super speeds which are geared towards online gamers. Super high speed is also a requirement for the Cyber Cafe's which are the social center of many of these online communities. Fiber is also widely connected across Seoul and even "smaller" operators like Dreamline are claiming network capacity in the 800Gpbs service range with 1.6Tbps coming soon.

Search

I'm at a search event at Microsoft.. of the 32 or so people.. there are about six search engine optimization people here. Kind of like being at the FBI and having criminals helping out in the room. We aren't really supposed to blog about what happens here.. but the event and attendee list are now out and known online. So far it's interesting and there are many smart people here.

The Church is not a conversation.� -- Peter Hirshberg

...from his blogpost: The Pope, The Word, and The Blogosphere where he discusses the whether the Pope might blog, and how the Vatican uses or doesn't use new media, as they attempt to mesh their style and goals with the distributed network online.

The score card, according to Peter:

Blogging: no.

Podcasting: yes.

Taking comments from the great unwashed: no.

Hearing from God and pronouncing: yes.

Vatican and cluetrain: no.

Broadcasting eternal truth to the masses: yes.

You get the idea.. basically they are into one way communication.. and to the extent that online distributed media can help with that, they'll use it, and otherwise likely opt out of other aspects.

Very funny post Peter!

Extreme Usability

A month ago I participated in the Open Source Usability Sprint.

For me it was revelatory. Something I started doing last August and September with Tantek Celik, at Technorati (I used to work there). We would sit, side-by-side, working on the usability of the site, where we picked through about 50 little niggling problems that I'd found over the previous 9 months (yes, I'd found more.. but fixing 50 was great) to make those little problems go away. We started out deciding just to fix a couple of things, so I took him through the user's perspective about why something might be broken from their perspective, and it was boring, tedious, time-consuming to do this.. and yet.. immediately as we refreshed the changes, we would both see the improvement and understand how users would like the change. So we fixed another and another.

These were problems that I knew about, had documented, or had found in several rounds of user testing. I did what is common in usability, documenting these issues. The engineers would read the reports, comment on them in conversation, quote the user's, and generally agree. But then, nothing would change. And that's not to say that these engineers at Technorati, or the ones I've worked with elsewhere, weren't brilliant or personable, or desiring of good usability and user satisfaction. They are.

But the reality is, written reports, while read and interesting to engineers, are hard to translate into change. But this extreme usability (we didn't call it that then) actually worked (though I left Technorati just after so it didn't continue there that long).

One thing to note is that this sort of extreme usability takes different forms depending on the stage of engineering development. I have worked with it at early state needs assessment and prototyping, in the form of rapid iteration during development of working software and web services, and even well after a site or software has been in use. What is important to know though, is that all the usability work that would normally be done, whether needs assessment, user profiling, interviews of one sort or another, or led discussion focus groups, has to be done on the usability side, and reporting and documentation is still necessary. But after that, extreme usability or pair programming with engineers is very effective, just as engineers will spend time coding and developing before they get to the pair programming session with a usability person.

This fall, I worked with several engineers, and I just insisted we sit down, and pick through the problems side-by-side. Wow, they said. This is fantastic, we are making really good progress and users are responding quickly and favorably to the changes!

By the time we got to the Usability Sprint in February, where we did this for three days, and named it as extreme usability, I have become fully convinced that this style of usability and engineering partnership is really key to the next generation of interface and information architecture development. It's pair programming. And I highly recommend it.

More Interesting Stuff This Week

Still catching up. Got no sleep Friday night, and ended up with a bad cold. In bed working.. but hopefully I'll make my meeting this afternoon. Oh and did I mention, a snow storm is rolling into NYC .. supposed to be slow moving, and so the airlines are reporting on their websites that flights may not go as planned today or tomorrow. Yeah. Did this delay thing out of here last month and now it appears I'm doing it all again. So it's snowing out the window.. lovely .. it reminded me of more things I'd meant to blog the last few days:

A podcast on the napsterization of TV (12.47 mb mp3, from Webtalk radio). One interesting point is that when the Supernova site was shut down a few months ago, it was over the distribution of movies and music, but the prosecutors didn't touch the TV aspects because of the perception that TV is free anyway and they didn't want to get into that argument. It was just easier to deal with the obvious movie and music copy-written content being distributed. They go off into podcasting about 20 minutes in.. or so.. so the title is a bit of a misnomer for the last 2/3.

Also, Adam Penenberg wrote last Thursday about the lack of attention the Wall Street Journal gets online.. because nobody can link to them. Adam and I talked about this a few months ago.. when I was at Technorati and he interviewed me for an article in August about the service. I mentioned that while the NY Times has tons of links, and is one of the most "authoritative" sources online, the WSJ is non-existent.. as far as linking and discussion attention go from bloggers, because they are a walled garden. I've blogged about it for a long time.

Adam takes an interesting view.. not about linking, though he does quote JD about the WSJ's lack of linkability, but rather the effects of this. Adam says that people are not finding the WSJ in google searches, or hearing it talked about, and so the WSJ is in danger of becoming irrelevant. And this may not be very reversible, if things continue as they are, because the WSJ.com biz model is based on the walled garden/paid subscription model. Their competitors like Forbes are free online, sans registration even, and therefore, it's allowed Forbes to get pretty entrenched as the source for online business news.

FN Server: Blog Posts Hold Image Wikis, With Image Annotation, Plus Text Posts

Note: cross-posted to FN Server.

Greg Elin has been working on this experimental project, FN Server. It is an extention, a rif, on FotoNotes which he's been working on for years. That annotation tool allows for notes to be made within a jpg image. The new FN Server project goes much further to make (in what is right now, a single site of posts) something where users can post images and text, into little blocks, each functioning similarly to a blog post as a single, linkable conversation.

Inside each block .. it's whatever you want: images, text, annotations .. nested within one another. And changeable by participants. For example, within a block is a wiki like variant for an image, annotatable by anyone. Or there is a text with links function like a more typical blog post, and not editable, though it would be fun to have comments. Tags can be used for either type of block, and then searched by others. There is a daily view, a view by user, by tag, and at some point, maybe individual or group pages because as more users come, there will be the desire to break down from the whole even further.

Most importantly, to really see what is there, you must use it, because the navigation and functionality can't be seen from one single vantage point, but rather, must be traversed. You must wander around, see things from different angles, find the trail to putting objects and notes together, making conversation and sharing information.

I love the navigation box... located within a single block in the upper left grid of blocks. It's simple.. and elegant, unlike the x-y axis navigation systems often used by webservice sites where the top and left and/or right navigation bars are cluttered with a million links to features, views and skews, so heavy and overwhelming, and giving no sense of the function or use of the tool because they attempt to textually link everything at once in a purely hierarchical metaphor. This nav box is lovely: it is represented in what it is also navigating, and is almost on the same level as the other blocks except that it's always first and therefore doesn't scroll off the screen with older entries. It gives just enough of a hint of each function, so that each choice brings me to a result I expect, or if unknown, leaves me with something I can make sense of and relate back to the choice I made.

As to what content is there in the posts now, I've been on group blog projects before, where we agreed as a group to certain topical or other editorial standards, or where by invitation from one person, the blog topic or focus was directed and owned. But this little experiment the last month has been fun, because there is no director or topic focus for this group. Each of us has posted on various topics, mostly through images, with annotations but with some text posts as well. It is random in a way, but personal, engaging and playful.

Update: now FN Server is on source forge... as an open source project!! We love that at napsterization.

Dumpster Diving OPML Feeds

Susan Mernit just told me about how she was diving for feeds in other people's OPML files, in Bloglines, and came across two sets she really likes, so she copied a few feeds, but then realized that a lot of the feeds seemed similar to her own. Then she looked at who was dumpster diving her feeds, and those two were on the list. One turned out to be a Russian programmer, and the other was from Brazil. Small world.

Kinda reminds me of Napster's old playlists, where you would find a couple cool songs in a speedy download, look over the rest of someone's collection and then pull down a few more... only to find back on your current transfers page that they had done the same to you.. and were currently downloading things. This of course, was back before Napster was outlawed.

Google Groups Beta, Group Lists Social Courtesy, or What We Really Don't Want

Over the last few days, I've been getting email from a Google Groups list I have been subscribed on... by someone unknown to me, that I did not agree to or ask to be included in. Dan Gillmor responded to one message yesterday after several came through, asking to be removed, because he didn't know how he got on it, or why, and simply expressed: "geez" in response to the situation. In fact, the group list appears to have started last May, and the first email I have is from August. As of yesterday, when I went to Google Groups to figure out what was going on, I found there were 28 people on the list, though unlike Yahoo groups, once I made a login, I could not find an administrator or a list of participants.

I decided to reply to this Google Groups email list, and the email text I sent is below, explaining what the problems were that I found, stating that I thought it was strange to be put into something without being asked, especially something that, by replying to it, would lead to the public recording of my email address and text, without my agreement or permission in a Usenet list. Not that I care that much because I do blog after all.. but still. This is weird, and not very sociable or considerate. Seems in fact like an excellent way to annoy a bunch of folks.

Another problem is that Google Groups does not explicitly link on the GG page to the support group or to a privacy policy, though apparently there are three ways to get some support.. by emailing support (labs-groups2 @ google.com) -- not intuitive by any stretch, or using the form here: http://groups-beta.google.com/support/bin/request.py or posting a message here: http://groups-beta.google.com/group/google-labs-groups2. None of these three options are linked on the GG pages, nor are they listed at the bottom of the email sent through the groups, but one user replied to every email sent yesterday restating the same three possibilities numerous mind-numbing times to the entire group.

I would suggest that Google Groups limit participation in groups by requiring a confirmation email-reply from list subscribers, and that it add an appropriate links to the bottom of group email lists.. explaining the privacy implications, the public nature of the group mail, and the support email address and de-subscribe information. All this would seem like pretty basic best practices for any group email tools.

Since I sent that reply email back to this group, I've received several email or otherwise heard from people like danah boyd and Chris Alden telling me they are on the list as well, but they don't know how they were put on it, or how to get off, because there is no unsubscribe link. And I've heard from a number of others.. who are not on the list, but who 'heard' about the email sent from Dan, and me.

This is the original letter I sent out yesterday to the list, after Dan's email, for those who want to see it (it's redundant to the above post):

I have no idea how I ended up on this list either. I've never signed up for Google Groups, until today.. when I coincidentally happened to get a whole lot of email via this group (which I did not subscribe to) and happened to be doing an analysis of group ware tools.. including GG and Yahoo Groups, just now. Interesting introduction to GG.

Looking at the "about groups" section (http://groups-beta.google.com/group/Testing-Google-Groups/about), there have been 19 email since May, when it started (I went back to see that I appear to have been added in August) and there are 28 members. However, once you join GG, the system will not allow you to see who the other members are, unlike Yahoo Groups, which does allow members to see other members and the administrator of the group. So we can't really tell who subscribed us or started this... other than to see the first two messages... by "mer...@gmail.com" and "krucoff" (see this link to find more about the group email for May: http://groups-beta.google.com/group/Testing-Google-Groups/browse_frm/ month/2004-05).

A hint, for people starting groups: don't subscribe people and their email without asking them first. If they don't agree, don't put them into it. It's rude. It's spam unless we agree. Especially since it's publicly searchable as a Usenet group.

Also, if anyone from Google is on this, your system has a serious problem. People can be added to groups without their being asked, or warned, and the email is received from the list with no remove link at bottom, or link to the particular Google Group they have been unwittingly subscribed to, in order to see what is going on to begin with. And for what I'm sure isn't the last problem, the whole thing is viewable live on the web published as a Usenet group. And for those who didn't voluntarily sign up, their email address and content (if they participate in the group) is available whether they agreed to or were aware of this "feature" or not. If they are used to Yahoo Groups, they may well believe, even if they know it's a group email list, that it's private, and you should warn people about this difference.

I know it's in beta, as is every project developed in the last few years, but could Google include support and a privacy policy? If the main Google privacy policy and support apply to Groups, how bout linking directly from Groups to make that clear?

Phil Wolfe Comment

From the previous post.

Just a note about the use case.

Mary's power use may be leading edge, but it is well within scope. If you spend hours of each workday online or at your laptop, then any RSS client becomes integral. It replaces bookmarks. It replaces addressbook entries. It replaces listerv archives. It becomes the place you look for your own blog posts, a backup brain. To pick up threads of conversation. To detect trends and be a little smarter. The place to find contact info, places to go, things to do.

So it's indispensable.

But it must scale too.

About the assumptions: disk is cheap (comparatively) so there's no cost to adding feeds and saving posts. Six months after I started adding feeds, I had 500. Six months later, more than a thousand. Ask Scoble how many he has these days. With time, and exploding feed availability, it will be nothing to pick up 20-100 a day. That's thousands a year, hundreds of thousands of posts before you know it. All reflecting your social and reading and writing behavior, your interests and values.

Quite an asset, unique in all the world. And personal. (Something you might be upset about losing.)

So don't scold Mary for pushing the envelope and holding the lovely developers accountable. It's in all of our interests. Because someday sooner than later, we'll all be in her shoes.

Thanks Phil.

UPDATE:
Note these follow ups to this post, that though unrelated to the post topic, didn't appear in trackbacks:
Brad DeLong
Frank Field

to this trackback: Backup Brain

Wow, So Fun! My Very Own Troll!

I have my very own troll! IP address: 141.155.153.110. So they've been posing as me and others leaving fake comments. I've deleted their comments because I don't want other's misled and the idea is to have a constructive discussion to get to something more useful about the post topics.

But how cool.. some person with so much time on their hands that they can sit around leaving fake comments on my blog. Love it! It's like having my own grinch. Hopefully they will come up with something that's entertaining instead of those sort of mundane posts from earlier. I'm just imagining them sitting at home, spending hours scheming up really creative pithy comments and then posting to try to fool us. Come on, try harder! Really put some time and effort into it and you can take that show on the road!

The Definition of Beta

Several people who've visited this site the last few days have defined 'beta' in the traditional way, as something distinctly unfinished, full of bugs and therefore to be regarded as something potentially dangerous, while others see it on everything and think it's lost it's meaning. Last night a developer friend called, and I mentioned the posts and the term 'beta' and asked for clarification and opinion.

Since everything is now marked 'beta', the term has lost it's meaning to users, according to many commenters as well as most of the folks I've talked with, including several developers. As this software developer said to me last night on the phone, it's like having Jordashe slapped on the back of your jeans in 1978 at Studio 54.. it's trendy, it's hip, and everyone does it to be cool. But do users pays attention? According to this person, it's just marketing, fashion, and a way to not commit... and since so many have issues committing to things in our society, users just sort of think developers are having trouble committing to finishing.

Google puts it on everything but search, so does Flickr, so do many software sellers, as Brent did at Ranchero. It's clear now that he obviously meant it in one strictly definied way, but since there is so much variety of definition and use by other developers, how should users take this overall? Isn't it up to developers to address a common definition so as not to confuse things.

Until then, blaming users for misunderstanding the definition of 'beta' seems unrealistic, and while developers can make users 'bad' for doing this, developers actually are the leaders here. They have the control and power to name and define. Developers have confused this issue, and users should ask them to get this aspect of software clear in the marketplace. Blame Google, et al, for causing problems like this. For now, users have been trained by the aggregate of developers to think 'beta' doesn't really mean anything.

Charging users for 'beta' reinforces the loose definition of 'beta', because we assume the software is for real. When I purchased NetNewsWire, the Ranchero site let me buy it, and didn't say anything about reporting bugs or that it was buggy on the payment page, was to be strictly, traditionally defined as a 'beta' or that there would be a loss of data, which they were aware was a distinct possiblity upon putting in the license code. I'm sorry that NetNewsWire continues to be the example here, because lots of other developers have done the same thing, but my experience was with NNW and so it's the concrete example I have to work with for this write up.

Not charging for beta means users will have a significantly lower chance of confusing the definition of 'beta' as it's traditionally held by developers.

User Developer Relations: What are The Social Norms

So yesterday's post is still raging down there in comments. I've been thinking about the larger issues, beyond my own loss of data on NetNewsWire. I got very upset about the loss of my data and posted a rant that was too harsh. But I would say that this is what a lot of users feel like when they run into system or software failure, especially if what the system lost is very important to them for some reason. I also realize that developers work very hard to structure and code their systems, and I respect that.

But there is something in-between, that is important to address. There are interface and translation between user and developer... and often these are given short shrift by developers, who appear to users to have all the control in making the software. Users feel that instead of starting user needs assessment and testing well before they begin to write the code, and doing iterative testing through the development process, developers often slap the interface on at the end of development, after deciding how to meet user's goals without actually talking with users, and done little or no user testing. For users, it often feels like the system was designed with only engineer's in mind.

There is also the issue of what the contract is when software is paid for... users often feel that paying for software or services means it should work, regardless of what the stated status of the software is. Developers often feel that their work is laborious, that users aren't appreciative or getting what is intended by the software, and that users should take more responsibility for backups and failures.

I want to know what people believe the social norms are for this new kind of software, that collects or enables social, multidirectional data, creations or communications. Not what the law says, and not what the licensing agreement says, but rather, what are the social norms and expectations each side has? How should the community understand these transactions and sales, and can we agree on some community norms for the sale of services and softwares? Or is this simply too much and therefore will the conflict continue unresolved? These questions of responsibility are not posed with money in mind, but rather, a social responsibility toward each other to either behave independently or interact in a certain way.

The discussion that has come up in the previous post on NetNewsWire is interesting. About 75% of respondents feel that users who pay for software have no recourse, are to blame for the failure, and because of their hostility toward the expression of the complaint, appear to feel that users shouldn't even complain publicly (or on blogs) about purchased software that malfunctions.

Regarding my specific case, I think it's likely that others have gotten software from parties or places other than the website selling the software and then paid for it ... and I think this will continue. People who develop things like it when their stuff spreads virally... and if that viralness takes the form of person-to-person sharing of demos, what is the responsibility of a software seller, who is just selling a license to someone, without the benefit of that person seeing the site, to users in order to notify them of known bugs?

Is a developer selling software responsible for having a Help section, or a bug reporting system (in NetNewsWire's case, their product area refers to a Help section, but 'Help' is not hyperlinked where it is described in the middle of this page, and a Help section is not available anywhere else. Apparently, they have bug tracking for users, but the link is not on this product page, though a commenter on the previous post listed it so it apparently does exist. However, how would users know how to find NNW-bugs)?

Are developers who