March 17, 2005
A month ago I participated in the Open Source Usability Sprint.
For me it was revelatory. Something I started doing last August and September with Tantek Celik, at Technorati (I used to work there). We would sit, side-by-side, working on the usability of the site, where we picked through about 50 little niggling problems that I'd found over the previous 9 months (yes, I'd found more.. but fixing 50 was great) to make those little problems go away. We started out deciding just to fix a couple of things, so I took him through the user's perspective about why something might be broken from their perspective, and it was boring, tedious, time-consuming to do this.. and yet.. immediately as we refreshed the changes, we would both see the improvement and understand how users would like the change. So we fixed another and another.
These were problems that I knew about, had documented, or had found in several rounds of user testing. I did what is common in usability, documenting these issues. The engineers would read the reports, comment on them in conversation, quote the user's, and generally agree. But then, nothing would change. And that's not to say that these engineers at Technorati, or the ones I've worked with elsewhere, weren't brilliant or personable, or desiring of good usability and user satisfaction. They are.
But the reality is, written reports, while read and interesting to engineers, are hard to translate into change. But this extreme usability (we didn't call it that then) actually worked (though I left Technorati just after so it didn't continue there that long).
One thing to note is that this sort of extreme usability takes different forms depending on the stage of engineering development. I have worked with it at early state needs assessment and prototyping, in the form of rapid iteration during development of working software and web services, and even well after a site or software has been in use. What is important to know though, is that all the usability work that would normally be done, whether needs assessment, user profiling, interviews of one sort or another, or led discussion focus groups, has to be done on the usability side, and reporting and documentation is still necessary. But after that, extreme usability or pair programming with engineers is very effective, just as engineers will spend time coding and developing before they get to the pair programming session with a usability person.
This fall, I worked with several engineers, and I just insisted we sit down, and pick through the problems side-by-side. Wow, they said. This is fantastic, we are making really good progress and users are responding quickly and favorably to the changes!
By the time we got to the Usability Sprint in February, where we did this for three days, and named it as extreme usability, I have become fully convinced that this style of usability and engineering partnership is really key to the next generation of interface and information architecture development. It's pair programming. And I highly recommend it.
Posted by Mary Hodder at March 17, 2005 08:46 AM
Hi Mary, great story. Did the engineers ever stop to ask for a larger framework of what usability meant. Or were they happy to just fix things that had been pointed out was wrong.
So, yes, engineers in my experience always want to know how things were done in the usability process.. they want to know the context of the study. But often this would be well documented in the reports they'd read, and we'd already talked about it in detail.. before getting to the pair programming work. However, during each little fix, I always explained why users interacted the ways they did about the issue at hand, and then often the larger framework of usability would come up. We would talk about the usability issues and fixes would be put into context.
I'm all for sharing this kind of information if people are receptive, or very quickly.. the issues we've fixed reappear. To give an example... at eTech this year, there is a fantastic attention stream on a plasma screen in the lobby (blogged two posts back) fed with Technorati, del.icio.us and flickr data streams as well as a little blue tooth data from cell phones that come close to it. So some of the fixes Tantek and I made last August and September have reappeared in the feed, because someone else besides Tantek made the feed. It's a simple mistake, and not a big deal.. but lots of people the last three days have asked me over and over what one or two things in the Technorati feed are, because they don't understand what they are. This is a pretty bloggy, wired crowd, so if they don't know what some of this stuff is, it's a problem. It's not wrong, it's just that it would be better if the problems were fixed, so that people could concentrate solely on the information being conveyed and not on the parts they don't get.
So yes, when fixing things with an engineer as my pair programming colleague, it's very important to discuss both the larger framework of usability, as well the specific rationale of users over a specific problem. If nothing else, it has a preventative effect, and my hope is that as they code further projects, they think more about the user perspective in other places.
Yes!!! Good work.
Now, for the next step. Somebody should collect (or have routed) and review all the usability (and minor functionality) ideas from users, work them over herself, pick the ones with the highest bang/buck, and then carry them to the development team in the same way that you did.
Frankly, as a user, I often send significant usability ideas to product feedback channels. But it always feels like dumping energy down a black hole. I know my ideas are good ones -- I try really hard to do high bang/buck -- and I bet a lot of others from users are too. But if the internal ones usually get ignored, you can imagine what happens those that arrive in random user e-mails.
Of course, some variation on what you suggest has been happening for some time including things like bugzilla being installed opening for users to report bugs (or worse, ideas -- it's really miserable for that).. and certainly users have been submitting ideas since the first system. But there are a lot of reasons why those ideas don't get used or at least considered. The worst is the lack of meaningful way to let users know that their ideas have been heard, so in effect there is the black hole effect from the user perspective, whether or not user ideas really are brought into the engineering process.
The other issue is that even with user idea consideration, many proposed solutions that are technically not possible or address too few people's needs for the cost, or they are solutions that won't really solve users real goals. One thing usability work is about is getting to the goals, the needs, of users and then translating that with engineers into solutions that make both usability and technical sense. So we need to figure out better ways to let users know their ideas are on the table, but maybe are being put through a needs assessment and technical feasibilty ranking process.
This brings up another point. When I work on projects with engineers, just as I pass usability info to them, I want to know as much about the technology as they are willing to give me, because I'll do better work as I learn more about this. So this is very much a two way street.. and in the post above, I don't mean to imply that information is only going in one direction, or that engineers are the only ones getting informed. I love learning about what the technical decisions are and what matters for the code, the site, performance, etc. I very much want as much from them as they get from me. It makes my work going forward so much better.
One of the things we did was to make a page available containing the following progress bars & graphs
1) Horizontal : Absolute number of RFUIs (Requests for Usability Improvements) and percentage implemented so far.
2) Vertical : Ongoing subjective user rating for usability with average and standard deviation.
That way the users got some feedback and the programmers knew they were being watched. Couple that with a pecuniary rewards scheme and you're set to go :-)