Christopher Fahey, one of the founders of a great design firm in NY, Behavior, has started a great series that takes a long hard look into the value of user research in web design.
He asks some tough questions and makes some very valid points that just might get quite a few people thinking differently about user research. For our part, we think that getting in front of your users is very important, be we’re not all that keen on formal usability (research or otherwise), for many of the same reasons Chris mentions here.
Empathy and Experience
Of course it depends on the project and what your looking to do, but when it comes to design “being expert designers who draw on deep experience and good instincts” is what it boils down to. I would assert that in order to get that deep experience and good instinct you need to spend time with people who use your design.
I’ve watched hundreds of people using web sites and I’d be lying to you if I said it didn’t help me as a designer. In fact, every time I talk to someone about their experiences on the web and/or sit with them using the web I think I learn something. If nothing else I build the necessary empathy needed to be a great designer.
But do you need much more than that? Usually the answer is no. So is there any value beyond that with user research?
Research for consensus and sign-off
Most user research has its value not in informing design specifically (again, unless it’s helping a designer build empathy and knowledge) but rather in things like building consensus among stakeholders, validation of design decisions, etc. We look to more formal research, in many cases, to back up what we already know or to support our own thinking to gain buy-off.
Hopefully our clients trust us to know the field we’ve all been working in for a long time and to make the right decisions based on empathy and experience (Oh, and a good understanding of the business goals of the project. Don’t want to forget those!) But lets face it, some times you’ve got to prove it.
Valid research can help build trust as well.
Questioning is good
All of this should make you question the value of formal research as it relates to web design. As you should! This kind of research is expensive and even when it’s working probably doesn’t justify the cost. Again, I don’t want to make any blanket statements here and I do think there are times when formal usability study could very well be worth it. But with most web design projects? Probably not.
Go informal. If you’re a designer, learn to observe people using the web. Get out there and talk to your users if you can do it, develop that empathy. If you’re looking to redesign and don’t want to spend a lot of money, do informal testing, or have someone do it for you.
You can do what I call Gorilla Usability (I wrote that back in 2002? Yikes!) or even something much less formal like cafe testing. I’ve done that for clients quite a few times and it’s been great.
There are lots of easy, low cost alternatives to formal user study, and IMHO, most of them will tell you all you need to know and more. What do you think?
Thanks Keith. I think we’re basically trying to reach a balance between too little measurement (relying exclusively on the designer’s instincts) and too much (evaluating designs based only on metrics).
Where this becomes a real problem is when designers neglect to do any research at all because they think that research has to be done by Phd ethnographic social scientists and fancy statisticians, and has to involve detailed quantitative analysis and strict scientific methodologies. They are intimidated by the prospect of doing all that expensive and time-consuming research, so instead they just do nothing.
Part of my point is to show that even many of the big expensive fancy research projects out in the wild are actually useless wastes of time anyway, so there is even less reason to be intimidated by them!
The middle “informal” ground you describe is ideal for the 90% of us who (or whose clients) can’t afford the time and money for an expensive properly-executed user research phase.
By the way, a long time ago (at a previous job) we made a video advertising our firm’s user research lab, in which we did user testing with a guy dressed up in a Gorilla suit. I’ll have to dig that up someday.
Exactly. Not all formal usability is bad, but given the cost…
Once, back in 2000 or so I was part of a project where we hired a research firm to come in and do studies for us. While some of the information they gathered was good, most of it was common sense or downright useless. What’s worse is that it was very, very expensive and they didn’t allow us (the designers in the group who hired them) to be part of the process. All we got was a 30-40 page report of the finding.
I’m pretty sure we all read and dismissed most of what was in there as useless. Ah, well, live and learn.
Thanks for the thoughtful series!
(I’d love to see that video by the way.)
Nick and I were just talking about this and it reminded me of another good story. When I was working for Children’s Hospital here is Seattle I did some informal “over-the-shoulder” type research with a bunch (maybe 15-20) of nurses to try and gain some insight into how they accessed policies and procedures.
I learned a lot and that “research” (me sitting down and talking with them and watching them pull up policies at their stations) could not have gleaned much of anything had it not been done informally at their desks. Put them into a lab, for example, and you lose all context of how they work and how they access the Web. People are talking to them, the phone’s ringing. They do the equivalent of “drive by” searching…
I learned things that drove design decisions with those conversations that I’m sure would not have been uncovered accurately via formal user study…
Ahhh, Keith beat me to the Children’s Hospital story. I just finished up some “over-the-shoulder” usability testing here with some administrative assistants. I too found that the conversations were often the most revealing instead of forcing them to follow the script I wrote down.
It doesn’t take many people either. I ended up meeting with 3 initially and they all had the same reactions to the design decsions I’d made (some things needed to be tweaked, but I wasn’t too far off). I ended up talking to 5 more because a few people here didn’t feel the sample size was adequate, but it turned out that they all held the same opinions.
We’ve been using this informal user testing a ton lately and it’s been really effective. We’ve been doing some more formal user testing as well but 90% of the time I feel I get more out of the “over-the-shoulder” stuff.
The “over-the-shoulder” work is also known as “contextual inquiry”, and can be quite extensively conducted. We recently did such work with about 30 college professors around the country. Very enlightening.
“Contextual Inquiry” sounds so formal though… ;0)