A co-worker of mine pointed me to this video on YouTube, Stone and Stone Wheel. I’ve been thinking a lot lately about innovating versus adhering to convention and when and when not to pay attention to user testing data, and this video really got me thinking.

I remember way back in 1999 observing my first user test, I was working for frog design on a web site for a large direct sales computer manufacturer. The client had taken it upon themselves to hire a third party to conduct user testing on the site which frog, in collaboration with the client’s own web team, had just launched. It was both awesome and horrifying to see users actually using that which we had created. Some of our design decisions succeeded exactly as we thought they would while some, that we had thought were pretty simple and straight forward, baffled users to a point where they could not complete necessary tasks. The whole experience was very enlightening.

When I got back from user testing I talked to one of the managers at frog about user testing and asked why frog didn’t as a matter of process include it in their projects. I was told that frog doesn’t do user testing. Period. The argument against it was users often do not respond well to new or innovative ideas (and frog had built its reputation on new and innovative ideas.) New or innovative work will rarely test well, thus negating the relevance of user testing.

Since I left frog I’ve heard that they have changed this policy (if it ever was an official policy, rather than just a general stance by the management.) And since I’ve left frog I’ve gotten to work with a lot of user experience professionals. Some of which have insisted that everything be tested. Every decision should be in some way be run by users. I’ve seen the User Centered Design methodology be taken to great lengths, where paper prototypes, then wireframes, then design comps, then the finished site are tested revised and retested until the data shows what the usability experts are looking for. If a user does not figure it out on the first try, it’s a fail.

I’ve also worked with user experience professionals who have demonstrated that there are ways to test innovative ideas and that not all data from user tests should be taken at face value. Just because someone doesn’t figure something out on the first try doesn’t always make it wrong. Sometimes it’s when a user doesn’t figure it out on the second or third try that it means something is wrong, and the fact that the user found something compelling enough to try to use it multiple times should be given some weight. Innovative ideas introduce a learning curve, it’s whether or not users are willing to overcome the learning curve that should be paid attention to.

Designers should always listen to their users. Designers should really listen to everyone, you never know where a good idea might come from. But being a good designer means knowing which ideas to use and which feedback to listen to. Good designers also know when to take a risk and try something new and when to stick to convention. My own philosophy on the subject is based on a quote from Charles and Ray Eames, “Innovate as a last resort.” If an method exists that users are familiar with, use it. Use it unless you think you’ve really got something new that works, and when you do, test it. And if no one gets it in testing, weigh the costs versus benefits before going forward anyway.