(Video) Rolf Molich About Discount Usability Testing and Quick-and-Dirty User Tests in the Real World

May 9, 2016

We had the pleasure of interviewing Rolf Molich on a recent visit to Denmark. He is an institution in the area of usability and has dedicated himself to this topic for more than 30 years. In this video Rolf explains:

• How to achieve valuable results with quick and dirty usability tests
• Why it is important to conduct tests with users “in the real world”
• Why bigger budgets are not always the solution to usability problems
• What impact the usage of multiple devices has on usability tests

 

Here’s the video – enjoy 11 minutes with Rolf:

Transcript of the interview:

Rolf: I started doing my first usability tests back in 1984. So, it is a lot longer than I really like to think of. What has surprised me in this field – and what keeps surprising me – is that many of the problems that we run into today are the same that I saw right when I started back in 1984. We have more things at our disposal right now, including more ways to do it wrong. But basic problems still remain.

Lotte: So what are the usability problems that still exist?

Rolf: Well, I think it’s still the classic mistake that everyone, including me, believes that they can do it right; they don’t have to use these complicated and cumbersome methods where they test with users. That takes time, and it’s awkward, and so on. There are still a lot of basic things that we have known for many years that people do wrong. Back in 1990, together with my friend and colleague, Jakob Nielsen, I formulated some of these rules and the so-called heuristics. And I am very glad and pleased to see, that now, 25 years later, these heuristics are still very popular.

Lotte: Does the usage of different devices have an impact on usability?

Rolf: Well, first of all, I must say I have not studied this; I have done few usability tests of big-screen user interfaces and small-screen user interfaces. And I have one basic principle, and that is: I don’t like to speak about things of which I have no knowledge, no data, like I usually say. But in my view – to try to answer it after all –  is that consistency becomes very important. So that if you have the same user interface or similar user interfaces on multiple devices, consistency becomes important, so users can transfer their knowledge from one device to the other one. But then again, the basic message is: you’ve got to test it. You’ve got to get users – real users – working with real tasks on these interfaces in order to find out where the problems are. And they might be something that’s not related to consistency at all.

Lotte: Are there specific usability challenges for small devices?

Rolf: Yes, it is true. The small-screen devices present special problems because of the limited area on the screen. And you’ve got to use widgets and items on the screen that are well-known to users and that will allow you to consistently control how the area on the screen is being used.

Lotte: Do we need bigger budgets to run usability tests on different devices?

Rolf: Well not necessarily bigger budgets. I think there are many things here. It is not necessarily solved by bigger budgets. It could also be better people. It could be more time. It could be better knowledge of modern, efficient methods – That’s  one of the things that I try to teach people a lot, that I like to focus on when I teach professionals – and that is knowledge of discount usability methods where you get more bang for your buck. You don’t necessarily use an expensive usability lab: you go out in the street and do a few quick-and-dirty tests, and then you try to get into a good iterative cycle where you correct problems, test, correct the new problems you have found, test again, and so on, and so forth.

A big budget is not necessarily a good solution, a way of solving more problems. One method that I sometimes use is that I go to a café or public place like a library and ask people: “Can I buy you a cup of coffee in return for 10 minutes of your time?” And then we do an admittedly quick and dirty test, where we don’t have a lot of control of the participants we get in. But in order to find many usability problems, you don’t need a complicated set up. If you can just get the basic, the low-hanging fruits right, you will already have a user interface that’s better or at least as good as that of your competitors.

One of the things I have learned is that bigger budgets are not always the solution to usability problems, because I have seen teams who had a lot of money at their disposal and still they did it awfully wrong.

Lotte: Should we move out of the research lab and test in more natural contexts?

Rolf: I think it is always important to test in realistic surroundings because if you test in a usability lab, for instance, there are certain advantages with a usability lab. Like, it is easy to get people in and observe what is happening, which has a tremendous effect in a company politically. But often the surroundings in a usability lab are not natural: they are artificial. It’s a very cold and – hmmm, not sure of the word… – sterile surrounding. So, there are many real problems that you will never encounter in the usability lab. So, I usually tell people to go out in the real, dirty world and find out how it works there, back in peoples’ home. That’s one of the reasons why I do a lot of unmoderated testing these days, where you don’t have a usability lab but in return people work with the user interface in their home, on their own browser, on their own computer, and you learn a lot of things – what it’s like in the real world where it’s not always the same computer and not always the same browser that you use.

I know of very few usability problems that by definition could not have been found in the lab. But the point is, that moderators – usability testers – are not always aware of these problems. If you test in people’s homes, you will find that these problems just come out naturally. Another good example of that is, for instance, the open-ended task, where you don’t have a specific task that you ask the user to do. But, for instance, you ask them – just to take a simple example – if you want to test the usability of a car rental website, you don’t say to people, “Rent a car in Frankfurt, starting tomorrow, returning it 48 hours later.” But you ask them in an open fashion, “Do you have any plans of travelling somewhere? Oh, you want to go to Strasbourg, for instance, or you want to go Vienna? Sounds interesting. When are you going? Would you please show me how you would rent a car that fits your needs and your purse in Vienna Airport?” And in that way, by having the user formulate the task, and not always having to use the same task, – or like with the unmoderated usability test, where it’s the users own equipment that often provides surprises –  you learn a lot more in that way.

Lotte: Is it true that “you only need to test with 5 users in an iterative cycle”?

Rolf: I think so. Just to clarify, there are actually 2 quotes here. The original quote was that 5 users are enough to find 85% of the problems. Now that has been shown, very clearly, not to be true. Some of the studies I have done – the comparative usability evaluations studies – showed that was definitely not true. But it is true that I have been advocating that, with 5 users, you can find so many problems, that, instead of testing more users than 5, it is better to go back to the development team and tell them, “Please correct this, and please correct that.” And if you still have more time for testing, then after these changes have been made, go out and test again. And that’s again one of the basic principles of discount usability testing. That is, do it in a way where you are almost certain that you will not solve, you will not correct, all problems, but it is still immensely helpful and highly cost-effective.

Lotte: How can we get more commitment from decision-takers for usability?

Rolf: The best way I know of in a company that’s not mature usability-wise, that does not really know what usability is would be to take one of their products and run a simple usability test, a usability test where the main purpose is to convince management that there are problems even in our product which we thought was so good. You can do that with just running 2 or 3 tests. You are not out to really find problems: you are out there to demonstrate to management and to your colleagues, to the development team, to marketing people, to educators and trainers, and so on, “There are problems in our interface, and the good news is there are simple ways to find these problems in time for them to be corrected.” So that’s the number-one thing that I try to do. So to speak, start from the end, start with the testing. And then, in the hope – after they have realized, “Yes, there are indeed problems in our products” – then come in and say, “Well, maybe the next time we should start a little bit earlier with a contextual inquiry, interviewing a few users, setting up usability requirements, writing prototypes and testing the prototypes, of course.

A BIG thank you to Rolf!

We’d like to thank Rolf for his kind invitation to his home in Denmark for this interview. Whether we meet in Vienna or in Denmark, it is always a pleasure. Every discussion with him inspires us.

About Rolf Molich:

Rolf Molich is an institution in the area of usability. He conducted his first usability tests back in 1983 and has been devoted to usability ever since. Together with Jakob Nielsen he developed heuristics for user interface design in 1990. Even today, more than a quarter of a century later, they are still among the most used heuristics. Rolf owns Dialog Design, a consultancy based in Denmark.

Want to get in touch with us?

– Subscribe to our newsletter to receive our case studies and tips.

– Call us or write us if you’d like to know how we can help you.