Re: Best Practices for Measuring Usability

Hi Sandeep, and list members

I have used the think-aloud technique before during usability audits, so will have a go at sharing some of my experiences and how myself and colleagues have learned to cope with some of the drawbacks. 

Firstly, In order to help ensure that participants were willing to be absolutely honest when they were unhappy with something, it was made clear at the beginning that the aim of the testing was to identify the strengths as well as weaknesses of the user experience. So the users were told that giving negative feedback is fine, and it is all part of the development and improvement process.

Ensure that your participants remain anonymous, and that they are aware of this, as this may make them more wary of what they are saying/doing during the test. I once had a participant with health problems, he was concerned that his employers would be able to identify him as a result from testing. If you are recording evidence (for example with audio/video), it is often a good idea to supply short quotes/overviews in your report and not the actual recorded evidence where your clients face/hand might be seen and voice heard.

In terms of the participant going quiet during think-aloud, this is quite normal, and often happens when they are focused and thinking hard about an action they are carrying out at the product/software interface. This can often be a good indicator that they may be experiencing a higher degree of cognitive load. Of course it is not the perfect solution, but you can gently try prompting them by asking - "Can you tell me exactly what is happening right now, what are you looking for/trying to do".

Any lab type test is going to be slightly artificial, simply as the user is not in their normal environment and will know they are being monitored in one way or another which might make them uneasy. If you have slightly more time, and depending on the particular outcomes you are assessing for, it might be an idea to follow up an initial lab based series of tests later on with some kind of field trial. For example, asking people to use your product for a period of time, say a couplpe of months and to keep a short diary of their experiences. In this way, they are much less focused on the assessment, and more on achieving their usual goals in their normal environment.

Hope that helps for a start, 

Marcia Gibson

Research student (Ex usability/accessibility consultant)

    --------------------------------------------------------------
    Tip of the Day: Suspend your subscription if using auto replies
     CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
              MODERATORS: mailto:[email protected]
       SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
    --------------------------------------------------------------