Re: Best Practices for Measuring Usability

Just to add the Sharon's warning about the use of a think aloud  
protocol, I have a colleague who uses layered voice stress analysis to  
analyze respondent comments.  He normally does it for market research  
work but was recently asked to perform an analysis on a think aloud  
protocol usability evaluation.  His findings were that people  
performing a think aloud protocol while trying to perform a task  
showed signs of a loss of concentration for both activities (not  
really unexpected).  He also found that people who had to talk while  
working out a problem showed "relatively high levels of physical  
tension... anticipation, fear, embarrassment, and guilt."  In other  
words, if participants are required to talk while working through a  
process, they cannot filter their responses and end up embarrassing  
themselves or fearing they will. This may seem like a good think to  
some (the ability to know their thought process without filtering),  
but the resultant effect on their emotional state and performance  
makes this more of an artifact of the protocol than data.

We only use think aloud in evaluations of conceptual designs (e.g.,  
storyboards or static prototype) or only after task completion.   We  
also know (recently proven again in a for Sharon as a matter of fact)  
that time-on-task does not correlate with performance so time a  
measure of interest only in special cases.    And TOT is never real if  
taken during a think aloud protocol.

Bill

On Sep 15, 2008, at 9:24 AM, Sharon Laskowski wrote:

> I would also encourage you to read some of the classic books.  Rubin  
> and Chisnell have just published the 2nd edition of "Handbook of  
> Usability Testing", for example.  And consider joing the Usability  
> Professionals' Association and attend their conference.
>
> I'm a bit concerned that you said you would be measuring time in  
> your earlier note, but here you say you expect a think-aloud.  Time  
> would not be very informative if they are thinking aloud--too much  
> variation in how people discuss.  Think alouds can also affect error  
> rates.    The above book has some good advice about using think- 
> alouds.
>
> Sharon
> Quoting [email protected]:
>
>> Hello All,
>>
>> I appreciate all the quick response; all the materials were really  
>> helpful
>> to get an idea and gathering knowledge about usability testing;  
>> however
>> would still be waiting if someone would like to share their  
>> experiences
>> regarding any major problems faced during the user testing; as while
>> conducting few of my tests, I noticed that few of the users were very
>> conscious when they knew that they are being noticed and the  
>> sessions are
>> being recorded. I believe this would not lead me to the results I am
>> targeting for; also I noticed that they are unable to think a loud  
>> as I
>> see them being conscious and hastate to do so in presence of a
>> representative taking notes; any guidance or shearing from the  
>> members
>> would be great help.
>>
>> Thanks once again.
>>
>> Best Regards,
>> Sandeep Rathod
>> Sr. HCI Designer
>> --------------------------------------------------------------------------------------------------
>>
>> Zycus - SAVE WITH EASE
>> Email: [email protected] | Tel. Desk: +91 22 6640 7676  
>> (Extn: 7689)
>> Website: http://www.zycus.com
>> --------------------------------------------------------------------------------------------------
>>
>>
>
>   --------------------------------------------------------------
>   Tip of the Day: Suspend your subscription if using auto replies
>    CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
>             MODERATORS: mailto:[email protected]
>      SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
>   --------------------------------------------------------------
>


Bill
---------------------------------------------------
Bill Killam, MA CHFP
President, User-Centered Design, Inc.
20548 Deerwatch Place
Ashburn, VA 20147
703-729-0998 (Office)
703-626-6318 (Mobile)
http://www.user-centereddesign.com





    --------------------------------------------------------------
        Tip of the Day: Email mailto:[email protected]
               with any comments, questions or problems
     CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
              MODERATORS: mailto:[email protected]
       SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
    --------------------------------------------------------------