Re: Best Practices for Measuring Usability

Sandeep,

Have you considered being in another room while the participant is doing the 
tasks? I have found that leaving the participant alone with the computer 
after I have welcomed them, shown them the set up, and helped them get 
settled makes it much easier for them to forget someone is watching. This 
requires you have have a set up in which the participant's screen is shared 
with you and there is an audio connection between the room you are in and 
the room the participant is in. This is not difficult to do if the two rooms 
are on the same network. There are several software applications that enable 
either or both of these forms of sharing.

I also begin any study by having the participant read aloud from a page that 
gives the general description of what they are going to do, tells them about 
confidentiality, and then explains the think aloud protocol and has a short 
task using notepad they can do to practice thinking aloud. Most participants 
I have worked with have little problem with it after that. Of course, I will 
gently coax them to think aloud if they become silent for too long.

Facilitating user observation studies is not an easy task; it takes lots of 
practice to feel comfortable doing it and learning how to coax a participant 
without instructing them. If you really need to keep time-on-task data, 
however, you might consider not using the think aloud protocol and, instead, 
recording what they do on the screen and then after they finish the tasks, 
run the video and ask them to talk about why they need this or that.

If you use a screen capture tool such as Morae that lets you index what is 
being captured in real time, the method of asking participants to explain 
actions is made easier because you can easily jump to specific places in the 
video where you want to ask follow-up questions. There have been some 
studies that indicate asking after the tasks gives just as good data as 
doing think-aloud.

You certainly have been given a challenging task, and the person who 
suggested getting the Rubin book, Handbook of Usability Testing, gave good 
advice. It is a step-by-step explanation of how to plan, design, and conduct 
a user observation test.

Good luck,

Mary Deaton
Deaton Interactive Design
Manager, Usability and User Experience SIG of STC


----- Original Message ----- 
From: <[email protected]>
... I noticed that few of the users were very
conscious when they knew that they are being noticed and the sessions are
 being recorded. I believe this would not lead me to the results I am
 targeting for; also I noticed that they are unable to think a loud as I
 see them being conscious and hastate to do so in presence of a
 representative taking notes; any guidance or shearing from the members
 would be great help.

    --------------------------------------------------------------
    Tip of the Day: Suspend your subscription if using auto replies
     CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
              MODERATORS: mailto:[email protected]
       SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
    --------------------------------------------------------------