Re: Usability Test Metrics

Hi Lynne,

There's no simple answer to this one.

On the one hand, it depends on the type of system you're developing. For
example, if you're developing a data entry system (or other system that gets
lots of frequent use by lots of users) then you might want to minimise the
time it takes users to do their tasks - over the period of a year your
client might save a lot of money by reducing task times (or errors, or
support calls, etc). So working towards targets might be useful.

Similarly, your client might want to demonstrate that their system is more
efficient, effective etc compared to competitors' products. So they might
like this data.

Metrics shouldn't be too difficult to collect if you're going to run
usability tests anyway. For instance, you just need to start a stop-watch
when your users start the task and stop it when they stop! Similarly, it's
easy to collect data on how many people manage to complete the tasks they
attempt. Also, after each task you could simply ask them to rate how
frustrated they were doing the task, how satisfying they found it, and how
confident they were that they had completed the task successfully. That will
only take you seconds to do. If you've not got a huge amount of users then
collating the data shouldn't take too long either. I wonder if you're
over-estimating the cost of collecting it?

It might be difficult to set usability targets up-front. Are you going to go
through more than one iteration of tests? If so, when you've collected your
first set of metrics you will get an intuitive feel if users are (say)
taking too long to do the tasks... and on that basis you could set targets
to aim for for future versions.

Another reason to collect quantitative data would be political. Do you think
you might be asked to justify the quality of the system at some point? Or to
justify making changes to it? If so, you might find it useful to have this
data to hand to support your arguments for making the changes you suggest,
e.g. "We have to make this change... 6 out of 10 users failed to do this
task...", "Most users took more than 7 minutes to do this... that's too
long." If you suspect that you might have to defend your decisions then you
might want to collect this stuff. (BUT: because of the small sample sizes we
tend to use in HCI, some people may dispute the reliability of your data, so
this can be a slippery slope to head down!)

Of course, in terms of informing future design iterations, the qualitative
data is what you will really need: it sounds like you're doing early
formative evaluations rather than summative ones. It really depends on how
important it is to collect these metrics: what purpose would they serve?

Best of luck.

Fraser
--
Fraser Hamilton
Usability Consultant & Researcher
Email: [email protected]
Tel:   +44 (0)20 7040 8418

    --------------------------------------------------------------
     POSTINGS (in plain text): [email protected]
           SUBSCRIPTION CHANGES:  http://lists.syntagm.co.uk
            (or send email to mailto:[email protected])
    --------------------------------------------------------------