Re: Metrics :: Usability Group Performance

Hugues wrote...

>I was thinking of the impact the usability staff has on end users could be
a measure that we take >into account. Likewise, so could user satisfaction,
perceived ease of use, and perceived usefulness.

I think its theoretically possible to measure the impact of a specific team
on a product, but difficult. (If it was easy, there wouldn't be whole books
on cost justifying usability, etc.)  The one thing I would caution about is
using subjective measures like perceived satisfaction or perceived ease of
use.  They seldom correlate with performance measures.  Then again, I do
know many people (and many I work with) who use these measures, but they are
mostly in marketing.  But marketing has been able to do exactly what you're
trying to do, so you might consult with them as well on how they measure the
impact of marketing efforts.  

I think for what you're trying to prove, more objective measures would be
more powerful.  But the issue will be knowing the cause.  There are too many
players and too many variables unless you can do it in a controlled study.
But it is possible, with some care and planning, to demonstrates product
improvements in a single case study. You'll just have to be a bit careful
about the method and the statistics.  (I just presented at our local
chapters on how design as study that would allow it and how to report
statistically valid conclusions.  It's not hard but it is rigorous.)  It
might also take some up front work to get a study approved since the
approach is counter to the typical study done as part of a design process.
(It's more like an IV&V process.)  If you are doing work on internal
products (which I assume you are), you have an even better chance of getting
the data you need since you can address ease of use and not be stuck with
only looking at ease of learning.  (Use of use being more significant since
ease of learning is so transient.)

Hope this helps.

Bill
-------------------------------------------------------
Bill Killam, MA CHFP
President, User-Centered Design, Inc.
20548 Deerwatch Place
Ashburn, VA 20147
email: [email protected]
Work/Fax: (703) 729-0998
Mobile: (703) 626-6318
Web: www.user-centereddesign.com


On 10/24/07, Belanger, Hugues <[email protected]> wrote:
> Bill,
> Generally speaking, we are trying to optimize our business value within
Enterprise. We are no exception since every group is required to do it and
will be competing for an award.
>
> The performance of the staff is ultimately what I want to measure. I was
thinking of the impact the usability staff has on end users could be a
measure that we take into account. Likewise, so could user satisfaction,
perceived ease of use, and perceived usefulness.
>
> In a perfect world, I would like to be able to counter balance speed with
doing the right thing. A priori, I do not have anything against measuring
"doing the right thing" in terms of user metrics.
>
> Regards,
> -hugues
>
> -----Original Message-----
> From: Bill Killam [mailto:[email protected]]
> Sent: Wednesday, October 24, 2007 8:44 AM
> To: Belanger, Hugues; [email protected]
> Subject: RE: Metrics :: Usability Group Performance
>
> >Has anyone else successfully used metrics to measure the performance of
> their group?
>
> I can't tell if you mean you want to measure the performance of the staff
who do usability work for you, the impact usability had had on end users, or
the performance of end users doing their work.  Are you trying to cost
justify (or optimize) the usability group within Enterprise?
>
> Bill
> -------------------------------------------------------
> Bill Killam, MA CHFP
> President, User-Centered Design, Inc.
> 20548 Deerwatch Place
> Ashburn, VA 20147
> email: [email protected]
> Work/Fax: (703) 729-0998
> Mobile: (703) 626-6318
> Web: www.user-centereddesign.com
>
>
> -----Original Message-----
> From: ACM SIGCHI WWW Human Factors (Open Discussion)
[mailto:[email protected]] On Behalf Of Belanger, Hugues
> Sent: Tuesday, October 23, 2007 12:44 PM
> To: [email protected]
> Subject: Metrics :: Usability Group Performance
>
> We will soon be tracking the performance of our usability group with
metrics.
>
> We want to balance timeliness, cost, and quality of the products we
develop.
> Finding a set of key metrics that will help drive the right behaviors
(e.g.
> do things faster but without cutting corners) is essential.
>
> Has anyone else successfully used metrics to measure the performance of
their group?
>
> Thanks,
>
> Hugues Belanger
> Manager :: Usability Architecture
> Enterprise Rent-A-Car
>
>
>
> ________________________________
> CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are
intended solely for the use of the individual or entity to whom they are
addressed and may contain confidential and privileged information protected
by law. If you received this e-mail in error, any review, use,
dissemination, distribution, or copying of the e-mail is strictly
prohibited. Please notify the sender immediately by return e-mail and delete
all copies from your system.
>
>
>
>     --------------------------------------------------------------
>         Tip of the Day: Forward out-of-office replies to
>                     mailto:[email protected]
>      CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
>               MODERATORS: mailto:[email protected]
>        SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
>     --------------------------------------------------------------
>
>     --------------------------------------------------------------
>            Tip of the Day: Postings must be in plain text
>      CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
>               MODERATORS: mailto:[email protected]
>        SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
>     --------------------------------------------------------------
>
>


-- 
Wendy Cown
[email protected]

    --------------------------------------------------------------
        Tip of the Day: Forward out-of-office replies to
                    mailto:[email protected]
     CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
              MODERATORS: mailto:[email protected]
       SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
    --------------------------------------------------------------

    --------------------------------------------------------------
           Tip of the Day: Postings must be in plain text
     CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
              MODERATORS: mailto:[email protected]
       SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
    --------------------------------------------------------------