Summary: Concept testing using focus groups

Hi All -

Thanks for all who responded.  Here is my original question, a quick
overview of the method we chose for the research and then a summary of
everyone's responses:

"I am in the initial phases of planning a research effort on new concept
ideas for our website.  Initially, my thought is to conduct small focus
groups with individual components (rather than the traditional one-one-one
usability testing of proposed concepts in prototype form) as this method
might elicit more discussion and ideation in a group setting.  I am
interested in hearing from people who have used focus groups as a method
for testing new concept ideas and recommendations for what worked well and
what didn't (best practices, etc.)"

I did end up conducting 5 small focus groups (4-5 participants each), with
individual components and it worked wonderfully.  We presented 9 concepts
to each group. (The concepts were truly at the 30,000 foot level - no
functionality, just purely conceptual).  The format we followed was we
provided each participant with 1) a written description of the concept
with callouts, 2) a larger visual representation of the concept and then
3) we had them fill out a quick questionnaire to gauge their initial
thoughts about the concept (wow factor, usefulness and "stickiness").
After each participant reviewed the materials and completed the
questionnaire, discussion followed.  At the end, we used one of
the "Innovation Games" (listed below - thanks Nancy!) - specifically
the "20/20" exercise to have participants stack rank the concepts, as a
group, based on which concept was most compelling.  It was an interesting
way to pull all the discussion together at the end.

=======================
I've been introduced to Innovation Games which are an intriguing set of
tools, some of which are familiar (with new names), or variants of
familiar activities.  Several are new to me.  Most work well with "focus
groups", but differ from a usual focus group by allowing people to
interact by playful engagement with one another.  (I think of focus groups
as all verbal expression, much of which is directed by and toward the
moderator.)  The games avoid some of the typical difficulties of focus
groups (e.g., dominant, vocal member or unassertive member).

Disclosure:  I am not the author of the book describing the games (Luke
Hohmann of Enthiosys is); I have been certified as a facilitator, have
used the games several times, hope to more, and do contribute to the
Innovation Games discussion forums http://www.innovationgames.com, where
you'll find more comments on
individual games, techniques and materials.   Or write/call me off
list.
--
Nancy Frishberg  +1 650.556.1948  [email protected]

=====================
Hi Mary,

My experience is that focus groups heavily depend on the skills of the
moderator.
I agree with you that for gathering new ideas a focus group is much better
than u-tests.

If you have to conduct a u-test for this I would recommend the co-
discovery method for this - you can see it as a micro-focus-group then.

Kind regards
Peter
___CURE - Center for Usability Research & Engineering___

Peter Wolkerstorfer
Usability Engineer
Hauffgasse 3-5, 1110 Wien, Austria

[Tel]  +43.1.743 54 51.46
[Fax]  +43.1.743 54 51.30

[Mail] [email protected]
[Web]  http://www.cure.at

=======================

Mary,

I would be wary of using focus groups for 'testing' - if by that you mean
assessing and comparing elements/features/functions. The discussion can e
too uncontrolled, and issues other than design come in to play too easily.

I think the group setting can be useful for helping you see the big
picture, understand users' conceptual view, capture the language and gauge
what's 'sensitive' what's 'important' and what's nice to have'.

good luck!

Avi

=======================

Hi Mary,
Are you asking if anyone has done user testing with more than one person
during a session? If so, the answer is yes. This testing method is called
co-discovery and it works quite well because you get to hear each person
discuss their decisions for doing something with the other. Co-discovery
is only done with two people though. I don't think you would get good data
from having a group greater than two test something. I believe that you
would see the same problems with traditional focus groups (i.e. one person
taking the reigns and leading, while the rest just follow along).

I hope this helps you.

Best,
Kyle

--
Kyle Pero
Usable Interface - Usability and User Interface Design Consulting
http://www.usableinterface.com
Phone: 267-241-4786
Email: [email protected]

========================

Hey, Mary Pat. How are you? I have done things like this but I did not
call them focus groups, I called them participatory design sessions.

What form would these ideas take? Would they be printed wireframes?

If you are looking for information on location of objects, and such, I
would make it possible for them to actually rearrange things. For
exapmple, but out the components of the layout, put them on stickynotes or
put magnetic backings on them and such so that people can then reassemble
them.

As people suggest changes, you can hear them talk about why.

If you have a board on which to use magnets, put the magnets up showing
the concept you all have decided on. Then, after hearing comments, ask
people, one at a time, come come up and rearrange things. Have them talk
aloud.
Then, let the rest of the groups respond to that that person did. If
someone says I have another idea, ask that person to come up.

Were you in the class where we looked at an online testing tool that
elicited people's emotional reactions to things? It was from MIT or Yale
or somewhere. By putting several images in front of people multiple times
and asking varying questions, you get a very comprehensive pattern
indicating likes and dislikes based on their choices, not on anything they
verbalize.
Maybe it was Dave Farkas' class. I wish I could remember.

Mary Deaton

=======================

Hi Mary--

It sounds like you are interested in applying concept testing to web page
development, which shouldn't be a problem. What stage will your project be
at that point? Do you want participant feedback on the type of information
you want to provide on each page? Or do you want feedback at a higher
level, having to do with the purpose of the site and its content?

Unless you are at the product prototype stage, where you have something
more concrete, the format should be the same. Gather a group of ten to
twelve people who represent your target audience in a room and ask for
their input. What you present to them and their response will depend on
what you're testing.

Feel free to send me follow-up information or questions.

Best regards,

Mike Elledge
Assistant Director
Usability & Accessibility Center
Michigan State University

=======================

hi mary;

one difference we've noticed, particularly with a new design/concept
product, is that we haven't gathered as many data points for "what works"
and "what doesn't work" and "how this could be improved" when using a
group discussion instead of an individual interview.  we've done projects
that included 6-12 individual interviews and 1-2 group discussions (each
group with 6-8 people).  the individual interviews, every single time,
have produced more interesting feedback and a wider breadth of feedback --
specifically for new designs and products and concepts.

when we do group discussions for reseach for new ideas/concepts/
designs/products then we tend to discuss the general product area and
elicit brainstorming before introducing a new concept -- helps the group
get in the mindset of thinking of new ideas so they don't get stuck
in "but i like this the way it is" -- that has resulted in more vibrant
feedback than showing new concept and discussing opinions of it.

we've seen more interesting and wider feedback in group discussion when
the topic is branding, advertising, logos, professional practices,
behaviors/opinions regarding existing products/practices and would could
be improved about what exists.

best,
leanne
=======================
END OF SUMMARY

    --------------------------------------------------------------
    Tip of the Day: Quote only what you need from earlier postings
     CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[email protected]
              MODERATORS: mailto:[email protected]
       SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
    --------------------------------------------------------------