Re: Who is in favour of empirical research approach?

You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0

This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe.  A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) 

Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html 

For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture.

I like José's distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well.

Miles Kimball
Texas Tech University


-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Caroline Jarrett
Sent: Friday, March 05, 2010 4:03 PM
To: 'Discussions about information design'
Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach?

José explained:

> I am developing a list of arguments in favour of (or against, if you like)
empirical research.
> But, I am focused on empirical research that involves statistical analyses
only.

Not sure if these examples are what you mean, but here goes anyway.

1. The Direct Marketing industry has focuses solely on A/B testing for ages.
Send out a mail shot with half of the pieces done in style A and half in
style B: whichever gets the best response wins.

I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail
Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely
available second hand. (Actually, I may buy one: I lent my copy to someone
who failed to return it).

The results tend to be pieces that are remarkably ugly but effective.

2. More recently, a client of mine decided that he wanted to do A/B testing,
which was easy for him to arrange, rather than anything else. We did end up
with a better form. I wrote it up: 
Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of
the Usability Professionals' Association Conference, Minneapolis, Minnesota,
USA.
Article is available as a .pdf from here:
http://www.formsthatwork.com/Articles

3. Quite a few of us have been using statistical and numerical methods in
usability, usually as adjunct or complementary techniques. For example, I do
quite a lot of search log analysis for the Open University and we use other
complementary numerical methods e.g. they do a lot of traffic analysis.
However, like the thoughtful people they are, they regard these as part of
an overall package that includes qual. methods as well. I extracted some
examples of the ways that we use quantitative methods in this paper:
Jarrett, C. et al " Using Measurements from Usability Testing, Search Log
Analysis and Web Traffic Analysis to Inform Development of a Complex Web
Site Used for Complex Tasks" 
http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF
ID=80667064&CFTOKEN=87243617

4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative
methods with qualitative methods, and also sometimes uses quant methods on
their own. His approach is written up in:
  
Tullis, T and Albert, W. "Measuring the User Experience: Collecting,
Analyzing, and Presenting Usability Metrics" Morgan Kaufmann

5. There is a sort of fashion for numerical-only justification of design
decisions going on in parts of the web design community. For example, just
in the last few days, there's been a bit of discussion around the "madlibs"
style form used by http://huffduffer.com/signup
Luke Wroblewski justified his design decisions to use this style with
conversion statistics:
http://www.lukew.com/ff/entry.asp?1007
but others have fought back with some further statistics:
http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o
wn-tests/
and scepticism about the way the stats have been generated:
http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver
sion-by-30-well-maybe/


Best
Caroline Jarrett
www.formsthatwork.com


___________________________________________________________________

Use the following address to post a message to all subscribers: 
 [email protected]

To subscribe, unsubscribe or change your options, visit:
 http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe

For all Information Design matters:
 http://InformationDesign.org

Problems? Write to:
 [email protected]
___________________________________________________________________

___________________________________________________________________

Use the following address to post a message to all subscribers: 
 [email protected]

To subscribe, unsubscribe or change your options, visit:
 http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe

For all Information Design matters:
 http://InformationDesign.org

Problems? Write to:
 [email protected]
___________________________________________________________________