[CAnet - news] Internet Peer Reviews-Foolishiness of Crowds vs Bursts of Wisdom

For more information on this item please visit the CANARIE CA*net 4 Optical
Internet program web site at http://www.canarie.ca/canet4/library/list.html
-------------------------------------------

[Interesting posting from Richard Ackerman-- BSA]

 Nature is having an online debate about peer review

http://www.nature.com/nature/peerreview/debate/

I contributed an article "Technical solutions: Evolving peer review for the
internet" http://www.nature.com/nature/peerreview/debate/nature04997.html

I have provided additional thoughts in my blog
http://scilib.typepad.com/science_library_pad/2006/06/my_article_on_p.html

--
Richard Akerman | [email protected]
my article on peer review for Nature

[From Richard Ackerman's Blog--BSA]

My brief opinion piece "Technical solutions: Evolving peer review for the
internet" has been posted to the Nature peer review debate.
You can leave your feedback about it on the Peer Review Debate Comments blog
section.

I have supplementary bookmarks at
http://www.connotea.org/user/scilib/tag/peerfocus

I want to explore some of the concepts in the article in a bit more depth.
Also at the end of this posting you will find some acknowledgements, as many
people contributed to this article.

There were three main points I made:

   1. the Wikipedia "Wisdom of Crowds" approach is not a replacement for
peer review
   2. peer review is a service that is currently bundled up with journal
publication, but it could in theory be separated out
   3. peer review doesn't give us any information about the article other
than "this is ok" - there are many additional metrics made possible by
technology that can help to rank articles, such rankings may be part of
overall systems of discovery and peer discussion around articles

There is a fourth item which was part of my original charge for the article,
but which I didn't really cover:

   4. in what ways can online peer discussions contribute to peer review?

Taking these one-by-one.

1. I think we have to be very careful about the Wisdom of Crowds idea. 
In Technical solutions: Wisdom of the crowds, Chris Anderson explored a lot
of these issues.

One of my main concerns is that Wisdom of Crowds is sometimes oversold, in
the way that Service-Oriented Architecture (SOA) is.  Just put together a
system, sprinkle some magic Wisdom of Crowds dust on it, and hey presto, the
system is continuously improved by everyone who uses it.

The stark reality of Wikipedia and of open-source projects is that in
general, a very small core group of people, with quite a strict set of
management and control, drives the creation and growth of the system.  If
you looked in depth at the stats on how many people are making major,
ongoing contributions to Wikipedia, I think you will find the core group is
quite small.  Similarly, if you look at the open-source projects on
Sourceforge, you will discover most of them are one-man shows, or tiny
teams.  The poster child for open source, Linux, is really quite a small
technical team of experts at its core.

2. Since one of my primary areas of activity is in Service-Oriented
Architecture for library technology, I really like the idea of applying
service thinking to the scientific workflow.  Herbert Van de Sompel explores
this in Technical solutions: Certification in a digital era. 

The way I think about it is that currently when an article is submitted to a
journal, you get peer review as a bundled service that is part of the
publication workflow.  I don't see any reason why this couldn't be a
separate, stand-alone service: I write my article, I submit it to the
Elsevier Peer Review Service, and it comes back certified as passing peer
review. 

The logical question then is, who pays?  I think you have the same options
as now: in the author (or "author's research funder") pays model, you pay up
front, to cover the costs of peer review (which are mostly associated with
the administration of the system - contacting reviewers, gathering their
feedback, managing the communications).  In the library (or repository) pays
model, the organization that is going to be the primary host for the article
pays to have it certified, in order to maintain the quality of the contents
of its repository.  I realize of course that I am glossing over a lot of
complexity.  One may certainly wonder about the possibility of then "buying"
peer review - in theory the market might drive the business to services that
have low certification standards.  But I think that scientific reputation
mitigates against this - certification from a well-established service is
going to have the most weight.

In any case, I think this is an idea that should be explored, as we try to
understand how scientific communication is evolving: does it become
article-centric? repository-centric? publisher-centric? author-centric?

3. Peer review is, in my opinion, central to maintaining the quality of
science.  But science is also about communications, advancing our knowledge.
An article may be wonderful and receive an excellent peer review, but if it
doesn't lead to further discussions and additional work, is it contributing
substantially to science as a whole?  This is one area where the Internet
may be able to shine, but it is also an area of huge challenges.

One can certainly imagine a dream research workflow environment, where as
you work, relevant articles appear in your workspace.  To make that happen
is very difficult.  What is most relevant, articles with a high number of
journal citations?  Linked to by a lot of web sites?  Being discussed in
science blogs?  Assigned 3-or-more stars by your science peers? 

That being said, discovering relevant information is one of the key
challenges of our age, as we drown in the Internet flood.  We need to create
tools that enable communities to discover the information of relevance to
them.  An article that is tremendously helpful to an ordinary citizen
suffering from cancer may be of less relevance to a leading-edge cancer
researcher.  The best scientific resources for teaching about cancer at a
high-school level may not be the same as those for a graduate course.  The
underlying articles may all be excellent peer-reviewed science, but their
relevance is highly contextual.

4. Discussion lists, discussion groups, discussion boards, repository
discussions etc.

In part, this is an extension of the article discover topic above, because
presumably one reason you want to discover articles is in order to discuss
them.  I deliberately avoided any detailed coverage of the discussion topic
in my opinion piece, because it would have led me into a multi-paragraph
morass around moderation.  Conducting discussions that are both open and
relevant is a huge challenge in an Internet environment.  Even back in the
original USENET days, when it was primarily university students
contributing, a group like sci.physics was a chaotic mess of useful peer
communications, frivolity, and pointless flamewars.  That's with a tiny,
specialized, self-selected community.  Try now to extend it to thousands and
millions of people.

This is a really hard problem.  The Internet is as prone to the Foolishiness
of Crowds as it is to the occasional bursts of Wisdom.  As I write this,
Science Blogs lists
More Hot Topics

    * stephen hawking
    * ann coulter
    * hurricane season
    * dwarf dinosaurs

I have to say, Stephen Hawking and Ann Coulter (a US right-wing pundit, for
those who are fortunate enough not to know) are not two people one would
often find in the same company, and Ms. Coulter may be many things, but I
think even her most partisan admirers would not count scientist amongst
them.  There is a whole side discussion this opens up about ranking -
popularity is about the most-discussed.  In the US, most-discussed may be
more about heated political debate.  Another example: for a while, a top
"most bookmarked" article in delicious and Furl was Scientists respond to
Gore's warnings of climate catastrophe, even meriting a Slashdot article.

The article reads as if it is a serious scientific counterpoint to Gore's
statements, and many of the comments on delicious reflect this. On Slashdot
on the other hand, the top rated comments converged on the information that
this was in fact a bit of partisan political hackery, cooked up by a
consultant whose funding comes from oil companies.

Peer review would, I think, have immediately rejected this article. Peer
discussion... well, it's all over the map.  In Slashdot, high-ranked
comments can bubble to the top, as part of a sophisticated moderation
system.  In my Slashdot view, it doesn't take much scrolling down to reach

And Who Happens to Fund the Article's Author?
(Score:5, Informative)
by goMac2500 (741295) on 17:46 Wednesday 14 June 2006 (#15535430)
Why Exxon Mobile of course!

del.icio.us and Furl have no such moderation systems for comments.  So on
delicious we have comments expressing enthusiasm for the article such as

"Actual Climatologists respond to Gore's warnings of climate catastrophe"
"the real climate experts weigh in on Al Gore's mis-representation and
distortion of climate science"

as well as some negative comments
"right wing website trashing gore"

On Furl, the comments are pretty negative about the article

"blabbering about science without offering anything legitimate. What crap."
"how does this bullshit get written?"

Three different communities, three different comment systems, three
different views of the article.
If you read delicious, you might get an impression the article was credible.
Reading Slashdot and Furl, you would get the (more accurate) impression that
the article is highly biased and its science is dubious at best.

What's next?

I think it would be very interesting to explore the idea of controlled
social discussion spaces for science - MySpace for scientists?  Stay
tuned...

I am deeply indebted to a number of people for in-depth background
discussions that helped me shape this article.

They are:
Dr. Bruce Dancik, Editor-in-Chief of NRC Research Press, about peer review
in general.

Cathy Norton, Library Director, MBLWHOI Library, and Director of Information
Systems Marine Biological Laboratory, Woods Hole Oceanographic Institution
and
David Remsen, Information Systems Program Developer, Marine Biological
Laboratory, Woods Hole Oceanographic Institution
about the uBioRSS tool (part of the uBio project)

Carl Lagoze, Senior Research Associate, Cornell Computing and Information
Science
and
Sandy Payette, Researcher and Co-Director of the Fedora Project, Cornell
Information Science
about the Pathways project

Euan Adie, bioinformatician, University of Edinburgh
about the PostGenomic tool

My organization, NRC CISTI, Canada's National Science Library, provided me
with time to work on this article.

I also had lots of help from Nature:

Thanks to Sian Lewis (Ichabod is itchy) for her help shepherding my article
through the Nature publication process, and to Maxine Clarke for her key
role in providing me with this opportunity.

Previously:
June 15, 2006  new articles up in Nature Peer Review Debate
June 05, 2006  the Nature of peer review

Posted by Richard Akerman on June 21, 2006 at 08:19 AM in Publishing,
Science | Permalink



-------------------------------------
To SUBSCRIBE:
send a blank e-mail message to
[email protected]

To UNSUBSCRIBE:
send a blank email message to
[email protected]
-------------------------------------

These news items and comments are mine alone and do not necessarily reflect
those  of the CANARIE board or management.
-----------
[email protected]
www.canarie.ca/~bstarn
skype: pocketpro
SkypeIn: +1 614 441-9603


_______________________________________________
news mailing list
[email protected]
http://lists.canarie.ca/mailman/listinfo/news