Submission deadline put back - Workshop on novel methodologies for evaluation in IR

After many requests, we have agreed to put the 
submission deadline back to Midnight 17th of February (UK time)


CALL FOR PAPERS

Workshop on novel methodologies for evaluation in information retrieval
30th March, Glasgow, Scotland, as part of ECIR 2008


OBJECTIVES
Information retrieval is an empirical science; 
the field cannot move forward unless there are 
means of evaluating the innovations devised by 
researchers. However the methodologies conceived 
in the early years of IR and used in the 
campaigns of today are starting to show their age 
and new research is emerging to understand how to 
overcome the twin challenges of scale and diversity.

Scale
The methodologies used to build test collections 
in the modern evaluation campaigns were 
originally conceived to work with collections of 
10s of thousands of documents. The methodologies 
were found to scale well, but potential flaws are 
starting to emerge as test collections grow 
beyond 10s of millions of documents. Support for 
continued research in this area is crucial if IR 
research is to continue to evaluate large scale search.

Diversity
With the rise of the large Web search engines, 
some believed that all search problems could be 
solved with a single engine retrieving from a one 
vast data store. However, it is increasingly 
clear that evolution of retrieval is not towards 
a monolithic solution, but instead to a wide 
range of solutions tailored for different classes 
of information and different groups of users or 
organizations. Each tailored system on offer 
requires a different mixture of component 
technologies combined in distinct ways and each solution requires evaluation.

This workshop calls for research papers (either 8 
page full papers or shorter 2 page papers) to be 
submitted on topics that address evaluation in 
Information Retrieval. Topics will include but are not limited to:

         • test collection building for diverse needs
         • new metrics and methodologies
         • evaluation of multilingual IR and/or multimedia IR systems
         • novel evaluation of related areas, such as QA or summarization
         • evaluation of commercial systems
         • Novel forms of user-centered evaluation

Papers will be peer reviewed by members of the 
workshop Programme Committee. They are...

Paul Clough                       University of Sheffield
Franciska de Jong                University of Twente
Thomas Deselaers                 RWTH Aachen University
Norbert Fuhr                      University of Duisburg
Gareth Jones                      Dublin City University
Jussi Karlgren                    Swedish Institute of Computer Science
Bernardo Magnini                  ITC-irst
Paul McNamee                      Johns Hopkins University
Henning Müller                    University & University Hospitals of Geneva
Stephen Robertson                Microsoft Research
Tetsuya Sakai                     NewsWatch Inc.
Diana Santos                      Linguateca  Sintef
Ellen Voorhees                    National 
Institute of Standards and Technology
Christa Womser-Hacker   University of Hildesheim
Maarten de Rijke                 University of Amsterdam
Frederic Gey                      U.C. Berkeley
Donna Harman                      National 
Institute of Standards and Technology
Noriko Kando                      National Institute of Informatics
Peter Schäuble                    Eurospider Information Technologies


SUBMISSION
Papers will be submitted as PDFs in ACM SIG Proceedings format

          <http://www.acm.org/sigs/publications/proceedings-templates>http://www.acm.org/sigs/publications/proceedings-templates

Submit final versions of paper to [email protected]



IMPORTANT DATES
Submission date Sunday 17 February
Notifications by 3 March
Final copy by 14 March
Workshop, 30 March



WEB SITE
http://www.trebleclef.eu/call_for_papers_ecir08.php


WORKSHOP ORGANISERS AND CONTACT DETAILS:
The Workshop Chair is Mark Sanderson. 
Co-organisers are Martin Braschler, Nicola Ferro and Julio Gonzalo.

The workshop will be sponsored by Treble-CLEF, a 
Coordination Action under 7FP which will promote 
R&D, evaluation and technology transfer in the 
multilingual information access domain.




Mark Sanderson
Reader in Information Retrieval

Room 225, Dept. of Information Studies
University of Sheffield, Regent Court
Portobello St, Sheffield, S1 4DP, UK
Tel: +44 (0) 114 22 22648, Fax: +44 (0) 114 27 80300
mailto:[email protected], http://dis.shef.ac.uk/mark/

Good judgement comes from experience, experience comes from bad judgement