CLEF 2022: Call for Lab Proposals

-------------------------------------------
CLEF 2022
Conference and Labs of the Evaluation Forum
Bologna, Italy, September 5-8, 2022
http://clef2022.clef-initiative.eu
-------------------------------------------


Call for Lab Proposals

We invite researchers and practitioners from all areas of information
access as well as closely related fields to submit proposals for the
organization of an Evaluation Lab (a.k.a. "Shared Task”) as part of CLEF
2022.

The CLEF conference will be hosted by Università di Bologna on September
5-9, 2022, in Bologna, Italy (http://clef2022.clef-initiative.eu/), and
accepted labs will be organized within the twelve months beforehand.



Background

CLEF is a leading annual international conference organized by the CLEF
Initiative (http://www.clef-initiative.eu), which builds on a
twenty-three-year history of successful and diverse labs exploring the
evaluation of information access systems.

Its main mission is to promote research, innovation, and development of
information access systems by providing an infrastructure for:

- independent evaluation of information access systems;
- investigation of the use of unstructured, semi-structured,
highly-structured, and semantically enriched data in information access;
- creation of reusable test collections for benchmarking;
- exploration of new evaluation methodologies and innovative ways of using
experimental data;
- discussion of results, comparison of approaches, exchange of ideas, and
transfer of knowledge.

In particular, CLEF welcomes labs studying low-resource languages, as well
as multilingual and multimodal information access, albeit, this is not a
mandatory requirement for a successful proposal.



Scope of CLEF Labs

Proposals are accepted for two different types of labs:

- Shared-task-style labs. Labs of this type implement an evaluation of
specific information-access tasks, adopting the best practices of shared
tasks. A lab can consist of multiple different tasks.

- Workshop-style labs. Labs of this type organize a workshop with the goal
of exploring issues of evaluation methodology, metrics, processes, etc.,
or, as the first step toward creating new Shared-task-style labs.

CLEF is interested in receiving and facilitating innovative lab proposals.
Potential lab proposers who are unsure of the suitability of their lab
proposal or its format for inclusion at CLEF are encouraged to contact the
CLEF 2022 Lab Organizing Committee Chairs to discuss its suitability or
design at an early stage, as well as participate in the lab mentorship
programme.



Proposal Submission

Lab proposals must provide sufficient information to judge the relevance,
timeliness, scientific quality, benefits for the research community, and
the competence of the proposers to coordinate the lab. Each lab proposal
should identify one or more organisers as responsible for ensuring the
timely execution of the lab. Proposals should be 3 to 4 pages long and
should provide the following information:

1. Title of the proposed lab.

2. The planned format of the lab; i.e., shared-task-style or workshop-style.

3. A brief description of the lab topic and goals, its relevance to CLEF,
and the significance for the field.

4. A brief but clear statement of usage scenarios or domain to which the
activity is intended to contribute. For shared-task-style labs, include the
evaluation setup and metrics. For workshop-style labs, describe the planned
outputs of the workshop.

5. Details on the lab organiser(s), including (for shared-task-style labs)
identifying the task chair(s) responsible for ensuring the running of the
task(s). This should include details of any previous involvement in
organising or participating in similar evaluation activities at CLEF or
similar venues.

6. Is the lab a continuation of an activity from previous year(s) or a new
activity?
a) For activities continued from previous year(s): Statistics from previous
years (number of participants/runs for each task), a clear statement on why
another edition is needed, an explicit listing of the changes proposed, and
a discussion of lessons to be learned or insights to be made.
b) For new activities: A statement on why a new lab is needed and how the
community would benefit from the activity.

7. Details of the expected target audience; i.e. who do you expect to
participate in the task(s), and how do you propose to reach them.

8. Shared-task-style labs only: Brief details of tasks to be carried out in
the lab. The proposal should clearly motivate the need for each of the
proposed tasks and provide evidence of its capability of attracting
sufficient participation. The dataset(s) forming the basis of each task
needs to be described and motivated in the perspective of the goals of the
lab; also providing indications on how the dataset(s) will be shared is
useful. It is fine for a lab to have a single task, but labs often contain
multiple closely related tasks. Having more than 3 tasks needs a strong
motivation to avoid fragmentation.

9. Expected length of the lab session at the CLEF conference: half-day, one
day, two days. This should include very high-level details of the planned
structure of the session; e.g., participant presentations, invited
speaker(s), panels, etc., to justify the requested session length. For
workshop-style labs, indicate methods used to foster discussion, encourage
audience participation, and gather information.

10. Arrangements and other relevant information for the organization of the
lab. E.g., for shared-task-style labs, who will be responsible for
activities within the task; how will data be acquired or created, what
tools or methods will be used;  how will necessary queries be created or
relevance assessment carried out?

11. If the lab proposes to set up a steering committee to oversee and
advise its activities, include names, addresses, and homepage links of
people you propose to be involved.

The CLEF 2022 program will reserve about half of the conference schedule
for lab sessions. The labs will present their overall results in "overview
presentations" during the plenary scientific paper sessions to allow
non-participants in the lab to get a sense of where the research frontiers
are moving.

Important considerations:
- Organisers new to CLEF, or shared-tasks in general, are highly encouraged
to first organise a workshop-style lab to discuss the format, the problem
space, and the practicalities of their envisioned future shared task.
- Organisers of each lab are expected to chair sessions for their lab at
the conference.
- Within these sessions, ample time should be given for general discussion
and engagement by all participants---not just those presenting results and
papers. For instance, this may include keynotes, panels, demos, poster
sessions, discussion rounds, etc., as appropriate.
- Organizers of accepted labs are expected to advertise their labs at both
CLEF 2021 (21-24 September 2021, Bucharest, Romania) and ECIR 2022 (10-14
April 2022, Stavanger, Norway). At least one lab representative should
participate in these events.
- Advertising at CLEF 2021 will consist of running a break-out session to
discuss the lab with prospective participants, and advertising/announcing
it during the closing session.
- Advertising at ECIR 2022 will consist of submitting a short lab
description to be included in the ECIR 2022 proceedings (submission by
October 7, 2021), advertising the lab in a booster session during ECIR
2022, and presenting a poster about the lab.

Lab proposals have to be submitted at the following address:
https://easychair.org/conferences/?conf=clef2022
choosing the "Lab Proposals" track.

Examples of previous labs can be found at
http://clef2021.clef-initiative.eu/index.php?page=Pages/labs.html.



Reviewing Process

Each submitted proposal will be reviewed by the CLEF 2021 Lab Organizing
Committee. The decision will be sent by email to the responsible organiser
by July 29, 2021. The final length of the lab session will be determined
based on the overall organization of the conference and the number of
participant submissions received by a lab.



Mentorship Program for Lab Proposals from Newcomers

As in previous editions, CLEF 2022 will run a mentorship program to support
the preparation of lab proposals for newcomers to CLEF. We encourage
newcomers to refer to Friedberg et al. (2015) for initial guidance on
preparing their proposal: Friedberg I, Wass MN, Mooney SD, Radivojac P. Ten
simple rules for a community computational challenge. PLoS Comput Biol.
2015 Apr 23;11(4):e1004150.

The CLEF newcomers mentoring program offers help, guidance, and feedback on
writing the proposal draft by assigning a mentor, helping with preparing
and maturing the proposal for submission. If your proposal falls into the
scope of an already existing CLEF lab, the mentor will make contact with
the respective lab organisers so that the lab’s teams can join forces.

Lab proposals for mentorship have to be submitted at the following address:
https://easychair.org/conferences/?conf=clef2022
choosing the "Lab Mentorship" track.



Important Dates

This year’s schedule for the lab proposal submission is as follows

- Requests for mentorship submission (only newcomers): 10 May 2021
- Mentorship period starts: 10 May 2021
- Lab proposals submission (both newcomers and veterans): 1 July 2021
- Notification of lab acceptance: 29 July 2021
- Advertising labs at CLEF 2021, Bucharest, Romania: 21-24 September 2021
- Submission of short lab description for ECIR 2022: 7 October 2021
- Lab registration opens: 1 December 2021
- Advertising labs at ECIR 2022: 10-14 April 2021, Stavenger, Norway


CLEF 2022 Lab Chairs
- Allan Hanbury, TU Wien, Austria
- Martin Potthast, Leipzig University, Germany


CLEF 2022 Lab Mentorship Chair
- Paolo Rosso, Universitat Politècnica de València, Spain

########################################################################

To unsubscribe from the IR list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=IR&A=1

This message was issued to members of www.jiscmail.ac.uk/IR, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/