RCIR (Reading Comprehension in Information Retrieval) Task at NTCIR-16

====================================
CALL FOR TASK PARTICIPATION: RCIR (Reading Comprehension in Information Retrieval) at NTCIR-16
====================================
 
Registration deadline:  1 Dec 2021
Paper due: 1 Mar 2022
Camera ready due: 1 May 2022
NTCIR-15 Conference: 14-17 June 2022 (NII, Tokyo, Japan) (remote online presentation option available)
 
RCIR (Reading Comprehension in Information Retrieval) is an NTCIR-16 pilot task. The RCIR pilot task aims to motivate the development of a first generation of personalised information retrieval techniques that integrate reading comprehension measures from eye trackers as a source of evidence for building personalised information retrieval models. Participating researchers in RCIR will develop and benchmark approaches to integrate multi-modal signals (e.g. eye tracking, screenshots, etc) into the retrieval process for two sub-tasks: a) a comprehension-evaluation task (CET) that aims to sort text documents in terms of comprehension levels, and b) a comprehension-based retrieval task (CRT) that aims to rank text documents (for a variety of topics) by integrating text comprehension-evidence into the ranking model. Both of these sub-tasks are exploratory in nature, but are designed to facilitate pioneering experimentation on the topic by the IR community.
 
Additionally we are especially interested in papers that apply novel or speculative approaches on the RCIR dataset e.g. for insight generation, interpretability, etc. Papers exploring such aspects of the RCIR dataset are also highly encouraged as part of the Insights sub-task.
A novel eye-tracking IR dataset has been generated by the organisers and consists of text documents selected from the RACE dataset (approximately 1000 content texts), measures of subjects’ comprehension when reading the text, as well as a range of preprocessed signal data e.g. eye movement features. We provide a working baseline approach,  a training set and pre-extracted features to reduce barriers to participation in this novel task.
There are three sub-tasks in the pilot RCIR task:
A) Comprehension-evaluation sub-Task (CET): that aims to sort texts in terms of comprehension levels.
B) Comprehension-based retrieval  sub-Task (CRT): that aims to rank texts (for a variety of topics) by integrating text comprehension-evidence into the IR process.
C) Insights  sub-Task (IT): that aims to explore alternative uses/analyses/approaches with the dataset.
Participating organisations will be expected to submit a paper outlining their approach that they will then present at the RCIR workshop in Tokyo, Japan in June 2022. Presentations may be physical or virtual.
To get access to the data/participate in the evaluation task, please complete the signup procedure via the NTCIR-16 website and contact the task organiser Graham Healy ([email protected]) for dataset access: http://research.nii.ac.jp/ntcir/ntcir-16/howto.html
For more information please visit: http://ntcir-rcir.computing.dcu.ie/
 
Important Dates:
15 Oct 2021 - Dataset release
1 Dec 2021 - Task registration due
28 Feb 2022 - Formal runs due
1 Mar 2022 - Participant papers due
1 May 2022 - Camera ready paper due
14-17 June 2022 - NTCIR-16 in NII, Tokyo
 
Regards,
Graham Healy

########################################################################

To unsubscribe from the IR list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=IR&A=1

This message was issued to members of www.jiscmail.ac.uk/IR, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/