Dataset: Release of LETOR Benchmark Datasets for Learning to Rank by Microsoft Research Asia

LETOR, Benchmark Datasets for Learning to Rank

Version 1.0 is now available at
http://research.microsoft.com/users/tyliu/LETOR/


INTRODUCTION
LETOR is a package of benchmark datasets for learning to rank, released by
Microsoft Research Asia.  LETOR version 1.0 contains features extracted
from two datasets: OHSUMED, and TREC (TD2003 and TD2004). It also provides
basic documents, evaluation tools, and baseline evaluation results.
Ranking is the central problem for many applications, and using machine
learning technologies to learn the ranking function has been a promising
research direction. However, the lack of public benchmark datasets (e.g.
standard features, relevance judgments, data partitioning, and evaluation
metrics) makes the existing work difficult to be compared with each other.
To tackle this problem, in LETOR version 1.0, we extracted features for
each query-document pair in the OHSUMED and TREC collections (which are
widely used in the literature of information retrieval (IR)). Our extracted
features cover most of the 'standard' features in IR, including classical
features (such as term frequency, inverse document frequency, BM25 and
language models for IR), and the features proposed in SIGIR papers these
years (such as HostRank, Feature propagation and Topical PageRank). Note
that from these features, one cannot reconstruct the original documents in
the OHSUMED and TREC collections.
We benchmarked several state-of-the-arts ranking models with these features
and provided baseline results for future studies. We also released an
evaluation tool which can compute precision (P@n and MAP) and normalized
discount cumulative gain (NDCG), hoping that by using this single tool, the
experimental results of different methods can be easily and impartially
compared.
DOWNLOAD

The LETOR package can be downloaded from the website of Microsoft Research.

ABOUT

   l  The LETOR dataset was developed by the Information Retrieval and
   Mining group at Microsoft Research Asia. Contributors include Tie-Yan
   Liu, Jun Xu, Tao Qin, Wenying Xiong, Taifeng Wang and  Hang Li.
   l  If you find any problem in this dataset, please kindly let us know.
   We will upgrade it accordingly to fix these problems. Our goal is to
   make the dataset reliable and useful for the community. We plan to
   release more datasets for learning to rank in the future versions of
   LETOR. For further information, please visit this website, or send email
   to Tie-Yan Liu.