Why New Datasets are Needed for Deep Learning-Enhanced IR

This section critiques existing information retrieval benchmarks, noting their lack of web-scale data, highly-skewed multilingual queries, and rich multi-modal information for advanced AI research.

Table of Links

Abstract and 1 Introduction

2 Background and Related work

2.1 Web Scale Information Retrieval

2.2 Existing Datasets

3 MS Marco Web Search Dataset and 3.1 Document Preparation

3.2 Query Selection and Labeling

3.3 Dataset Analysis

3.4 New Challenges Raised by MS MARCO Web Search

4 Benchmark Results and 4.1 Environment Setup

4.2 Baseline Methods

4.3 Evaluation Metrics

4.4 Evaluation of Embedding Models and 4.5 Evaluation of ANN Algorithms

4.6 Evaluation of End-to-end Performance

5 Potential Biases and Limitations

6 Future Work and Conclusions, and References

2.2 Existing Datasets

To encourage innovation in the information retrieval area, the community has collected several datasets for public benchmarking ...