TASK DESCRIPTION

One critical but challenging problem in natural language understanding (NLU) is to develop a question answering system which could consistently understand and correctly answer general questions about the world. "Multi-choice Question Answering in Exams" is a typical question answering task that aims to test how accurately the participants could answer the questions in exams. The questions in this competition come from real exams. We believe that this challenge is an important step towards a rational, quantitative assessment of NLU’s capabilities.

This challenge collects multiple choice questions from a typical science and history curriculum. All questions are restrained within the elementary and middle school level. For each question, there are four possible answers, where each of them may be a word, a value, a phrase or even a sentence. The participants are required to select the best one from these four candidates. To answer these questions, participants could utilize any public toolkits and any resources on the Web, but manually annotation is not permitted. The following is an example.

DATA FORMAT/SIZE

In order to explore the influence of diversity of questions, we collect questions from seven subjects in two languages, including an English subset and a Chinese subset. The subjects of English subset contain biology, chemistry, physics, earth science and life science. And the subjects of Chinese subset contain biology and history. The total number of questions is 14,447.

Data Format

Each line of data is like:

id,”question”,”answerA”,”answerB”,”answerC”,”answerD”,correctAnswer

The meaning of each field is shown as follows.

• id - unique integer id for each question
• question - the question text
• answerA - the text for answer option A
• answerB - the text for answer option B
• answerC - the text for answer option C
• answerD - the text for answer option D
• correctAnswer - the correct answer (0,1,2,3, which correspond to four options. Only available for Train set and Valid Set)

Data Size

The dataset totally contains 14,447 multiple choice questions. In detail, English subset contains 5,367 questions and Chinese subset contains 9,080 questions. We randomly split the dataset into Train, Validation and Test sets. And more detail statistics is showed as follows.

Corpus

We encourage participants utilize any resources in the Internet, including software, toolbox, and corpus. There may be useful corpus shown as follows.

DATA LICENSE

We collect questions from Internet and all questions only can be used for this Task. It is forbidden to distribute this dataset and to use this dataset for commercial purposes.

RELATED WORK

Aristo is a system to answer a variety of science questions from standardized exams for students in multiple grade levels. It is developed by Allen Institute for Artificial Intelligence (AI2), and it integrates several promising solvers, including IR Solver, PMI Solver, SVM Solver, Rule-based solver, ILP solver. There are related materials at http://allenai.org/papers.html#aristo.

SCORER

This challenge employs the accuracy of a method on answering questions in test set as the metric, the accuracy is calculated as

English subset and Chinese subset are evaluated individually, and participants can submitted results for one or both two subsets.

BASELINE

We employ a simple retrieval based method as a baseline, and it is implemented based on Lucene which is an open-source information retrieval software library. We employ the method to build reverse-index on the whole Wikipedia dump for English questions and on the Baidu Baike corpus for Chinese questions.

This method scores pairs of the question and each of its option, the detail steps are shown as follows.

• Concatenate a question with an option as the query;
• Use Lucene to search relevant documents with the query;
• Score relevant documents by the similarity between the query q and the document d, noted as Sim(q,d)
• Choose at most three highest scores to calculate the score of the pair of the question and the option as



where n is less than or equals to 3 and if n = 0, score(q, a)=0;
• The pair with highest score(q,a) is the answer outputted by this method.