Abstract

In addition to machines answering questions, question answering (qa) research creates interesting, challenging questions that reveal the best systems. We argue that creating a qa dataset—and its ubiquitous leaderboard—closely resembles running a trivia tournament: you write questions, have agents—humans or machines—answer questions, and declare a winner. However, the research community has ignored the lessons from decades of the trivia community creating vibrant, fair, and effective qa competitions. After detailing problems with existing qa datasets, we outline several lessons that transfer to qa research: removing ambiguity, discriminating skill, and adjudicating disputes.