Sherlock: a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure

Chenghua Lin, Dong Liu, Wei Pang, Zhe Wang

Research output: Contribution to journalArticlepeer-review

21 Citations (Scopus)
11 Downloads (Pure)


In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 % gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.
Original languageEnglish
Pages (from-to)667-679
Number of pages13
JournalCognitive Computation
Issue number6
Early online date4 Aug 2015
Publication statusPublished - Dec 2015

Bibliographical note

This work is supported by the BBC Connected Studio programme ( nectedstudio/), the award made by the RCUK Digital Economy theme
to the dot.rural Digital Economy Hub; award reference EP/G066051/1, the award made by UK Economic & Social Research Council (ESRC); award reference ES/M001628/1, National Natural Science Foundation of China (NSFC) under Grant No. 61373051, and the China National Science and Technology Pillar Program (Grant No. 2013BAH07F05). The authors would like to thank Ryan Hussey for the work on the user interface design and Tom Cass and James Ruston for the help in developing the Sherlock application. We are also grateful to Herm Baskerville for creating the editorial quizzes and Nava Tintarev for many helpful discussions on the human evaluation.


  • quiz generation
  • linked data
  • RDF
  • educational games
  • semantic similarity
  • text analytics


Dive into the research topics of 'Sherlock: a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure'. Together they form a unique fingerprint.

Cite this