Sherlock: a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure

Chenghua Lin; Dong  Liu; Wei Pang; Zhe Wang

doi:10.1007/s12559-015-9347-7

Sherlock: a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure

Chenghua Lin, Dong Liu, Wei Pang, Zhe Wang

Research output: Contribution to journal › Article › peer-review

24 Citations (Scopus)

15 Downloads (Pure)

Abstract

In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 % gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.

Original language	English
Pages (from-to)	667-679
Number of pages	13
Journal	Cognitive Computation
Volume	7
Issue number	6
Early online date	4 Aug 2015
DOIs	https://doi.org/10.1007/s12559-015-9347-7
Publication status	Published - Dec 2015

Bibliographical note

Acknowledgments
This work is supported by the BBC Connected Studio programme (http://www.bbc.co.uk/partnersandsuppliers/con nectedstudio/), the award made by the RCUK Digital Economy theme
to the dot.rural Digital Economy Hub; award reference EP/G066051/1, the award made by UK Economic & Social Research Council (ESRC); award reference ES/M001628/1, National Natural Science Foundation of China (NSFC) under Grant No. 61373051, and the China National Science and Technology Pillar Program (Grant No. 2013BAH07F05). The authors would like to thank Ryan Hussey for the work on the user interface design and Tom Cass and James Ruston for the help in developing the Sherlock application. We are also grateful to Herm Baskerville for creating the editorial quizzes and Nava Tintarev for many helpful discussions on the human evaluation.

Keywords

quiz generation
linked data
RDF
educational games
semantic similarity
text analytics

Access to Document

10.1007/s12559-015-9347-7Licence: CC BY

Sherlock
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Final published version, 2.2 MBLicence: CC BY

Cite this

@article{06c71a7aedec4d95846c5ae5e80a3283,

title = "Sherlock: a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure",

abstract = "In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 % gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.",

keywords = "quiz generation, linked data, RDF, educational games, semantic similarity, text analytics",

author = "Chenghua Lin and Dong Liu and Wei Pang and Zhe Wang",

note = "Acknowledgments This work is supported by the BBC Connected Studio programme (http://www.bbc.co.uk/partnersandsuppliers/con nectedstudio/), the award made by the RCUK Digital Economy theme to the dot.rural Digital Economy Hub; award reference EP/G066051/1, the award made by UK Economic & Social Research Council (ESRC); award reference ES/M001628/1, National Natural Science Foundation of China (NSFC) under Grant No. 61373051, and the China National Science and Technology Pillar Program (Grant No. 2013BAH07F05). The authors would like to thank Ryan Hussey for the work on the user interface design and Tom Cass and James Ruston for the help in developing the Sherlock application. We are also grateful to Herm Baskerville for creating the editorial quizzes and Nava Tintarev for many helpful discussions on the human evaluation.",

year = "2015",

month = dec,

doi = "10.1007/s12559-015-9347-7",

language = "English",

volume = "7",

pages = "667--679",

journal = "Cognitive Computation",

issn = "1866-9956",

publisher = "Springer New York",

number = "6",

}

TY - JOUR

T1 - Sherlock

T2 - a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure

AU - Lin, Chenghua

AU - Liu, Dong

AU - Pang, Wei

AU - Wang, Zhe

N1 - Acknowledgments This work is supported by the BBC Connected Studio programme (http://www.bbc.co.uk/partnersandsuppliers/con nectedstudio/), the award made by the RCUK Digital Economy theme to the dot.rural Digital Economy Hub; award reference EP/G066051/1, the award made by UK Economic & Social Research Council (ESRC); award reference ES/M001628/1, National Natural Science Foundation of China (NSFC) under Grant No. 61373051, and the China National Science and Technology Pillar Program (Grant No. 2013BAH07F05). The authors would like to thank Ryan Hussey for the work on the user interface design and Tom Cass and James Ruston for the help in developing the Sherlock application. We are also grateful to Herm Baskerville for creating the editorial quizzes and Nava Tintarev for many helpful discussions on the human evaluation.

PY - 2015/12

Y1 - 2015/12

N2 - In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 % gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.

AB - In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 % gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.

KW - quiz generation

KW - linked data

KW - RDF

KW - educational games

KW - semantic similarity

KW - text analytics

U2 - 10.1007/s12559-015-9347-7

DO - 10.1007/s12559-015-9347-7

M3 - Article

SN - 1866-9956

VL - 7

SP - 667

EP - 679

JO - Cognitive Computation

JF - Cognitive Computation

IS - 6

ER -

Sherlock: a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure

Abstract

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this