QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

Kuldeep Singh; Muhammad Saleem; Abhishek Nadgeri; Felix Conrads; Jeff Z. Pan; Axel Cyrille Ngonga Ngomo; Jens Lehmann

doi:10.1007/978-3-030-30796-7_18

QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

Kuldeep Singh^* (Corresponding Author), Muhammad Saleem, Abhishek Nadgeri, Felix Conrads, Jeff Z. Pan, Axel Cyrille Ngonga Ngomo, Jens Lehmann

^*Corresponding author for this work

Computing Science

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

8 Citations (Scopus)

3 Downloads (Pure)

Abstract

Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for microbenchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components. Resource Type: Evaluation benchmarks or Methods Repository: https://github.com/dice-group/qald-generator License: GNU General Public License v3.0

Original language	English
Title of host publication	The Semantic Web
Subtitle of host publication	ISWC 2019
Editors	Chiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtech Svátek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, Fabien Gandon
Publisher	Springer
Pages	277-292
Number of pages	16
ISBN (Electronic)	978-3-030-30796-7
ISBN (Print)	9783030307950
DOIs	https://doi.org/10.1007/978-3-030-30796-7_18
Publication status	E-pub ahead of print - 17 Oct 2019
Event	International Semantic Web Conference 2019 - The University of Auckland, Auckland, New Zealand Duration: 26 Oct 2019 → 30 Oct 2019 Conference number: 18 https://link.springer.com/book/10.1007/978-3-030-30793-6 https://iswc2019.semanticweb.org/

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11779 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	International Semantic Web Conference 2019
Abbreviated title	ISWC 2019
Country/Territory	New Zealand
City	Auckland
Period	26/10/19 → 30/10/19
Internet address	https://link.springer.com/book/10.1007/978-3-030-30793-6 https://iswc2019.semanticweb.org/

Bibliographical note

This work has been supported by the project LIMBO (Grant no. 19F2029I), OPAL (no. 19F2028A), KnowGraphs (no. 860801), and SOLIDE (no. 13N14456)

Access to Document

10.1007/978-3-030-30796-7_18Licence: GNU GPL

Ghidini_etal_ISWC__The_Semantic_Web_VOR
© Springer Nature Switzerland AG 2019 License: GNU General Public License v3.0 The chapter “The SEPSES Knowledge Graph: An Integrated Resource for Cybersecurity” is Open Access. This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). For further details see license information in the chapter. This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
Final published version, 43.9 MBLicence: GNU GPL

Cite this

Singh, K., Saleem, M., Nadgeri, A., Conrads, F., Pan, J. Z., Ngomo, A. C. N., & Lehmann, J. (2019). QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. In C. Ghidini, O. Hartig, M. Maleshkova, V. Svátek, I. Cruz, A. Hogan, J. Song, M. Lefrançois, & F. Gandon (Eds.), The Semantic Web: ISWC 2019 (pp. 277-292). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11779 LNCS). Springer . https://doi.org/10.1007/978-3-030-30796-7_18

QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. / Singh, Kuldeep (Corresponding Author); Saleem, Muhammad; Nadgeri, Abhishek et al.
The Semantic Web: ISWC 2019. ed. / Chiara Ghidini; Olaf Hartig; Maria Maleshkova; Vojtech Svátek; Isabel Cruz; Aidan Hogan; Jie Song; Maxime Lefrançois; Fabien Gandon. Springer , 2019. p. 277-292 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11779 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Singh, K, Saleem, M, Nadgeri, A, Conrads, F, Pan, JZ, Ngomo, ACN & Lehmann, J 2019, QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. in C Ghidini, O Hartig, M Maleshkova, V Svátek, I Cruz, A Hogan, J Song, M Lefrançois & F Gandon (eds), The Semantic Web: ISWC 2019. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11779 LNCS, Springer , pp. 277-292, International Semantic Web Conference 2019, Auckland, New Zealand, 26/10/19. https://doi.org/10.1007/978-3-030-30796-7_18

Singh K, Saleem M, Nadgeri A, Conrads F, Pan JZ, Ngomo ACN et al. QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. In Ghidini C, Hartig O, Maleshkova M, Svátek V, Cruz I, Hogan A, Song J, Lefrançois M, Gandon F, editors, The Semantic Web: ISWC 2019. Springer . 2019. p. 277-292. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2019 Oct 17. doi: 10.1007/978-3-030-30796-7_18

Singh, Kuldeep ; Saleem, Muhammad ; Nadgeri, Abhishek et al. / QaldGen : Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. The Semantic Web: ISWC 2019. editor / Chiara Ghidini ; Olaf Hartig ; Maria Maleshkova ; Vojtech Svátek ; Isabel Cruz ; Aidan Hogan ; Jie Song ; Maxime Lefrançois ; Fabien Gandon. Springer , 2019. pp. 277-292 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{13cbe2b618ba40bf8f8b5ef5e7a8b65c,

title = "QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs",

abstract = "Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for microbenchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components. Resource Type: Evaluation benchmarks or Methods Repository: https://github.com/dice-group/qald-generator License: GNU General Public License v3.0",

author = "Kuldeep Singh and Muhammad Saleem and Abhishek Nadgeri and Felix Conrads and Pan, {Jeff Z.} and Ngomo, {Axel Cyrille Ngonga} and Jens Lehmann",

note = "This work has been supported by the project LIMBO (Grant no. 19F2029I), OPAL (no. 19F2028A), KnowGraphs (no. 860801), and SOLIDE (no. 13N14456); International Semantic Web Conference 2019, ISWC 2019 ; Conference date: 26-10-2019 Through 30-10-2019",

year = "2019",

month = oct,

day = "17",

doi = "10.1007/978-3-030-30796-7_18",

language = "English",

isbn = "9783030307950",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer ",

pages = "277--292",

editor = "Chiara Ghidini and Olaf Hartig and Maria Maleshkova and Vojtech Sv{\'a}tek and Isabel Cruz and Aidan Hogan and Jie Song and Maxime Lefran{\c c}ois and Fabien Gandon",

booktitle = "The Semantic Web",

url = "https://link.springer.com/book/10.1007/978-3-030-30793-6, https://iswc2019.semanticweb.org/",

}

TY - GEN

T1 - QaldGen

T2 - International Semantic Web Conference 2019

AU - Singh, Kuldeep

AU - Saleem, Muhammad

AU - Nadgeri, Abhishek

AU - Conrads, Felix

AU - Pan, Jeff Z.

AU - Ngomo, Axel Cyrille Ngonga

AU - Lehmann, Jens

N1 - Conference code: 18

PY - 2019/10/17

Y1 - 2019/10/17

N2 - Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for microbenchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components. Resource Type: Evaluation benchmarks or Methods Repository: https://github.com/dice-group/qald-generator License: GNU General Public License v3.0

AB - Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for microbenchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components. Resource Type: Evaluation benchmarks or Methods Repository: https://github.com/dice-group/qald-generator License: GNU General Public License v3.0

UR - http://www.scopus.com/inward/record.url?scp=85081092804&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-30796-7_18

DO - 10.1007/978-3-030-30796-7_18

M3 - Published conference contribution

AN - SCOPUS:85081092804

SN - 9783030307950

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 277

EP - 292

BT - The Semantic Web

A2 - Ghidini, Chiara

A2 - Hartig, Olaf

A2 - Maleshkova, Maria

A2 - Svátek, Vojtech

A2 - Cruz, Isabel

A2 - Hogan, Aidan

A2 - Song, Jie

A2 - Lefrançois, Maxime

A2 - Gandon, Fabien

PB - Springer

Y2 - 26 October 2019 through 30 October 2019

ER -

QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this