QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

Kuldeep Singh* (Corresponding Author), Muhammad Saleem, Abhishek Nadgeri, Felix Conrads, Jeff Z. Pan, Axel Cyrille Ngonga Ngomo, Jens Lehmann

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

8 Citations (Scopus)
3 Downloads (Pure)


Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for microbenchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components. Resource Type: Evaluation benchmarks or Methods Repository: License: GNU General Public License v3.0

Original languageEnglish
Title of host publicationThe Semantic Web
Subtitle of host publication ISWC 2019
EditorsChiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtech Svátek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, Fabien Gandon
Number of pages16
ISBN (Electronic)978-3-030-30796-7
ISBN (Print)9783030307950
Publication statusE-pub ahead of print - 17 Oct 2019
EventInternational Semantic Web Conference 2019 - The University of Auckland, Auckland, New Zealand
Duration: 26 Oct 201930 Oct 2019
Conference number: 18

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11779 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceInternational Semantic Web Conference 2019
Abbreviated titleISWC 2019
Country/TerritoryNew Zealand
Internet address

Bibliographical note

This work has been supported by the project LIMBO (Grant no. 19F2029I), OPAL (no. 19F2028A), KnowGraphs (no. 860801), and SOLIDE (no. 13N14456)


Dive into the research topics of 'QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs'. Together they form a unique fingerprint.

Cite this