Quantified Reproducibility Assessment of NLP Results

Anja Belz, Maja Popovic, Simon Mille

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

22 Citations (Scopus)

Abstract

This paper describes and tests a method for carrying out quantified reproducibility assessment (QRA) that is based on concepts and definitions from metrology. QRA produces a single score estimating the degree of reproducibility of a given system and evaluation measure, on the basis of the scores from, and differences between, different reproductions. We test QRA on 18 different system and evaluation measure combinations (involving diverse NLP tasks and types of evaluation), for each of which we have the original results and one to seven reproduction results. The proposed QRA method produces degree-of-reproducibility scores that are comparable across multiple reproductions not only of the same, but also of different, original studies. We find that the proposed method facilitates insights into causes of variation between reproductions, and as a result, allows conclusions to be drawn about what aspects of system and/or evaluation design need to be changed in order to improve reproducibility.
Original languageEnglish
Title of host publicationProceedings of the 60th Annual Meeting of the Association for Computational Linguistics
Place of PublicationDublin, Ireland
PublisherAssociation for Computational Linguistics
Pages16-28
Number of pages13
Volume1
EditionLong Papers
ISBN (Electronic)978-1-955917-21-6
DOIs
Publication statusPublished - 24 May 2022
Externally publishedYes
EventACL 2022: 60th Annual Meeting of the Association for Computational Linguistics - The Convention Centre Dublin , Dublin, Ireland
Duration: 22 May 202227 May 2022
Conference number: 60
https://www.2022.aclweb.org/

Conference

ConferenceACL 2022
Abbreviated titleACL
Country/TerritoryIreland
CityDublin
Period22/05/2227/05/22
Internet address

Bibliographical note

Anya Belz, Maja Popovic, and Simon Mille. 2022. Quantified Reproducibility Assessment of NLP Results. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16–28, Dublin, Ireland. Association for Computational Linguistics.

Fingerprint

Dive into the research topics of 'Quantified Reproducibility Assessment of NLP Results'. Together they form a unique fingerprint.

Cite this