A Systematic Review of Reproducibility Research in Natural Language Processing

Anya  Belz; Anastasia  Shimorina; Shubham  Agarwal; Ehud Reiter

doi:10.18653/v1/2021.eacl-main.29

A Systematic Review of Reproducibility Research in Natural Language Processing

Anya Belz, Anastasia Shimorina, Shubham Agarwal, Ehud Reiter

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

30 Citations (Scopus)

Abstract

Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

Original language	English
Title of host publication	Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics
Subtitle of host publication	Main Volume
Publisher	ACL Anthology
Pages	381-393
Number of pages	13
ISBN (Electronic)	978-1-954085-02-2
DOIs	https://doi.org/10.18653/v1/2021.eacl-main.29
Publication status	Published - 23 Apr 2021
Event	EACL 2021 : 16th Conference of the European Chapter of the Association for Computational Linguistics - Virtual event Duration: 19 Apr 2021 → 23 Apr 2021 https://2021.eacl.org/

Conference

Conference	EACL 2021
Abbreviated title	EACL21
Period	19/04/21 → 23/04/21
Internet address	https://2021.eacl.org/

Bibliographical note

Acknowledgments
We thank our reviewers for their valuable feedback. Shubham Agarwal’s PhD fees are supported by Adeptmind Inc., Toronto, Canada.

Access to Document

10.18653/v1/2021.eacl-main.29Licence: CC BY

Belz_etal_EACL2021_A-Systematic_Review_VOR
The ACL materials that are hosted in the Anthology are licensed to the general public under a liberal usage policy that allows unlimited reproduction, distribution and hosting of materials on any other website or medium, for non-commercial purposes. Prior to 2016, all ACL materials are licensed using the Creative Commons 3.0 BY-NC-SA (Attribution, Non-Commercial, Share-Alike) license. As of 2016, this policy has been relaxed further, and all subsequent materials are available to the general public on the terms of the Creative Commons 4.0 BY (Attribution) license; this means both commercial and non-commercial use is explicitly licensed to all. https://creativecommons.org/licenses/by/4.0/
Final published version, 393 KBLicence: CC BY

https://arxiv.org/abs/2103.07929Licence: CC BY

Cite this

A Systematic Review of Reproducibility Research in Natural Language Processing. / Belz, Anya ; Shimorina, Anastasia ; Agarwal, Shubham et al.
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. ACL Anthology, 2021. p. 381-393.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

@inproceedings{24335b89bfb449a185cc94483cc1fa8c,

title = "A Systematic Review of Reproducibility Research in Natural Language Processing",

abstract = "Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.",

author = "Anya Belz and Anastasia Shimorina and Shubham Agarwal and Ehud Reiter",

note = "Acknowledgments We thank our reviewers for their valuable feedback. Shubham Agarwal{\textquoteright}s PhD fees are supported by Adeptmind Inc., Toronto, Canada.; EACL 2021 : 16th Conference of the European Chapter of the Association for Computational Linguistics, EACL21 ; Conference date: 19-04-2021 Through 23-04-2021",

year = "2021",

month = apr,

day = "23",

doi = "10.18653/v1/2021.eacl-main.29",

language = "English",

pages = "381--393",

booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics",

publisher = "ACL Anthology",

url = "https://2021.eacl.org/",

}

TY - GEN

T1 - A Systematic Review of Reproducibility Research in Natural Language Processing

AU - Belz, Anya

AU - Shimorina, Anastasia

AU - Agarwal, Shubham

AU - Reiter, Ehud

N1 - Acknowledgments We thank our reviewers for their valuable feedback. Shubham Agarwal’s PhD fees are supported by Adeptmind Inc., Toronto, Canada.

PY - 2021/4/23

Y1 - 2021/4/23

N2 - Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

AB - Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

U2 - 10.18653/v1/2021.eacl-main.29

DO - 10.18653/v1/2021.eacl-main.29

M3 - Published conference contribution

SP - 381

EP - 393

BT - Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics

PB - ACL Anthology

T2 - EACL 2021

Y2 - 19 April 2021 through 23 April 2021

ER -

A Systematic Review of Reproducibility Research in Natural Language Processing

Abstract

Conference

Bibliographical note

Access to Document

Fingerprint

Cite this