Against a background of growing interest in reproducibility in NLP and ML, and as part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the second shared task on reproducibility of evaluations in NLG, ReproGen 2022. This paper describes the shared task, summarises results from the reproduction studies submitted, and provides further comparative analysis of the results. Out of six initial team registrations, we received submissions from five teams. Meta-analysis of the five reproduction studies revealed varying degrees of reproducibility, and allowed further tentative conclusions about what types of eval- uation tend to have better reproducibility.
|Title of host publication||Proceedings of the 15th International Conference on Natural Language Generation|
|Subtitle of host publication||Generation Challenges|
|Place of Publication||Waterville, Maine, USA and virtual meeting|
|Publisher||Association for Computational Linguistics|
|Number of pages||9|
|Publication status||Published - 1 Jul 2022|
|Event||15th International Natural Language Generation Conference: Generation Challenges - Colby College, Waterville, United States|
Duration: 18 Jul 2022 → 22 Jul 2022
|Conference||15th International Natural Language Generation Conference|
|Abbreviated title||INLG 2022|
|Period||18/07/22 → 22/07/22|
Bibliographical noteWe thank the authors of the five original papers that were up for reproduction in Track A. And of course the authors of the reproduction papers, without whom there would be no shared task. Our work was carried out as part of the ReproHum project on Investigating Reproducibility of Human Evaluations in Natural Language Processing, funded by EPSRC (UK) under grant number EP/V05645X/1. Popovic’s work is directly funded by the ADAPT ´SFI Centre for Digital Media Technology which is funded by Science Foundation Ireland through
the SFI Research Centres Programme and is cofunded under the European Regional Development Fund (ERDF) through Grant 13/RC/2106. Both Popovic and Belz also benefit in other ways from ´being members of ADAPT.