GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Sebastian Gehrmann; Abhik Bhattacharjee; Abinaya Mahendiran; Alex Wang; Alexandros Papangelis; Aman Madaan; Angelina McMillan-Major; Anna Shvets; Ashish Upadhyay; Bernd Bohnet; Bingsheng Yao; Bryan Wilie; Chandra Bhagavatula; Chaobin You; Craig Thomson; Cristina Garbacea; Dakuo Wang; Daniel Deutsch; Deyi Xiong; Di Jin; Dimitra Gkatzia; Dragomir Radev; Elizabeth Clark; Esin Durmus; Faisal Ladhak; Filip Ginter; Genta Indra Winata; Hendrik Strobelt; Hiroaki Hayashi; Jekaterina Novikova; Jenna Kanerva; Jenny Chim; Jiawei Zhou; Jordan Clive; Joshua Maynez; João Sedoc; Juraj Juraska; Kaustubh Dhole; Khyathi Raghavi Chandu; Laura Perez-Beltrachini; Leonardo F.R. Ribeiro; Lewis Tunstall; Li Zhang; Mahima Pushkarna; Mathias Creutz; Michael White; Mihir Sanjay Kale; Moussa Kamal Eddine; Nico Daheim; Nishant Subramani; Ondrej Dusek; Paul Pu Liang; Pawan Sasanka Ammanamanchi; Qi Zhu; Ratish Puduppully; Reno Kriz; Rifat Shahriyar; Ronald Cardenas; Saad Mahamood; Salomey Osei; Samuel Cahyawijaya; Sanja Štajner; Sebastien Montella; Shailza Jolly; Simon Mille; Tahmid Hasan; Tianhao Shen; Tosin Adewumi; Vikas Raunak; Vipul Raheja; Vitaly Nikolaev; Vivian Tsai; Yacine Jernite; Ying Xu; Yisi Sang; Yixin Liu; Yufang Hou

doi:10.18653/v1/2022.emnlp-demos.27

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Sebastian Gehrmann
, Abhik Bhattacharjee
, Abinaya Mahendiran
, Alex Wang
, Alexandros Papangelis
, Aman Madaan
, Angelina McMillan-Major
, Anna Shvets
, Ashish Upadhyay
, Bernd Bohnet
, Bingsheng Yao
, Bryan Wilie
, Chandra Bhagavatula
, Chaobin You
, Craig Thomson
, Cristina Garbacea
, Dakuo Wang
, Daniel Deutsch
, Deyi Xiong
, Di Jin

Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez-Beltrachini, Leonardo F.R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou

Computing Science

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

23 Citations (Scopus)

3 Downloads (Pure)

Abstract

Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.

Original language	English
Title of host publication	EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing
Subtitle of host publication	Proceedings of the Demonstrations Session
Editors	Wanxiang Che, Ekaterina Shutova
Publisher	Association for Computational Linguistics (ACL)
Pages	266-281
Number of pages	16
ISBN (Electronic)	9781959429418
DOIs	https://doi.org/10.18653/v1/2022.emnlp-demos.27
Publication status	Published - 11 Dec 2022
Event	2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 - Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022

Conference

Conference	2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	7/12/22 → 11/12/22

Access to Document

10.18653/v1/2022.emnlp-demos.27Licence: CC BY

Gehrman_etal_EMNLP_GEMv2_VOR
Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. https://creativecommons.org/licenses/by/4.0/
Final published version, 1.45 MBLicence: CC BY

Cite this

Gehrmann, S., Bhattacharjee, A., Mahendiran, A., Wang, A., Papangelis, A., Madaan, A., McMillan-Major, A., Shvets, A., Upadhyay, A., Bohnet, B., Yao, B., Wilie, B., Bhagavatula, C., You, C., Thomson, C., Garbacea, C., Wang, D., Deutsch, D., Xiong, D., ... Hou, Y. (2022). GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. In W. Che, & E. Shutova (Eds.), EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing: Proceedings of the Demonstrations Session (pp. 266-281). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.emnlp-demos.27

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. / Gehrmann, Sebastian; Bhattacharjee, Abhik; Mahendiran, Abinaya et al.
EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing: Proceedings of the Demonstrations Session. ed. / Wanxiang Che; Ekaterina Shutova. Association for Computational Linguistics (ACL), 2022. p. 266-281.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Gehrmann, S, Bhattacharjee, A, Mahendiran, A, Wang, A, Papangelis, A, Madaan, A, McMillan-Major, A, Shvets, A, Upadhyay, A, Bohnet, B, Yao, B, Wilie, B, Bhagavatula, C, You, C, Thomson, C, Garbacea, C, Wang, D, Deutsch, D, Xiong, D, Jin, D, Gkatzia, D, Radev, D, Clark, E, Durmus, E, Ladhak, F, Ginter, F, Winata, GI, Strobelt, H, Hayashi, H, Novikova, J, Kanerva, J, Chim, J, Zhou, J, Clive, J, Maynez, J, Sedoc, J, Juraska, J, Dhole, K, Chandu, KR, Perez-Beltrachini, L, Ribeiro, LFR, Tunstall, L, Zhang, L, Pushkarna, M, Creutz, M, White, M, Kale, MS, Eddine, MK, Daheim, N, Subramani, N, Dusek, O, Liang, PP, Ammanamanchi, PS, Zhu, Q, Puduppully, R, Kriz, R, Shahriyar, R, Cardenas, R, Mahamood, S, Osei, S, Cahyawijaya, S, Štajner, S, Montella, S, Jolly, S, Mille, S, Hasan, T, Shen, T, Adewumi, T, Raunak, V, Raheja, V, Nikolaev, V, Tsai, V, Jernite, Y, Xu, Y, Sang, Y, Liu, Y & Hou, Y 2022, GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. in W Che & E Shutova (eds), EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing: Proceedings of the Demonstrations Session. Association for Computational Linguistics (ACL), pp. 266-281, 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, 7/12/22. https://doi.org/10.18653/v1/2022.emnlp-demos.27

Gehrmann S, Bhattacharjee A, Mahendiran A, Wang A, Papangelis A, Madaan A et al. GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. In Che W, Shutova E, editors, EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing: Proceedings of the Demonstrations Session. Association for Computational Linguistics (ACL). 2022. p. 266-281 doi: 10.18653/v1/2022.emnlp-demos.27

Gehrmann, Sebastian ; Bhattacharjee, Abhik ; Mahendiran, Abinaya et al. / GEMv2 : Multilingual NLG Benchmarking in a Single Line of Code. EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing: Proceedings of the Demonstrations Session. editor / Wanxiang Che ; Ekaterina Shutova. Association for Computational Linguistics (ACL), 2022. pp. 266-281

@inproceedings{2c6bc03c3f254edcabe32e7b8b260096,

title = "GEMv2: Multilingual NLG Benchmarking in a Single Line of Code",

abstract = "Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.",

author = "Sebastian Gehrmann and Abhik Bhattacharjee and Abinaya Mahendiran and Alex Wang and Alexandros Papangelis and Aman Madaan and Angelina McMillan-Major and Anna Shvets and Ashish Upadhyay and Bernd Bohnet and Bingsheng Yao and Bryan Wilie and Chandra Bhagavatula and Chaobin You and Craig Thomson and Cristina Garbacea and Dakuo Wang and Daniel Deutsch and Deyi Xiong and Di Jin and Dimitra Gkatzia and Dragomir Radev and Elizabeth Clark and Esin Durmus and Faisal Ladhak and Filip Ginter and Winata, \{Genta Indra\} and Hendrik Strobelt and Hiroaki Hayashi and Jekaterina Novikova and Jenna Kanerva and Jenny Chim and Jiawei Zhou and Jordan Clive and Joshua Maynez and Jo{\~a}o Sedoc and Juraj Juraska and Kaustubh Dhole and Chandu, \{Khyathi Raghavi\} and Laura Perez-Beltrachini and Ribeiro, \{Leonardo F.R.\} and Lewis Tunstall and Li Zhang and Mahima Pushkarna and Mathias Creutz and Michael White and Kale, \{Mihir Sanjay\} and Eddine, \{Moussa Kamal\} and Nico Daheim and Nishant Subramani and Ondrej Dusek and Liang, \{Paul Pu\} and Ammanamanchi, \{Pawan Sasanka\} and Qi Zhu and Ratish Puduppully and Reno Kriz and Rifat Shahriyar and Ronald Cardenas and Saad Mahamood and Salomey Osei and Samuel Cahyawijaya and Sanja {\v S}tajner and Sebastien Montella and Shailza Jolly and Simon Mille and Tahmid Hasan and Tianhao Shen and Tosin Adewumi and Vikas Raunak and Vipul Raheja and Vitaly Nikolaev and Vivian Tsai and Yacine Jernite and Ying Xu and Yisi Sang and Yixin Liu and Yufang Hou",

year = "2022",

month = dec,

day = "11",

doi = "10.18653/v1/2022.emnlp-demos.27",

language = "English",

pages = "266--281",

editor = "Wanxiang Che and Ekaterina Shutova",

booktitle = "EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing",

publisher = "Association for Computational Linguistics (ACL)",

note = "2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",

}

TY - GEN

T1 - GEMv2

T2 - 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

AU - Gehrmann, Sebastian

AU - Bhattacharjee, Abhik

AU - Mahendiran, Abinaya

AU - Wang, Alex

AU - Papangelis, Alexandros

AU - Madaan, Aman

AU - McMillan-Major, Angelina

AU - Shvets, Anna

AU - Upadhyay, Ashish

AU - Bohnet, Bernd

AU - Yao, Bingsheng

AU - Wilie, Bryan

AU - Bhagavatula, Chandra

AU - You, Chaobin

AU - Thomson, Craig

AU - Garbacea, Cristina

AU - Wang, Dakuo

AU - Deutsch, Daniel

AU - Xiong, Deyi

AU - Jin, Di

AU - Gkatzia, Dimitra

AU - Radev, Dragomir

AU - Clark, Elizabeth

AU - Durmus, Esin

AU - Ladhak, Faisal

AU - Ginter, Filip

AU - Winata, Genta Indra

AU - Strobelt, Hendrik

AU - Hayashi, Hiroaki

AU - Novikova, Jekaterina

AU - Kanerva, Jenna

AU - Chim, Jenny

AU - Zhou, Jiawei

AU - Clive, Jordan

AU - Maynez, Joshua

AU - Sedoc, João

AU - Juraska, Juraj

AU - Dhole, Kaustubh

AU - Chandu, Khyathi Raghavi

AU - Perez-Beltrachini, Laura

AU - Ribeiro, Leonardo F.R.

AU - Tunstall, Lewis

AU - Zhang, Li

AU - Pushkarna, Mahima

AU - Creutz, Mathias

AU - White, Michael

AU - Kale, Mihir Sanjay

AU - Eddine, Moussa Kamal

AU - Daheim, Nico

AU - Subramani, Nishant

AU - Dusek, Ondrej

AU - Liang, Paul Pu

AU - Ammanamanchi, Pawan Sasanka

AU - Zhu, Qi

AU - Puduppully, Ratish

AU - Kriz, Reno

AU - Shahriyar, Rifat

AU - Cardenas, Ronald

AU - Mahamood, Saad

AU - Osei, Salomey

AU - Cahyawijaya, Samuel

AU - Štajner, Sanja

AU - Montella, Sebastien

AU - Jolly, Shailza

AU - Mille, Simon

AU - Hasan, Tahmid

AU - Shen, Tianhao

AU - Adewumi, Tosin

AU - Raunak, Vikas

AU - Raheja, Vipul

AU - Nikolaev, Vitaly

AU - Tsai, Vivian

AU - Jernite, Yacine

AU - Xu, Ying

AU - Sang, Yisi

AU - Liu, Yixin

AU - Hou, Yufang

PY - 2022/12/11

Y1 - 2022/12/11

N2 - Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.

AB - Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.

UR - https://www.scopus.com/pages/publications/85149671144

U2 - 10.18653/v1/2022.emnlp-demos.27

DO - 10.18653/v1/2022.emnlp-demos.27

M3 - Published conference contribution

AN - SCOPUS:85149671144

SP - 266

EP - 281

BT - EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing

A2 - Che, Wanxiang

A2 - Shutova, Ekaterina

PB - Association for Computational Linguistics (ACL)

Y2 - 7 December 2022 through 11 December 2022

ER -

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Abstract

Conference

Access to Document

Other files and links

Fingerprint

Cite this