A taxonomy and review of generalization research in NLP

Dieuwke Hupkes; Mario Giulianelli; Verna Dankers; Mikel Artetxe; Yanai Elazar; Tiago Pimentel; Christos Christodoulopoulos; Karim Lasri; Naomi Saphra; Arabella Sinclair; Dennis Ulmer; Florian Schottmann; Khuyagbaatar Batsuren; Kaiser Sun; Koustuv Sinha; Leila Khalatbari; Maria Ryskina; Rita Frieske; Ryan Cotterell; Zhijing Jin

doi:10.1038/s42256-023-00729-y

A taxonomy and review of generalization research in NLP

Dieuwke Hupkes^*, Mario Giulianelli^*, Verna Dankers^*, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, Zhijing Jin

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

12 Citations (Scopus)

2 Downloads (Pure)

Abstract

The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what ‘good generalization’ entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.

Original language	English
Pages (from-to)	1161-1174
Number of pages	14
Journal	Nature Machine Intelligence
Volume	5
Issue number	10
Early online date	19 Oct 2023
DOIs	https://doi.org/10.1038/s42256-023-00729-y
Publication status	Published - 19 Oct 2023

Bibliographical note

Funding Information:
We thank A. Williams, A. Joulin, E. Bruni, L. Weber, R. Kirk and S. Riedel for providing feedback on the various stages of this paper, and G. Marcus for providing detailed feedback on the final draft. We also thank the reviewers of our work for providing useful comments. We thank E. Hupkes for making the app that allows searching through references, and we thank D. Haziza and E. Takmaz for other contributions to the website. M.G. was supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 819455). V.D. was supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant no. EP/S022481/1) and the University of Edinburgh. N.S. was supported by the Hyundai Motor Company (under the project Uncertainty in Neural Sequence Modeling) and the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI).

Publisher Copyright:
© 2023, The Author(s).

Data Availability Statement

Data availability
The full annotated list of articles included in our survey is available through the GenBench website (https://genbench.org/references), where articles can be filtered through a dedicated search tool. This is an evolving survey: we encourage authors to submit new work and to request annotation corrections through our contributions page (https://genbench.org/contribute). The exact list used at the time of writing can be retrieved from https://github.com/GenBench/GenBench.github.io/blob/cea0bd6bd8af6f2d0f096c8f81185b1dfc9303b5/taxonomy_clean.tsv. We also release interactive tools to visualize the results of our survey at https://genbench.org/visualisation. Source data are provided with this paper.
Source data:
https://static-content.springer.com/esm/art%3A10.1038%2Fs42256-023-00729-y/MediaObjects/42256_2023_729_MOESM3_ESM.csv

Access to Document

10.1038/s42256-023-00729-y

Hupkes_Etal_A_Taxonomy_And_VOR
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
Final published version, 2.06 MBLicence: CC BY

Cite this

Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., Ulmer, D., Schottmann, F., Batsuren, K., Sun, K., Sinha, K., Khalatbari, L., Ryskina, M., Frieske, R., Cotterell, R., & Jin, Z. (2023). A taxonomy and review of generalization research in NLP. Nature Machine Intelligence, 5(10), 1161-1174. https://doi.org/10.1038/s42256-023-00729-y

Hupkes, D, Giulianelli, M, Dankers, V, Artetxe, M, Elazar, Y, Pimentel, T, Christodoulopoulos, C, Lasri, K, Saphra, N, Sinclair, A, Ulmer, D, Schottmann, F, Batsuren, K, Sun, K, Sinha, K, Khalatbari, L, Ryskina, M, Frieske, R, Cotterell, R & Jin, Z 2023, 'A taxonomy and review of generalization research in NLP', Nature Machine Intelligence, vol. 5, no. 10, pp. 1161-1174. https://doi.org/10.1038/s42256-023-00729-y

@article{2eb8d0d6fe754237a37193d562a096fd,

title = "A taxonomy and review of generalization research in NLP",

abstract = "The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what {\textquoteleft}good generalization{\textquoteright} entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.",

author = "Dieuwke Hupkes and Mario Giulianelli and Verna Dankers and Mikel Artetxe and Yanai Elazar and Tiago Pimentel and Christos Christodoulopoulos and Karim Lasri and Naomi Saphra and Arabella Sinclair and Dennis Ulmer and Florian Schottmann and Khuyagbaatar Batsuren and Kaiser Sun and Koustuv Sinha and Leila Khalatbari and Maria Ryskina and Rita Frieske and Ryan Cotterell and Zhijing Jin",

note = "Funding Information: We thank A. Williams, A. Joulin, E. Bruni, L. Weber, R. Kirk and S. Riedel for providing feedback on the various stages of this paper, and G. Marcus for providing detailed feedback on the final draft. We also thank the reviewers of our work for providing useful comments. We thank E. Hupkes for making the app that allows searching through references, and we thank D. Haziza and E. Takmaz for other contributions to the website. M.G. was supported by the European Research Council (ERC) under the European Union{\textquoteright}s Horizon 2020 research and innovation programme (grant agreement no. 819455). V.D. was supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant no. EP/S022481/1) and the University of Edinburgh. N.S. was supported by the Hyundai Motor Company (under the project Uncertainty in Neural Sequence Modeling) and the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI). Publisher Copyright: {\textcopyright} 2023, The Author(s).",

year = "2023",

month = oct,

day = "19",

doi = "10.1038/s42256-023-00729-y",

language = "English",

volume = "5",

pages = "1161--1174",

journal = "Nature Machine Intelligence",

issn = "2522-5839",

publisher = "Springer Nature Switzerland AG",

number = "10",

}

TY - JOUR

T1 - A taxonomy and review of generalization research in NLP

AU - Hupkes, Dieuwke

AU - Giulianelli, Mario

AU - Dankers, Verna

AU - Artetxe, Mikel

AU - Elazar, Yanai

AU - Pimentel, Tiago

AU - Christodoulopoulos, Christos

AU - Lasri, Karim

AU - Saphra, Naomi

AU - Sinclair, Arabella

AU - Ulmer, Dennis

AU - Schottmann, Florian

AU - Batsuren, Khuyagbaatar

AU - Sun, Kaiser

AU - Sinha, Koustuv

AU - Khalatbari, Leila

AU - Ryskina, Maria

AU - Frieske, Rita

AU - Cotterell, Ryan

AU - Jin, Zhijing

N1 - Funding Information: We thank A. Williams, A. Joulin, E. Bruni, L. Weber, R. Kirk and S. Riedel for providing feedback on the various stages of this paper, and G. Marcus for providing detailed feedback on the final draft. We also thank the reviewers of our work for providing useful comments. We thank E. Hupkes for making the app that allows searching through references, and we thank D. Haziza and E. Takmaz for other contributions to the website. M.G. was supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 819455). V.D. was supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant no. EP/S022481/1) and the University of Edinburgh. N.S. was supported by the Hyundai Motor Company (under the project Uncertainty in Neural Sequence Modeling) and the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI). Publisher Copyright: © 2023, The Author(s).

PY - 2023/10/19

Y1 - 2023/10/19

N2 - The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what ‘good generalization’ entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.

AB - The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what ‘good generalization’ entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.

UR - http://www.scopus.com/inward/record.url?scp=85174518288&partnerID=8YFLogxK

U2 - 10.1038/s42256-023-00729-y

DO - 10.1038/s42256-023-00729-y

M3 - Article

AN - SCOPUS:85174518288

SN - 2522-5839

VL - 5

SP - 1161

EP - 1174

JO - Nature Machine Intelligence

JF - Nature Machine Intelligence

IS - 10

ER -

A taxonomy and review of generalization research in NLP

Abstract

Bibliographical note

Data Availability Statement

Access to Document

Other files and links

Fingerprint

Data From A taxonomy and review of generalization research in NLP

Cite this

A taxonomy and review of generalization research in NLP

Abstract

Bibliographical note

Data Availability Statement

Access to Document

Other files and links

Fingerprint

Datasets

Data From A taxonomy and review of generalization research in NLP

Cite this