A taxonomy and review of generalization research in NLP

The ability for NLP models to generalize well is one of the main desiderata of current NLP research. However, there is currently no consensus as to what 'good generalization' entails and how it should be evaluated. The rough definition is the ability to successfully transfer representations, knowledge and strategies from past to new experiences, but different researchers use different definitions and there are currently no common standards to evaluate generalization. As a consequence, newly proposed NLP models are usually not systematically tested for their ability to generalize.

To help overcome this problem, an international team of researchers, including researchers from the ILLC, has now published an Analysis in Nature Machine Intelligence[/i] in which they present a taxonomy for characterizing and understanding generalization research in NLP. The publication is the first result of the larger project GenBench, led by UvA-ILLC alumna Dieuwke Hupkes.