Multilinguality and Multiculturalism: Towards more Effective and Inclusive Neural Language Models
Rochelle Choenni

Abstract:
Large-scale pretraining requires vast amounts of text in a given language, which limits the applicability of such techniques to a handful of high-resource languages. Therefore, researchers have focused on the development of models with a wider cross-lingual applicability, leading to the development of single models that are jointly trained on texts from multiple languages i.e., multilingual language models (MLMs). The intuition behind multilingual joint training is that it facilitates information sharing between languages, such that languages can learn to support one another by leveraging language commonalities. 
However, while LMs have become increasingly multilingual, the state-of-the-art modeling approaches have come with a new set of technical and social challenges. In particular, joint training reduces the model capacity available per language, and consequently, languages start competing for limited resources. In turn, this can cause languages to negatively affect each other, which undercuts the benefits of cross-lingual sharing. Moreover, to deploy MLMs in culturally-diverse communities, their output needs to be sensitive to the sociocultural norms and biases of those communities, necessitating MLMs to become inherently multicultural as well.
In this thesis, we therefore study MLMs with respect to both their technical and social challenges. In particular, we investigate how to build more effective MLMs that mitigate negative cross-language interference and study the effect that joint multilingual training has on the social biases and cultural values that MLMs encode.