16 October 2024, Computational Linguistics Seminar, Verna Dankers

Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, but which examples require memorisation and where in the millions (or billions) of parameters memorisation occurs are questions that remain largely unanswered. In this talk, I first elaborate on the localisation question by examining memorisation in the context of classification in fine-tuned PLMs, using 12 tasks. Our findings give nuance to the generalisation-first memorisation-second hypothesis dominant in the literature and find memorisation to be a gradual process rather than a localised one. Secondly, I discuss memorisation from the viewpoint of the data using neural machine translation (NMT) models by putting individual data points on a memorisation-generalisation map. I illustrate how the data points' characteristics are predictive of memorisation in NMT and describe the influence that subsets of that map have on NMT systems' performance.