【Team】Explainable AI Team
【Date】2026/October/14(Wednesday) 17:30-18:30(JST)
【Speaker】Talk by Étienne Simon, University of Oslo
Title:Evaluating Compositional Generalization of Language Models
Abstract:
Compositionality is the principle that the meaning of a complex expression is determined by the meanings of its parts and how they are combined. Turning this intuition into practical assessments of language models, however, remains challenging. This talk brings together three approaches to assessing compositional abilities. We first examine whether language models grounded in knowledge graphs generalize to longer sequences and novel combinations of familiar components, finding limitations in both settings. We then explore two ways of moving beyond a strict notion of compositionality. The first examines systematic generalization across datasets, connecting model performance to the entropy of component distributions in training data. The second relaxes synonymy in the formal semantic definition of compositionality to introduce ε-compositionality, a continuous measure that we apply to text embeddings of idiomatic and non-idiomatic expressions. Together, these approaches connect theoretical accounts of compositionality with empirical evaluations of model behavior.
