How Linguistics Learned to Stop Worrying and Love the Language Models

Language models can produce fluent, grammatical text. Nonetheless, some maintain that language models don’t really learn language and also that, even if they did, that would not be informative for the study of human learning and processing. On the other side, there have been claims that the success of LMs obviates the need for studying linguistic theory and structure. In a recent position piece with Richard Futrell, we argue that both extremes are wrong. LMs can contribute to fundamental questions about linguistic structure, language processing, and learning. They require rethinking arguments about learning and are informative for major questions in linguistic theory. But they do not replace linguistic structure and theory. We offer an optimistic take on the relationship between language models and linguistics. As part of this take, I will discuss some of my own work on grammaticality judgments in LMs as well as on controlled pretraining paradigms, in which small models are trained on systematically manipulated input corpora.