Forrest Davis (original) (raw)

322 Bernstein Hall

Colgate University

Hamilton, NY 13346

I am an Assistant Professor of Computer Science at Colgate University. Before joining Colgate, I was a Postdoctoral Associate in the Linguistics and Philosophy Department at the Massachusetts Institute of Technology. I received my Ph.D. from the Department of Linguistics at Cornell University, where I was primarily advised by Marten van Schijndel, and my B.A. in Computer Science and Mathematics from Columbia University.

I am broadly interested in mismatches between our experiences with language and our knowledge of language. That is, I try to find in linguistic data systematic deviations from what we might expect given our knowledge of grammar. Findings like this excite me because they suggest cases where our minds extend beyond mere correspondence with experience. My primary tool at the moment is neural language models (e.g., large language models) trained on text data. Drawing on insights from psycholinguistics, linguistic theory, and cross-linguistic variation, I expose limitations in current AI models and tie these limitations to properties of training data. My dissertation titled “On the Limitations of Data: Mismatches between Neural Models of Language and Humans” sketches out my perspective with case studies.

Besides research and teaching, I love coffee, watching movies, swimming, hiking, and skiing. I cohabitate with a cat named Figaro, who spends his time sleeping throughout our home, sitting on my keyboard, and meowing for food.

news

selected publications

2024

1. Training an NLP Scholar at a Small Liberal Arts College: A Backwards Designed Course Proposal
   In Proceedings of the Sixth Workshop on Teaching NLP, Aug 2024
   The rapid growth in natural language processing (NLP) over the last couple yearshas generated student interest and excitement in learning more about the field. In this paper, we present two types of students that NLP courses might want to train. First, an “NLP engineer” who is able to flexibly design, build and apply new technologies in NLP for a wide range of tasks. Second, an “NLP scholar” who is able to pose, refine and answer questions in NLP and how it relates to the society, while also learning to effectively communicate these answers to a broader audience. While these two types of skills are not mutually exclusive — NLP engineers should be able to think critically, and NLP scholars should be able to build systems — we think that courses can differ in the balance of these skills. As educators at Small Liberal Arts Colleges, the strengths of our students and our institution favors an approach that is better suited to train NLP scholars. In this paper we articulate what kinds of skills an NLP scholar should have, and then adopt a backwards design to propose course components that can aid the acquisition of these skills.

2023

1. Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics
   In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), Dec 2023
   Language models (LMs) have been argued to overlap substantially with human beings in grammaticality judgment tasks. But when humans systematically make errors in language processing, should we expect LMs to behave like cognitive models of language and mimic human behavior? We answer this question by investigating LMs’ more subtle judgments associated with “language illusions” – sentences that are vague in meaning, implausible, or ungrammatical but receive unexpectedly high acceptability judgments by humans. We looked at three illusions: the comparative illusion (e.g. “More people have been to Russia than I have”), the depth-charge illusion (e.g. “No head injury is too trivial to be ignored”), and the negative polarity item (NPI) illusion (e.g. “The hunter who no villager believed to be trustworthy will ever shoot a bear”). We found that probabilities represented by LMs were more likely to align with human judgments of being “tricked” by the NPI illusion which examines a structural dependency, compared to the comparative and the depth-charge illusions which require sophisticated semantic understanding. No single LM or metric yielded results that are entirely consistent with human behavior. Ultimately, we show that LMs are limited both in their construal as cognitive models of human language processing and in their capacity to recognize nuanced but critical information in complicated language materials.

2022

1. Incremental Processing of Principle B: Mismatches Between Neural Models and Humans
   Forrest Davis
   In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), Dec 2022
   Despite neural language models qualitatively capturing many human linguistic behaviors, recent work has demonstrated that they underestimate the true processing costs of ungrammatical structures. We extend these more fine-grained comparisons between humans and models by investigating the interaction between Principle B and coreference processing. While humans use Principle B to block certain structural positions from affecting their incremental processing, we find that GPT-based language models are influenced by ungrammatical positions. We conclude by relating the mismatch between neural models and humans to properties of training data and suggest that certain aspects of human processing behavior do not directly follow from linguistic data.

2020

1. Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment
   In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jul 2020
   A standard approach to evaluating language models analyzes how models assign probabilities to valid versus invalid syntactic constructions (i.e. is a grammatical sentence more probable than an ungrammatical sentence). Our work uses ambiguous relative clause attachment to extend such evaluations to cases of multiple simultaneous valid interpretations, where stark grammaticality differences are absent. We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. Thus, English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences. We conclude by relating these results to broader concerns about the relationship between comprehension (i.e. typical language model use cases) and production (which generates the training data for language models), suggesting that necessary linguistic biases are not present in the training signal at all.