Political DEBATE: Efficient Zero-shot and
Few-shot Classifiers for Political Text
Speaker: (opens in a new window)Michael Burnham (Princeton University)
Co-authors: Kayla Kahn , Ryan Yang Wang , and Rachel X. Peng
Wednesday, October 9, 14:00–14:45 (Irish time)
Please register (opens in a new window)here to receive the link and password to the online meeting and information on the room at UCD.
Abstract: Social scientists quickly adopted large language models due to their ability to annotate documents without supervised training, an ability known as zero-shot learning. However, due to their computing demands, cost, and often proprietary nature, these models are often at odds with replication and open science standards. This paper introduces the Political DEBATE (DeBERTa Algorithm for Textual Entailment) language models for zero-shot and few-shot classification of political documents. These models are not only as good, or better than, state-of-the art large language models at zero and few-shot classification, but are orders of magnitude more efficient and completely open source. By training the models on a simple random sample of 10-25 documents, they can outperform supervised classifiers trained on hundreds or thousands of documents and state-of-the-art generative models with complex, engineered prompts. Additionally, we release the PolNLI dataset used to train these models – a corpus of over 200,000 political documents with highly accurate labels across over 800 classification tasks.
About the speaker: (opens in a new window)Michael Burnham is a postdoctoral researcher at Princeton University studying American politics and computational research methods. He has a dual-title Ph.D. in American politics and social data analytics from Penn State. He is particularly interested in polarization, democratic accountability, and electoral incentives. His research touches on various topics, including social media, judicial politics, legislatures, and political communication. Methodologically, He focuses on natural language processing and machine learning. His dissertation advances methods of using language models in social science and demonstrates how they can be used in conjunction with Bayesian item response theory to estimate concepts like affective polarization from text. He authored the Python package Entss to implement this method.