What is a good conversation? Improving the Operationalization and Measurement of Deliberative Quality by Analyzing Interactions
Speaker: (opens in a new window)Nicolai Berk (ETH Zürich)
Wednesday, November 6, 14:00–14:45 (Irish time)
Please register (opens in a new window)here to receive the link and password to the online meeting and information on the room at UCD.
Abstract: Deliberation - the exchange of arguments to inform a collective understanding of an issue - is a key component of opinion formation and, therefore, democratic decision-making. Theorists of deliberation conceptualize deliberative quality as a property of entire communicative interactions, involving respect, justification, and reciprocal engagement. However, prior attempts of operationalization - both in the interpretive and computer science traditions - have treated each debate contribution, like a speech or online comment, as a single unit of analysis, ignoring the interactive nature of this process and failing to measure the deliberative quality of the entire communicative exchange. In this paper, we empirically show that operationalizing deliberation at the interaction level captures important aspects of the concept which cannot be measured when treating single debate contributions as distinct entities. Using data from three different online outlets, we show that large language models (LLMs) can achieve performance comparable to human annotators when evaluating deliberative quality in online conversations. Furthermore, we show that LLMs better identify the deliberative quality of an interaction, especially with regards to its reciprocity and respectfulness, when shown entire conversations, rather than individual comments separately. Based on a synthetic-labeling approach, we train and publish adapter modules, providing an off-the-shelf tool to researchers interested in the measurement of deliberative quality and its underlying dimensions in online discourse.
About the speaker: (opens in a new window)Nicolai Berk is a Postdoctoral Researcher at the Immigration Policy Lab and the Public Policy Group at ETH Zürich. Previously, he was a PhD Candidate at the Dynamics Research Training Group and the Chair for Comparative Politics at Humboldt University Berlin. His dissertation was supervised by Thomas Meyer, as well as Heike Klüver and Rune Slothuus. His research focuses on voters’ partisan perceptions of politics, public and media discourse and their effects on attitude formation, as well as the application of Natural Language Processing (NLP) techniques to study social phenomena.