Moderating with the Mob: Evaluating the Efficacy of Real-Time Crowdsourced Fact-Checking (original) (raw)
Related papers
2017
A major challenge for fact-checking platforms to survive is a lack of resources. Platforms relying solely on professionals may not be able to become self-sufficient because of the amount of resources they require. We propose that a sustainable model requires incorporating the wisdom of the crowd and automated assisting tools into the process, which will increase efficiency and decrease costs. In this study, we examined a crowd curated political fact-checking platform, reddit.com/politicalfactchecking, and identified various roles crowds and professionals play in fact-checking. We’ve also developed an automated argument classification model and identified some steps in fact-checking, which could be automated.
Studies in Media and Communication, 2023
Fake news, misinformation, and non-true stories create a definite threat to the world's public sphere. Fake news contaminates democracy by blurring the sight and the vision, or by altering the beliefs of citizens on simple everyday matters but also on significant matters such as vaccination, politics, social issues, or public health. Lots of efforts have been conducted in order to tackle the phenomenon. Fact-checking platforms consist of a major step in this issue. Certain cases of fact-checking platforms worldwide seem to work properly and fulfill their strategic goals, although functional and other issues might emerge. This study comes to take the fact-checking platform evolution one step beyond by proposing a new communication model for fake news detection and busting. The proposed model's blueprint is based on the Greek "Ellinika Hoaxes" fact-checking platform with some critical reinforcements: More extensive use of crowdsourcing strategies for detecting and busting non-true stories with the aid of AI chatbots in order not only to bust non-true stories but also to rank news outlets, writers, social media personas and journalists for their credibility. This way, serious news outlets, journalists, and media professionals can build their trust and be ranked for the credibility of their services for a more trustful and democratic public sphere.
The Effects of Crowd Worker Biases in Fact-Checking Tasks
2022 ACM Conference on Fairness, Accountability, and Transparency
Due to the increasing amount of information shared online every day, the need for sound and reliable ways of distinguishing between trustworthy and non-trustworthy information is as present as ever. One technique for performing fact-checking at scale is to employ human intelligence in the form of crowd workers. Although earlier work has suggested that crowd workers can reliably identify misinformation, cognitive biases of crowd workers may reduce the quality of truthfulness judgments in this context. We performed a systematic exploratory analysis of publicly available crowdsourced data to identify a set of potential systematic biases that may occur when crowd workers perform fact-checking tasks. Following this exploratory study, we collected a novel data set of crowdsourced truthfulness judgments to validate our hypotheses. Our findings suggest that workers generally overestimate the truthfulness of statements and that different individual characteristics (i.e., their belief in science) and cognitive biases (i.e., the affect heuristic and overconfidence) can affect their annotations. Interestingly, we find that, depending on the general judgment tendencies of workers, their biases may sometimes lead to more accurate judgments.
Information Processing & Management, 2021
Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased ("cherrypicked"), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker's Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness.
Towards Crowdsourcing Tasks for Accurate Misinformation Detection
2020
For all the recent advancements in Natural Language Processing and deep learning, current systems for misinformation detection are still woefully inaccurate in real-world data. Automated misinformation detection systems —available to the general public and producing explainable ratings— are therefore still an open problem and involvement of domain experts, journalists or fact-checkers is necessary to correct the mistakes such systems currently make. Reliance on such expert feedback imposes a bottleneck and prevents scalability of current approaches. In this paper, we propose a method —based on a recent semantic-based approach for misinformation detection, Credibility Reviews (CR)—, to (i) identify real-world errors of the automatic analysis; (ii) use the semantic links in the CR graphs to identify steps in the misinformation analysis which may have caused the errors and (iii) derive crowdsourcing tasks to pinpoint the source of errors. As a bonus, our approach generates real-world t...
Interactive crowdsourcing to fact-check politicians
The discourse of political leaders often contains false information that can misguide the public. Fact-checking agencies around the world try to reduce the negative influence of politicians by verifying their words. However, these agencies face a problem of scalability and require innovative solutions to deal with their growing amount of work. While previous studies have shown that crowdsourcing is a promising approach to fact-check news in a scalable manner, it remains unclear whether crowdsourced judgements are useful to verify the speech of politicians. This paper fills that empirical gap and also studies the effect of social influence on the accuracy of collective judgements about the veracity of phrases pronounced by politicians. Participants (N=180) first read 20 politically-balanced phrases and made individual judgements. Then, they were randomly assigned to discuss the same phrases with another politically homogeneous or heterogeneous person, or to a control condition with n...
Diffusion of Community Fact-Checked Misinformation on Twitter
Cornell University - arXiv, 2022
The spread of misinformation on social media is a pressing societal problem that platforms, policymakers, and researchers continue to grapple with. As a countermeasure, recent works have proposed to employ non-expert fact-checkers in the crowd to fact-check social media content. While experimental studies suggest that crowds might be able to accurately assess the veracity of social media content, an understanding of how crowd fact-checked (mis-)information spreads is missing. In this work, we empirically analyze the spread of misleading vs. not misleading community fact-checked posts on social media. For this purpose, we employ a dataset of community-created fact-checks from Twitter's "Birdwatch" pilot and map them to resharing cascades on Twitter. Different from earlier studies analyzing the spread of misinformation listed on third-party fact-checking websites (e. g., snopes.com), we find that community fact-checked misinformation is less viral. Specifically, misleading posts are estimated to receive 36.62 % fewer retweets than not misleading posts. A partial explanation may lie in differences in the fact-checking targets: community fact-checkers tend to factcheck posts from influential user accounts with many followers, while expert fact-checks tend to target posts that are shared by less influential users. We further find that there are significant differences in virality across different sub-types of misinformation (e. g., factual errors, missing context, manipulated media). Moreover, we conduct a user study to assess the perceived reliability of (real-world) community-created fact-checks. Here, we find that users, to a large extent, agree with community-created fact-checks. Altogether, our findings offer insights into how misleading vs. not misleading posts spread and highlight the crucial role of sample selection when studying misinformation on social media. CCS Concepts: • Human-centered computing → Empirical studies in collaborative and social computing; Social media; • Information systems → Crowdsourcing.
Scalable Fact-checking with Human-in-the-Loop
2021 IEEE International Workshop on Information Forensics and Security (WIFS), 2021
Researchers have been investigating automated solutions for fact-checking in a variety of fronts. However, current approaches often overlook the fact that the amount of information released every day is escalating, and a large amount of them overlap. Intending to accelerate fact-checking, we bridge this gap by grouping similar messages and summarizing them into aggregated claims. Specifically, we first clean a set of social media posts (e.g., tweets) and build a graph of all posts based on their semantics; Then, we perform two clustering methods to group the messages for further claim summarization. We evaluate the summaries both quantitatively with ROUGE scores and qualitatively with human evaluation. We also generate a graph of summaries to verify that there is no significant overlap among them. The results reduced 28,818 original messages to 700 summary claims, showing the potential to speed up the fact-checking process by organizing and selecting representative claims from massive disorganized and redundant messages.
A Reliable Weighting Scheme for the Aggregation of Crowd Intelligence to Detect Fake News
Information
Social networks play an important role in today’s society and in our relationships with others. They give the Internet user the opportunity to play an active role, e.g., one can relay certain information via a blog, a comment, or even a vote. The Internet user has the possibility to share any content at any time. However, some malicious Internet users take advantage of this freedom to share fake news to manipulate or mislead an audience, to invade the privacy of others, and also to harm certain institutions. Fake news seeks to resemble traditional media to establish its credibility with the public. Its seriousness pushes the public to share them. As a result, fake news can spread quickly. This fake news can cause enormous difficulties for users and institutions. Several authors have proposed systems to detect fake news in social networks using crowd signals through the process of crowdsourcing. Unfortunately, these authors do not use the expertise of the crowd and the expertise of a...
Identifying Fake News from Twitter Sharing Data: {A} Large-Scale Study
Social networks offer a ready channel for fake and misleading news to spread and exert influence. This paper examines the performance of different reputation algorithms when applied to a large and statistically significant portion of the news that are spread via Twitter. Our main result is that simple crowdsourcing-based algorithms are able to identify a large portion of fake or misleading news, while incurring only very low false positive rates for mainstream websites. We believe that these algorithms can be used as the basis of practical, large-scale systems for indicating to consumers which news sites deserve careful scrutiny and skepticism.