Is It Biased? Empirical Analysis of Various Phenomena That Affect Survey Results (original) (raw)

Response Styles in Survey Research: A Literature Review of Antecedents, Consequences, and Remedies

… Journal of Public Opinion Research, 2012

Although the purpose of questionnaire items is to obtain a person’s opinion on a certain matter, a respondent’s registered opinion may not reflect his or her ‘‘true’’ opinion because of random and systematic errors. Response styles (RSs) are a respondent’s tendency to respond to survey questions in certain ways regardless of the content, and they contribute to systematic error. They affect univariate and multivariate distributions of data collected by rating scales and are alternative explanations for many research results. Despite this, RSs are often not controlled in research. This article provides a comprehensive summary of the types of RSs, lists their potential sources, and discusses ways to diagnose and control for them. Finally, areas for further research on RSs are proposed.

An Empirical Test of Alternative Theories of Survey Response Behaviour

Market Research Society. Journal., 1999

This study examines the extent to which the theories of exchange, cognitive dissonance, self-perception and commitment/involvement, when used to design surveys, can influence potential respondents to participate in a survey. The results from an experiment involving a total of 403 subjects in Hong Kong and Australia expands what is known about the role played by theory by examining consumer responses to participation requests made on the basis of each theoretical framework. Specific results support the relatively high positive impact of two of the frameworks that has been reported in a study of research practitioners.

A Catalog of Biases in Questionnaires

Bias in questionnaires is an important issue in public health research. To collect the most accurate data from respondents, investigators must understand and be able to prevent or at least minimize bias in the design of their questionnaires. This paper identifies and categorizes 48 types of bias in questionnaires based on a review of the literature and offers an example of each type. The types are categorized according to three main sources of bias: the way a question is designed, the way the questionnaire as a whole is designed, and how the questionnaire is administered. This paper is intended to help investigators in public health understand the mechanism and dynamics of problems in questionnaire design and to provide a checklist for identifying potential bias in a questionnaire before it is administered.

Assessing bias in online surveys using alternative survey modes

Work Organisation, Labour & Globalisation

Due to concerns that respondents to online surveys are different from populations of interest, parallel offline surveys can be undertaken and results compared. In this article we create a set of principles to compare results from online surveys with those from surveys using other survey modes. Rather than just comparing estimates and confidence intervals from the different modes, these principles consider biases that each survey mode introduces and whether the results obtained are compatible with each other, given these different biases. Using the example of a survey of platform work, we demonstrate that this approach can be used effectively and be applied to a variety of social science studies that use online surveys.

Survey Satisficing Inflates Reliability and Validity Measures

Educational and Psychological Measurement, 2016

This study examined the predictors and psychometric outcomes of survey satisficing, wherein respondents provide quick, “good enough” answers (satisficing) rather than carefully considered answers (optimizing). We administered surveys to university students and respondents—half of whom held college degrees—from a for-pay survey website, and we used an experimental method to randomly assign the participants to survey formats, which presumably differed in task difficulty. Based on satisficing theory, we predicted that ability, motivation, and task difficulty would predict satisficing behavior and that satisficing would artificially inflate internal consistency reliability and both convergent and discriminant validity correlations. Indeed, results indicated effects for task difficulty and motivation in predicting survey satisficing, and satisficing in the first part of the study was associated with improved internal consistency reliability and convergent validity but also worse discrimi...

Survey Mode as a Source of Instability in Responses across Surveys

Field Methods, 2005

Changes in survey mode for conducting panel surveys may contribute significantly to survey error. This article explores the causes and consequences of such changes in survey mode. The authors describe how and why the choice of survey mode often causes changes to be made to the wording of questions, as well as the reasons that identically worded questions often produce different answers when administered through different modes. The authors provide evidence that answers may changeas a result of different visual layouts for otherwise identical questions and suggest ways to keep measurement the same despite changes in survey mode.

Implicit theories of performance as artifacts in survey research: A replication and extension

Organizational Behavior and Human Performance, 1978

has presented laboratory data which indicate that people may respond to questionnaire items concerning their behavior based on "theories" they have formed about the relationships among variables. The implication that data collected from field surveys may simply reflect stereotypes based on these theories, poses serious problems for survey research. The present study attempted to examine rival alternative explanations for Staw's results. His results showed a strong effect for false feedback on questionnaire responses, but the present study indicates that by allowing subjects to work together long enough to make more accurate judgments of process variables, Staw's effects can virtually be eliminated. Conditions under which respondent stereotypes should and should not be important are discussed.

Non-Response Error within Internet Surveys: A Cautionary Note

The Journal of International Information Management, 2005

The use of the internet as a method to conduct survey research has expanded rapidly over the past decade. High speeds of response and lower expenses have driven this rapid growth. Relatively low response rates, however, suggest online surveys may be compromised by high levels of non-sampling error. This paper examines a major component of non-sampling error and the consequences that may be associated with internet survey non-response. Known population parameters are compared to point estimates from a census as well as a random sample of non-respondents in order to provide insight on the magnitude and direction of nonresponse error. Issue salience and response latency are found to exhibit a significant relationship to self-selection and response valance biases. Specifically, lower rates of nonresponse were obtained from respondents who perceived the topic of the survey as more important and patterns of response were more favorable among initial study participants. INTRODUCTION Usage ...