Problems in the estimation and interpretation of the reliability of survey data (original) (raw)
Related papers
2015
When designing questionnaires, many choices have to be made. Because the consequences of these choices for the quality of the questions are largely unknown, it has often been said that designing a questionnaire is an art. To make it a more scientific activity we need to know more about the consequences of these choices. In order to further such an approach we have: 1. made an inventory of the choices to be made when designing survey questions and created a code book to transform these question characteristics into the independent variables for explaining quality of survey questions; 2. assembled a large set of studies that use Multi-trait Multi-method experiments to esti-mate the reliability and validity of questions 3. carried out a meta-analysis that relates these question characteristics to the reliability and validity estimates of the questions. On the basis of the results of these efforts we have constructed a database. This data base con-tains at present 1023 measurement instr...
Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions
Survey Research Methods, 2007
When designing questionnaires, many choices have to be made. Because the consequences of these choices for the quality of the questions are largely unknown, it has often been said that designing a questionnaire is an art. To make it a more scientific activity we need to know more about the consequences of these choices. In order to further such an approach we have:
Evaluating Surveys and Questionnaires
Critical Thinking in Psychology, 2006
Much of what we know about human behavior is based on self-reports. When we want to learn about individuals' health behaviors, consumer habits, family problems, media consumption, values or political beliefs, we ask appropriate questions. The answers provided to these questions serve as input into scientific analyses and provide the basis of statistical indicators used to describe the state of a society. Obviously, these data are only as meaningful as the questions we ask and the answers we receive. Moreover, whom we ask is of crucial importance to our ability to draw conclusions that extend beyond the particular people who answered our questions. Accordingly, the processes underlying question answering and the appropriate selection of respondents are of great importance to many areas of social research.
Some Consequences of Measurement Error in Survey Data
American Journal of Political Science, 1974
socialization data, this paper first presents some estimates of the extent of measurement error in several standard face sheet items. After the presence of measurement error is demonstrated, two techniques involving multiple indicators and observations over time are employed to estimate the effects of measurement error on bivariate correlation coefficients with party identification providing the substantive vehicle of the analysis. In general, the analysis suggests that random measurement error may have a major impact on our coefficients and thereby result in misleading inferences. The advent of data archives such as the Inter-University Consortium for Political Research has been a boon to researchers wishing to engage in secondary analysis.' However, the reliance on data collected by others has a number of limitations, some quite obvious and others less so. In the former category is the likelihood that important variables were omitted in the data collection or that key concepts were not operationalized in a way suitable for the secondary analyst. But a more subtle problem of secondary analysis is that the investigator often has little feel for the quality of the data, for the extent and nature of the measurement error in the data. Hence, this paper will present some estimates of the amount of measurement error for some standard face sheet items in two survey data sets collected by a social science institute renowned for its quality control procedures. Then the effects of measurement error on correlation coefficients will be evaluated by a multiple-indicator approach and an observations-overtime strategy, both of which involve the use of path analysis techniques. By measurement error is meant any deviation from the true value of a *I am grateful to Aage Clausen, David Leege, and Robert Lehnen for their helpful comments and suggestions, and to M.
Enhancing the validity and cross-cultural comparability of measurement in survey research
… der vergleichenden Politik- …, 2009
W e address two long-standing survey research problems: measuring complicated concepts, such as political freedom and efficacy, that researchers define best with reference to examples; and what to do when respondents interpret identical questions in different ways. Scholars have long addressed these problems with approaches to reduce incomparability, such as writing more concrete questions-with uneven success. Our alternative is to measure directly response category incomparability and to correct for it. We measure incomparability via respondents' assessments, on the same scale as the self-assessments to be corrected, of hypothetical individuals described in short vignettes. Because the actual (but not necessarily reported) levels of the vignettes are invariant over respondents, variability in vignette answers reveals incomparability. Our corrections require either simple recodes or a statistical model designed to save survey administration costs. With analysis, simulations, and cross-national surveys, we show how response incomparability can drastically mislead survey researchers and how our approach can alleviate this problem.
Dhaulagiri Journal of Sociology and Anthropology
This research note briefly describes the levels of measurement of variables and their applications in the quantitative analysis of survey data. It first presents the concept of the measurement of variables. Second, the four levels of measurements, namely, nominal, ordinal, interval, and ratio, with examples are offered. Then, the application of these measurement levels to the statistical analysis of data at the univariate (descriptive statistics), bivariate, and multivariate (e.g., binary logistic and multiple linear regression) levels are discussed. This note is expected to be useful to the beginning (naïve) scholars for real-world application of statistical tools to analyze survey data.
A Critique of the Conventional Methods of Survey Item Transformations, with an Eye to Quantification
The Pope of Happiness, 2021
Our contribution to the Festschrift for Professor Ruut Veenhoven in celebration of his many contributions to happiness and social indicators research over the past fifty years focuses on the theme of making valid claims based on questionnaire response data from social surveys and assessments. In a series of publications since the early 1990s, Veenhoven has been calling for the development of survey research methods, as well as statistical and mathematical techniques, aimed at constructing reliable and valid measures to assess progress in societies using large-scale questionnaire studies among samples of the general population in different countries (Veenhoven 1993, 2008). The motivation for his call to action stems from the observation that many of us have had, that various surveys ask about the same topic, but the survey questions may not be the same. This difference could be due to personal preferences or styles for writing survey questions within the same country, or cross-cultural or crosslingual differences between countries. Even a cursory review of different social surveys shows that there is a mélange of ways of asking about and recording a survey respondent's characteristics such as age, not to mention the jumble of survey questions and response formats for measures of wellbeing and quality of life. Veenhoven remarked on several occasions that this methodological question was motivated by his need to prepare the data in his World Database of Happiness for a research synthesis such as a meta-analysis. The fundamental problem comes down to the fact that the questions that make up social surveys have response formats (also referred to as response rating scales) that may differ for different questionnaires within or between countries. The different response options of the rating scales for various surveys are not a problem, from a statistical point of view when analyses are
Development and application of a survey quality assessment model
2014
Full bibliographic details must be given when referring to, or quoting from full items including the author's name, the title of the work, publication details where relevant (place, publisher, date), pagination, and for theses or dissertations the awarding institution, the degree type awarded, and the date of the award.