Beyond Alpha: An Empirical Examination of the Effects of Different Sources of Measurement Error on Reliability Estimates for Measures of Individual Differences Constructs (original) (raw)
Related papers
2010
43 A basis of knowledge in any scientific domain is measurement. A huge body of literature has been written thus far on the nature of measurement. There is, however, an obvious diversity in the ways that measurement has been defined in traditional quantitative science, among the behavioural sciences, or under various conceptions within philosophy of science (cf. Andrich, 1988; Brennan, 2001; McDonnald, 1999; Michell, 2005; Nunnally & Bernstein, 1994; Stemler & Birney, 2006; Zumbo, 2007). Notwithstanding the definitional and conceptual differences, a clear distinction, as stressed by Nunnally and Bernstein (1994), should be made between measurement as a process and the standards for validating measures. Whereas measurement process involves issues on quantification of attributes and/or classification of objects with regard to given attributes, measurement or test score validation involves considerations as to how well the measurement method can explain important phenomena. Having in m...
Educational and Psychological Measurement, 2000
Meta-analysis was used to cumulate reliabilities of personality scale scores. A total of 848 coefficients of stability and 1,359 internal consistency reliabilities across the Big Five factors of personality were examined. The frequency-weighted mean coefficients of stability were .75 (SD = .10, K = 221), .76 (SD = .12, K = 176), .71 (SD = .13, K = 139), .69 (SD = .14, K = 119), and .72 (SD = .13, K = 193) for Emotional Stability, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness, respectively. The corresponding internal consistency reliabilities were .78 (SD = .11, K = 370), .78 (SD = .09, K = 307), .73 (SD = .12, K = 251), .75 (SD = .11, K = 123), and .78 (SD = .10, K = 307). Sample-size-weighted means also were computed. The dimension of personality being rated does not appear to strongly moderate either the internal consistency or the test-retest reliabilities. Implications for personality assessment are discussed.
2017
Recent personality literature has proposed that Cronbach’s generalisability theory offers a substantive ground for the integration of inter-individual differences and intra¬personal process approaches to personality trait research. Generalisability theory has the advantage of maximising psychometric dependability of behavioural measurements, but does not demand reconciliation to the environment that is relevant to the life circumstances of the participant. A conceptual analysis contrasting generalisability theory with Egon Brunswik’s conceptual framework for psychology is presented. Recommendations regarding the connection between theory and methodological practice follow, in light of an evolutionary approach to personality and individual differences. An example that highlights the contrast between Cronbachian and Brunswikian approaches is presented, resulting in a recommendation to revisit the concept of ecological validity and representational research design, to better account fo...
Applied individual differences research and its quantitative methods
Psychology, Public Policy, and Law, 1996
Applied individual differences research is one of the few branches of psychological science that have systematically amassed a coherent body of empirical knowledge withstanding the test of time. Furthermore, it has exemplified how practice can facilitate basic science. While examining relatively stable behavioral attributes (abilities, personality, vocational preferences), which form the bases of thenlongitudinal forecasts pertaining to broad behavioral patterns and proclivities, differential psychologists appear to have uncovered keys to behavioral development. They also have contributed to a better understanding of human diversity and its many unique ("idiographic") manifestations. This issue of the journal tells a story about this domain of knowledge; its history in facilitating practice in educational, clinical, and counseling psychology; and its role in personnel development and selection in business, industry, and the military. Some ways in which individual differences dimensions converge on other psychological viewpoints (phenomenology, Skinnerian behaviorism) are reflected, as is then-bearing on theories of human development more generally. When examining human behavioral phenomena from a psychological point of view, taking a multifaceted (individual differences) approach appears to be critical; this is especially important when designing opportunities and policies for positive development in clinics, in schools, at work, and in society at large. Applied individual differences research and its quantitative methods has a long and impressive history. Over the years, several names have been offered for this branch of psychology. Binet called it individual psychology, Stern preferred differential psychology, but it was E. L. Thorndike who christened the most frequently used descriptor, the psychology of individual differences. Much of the early work in this area was aimed at assessing behavioral dimensions of human variation-abilities, personality, and vocational preferences-but not just any dimensions. Concentration was restricted to uncovering parameters that carried real-world significance. Dimensions that never traveled outside the laboratory of brass instruments or the factor analyst's Euclidean space were paid little attention. Examining elegant trivia, or what Truman Kelley (1939) called psychological factors-of-no-importance, was shunned. Hugo Munsterberg's well-known substantive appraisal of structuralism ("Yes, it is precise but it is not useful;" cf. Landy, 1992, p. 788) also describes the type of work that was to be avoided in the psychology of individual differences. Dimensions holding prophecy were what was aimed for-but prophecy for useful things. Thus, validation criteria typically consisted of behaviors observed after large temporal gaps from initial assessments. Because early differential
Individual differences and their measurement: A review of 100 years of research
The Journal of applied psychology, 2017
This article reviews 100 years of research on individual differences and their measurement, with a focus on research published in the Journal of Applied Psychology. We focus on 3 major individual differences domains: (a) knowledge, skill, and ability, including both the cognitive and physical domains; (b) personality, including integrity, emotional intelligence, stable motivational attributes (e.g., achievement motivation, core self-evaluations), and creativity; and (c) vocational interests. For each domain, we describe the evolution of the domain across the years and highlight major theoretical, empirical, and methodological developments, including relationships between individual differences and variables such as job performance, job satisfaction, and career development. We conclude by discussing future directions for individual differences research. Trends in the literature include a growing focus on substantive issues rather than on the measurement of individual differences, a d...
Self-discrepancies and negative affect: A primer on when to look for specificity, and how to find it
Australian Journal of Psychology, 2005
There is substantial evidence that discrepancies within the self-system produce emotional distress. However, whether specific types of discrepancy are related to different types of negative affect remains contentious. At the heart of self-discrepancy theory (SDT: Higgins, 1987 is the assumption that different types of discrepancies are related to distinctive emotional states, with discrepancies between the actual and ideal selves being uniquely related to dejection-related emotion and discrepancies between the actual and ought selves being uniquely related to agitation-related emotion. Research examining this proposition has demonstrated that the magnitudes of these discrepancies are substantially correlated. As a result, some researchers have questioned whether they are functionally independent (e.g., . In addition, other researchers have failed to support the hypothesized unique relationships (e.g., Ozgul, Heubeck, . Together these two types of research finding have been interpreted as presenting a challenge to SDT. It is our contention that this interpretation is inaccurate. In this paper, we review the assumptions made when testing for these distinct relationships. Specifically, we examine the necessary conditions under which the functional independence of discrepancies is apparent, and the statistical methods appropriate to test these relationships. We also comment on the measurement of self-discrepancies, and fundamental problems in the interpretation of null findings. We conclude that studies using appropriate methodological and statistical procedures have produced ample evidence that discriminant relationships exist, and we encourage researchers to further investigate the conditions under which these relationships are most apparent.
Confidence judgments in studies of individual differences
Personality and Individual Differences, 1996
properties of scores derived from calibration curves (overconfidence, calibration, resolution, and slope) and an analogue of overconfidence that is based on a posttest estimate of the proportion of correctly solved items. Four tests from the theory of fluid and crystallized intelligence were used, and two of these tests employed both sequential and simultaneous methods of item presentation.
European Journal of Psychological Assessment
Although Objective Personality Tests (OPTs) have a long history in psychology and the field of psychological assessment, their validity, and reliability have not yet been sufficiently studied. In this study, we examined the convergent and discriminant validity of objective (personality) tests, Implicit Association Tests (IATs), and self-report measures for the assessment of conscientiousness and intelligence. Moreover, the convergent and discriminant validity of these measures was assessed on the trait (stable) and occasion specific (momentary) level by using the multimethod latent state-trait (MM-LST) model proposed by Courvoisier, Nussbeck, Eid, and Cole (2008) which allows for the decomposition of different sources of variance. Data from 367 students assessed on three different measurement occasions was incorporated. Results indicate generally low convergence of OPTs with data gained by other approaches. Additional analyses revealed that the OPTs used assess stable rather than mo...
Situational effects in trait assessment: The FPI, NEOFFI, and EPI questionnaires
European Journal of Personality, 1995
while most researchers do agree now that situations may have an effect in the assessment of traits, the consequences have been neglected, so far: if situations aflect the assessment of traits we have to take this fact into account in studies on reliability and validity of measurement instruments and their application. In the theoretical part of this article we provide a more formal exposition of this point, introducing the basic concepts of latent state-trait (LST) theory. LST theory and the associated models allow for the estimation of the situational impact on trait measures in non-experimental, correlational studies. In the empirical part, LST.theory is applied to three well known trait questionnaires: the Freiburg Personality Inventory, the NEO Five-Factor Inventory and the Eysenck Personality Inventory. It is shown that significant proportions of the variances of the scales of these questionnaires are due to situational effects. The following consequences of this finding are discussed (i) Instead of the reliability coefficient, the proportion of variance due to the latent trait, the consistency coefjcient, should be used for the estimation of confi&nce intervals for trait scores. (ii) To reduce the situational effects on trait estimates it may be useful to base such an estimate on several occasions, ie., to aggregate data across occasions. (iii) Reliability and validity studies should not only be based on a sample of persons representative of those to whom the test will be applied; they should also be conducted in situational contexts representative of the intended applications.