Incremental Validity of Situational Judgment Tests for Task and Contextual Job Performance (original) (raw)
Related papers
Situational Judgment Tests: A Review of Practice and Constructs Assessed
International Journal of Selection and Assessment, 2001
In this article, we seek to summarize current practice concerning situational judgment tests in personnel selection. We begin by describing the manner in which situational judgment tests are developed and examining the diverse ways in which situational items are presented and scored. We then offer speculation concerning constructs assessed by situational judgment tests as well as discuss the legal aspect of situational judgment measures. We also review meta-analytic evidence concerning the construct validity of situational judgment tests and offer several new meta-analytic findings. Situational judgment tests are shown to be typically correlated moderately with general mental ability. Their primary personality correlates are emotional stability, conscientiousness, and agreeableness. Situational test scores also tend to increase with increasing years of job experience. The article concludes with a list of areas that need addressed in future research.
Journal of Occupational and Organizational Psychology, 2016
Despite their popularity and capacity to predict performance, there is no clear consensus on the internal measurement characteristics of situational judgment tests (SJTs). Contemporary propositions in the literature focus on treating SJTs as methods, as measures of dimensions, or as measures of situational responses. However, empirical evidence relating to the internal structure of SJT scores is lacking. Using generalizability theory, we decomposed multiple sources of variance for three different SJTs used with different samples of job candidates (N 1 = 2,320; N 2 = 989; N 3 = 7,934). Results consistently indicated that (a) the vast majority of reliable observed score variance reflected SJT-specific candidate main effects, analogous to a general judgment factor and that (b) the contribution of dimensions and situations to reliable SJT variance was, in relative terms, negligible. These findings do not align neatly with any of the proposals in the contemporary literature; however they do suggest an internal structure for SJTs. Practitioner Points To help optimize reliable variance, overall-level aggregation should be used when scoring SJTs. The majority of reliable variance in SJTs reflects a general performance factor, relative to variance pertaining to specific dimensions or situations. SJT developmental feedback should be delivered in terms of general SJT performance rather than on performance relating to specific dimensions or situations. Generalizability theory, although underutilised in multifaceted measurement, offers an approach to informing on the psychometric properties of SJTs that is well-suited to the complexities of SJT measurement designs.
Relationship Between a Situational Judgement Test and Ability and Personality Measures
Journal of Psychology in Africa, 2011
This study investigated the construct validity of a personnel selection instrument known as Scenarios. Data were collected from 93 managers employed by a leading international tobacco company using a situational judgement test (SJT), Scenarios (Howard & Choi, 2004), and two cognitive ability measures, namely Verbal Evaluation (VC1.1: Wilson, Baron, Borkowski, 1991), Interpreting Data (NC2.1: Wilson, Baron, Borkowski, 1991) and the Occupational Personality Questionnaire (OPQ32i: Bartram, Brown, Fleck, Inceoglu & Ward, 2006). Correlations among the measures were computed. The sample consisted of 71 males and 22 females. 52.86% of the sample was African, 8.57% Indian and 38.57% White. The SJT Scenarios was uncorrelated with cognitive ability tests and trait personality measures.
Journal of Applied Psychology, 2019
Recent theorizing and empirical evidence suggesting that situational judgment tests (SJTs) are more context-independent than previously thought has sparked a debate about the role of situation descriptions in SJTs. To contribute to this debate and add to our understanding of how SJTs work, this paper conceptually embeds SJT performance in a situation construal model and examines the effects of situation descriptions on the construct saturation and predictive validity of SJT scores, as well as on applicant perceptions. Across two studies (N = 1,092 and 578) and different SJTs, personality and cognitive ability were equally important determinants of SJT performance regardless of whether situation descriptions were presented or omitted. The effects of removing situation descriptions on the criterion-related validity of SJT scores differed depending on the breadth of the criteria. For predicting global job performance criteria (in-role performance and OCB), SJT validity was not significantly affected, whereas it decreased for predicting more specific criteria (interpersonal adaptability, efficacy for teamwork). Finally, the effects of omitting situation descriptions in SJTs on applicant perceptions were either negligible or small. Implications for SJT theory, research, and design are discussed.
Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis
Human Performance, 2008
In this article, we describe a systematic review of mean race and sex differences in situational judgment test (SJT) performance. On average, White test takers perform better on SJTs than Black, Hispanic, and Asian test takers. Female examinees perform slightly better than male test takers on SJTs. We investigate two moderators of these differences: loading of g or personality on the SJT, and response instructions. Mean race differences between Black, Hispanic, Asian and White examinees in SJT performance are largely explained by the cognitive loading of the SJT such that the larger the cognitive load, the larger the mean race differences. Regarding the effect of personality loadings of SJTs on race differences, Black-White and Asian-White differences are smaller to the extent that the SJT is correlated with emotional stability and Hispanic-White differences are smaller to the extent that SJTs are correlated with conscientiousness and agreeableness. Cognitive loading has minimal effect on male-female SJT score differences; however, SJT score differences are larger, favoring women, when SJTs are correlated with conscientiousness and agreeableness. Concerning response instructions, knowledge response instructions showed greater race differences than behavioral tendency instructions. The mean correlations show that these differences are largely because of the greater g loading of knowledge instructions. A second study showed that when used in hiring, SJTs are likely to have adverse impact by race but not by sex. Situational judgment tests (SJTs) assess an applicant's judgment regarding situations encountered in the work place (
Situational judgment tests: An overview of current research
Human Resource Management Review, 2009
Situational judgment tests (SJTs) are popular personnel selection tests. To aid researchers, the paper summarizes the current knowledge and where knowledge gaps exist. To guide practice, the paper provides evidence-based recommendations. The paper begins with a brief history of SJTs, presents likely reasons for the resurgence of SJT research and practice, and summarizes the theoretical basis of SJTs. Then, the distinction between personnel selection methods and constructs is reviewed as it is particularly important in understanding SJTs. SJT research relevant to reliability and validity is summarized as is research relevant to the implementation of SJTs. The paper concludes with recommendations for practice and an agenda for future research.
International Journal of Selection and Assessment, 2024
To date, a limited set of studies have compared the criterion-related validity of low-fidelity (SJT) versus high-fidelity (AC) simulations for predicting job performance. Unfortunately, these studies validated these simulations through the overall assessment rating (OAR) instead of on the basis of specific dimensions. Given SJTs and ACs were compared that measured different dimensions, our understanding of the relative and comparative validity of these assessment approaches in measuring the same set of dimensions is still limited. Therefore, this study aims to conduct a head-to-head comparison of the criterion-related validity of the AC and the SJT (and their incremental validity) while keeping the performance dimensions under investigation constant. Data were collected from 406 applicants for supervisory and management positions in a large Iranian steel industry company. In this process, a general mental ability test, a personality inventory, an SJT, and an AC were used as predictors, and supervisory ratings of job performance dimensions (Thinking, Feeling, and Power) served as criteria. The AC had relatively high validity for all three dimensions, whereas the SJT had a similar validity only for the Thinking dimension. So, the SJT was significantly weaker in assessing the Feeling and Power dimensions. These results were confirmed by incremental validity analyses. Overall, this study shows that understanding the relationships between predictor and criterion dimensions plays a critical role in developing valid selection systems.
Situational judgment tests, response instructionns, and validity: A meta-analysis
2007
Situational judgment tests (SJTs) are personnel selection instruments that present job applicants with work-related situations and possible responses to the situations. There are typically 2 types of instructions: behavioral tendency and knowledge. Behavioral tendency instructions ask respondents to identify how they would likely behave in a given situation. Knowledge instructions ask respondents to evaluate the effectiveness of possible responses to a given situation. Results showed that response instructions influenced the constructs measured by the tests. Tests with knowledge instructions had higher correlations with cognitive ability. Tests with behavioral tendency instructions showed higher correlations with personality constructs. Results also showed that response instructions had little moderating effect on criterion-related validity. Supplemental analyses showed that the moderating effect of response instructions on construct validity was not due to systematic differences in item content. SJTs have incremental validity over cognitive ability, the Big 5, and over a composite of cognitive ability and the Big 5. Research on SJTs for employee selection has increased dramatically in recent years (Weekley & Ployhart, 2006). SJTs present applicants with work-related situations and possible responses to the situations. The criterion-related validity of SJTs has been evaluated in several primary studies (
2007
In this chapter, we give an overview of situational judgment tests (SJTs) as selection instruments. Their history, basic characteristics, and development are presented. The available research evidence regarding their reliability, construct-related validity, criterionrelated validity, incremental validity, subgroup differences, and test-taker perceptions is also reviewed. As a general conclusion, the increasing popularity of SJTs in personnel selection seems to be accredited to their potential to capture a variety of constructs and for different purposes. Additionally, SJTs are able to predict several job-related and/or academic criteria while at the same time offering prospects permitting to select for diversity.
International Journal of Selection and Assessment, 2011
We report two studies that investigate single-response situational judgment tests (SJTs) as measures of job knowledge. Study 1 examines relationships between job knowledge measured by a single-response SJT, personality, and performance for museum tour guides. Study 2 extends Study 1's findings with a sample of volunteers using a single-response SJT about volunteerism. In both studies, personality was related to knowledge, and knowledge predicted performance. In Study 2, knowledge accounted for incremental variance in performance beyond personality, but personality added no incremental variance beyond knowledge. Results suggest that knowledge of effective behavior and knowledge of ineffective behavior are separate constructs.These studies demonstrate the validity of single-response SJTs and provide evidence that knowing what to do and what not to do are separate domains.