Alexander Robitzsch - Academia.edu (original) (raw)

Papers by Alexander Robitzsch

Research paper thumbnail of Measuring the Mind: (2005

Zeitschrift Fur Padagogische Psychologie, 2007

Research paper thumbnail of Have Cognitive Diagnostic Models Delivered Their Goods? Some Substantial and Methodological Concerns

Measurement: Interdisciplinary Research & Perspective, 2009

      ... more                                                          

Research paper thumbnail of Intelligence Differentiation in Early Childhood

Journal of Individual Differences, 2011

We investigated the usefulness of the Over-Claiming Questionnaire (OCQ) as a measure of cognitive... more We investigated the usefulness of the Over-Claiming Questionnaire (OCQ) as a measure of cognitive abilities. In OCQs respondents are asked to rate their familiarity with items of academic or everyday knowledge . Some items exist in reality (reals), and others do not (foils). We developed four OCQs, each consisting of 40 reals and 8 foils from the domains of Science, Humanities and Civics. The OCQs were administered in a longitudinal rotation design to 112 participants who attended the 9th school grade at the beginning of the study. In latent variable regression analyses 53% of variation in the reals could be explained by fluid and crystallized intelligence and over-claiming as indicated by responses to foils. Further variation in responses to reals and foils was explained by intellectual engagement. Our results show that self-reported knowledge, although positively related to measures of ability, to a large extent reflects over-claiming.

Research paper thumbnail of Gender differences in the mathematics achievements of German primary school students: results from a German large-scale study

Zdm, 2008

In Germany, national standards for mathematics for the end of primary school were established in ... more In Germany, national standards for mathematics for the end of primary school were established in 2004. In the present study, data were collected to evaluate these standards, and were used to compare the mathematical skills of girls and boys. Many studies have shown that gender differences are strongest at the highest levels of education. The findings from primary school are less consistent. Thus, in our study we analyzed achievement differences in a sample of approximately 10,000 third and fourth graders, representative of the German elementary school population. Gender-specific competencies were compared in the different content domains, both for the general mathematical competence, and for the cognitive levels of the tasks. Overall, boys outperformed girls, but substantial variation was found between the content domains and general mathematical achievement. Differences were higher in grade three than in grade four. The proportion of boys in the classroom did not appear to affect the individual level of performance. Analysis of the items on which boys or girls clearly outperformed each other reproduced a pattern of specific item characteristics predicting gender bias consistent with those reported in previous studies in other countries.

Research paper thumbnail of Prognostische Relevanz krisenhafter Behandlungsepisoden für den Behandlungserfolg stationärer Psychotherapie im psychodynamischen Grundverfahren

Psychotherapie Psychosomatik Medizinische Psychologie, 2007

Research paper thumbnail of Prozeßdynamik stationärer Psychotherapie – Untersuchung von Behandlungsverläufen mittels Funktionaler Datenanalyse FDA

Psychotherapie Psychosomatik Medizinische Psychologie, 2008

Research paper thumbnail of The Multilevel Latent Covariate Model: A New, More Reliable Approach to Group-Level Effects in Contextual Studies

Psychological Methods, 2008

In multilevel modeling (MLM), group level (L2) characteristics are often measured by aggregating ... more In multilevel modeling (MLM), group level (L2) characteristics are often measured by aggregating individual level (L1) characteristics within each group as a means of assessing contextual effects (e.g., group-average effects of SES, achievement, climate). Most previous applications have used a multilevel manifest covariate (MMC) approach, in which the observed (manifest) group mean is assumed to have no measurement error. This paper shows mathematically and with simulation results that this MMC approach can result in substantially biased estimates of contextual effects and can substantially underestimate the associated standard errors, depending on the number of L1 individuals in each of the groups, the number of groups, the intraclass correlation, the sampling ratio (the percentage of cases within each group sampled), and the nature of the data. To address this pervasive problem, we introduce a new multilevel latent covariate (MLC) approach that corrects for unreliability at L2 and results in unbiased estimates of L2 constructs under appropriate conditions. However, our simulation results also suggest that the contextual effects estimated in typical research situations (e.g., fewer than 100 groups) may be highly unreliable. Furthermore, under some circumstances when the sampling ratio approaches 100%, the MMC approach provides more accurate estimates. Based on three simulations and two real-data applications, we critically evaluate the MMC and MLC approaches and offer suggestions as to when researchers should most appropriately use one, the other, or a combination of both approaches.

Research paper thumbnail of Doubly-Latent Models of School Contextual Effects: Integrating Multilevel and Structural Equation Approaches to Control Measurement and Sampling Error

Multivariate Behavioral Research, 2009

Broadly, contextual studies evaluate whether group-level (L2) characteristics (e.g., family, peer... more Broadly, contextual studies evaluate whether group-level (L2) characteristics (e.g., family, peer group, classroom, school, workplace, country) contribute to out-FIGURE 2A Set of four multilevel latent contextual models (see ) that are latent or manifest in relation to sampling items (and correction for measurement error) and latent or manifest in relation to sampling students (and correction for sampling error).

Research paper thumbnail of Umgang mit fehlenden Werten in der psychologischen Forschung

Psychologische Rundschau, 2007

Research paper thumbnail of Steht Transparenz einer adäquaten Datenauswertung im Wege?: Eine Antwort auf

Psychologische Rundschau, 2008

... auf (vgl. McKnight, McKnight, Souraya & Figueredo, 2007). Unser ... MCAR sein. Der hä... more ... auf (vgl. McKnight, McKnight, Souraya & Figueredo, 2007). Unser ... MCAR sein. Der häufigere Fall dürfte darin bestehen, dass nicht klar ist, ob die fehlenden Angaben nun MAR oder MNAR (Missing Not At Ran-dom) sind. Daraus ...

Research paper thumbnail of Assessing the impact of learning environments: How to use student ratings of classroom or school characteristics in multilevel modeling

Contemporary Educational Psychology, 2009

... of the learning environment (private beta press), and the classroom level, reflecting the ave... more ... of the learning environment (private beta press), and the classroom level, reflecting the averageperception of the ... In the case of student ratings, the ICC(1) can be seen as a measure of effect ... students in each class might be asked to rate the quality of the homework assigned by ...

Research paper thumbnail of Steht Transparenz einer adäquaten Datenauswertung im Wege?: Eine Antwort auf

Psychologische Rundschau, 2008

... auf (vgl. McKnight, McKnight, Souraya & Figueredo, 2007). Unser ... MCAR sein. Der hä... more ... auf (vgl. McKnight, McKnight, Souraya & Figueredo, 2007). Unser ... MCAR sein. Der häufigere Fall dürfte darin bestehen, dass nicht klar ist, ob die fehlenden Angaben nun MAR oder MNAR (Missing Not At Ran-dom) sind. Daraus ...

Research paper thumbnail of A 2 × 2 Taxonomy of Multilevel Latent Contextual Models: Accuracy–Bias Trade-Offs in Full and Partial Error Correction Models

Psychological Methods, 2011

In multilevel modeling, group-level variables (L2) for assessing contextual effects are frequentl... more In multilevel modeling, group-level variables (L2) for assessing contextual effects are frequently generated by aggregating variables from a lower level (L1). A major problem of contextual analyses in the social sciences is that there is no error-free measurement of constructs. In the present article, 2 types of error occurring in multilevel data when estimating contextual effects are distinguished: unreliability that is due to measurement error and unreliability that is due to sampling error. The fact that studies may or may not correct for these 2 types of error can be translated into a 2 × 2 taxonomy of multilevel latent contextual models comprising 4 approaches: an uncorrected approach, partial correction approaches correcting for either measurement or sampling error (but not both), and a full correction approach that adjusts for both sources of error. It is shown mathematically and with simulated data that the uncorrected and partial correction approaches can result in substantially biased estimates of contextual effects, depending on the number of L1 individuals per group, the number of groups, the intraclass correlation, the number of indicators, and the size of the factor loadings. However, the simulation study also shows that partial correction approaches can outperform full correction approaches when the data provide only limited information in terms of the L2 construct (i.e., small number of groups, low intraclass correlation). A real-data application from educational psychology is used to illustrate the different approaches.

Research paper thumbnail of Large-Scale Assessment of Change in Student Achievement: Dutch Primary School Students’ Results on Written Division in 1997 and 2004 as an Example

Psychometrika, 2009

This article discusses large-scale assessment of change in student achievement and takes the stud... more This article discusses large-scale assessment of change in student achievement and takes the study by Hickendorff, Heiser, Van Putten, and Verhelst (2009) as an example. This study compared the achievement of students in the Netherlands in 1997 and 2004 on written division problems. Based on this comparison, they claim that there is a performance decline in this subdomain of mathematics, and that there is a move from applying the digit-based long division algorithm to a less accurate way of working without writing down anything. In our discussion of this study, we address methodological challenges that come in when investigating long-term trends in student achievements, such as the need for adequate operationalizations, the influence of the time of measurement and the necessity of the comparability of assessments, the effect of the assessment format, and the importance of inclusion relevant covariates in item response models. All these issues matter when assessing change in student achievement.

Research paper thumbnail of LARGE-SCALE ASSESSMENT OF CHANGE IN STUDENT ACHIEVEMENT: DUTCH PRIMARY SCHOOL STUDENTS' RESULTS ON WRITTEN DIVISION IN 1997 AND 2004 AS AN EXAMPLE MARJA VAN DEN HEUVEL-PANHUIZEN FREUDENTHAL INSTITUTE FOR SCIENCE AND MATHEMATICS EDUCATION (FISME) AND INSTITUTE FOR EDUCATIONAL PROGRESS, HUMBOLDT UN...

This article discusses large-scale assessment of change in student achievement and takes the stud... more This article discusses large-scale assessment of change in student achievement and takes the study by as an example. This study compared the achievement of students in the Netherlands in 1997 and 2004 on written division problems. Based on this comparison, they claim that there is a performance decline in this subdomain of mathematics, and that there is a move from applying the digit-based long division algorithm to a less accurate way of working without writing down anything. In our discussion of this study, we address methodological challenges that come in when investigating long-term trends in student achievements, such as the need for adequate operationalizations, the influence of the time of measurement and the necessity of the comparability of assessments, the effect of the assessment format, and the importance of inclusion relevant covariates in item response models. All these issues matter when assessing change in student achievement.

Research paper thumbnail of Exploratory Structural Equation Modeling, Integrating CFA and EFA: Application to Students' Evaluations of University Teaching

Structural Equation Modeling-a Multidisciplinary Journal, 2009

Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), path analysis, and stru... more Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), path analysis, and structural equation modeling (SEM) have long histories in clinical research. Although CFA has largely superseded EFA, CFAs of multidimensional constructs typically fail to meet standards of good measurement: goodness of fit, measurement invariance, lack of differential item functioning, and well-differentiated factors in support of discriminant validity. Part of the problem is undue reliance on overly restrictive CFAs in which each item loads on only one factor. Exploratory SEM (ESEM), an overarching integration of the best aspects of CFA/SEM and traditional EFA, provides confirmatory tests of a priori factor structures, relations between latent factors and multigroup/multioccasion tests of full (mean structure) measurement invariance. It incorporates all combinations of CFA factors, ESEM factors, covariates, grouping/multiple-indicator multiple-cause (MIMIC) variables, latent growth, and complex structures that typically have required CFA/SEM. ESEM has broad applicability to clinical studies that are not appropriately addressed either by traditional EFA or CFA/SEM.

Research paper thumbnail of Are There Test Administrator Effects in Large-Scale Educational Assessments?: Using Cross-Classified Multilevel Analysis to Probe for Effects on Mathematics Achievement and Sample Attrition

Methodology: European Journal of Research Methods for The Behavioral and Social Sciences, 2007

ABSTRACT

Research paper thumbnail of Statistische Artefakte bei Kontexteffekten in derpädagogisch-psychologischen Forschung

Zeitschrift Fur Padagogische Psychologie, 2002

... Oliver Lüdtke. Max-Planck-Institut für Bildungsforschung, Berlin. Alexander Robitzsch. Techni... more ... Oliver Lüdtke. Max-Planck-Institut für Bildungsforschung, Berlin. Alexander Robitzsch. Technische Universität, Dresden. Olaf Köller. Max-Planck-Institut für ... Für den Einfluss der Intelligenz auf Schulleistungen zeigten Mortimore, Simmons, Stoll, Lewis und Ecob (1989), dass nach ...

Research paper thumbnail of Measuring the Mind: (2005

Zeitschrift Fur Padagogische Psychologie, 2007

Research paper thumbnail of Have Cognitive Diagnostic Models Delivered Their Goods? Some Substantial and Methodological Concerns

Measurement: Interdisciplinary Research & Perspective, 2009

      ... more                                                          

Research paper thumbnail of Intelligence Differentiation in Early Childhood

Journal of Individual Differences, 2011

We investigated the usefulness of the Over-Claiming Questionnaire (OCQ) as a measure of cognitive... more We investigated the usefulness of the Over-Claiming Questionnaire (OCQ) as a measure of cognitive abilities. In OCQs respondents are asked to rate their familiarity with items of academic or everyday knowledge . Some items exist in reality (reals), and others do not (foils). We developed four OCQs, each consisting of 40 reals and 8 foils from the domains of Science, Humanities and Civics. The OCQs were administered in a longitudinal rotation design to 112 participants who attended the 9th school grade at the beginning of the study. In latent variable regression analyses 53% of variation in the reals could be explained by fluid and crystallized intelligence and over-claiming as indicated by responses to foils. Further variation in responses to reals and foils was explained by intellectual engagement. Our results show that self-reported knowledge, although positively related to measures of ability, to a large extent reflects over-claiming.

Research paper thumbnail of Gender differences in the mathematics achievements of German primary school students: results from a German large-scale study

Zdm, 2008

In Germany, national standards for mathematics for the end of primary school were established in ... more In Germany, national standards for mathematics for the end of primary school were established in 2004. In the present study, data were collected to evaluate these standards, and were used to compare the mathematical skills of girls and boys. Many studies have shown that gender differences are strongest at the highest levels of education. The findings from primary school are less consistent. Thus, in our study we analyzed achievement differences in a sample of approximately 10,000 third and fourth graders, representative of the German elementary school population. Gender-specific competencies were compared in the different content domains, both for the general mathematical competence, and for the cognitive levels of the tasks. Overall, boys outperformed girls, but substantial variation was found between the content domains and general mathematical achievement. Differences were higher in grade three than in grade four. The proportion of boys in the classroom did not appear to affect the individual level of performance. Analysis of the items on which boys or girls clearly outperformed each other reproduced a pattern of specific item characteristics predicting gender bias consistent with those reported in previous studies in other countries.

Research paper thumbnail of Prognostische Relevanz krisenhafter Behandlungsepisoden für den Behandlungserfolg stationärer Psychotherapie im psychodynamischen Grundverfahren

Psychotherapie Psychosomatik Medizinische Psychologie, 2007

Research paper thumbnail of Prozeßdynamik stationärer Psychotherapie – Untersuchung von Behandlungsverläufen mittels Funktionaler Datenanalyse FDA

Psychotherapie Psychosomatik Medizinische Psychologie, 2008

Research paper thumbnail of The Multilevel Latent Covariate Model: A New, More Reliable Approach to Group-Level Effects in Contextual Studies

Psychological Methods, 2008

In multilevel modeling (MLM), group level (L2) characteristics are often measured by aggregating ... more In multilevel modeling (MLM), group level (L2) characteristics are often measured by aggregating individual level (L1) characteristics within each group as a means of assessing contextual effects (e.g., group-average effects of SES, achievement, climate). Most previous applications have used a multilevel manifest covariate (MMC) approach, in which the observed (manifest) group mean is assumed to have no measurement error. This paper shows mathematically and with simulation results that this MMC approach can result in substantially biased estimates of contextual effects and can substantially underestimate the associated standard errors, depending on the number of L1 individuals in each of the groups, the number of groups, the intraclass correlation, the sampling ratio (the percentage of cases within each group sampled), and the nature of the data. To address this pervasive problem, we introduce a new multilevel latent covariate (MLC) approach that corrects for unreliability at L2 and results in unbiased estimates of L2 constructs under appropriate conditions. However, our simulation results also suggest that the contextual effects estimated in typical research situations (e.g., fewer than 100 groups) may be highly unreliable. Furthermore, under some circumstances when the sampling ratio approaches 100%, the MMC approach provides more accurate estimates. Based on three simulations and two real-data applications, we critically evaluate the MMC and MLC approaches and offer suggestions as to when researchers should most appropriately use one, the other, or a combination of both approaches.

Research paper thumbnail of Doubly-Latent Models of School Contextual Effects: Integrating Multilevel and Structural Equation Approaches to Control Measurement and Sampling Error

Multivariate Behavioral Research, 2009

Broadly, contextual studies evaluate whether group-level (L2) characteristics (e.g., family, peer... more Broadly, contextual studies evaluate whether group-level (L2) characteristics (e.g., family, peer group, classroom, school, workplace, country) contribute to out-FIGURE 2A Set of four multilevel latent contextual models (see ) that are latent or manifest in relation to sampling items (and correction for measurement error) and latent or manifest in relation to sampling students (and correction for sampling error).

Research paper thumbnail of Umgang mit fehlenden Werten in der psychologischen Forschung

Psychologische Rundschau, 2007

Research paper thumbnail of Steht Transparenz einer adäquaten Datenauswertung im Wege?: Eine Antwort auf

Psychologische Rundschau, 2008

... auf (vgl. McKnight, McKnight, Souraya & Figueredo, 2007). Unser ... MCAR sein. Der hä... more ... auf (vgl. McKnight, McKnight, Souraya & Figueredo, 2007). Unser ... MCAR sein. Der häufigere Fall dürfte darin bestehen, dass nicht klar ist, ob die fehlenden Angaben nun MAR oder MNAR (Missing Not At Ran-dom) sind. Daraus ...

Research paper thumbnail of Assessing the impact of learning environments: How to use student ratings of classroom or school characteristics in multilevel modeling

Contemporary Educational Psychology, 2009

... of the learning environment (private beta press), and the classroom level, reflecting the ave... more ... of the learning environment (private beta press), and the classroom level, reflecting the averageperception of the ... In the case of student ratings, the ICC(1) can be seen as a measure of effect ... students in each class might be asked to rate the quality of the homework assigned by ...

Research paper thumbnail of Steht Transparenz einer adäquaten Datenauswertung im Wege?: Eine Antwort auf

Psychologische Rundschau, 2008

... auf (vgl. McKnight, McKnight, Souraya & Figueredo, 2007). Unser ... MCAR sein. Der hä... more ... auf (vgl. McKnight, McKnight, Souraya & Figueredo, 2007). Unser ... MCAR sein. Der häufigere Fall dürfte darin bestehen, dass nicht klar ist, ob die fehlenden Angaben nun MAR oder MNAR (Missing Not At Ran-dom) sind. Daraus ...

Research paper thumbnail of A 2 × 2 Taxonomy of Multilevel Latent Contextual Models: Accuracy–Bias Trade-Offs in Full and Partial Error Correction Models

Psychological Methods, 2011

In multilevel modeling, group-level variables (L2) for assessing contextual effects are frequentl... more In multilevel modeling, group-level variables (L2) for assessing contextual effects are frequently generated by aggregating variables from a lower level (L1). A major problem of contextual analyses in the social sciences is that there is no error-free measurement of constructs. In the present article, 2 types of error occurring in multilevel data when estimating contextual effects are distinguished: unreliability that is due to measurement error and unreliability that is due to sampling error. The fact that studies may or may not correct for these 2 types of error can be translated into a 2 × 2 taxonomy of multilevel latent contextual models comprising 4 approaches: an uncorrected approach, partial correction approaches correcting for either measurement or sampling error (but not both), and a full correction approach that adjusts for both sources of error. It is shown mathematically and with simulated data that the uncorrected and partial correction approaches can result in substantially biased estimates of contextual effects, depending on the number of L1 individuals per group, the number of groups, the intraclass correlation, the number of indicators, and the size of the factor loadings. However, the simulation study also shows that partial correction approaches can outperform full correction approaches when the data provide only limited information in terms of the L2 construct (i.e., small number of groups, low intraclass correlation). A real-data application from educational psychology is used to illustrate the different approaches.

Research paper thumbnail of Large-Scale Assessment of Change in Student Achievement: Dutch Primary School Students’ Results on Written Division in 1997 and 2004 as an Example

Psychometrika, 2009

This article discusses large-scale assessment of change in student achievement and takes the stud... more This article discusses large-scale assessment of change in student achievement and takes the study by Hickendorff, Heiser, Van Putten, and Verhelst (2009) as an example. This study compared the achievement of students in the Netherlands in 1997 and 2004 on written division problems. Based on this comparison, they claim that there is a performance decline in this subdomain of mathematics, and that there is a move from applying the digit-based long division algorithm to a less accurate way of working without writing down anything. In our discussion of this study, we address methodological challenges that come in when investigating long-term trends in student achievements, such as the need for adequate operationalizations, the influence of the time of measurement and the necessity of the comparability of assessments, the effect of the assessment format, and the importance of inclusion relevant covariates in item response models. All these issues matter when assessing change in student achievement.

Research paper thumbnail of LARGE-SCALE ASSESSMENT OF CHANGE IN STUDENT ACHIEVEMENT: DUTCH PRIMARY SCHOOL STUDENTS' RESULTS ON WRITTEN DIVISION IN 1997 AND 2004 AS AN EXAMPLE MARJA VAN DEN HEUVEL-PANHUIZEN FREUDENTHAL INSTITUTE FOR SCIENCE AND MATHEMATICS EDUCATION (FISME) AND INSTITUTE FOR EDUCATIONAL PROGRESS, HUMBOLDT UN...

This article discusses large-scale assessment of change in student achievement and takes the stud... more This article discusses large-scale assessment of change in student achievement and takes the study by as an example. This study compared the achievement of students in the Netherlands in 1997 and 2004 on written division problems. Based on this comparison, they claim that there is a performance decline in this subdomain of mathematics, and that there is a move from applying the digit-based long division algorithm to a less accurate way of working without writing down anything. In our discussion of this study, we address methodological challenges that come in when investigating long-term trends in student achievements, such as the need for adequate operationalizations, the influence of the time of measurement and the necessity of the comparability of assessments, the effect of the assessment format, and the importance of inclusion relevant covariates in item response models. All these issues matter when assessing change in student achievement.

Research paper thumbnail of Exploratory Structural Equation Modeling, Integrating CFA and EFA: Application to Students' Evaluations of University Teaching

Structural Equation Modeling-a Multidisciplinary Journal, 2009

Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), path analysis, and stru... more Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), path analysis, and structural equation modeling (SEM) have long histories in clinical research. Although CFA has largely superseded EFA, CFAs of multidimensional constructs typically fail to meet standards of good measurement: goodness of fit, measurement invariance, lack of differential item functioning, and well-differentiated factors in support of discriminant validity. Part of the problem is undue reliance on overly restrictive CFAs in which each item loads on only one factor. Exploratory SEM (ESEM), an overarching integration of the best aspects of CFA/SEM and traditional EFA, provides confirmatory tests of a priori factor structures, relations between latent factors and multigroup/multioccasion tests of full (mean structure) measurement invariance. It incorporates all combinations of CFA factors, ESEM factors, covariates, grouping/multiple-indicator multiple-cause (MIMIC) variables, latent growth, and complex structures that typically have required CFA/SEM. ESEM has broad applicability to clinical studies that are not appropriately addressed either by traditional EFA or CFA/SEM.

Research paper thumbnail of Are There Test Administrator Effects in Large-Scale Educational Assessments?: Using Cross-Classified Multilevel Analysis to Probe for Effects on Mathematics Achievement and Sample Attrition

Methodology: European Journal of Research Methods for The Behavioral and Social Sciences, 2007

ABSTRACT

Research paper thumbnail of Statistische Artefakte bei Kontexteffekten in derpädagogisch-psychologischen Forschung

Zeitschrift Fur Padagogische Psychologie, 2002

... Oliver Lüdtke. Max-Planck-Institut für Bildungsforschung, Berlin. Alexander Robitzsch. Techni... more ... Oliver Lüdtke. Max-Planck-Institut für Bildungsforschung, Berlin. Alexander Robitzsch. Technische Universität, Dresden. Olaf Köller. Max-Planck-Institut für ... Für den Einfluss der Intelligenz auf Schulleistungen zeigten Mortimore, Simmons, Stoll, Lewis und Ecob (1989), dass nach ...