Matthew Gaertner | WestEd - Academia.edu (original) (raw)

Papers by Matthew Gaertner

To support students’ progress toward scientific literacy, the Next Generation Science Standards (... more To support students’ progress toward scientific literacy, the Next Generation Science Standards (NGSS) demand that science and engineering practices, disciplinary core ideas, and crosscutting concepts be interwoven into every aspect of science education – curriculum, instruction, and assessment. Because assessments can signal priorities and guide instruction, practices and concepts rarely gain traction in schools if they are not emphasized in assessment. Unfortunately, there are few assessments available that provide the level depth and integration required by the NGSS and that can be scored economically. To date, NGSS standards have been assessed largely through technology-enhanced items. However, these items do not assess the standards requiring explanation and argument. To tap standards related to explanation and argument, items need to elicit written responses, which present challenges to test developers due to hand-scoring requirements and the extensive rater training involved. To address this gap in assessment research and practice, this study focused on the development and validation of constructed-response items that can be machine-scored. Through this study, we ultimately intend to develop items that can better assess a student’s ability to construct explanations and arguments and that can be reliably, automatically scored. This will yield rich, authentic assessment tasks that generate the evidence of the integrated science knowledge the NGSS demand, but that can be scored and reported efficiently to support progress monitoring and accountability.

Monitoring trends in student achievement depends heavily upon establishing stable linkages betwee... more Monitoring trends in student achievement depends heavily upon establishing stable linkages between assessments over time. In state-level large-scale assessment programs this is typically accomplished through the application of item response theory (IRT) modeling. The key feature that makes IRT modeling attractive is the property of parameter invariance, which holds that item parameters are invariant to the particular sample of examinees that is the basis for their estimation, and vice-versa. In some cases, however, the property of parameter invariance will not hold. The phenomenon in which the parameter values for the same test items change systematically over multiple testing occasions is known as is known as item parameter drift (IPD). There are a variety of sources that could cause the parameters for item difficulty to drift upwards or downwards; some are desirable (e.g., assessment-driven curricular changes or more focused instruction), and others (e.g., item exposure or cheating, curricular misalignment with state standards) are not. Whatever the cause of IPD, several studies have shown that its presence can lead to biased estimates of ability and ultimately improper classification of examinees, and this could have important ramifications when test scores are used for high-stakes purposes. As such, it is important to evaluate the presence of IPD when linking assessments over time, especially when the same common items are being used repeatedly. This literature review will focus on IPD in test equating scenarios. We address three central questions:

What methods can and should be used to detect IPD?
How large does IPD need to be before it is considered practically significant?
If practically significant levels of IPD are detected, what should be done?

National Education Policy Center, 2017

This report provides an extensive analysis based on the most comprehensive dataset ever assembled... more This report provides an extensive analysis based on the most comprehensive dataset ever assembled for school closure research, including 1,522 low-performing schools that were closed across 26 states between 2006 and 2013. The report finds that even when holding constant academic performance, schools were more likely to be closed if they enrolled higher proportions of minority and low-income students. It also finds test score declines, relative to the comparison group, for two groups of students displaced by closures: those who transferred to schools with a prior record of relatively lower test-score performance and those who transferred to schools with equivalent past test-score performance. The slightly less than half of students who transferred to higher performing schools showed academic improvement relative to their matched peers. In general, although the reviewers found this to be a careful and rigorous study, they see a few missed opportunities. First, the report’s focus on some tenuous analyses (involving pre-closure transfers) obscures its most important findings – disproportionality in school closures and inadequate numbers of higher quality receiving schools, leading to performance declines for most. Second, the reviewers are concerned about statistical modeling choices and matching challenges that may threaten the validity of subgroup analyses (charter school students). Finally, the reviewers would have liked to see the report acknowledge the inescapable moral dimensions of school closure: The communities most likely to be negatively affected are unlikely to have participated in closure decisions.

Though higher education costs continue to rise, the returns of a college education are clear. Col... more Though higher education costs continue to rise, the returns of a college education are clear. College graduates experience lower levels of unemployment and higher annual wages, resulting in over $1 million more in lifetime earnings than high school graduates. Moreover, regions with higher proportions of college graduates benefit from lower crime rates, better health, and greater civic participation. However, just 40% of adults hold a college degree. Each year 60% of new community college students are referred to non-credit, developmental education courses. Unfortunately this path is often a dead end; only 28% of these students who take developmental courses complete their degree within eight years. Boosting postsecondary readiness has become a centerpiece of education reforms. Though college and career readiness is common discourse among educators, researchers, policy makers, and the general public, that discourse tends to lack consistency and clarity about what college and career readiness actually means. With this book, we brought together a cross-disciplinary group of experts to synthesize the current state of college- and career-readiness research, best practices in measurement and diagnostics, and leading intervention practices designed to prepare students for life after high school. The book is divided into three sections. The first describes knowledge, skills, and attributes associated with college and career readiness as well as how those constructs are measured. The second focuses on defining, validating, and using college- and career-ready performance levels, and the final section is devoted to both general and specific strategies for improving college and career readiness.

Preparing all students for postsecondary endeavors has become a top priority in national and stat... more Preparing all students for postsecondary endeavors has become a top priority in national and state educational policy. To serve this goal, states regularly evaluate both the rigor of their academic standards and the extent to which their assessments and associated performance levels align with those standards. The goal is laudable: all states should adopt rigorous standards that put students on a path toward college and career readiness. The execution, however, is unavoidably inconsistent. States vary widely in their definitions of college and career readiness – witness, for example, the sizeable differences between individual states’ test results and national indicators of college and career readiness. Ultimately, mixed messages hamper U.S. efforts to evaluate and improve college and career readiness. Despite ostensibly open-minded calls for more local control in assessment, variation in those assessments causes confusion, promotes inefficiencies, and masks large inequities across states. Moreover, inconsistent readiness standards send remarkably inconsistent messages to students, parents, and the general public. Depending on which feedback a student receives or attends to, he or she may cruise through high school with confidence, unknowingly ill-prepared for what lies ahead. This chapter will focus on the difficulties inherent in broadly assessing college and career readiness when generalizations are based on different assessments with different constructs, uses, performance levels, populations (e.g., self-selected SAT-takers versus statewide populations), and consequences (e.g., high- versus low-stakes). In addition, we will suggest simple principles states can adopt to better align their assessments and readiness diagnoses, both with each other and with national frameworks.

Although educational catchphrases like “reading, writing, and arithmetic” seem dated and quaint, ... more Although educational catchphrases like “reading, writing, and arithmetic” seem dated and quaint, modern assessment systems still place inordinate emphasis on these narrow domains. Focusing on academic content knowledge to the exclusion of noncognitive skills leaves a lot of unexplained variance on the table. Broadening our definition of readiness to include noncognitive dimensions could yield more accurate and actionable intelligence for students and educators. This chapter focuses on the measurement of noncognitive skills, and its arguments center on three theses. First, when combined with achievement measures, noncognitive attributes like motivation substantially improve predictions of future outcomes. Second, noncognitive measures provide more nuanced diagnoses than cognitive tests alone. Third, noncognitive measures are highly actionable; students can do something about them, which might not be true of conventional aptitude measures. Armed with more accurate predictions based on more diverse data, educators can more reliably determine which students are on track to college and career readiness, and which will need significant support to achieve their future goals. We cite specific examples of noncognitive and psychosocial skills, with particular focus on the Five Factor Model of personality. Each of these constructs has been operationalized and measured in the literature and subsequently shown to offer powerful diagnostic information about students’ chances of future success. In addition, we outline implications for policy, practice, and research, highlighting the rationales and risks associated with putting noncognitive measures to use in accountability systems.

When Katie, Matt, and I first sat down to brainstorm ideas for a book prospectus on college and c... more When Katie, Matt, and I first sat down to brainstorm ideas for a book prospectus on college and career readiness, we articulated our hopes for what another book on the topic could contribute to the literature. College and career readiness had been generating substantial interest and enthusiasm among educators, researchers, policymakers, and the general public, but there were already quite a few books on the topic. How could this book be different, and what could we contribute to the dialogue? We set out to produce an edited book on college and career readiness that was grounded in scientific research. That is, we felt the college- and career-readiness discourse could benefit from more systematic empirical evidence. Debates too often hinge on anecdotes or case studies that have not been replicated or generalized, so we wanted a volume with a balance of theory, research, and practice, supported by rigorous scientific research findings. It quickly became apparent that there were three questions the book would need to address: (1) How should we define and measure college and career readiness? (2) What are some best practices for validating college- and career-readiness performance levels and uses? (3) What interventions show the most promise for improving college and career readiness among students? These questions dictated the structure of the book and the information conveyed within. As authors began to submit their chapters, themes began to emerge within each section as well as some themes that spanned the entire book. We will use this closing chapter to summarize some of those themes. We believe that in doing so, we can highlight current issues and trends in college- and career-readiness research as well as elucidate future directions for research and practice.

This chapter explores the utility of pass rate analysis – the study of year-to-year changes in th... more This chapter explores the utility of pass rate analysis – the study of year-to-year changes in the percentage of students school-wide who reach established proficiency targets. Pass rate analyses are particularly sensitive to small unexpected changes in students’ scale scores that (1) are undetectable via examination of scale score patterns alone, and (2) nonetheless result in substantial changes in school-level proficiency rates. In this chapter, the authors compare the efficiency of two statistical approaches – the two-proportion z-score and multilevel logistic regression – for detecting abnormal changes in pass rates over time. Through a simulation and an analysis of statewide education data, the authors evaluate which technique best identifies schools where cheating has occurred and propose a framework educational agencies can use to set thresholds triggering further investigation of potential test fraud.

Harvard Educational Review, 2013

In this article, Amy N. Farley, Matthew N. Gaertner, and Michele S. Moses examine the use of ball... more In this article, Amy N. Farley, Matthew N. Gaertner, and Michele S. Moses examine the use of ballot initiatives as a particularly attractive form of direct democracy for opponents of affirmative action in higher education. Building on previous scholarship, the authors question whether anti-affirmative ballot initiatives validly reflect voters' attitudes toward affirmative action. The authors examine the case of Colorado's Amendment 46, an anti-affirmative action ballot initiative. They investigate the language of the initiative itself, as well as voters' perceptions of and confusion around its intent, and the factors that influenced voting behavior. They employ item response theory to estimate voters' attitudes toward affirmative action. The authors then describe the prevalence of voter confusion around the initiative's intent. Finally, employing a binary logistic regression model, they analyze survey data to determine which factors influenced voting behavior. They find that the initiative's language was successful at confusing voters who intended to support affirmative action. Following the U.S. Supreme Court's recent decision in Fisher v. University of Texas and in anticipation of its decision in Schuette v. Coalition to Defend Affirmative Action, the authors call for greater scrutiny with regard to the use of initiatives to craft education policies that have a disproportionately negative impact on members of disadvantaged populations. Since 1996, six states-California, Washington, Michigan, Nebraska, Arizona, and Oklahoma-have considered ballot initiatives seeking to ban affirmative Democracy under Fire farley, gaertner, and moses Democracy under Fire farley, gaertner, and moses Democracy under Fire farley, gaertner, and moses Democracy under Fire farley, gaertner, and moses

This article provides further detail on a middle school college readiness index, which was devise... more This article provides further detail on a middle school college readiness index, which was devised to generate earlier and more nuanced readiness diagnoses to K-12 students. Issues of reliability and validity (including construct underrepresentation and construct-irrelevant variance) are discussed in detail. In addition, comments from Lazowski et al. and Mattern et al. provide a useful summary of areas where more college readiness research is needed. This rejoinder elaborates on those essential next steps, including deploying readiness indicators in school settings. Advances in the measurement of noncognitive skills will require more validity research using more contemporary data from authentic school settings. Still, the middle school college readiness index is a promising proof of concept, demonstrating the utility of diverse, early measures to support targeted, timely interventions.

In this paper, we attempt to clarify the readiness landscape. We introduce three readiness paradigms—the college readiness index for middle school students, the Conley Readiness Index, and GRIT—and review their goals, theoretical
foundations, and empirical support. This paper dedicates particular focus to strands of convergence and divergence between these three approaches. A few core tenets underpin each of the three readiness paradigms: (1) scholars, educational practitioners, and employers must develop an expanded definition of readiness and success, (2) useful readiness paradigms should empower the learner, and (3) rigorous measurement still matters.

The paper concludes with a short set of recommendations focused on how new
approaches to college and career readiness can be used to support smarter and earlier interventions and open college and career pathways to all learners.

Popular conceptions of college and career readiness are broadening beyond strictly academic compe... more Popular conceptions of college and career readiness are broadening beyond strictly academic competencies like literacy and numeracy. New thinking on the many dimensions of preparedness has produced volumes of research and scores of new products. In fact, educators and employers may find it difficult to separate signal from noise and focus on the readiness paradigms that suit their needs. In this paper, we attempt to clarify the readiness landscape. We introduce three readiness paradigms—the college readiness index for middle school students, the Conley Readiness Index, and GRIT—and review their goals, theoretical foundations, and empirical support. This paper dedicates particular focus to strands of convergence and divergence between these three approaches. A few core tenets underpin each of the three readiness paradigms: (1) scholars, educational practitioners, and employers must develop an expanded definition of readiness and success, (2) useful readiness paradigms should empower the learner, and (3) rigorous measurement still matters. The paper concludes with a short set of recommendations focused on how new approaches to college and career readiness can be used to support smarter and earlier interventions and open college and career pathways to all learners.

Advances in Multilevel Modeling for Educational Research: Addressing Practical Issues Found in Re... more Advances in Multilevel Modeling for Educational Research: Addressing Practical Issues Found in Real‐World Applications is a resource intended for advanced graduate students, faculty and/or researchers interested in multilevel data analysis, especially in education, social and behavioral sciences. The chapters are written by prominent methodological researchers across diverse research domains such as educational statistics, quantitative psychology, and psychometrics. Each chapter exposes the reader to some of the latest methodological innovations, refinements and state‐of‐the‐art developments and perspectives in the analysis of multilevel data including current best practices of standard techniques.

Access and diversity initiatives are university-wide enterprises, but in practice these initiativ... more Access and diversity initiatives are university-wide enterprises, but in practice these initiatives usually fall to collegiate admissions and enrollment management leaders. In the past two decades their jobs have been complicated by a string of ballot initiatives and lawsuits that threaten to outlaw the consideration of race in the admissions process. How are admissions and enrollment management leaders responding? What is working? Where are admissions professionals seeking guidance? These are the questions that motivated our new report Race, Class, and College Access: Achieving Higher Education Diversity in a Shifting Legal Landscape.

In collaboration with Dr. Gary Orfield of the Civil Rights Project at UCLA and other national higher education organizations, we surveyed 338 admissions and enrollment management leaders at nonprofit four-year colleges across the country. We asked about commonly used diversity strategies, the efficacy of those strategies, and changes in practice following recent Supreme Court cases and statewide affirmative-action bans. What we found surprised us.

The American Council on Education’s Center for Policy Research and Strategy, in collaboration wit... more The American Council on Education’s Center for Policy Research and Strategy, in collaboration with Pearson’s Center for College & Career Success and The Civil Rights Project at the University of California, Los Angeles, recently embarked on a groundbreaking study to examine how legal challenges to race-conscious admissions are influencing contemporary admissions practices at selective colleges and universities.

In educational research and policy circles, college and career readiness is generating great inte... more In educational research and policy circles, college and career readiness is generating great interest. States are adopting various policy initiatives, such as rigorous curricular requirements, to increase students’ preparedness for life after high school. Implicit in many of these initiatives is the idea that college readiness and career readiness are essentially the same thing. This assumption has persisted, largely untested. Our paper explores this assumption in greater depth. Using two national datasets and an instrumental variables approach to mitigate selection bias, we evaluated the effects of completing Algebra II in high school on subsequent college and career outcomes (i.e., persistence and graduation as well as wages and career advancement). Results suggest Algebra II matters more for college outcomes than career outcomes and more for students completing Algebra II in the early 1990s than in the mid-2000s. Study limitations are discussed along with directions for future research, such as evaluating the opportunity cost associated with taking Algebra II for students seeking careers upon high school completion.

Educational Measurement: Issues and Practice, 2015

Although college readiness is a centerpiece of major educational initiatives such as the Common C... more Although college readiness is a centerpiece of major educational initiatives such as the Common Core State Standards, few systems have been implemented to track children's progress toward this goal. Instead, college-readiness information is typically conveyed late in a student's high-school career, and tends to focus solely on academic accomplishments-grades and admissions test scores. Late-stage feedback can be problematic for students who need to correct course, so the purpose of this research is to develop a system for communicating more comprehensive college-readiness diagnoses earlier in a child's K-12 career. This article introduces college-readiness indicators for middle-school students, drawing on the National Education Longitudinal Study of 1988 (NELS), a nationally representative longitudinal survey of educational inputs, contexts, and outcomes. A diversity of middle-school variables was synthesized into six factors: achievement, behavior, motivation, social engagement, family circumstances, and school characteristics. Middle-school factors explain 69% of the variance in college readiness, and results suggest a variety of factors beyond academic achievement-most notably motivation and behavior-contribute substantially to preparedness for postsecondary study. The article concludes with limitations and future directions, including the development of college-readiness categories to support straightforward communication of middle-school indicators to parents, teachers, and students.

On July 28, 2015, Hobsons hosted a small invitation-only convening of admissions and enrollment m... more On July 28, 2015, Hobsons hosted a small invitation-only convening of admissions and enrollment management leaders in partnership with the American Council on Education (ACE), to discuss college access and success for underrepresented minority and low-income students. A series of panel discussions, composed of leading higher education experts, framed the issues. What emerged was a rich and robust discussion on the challenges, successes, and promising practices that will allow education professionals to meet mission-critical diversity goals and advance equity in postsecondary education.

American Institutes for Research conducted in-depth case studies of eight Magnet Schools Assistan... more American Institutes for Research conducted in-depth case studies of eight Magnet Schools Assistance Program (MSAP) projects in school districts identified as Districts A to H. The purpose of these case studies is to help illuminate and illustrate results obtained from the data collected on all 57 MSAP projects.

In 2008 the University of Colorado Boulder, fearing that a state anti-affirmative action referend... more In 2008 the University of Colorado Boulder, fearing that a state anti-affirmative action referendum banning considerations of race would pass, turned to Matthew Gaertner to help devise a race-neutral alternative that provided a leg up to socioeconomically disadvantaged students of all races. Based on national research, Colorado devised indices of disadvantage and overachievement that incorporated a variety of socioeconomic indicators. Under the program, socioeconomically disadvantaged students receive a sizeable preference in admissions. When simulations were run, socioeconomic diversity increased, as expected, but surprisingly, the acceptance rates of underrepresented minority applicants also increased. But did the sizable socioeconomic boost create a new academic mismatch problem by admitting too many unprepared students? Although class-based admits as an undifferentiated group were less likely to graduate in six years (53 percent versus 66 percent for the general population), those who were identified by the overachievement index actually outperformed typical undergraduates. The university has not seen the lower graduation rates of disadvantaged students as inevitable or as reason to discontinue the program but rather has moved to beef up academic supports for such students.

What methods can and should be used to detect IPD?
How large does IPD need to be before it is considered practically significant?
If practically significant levels of IPD are detected, what should be done?

National Education Policy Center, 2017

Harvard Educational Review, 2013

Educational Measurement: Issues and Practice, 2015

In November 2008, Colorado voters considered a ballot initiative intended to end affirmative acti... more In November 2008, Colorado voters considered a ballot initiative intended to end affirmative action in public education, employment, and contracting in the state. Known by proponents as the “Colorado Civil Rights Initiative,” Amendment 46 would have prohibited “discrimination or preferential treatment in public employment, public education, and public contracting,” effectively ending affirmative action in Colorado. In a reversal of past trends and contradictory to polling predictions, Coloradans voted to defeat the ballot measure by a margin of fewer than 40,000 votes. This study aimed to gain detailed understanding of what led Coloradans to defeat Amendment 46, using a multiple methods research design including statistical analyses of a large-scale survey of Colorado voters, in-depth qualitative analyses of interview data from 20 key opponents and proponents of the ballot initiative, and qualitative analyses of print, electronic, and broadcast media content examining what public information was available to voters, how media depicted Amendment 46, and which experts were consulted most often. We found three primary factors that influenced the outcome of the vote: (1) Voter attitudes about affirmative action; (2) Voter confusion about the intent, meaning, and consequences of Amendment 46; and (3) Specific oppositional efforts such as proposed alternative initiatives—Initiatives 61 and 82—and newspaper editorial stances.

Preparing Students for College and Careers addresses measurement and research issues related to c... more Preparing Students for College and Careers addresses measurement and research issues related to college and career readiness. Educational reform efforts across the United States have increasingly taken aim at measuring and improving postsecondary readiness. These initiatives include developing new content standards, redesigning assessments and performance levels, legislating new developmental education policy for colleges and universities, and highlighting gaps between graduates’ skills and employers’ needs. In this comprehensive book, scholarship from leading experts on each of these topics is collected for assessment professionals and for education researchers interested in this new area of focus. Cross-disciplinary chapters cover the current state of research, best practices, leading interventions, and a variety of measurement concepts, including construct definitions, assessments, performance levels, score interpretations, and test uses.

The closure of urban public schools in poor and working class communities of color has, since 200... more The closure of urban public schools in poor and working class communities of color has, since 2008, become normalized as a school reform strategy. Although sometimes motivated by necessity, such as budget deficits or diminishing enrollment, closures have also been touted as a strategy for improving educational options for youth attending low-performing schools. This chapter reviews research about the experiences of students displaced by closure and finds mixed evidence. There is some evidence that, under certain conditions (e.g., transferring from a low-performing to a high-performing school), students experience increased academic learning, particularly in math, relative to their test performance prior to closure. Often, however, because higher performing schools are not available or accessible to displaced students, displaced students either do not show improvement or show declines. Moreover, there are a number of other variables that complicate findings about student outcomes, including the age of students, the political context of the closure, the specific academic subject measured, and the experiences of students with transportation and after-school opportunities. The chapter argues that in addition to weighing empirical findings, policy decisions about closure raise normative questions about people’s right to participate in decisions that affect their lives. The conclusion suggests ways that more robust opportunities for public deliberation could be supported through policy.

In early 2006, the Riverside School Board voted to close Jefferson High School – an institution t... more In early 2006, the Riverside School Board voted to close Jefferson High School – an institution that had served the urban district’s Black and Latino populations for more than a century. In light of Jefferson students’ academic struggles, the school district touted the closure as a “rescue mission” – displaced students would have the opportunity to relocate to higher performing schools and would therefore stand a better chance of progressing academically and graduating on time. Our research, based on quantitative analyses of academic outcomes and qualitative analyses of student experiences, suggests the closure had the opposite effect for most students. Displaced students’ test scores – rising prior to the closure – declined in the years that followed it. Graduation and dropout trends followed suit. Displaced students were more than twice as likely to exit early and 31% less likely to graduate on time, compared to similar students who had attended Jefferson prior to the closure.