Test Score Reporting: Perspectives From the ETS Score Reporting Conference (original) (raw)

IMPROVING TEST SCORE REPORTING: PERSPECTIVES FROM THE ETS SCORE REPORTING CONFERENCE

ETS Research Report Series, 2011

November 4th, 2010, to explore some issues that influence score reports and new advances that contribute to the effectiveness of these reports. Jessica Hullman, Rebecca Rhodes, Fernando Rodriguez, and Priti Shah present the results of recent research on graph comprehension and data interpretation, especially the role of presentation format, the impact of prior quantitative literacy and domain knowledge, the trade-off between reducing cognitive load and increasing active processing of data, and the affective influence of graphical displays. Rebecca Zwick and Jeffrey Sklar present the results of the Instructional Tools in Educational Measurement and Statistics for School Personnel (ITEMS) project, funded by the National Science Foundation and conducted at the University of California, Santa Barbara to develop and evaluate 3 web-based instructional modules intended to help educators interpret test scores. Zwick and Sklar discuss the modules and the procedures used to evaluate their effectiveness. Diego Zapata-Rivera presents a new framework for designing and evaluating score reports, based on work on designing and evaluating score reports for particular audiences in the context of the CBAL (Cognitively Based Assessment of, for, and as Learning) project (Bennett & Gitomer, 2009), which has been applied in the development and evaluation of reports for various audiences including teachers, administrators and students.

Designing and evaluating an interactive score report for students

In an effort to make students active participants in their learning and to consider students' score reporting needs, we have designed an interactive student score report for middle school students that implements a guided-instructional activity aimed at facilitating student understanding of score report information and improving student engagement. This paper showcases an interactive student score report developed for the Cognitively Based Assessment of, for, and as Learning (CBAL™) initiative and reports on results from an initial usability study, as well as revisions made to the original version of the score report based upon results from the initial usability study and expert feedback. In addition, results are presented from a follow-up usability study comparing the graphical representations in Version 1 of the score report to Version 2 of the report.

Applying Score Design Principles in the Design of Score Reports for CBAL™ Teachers

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Memorandum series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes.

A Review of Recent Research on Individual-Level Score Reports

Educational Measurement: Issues and Practice, 2018

As the primary interface between test developers and multiple educational stakeholders, score reports are a critical component to the success (or failure) of any assessment program. The purpose of this review is to document recent research on individual-level score reporting to advance the research and practice of score reporting. We conducted a search for research studies published or presented between 2005 and 2015, examining 60 scholarly works for (1) the research focus, (2) stated or implied theoretical frameworks of communication, and (3) the characteristics of data sets employed in the studies. Results show that research on score properties, especially subscores, and score report design/layout are well-represented in the literature base. The predominant approach to score reporting has been through a cybernetics tradition of communication. Data sets were often small or localized to a single context. We present example research questions from novel communication frameworks, and encourage our colleagues to adopt new roles in their relationships to stakeholders to advance score reporting research and practice.

What Teachers Need to Know about Assessment

new material can be presented. These tests help the teacher gain a perspective of the range of attained learning as well as individual competence. Tests can be used to help make promotion and retention decisions. Many factors enter into the important decision of moving a student into the next grade. Intuition is an important part of any decision but that intuition is enhanced when coupled with data. Standardized tests, and records of classroom performance on less formal tests are essential for supplying much of the data upon which these decisions are based. Test results are important devices to share information with boards of education, parents, and the general public through the media. Rudner, L. and W. Schafer (2002) What Teachers Need to Know About Assessment. Washington, DC: National Education Association. From the free on-line version. To order print copies call 800 229-4200 3 some criterion. This section ends includes a discussion of norm-referenced and criterion referenced tests. This section also includes standardized and large scale assessments-typically the types of tests sponsored by state education agencies, reported in the popular press, and unfortunately, often inappropriately used as the sole measure to judge the worth of a school. We start with a discussion of the different types of scores used to report standardized test results. You will learn the advantages, disadvantages of each along with how the different types of scores should be used. A key feature of state assessments is that they are almost always accompanied by a careful delineation of endorsed educational goals. There should be no ambiguity with regard to what is covered by such tests. The next chapter discusses aligning one's instruction to the test and making the test into a valuable instructional planning tool. There is often a debate with regard to teaching to a test. Some argue that since the test identifies goals, teaching to the test is equivalent to teaching goals and should be done. Others argue that teaching to a test is an attempt to short circuit the educational process. The next chapter identifies a continuum of acceptable and unacceptable practices for preparing students to take standardized achievement tests. Lastly, with testing so prominent in the popular press, we provide an overview of some of the politics of national testing. Section 2: Essential Concepts for Classroom Assessment. The most frequent and most valuable types of tests are those developed and used by classroom teachers. This section is designed to help you develop you write better multiple choice and better performance tests. You will learn to examine what it is that you want to assess, how to write questions that assess those concepts. Special attention is paid to the development of analytic and holistic scoring rubrics. Consistent with the view of testing as a form of data gathering and communication, chapters have been included on asking classroom questions as part of routine instruction and on writing comments on report cards. Rudner, L. and W. Schafer (2002) What Teachers Need to Know About Assessment. Washington, DC: National Education Association. From the free on-line version. To order print copies call 800 229-4200 4 the reasonable expectations that those involved in the testing enterprise-test producers, test users, and test takers-should have of each other. The document is applicable to classroom tests as well as standardized tests.

Keeping your audience in mind: applying audience analysis to the design of interactive score reports

Assessment in Education: Principles, Policy & Practice, 2014

Score reports have one or more intended audiences: the people who use the reports to make decisions about test takers, including teachers, administrators, parents and test takers. Attention to audience when designing a score report supports assessment validity by increasing the likelihood that score users will interpret and use assessment results appropriately. Although most design guidelines focus on making score reports understandable to people who are not testing professionals, audiences should be defined by more than just their lack of statistical knowledge. This paper introduces an approach to identifying important audience characteristics for designing computer-based, interactive score reports. Through three examples, we demonstrate how an audience analysis suggests a design pattern, which guides the overall design of a report, as well as design details, such as data representations and scaffolding. We conclude with a research agenda for furthering the use of audience analysis in the design of interactive score reports.

Every Teacher's Guide to Assessment

It's not a stretch to say that assessment is a hot button issue in education; however, you'd be hard pressed to find an educator who doesn't see the value in measuring student progress.

What Do the Test Scores Really Mean? Critical Issues in Test Design

1986

Issues in designing valid tests for the National Assessment of Educational Progress (NAEP) are discussed. Test scores are often provided without any information on the nature of the tasks represented. Because test domains are defined by individual item writers, the generalizabP.ity between tests and items is suspect. While typical content validation procedures help assure that the included items are important, they still might not represent the full range of knowledge and skills constituting given domains. As a result, the underlying meaning of what is tested is vague, and the specific definition of what is to be tested escapes public scrutiny. This is especially important when matching particular tests and curricula among states. Better specification of test content and task structure is recommended. Elements in good task structure should include: task description; content limits; linguistic features; cognitive complexity; and format. Recent NAEP assessments defined four different types of context for test items: (1) scientific, (2) personal, (3) societal, and (4) technological. Three levels of cognitive complexity items-were defined: (1) knows, (2) uses, and (3) integrates. Six categories of subject content were specified. In conclusion, NAEP planners should emphasize content validity; define more specifically what is to be tested; provide better models for item construction; and assure that the entire domain is represented.