Constructing Parallel Simulation Exercises for Assessment Centers and Other Forms of Behavioral Assessment (original) (raw)
Related papers
A cognitive task analysis with implications for designing simulation-based performance assessment
Computers in Human Behavior, 1999
To function effectively as a learning environment, a simulation system must present learners with situations in which they use relevant knowledge, skills, and abilities. To function effectively as an assessment, such a system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and fitting to the purpose at hand (e.g., licensure, achievement testing, coached practice). This article concerns an evidence-centered approach to designing a computer-based performance assessment of problem-solving. The application is a prototype licensure test, with supplementary feedback, for prospective use in the field of dental hygiene. We describe a cognitive task analysis designed to (a) tap the knowledge hygienists use when they assess patients, plan treatments, and monitor progress, and (b) elicit behaviors that manifest this knowledge. After summarizing the results of the analysis, we discuss implications for designing student models, evidentiary structures, task frameworks, and simulation capabilities required for the proposed assessment.
How Do Test Takers Interact With Simulation-Based Tasks? A Response-Time Perspective
Frontiers in Psychology
Many traditional educational assessments use multiple-choice items and constructed-response items to measure fundamental skills. Virtual performance assessments, such as game-or simulation-based assessments, are designed recently in the field of educational measurement to measure more integrated skills through the test takers' interactive behaviors within an assessment in a virtual environment. This paper presents a systematic timing study based on data collected from a simulation-based task designed recently at Educational Testing Service. The study is intended to understand the response times in complex simulation-based tasks so as to shed light on possible ways of leveraging response time information in designing, assembling, and scoring of simulation-based tasks. To achieve this objective, a series of five analyses were conducted to first understand the statistical properties of the timing data, and then investigate the relationship between the timing patterns and the test takers' performance on the items/task, demographics, motivation level, personality, and test-taking behaviors through use of different statistical approaches. We found that the five analyses complemented each other and revealed different useful timing aspects of this test-taker sample's behavioral features in the simulation-based task. The findings were also compared with notable existing results in the literature related to timing data.
Computers & Education, 2015
Researchers have shown in multiple studies that simulations and games can be effective and powerful tools for learning and instruction (cf. Mitchell some area. However, the last decade researchers have started focusing on what is happening inside the black box to an increasing extent and the literature on the topic is growing. To our knowledge, no systematic reviews have been published that investigate the psychometric analysis of performance data of simulation-based assessment (SBA) and game-based assessment (GBA). Therefore, in Part I of this article, a systematic review on the psychometric analysis of the performance data of SBA is presented. The main question addressed in this review is: 'What psychometric strategies or models for treating and analyzing performance data from simulations and games are documented in scientific literature?'. Then, in Part II of this article, the findings of our review are further illustrated by presenting an empirical example of the e according to our review e most applied psychometric model for the analysis of the performance data of SBA, which is the Bayesian network. Both the results from Part I and Part II assist future research into the use of simulations and games as assessment instruments.
The effect of repeat exposure to simulation based items
Practical Assessment, Research and Evaluation, 2020
This study aims to examine the potential impacts on repeat examinees' performance by reusing simulation-based items in a high-stakes standardized assessment. We examined change patterns of item scores, ability estimate, score pattern change, response time and compared the performance of repeat examinees who have received repeat items and those who haven't. Results suggest that there are limited benefits from encountering the same items. The practical implications to licensing/certification assessments are discussed.
Rater Training to Support High-Stakes Simulation-Based Assessments
Journal of Continuing Education in the Health Professions, 2012
Competency-based assessment and an emphasis on obtaining higher-level outcomes that reflect physicians' ability to demonstrate their skills has created a need for more advanced assessment practices. Simulation-based assessments provide medical education planners with tools to better evaluate the 6 Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties (ABMS) core competencies by affording physicians opportunities to demonstrate their skills within a standardized and replicable testing environment, thus filling a gap in the current state of assessment for regulating the practice of medicine. Observational performance assessments derived from simulated clinical tasks and scenarios enable stronger inferences about the skill level a physician may possess, but also introduce the potential of rater errors into the assessment process. This article reviews the use of simulation-based assessments for certification, credentialing, initial licensure, and relicensing decisions and describes rater training strategies that may be used to reduce rater errors, increase rating accuracy, and enhance the validity of simulation-based observational performance assessments.
2014
AUTOMATED SCORING IN ASSESSMENT CENTERS: EVALUATING THE FEASIBILITY OF QUANTIFYING CONSTRUCTED RESPONSES Automated scoring has promised benefits for personnel assessment, such as faster and cheaper simulations, but there is yet little research evidence regarding these claims. This study explored the feasibility of automated scoring for complex assessments (e.g., assessment centers). Phase 1 examined the practicality of converting complex behavioral exercises into an automated scoring format. Using qualitative content analysis, participant behaviors were coded into sets of distinct categories. Results indicated that variations in behavior could be described by a reasonable number of categories, implying that automated scoring is feasible without drastically limiting the options available to participants. Phase 2 compared original scores (generated by human assessors) with automated scores (generated by an algorithm based on the Phase 1 data). Automated scores had significant convergence with and could significantly predict original scores, although the effect size was modest at best and varied significantly across competencies. Further analyses revealed that strict inclusion criteria are important for filtering out contamination in automated scores. Despite these findings, we cannot confidently recommend implementing automated scoring methods without further research specifically looking at the competencies in which automated scoring is most effective.
A methodology for simulation-based job performance assessment
2008
Job performance measurement is of critical importance to any organization"s health. It is important not only to recognize and reward good performance, but also to groom future leaders. Developing effective assessment techniques that are valid, effective and fair is an ongoing challenge. Assessing factual knowledge using multiplechoice test batteries relatively inexpensive and tends to be commonly used. Hands-on assessment is the most effective in assessing task proficiency but is very resource intensive and expensive. Computer-based simulations provide an alternative where users can be assessed in the context of skill application under controlled conditions. However, simulations are expensive to produce and maintain. Validated guidelines and methodologies are needed to help organizations develop effective assessment simulations. In this paper we present a standard, prescriptive methodology for developing simulations for job performance assessment. We then describe a performance assessment simulation for Light-Wheeled Vehicle Maintenance constructed according to this methodology. This simulation includes automated assessment methods that borrow heavily from existing work in intelligent tutoring systems. Finally, we discuss future research directions based on the results of this initial methodology and assessment.
Reliability and validity of scenario-specific versus generic simulation assessment rubrics
Background: This study assessed reliability and validity of scenario-specific and generic simulation assessment rubrics used in two different deteriorating patient simulations, and explored learner and instructor preferences. Methods: Learner performance was rated independently by three instructors using two rubrics. Results: A convenience sample of 29 nursing students was recruited. Inter-rater reliability was similar but slightly higher for the generic rubric than the scenario-specific learning outcomes assessment rubric (ICC = .759 vs .748 and IRR = .693 vs .641) for two different scenarios. Most students found the scenario-specific rubric more helpful to their learning (59%), and easier to use (52%). Instructors (3/3) found the scenario-specific rubric more helpful to guide debriefing. Conclusions: Scenario-specific rubrics may be more valuable for learners to help them identify their own knowledge and performance gaps and assist them in their preparation for simulation. Additionally, scenario-specific rubrics provide direction for both learners and instructors during debriefing sessions.