Cross-institutional evaluation of a mastoidectomy assessment instrument (original) (raw)
Related papers
Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument
Journal of Surgical Education, 2017
A method for rating surgical performance of a mastoidectomy procedure that is shown to apply universally across teaching institutions has not yet been devised. This work describes the development of a rating instrument created from a multi-institutional consortium. DESIGN Using a participatory design and a modified Delphi approach, a multi-institutional group of expert otologists constructed a 15 element task-based checklist for evaluating mastoidectomy performance. This instrument was further refined into a 14 element checklist focusing on the concept of safety after using it to rate a large and varied population of performances. SETTING Twelve Otolaryngological surgical training programs in the United States. PARTICIPANTS 14 surgeons from 12 different institutions took part in the construction of the instrument. RESULTS By using 14 experts from 12 different institutions and a literature review, individual metrics were identified, rated as to the level of importance and operationally defined to create a rating scale for mastoidectomy performance. Initial use of the rating scale showed modest rater agreement. The operational definitions of individual metrics were modified to emphasize "safe" as opposed to "proper" technique. A second rating instrument was developed based on this feedback. CONCLUSIONS Using a consensus building approach with multiple rounds of communication between experts is a feasible way to construct a rating instrument for mastoidectomy. Expert opinion alone using a Delphi method provides face and content validity evidence, however, this is not sufficient to develop a universally acceptable rating instrument. A continued process of development and
Objective Assessment of Mastoidectomy Skills in the Operating Room
Otology & Neurotology, 2010
To determine the feasibility and validity of an objective assessment tool designed to measure the development of mastoidectomy skills by resident trainees in the operating room. Study Design: Prospective longitudinal validation study. Setting: Tertiary referral center and residency training program. Subjects: Otolaryngology residents. Main Outcome Measure: Technical performance as measured over time using Task-Based Checklist (TBC) and Global Rating Scale (GRS) developed for assessment of mastoidectomy skills. Results: Seventy pairs of evaluations were completed on 15 residents, showing strong correlation between both instruments (r = 0.93; p G 0.0001). Our instrument demonstrated construct validity for both TBC and GRS, showing higher scores with increasing surgical experience in otology. Both instruments showed high interitem reliability with Cronbach > coefficients of 0.98 and 0.95 for TBC and GRS, respectively. Regression analysis showed that thinning posterior external auditory canal ( p G 0.05) and opening antrum to deepen dissection at sinodural angle ( p G 0.05) were the strongest predictors of overall surgical performance. Conclusion: Our assessment tool is a feasible and valid method of evaluating acquisition of mastoidectomy skills in the operating room. It can be integrated into surgical teaching in the operating room and yields information for direct formative feedback. Key Words: CompetenceVEvaluationV MastoidectomyVObjective structured assessment of technical skillsVOperating roomVOtolaryngologyVSurgical education.
Annals of Otology, Rhinology & Laryngology, 2019
Objective: Competency-based surgical training involves progressive autonomy given to the trainee. This requires systematic and evidence-based assessment with well-defined standards of proficiency. The objective of this study is to develop standards for the cross-institutional mastoidectomy assessment tool to inform decisions regarding whether a resident demonstrates sufficient skill to perform a mastoidectomy with or without supervision. Methods: A panel of fellowship-trained content experts in mastoidectomy was surveyed in relation to the 16 items of the assessment tool to determine the skills needed for supervised and unsupervised surgery. We examined the consensus score to investigate the degree of agreement among respondents for each survey item as well as additional analyses to determine whether the reported skill level required for each survey item was significantly different for the supervised versus unsupervised level. Results: Ten panelists representing different US trainin...
Creating a cross-institutional grading scale for temporal bone dissection
The Laryngoscope, 2010
Objectives/Hypothesis-There is increasing interest in objective assessment of surgeon competence. In the field of otolaryngology, several surgical training programs, including The Ohio State University, the University of Toronto, and Stanford University, have pursued standardized criteria to rate their trainees' performance in the initial steps of temporal bone dissection (complete mastoidectomy with facial recess approach). Although these assessment metrics require the completion of similar basic components integral to successful temporal bone dissection, certain listed criteria are unique to each institution. Our aim was to establish a more standardized set of criteria that can be used across different institutions to objectively assess temporal bone dissection. We translated these new criteria into automated metrics in our temporal bone dissection simulator to achieve even more objective grading of temporal bone dissections.
Determinants of resident competence in mastoidectomy: Role of interest and deliberate practice
The Laryngoscope, 2013
Objectives/Hypothesis: This study explores the influence of selected factors on achievement of competency in mastoid surgery. Study Design: A prospective study of surgical performance and a retrospective survey of learner and training factors. Methods: The longitudinal performance of 15 residents was evaluated using the mastoidectomy task-based checklist (TBC). The influence of surgical experience, resident interest, and training factors (course attendance, voluntary use of simulation laboratory) was also examined and compared for the acquisition of distinct levels of technical skill difficulty (cortical mastoidectomy vs. facial recess). Results: Ninety-six observations made during the first otology rotation were analyzed. Cortical mastoidectomy tasks showed positive associations with cumulated case numbers (
Automatic scoring of virtual mastoidectomies using expert examples
International Journal of Computer Assisted Radiology and Surgery, 2012
Purpose-Automatic scoring of resident performance on a virtual mastoidectomy simulation system is needed to achieve consistent and efficient evaluations. By not requiring immediate expert intervention, the system provides a completely objective assessment of performance as well as a self-driven user assessment mechanism.
Feasibility of a Synthetic Temporal Bone for Training in Mastoidectomy
Otology & Neurotology, 2014
Objective: To investigate the face, content, and concurrent validity of the synthetic Pettigrew temporal bone (PTB) for mastoidectomy training as compared with cadaveric temporal bone (CTB). Study Design: A prospective evaluation study. Methods: Participants were invited to perform a step-by-step modified radical mastoidectomy using both bones and complete a 22-item, 5-point Likert scale questionnaire. The questionnaire is divided into 4 domains: face validity (FV), global content (GC), task-specific content (TSC), and curriculum recommendation (CR). Results: Thirty-six experts and 89 trainees completed all tasks, 63 using CTB and 62 using PTB.
Pre-Operative Temporal Bone CT Scan Readings and Intraoperative Findings During Mastoidectomy
Philippine Journal of Otolaryngology Head and Neck Surgery
Objective: To determine the correlation between pre-operative in-house temporal bone CT scan readings and intraoperative findings during mastoidectomy for cholesteatoma in a tertiary government hospital from January 2018 to December 2019. Methods: Design: Review of Records Setting: Tertiary Government Hospital Participants: A total of 25 charts were included in the study. Surgical memoranda containing intraoperative findings were scrutinized. Data on key structures or locations were filled into a data gathering tool. Categorical descriptions were used for surgical findings: “present” or “absent” for location, and “intact” or “eroded” for status of ossicles and critical structures. Radiological readings to describe location and extent of disease were recorded as either “involved” or “uninvolved,” while “intact” or “eroded” were used to describe the status of ossicles and critical structures identified. Statistical correlations were computed using Cohen kappa coefficient. Sensitivity,...
Virtual mastoidectomy performance evaluation through multi-volume analysis
International Journal of Computer Assisted Radiology and Surgery, 2013
Purpose Development of a visualization system that provides surgical instructors with a method to compare the results of many virtual surgeries (n > 100). Methods A masked distance field models the overlap between expert and resident results. Multiple volume displays are used side-by-side with a 2D point display.
Evaluating drilling and suctioning technique in a mastoidectomy simulator
Studies in health technology and informatics, 2007
This paper presents several new metrics related to bone removal and suctioning technique in the context of a mastoidectomy simulator. The expertise with which decisions as to which regions of bone to remove and which to leave intact is evaluated by building a Naïve Bayes classifier using training data from known experts and novices. Since the bone voxel mesh is very large, and many voxels are always either removed or not removed regardless of expertise, the mutual information was calculated for each voxel and only the most informative voxels used for the classifier. Leave-out-one cross validation showed a high correlation of calculated expert probabilities with scores assigned by instructors. Additional metrics described in this paper include those for assessing smoothness of drill strokes, proper drill burr selection, sufficiency of suctioning, two-handed tool coordination, and application of appropriate force and velocity magnitudes as functions of distance from critical structures.