An explanatory mixed methods study on the validity and validation of students' assessment results in the undergraduate surgery course (original) (raw)
Related papers
Journal of American Science, 2016
Background/ Purpose: Evidential bases were not performed en masse to validate assessment results in the undergraduate Surgery clerkship in King Abdulaziz University (KAU). This study aimed at producing a comprehensive package of evidence to prove validity of students' clinical performance assessment results (as defined by Messick's framework). Method: Guided by Messick's conceptual framework, the problem was analyzed. Hands-on faculty development on creating an exam blueprint was done: 1. Learning objectives (LOs) revised; 2. Alignment secured; 3. Weight of (LOs) determined; 4. Number of items/topic/domain calculated; and 5. Appropriate assessment methods selected..Quantitative evidences as reliability and correlation coefficients of various validity components were calculated. The underlying values that scaffold validity evidences were explored via a Focus Group Discussion and the results analyzed by content analysis. Results: 1. The weight of different domains in the test equally reflected their weight in the curriculum (content validity); 2. Positive unintended consequences resulted from the new assessment approach (consequential validity); 3. There was a statistically significant correlation among various assessment methods that provided evidence for concurrent and predictive validity; 4. Success rates and grades distribution alone could not provide evidence to advocate an argument on validity of results. Conclusion: A newly introduced assessment plan with new tools had to be validated by pursuing a comprehensive, unified approach to create evidence from multiple sources of data in order to support the argument of advocating the assessment results.
Southern Medical Journal, 2007
Background: Despite calls for competency based education, a dearth of validated instruments for measuring basic skills currently exists. We developed an instrument to assess competency in basic surgical skills in second-year medical students and tested it for psychometric reliability and validity. Methods: From a review of the literature, an instrument comprised of numerically scaled items was constructed. After initial tests, several items were divided to produce a final instrument more specific and more appropriate for providing feedback to students. The final instrument was empirically tested for reliability and validity. Results: The final 10-item instrument is presented here along with all of the empirical evidence including internal consistency reliability and interrater reliability, and content, criterion-related, and construct validity. Overall alpha reliability was 0.84 and interrater reliability was r ϭ 0.83, P Ͻ 0.01 for the total scores. Factor analysis provided evidence of construct validity. Conclusions: The instrument has psychometric properties adequate for use as one criterion for summative evaluation and is educationally practical enough to provide focused and detailed feedback for student improvement.
Education in Medicine Journal, 2012
Background: Summative assessment in postgraduate examination globally employs multiple measures. A standard-setting method decides on pass or fail based on an arbitrarily defined cutoff point on a test score, which is often content expert's subjective judgment. Contrary to this a standard-setting strategy primarily practices two approaches, a compensatory approach, which decides on overall performance as a sum of all the test scores and a conjunctive approach that requires passing performance for each instrument. However, the challenge using multiple measures is not due to number of measurement tools but due to logic by which the measures are combined to draw inferences on pass or fail in summative assessment. Conjoint University Board of Examination of Masters' of Otolaryngology and Head-Neck Surgery (ORL-HNS) in Malaysia also uses multiple measures to reach a passing or failing decision in summative assessment. However, the standard setting strategy of assessment is loosely and variably applied to make ultimate decision on pass or fail. To collect the evidences, the summative assessment program of Masters' of ORL-HNS in School of Medical Sciences at Universiti Sains Malaysia was analyzed for validity to evaluate the appropriateness of decisions in postgraduate medical education in Malaysia. Method: A retrospective study was undertaken to evaluate the validity of the conjoint summative assessment results of part II examination of USM candidates during May 2000-May 2011. The Pearson correlation and multiple linear regression tests were used to determine the discriminant and convergent validity of assessment tools. Pearson's correlation coefficient analyzed the association between assessment tools and the multiple linear regression compared the dominant roles of factor variables in predicting outcomes. Based on outcome of the study, reforms for standard-setting strategy are also recommended towards programming the assessment in a surgical-based discipline. Result: The correlation coefficients of MCQ and essay questions were found not significant (0.16). Long and short cases were shown to have good correlations (0.53). Oral test stood as a component to show fair correlation with written (0.39-0.42) as well as clinical component (0.50-0.66). The predictive values in written tests suggested MCQ predicted by oral (B=0.34, P<0.01) and essay predicted by long case (B= 0.23, p<0.01). In clinical components long case predicted by oral (B= 0.71, p<0.05) and short cases predicted by long case (B= 0.31, p<0.001). Conclusion: The recorded discriminant and convergent validity evidences conclude that MCQ and essay do not correlate, nor do they predict each other. Long case and short cases significantly correlate with each other however, short cases are predicted by long case. All components though predict the overall performance, long case has the dominant role. The study outcome provides enough evidence to reconsider role of quantitative as well as qualitative evaluation in high stake examination of surgical-based discipline of ORL-HNS.
BMC Medical Education, 2013
Background We evaluated a newly designed electronic portfolio (e-Portfolio) that provided quantitative evaluation of surgical skills. Medical students at the University of Seville used the e-Portfolio on a voluntary basis for evaluation of their performance in undergraduate surgical subjects. Methods Our new web-based e-Portfolio was designed to evaluate surgical practical knowledge and skills targets. Students recorded each activity on a form, attached evidence, and added their reflections. Students self-assessed their practical knowledge using qualitative criteria (yes/no), and graded their skills according to complexity (basic/advanced) and participation (observer/assistant/independent). A numerical value was assigned to each activity, and the values of all activities were summated to obtain the total score. The application automatically displayed quantitative feedback. We performed qualitative evaluation of the perceived usefulness of the e-Portfolio and quantitative evaluation ...
2020
Introduction the outcome of the undergraduate medical training programme in South Africa is to produce competent medical doctors who can integrate knowledge, skills and attitudes relevant to the South African context. Training facilities have a responsibility to ensure that they perform this assessment of competence effectively and defend the results of high-stakes assessments. This study aimed to obtain qualitative data to suggest practical recommendations on best assessment practices to address the gaps between theoretical principles that inform assessment and current assessment practices. Methods a focus group interview was used to gather this data. The teaching and learning coordinators for five of the six modules that are offered in the clinical phase of the undergraduate medical programme participated in the focus group interview. The focus group interview proceeded as planned and took 95 minutes to complete. The responses were transcribed and recorded on a matrix. Results the...
The American Journal of Surgery, 2014
BACKGROUND: The Surgery Clerkship Clinical Skills Examination (CSE) is a novel written examination developed to assess the surgical knowledge, clinical decision making, communication skills, and professionalism of medical students on the surgery clerkship. This study was undertaken to determine its validity. METHODS: Data were prospectively collected from July 2011 through February 2013. Multivariate linear and logistic regression analyses were used to assess score trend; convergent validity with National Board of Medical Examiners surgery and medicine subject scores, United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge scores, and evaluation of clinical reasoning and fund of knowledge; and the effect of clerkship order. Exam reliability was assessed using a modified Cronbach's a statistic. RESULTS: During the study period, 262 students completed the CSE, with a normal distribution of performance. United States Medical Licensing Examination Step 2 Clinical Knowledge score and endof-clerkship evaluations of fund of knowledge and clinical reasoning predicted CSE score. Performance on the CSE was independent of clerkship order or prior clerkships. The modified Cronbach's a value for the exam was .67. CONCLUSIONS: The CSE is an objective, valid, reliable instrument for assessing students on the surgery clerkship, independent of clerkship order.
Education in Medicine Journal, 2024
The existence of a good clerkship system and quality lecturers will undoubtedly enhance the medical students' knowledge and skills that align with their competencies. This study aimed to evaluate the achievement of surgical competence based on the expectations of students following clinical clerkships in a surgical department. This study included medical students who participated in a clinical clerkship in the surgical department using a mixed method with a sequential explanatory design as a quantitative approach. Pre-and post-tests were also used to assess changes in student knowledge, qualitative results were obtained from focus group discussions (FGD), and standardised questionnaires were used to assess student skills expectations and achievements. The results showed a significant increase in students' knowledge after participating in a clinical clerkship, with a mean difference of 3.94 [standard deviation (SD) = 10.67]. However, only three of the skills met students' expectations during clinical clerkship. This was attributed to a lack of opportunities to practice the skills that students had acquired, the presence of too many students, limited time, and a shortage of practical placements in regional hospitals. Most students perceived the majority of lecturers as nurturing and proficient. However, some lecturers are often occupied with their commitment, resulting in infrequent student interactions. The clinical clerkship system for the Department of Surgery at the Faculty of Medicine, Hasanuddin University effectively enhanced student knowledge, but it needs to be improved to meet their skill expectations.