Creating Evidence to Advocate the Validity of Results of Clinical Performance in the Undergraduate Surgery Clerkship (original) (raw)
Related papers
Medical teacher, 2018
There is inadequate evidence of reported validity of the results of assessment instruments used to assess clinical competence. This study aimed at combining multiple lines of quantitative and qualitative evidence to support interpretation and use of assessment results. This study is a mixed methods explanatory research set in two stages of data collection and analysis (QUAN : qual). Guided by Messick's conceptual model, quantitative evidences as reliability and correlation coefficients of various validity components were calculated using students' scores, grades and success rates of the whole population of students in 2012/2013 and 2013/2014 (n= 383; 326). The underlying values that scaffold validity evidences were identified via Focus Group Discussions (FGD) with faculty and students; sampling technique was purposive; and results were analyzed by content analysis. (1) Themes that resulted from content analysis aligned with quantitative evidences. (2) Assessment results show...
Education in Medicine Journal, 2012
Background: Summative assessment in postgraduate examination globally employs multiple measures. A standard-setting method decides on pass or fail based on an arbitrarily defined cutoff point on a test score, which is often content expert's subjective judgment. Contrary to this a standard-setting strategy primarily practices two approaches, a compensatory approach, which decides on overall performance as a sum of all the test scores and a conjunctive approach that requires passing performance for each instrument. However, the challenge using multiple measures is not due to number of measurement tools but due to logic by which the measures are combined to draw inferences on pass or fail in summative assessment. Conjoint University Board of Examination of Masters' of Otolaryngology and Head-Neck Surgery (ORL-HNS) in Malaysia also uses multiple measures to reach a passing or failing decision in summative assessment. However, the standard setting strategy of assessment is loosely and variably applied to make ultimate decision on pass or fail. To collect the evidences, the summative assessment program of Masters' of ORL-HNS in School of Medical Sciences at Universiti Sains Malaysia was analyzed for validity to evaluate the appropriateness of decisions in postgraduate medical education in Malaysia. Method: A retrospective study was undertaken to evaluate the validity of the conjoint summative assessment results of part II examination of USM candidates during May 2000-May 2011. The Pearson correlation and multiple linear regression tests were used to determine the discriminant and convergent validity of assessment tools. Pearson's correlation coefficient analyzed the association between assessment tools and the multiple linear regression compared the dominant roles of factor variables in predicting outcomes. Based on outcome of the study, reforms for standard-setting strategy are also recommended towards programming the assessment in a surgical-based discipline. Result: The correlation coefficients of MCQ and essay questions were found not significant (0.16). Long and short cases were shown to have good correlations (0.53). Oral test stood as a component to show fair correlation with written (0.39-0.42) as well as clinical component (0.50-0.66). The predictive values in written tests suggested MCQ predicted by oral (B=0.34, P<0.01) and essay predicted by long case (B= 0.23, p<0.01). In clinical components long case predicted by oral (B= 0.71, p<0.05) and short cases predicted by long case (B= 0.31, p<0.001). Conclusion: The recorded discriminant and convergent validity evidences conclude that MCQ and essay do not correlate, nor do they predict each other. Long case and short cases significantly correlate with each other however, short cases are predicted by long case. All components though predict the overall performance, long case has the dominant role. The study outcome provides enough evidence to reconsider role of quantitative as well as qualitative evaluation in high stake examination of surgical-based discipline of ORL-HNS.
A clear concept and understanding about the measure and the measuring tools is essential for good practice of assessment. Assessors need to have information about the full range of assessment tools inclusive of psychometric validity and purpose of its use. Subjective inferences drawn from the readily available data as numbers of summative scores over the years and statistical evidences of reliability and validity of assessment tools used to measure student's performance are good sources of feedback for competent assessment program. It also provides meaningful evaluation of learning and teaching in medical education. Method: A retrospective study of 119 candidates was carried out to analyze the summative assessment scores of their certifying examination of Masters of Surgery in School of Medical Sciences (SMS) at Universiti Sains Malaysia. Subjective judgment of raw data followed by internal consistency as reliability, convergent validity and discriminant validity as constructs of individual assessment tool was analyzed. Finally each assessment tool as a measure of written or clinical construct was evaluated against six aspects of Messick's criteria for quality control. Result: The correlation coefficient for validity and Cronbach's alpha for reliability was evaluated for clinical measures. However, the test of internal reliability was not possible for essay being the only measure in written construct of summative assessment in surgery. All measures of clinical construct were found highly reliable with Cronbach's alpha between 0.962-0.979. Long case and the short cases have shown excellent correlations (r=0.959 at p<0.001). Viva stood on its own and showed good correlation with long case (r=0.933 at p<0.001) as well as with short cases (r=0.926 at p<0.001). The linear regression analysis of essay measure was not predicted by any of the clinical measure. In clinical construct long case was strongly predicted by short case and vice versa (B=0.640 at p<0.001). Viva was predicted by the long case only (B=. 245 at p<. 001). All measures have positively predicted the overall performance however, the long case predominantly more than the other measure of this construct (r 2 =0.973 at p<. 001) Conclusion: Suggestions to improve the framework of assessment are proposed for future practice of competent assessment program in surgery.
Assessment of the Clinical Performance of Medical Students: A Survey of Methods
1983
This review of methods used to assess the Clinical performance of medical students foCuses on four common assessment approaches: (1) the examination developed by the National Board of Medical Examiners (NBME); (2) systematic, multifactor evaluation methods; (3) observation techniques; and (4) problem based methods. Analyzed in conjunction with each approach were reliability and validity data as we/1 as practicality of'the assessment approaches. The reliability and validity data are extensive and high for the NBME.'The NBME is theleast complicated instrument to administer and score. However, it cannot assess client-clinician interactions, utilizing live subjects. Reliability and validity data on observation methods are sparse; and where data exists, the coefficients are generally low. Mgltifadeted evaluation techniques have provided more accurate assessments of student competence, but require more time apd more people .to administer multiple assessments. .A final issue related to the assessment of' clinical competence involves the determination of a generally acceptable definition of competence. (Author /BW)
The American Journal of Surgery, 2014
BACKGROUND: The Surgery Clerkship Clinical Skills Examination (CSE) is a novel written examination developed to assess the surgical knowledge, clinical decision making, communication skills, and professionalism of medical students on the surgery clerkship. This study was undertaken to determine its validity. METHODS: Data were prospectively collected from July 2011 through February 2013. Multivariate linear and logistic regression analyses were used to assess score trend; convergent validity with National Board of Medical Examiners surgery and medicine subject scores, United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge scores, and evaluation of clinical reasoning and fund of knowledge; and the effect of clerkship order. Exam reliability was assessed using a modified Cronbach's a statistic. RESULTS: During the study period, 262 students completed the CSE, with a normal distribution of performance. United States Medical Licensing Examination Step 2 Clinical Knowledge score and endof-clerkship evaluations of fund of knowledge and clinical reasoning predicted CSE score. Performance on the CSE was independent of clerkship order or prior clerkships. The modified Cronbach's a value for the exam was .67. CONCLUSIONS: The CSE is an objective, valid, reliable instrument for assessing students on the surgery clerkship, independent of clerkship order.
Pakistan Journal of Medical Sciences, 1969
Medical education is the art and science of medical learning which has rapidly progressed in the recent years. Adequately measuring core competencies of the medical students is a vital constituent for evaluation, provision of reliable feedback and improving medical education. 1 Assessment is an essential compenent in medical education because of its implications on the students, teachers, communities and the whole world; after graduation from their medical schools. 2 Assessment is an integral component of overall educational activities