Susan Humphrey-murto - Academia.edu (original) (raw)
Papers by Susan Humphrey-murto
Teaching and learning in medicine, Jan 10, 2016
Construct: This article describes the development and validity evidence behind a new rating scale... more Construct: This article describes the development and validity evidence behind a new rating scale to assess feedback quality in the clinical workplace. Competency-based medical education has mandated a shift to learner-centeredness, authentic observation, and frequent formative assessments with a focus on the delivery of effective feedback. Because feedback has been shown to be of variable quality and effectiveness, an assessment of feedback quality in the workplace is important to ensure we are providing trainees with optimal learning opportunities. The purposes of this project were to develop a rating scale for the quality of verbal feedback in the workplace (the Direct Observation of Clinical Skills Feedback Scale [DOCS-FBS]) and to gather validity evidence for its use. Two panels of experts (local and national) took part in a nominal group technique to identify features of high-quality feedback. Through multiple iterations and review, 9 features were developed into the DOCS-FBS....
International Journal of Nursing Studies, 2016
Where a licence is displayed above, please note the terms and conditions of the licence govern yo... more Where a licence is displayed above, please note the terms and conditions of the licence govern your use of this document. When citing, please reference the published version. Take down policy While the University of Birmingham exercises care and attention in making items available there are rare occasions when an item has been uploaded in error or has been deemed to be commercially or otherwise sensitive.
Medical Education, 2016
Progress tests, in which learners are repeatedly assessed on equivalent content at different time... more Progress tests, in which learners are repeatedly assessed on equivalent content at different times in their training and provided with feedback, would seem to lend themselves well to a competency-based framework, which requires more frequent formative assessments. The objective structured clinical examination (OSCE) progress test is a relatively new form of assessment that is used to assess the progression of clinical skills. The purpose of this study was to establish further evidence for the use of an OSCE progress test by demonstrating an association between scores from this assessment method and those from a national high-stakes examination. The results of 8 years' of data from an Internal Medicine Residency OSCE (IM-OSCE) progress test were compared with scores on the Royal College of Physicians and Surgeons of Canada Comprehensive Objective Examination in Internal Medicine (RCPSC IM examination), which is comprised of both a written and performance-based component (n = 180). Correlations between scores in the two examinations were calculated. Logistic regression analyses were performed comparing IM-OSCE progress test scores with an 'elevated risk of failure' on either component of the RCPSC IM examination. Correlations between scores from the IM-OSCE (for PGY-1 residents to PGY-4 residents) and those from the RCPSC IM examination ranged from 0.316 (p = 0.001) to 0.554 (<.001) for the performance-based component and 0.305 (p = 0.002) to 0.516 (p < 0.001) for the written component. Logistic regression models demonstrated that PGY-2 and PGY-4 scores from the IM-OSCE were predictive of an 'elevated risk of failure' on both components of the RCPSC IM examination. This study provides further evidence for the use of OSCE progress testing by demonstrating a correlation between scores from an OSCE progress test and a national high-stakes examination. Furthermore, there is evidence that OSCE progress test scores are predictive of future performance on a national high-stakes examination.
![Research paper thumbnail of iMedLearn – Developing and Evaluating an Innovative Automated Tool to Support Self-Directed Learning in Family Medicine Resident Training [POSTER]](https://attachments.academia-assets.com/83921583/thumbnails/1.jpg)
](
Medical teacher, Jan 24, 2015
The purpose of this study was to explore the use of an objective structured clinical examination ... more The purpose of this study was to explore the use of an objective structured clinical examination for Internal Medicine residents (IM-OSCE) as a progress test for clinical skills. Data from eight administrations of an IM-OSCE were analyzed retrospectively. Data were scaled to a mean of 500 and standard deviation (SD) of 100. A time-based comparison, treating post-graduate year (PGY) as a repeated-measures factor, was used to determine how residents' performance progressed over time. Residents' total IM-OSCE scores (n = 244) increased over training from a mean of 445 (SD = 84) in PGY-1 to 534 (SD = 71) in PGY-3 (p < 0.001). In an analysis of sub-scores, including only those who participated in the IM OSCE for all three years of training (n = 46), mean structured oral scores increased from 464 (SD = 92) to 533 (SD = 83) (p < 0.001), physical examination scores increased from 464 (SD = 82) to 520 (SD = 75) (p < 0.001), and procedural skills increased from 495 (SD = 99) ...
Radiology, 2015
Imaging Findings Figure 1 shows sagittal T2-weighted (Fig 1a) and pre-(Fig 1b) and postgadolinium... more Imaging Findings Figure 1 shows sagittal T2-weighted (Fig 1a) and pre-(Fig 1b) and postgadolinium (Fig 1c) T1-weighted MR images of the cervical and upper thoracic spine acquired with a 1.5-T MR imager. The images revealed a broad dural-based ex tramedullary mass spanning vertebral levels C7 through T7. The mass partially encircled and compressed the spinal cord at these levels. This resulted in cord edema, seen as T2 hyperintense signal intensity that progressively increased at follow-up. The bone marrow was not involved, and the mass did not appear to invade the spinal cord parenchyma. The lesion appeared profoundly hypointense on T2-weighted images (Fig 1a). On T1-weighted images, the mass
Advances in Health Sciences Education, 2014
Medical Education, 2014
The merits of informal learning have been widely reported and embraced by medical educators. Howe... more The merits of informal learning have been widely reported and embraced by medical educators. However, research has yet to describe in detail the extent to which informal intraprofessional or informal interprofessional education is part of graduate medical education (GME), and the nature of those informal education experiences. This study seeks to describe: (i) who delivers informal education to residents; (ii) how often they do so; (iii) the content they share; and (iv) the teaching techniques they use. This study describes instances of informal learning in GME captured through non-participant observations in two contexts: a palliative care hospice and a paediatric hospital. Analysis of 60 hours of observation data involved a process of collaborative team consensus to: (i) identify instances of informal intraprofessional and informal interprofessional education, and (ii) categorise these instances by CanMEDS Role and teaching technique. Findings indicate that 84.8% of GME-level informal education that takes place in these two settings is physician-led and 15.2% is nurse-led. Organised by CanMEDS Role, findings reveal that, although all Roles are addressed by both physicians and nurses, those most commonly addressed are Medical Expert (physicians: 35.7%; nurses: 27.5%) and Communicator (physicians: 22.3%; nurses: 25.0%). Organised by teaching technique, findings reveal that physicians and nurses favour similar techniques. Although it is not surprising that informal interprofessional education plays a lesser role than informal intraprofessional education in GME, these findings suggest that the role of informal interprofessional education is worthy of support. Echoing the calls of others, we posit that medical education should recognise and capitalise on the contributions of informal learning, whether it occurs intra- or interprofessionally.
Teaching and Learning in Medicine, 2004
Effective education of clinical skills is essential if doctors are to meet the needs of patients ... more Effective education of clinical skills is essential if doctors are to meet the needs of patients with rheumatic disease, but shrinking faculty numbers has made clinical teaching difficult. A solution to this problem is to utilize patient educators. This study evaluates the teaching effectiveness of patient educators compared to rheumatology faculty using the musculoskeletal (MSK) examination. Sixty-two 2nd-year medical students were randomized to receive instruction from patient educators or faculty. Tutorial groups received instructions during three, 3-hr sessions. Clinical skills were evaluated by a 9 station objective structured clinical examination. Students completed a tutor evaluation form to assess their level of satisfaction with the process. Faculty-taught students received a higher overall mark (66.5% vs. 62.1%,) and fewer failed than patient educator-taught students (5 vs. 0, p = 0.02). Students rated faculty educators higher than patient educators (4.13 vs. 3.58 on a 5-point Likert scale). Rheumatology faculty appear to be more effective teachers of the MSK physical exam than patient educators.
Medicine, 1997
The present survey of cytotoxic therapy from a single large lupus clinic has shown that approxima... more The present survey of cytotoxic therapy from a single large lupus clinic has shown that approximately 33% of the patients have received cytotoxic therapy at some point in their course. These agents were initiated for a variety of manifestations, with renal manifestations being the major indication, accounting for 28.2% of the cytotoxic agents used. Other common indications for initiation of cytotoxic therapy included steroid sparing (18.4%), global flare (12.5%), neurologic manifestations (11.4%), and musculoskeletal (8.6%). Azathioprine, methotrexate, and cyclophosphamide accounted for 98% of all cytotoxic agents used. Azathioprine was the most frequently used cytotoxic drug (70%), followed by methotrexate (21.5%) and cyclophosphamide (9.4%). Cytotoxic agents were used sequentially in 12.5% of patients and in combination in 4.2% of the patients. Overall, the use of cytotoxic therapy appears to be beneficial in reducing global disease activity, as the mean SLEDAI fell by 2.59 (33%) over 6 months of cytotoxic therapy, and the mean steroid dose was reduced by 37% over the same time period. There was also an improvement in most organ-specific indications with the use of cytotoxic agents. Overall the cytotoxic agents were well tolerated, with 17% of the courses being discontinued due to a side effect. Cytopenia was the most common side effect necessitating discontinuation of cytotoxic agents.
Medical Teacher, 2005
Recruitment of physician examiners for an objective structured clinical examination (OSCE) can be... more Recruitment of physician examiners for an objective structured clinical examination (OSCE) can be difficult. The following study will explore reasons why physicians volunteer their time to be OSCE examiners. A questionnaire was collected from 110 examiners including a fourth year formative student OSCE (SO) (n=49), formative internal medicine OSCE (IM) (n=21) and the Medical Council of Canada Qualifying Exam Part II (MCCQE II) (n=40). A 5-point Likert scale was used. Statements with high mean ratings overall included: enjoy being an examiner (4.05), gain insights into learners&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; skills and knowledge (4.27), and examine out of a sense of duty (4.10). The MCC participants produced higher ratings (p&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;0.05). Overall, OSCE examiners volunteer their time because they enjoy the experience, feel a sense of duty and gain insight into learners&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; skills and knowledge. The MCC examiners appear to value the experience more. The ability to provide feedback and the provision of CME credits were not significant factors for increasing examiner satisfaction.
Medical Education, 2014
The shift from a time-based to a competency-based framework in medical education has created a ne... more The shift from a time-based to a competency-based framework in medical education has created a need for frequent formative assessments. Many educational programmes use some form of written progress test to identify areas of strength and weakness and to promote continuous improvement in their learners. However, the role of performance-based assessments, such as objective structured clinical examinations (OSCEs), in progress testing remains unclear. The aims of this paper are to describe the use of an OSCE to assess learners at different stages of training, describe a structure for reporting scores, and provide evidence for the psychometric properties of different rating tools. A 10-station OSCE was administered to internal medicine residents in postgraduate years (PGYs) 1-4. Candidates were assessed using a checklist (CL), a global rating scale (GRS) and a training level rating scale (TLRS). Reliability was calculated for each measure using Cronbach&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39;s alpha. Differences in performance by year of training were explored using analysis of variance (anova). Correlations between scores obtained using the different rating instruments were calculated. Sixty-nine residents participated in the OSCE. Inter-station reliability was greater (0.88) using the TLRS compared with the CL (0.84) and GRS (0.79). Using all three rating instruments, scores varied significantly by year of training (p &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt; 0.001). Scores from the different rating instruments were highly correlated: CL and GRS, r = 0.93; CL and TLRS, r = 0.90, and GRS and TLRS, r = 0.94 (p &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt; 0.001). Candidates received feedback on their performance relative to examiner expectations for their PGY level. Scores were found to have high reliability and demonstrated significant differences in performance by year of training. This provides evidence for the validity of using scores achieved on an OSCE as markers of progress in learners at different levels of training. Future studies will focus on assessing individual progress on the OSCE over time.
Medical Education, 2009
CONTEXT The objective structured clinical examination (OSCE) requires the use of standardised pat... more CONTEXT The objective structured clinical examination (OSCE) requires the use of standardised patients (SPs). Recruitment of SPs can be challenging and factors assumed to be neutral may vary between SPs. On stations that are considered gender-neutral, either male or female SPs may be used. This may lead to an increase in measurement error. Prior studies on SP gender have often confounded gender with case. OBJECTIVE The objective of this study was to assess whether a variation in SP gender on the same case resulted in a systematic difference in student scores. METHODS At the University of Ottawa, 140 Year 3 medical students participated in a 10-station OSCE. Two physical examination stations were selected for study because they were perceived to be 'gender-neutral'. One station involved the physical examination of the back and the other of the lymphatic system. On each of the study stations, male and female SPs were randomly allocated. RESULTS There was no difference in mean scores on the back examination station for students with female (6.96 ⁄ 10.00) versus male (7.04 ⁄ 10.00) SPs (P = 0.713). However, scores on the lymphatic system examination station showed a significant difference, favouring students with female (8.30 ⁄ 10.00) versus male (7.41 ⁄ 10.00) SPs (P < 0.001). Results were not dependent on student gender. CONCLUSIONS The gender of the SP may significantly affect student performance in an undergraduate OSCE in a manner that appears to be unrelated to student gender. It would be prudent to use the same SP gender for the same case, even on seemingly gender-neutral stations.
Medical Education, 2014
Selected-response (SR) formats (e.g. multiple-choice questions) and constructed-response (CR) for... more Selected-response (SR) formats (e.g. multiple-choice questions) and constructed-response (CR) formats (e.g. short-answer questions) are commonly used to test the knowledge of examinees. Scores on SR formats are typically higher than scores on CR formats. This difference is often attributed to examinees being cued by options within an SR question, but there could be alternative explanations. The purpose of this study was to expand on previous work with regards to the cueing effect of SR formats by directly contrasting conditions that support cueing versus memory of previously seen questions. During an objective structured clinical examination, students (n = 144) completed two consecutive stations in which they were presented with the same written cases but in different formats. Group 1 students were presented with CR questions followed by SR questions. Group 2 students were presented with questions in reverse order. Participants were asked to describe their testing experience. Selected-response scores (M = 4.21/10) were statistically higher than the CR scores (M = 3.82/10). However, there was no significant interaction between sequence and format (F(1,142) = 1.59, p = 0.21, ηp2 = 0.01) with scores increasing from 3.49/10 to 4.06/10 in the group that started with CR and decreasing (4.38/10-4.15/10) in the group that started with SR first. Correlations between SR scores and CR scores were high (CR first = 0.78, SR first = 0.89). Questionnaire results indicated that students felt the SR format was easier and led to cueing. To better understand test performance, it is important to know how different response formats could influence results. Because SR scores were higher than CR scores, irrespective of the format seen first, the pattern is consistent with what would be expected if cueing rather than memory for prior questions led to higher SR scores. This could have implications for test designers, especially when selecting question formats.
Medical Education, 2014
Currently, a &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp... more Currently, a &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39;pedagogical gap&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; exists in distributed medical education in that distance educators teach medical students but typically do not have the opportunity to assess them in large-scale examinations such as the objective structured clinical examination (OSCE). We developed a remote examiner OSCE (reOSCE) that was integrated into a traditional OSCE to establish whether remote examination technology may be used to bridge this gap. The purpose of this study was to explore whether remote physician-examiners can replace on-site physician-examiners in an OSCE, and to determine the feasibility of this new examination method. Forty Year 3 medical students were randomised into six reOSCE stations that were incorporated into two tracks of a 10-station traditional OSCE. For the reOSCE stations, student performance was assessed by both a local examiner (LE) in the room and a remote examiner (RE) who viewed the OSCE encounters from a distance. The primary endpoint was the correlation of scores between LEs and REs across all reOSCE stations. The secondary endpoint was a post-OSCE survey of both REs and students. Statistically significant correlations were found between LE and RE checklist scores for history taking (r = 0.64-r = 0.80), physical examination (r = 0.41-r = 0.54), and management stations (r = 0.78). Correlations between LE and RE global ratings were more varied (r = 0.21-r = 0.77). Correlations on three of the six stations reached significance. Qualitative analysis of feedback from REs and students showed high acceptance of the reOSCE despite technological issues. This preliminary study demonstrated that OSCE ratings by LEs and REs were reasonably comparable when using checklists. Remote examination may be a feasible and acceptable way of assessing students&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; clinical skills, but further validity evidence will be required before it can be recommended for use in high-stakes examinations.
Medical Education, 2011
To help reduce pressure on faculty staff, medical students have been used as raters in objective ... more To help reduce pressure on faculty staff, medical students have been used as raters in objective structured clinical examinations (OSCEs). There are few studies regarding their ability to complete checklists and global rating scales, and a paucity of data on their ability to provide feedback to junior colleagues. The objectives of this study were: (i) to compare expert faculty examiner (FE) and student-examiner (SE) assessment of students' (candidates') performances on a formative OSCE; (ii) to assess SE feedback provided to candidates, and (iii) to seek opinion regarding acceptability from all participants. METHODS Year 2 medical students (candidates, n = 66) participated in a nine-station formative OSCE. Year 4 students (n = 27) acted as SEs and teaching doctors (n = 27) served as FEs. In each station, SEs and FEs independently scored the candidates using checklists and global rating scales. The SEs provided feedback to candidates after each encounter. The FEs evaluated SEs on the feedback provided using a standardised rating scale (1 = strongly disagree, 5 = strongly agree) for several categories, according to whether the feedback was: balanced; specific; accurate; appropriate; professional, and similar to feedback the FE would have provided. All participants completed questionnaires exploring perceptions and acceptability. RESULTS There was a high correlation on the checklist items between raters on each station, ranging from 0.56 to 0.86. Correlations on the global rating for each station ranged from 0.23 to 0.78. Faculty examiners rated SE feedback highly, with mean scores ranging from 4.02 to 4.44 for all categories. There was a high degree of acceptability on the part of candidates and examiners.
The Journal of Rheumatology, 2009
Objective.The in-training evaluation report (ITER) is widely used to assess clinical skills, but ... more Objective.The in-training evaluation report (ITER) is widely used to assess clinical skills, but has limited validity and reliability. The purpose of our study was to assess the feasibility, validity, reliability, and effect on feedback of using daily evaluation forms to evaluate residents in ambulatory rheumatology clinics.Methods.An evaluation form was developed based on the Royal College of Physicians and Surgeons of Canada CanMEDS roles. There were 12 evaluation items including overall clinical competence. They were rated on a 5-point scale from unsatisfactory to outstanding. All internal medicine residents rotating on rheumatology were strongly encouraged to provide the form to their preceptor at the end of each clinic. A questionnaire was administered to residents and faculty.Results.Seventy-three internal medicine residents completed a 1-month rotation at University of Ottawa (n = 26) and McMaster University (n = 47). Faculty members completed a total of 637 evaluation forms....
The Journal of Rheumatology, 2011
Teaching and learning in medicine, Jan 10, 2016
Construct: This article describes the development and validity evidence behind a new rating scale... more Construct: This article describes the development and validity evidence behind a new rating scale to assess feedback quality in the clinical workplace. Competency-based medical education has mandated a shift to learner-centeredness, authentic observation, and frequent formative assessments with a focus on the delivery of effective feedback. Because feedback has been shown to be of variable quality and effectiveness, an assessment of feedback quality in the workplace is important to ensure we are providing trainees with optimal learning opportunities. The purposes of this project were to develop a rating scale for the quality of verbal feedback in the workplace (the Direct Observation of Clinical Skills Feedback Scale [DOCS-FBS]) and to gather validity evidence for its use. Two panels of experts (local and national) took part in a nominal group technique to identify features of high-quality feedback. Through multiple iterations and review, 9 features were developed into the DOCS-FBS....
International Journal of Nursing Studies, 2016
Where a licence is displayed above, please note the terms and conditions of the licence govern yo... more Where a licence is displayed above, please note the terms and conditions of the licence govern your use of this document. When citing, please reference the published version. Take down policy While the University of Birmingham exercises care and attention in making items available there are rare occasions when an item has been uploaded in error or has been deemed to be commercially or otherwise sensitive.
Medical Education, 2016
Progress tests, in which learners are repeatedly assessed on equivalent content at different time... more Progress tests, in which learners are repeatedly assessed on equivalent content at different times in their training and provided with feedback, would seem to lend themselves well to a competency-based framework, which requires more frequent formative assessments. The objective structured clinical examination (OSCE) progress test is a relatively new form of assessment that is used to assess the progression of clinical skills. The purpose of this study was to establish further evidence for the use of an OSCE progress test by demonstrating an association between scores from this assessment method and those from a national high-stakes examination. The results of 8 years&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; of data from an Internal Medicine Residency OSCE (IM-OSCE) progress test were compared with scores on the Royal College of Physicians and Surgeons of Canada Comprehensive Objective Examination in Internal Medicine (RCPSC IM examination), which is comprised of both a written and performance-based component (n = 180). Correlations between scores in the two examinations were calculated. Logistic regression analyses were performed comparing IM-OSCE progress test scores with an &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39;elevated risk of failure&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; on either component of the RCPSC IM examination. Correlations between scores from the IM-OSCE (for PGY-1 residents to PGY-4 residents) and those from the RCPSC IM examination ranged from 0.316 (p = 0.001) to 0.554 (&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;.001) for the performance-based component and 0.305 (p = 0.002) to 0.516 (p &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt; 0.001) for the written component. Logistic regression models demonstrated that PGY-2 and PGY-4 scores from the IM-OSCE were predictive of an &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39;elevated risk of failure&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; on both components of the RCPSC IM examination. This study provides further evidence for the use of OSCE progress testing by demonstrating a correlation between scores from an OSCE progress test and a national high-stakes examination. Furthermore, there is evidence that OSCE progress test scores are predictive of future performance on a national high-stakes examination.
Medical teacher, Jan 24, 2015
The purpose of this study was to explore the use of an objective structured clinical examination ... more The purpose of this study was to explore the use of an objective structured clinical examination for Internal Medicine residents (IM-OSCE) as a progress test for clinical skills. Data from eight administrations of an IM-OSCE were analyzed retrospectively. Data were scaled to a mean of 500 and standard deviation (SD) of 100. A time-based comparison, treating post-graduate year (PGY) as a repeated-measures factor, was used to determine how residents' performance progressed over time. Residents' total IM-OSCE scores (n = 244) increased over training from a mean of 445 (SD = 84) in PGY-1 to 534 (SD = 71) in PGY-3 (p < 0.001). In an analysis of sub-scores, including only those who participated in the IM OSCE for all three years of training (n = 46), mean structured oral scores increased from 464 (SD = 92) to 533 (SD = 83) (p < 0.001), physical examination scores increased from 464 (SD = 82) to 520 (SD = 75) (p < 0.001), and procedural skills increased from 495 (SD = 99) ...
Radiology, 2015
Imaging Findings Figure 1 shows sagittal T2-weighted (Fig 1a) and pre-(Fig 1b) and postgadolinium... more Imaging Findings Figure 1 shows sagittal T2-weighted (Fig 1a) and pre-(Fig 1b) and postgadolinium (Fig 1c) T1-weighted MR images of the cervical and upper thoracic spine acquired with a 1.5-T MR imager. The images revealed a broad dural-based ex tramedullary mass spanning vertebral levels C7 through T7. The mass partially encircled and compressed the spinal cord at these levels. This resulted in cord edema, seen as T2 hyperintense signal intensity that progressively increased at follow-up. The bone marrow was not involved, and the mass did not appear to invade the spinal cord parenchyma. The lesion appeared profoundly hypointense on T2-weighted images (Fig 1a). On T1-weighted images, the mass
Advances in Health Sciences Education, 2014
Medical Education, 2014
The merits of informal learning have been widely reported and embraced by medical educators. Howe... more The merits of informal learning have been widely reported and embraced by medical educators. However, research has yet to describe in detail the extent to which informal intraprofessional or informal interprofessional education is part of graduate medical education (GME), and the nature of those informal education experiences. This study seeks to describe: (i) who delivers informal education to residents; (ii) how often they do so; (iii) the content they share; and (iv) the teaching techniques they use. This study describes instances of informal learning in GME captured through non-participant observations in two contexts: a palliative care hospice and a paediatric hospital. Analysis of 60 hours of observation data involved a process of collaborative team consensus to: (i) identify instances of informal intraprofessional and informal interprofessional education, and (ii) categorise these instances by CanMEDS Role and teaching technique. Findings indicate that 84.8% of GME-level informal education that takes place in these two settings is physician-led and 15.2% is nurse-led. Organised by CanMEDS Role, findings reveal that, although all Roles are addressed by both physicians and nurses, those most commonly addressed are Medical Expert (physicians: 35.7%; nurses: 27.5%) and Communicator (physicians: 22.3%; nurses: 25.0%). Organised by teaching technique, findings reveal that physicians and nurses favour similar techniques. Although it is not surprising that informal interprofessional education plays a lesser role than informal intraprofessional education in GME, these findings suggest that the role of informal interprofessional education is worthy of support. Echoing the calls of others, we posit that medical education should recognise and capitalise on the contributions of informal learning, whether it occurs intra- or interprofessionally.
Teaching and Learning in Medicine, 2004
Effective education of clinical skills is essential if doctors are to meet the needs of patients ... more Effective education of clinical skills is essential if doctors are to meet the needs of patients with rheumatic disease, but shrinking faculty numbers has made clinical teaching difficult. A solution to this problem is to utilize patient educators. This study evaluates the teaching effectiveness of patient educators compared to rheumatology faculty using the musculoskeletal (MSK) examination. Sixty-two 2nd-year medical students were randomized to receive instruction from patient educators or faculty. Tutorial groups received instructions during three, 3-hr sessions. Clinical skills were evaluated by a 9 station objective structured clinical examination. Students completed a tutor evaluation form to assess their level of satisfaction with the process. Faculty-taught students received a higher overall mark (66.5% vs. 62.1%,) and fewer failed than patient educator-taught students (5 vs. 0, p = 0.02). Students rated faculty educators higher than patient educators (4.13 vs. 3.58 on a 5-point Likert scale). Rheumatology faculty appear to be more effective teachers of the MSK physical exam than patient educators.
Medicine, 1997
The present survey of cytotoxic therapy from a single large lupus clinic has shown that approxima... more The present survey of cytotoxic therapy from a single large lupus clinic has shown that approximately 33% of the patients have received cytotoxic therapy at some point in their course. These agents were initiated for a variety of manifestations, with renal manifestations being the major indication, accounting for 28.2% of the cytotoxic agents used. Other common indications for initiation of cytotoxic therapy included steroid sparing (18.4%), global flare (12.5%), neurologic manifestations (11.4%), and musculoskeletal (8.6%). Azathioprine, methotrexate, and cyclophosphamide accounted for 98% of all cytotoxic agents used. Azathioprine was the most frequently used cytotoxic drug (70%), followed by methotrexate (21.5%) and cyclophosphamide (9.4%). Cytotoxic agents were used sequentially in 12.5% of patients and in combination in 4.2% of the patients. Overall, the use of cytotoxic therapy appears to be beneficial in reducing global disease activity, as the mean SLEDAI fell by 2.59 (33%) over 6 months of cytotoxic therapy, and the mean steroid dose was reduced by 37% over the same time period. There was also an improvement in most organ-specific indications with the use of cytotoxic agents. Overall the cytotoxic agents were well tolerated, with 17% of the courses being discontinued due to a side effect. Cytopenia was the most common side effect necessitating discontinuation of cytotoxic agents.
Medical Teacher, 2005
Recruitment of physician examiners for an objective structured clinical examination (OSCE) can be... more Recruitment of physician examiners for an objective structured clinical examination (OSCE) can be difficult. The following study will explore reasons why physicians volunteer their time to be OSCE examiners. A questionnaire was collected from 110 examiners including a fourth year formative student OSCE (SO) (n=49), formative internal medicine OSCE (IM) (n=21) and the Medical Council of Canada Qualifying Exam Part II (MCCQE II) (n=40). A 5-point Likert scale was used. Statements with high mean ratings overall included: enjoy being an examiner (4.05), gain insights into learners&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; skills and knowledge (4.27), and examine out of a sense of duty (4.10). The MCC participants produced higher ratings (p&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;0.05). Overall, OSCE examiners volunteer their time because they enjoy the experience, feel a sense of duty and gain insight into learners&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; skills and knowledge. The MCC examiners appear to value the experience more. The ability to provide feedback and the provision of CME credits were not significant factors for increasing examiner satisfaction.
Medical Education, 2014
The shift from a time-based to a competency-based framework in medical education has created a ne... more The shift from a time-based to a competency-based framework in medical education has created a need for frequent formative assessments. Many educational programmes use some form of written progress test to identify areas of strength and weakness and to promote continuous improvement in their learners. However, the role of performance-based assessments, such as objective structured clinical examinations (OSCEs), in progress testing remains unclear. The aims of this paper are to describe the use of an OSCE to assess learners at different stages of training, describe a structure for reporting scores, and provide evidence for the psychometric properties of different rating tools. A 10-station OSCE was administered to internal medicine residents in postgraduate years (PGYs) 1-4. Candidates were assessed using a checklist (CL), a global rating scale (GRS) and a training level rating scale (TLRS). Reliability was calculated for each measure using Cronbach&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39;s alpha. Differences in performance by year of training were explored using analysis of variance (anova). Correlations between scores obtained using the different rating instruments were calculated. Sixty-nine residents participated in the OSCE. Inter-station reliability was greater (0.88) using the TLRS compared with the CL (0.84) and GRS (0.79). Using all three rating instruments, scores varied significantly by year of training (p &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt; 0.001). Scores from the different rating instruments were highly correlated: CL and GRS, r = 0.93; CL and TLRS, r = 0.90, and GRS and TLRS, r = 0.94 (p &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt; 0.001). Candidates received feedback on their performance relative to examiner expectations for their PGY level. Scores were found to have high reliability and demonstrated significant differences in performance by year of training. This provides evidence for the validity of using scores achieved on an OSCE as markers of progress in learners at different levels of training. Future studies will focus on assessing individual progress on the OSCE over time.
Medical Education, 2009
CONTEXT The objective structured clinical examination (OSCE) requires the use of standardised pat... more CONTEXT The objective structured clinical examination (OSCE) requires the use of standardised patients (SPs). Recruitment of SPs can be challenging and factors assumed to be neutral may vary between SPs. On stations that are considered gender-neutral, either male or female SPs may be used. This may lead to an increase in measurement error. Prior studies on SP gender have often confounded gender with case. OBJECTIVE The objective of this study was to assess whether a variation in SP gender on the same case resulted in a systematic difference in student scores. METHODS At the University of Ottawa, 140 Year 3 medical students participated in a 10-station OSCE. Two physical examination stations were selected for study because they were perceived to be 'gender-neutral'. One station involved the physical examination of the back and the other of the lymphatic system. On each of the study stations, male and female SPs were randomly allocated. RESULTS There was no difference in mean scores on the back examination station for students with female (6.96 ⁄ 10.00) versus male (7.04 ⁄ 10.00) SPs (P = 0.713). However, scores on the lymphatic system examination station showed a significant difference, favouring students with female (8.30 ⁄ 10.00) versus male (7.41 ⁄ 10.00) SPs (P < 0.001). Results were not dependent on student gender. CONCLUSIONS The gender of the SP may significantly affect student performance in an undergraduate OSCE in a manner that appears to be unrelated to student gender. It would be prudent to use the same SP gender for the same case, even on seemingly gender-neutral stations.
Medical Education, 2014
Selected-response (SR) formats (e.g. multiple-choice questions) and constructed-response (CR) for... more Selected-response (SR) formats (e.g. multiple-choice questions) and constructed-response (CR) formats (e.g. short-answer questions) are commonly used to test the knowledge of examinees. Scores on SR formats are typically higher than scores on CR formats. This difference is often attributed to examinees being cued by options within an SR question, but there could be alternative explanations. The purpose of this study was to expand on previous work with regards to the cueing effect of SR formats by directly contrasting conditions that support cueing versus memory of previously seen questions. During an objective structured clinical examination, students (n = 144) completed two consecutive stations in which they were presented with the same written cases but in different formats. Group 1 students were presented with CR questions followed by SR questions. Group 2 students were presented with questions in reverse order. Participants were asked to describe their testing experience. Selected-response scores (M = 4.21/10) were statistically higher than the CR scores (M = 3.82/10). However, there was no significant interaction between sequence and format (F(1,142) = 1.59, p = 0.21, ηp2 = 0.01) with scores increasing from 3.49/10 to 4.06/10 in the group that started with CR and decreasing (4.38/10-4.15/10) in the group that started with SR first. Correlations between SR scores and CR scores were high (CR first = 0.78, SR first = 0.89). Questionnaire results indicated that students felt the SR format was easier and led to cueing. To better understand test performance, it is important to know how different response formats could influence results. Because SR scores were higher than CR scores, irrespective of the format seen first, the pattern is consistent with what would be expected if cueing rather than memory for prior questions led to higher SR scores. This could have implications for test designers, especially when selecting question formats.
Medical Education, 2014
Currently, a &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp... more Currently, a &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39;pedagogical gap&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; exists in distributed medical education in that distance educators teach medical students but typically do not have the opportunity to assess them in large-scale examinations such as the objective structured clinical examination (OSCE). We developed a remote examiner OSCE (reOSCE) that was integrated into a traditional OSCE to establish whether remote examination technology may be used to bridge this gap. The purpose of this study was to explore whether remote physician-examiners can replace on-site physician-examiners in an OSCE, and to determine the feasibility of this new examination method. Forty Year 3 medical students were randomised into six reOSCE stations that were incorporated into two tracks of a 10-station traditional OSCE. For the reOSCE stations, student performance was assessed by both a local examiner (LE) in the room and a remote examiner (RE) who viewed the OSCE encounters from a distance. The primary endpoint was the correlation of scores between LEs and REs across all reOSCE stations. The secondary endpoint was a post-OSCE survey of both REs and students. Statistically significant correlations were found between LE and RE checklist scores for history taking (r = 0.64-r = 0.80), physical examination (r = 0.41-r = 0.54), and management stations (r = 0.78). Correlations between LE and RE global ratings were more varied (r = 0.21-r = 0.77). Correlations on three of the six stations reached significance. Qualitative analysis of feedback from REs and students showed high acceptance of the reOSCE despite technological issues. This preliminary study demonstrated that OSCE ratings by LEs and REs were reasonably comparable when using checklists. Remote examination may be a feasible and acceptable way of assessing students&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;#39; clinical skills, but further validity evidence will be required before it can be recommended for use in high-stakes examinations.
Medical Education, 2011
To help reduce pressure on faculty staff, medical students have been used as raters in objective ... more To help reduce pressure on faculty staff, medical students have been used as raters in objective structured clinical examinations (OSCEs). There are few studies regarding their ability to complete checklists and global rating scales, and a paucity of data on their ability to provide feedback to junior colleagues. The objectives of this study were: (i) to compare expert faculty examiner (FE) and student-examiner (SE) assessment of students' (candidates') performances on a formative OSCE; (ii) to assess SE feedback provided to candidates, and (iii) to seek opinion regarding acceptability from all participants. METHODS Year 2 medical students (candidates, n = 66) participated in a nine-station formative OSCE. Year 4 students (n = 27) acted as SEs and teaching doctors (n = 27) served as FEs. In each station, SEs and FEs independently scored the candidates using checklists and global rating scales. The SEs provided feedback to candidates after each encounter. The FEs evaluated SEs on the feedback provided using a standardised rating scale (1 = strongly disagree, 5 = strongly agree) for several categories, according to whether the feedback was: balanced; specific; accurate; appropriate; professional, and similar to feedback the FE would have provided. All participants completed questionnaires exploring perceptions and acceptability. RESULTS There was a high correlation on the checklist items between raters on each station, ranging from 0.56 to 0.86. Correlations on the global rating for each station ranged from 0.23 to 0.78. Faculty examiners rated SE feedback highly, with mean scores ranging from 4.02 to 4.44 for all categories. There was a high degree of acceptability on the part of candidates and examiners.
The Journal of Rheumatology, 2009
Objective.The in-training evaluation report (ITER) is widely used to assess clinical skills, but ... more Objective.The in-training evaluation report (ITER) is widely used to assess clinical skills, but has limited validity and reliability. The purpose of our study was to assess the feasibility, validity, reliability, and effect on feedback of using daily evaluation forms to evaluate residents in ambulatory rheumatology clinics.Methods.An evaluation form was developed based on the Royal College of Physicians and Surgeons of Canada CanMEDS roles. There were 12 evaluation items including overall clinical competence. They were rated on a 5-point scale from unsatisfactory to outstanding. All internal medicine residents rotating on rheumatology were strongly encouraged to provide the form to their preceptor at the end of each clinic. A questionnaire was administered to residents and faculty.Results.Seventy-three internal medicine residents completed a 1-month rotation at University of Ottawa (n = 26) and McMaster University (n = 47). Faculty members completed a total of 637 evaluation forms....
The Journal of Rheumatology, 2011