Ryan Brydges | University of Toronto (original) (raw)

Papers by Ryan Brydges

Journal of Graduate Medical Education, 2014

Advances in health sciences education : theory and practice, Jan 15, 2015

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2015

Instructor-led simulation-based mastery learning of advanced cardiac life support (ACLS) skills i... more Instructor-led simulation-based mastery learning of advanced cardiac life support (ACLS) skills is an effective and focused approach to competency-based education. Directed self-regulated learning (DSRL) may be an effective and less resource-intensive way to teach ACLS skills. Forty first-year internal medicine residents were randomized to either simulation-based DSRL or simulation-based instructor-regulated learning (IRL) of ACLS skills using a mastery learning model. Residents in each intervention completed pretest, posttest, and retention test of their performance in leading an ACLS response to a simulated scenario. Performance tests were assessed using a standardized checklist. Residents in the DSRL intervention were provided assessment instruments, a debriefing guide, and scenario-specific teaching points, and they were permitted to access relevant online resources. Residents in the IRL intervention had access to the same materials; however, the teaching and debriefing were instructor led. Skills of both the IRL and DSRL interventions showed significant improvement after the intervention, with an average improvement on the posttest of 21.7%. After controlling for pretest score, there was no difference between intervention arms on the posttest [F(1,37) = 0.02, P = 0.94] and retention tests [F(1,17) = 1.43, P = 0.25]. Cost savings were realized in the DSRL intervention after the fourth group (16 residents) had completed each intervention, with an ongoing savings of $80 per resident. Using a simulation-based mastery learning model, we observed equivalence in learning of ACLS skills for the DSRL and IRL conditions, whereas DSRL was more cost effective.

Medical Education, 2015

Assessment is central to medical education and the validation of assessments is vital to their us... more Assessment is central to medical education and the validation of assessments is vital to their use. Earlier validity frameworks suffer from a multiplicity of types of validity or failure to prioritise among sources of validity evidence. Kane's framework addresses both concerns by emphasising key inferences as the assessment progresses from a single observation to a final decision. Evidence evaluating these inferences is planned and presented as a validity argument. We aim to offer a practical introduction to the key concepts of Kane's framework that educators will find accessible and applicable to a wide range of assessment tools and activities. All assessments are ultimately intended to facilitate a defensible decision about the person being assessed. Validation is the process of collecting and interpreting evidence to support that decision. Rigorous validation involves articulating the claims and assumptions associated with the proposed decision (the interpretation/use argument), empirically testing these assumptions, and organising evidence into a coherent validity argument. Kane identifies four inferences in the validity argument: Scoring (translating an observation into one or more scores); Generalisation (using the score[s] as a reflection of performance in a test setting); Extrapolation (using the score[s] as a reflection of real-world performance), and Implications (applying the score[s] to inform a decision or action). Evidence should be collected to support each of these inferences and should focus on the most questionable assumptions in the chain of inference. Key assumptions (and needed evidence) vary depending on the assessment's intended use or associated decision. Kane's framework applies to quantitative and qualitative assessments, and to individual tests and programmes of assessment. Validation focuses on evaluating the key claims, assumptions and inferences that link assessment scores with their intended interpretations and uses. The Implications and associated decisions are the most important inferences in the validity argument.

BMJ open, Jan 30, 2014

Research in medical education has increased in volume over the past decades but concerns have bee... more Research in medical education has increased in volume over the past decades but concerns have been raised regarding the quality of trials conducted within this field. Randomised controlled trials (RCTs) involving educational interventions that are reported in biomedical journals have been criticised for their insufficient conceptual, theoretical framework. RCTs published in journals dedicated to medical education, on the other hand, have been questioned regarding their methodological rigour. The aim of this study is therefore to assess the quality of RCTs of educational interventions reported in 2012 and 2013 in journals dedicated to medical education compared to biomedical journals with respect to objective quality criteria. RCTs published between 1 January 2012 and 31 December 2013 in English are included. The search strategy is developed with the help of experienced librarians to search online databases for key terms. All of the identified RCTs are screened based on their titles ...

Simulation in healthcare : journal of the Society for Simulation in Healthcare, 2012

Educators often simplify complex tasks by setting learning objectives that focus trainees on isol... more Educators often simplify complex tasks by setting learning objectives that focus trainees on isolated skills rather than the holistic task. We designed 2 sets of learning objectives for intravenous catheterization using goal setting theory. We hypothesized that setting holistic goals related to technical, cognitive, and communication skills would result in superior holistic performance, whereas setting isolated goals related to technical skills would result in superior technical performance. We randomly assigned practicing health care professionals to set holistic (n = 14) or isolated (n = 15) goals. All watched an instructional video and studied a list of 9 goals specific to their group. Participants practiced independently in a hybrid simulation (standardized patient combined with an arm simulator). The first and the last practice trials were videotaped for analysis. One-week later, participants completed a transfer test in another hybrid simulation scenario. Blinded experts evalu...

Journal of the American College of Surgeons, 2014

Advances in health sciences education : theory and practice, Jan 22, 2015

In order to construct and evaluate the validity argument for the Objective Structured Assessment ... more In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision)....

Journal of Interprofessional Care, 2015

Evidence suggests that breakdowns in communication and a lack of situation awareness contribute t... more Evidence suggests that breakdowns in communication and a lack of situation awareness contribute to poor performance of medical teams. In this pilot study, three interprofessional obstetrical teams determined the feasibility of using the situation awareness global assessment technique (SAGAT) during simulated critical event management of three obstetrical scenarios. After each scenario, teams were asked to complete questionnaires assessing their opinion of how their performance was affected by the introduction of questions during a SAGAT stop. Fifteen obstetrical professionals took part in the study and completed the three scenarios in teams consisting of five members. At nine questions per stop, more participants agreed or strongly agreed that there were too many questions per stop (57.1%) than when we asked six questions per stop (13%) and three questions per stop (0%). A number of interprofessional differences in response to this interprofessional experience were noted. A team SAGAT score was determined by calculating the proportion of correct responses for each individual. Higher scores were associated with better adherence to outcome times, although not statistically significant. A robust study design building on our pilot data is needed to probe the differing interprofessional perceptions of SAGAT and the potential association between its scores and clinical outcome times.

Studies in health technology and informatics, 2007

Novices' learning of temporal bone drilling on a simple (2D) model was compared to learning o... more Novices' learning of temporal bone drilling on a simple (2D) model was compared to learning of the skill on a complex model (3D). Trainees practicing on 2D model required an additional 3D practice to achieve similar final products as those practicing on a 3D model. This implies that during learning fundamental psychomotor adaptations take place, regardless of the exact skill practiced. However to achieve optimal products the depth of drilling must be practiced separately on high fidelity models.

Journal of Graduate Medical Education, 2012

Cleland/Researching Medical Education, 2015

Surgery, 2013

The costs involved with technology-enhanced simulation remain unknown. Appraising the value of si... more The costs involved with technology-enhanced simulation remain unknown. Appraising the value of simulation-based medical education (SBME) requires complete accounting and reporting of cost. We sought to summarize the quantity and quality of studies that contain an economic analysis of SBME for the training of health professions learners. We performed a systematic search of MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, and previous review bibliographies through May 2011. Articles reporting original research in any language evaluating the cost of simulation, in comparison with nonstimulation instruction or another simulation intervention, for training practicing and student physicians, nurses, and other health professionals were selected. Reviewers working in duplicate evaluated study quality and abstracted information on learners, instructional design, cost elements, and outcomes. From a pool of 10,903 articles we identified 967 comparative studies. Of these, 59 stud...

Simulation in healthcare : journal of the Society for Simulation in Healthcare, 2012

To determine the comparative effectiveness of technology-enhanced simulation, we summarized the r... more To determine the comparative effectiveness of technology-enhanced simulation, we summarized the results of studies comparing technology-enhanced simulation training with nonsimulation instruction for health professions learners. We systematically searched databases including MEDLINE, Embase, and Scopus through May 2011 for relevant articles. Working in duplicate, we abstracted information on instructional design, outcomes, and study quality. From 10,903 candidate articles, we identified 92 eligible studies. In random-effects meta-analysis, pooled effect sizes (positive numbers favoring simulation) were as follows: satisfaction outcomes, 0.59 (95% confidence interval, 0.36-0.81; n = 20 studies); knowledge, 0.30 (0.16-0.43; n = 42); time measure of skills, 0.33 (0.00-0.66; n = 14); process measure of skills, 0.38 (0.24-0.52; n = 51); product measure of skills, 0.66 (0.30-1.02; n = 11); time measure of behavior, 0.56 (-0.07 to 1.18; n = 7); process measure of behavior, 0.77 (-0.13 to 1...

Academic medicine : journal of the Association of American Medical Colleges, Jan 4, 2014

To examine the evidence supporting the use of simulation-based assessments as surrogates for pati... more To examine the evidence supporting the use of simulation-based assessments as surrogates for patient-related outcomes assessed in the workplace. The authors systematically searched MEDLINE, EMBASE, Scopus, and key journals through February 26, 2013. They included original studies that assessed health professionals and trainees using simulation and then linked those scores with patient-related outcomes assessed in the workplace. Two reviewers independently extracted information on participants, tasks, validity evidence, study quality, patient-related and simulation-based outcomes, and magnitude of correlation. All correlations were pooled using random-effects meta-analysis. Of 11,628 potentially relevant articles, the 33 included studies enrolled 1,203 participants, including postgraduate physicians (n = 24 studies), practicing physicians (n = 8), medical students (n = 6), dentists (n = 2), and nurses (n = 1). The pooled correlation for provider behaviors was 0.51 (95% confidence int...

American Journal of Surgery, 2000

Background:We aimed to determine if technical surgical experience enhances learning of new cognit... more Background:We aimed to determine if technical surgical experience enhances learning of new cognitive information, under multitasking conditions.

Surgical Endoscopy and Other Interventional Techniques, 2006

Background Process measures describing the generation of movement are useful for evaluation and ... more Background Process measures describing the generation of movement are useful for evaluation and performance feedback purposes. This study aimed to identify process measures that differ between novice and advanced laparoscopists while completing a suturing skill. Methods A group of junior and a group of senior residents and fellows in surgery (n = 6) placed 10 laparoscopic sutures in a synthetic model. Process

Journal of Graduate Medical Education, 2014

Advances in health sciences education : theory and practice, Jan 15, 2015

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2015

Instructor-led simulation-based mastery learning of advanced cardiac life support (ACLS) skills i... more Instructor-led simulation-based mastery learning of advanced cardiac life support (ACLS) skills is an effective and focused approach to competency-based education. Directed self-regulated learning (DSRL) may be an effective and less resource-intensive way to teach ACLS skills. Forty first-year internal medicine residents were randomized to either simulation-based DSRL or simulation-based instructor-regulated learning (IRL) of ACLS skills using a mastery learning model. Residents in each intervention completed pretest, posttest, and retention test of their performance in leading an ACLS response to a simulated scenario. Performance tests were assessed using a standardized checklist. Residents in the DSRL intervention were provided assessment instruments, a debriefing guide, and scenario-specific teaching points, and they were permitted to access relevant online resources. Residents in the IRL intervention had access to the same materials; however, the teaching and debriefing were instructor led. Skills of both the IRL and DSRL interventions showed significant improvement after the intervention, with an average improvement on the posttest of 21.7%. After controlling for pretest score, there was no difference between intervention arms on the posttest [F(1,37) = 0.02, P = 0.94] and retention tests [F(1,17) = 1.43, P = 0.25]. Cost savings were realized in the DSRL intervention after the fourth group (16 residents) had completed each intervention, with an ongoing savings of $80 per resident. Using a simulation-based mastery learning model, we observed equivalence in learning of ACLS skills for the DSRL and IRL conditions, whereas DSRL was more cost effective.

Medical Education, 2015

Assessment is central to medical education and the validation of assessments is vital to their us... more Assessment is central to medical education and the validation of assessments is vital to their use. Earlier validity frameworks suffer from a multiplicity of types of validity or failure to prioritise among sources of validity evidence. Kane's framework addresses both concerns by emphasising key inferences as the assessment progresses from a single observation to a final decision. Evidence evaluating these inferences is planned and presented as a validity argument. We aim to offer a practical introduction to the key concepts of Kane's framework that educators will find accessible and applicable to a wide range of assessment tools and activities. All assessments are ultimately intended to facilitate a defensible decision about the person being assessed. Validation is the process of collecting and interpreting evidence to support that decision. Rigorous validation involves articulating the claims and assumptions associated with the proposed decision (the interpretation/use argument), empirically testing these assumptions, and organising evidence into a coherent validity argument. Kane identifies four inferences in the validity argument: Scoring (translating an observation into one or more scores); Generalisation (using the score[s] as a reflection of performance in a test setting); Extrapolation (using the score[s] as a reflection of real-world performance), and Implications (applying the score[s] to inform a decision or action). Evidence should be collected to support each of these inferences and should focus on the most questionable assumptions in the chain of inference. Key assumptions (and needed evidence) vary depending on the assessment's intended use or associated decision. Kane's framework applies to quantitative and qualitative assessments, and to individual tests and programmes of assessment. Validation focuses on evaluating the key claims, assumptions and inferences that link assessment scores with their intended interpretations and uses. The Implications and associated decisions are the most important inferences in the validity argument.

BMJ open, Jan 30, 2014

Research in medical education has increased in volume over the past decades but concerns have bee... more Research in medical education has increased in volume over the past decades but concerns have been raised regarding the quality of trials conducted within this field. Randomised controlled trials (RCTs) involving educational interventions that are reported in biomedical journals have been criticised for their insufficient conceptual, theoretical framework. RCTs published in journals dedicated to medical education, on the other hand, have been questioned regarding their methodological rigour. The aim of this study is therefore to assess the quality of RCTs of educational interventions reported in 2012 and 2013 in journals dedicated to medical education compared to biomedical journals with respect to objective quality criteria. RCTs published between 1 January 2012 and 31 December 2013 in English are included. The search strategy is developed with the help of experienced librarians to search online databases for key terms. All of the identified RCTs are screened based on their titles ...

Simulation in healthcare : journal of the Society for Simulation in Healthcare, 2012

Educators often simplify complex tasks by setting learning objectives that focus trainees on isol... more Educators often simplify complex tasks by setting learning objectives that focus trainees on isolated skills rather than the holistic task. We designed 2 sets of learning objectives for intravenous catheterization using goal setting theory. We hypothesized that setting holistic goals related to technical, cognitive, and communication skills would result in superior holistic performance, whereas setting isolated goals related to technical skills would result in superior technical performance. We randomly assigned practicing health care professionals to set holistic (n = 14) or isolated (n = 15) goals. All watched an instructional video and studied a list of 9 goals specific to their group. Participants practiced independently in a hybrid simulation (standardized patient combined with an arm simulator). The first and the last practice trials were videotaped for analysis. One-week later, participants completed a transfer test in another hybrid simulation scenario. Blinded experts evalu...

Journal of the American College of Surgeons, 2014

Advances in health sciences education : theory and practice, Jan 22, 2015

In order to construct and evaluate the validity argument for the Objective Structured Assessment ... more In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision)....

Journal of Interprofessional Care, 2015

Evidence suggests that breakdowns in communication and a lack of situation awareness contribute t... more Evidence suggests that breakdowns in communication and a lack of situation awareness contribute to poor performance of medical teams. In this pilot study, three interprofessional obstetrical teams determined the feasibility of using the situation awareness global assessment technique (SAGAT) during simulated critical event management of three obstetrical scenarios. After each scenario, teams were asked to complete questionnaires assessing their opinion of how their performance was affected by the introduction of questions during a SAGAT stop. Fifteen obstetrical professionals took part in the study and completed the three scenarios in teams consisting of five members. At nine questions per stop, more participants agreed or strongly agreed that there were too many questions per stop (57.1%) than when we asked six questions per stop (13%) and three questions per stop (0%). A number of interprofessional differences in response to this interprofessional experience were noted. A team SAGAT score was determined by calculating the proportion of correct responses for each individual. Higher scores were associated with better adherence to outcome times, although not statistically significant. A robust study design building on our pilot data is needed to probe the differing interprofessional perceptions of SAGAT and the potential association between its scores and clinical outcome times.

Studies in health technology and informatics, 2007

Novices' learning of temporal bone drilling on a simple (2D) model was compared to learning o... more Novices' learning of temporal bone drilling on a simple (2D) model was compared to learning of the skill on a complex model (3D). Trainees practicing on 2D model required an additional 3D practice to achieve similar final products as those practicing on a 3D model. This implies that during learning fundamental psychomotor adaptations take place, regardless of the exact skill practiced. However to achieve optimal products the depth of drilling must be practiced separately on high fidelity models.

Journal of Graduate Medical Education, 2012

Cleland/Researching Medical Education, 2015

Surgery, 2013

The costs involved with technology-enhanced simulation remain unknown. Appraising the value of si... more The costs involved with technology-enhanced simulation remain unknown. Appraising the value of simulation-based medical education (SBME) requires complete accounting and reporting of cost. We sought to summarize the quantity and quality of studies that contain an economic analysis of SBME for the training of health professions learners. We performed a systematic search of MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, and previous review bibliographies through May 2011. Articles reporting original research in any language evaluating the cost of simulation, in comparison with nonstimulation instruction or another simulation intervention, for training practicing and student physicians, nurses, and other health professionals were selected. Reviewers working in duplicate evaluated study quality and abstracted information on learners, instructional design, cost elements, and outcomes. From a pool of 10,903 articles we identified 967 comparative studies. Of these, 59 stud...

Simulation in healthcare : journal of the Society for Simulation in Healthcare, 2012

To determine the comparative effectiveness of technology-enhanced simulation, we summarized the r... more To determine the comparative effectiveness of technology-enhanced simulation, we summarized the results of studies comparing technology-enhanced simulation training with nonsimulation instruction for health professions learners. We systematically searched databases including MEDLINE, Embase, and Scopus through May 2011 for relevant articles. Working in duplicate, we abstracted information on instructional design, outcomes, and study quality. From 10,903 candidate articles, we identified 92 eligible studies. In random-effects meta-analysis, pooled effect sizes (positive numbers favoring simulation) were as follows: satisfaction outcomes, 0.59 (95% confidence interval, 0.36-0.81; n = 20 studies); knowledge, 0.30 (0.16-0.43; n = 42); time measure of skills, 0.33 (0.00-0.66; n = 14); process measure of skills, 0.38 (0.24-0.52; n = 51); product measure of skills, 0.66 (0.30-1.02; n = 11); time measure of behavior, 0.56 (-0.07 to 1.18; n = 7); process measure of behavior, 0.77 (-0.13 to 1...

Academic medicine : journal of the Association of American Medical Colleges, Jan 4, 2014

To examine the evidence supporting the use of simulation-based assessments as surrogates for pati... more To examine the evidence supporting the use of simulation-based assessments as surrogates for patient-related outcomes assessed in the workplace. The authors systematically searched MEDLINE, EMBASE, Scopus, and key journals through February 26, 2013. They included original studies that assessed health professionals and trainees using simulation and then linked those scores with patient-related outcomes assessed in the workplace. Two reviewers independently extracted information on participants, tasks, validity evidence, study quality, patient-related and simulation-based outcomes, and magnitude of correlation. All correlations were pooled using random-effects meta-analysis. Of 11,628 potentially relevant articles, the 33 included studies enrolled 1,203 participants, including postgraduate physicians (n = 24 studies), practicing physicians (n = 8), medical students (n = 6), dentists (n = 2), and nurses (n = 1). The pooled correlation for provider behaviors was 0.51 (95% confidence int...

American Journal of Surgery, 2000

Background:We aimed to determine if technical surgical experience enhances learning of new cognit... more Background:We aimed to determine if technical surgical experience enhances learning of new cognitive information, under multitasking conditions.

Surgical Endoscopy and Other Interventional Techniques, 2006

Background Process measures describing the generation of movement are useful for evaluation and ... more Background Process measures describing the generation of movement are useful for evaluation and performance feedback purposes. This study aimed to identify process measures that differ between novice and advanced laparoscopists while completing a suturing skill. Methods A group of junior and a group of senior residents and fellows in surgery (n = 6) placed 10 laparoscopic sutures in a synthetic model. Process