Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes (original) (raw)

ABSTRACT

OBJECTIVE

Valid teamwork assessment is imperative to determine physician competency and optimize patient outcomes. We systematically reviewed published instruments assessing teamwork in undergraduate, graduate, and continuing medical education in general internal medicine and all medical subspecialties.

DATA SOURCES

We searched MEDLINE, MEDLINE In-process, CINAHL and PsycINFO from January 1979 through October 2012, references of included articles, and abstracts from four professional meetings. Two content experts were queried for additional studies.

STUDY ELIGIBILITY

Included studies described quantitative tools measuring teamwork among medical students, residents, fellows, and practicing physicians on single or multi-professional (interprofessional) teams.

STUDY APPRAISAL AND SYNTHESIS METHODS

Instrument validity and study quality were extracted using established frameworks with existing validity evidence. Two authors independently abstracted 30 % of articles and agreement was calculated.

RESULTS

Of 12,922 citations, 178 articles describing 73 unique teamwork assessment tools met inclusion criteria. Interrater agreement was intraclass correlation coefficient 0.73 (95 % CI 0.63–0.81). Studies involved practicing physicians (142, 80 %), residents/fellows (70, 39 %), and medical students (11, 6 %). The majority (152, 85 %) assessed interprofessional teams. Studies were conducted in inpatient (77, 43 %), outpatient (42, 24 %), simulation (37, 21 %), and classroom (13, 7 %) settings. Validity evidence for the 73 tools included content (54, 74 %), internal structure (51, 70 %), relationships to other variables (25, 34 %), and response process (12, 16 %). Attitudes and opinions were the most frequently assessed outcomes. Relationships between teamwork scores and patient outcomes were directly examined for 13 (18 %) of tools. Scores from the Safety Attitudes Questionnaire and Team Climate Inventory have substantial validity evidence and have been associated with improved patient outcomes.

LIMITATIONS

Review is limited to quantitative assessments of teamwork in internal medicine.

CONCLUSIONS

There is strong validity evidence for several published tools assessing teamwork in internal medicine. However, few teamwork assessments have been directly linked to patient outcomes.

KEY WORDS: teamwork, systematic review, medical education, interprofessional teams

Effective teamwork among health professionals improves patient safety.1,2 A substantial proportion of preventable errors in United States hospitals are attributable to teamwork and communication failures.3,4 Recognizing this, the Institute of Medicine, the Joint Commission, the Agency for Healthcare Research and Quality (AHRQ), and others have made teamwork a top priority in their recommendations for improving healthcare.5–9

Teamwork is also prominently positioned within the American Board of Internal Medicine (ABIM) requirements for maintenance of certification for internists,10 as well as the Accreditation Council for Graduate Medical Education’s core competencies,11 milestones,12 and medical student competencies.13 As such, every physician at the undergraduate, graduate, and continuing professional level must demonstrate competency in teamwork.

While there is broad agreement on the imperative to improve teamwork, there is little consensus regarding how to measure it. Internal medicine teams vary substantially in composition, setting, function and charge. The knowledge, skills, and attitudes required for optimal teamwork within an inpatient medical team may differ from those necessary for successful interprofessional collaboration among undergraduate students in a classroom.13,14 Additionally, there are numerous purposes for teamwork assessment, including determining individual physician competence as well as measuring the effectiveness of teams as a whole.15

Given the heterogeneity of healthcare teams within internal medicine, it is logical that no single teamwork measurement tool will suit all clinical and educational situations. Yet, any endeavor to measure teamwork is likely to be most successful if it is grounded in the literature, built upon prior work, reliable and valid.15 Prior reviews have examined teamwork training and interventions, as well as the outcomes of effective teams.1,16–24 These reviews have advanced the understanding of ‘what works’ to improve teamwork (i.e. curricula and interventions), but they do not fully answer the critical question of how teamwork is best measured in healthcare.

Therefore, the objective of this systematic review is to provide a synthesis of published instruments that have been used to assess teamwork in internal medicine. Given the breadth and marked heterogeneity of literature on teamwork assessment within healthcare as a whole, this review was limited to a synthesis of teamwork tools used in internal medicine. It encompasses all instruments used in undergraduate, graduate, and continuing medical education in general internal medicine and internal medicine subspecialties. To capture all published validity evidence for each tool, we also included articles from non-internal medicine specialties that reported additional validity evidence. This paper is intended to serve as a resource to help educators, clinicians, and other health professionals identify appropriate teamwork measurement tools to apply to their own internal medicine settings and teams.

METHODS

Although there are no standard reporting guidelines specific to systematic reviews of assessment tools, this review is reported according to applicable sections of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) standards25 and similar reviews of assessment tools in medical education.26

Data Sources and Search Strategy

We searched MEDLINE, MEDLINE In-process, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and PsycINFO for English language studies from January 1, 1979 through October 31, 2012. To identify studies related to teamwork, the term team was exploded to include all Medical Subject Headings (MeSH) and keywords containing “team” (e.g. team, teamwork, teamworking, team behavior, team climate, team culture, team collaboration, team effectiveness). Other MeSH terms and keywords related to teamwork included interprofessional relations, patient care team, cooperative behavior, crew resource, crisis resource and non-technical skills. These terms were combined with measurement terms including: scale, measure, inventory, questionnaire, tool, instrument, assessment, evaluation, profile, indicator, index and survey. Last, terms for teamwork and measurement were combined with terms pertaining to medical education and health professionals, including MeSH students, health occupations, health personnel, education professional, internship and residency, healthcare facilities, manpower and services, and exploded terms doctor, physician, nurse, student, intern, resident, registrar, house officer, medical, surgeon, operating, health, clinic, patient, interdiscliplinary, multidiscliplinary and interprofessional. An expert librarian with experience conducting literature searches for systematic reviews assisted in development and implementation of the search. The exact search strategies for each database are available from the authors.

The reference lists of all included articles were reviewed for additional studies. To identify in-press and unpublished studies, we searched scientific abstracts from national meetings between 2010 and 2012 of the Association of American Medical Colleges (AAMC), Association of Medical Educators of Europe (AMEE), Society of General Internal Medicine (SGIM), and the International Meeting on Simulation in Healthcare (IMSH). Authors of relevant abstracts were contacted for unpublished manuscripts. Finally, two experts who have published prior systematic reviews of teamwork or empiric studies of teamwork assessment reviewed the list of included articles to identify additional studies.

Study Selection

Articles were included if they were original research describing a quantitative tool designed for measuring teamwork within healthcare teams involving physicians and/or trainees in general internal medicine or an internal medicine subspecialty. To provide a comprehensive synthesis, we also included articles from non-internal medicine specialties that reported validity evidence for included tools. Studies of interprofessional teams (defined as two or more professions working together as a team)27,28 were included as long as internal medicine physicians (or medical students, residents, fellows) were one of the professions studied. Tools were considered measures of teamwork, based on authors’ descriptions of tools as measuring teamwork, collaboration, team process or function, team behavior, team effectiveness, team climate/environment, team culture, non-technical skills, or crew/crisis management. We excluded studies that measured just one specific aspect of team function, such as conflict, negotiation, leadership, communication, disruptive behavior and harassment. Studies of patient hand-over were excluded since recent reviews on this topic have been published.29–31

Title and Abstract Review

The search yielded 12,922 citations (Fig. 1). Each title/abstract was reviewed and we erred on the side of full article retrieval if titles/abstracts were insufficient to determine eligibility. A total of 892 articles was included for full article review. All uncertainties regarding inclusion were resolved by consensus.

Figure 1.

Figure 1

Article search and selection.

Data Extraction

Data were entered into a structured extraction form that included information on articles (study location, design, participants, setting) and tool characteristics (content, validity, and outcomes). Five authors extracted data. These authors met weekly during the study period and uncertainties were resolved by consensus. Thirty percent of articles were independently extracted by two authors to verify consistency in coding and determine interrater agreement using an intra-class correlation coefficient (ICC). The remaining 70 % of articles were extracted by a single reviewer.

We used an established framework to categorize the validity of instruments32–34 that has been used in similar evaluations of assessment tools.26 This framework includes five categories of validity evidence: 1) content (the degree to which the tool content reflects the construct being measured); 2) response process (training of raters to use the tool); 3) internal structure (instrument reliability including internal consistency, interrater, intrarater, and test-retest reliability); 4) relationships to other variables (relationship between scores and other variables measuring the same construct); and 5) consequences (outcomes associated with tool scores). Kirkpatrick’s hierarchy was used to categorize outcomes as satisfaction/opinion, knowledge and skills, behaviors, and patient outcomes.35 Patient measures were recorded as outcomes only if the study reported a direct quantitative association between the teamwork assessment score and the patient outcome.

To evaluate the methodological quality of studies, we used criteria from the ten-item Medical Education Research Study Quality Instrument (MERSQI),36 which encompasses basic methodological components (e.g. study design, sampling, analysis). Validity evidence for the MERSQI includes content, interrater, intrarater, and internal consistency reliability, and relationships to other variables, including correlations between instrument items and journal impact factor, 3-year citation rate, and journal editors’ quality ratings,36 as well as predictive validity based on associations with editors’ decisions to accept or reject manuscripts for publication.37 We tallied the number of studies that fully, partially or failed to satisfy each of the ten quality criteria in the MERSQI.

Data Synthesis

Characteristics of studies and teamwork measurement tools were synthesized qualitatively and reported in evidence tables. Articles describing identical tools were grouped to enable examination and presentation of all validity evidence and outcomes for each unique tool. Frequencies and percentages were used to describe study and tool characteristics. Means and standard deviations were used to summarize quality scores. Meta-analysis was not possible nor logical, given that this was a review of assessment tools with obvious heterogeneity among instruments, study designs, and outcomes.

RESULTS

Of the 12, 922 citations, 12,629 were identified through the electronic database searches, 16 from reference lists of included articles, two from expert review, and 275 from relevant meeting abstracts. We identified 98 articles from non-internal medicine specialties that contained validity evidence for included tools. The total number of articles meeting inclusion criteria was 178 (Fig. 1). Interrater agreement for data extraction was very good (ICC = 0.73, 95 % CI: 0.63–0.81).

Table 1 shows the characteristics of the 178 included studies. Approximately half of studies were conducted in the U.S. and one-third in Europe. Most (142, 80 %) of studies included practicing physicians as participants, followed by residents (68, 38 %) and medical students (11, 6 %). The majority (152, 85 %) of studies also assessed non-physician professionals (e.g. nurses, pharmacists, mid-level providers, therapists, social workers, administrators) in interprofessional teams. Although most studies took place in actual inpatient or outpatient practice settings, 37 (21 %) of studies were simulation-based and 13 (7 %) took place in classrooms.

Table 1.

Characteristics of 178 Studies Describing 73 Tools for Measuring Teamwork

Characteristics No. (%)
Location*
United States 101 (57)
Europe 58 (33)
Canada 14 (8)
Australia 9 (5)
Asia 1 (1)
Other† 8 (4)
Single/multi-institution
Single institution 82 (46)
Multi-institution 96 (54)
Setting*
Inpatient 77 (43)
Outpatient 42 (24)
Operating room 23 (13)
Emergency room 2 (1)
Simulation 37 (21)
Classroom 13 (7)
Other 5 (3)
Level of participants*
Medical students 11 (6)
Residents 68 (38)
Fellows 10 (6)
Practicing physicians 142 (80)
Interprofessional team‡
Single profession 26 (15)
Interprofessional team 152 (85)
Study design
Single group cross sectional 158 (89)
Single group post-test only 0 (0)
Single group pre-post test 0 (0)
Prospective cohort 10 (6)
Nonrandomized two-group experiment 6 (3)
Randomized two-group experiment 10 (6)

Study Quality

Figure 2 shows the proportion of studies satisfying the ten MERSQI quality criteria. Ten (6 %) studies fully satisfied, 59 (33 %) partially satisfied, and 109 (61 %) did not satisfy quality criteria for study design. The most frequent study design was single group cross-sectional (89 %). Ten studies were randomized controlled experiments.38–47 A majority (153, 86 %) of studies fully satisfied at least one validity criterion: 122 (69 %) studies reported content validity, 115 (65 %) reported internal structure, and 47 (26 %) described relationships to other variables. Twenty-nine (16 %) studies fully satisfied all three of these validity criteria. Most studies (140, 79 %) relied on subjective assessments by study participants for measuring teamwork.

Figure 2.

Figure 2

Methodological quality of 178 studies describing teamwork assessment tools.

Description and Validity Evidence for Teamwork Assessment Tools

The 178 included articles described 73 unique tools designed to measure teamwork (Table 2). Of the 73 tools, 15 (21 %) measured the teamwork of individuals working within teams, 43 (60 %) measured the teamwork of teams as a whole, and 15 (21 %) assessed both individuals and teams.

Table 2.

Characteristics and Validity Evidence for 73 Teamwork Measurement Tools

Tool* Setting Specialty Non-Physician Professions Data Assesses Individual or Team Content Validity Response Process Internal Structure Relation- ships to Other Variables Outcomes
Tools used with Practicing Physicians, Residents/Fellows, and Medical Students
Anesthetists’ Non-Technical Skills 41 – 44 , 47 – 57 Simulation NS; OR Critical care, Anesthesia, Pediatrics, General surgery Nurse, Technician Objective Both Yes Yes Alpha 0.79–0.87 42 , 49 , 52 , 55; Interrater reliability 0.88–9253 , 56 Yes 43 , 48 , 50 Satisfaction/Opinion 48 , 56; Skill41 – 44 , 49 – 51 , 53 , 56 , 57
Attitudes Toward Health Care Teams Scale58 – 67 Classroom; Inpatient; Outpatient General medicine,Pain medicine, Geriatrics Multi-disciplinary students†; Nurse, Pharmacist, Social work, Other‡; Subjective Team Yes No Factor analysis58 , 62 , 66Alpha 0.61–0.8 58 , 60 , 62 , 66 , 61 , 65 Test-retest reliability 0.7165 Yes65 Satisfaction/Opinion
Jefferson Scale of Attitudes toward Physician-Nurse Collaboration68 – 79 Classroom; Simulation NS; Inpatient; Outpatient; OR Anesthesia, General surgery, Family medicine, General medicine Nurse, Midlevel provider§, Nursing students Subjective Team Yes No Factor analysis 74 , 76 , 77 Alpha 0.7–0.9 68 , 71,78 , 70 , 73 , 77 , 79 Interrater reliability 0.02–0.6470 Yes71 , 72 , 78 Satisfaction/Opinion
TeamSTEPPS Team Assessment Questionnaire80 – 82 Simulation, technology-assisted, OR, Inpatient, General surgery, Anesthesia, Critical care, Ob/Gyn Midlevel provider§, Nurse, Pharmacist, Technician Subjective Both Yes No Alpha 0.9–0.9881 , 82 No Satisfaction/Opinion; Behavior
Tools used with Practicing Physicians and Residents/Fellows
Collaboration & Satisfaction about Care Decisions49 , 83 – 87 Inpatient; Simulation, technology-assisted, OR AnesthesiaPediatrics, Critical care, General surgery Nurse Subjective Team Yes No Factor Analysis49 , 84, Alpha 0.90–0.9649 , 83 – 86 No Satisfaction/Opinion49 , 83 – 87; Patient Outcome83
Explicit Professional Oral Communication Measurement88 Inpatient Critical care Non-physicians NS Objective Individuals No Yes No No None
Group Environment Scale 89 – 91 Inpatient; Outpatient PM&R, IM specialty Nurse, Therapist, Psychologist, Social work Subjective Team Yes 89 No Alpha 0.84–89 No Satisfaction/Opinion 89 , 90 , 91
ICU Nurse-Physician Questionnaire 92 – 95 Simulation NS; Inpatient; ER Critical care, EM Nurse Subjective Team Yes 92 – 95 Yes 92 Factor analysis 95 Alpha 0.62–0.9 93 , 94 , 95 Yes92 , 93 , 95 Satisfaction/Opinion 92 – 95 Behavior 93 , 95
Intensity of Inter-professional Collaboration questionnaire96 Inpatient IM specialty Nurse Subjective Team No No Alpha 0.81 Yes Satisfaction/Opinion; Patient Outcome
Interdisciplinary Collaboration Questionnaire97 Inpatient Critical care Nurse, Nursing students Subjective Team Yes No Alpha > 0.7 Yes Knowledge
Management Attitudes Questionnaire, Operating Room or ICU98 – 104 Simulation NS; Inpatient; OR General surgery, Surgical specialty, Anesthesia Critical care Nurse, Technician Subjective Individuals, Both 99 , 104 Yes 98 , 100 – 104 No Factor analysis 98 , 104;Alpha 0.54–0.82 99 – 101 , 103 , 104; Intrarater reliability 0.34 103;Test-retest reliability 103 Yes 103 Satisfaction/Opinion 98 – 102
Non-technical Skills for Surgeons40 , 57 , 105 – 109 Simulation, technology-assisted; OR General surgery, Surgical specialty, IM specialty, Anesthesia None Objective Individuals Yes 40 , 105 – 109 Yes 105 , 107 – 109 Factor analysis108; Interrater reliability 0.68–0.94 40 , 107;D-study coefficient 0.82 108 No Satisfaction/Opinion 57 , 106 , 108 , 109 ; Skill 40 , 106 – 108
Organizational Culture Inventory 95 , 110 Inpatient; ER Critical care, EM Nurse, Other‡ Subjective Team Yes 95 No Factor analysis 95 , 110 Yes110 Satisfaction/Opinion 95 , 110
Organizational Management of the ICU Questionnaire111 Inpatient Hospital Medicine Nurse Subjective Team Yes No Alpha 0.90–0.92 No Patient Outcome
Practice Environment Checklist- Short Form112 Outpatient General medicine Nurse, Midlevel provider§, Other‡ Subjective Team Yes No Factor analysis Alpha 0.8 No Validity only
SafeTeam Questionnaire88 Inpatient Critical care Non-physicians NS Subjective Both Yes No No No None
Safety Attitudes Questionnaire45 , 46 , 113 – 137 Classroom; Simulation NS; Inpatient; Outpatient; OR Pediatrics, Anesthesia Ob/Gyn, General surgery, Surgical specialty, General medicine, Critical care Nurse, Midlevel provider§ Technician, Therapist, Pharmacist, Social work, Other‡ Subjective Team Yes 46 , 113 , 114 , 116 , 118 , 122 , 123 , 125 , 127 , 129 – 132 , 134 , 135 Yes115 Factor analysis 46 , 113 , 123 , 127 , 129 , 134;Alpha 0.54–0.9 116 , 119 , 127 , 123 , 129 , 134 , 117 Interrater reliability 0.78–0.86 129 , 123 , 134 Yes127 , 134 , 136 , 123 , 136 Satisfaction/Opinion45 , 46 , 113 , 114 , 117 , 120 – 123 , 125 – 130 , 132 – 137; Behavior91, 97, 106; Skill 131; Patient Outcome116 – 118
Simulation Team Assessment Tool138 Simulation, technology-assisted Critical care, Pediatrics None Objective Team Yes Yes Inter-rater reliability 0.81 No Skill
SPRinT Questionnaire139 Simulation, technology-assisted Anesthesia, Critical care, General surgery, Surgery specialty Nurse, Non-physicians NS Subjective Individuals No No No No Satisfaction/Opinion
Teamwork Behavioral Rater39 Simulation, technology-assisted Critical care Nurse Objective Team Yes No No No Skill
Unnamed140 Inpatient General medicine, IM specialty Nurse Subjective Team No No No No None
Unnamed141 Inpatient IM specialty, Nurse, Therapist, Other‡ Objective Individuals No No Factor analysis Alpha 0.9 No Patient Outcome
Unnamed142 Inpatient IM specialty Nurse, Therapist Subjective Team No No No No None
Unnamed143 Classroom General medicine None Objective Team No No No No Knowledge
Unnamed143 Classroom General medicine None Subjective Team No No No No Satisfaction/Opinion
Unnamed144 Simulation, technology-assisted Critical care Nurse Objective Team Yes Yes Factor analysis Alpha 0.89 No Skill
Unnamed88 Inpatient Critical care Non-physicians NS Subjective Team Yes No No No None
Unnamed145 Outpatient General medicine Nurse, Therapist, Pharmacist, Psychologist, Social work, Optometrist Subjective Team Yes No Factor analysis No Satisfaction/Opinion; Behavior
T ools used with Practicing Physicians Only
Collaboration Skills Simulation Vignette Test92 Simulation, technology-assisted Critical care Nurse Objective Individuals Yes No Interrater reliability 0.82 Yes Skill
Collaboration with Medical Staff Scale146 Inpatient Critical care Nurse Subjective Team Yes No Alpha 0.8 No None
Collaborative Behavior Scale147 Inpatient Critical care Nurse Subjective Team No No Alpha 0.97 No Behavior
Collaborative Practice Scale147 – 152 Inpatient; Outpatient Critical care, Not specified Nurse, Mid-level provider§ Subjective Team No No Factor analysis147; Alpha 0.80–0.83147 , 152 Yes152 Satisfaction/Opinion
Doctors’ Opinions on Collaboration153 Outpatient Multi-specialty‖ None Subjective Individuals Yes No Factor analysis Alpha 0.64–0.9 No Validity only
Group Behavior Inventory154 Inpatient IM specialty, Surgical specialty, Radiology, Pathology Nurse Subjective Team No No No No None
Group Development Questionnaire155 Inpatient Critical care Nurse, Other‡ Subjective Team No No No No Patient Outcome
Healthcare Team Vitality Instrument156 Inpatient Hospital medicine Nurse, Non-physicians NS Subjective Both Yes No Factor analysis Yes Satisfaction/Opinion
Human Factors Attitude Survey157 Classroom General medicine, IM specialty, Pediatrics, General surgery, EM Nurse, Technician, Other‡ Subjective Both No No Alpha 0.8 No Satisfaction/Opinion
Inter-professional Collaborative Learning Series158 Inpatient General medicine Nurse, Therapist, Pharmacist, Non-physicians NS Subjective Both Yes No No No Satisfaction/Opinion
Leader Communication Self-Report of Collaboration92 Inpatient Critical care Nurse Subjective Both Yes No Alpha 0.7–0.9 No Satisfaction/Opinion
Multi-disciplinary Collaboration Instrument159 Inpatient Hospital medicine Nurse, Therapist, Pharmacist, Other‡ Subjective Team Yes No Alpha 0.8 Yes Validity only
Multi-disciplinary Team Performance Assessment Tool160 , 161 Inpatient; Outpatient IM specialty, General surgery, Radiology, Pathology Nurse, Midlevel provider§, Other‡ Objective Both Yes Yes160 Interrater reliability ≥ 0.7161 Yes161 Validity only
Nursing Home Survey on Patient Safety Culture162 Outpatient Geriatrics Midlevel provider, Nurse, Therapist, AdministratorSocial work Subjective Both No No No No Satisfaction/Opinion
Observational Skill-based Clinical Assessment Tool for Resuscitation163 Simulation, technology-assisted General medicine Nurse, Therapist Objective Individuals Yes No Factor analysis Alpha 0.74–0.96Interrater reliability 0.65–0.91 No Validity only
Physician/ Pharmacist Collaboration Index 38 , 164 – 170 Outpatient; Inpatient General medicine, Pediatrics, Family Medicine, Ob/Gyn, Critical care Pharmacist Subjective Team Yes164 , 165 , 169 No Factor analysis 164 , 165;Alpha 0.70–0.90164 , 165 , 168 , 169;Interrater reliability 0.89167 Yes164 , 165 , 167 Satisfaction/Opinion164 – 168; Behavior167 , 169; Patient Outcome38
Practice Climate Scale for Practitioners171 Outpatient IM specialty Midlevel provider§, Nurse, Other‡ Subjective Both No No Alpha 0.90 Yes Patient Outcome
Team Climate Inventory172 – 191 Outpatient; Inpatient Multi-specialty‖ Nurse, Midlevel provider§, Therapist, Pharmacist, Social work, Other‡ Subjective Team Yes No Alpha 0.80–0.90 174 , 177 , 180 , 184 – 186 , 189 , 190 , 192 Interrater reliability 0.60– 0.98175 , 174 , 182 , 189 , 190 , 187 Yes172 , 174 , 176 , 178 , 182 , 186 , 187 , 189 , 190 , 192 Satisfaction/Opinion173 , 174 , 176 – 183 , 186 – 189 , 192; Knowledge 175; Behavior175 , 176 , 189; Patient Outcome172 – 175 , 185 , 187
Team Emergency Assessment Measure193 Inpatient, technology-assisted simulation General medicine, EM Nursing students Objective Both Yes No Factor analysis Alpha 0.89–0.90Interrater reliability 0.53–0.80 No Validity only
Team Functioning Assessment Tool194 Inpatient Hospital medicine Non-physicians NS Objective Individuals Yes Yes Interrater reliability 0.67–0.99 No Validity only
Team Observation Scale 195 , 196 Outpatient Geriatrics, Hospice Nurse, Social work, Other‡ Objective Team Yes 195 Yes 195 No No Skill 195 Behavior 196
TeamSTEPPSTeamwork Attitudes Questionnaire197 Classroom Critical care, EM Nurse Subjective Both Yes No Alpha 0.70 No Validity only
Teamwork Evaluation of Non-technical skills198 Inpatient Pediatrics, General surgery, Critical care Nurse, Therapist Objective Team Yes No Interrater reliability 0.96 Yes Behavior; Patient Outcome
Teamwork Scale199 Inpatient Hospital medicine Non-physicians NS Subjective Team Yes No Alpha 0.80 Yes Satisfaction/Opinion
Trainee Test of Team Dynamics200 Outpatient simulation, SP simulation Geriatrics Midlevel provider§, Pharmacist, Social work Objective Team Yes No Factor analysis No Validity only
Working as a Team201 Outpatient General medicine, IM specialty, Surgical specialty Non-physicians NS Subjective Individuals Yes No Factor analysis Alpha 0.71–0.90 No Behavior
Unnamed165 Outpatient General medicine, IM specialty, Family Medicine, Ob/Gyn Pharmacist Subjective Team Yes No Alpha 0.90 Yes Satisfaction/Opinion
Unnamed202 Inpatient IM specialty Nurse, Therapist Subjective Both Yes No No No Patient Outcome
Unnamed203 Inpatient General medicine, General surgery Nurse, Other‡ Subjective Team Yes No No No Satisfaction/Opinion
Unnamed204 Simulation, technology-assisted General medicine, General surgery Midlevel provider§, Nurse, Technician Subjective Individuals Yes No Factor analysisAlpha 0.94 No Satisfaction/Opinion
Unnamed205 Outpatient General medicine Midlevel provider§, Nurse, Other‡ Subjective Team No No Alpha 0.73–0.80Interrater reliability > 0.4 No Patient Outcome
Unnamed206 Inpatient General medicine, General surgery, Critical care, Anesthesia Nurse, Other‡ Subjective Team Yes No Factor analysis No Satisfaction/Opinion
Unnamed207 Inpatient General medicine Nurse, Pharmacist Objective Both No No No No Satisfaction/Opinion; Patient Outcome
Unnamed208 Inpatient General medicine, General surgery, Critical care Nurse, Pharmacist, Clerical Subjective Team Yes No No No Satisfaction/Opinion
Tools used with Residents/Fellows and Medical Students
Team Skills Scale58 , 209 Classroom General medicine58 Students of Nursing, Pharmacy, Social work, Other‡ Subjective Team Yes No Factor analysis58; Alpha 0.9 58 No Skill58 , 209
Tools used with Residents/Fellows Only
Communication, Collaboration and Critical Thinking for Quality Patient Outcomes Survey79 Inpatient General medicine Nurse Subjective Team No No Alpha 0.9 No Satisfaction/Opinion
Decision About Transfer210 Inpatient Critical care Nurse Subjective Team Yes No No Yes Patient Outcome
Geriatric Interdisciplinary Team Training Videotape Test of Team Dynamics58 Classroom General medicine Students of Nursing, Social work Objective Team Yes No No No Knowledge
Ottawa Crisis Resource Management Checklist211 Simulation NS Multi- specialty▪ Nurse Objective Individuals Yes Yes Interrater reliability 0.63 Yes Skill
Ottawa Global Rating Scale 48 , 211 , 212 Simulation NS Multi-specialty¶ None Objective Individuals Yes 48 , 211 Yes 211 , 212 Interrater reliability 0.49–0.8548 , 211 , 212 Yes 211 Satisfaction/Opinion 48; Skill 211 , 212
Resident Leadership Scale213 Inpatient General medicine None Objective Individuals Yes No Factor analysis Alpha 0.90 Yes Skill
Team Diagnostic Survey214 Inpatient Critical care Nurse Subjective Team Yes No Factor analysis Alpha 0.72–0.80 No Satisfaction/Opinion
Unnamed214 Inpatient Critical care Nurse Subjective Team Yes No Factor analysis Alpha 0.77–0.80 Yes Satisfaction/Opinion
Unnamed114 Inpatient General medicine Nurse Subjective Individuals Yes No No No Satisfaction/Opinion
Tools used with Medical Students Only
Situation Awareness Global Assessment Technique215 Simulation, Technology-assisted Critical care None Objective Individuals No No No No Knowledge

Content validity was demonstrated for 54 (74 %) of tools (Table 2) and generally consisted of developing instrument content from expert panels, existing instruments, and literature review. The TeamSTEPPS Teamwork Attitudes Questionnaire80 is an example of an assessment tool with strong content validity designed to assess the teamwork attitudes, knowledge and skills of learners participating in the TeamSTEPPS curriculum. TeamSTEPPS is a training program developed by the United States Department of Defense and the AHRQ that encompasses leadership, situation monitoring, mutual support and communication.7,80,198 The TeamSTEPPS Teamwork Perception Questionnaire is a second instrument associated with this curriculum that measures individuals’ perceptions of organizational teamwork.216

Few tools (12, 16 %) reported response process, which included training raters to correctly use tools. The Multi-disciplinary Team Performance Assessment Tool160,161 is an observational teamwork assessment of cancer teams modified from an established teamwork assessment tool in the surgical literature (Observational Teamwork Assessment in Surgery).217–219 Assessors were trained in the use of this tool by an expert psychologist with experience using the tool.160

Reliability of tools was demonstrated by internal consistency (38, 52 %), interrater reliability (16, 22 %), intrarater reliability (1, 1 %), and test-retest reliability (2, 3 %). Reliability estimates for most tools were very good (> 0.7).220 The Physician/Pharmacist Collaboration Index is an example of a tool assessing interactions between internists and pharmacists that has extensive reliability evidence, including factor analysis, internal consistency (Crohnbach alpha 0.70-0.90) and interrater reliability (ICC 0.89).38,164–170 This tool measures the pharmacist’s view of collaboration among physicians and other health professionals in both inpatient and outpatient settings.

Relationships between teamwork scores and other variables reflecting the construct of teamwork were reported for 25 (34 %) tools (Table 2). Studies varied widely with regard to the specific variables reported. The Attitudes Toward Health Care Teams Scale (ATHCTS) has been used in ten studies measuring attitudes towards interprofessional collaboration in a variety of settings, most commonly interprofessional education.58–67 It consists of three subscales assessing attitudes about team value, team efficiency and the physician’s shared role on the team. ATHCTS scores have been shown to correlate with other measures of team process.65 The Ottawa Global Rating Scale has been used in multi-specialty education as an objective measurement of an individual’s crisis resource management skills in simulated scenarios.211,212 This tool has been shown to differentiate among residents’ level of training when applied in simulated medical crisis scenarios.211

Consequences validity refers to the outcomes associated with scores from teamwork tools. For many tools (35, 48 %), outcomes included satisfaction or opinion of participants (Table 2). Twelve (16 %) tools measured participants’ teamwork skills. Teamwork skills such as leadership, communication and crisis management were assessed through simulation;41,138,209 however, other tools involved direct observation of skills in actual practice settings, such as medical residents’ abilities to lead ward teams213 and palliative care physicians’ communication in team meetings.195 Behaviors of students, residents/fellows, or practicing physicians were reported outcomes for ten (14 %) tools.

Teamwork Tools Associated with Patient Outcomes

Relationships between teamwork scores and patient outcomes have been directly examined for 13 (18 %) of teamwork tools (Table 3). Teamwork tools by Baggs83,210 and Wheelan155 show inverse relationships between positive teamwork and mortality rates.

Table 3.

Relationships Between Scores from Teamwork Measurement Tools (n = 13) and Patient Outcomes

Tool* Participants Patient Outcomes
Collaboration & Satisfaction about Care Decisions83 Residents, Practicing physicians, Nurse Nurses’ ratings of collaboration negatively associated with patient death or readmission to the Intensive Care Unit (p = 0.037).
Decision About Transfer210 Residents, Nurse Nurse ratings of collaboration negatively associated with patient death or readmission to the Intensive Care Unit (p = 0.02). Resident-reported collaboration was not associated with death or readmission.
Group Development Questionnaire155 Practicing physicians, Nurse, Other† Group development (teamwork) inversely associated with mortality rate (r = −0.662, p = 0.004).
Intensity of Interprofessional Collaboration Questionnaire96 Residents, Practicing physicians, Nurse Patients cared for by teams with high intensity collaboration had higher mean patient satisfaction (0.501, 95 % CI 0.286-0.715) and lower mean uncertainty scores (0.138, 95 % CI 0.002–0.275) compared to teams with low intensity collaboration. High intensity collaboration teams demonstrated better pain management (92.6 %, 95 % CI 87.9–97.3 %) compared to low intensity collaboration teams (82.7 %, 95 % CI 76.3–89.2 %). There were no significant differences in length of hospital stay among teams with high versus low intensity collaboration
Organizational Management of the ICU Questionnaire111 Residents, Practicing physicians, Nurse Perceptions of increased nurse–physician collaboration were associated with increased length of stay (p < 0.001).
Physician/ Pharmacist Collaboration Index38 Practicing physicians, Pharmacist Teamwork scores inversely correlated with blood pressure (Spearman = −0.153).
Practice Climate Scale for Practitioners171 Practicing physicians, Midlevel provider‡, Nurse, Other† Positive perceptions of teamwork practice climate were positively associated with patient ratings of trust in primary care physicians (p = 0.04).
Safety Attitudes Questionnaire (SAQ)116 – 118 Residents, Fellows, Practicing physicians, Nurse, Non-physicians NS SAQ teamwork and communication scores were inversely correlated with risk-adjusted morbidity (defined as patient having one or more of 21 specific postoperative complications up to 30 days after surgery), r = −0.38, p < 0.0.116 Improvement in SAQ score correlated with lower postoperative complication rate (r = 0.7143, p = 0.0381).117 No association between SAQ scores and hospital reported patent safety events.118
Team Climate Inventory 172 – 175 , 185 , 187 Practicing physicians, Midlevel‡,Nurse, Therapist, Pharmacist, Clerical, Non-physicians NS Team Climate scores were positively associated with overall patient satisfaction (regression coefficient 1.35, 95 % CI: 0.43–2.26, p = 0.005) and higher quality diabetes management (2.13, 95 % CI: 0.20–4.05, p = 0.031).172 Team Climate scores were positively associated with patients’ experience of improved patient care (p < 0.02).175 Team Climate scores in the participation subscale were positively associated with patients’ perceptions of continuity [regression coefficient 3.72 (95 % CI 0.56,6.87, p = 0.02)].173 Patient satisfaction increased by 0.042 (95 % CI = 0.047–0.129, p = 0.014) for an increase of one standard deviation in overall Team Climate score.174 No relationships between overall Team Climate score and diabetes care or overall patient satisfaction.174 No relationship between Team Climate score and prevalence of pressure ulcers (OR 1.00, 95 % CI 0.50–2.02).185
Unnamed141 Residents, Practicing physicians, Nurse, Therapist, Other† Physician relational coordination was inversely associated with length of stay (r = −0.46, p < 0.01) and log total costs per stay (r = − 0.08, p = 0.03).
Unnamed207 Practicing physicians, Nurse, Pharmacist Improved teamwork was associated with patients’ ratings of satisfaction with physicians treating them with respect and nurses listening to them (p = 0.001 and 0.0003, respectively)
Unnamed202 Practicing physicians, Nurse, Therapist Team organization and task orientation were positively associated with improvement in motor function among stroke rehabilitation patients (both p < 0.05).Higher team effectiveness was associated with shorter length of stay among stroke rehabilitation patients (p = 0.02)No association between teamwork scores and patient discharge destination.
Unnamed205 Practicing physicians, Midlevel provider‡, Nurse, Other† Patient physical function was higher among patients impanelled to teams with higher teamwork scores (p = 0.05)No association between teamwork scores and patient physical and mental quality of life scores.

Of the tools shown to correlate with patient outcomes, the Safety Attitudes Questionnaire (SAQ)113 has the strongest validity evidence, and has been adapted for use across multiple settings and learner levels. The SAQ contains six domains, one of which is teamwork. Twenty-seven studies have reported validity evidence for the SAQ.45,46,113–137 SAQ scores have been correlated with reduced postoperative complications;116,117 however, studies have not shown associations between the SAQ and mortality or patient safety events.116,118

The Team Climate Inventory (TCI) has been used to assess teamwork among inpatient and outpatient interprofessional teams in 21 studies.172–192 The TCI has four subscales: vision, participative safety, task orientation, and support for innovation.190 A study by Bower et al. found that ratings on the Team Climate Inventory (TCI) were associated with better diabetes care,172 while another study showed no relationship between the TCI and diabetes management.173

The Intensity of Interprofessional Collaboration Questionnaire is an instrument that measures the nurse–physician collaboration in the inpatient setting. Patients cared for by teams with high intensity collaboration on this scale reported higher satisfaction, lower uncertainty, and better pain management.96 However, there was no relationship between collaboration and patient length of stay.96

DISCUSSION

Assessing teamwork is imperative for determining physician competency11,13,221 and ensuring patient safety.3,5 Valid and reliable measurement of teamwork is necessary to understand connections between teamwork and patient safety, and to maximize gains achieved through teamwork education.

Together, the 178 studies and 73 teamwork tools summarized in this review constitute a resource for internists who wish to apply teamwork assessment tools to their local settings and teams. Although there is considerable validity evidence for many of these teamwork tools, most assessments consisted of participants’ subjective reports of satisfaction, attitude, or opinion. A thorough understanding of attitudes is prerequisite to improving teamwork; yet, tools examining teamwork behaviors in actual practice provide scores that may be more readily linked to important patient safety outcomes.83,155,202 Unfortunately, these assessments often require extensive rater training to achieve adequate reliability,105 which can be time consuming and costly. Implementing existing tools, rather than creating new ones, should reduce the cost of tool development so that these resources can be allocated to rater training and implementation. Furthermore, the trustworthiness of validity information depends upon the methodological quality of studies from which it is derived. Based on MERSQI criteria, further studies should aim to improve rigor of study design and outcome assessment.

Evidence suggests that teamwork training should improve patient safety,1,2 yet our review indicates that most studies examining teamwork in internal medicine do not directly link teamwork measures to reported patient outcomes. Several studies in this review described concurrent changes in patient outcomes and teamwork scores (e.g. pre/post teamwork training), but did not actually examine relationships between outcomes and teamwork scores, thus making it difficult to attribute gains in patient safety to teamwork improvements. To advance the understanding of how to improve safety through collaboration, future studies should not only apply valid teamwork assessments, but should directly examine relationships between these assessments and patient outcomes. Robust teamwork assessments and appropriate conceptual frameworks are essential to meaningful evaluations of relationships between teamwork and patient outcomes.

The majority of teamwork tools in this review were applied to groups of individuals working together to achieve a common goal within traditional team structures (e.g. physically side by side/face to face).23,222 However, the concept of ‘team’ in healthcare is rapidly evolving to include a greater emphasis on interprofessional collaboration,223 as well as new team structures. With the advent of restricted duty hours,224 and frequent hand-offs,29,30 team members are often working in shifts225–227 and are also becoming dispersed geographically. The telemedicine intensive care unit is an example in which intensivists and nurses use telemetry and electronic medical records to provide care to patients hospitalized remotely.228 Teams dispersed over distance and/or time face unique teamwork challenges119 that may require new or adapted assessment tools.

There are limitations to this review. First, although our search was comprehensive, we may have failed to capture some non-indexed or unpublished studies. We attempted to limit this possibility by reviewing abstracts from four professional meetings that are likely to include teamwork content, reviewing reference lists of included articles, and by having two content experts examine our reference list. Also, our electronic search included terms such as “registrar” that should have helped capture studies across countries. Second, to make the scope of the review manageable, it was limited to tools published in the field of internal medicine. However, some validity evidence was obtained from studies conducted in other specialties such as surgery and anesthesia. Validity is not a property of an instrument itself; rather it is a property of inferences derived from implementation of the instrument within specific contexts.34 As such, the setting in which tools are applied influences the validity information acquired. When selecting a tool for use in a new setting, it is important to consider the degree to which existing validity evidence may apply to the new context.

Third, this review included only quantitative measurement tools; however, qualitative studies provide valuable frameworks for understanding team behaviors and processes28,229,230 that are essential to the development of meaningful assessment tools. A synthesis of findings from the qualitative literature on teamwork would be a useful next step. Fourth, although we used an extremely broad definition and search strategy for teamwork, we excluded studies that examined just one specific element of interpersonal interaction, such as disruptive behavior and harassment. These behaviors alone do not constitute teamwork; however, they certainly may influence team interactions.231–233 Finally, we used established frameworks for abstracting tool validity32,33 and study quality;36 however, these frameworks do not encompass every aspect of validity and/or quality present in studies.

In conclusion, this systematic review provides a synthesis of teamwork assessment tools in internal medicine that may serve as a resource for educators who wish to assess teamwork for various learner levels and settings. Valid teamwork assessment is essential to determine physician competency and to ensure patient safety. Future research should expand the validity evidence for existing tools and further explore relationships between teamwork assessment and important patient safety outcomes.

Acknowledgements

Contributors

Dr. Paul Haidet of Pennsylvania State Hershey College of Medicine and Dr. Chayan Chakraborti, MD FACP of Tulane University School of Medicine conducted expert reviews of the reference list. Karen Larsen of Mayo Clinic assisted with the electronic searches. Kathy Thompson of Mayo Clinic assisted with article retrieval.

Funding /Support

This study was supported in part by the Mayo Clinic Program in Professionalism and Ethics and the Mayo Clinic-Rochester Internal Medicine Residency Office of Educational Innovations as part of the ACGME Educational Innovations Project.

Role of the Sponsor

The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.

Prior Presentations

This review was presented in an oral session at the Society of General Internal Medicine Annual Meeting in Denver, Colorado in April 2013.

Conflict of Interest

The authors declare that they do not have a conflict of interest.

REFERENCES