Evaluation of a Palliative Care Longitudinal Curriculum for Medical Students Using the Context-Input-Process-Product Model (original) (raw)
Abstract
Palliative care (PC) longitudinal curricula are increasingly being recognized as important in Undergraduate Medical Education (UME). They are however, not yet commonplace, and where they do exist may be implemented without a systematic, prospective approach to curriculum evaluation. This paper describes an implementation of a new longitudinal curriculum at the Perelman School of Medicine (PSOM) at the University of Pennsylvania. We used the Context Input Process Product (CIPP) model, a holistic evaluation model, to assess the local environment, design the curriculum, guide the improvement process, and evaluate outcomes. Comprehensive models such as CIPP provide a more robust approach to curriculum evaluation than outcomes-only models and may be of use to other programs who are implementing new curricula or improving upon existing programs.
Keywords: palliative care, undergraduate medical education, program evaluation, context input process product
Introduction
A palliative care (PC) workforce shortage and a call for all physicians to have primary PC skills highlight the need for longitudinal PC curricula in undergraduate medical education (UME).1-3 Such curricula align with accreditation standards and other expert recommendations and are successful in improving student knowledge, skills and attitudes.4-8 However, U.S. medical school deans, students and researchers report that integrated longitudinal UME curricula in PC are underutilized.9-16 Among existing curricula, few include a structured approach to evaluation of program context, design and process in addition to short-term and long-term outcomes.17-19 Without a comprehensive approach to evaluation from the inception of a program, educators run the risk of missing important information that might explain how and why outcomes were achieved. For example, a local context may have resources in terms of protected curricular time that could help explain why students gained knowledge (vs less notable knowledge gains in a context where students had to study on their own time). This paper describes the use of the Context Input Process Product (CIPP) Model to design, implement and evaluate a new longitudinal curriculum in PC at the Perelman School of Medicine (PSOM). 20 The program is named CARE-7; the CARE acronym stands for “Caring, Attention, Responsiveness and Empathy” while 7 references the 7 overarching goals of the curriculum (Table 1).
Table 1.
CARE-7 Goals and Objectives.
Goals | Objectives | LCME standards |
---|---|---|
1) Caring for patients throughout their disease process through knowledge of specialist palliative and hospice care | a) Defines the philosophy and role of palliative care across the life cycle and differentiates hospice from PC | 7.1 |
b) Describes the rationale behind the inter-disciplinary nature of the PC team and describe the roles of various team-members | 7.2 | |
c) Describes disease trajectories for common serious illnesses | 7.9 | |
2) Effective communication that incorporates the patient’s values, cultural context and goals | d) Explores patient and family understanding of illness, goals, values and cultural beliefs and practices that inform the plan of care | 7.6 |
e) Identifies and responds to emotion in patients and families facing serious illness | 7.8 | |
f) Demonstrates patient-centered communication techniques when giving bad news and discussing medical decisions | ||
3) Consideration of psychosocial and spiritual distress and bereavement in the care of seriously ill patients and their families | g) Identifies grief and psychosocial distress in patients and families | 7.1 |
h) Identifies spiritual and existential suffering in patients and families | 7.6 | |
4) Connecting with one’s own emotional response to patients with serious illness | i) Reflects on personal emotional reactions to encountering patients with serious illness | 7.1 |
7.6 | ||
5) Comprehensive pain management for patients with serious illness | j) Describes an approach to pain management in serious illness including knowledge about opioid initiation and titration | 7.1 |
k) Assesses pain systematically and distinguishes nociceptive from neuropathic pain | 7.2 | |
7.4 | ||
6) Providing comfort through excellent symptom management | l) Describes an approach to the evaluation and management of common non-pain symptoms such as constipation and nausea | 7.2 |
7.4 | ||
7) Care of the patient at the end-of-life | m) Identifies common signs of the dying process and describes treatments for common symptoms at the end of life | 7.2 |
n) Describes tasks prior to and after a patient dies, including rationale behind medical decision making, anticipatory guidance, death pronouncement and family notification | 7.7 | |
7.8 |
Program History
Prior to the implementation of CARE-7, typical PC content at PSOM was scant, often voluntary, and designed by educators who were not always content experts. The structure of oversight for the PSOM curriculum was changed in July 2019 coincident with the development of a new leadership team in Medical Education under the leadership of a new Senior Vice Dean. At the same time, a new paradigm for the development of longitudinal courses was developed beginning with a list of key curricular topics that needed enhancements. A plan for definitive direction for longitudinal curricular development across the entire curriculum with defined goals, objectives and curricular elements was put into place. In the Spring 2021, 1 of the authors (S.R.) began discussions with an interested donor to support the longitudinal development in PC. Upon development and submission of a CARE-7 curriculum proposal (N.B.), this came to successful fruition. In the Fall of 2021, a curriculum director (A.S.) and co-directors (R.M., J.O.) were named, and a curriculum evaluation committee was assembled. Curriculum directors are board-certified hospice and palliative medicine physicians who practice specialty PC. The curriculum evaluation committee is composed of the CARE-7 directors, school of medicine leadership, PC leadership as well as curriculum evaluation experts.
CIPP and the Program Evaluation Model
CIPP is a program evaluation model that evaluates 4 components: characteristics of the environment in which the curriculum is to be created (Context), design of content based on the environmental context (Input), monitoring and refining the curriculum (Process) and finally outcomes assessment (Product). 20 We used the CIPP model because of its broad scope, its adaptability and its attention to design and process. CIPP includes outcomes assessment but avoids a narrow outcomes-only focus. Instead CIPP emphasizes a context-driven design and process that encourages diverse stakeholder involvement. Such features make it well-suited for the evaluation of complex longitudinal curricula where the evaluation model can serve as a guide for curriculum creation and implementation as well as capture of outcomes. Table 2 summarizes the questions posed and tools used in each step of the model. The curriculum evaluation protocol was submitted and deemed exempt by the Penn IRB.
Table 2.
CIPP Model Questions and Tools.
Context | Input | Process | Product | |
---|---|---|---|---|
Questions | What are the educational needs of students in this environment and what should the goals/priorities of intervention be? | What approaches and strategies should we use in our curriculum to meet our context specific goals? | Is the curriculum being implemented as intended? If not, why not? | Is program well-received by students/faculty/others? |
What are the assets and opportunities of the environment? | Are resources adequate to deliver and evaluate the proposed curriculum? | Are system capabilities strained by the curriculum? | Are students gaining knowledge/skills/attitudes laid out in curriculum objectives? | |
What are the problems and barriers that might inhibit meeting the goals? | What else is happening that might facilitate or inhibit implementation? | What else is being accomplished? Are there un-intended consequences? | ||
Do early indicators suggest we are on target to meet our goals? | Why was our curriculum successful or not in the above measures? Does our conceptual framework explain our results? Is there another explanation? | |||
Tools | Faculty and student interviews | Literature review | Faculty interviews | Student yearly focus groups |
Curricular mapping | Interviews with PC curriculum directors at outside institutions | Student evaluations | Surveys to assess lifelong learning indicators | |
Student focus group and survey | Session observations at outside institutions | Student pilot-specific focus groups | Student evaluations | |
Hospital of the university of Pennsylvania’s patient advisory council Co-chair interview | Discussion with leadership team/content experts | Student written reflections | Meetings with PSOM educational leaders | |
Residency director interview | Inventory and analyze available resources | Requests for clinical PC elective/area of concentration | Student written reflections | |
Piloting of interventions | Quiz scores | Requests for clinical PC elective/area of concentration | ||
Student focus group | Activity of PC specialty interest group | |||
PC consult numbers on select surgical floors | ||||
Student medical record notes | ||||
Quiz scores |
Context Evaluation
Context evaluation focuses on assessing the needs, problems, and opportunities of the educational environment, acknowledging that a curriculum’s design, implementation, and evaluation process will depend upon contextual factors. Our context evaluation questions centered on educational needs as well as assets/opportunities and barriers. Here, we use the terms context, environment, and environmental context interchangeably.
Context Evaluation Methods
To answer context evaluation questions, we conducted semi-structured interviews with twenty-one PSOM faculty, 6 students, the internal medicine residency director, and the co-chair of the Hospital of the University of Pennsylvania’s Patient Advisory Council, to map existing PC content in the curriculum and assess stakeholder viewpoints via 5 questions ranging from current state to opportunities for improvement. A student focus group was also conducted with seven fourth-year students for the purpose of eliciting student perceptions about the current state of palliative care education at PSOM, perceived opportunities to improve and opinions about potential new curricular content. These students were given surveys assessing their thoughts on the importance of learning various palliative care topics and their current comfort with those topics (Table 3). A document mapping existing PC content at PSOM was also created.
Table 3.
Student Survey.
How important do you think it is for the following skills to be included in a medical school curriculum? (N = 6) | ||||
---|---|---|---|---|
Mean | Median | Mode | Range | |
Pain/symptom management | 4.71 | 5 | 5 | 4-5 |
Communication with patients/families with serious illness | 4.71 | 5 | 5 | 4-5 |
Knowledge about specialist hospice and PC | 3.43 | 3 | 3 | 2-5 |
Reflecting on personal emotions around patients with serious illness | 4.43 | 5 | 5 | 3-5 |
Caring for patients undergoing the active dying process | 4.29 | 4 | 5 | 3-5 |
How strong do you feel in each of these competencies? (N = 6) | ||||
Mean | Median | Mode | Range | |
Pain/symptom management | 3.71 | 4 | 3 | 3-5 |
Communication with patients/families with serious illness | 4.00 | 4 | 4 | 2-5 |
Knowledge about specialist hospice and PC | 2.71 | 3 | 3 | 1-4 |
Reflecting on personal emotions around patients with serious illness | 3.86 | 4 | 3 | 3-5 |
Caring for patients undergoing the active dying process | 2.71 | 2 | 2 | 1-4 |
Context Evaluation Findings
All PSOM stakeholders agreed that communication skill training should be a priority. While some serious illness communication skills training existed, there was no cohesive educational plan with longitudinal integration. Training in pain and symptom management, knowledge of PC and hospice, and mandatory clinical experiences in hospice and palliative medicine with time to debrief these experiences were also identified as curricular needs.
The student survey showed that students considered symptom management and communication with patients and families to be the most critical competencies associated with PC. Knowledge about hospice and PC was considered less important, but students felt relatively weak in their skills in this domain. Students also reported weakness in their ability to care for dying patients, and this was consistent with student reports of lack of exposure to patients near the end of life. Table 2 summarizes these survey results.
We identified many assets and opportunities in the context evaluation dataset, including donor funding which provided for dedicated time for curriculum directors. PSOM had a large network of PC clinical programs with providers who were also skilled educators, and leadership who were supportive of longitudinal PC education. We also identified contextual barriers, such as a need for administrative support and the planned closure of Penn Hospice’s inpatient unit.
While context evaluation tended to validate our own experience of the environment, it elucidated various points. For example, we realized that resources would not allow us to offer a dedicated palliative care clinical rotation; therefore, it was important to provide other opportunities for PC clinical immersion. We also felt empowered to put much of our focus in our ongoing curriculum evaluation on communication skills training, as our stakeholders felt this to be a crucial piece of the curriculum.
Input Evaluation
Input focuses on the implementation and design of interventions based on the environmental context. Our input evaluation questions asked about approaches/strategies to meet curricular goals and key resources.
Input Evaluation Methods
To answer input evaluation questions, we referred to our literature review, 7 interviews with outside curriculum directors and observations of outside programming. Summaries of best practices were collected, and initial curricular design was discussed with PC leadership, content experts and students. Finally, smaller pilots were planned for most parts of the curriculum prior to implementation for all students. Student focus groups and faculty interviews were held after each pilot session to collect feedback along with other data points. For large, single-day teaching interventions, faculty interviews happened directly or soon after the session. For small recurring sessions, faculty group interviews were scheduled once all faculty had a chance to participate. Student focus groups were scheduled in a similar fashion.
Input Evaluation Findings
We used the input evaluation data along with context data to write goals and objectives (Table 1) and an outline of key sessions (Table 4). Goals and objectives were created based on our literature review and relied largely on a 2014 article by Schafer et al who surveyed national PC experts and identified important medical student PC competencies.21-23 They were then modified based on our context data; for example, the ethics around end-of-life decision making and management of certain symptoms were de-emphasized as they were already covered in other areas of the general curriculum. Finally, goals and objectives were cross-referenced with curricular content standards from the Liaison Committee on Medical Education (LCME) and PSOM Objectives.8,24
Table 4.
CARE-7 Core Sessions.
Session title | Description | Placement in PSOM curriculum | Student evaluation | Session evaluation | Changes made after pilot |
---|---|---|---|---|---|
Introduction to palliative care part one | Didactic session introducing basic concepts in PC | Pre-clerkship: Fall of 1st year | # Large group discussion | # Student evaluations | # Decision to split up material into 2 sessions rather than 1 |
# Faculty interviews | |||||
# Student focus group | |||||
Introduction to palliative care part two | Didactic session reviewing basic concepts in PC and introducing more complex topics such as disease trajectories and PC across settings | Pre-clerkship: Spring of 1st year | # Large group discussion | # Student evaluations | NA |
# Faculty interviews | |||||
# Student focus group | |||||
Hospice and palliative medicine experience | Multi-part experience including an immersion experience in a hospice or PC setting followed by a small group session during which students interview bereaved family members and then debrief their experiences | Pre-clerkship: Fall of 2nd year, part of doctoring course | # Small group discussion | # Student evaluations | # Expansion of immersion experiences to include home visits |
# Faculty interviews | # Creation of short, recorded talks to prepare students prior to immersion and small-group sessions | ||||
# Student focus group | |||||
Pain and symptom management | Asynchronous didactic session on pain and symptom management followed by a small group evolving video case simulation | Occurs during “PC skills practicum day,” a full day of required PC content midway through clerkship year | # Quizzes | # Student evaluations | # Additional asynchronous didactic sessions added to create more time for discussion during in-person sessions |
# Small group discussion | # Faculty interviews | ||||
# Student focus group # quizzes | |||||
Coping with serious illness and end-of-life | Small group session where students read and discuss a poem and then discuss their own emotional responses to encountering patients with serious illness | Medicine clerkship | # Small group discussion | # Student evaluations | # Addition of a second poem to the session |
# Faculty interviews | # Increased emphasis on student discussion over facilitator tips or suggestions | ||||
# Student focus group | |||||
Palliative care assessment | Students choose 1 patient with serious illness to do a PC holistic assessment with the goal of understanding their experience of illness and practicing communication skills | Surgical clerkship | # Reflection assignment | # Student evaluations | # Improved messaging to students and faculty that the assignment is about understanding the patient experience and is not a goals of care discussion |
# Student medical record notes | # Faculty interviews | ||||
# Discussion with surgical team | # Medical record notes | ||||
# Student reflections | |||||
# Number of PC consults on select surgical services | |||||
# Student focus group | |||||
Nuts and bolts of end-of-life care | Didactic session on caring for patients at the end of life | Post-clerkship: Takes place during residency preparation electives in 4th year | # Small group discussion | # Student evaluations | # Increased discussion about the role of hospice care |
# Faculty interviews | |||||
# Student focus group | |||||
Communication curriculum | “Delivering serious news”: Didactic followed by small group skills practice using drills and video case simulation | Occurs during “PC skills practicum day,” a full day of required PC content midway through clerkship year | # Formative small group assessment | # Faculty interviews | # Increased student engagement/interaction during didactic sessions |
# Student evaluations | |||||
# Surveys | |||||
# Focus groups | |||||
“Goals of care conversations part one: drills”: Didactic followed by small group sessions where students practice using skills through evolving cases | Integrated into 4th year required specialty-specific tracks | # Formative small group assessment | # Faculty interviews # Student evaluations # Surveys # Focus groups | NA | |
“Goals of care conversations part two: skills practice”: Small group sessions using live simulated patients and VitalTalk© methodology | Integrated into 4th year required specialty-specific tracks | # Formative small group assessment | # Faculty Interviews # Student evaluations # Surveys # Focus groups | # Inclusion of time for short didactic content review |
Based on what we learned in the literature review and discussion with outside institutions, we included clinical exposure to patients with PC needs, simulation, and a range of teachers across specialties and disciplines.5,18,25 For communication skills sessions we used methodology from VitalTalk©, a nationally recognized communication skills training program. 26 As other successful curricula have done, 4 we took a step-wise approach to designing the curriculum: pre-clerkship students are introduced to core concepts and skills and exposed to clinical PC; clerkship students then build on these skills by learning more advanced skills and concepts that they are prompted to apply clinically; and post-clerkship students practice advanced skills and have opportunities to pursue additional training through PC clinical electives or a PC area of concentration. CARE-7 coaches mentor students in the PC area of concentration and help teach curricular sessions. Coaches are all faculty members who practice primary palliative care rather than primarily specialist palliative care; this choice was intentional as we were hoping that coaches could model for students the importance of primary palliative care skills regardless of specialty.
Based on our input evaluation we created the following mission statement: “The CARE-7 Curriculum will prepare all medical students, regardless of future specialty, to care for people with serious illness in a multidisciplinary team through simulation and experiential sessions that foster lifelong learning of empathic patient and family-centered communication skills, holistic symptom management and support.” We felt that it was important to include lifelong learning as a principle in our curriculum vision because while students will not be experts in PC skills or topics by the time they graduate, we hope to foster an approach to learning that will help students to continue to develop these skills over their careers.
As with our context evaluation, our input evaluation revealed some unanticipated information. There was variation in expert opinion about the extent to which medical students should be expected to master advanced communication skills such as discussions about goals of care, for example, and whether students should have a mandatory PC rotation. We opted to give students an opportunity to practice having a goals of care conversation. Because our PC and hospice services did not have the bandwidth for a mandatory PC rotation, we created a short early-immersion experience and embedded PC assessment of a patient in an existing clerkship. Furthermore, advice from outside programs that signposting the cohesiveness of a longitudinal curriculum can be difficult led to the creation of a CARE-7 logo.
Process Evaluation
Process evaluation is the observation of how the implementation process is proceeding, noting any areas that diverge from the plan or that require modification. In our process evaluation, we asked if our PC curriculum was being implemented as intended and sought to learn about early indicators of desired outcomes. Given our ambitious roll-out plan, a process evaluation was especially important.
Process Evaluation Methods
A large part of process evaluation focused on faculty interviews, student evaluations and focus groups that took place after session pilots. Detailed notes were taken during these meetings and used in leadership discussions to guide changes for future sessions. One session, called the “PC Assessment,” asks students to perform a comprehensive evaluation of 1 of their patients during Surgical Clerkship and evaluate key PC domains. Students write a note in the medical record, bring pertinent findings back to their team, and write a reflection on their experience. Two CARE-7 team members are currently working on qualitative analysis of electronic medical record (EMR) notes and reflections to identify common themes. At the time of this writing 42 student reflections and EMR notes have been collected and coding is ongoing. A new PC Specialty Interest Group was created in the Fall of 2021, and we are monitoring activity as well as clinical PC elective and area of concentration requests.
Process Evaluation Findings
At the time of this writing 8 out of 10 curricular sessions have been piloted or implemented for all students. Based on the corpus of process evaluation data, curricular sessions have gone as intended and been well-received by students as well as faculty. Reflections and medical record notes show that students are applying knowledge and shifting attitudes in the desired direction. The PC specialty interest group had 7 meetings in its first year and started a program to help hospice patients write a memoir, and multiple students signed up for the PC area of concentration directly after its approval.
All sessions have undergone changes after session pilots (Table 4); changes are tracked and discussed at curriculum director meetings. Some early challenges have included faculty availability to teach sessions and attend faculty development, and logistical hurdles of having pre-clinical medical students spend time in clinical settings.
There are many changes that we made to sessions based on student and faculty input that would have been missed if we had focused on outcomes data alone. For example, we were surprised to learn from students that they valued having evolving video cases to learn communication skills rather than using simulated patient encounters for every session. Another surprise was the interest and engagement of medical students, residents, and fellows in the process of evaluating CARE-7; learners have helped by designing the logo, running focus groups, evaluating reflections and quiz scores and volunteering to help teach sessions. While all planned sessions were given and well-attended, the observation that some students are not able to attend all sessions led to the creation of make-up assignments.
Product Evaluation
Product evaluation refers to the learning outcomes of a curriculum and determines how well it has met the needs of its stakeholders. Our product assessment questions attempt to answer whether sessions are well received, effective in imparting curricular objectives, and to attempt to uncover any un-intended consequences of the curriculum. It also seeks to answer why our curriculum was successful or not in the above measures.
Product Evaluation Methods
Student surveys are being administered at various points in their medical school training to assess student indicators of a lifelong learning approach to communication skill building and to assess the learning environment. Because a growth mindset has been associated with a lifelong learning approach in medical students, surveys include an instrument measuring students’ growth mindset toward communication skills. 27 Yearly student focus groups are planned to assess student’s thoughts about the value of the curriculum, gain insight about their attitudes to learning curricular content, and assess the impact of the hidden curriculum. These yearly focus groups have a broader scope than the pilot-specific focus groups used in process evaluation and will undergo thematic analysis. Bi-annual meetings of PSOM faculty involved in communication skills sessions are planned to ensure integration of CARE-7 communication skills content with other communication skills sessions and to assess the impact of these sessions. Given that 1 of the CARE-7 sessions takes place during the surgical clerkship, palliative care consults to select surgical services will be monitored to assess for any changes to clinical care that might arise. Other tools used in product evaluation are listed in Table 2. Data is collected in a database and qualitative analysis of focus groups and reflections is summarized.
Product Evaluation Findings
At the time of this writing, product evaluation is still in early stages. We are encouraged by initial student enthusiasm for sessions as assessed by session evaluations, focus groups, reflections, and interest in the specialty interest group and area of concentration. Evaluations showed that the “PC Skills Day,” a part of the clerkship year “Practicum Series” was the most highly rated practicum day of the clerkship year. Initial student surveys surrounding communication skills sessions show that students feel sessions create a safe learning environment, and that student indicators of a lifelong learning approach are improved after CARE-7 skills sessions. Quiz scores show that students are learning key concepts. Further data collection as students progress through medical school is needed to understand survey implications and general curricular impact.
Conclusion
Our use of the CIPP model has been invaluable in the process of designing and implementing CARE-7. Because every curriculum will be developed in a context with unique resources, unique barriers and unique stakeholder needs, a one-size-fits-all approach to a PC longitudinal curriculum would be impractical and limiting. No curriculum is a static creation. Ideally, curriculum directors are responsive to the need to change and constantly seek to improve. A successful curriculum will design inputs based upon the context, involve stakeholders in process evaluation and design an outcomes evaluation informed by stakeholder needs. CIPP is a holistic evaluation model that can be applied across contexts and facilitates iterative review of the the curricular processes while paying attention to curricular products.
While longitudinal palliative care curricula are increasingly recognized as critical to medical training, there is a lag in emphasis on robust evaluation to support such curricula. 28 When such curricula are implemented, evaluators often cite prospective program evaluation as a challenge.6,29 To our knowledge, CARE-7 is the first U.S. palliative care UME longitudinal curriculum to describe the use of an evaluation model for prospective program development as well as outcomes assessment. The resulting curriculum is an ambitious program composed of 10 core sessions over 4 years utilizing instructional tools including clinical experiences, various types of simulation, VitalTalk© methodology and interdisciplinary teachers who can role model both primary and specialist palliative care skills.
In our own use of the CIPP framework to evaluate CARE-7 we have identified lessons learned. First, having a curriculum evaluation committee comprised of some members with extensive experience with curriculum evaluation was hugely helpful in guiding our efforts. Next, stakeholder involvement in the process has been helpful both in terms of data collection as well as fostering ongoing stakeholder investment in the project. Finally, we have found that the focus on fostering students primed for lifelong learning has provided further direction to our curriculum evaluation and shaped how we improve upon and evaluate sessions.
In conclusion, the use of a comprehensive program evaluation model such as CIPP can help with the design and implementation of context-specific PC curricula and improve the quality of evaluation. Utilizing the structured approach of the CIPP model allowed us to uncover findings we might not have otherwise and dynamically improve the curriculum, ultimately making the educational experience more relevant and impactful for our students. We look forward to learning more from product evaluation and plan for future scholarship to describe findings in this component of CIPP.
Acknowledgments
The CARE-7 curriculum is generously funded by Barbara M. Jordan, a Penn Medicine Board member and spouse of the late Henry A. Jordan, MD, a Penn alumnus (M’62 from the School of Medicine).
Footnotes
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Barbara M. Jordan.
ORCID iD
Alana Sagin https://orcid.org/0000-0003-1021-6414
References
- 1.Kamal AH, Wolf SP, Troy J, Leff V, Dahlin C, Rotella JD, et al. Policy changes key to promoting sustainability and growth of the specialty palliative care workforce. Health Aff. 2019;38(6):910-918. doi: 10.1377/hlthaff.2019.00018. PubMed PMID: 31158018 [DOI] [PubMed] [Google Scholar]
- 2.Head BA, Schapmire TJ, Earnshaw L, Chenault J, Pfeifer M, Sawning S, et al. Improving medical graduates’ training in palliative care: Advancing education and practice. Adv Med Educ Pract. 2016;7:99-113. doi: 10.2147/amep.S94550. Epub 20160224. PubMed PMID: 26955298; PMCID: PMC4772917. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Lloyd-Williams M, Carter YH. Can medical education extend palliative care? Palliat Med. 2003;17(7):640-642. doi: 10.1191/0269216303pm831xx. PubMed PMID: 14594157. [DOI] [PubMed] [Google Scholar]
- 4.Ellman MS, Fortin AH, Putnam A, Bia M. Implementing and evaluating a four-year integrated end-of-life care curriculum for medical students. Teach Learn Med. 2016;28(2):229-239. doi: 10.1080/10401334.2016.1146601. PubMed PMID: 27064725. [DOI] [PubMed] [Google Scholar]
- 5.Bush SH, Gratton V, Kabir M, Enright P, Grassau PA, Rice J, et al. Building a medical undergraduate palliative care curriculum: lessons learned. J Palliat Care. 2021;36(1):29-37. doi: 10.1177/0825859720916565. PubMed PMID: 32284024. Epub 20200413. [DOI] [PubMed] [Google Scholar]
- 6.Radwany SM, Stovsky EJ, Frate DM, Dieter K, Friebert S, Palmisano B, et al. A 4-year integrated curriculum in palliative care for medical undergraduates. Am J Hosp Palliat Care. 2011;28(8):528-535. doi: 10.1177/1049909111406526. PubMed PMID: 21518705. Epub 20110425. [DOI] [PubMed] [Google Scholar]
- 7.Weissman DE, Quill TE, Block SD. Missed opportunities in medical student education. J Palliat Med. 2010;13(5):489-490. PubMed PMID: 20420547. doi: 10.1089/jpm.2010.9838 [DOI] [PubMed] [Google Scholar]
- 8.Liason committee on medical education. https://lcme.org/. Available from: [November 17, 2022].
- 9.Sullivan AM, Warren AG, Lakoma MD, Liaw KR, Hwang D, Block SD. End-of-life care in the curriculum: A national study of medical education deans. Acad Med. 2004;79(8):760-768. doi: 10.1097/00001888-200408000-00011. PubMed PMID: 15277133. [DOI] [PubMed] [Google Scholar]
- 10.Sullivan AM, Lakoma MD, Block SD. The status of medical education in end-of-life care: A national report. J Gen Intern Med. 2003;18(9):685-695. doi: 10.1046/j.1525-1497.2003.21215.x. PubMed PMID: 12950476; PMCID: PMC1494921. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Chiu N, Cheon P, Lutz S, Lao N, Pulenzas N, Chiu L, et al. Inadequacy of palliative training in the medical school curriculum. J Cancer Educ. 2015;30(4):749-753. doi: 10.1007/s13187-014-0762-3. PubMed PMID: 25487030. [DOI] [PubMed] [Google Scholar]
- 12.Hammel JF, Sullivan AM, Block SD, Twycross R. End-of-life and palliative care education for final-year medical students: A comparison of Britain and the United States. J Palliat Med. 2007;10(6):1356-1366. doi: 10.1089/jpm.2007.0059. PubMed PMID: 18095815. [DOI] [PubMed] [Google Scholar]
- 13.Smith GM, Schaefer KG. Missed opportunities to train medical students in generalist palliative care during core clerkships. J Palliat Med. 2014;17(12):1344-1347. doi: 10.1089/jpm.2014.0107. PubMed PMID: 24971609. [DOI] [PubMed] [Google Scholar]
- 14.Horowitz R, Gramling R, Quill T. Palliative care education in U.S. medical schools. Med Educ. 2014;48(1):59-66. doi: 10.1111/medu.12292. PubMed PMID: 24330118. [DOI] [PubMed] [Google Scholar]
- 15.Billings ME, Engelberg R, Curtis JR, Block S, Sullivan AM. Determinants of medical students' perceived preparation to perform end-of-life care, quality of end-of-life care education, and attitudes toward end-of-life care. J Palliat Med. 2010;13(3):319-326. doi: 10.1089/jpm.2009.0293. PubMed PMID: 20178433; PMCID: PMC2883506. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Dickinson GE. Thirty-five years of end-of-life issues in US medical schools. Am J Hosp Palliat Care. 2011;28(6):412-417. doi: 10.1177/1049909110397608. PubMed PMID: 21282128. Epub 20110130. [DOI] [PubMed] [Google Scholar]
- 17.DeCoste-Lopez J, Madhok J, Harman S. Curricular innovations for medical students in palliative and end-of-life care: A systematic review and assessment of study quality. J Palliat Med. 2015;18(4):338-349. doi: 10.1089/jpm.2014.0270. PubMed PMID: 25549065. Epub 20141230. [DOI] [PubMed] [Google Scholar]
- 18.Kozhevnikov D, Morrison LJ, Ellman MS. Simulation training in palliative care: state of the art and future directions. Adv Med Educ Pract. 2018;9:915-924. doi: 10.2147/amep.S153630. PubMed PMID: 30574008; PMCID: PMC6292390. Epub 20181207. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Fitzpatrick D, Heah R, Patten S, Ward H. Palliative care in undergraduate medical education-how far have we come? Am J Hosp Palliat Care. 2017;34(8):762-773. doi: 10.1177/1049909116659737. PubMed PMID: 27468933. Epub 20160728. [DOI] [PubMed] [Google Scholar]
- 20.Stufflebeam CFMaTK DL. Evaluation Models. Boston, MA: Kluwer Academit Publishers; 2000. [Google Scholar]
- 21.Meekin SA, Klein JE, Fleischman AR, Fins JJ. Development of a palliative education assessment tool for medical student education. Acad Med. 2000;75(10):986-992. doi: 10.1097/00001888-200010000-00011. PubMed PMID: 11031142. [DOI] [PubMed] [Google Scholar]
- 22.Schaefer KG, Chittenden EH, Sullivan AM, Periyakoil VS, Morrison LJ, Carey EC, et al. Raising the bar for the care of seriously ill patients: Results of a national survey to define essential palliative care competencies for medical students and residents. Acad Med. 2014;89(7):1024-1031. doi: 10.1097/acm.0000000000000271. PMC4077186. PMCID. PubMed PMID: 24979171. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Bush SH, Roze des Ordons A, Chary S, Boyle AB. The development and validation of updated palliative and end-of-life care competencies for medical undergraduates in Canada. J Palliat Med. 2019;22(12):1498-1500. doi: 10.1089/jpm.2019.0160. PubMed PMID: 31486699. Epub 20190905. [DOI] [PubMed] [Google Scholar]
- 24.Perelman school of medicine: Medical education program objectives webpage. https://www.med.upenn.edu/student/medical-education-program-objectives.html. Available from: [cited 2023 January 21].
- 25.Boland JW, Barclay S, Gibbins J. Twelve tips for developing palliative care teaching in an undergraduate curriculum for medical students. Med Teach. 2019;41(12):1359-1365. doi: 10.1080/0142159x.2018.1533243. PubMed PMID: 30689479. Epub 20190128. [DOI] [PubMed] [Google Scholar]
- 26.VitalTalk. https://www.vitaltalk.org. Available from: [November 9, 2022].
- 27.Babenko O, Daniels LM, Ross S, White J, Oswald A. Medical student well-being and lifelong learning: a motivational perspective. Educ Health. 2019;32(1):25-32. doi: 10.4103/efh.EfH_237_17. PubMed PMID: 31512589. [DOI] [PubMed] [Google Scholar]
- 28.Lloyd-Williams M, MacLeod RDM. A systematic review of teaching and learning in palliative care within the medical undergraduate curriculum. Med Teach. 2004;26(8):683-690. doi: 10.1080/01421590400019575. PubMed PMID: 15763870. [DOI] [PubMed] [Google Scholar]
- 29.Denney-Koelsch EM, Horowitz R, Quill T, Baldwin CD. An integrated, developmental four-year medical school curriculum in palliative care: A longitudinal content evaluation based on national competency standards. J Palliat Med. 2018;21(9):1221-1233. doi: 10.1089/jpm.2017.0371. PubMed PMID: 29768072. Epub 20180516. [DOI] [PubMed] [Google Scholar]