Introduction

In 2013, the Accreditation Council for Graduate Medical Education (ACGME) implemented its Next Accreditation System (NAS) that requires semiannual evaluation of the milestones that are expected of residents throughout their training ( 1). This move was due in part to shifting attitudes regarding medical education, specifically moving towards an outcome-driven system of evaluating success ( 2). For Emergency Medicine (EM), the American Board of Emergency Medicine describes 23 sub-competencies with milestones that range from level 1 (expected of an incoming resident/intern) to level 5 (exceptional residents that demonstrate abilities of an attending). 

Since implementation of NAS, studies of incoming interns have found that many fall short of meeting level 1 milestones ( 3- 4). Santen et al. previously surveyed EM interns across the United States and reported up to 39% of interns reported never receiving instruction on certain milestones ( 3). Additionally, a previous observational study on incoming EM interns found a wide variability ranging from 48-93% competency in the milestones assessed ( 4). Challenges in implementing competency-based medical education such as NAS include barriers to creating curricula that individualize learning plans and inconsistent assessment of milestones ( 5). To our knowledge, there have not yet been studies on introducing a milestones-based curriculum in the pre-clinical years, typically the first and second year of traditional medical school curricula. Here, we describe a curriculum developed for medical students to introduce milestones prior to entering the residency stage.

This study took place at a medical school with a robust ultrasound curriculum that showed success in early integration and longitudinal development of ultrasound skills throughout medical school ( 7- 8). We sought to determine the effects that a pre-clinical Emergency Medicine Interest Group (EMIG) Milestones Elective would have on preparing students interested in EM as a specialty to meet the level 1 milestones prior to graduating medical school.

The EMIG Milestones Elective’s objective was to prepare the students who complete the elective to meet 19 of the 23 level 1 milestones (4 were omitted, as they are better suited for MS3 and MS4 years in a clinical setting). As a result of the study, we sought to determine the effect, if any, that the elective had on 1) preparing students to meet milestone expectations, and 2) impacting the level of interest expressed by the student in pursuing Emergency Medicine as a specialty. Additionally, the results of the survey would allow us to identify parts of the curriculum to improve for future years. Here, we report significant increases in self-reported preparedness for meeting the majority of the level 1 milestones included in the study. 

Methods

This study was reviewed by the Institutional Review Board and classified as exempt with a waived requirement for signed informed consent. A Study Information Sheet was provided to students via email and on the first page of the electronic survey with response buttons to indicate consent. Students were allowed to participate in the elective regardless of participation in our study without penalty. 

There were 23 sub-competencies outlined by the ACGME. Four of the 23 were omitted in designing the curriculum because they were better suited for third- and fourth-year training. The four omitted sub-competencies were PC8 multi-tasking, SBP1 patient safety, SBP3 technology, and PROF2 accountability as these are better taught in a clinical setting during the third- and fourth-year medical school training. The decision to omit these sub-competencies was made by the elective coordinator with guidance from the faculty advisor who was also associate residency program director at the time and well-versed in milestone requirements and residency education. The remaining 19 sub-competencies were more broadly categorized into 6 competencies based on ACGME guidelines: patient care, medical knowledge, system-based practice, practice-based learning and improvement, professionalism, and interpersonal and communication skills. The 19 level 1 milestones are outlined in Table 1. Data regarding competencies 5 and 6 were combined in our analysis due to the similarity in competency features. 

Convenience sampling was used to gather our data. Participants were first- and second-year medical students enrolled in the milestones elective. A survey assessing the level of interest in EM, overall preparedness for an intern year in EM, and readiness for meeting the Level 1 skills for each of the 19 included milestones was designed in Qualtrics Survey Software. The responses were scored based on a Likert scale (i.e. not prepared at all, somewhat prepared, neutral, very prepared, and extremely prepared). A repeated measures design was used, in which the same variables were measured on the same sample before and after the curriculum. The survey was first administered at the beginning of the academic year prior to any elective events (pre-curriculum survey). Respondents taking the pre-curriculum survey were de-identified and assigned an anonymous, random 6-digit identifier used to track the survey results at the end of the year (post-curriculum survey). The post-curriculum survey contained the same questions and answer choices as the pre-curriculum survey. Due to the novelty and specific focus of our study design, the questions used in our survey have not been validated.  

There were 15 events throughout the year: Wilderness Medicine, Intro to EM Talk, Procedures Workshop, five Talk Shops with EM attendings, Research Opportunities dinner, Shadowing, Jeopardy, Matching into EM Panel, Disaster Medicine, Cadaver Workshop, and Post-Match Panel. Sub-competencies were assigned to events based on the event type (Table 2). These assignments were also discussed with the faculty member overseeing the elective. This advisor was also well-versed in what each event entailed. Students earned credit for sub-competencies assigned to a particular event by attending. Credit for the elective was earned by attending the combination of events to satisfy all 19 sub-competencies and at least eight events. After all events were held, the post-curriculum survey was administered using the 6-digit identifier for longitudinal tracking. 

The elective was graded based on completion for transcript notation and without penalty if a student did not complete the elective. No letter grades were assigned, and participation in the survey was voluntary and anonymous. Forty-six first- and second-year medical students signed up for the elective, and 22 of the 46 completed the elective for credit. Twenty-two students agreed to participate in the study and took the pre-curriculum survey. Thirteen out of those 22 students who completed the initial survey also completed the post-curriculum survey. Students who did not complete the elective were not included in our study due to an incomplete data set. We used these 13 sets of data for our analysis. 

Statistical analysis

The responses were re-coded into 1 (not prepared/interested at all) to 5 (extremely prepared/interested). Distribution of milestones are presented as median and interquartile range (quartile 1 to quartile 3). In case of competencies 1 and 6, the median and interquartile range (IQR) of all milestones that comprised those competencies were calculated. The statistical significance of any difference between pre- and post- curriculum was calculated by using Wilcoxon Signed Ranks Test. The Wilcoxon Signed Ranks Test was used because the survey design repeated measurements before and after the curriculum on the same sample, as the surveys administered asked identical questions to the same respondents. We used IBM SPSS statistics 26 for data analysis.

Ethical Consideration

Some of the members involved in the creation of this study were faculty who worked directly with the study participants in other aspects of training. Participation in the study was optional and participants could withdraw at any time without penalty or academic repercussions. The data collected was not traceable to individual respondents due to the use of anonymous, randomly-assigned identifiers. The survey questions assessed self-identified level of interest/preparedness in the field of emergency medicine in general and not specific to our academic institution. There was no reference to our institution's faculty or affiliates, so participants could answer survey questions honestly without any impact on transcript grades.

Results

There were 13 sets of data included for analysis. The median (IQR) for overall preparedness for an intern year in an EM residency was 1 (1 to 2) before the curriculum and 2 (2 to 3) after the curriculum (Figure 1, p=0.027). The median level of interest in EM was 4 (3 to 5) before the curriculum. The median level of interest in EM remained 4 (3 to 5) after the curriculum (Figure 1, p=0.317).

Figure 1. Self-reported level of preparedness and level of interest in emergency medicine (EM), before and after completing the EMIG Milestones Elective, EMIG : Emergency Medicine Interest Group, ACGME: Accreditation Council for Graduate Medical Education

In assessing perceived preparedness for each of the 19 level 1 milestones grouped by competency, there were significant increases in competencies 1-4 (Figure 2). The categories and their corresponding statistical p-values are displayed in Table 3. For competency 1 (patient care, sub-competencies PC1-PC14), the median score was 2 (1 to 3) prior to the elective and increased to 3 (2 to 4) after elective (p=0.004) (Table 3). 

Figure 2. Level of readiness for level 1 milestones, before and after the EMIG Milestones Elective. Increased readiness was found in competencies 1, 2, 3, and 4. 

The median score for preparedness in competency 2 (medical knowledge, MK) was 1 prior to the elective; median level of preparedness increased to 3 (2 to 4) after the elective (p=0.002) (Figure 2). Competency 3 (system-based practice, SBP2) showed a median increase from 2 to 3 (p=0.037), and competency 4 (practice-based learning and improvement, PBLI) also showed a median increase from 2 to 3, (p=0.004) (Figure 2). Competencies 5 and 6 (professionalism and interpersonal and communication skills, PROF1 and ICS1-ICS2) did not show a significant increase from the post-curriculum survey. For these milestones, the medians for pre- and post-curriculum were 4 (p=0.234) (Figure 2). 

Discussion

Significant increases in the students’ perceived preparedness were seen in 16 out of the 19 level 1 milestones included in the elective as well as overall. The competency that showed the most significant increase was medical knowledge (sub-competency MK). Within the elective, PC1 and MK (medical knowledge) had the highest number of events that qualified for milestone credit, which may have played a proportional role in preparing students to feel they could meet the level 1 milestones for those sub-competencies. As an elective geared towards first- and second-year students, it is also expected that initial medical knowledge may naturally increase with more time and exposure to medicine, not only in the field of EM, but generalizable to the time spent in the medical school environment. 

The next competencies to show the highest significance in increased preparedness were patient care (sub-competencies PC1-PC14) and practice-based learning and improvement (sub-competency PBLI). Patient care encompasses the greatest number of sub-competencies, and 11 out of the 15 events covered at least one sub-competency in this broader competency. Therefore, an increased frequency of events that incorporate patient care may positively influence preparedness in those level 1 milestones. Additionally, several of these 11 events contained hands-on skills training, so this finding could also indicate that students show better response to an active learning setting, allowing them to feel more confident in these categories. 

Lamba et al. previously studied the effects on self-reported confidence after hosting a procedures workshop on intubation, thoracostomy, and central venous catheterization, which fall under the competency of patient care ( 8). The previous study was similar to ours in having a small sample size surveyed before and after an intervention workshop; however, the study was performed on EM-bound senior medical students rather than first- or second-year students. Our study showed similar results in that there was a significant increase in self-reported preparedness for patient care after the elective.

Practice-based performance improvement is a theme seen throughout events in the elective as this study took place in an academic institution with several areas of ongoing research and emphasis on research-based improvement practices. This environment may have played a role in increasing preparedness in this competency. 

System-based practice also showed significant increases in perceived preparedness. The level 1 milestone for this sub-competency SBP2 is that a student is able to describe members of the ED team and their roles (e.g., nurses, techs, security). This skill is best taught through interactions with attendings or students in their third or fourth years and through shadowing. Though skills acquired through shadowing were left open for students to select which ones they would receive credit for, the other events that covered SBP2 involved direct interactions with attendings and/or fourth-year medical students. 

The remaining competencies of professionalism and interpersonal and communication skills (PROF1, ICS1, ICS2) require consistent training and are not readily addressed in the types of events offered through the elective that are more geared towards procedural skills and talk shops. The post-match panel was notable for being moved to a virtual setting this year due to COVID-19 pandemic restrictions, which may have resulted in less engagement with the panelists than has been observed in similar events in previous years. Overall, this may point to the notion that professionalism and interpersonal communications skills are not easily taught or improved upon by a short elective course.  In a survey conducted by Stehman et al. regarding assessment of the competency of professionalism, non-technical skills were most commonly assessed by faculty evaluation and only 11.2% of the survey respondents felt that this method of assessment of professionalism was very effective ( 9, 10). In our study, there was no significant increase in perceived preparedness in the non-technical competencies; taken together with the previous study, our results suggest that it may be difficult to objectively assess these skills.

There was no significant increase in interest in EM after the curriculum, though the baseline median level of interest was already the highest of all the survey categories at “very interested.” Students who were already interested in EM may have been more likely to take the elective initially, and those whose interest was consistent throughout the year may have been more likely to complete the elective, resulting in the insignificant change at the end. Students who lost interest in emergency medicine might have dropped the elective and, therefore, potentially biased the sample by not completing research surveys.

Limitations

One of the limitations of our study is its sample size. Although 22 students completed the elective, we only received 13 complete pre- and post-curriculum sets of data. It is likely that the respondents who were lost to follow-up were those who did not complete the elective. Within the 13 complete data sets, it is also impossible to track which event each of the respondents attended due to the de-identified format of the survey. This absence of tracking is important because it is possible to achieve all the milestones through a number of combinations of events, so the learning environments for each of the 13 students may not have been identical. 

Additionally, there may be volunteer bias due to our use of convenience sampling. The students participating in the surveys were those already enrolled in the elective at the beginning of the academic year and completed it by the end of the year. The students who completed the elective may have been more likely to report increased preparedness due to higher interest in EM-related topics at baseline. Another limitation of our study was the fact that our surveys were not previously validated. Due to the novel design and focus of our study, there were no previously validated surveys that addressed the main hypothesis of this study. 

In our study design, students self-reported their perceived level of preparedness in achieving level 1 milestones, rather than having a standardized method of measuring preparedness in the form of a post-curriculum assessment. For addressing this limitation, future elective courses could have third- or fourth-year medical students or EM residents evaluate the first- and second-year students before and after the elective in a practical skills session or through written exam. Quinn et al. report a method of evaluating students rotating through EM based on milestone achievement through the use of faculty evaluations and quizzes ( 9). Future studies using our elective could develop a more formal assessment of skills in a similar way.

Another limitation was accommodating for COVID-19 social distancing restrictions and implementation of remote learning so that our final event (Post-Match Panel) was held over video conferencing and may have limited the interaction that normally would be facilitated in a live event. Lastly, it is important to acknowledge the possibility of confounding, as this elective ran concurrently with the MS1 and MS2 medical school curricula. It is possible that increases in perceived preparedness observed in the survey data could be influenced by reinforcement of material in mandatory courses. 

Conclusion

Implementing a milestones-based curriculum during the pre-clinical years may better prepare the students interested in pursuing EM for meeting level 1 milestones prior to residency stage. This elective can be readily recreated in other programs by creating events that broadly encompass many aspects of EM. Examples include case studies, skills workshops, and interactions with attendings (e.g., lunch/dinner talks). To capture more abstract skills such as professionalism and interpersonal communication, events targeting these skills should be offered. Further investigation in the form of pre- and post-curriculum testing specific to each emergency medicine milestone is warranted to validate these results and better assess the true extent of achieving level 1 milestones. 

References

1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system – rationale and benefits. N Engl J Med. 2012; 366:1051-6.
2. Desy JR, Reed DA, Wolanskyj AP. Milestones and millennials: a perfect pairing - competency-based medical education and the learning preferences of generation Y. Mayo Clin Proc. 2017; 92(2):243-50.
3. Santen SA, Rademacher N, Heron SL, Khandelwal S, Hauff S, Hopson L. How competent are emergency medicine interns for level 1 milestones: who is responsible?. Acad Emerg Med. 2013; 20(7):736-9.
4. Hauff SR, Hopson LR, Losman E, Perry MA, Lypson ML, Fischer J, et al. Programmatic assessment of level 1 milestones in incoming interns. Acad Emerg Med. 2014; 21(6):694-8.
5. Hawkins RE, Welcher CM, Holmboe ES, Kirk LM, Norcini JJ, Simons KB, et al. Implementation of competency-based medical education: are we addressing the concerns and challenges?. Med Educ. 2015; 49(11):1086-102.
6. Fox JC, Schlang JR, Maldonado G, Lotfipour S, Clayman RV. Proactive medicine: the “UCI 30,” an ultrasound-based clinical initiative from the University of California, Irvine. Acad Med. 2014; 89(7):984-9.
7. Fox JC, Chiem AT, Rooney KP, Maldonado G. Web-based lectures, peer instruction and ultrasound-integrated medical education. Med Educ. 2012; 46(11):1109-10.
8. Lamba S, Wilson B, Natal B, Nagurka R, Anana M, Sule H. A suggested emergency medicine boot camp curriculum for medical students based on the mapping of Core Entrustable Professional Activities to Emergency Medicine Level 1 milestones.  Adv Med Educ Pract. 2016; 7:115-24.
9. Quinn SM, Worrilow CC, Jayant DA, Bailey B, Eustice E, Kohlhepp J, et al. Using Milestones as Evaluation Metrics During an Emergency Medicine Clerkship. J Emerg Med. 2016; 51(4):426-31.
10. Stehman CR, Hochman S, Fernández-Frackelton M, Volz EG, Domingues R, Love JN, et al. Professionalism milestones assessments used by emergency medicine residency programs: a cross-sectional survey.  West J Emerg Med. 2019; 21(1):152-9.