Introduction

Emergency Medicine (EM) residency programs first started using standardized letters of recommendations in place of traditional narrative letters of recommendations (NLORs) for their medical school applicants in 1995 ( 1 ). NLORs were largely subjective and difficult to interpret. This created a need to increase objectivity, as well as provide a way to evaluate an applicant’s interpersonal and clinical skills ( 2 - 4 ). Standard letters of recommendation have gone through several iterations, with the most recent version being the standardized letter of evaluation (SLOE) developed by the Council of Residency Directors in Emergency Medicine (CORD) in 2014 ( 5 ). SLOEs are completed by EM physicians or a committee of EM physicians and use ranking for specific categories, along with narrative feedback to combine objective and subjective measures of a candidate. A survey of EM directors in 2018 showed that SLOEs have developed a crucial role in the application process with 80% of directors requiring at least one SLOE to offer an interview and 38% requiring at least two SLOEs.  Furthermore, most directors ranked SLOEs among the top considerations for ranking an applicant ( 4 , 6 , 7 ) .

  During the COVID-19 pandemic, the Association of American Medical Colleges issued various non-clinical mandates and individual universities and hospitals adopted policies for their own students’ safety ( 8 - 10 ). With safety in mind, the EM community encouraged students to only participate in home EM rotations, decreasing students’ ability to obtain multiple SLOEs or even a single SLOE if no EM rotations were available ( 11 ). To compensate, CORD implemented “off-service” or “other rotation” SLOEs (OSLOEs), which could be completed by non-EM attendings. This option provided additional opportunities for students to receive both narrative feedback and objective rankings on an off-service, or non-EM clinical rotation, to be included in their application to the EM match. This letter allows for details about the student’s performance during an off-service rotation, including data regarding grades distributed, categorical clinical rankings among student peers, global assessment ranking, as well as narrative comments regarding the student.

 With the addition of non-EM physicians contributing to standardized letters of evaluation for EM residency applicants, there is concern related to the utility of OSLOEs as a substitute for a SLOE. The purpose of this study was to assess the utility of OSLOEs for candidate selection for a single EM residency during the 2020-21 application cycle.

Methods

A retrospective cohort study utilizing data from OSLOEs submitted during the 2020-21 academic year to an EM residency program was performed. Students’ applications were downloaded from the Electronic Residency Application Service (ERAS) following submission to the residency program. OSLOEs submitted within the applications were de-identified and data points from these letters were recorded. Data collected included global rank (top 10%: star student, chief resident potential; top 1/3: above average student; middle 1/3: solid student, average; and bottom 1/3: below average student), grade (honors, high pass, pass, low pass, fail), and specific category (knowledge, work ethic, communication, teachability, respectfulness, admits mistakes, accountable, and reliability) rank (top 1/3: above peers; middle 1/3: at level of peers; bottom 1/3: below peers). OSLOEs were excluded from review if they were submitted from EM or EM subspecialty rotations, no rotation was identified or the rotation was non-clinical or obscure (e.g., research, administrative). Recorded data were reviewed and checked for accuracy by the principal investigator prior to statistical summary. A total of 296 OSLOEs were submitted with 598 applications, with 270 meeting the eligibility criteria for review. Summary statistics were calculated using IBM SPSS Statistics, v. 23 (Armonk, NY; IBM Corp).

Ethical Consideration

The study was reviewed and determined to be non-human subjects’ research by the Institutional Review Board.

Results

A total of 598 applications were submitted. Of these, 296 OSLOEs were submitted from 232 (38.8%) applicants (1 OSLOE: 74.6%, 2 OSLOEs: 23.3%, 3 OSLOEs: 1.3%). Twenty-six OSLOEs were excluded due to EM or EM subspecialty rotations, no rotation identified, or non-clinical or obscure rotations, leaving 270 OSLOEs for review.  

For the overall global assessment rating comparing the student “to all students you have worked with over the past few years” 95% of the letters ranked the candidate in the top 2 categories (Top 10%: star student, chief resident potential; Top 1/3: above average student), with over 60% rated in the highest category (Figure 1). There were 211 OSLOEs with grades given, with over 90% receiving the highest two grades (honors, high pass) and 0% in the bottom 2 categories of low pass or fail (Figure 2). Over 75% of the OSLOEs had rankings of top 1/3 for specific characteristics as compared to their peers and 0% ranked as bottom 1/3 (Figure 3).

Figure 1. Global assessment rating

Figure 2. Distribution of grade

Figure 3. Rank for specific category within the OSLOE (Off Standardized Letters of Evaluation)

Discussion

Given the overall ranking and grade distributions seen in our study, the utility of the objective portion of the OSLOE seems quite low. Our findings showed that the majority of submitted OSLOEs ranked applicants in the top 10 % (star student, chief resident potential) and 95% ranked students in the top two categories. The majority of students were given the highest grade (honors) and highest ranking (top 1/3) in specific categories as well. In contrast, authors reviewing rank distributions of an electronic version of the SLOE reported findings of only 18% of candidates receiving a rank of top 10% and only 55% received the highest two rankings ( 5 ). Further, only 39% received an honors grade. This earlier study also demonstrated that prior to the revision and training efforts related to the SLOE, it suffered from over-ranking of applicants.

We suspect a contributing factor to the lack of utility regarding OSLOEs submitted to our program is the evaluators’ lack of familiarity with how to use the tool, especially as it relates to the EM residency application process. A previous study found less experienced evaluators were more likely to grade favorably, even among EM faculty who write SLOEs annually ( 12 ). Gender, duration of time the letter writer knew the applicant, inexperience and rotations at the home institution have also been shown to cause bias when evaluating candidates ( 13 - 15 ). SLOEs have been through multiple iterations, with an educational campaign and series of best practices for the people utilizing them most ( 16 ).

Selection bias may also play a role in the lack of utility of the OSLOE. Given the freedom of a student to choose the faculty at their home institution to complete the OSLOE, it would be foolish not to consider that they would select someone they believe will rank them highly. This is fundamentally different than the SLOE, in which only certain faculty / faculty groups can prepare SLOEs within EM, and students are not allowed to choose who completes the evaluation. Evaluators also may have graded higher given the situation the students were put in due to the pandemic.

A limitation of our findings is that results are from one academic cycle at a single institution. However, the data reported was generated from various specialties from all over the country within these OSLOEs. A larger multi-institutional comparison would help to elucidate the utility of the OSLOE.

Conclusion

In an attempt to adapt quickly to the lack of availability of in-person EM rotations due to the COVID-19 pandemic, the OSLOE was a logical alternative. However, the utility of the OSLOE in place of a traditional SLOE remains in question. Our findings provide evidence that the ranking distribution of the OSLOE may have little value in the evaluation of student performance. If the OSLOE continues to be utilized, educational efforts related to completing these letters of recommendation may be warranted.

Authors' contribution

J.G, B.D, J.C, Contributed to data collection and manuscript review; T.K, Performed statistical analysis and manuscript review; R.O, Performed manuscript review; J.B Implemented study design, and contributed to manuscript review. All Authors contributed in drafting and revising the manuscript critically for important intellectual content. All authors have read and approved the final manuscript and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflict of Interest:

None declared.

References

1. Keim SM, Rein JA, Chisholm C, Dyne PL, Hendey GW, Jouriles NJ, et al. A standardized letter of recommendation for residency application.  Acad Emerg Med. 1999; 6(11):1141-6.
2. Girzadas D, Harwood R, Dearie J, Garrett S. A comparison of standardized and narrative letters of recommendation. Acad Emerg Med. 1998; 5(11):1101-4.
3. Martin DR, McNamara R. The CORD standardized letter of evaluation: Have we achieved perfection or just a better understanding of our limitations?. J Grad Med Educ. 2014; 6(2):353-4.
4. Pelletier-Bui A, Franzen D, Smith L, Hopson L, Lutfy-Clayton L, Parekh K, et al. COVID-19: A driver for disruptive innovation of the emergency medicine residency application process. West J Emerg Med. 2020; 21(5):1105-13.
5. Jackson JS, Bond M, Love JN, Hegarty C. Emergency medicine standardized letter of evaluation (SLOE): Findings from the new electronic SLOE format. J Grad Med Educ. 2019; 11(2):182-6.
6. Negaard M, Assimacopoulos E, Harland K, Van Heukelom J. Emergency medicine residency selection criteria: An update and comparison. AEM Educ Train. 2018; 2(2):146-53.
7. Kukulski P, Ahn J. Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation. J Grad Med Edu. 2021; 13(4):490-9.
8. Council of Residency Directors in Emergency Medicine. Consensus statement regarding SLOEs and away rotations from the CORD advising students committee in emergency medicine. (n.d.) [Internet]. [Retrieved November 21, 2021]. Available from: https://www.cordem.org/resources/residency-management/sloe/osloe-instructions/.
9. Katirji L, Smith L, Pelletier-Bui A, Hillman E, Zhang X, Pasirstein M, et al. Addressing Challenges in Obtaining Emergency Medicine Away Rotations and Standardized Letter of Evaluation Due to COVID-19 Pandemic. West J Emerg Med. 2020; 21(3):538-41.
10. Stark N, Hayirli T, Bhanja A, Kerrissey M, Hardy J, Peabody ChR. Unprecedented training: Experiences of Residents During the Covid-19 Pandemic. Annals of Emergency Medicine. 2022; 79(5):488-94.
11. Shandro J, Kessler R, Schrepel C, Jauregui J. Advising Medical Students During COVID-19: The Case for a Single Emergency Medicine Rotation for All. AEM Educ Train. 2020; 4(3):318-20.
12. Beskind DL, Hiller KM, Stolz U, Bradshaw H, Berkman M, Stoneking LR, et al. Does the experience of the writer affect the evaluative components on the standardized letter of recommendation in emergency medicine?.   J Emerg Med. 2014; 46(4):544-50.
13. Girzadas DV, Harwood RC, Davis N, Schulze L. Gender and the council of emergency medicine residency directors standardized letter of recommendation. Acad Emerg Med. 2004; 11(9):988-91.
14. Boysen‐Osborn M, Andrusaitis J, Clark C, Saadat S, Billimek J, Paradiseet al. A retrospective cohort study of the effect of home institution on emergency medicine Standardized Letters of Evaluation. AEM Educ Train. 2019; 3(4):340-6.
15. Andrusaitis J, Clark C, Saadat S, Billimek J, Paradise S, Wray A, et al. Does Applicant Gender Have an Effect on Standardized Letters of Evaluation Obtained During Medical Student Emergency Medicine Rotations?. AEM educ train. 2019; 4(1):18-23.
16. Love JN, DeIorio NM, Ronan‐Bentle S, Howell JM, Doty CI, Lane DR, et al. Characterization of the Council of Emergency Medicine Residency Director’s Standardized Letter of Recommendation in 2011-2012. Acad Emerg Med. 2013; 20(9):926-32.