Introduction

Telemedicine or “Telehealth” has grown exponentially in the U.S. over the past few decades and current trends in the healthcare environment fuel this expansion into the future ( 1 , 2 ). Telehealth involves the use of telecommunications and virtual technology to deliver healthcare outside of the traditional brick and mortar healthcare facility. This platform has been shown to reduce inefficiencies in the delivery of healthcare by improving access for underserved and rural populations ( 3 , 4 ). Telehealth has also been shown to be cost-effective, improve health outcomes, and enhance patient social support ( 5- 8). Kansas, is one of the longest running practices of its kind worldwide. The practice began in 1995 and connected an oncologist at KUMC with a rural medical center in Hays, Kansas. Fifteen years later, the practice continues to thrive at Hays Medical Center and has also expanded to include two additional sites within the state-the Northeast Kansas Center for Health and Wellness in Horton and Goodland Regional Medical Center in Goodland-that offer regularly scheduled teleoncology clinics. While the KUMC practice has witnessed an expansion in service sites throughout its history, the practice has seen a significant decrease in the costs associated with providing such services since its inception. The cost decrease can, in part, be attributed to an increase in the number of teleoncology visits conducted through the practice since it began (In Fiscal Year 1995). Recently, telehealth has also proven to be rather dynamic as it has been applied in both primary care settings and for specialty consultations ( 9 ).

Despite the multiple innovative applications of digital technology in healthcare, telehealth is not readily adopted by all healthcare providers and many medical schools still do not include telehealth in their curricula ( 2 , 10 ). Additionally, the American Medical Association (AMA) recommends telehealth training of medical students and residents. However, a recent survey by the Association of American Medical Colleges (AAMC) found that only 58% of medical schools currently provide telehealth training ( 11 , 12 ).

Due to the new challenges faced by medical schools during the COVID-19 pandemic, novel curricula must be considered ( 13 , 14 ). Medical schools have been challenged with adapting and innovating new curricula aimed at creating clinical experiences for students that are remote from the hospital ( 15 ). To ensure the delivery of quality clinical education despite the pandemic, we created a Telehealth Education Curriculum (TEC) for the Obstetrics and Gynecology (Ob/Gyn) third year clinical rotation at New York Medical College, which was implemented in April 2020. The novel TEC included medical student telehealth encounters with real patients and a virtual OSCE (vOSCE) with standardized patients to resemble a telehealth encounter. We chose to incorporate an OSCE into the TEC curriculum as they have been shown to improve clinical skills in educational settings ( 16 ). Students were asked to complete surveys on their perceptions of the TEC and responses were used to assess its educational value.

The objective of this study was to measure student perceptions of a novel Telehealth Education Curriculum for the virtual obstetrics and gynecology core clerkship during the COVID-19 pandemic and evaluate best practices for future telehealth curriculum implementation. A secondary objective was to contribute to current medical education literature on telehealth curricula, and their utility, to continue finding new innovations in medical student education during the COVID-19 pandemic and beyond.

Methods

Study Design/ Participants

This prospective mixed methods study of 52 medical students was conducted at New York Medical College School of Medicine (NYMC SOM), located in Valhalla, New York. It is one of the nation’s oldest private health sciences universities (est. 1860) and provides a comprehensive educational program with the goal of developing well-rounded medical students who will become resilient, compassionate, and skilled physicians. The SOM is proud of its strong foundational science education, diverse affiliated clinical training sites, and the commitment of its faculty and administration to medical student education.

In light of the COVID-19 pandemic, which provided a significant barrier to undergraduate medical education, and specifically clinical education, the third year Obstetrics and Gynecology (Ob/Gyn) medical student core clerkship was converted to a four-week virtual format in April 2020. The students originally scheduled to complete the standard Ob/Gyn clerkship in April and May 2020 were automatically enrolled in the virtual Ob/Gyn clerkship. This put the onus on faculty to rapidly develop and implement a new virtual curriculum that allowed for clinical learning and interaction in this new digital era. This new curriculum featured didactic sessions, case-based presentations, and interactive learning workshops via Zoom. Didactic activities included presentation of video-based surgical cases, interactive board question review, and discussion of general obstetrics and gynecology topics. Additionally, a novel Telehealth Education Curriculum (TEC) consisting of virtual Observed Structured Clinical Examinations and actual telehealth patient encounters was implemented as part of the virtual Ob/Gyn third year clerkship. We evaluated the student perceptions of the novel TEC following their successful completion of the virtual Ob/Gyn clerkship.

To be considered eligible to participate in the study, the students were required to be in their third year at NYMC SOM, enrolled in the virtual Ob/Gyn clerkship, and participating in the virtual Ob/Gyn clerkship and TEC. Students were provided with explicit information explaining that their participation in surveys was voluntary and their responses would be utilized for research purposes only. They were also informed that responses would be de-identified and accessed only by the research team data handlers with no impact on their grading. Two authors, JM and LF, were responsible for all data handling and were not involved in any aspect of medical student grading for the clerkship. NL and JP were able to access student data after it was processed and fully de-identified without the ability to re-identify students given their roles at the School of Medicine. All study procedures were approved by and compliant with the New York Medical College Institutional Review Board (protocol #14323).

Novel Telehealth Education Curriculum (TEC) Design and Assessment

The novel Telehealth Education Curriculum (TEC) consisted of two components: the telehealth encounter and the virtual Observed Structured Clinical Examination (vOSCE). For the telehealth encounter, each student completed at least one telehealth visit via a 3-way video conference call with an Ob/Gyn patient and attending physician, before the end of the 4-week virtual clerkship. Following the telehealth visit, students completed a clinical note in the Subjective, Objective, Assessment, and Plan (SOAP) note structure and a survey entitled Telehealth Evaluation for 3^rd year Ob/Gyn Clerkship. Students then virtually met with their attendings for a telehealth debrief in which verbal and written feedback was provided.

The second component of the novel TEC was the virtual OSCE designed to mimic a telehealth patient encounter, conducted via Zoom Video Communications, Inc. The blueprint for the vOSCE included completion of a standardized patient encounter with a physical exam and history (20 minutes), a post-encounter note that included a differential diagnoses, assessment, and plan (15 minutes), followed by two debriefing sessions with faculty focused on communication skills (15 minutes) and clinical reasoning (40 minutes). Students also had the opportunity to collaborate with another student who had independently completed the same case, while individually writing their assessment. The standardized patients presented with a common obstetric and gynecologic complaint (i.e. primary amenorrhea, postmenopausal bleeding, antenatal visit). Faculty observed the vOSCEs, reviewed post-encounter assessments, and evaluated students using the following criteria: medical knowledge, interpersonal skills and communication, empathy, and development of a differential diagnoses. Following the vOSCE, students completed the Virtual OSCE Evaluation Form for the 3rd year Ob/Gyn clerkship to provide feedback about their experience with the vOSCE and its value as an educational tool.

Data Collection

To assess student perceptions of the novel TEC curriculum, students in the first two cohorts of the virtual Ob/Gyn third year clerkship were asked to complete two 4-point Likert surveys consisting of nine student perception questions and four free response questions. Possible answer choices for the Likert survey questions were as follows:1=Strongly agree, 2=Moderately agree, 3=Moderately disagree, 4=Strongly disagree. Free response questions for both surveys asked the following: 1) What was most successful about the session? 2) What was most challenging about the session? 3) What were my top takeaways from this session (clinical and communication skills based)? 4) Any Additional comments or suggestions for improvements?

The two surveys were distributed to students at the conclusion of their virtual Ob/Gyn clerkship. One survey corresponded to the telehealth encounter and the other to the virtual OSCE. Responses were de-identified upon completion and stored in a secure database.

Analysis

We conducted all analyses using the Statistical Package for the Social Sciences (SPSS; IBM version 19). Likert data were analyzed by computing means, and response distributions, for each question to compare self-reported student agreement with survey statements.

For qualitative analysis of free response questions, the research team reviewed all qualitative data individually to identify salient themes. The research team then met as a group to resolve discrepancies in interpretations and reach final theme interpretation. Qualitative data was reviewed until no new themes emerged from the analysis. Illustrative quotes were collected to represent all salient themes.

Results

A total of 52 students participated in the virtual Ob/Gyn clerkship with 25 in the first cohort and 27 in the second. The telehealth encounter component of the TEC was still being piloted with the first cohort of students, so their participation in the telehealth encounter was optional. Based on faculty availability and student interest, 9 out of the 25 students in the first cohort completed the telehealth encounter. However, all 25 participated in the vOSCE because it was not optional. All 27 students in the second cohort completed both components of the TEC: the telehealth encounter and vOCSE. Of the 52 students who participated in the first two cohorts of the virtual Ob/Gyn clerkship, 36 completed the Telehealth encounter and 52 completed the vOSCE. Of these students, 33 completed the Telehealth survey (92%) and 48 completed the vOSCE survey (92%) (Tables 1 and 2).

Student responses to the Telehealth survey revealed that 76% strongly or moderately agreed that the educational value of a telehealth patient interview was similar to an in-person patient interview. Additionally, 97% strongly or moderately agreed that their telehealth encounter provided them with a valuable patient interaction. When asked if the telehealth encounter should continue during the COVID-19 pandemic, 97% strongly or moderately agreed. However, when asked if the telehealth encounter should be incorporated into the Ob/Gyn clerkship curriculum permanently, when the pandemic comes to an end, 82% strongly or moderately agreed.

Student answers to free response questions on the Telehealth survey revealed key themes for the first three out of four questions. When asked “What was most successful about the session?” students mostly reported: getting virtual clinical exposure to “real patients” during a pandemic, practicing Ob/Gyn history taking, and being taught by knowledgeable attendings who helped develop clinical reasoning skills. When asked “What was most challenging about the session?” most students identified: technical challenges, inability to physically examine the patient, and difficulty communicating nonverbally. When asked “What were my top takeaways from this session?” most students reported: telehealth visits are similar to in person visits, telehealth visits are a valuable tool and save time for patients, communication skills are important in a virtual format, and time management is important when acquiring patient histories virtually (Table 3).

Medical student responses to the vOSCE survey revealed that 83% strongly or moderately agreed the vOSCE standardized patient experience provided a valuable patient interaction. When asked if the vOSCE session increased their confidence navigating a patient interview using a virtual platform, 90% of students strongly or moderately agreed. Additionally, 83% strongly or moderately agreed that the vOSCE is a valuable tool to train students in telemedicine. Finally, 65% of respondents strongly or moderately agreed the vOSCEs should be part of the Ob/Gyn clerkship going forward, including after the COVID-19 pandemic.

Student answers to free response questions on the vOSCE survey again revealed key themes for the first three out of four questions. When asked “What was most successful about the vOSCE session?” students mostly reported: increased comfort and familiarity with the virtual encounter format, opportunity to experience Ob/Gyn patient encounters during COVID-19, and collaboration with peers. When asked “What was most challenging about the session?” students identified: technical difficulties/adjusting to a virtual format, the virtual physical exam, and difficulty connecting with patients given limited ability to read body language and communicate nonverbally. When asked “What were my top takeaways from this session?” students mostly reported: the vOSCE was helpful for practicing Ob/Gyn history taking and developing a differential diagnosis, ability to experience and practice using telemedicine, telemedicine is a good tool for student learning, and practice with nonverbal communication (Table 4).

Discussion

When the COVID-19 pandemic hit the Tristate Area and students were dismissed from in person clinical rotations, faculty and student researchers in the Department of Obstetrics & Gynecology at New York Medical College drafted and implemented a virtual Ob/Gyn clerkship and novel Telehealth Education Curriculum (TEC) in a matter of weeks. After the first two cohorts of students completed the virtual Ob/Gyn clerkship and novel TEC, we assessed student perceptions of the TEC. In general, most students agreed that the telemedicine encounters provided similar educational opportunities as in-person encounters and were a valuable clinical tool that should be incorporated into future Ob/Gyn clerkship curricula, during and after the pandemic. Furthermore, student responses were both positive and thankful for the chance to continue their learning and experience using an emerging healthcare platform in the midst of a rapidly changing learning environment. While students indicated initial difficulty adjusting to a virtual format and communicating with fewer nonverbal cues, most ultimately expressed increased comfort using telehealth after completing the TEC.

The information presented in this study including the tools used to implement the curriculum and format by which student perceptions were assessed may be utilized by other institutions interested in developing a telehealth curriculum for their students. While some US medical schools have created virtual student rotations, or exams, and proven them effective for clinical learning, few have incorporated focused teaching on the practice of telehealth itself ( 17- 19). Proceeding with assessment to determine competency for graduation from medical school, and maintaining performance standards for graduating doctors is an unprecedented challenge under pandemic conditions. This challenge is hitherto uncharted territory for medical schools and there is scant guidance for medical educators. In early March 2020, Duke-National University Singapore Medical School embraced the challenge for ensuring competent final year medical students could complete their final year of studies and graduate on time, to enter the medical workforce in Singapore without delay. This paper provides details of how the final year clinical performance examinations were planned and conducted during the COVID-19 pandemic. The aim of the paper is to provide guidance to other medical schools in similar circumstances who need to plan and make suitable adjustments to clinical skills examinations under current pandemic conditions. The paper illustrates how it is possible to design and implement clinical skills examinations. Our study findings are consistent with the few other studies reporting on medical student telehealth education including increased student comfort using telehealth following the curriculum and student agreement about the value of telehealth education ( 20 , 21 ). Patient interest in and satisfaction with telehealth are high. No comprehensive U.S. undergraduate medical education curriculum teaching telehealth University of the Health Sciences (USU). Other studies also reveal initial student discomfort using telehealth platforms, student-reported technological issues, and student agreement that telehealth provides similar learning opportunities compared to in-person encounters ( 20 - 22 ). Interestingly, our students identified challenges communicating with patients nonverbally and expressing empathy via a virtual platform: “rapport building and demonstrating empathy must be even more intentional on a virtual platform because subtleties can be missed.” However, with instruction and continued practice, students were able to overcome this obstacle,  and learned “there is a different way of expressing empathy in these interactions and there’s never not an opportunity to show empathy and compassion to a patient. ” The need for effective communication adapted for the virtual healthcare setting has been recognized but further studies in the clinical setting are necessary ( 23 , 24 ). Lastly, to our knowledge, our study is the first to evaluate medical student desire for telehealth training during and after the COVID-19 pandemic. Thus, the present study underscores the importance of training future physicians to use this ever evolving and increasingly popular medical technology.

In recent years, telehealth has been increasingly employed to deliver quality healthcare to diverse patient groups ( 2 , 8 ). Pre-COVID-19 pandemic trends supported its continuous growth but the current health crisis has accelerated the rate at which healthcare facilities are adopting and utilizing telehealth across practice sites ( 25 , 26 ). This rapid integration is unlikely to be a fad. Evidence suggests that following the pandemic, telehealth will prevail given its copious benefits for patients and providers alike ( 27- 29). Because telehealth is here to stay as another healthcare delivery modality, it is important to train future providers to successfully use this interface ( 10 , 30 ). In fact, many medical students are already expressing interest in telehealth education and have plans to use telehealth in their future careers as physicians ( 20 , 31 , 32 ). By delivering telehealth training to medical student, our novel TEC provides an opportunity to bridge a much-anticipated knowledge gap between future physicians and medical technologies they will be expected to know and use. Overall, the COVID-19 pandemic provided an educational incentive to address the already existing need for medical student telehealth education through innovative educational curricula.

Conclusion

Our data strongly support an urgent need for current and future medical student telehealth education curricula. Still, our study presents with some limitations including a modest sample size. Additionally, we solely evaluated medical students enrolled in the third-year obstetrics and gynecology rotation during the COVID-19 pandemic at a single institution. Therefore, more studies evaluating medical student perceptions of telehealth education curricula are needed to examine if our findings and conclusions can be extrapolated to other fields of medical education. Student free responses indicate areas for continued improvement when implementing telehealth curricula including a demand for more telehealth encounters, more vOSCE cases and virtual SP feedback, and follow up telehealth visits. Other than these area of improvement, medical students indicated overall great satisfaction with our novel TEC.

Given how well received our TEC was at New York Medical College during the COVID-19 pandemic and that students expressed interest in continued telehealth learning, we plan to incorporate the TEC into future clerkship curricula. Currently, the third year Ob/Gyn clerkship is split between virtual and in-person learning. We plan to integrate student feedback from this pilot project by including more telehealth encounters and modify/adapt the curriculum according to emerging student needs and healthcare trends as appropriate.

Funding/Support:

The authors received no specific funding for this work.

References

1. Conde JG, De S, Hall RW, Johansen E, Meglan D, Peng GCY. Telehealth innovations in health education and training. Telemedicine and e-Health: Germany; 2010.
2. Waseh S, Dicker AP. Telemedicine training in undergraduate medical education: Mixed-methods review. J Med Internet Res. 2019; 21(4)
3. Nelson R. Telemedicine and telehealth: The potential to improve rural access to care. American Journal of Nursing. 2017; 117:17-8.
4. Francis E, Kraschnewski J, Hogentogler R, Buckner K, Sabol J, Bowers K. A Telehealth Approach to Improving Healthcare to Rural and Underserved Populations. J Clin Transl Sci. 2020; 4(s1)
5. Doolittle GC, Spaulding AON, Williams AR. The decreasing cost of telemedicine and telehealth. Telemed J E Health. 2011; 17(9)
6. Soegaard Ballester JM, Scott MF, Owei L, Neylan C, Hanson CW, Morris JB. Patient preference for time-saving telehealth postoperative visits after routine surgery in an urban setting. Surg (United States). 2018; 163(4):672-9.
7. Campbell SB, Erbes C, Grubbs K, Fortney J. Social Support Moderates the Association Between Posttraumatic Stress Disorder Treatment Duration and Treatment Outcomes in Telemedicine-Based Treatment Among Rural Veterans. J Trauma Stress. 2020; 33(4):391-400.
8. Sly J, Clarke B, Sanabria A. Telemedicine Tools for Patients and Providers: Systematic Review. Iproceedings. 2018; 4(2):e11866.
9. Nandra K, Koenig G, DelMastro A, Mishler EA, Hollander JE, Yeo CJ. Telehealth provides a comprehensive approach to the surgical patient. Am J Surg. 2019; 218(3):476-9.
10. Pourmand A, Ghassemi M, Sumon K, Amini SB, Hood C, Sikka N. Lack of Telemedicine Training in Academic Medicine: Are We Preparing the Next Generation?. Telemed e-Health. 2020; 27(1):62-7.
11. Kelly J. AMA encourages telemedicine training for medical students, residents [Internet]. American Medical Association. 2016 [cited 2020 Oct 14]. Available from: https://www.ama-assn.org/press-center/press-releases/ama-encourages-telemedicine-training-medical-students-residents.
12. Warshaw R. From Bedside to Website: Future Doctors Learn How to Practice Remotely [Internet]. AAMCNEWS. 2018 [cited 2020 Oct 14]. Available from: https://www.aamc.org/news-insights/bedside-webside-future-doctors-learn-how-practice-remotely.
13. Rose S. Medical Student Education in the Time of COVID-19. JAMA, J Am Med Assoc. 2020; 323(21):2131-2.
14. Goldenberg MN, Hersh DC, Wilkins KM, Schwartz ML. Suspending Medical Student Clerkships Due to COVID-19. Med Sci Educ. 2020; 30(3):1273-6.
15. Whelan A. Updated Interim Guidance for Medical Students’ Participation in Patient Care During the Coronavirus (COVID-19) Outbreak [Internet]. Aamc. 2020 [cited 2020 Oct 14]. Available from: https://www.aamc.org/news-insights/press-releases/updated-interim-guidance-medical-students-participation-patient-care-during-coronavirus-covid-19.
16. Joolaee H, Alizadeh M, Fallahzadeh MH, Baseri A, Sayadi M. Evaluation of Clinical Skills of Family Physicians in Fars Province by Means of Observed Structured Clinical Evaluation. Strides Dev Med Educ. 2011; 7(2):92-8.
17. Boursicot K, Kemp S, Ong TH, Wijaya L, Goh SH, Freeman K, et al. Conducting a high-stakes OSCE in a COVID-19 environment. MedEdPublish. 2020; 9(1)
18. Hannon P, Lappe K, Griffin C, Roussel D, Colbert-Getz J. An objective structured clinical examination: From examination room to Zoom breakout room. Med Educ. 2020; 54(9):861.
19. Durfee SM, Goldenson RP, Gill RR, Rincon SP, Flower E, Avery LL. Medical Student Education Roadblock Due to COVID-19: Virtual Radiology Core Clerkship to the Rescue. Acad Radiol. 2020; 27(10):1041-6.
20. Jonas CE, Durning SJ, Zebrowski C, Cimino F. An Interdisciplinary, Multi-Institution Telehealth Course for Third-Year Medical Students. Acad Med. 2019; 94(6):833-7.
21. Abraham HN, Opara IN, Dwaihy RL, Acuff C, Brauer B, Nabaty R, et al. Engaging Third-Year Medical Students on Their Internal Medicine Clerkship in Telehealth During COVID-19. Cureus. 2020; 12(6)
22. Weber AM, Dua A, Chang K, Jupalli H, Rizwan F, Chouthai A, et al. An Outpatient Telehealth Elective for Displaced Clinical Learners during the Coronavirus Pandemic. Medrxiv. 2020; 3:2-17.
23. Liu C, Lim RL, McCabe KL, Taylor S, Calvo RA. A web-based telehealth training platform incorporating automated nonverbal behavior feedback for teaching communication skills to medical students: A randomized crossover study. J Med Internet Res. 2016; 18(9):246.
24. Begasse de Dhaem O, Bernstein C. Headache Virtual Visit Toolbox: The Transition From Bedside Manners to Webside Manners. Headache. 2020; -60:1743-6.
25. Loeb AE, Rao SS, Ficke JR, Morris CD, Riley LH, Levin AS. Departmental Experience and Lessons Learned With Accelerated Introduction of Telemedicine During the COVID-19 Crisis. The Journal of the American Academy of Orthopaedic Surgeons. 2020; 28(11):469-76.
26. Lee S, Hitt WC. Clinical Applications of Telemedicine in Gynecology and Women’s Health. Obstetrics and Gynecology Clinics of North America. 2020; 47(2):259-70.
27. Camhi SS, Herweck A, Perone H. Telehealth Training Is Essential to Care for Underserved Populations: a Medical Student Perspective. Med Sci Educ. 2020; 30(3):1-4.
28. Iancu AM, Kemp MT, Alam HB. Unmuting medical students’ education: Utilizing telemedicine during the COVID-19 pandemic and beyond. Journal of Medical Internet Research. 2020; 22(7):19667.
29. Wosik J, Fudim M, Cameron B, Gellad ZF, Cho A, Phinney D, et al. Telehealth transformation: COVID-19 and the rise of virtual care. Journal of the American Medical Informatics Association. 2020; 27(6):957-62.
30. Pit SW, Bailey J. Medical students’ exposure to, knowledge and perceptions of telehealth technology: is our future workforce ready to embrace telehealth service delivery?. Heal Educ Pract J Res Prof Learn. 2018; 1(2):73-87.
31. Theoret C, Ming X. Our education, our concerns: The impact on medical student education of COVID-19. Med Educ. 2020; 57(4):591-2.
32. Rhone K, Lindgren J, Mack LJ, Nelson ME, Spencer L, Anderson SM. Avera eCARE: Medical Student Education in Telemedicine. Telehealth Med Today. 2020; 5