Introduction

During the Covid-19 pandemic, medical schools have had to transform medical education instructional content for virtual delivery in online learning environments, and have been evaluating their curricula using the Association of American Medical Colleges (AAMC) recommendations [1, 2]. While contingency planning is an important strategy for abrupt changes in curricula, the infrequency of pandemics and other natural disasters invariably leads to inadequate preparedness [3, 4]. In addition to delivering the pre-established medical curriculum, students should also receive education and training relevant to the underlying circumstances precipitating the disruption to the typical methods of instruction [5, 6]. Virtual learning, defined as the delivery of learning materials and instruction using the internet, has previously been used as a solution in resource-constrained countries and in response to disasters such as the Ebola epidemic [7, 8]. Blended learning combines virtual education with traditional classroom-based education and is preferred for curricular elements that cannot be substituted with virtual components [9,10,11]. Given social distancing restrictions during the COVID-19 pandemic, instruction using virtual and blended models became the educational methods of choice for pre-clinical students.

There is literature about the implications of this major shift in instructional methodology in the context of medical education [1, 5, 12,13,14,15,16,17,18,19,20,21,22]. Within this literature, there is indication that students view the flexibility afforded by virtual adaptation of the curriculum positively. However, virtual learning could negatively impact student connectedness [12, 18, 23, 24]. Student connectedness encompasses establishing social relationships and integration, which impact academic success, wellness, and the ability to manage stress especially for beginning undergraduate students [15, 25].

The specific aims of our study were to determine the impact of the COVID-19 pandemic on pre-clerkship students at the Texas Tech University Health Sciences Center at El Paso (TTUHSC El Paso) Paul L. Foster School of Medicine (PLFSOM). This study included a survey that was conducted with first- and second-year medical students (MS1s and MS2s) to determine: (1) perceptions and attitudes about overall wellness and connectedness; (2) perceptions about and satisfaction with virtual and blended curricular components; and (3) satisfaction with instructional coverage relevant to the COVID-19 pandemic. The study also compared satisfaction scores from course evaluations of the curricular unit that was delivered online (2021) as a response to the pandemic, with course evaluations during the previous academic years. Finally, the study compared student performance on the unit summative exam in the first full academic unit delivered in the virtual environment to historic performance in the same unit by previous cohorts. By highlighting differences between first and second-year students’ experience with virtual and blended instruction, this study can help inform educators who are designing future learning models.

Method

Environment

The pre-clerkship curriculum at TTUHSC El-Paso PLFSOM spans 2 years and consists of 10 integrated, systems-based learning units. Each week of each academic unit is taught within the framework of a common clinical presentation that illustrates essential basic and clinical knowledge. Both the basic science and clinical skills components are taught within a weekly integrated instructional framework. In addition, students are assigned to four individual learning communities (colleges) [26].

The pre-clerkship phase includes the following core courses: (1) Scientific Principles of Medicine (SPM), which consists of lectures, team-based learning, and laboratory-based sessions (e.g., Anatomy); (2) Medical Skills (MS), which incorporates standardized patient encounters that correspond to SPM content, learning medical examination skills, and small group discussions; (3) College Colloquium, which is a discussion-based course covering topics on humanities, ethics, economics, etc.; and (4) Society, Community and the Individual (SCI), which covers epidemiology and biostatistics, community engagement, and medical and conversational Spanish instruction.

In response to the COVID-19 pandemic, virtual synchronous and asynchronous activities were utilized to deliver all pre-clerkship instruction, except for the Medical Skills which was converted to a blended modality [9,10,11]. In the Medical Skills course, physical examination skills were taught utilizing both online video conferencing and face-to-face instructional methods, while enforcing social distancing measures in live settings. In addition, encounters with standardized patients were adapted into virtual experiences (Table 1).

Table 1 Pre-clerkship phase curriculum modes of instruction delivery before and after the COVID-19 pandemic
Full size table

Study Design

To evaluate the efficacy of the pre-clerkship curriculum during the COVID-19 pandemic, we utilized a mixed-method research design, including both quantitative and qualitative research methods. Within this mixed-methods approach, the quantitative methodology followed a survey design approach, seeking to describe the status of a variable or phenomenon [27]. The qualitative component of this study utilized a case study approach, an empirical inquiry about a contemporary phenomenon, set within its real-world context [28]. In addition, direct measures of student course evaluations and performance were compared to previous cohorts. The study was reviewed and deemed exempt by the institution’s Institutional Review Board.

Participants

The study was conducted at TTUHSC El Paso PLFSOM, an accredited allopathic medical school located in El Paso, TX. Study participants were 223 medical students, enrolled in the pre-clerkship phase of the PLFSOM medical education program (Years 1 and 2) during Academic Year 2020–2021. This represents students from two medical school cohorts (Classes of 2023 and 2024), including 115 Year 1 students (MS1s) and 108 Year 2 students (MS2s).

Instruments

The following instruments were utilized in the collection of quantitative and qualitative data:

Survey

Student perceptions regarding the impact of virtual and blended learning, student wellness, and topical coverage of SARS-CoV-2/COVID-19 in the curriculum were examined with a survey designed to provide ratings of criteria using modified Likert scales and narrative responses to open-ended questions. The survey (Appendix 1) was developed based on prior TTUHSC EP survey designs, and was reviewed by faculty to optimize face validity, as described in prior medical student survey studies [29].

Course Evaluations

Course evaluations are routinely collected as part of the ongoing evaluation of the curriculum at the PLFSOM. Course evaluations are designed by the responsible medical school faculty and are administered to all students at the time of course completion. Course evaluation items pertaining to the overall satisfaction with the course were included in the analysis. Responses were on a 6-point Likert-type scale and ranged from “Strongly Disagree” to “Strongly Agree.”

Summative Exam Scores

Summative exams consist of National Board of Medical Examiners (NBME)-style questions for each unit (e.g., 150 multiple-choice questions for the unit). The exam questions were written by the medical school faculty based on the educational objectives that were taught during the corresponding unit.

Procedure

Survey

We utilized a cross-sectional survey instrument approach, assessing student perceptions in two student cohorts enrolled in the medical education program. Surveys were administered in November of 2020, midway through the first semester for MS1s and MS2s. A follow-up survey was administered in the summer of 2021. Students were invited to participate in the survey via email. The survey was hosted on an online survey software platform (Qualtrics, Provo, UT) [30]. The survey was anonymous and optional. To maintain confidentiality, student information was de-identified during data analyses.

We performed descriptive analyses, generating counts, frequency percent, and means. For Likert scale responses, we combined positive ratings (somewhat agree, agree, or strongly agree) into an overall agreement percentage. To compare differences between first- and second-year medical students, we used Fisher’s exact test with a p value < 0.05 to show statistical difference. We used GraphPad Prism for all statistical analyses [31].

Qualitative analyses were conducted on narrative answers resulting from open-ended survey items to identify major themes in student feedback and perceptions. To achieve interpreter reliability, two members of the research personnel who are trained in qualitative methods analyzed and coded qualitative data generated. They reviewed the open-ended responses independently, utilizing an inductive method for narrative analysis to code student responses [32]. Codes were generated by following the six phases described by Braun and Clarke in 2006 [33]: (1) familiarizing with the data; (2) writing initial codes; (3) searching for themes; (4) reviewing themes; (5) defining themes; and (6) writing the analysis. This was an iterative process which involved moving back and forth among the phases described and peer debriefing to reach consensus [34]. The credibility of analysis was enhanced by having two research team members analyze the data and using a systematic reiterative process.

Course Evaluations

All medical students are required to complete curricular course evaluations during the pre-clerkship phase of the PLFSOM medical program at the end of each academic unit. Analysis of data pertaining to the overall satisfaction with the Renal System unit was performed. This particular academic unit was chosen as it was the first to be delivered remotely for the Class of 2022. Student’s overall satisfaction was compared to historic cohorts using one-way ANOVA.

Student Exam Performance

To determine the effect of the change in instructional delivery of the first SPM unit taught virtually (Renal System unit) in April 2020, student performance on the unit summative examination for the Class of 2023 was analyzed. Students in this course took a National Board of Medical Examiners (NBME)-style exam for each unit. The final grade for each academic unit is solely determined by the performance on this examination, with the passing score set at 65%. Exam performance is reported as the mean summative percent performance for all students who took the exam in each cohort. The range (i.e., maximum and minimum) scores are included for each group.

We compared the mean score of the Class of 2024, which received virtual instruction, with the mean score achieved by the previous academic year’s cohort (Class of 2023), which received in-person instruction. We performed two-tailed unpaired Student’s t-tests to compare mean performance between the two cohorts. Statistical significance was set at a p-value less than 0.05. The exam format changed from NBME-style questions that were written by faculty to questions from the NBME question-bank for the pre-clerkship students in August 2020. As a result, only the Renal unit which was delivered completely virtually in April 2020 had the same exam format as previous years, and as a result was used for comparison.

Results

Survey Quantitative Results

Demographic information was completed by 127 students in the initial survey (Table 2). There were approximately 125 out of 223 students (56%) who completed the first section of the survey which asked about their overall experience with virtual instruction. Respondents consisted of 65 of 115 (57%) MS1s and 60 of 108 (56%) MS2s. Survey results revealed the majority of students reported changes in their learning experience. Specifically, the majority of students (78%) reported having to change their study methods, experiencing technical issues (85%), and having difficulty communicating with faculty (62%) during the pandemic. Students also reported the virtual environment negatively affected their social skills (77%), connectedness to peers (89%), professional development (62%), and sense of wellness (62%). When comparing first- and second-year students, MS1s were significantly more likely than MS2s to report difficulty adjusting to virtual instruction (p = 0.037), negative effects on their learning skills (p = 0.005), and negative effects on academic performance (p = 0.003). MS1s were also significantly more likely than MS2s to perceive a negative effect on their sense of wellness (0.002) (Table 3).

Table 2 Demographics
Full size table
Table 3 Pre-clerkship medical students experience during the COVID19 pandemic. Sixty-five of MS1s and 60 of MS2s responded to survey questions pertaining to their pre-clerkship experience in the initial survey in the fall of 2020. Forty-two of 115 (37%) MS1s and 25 of 108 (23%) MS2s responded to the follow-up survey in the summer of 2021. Questions that were not asked on the follow-up survey are blank
Full size table

Sixty-seven students (30%) completed a short follow-up survey regarding their experience during the second semester at the end of the year. Respondents consisted of 42 of 115 (37%) MS1s and 25 of 108 (23%) MS2s. In the follow-up survey, students continued to report technical issues (84%), having difficulty communicating with faculty (73%), and having difficulty with adjusting to virtual instruction (49%). They also reported a negative effect on their professional development (58%), and academic performance (45%). MS1s were significantly more likely than MS2s to report a negative effect in their connectedness to peers (p = 0.044) and sense of wellness (p = 0.001). There were no significant differences in responses for each class between the first and follow-up surveys.

In-person vs. Virtual Course Sessions

Students were asked if the virtual activities for the various courses were adequate for their learning followed by a question asking if they would have preferred in-person activities. For basic science sessions, most students (80%) attended asynchronously and only 26% of students found synchronous sessions useful; though, 46% would have preferred in-person lectures. For team-based sessions covering case-based scenarios, only 23% of students found the virtual sessions adequate for their learning and 57% would have preferred in-person activities. For topics covered in the Society, Community, and the Individual (SCI) Course (Epidemiology, Biostatistics, Spanish, etc.), 52% found virtual sessions adequate and 55% of students preferred virtual sessions. Overall, there was no significant difference between first- and second-year medical students’ responses regarding in-person compared to virtual sessions.

Colloquium Course

Sixty-five (57%) of MS1s and 59 (55%) of MS2s responded to survey questions pertaining to the Colloquium course. Twenty-eight of 65 (43%) MS1s and 39 of 59 (66%) MS2s found virtual sessions adequate to address ethical topics. Twenty-four of 65 (37%) MS1s and 31 of 59 (53%) MS2s found virtual sessions adequate to establish a dynamic discussion. Thirty-seven of 65 (57%) MS1s and 26 of 59 (44%) MS2s found virtual sessions inadequate to promote social connectedness. Results also revealed MS1s were less likely than MS2s to engage in the online discussion using the chat box function (p = 0.004). Overall, seventy percent of students would have preferred the Colloquium course to have been delivered in person (face-to-face) (Table 4).

Table 4 Colloquium experience. Sixty-five of MS1s and 59 of MS2s responded to survey questions pertaining to the Colloquium course
Full size table

Blended Medical Skills Course

Sixty-four of 115 (57%) MS1s and 58 of 108 (54%) MS2s responded to survey questions pertaining to the Medical Skills course. Six of 64 (9%) MS1s and 17 of 58 (29%) MS2s agreed that learning physical examination skills through videoconferencing was adequate for their learning (p < 0.002). Forty-one of 64 (64%) MS1s and 49 of 58 (84%) MS2s agreed that virtual SP encounters were adequate for their learning (p < 0.020).

Pandemic-Related Course Content

Eighty-seven of 125 (70%) first- and second-year medical student respondents agreed that the curriculum addressed important ethical and social issues related to the COVID-19 pandemic, as well as societal and public health issues in the Colloquium and SCI courses (61%). Given that the respiratory system is covered during the second half of the first year, the survey for the second-year students enquired about inclusion of learning activities relevant to the pandemic. The majority (80%) of second-year medical students agreed that the Medical Skills course included important clinical activities addressing COVID-19. Thirty-six percent of students agreed that SPM included important basic science information.

Survey Narrative Results

Students were asked to describe the impact of the pandemic on their studying and learning and how the virtual learning experience affected them, and had an opportunity to suggest areas for improvement. Open-ended responses were received from a total of 271 students, which included 147 MS1s and 124 MS2s. Qualitative analysis of these responses revealed several themes.

A major theme that emerged from responses provided by both MS1s and MS2s was that they preferred learning the course material asynchronously. Students described that this allowed them to self-pace their studying, in the comfort of their homes. It reduced commute time, which also saved money, and allowed more time for family and self-care. For example, students preferred switching “live” team-based sessions to asynchronous learning modules. Only MS2s experienced in-person team-based sessions prior to the COVID-19 pandemic and reflected that “live” sessions were not as useful as when they were in-person. Students preferred having the option to pause the material or fast forward and disliked the slow pace of mandatory “live” sessions. Students also indicated preference for in-person components of the Medical Skills course. Students reported technical and connectivity issues, and lack of a dedicated home study environment as negative aspects of virtual learning. Students perceived the lack of social interaction negatively on their wellness, as well as their learning skills, by limiting opportunities to study in groups and direct communication with faculty.

To improve the virtual learning experience, students suggested decreasing the number of mandatory synchronous sessions, training faculty in online learning platforms, and tailoring the material to be more conducive for independent learning. Main themes from the qualitative analysis are presented according to their frequency in Tables 3 and 4. Verbatim examples are provided alongside each theme (Table 5).

Table 5 Main themes from 147 first-year medical students’ (MS1s) and 124 s-year medical students’ (MS2s) responses
Full size table

Course Evaluations

Analysis of student perceptions with course evaluations revealed no significant differences in the students’ satisfaction with the virtual and blended components compared to previous years, in which courses were delivered face-to-face. Table 6 provides a summary of students’ satisfaction levels. Percentages represent combined satisfaction levels (responses ranging from “somewhat satisfied” to “very satisfied”) for four class cohorts.

Table 6 Student satisfaction with renal system unit courses by academic year
Full size table

Academic Performance

The effect of virtual instruction on academic performance was determined by comparing the summative exam scores in the renal system unit for the Class of 2023 (virtual) versus the Class of 2022 (in person). Data is presented as means ± maximum and minimum values. Passing threshold was set at 65%. The overall performance for the Class of 2023 was significantly lower than that for the previous class cohort, with a mean score of 76.74 compared to a mean score of 80.07 for the Class of 2022 (p = 0.0066) (Fig. 1).

Fig. 1
figure 1

Unit summative exam performance by delivery mode. The effect of virtual instruction on academic performance was determined by comparing the summative exam scores in the renal system unit for the Class of 2023 (virtual) versus the Class of 2022 (in person). Data is presented as means ± maximum and minimum values. Passing threshold is set at 65%

Full size image

Discussion

This study included an evaluation process of the impact of the pandemic on first- and second-year medical students (MS1s and MS2s) at TTUHSC El-Paso PLFSOM through analyses of a student survey, course evaluation data, and academic performance.

Other studies have examined students’ perceptions of the effects of the pandemic on the pre-clerkship curriculum, and similarly demonstrated satisfaction with the increased flexibility afforded by virtual instruction, along with dissatisfaction due to reduced opportunities to practice medical skills and participate in laboratory-based sessions such as anatomy [1, 12, 14, 17, 21]. Our results additionally show that MS2s, who experienced the majority of their first year in person prior to transitioning to the blended model, were able to adapt better to the change in delivery mode. In contrast, MS1s, who from the beginning of medical school did not have the opportunity to connect in person with peers and faculty, had more difficulty adapting to the conditions imposed by the pandemic.

MS2s were less likely to perceive a negative impact on their academic performance and sense of wellness. Overall, data from MS2s indicated higher satisfaction. This outcome possibly reflects that MS2s viewed the efficiency in time management afforded by the pandemic with a more positive attitude than MS1s. In addition, it is possible that MS2s who did not regularly attend class during their MS1 year had already honed their virtual study skills, thus making the transition to the virtual asynchronous environment easier. MS2s commented that they were more sophisticated in identifying strengths and weaknesses in their study habits and adjusted accordingly. By contrast, MS1s had little experience in adapting to the workload of a medical curriculum and optimizing their study habits. Thus, the added time available under pandemic conditions was less effective toward promoting self-directed learning than for MS2s.

Students’ perceptions of the learning environment and virtual instruction showed similar trends for both MS1 and MS2 survey respondents. Students were evenly split between preferring in-person vs. virtual sessions for basic science topics, epidemiology, biostatistics, and Spanish. Students preferred in-person sessions for Anatomy and learning Medical Skills, discussion-based and team-based activities. Students described mandatory synchronous team-based activities as slow, and experienced technical difficulties. Narrative results showed that some students preferred interactive online modules over “live” virtual sessions. As a result, after returning to in-person instruction, PLFSOM incorporated both asynchronous interactive modules supplemented with in-person team-based activities.

Moving the curriculum into a virtual environment greatly reduced interactions among students and between students and faculty for both MS1s and MS2s, which is consistent with previous studies [12, 20, 21]. More than half of the MS1s responded that virtual instruction had a negative impact on their feelings of connectedness. A similar difference between the groups was seen with respect to their sense of wellness. A possible explanation for the disparity in responses between the MS1s and the MS2s for both connectedness and wellness could be the fact that MS2s had experienced most of their first year under normal social conditions, including in-person classes and involvement with their learning communities (Colleges), along with a more normal spectrum of non-school social interactions [35]. By contrast, MS1s lacked the socialization aspect of medical school that usually occurs during the first year, and were likely more impacted by the isolation imposed by the pandemic.

A blended model for the Medical Skills component of the curriculum was feasible during the pandemic, and results of our study indicated that this model did not significantly change the overall student satisfaction with the course. Student perceptions of virtual standardized patient encounters were more favorable than learning physical examination skills through videoconferencing. Our results further revealed MS1s are less likely than MS2s to find the activities conducted in the virtual environment adequate for their learning. This possibly reflects that MS2s had the opportunity to learn many of the basic exam skills in-person prior to the pandemic and prior to transitioning to the blended format, which enabled them to better adapt.

There were positive components to the change in curriculum delivery that were identified from the thematic analysis of the students’ open-ended survey answers. Virtual learning provided flexibility in schedules and the option to review recorded lectures. It reduced commute time, saved money, and allowed more time for family and self-care. There may be a benefit to maintaining the flexibility of virtual learning in the future, if there are opportunities for students to establish social connections. Students benefit from the availability of pre-recorded lectures, combined with live, optional, question-and-answer sessions during which students can join and interact with the faculty [19]. On the other hand, it is important to understand the challenges and potential barriers to achieve wide adoption of these strategies. Virtual instruction demands that educators revise their teaching materials in a way that is more conducive to support independent learning, and in our study, students clearly identified the need for faculty training in this area. To this end, more interactive and engaging materials that follow best practices in instructional design and development need to be created. Thus, educators need to be trained in the effective use of different technologies and platforms to create successful remote learning experiences.

Student academic performance, determined by exam scores for the first SPM unit delivered virtually, demonstrated a negative impact. Compared to the previous academic year, the Class of 2024 had lower exam scores. This may have resulted from the abrupt transition to virtual delivery and the psychological impact of the COVID-19 pandemic. Moreover, as a direct result of the stay-at-home orders, students had to take their exam at their place of shelter and under remote proctoring conditions. This is consistent with the results reported by Andersen et al. which showed that MS1s had lower exam scores in the first semester of the COVID-19 pandemic [20]. However, other studies demonstrated that exam scores did not show significant differences for medical students during the COVID-19 pandemic [21, 22].

There are limitations to this study. This survey was conducted at a single institution in the 2020–2021 academic year. Additional surveys and focus groups would be needed to evaluate the ongoing impact of the pandemic on this student cohort’s educational experience. The response rate was 56% for the initial survey and only 30% on follow-up, which may have skewed results, either positively or negatively, toward individuals who are more likely to voice their perceptions about the effect of the pandemic on the curriculum. Variable number of students answered different sections in the survey likely due to the lengthiness of the survey. The initial and follow-up survey did not track student survey responses to determine if there is overlap. Student academic performance was only compared for the first academic unit that was delivered virtually and did not include additional units given the change in the exam format in August 2020. Lower exam scores in our cohort of students could have been due to factors that were not accounted for in this study. Additional analysis on the effect of medical student exam scores during the COVID-19 pandemic should take into consideration other factors such as pre-matriculation data [36].

Conclusion

This study highlights the implications of virtual and blended instruction in the setting of pre-clerkship medical education. We recommend that educators take the difference between the ability of first-year versus second-year students to adjust to blended and virtual models into consideration when designing curricula. The difference between MS1s and MS2s also emphasizes the importance of the first year for medical students to establish social connections with peers and faculty and could inform extracurricular program design. Having considered these differences, this study contributes to the overall understanding of the impact of the COVID-19 pandemic on pre-clerkship medical education, about the benefits of asynchronous learning, and pre-clerkship student preferences. Additional benefits and drawbacks were identified which may help to inform design improvements for future learning models.