Introduction

Countries around the globe have faced unprecedented challenges in trying to support children’s learning amidst and beyond the COVID-19 era. The global pandemic outbreak has forced school closures to prevent the transmission of the coronavirus; this heavily disrupted children’s learning opportunities, methods, and resources. Recent studies that estimated the impact of pandemic-related school closures on student learning progress among U.S. students from multiple states (e.g., Domingue et al., 2022; Education Policy Innovation Collaborative [EPIC], 2021; Kuhfeld et al., 2022; Pier et al., 2021) indicate that children’s learning and academic development have suffered substantial setbacks during the pandemic school year when compared to a typical year unaffected by COVID-19. A report from the North Carolina Department of Public Instruction (2021) shows that the average proficiency rates in reading in spring 2021 declined significantly, ranging from 7.4% (Grade 8) to 25.5% (Grade 6), compared to spring 2019, which means that fewer students were proficient in reading during the pandemic than a non-pandemic school year. Educators, researchers, and policymakers have expressed mounting concerns that short-term learning loss could continue to accumulate, even after school re-opening, resulting in prolonged learning loss over years (e.g., Bailey et al., 2021; Kuhfeld et al., 2020b).

Although the existing projections of educational outcomes provide information and insights on the potential overall impact on students’ academic performance, the scope of the pandemic’s impact on academic achievement levels and growth, particularly in reading, is currently preliminary and scant. There is a common belief that many children from historically marginalized or vulnerable groups are disproportionately affected by the COVID-19 school disruptions (Amplify Education, 2021), yet limited robust evidence exists to support our understanding of the extent to which learning losses or gains have occurred to at-risk student population groups in the United States. In a recent study, Kuhfeld et al. (2022) explored racial-ethnic group differences in reading gains during the 2020–2021 school year and revealed that Black students exhibited significantly less gains than White students, resulting in widening racial/ethnic inequality gaps over time. Although this study provides insight into how the pandemic has affected historically marginalized race/ethnic groups of students, further evidence is needed to determine the extent to which learning losses or gains have occurred within other vulnerable groups of students, such as English learners and students with disabilities, over the pandemic period.

Furthermore, as school and district leaders currently concentrate on making important decisions for pandemic-related recovery efforts, it is important to comprehensively understand for whom, and to what extent, reading losses or gains have occurred during the pandemic year within a school district to effectively target recovery strategies and resources to the students most in need. Although available evidence has documented variation in students’ reading levels and growth during the pandemic based on nationwide samples (e.g., Curriculum Associates, 2020; Kuhfeld et al., 2022; Renaissance Learning, 2020), inferences founded on analyses of national databases may mislead or be insufficient for a school district to accurately assess and target student learning needs. Recent analyses that include 16% of the U.S. public schools serving Grade 3–8 students reveal substantial between-district variability in students’ reading achievement and growth distributions during the pandemic (Goldhaber et al., 2022).

Therefore, assuming that national trends apply to a specific school district may lead to inaccurate inferences about the predicted impact of COVID-19 on student reading growth. More importantly, school district policymakers and educators can benefit from a case study of a single school district in learning about how district-level education policies in response to COVID-19 have impacted student reading outcomes and progress and whether the impacts have differed across student population groups. Specifically, some school districts in North Carolina offered students and parents/guardians the option of starting the fall 2020 semester with in-person or remote instruction, but it is unknown whether the impact of COVID-19 on students’ reading growth varied by the instructional modality that students experienced. Research evidence on the influence of instructional modality (e.g., in-person or remote instruction) on students’ reading outcomes and growth during the 2020–2021 school year will enlarge understanding of the association between instructional modality and reading performance, thus influencing a school district’s policy implementation and evaluation efforts.

In the current study, drawing upon administrative data from a large urban school district in North Carolina, we examined the extent of learning losses or gains in reading that occurred among Grade 3–5 students during the pandemic and how it varied across demographic subgroups of students [e.g., socioeconomic status (SES), language status, disability status] within the school district. Specifically, we estimated reading losses or gains by comparing two same-grade cohorts: (a) the COVID-19 cohort of students who experienced COVID-19-related school closures and distance learning during the 2020–2021 school year and (b) the pre-COVID-19 cohort of students in the 2018–2019 school year. With the COVID-19 cohort of students, we further explored reading growth over the 2020–2021 school year to gauge the extent to which reading growth varied as a function of the instructional modality that students received, and how their demographic characteristics interacted with instructional modality.

Reading achievement during COVID-19

Learning loss can be conceptualized as the discrepancy between students’ assessed academic knowledge and skills and grade-level curricular expectations due to extended gaps or discontinuities in students’ education progress (Pier et al., 2021). This concept has often been discussed with reference to summer slides or setbacks even before COVID-19. There is well-documented evidence that the absence of formal schooling over the summer months has resulted in significant learning losses or slowdowns (e.g., Alexander et al., 2001; Downey et al., 2004; Quinn et al., 2016). Cooper et al.’s (1996) meta-analysis of 39 studies concerning summer learning loss indicates that U.S. students, on average, make one month of academic progress during the three-month summer break. Likewise, Atteberry and McEachin (2021) have found that the average U.S. students in Grade 1–8 achieve nearly 25–34% of school-year learning gains during the summer months. They have also found much higher variability in summer learning gains across students than during school years, which can contribute to widening race/ethnicity and socioeconomic achievement disparities in later school years (von Hippel & Hamrock, 2019). The negative effect of the absence or interruption of all schooling on student learning appears to accumulate over time, which may lead to a substantial impact on academic performance and social and educational inequalities (Hernandez, 2011; Lloyd, 1978).

School lockdown for nearly one-third of the school year in the wake of COVID-19 can be considered an extended time of summer break for many students. There is consensus that the historic interrupted or unfinished schooling has largely exerted a negative influence on students’ academic achievement levels and growth to an even greater degree than during summer break. Recent estimates of the COVID-19 learning slide or loss, drawn upon the NWEA Measure of Academic Progress (MAP) Growth assessment from multiple states in the United States (Kuhfeld et al., 2022), show that Grade 3–8 students’ average reading scores at the end of the 2020–2021 school year were, on average, 0.06–0.11 standard deviations lower than those from the 2018–2019 school year, with the largest year-difference for Grade 4 and 5 students. Kuhfeld et al. (2022) also found that students exhibited a positive, but modest, growth in reading, yet variability in growth rates within a grade level in the 2020–2021 school year was larger than that observed in the 2018–2019 school year.

A serious concern is that these short-term learning slowdowns can continue to accumulate over time, which might lead to much larger and long-lasting consequences in that many students who fell behind during the pandemic would struggle to catch up. For example, current Grade 3 students could fall further behind pre-pandemic expectations, resulting in a loss of 1.5 years’ worth of learning by the time they reach Grade 10 (Kaffenberger, 2020).

The COVID-19 impact on students with diverse backgrounds

To obtain a more comprehensive understanding of the profound impact of the pandemic on students’ academic attainment and growth, it is critical to consider the heterogeneous effects on different groups of students. Despite a rapidly growing number of studies on the COVID-19 impact, only a few studies to date have rigorously explored the heterogeneity of the pandemic-induced learning losses or gains as a function of students’ demographic characteristics.

Recent evidence suggests that school closures and rapid transition to home-based virtual learning during the pandemic disproportionately affected elementary and middle school students, especially Black and Hispanic students and those in high-poverty schools (e.g., Goldhaber et al., 2022). However, the negative impact of the pandemic on reading achievement is likely more profound for students in the early elementary grades as compared to upper elementary and secondary grades (e.g., Amplify Education, 2021; Georgiou, 2021; Kuhfeld et al., 2022; Tomasik et al., 2020). This may be because younger children require more instructional support and systemic scaffolding and, at the COVID-19 outbreak, their competencies for independent and self-regulated online learning had not yet sufficiently developed.

Moreover, the COVID-19 slide has had a particularly harmful effect on the academic achievement of students from low-income backgrounds, in general, amplifying existing income-based achievement disparities and inequalities (e.g., Engzell et al., 2020). Therefore, the COVID-19 disruptions had disproportionately detrimental impacts on English learners’ learning, yet it is unclear to what extent English learners’ English reading achievement and growth have been affected by the pandemic-related school closures.

Likewise, students with disabilities represent a uniquely vulnerable group of students who may have been significantly affected by COVID-19 school closures. The shift to remote instruction due to school lockdown can be immensely challenging for many students with disabilities who often experience difficulties with information processing or sustaining attention and focus to complete instructional tasks (Swanson, 1987). Particularly, for students with attention deficit hyperactivity disorder (ADHD) who experience inattention, hyperactivity, and impulsivity, their condition makes it hard to pay attention or control behaviors in an online learning environment (Lupas et al., 2021). Special education services or individualized education programs (IEP) were suspended to mitigate the spread of COVID-19. Regardless of how well an online learning curriculum was designed, reasonable accommodations and accessibility for students with disabilities and their needs were not sufficiently considered (Petretto et al., 2020). Consequently, most teachers faced many challenges in teaching remotely while trying to accommodate the unique needs of students with disabilities. Students with disabilities typically attain lower-than-average achievement scores (Gilmour et al., 2019) and the disability-based disparities in academic achievement may have been exacerbated by the pandemic.

How might instructional modality affect student reading outcomes?

Pre-pandemic studies on the effects of remote instruction on students’ academic achievement often reported a negative association between an online or distance learning mode and students’ academic achievement (e.g., Ahn & McEachin, 2017; Buddin & Zimmer, 2005; Center for Research on Education Outcomes [CREDO], 2015; Fitzpatrick et al., 2020). Despite the advent of new technologies that elevated students’ learning and engagement, research evidence shows that K-12 students who have attended online schooling are likely to perform lower on reading and mathematics assessments than their peers in traditional face-to-face learning environments (e.g., Ahn & McEachin, 2017; CREDO, 2015). In most virtual learning environments, students tend to participate in self-paced instruction with limited student–teacher and peer-to-peer interactions (Gill et al., 2015) such that students in online learning environments generally learn less than their peers who physically participate in active learning in their schools.

Even if internet access and the quality of remote learning improved over the pandemic, a lack of engagement and chronic absenteeism was more pronounced among students from high-poverty backgrounds, those who were English learners, and students with disabilities, when they were learning virtually (Patrick et al., 2021). In an online learning environment, students may need to work more independently through curriculum and lesson materials which increasingly requires self-regulatory learning and metacognitive skills to manage their learning (Azevedo, 2005). With limited scaffolding and guidance in distance settings, these skills may not be developed enough to foster learning for some students, particularly those younger and more vulnerable groups of children.

Emerging research evidence suggests that students who spent more in-person school days during the pandemic attained higher academic outcomes than peers who chose a full-distance learning option (e.g., Goldhaber et al., 2022; Halloran et al., 2021; Molnar, 2021; Tomasik et al., 2020). In the current study, we sought to examine the differential impact of instructional modality (i.e., in-person vs. remote instruction) on students’ reading growth rate over the pandemic school year. Figure 1 displays a conceptual framework of how types of instructional modality would affect students’ reading gains over time. It is expected that, among students with similar demographic characteristics, lower-achieving students are more likely to choose the in-person schooling option, while higher-achieving students tend to prefer the remote instruction option (National Center for Education Statistics, 2022). This may be because remote learning environments require high levels of independent and self-regulated learning skills to learn and access academic content with a limited amount of support from teachers and administrators, and these skills are more feasible for higher-achieving students than for lower-achieving students. However, we hypothesize that lower-achieving students would benefit from in-school learning experiences that can stimulate cognitive and social development, making greater reading gains than their higher-achieving peers who tended to participate in remote learning instruction during the pandemic. In-person schooling may thus offset inequalities in learning opportunities, and consequently, result in narrowing achievement differences to some degree.

Fig. 1
figure 1

Conceptual framework of how instructional modality affects reading achievement levels and growth rates during the 2020–2021 school year

Full size image

This conceptualization aligns with the faucet theory (Entwisle et al., 1997) and an accumulating body of knowledge about seasonal learning patterns (e.g., Alexander et al., 2001; Downey et al., 2004). During the school year, the resource faucet is turned on for all children; as a result, children with varying economic backgrounds benefit nearly equally. However, when a school session ends or is canceled, the resource faucet is turned off, thereby creating inequalities in educational opportunities and widening achievement gaps between students from high-poverty and low-poverty backgrounds. In out-of-school learning environments, the accumulation of learning losses and achievement gaps due to school closures occurs more substantively among low-achieving students, students from high-poverty environments, or students from historically marginalized vulnerable groups who may have unequal access to resources both inside and outside schools. Existing research suggests that high-quality summer school programs can serve to prevent learning losses and mitigate educational inequalities (Borman et al., 2005; Cooper et al., 1996; Kim & Quinn, 2013). We hypothesize that under pandemic circumstances, in-person schooling may serve as an equalizer for lower-achieving students, particularly from historically marginalized or vulnerable student populations (Alexander et al., 2001; Downey et al., 2004; Raudenbush & Eschmann, 2015).

The current study

The current study aimed to assess the COVID-19 impact on reading achievement levels and growth rates of Grade 3–5 students in the U.S. and whether the impact differed by students’ demographic characteristics (i.e., SES, language status, disability status) and instructional modality (i.e., in-person, remote instruction). Although a growing number of studies have documented COVID-19 learning loss or gain phenomenon around the globe, there is limited evidence of quantifying differential impacts on reading achievement gains and growth. Focusing on demographic subgroups can provide insights into the heterogeneity of the pandemic impact on reading attainment and can inform reading instruction and intervention as school districts continue to address local learning recovery needs.

Using administrative data drawn from an urban school district in North Carolina, we investigated to what extent upper elementary grades students made gains in reading during the 2020–2021 school year relative to those students in the same grade level who did not experience the pandemic in the 2018–2019 school year. We were particularly interested in inter- and intra-group differences to determine the dynamics of the impact of COVID-19 school closures on reading achievement in Grade 3 to 5 to contextualize our findings with other state (e.g., Pier et al., 2021) and national (e.g., Goldhaber et al., 2022; Kuhfeld et al., 2022) analyses focusing on those upper elementary grades. Furthermore, we examined whether students’ instructional modality affected the rates of reading growth over the 2020–2021 school year, particularly by focusing on whether the effects of instructional modality on reading growth varied by students’ SES, language status, and disability status. In pursuing this endeavor, our goals were not only to contribute to the literature in the field of the COVID-19 impact analysis but also to offer insights to educators and school and district leaders that are grounded by district-specific administrative data and evidence. Two research questions that guide this study are as follows:

  1. 1.

    To what extent did Grade 3–5 students’ reading gains during the 2020–2021 school year vary by grade level, SES, language status, and disability status compared to the 2018–2019 school year?

  2. 2.

    Did the association between Grade 3–5 students’ instructional modality and reading growth rates during the 2020–2021 school year differ by SES, language status, and disability status?

Method

Data source

This study used administrative data drawn from 180 elementary schools in an urban school district in North Carolina, USA, from the 2018–2019 and 2020–2021 school years. The primary data source for this study was the Measure of Academic Progress (MAP) Growth Reading assessment from the Northwest Evaluation Association (NWEA), a nationally normed, anonymous assessment database. We accessed the data based on a data-sharing agreement stemming from a research-practice partnership with the school district.

Analytic sample

The full analytic sample comprised 52,525 students from the two cohorts: 28,924 students from the pre- COVID-19 cohort (2018–2019) and 23,601 students from the COVID-19 cohort (2020–2021). Table 1 displays the demographic characteristics of the two-cohort samples by grade. The demographic characteristics of the two cohorts were similar across grade levels: 50% male, 34–36% Black, 26–28% White, 27–28% Hispanic, 7–9% Asian, 17–20% English learners, and 8–9% students with disabilities. The proportion of students from low SES neighborhoods (35–37%) was slightly higher than those of students from medium (30–32%) and high (29–31%) SES backgrounds.

Table 1 Demographic characteristics of the sample of two cohorts: 2018–2019 and 2020–2021 school year
Full size table

Table 2 shows demographic characteristics of the COVID-19 cohort in 2020–2021 by a choice of instructional modality. Students and parents/guardians were given the option to select either in-person or fully remote instruction options for the 2020–2021 school year in summer 2020. Overall, 62% of students opted to receive in-person instruction, while 38% selected the full remote instruction option. Within-subgroup variability existed in instructional modality preference. Specifically, nearly 59% of students from low SES neighborhoods chose in-person instruction, whereas 57% and 71% of students from medium and high SES, respectively, opted for in-person instruction. Approximately 66% of English learners and 66% of students with disabilities opted to participate in in-person instruction.

Table 2 Demographic characteristics of the COVID-19 cohort of students by instructional modality: in-person instruction and remote instruction
Full size table

Measures

Reading achievement

The NWEA MAP Growth assessment on student reading achievement is a computer-adaptive test aligned to the Common Core and state standards and is designed to serve as a benchmarking assessment to monitor and analyze students’ progress and needs throughout the school year (NWEA, 2019). The MAP reading scores are calculated using the Rasch unit (RIT) vertical scale that places a student’s ability and item difficulty estimates on the same scale. This vertical scale allows for comparisons of students’ learning growth within and across grades over time. The MAP reading composite score is computed based on the four strands: foundational skills, language and writing, vocabulary usage and functions, and narrative and informational text comprehension. As an adaptive test, the MAP assessment was designed to initially provide a student with question items appropriate for the student’s grade level, and then adjust the difficulty of each item depending on the student’s responses to previous items. Although this computer-adaptive assessment was administered remotely for many students in the beginning of the 2020–2021 school year, the test mode (i.e., in-classroom vs. remote) did not compromise the test quality (Kuhfeld et al., 2020a). Test–retest reliabilities, calculated by the vendor, range from .89 to .96 (NWEA, 2019). The concurrent validity estimates show that Grade 3–5 MAP reading scores are highly correlated (r = .79 to .80) with other U.S. state-specific assessments, including ACT Aspire, Partnership for Assessment of Readiness for College and Careers, and Smarter Balanced Assessment Consortium assessments) (NWEA, 2019). The MAP testing periods during a school year occurred in fall (late September), winter (late January), and spring (mid-April).

Student demographic characteristics

Three types of student demographic characteristics of interest were obtained from school district administrative data: SES, language status, and disability status. The SES variable had three categories—low, medium, and high SES—based on the census tract information. Students’ language status was to identify whether an individual was an English learner who came from households where a language other than English was primarily spoken. Disability status was to determine students with disabilities who received special education and related services under the Individuals with Disabilities Education Act according to an Individualized Education Program or other services plans.

Instructional modality

Instructional modality was operationalized as the assignment of students to either an (a) in-person instruction option or (b) remote instruction option for the 2020–2021 school year when both options were offered to students and their parents/guardians in summer 2020. Students who opted into the in-person schooling option physically attended school face-to-face for at most 10 days in the fall 2020 semester (2 days per week between November 2 and December 14) and 48 days in the spring 2021 semester (2–4 days per week between February 15 and May 28). They participated in remote instruction at home throughout the remainder of the school year. By contrast, students who chose the remote instruction option exclusively received virtual instruction without physical school attendance throughout the 2020–2021 school year.

Data analysis

Research questions 1: reading gains and variability

To address the first research question regarding Grade 3–5 students’ reading gains during the 2020–2021 school year and the variation across subgroups, we first obtained 9-month MAP reading gain scores for individual students in the COVID-19 cohort (2020–2021 school year) and pre-COVID-19 cohort (2018–2019 school year) by subtracting the score at the beginning of the school year (late September) from the score at the end of the school year (mid-April). To further contextualize how reading gains prior to the pandemic compared to reading gains during the pandemic for each grade level, we estimated the standardized difference (in 2018–2019 standard deviation units) between 2018–2019 and 2020–2021 means by grade level by standardizing the 9-month gains for 2020–2021 to the mean and standard deviation of the 9-month gains for 2018–2019. Then, we calculated the means and standard deviations of the gained scores by the subgroup samples (e.g., SES, language status, disability status) of the two cohorts. Subsequently, we estimated the percentage increase in means and standard deviations achieved by the COVID-19 cohort relative to the pre-COVID-19 cohort within the subgroups.

Research question 2: instructional modality difference in reading growth

To examine the effects of different instructional modality use, either in-person or fully remote instruction, on reading growth rates during the 2020–2021 school year, we employed a series of piecewise growth curve models (Singer & Willett, 2003). An initial inspection of the average MAP reading scores at the three assessment time points (i.e., beginning, middle, and end of the school year) (see Table 3) indicated that students’ reading progression patterns across the three-time points appeared to be nonlinear. We specified linear growth slopes for two separate intervals: (a) fall semester: between the beginning (fall 2020) and middle of the school year (winter 2021) and (b) spring semester: between the middle and end of the school year (spring 2021). Three-level piecewise growth curve models were specified (time nested within students within schools). The level 1 (within individual) model is expressed as follows:

$${\text{Level}}\;{1}:\quad Y_{tij} = {\uppi }_{{{0}ij}} + \pi_{{{1}ij}} FALL_{{{1}tij}} +{{ \pi }}_{{{2}ij}} SPRING_{{{2}tij}} + \varepsilon_{tij} { }\quad {\upvarepsilon }_{tij} \sim {\text{ N}}(0,\upsigma _{{\upvarepsilon }}^{2} )$$

where Ytij represents the MAP reading score at time t for student i in school j; \(\uppi _{0ij}\) denotes the predicted score for student i in school j at fall 2020; and \(\uppi _{1ij}\) and \(\uppi _{2ij}\) refer to monthly learning rates for student ij over the fall 2020 and spring 2021 semesters, respectively. The error term, \({\upvarepsilon}_{{tij}}\) is assumed to be normally distributed with a mean of zero.

Table 3 Descriptive statistics of Measure of Academic Progress (MAP) reading scores of the COVID-19 cohort of students (2020–2021 school year) by grade levels, instructional modalities, and assessment time points
Full size table

At level 2 (between individual), we included the instructional modality variable (i.e., REMOTE) and demographic indicators such as SES (low SES vs. medium/high SES), language status [English learners (EL) vs. English-fluent students], and disability status [students with (SwD) vs. without disabilities] as a main-effect predictor of intercept (the beginning of the 2020–2021 school year) and growth rates over the fall and spring semesters. To examine the interaction effects between instructional modality and demographic characteristics on MAP reading level at intercept and growth rates in the fall and spring semesters, we additionally included a set of interaction terms. The level 3 model was specified to represent the variability among schools. The equations for level 2 and 3 are presented below:

$${\text{Level}}\;2:\quad\uppi _{pij} = \upgamma _{p0j} + \upgamma _{p1j} REMOTE_{ij} + \upgamma _{p2j} LowSES_{ij} + \upgamma _{p3j} EL_{ij} +\upgamma _{p4j} SwD_{ij} + \upgamma _{p5j} (REMOTE \times LowSES)_{ij} + \upgamma _{p6j} (REMOTE \times EL)_{ij} + \upgamma _{p7j} (REMOTE \times SwD)_{ij} + \mathop \sum \limits_{w = 8}^{15}\upgamma _{0wj} COV_{ij} + \upzeta _{0ij(p = 0)} + \upzeta _{1ij(p = 1)} \quad (p = 0,1,2)$$
$${\text{Level}}\;3:\quad\upgamma _{pqj} =\upbeta _{pq0} + \upnu _{00j} \quad (q = 0, \ldots, 8)$$
$$\left[ {\begin{array}{*{20}c} {\upzeta _{0ij} } \\ {\upzeta _{1ij} } \\ \end{array} } \right] \sim {\text{N}}\left( {\begin{array}{*{20}c} {\left[ {\begin{array}{*{20}c} 0 \\ 0 \\ \end{array} } \right]} & {\left[ {\begin{array}{*{20}c} {\uptau _{00} } & {\uptau _{01} } \\ {\uptau _{10} } & {\uptau _{11} } \\ \end{array} } \right]} \\ \end{array} } \right), \upnu _{00j} \sim {\text{N}}\left( {0, \uptau _{\beta 00} } \right).$$

Note that β000 is the overall mean at the beginning of the 2020–2021 school year; β010 denotes the initial difference between in-person and remote instruction students; β110 and β210 represent monthly reading growth rates over the fall and spring semester, respectively; β150, β160, and β170 denote the interaction effects of instructional modality with subgroups (Low SES, EL, and SwD, respectively) on the fall-semester growth rate, while β250, β260, and β270 refer to the interaction effects on the spring-semester growth rate, controlling for the effects of covariates (COV; i.e., gender, race/ethnicity).

Results

Research question 1: COVID-19 reading gains and variability

Table 4 shows means and standard deviations of the MAP reading achievement scores at the beginning and end of the school year for Grade 3–5 students in the 2018–2019 and 2020–2021 school year cohorts. Figure 2 displays the percentages of MAP reading achievement score gains of the 2020–2021 school year (or COVID-19) cohort relative to the 2018–2019 school year (or pre-COVID-19) cohort by student grade levels and demographic subgroups. Overall, the COVID-19 cohort achieved lower 9-month reading gains than the pre-COVID-19 cohort, with a learning loss effect size of 0.54, 0.27, and 0.28 standard deviation units for Grade 3, 4, and 5, respectively. Among the COVID-19 cohort students, reading losses were evident compared to the typical school year (i.e., 2018–2019), particularly for Grade 3 students. Overall, Grade 3 students in the COVID-19 cohort achieved 48% gains of the pre-COVID-19 cohort in reading, on average, whereas Grade 4 and 5 students achieved 65% and 58% gains, respectively. Moreover, there was much more variability in reading gains for the COVID-19 cohort, especially in the earlier grades. As shown in Fig. 3, the standard deviation of reading scores of the COVID-19 cohort increased by 56%, 40%, and 29% for Grade 3, 4, and 5 students, respectively.

Table 4 Measure of Academic Progress (MAP) reading score means and standard deviation at the beginning of the school year (BOY) and end of the school year (EOY) for the cohorts of the 2018–2019 and 2020–2021 school year
Full size table
Fig. 2
figure 2

Percentages of Measure of Academic Progress (MAP) reading gains between beginning of year and end of year for the 2020–2021 school year cohort relative to the 2018–2019 school year cohort by student grade levels and demographic Characteristics. Note. SES socioeconomic status

Full size image
Fig. 3
figure 3

Percentages of increase in Measure of Academic Progress (MAP) Reading variability for the 2020–2021 school year cohort relative to the 2018–2019 school year cohort by student grade levels and demographic characteristics. Note. SES socioeconomic status

Full size image

We further examined relative reading gains and variability of the COVID-19 cohort within a grade level across subgroups. Among Grade 3 students, the COVID-19 cohort students from high SES backgrounds achieved 61% of pre-COVID-19 cohort reading gains, while students from low and medium SES backgrounds made 40% and 43% of the typical gains, respectively, during the pandemic. Moreover, low- and medium-SES students’ reading gains showed much greater variabilities (62% and 63% respectively) than high-SES students (38%). Likewise, Grade 4 students from high SES environments attained over 70% of typical reading gains whereas their peers from the low and medium SES groups made nearly 60% of typical gains. Reading gains variabilities for low- and medium-SES students (41% and 43% respectively) were slightly higher than that for high-SES students (34%). However, for Grade 5 students, conversely, low SES group ended the 2020–2021 school year with 63% of their prior-year reading gains compared to medium- and high-SES groups who made 52% and 58% of typical reading gains, respectively. The increase in variability for Grade 5 was smaller than that for Grade 3 and 4 students and consistent across SES groups (28–32%).

In terms of relative reading gains among English learners and English-fluent learners, Grade 3 and 4 English learners experience 41% and 60% of typical reading gains, respectively, lower than their English-fluent peers (49% and 66%, respectively). Notably, Grade 5 English learners showed 68% of typical gains with a small increase (16%) in variability, while English-fluent students made 54% of typical gains with a twice larger variability (32%) than their counterparts.

Finally, students with disabilities demonstrated much lower gains in reading than what would have been observed in normal conditions. Grade 3, 4, and 5 students with disabilities achieved only 18%, 28%, and 53%, respectively, of pre-COVID-19 reading gains, whereas students without disabilities made 50%, 68%, and 59% of typical gains for the respective grades. The increase in spread of reading scores was especially stark for Grade 3 and 4 students with disabilities (87% and 86%, respectively), compared to students without disabilities (53% and 34%, respectively).

Research question 2: association between instructional modality and reading growth rates by subgroups

Table 5 shows the results of the full piecewise growth curve models by grade level. Overall, across the Grade 3, 4, and 5 models, there was a statistically significant difference between in-person and remote instruction modality groups at the beginning of the 2020–2021 school (Grade 3: β010 = 4.83, SE = 0.59; Grade 4: β010 = 3.47, SE = 0.53; Grade 3: β010 = 3.86, SE = 0.50; ps < .001), indicating that students who opted for remote instruction started the school year with higher MAP reading scores than their peers who chose in-person instruction. As depicted in Fig. 4, during the fall semester, reading growth rates were not statistically significantly different between in-person and remote instruction groups across grade levels and subgroups (ps > .05), holding all else constant. However, variations in reading growth rates became apparent over the spring semester (between winter and spring 2021). Students who participated in remote instruction exhibited significantly lower growth rates than their peers who received in-person instruction during the spring semester (Grade 3: β210 = − 0.51, SE = 0.10; Grade 4: β210 = − 0.55, SE = 0.08; Grade 3: β210 = − 0.56, SE = 0.08; ps < .001). To shed light on whether the association between instructional modality and reading growth rates varied by students’ demographic subgroups, we further examined the interactions between instructional modality and subgroup (i.e., SES, language status, disability status) in each grade level.

Table 5 Results of piecewise growth curve modeling for the interaction effects between demographic characteristics (socioeconomic, language status, and disability status) and instructional modality on MAP reading levels and growths for grade 3, 4, and 5 students
Full size table
Fig. 4
figure 4

Piecewise growth curve trajectories of COVID-19 cohort students’ Measure of Academic Progress (MAP) reading by instructional modality (in-person or remote instruction) for A Grade 3, B Grade 4, and C Grade 5

Full size image

Grade 3

The interaction between low SES and remote instruction was not statistically significant in predicting intercept (beginning of fall 2020) and growth rates over the fall and spring semesters (ps > .05). The interaction between English learner and remote instruction was not statistically significant in predicting intercept and growth rate in fall (ps > .05), but significantly predicted growth rate in spring (β260 = − 0.30, SE = 0.15, p < .05). Likewise, the interaction between student with disabilities and remote instruction statistically significantly predicted growth rate in spring (β270 = − 0.52, SE = 0.23, p < .05), but not intercept and growth rate in fall (ps > .05).

Figure 5 displays these significant differences in fitted growth trajectories in the spring semester. As shown in Fig. 5A, both Grade 3 English-fluent students and English learners who participated in in-person instruction showed a steady increase in reading, while their peers who received fully remote instruction had a decrease in reading growth rate during the spring semester. By the end of the school year, English learners with in-person instruction narrowed the initial differences in reading with their English learners and English-fluent peers who received remote instruction. Additionally, among English-fluent students, the initial reading achievement difference between in-person and remote instruction groups narrowed at the end of the school year. In Fig. 5B, a similar pattern of the closed gap between instructional modality groups was observed among students without disabilities. However, students with disabilities who received remote instruction exhibited a decline in reading over the spring semester, while students with disabilities with in-person instruction made very little reading growth in reading over time.

Fig. 5
figure 5

Piecewise growth curve trajectories of Grade 3 COVID-19 cohort students’ Measure of Academic Progress (MAP) reading by A language status and instructional modality and B disability status and instructional modality

Full size image

Grade 4

The interaction between low SES and remote instruction was statistically significant in predicting intercept (β050 = 3.67, SE = 0.84, p < .001) and growth rate in spring (β250 = − 0.25, SE = 0.11, p < .05), but not in fall (p < .05). As shown in Fig. 6A, there were substantial variations in reading levels at the outset and growth trajectories over the spring semester based on the interaction between low SES and remote instruction. Specifically, among students from low SES neighborhoods, those with remote instruction started the school year with a higher reading level than their peers with in-person instruction, yet their difference in reading became indistinguishable as the remote instruction group made slower progress, while the in-person group continued to grow over the spring semester. A similar pattern emerged between the in-person and remote instruction groups among students from medium/high SES backgrounds.

Fig. 6
figure 6

Piecewise growth curve trajectories of Grade 4 COVID-19 cohort students’ Measure of Academic Progress (MAP) reading by A socioeconomic status (SES) and instructional modality, B language status and instructional modality, and C disability status and instructional modality

Full size image

The interaction between English learners and remote instruction was statistically significant in predicting reading growth rate in spring (β260 = 0.39, SE = 0.13, p < .01), but not intercept and growth rate in fall (ps > .05). This significant difference in spring may be particularly attributable to English-fluent students, in which those with the in-person option made continuous growth, whereas those with the remote instruction option showed a slowdown (see Fig. 6B). Notably, the initial and persistent reading difference that existed between English-fluent students with remote instruction and English learners with in-person instruction over the fall semester gradually diminished during the spring semester.

Similarly, the interaction between students with disabilities and remote instruction statistically significantly predicted growth rate only in spring (β270 = − 0.31, SE = 0.19, p < .01). As displayed in Fig. 6C, the pre-existing difference between the in-person and remote groups among students without disabilities disappeared by the end of spring as those who received in-person instruction continuously grew through the spring semester. However, both instructional modality groups among students with disabilities experienced negative growth in spring with their growth trajectories parallel to each other.

Grade 5

The interaction effect between low SES and remote instruction was marginally significant on intercept (p < .10) and statistically significant on growth rate only for the fall semester (β150 = − 0.25, SE = 0.09, p < .01). Figure 7A depicts that among Grade 5 students from low SES environments, those who opted for remote instruction started the fall semester with nearly 5 RIT higher than their peers who chose in-person instruction. However, the low-SES group students who participated in in-person instruction achieved a positive growth, while those who received fully remote instruction hardly showed any gains in reading. As a result, the gap identified in fall between the instruction modality groups vanished by the end of spring. A similar pattern was observed among students from medium/high SES backgrounds. Notably, in the group of students who decided to receive fully remote instruction, an initial difference between low and medium/high SES groups at the beginning of fall became slightly larger by the end of spring, whereas the SES-based difference within the in-person group remained persistent.

Fig. 7
figure 7

Piecewise growth curve trajectories of the Grade 5 COVID-19 cohort (2020–2021 school year) students’ Measure of Academic Progress (MAP) reading by A socioeconomic status (SES) and instructional modality and B language status and instructional modality

Full size image

In addition, the interaction effect between language status and instructional modality was statistically significant on intercept (β060 = 5.01, SE = 0.99, p < .001) but not growth rates (ps > .05). Notably, as shown in Fig. 7B, among English learners, the initial difference between the in-person and remote instruction groups was nearly 6 RIT and this difference sustained throughout the fall semester. Yet, the difference narrowed by about half by the end of spring as the reading growth rate for English learners who participated in in-person instruction accelerated over the spring semester, while English learners who received remote instruction experienced a growth plateau during that time.

Discussion

Drawing upon the school district administrative data, the present study explored Grade 3–5 students’ reading gains during the 2020–2021 school year and the association between instructional modality and reading growth rates, focusing on the variations across demographic characteristics. Previous analyses on the pandemic-related impact on student academic achievement and growth have focused on students’ racial and ethnic backgrounds (e.g., Kuhfeld et al., 2022) and poverty levels (e.g., Maldonado & De Witte, 2020; Pier et al., 2021) with limited attention being paid to English learners and students with disabilities. Two main findings emerged from the study. First, the COVID-19 cohort students’ reading achievement gains from the beginning to end of the 2020–2021 school year were lower than reading gains of the pre-COVID-19 cohort students in the 2018–2019 school year with substantially reduced gains for younger students, students from low SES backgrounds, English learners, and students with reading disabilities. Second, among students with similar demographic characteristics, higher-achieving students and their parents/guardians tended to choose the remote instruction option, while lower-achieving students appeared to opt for in-person instruction at the beginning of the 2020–2021 school year. However, those students who received in-person instruction most likely demonstrated positive growth continuously over the school year, whereas initially higher-achieving students who received remote instruction showed stagnation or decline in reading in the spring semester. We found substantial variation in reading levels and growth rates as a function of the interaction between instructional modality and students’ demographic subgroups.

COVID-19 reading gains and variability

With the current data from an urban school district in the United States, we provide evidence that Grade 3, 4, and 5 students ended the 2020–2021 school year with 0.54, 0.27, and 0.28 standard deviations behind the 2018–2019 school year reading, suggesting that students’ reading achievement levels declined during the pandemic school closures. The degree of reading loss experienced by students in the urban school district in North Carolina over the 9-month school year was larger than the 12-month-based estimates of learning loss obtained from the results from multiple states in the U.S. (cf. Kuhfeld et al., 2022). Furthermore, consistent with recent evidence on COVID-19 learning loss by grade level (e.g., Goldhaber et al., 2022; Kuhfeld et al., 2022; Tomasik et al., 2020), our cross-cohort comparisons of reading gains in the pandemic (2020–2021) and typical (2018–2019) school year suggest that younger students lost substantially more ground in reading relative to older students during school lockdowns. Grade 3 students achieved only 48% of the learning gains in reading over the 9-month pandemic school year compared to the pre-pandemic school year, indicating nearly five months behind where they would have been under normal circumstances (cf. Dorn et al., 2020). This estimated magnitude of pandemic-related reading loss for Grade 3 students was much lower than those for Grade 4 and 5 students (65% and 58%, respectively). Grade 3 students’ substantial reading loss is plausibly associated with the reduction in daily instructional time usually devoted to develo** foundational literacy skills and promoting language and reading comprehension. From a developmental perspective, Grade 3 is a stage in which students develop more advanced phonemic awareness, phonics knowledge, and word decoding skills to be fluent readers with greater comprehension skills (Chall, 1983; Ehri, 2014; Kilpatrick, 2015). This requires sufficient instructional time in which children are actively and repeatedly involved in engaging, efficient, and systematic literacy practice. With the significant amount of disruption to instructional time during the extended school closures, Grade 3 students experienced a lack of opportunity to gain and build foundational reading skills that are essential to effective comprehension, critical thinking, and content knowledge development, which may potentially lead to negative long-term consequences in future years (Kaffenberger, 2020).

The large average reductions in reading gains during the pandemic have been compounded with substantially increased variation in scores. The circumstances of COVID-19 created a much greater spread in scores compared to the pre-pandemic, particularly with earlier grade (e.g., Grade 3) students and more vulnerable students (e.g., low-SES group, English learners, students with disabilities) who attained a much wider range of scores relative to later grade (i.e., Grade 4 or 5) students and less vulnerable students (e.g., high-SES group, English-fluent students, students without disabilities).

Our findings suggest that the negative impact of pandemic-related school closures on reading was especially profound for students from low SES environments, English learners, and students with disabilities. Young children with a high poverty status, English learner status, and disability status appear more vulnerable to the pandemic school disruptions. This finding converges with previous projections, in which the detrimental pandemic influence on student learning may have disproportionally affected the historically marginalized and vulnerable groups of students (Amplify Education, 2021). For students from low SES backgrounds, particularly in Grade 3 and 4, the estimated percentages of increase in reading between the beginning and end of the pandemic school year relative to the typical year were even lower than high-SES group students. For example, Grade 3 low-SES group students made only 40% of the pre-pandemic reading gains while medium- and high-SES students achieved more than 60%. This finding supports the notion that COVID-19 has magnified pre-existing SES-based achievement gaps and inequalities (e.g., Gore et al., 2021; Maldonado & De Witte, 2020) due to a lack of access to learning opportunities, appropriate digital devices, and reliable internet at home that students from high-poverty neighborhoods faced during school closures.

Similarly, English learners who were most likely from low-SES immigrant families and under-served communities demonstrated positive gains in 2020–2021, but their reading gains lagged relative to the pre-pandemic school year. We provide evidence that Grade 3 and 4 English learners’ relative reading gains in percentage (41% and 60% of the pre-pandemic reading gains, respectively) were smaller than their English-fluent peers’ relative reading gains (49% and 66%, respectively). This finding is consistent with pre-pandemic research evidence (e.g., Lawrence, 2012) that English learners experience greater summer setback in their English vocabulary development than English-fluent students during the summer months. This is partially because for many English learners, school is their primary context for exposure to, and development of, academic language that is central to academic success. However, the detrimental impact of COVID-19-related school disruptions on English learners could be even more pronounced because the absence of formal schooling and a lack of collaborative peer learning opportunities can influence English language and literacy development years later (Sugarman & Lazarín, 2020).

In addition, we found that students with disabilities were likely to struggle the most. Particularly, Grade 3 and 4 students with disabilities ended the 2020–2021 school year with only 18% and 28% of the pre-pandemic-year reading gains, leaving them nearly seven to eight months behind in reading. They may have experienced reduced access to differentiated instructional support and inadequate accommodation and accessibility during COVID-19 (Petretto et al., 2021). As many students with special needs rely on established routines and a vibrant network of services in their communities, dramatic decreases in services from school staff and community organizations and remote instruction have been a significant challenge to attention and motivation in reading (Sciberras et al., 2020).

Instructional modality and reading growth during COVID-19

Our second major finding based on the COVID-19 (2020–2021) school year cohort students indicates that when schools began to re-open in the fall of the school year, there existed educational disparities from the choice of instructional modality. We found that, conditional on students’ demographic characteristics, higher-achieving students were likely to start the school year with the online schooling option in contrast to lower-achieving students who tended to choose the in-person option. However, our results indicate that there was some variation in the magnitude of these disparities. The reading achievement gap between lower-achieving students (or students with the in-person option) and higher-achieving students (or students with the remote instruction option) was particularly bigger among English learners compared to other subgroups such as low SES and students with disabilities. This may be because young English learners with relatively low reading ability in English in the urban areas were likely coming from immigrant or refugee families who were mostly constrained in their educational options and tended to opt into in-person schooling mode.

There has been a concern that the pre-existing academic achievement gaps would be exacerbated in the absence of schooling during the pandemic (e.g., Bailey et al., 2021). However, we provide evidence that many lower-achieving students who had in-person schooling experience showed steeper reading growth trajectories than higher-achieving peers who did not, especially during the spring 2021 semester. This finding suggests that schools helped lower-achieving groups of students with similar demographic characteristics catch up to higher-achieving groups over the COVID-19 school year, supporting our conceptual framework (Fig. 1) grounded by the notion that schools generally play an equalizing role in academic disparities between student groups (e.g., Alexander et al., 2001; Downey et al., 2004; Quinn et al., 2016). It is noteworthy that the major difference between the in-person and fully remote instructional modality in this study was the duration of in-person school attendance. Students with the face-to-face schooling option physically attended schools for 10 days in the fall semester and 48 days in the spring semester, while peers with the fully remote instruction option exclusively participated in school instruction virtually. With the 10-day school attendance during the fall semester, the average reading scores for both instructional modality groups increased gradually in parallel, yet in-person school attendance for 48 days over the spring semester appeared to make a substantial difference, contributing to reading growth trajectories. The relative benefits of in-person instruction align with recent research evidence on the association of instructional modality with learning outcomes during the pandemic (e.g., Goldhaber et al., 2022; Halloran et al., 2021; Molnar, 2021; Tomasik et al., 2020).

Students who began the pandemic school year with relatively weaker reading ability benefitted from the opportunities to develop language and literacy skills by interacting with educators and peers in in-person environments, resulting in making greater gains in reading over time. Particularly, Grade 3–5 English learners in the in-person instructional modality group experienced continuously positive growth over the school year, but their English-fluent students in the remote instruction group remained stagnant or declined during the spring semester.

However, an inconsistent pattern of schools as equalizers emerged for students with disabilities who participated in in-person instruction. Despite their face-to-face attendance to general classroom instruction, their reading growth stagnated or fell especially over the spring semester, possibly due to limited special education services or IEP offered to students with special needs during the pandemic. This pattern of school attendance not contributing to learning trajectories for students with disabilities is consistent with recent research evidence on summer learning rates (e.g., Cooc & Quinn, 2022; Gershenson & Hayes, 2017). Furthermore, the widening academic inequality between students with and without disabilities observed regardless of instructional modality during the pandemic school year is aligned with evidence of the Matthew effect (Stanovich, 2009), in which students with initially higher levels of reading ability experience greater learning gains than their counterparts, leading to growing disparities over time.

Limitation and future research

The current study findings must be interpreted within several limitations of the study that can inform future research. First, an important caveat for interpreting the results of the current study is that descriptive comparisons of reading gains between the two cohorts do not make causal claims about the COVID-19 impact on reading gains. Thus, we acknowledge that any causal interpretations of our findings should be made with caution. Second, a lack of contextual information on in-person and remote instructional settings in the current study is an important study limitation to note. Although the current study used existent administrative data as a source of large quantitative information readily available from an urban U.S. school district, administrative records that contain vast amounts of qualitative information on students, families/homes, teachers, and schools obtained during COVID-19 may provide insights into mechanisms leading to pandemic-related reading losses. Particularly, to provide a more comprehensive picture of how and why in-person instruction was positively associated with students’ reading growth over the pandemic school year, future research should delve into features of instructional practices and students’ interactions with peers and teachers in face-to-face settings, distinctive from those via an online platform. For example, there is emerging causal evidence that in-person tutoring (Nickow et al., 2020) has substantially larger effects on students’ reading achievement than online or remote tutoring (Kraft et al., 2022). More causal intervention studies that compare in-person to face-to-face instruction along with detailed contextual information would permit a deeper understanding of how and why the in-person learning mode provides enhanced learning opportunities for students to make continuous growth in reading during the pandemic, particularly for lower-achieving students, and what online instructional approaches and resources need to be considered in remote schooling to meet the diverse learning needs of students.