Background

Over the last several decades, efforts to develop and test mental health prevention and treatment programs have resulted in an impressive list of evidence-based interventions (EBIs). However, individuals in need of mental health services remain unlikely to receive an EBI. This public health dilemma has prompted increased consideration of factors that promote EBI implementation in the complex settings where most people receive mental health services, including universities and colleges (hereafter, colleges). Implementation involves several processes and outcomes, among them initial adoption, fidelity, feasibility, acceptability, effectiveness, and sustainability [1]. Of these, sustainability—the “extent to which a newly implemented treatment is maintained or institutionalized within a service setting’s ongoing, stable operations” [1] (p70)—has received relatively little attention [2]. Given the high costs associated with initial training and support of EBIs [3, 4], sustainability is an important way to measure return on investment. Additionally, for EBIs to meaningfully reduce the burden of mental illness they must be implemented in a sustained fashion. Few studies have compared different implementation strategies using experimental designs and evaluated their impact on EBI sustainability [2, 5], and even fewer have studied EBI sustainability in college settings [6].

Most young adults attend college [7], making college campuses a natural place to intervene with mental health disorders—especially given the high prevalence of these disorders among college students [8, 9] which has increased following COVID-19 [10]. Prolonged isolation during the pandemic has been linked to various mental health concerns, including body image distress and eating disorders for adolescents and young adults [11, 12]. Eating disorder prevention efforts are particularly well-suited for college settings, since access to treatment services at college mental health clinics is limited [13, 14], and eating disorders, once developed, are often chronic and associated with high levels of distress, medical complications, impairment, and mortality [15, 16]. Due to the low ratio of mental health providers to college students in need, prevention programs that can be provided by college peer educators could dramatically increase access to services.

Among eating disorder prevention programs, the Body Project has a robust evidence base [17]. The Body Project is an EBI for the prevention of eating disorders wherein adolescent girls/young women with body image concerns collectively critique pursuit of the thin beauty ideal in verbal, written, and behavioral exercises based on cognitive dissonance principles. Prior studies of the Body Project with college peer educators [18,19,20,21,22] have demonstrated significant reductions in eating disorder symptoms and future eating disorder onset. Moreover, a recent randomized trial found that the Body Project delivered by college peer educators remained effective across varying levels of implementation support [23] and had similar results on other key implementation outcomes (reach, adherence, competence). Although the Body Project demonstrated substantial benefits, the factors that predict sustainability are unknown. A mixed-methods naturalistic study of college clinician reactions to the Body Project two years after an effectiveness trial found that while 50% of colleges continued to implement the Body Project at one-year follow-up, by year 2 only one college (12%) sustained the practice [24].

Most research related to sustainability of EBIs has been in the form of qualitative interviews and retrospective surveys after active implementation efforts [5, 25], alongside conceptual papers. Results of studies set in mental health clinics [26, 27], Veteran’s Affairs hospitals [28], primary care [29] K-12 schools [30, 31] and colleges [24] suggest that numerous factors influence whether an intervention is sustained, including characteristics of the intervention and the practitioners who provide it. This is consistent with theoretical frameworks of sustainability, such as the dynamic sustainability framework (DSF [32]), which proposes that features of the interventions, practice settings, and the broader ecological system affect sustainability both directly and interactively. The DSF further contends that the better the fit between the intervention and the setting, the more likely that the intervention will be maintained32(p6). A unique aspect of the DSF is the expectation that as the practice setting and broader context change dynamically, so too must the intervention to maintain fit. Thus, even interventions that are considered already “optimized” based on efficacy and effectiveness testing may be difficult to sustain amidst ever-changing service settings. Perhaps that is why providers often report modifying interventions when they are sustained [26, 33, 34].

Implementation strategies like provider training, ongoing support, and quality assurance are embedded in conceptual frameworks like the DSF and could impact intervention sustainability. A study of parent–child interaction therapy that randomized 50 mental health organizations to a cascading “train-the-trainer” (TTT) model of training, a learning collaborative model, or a distance learning model found that clinicians in the TTT model implemented the treatment approach with more clients at the 24-month follow-up assessment [35]. Continued reach of the EBI over time is a critical marker of sustainability.

Adequate staffing of individuals who are trained to provide the EBI is another key indicator of a setting’s ability to sustain a practice [36]. Lack of training opportunities following initial implementation has been identified as a barrier to sustainability [37], despite evidence that later generations of providers trained internally can implement EBIs with a comparable level of fidelity as those originally trained by intervention developers [38]. Supervisory support within a setting has also been found to promote adherent practice and client benefit [39, 40]; in contrast, when internal supervisors are not adequately trained in the EBI being offered by their supervisees, working alliance is lessened which may affect service quality [41]. Thus, an organization’s ability to continue to offer an EBI is also affected by their ability to maintain a trained staff of providers and supervisors beyond those initially trained despite the high rates of expected turnover among mental health providers [42], which can undermine sustainability [31]. On college campuses where mental health promotion and prevention efforts are often provided by peer educators [43], sustainability efforts are made more complicated by the transient nature of college students. We were unable to find any studies that measured the extent to which activities like trainings continued after initial EBI implementation, or that examined variables associated with those activities, despite their relevance to an organization’s internal capacity to sustain services.

The current study

Considerable resources have been invested in the development, testing, and initial implementation of EBIs, but without attention to sustainability their potential for real impact is limited. As noted, college students are a population with a high prevalence of mental health problems, including eating disorders, and often lack access to EBIs. The Body Project is a highly effective preventive intervention that can be efficaciously provided by college peer educators, however a rigorous evaluation of its sustainability within college peer educator programs has not been conducted. As part of a larger implementation study [44], we recruited 63 colleges with peer educator programs and randomly assigned them to one of three implementation strategies: Train-the-Trainer (TTT) training, TTT training plus a technical assistance workshop (TTT + TA), and TTT plus the TA workshop and one year of quality assurance (TTT + TA + QA). After one year of active implementation, results indicated that implementation support strategies did not significantly predict primary outcomes including attendance, adherence, competence, and reach. However, adding technical assistance and quality assurance strategies was related to larger reductions in eating disorder risk factors and symptoms. Following active implementation, all participating colleges entered a two-year sustainability phase. The current study compared the impact of different levels of implementation support for the Body Project on secondary outcomes related to sustainability.

Our main research questions concerned whether level of implementation support would predict whether Body Project groups were conducted during the two-year sustainability phase, and secondarily whether level of implementation support predicted two key ongoing implementation activities: training new peer educators and training new supervisors. Based on main effects reported by Stice et al. [44], we hypothesized that the TTT + TA + QA implementation support strategy would be superior to both the TTT alone and TTT + TA strategies in terms of conducting Body Project groups and conducting internal trainings for new peer educators and staff. Although the technical assistance (TA) workshop was intended to enhance the fit of the intervention to each individual context, which has been theorized to result in better sustainability [32], differences between the TTT and TTT + TA strategies during implementation were nonsignificant or small in magnitude. In contrast, the addition of QA consultations, which explicitly focused on managing barriers to implementation and planning for sustainability, resulted in stronger effects during implementation. In alignment with the DSF, the QA consultations permitted progress review, problem-solving of implementation barriers, and sustainability planning, which theoretically would improve the “optimization” of the Body Project for long-term sustainment.

Because intervention factors (perceived characteristics of the EBI), provider characteristics (attitudes and beliefs) and implementation process (the completion of implementation activities) have emerged as possible predictors of sustainability in existing qualitative and naturalistic studies of EBT sustainability, we also explored whether provider’s perceived characteristics of the intervention, supervisor attitudes towards the delivery of evidence-based practices, and degree of completion of activities during implementation predicted sustainability outcomes. Because they predicted initial implementation outcomes in a previously published report of the same study [23], we hypothesized that supervisor attitudes towards evidence-based practices and perceived intervention characteristics, and a greater number of completed implementation activities would predict all three sustainability outcomes. All models included covariates related to the organizational context of the colleges since prior research on the Body Project found that available resources are related to some implementation outcomes [45], and because theoretical models such as the DSF and others describe these outer and inner settings implementation indicators as influential to sustainability [32, 46].

Method

Participants and procedures

US colleges with peer educator programs not already implementing the Body Project were contacted to participate. To be included, schools had to have (a) two or more peer education supervisors (college employees who oversee peer educator programming), and (b) eight or more peer educators interested in this training. Sixty-three colleges agreed to participate and were trained. Once the peer educator supervisors and peer educators were trained by project staff, schools recruited undergraduate participants to complete a body acceptance program (which is how the Body Project is typically described). Interested female-identifying students were directed to a screening web page to confirm that they had body image concerns (the sole inclusion criterion) and administered the Eating Disorder Diagnostic Scale [44]. No exclusion criteria were recommended. This study’s design and hypotheses were preregistered; see https://clinicaltrials.gov/ct2/show/NCT03409809. The Standards for Reporting Implementation Studies checklist has been included as an additional file. Other details of the study, including student group member characteristics, recruitment procedures, inclusion criteria, and the consideration of sample size are provided by Stice et al. [23] and Rohde et al. [45].

Although student group members were research participants for other aspects of this study, they did not provide any data during the sustainability period. Thus, the present report relies solely on outcome data provided by the peer educators and supervisors supervisors. Supervisors completed an interview and surveys before and after the initial TTT training assessing knowledge and attitudes towards eating disorder prevention, as well as the needs and resources (including the peer education program) on their campus. Supervisors then completed Qualtrics surveys assessing implementation progress at one-year (i.e., end of implementation), two-year, and three-year (i.e., end of Sustainability) follow-ups; the two- and three-year sustainability surveys were completed by one peer education supervisor at each college who had knowledge of the program’s delivery on campus but may not have completed previous assessments (due to staff turnover). Peer educators completed a survey after training assessing their perceptions of the characteristics of the Body Project. Neither supervisors nor peer educators received monetary compensation.

Sustainability outcome measures

Implementation and Sustainability Progress. Because we were unable to locate a measure that assesses adoption and implementation of prevention programs, we created the Prevention Implementation Progress Scale (PIPS). We drew general principles for our scale from the Progress Monitoring Instrument (PMI [47]) and the Stages of Implementation Completion measure (SIC [48]). Following the strategies used by these research teams, we delineated the major milestones of the implementation procedure for this prevention program. This measure assesses the following discrete and observable steps in the implementation process: (1) engagement, (2) consideration of feasibility, (3) readiness planning, (4) staff recruitment and training, (5) service initiation, (6) ongoing service provision, (7) adherence monitoring and (8) competency. Most steps were assessed with multiple items that were obtained by a Qualtrics survey (followed up with clarifying emails, if necessary). We used a preliminary version of the PIPS in a previous implementation project with the Body Project, where it showed predictive validity; Body Project participants in organizations that completed 90% or more versus fewer of the activities in the implementation process showed significantly greater pre-to-post declines in avoidance of life activities due to body image concerns and eating disorder symptoms (M d = 0.27). Herein, we report three outcome measures from the PIPS occurring in the two-year sustainability period: (1) whether the school conducted one or more Body Project groups (primary outcome), (2) whether the school trained new peer educators to conduct Body Project groups, and (3) whether the school trained one or more new peer education supervisors on this EBI.

Predictor variables

Implementation support strategies

Colleges were randomly assigned to receive TTT only (n = 21), TTT + TA (n = 21), or TTT + TA + QA (n = 21), which resulted in 264 Body Project groups implemented by 613 peer educators to 1,387 students in the one-year Implementation period following TTT. Peer educators and supervisors in all three support strategy arms were given the tools needed to conduct Body Project groups, including the intervention script and access to a facilitator support website (https://sticebodyprojectsupport.weebly.com). Implementation support strategies pairwise contrasts using dummy variables were included in the models described below and include TTT (= 0) vs. TTT + TA (= 1), TTT (= 0) vs. TTT + TA + QA (= 1), and TTT + TA (= 0) vs. TTT + TA + QA (= 1).

Train-The-Trainer strategy

All schools received TTT training via an on-campus 2-day TTT training that has been previously validated [21, 22] by an expert trainer. Training included: (1) reviewing the intervention theory, components, and evidence base, (2) roleplay delivery of sessions by teams of peer educators, (3) supervisory feedback from the trainer and supervisors to train peer educators, (4) completion of homework assignments, and (5) reviewing facilitation skills, common problems, assessment, and crisis response plans.

Technical assistance strategy

Schools randomized to TTT + TA received an additional ½ day of training to articulate goals, needs, leadership structure, communication, adoption options, and recruitment strategies. The expert trainer met with trainees to assess the practical needs and resources for successful implementation and sustainability. They also answered remaining questions.

Quality assurance strategy

Supervisors at schools randomized to TTT + TA + QA were offered monthly consultation calls with their trainer over a 1-year period following the TTT and TA training. These phone or video consultation meetings reviewed progress in conducting groups, problem-solving barriers, planning future implementation events, knowledge sharing, and sustainability planning.

Intervention, contextual, and implementation process factors

Perceived Characteristics of the Intervention. Following TTT training, peer educators and their supervisors completed a survey assessing perceptions of relative advantage, compatibility, complexity, observability, and trialability [49] in an organization. We used the 28-item adoption scale (the Provider Intervention Adoption Scale [50]), modified to refer to the Body Project. Response options were on a 5-point scale (1 = strongly disagree, 5 = strongly agree). Exploratory factor analyses identified two factors in the present data. The first factor included 19 items that measured positive perceptions of Body Project characteristics (α = 0.96 and 0.93 for peer educators and supervisors, respectively). The second factor included 9 items that measured negative perceptions of Body Project characteristics (α = 0.87 and 0.77 for peer educators and supervisors, respectively). All scales were averages across items unless otherwise indicated.

The Modified Practice Attitudes Scale (MPAS [51]) is an 8-item measure designed to assess therapists’ attitudes towards evidence-based practices. The eight items use a five-point Likert scale to indicate the extent to which the participant agrees or disagrees with each statement (0 = “not at all” and 4 = “to a very great extent”). Higher scores indicate more favorable attitudes towards evidence-based practices. The MPAS has demonstrated acceptable internal consistency (α = 0.77-0.80) in studies of community mental health providers [52]. A mean score was computed, and higher scores indicate greater support for evidence-based practices (α = 0.77).

Implementation process factors. We used 11 indicators of implementation progress collected from the PIPS during the 1-year Implementation period as a potential predictor of greater sustainability. Items addressed the school beginning recruitment for a Body Project group, beginning each of up to six groups during implementation, training new peer educators and supervisors in the intervention, and providing structured (observation-based) supervision of peer educator delivery of the Body Project. Schools received one point for completion of each activity, with scores ranging from 0–11.

Covariate adjustments

Outer and inner setting implementation indicators

Covariates hypothesized to contribute to the variance of sustainability outcomes are derived from the outer and inner settings domains of the Consolidated Framework for Implementation Research (CFIR [46]). For a longer description, see Rohde et al [53]. At baseline, we evaluated the presence or absence of formal policies related to EBPs, fiscal and other organizational resources for peer educator programs and the student mental health centers, and the mental health needs and resources of students using a standardized qualitative interview conducted with peer educator supervisors prior to the TTT training to assess the following CFIR domains: (1) Available Resources (i.e., How well-resourced [staffing, time, and budget] is the university?); (2) Access to Knowledge and Information (i.e., Does the supervisor feel that they can easily find new information and do they seek it out?); and (3) Leadership Engagement (i.e., How supported do supervisors feel by their supervisor at the university?). Interviews were audio recorded and transcribed, and responses were coded into 3-point quantitative scores per domain, with double coding to ensure acceptable inter-rater reliability: Available resources (0 = not supported in staffing, time budget; 1 = support in one area, 2 = support in 2 + areas); Access to knowledge and information (0 = no access, 1 = one outlet they can use to access information, 2 = materials are available to them from multiple outlets); and Leadership Engagement (0 = minimal/no involvement, 1 = feels support but leader could be more involved, 2 = feels supported).

The Team Climate Inventory [54] was completed by peer educators and supervisors at baseline to measure perceived organizational climate, an inner setting factor. This 14-item survey assesses team Vision (e.g., I am in agreement with the objectives of our peer education program), Participative Safety (e.g., People in the peer education program feel understood and accepted by each other), Task Orientation (e.g., Peer education team members are prepared to question the basis of what the team is doing), and Support for Innovation (e.g., In this peer education team we take the time needed to develop new ideas). Response options were on a 5-point Likert scale of agreement (1 = strongly disagree; 5 = strongly agree). It has shown internal consistency (α = 0.90—0.93) and predictive validity for team innovation and effectiveness [54,55,56]. For the current study, a total composite mean score was computed, with a higher score indicating greater levels of team climate. Internal consistency was high for peer educators and supervisors (α = 0.96 and 0.95, respectively), and scores were grand mean centered.

Data analysis

Preliminary analyses

We examined the distribution of variables, extent of missing data, summarized the sample of supervisors and peer educators, and examined the distribution and intercorrelations of the predictors.

Prediction models

We relied on colleges to provide data matching supervisor and peers to groups, however, less than 50% of colleges provided those data. Thus, multilevel modeling of supervisors and peers nested with groups was not possible and we conducted non-nested models with mean supervisor and peer-educator predictors for each school. We examined logistic regression models for dichotomous outcomes that were estimated with a logit link using SPSS (Version 26). All continuous predictor variables were mean-centered. For each outcome, we estimated one model for each of the nine predictors and each model simultaneously adjusted for the four covariates. The likelihood ratio test was used to evaluate model fit. The likelihood ratio test does not always perform well [57] and others have suggested the Hosmer–Lemeshow test [58] as an alternative, however, these require larger sample sizes, with some recommending up to 400 [59]. The Nagelkerke pseudo R2 was used to approximate percentage of variance accounted for [60] and was selected over the Cox and Snell pseudo R2 because the Cox and Snell pseudo R2 value cannot reach 1.0 [61]. Post modeling, we examined for the presence of multicollinearity by inspecting the variance inflation factor (VIF) for model predictors, with values greater than 10.0 indicating the presence of multicollinearity [62]. For effect sizes, we report odds ratios with 95% confidence intervals. Although we did not correct for multiple testing given limited sensitivity with the uncorrected analyses, we computed and report the Benjamini–Hochberg false discovery rate adjusted p-values [63] along with the unadjusted p-values in Table 3.

Results

Preliminary analyses

Table 1 provides a descriptive summary for the three outcomes. Due to the low endorsement of non-zero categories, we created three dichotomous (Yes/No) outcome measures for the variables. Complete data were available for all 63 colleges.

Table 1 Descriptive Summary of Outcome Measures
Full size table

We collected data from 232 supervisors and 510 peer educators. On average, there were 4.1 (SD = 2.0, minimum = 1, maximum = 12) supervisors and 10.0 (SD = 3.86, minimum = 3, maximum = 26) peer educators per college. The supervisors were mostly female (93%) and primarily reported their age as between 26 – 30 years of age (23%) followed by 31 – 35 years of age (20%) and 51 years or older (14%). The majority identified as White (74%), followed by Black (8%), Hispanic (7%), Asian (6%) and multi-racial (4%). Peer educators were also mostly female (96%), with a mean age of 21.0 years (SD = 3.5) and identified primarily as White (51%), followed by Hispanic (19%), Black (15%), Asian (10%), Native American (3%), and Pacific Islander (2%). A minority of the supervisors (13%) and peer educators (21%) reported a history of eating disorders, and most peer educators (75%) indicated that body image concerns kept them from doing something they enjoyed. We examined whether supervisor or peer educator self-reported demographic characteristics and history of eating disorder or body image concerns were significantly associated with randomized implementation support strategies. No significant associations were found (at p < 0.05).

The intercorrelations and descriptive statistics for the 9 predictors are summarized in Table 2. No multicollinearity among the predictors was detected and the largest bivariate correlation was between peer educator reports of positive and negative implementation of the Body Project (r = -0.56).

Table 2 Pearson Correlation Matrix and Descriptive Summary for Predictors
Full size table

Prediction models

Annotated parameter estimates from the logistic regression models for the three sustainability outcomes are shown in Table 3 and only include results for the model predictors. See Additional File 1 for results of the full models including parameter estimates for the covariates, and for each model the fit, estimated variance accounted for, and VIF for the predictors and covariates. One variable significantly predicted the primary outcome of a college peer education program conducting one or more Body Project groups in the 2-year sustainability period: completion of a greater number of PIPS activities during the first year of implementation (OR = 1.48).

Table 3 Annotated Results of Logistic Regression Models for College-level Predictors
Full size table

Two variables significantly predicted whether a new peer educator was trained during sustainability, which included the greater number of PIPS implementation activities (OR = 1.39). In addition, higher scores on peer educator positive report of the Body Project characteristics were significantly associated with greater odds of whether new peer educators were trained during sustainability (OR = 18.42).

No variables were identified that significantly predicted whether a new peer education supervisor was trained during the 2-year sustainability period. The only predictor variable that approached significance (OR = 5.46; p = 0.066) was whether the college had received the complete package of implementation support strategies (i.e., ongoing QA in addition to TTT and TA).

Discussion

The Body Project is a group preventive intervention for eating disorders that has proven effectiveness when offered as a peer educator-led service [18,19,20,21,22] and with differing levels of implementation support [23]. The current study is the first to test whether implementation support predicts the sustainability of the Body Project on college campuses. Sixty-three college campuses were randomly assigned to three implementation support strategies: TTT only, TTT + TA, and TTT + TA + QA. During a two-year sustainability period, supervisors reported on the number of additional Body Project groups conducted, as well as whether they trained new peer educators and supervisors. We hypothesized that schools randomized to the TTT + TA + QA strategy would be more likely to offer Body Project groups and would train additional peer educators and supervisors during the sustainability phase. Because theories of sustainability [32] and naturalistic studies of EBI implementation [27] suggest that intervention, provider, and implementation process factors may also influence whether interventions are continued, we also tested whether provider perceived characteristics of the Body Project, supervisor attitudes towards evidence-based practices, and the extent to which colleges completed implementation activities in the earlier stages predicted sustainability outcomes. Although the highest level of implementation support approached significance as a predictor of training new supervisors, completion of implementation activities emerged as the strongest predictor of sustainability.

Of the 63 colleges that participated in the study, 38 of them (60%) did not offer any Body Project groups following the active year of implementation and reported that no students participated in groups, perhaps in part due to disruptions caused by COVID-19 during follow-up. Among the colleges that did continue to offer groups, the number of groups reported ranged from one to 13, with an average of 4.3 groups representing on average 0.76% (minimum = 0.09, maximum = 4.34) of the eligible female student population. Likewise, most colleges (54%) did not train new peer educators or peer educator supervisors (76%). The finding that only 40% of colleges implemented the Body Project during the sustainability phase was not unexpected, as a prior review of EBI implementation efforts noted that programs are rarely sustained in full over time [25]. Indeed, the rates of Body Project sustainment compare favorably to the rates of sustainment described in a study of over 1,000 sites using the Universal Stages of Implementation Completion scale, which averaged between 25 to 37%, depending on the definition of sustainment used [64]. Nonetheless, this rate of discontinuation underscores the need for increased attention to the factors that bolster program continuation, particularly those factors that may be malleable.

We hypothesized that level of implementation support might be one malleable factor, however there were no significant differences by implementation support strategies for whether Body Project groups were offered or peer educators trained. Colleges that were randomly assigned to receive QA in addition to TTT and TA during the implementation stage appeared to be more likely to train new supervisors during the two-year sustainability period, relative to those who received TTT and TTT + TA, but this was only a trend. The odds ratio (OR = 5.46) corresponds to a moderate-to-large association [65] with a large confidence interval that may reflect the small sample size. Nonetheless, this trend makes intuitive sense given that QA calls specifically included a focus on sustainability planning, which would address continued training of supervisors. Supervisors in the study were trained not only to provide clinical oversight of peer educator-led groups but also to train new peer educators, an activity critical to the longevity of the Body Project. In a randomized trial of parent–child interaction therapy training, the training arm in which internal supervisors were trained to provide ongoing training and support resulted in more clients seen at 24 months following active implementation, even though there were no differences between this arm and the others (Learning Collaborative and Distance Education) in number of total clients seen since the initial training [35]. It may be that investing in trainers and supervisors results in more durable services over time, even if it does not initially predict greater reach.

The number of completed activities during the active implementation stage was the most robust predictor of sustainability outcomes. Prior research has shown that completing a greater proportion of activities in the early stages of implementation predicted program start-up in a study of multidimensional treatment foster care in 53 child-welfare serving sites, using different stages of the Stages of Implementation Completion [66]. Our measure of implementation progress likewise measured the proportion of completed activities across the different stages of implementation and we found that greater completion of critical activities during the implementation stage predicted whether groups were offered during the sustainability stage (OR = 1.53), as well as whether new peer educators were trained (OR = 1.39), both small but durable effects [65] that remained significant even after correction for multiple tests. This relationship could have emerged for a variety of reasons. Colleges that are more committed to an EBI may be more likely to complete both implementation activities and those during sustainability. Alternatively, colleges that completed fewer implementation activities may have encountered more challenges, including financial constraints, turnover of leaders or EBI champions, or other emergent priorities [66]. Regardless, completion of implementation activities may be a valuable litmus test for later sustainability and could inform decisions about when to invest resources over time.

Consistent with the DSF [32], peer educators’ perceptions of the characteristics of the Body Project following their initial training predicted whether new peer educators were trained (OR = 18.42) in the expected direction. This effect was very large in magnitude [65], though it became non-significant after correcting for multiple tests. Peer educators’ perceptions that the Body Project offered advantages relative to other programs, was compatible with the setting, was appropriately complex, and offered observable benefits predicted the training of new peer educators. This dovetails with the results of a study by Massey Combs and colleagues [31] in which higher teacher ratings of an intervention’s benefit, compatibility and complexity predicted intervention sustainability. Similarly, a study by Lau et al. [27] found that community therapists’ negative perception of particular EBIs predicted their discontinuation. In contrast, supervisors’ perceptions of the Body Project characteristics and evidence-based practices in general did not significantly predict any sustainability outcomes. In other studies, both provider and supervisor endorsement of EBIs has been suggested to foster sustainment [67,68,69]. The DSF specifically proposes that the better the fit between the intervention and the setting, the more likely that the intervention will be maintained over time. Peer educators are both the intervention providers and members of the student body for whom the Body Project is intended, perhaps lending them particular insight into the extent to which the Body Project was an optimal fit with their settings’ needs. For colleges where the Body Project wasn’t sustained, it is possible that program modifications could increase the likelihood of sustainment, as the DSF suggests that even programs considered already “optimized” may require adjustment to be maintained [32].

Limitations

The current study is among very few that have used randomization to test the effects of different implementation support strategies on the sustainability of an evidence-based intervention; even fewer studies have studied the sustainability of prevention interventions on college campuses. Despite this, several limitations should be considered. First, all sustainability outcomes were examined at the college level (N = 63) which resulted in less-than-optimal power to detect more than large effects. Although 63 organizations is a respectable sample for dissemination and implementation research projects, it is still a small sample in terms of statistical power. We only had a power of 0.68 to detect medium-sized effects (e.g., OR = 3.47), so we were likely unable to detect clinically meaningful effects that were small to medium in magnitude. Second, the active implementation stage of this study for all colleges took place in early 2020 at the start of the COVID-19 pandemic. This disrupted typical procedures for most college campuses and may have influenced a lack of initial implementation for a sizable number of colleges, 17% of whom did not implement the Body Project in the active phase. Perhaps related, most colleges did not engage in any of the sustainability activities examined as outcomes in this study. This resulted in low endorsement of non-zero categories and led us to use dichotomous (Yes/No) outcomes for most sustainability measures rather than continuous, which may have reduced statistical power that was already limited due to the relatively modest number of colleges that participated. In addition, during the sustainability phase, we no longer collected participant data so we cannot assess whether Body Project groups remained effective over time or whether there were differences in effectiveness by implementation support strategies. Intervention effectiveness is a critical sustainability outcome, and future studies should consider this alongside the outcomes measured in the current study. Third, the study sample size was small, especially in the context of dichotomous outcomes and logistic regression analyses. The small sample hampered the study in a few statistical ways. We were not able to use more appropriate indicators of model fit (e.g., Hosmer–Lemeshow test), explore potential better modeling options such as robust Poisson regression with estimates of risk ratios [70] or include more covariates adjustments in our prediction models. Finally, this study relied on peer educator supervisor self-reports of all outcomes, which may have biased the results.

Conclusions

The sustainability of evidence-based interventions has been described as “one of the most significant translational research problems of our time” [71] (p2). Any benefits yielded by the development, testing, adoption, and deployment of interventions are short-lived if practices are ultimately abandoned following the initial stages of implementation. In the current study, 60% of participating colleges did not sustain the Body Project after the initial, active implementation stage. Although sustainability rates in this study compared favorably to many EBI implementation efforts [64], even intentional efforts to plan for sustainability with the most intensive implementation support strategy did not significantly increase the likelihood of activities such as continuing to offer groups or training new providers.

It is likely that some of the factors that influence sustainability, such as funding and competing organizational priorities, were beyond the reach of the implementation support interventions in the current study [25]. In a qualitative study by Massatti et al [72], key warning signs that an organization might de-adopt a practice included lack of financial resources and evidence that external agents did not support continued implementation efforts. The current study included such outer and inner setting implementation indicators as covariates [46]; though very few were significant predictors of outcomes in any models (see Additional File 1). Other risks included goodness-of-fit of the intervention with employees’ skills and beliefs and perceptions about ease of implementation [72]. Assessing these risks during implementation, using measures like the Evidence Based Practice Sustainability Index [36], could allow for more personalized approaches to increase sustainability and prevent the de-adoption of EBIs.

The proportion of activities that a college completed during the active implementation stage emerged as the most robust predictor of whether Body Project groups were offered during the sustainability stage, as well as whether new peer educators were trained to lead future groups. This aligns with conceptual models that describe implementation as a process of moving through distinct stages in a potentially non-linear and recursive fashion, with each stage interacting with and influencing subsequent stages [73]. Studies of other EBIs in settings different from the current study have also found that the proportion of completed activities in one stage predicted overall success [66]; perhaps this type of progress assessment can be used to guide decision-making about whether an organization has received an adequate dose of support during one stage or whether additional efforts to assist with initial implementation may be required to yield ongoing results.

When implementing EBIs in complex settings, results of the current study suggest that provider positive perceptions of the intervention’s characteristics should be intentionally bolstered during training. Likewise, provider concerns about the relative advantage, compatibility, complexity, and benefit of interventions should be assessed and addressed—perhaps through modifications that increase goodness of fit. Future research should consider whether pre-implementation interventions that promote provider positive perceptions of the intervention might improve sustainability for organizations where baseline attitudes are less positive. Ultimately, sustainment of EBIs in complex service settings is critical to realizing the potential of intervention development, testing, and implementation and supporting meaningful improvements in mental health care.