Introduction

The US Department of Veterans Affairs (VA) Office of Mental Health and Suicide Prevention (OMHSP) oversees mental health services across 140 VA medical centers in the United States. In 2013, OMHSP launched the Behavioral Health Interdisciplinary Program (BHIP), which aimed to organize outpatient mental health services for Veterans into a team-based format. Each BHIP team was intended to include a range of interdisciplinary staff (approximating 7.5 total staff, including both licensed and non-licensed clinicians, as well as administrative support personnel), responsible for providing mental health care to a panel of about 1,000 Veteran patients (Bauer et al., 2016).

More recently, OMHSP designated the Collaborative Chronic Care Model (CCM) as the guiding framework for organizing the clinical care provided by these BHIP teams (Smith et al., 2019; Von Korff et al., 1997; Wagner, Austin, & Von Korff, 1996; Woltmann et al., 2012). The overarching goal of CCM-based care is to ensure that patients are treated in a coordinated, patient-centered, and anticipatory manner. There are six elements in the CCM including: work role redesign, patient self-management support, provider decision support, clinical information systems, linkages to community resources, and organization and leadership support (Coleman et al., 2009). Randomized trials suggest that care organized around these principles results in improved patient outcomes in a cost-saving or cost-neutral manner (Miller et al., 2013; Woltmann et al., 2012).

In 2016 we partnered with OMHSP to conduct a stepped wedge hybrid implementation-effectiveness trial (the “BHIP-CCM Enhancement Project”) to implement CCM-based care in nine BHIP teams across the US. We used implementation facilitation—featuring an internal and external facilitator for each BHIP team (Kirchner et al., 2014)—as our implementation strategy, and found that clinicians in the enrolled BHIP teams experienced improvement in some aspects of collaborative team functioning (role clarity and an emphasis on team-wide goals). Veterans treated by the participating BHIP teams experienced fewer mental health hospitalizations than Veterans treated in other general mental health clinics within the same medical centers (Bauer, Weaver, et al., 2019a), at substantial cost savings to VA (Miller et al., 2020). CCM-based BHIP care was also associated with improved mental health-related quality of life for Veterans with three or more comorbid mental health conditions, although Veterans with fewer mental health diagnoses did not derive as much benefit (Bauer, Weaver, et al., 2019a).

Follow-up quantitative analyses, however, suggested that clinical gains from the BHIP-CCM Enhancement Project (e.g., reduced mental health hospitalization) had faded one year after facilitation ended (Bauer et al., 2021). This is consistent with other literature in demonstrating the difficulty of sustaining clinical improvements after implementation support is removed (Chinman et al., 2021; Jackson et al., 2021; Palinkas et al., 2013; Scheirer & Dearing, 2011; Shelton et al., 2018). Therefore, the goal of the current project was to use rigorous qualitative methods to determine the extent to which each of the six CCM elements was still present within the nine BHIP teams that participated in the BHIP-CCM Enhancement Project, three to four years after facilitation ended, to inform future CCM-oriented implementation efforts.

Method

Overview

The original BHIP-CCM Enhancement Project, conducted from 2016 to 2018, was a joint research-operations project, the research components of which were approved by the VA Central Institutional Review Board (IRB). Details of that study’s procedures, including implementation facilitation approach, can be found elsewhere (Bauer et al., 2016, 2019a, 2019b). The current analyses were approved as non-research by the VA Boston Research and Development (R&D) Service.

Sampling Approach

Consistent with other qualitative work (Hamilton & Finley, 2019; Miller et al., 2019; Sullivan et al., 2021), we aimed to recruit about three to four staff at each of the nine VA medical centers, including a multidisciplinary mix of mental health leaders and frontline BHIP staff. Whenever possible, for each site we aimed to recruit at least one BHIP clinician who had been present during the original BHIP-CCM Enhancement Project (i.e., had participated in implementation facilitation between 2016 and 2018). We began recruitment by contacting mental health leaders at each site and used snowball sampling to identify staff who were familiar with the ways that each site’s BHIP team had evolved since the completion of the BHIP-CCM Enhancement Project. Recruitment was conducted by email, and we obtained verbal informed consent from all participants.

Data Collection

We developed the interview guide collaboratively as a project team, based on the following: (a) the six CCM elements, (b) core constructs of the i-PARIHS implementation framework (Harvey & Kitson, 2015; Ritchie et al., 2022), (c) the current project’s focus on sustainability, and (d) the interview guides used in previous studies of BHIP-CCM implementation (Miller et al., 2019; Sullivan et al., 2021; see Supplemental File). Interview questions asked about the extent to which care processes within the BHIP team remained aligned with the 6 CCM elements described above. We supplemented these questions with open-ended items regarding the broader functioning and structure of the BHIP teams, to make sure that we captured potentially important constructs not explicitly reflected in our interview guide. Prior to the interviews, we developed an “Informational Sheet” for each site, summarizing the CCM-consistent care processes that had been implemented during the original BHIP-CCM Enhancement Project. We referenced this Informational Sheet for interview participants who had not been at the site during the original implementation effort, and asked them to reflect on the extent to which those CCM-consistent care processes were still in place, as well as any modifications that may have been made to these care processes over time. Interviews were recorded and professionally transcribed verbatim.

Data Analysis

Based on our previously developed codes related to the CCM framework (Sullivan et al., 2021), we used a directed content analysis approach (Hsieh & Shannon, 2005) to code interview transcripts using NVivo 12 software. Each member of our qualitative analytic team (JS, BK, CM, SC, ES, MB) independently coded two transcripts utilizing the CCM framework and definitions. We met weekly to discuss and come to 100% consensus on our coding structure.

After determining our coding structure was credible, each transcript was coded independently by two team members using our a priori coding structure. Upon completion of coding, each dyad met to come to 100% consensus on any coding disagreements. The coding assignments alternated among the analytic team members.

Upon completion of coding, one analytic team member summarized the evidence for each of the six CCM elements within each site across multiple participants. The data were reduced to create a summary describing the evidence within each CCM element and providing exemplar quotes. Then, the analytic team reviewed each site summary, which consisted of CCM summaries and supporting exemplar quotes during a full-team meeting. The full team reviewed the summarization of evidence and exemplar quotes. In cases where more information about the CCM element was needed, the lead team member from that site revisited the summaries displaying all coded transcript data for that element so that additional data describing processes within a CCM element could be appended.

Thus, the CCM element descriptions across all nine sites were confirmed via 100% consensus by 6 analytic team members. Once the team had come to consensus on CCM element sustainability across all nine sites, the analytic team utilized a cross-site matrix to compare the extent of CCM element sustainability across the nine sites looking for similarities and differences within each CCM element (Kim et al., 2020). The cross-site themes were then reviewed in weekly full-team meetings to come to 100% consensus on sustainability within each CCM element.

Results

Project Sample

We completed 30 interviews across the nine sites between February and July of 2021; each site had between 1 and 6 participants. The participant sample consisted of 14 psychologists (47%), 6 psychiatrists (20%), 7 social workers (23%), 2 nurses (7%), and 1 vocational rehabilitation specialist (3%). These participants represented a mix of frontline clinicians, team leads, and higher-level mental health managers: of the participants, twelve (36%) had been involved in the original BHIP-CCM Enhancement Project. We had initially attempted to recruit an additional 3 staff at the sites who either declined to participate or did not respond after three recruitment emails. We had an additional 23 contacts for potential recruitment but reached our recruitment goals before needing to contact them.

Sustainability of Care Practices Aligned with the CCM Elements

In describing our results below, we use words like “a few” and “several” to generally mean two to three or three to four sites, respectively. Table 1 summarizes results for each CCM element, described in more detail below. In each case below, and in Table 1, we briefly summarize how teams were doing prior to the sustainability phase for comparison (for more detail see Miller et al., 2019 and Sullivan et al., 2021).

Table 1 Key findings regarding sustainability by CCM element
Full size table

Work Role Redesign

At the completion of the BHIP-CCM Enhancement Project (Sullivan et al., 2021), several participating BHIP teams had established modified work roles to support more anticipatory, continuous mental health care. These efforts included: care coordinator roles to check in with patients or administer symptom questionnaires between sessions; orientation groups or open access slots to increase access to timely care; and standard operating procedures for following up after patients’ missed appointments to ensure continuity of care.

We found mixed results regarding the extent to which these CCM-consistent care practices were maintained three to four years later. At some sites, the retention (or even expansion) of nurse care coordinator roles was seen as helpful for covering crucial care management tasks (e.g., providing same-day services for patients who present for care without an appointment, coordinating care after Veterans were discharged from inpatient admissions, conducting follow-up appointments for high-risk patients, completing treatment plans, and following up with Veterans who had stopped engaging in mental health care). At other sites, however, nurses or social workers who had been assigned to care coordinator roles had to drop those roles, as workload pressures and high caseloads meant that they instead were required to deliver one-on-one clinical services to their own individually assigned patients. Even if they retained their care coordinator roles, at times they needed to significantly narrow the scope of their care coordination activities (e.g., to focus solely on monitoring Veterans taking antidepressants).

Similarly, in some cases, orientation groups for new patients were eliminated, with the burden of introducing new patients to the clinic instead falling on whichever clinician was conducting an individual intake assessment with that patient. One provider attributed this process to a combination of staff turnover and the onset of the COVID-19 pandemic, even though orientation groups were seen as valuable for Veterans:

“I'm disappointed because one of the things I really liked about the orientation classes is that most Veterans who came into them appreciated them… a lot of Veterans said to us explicitly, “this is the first time that I’ve gotten this much information about what's available” … And then as we became more short-staffed with turnover over the last year and a half, and then with COVID, maintaining those groups just became more and more of a challenge to do.” (Psychiatrist, Site #8)

Patient Self-Management Support

At the completion of the BHIP-CCM Enhancement Project, the most common manifestations of this CCM element consisted of an emphasis on evidence-based psychotherapies (EBPs) with self-management components (e.g., cognitive behavioral therapy), alongside individual treatment plans that incorporated co** skills patients could use to manage their symptoms between sessions. In some cases, BHIP teams had initially developed brochures or other patient education documents describing available services.

We found that many of these CCM-consistent care practices remained in place three to four years after facilitation support ended. Participants across several sites noted a consistent emphasis on EBPs that in some cases preceded the BHIP-CCM Enhancement Project, frequently supported by local champions responsible for reinforcing their use. For example, one participant described a robust approach to EBP delivery within their BHIP team, although ensuring ready access to those services remained a challenge:

“So training [in EBPs] we always support and recommend and encourage. We have a new EBP Coordinator and we have been working on approaching evidence-based therapies [that align with] the clinic’s needs… So that’s been really great. The challenge for the providers for the EBPs is access. I talk to all the providers like weekly, biweekly, and they say, “Yeah, I have two people I’d really like to do an EBP with. I don’t have room.” (Social Worker, Site #9)

In contrast, other clinical processes oriented toward hel** Veterans be active participants in their care (e.g., patient brochures or education documents) appeared to fall by the wayside. We note as well that patient orientation groups had also fulfilled a patient self-management role by informing Veterans of their available treatment options in the clinic. As noted under “Work Role Redesign” above, however, these groups were in many cases eliminated based on workload pressures or COVID-19 restrictions.

Provider Decision Support

At the completion of the BHIP-CCM Enhancement Project, provider decision support was provided in the form of: ongoing trainings in EBPs (mentioned above as also relevant to patient self-management support), medication algorithms to inform choices of psychotropic medications, and streamlined processes for soliciting feedback from, or referring patients to, other mental health clinics within the medical center.

Three to four years post-implementation, we found that EBP use and referral processes to other clinics generally remained in place (although in many cases these processes had in fact predated the BHIP-CCM Enhancement Project). However, respondents at one site noted that referral procedures within the BHIP team itself presented ongoing challenges, especially between disciplines (e.g., therapy referrals from psychiatrists to psychologists): in that case, within-team referrals required discussion within the team meeting along with two separate notes in the medical record to avoid the consult “getting lost” (Psychiatrist, Site 6).

Clinical Information Systems

At the completion of the BHIP-CCM Enhancement Project, clinical information systems typically consisted of measurement-based care (MBC, i.e. incorporation of routine screening measures or symptom assessments to guide ongoing care). Efforts to aggregate data across the entire BHIP team’s panel were less common. Three to four years post-implementation, data aggregation across the team’s panel of patients remained uncommon, and some sites that had made initial progress toward such aggregation had abandoned their efforts after facilitation ended.

In contrast, we found increased emphasis on MBC within BHIP teams, with the administration of many measures being more fully automated and incorporated into clinic workflows and the medical record. At some sites this increased emphasis on MBC was seen as helpful for addressing clinic workflows, potentially with the help of a separate MBC Champion from outside of the BHIP team, with one respondent noting:

“We had an MBC champion here, a psychologist, who really did a great job with it. And she'd present every month at the staff meeting and talk about measurement-based care and a case that she had used it successfully in… so we were really trying to use measurement-based care to help us to move people through and maybe out the back door… to a group, or something else, or just back to primary care.” (Mental Health Leader, Site #9)

Linkages to Community Resources

At the completion of the BHIP-CCM Enhancement Project, connections to community resources (e.g. Vet Centers that provide mental health services to Veterans outside of larger VA medical centers, non-VA clinicians, Alcoholics Anonymous groups, or Veteran Service Organizations) were frequently handled on an individual clinician basis. In some cases, BHIP teams had attempted to consolidate their knowledge about resources available in the community (e.g. by develo** compendia based on their collective knowledge, or by designating one BHIP team member [typically a social worker] to serve as their primary source of information about community resources).

Three to four years post-implementation, there was significant diversity among our respondents regarding how these community linkages were established, documented, and maintained. In some cases, social workers or nurses remained the primary resource for BHIP team members seeking knowledge about local community resources. Other sites had redoubled efforts to develop a consolidated pamphlet or guidebook of local resources that was easily accessible for providers:

"The [community resources pamphlet is] in [a VA-approved messaging software], which is really easy to access since we have [the messaging software] up all day. That's one of those documents that, you know, we can just turn to in a flash." (Psychologist, Site #6)

Others left it up to individual BHIP clinicians to decide whether and how to seek those resources for their patients. Two sites had “good working relations” with local Vet Centers, which were maintained by having monthly meetings involving both VA and Vet Center staff.

We also note that, consistent with the goal of giving Veterans greater choice over their healthcare, many medical centers had established separate community care offices explicitly tasked with managing referrals to non-VA clinics. Thus, participants noted that their connections to these community resources were in some cases mediated through those community care offices.

Organization and Leadership Support

At the completion of the BHIP-CCM Enhancement Project, BHIP team members noted mixed leadership support for their teams. Three to four years post-implementation, participants described further divergence, with some sites noting reduced support from leadership—one respondent noted that a newly assigned Associate Chief of Staff “pretty much single-handedly dismantled BHIP” (Staff Psychologist, Site #1) at their site.

Meanwhile, others reported continued or even increased emphasis on delivering CCM-based mental health care within their BHIP teams. This support could include: blocking clinic schedules to accommodate daily huddles or weekly team meetings; funding BHIP manager or team lead roles; working with Human Resources to prioritize hiring of additional BHIP clinicians; and supporting workflow projects to help BHIP teams discharge patients who have met their treatment goals to a less intensive level of care (e.g., by shifting the locus of their mental health care back to primary care). Among these components, two emerged as particularly salient for our respondents: (a) blocking time for BHIP staff to meet (described as “really crucial” by one respondent [Staff Psychologist, Site #8]), and (b) appropriately staffing the BHIP team to ensure a manageable workload for the BHIP team members (described as a “huge” issue by another respondent, a BHIP Program Manager from Site #2). Another respondent concluded that executive or leadership engagement was the single biggest factor in securing sustainability of this approach to care:

“If I was ever to make recommendations on how to make BHIP better on a more macro level, you have to get your hospital leadership on board. You’ve got to [convince them that] we do have research and evidence now that concludes that, when you do this model successfully, it can keep people from utilizing hospital resources as much.” (BHIP Team Lead, Site #7)

Several participating sites with robust leadership support were able to maintain CCM-consistent care practices even in the face of staff turnover and disruptions associated with the COVID-19 pandemic.

In addition to findings related to the sustainability of the CCM elements described above, we note that many of the BHIP teams that had participated in the BHIP-CCM Enhancement Project underwent important structural changes since that project ended. For example, one team had been disbanded entirely after several team members retired or were reassigned in a three-month span. Most of the remaining BHIP teams maintained weekly team meetings and/or brief daily huddles to discuss case assignment or patients in need of more intensive follow-up, but these meetings transitioned to a virtual format during the COVID-19 pandemic.

Discussion

Sustaining evidence-based practices is challenging, and even successful health interventions may be abandoned in the face of changing priorities, workload pressures, and staff turnover (Chambers et al., 2013; Chinman et al., 2021; Jackson et al., 2021; Palinkas et al., 2013; Scheirer & Dearing, 2011; Shelton et al., 2018). Thus, increasing the sustainability of such health interventions—thereby maintaining the clinical gains associated with them—is a major public health goal. In this manuscript we described the sustainability of care practices aligned with the collaborative chronic care model (CCM) across nine outpatient mental health teams (Bauer et al., 2019a). Previously analyzed quantitative data suggested that clinical gains from our implementation efforts (e.g. reduced mental health hospitalizations for Veterans being treated by the team) had faded by one year post-implementation, suggesting the need to evaluate the extent to which CCM-consistent care practices had been sustained. While other studies have assessed CCM sustainability over shorter time frames in primary care settings (Moise et al., 2018; Palinkas et al., 2010), to our knowledge this is the first qualitative assessment of such sustainability in a mental health setting three to four years post-implementation. Specifically, our analyses suggested variable sustainability of CCM-consistent care practices within nine outpatient mental health (BHIP) teams that had undergone one year of implementation facilitation (Kirchner et al., 2014).

A highlight of our findings, seen relatively consistently across these sites, was that these teams continued to emphasize the delivery of EBPs to their Veterans (relevant to the CCM elements of patient self-management support and provider decision support). This emphasis could include periodic trainings on EBP delivery for clinical staff, frequently established by a facility-level EBP Coordinator, as well as scheduling procedures to allow Veterans to complete these EBPs despite heavy clinic workloads. Generally, consult procedures to refer Veterans to other clinics within the medical center (relevant to the CCM element of provider decision support) were widely used, having been embedded in the medical record system at all participating sites. Similarly, periodic clinical assessments in the context of ongoing clinical care (MBC, relevant to the CCM elements of provider decision support and clinical information systems) remained widespread within these BHIP teams, having been supported by local MBC champions.

In contrast, other CCM-consistent care practices—which had in some cases been developed with considerable effort during the original BHIP-CCM Enhancement Project—did not demonstrate robust sustainability across participating sites. For example, care manager roles (relevant to the CCM element of work role redesign) were in several cases either eliminated completely or vastly diminished in scope in response to workload pressures, the need for more frontline care delivery, or the COVID-19 pandemic. Similarly, several processes relevant to the CCM element of patient self-management support (e.g. Veteran orientation groups, brochures, or processes for allowing Veterans to participate in clinical team meetings) had generally been scaled back or eliminated at these sites.

We observed the CCM element of leadership and organizational support was a key support undergirding each of the other CCM elements as hypothesized in the CCM. Specifically, interview respondents noted that consistent leadership support for CCM-based BHIP teams, especially in the form of appropriate staffing and dedicated time for BHIP team meetings, was crucial to sustaining this model of care delivery. Other related work (Miller et al., 2022) has suggested that a “two-by-two” approach to team staffing, with each discipline having at least two representatives on the clinical team, may help protect against the negative impacts of staff turnover, as it allows for some level of continuity within the team even if one person leaves. In contrast, in the current project, in one case the presence of one influential non-supportive leader (e.g., an Associate Chief of Staff) resulted in the dismantling of several years of BHIP-related process improvement work.

In summary, many of the CCM-consistent care practices that appeared to achieve robust sustainability—including EBP delivery, MBC, and established referral processes to other clinics embedded in the medical record—benefited from system-wide emphasis. This was frequently demonstrated with the support of one or more facility-wide support roles (e.g. EBP Coordinator, MBC champion, and Clinical Applications Coordinator). In many cases these roles predated the BHIP-CCM Enhancement Project. This is consistent with a conceptualization of the BHIP team as a clinical microsystem (Barach & Johnson, 2006), and suggests that establishing effective, CCM-oriented BHIP teams likely requires infrastructure, organizational readiness, and coordination at the level of the mental health clinic or service, rather than being conceptualized primarily as a team-level project (Bauer, Weaver, et al., 2019b).

Strengths and Limitations

As noted above, the follow-up period (three to four years after completion of the BHIP-CCM Enhancement Project’s facilitation) is a strength of these analyses, as previous analyses of care model sustainability (particularly in mental health settings) have typically featured shorter follow-up times, with a few notable exceptions (e.g. Peterson et al., 2014). In addition, more than one-third of our interview respondents had been present at these sites when implementation facilitation was originally conducted, and we were able to present our current findings in the context of previous interviews at these same sites (Sullivan et al., 2021) to present a robust picture of CCM sustainability.

Nonetheless, our findings should be interpreted in the context of certain limitations. First, while our sample size compares favorably with other recent reports (Young & Casey, 2019), we had between one and three interview respondents at several participating sites. It is possible that a more robust sample would have uncovered additional findings. Second, our analysis focused on identifying commonalities across sites regarding sustainability of CCM elements. While this allowed us to identify broad-based patterns, important differences between sites were not highlighted. Third, while we note some factors that may have impacted sustainability of these efforts (e.g. leadership support, facility-level roles such as MBC champions or EBP Coordinators), our analyses do not allow us to comprehensively describe the full range of influencing factors that may explain how CCM sustainability was (or was not) achieved. We are currently conducting follow-up analyses to address these latter two limitations, by applying a matrixed multiple case study approach (Kim et al., 2020), built around the i-PARIHS framework (Harvey & Kitson, 2015), to better identify the contextual factors (e.g. staff morale and attrition, workload pressures, and COVID-related disruptions in clinical operations) that may explain why some sites may have achieved better CCM sustainability than others. Fourth, we note that the COVID-19 pandemic impacted CCM sustainability at these sites. However, most respondents noted that, to the extent that CCM elements were not sustained at their sites, the shift away from such care practices typically predated the pandemic. Finally, this project was conducted within VA, an integrated, national, capitated care health system. While this is of course ideal for learning how to improve VA-based care (or similarly structured healthcare systems) in the future, we urge caution in generalizing these results to less similar settings.

Implications for CCM Sustainability

These findings have two noteworthy implications for improving the sustainability of CCM-consistent care practices in mental health teams. First, consistent with previous literature (and the CCM itself), strong leadership support is crucial to the sustainability of care models like the CCM. Given the potential impact of leadership turnover on sustainability in mental health (Peterson et al., 2014), future efforts to implement CCM-based mental health care should strive to involve and engage multiple leaders across organizational levels in implementation and sustainment efforts, so sustainability efforts are not halted by new or unsupportive leaders. Second, our findings suggest that sustaining CCM-based care practices within a specific mental health team may be most effectively pursued at the clinic or service level, as many of the factors that impact a given team’s care practices may ultimately be decided at those levels. For example, CCM-consistent care processes related to clinical referrals, linkages outside of the medical center, or EBP trainings for clinicians may be best coordinated by the mental health clinic more broadly, rather than having each BHIP team develop their own independent processes. It remains to be seen how best to accomplish this while also incorporating the views, feedback, and clinical experience of the frontline care team itself. Future CCM implementation efforts by our group within VA are currently being pursued, with increased emphasis on discussing sustainability in a robust way at the outset of, and throughout, the implementation process. Ideally, incorporating these findings will enhance the sustainability of similar efforts within and beyond VA.