SpringerLink
Log in

Factors Affecting Post-trial Sustainment or De-implementation of Study Interventions: A Narrative Review

  • Narrative Review
  • Published:
Journal of General Internal Medicine Aims and scope Submit manuscript

Abstract

In contrast to traditional randomized controlled trials, embedded pragmatic clinical trials (ePCTs) are conducted within healthcare settings with real-world patient populations. ePCTs are intentionally designed to align with health system priorities leveraging existing healthcare system infrastructure and resources to ease intervention implementation and increase the likelihood that effective interventions translate into routine practice following the trial. The NIH Pragmatic Trials Collaboratory, funded by the National Institutes of Health (NIH), supports the conduct of large-scale ePCT Demonstration Projects that address major public health issues within healthcare systems. The Collaboratory has a unique opportunity to draw on the Demonstration Project experiences to generate lessons learned related to ePCTs and the dissemination and implementation of interventions tested in ePCTs. In this article, we use case studies from six completed Demonstration Projects to summarize the Collaboratory’s experience with post-trial interpretation of results, and implications for sustainment (or de-implementation) of tested interventions. We highlight three key lessons learned. First, ineffective interventions (i.e., ePCT is null for the primary outcome) may be sustained if they have other measured benefits (e.g., secondary outcome or subgroup) or even perceived benefits (e.g., staff like the intervention). Second, effective interventions—even those solicited by the health system and/or designed with significant health system partner buy-in—may not be sustained if they require significant resources. Third, alignment with policy incentives is essential for achieving sustainment and scale-up of effective interventions. Our experiences point to several recommendations to aid in considering post-trial sustainment or de-implementation of interventions tested in ePCTs: (1) include secondary outcome measures that are salient to health system partners; (2) collect all appropriate data to allow for post hoc analysis of subgroups; (3) collect experience data from clinicians and staff; (4) engage policy-makers before starting the trial.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Weinfurt K. What is a pragmatic clinical trial: Why are we talking about pragmatic trials? NIH Pragmatic Trials Collaboratory. Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials Web site. https://rethinkingclinicaltrials.org/chapters/pragmatic-clinical-trial/what-is-a-pragmatic-clinical-trial/. Updated February 8, 2023. Accessed March 15, 2023.

  2. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: Understanding time lags in translational research. J R Soc Med. 2011;104(12):510-520.

    Article  PubMed  PubMed Central  Google Scholar 

  3. NIH Pragmatic Trials Collaboratory. About NIH collaboratory. https://rethinkingclinicaltrials.org/about-nih-collaboratory/. Accessed March 15, 2023.

  4. NIH Pragmatic Trials Collaboratory. Demonstration projects. https://rethinkingclinicaltrials.org/demonstration-projects/. Accessed March 15, 2023.

  5. Hamilton AB, Finley EP. Qualitative methods in implementation research: An introduction. Psychiatry Res. 2019;280:112516.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Taylor B, Henshall C, Kenyon S, Litchfield I, Greenfield S. Can rapid approaches to qualitative analysis deliver timely, valid findings to clinical leaders? A mixed methods study comparing rapid and thematic analysis. Bmj Open. 2018;8(10).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Huang SS, Septimus E, Kleinman K, et al. Chlorhexidine versus routine bathing to prevent multidrug-resistant organisms and all-cause bloodstream infections in general medical and surgical units (ABATE infection trial): A cluster-randomised trial. Lancet. 2019;393(10177):1205-1215.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Agency for Healthcare Research and Quality. Toolkit for decolonizaion of non-icu patients with devices. Agency for Healthcare Research and Quality. https://www.ahrq.gov/hai/tools/abate/index.html. Updated March 2022. Accessed March 15, 2023.

  9. Jarvik JG, Meier EN, James KT, et al. The effect of including benchmark prevalence data of common imaging findings in spine image reports on health care utilization among adults undergoing spine imaging a stepped-wedge randomized clinical trial. JAMA Network Open. 2020;3(9).

    Article  PubMed  PubMed Central  Google Scholar 

  10. Mitchell SL, Volandes AE, Gutman R, et al. Advance care planning video intervention among long-stay nursing home residents: A pragmatic cluster randomized clinical trial. JAMA Intern Med. 2020;180(8):1070-1078.

    Article  PubMed  PubMed Central  Google Scholar 

  11. McCreedy EM, Yang X, Mitchell SL, et al. Effect of advance care planning video on do-not-hospitalize orders for nursing home residents with advanced illness. BMC Geriatr. 2022;22(1):298.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Moyo P, Loomer L, Teno JM, et al. Effect of a video-assisted advance care planning intervention on end-of-life health care transitions among long-stay nursing home residents. J Am Med Dir Assoc. 2022;23(3):394-398.

    Article  PubMed  Google Scholar 

  13. Palmer JA, Parker VA, Barre LR, et al. Understanding implementation fidelity in a pragmatic randomized clinical trial in the nursing home setting:A mixed-methods examination. Trials. 2019;20(1):656.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Loomer L, McCreedy E, Belanger E, et al. Nursing home characteristics associated with implementation of an advance care planning video intervention. J Am Med Dir Assoc. 2019;20(7):804-809 e801.

    Article  PubMed  PubMed Central  Google Scholar 

  15. DeBar L, Mayhew M, Benes L, et al. A primary care-based cognitive behavioral therapy intervention for long-term opioid users with chronic pain : A randomized pragmatic trial. Ann Intern Med. 2022;175(1):46-55.

    Article  PubMed  Google Scholar 

  16. Smith DH, O'Keeffe-Rosetti M, Leo MC, et al. Economic evaluation: A randomized pragmatic trial of a primary care-based cognitive behavioral intervention for adults receiving long-term opioids for chronic pain. Med Care. 2022;60(6):423-431.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Coronado GD, Petrik AF, Vollmer WM, et al. Effectiveness of a mailed colorectal cancer screening outreach program in community health clinics: The STOP CRC cluster randomized clinical trial. JAMA Intern Med. 2018;178(9):1174-1181.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Petrik AF, Green B, Schneider J, et al. Factors influencing implementation of a colorectal cancer screening improvement program in community health centers: An applied use of configurational comparative methods. J Gen Intern Med. 2020;35(Suppl 2):815-822.

    Article  PubMed  PubMed Central  Google Scholar 

  19. O'Leary MC, Lich KH, Gu Y, et al. Colorectal cancer screening in newly insured medicaid members: A review of concurrent federal and state policies. BMC Health Serv Res. 2019;19(1):298.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Coronado GD GB. Direct mail programs work. . . But will health centers implement them? Findings from STOP CRC. In: Collaboraoty. NPT, ed. NIH Pragmatic Trials Collaboratory Grand Rounds WebinarFebruary 23, 2018.

  21. Coronado GD, Green BB, West, II, et al. Direct-to-member mailed colorectal cancer screening outreach for medicaid and medicare enrollees: Implementation and effectiveness outcomes from the benefit study. Cancer. 2020;126(3):540-548.

    Article  PubMed  Google Scholar 

  22. Curtis L. Implementation experiences from NIH pragmatic trials collaboratory. Paper presented at: Pain Management Collaboratory Steering Committee Annual Meeting; 2022; Bethesda, MD.

  23. National Association of Chronic Disease Directors, Kaiser Permanente Center for Health Research. Mailed FIT implementation guide. https://chronicdisease.org/wp-content/uploads/2022/06/Mailed-FIT-Guide-Revised-2022.pdf. Published 2022. Updated June 14. Accessed May 2, 2023.

  24. Hitchcock ME, Green BB, Anderson DS. Advancing health equity for medicaid beneficiaries by adding colorectal cancer screening to the centers for medicare and medicaid services adult core set. Gastroenterology. 2022;162(3):710-714.

    Article  PubMed  Google Scholar 

  25. Zatzick D, Moloney K, Palinkas L, et al. Catalyzing the translation of patient-centered research into United States trauma care systems: A case example. Med Care. 2021;59(Suppl 4):S379-S386.

    Article  PubMed  PubMed Central  Google Scholar 

  26. American College of Surgeons. Resources for optimal care of the injured patient (2022 standards). Bull Am Coll Surg. 2022.

  27. Zatzick D, Palinkas L, Chambers DA, et al. Integrating pragmatic and implementation science randomized clinical trial approaches: A pragmatic explanatory continuum indicator summary-2 (PRECIS-2) analysis. Trials. 2023;24(1):288.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated consolidated framework for implementation research based on user feedback. Implement Sci. 2022;17(1):75.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This work is supported within the National Institutes of Health (NIH) Health Care Systems Research Collaboratory by the NIH Common Fund through cooperative agreement U24AT009676 from the Office of Strategic Coordination within the Office of the NIH Director. This work is also supported by the NIH through the NIH HEAL Initiative under award number U24AT010961.

Author information

Authors and Affiliations

  1. Duke Clinical Research Institute, Duke University School of Medicine, Durham, NC, USA

    Terren Green BA, Hayden B. Bosworth PhD & Kevin P. Weinfurt PhD

  2. Department of Population Health Sciences, Duke University School of Medicine, Duke University, 215 Morris St., Suite 210, Durham, NC, 27708, USA

    Hayden B. Bosworth PhD, Kevin P. Weinfurt PhD & Devon K. Check PhD

  3. Department of Medicine, Duke University School of Medicine, Duke University, Durham, NC, USA

    Hayden B. Bosworth PhD

  4. Center of Innovation to Accelerate Discovery and Practice Transformation, Durham Veterans Affairs Health Care System, Durham, NC, USA

    Hayden B. Bosworth PhD

  5. Kaiser Permanente Center for Health Research, Portland, OR, USA

    Gloria D. Coronado PhD & Lynn DeBar PhD, MPH

  6. Kaiser Permanente Washington Health Research Institute, Seattle, WA, USA

    Beverly B. Green MD, MPH

  7. Irvine School of Medicine, University of California, Irvine, CA, USA

    Susan S. Huang MD, MPH

  8. Department of Radiology, University of Washington School of Medicine, University of Washington, Seattle, WA, USA

    Jeffrey G. Jarvik MD, MPH

  9. Department of Health Services, Policy, and Practice, School of Public Health, Brown University and Providence Veterans Administration Medical Center, Providence, RI, USA

    Vincent Mor PhD

  10. Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine, University of Washington, Seattle, WA, USA

    Douglas Zatzick MD

Authors
  1. Terren Green BA
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hayden B. Bosworth PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gloria D. Coronado PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lynn DeBar PhD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Beverly B. Green MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Susan S. Huang MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jeffrey G. Jarvik MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Vincent Mor PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Douglas Zatzick MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Kevin P. Weinfurt PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Devon K. Check PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Devon K. Check PhD.

Ethics declarations

Conflict of Interest:

Terren Green has nothing to declare.

Dr. Bosworth receives funding from the VA HSRD 08–027 senior career scientist award; consulting fees from Abbott, Imatar, Novartis, Sanofi, Vidya, Walmart, and WebMed; contracts with BeBetter therapeutics, Boehringer Ingelheim, Improved Patient Outcomes, Merck, NIH, Novo Nordisk, Otsuka, Sanofi, Elton John Foundation, Hilton Foundation, Pfizer, and Esperion. Dr. Bosworth served as a member of the board of directors of Preventric Diagnostic.

Dr. Coronado served as a scientific advisor for Exact Sciences from 2020 to 2022, and from 2021 to 2023, she received funding from Guardant Health through a contract with the Kaiser Permanente Center for Health to conduct a research study on the adherence to a commercially available blood test for colorectal cancer.

Dr. DeBar receives support for travel expenses to the National Colorectal Cancer Round Table meeting.

Dr. Beverly Green is a member of the National Colorectal Cancer Round Table Steering Committee, and receives reimbursement for travel to an annual in-person steering committee meeting, and no other compensation.

Dr. Huang receives grants from NIH, AHRQ, CDC, and CDC Foundation; consulting fees from Tennessee Hospital Association; speaker honoraria from Hong Kong Public Health Authority, Infection Control Branch, and Kuwait Ministry of Health (2nd Kuwait Infection Prevention & Control Conference); travel reimbursement for the Kuwait Infection Prevention & Control Conference; and conducted clinical trials and studies in which participating nursing homes or patients in hospitals have received contributed antiseptic soap or environmental cleaning products from Medline Inc and/or Xttrium.

Dr. Jarvik receives grant funding from the NIAMS/NIH grant number P30AR072572; royalties from Springer Publishing and Wolters Kluwer/UpToDate as an author; travel support for GE Healthcare/ GE-Association of University Radiologists Radiology Research Academic Fellowship (GERRAF); and serves as the President-elect of the Association of University Radiologists.

Dr. Mor has nothing to declare.

Dr. Zatzick receives grant funding from the NIH under grant numbers UH2MH106338-01/UH3MH106338-02 and R01 MH130460.

Dr. Weinfurt receives grant funding from the NIH.

Dr. Check is a co-investigator at the NIH Pragmatic Trials Collaboratory Coordinating Center funded by NIH grants U24AT009676 and U24AT010961.

Disclaimer:

The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or its HEAL Initiative.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Green, T., Bosworth, H.B., Coronado, G.D. et al. Factors Affecting Post-trial Sustainment or De-implementation of Study Interventions: A Narrative Review. J GEN INTERN MED 39, 1029–1036 (2024). https://doi.org/10.1007/s11606-023-08593-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11606-023-08593-7

KEY WORDS

Search

Navigation