Background

Randomized controlled trials (RCTs), including individual-, cluster-, and stepped-wedge, are considered the “gold standard” study design for identifying effective health-related innovations [1]. Compared to other types of study designs (e.g., quasi-experimental), RCTs are best equipped to minimize the multitude of threats to internal validity that can compromise the integrity and interpretation of trial results [2]. However, RCTs are often criticized for having poor external validity—that is, trial results do not accurately reflect the circumstances to which the results should or could be applied once the study is completed. The potential for trials to impact patient outcomes, care delivery, and population health is limited if trial results are neither generalizable nor applicable to the contexts in which they are intended to apply.

In recognition of this criticism, methodologists have considered ways in which RCTs can be designed to better balance both internal and external validity. First introduced by Schwartz and Lellouch (1967), one way of conceptualizing trials with more or less emphasis on external validity is along a multi-axial continuum from more explanatory to more pragmatic [3]. On one end of the continuum, explanatory trials are those that emphasize internal validity. Explanatory trials seek to understand if an intervention is effective under ideal contexts, which are often characterized as highly resourced, tightly controlled, and conducted in somewhat artificial settings. These trials are mainly concerned with understanding and testing hypotheses on the existence of particular mechanisms of action for a given innovation. On the other end of the continuum are pragmatic trials. These trials emphasize a balance between internal and external validity. Pragmatic trials seek to understand if innovations work in real-world contexts that closely reflect the settings in which the innovation is intended to be used [4].

Interest in pragmatic RCTs (pRCTs) has increased substantially in recent years as researchers, practitioners, and funders continue to recognize the need for innovations that are both effective and generalizable beyond a single trial. Many opportunities now exist to support the conduct of pRCTs, including educational courses (e.g., massive open online course, Pragmatic Randomized Controlled Trials in Health Care, hosted by edX [5]), trainings and research networks (e.g., National Institutes of Health [NIH] Health Care Systems Research Collaboratory [6]), online resources (e.g., interactive eBook, Pragmatic Trials: A Workshop Handbook [7];), and funding opportunity announcements [8, 9]. Tools for hel** research teams plan for trials along the explanatory-pragmatic continuum to match the overall intent and purpose of the trial are also available. One such tool is the PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2 [10]).

The PRECIS tool was first developed by Thorpe and colleagues (2009) and revised by Loudon and colleagues (2015) through a collaborative, iterative process involving 80 international trialists [10, 11]. Briefly, the PRECIS-2 tool operationalizes elements or characteristics of trials that make them more or less pragmatic; in doing so, it encourages trialists to make purposeful decisions about the trial design to match the intent and purpose of the trial along the explanatory-pragmatic continuum. PRECIS-2 identifies nine domains of a trial that can make it more explanatory or more pragmatic: eligibility, recruitment, setting, organization, flexibility (delivery), flexibility (adherence), follow-up, primary outcome, and primary analysis.

The PRECIS-2 tool is intended to be used by the research team when planning for a trial. Through interactive, team-based discussions, each of the nine domains of the tool are scored on a five-point scale, with a score of one reflecting characteristics of that domain as very explanatory and a score of five reflecting characteristics of that domain as very pragmatic. Scores for each trial domain are represented on a wheel, where trials that are more explanatory have scores toward the center of the wheel, and trials that are more pragmatic have scores toward the periphery of the wheel. Scores for each domain can vary across the explanatory-pragmatic continuum, such that a score for one domain in one trial may be very pragmatic and a score for another domain within the same trial may be very explanatory. Rarely are all nine domains in a single trial scored as a one (very explanatory) or scored as a five (very pragmatic). Scores may also fluctuate throughout the duration of the trial, as planned or unplanned changes to specific domains may be necessary (e.g., recruitment, eligibility). As there are no objectively preferred individual or collective domain scores for a trial, each domain should match the intent and purpose of that particular trial.

The PRECIS-2 tool has been used to help design over 500 RCTs (personal communication, K. Loudon, August 15, 2020), with hundreds more retrospective assessments reported in published papers [12, 13]. It has demonstrated good interrater reliability and modest discriminant validity [14] and is often a major component of trainings and workshops for both research teams [5, 6] and funding agency staff [15]. With increased use and application of the PRECIS-2 tool to a wide range of trials, some users have identified a few aspects of the tool that would benefit from greater clarity and guidance.

One common source of confusion encountered by users of the PRECIS-2 tool is how to apply and interpret all nine domains to trials where participants are healthcare professionals (rather than patients) and trials are testing provider-focused strategies (rather than health interventions [16, 17]). While the eligibility and recruitment domains are similarly interpreted for trials involving patients or providers, it is less clear how to differentiate and operationalize all domains for trials testing health interventions that target patients compared to trials testing strategies that target healthcare professionals. Although the PRECIS-2 toolkit [10] states that “participants may be patients and/or healthcare professionals, interventions may target patients (e.g., medication) or healthcare professionals (e.g., continuing education),” it stops short of specifying how to conceptualize and interpret all domains for trials with healthcare professionals and provider-focused strategies. As trials designed to test provider-focused strategies vary from those that test patient-level interventions, the PRECIS-2 tool should vary, as well.

To clarify this issue, we propose an extension and adaptation of the PRECIS-2 tool tailored to trials that test strategies to change providers’ behavior. Such trials are quite common in implementation science, healthcare delivery research, and quality improvement research, where the overall goal is often to increase the adoption of evidence-based health interventions, de-implement ineffective interventions, and/or improve healthcare delivery and service provision. Consistent with the literature, we consider provider-focused strategies as those that target providers’ behavior; examples include audit and feedback, continuing education, and external coaching, among others [18].

The PRECIS-2-Provider Strategies (PS) tool was developed through a multi-step process involving experts from the original PRECIS tool (2009; MZ), revised PRECIS-2 tool (2015; KL, MZ), and implementation scientists (WEN, DAC). A series of 2-h discussions were held to deliberate the proposed domains, descriptions, and scores of PRECIS-2-PS. The PRECIS-2 worksheet and domain examples were used as points-of-comparison to contrast domains for trials where participants are patients and the target of health interventions vs. trials where participants are providers and the target of strategies. We leveraged our content expertise and trial experience to generate examples of a score of 1 (more explanatory) and a score of 5 (more pragmatic) for all nine domains. While formal reliability and validity testing was considered beyond the scope of the preliminary development of the tool, we nonetheless applied the PRECIS-2-PS tool to a small sample of four diverse trial protocols for pilot testing. Three of us independently read each trial protocol, scored each domain, and highlighted relevant text that informed the domain score. Scores and selected text were compared across coders; discrepancies were discussed until consensus was reached, and refinements to the tool were made accordingly (e.g., examples of domain scores). The implementation resources (#4) and data collection (#7) domains generated the most discussion. As the conceptualization of these two domains deviates the most from PRECIS-2, additional discussion of scores and refinement of PRECIS-2-PS was not unexpected. The remaining seven domains (e.g., eligibility, recruitment) generated fewer discrepancies and discussions.

We describe the PRECIS-2-PS tool below. We define and operationalize each of the nine domains of the PRECIS-2-PS tool and provide examples of characteristics that make that domain more explanatory (score 1) or more pragmatic (score 5). We provide guidance for how to use the tool during the trial planning phase, including a comprehensive toolkit (see Additional file). Finally, we build on our experience using the PRECIS-2 tool in training workshops and trial consultations to emphasize emergent issues and apply them to the PRECIS-2-PS tool. These include the complementary role of different stakeholder groups for trial design and the importance of understanding and describing implementation-as-usual.

Discussion

PRECIS-2-PS domains

Consistent with the PRECIS-2 tool, we include a total of nine domains for PRECIS-2-PS. We changed the domain name, key question, and/or description for all domains to match the specific nature of provider-targeted trials. A comparison on domain names, key questions, and score examples between PRECIS-2 and PRECIS-2-PS can be found in Table 1. We used the same wheel-and-spoke visualization of PRECIS-2 for the PRECIS-2-PS tool. A blank version of the PRECIS-2-PS wheel is shown in Fig. 1.

Table 1 Comparison of PRECIS-2 and PRECIS-2-PS on domains, key questions, and scores
Full size table
Fig. 1
figure 1

The PRagmatic-Explanatory Continuum Indicator Summary 2 Provider Strategies (PRECIS-2-PS) wheel

Full size image

Domain #1: Eligibility

The eligibility domain refers to characteristics of healthcare professionals who would qualify to participate in the trial. Healthcare professionals may include nurses, physicians, allied health professionals, specialists, patient navigators, social workers, community health workers, and other individuals who provide relevant services. Reflecting a very explanatory intent of the trial, a score of 1 for the eligibility domain would have extensive exclusion criteria. The sample of healthcare professionals eligible for and included in the trial would be a restricted subset of the full population of healthcare professionals to whom the findings of the trial would apply outside the context of a trial. Reflecting a very pragmatic intent of the trial, a score of 5 for this domain would have minimal exclusion criteria. The sample of healthcare professionals eligible for and included in the trial would be a representative subset of the full population of healthcare professionals to whom the findings of the trial would be expected to apply outside the context of a trial.

Domain #2: Recruitment

The recruitment domain refers to approaches used to recruit and enroll eligible healthcare professionals into the trial. The recruitment domain assesses to what extent additional time, effort, and resources are used to recruit participants into the trial compared to approaches that would be readily available in similar settings. Reflecting a very explanatory intent of the trial, a score of 1 would include extensive approaches in terms of additional time, effort, resources, and personnel for recruiting providers into the trial. Examples might include personalized invitation letters, free continuing medical education credits, monetary incentives, or other approaches that are uncommon or unavailable in routine care settings. Reflecting a very pragmatic intent of the trial, a score of 5 for this domain would include relatively few approaches. These approaches would be feasible in routine care settings, such as announcements at staff meetings, word-of-mouth, or flyers posted in break rooms.

Domain #3: Setting

The setting domain refers to characteristics of the setting (namely, organizations) in which the trial is conducted. Examples of organizations include hospitals, clinics, health centers, health departments, and community-based organizations, among others. Organizational characteristics may include location, size, resources, payment structures, culture, climate, and performance metrics. This domain assesses to what extent the organizations in the trial reflect organizations to which the results would apply; that is, are organizations included in the study a representative sample of organizations to which the results would apply, or are they unique in some way that makes them a poor representation of organizations to which the results are intended to apply?

Reflecting a very explanatory intent of the trial, a score of 1 for the setting domain would include a sample of organizations that are unusual or atypical in ways that make them a poor reflection of the full population of organizations to which the results are intended to be applied. Examples include organizations that are limited to a small geographic region, significantly under or over resourced, exceptionally high or low performers on quality metrics, or a single rarified delivery system. Reflecting a very pragmatic intent of the trial, a score of 5 for this domain would include organizations in the trial that closely represent the total population of organizations to which the trial is intended to be applied. Examples include organizations that are dispersed across a large geographic region, located in rural, urban, and peri-urban settings, and average performance on quality metrics.

Domain #4: Implementation resources

The implementation resources domain refers to the time, effort, and personnel needed to support the delivery of provider-focused strategies. This domain compares the intensity of strategies tested in the trial relative to what is or would likely be available to deliver those strategies outside the context of the trial. A very explanatory intent of the trial (score of 1) would reflect resources to support the delivery of strategies above and beyond what would be reasonable or available in similar care settings. This would include the use of strategies that are time-intensive, costly, extensive, and frequent, such as one-on-one in-person coaching sessions, and expensive expert-led external facilitation. A very pragmatic intent of the trial (score of 5) would use resources to deliver provider-focused strategies that are available or accessible in these settings, such as quality improvement teams and collaboratives, educational sessions, skill-building seminars or workshops, or electronic health record systems.

Domain #5: Flexibility of provider-focused strategies

The flexibility of delivery of provider-focused strategies domain refers to the flexibility in how, when, and by whom the strategies are delivered to providers within the trial. This domain reflects the extent to which individuals who deliver the strategies, and the detailed aspects of delivery of the strategies themselves, are pre-specified (e.g., training, experience, credentials), the format in which they can be delivered (e.g., in-person, online), the frequency and sequence in which they can be delivered, and the flexibility in selection, use, and adaptation of strategies. Reflecting a very explanatory trial, a score of 1 would include trials with strict protocols and structures that do not allow for or encourage adaptations to strategies, regardless of context. Reflecting a very pragmatic trial, a score of 5 would include trials with flexible guides, suggestions, and manuals for delivering provider-focused strategies that allow for or even encourage the adaptation of strategies to meet the context of the organization and the needs of healthcare professionals.

Domain #6: Flexibility of intervention

The flexibility of intervention domain refers to the degree to which healthcare professionals are able to adapt their use of the patient-focused intervention to their situation relative to the likely flexibility that would be available during usual care. A score of 1 on the flexibility of intervention domain, reflecting a very explanatory approach, would include strict protocols or structures in place to discourage or limit the extent to which the patient-focused intervention could be adapted by the provider. A score of 5 on the flexibility of intervention domain, reflecting a very pragmatic approach, would include explicit suggestions or encouragement for healthcare professionals to adapt the patient-focused intervention to the context.

Domain #7: Data collection

The data collection domain is a function of both the frequency and intensity of data collected throughout the duration of the trial from baseline through follow-up. This domain includes how often data are collected from participants as well as how extensive, intrusive, or time-consuming it is for participants to provide data compared to what would be considered routine within similar care settings. In a very explanatory trial (a score of 1), data would be collected quite often and require extensive time and effort on behalf of participants to complete. Very explanatory trials would also likely include (or be limited to) original data collection, including quantitative and/or qualitative data. In a very pragmatic trial (a score of 5), data would be collected less often and require little effort and time for completion. Very pragmatic trials would rely heavily on secondary data collection—that is, data that are readily available within routine care (e.g., electronic health records), with minimal or no original data collection.

Domain #8: Primary outcome

This domain is the extent to which the primary outcome is of interest and importance to healthcare professionals. Primary outcomes may include direct provider-level outcomes or indirect patient-level outcomes. Reflecting a very explanatory approach, a score of 1 on this domain would include trials where the primary outcome is some type of process or proxy variable, where the overall intent is to better explain or understand a mechanism or mediating variable that might predict the degree to which the provider-focused strategy is effective. Outcomes that would reflect a very explanatory trial include providers’ knowledge about an intervention, perceptions of leadership, or patients’ completeness of laboratory testing (where these tests may not be part of routine care). Reflecting a very pragmatic approach, a score of 5 on this domain would include trials where the primary outcome is of obvious interest and importance to healthcare professionals, such as their quality of care or job satisfaction, or their patients’ health status or quality of life.

Domain #9: Primary analysis

Primary analysis refers to the extent to which all data are used to assess the primary outcome of the trial. A trial that is more explanatory (a score of 1) would use an on-protocol approach to test the main study hypothesis, where only data from healthcare professionals who consented, received the provider-focused strategy (for experimental condition only), and completed all process and outcome measures of the study are included in the primary analysis. A more explanatory trial may also use less-robust data imputation techniques to account for attrition and missing data. Trials on the pragmatic end of the continuum (a score of 5) would use an intent-to-treat (ITT) or robust modified ITT analysis for comparing the trial arms on the primary outcome variable for all participants who consented into the trial [19,20,21,22].

Enhancing the use of the PRECIS-2-PS tool

The PRECIS-2-PS tool is intended to be used during the trial planning phase to help team members consider structured, contextual elements of the trial that would match the overall intent and purpose of the trial. Building on our collective experience with PRECIS-2 and the published literature [23,24,25], we make explicit two major recommendations for enhancing the use of the tool that were not fully articulated in previous versions. The first recommendation is to involve stakeholders throughout the planning, execution, and interpretation of the trial. The second recommendation is to describe implementation-as-usual at the outset of the trial and document major changes that may occur as the trial unfolds. Together, we believe these approaches strengthen the ability of the tool to better understand and characterize the context in which the trial will take place, ultimately providing better guidance to decision-makers. These two recommendations do not themselves constitute domains of PRECIS-2-PS, but do facilitate its effective use in the design of trials of provider-focused strategies.

Stakeholders

Individuals involved in the trial planning phase should include representatives from research and practice to ensure that the overall intent of the trial reflects important perspectives in the decision-making process of the trial [23, 24]. Stakeholders would include (but not be limited to) members of the interdisciplinary scientific team, organizational and healthcare professional partners participating in the trial, and representatives from the settings in which the trial results are intended to be applied and adopted. All stakeholders would provide unique yet complementary input during the trial planning phase [26]. Researchers and practitioners would rely on expert and tacit knowledge about specific topics that would otherwise be inaccessible or likely inaccurate if not provided by that group. Examples include healthcare professionals identifying priority topic areas and important outcomes; statisticians providing power calculations to determine sample size; and health system leaders describing the feasibility and acceptability of the proposed provider-focused strategies. See Table 2 for more examples of the roles that stakeholders may play when planning for the trial. By involving these stakeholder groups, one would be able to address another important aspect of using PRECIS-2-PS: understanding, describing, and measuring implementation-as-usual.

Table 2 Stakeholder questions for designing provider-focused trials to match the overall purpose and intent of the trial
Full size table

Implementation-as-usual

To date, most RCTs of multi-level implementation strategies (including but not limited to provider-focused strategies) use either an implementation-as-usual control condition or provide basic implementation support (e.g., 1-h training workshop, educational materials) as an enhanced implementation comparison condition. In reflecting generalizability to implementation-as-usual, it is important for trial teams to explicitly identify to what populations and settings the intent is to be generalizable, and to carefully consider the current approach to implementation in those settings. Given the importance of planning for and interpreting the results of trials, describing and defining implementation-as-usual is critically important [27, 28].

Of course, as with usual care for patients, implementation-as-usual is by no means the same across delivery sites or healthcare professionals. Improved characterization of implementation-as-usual at all trial sites at baseline and over time will offer greater understanding of to whom and where the results of the trial may be applicable. This also highlights the importance of tracking how implementation-as-usual may change over time as a result of changes to policies, reimbursement structures, or reorganization, which may enhance or limit the generalizability of study results to settings in which they are intended to be used. Note that all of these considerations similarly apply to usual care for PRECIS-2-PS, where applicable (i.e., flexibility of intervention).

Conclusion

The PRECIS-2-PS tool for provider-focused implementation trials aims to help research and practice stakeholders plan for trials where design decisions are matched with the overall intent and purpose of the trial. The PRECIS-2-PS tool builds on PRECIS-2 by adapting all nine domains to trials testing provider-focused strategies. In doing so, it enables planning for provider-focused trials along the explanatory-pragmatic continuum, with implications for advancing the field, and ultimately maximizing the impact of trial results on improving patient outcomes, care delivery, and population health. The PRECIS-2-PS tool also responds to recent calls for a better understanding of implementation strategies. We incorporate this thinking into the PRECIS-2-PS tool by encouraging trial teams to better specify strategies [29, 30], consider adaptation of strategies [31, 32], costs of strategies [33], and describe and track changes to strategies, usual care, and implementation-as-usual over time [34] before and during the trial.

Research is needed to further develop, validate, and apply the PRECIS-2-PS tool to diverse provider-focused implementation trials. The PRECIS-2-PS tool should be formally tested for interrater reliability and discriminant validity, following the sequence of development of PRECIS-2. Applying the PRECIS-2-PS tool prospectively during trial planning, and retrospectively to published protocols, would generate an abundance of examples of domain scores, and identify aspects of the tool that may need to be refined or further specified. In addition to focusing on aspects of the tool itself, research is needed on how best to communicate trial domain scores from the PRECIS-2-PS tool. Dissemination research can help elucidate what type of information generated from the PRECIS-2-PS toolkit may be most useful for decision-makers and in what format.

Compendiums of trial case studies of PRECIS-2-PS scores would also better enable calibration of scores in the planning process and facilitate training in how to use the tool. This would help identify which PRECIS-2-PS domains are most likely to change over time and in what direction along the continuum. It would help characterize the state of provider-focused trials along the explanatory-pragmatic continuum, and identify what additional trials are needed to enhance the generalizability and applicability of strategies. It is important to remember, however, that not all provider-focused trials should be pragmatic. Explanatory trials are not inherently “bad” and pragmatic trials are not inherently “good.” Rather, each type of trial serves a purpose. Given the relatively nascent state-of-the-science of implementation strategies, one might expect to see more explanatory trials testing mechanisms of change now and more pragmatic trials testing strategies in diverse, representative settings later.

The PRECIS-2-PS tool is intended to encourage research and practice stakeholders to design provider-focused trials that match the overall intent and purpose of trials. The tool creates an opportunity to enhance research-practice partnerships by making explicit the importance and complementary role of stakeholders in the trial design phase. It also highlights the need for more and better provider-centered research [35] by focusing on priority issues, questions, and outcomes that are of interest to them. This approach is necessary for maximizing the potential of provider-focused implementation trials to have an impact once studies have been completed and papers have been published. The PRECIS-2-PS tool provides a structured approach for bringing stakeholders together in designing elements of a trial to match the intent and purpose of that trial, whether it be more explanatory, more pragmatic, or somewhere in between.