Background

Sco** studies (or sco** reviews) represent an increasingly popular approach to reviewing health research evidence [1]. However, no universal sco** study definition or purpose exists (Table 1) [1, 2]. Definitions commonly refer to 'map**,' a process of summarizing a range of evidence in order to convey the breadth and depth of a field. Sco** studies differ from systematic reviews because authors do not typically assess the quality of included studies [3–5]. Sco** studies also differ from narrative or literature reviews in that the sco** process requires analytical reinterpretation of the literature [1].

Table 1 Definitions and purposes of sco** studies
Full size table

Researchers can undertake a sco** study to examine the extent, range, and nature of research activity, determine the value of undertaking a full systematic review, summarize and disseminate research findings, or identify gaps in the existing literature [6]. As such, researchers can use sco** studies to clarify a complex concept and refine subsequent research inquiries [1]. Sco** studies may be particularly relevant to disciplines with emerging evidence, such as rehabilitation science, in which the paucity of randomized controlled trials makes it difficult for researchers to undertake systematic reviews. In these situations, sco** studies are ideal because researchers can incorporate a range of study designs in both published and grey literature, address questions beyond those related to intervention effectiveness, and generate findings that can complement the findings of clinical trials.

In an effort to provide guidance to authors undertaking sco** studies, Arksey and O'Malley [6] developed a six-stage methodological framework: identifying the research question, searching for relevant studies, selecting studies, charting the data, collating, summarizing, and reporting the results, and consulting with stakeholders to inform or validate study findings (Table 2). While this framework provided an excellent methodological foundation, published sco** studies continue to lack sufficient methodological description or detail about the data analysis process, making it challenging for readers to understand how study findings were determined [1]. Arksey and O'Malley [6] encouraged other authors to refine their framework in order to enhance the methodology.

Table 2 Overview of the Arksey and O'Malley methodological framework for conducting a sco** study
Full size table

In this paper, we apply our experiences using the Arksey and O'Malley framework to build on the existing methodological framework. Specifically, we propose recommendations for each stage of the framework, followed by considerations for the advancement, application, and relevance of sco** studies in health research. Continual refinement of the framework stages may provide greater clarity about sco** study methodology, encourage researchers and clinicians to engage in this process, and help to enhance the methodological rigor with which authors undertake and report sco** studies [1].

Discussion

We each completed a sco** study in separate areas of rehabilitation using the Arksey and O'Malley framework [6]. Goals of these studies included: identifying research priorities within HIV and rehabilitation [7], applying motor learning strategies within pediatric physical and occupational therapy intervention approaches [8], and exploring the use of theory within studies of knowledge translation [9]. The amount of literature reviewed in our studies ranged from 31 (DL) to 146 (KO) publications. Upon discovering that we had similar challenges implementing the sco** study methodology, we decided to use our experiences to further develop the existing framework. We conducted an informal literature search on sco** study methodology. We searched CINAHL, MEDLINE, PubMed, ERIC, PsycInfo, and Web of Science databases using the search terms 'sco**,' 'sco** study,' 'sco** review,' and 'sco** methodology' for papers published in English between January 1990 and May 2010. Reference lists of pertinent papers were also searched. This search yielded seven citations that reflected on sco** study methodology, which were reviewed by one author (DL). After independently considering our own experiences utilizing the Arskey and O'Malley [6] framework, we met on seven occasions to discuss the challenges and develop recommendations for each stage of the methodological framework.

Recommendations to enhance sco** study methodology

We outline the challenges and recommendations associated with each stage of the methodological framework (Table 3).

Table 3 Summary of challenges and recommendations for sco** studies
Full size table

Framework stage one: Identifying the research question

Sco** study research questions are broad in nature as the focus is on summarizing breadth of evidence. Arksey and O'Malley [6] acknowledge the need to maintain a broad scope to research questions, however we found our research questions lacked the direction, clarity, and focus needed to inform subsequent stages of the research process, such as identifying studies and making decisions about study inclusion. To clarify this stage, we recommend that researchers combine a broad research question with a clearly articulated scope of inquiry. This includes defining the concept, target population, and health outcomes of interest to clarify the focus of the sco** study and establish an effective search strategy. For example, in one author's (KO) sco** study, the research question was broadly 'what is known about HIV and rehabilitation?' Defining the concept of 'rehabilitation' was essential in order to establish a clear scope to the study, guide the search strategy, and establish parameters around study selection in subsequent stages of the process [7].

Although Arskey and O'Malley [6] outline four main purposes for undertaking a sco** study, they do not articulate that purpose be specified within a specific framework stage. We recommend researchers simultaneously consider the purpose of the sco** study when articulating the research question. Linking a clear purpose for undertaking a sco** study to a well-defined research question at the first stage of the framework will help to provide a clear rationale for completing the study and facilitate decision making about study selection and data extraction later in the methodological process. A helpful strategy may be to envision the content and format of the intended outcome that may assist researchers to clearly determine the purpose at the beginning of a study. In the abovementioned HIV study, authors linked the broadly stated research question with a more specific purpose 'to identify the key research priorities in HIV and rehabilitation to advance policy and practice for people living with HIV in Canada' [7]. The envisioned outcome was a thematic framework that represented strengths and opportunities in HIV rehabilitation research, followed by a list of the key research priorities to pursue in future work.

Finally, the purposes put forth by Arksey and O'Malley [6] require more debate. We concur with Anderson et al.[2] and Davis et al.[1], who state that researchers may benefit from further clarification of the purposes for undertaking a sco** study. The first purpose, as articulated by Arksey and O'Malley [6], is to summarize the extent, range, and nature of research activity; however, researchers are not required to reflect on their underlying motivation for doing so. We recommend that researchers consider the rationale for why they should summarize the activity in a field and the implications that this will have on research, practice, or policy. The second purpose is to assess the need for a full systematic review. However, it is difficult to determine whether a systematic review is advantageous when a sco** study does not involve methodological quality assessment of included studies. Furthermore, it is unclear how this purpose differs from existing methods of determining feasibility for a systematic review. The third purpose is to summarize and disseminate research findings, but we question how this differs from other narrative or systematic literature reviews. Lastly, the fourth purpose of undertaking a sco** study -- to identify gaps in the existing literature -- may yield false conclusions about the nature and extent of those gaps if the quality of the evidence is not assessed. The purpose 'to identify the key research priorities in HIV and rehabilitation to advance policy and practice for people living with HIV in Canada' does not explicitly align with one of the four Arskey and O'Malley purposes [7]. However, it appears authors inherently first summarized the extent, range, and nature of research (purpose one) and identified gaps in the existing literature (purpose four) in order to subsequently identify the key research priorities in HIV and rehabilitation (author purpose). This suggests authors might have an overall study purpose with multiple objectives articulated by Arksey and O'Malley that are required in order to help achieve their overall purpose.

Framework stage two: Identifying relevant studies

A strength of sco** studies includes the breadth and depth, or comprehensiveness, of evidence covered in a given field [1]. However, practical issues related to time, funding, and access to resources often require researchers to consider the balance between feasibility, breadth, and comprehensiveness. Brien et al.[5] reported that their search strategy yielded a vast amount of literature, making it difficult to determine how in depth to carry out the information synthesis. Although Arksey and O'Malley [6] identify these concerns and provide some suggestions to support these decisions, we also struggled with the trade-off between breadth and comprehensiveness and feasibility in our sco** studies. As such, we recommend that researchers ensure decisions surrounding feasibility do not compromise their ability to answer the research question or achieve the study purpose. Second, we recommend that a sco** study team be assembled whose members provide the methodological and context expertise needed for decisions regarding breadth and comprehensiveness. When limiting scope is unavoidable, researchers should justify their decisions and acknowledge the potential limitations of their study.

Framework stage three: Study selection

Arksey and O'Malley [6] provide suggestions to manage the time-consuming process of determining which studies to include in a sco** study. We experienced this stage as more iterative and requiring additional steps than implied in the original framework. While Arksey and O'Malley [6] do not indicate a team approach is imperative, we agree with others and suggest sco** studies involve multidisciplinary teams using a transparent and replicable process [2, 10]. In two of our studies (HC and DL) where decision making was primarily completed by a single author, we faced several challenges, including uncertainty about which studies to include, variables to extract on the data-charting form, and the nature and extent of detail to conduct the data extraction process. This raised questions related to rigor and led to our recommendations for undertaking a systematic team approach to conducting a sco** study.

Specifically, we recommend that the team meet to discuss decisions surrounding study inclusion and exclusion at the beginning of the sco** process. Refining the search strategy based on abstracts retrieved from the search and reviewing full articles for study inclusion is also a critical step. We recommend that at least two researchers each independently review abstracts yielded from the search strategy for study selection. Reviewers should meet at the beginning, midpoint, and final stages of the abstract review process to discuss any challenges or uncertainties related to study selection and to go back and refine the search strategy if needed. This can help to alleviate potential ambiguity with a broad research question and to ensure that abstracts selected are relevant for full article review. Next, two reviewers should independently review the full articles for inclusion. When disagreements occur, a third reviewer can be consulted to determine final inclusion.

Framework stage four: Charting the data

This stage involves extracting data from included studies. Based on our experiences, we were uncertain about the nature and extent of information to extract from the included studies. To clarify this stage, we recommend that the research team collectively develop the data-charting form to determine which variables to extract that will help to answer the research question. Secondly, we recommend that charting be considered an iterative process in which researchers continually update the data-charting form. This is particularly true for process-oriented data, such as understanding how a theory or model has been used within a study. Uncertainty about the nature and extent of data that should be extracted may be resolved by researchers beginning the charting process and becoming familiar with study data, and then meeting again to refine the form. We recommend an additional step to charting the data in which two researchers independently extract data from the first five to ten studies using the data-charting form and meet to determine whether their approach to data extraction is consistent with the research question and purpose. Researchers may review one study several times within this stage. The number of researchers involved in the data extraction process will likely depend upon the number of included studies. For example, in one study, authors had difficulty develo** one data-charting form that could apply to all included studies representing a range study designs, reviews, reports, and commentaries [7]. As a preliminary step, authors decided to classify the included studies into three areas --HIV disability, interventions, and roles of rehabilitation professionals in HIV care -- to help determine the nature and extent of information to extract from each of the types of studies [7].

Arksey and O'Malley [6] refer to a 'descriptive analytical method' that involves summarizing process information, such as the use of a theory or model in a meaningful format. Our experiences indicated that this is a highly valuable, though challenging aspect of sco** studies, as we struggled to chart and summarize complex concepts in a meaningful way. Arksey and O'Malley [6] indicate that synthesis of material is critical as sco** studies are not a short summary of many articles. We agree, and feel that additional direction in the framework might help to navigate this crucial but challenging stage. Perhaps synthesizing process information may benefit from utilization of qualitative content analysis approaches to make sense of the wealth of extracted data [11]. This issue also highlights the overlap with the next analytical stage. The role and relevance of analyzing process data and using qualitative content analysis within sco** study methodology requires further discussion.

Framework stage five: Collating, summarizing, and reporting the results

Stage five is the most extensive in the sco** process, yet it lacks detail in the Arksey and O'Malley framework. Sco** studies have been criticized for rarely providing methodological detail about how results were achieved [1]. We appreciate the importance of breaking the analysis phase into meaningful and systematic steps so that researchers can provide this undertake sco** studies and report on findings in a rigorous manner. As a result, we recommend three distinct steps in framework stage five to increase the consistency with which researchers undertake and report sco** study methodology: analyzing the data, reporting results, and applying meaning to the results. As described in the existing framework, analysis (otherwise referred to as collating and summarizing) should involve a descriptive numerical summary and a thematic analysis. Arksey and O'Malley [6] describe the need to provide a descriptive numerical summary, stating that researchers should describe the characteristics of included studies, such as the overall number of studies included, types of study design, years of publication, types of interventions, characteristics of the study populations, and countries where studies were conducted. However, the description of thematic analysis requires additional detail to assist authors in understanding and completing this step. In our experience, this analytical stage resembled qualitative data analytical techniques, and researchers may consider using qualitative content analytical techniques [10] and qualitative software to facilitate this process.

Second, when reporting results, we recommend that researchers consider the best approach to stating the outcome or end product of the study and how the sco** study findings will be articulated to readers (e.g., through themes, a framework, or a table of strengths and gaps in the evidence). This product should be tied to the purpose of the sco** study as recommended in framework stage one.

Finally, in order to advance the legitimacy of sco** study methodology, we must consider the implications of findings within the broader context. As a result, we recommend that researchers consider the meaning of their sco** study results and the broader implications for research, policy, and practice. For example, for the question 'how are motor-learning strategies used within contemporary physical and occupational therapy intervention approaches for children with neuromotor conditions?,' the author (DL) presented themes that described strategy use. Results yielded insights into how researchers should better describe interventions in their publications and provided further considerations for clinicians to make informed decisions about which therapeutic approach might best fit their clients' needs. Considering the overall implications of the results as an explicit framework stage will help to ensure that sco** study results have practical implications for future clinical practice, research, and policy. This recommendation leads to the final stage of the framework.

Optional stage six: Consultation

Arksey and O'Malley [6] suggest that consultation is an optional stage in conducting a sco** study. Although only one of our three sco** studies incorporated this stage, we argue that it adds methodological rigor and should be considered a required component. Arksey and O'Malley [6] suggest that the purposes of consulting with stakeholders are to offer additional sources of information, perspectives, meaning, and applicability to the sco** study. However, it is unclear when, how, and why to consult with stakeholders, and how to analyze and integrate these data with the findings. We recommend researchers clearly establish a purpose for the consultation, which may include sharing preliminary findings with stakeholders, validating the findings, or informing future research. We suggest researchers use preliminary findings from stage five (either in the form of a framework, themes, or list of findings) as a foundation from which to inform the consultation. This will enable stakeholders to build on the evidence and offer a higher level of meaning, content expertise, and perspective to the preliminary findings. We also recommend that researchers clearly articulate the type of stakeholders with whom they wish to consult, how they will collect the data (e.g., focus groups, interviews, surveys), and how these data will be analyzed, reported, and integrated within the overall study outcome.

Finally, given that consultation requires researchers to orient stakeholders on the sco** study purpose, research question, preliminary findings, and plans for dissemination, we recommend that this stage additionally be considered a knowledge transfer mechanism. This may address Brien et al.'s [5] concern about the usefulness of sco** studies for stakeholders and how to translate knowledge about sco** studies. Given the importance of knowledge transfer and exchange in the uptake of research evidence [12, 13], the consultation stage can be used to specifically translate the preliminary sco** study findings and develop effective dissemination strategies with stakeholders in the field, offering additional value to a sco** study.

One sco** study included a consultation phase comprised of focus groups and interviews with 28 stakeholders including people living with HIV, researchers, educators, clinicians, and policy makers [7]. Authors shared preliminary findings from the literature review phase of the sco** study with stakeholders and asked whether they may be able to identify any additional emerging issues related to HIV and rehabilitation not yet published in the evidence. The team proceeded to conduct a second consultation with 17 new and returning stakeholders whereby the team presented a preliminary framework of HIV and rehabilitation research and stakeholders refined the framework to further identify six key research priorities on HIV and rehabilitation. This series of consultations engaged community members in the development of the study outcome and provided opportunities for knowledge transfer about HIV and rehabilitation research. This process offered an ideal mechanism to enhance the validity of the study outcome while translating findings with the community. Nevertheless, further development of steps for undertaking knowledge translation as a part of the sco** study framework is required.

Additional considerations for sco** studies to support the advancement, application, and relevance of sco** studies in health research

Sco** study terminology

Discrepancies in nomenclature between 'sco** reviews,' 'sco** studies,' 'sco** literature reviews,' and 'sco** exercises' lead to confusion. Despite our collective use of the Arksey and O'Malley framework, two authors (DL, HC) titled their studies as 'sco** reviews' while the other used 'sco** study.' In this paper, we use 'sco** studies' for consistency with Arksey and O'Malley's original framework. Nevertheless, the potential differences (if any) among the terms merit clarification. Lack of a universal definition for sco** studies is also problematic to researchers trying to clearly articulate their reasons for undertaking a sco** study. Finally, we advocate for labeling the methodology as the 'Arksey and O'Malley framework' to provide consistency for future use.

Quality assessment

Another consideration for sco** study methodology is the potential need to assess included studies for methodological quality. Brien et al.[5] state that this lack of quality assessment makes the results of sco** studies more challenging to interpret. Grant and Booth [4] imply that a lack of quality assessment limits the uptake of sco** study findings into policy and practice. While our research questions did not directly relate to any quality assessment debate, we recognize the challenges in assessing quality among the vast range of published and grey literature that may be included in sco** studies. This also raises the question of whether and how evidence from stakeholder consultation is evaluated in the sco** study process. It remains unclear whether the lack of quality assessment impacts the uptake and relevance of sco** study findings.

A final consideration for legitimization of sco** study methodology includes the development of a critical appraisal tool for sco** study quality [5]. Anderson et al.[2] offer criteria for assessing the value and utility of a commissioned sco** study in health policy contexts, but these criteria are not necessarily applicable to sco** studies in other areas of health research. Develo** a critical appraisal tool would require the elements of a methodologically rigorous sco** study to be defined. This could include, but would not be limited to, the minimum level of analysis required and the requirements for reporting results. Overall, the issues surrounding quality assessment of included studies and subsequent sco** studies require further discussion.

Limitations

This paper responds to Arksey and O'Malley's [6] request for feedback to their proposed methodological framework. However, the recommendations that we propose are derived from our subjective experiences undertaking sco** studies of varying sizes in the rehabilitation field, and we recognize that they may not represent the opinions of all sco** study authors. Other than our individual experiences with our own studies, we have not yet implemented the full framework recommendations. Hence, readers can determine how strongly to interpret and implement these recommendations in their sco** study research. We invite others to trial our recommendations and continue the process of refining and improving this methodology.

Summary

Sco** studies present an increasingly popular option for synthesizing health evidence. Brien et al.[5] argue that guidelines are required to facilitate sco** review reporting and transparency. In this paper, we build on the existing methodological framework for sco** studies outlined by Arksey and O'Malley [6] and provide recommendations to clarify and enhance each stage, which may increase the consistency with which researchers undertake and report sco** studies. Recommendations include: clarifying and linking the purpose and research question; balancing feasibility with breadth and comprehensiveness of the sco** process; using an iterative team approach to selecting studies and extracting data; incorporating a numerical summary and qualitative thematic analysis; identifying the implications of the study findings for policy, practice, or research; and adopting consultation as a required component of sco** study methodology. Ongoing considerations include: establishing a common accepted definition and purpose(s) of sco** studies; defining methodological rigor for the assessment of sco** study quality; debating the need for quality assessment of included studies; and formalizing knowledge translation as a required element of sco** methodology. Continued debate and development about sco** study methodology will help to maximize the usefulness of sco** study findings within healthcare research and practice.

Authors' information

DL is a physical therapist and doctoral candidate in the School of Rehabilitation Science at McMaster University. HC is an occupational therapist and doctoral candidate in the School of Rehabilitation Science at McMaster University. KO is a clinical epidemiologist, physical therapist, and postdoctoral fellow in the School of Rehabilitation Science at McMaster University. She is also a Lecturer in the Department of Physical Therapy at the University of Toronto.