Large-scale data-sharing strategies are a valuable asset for improving quality of care in this technologically evolving era of health care.1 Thoughtful quality databases are cost advantageous and efficient for researchers, stakeholders, and health care systems worldwide.2 Patient databases refer to uniform individual patient data collected in a systematic and comprehensive way to serve a predetermined purpose.3,4 In pediatrics, the limited number of patients with specific conditions often precludes conducting large randomized trials. Pediatric patient registries are particularly important because they facilitate the collection of high-quality data from the limited number of pediatric patients with different conditions.5,6 This is especially true for pediatric subspecialties like pediatric critical care.7 There are currently multiple well-established pediatric intensive care unit (PICU) registries worldwide8,9,10,11,12 These PICU registries have become invaluable for: 1) benchmarking and tracking quality-of-care indicators,13,14 2) informing health policymakers, 3) performing high-quality observational research, 4) health care planning and quantification of resource requirements (e.g., informing the pediatric COVID-19 response), and 5) efficiently evaluating patient care and/or health care interventions. Furthermore, network pediatric critical care database trials have been shown to be larger, more likely to be funded, and have a greater dissemination of findings.15 Given the importance of these large-scale databases, the consensus methods used to create them should be highlighted.

To inform and improve our distinct practices, all 17 Canadian PICUs actively endorse and are participating in the creation of a national database; collectively entitled Canadian Pediatric Intensive Care Consortium (CanPICC). Our overall aims are to build on individual PICU data collection strategies to include in a standardized data set, establish a data-sharing platform, and establish a governance model for a nationwide pediatric critical care database in Canada. A fundamental early aim of the national data initiative is to determine the common data elements for this CanPICC database. Design goals include capitalizing on existing clinical data collection, establishing broad ontology and standardizing data elements,16 and creating a dynamic infrastructure for linking data. The objective of this study was to develop national consensus-based common data elements for inclusion in a database of patients admitted to Canadian PICUs.

Methods

Study design

The study outlines the methodological framework for a modified Delphi consensus of variable selection for a national pediatric critical care database. The study had two phases: 1) a process to determine the initial data elements, and 2) a modified Delphi method to develop consensus with three iterations. The Delphi method is recommended in health care as a reliable approach to achieving consensus for a clinical problem.17 The study reporting followed the Checklist for Reporting of Survey Studies (Electronic Supplementary Material [ESM] eAppendix 1).

Phase 1: survey creation and determining initial data elements

The Delphi survey was created using the methodology outlined by Burns et al.18 We employed a modified Delphi consensus approach.17 An initial list of data elements for consideration was generated at a virtual meeting by a focus group (made up of physicians and nurses) separate from the expert panel used in phase 2. The group used a review of the literature, local PICU databases, existing national PICU registries,8,9,10,12,19 and their clinical expertise to generate the domains and the item variables for inclusion in the survey. While the review was not systematic, we searched for all relevant publications related to similar data sets/registries and contacted authors for data elements if these were not available on a website or link. All item sources were reviewed and classified into six data element domains (demographics, PICU admission characteristics, severity of illness scores, PICU therapies, PICU adverse events, and disposition). Item reduction was then performed to remove redundant elements. For data elements made up of multiple items (e.g., PRISM3 scores), all individual items were displayed. Patient and family insights and suggestions were provided by two family representatives (from two centres), along with stakeholder input from the Canadian Institute for Health Information (CIHI) solicited to review the survey draft, prior to dissemination.20 At the end of each domain, there was an open text field for suggestions or comments. The survey was piloted for face and content validity, and readability (n = 3), and then pretested among a small sample of clinicians (n = 3). Internal consistency was excellent with a Cronbach’s alpha of 0.90, but test-retest measure of reliability with Pearson’s r was low (n = 0.5).

The survey also included a selection of potential modules, with open-ended options, that panel members had to rank in terms of importance. Modules were a group of more specific nonmandatory data elements that could be added to the minimal data set for a distinct research project, or as a future expansion of the national database. A sample survey is available in the ESM eAppendix 2.

Phase 2: a modified Delphi method to develop consensus

We then conducted a prospective multicentre iterative survey of health care professionals and experts in pediatric critical care from March to June 2021. The Delphi study is reported according to the Checklist for Reporting Survey Studies, endorsed by the EQUATOR network.21 This initiative was funded by the Canadian Critical Care Trials Group (CCCTG) and the study was approved by the Research Ethics Board of the CHU Sainte-Justine Hospital and the Montreal Children’s Hospital (Montreal, QC, Canada; # MP-2021-3285).

Study sites and participants

We invited participants from all PICUs across Canada to complete the surveys. Canadian PICU study sites included BC Children’s Hospital and Victoria Children’s Hospital in British Columbia; Stollery Children’s Hospital and Alberta Children’s Hospital in Alberta; Royal University Hospital in Saskatchewan; Children’s Hospital of Winnipeg in Manitoba; The Hospital for Sick Children in Toronto, McMaster Children’s Hospital, Children’s Hospital at London Health Sciences Centre, Children’s Hospital of Eastern Ontario, and Kingston Health Sciences Centre in Ontario; CHU Sainte-Justine, Montreal Children’s Hospital, Centre Mère Enfant Soleil du CHU de Québec-Université Laval, and Centre Hospital Universitaire de Sherbrooke in Quebec; IWK Health Center in Nova Scotia; and Janeway Children’s Health and Rehabilitation Centre in Newfoundland and Labrador. We sought to include a diverse sample of stakeholder participants with relevant expertise in clinical care, quality improvement, or research. Lists of CCCTG members with an interest in pediatric critical care were used to identify at least one contact person in each site, who then provided the study team with contacts of health care experts to be invited. Health care experts included physicians, scientists, research assistants, nurse managers, advanced practice nurse clinicians, hospital administrators, health care administrators, and public health and patient/family representatives. For those not working directly in pediatric critical care, a knowledge of the Canadian PICU landscape and functioning were necessary for inclusion. There were no specific exclusion criteria for participants. Participation and engagement of potentially eligible expert participants was solicited with an online recruitment letter. Consent to participate was implied by answering and returning the survey.

We aimed to obtain a final sample size of 60 PICU health care experts to ensure that we sampled one to three experts from each of the 17 Canadian PICUs, with parents and stakeholder group members making up the remaining participants. Sending invitations to 85 potential participants would allow for a 10–15% nonresponse rate and 10% drop-out rate. This is in kee** with the recommended Delphi methodology suggesting a minimum of 12–35 participants.16,22

Survey process

Three rounds of surveys were planned for this process, each one month apart. Participant demographics were collected at the start of the first-round survey. In the first-round, participants selected which data elements would ideally be included in the data set. For the first two rounds of the Delphi process, all questions were evaluated on a five-point nominal scale by selecting the level of importance for inclusion of each data element (“mandatory,” “important,” “optional,” “remove,” “not my area of expertise”). “Consensus” was obtained if 70% or more of eligible responses were within the same response category on the five-point scale.23 If consensus was achieved in the “mandatory” category, these elements were included in the data set. “Disagreement” was defined as 35% or more of responses falling in both of the two extreme ranges of possible options (“mandatory” and “remove”) on the nominal scale. All other combinations of panel answers were considered “partial agreement.” Where consensus was achieved for a data element, that question was removed from subsequent survey rounds, including all of the items that made up that data element. For the second and third round, participants were asked to revisit their consideration of each remaining data element with anonymized results from the prior rounds, shared in the form of proportions of participants selecting each response for each data element. Each of the first two rounds allowed participants to add comments and potential additional data elements. In the third round, to finalize item selection, response options were modified to a four-point scale to “include,” “exclude,” “add to a module,” or “not my area of expertise,” with consensus maintained as a 70% response in the same category. In all rounds, participants were instructed not to account for feasibility of data element collection but rather to select the most important data elements for inclusion in the minimal mandatory data set. This was specified so as not to discourage inclusion of elements related to social determinants of health, including self-reported data elements.

Completion of the survey was anonymous. Study data were collected, managed, and confidentially stored using REDCap electronic data capture tools hosted at the Montreal Children’s Hospital.24,25

Analysis

Results were summarized as counts and proportions. The primary outcome was consensus on the importance of data element inclusion into the database. In the calculation of the proportion of consensus for a data element, if a participant selected “Not my area of expertise” for a given question, they were not included in the denominator calculation for that question, or in the number of participants who answered that question.

Results

Phase 1

The initial survey produced by the group included 125 items (variables) in the six domains (demographics, n = 22; PICU admission characteristics, n = 26; severity of illness scores, n = 16; PICU therapies, n = 25; PICU adverse events, n = 19; outcomes, n = 16). There were some repeated items with different definitions to allow the Delphi expert group to select the most appropriate definition (e.g., PICU discharge was defined first as the date and time patients were cleared for discharge, and second as the date and time patients physically left the intensive care unit [ICU]). In addition, 17 modules were included for ranking after the Delphi survey. The final survey included 117 data elements divided into six domains: 1) patient demographics, 2) PICU admission, 3) severity of illness, 4) interventions in the PICU, 5) quality assurance, and 6) outcomes.

Phase 2

We invited 86 experts to participate in the modified Delphi method phase, 68 (79%) of whom engaged and agreed to participate. There was proportional representation of participants from PICUs across the country, as well as a distributed variety of expertise and lengths of PICU practice (Table 1). Participants were predominantly physicians (63%), recommended by their units for their expertise in acute care data collection, clinical content, research and quality metrics. The 68 expert participants in the panel were sent all three rounds of the survey with response rates of 91% (62/68) in round one, 90% (60/68) in round two, and 81% (55/68) in round three (Figure).

Figure
figure 1

Flow diagram of expert participant inclusion, response rate, and selected variable count over three Delphi rounds. *Duplicates removed include night discharge, unplanned readmission, early cardiopulmonary resuscitation (CPR), Glasgow Coma Scale (GCS), and death within 6 hr

Full size image
Table 1 Expert participants included in the Delphi process (N = 62/68 respondents round 1)
Full size table

After the first round, 46 of the 117 (39%) data elements were included by consensus, and seven suggested items were added by participants for assessment in the subsequent rounds (Figure). After round two, an additional seven data elements were included by consensus. After limiting the answer options for round three, 24 additional data elements were added to give a total of 77. Of these, data elements that could be derived from other selected items were removed (e.g., Glasgow Coma Scale included in PRISM3 score, night admission/discharge included in time of admission/discharge, readmission), giving a total of 72 distinct included data elements (Table 2). The most highly ranked modules determined to be valuable additions to the minimal database were ventilation (n = 10), sepsis (n = 8), and traumatic brain injury (n = 8).

Table 2 Variables selected by consensus (≥ 70%) as mandatory for inclusion in minimal PICU database (round of selection, proportion selected as mandatory)
Full size table

Discussion

The present modified Delphi consensus process included representatives from all different PICUs across Canada, and identified 72 measurable data elements from six domains, for inclusion in a national PICU database that is currently being operationalized. Active recruitment and engagement of participants across the pediatric critical care community of practice resulted in high response rates from a diverse and heterogenous group of experts, and emphasizes the importance of this work. In contrast to national PICU data sets in other countries created by a small group of individuals, we have shown that development of consensus within a diverse group of experts was possible and resulted in a broad and inclusive group of data elements. Furthermore, we have established a framework to apply this methodology to the creation of other data registries.

The modified Delphi method phase used an initial small focus group of physicians and nurses from various centres to generate a list of variables based on literature, expertise, and other PICU databases. The larger Delphi panel of experts then suggested seven further variables to be added, but none of these were ultimately included in the mandatory data set based on consensus. The small focus group therefore likely appropriately reviewed evidence to select the variables, with their expertise in clinical research, data quality, and data acquisition likely strengthening this selection. The modification of this step in the Delphi process is therefore an efficient and appropriate method of variable preselection prior to consensus generation.

The exclusion of certain data elements in the consensus highlights whether relying solely on consensus methodology is the best approach for selecting a minimal data set. While consensus methodology effectively aggregates the anonymous opinion of the majority, other valuable but less commonly collected or reported data elements may be omitted. Furthermore, there is not a “correct” answer in a Delphi process, but rather a reflection of what the selected participants deem is important. In this case, the majority of selected elements reflected diagnostic categories, advanced medical care delivered, and PICU complications. These choices may reflect the potential of certain interventions to cause significant unintended harm or long-term impacts in patients.

A striking feature of the Delphi results was the lack of consensus regarding inclusion of elements describing social determinants of health. While race, sex, gender, and the first three digits of the postal code (geographic location) were included, elements such as minority status, indigenous status, primary language, family income, education, and ethnicity did not achieve consensus. There may be several reasons for reticence to include these data elements in the data set, including challenges with standardizing data collection, concerns for patient privacy, or lack of understanding of the item’s importance in the child’s health status. Furthermore, physiologic clinical variables may be as easier to standardize and collect than self-reported data of social constructs such as race and ethnicity. Regardless, these data elements remain important determinants of child health inequities in Canada,26 and serve as a foundation for future work in pediatrics in Canada.

Despite the significant increase in research on understanding outcomes after PICU, no scales associated with global or cerebral functional status were included. In addition, despite data to suggest the importance of transport on PICU patient outcomes,27 none of the transport data elements were included. Given that these pre-ICU admission data are not a part of the ICU electronic medical record, participants may not have included these due to perceived feasibility challenges related to collection and accuracy. Lastly, although patient weight is an included data element, there are no included measures of ideal body weight, body mass index, or height. Given the importance that obesity can have on outcomes of critical illness and long-term health, this exclusion will remain an important variable for future consideration. Data elements that did not achieve the vote of all members (e.g., “master ID” or “primary diagnosis”) highlight the heterogeneity, selective expertise, and knowledge bases of the panel members; each with their own use for the database. In this case, a hybrid approach for common data element inclusion, with oversight of a larger steering committee for example, may be necessary to avoid the omission of key data elements in addition to all those voted on by panel members.

Our study had some limitations. First, a low test-retest value (0.5) suggests a low reliability with equal true and error-related variance. It is unclear whether retesting would produce the same results given the possibility of measurement error.28 Second, despite the diversity of participants and their areas of expertise, the participant numbers were likely over represented by physicians, and limited in allied health professionals such as social workers and nutritionists. An increased presence of stakeholders, patients, and members specialized in equity, diversity, and inclusion may have balanced the perspectives of the group with regard to social determinants of health items. To overcome this limitation, we consulted the CIHI and the Public Health Agency of Canada to discuss adding important items for exploring social determinants of health, such as ethnicity. Improved understanding of the social determinants of health is essential to illuminate health care inequalities that exist for children in Canada, and to inform policy changes to improve health care access and quality for marginalized children. Ensuring that routinely collected data elements reflect the needs of a diverse range of stakeholders will amplify the sustainability of data collaboration. The aim to have a national, population-level data coverage, engaging public health and policy-makers has been emphasized as a means to strengthen our processes. Last, family perspectives were limited in the study as only one patient-partner was involved in this Delphi process.

Ongoing work includes clear ontogeny, definitions, and timing of data collection for the selected data elements, as some of these are not well established in children. In conjunction with pediatric members of the CCCTG, who conducted multiple randomized trials in children, we are harmonizing definitions for included data elements based on multiple recent trials29,30 to standardize data element inclusion. A parallel group has created a governance structure for how national data would be validated, accessed, and used for research and quality improvement. In terms of infrastructure, national partners in data management and adult critical care are creating an infrastructure for an open-source platform at each hospital. To overcome barriers preventing record-level data from leaving the provinces, data storage and analysis would be conducted behind a firewall at each individual site, and only aggregate data would exit the sites.

The core data elements selected in this Delphi study will be instrumental in assessing the determinants, clinical outcomes, and quality of care, of critically ill children in Canada. By pooling granular, standardized, anonymous data that goes beyond what is submitted by hospitals using diagnostic codes, we will have an unprecedented ability to ask targeted questions and learn from and improve our practices through quality improvement initiatives and research. Ensuring that variable coding can be harmonized with other data sources and is standardized to align with pre-existing data standards and international collaborations16 will ease linkage, sharing, and comparability across sites and registries. Overall, this will lead to greater insight in the effectiveness, safety, and quality of critical care.

Conclusion

We present the consensus methodology used to derive the data elements for a national pediatric critical care database, with participation from a diverse group of experts, caregivers, and stakeholders representing every pediatric critical care program in the country. The above-outlined process provides a framework for both stakeholders and clinicians that can be applied to similar initiatives to ensure consensual data element selection for a sustainable, feasible creation of a national database. The selected core data elements will provide standardized and synthesized data for research, benchmarking, and quality improvement initiatives in critically ill children.