Introduction

Over a decade ago, the publication of the first in a series of seminal Institute of Medicine reports ushered in the modern era of patient safety with a focus on transparency and eliminating harm in health care settings [1, 2]. Since that time, numerous organizations have defined key patient safety measures, resulting in a proliferation of measures; by 2010, over 100 patient safety measures had been endorsed by the National Quality Forum [3]. The Centers for Medicare and Medicaid Services (CMS) has over 65 measures [4]. Other organizations like Leapfrog and Institute for Healthcare Improvement (IHI) similarly have proposed their own set for public reporting.

A large part of the challenge is the ongoing lack of clear, concise definitions and measurements of safety. In addition, the key question of what really matters in patient safety measurement has not been fully addressed. This paper describes the current state of patient safety measurement and our view of what really matters in patient safety measurement.

Patient safety measurement

Safety measures include elements in three domains: structure, process, and outcome of care. Many of the structure and process measures are necessary because they can identify areas where improvement efforts are needed. However, they are not sufficient because their relation to patient outcomes often is tentative or indirect. Outcome measures may be difficult to evaluate because harm events may occur infrequently, may not be evident immediately, or may reflect reporting bias (more serious, more likely to be reported). Current measurement relies almost exclusively on self- or voluntary reporting; thus, it is difficult to be certain what comprises either the numerator or denominator when calculating harm rate to evaluate patient safety. In order to overcome these limitations, safety researchers have proposed a different process to identify and analyze safety events. Pronovost and colleagues at Johns Hopkins suggest that a structured process focusing on four domains would provide better measurement which could guide improvement initiatives. These authors propose to understand (a) how often patients are harmed, (b) how often clinicians provide appropriate interventions, (c) whether clinicians learned from mistakes, and (d) how successful clinicians and health care systems are in creating a culture of safety [5]. Using this four-part approach, institutions may better be able to understand their safety environment and possible areas for improvement. However, the model remains theoretical because of the difficulty of defining precise, validated measures for many of the contributing factors.

Thomas and Classen have proposed a similar three-part approach: (a) better understand the epidemiology of adverse events and revise current methods to better identify underlying causes, (b) encourage national organizations to improve measurement of safety culture within an organization using validated tools, and (c) educate front line staff on safety and improvement science and use of improvement tools so that staff can focus on what really matters. “Good evidence exists that educating caregivers about safety science and improving safety culture is the foundation of improvement efforts. Of course, reliance on the frontline worker is a long-standing tenet of quality improvement across many industries. With the emerging evidence that safety is not improving and is too heterogeneous to be assessed by externally mandated measures, we conclude that external top-down efforts to measure safety should cease to expand.” [6].

Patient and family expectation (Don’t hurt me…)

Over the past two decades, the public expectation has evolved to expect more transparency and even comparison of results among providers and hospitals. Yet, patients and families still have limited access to basic patient safety and quality of care data. This lack of public access and transparency was highlighted in the June 4, 2015, CNN report “Surgical death rates for babies kept secret from parents” [7]. At most hospitals in the USA, despite clarion calls by both patient safety advocates [8] and professional organizations [9], patients and families remain at a disadvantage when seeking unbiased information regarding safety and quality of care data.

In this context, patients and families are searching to evaluate and compare providers and hospitals. Healthgrades™, an online resource comparing short-term acute care hospitals on Medicare patient safety indicators, reports more than 225 million annual visits to its websites [10]. In Massachusetts, nearly three quarters of respondents to an independent study commissioned by the Betsy Lehman Center for Patient Safety and Medical Error Reduction strongly agreed that the public should have information on medical errors made by hospitals and providers; however, only 40 % of hospital-care users reported to have searched for patient safety or quality of medical care information. Interestingly, nearly all agreed that hospitals and providers should be required to report medical errors to a state agency, but less than half strongly agreed that they would base their choice of a hospital upon the hospital reporting fewer medical errors [11]. Thus, patients and families may desire to have access to provider or hospital patient safety data; it is less clear that they utilize this information in health care decision-making.

Little is known about what information regarding safety and quality is most helpful or meaningful for patients and families. They do need a plain language explanation of patient safety and a cohesive framework from which to understand its importance to their care. This is particularly important to help individuals from diverse socioeconomic and health backgrounds better comprehend definitions of key metrics and findings of comparative reports [12]. We also know little regarding how patient safety metrics should be presented to patients and families. Instead of overwhelming patients with data that are hard to interpret, clear simple graphs, ranking systems, color coding, and arrows to indicate direction of better outcome may improve comprehension of the patient safety metrics and comparison of providers and hospitals.

As public access to safety data increases, numerous challenges remain for an individual to be able to make meaningful decisions on his/her own health care. For example, the Joint Commission collects quality and safety data from US hospitals and makes some of the quality data available to the public through the Quality Check® website. For adult hospitals, one can obtain an almost overwhelming amount of information regarding percentage of patients who receive specific patient safety and quality of care practices, but not general or specific patient safety outcome data. For children’s hospitals, one can obtain information regarding whether or not the organization implemented certain safety goals. One must specifically request in writing information about occurrences of sentinel events or aggregate complaint information about alleged patient safety or quality of care issues [13]. Medicare allows families and the public to obtain and compare specific patient safety and quality of care data regarding individual Medicare-certified hospitals at their Hospital Compare website. Specific patient safety metrics are available including unplanned readmission and death rates, surgical complications, and health care-associated infections; however, data regarding children’s hospitals is lacking. Several advocacy groups also provide patient safety data [14]. Healthgrades has already been mentioned. Healthgrades, utilizing Medicare data, lists specific patient safety indicators for acute care hospitals and provides a three-category rating for each indicator; however, again, data regarding children’s hospitals is lacking [10]. The Leapfrog group, also utilizing Medicare data, provides an overall Hospital Safety Score (ranking system of A–F) that consists of several specific patient safety metrics. Each metric is displayed as either red, yellow, or green with red being worse and green being better. Unfortunately, again, data regarding children’s hospitals is lacking [15]. At least one professional organization, the Society of Thoracic Surgeons (STS), publicly reports online voluntarily collected patient safety data. For adult cardiac programs, the ratings are based on a star system and simply look at absence of mortality and major morbidity. Pediatric cardiac programs currently report only risk-adjusted mortality, and the individual program’s mortality rate is compared to an expected mortality rate for each risk strata. However, of the approximately 100 pediatric cardiac surgery programs, only 19 children’s hospitals voluntarily report their operative mortality data [9]. In the USA, very few children’s hospitals voluntarily report selected patient safety metrics such as adverse drug events, various health care-associated conditions, serious safety events, etc. (examples of reporting hospitals include Nationwide Children’s Hospital [16], Cincinnati Children’s Hospital Medical Center [17], and Texas Children’s Hospital [18]).

Health care provider perspective

Primum non nocere (first do no harm) has been the prime tenet of medical care for centuries. Making health care safe in the era of inpatient medicine, increasingly stronger medications, and complex health care settings remains challenging. The IOM reports [1, 2] pointed out the extent of the challenge. Based on available data principally from a Medicare patient population, the IOM estimated that nearly 100,000 patients die each year because of medical errors. A more recent reassessment estimates this number to be almost fourfold higher. Health care professionals gauge safe care not only by the number of patients that die each year from medical error but also by the number of patients that suffer preventable harm during the course of receiving care. Using this broader definition, health care-associated patient harm may occur at a rate four- to tenfold higher. Since the publication of the IOM reports, individuals, institutions, and national organizations have invested heavily in identifying and reducing preventable harm and in increasing patient safety. Yet, a clear definition of patient safety and how to measure it remain a challenge.

As discussed previously, there are hundreds of available patient safety measures. Institutions and providers face a real challenge to keep up with reporting burden while wondering which of these are necessary and core to evaluating patient safety. Meyer and colleagues propose a parsimonious set of measures which may provide a reasonable set of quality and safety metrics which balance the providers’ need for usable data to guide improvement and the payers and public need to judge quality and safety of the care provided [19]. To guide institutions, providers, and public, the Agency for Healthcare Research and Quality (AHRQ) engaged the RAND Corporation, which guided a consensus process to identify key patient safety measures. Of 501 possible measures, some from published reports and others proposed by national organizations, the group reached agreement that only 13 inpatient measures were important and had sufficient validity to recommend their use (see Table 1) to gauge internal progress and compare performance with other organizations [20].

Table 1 RAND health consensus process-determined measures with sufficient validity [20]
Full size table

In order to ameliorate the current situation, it is imperative first to know what we mean by patient harm and increase patient safety. Health care professionals usually focus on outcomes of medical care, with lapses in safety that result in unplanned procedures, long-term effects (treatment, morbidity), or death. Furthermore, many health care providers who deal with serious, life-threatening conditions continue to believe that a certain level of complications is to be expected as part of the care process and, therefore, not preventable. Over the past decade, the science of patient safety and concurrently the definitions have evolved to include different levels of harm (see Table 2). Using the Triple Aim as the goal for treatment (Care, Health, Cost or Value), harm (or failure in safety) can be defined as any preventable temporary or permanent injury, preventable deviation from intended outcome, or unnecessary increase in cost. However, such a definition is too broad for effectively measuring current level of safety and impact of improvement efforts. Nationally, many current efforts at improving patient safety are focused on reducing certain conditions (see Table 3) which have been identified as sufficiently frequent and costly to warrant attention. Vast resources and efforts have been allocated to reduce the frequency of these conditions. Yet, the level of patient safety remains nearly unchanged since the publication of the IOM reports [21, 22]. And the only valid measures seem to be health care-associated infections (HAI) and a measure of safety culture, using either the Agency for Healthcare Research and Quality’s Hospital Survey on Patient Safety Culture (HSOPS) or the Safety Attitudes Questionnaire developed by the University of Texas, Houston [23, 24]. Currently, many national organizations require institutions to report measures of patient safety; however, even measures have limited value in assessing how well institutions perform on the Triple Aim. A study by the Office of Inspector General [25] found that 13.5 % of Medicare beneficiaries had adverse events during their hospital stays, but only 1.6 % of these events would have been captured as serious reportable events or hospital-acquired conditions. The patient safety indicators have also been shown to miss most adverse events [22]. This degree of measurement error is extremely concerning because the Centers for Medicare & Medicaid Services is initiating a value-based reimbursement program that will link hospital reimbursement to performance on the hospital-acquired condition and patient safety indicator measures. Even if hospitals were to reduce or eliminate the events tracked by these measures, they may have done little to improve overall patient safety and may be continuing to harm patients on a daily basis [26]. Thomas and Classen call for a change in the way we approach safety metrics, saying that “The adverse consequences of mandated external safety measures and their inability to accurately measure patient safety should persuade national patient safety organizations to stop promulgating more measures.” [26].

Table 2 AHRQ (v1.2) Common Formats Harm Scale includes level of harm and duration
Full size table
Table 3 Hospital-acquired conditions which are the focus in CMS-guided initiatives (2015 final list)
Full size table

Payer perspective

In this environment, payment models have shifted from fee for service to pay-for-performance to curtail costs and improve quality. Pay-for-performance models provide bonuses for meeting or exceeding performance measures and impose penalties for failing to do so. While pay-for-performance initiatives existed prior to the Patient Protection and Affordable Care Act (ACA), the ACA expanded use of the approach in Medicare [27]. Currently, the Centers for Medicare and Medicaid Services pay for performance programs, and the Hospital Value-Based Purchasing program, the Hospital Readmissions Reduction Program, and the Hospital-Acquired Conditions Reduction Program require acute care hospitals to report quality measures. Medicare reimbursement is impacted by performance on the measures, with impacts of up to 5.5 % of a hospital’s total Medicare reimbursement [28]. In the private sector, more than 40 pay-for-performance measures exist including the California Pay for Performance Program, managed by the Integrated Health System and the Alternative Quality contract, a collaboration between Blue Cross Blue Shield of Massachusetts and over ten provider groups [28].

Across sectors, pay-for-performance programs include patient safety measures, often rates of health care-acquired conditions. For example, the AHRQ Patient Safety Indicator-90, required for both the Hospital-Acquired Condition Reduction Program and the Hospital Value-Based Purchasing program, includes rates of postoperative venous thromboembolism, pressure ulcers, central line-associated bloodstream infection, accidental punctures or lacerations, postoperative sepsis, iatrogenic pneumothorax, postoperative wound dehiscence, and postoperative hip fracture [29]. When viewing patient safety measurement from a payer perspective, health care-acquired conditions are key measures given their financial impact.

What really matters

While currently available measures have value and identify important outcomes, they have had little impact on the overall quality and safety of health care. What really matters may be the cultural key drivers of outcomes, including the culture of safety and team culture and behavior that supports listening to one another, learning from one another, and learning from mistakes.

Safety culture within an organization has emerged as the key element in reducing harm and improving patient safety. The definition of safety culture advanced by the Advisory Committee on the Safety of Nuclear Installations seems quite applicable to health care as well (Table 4) [30]. The primary measurement of how safe care is in any organization is most closely correlated with the organization’s safety culture as reflected in a safety survey. There are two validated survey instruments—the AHRQ’s HSOPS and the Safety Attitudes Questionnaire developed by the University of Texas, Houston. Both of the surveys can be used to predict the safety culture and overall safety grade of an organization. Components within the HSOPS better predict the frequency of error reporting and overall perceptions of safety in an organization [31].

Table 4 Safety culture definition
Full size table

Core to creating a culture of safety is trust built on mutual respect, learning from each other, and learning from mistakes. Leape and colleagues recently presented a thoughtful perspective on the culture of respect and its impact on safety in health care [32]. Disrespectful behavior, broadly defined, can undermine every effort to improve safety and quality in health care and may explain the lack of substantial improvement in safety since the publication of the IOM report in 2009. The authors discuss several categories of disrespectful behavior: disruptive behavior; humiliating, demeaning treatment of subordinates; passive-aggressive behavior; passive disrespect; dismissive treatment of patients; and systemic disrespect. Such behavior subverts efforts to inculcate elements of a high-reliability safe organization which have been shown to increase safety (see Table 5) [33]. Teamwork is essential to modern health care delivery, with doctors, nurses, and others collaborating on a treatment or care plan. Lack of respect in its various forms leads staff to avoid the disruptive person which negatively impacts the forthright closed-loop communication required to provide timely, effective care. Staff and patients may be reluctant to volunteer important information, cutting the physician off from important information that only the patient, family, or staff can provide about symptoms or complications of therapy and observed failures [32]. In order to make meaningful advancement in patient safety, institutions should develop and foster a culture of respect. The organization must have clear and consistent means to manage disruptive behavior (see Table 6). Additional efforts include setting the expectations (e.g., code of conduct), empowering front line staff, and creating a learning environment built on a questioning attitude [34].

Table 5 Five key traits of a high reliability organization
Full size table
Table 6 Graduated approach to managing disruptive behavior
Full size table

Patient-reported outcome measures

What really matters to patients and families may differ significantly from what physicians and the health care system may focus on. Outcomes measured merely in survival, number of complications, or degree of morbidity as defined by health care professionals may miss a lot of what is important to patients and families. There is a growing realization that patient-reported outcome measures (PROMs) are critical to measure when assessing effectiveness of therapy. The Cochrane handbook defines PROMs as “…any reports coming directly from patients about how they function or feel in relation to a health condition and its therapy, without interpretation of the patient’s responses by a clinician, or anyone else. PROs include any treatment or outcome evaluation obtained directly from patients through interviews, self-completed questionnaires, diaries or other data collection tools such as hand-held devices and web-based forms (US Food and Drug Administration 2006). Proxy reports from caregivers, health professionals, or parents and guardians (necessary in some conditions such as advanced cancer and cognitive impairment) cannot be considered PROs and should be considered as a separate category of outcomes.

PROs provide patients’ perspectives on treatment benefit; directly measure treatment benefit beyond survival, disease, and physiologic markers; and are often the outcomes of greatest importance to patients. Reports from patients may include the signs and symptoms reported in diaries, the evaluation of sensations (most commonly classified as symptoms), reports of behaviors and abilities (most commonly classified as functional status), general perceptions or feelings of well-being, and other reports including satisfaction with treatment, general or health-related quality of life, and adherence to treatments. Reports may also include adverse or side effects…” [35].

The rising importance of measuring PROMs is highlighted by substantial investment and numerous initiatives by major national agencies to standardize definitions and use of PROMs in clinical care and research. The Patient-Centered Outcomes Research Institute (PCORI) was created to highlight the importance of measuring outcomes that are important to the patient and family. In 2004, the National Institutes of Health launched the PROMIS (Patient Reported Outcomes Measurement Information System) to create and make available to clinicians and researchers “…highly reliable, precise measures of patient–reported health status for physical, mental, and social well–being. PROMIS tools measure what patients are able to do and how they feel by asking questions.” [36]. Patient perspective on outcomes continues to gain importance as reflected in the use of HCAHPS (Hospital Consumer Assessment of Healthcare Providers and Systems) and the pediatric version (child HCAHPS) by the Centers for Medicare and Medicaid Services (CMS) to measure the quality of care as judged by patients. Though not yet universally in use to gauge and compare patient safety and outcomes of care, the emerging science on PROMs promises to provide health care with better, valid tools to measure what really matters to patients and families.

Conclusions

In 1917, Dr. Ernest Amory Codman, a surgeon at Massachusetts General Hospital stated: “So I am called eccentric for saying in public that hospitals, if they wish to be sure of improvement must find out what their results are, must analyze their results, must compare their results with those of other hospitals, must welcome publicity not only for their successes, but (also) for their errors. Such opinions will not be eccentric a few years hence” (emphasis added) [37]. Almost 100 years later, public reporting of errors and patient safety metrics for hospitals and providers has not been fully realized. Patients and families want access to patient safety data, and they have an ethical right to that data to make informed decisions about their health care. While payers and providers have access to data, the focus on health care-acquired conditions has been narrow and not resulted in meaningful improvements in health care quality and safety. Public reporting of performance data does increase quality improvement activities [38]. Some of these quality improvement activities, particularly multi-center collaborative projects, may ultimately result in improved patient safety [39]. To improve the understanding of families and the public regarding patient safety and its importance to their care, the patient safety movement must provide a simple cohesive framework of well-defined and standardized general and specific easily understood patient safety metrics. This should include not only health care-acquired conditions, which are very important to payer and provider groups, but also the indicators of the organizational culture. Further, to make patient safety metrics and their presentation more meaningful, cross-sector input from patients, families, providers, and payers should be actively sought. Patient-reported outcome measures may prove to be better metrics for patient safety and quality than any currently in use. Finally, to promote patient autonomy and transparency, the reporting of standardized patient safety metrics for hospitals should be mandatory [40]. Mandatory reporting places everyone on the same playing field and indicates that the health care profession is after almost 100 years finally committed to eliminating medical errors and patient harm. The key concepts of this article are summarized in Table 7.

Table 7 Key points
Full size table