Differentiating quality of care across health care providers: signal or noise?

Introduction: Health care providers, including hospitals and surgeons, are increasingly held accountable to a wide variety of quality metrics. However, differences in measured performance across providers might not be a reliable indication of true differences in quality of care. Reliability is a key scientific criterion by which measures are evaluated before use in public reporting and payment models administered by the Centers for Medicare and Medicaid Services (CMS). However, methods for calculating the reliability of quality measures can disagree and might vary by context.

Methods: We calculated the performance of Veterans Health Administration (VHA) facilities on two clinical outcome measures: (1) hospital-level risk-standardized complication rate following elective primary total hip arthroplasty and/or total knee arthroplasty (THA/TKA) and (2) hospital 30-day all-cause, risk-standardized mortality rate following acute myocardial infarction hospitalization (AMI). Both measures are used in CMS’ Hospital Value-Based Purchasing Program. We estimated reliability for these measures using 4 different methods.

Results: For the THA/TKA complication rate measure, reliability estimates ranged from 0 to 0.23 across methods. For the AMI mortality rate measure, reliability estimates ranged from 0.10 to 0.22. All estimates were substantially lower than a commonly used threshold of 0.7 to indicate good reliability.

Conclusion: Although the two measures were endorsed for use in CMS programs, VHA facilities could not be distinguished from one another reliably on either measure. Evaluating reliability in the specific context in which it is applied is necessary to avoid making decisions in high-stakes accountability programs that are driven by statistical noise rather than signal.