Comparing Nurse- vs Machine-Extracted Risk Adjustment Variables for Hospital Outlier Detection using the VA CABG Only 30-day Mortality Model

Introduction: The Veterans Affairs (VA) Surgical Quality Improvement Program (VASQIP) collects data on all cardiac surgeries at VA hospitals. Surgical quality nurses (SQNs) manually extract 187 variables from medical records, 15 of which are used in a 30-day mortality model for coronary artery bypass grafting (CABG) to identify facilities with higher-than-expected 30-day mortality rates. We aimed to develop automated methods to extract these variables and compare outlier facilities identified using nurse- versus machine-extracted variables.

Methods: We developed rule-based methods to machine-extract 14 of the 15 risk-adjustment variables using the VA Corporate Data Warehouse. The VASQIP variable, functional status, could not be extracted with adequate accuracy. Using cardiac cases (FY2018-2021) across 40 facilities, we compared facility-level outlier status—defined as an observed-to-expected ratio significantly greater than 1.0—between two methods: one using all nurse-extracted variables and another using 14 machine-extracted variables and 1 nurse-extracted variable.

Results: Across 154 fiscal year facilities, 119 (77.3%) were not flagged as outliers by either method, 25 (16.2%) were flagged as outliers by both methods, 7 (4.5%) were flagged as outliers only by nurse-extracted method, and 3 (1.9%) were flagged as outliers only by machine-extracted method.

Conclusion: Machine-extraction of risk variables may be more efficient than nurse-extraction and identifies more facilities as outliers. Whether additional facilities flagged by the machine-extracted method are true outliers or false alarms is unknown. These results can inform decisions about possible tradeoffs between efficiency, accuracy, and allocation of resources in the VASQIP program.