Developing and Validating Methods to Automate Extraction of Veterans Affairs Surgical Quality Improvement Program (VASQIP) Registry Variables

Introduction: The Veterans Affairs Surgical Quality Improvement Program (VASQIP) cardiac dataset relies on trained surgical quality nurses (SQNs) at each VA Hospital to manually extract and/or verify data for 187 variables from the medical records of every cardiac surgery patient. This study seeks to develop and validate methods to automate variable extraction, as well as evaluate the quality of SQN extracted data.

Methods: We developed methods to automate the extraction of thirteen variables utilizing procedure dates paired with clinical notes for natural language processing, lab values, and diagnoses as needed for each variable.

Results: For all thirteen variables considered so far (10 laboratory values, pre- and post-operative atrial fibrillation, and smoking status), we have achieved high congruence with the SQN-extracted values. For the majority of the variables, the automated approach yielded fewer missing values. In cases where manual annotation had fewer missing values, the automated method captured more instances meeting the definition time criteria. Early indications suggest that the accuracy and timeliness of certain SQN-extracted data could be improved by automation.

Conclusions: While SQN-extracted data are currently considered the gold standard for national surgery quality improvement programs, early indications suggest that fully automated systems can, in some cases, perform comparably with the work of SQNs while others may actually outperform SQN designations due to fewer errors and less missing data.