An Explainable Machine Learning Model to Predict Acute Kidney Injury After Cardiac Surgery: A Retrospective Cohort Study

Introduction

The impact of acute kidney injury (AKI) is up to 40% in patients undergoing cardiac surgery.¹ Delayed diagnosis and intervention allow AKI to progress to more severe stages and contribute to the development of chronic kidney disease after hospital discharge.² Both mortality and length of stay increase with the progressive severity of kidney injury. In clinical trials, no pharmacological or non-pharmacological prevention strategies have been shown to reduce the occurrence of AKI.^3,4 Therefore, an accurate postoperative AKI risk assessment is crucial for the postoperative strategy for monitoring and disposition.

In recent years, there are many studies have attempted to predict AKI following cardiac surgery.⁵ Most AKI prediction efforts focus on new blood- or urine-based biomarkers of injury, stress, and metabolomics.^6–8 In addition to being costly, these markers are not widely available and often lack sensitivity or specificity. Most prediction models currently used logistic regression, which has a general predictive ability.^9–11 With the current growth of electronic medical records coupled with machine learning presents an opportunity to improve the performance of established risk models. Compared to traditional risk scores, machine learning (ML) algorithms can easily identify nonlinear relationships and interactions between variables. ML is increasingly being used in AKI prediction, and many models use only small datasets, limiting the predictive performance of the models.^12–14 Even though a few have used large datasets, many of them suffer from the “black box” phenomenon, which could limit the acceptance among clinicians and raise ethical concerns.^15,16 SHAP is a model agnostic representation of feature importance where the impact of each feature on a particular prediction is represented using Shapley values inspired by cooperative game theory.¹⁷ Recently, Shapley additive values as an explainable technique in artificial intelligence have been developed that may overcome the black box problem of ML models.^18,19 This new interpretable approach has been successfully applied to explain ML models related to ICU mortality prediction,¹⁷ poststroke atrial fibrillation,²⁰ and postoperative complications.²¹

In this study, we aimed to demonstrate an EHR data-driven, ML approach for the prediction of postsurgical AKI. Shapley values were also reported to identify the variables that contribute most to the ML model. We hypothesize that an ML model for AKI risk prediction would outperform to a traditional logistic risk calculator.

Materials and Methods

Study Population

Consecutive adult patients who underwent cardiac surgery, admitted between January 2017 and December 2018, were recruited from Fuwai Hospital. Patients were excluded if they underwent dialysis or mechanical circulatory support before surgery. In addition, patients with missing preoperative serum creatinine level or high-level creatine (>4ng/mL) were also excluded from the analysis. The Fuwai hospital Institutional Review Board approved this project and waived consent for patients who provided research authorization compliant. We reported our work following the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement guidelines. The study was complied with the Declaration of Helsinki.

Data Collection

Baseline demographics, clinical comorbidities, and laboratory tests were extracted from the EHR. Laboratory data, including creatinine, glucose and routine blood test measured within 7 days closest to surgery were considered to be baseline values. Estimated glomerular filtration rate (eGFR) was calculated according to the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation based on serum creatinine levels.²² Intraoperative data included transfusion, CPB data and medication were also routinely collected.

Primary Outcome

Our primary outcome of interest was AKI after surgery. AKI is diagnosed using Kidney Disease Improving Global Outcomes (KDIGO) clinical practice guidelines, based on serum creatinine: an increase in serum creatinine of 26.5 μmol/L within 48 hours after surgery or an increase at least 1.5 times the baseline measurement within 7 days after surgery. In this study, the baseline Scr of patients was measured at hospital admission.

Statistical Analyses

The normal distribution of continuous variables was determined by the Kolmogorov–Smirnov test. The continuous data were described by the median and interquartile range (IQR) or mean and standard deviation, while the categorical data were expressed by frequency and percentage. Details of the missing variables are shown in Table S1. Multiple imputations by chained equation were used to impute missing values.

Dataset was randomly divided into training set and testing set with a ratio of 7:3. We used the multivariable logistic regression (LR) with Stepwise Akaike information criterion (stepAIC) algorithm to determine the most significant variables in LR model. The ML methods eXtreme Gradient Boosting (Xgboost) and LR were used to develop and validate the models for assessing risk of AKI. The statistical details of Xgboost methods were displayed in Supplementary methods section. Hyperparameter tuning of the Xgboost model was performed using 5-fold cross-validation during training with 70% of the data. The remaining 30% of the test set was used for validation and to compute performance results. Table S2 showed the best hyperparameters results of the Xgboost. Our final candidate model was selected based on the AUC. DeLong’s test was performed to assess the differences in AUC between these two models. The calibration curve was used to compare the prediction probability of the models and the ground truth. To facilitate the interpretation of the ML model with highest AUROC, we report Shapley values. We used the SHAP values to visualize the significant features that influence the risk of AKI, to analyze the importance of individual features affecting the output of the model and to visualize the impact of key features on the final model in individuals. The caret, xgboost, SHAPforxgboost and plyr packages in R were used for XGBoost model.

R version 4.1.1 (R Foundation for Statistical Computing, Vienna, Austria) was used for statistical analysis with a 2-tailed P value of less than 0.05 indicating significance.

Results

Study Participants

For model development, we obtained data on 15,880 patients, of which we allocated approximately 30% of patients to the validation dataset. The flowchart of study population was presented in Figure 1. The AKI prevalence of the 15,880 patients involved in our study was 30.5%, 2.8% were stage 2 AKI, 1.6% were stage 3 AKI.

Figure 1 Study cohort description.

In the development dataset, the median age was 58 (IQR, 50–65) and 66.4% were male. Patients had a median body mass index of 24.9 (IQR, 22.8–27.3). The top 3 comorbidities of this cohort were hypertension (51.9%), hyperlipidemia (40.2%) and diabetes mellitus (36.0%). Coronary artery bypass grafting and valve surgery were the two most common procedures in this study. Baseline and surgical characteristics were almost well balanced between the derivation and validation datasets (Table 1). The details of derivation cohort are shown in Table 2.

Table 1 Baseline Characteristics of Patients Undergoing Cardiac Surgery in Derivation and Validation Cohorts

Table 2 Baseline Characteristics in Derivation Cohort

Model Performance

We used the variables selected by StepAIC as input factors to develop LR and Xgboost models to predict AKI after cardiac surgery. In the training cohort, the Xgboost model exhibited a maximum AUC of 0.850 (95% CI, 0.842–0.858), which performed significantly better than LR of 0.807 (95% CI, 0.798–0.816, p<0.001). Overall, the predictive ability of models was great in the validation dataset with AUC of was 0.849 (95% CI, 0.837–0.861) in Xgboost, and 0.803 (95% CI, 0.790–0.817) in LR (Table 3). The performance of these two models on the derivation and validation cohorts is proved in Figure 2. Calibration curves were plotted to show the relationship between the predictions of these two models and the observations of the cohort, where a fully calibrated model follows a 45° line (Figure 3). The variance inflation factor and variable contribution of the LR model are displayed in Table S3 and Figure S1 respectively.

Table 3 Prediction Performance of the Models

Figure 2 Comparison of AUC between LR and Xgboost models in predicting postoperative AKI. (a) derivation cohort. (b) testing cohort. Abbreviations: AUC, area under curve; Xgboost, eXtreme gradient boosting; LR, logistic regression.

Figure 3 Calibration plot of Logistic regression ((a) derivation cohort, (c) testing cohort) and Xgboost model ((b) derivation cohort, (d) testing cohort).

Model Explainability

The contribution of the top 15 feature in the model was presented by the SHAP summary plot (Figure 4a). The plots also identify the features that influenced the model predictions the most. Figure 4b shows the variable importance in the Xgboost model, which was the best performing model for postoperative AKI. Additionally, SHAP dependence analysis was performed to depict how a single predictor affected the output of the Xgboost model. The top six clinical features that have the most important impact on the output of the Xgboost prediction model have been shown in detail in Figure 5.

Figure 4 (a) sHapley Additive exPlanations (SHAP) summary plot of the top 15 features in the Xgboost model. The higher the SHAP value of a feature, the higher the probability of postoperative AKI development. Each line represents a feature, and a single dot represent each value for each variable observed in the cohort. Purple represents higher feature values, and yellow represents lower feature values. (b) Variable importance of features included in the Xgboost model for prediction of AKI. Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate; NT-proBNP, N-terminal pro-B-type natriuretic peptide; CPB, cardiopulmonary bypass; LVRWT, left ventricular relative wall thickness.

Figure 5 SHAP dependence plot of the Xgboost model. The SHAP dependence plot shows how a single feature affects the output of the Xgboost prediction model. SHAP values for specific features exceed zero, representing an increased risk of AKI development. (a) Postoperative eGFR; (b) Postoperative creatinine; (c) Preoperative eGFR; (d) Postoperative NT-proBNP; (e) Surgical time; (f) CPB time. Abbreviations: eGFR, estimated glomerular filtration rate; SHAP, shapley additive explanations.

Discussion

Data-driven prediction is not new in medicine and is already routinely used in clinical practice. We developed and validated a machine learning model with postoperative AKI as the outcome and evaluated the performance in this cohort study. ML model demonstrated excellent performance in predicting AKI among individuals with cardiac surgery. The most significant advancement with our risk models is the use of explainable ML approach to risk prediction. This study contributes to the growing body of retrospective machine learning literature for the prediction of AKI.

Prior risk prediction models have been limited to revealing the linear relationship between features and AKI.^11,23 Besides traditional AKI prediction models, the number of studies applying ML to predict AKI has grown steadily over the past decade.²⁴ However, these studies have been mainly developed in small cohorts, and absence of model interpretability.^12,16 To this point, the current study was important because it proposed and internally validated a ML algorithm to predict the risk of AKI based on a large prospectively collected dataset focused on a population of patients undergoing cardiac surgery.

The performance of ML algorithms against conventional statistical methods such as LR to predict clinical outcome remains controversial, some studies showed that both models performed equally well,^25,26 while other studies reported better performance with ML models.^27,28 Luo et al²⁹ were the first to apply machine learning approaches to predict the cardiac surgery-associated acute kidney injury (CSA-AKI) in pediatric patients undergoing cardiac surgery. The Xgboost model achieved the best predictive performance among the considered machine learning models. The results of external validation and sensitivity analysis demonstrated the robustness and applicability of the Xgboost model for predicting pediatric postoperative cardiac AKI. Our study further confirmed the excellent performance of the Xgboost model in predicting postoperative AKI in adults undergoing cardiac surgery.

Many of the most predictive features (eg, creatinine, eGFR and surgical time) identified in previous work were confirmed in the models presented here. Except preoperative and intraoperative characteristics, we also included a number of postoperative laboratory tests at ICU arrival, which may provide some additional information on the kidney function of patients in the early postoperative period. Additionally, our ML model reflects complex nonlinear relationships, which can assist in comprehension the association between the changes in predictors and the risk of AKI. A growing amount of research suggests that baseline NT-proBNP can accurately predict AKI in both surgical and medical patient.^30–32 We found that postoperative NT-proBNP can also have a contribution of AKI development. Almost 14% patients without preoperative NT-proBNP level in our study, we can speculate that early postoperative NT-proBNP assessment may contribute to the risk prediction of AKI. Early postoperative prediction helps to optimize postoperative management and care planning, such as continuous assessment of renal function, hemodynamic monitoring, avoidance of renal toxins, or renal replacement therapy. This study used preoperative, intraoperative, and postoperative variables routinely collected in clinical practice and does not add additional laboratory tests and financial burden to standard clinical procedures.

SHAP analysis was employed in ML application, providing a visualizable prediction model that can be easily accepted by clinicians or decision makers. The algorithm of model interpretability that allowed a quick understanding of the impact of single features on model predictions. In the present study, the SHAP method was used to interpret the XGBoost model, and the results showed that eGFR, creatinine and NT-proBNP at the time of ICU admission were the three most important variables in predicting postoperative cardiac AKI. Our results have emphasized the essentiality of early postoperative laboratory biomarkers in reflecting the acute pathophysiology of kidney injury. Notably, 80% of the top 5 most important predictor variables were intraoperative and postoperative variables, suggesting a significant effect of intraoperative management and surgical operations on the occurrence of AKI.

Several limitations should be mentioned in this study. First, this study is retrospective and relates to only one center, although our dataset was relatively large, complete and accurate. Study findings may not be generalizable for other settings. Second, some residual confounders such as intraoperative time-series variables are unrecorded, may have biased the results. Third, we did not use urine output to define AKI because it was not available to most patients. The lack of urine output criteria may have resulted in lower prevalence of AKI than previous studies. However, the clinical use of diuretics will affect this diagnostic criterion. Lastly, we encourage validation of the findings and algorithm of this study in other cohort, as well as the inclusion of novel peri-operative data types in the analysis. Future research will shift from risk stratification to therapeutic interventions, which will be a milestone in clinical practice

Conclusion

A robust model for predicting AKI after cardiac surgery has been obtained using machine learning techniques. This tool can provide relevant information to patients and assists in clinical decision-making process. However, the model still needs to be further verified by external validation or a randomized clinical trial before using in the daily clinical practice.