Derivation and Validation of a Risk Prediction Model for Vancomycin-Associated Acute Kidney Injury in Chinese Population

Introduction

Methicillin-resistant Staphylococcus aureus (MRSA) has undergone a rapid epidemiologic expansion to become a major cause of hospital and community-acquired infections worldwide.¹ Glycopeptide vancomycin has been the mainstay of treatment for four decades. However, increased nephrotoxicity remains the most common adverse event after vancomycin administration with an average incidence of 15%. And it reaches up to 40% in critically-ill population.² Once acute kidney injury (AKI) occurs, vancomycin treatment has to be modified or even discontinued. Besides, it also leads to greater incidence of end-stage kidney disease, prolonged hospitalization, poorer prognosis and even higher mortality.³

Risk factors of vancomycin-associated nephrotoxicity have been extensively studied. Higher vancomycin daily dosage, longer duration of therapy, and elevated plasma concentrations of vancomycin are widely known predisposing factors.^4–7 The most recent guidelines recommended that clinicians should monitor vancomycin nephrotoxicity by calculating the AUC/MIC ratio.⁸ Some other conditions have also been proposed as risk factors, like renal dysfunction, admission to ICU and concomitant nephrotoxic agents.⁶ For instance, ICU residence increased the incidence of AKI from 9.3% to 13.3%.^6,9 Concurrent piperacillin/tazobactam significantly added the risk of AKI with a hazard ratio of 4.27.¹⁰

By integrating multiple predisposing factors, risk prediction models can be developed to help clinicians estimate the probability or risk of AKI after vancomycin administration, thus to make informed clinical decisions. As an effective strategy to forecast future event, risk prediction models have been extensively applied to cardiovascular events,¹¹ type 2 diabetes, diabetic retinopathy,¹² cancer,¹³ etc. However, in terms of vancomycin-associated nephrotoxicity, its application is limited.

After comprehensive literature searching, only two prediction models were published for prognosis of vancomycin-associated nephrotoxicity. One was established by logistic regression (LR) and presented as a mathematical formula.¹⁴ It was proposed for elderly patients and could not be applied in general patients. The other model was built by a decision tree (DT) method to evaluate the risk of AKI in Japanese patients.^15,16 DT model is nonparametric and weak in clinical interpretation. Besides, neither of two models were validated externally. A novel model with stronger applicability and generality in Chinese population remains to be explored.

Our study aims to explore potential predisposing factors of vancomycin-associated AKI and establish a risk prediction model in Chinese population. We hope to provide an effective strategy for early recognition and prompt intervention of AKI in patients treated with vancomycin.

Method

Study Setting and Design

This was a retrospective cohort study. Following the TRIPOD statement (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) in 2015,¹⁷ the study was designed for model development and validation. This study was implemented in The First Affiliated Hospital, Zhejiang University School of Medicine. The protocol was approved by the Ethics Committee of The First Affiliated Hospital, Zhejiang University School of Medicine with the approval letter (No.20191540). We strictly conformed to the ethical standard in Declaration of Helsinki during the study. All the information was gathered anonymously and used harmlessly. And the informed consent was exempted by the ethics committee.

Data Source

All the data in this study were obtained from hospital information system (HIS), including electronic medical record (EMR), laboratory information system (LIS) and nursing information system (NIS). In detail, general information such as age, gender, and clinical features (blood pressure, heart rate and the history of diseases) were derived from admission record in the EMR. Heights and weights were exported from the NIS. Comorbidities were judged from admission and discharge diagnosis recorded in the EMR. Concomitant medications were obtained from the electronic prescription system. Data about laboratory examinations, like serum concentration of vancomycin, serum creatinine (Scr) and others, originated from the LIS. Two researchers (Xu and Zhang) were trained to gather the information, which were firstly collected by one researcher (Xu) and then checked by the other (Zhang).

Study Population

Patients who were admitted to The First Affiliated Hospital, Zhejiang University School of Medicine from January 1, 2016 to October 11, 2019 and received vancomycin treatment for ≥48h during the period of hospitalization were eligible. Patients were excluded 1) if their ages were below 18; 2) if they already suffered from renal diseases at the start of vancomycin, because renal diseases interfered the judgment of the main outcome-AKI. And renal diseases were diagnosed by clinicians according to International Classification of Diseases-10 (ICD-10), including uremia or dialysis, kidney transplant, kidney failure, chronic kidney disease and acute renal injury and they were detailed in the Appendix; 3) if estimated creatinine clearance ≤45mL/min, which was calculated by Cock Croft-Gault formula according to baseline Scr, age, weight and sex; 4) if patients already had a rising Scr before or right at the time of vancomycin start; 5) if vancomycin was orally instead of intravenously administrated; (6) if important information, like vancomycin trough concentration, baseline and subsequent Scr were missing.

Patients in this study were divided into two cohorts according to their admission time: derivation cohort (from January 1, 2016 to December 31, 2018) and validation cohort (from January 1, 2019 to October 11, 2019). The Model was established in the derivation cohort and independently validated in the validation cohort. The inclusion and exclusion criteria were the same in the two cohorts. And data were also collected by the same investigators, typically using the same predictors and outcome definitions and measurements. Details about patients’ selection are shown in Figure 1.

Figure 1 Flowing chart of population enrollment.

Study Outcomes

The primary outcome of this study was the occurrence of AKI during the period of vancomycin treatment. And AKI was defined according to Kidney Disease Improving Global Outcome (KDIGO) Clinical Practice Guidelines.¹⁸ That is, patients were diagnosed with AKI if the Scr increased by 26.5μmol/L within 48h, or Scr rose to 1.5 times of the baseline value within 7 days. Additionally, the AKI criterion judged by urine volume in the KDIGO guideline was omitted, because urine volume per hour could not be obtained. Baseline Scr in this study was measured the day before vancomycin administration. If it was not available, the one measured 3 days before administration served as a substitute.

Variable Classification and Definition

We collected five types of variables for variables screening: demographic information, vancomycin regimen, comorbidities, concomitant medications and laboratory tests, which could put some effect on vancomycin-associated AKI. Among these, variables associated with vancomycin therapy like dosage, frequency and duration were collected and daily dosage was calculated according to dosage and frequency. If dosage was adjusted during the treatment, daily dosage was averaged for analysis.

Patients’ comorbidities were diagnosed by the clinicians based on ICD-10 and recorded in the EMR. Comorbidities analyzed in this study included hypertension, diabetes, heart failure, arrhythmia, atrial fibrillation, coronary atherosclerotic heart disease, hepatic insufficiency, viral hepatitis, lung disease, shock, trauma and cancer. And other comorbidities with a lower occurrence (≤5%) were excluded for analysis.

As for concomitant medications, some well-known nephrotoxic agents, such as aminoglycosides, non-steroidal-anti-inflammatory drugs (NSAIDs), diuretics, antiviral agents and contrast agents were recorded if they were used together with vancomycin. Other concurrent drugs like antibiotics, β-blockers, angiotensin-II receptor antagonists, immunosuppressive agents and antifungal agents were also collected. Co-medications were not included for analysis if less than 5% patients took these agents. All the agents were prescribed in electronic prescription system and taken under the supervision of a nurse. These agents were classified according to Drug Classification Code of the First Affiliated Hospital, Zhejiang University School of Medicine, which was compiled referring to Anatomical Therapeutic Chemical (ATC) classification system. Details about drugs and classifications in this study were shown in Appendix.

Baseline laboratory tests referred to the values measured the day before vancomycin administration, including Scr, vancomycin serum concentrations, total protein, albumin, aspartate aminotransferase (AST), alanine aminotransferase (ALT), hemoglobin and so on (Appendix). All the blood samples were collected by nurse according to medical orders and measured by the laboratory in one day. As for vancomycin trough concentrations, samples were collected in half an hour before vancomycin administration. On the condition that serum trough concentrations were measured for several days, only the initial one was selected for analysis. As all the patients were given vancomycin by intermittent medication, trough concentration measurement for continuous medication was not considered.

Statistical Analysis

We use IBM SPSS^® Statistics, version 22.0 and R version 3.6.2 to perform the statistical analysis and model development. Numerical variables were presented with mean and standard deviation (SD). Comparison between two groups was performed via Student’s t-test or Wilcoxon rank-sum test. Categorical variables were described by frequency and percentage. Comparison was performed via Chi-square or Fisher exact test. Difference was considered significant if P value was below 0.05.

In the derivation set, in total of 94 variables were screened via univariate regression. If P<0.1, this variable would progress to multivariable logistic regression with backward method (in 0.05 and out 0.10). And independent predictors were determined when P<0.05 in multivariable regression. Then, they were integrated to be an initial risk prediction model. This initial model was presented as a mathematical formula:

P represents the probability of AKI occurrence. X₁, X₂, …, X_n represent predictive variables we have selected. , …, refer to regression coefficients of corresponding variables.

Besides, we plotted a nomogram with R to visualize our initial model. By the nomogram, each predictor value was assigned one point from 0 to 100 based on the magnitude of association of each predictor with AKI. Then we added the points of three predictors together to calculate the total point or risk score, which corresponded to the probability of the outcome. And all the patients were stratified into 4 groups by quartile of risk scores: very low (score<25), low (score 25–50), moderate (score 50–75), and high risk (score>75). The actual incidence of AKI in the four groups stratified by risk score quartile was compared in both internal and external cohort.

The established model was internally validated by bootstrap method (N_samples=1000) and externally in the validation cohort. Discrimination and calibration were reported to evaluate model predictive performance. And discrimination indicated the ability of a model to distinguish patients who would develop AKI from those who would not, which was a comprehensive reflection of sensitivity and specificity. Discrimination was measured by the area under the receiver operating characteristic (ROC) curve (AUC). AUC ranged from 0 to 1, and the higher AUC, the better discrimination. Besides, sensitivity and specificity at the optimal cutoff point with a maximum Youden’s index (Youden’s index = sensitivity+specificity-1)¹⁹ were also reported. Model calibration demonstrated the consistency between the observed incidences and the predicted risks. Calibration was measured by two methods: calibration plot and Hosmer-Lemeshow goodness-of-fit test with a higher P value indicating a better calibration.

Our model was compared with the published model,¹⁵ prediction by C_min alone and other models. Net reclassification improvement (IRI), integrated discrimination improvement (IDI) and Akaike Information Criterion (AIC) were calculated to quantitatively compare predictive accuracy with P<0.05 showing a significant improvement or a lower AIC showing a priority of model selection.

As for missing data, glomerular filtration rate (GFR) was missed for about 54.9%. This variable was removed from our analysis. Height and weight were missed for about 29.3% and 28.3%, separately. They were filled with multiple imputation method. Systolic blood pressure (SBP), diastolic blood pressure (DBP) and HR (heart rate) were missed for less than 0.4%. ALT was missing for about 0.6%. Drinking and smoking were missing for about 0.4%. Those data with missing rates below 1% were filled with average. Especially, some important data, like vancomycin trough concentration, baseline and subsequent Scr were complete because individuals with missing data were excluded. Besides, 53% patients had their Scr measured daily, 94.9% patients had Scr measured at least four times and all the patients had at least twice during vancomycin treatment.

Results

Baseline Characteristics and Outcomes

Three hundred and forty-one patients were included in the derivation cohort and AKI occurred in 55 (16.1%) patients. In the external validation cohort, 183 patients were enrolled and the incidence of AKI was 16.4% (30/183). As is shown in Table 1, baseline characteristics in the two cohorts including age, sex, BMI and blood pressure and so on, were mostly similar, while patients from neurosurgery ward accounted for the largest proportion in derivation cohort (31.8%) and patients in validation cohort mainly came from hepatobiliary pancreatic surgery ward (46.4%). Therefore, percentage of patients suffering from liver diseases in the validation cohort was higher than that in the derivation cohort (57.9% vs. 23.0%, P<0.001).

Table 1 Baseline Characteristics in the Derivation and Validation Cohorts

AKI Predictors and Model Establishment

Twelve potential predictors with P<0.1 were selected by univariate analysis and listed in Table S1. According to multivariable logistic regression, 4 factors were found to be independently associated with the AKI outcome: C_min, concomitant piperacillin/tazobactam, furosemide, and dosage adjustment (P<0.05). And three variables were eventually integrated to predict the occurrence of AKI in the final model (shown in Table 2). Vancomycin dosage adjustment (OR=2.286, 95% CI: 1.169–4.470, P=0.016) was excluded from our prediction model, because dosage adjustment might occur after AKI.

Table 2 Predictors Included in the Multivariable Logistic Regression Model

The final logistic regression model was exhibited as following:

Additionally, an alternative model was built by adding two variables (cardiac surgery and baseline Scr) into the three-item model, considering that these two variables were recognized as important risk factors for AKI.^20,21

The alternative model was shown below:

We transformed the multivariable logistic regression model into a three-item AKI risk score (C_min, furosemide and piperacillin/tazobactam) by assigning points to each predictive variable based on a nomogram (Figure 2). According to the nomogram, separate point was assigned when dichotomous variables (furosemide and piperacillin/tazobactam) equal 1 or 0. And one point was assigned to each value of the continuous variable (C_min) from 0 to 60μg/mL. By adding points of three predictors, we calculated the total point (risk score) for each patient. According to the risk score, all the patients in the derivation cohort were separated into 10 groups by decile. The frequency and incidence of AKI in each group were calculated and shown in Figure 2, demonstrating that the incidence of AKI increased with the growth of risk score. Similarly, the nomogram for the alternative model was presented in Figure S1.

Figure 2 The AKI risk score nomogram for clinical application and AKI incidence by score decile.

Model Evaluation

Our model showed a comparable discrimination in both derivation and validation cohort. AUC of our model was 0.793 (95% CI: 0.732–0.855) (Figure 3), and sensitivity and specificity were respectively 83.6% and 62.6% at the optimal cutoff point with a maximum of Youden’s index of 0.462 in the derivation cohort. AUC in internal and external validation were 0.786 (95% CI: 0.783–0.788) and 0.788 (95% CI: 0.698–0.877), respectively (Figure 3). Besides, sensitivity and specificity were 80.0% and 60.1% in the validation cohort.

Figure 3 The ROC curve in the derivation (A) and external validation sets (B).

The established model also indicated a good calibration according to goodness-of-fit test and calibration plot. In the derivation cohort, and in internal and external validation, goodness-of-fit test showed χ²=6.079 (P=0.638>0.05), χ²=5.074 (P=0.742>0.05) and χ²=5.665 (P=0.686>0.05), respectively, suggesting a consistency between the predicted probabilities and the actually observed incidences. This result was confirmed by calibration plot (Figure 4).

Figure 4 Calibration plot in the derivation and validation sets.

All the patients in the derivation and validation cohorts were stratified into four groups by risk score quartiles: very low risk (score<25), low risk (score 25–50), moderate risk (score 50–75) and high risk (score>75). Actual incidence of vancomycin-associated AKI in each group was presented in Figure 5. It showed that predicted risk was consistent with the actual incidence of AKI in both derivation and validation cohorts. However, in moderate and high-risk groups, actual incidences of AKI in the validation cohort were a bit lower than that in the derivation cohort.

Figure 5 Incidence of AKI after treatment with vancomycin in both derivation and validation sets stratified by score quartiles: score <25 (very low risk), score 25–50 quartile (low risk), score 50–75 (moderate risk), and score ≥75 (high risk).

Comparison with Existing Models

Our model was compared with the alternative model, Imai’s model and prediction by C_min alone in terms of NRI, IDI and AIC. In the derivation cohort, by referring to prediction by Cmin alone, NRIs (95% CIs) of our model, the alternative model and Imai’s model were 0.166 (0.044–0.289), 0.155 (0.021–0.288) and 0.052 (−0.060–0.165), separately. And I(95% CIs) were 0.050 (0.024–0.076), 0.048 (0.019–0.077) and 0.022 (−0.022–0.066), separately. As for AIC, they were calculated to be 252.614, 255.429, 263.052 and 267.494 by our model, the alternative model, Imai’s model and prediction by Cmin alone. The results in the validation cohort were similar with those in the derivation cohort (see in Table 3), suggesting that our three-item model showed the best predictive accuracy of AKI in both derivation and validation cohort.

Table 3 Discriminative, Reclassification Ability and AIC of Our Established Risk Score in the Derivation and Validation Cohort in Comparison with Existing AKI Prediction Models in Patients Treated with Vancomycin

Discussion

In this study, we confirmed the role of serum trough concentration and concomitant piperacillin/tazobactam in vancomycin-associated AKI, which had been previously reported, and featured the potential causality of concomitant furosemide with AKI. Then we established a risk score by integrating these three predictors by multivariable logistic regression and nomogram. Finally, this novel risk score was internally and externally validated and was compared with the existing models. Discussions were made from four aspects: outcomes and predictors, comparison with the existing models, clinical applications and strengths and limitations.

Outcomes and Predictors

Establishing the causality between vancomycin and AKI is challenging. In this study, we confirmed their association mainly according to time sequence. That is, AKIs that happened before vancomycin administration were excluded. Besides, possible confounding factors like cardiac surgery, combination of some other nephrotoxic drugs, ICU admission and so on were also taken full consideration to determine the role of vancomycin in AKI.

In our risk score, vancomycin serum trough concentration acted as an essential predictor for AKI, which has been stated in many studies. A study from Dong reported that serum trough level of vancomycin was an independent factor for the development of nephrotoxicity.²² Burgess et al found that higher trough concentration significantly increased the risk of vancomycin-associated nephrotoxicity.²³ However, the causality between high vancomycin level and AKI remained to be further revealed. Due to the fact that serum vancomycin was mainly excreted by kidney, renal injury from any reasons could lead to a high vancomycin level. In turn, vancomycin accumulation resulted in aggravating kidney injury.²⁴ Due to the interaction of vancomycin accumulation and kidney injury,² steady trough concentration may change, especially increase afterwards. As a result, initial vancomycin trough concentration was selected as a predictor in this study. Besides, a model developed based on an earlier serum concentration could contribute to prompt interventions.

Our study also showed that concurrent furosemide increased the risk of vancomycin-associated AKI (OR=3.609, 95% CI: 1.630–7.991). According to Huang et al,²⁵ the combination of diuretics (furosemide≥40 mg/d) and vancomycin was an independent risk factor of nephrotoxicity in elderly critically-ill patients. Although some studies hold that diuretic was also a kind of nephrotoxic agents, co-medication of vancomycin and furosemide remained common in clinical practice. As showing in the present study, 63.3% of patients in the derivation cohort took furosemide along with vancomycin and the percentage was even higher in the validation cohort (70.5%). Therefore, our study reminded that the use of furosemide should be cautious when combined with vancomycin in case of AKI.

A lot of studies reported that piperacillin/tazobactam had an additive effect on vancomycin-associated nephrotoxicity.^23,26–29 Consistent with this result, our study further confirmed the association between AKI and concurrent use of vancomycin and piperacillin/tazobactam. And we integrated it into our risk score model for vancomycin-associated AKI to achieve an improved model performance, compared with prediction by C_min alone.

Comparison with Existing Models

After comprehensive literature retrieval, only two prediction models about vancomycin-associated nephrotoxicity have been published. One model by Pan in 2017 was proposed for elderly patients who received vancomycin therapy.¹⁴ This model was not validated in general patients, and could not be applied extensively. Another model by Imai in 2019 was a decision tree model, which was developed in Japanese patients.^15,16 And only categorical variables were utilized in DT algorithm. In addition, neither of the two models was validated by independent external data, though the predictive performance proved good in the training data. Different from these two models, our risk score was developed in Chinese adult patients and validated in an external data. The predictive accuracy of our model was improved according to the calculated NRI, IDI and AIC, compared with Imai’s model and prediction by C_min alone.

Clinical Application of the Risk Score

Our risk score predicted the occurrence of vancomycin-associated AKI by integrating three variables: C_min, furosemide and piperacillin/tazobactam. According to our risk score, when C_min of vancomycin was above 45μg/mL or 30μg/mL, the risk of AKI was classified to be high (average probability of 75%) or moderate (average probability of 40%), no matter furosemide and/or piperacillin/tazobactam were combined or not. Based on our risk score, when C_min was 10–20μg/mL, the risk of AKI would still be high if furosemide and piperacillin/tazobactam were used together, though C_min between 10 to 20μg/mL was widely recommended in clinical practice.⁷ But the risk would be reduced to low and very low if none of the two agents were taken. Thus, co-medication of these two agents should be avoided when vancomycin had to be administered. In turn, if furosemide and/or piperacillin/tazobactam were necessary in clinical practice, a suggestion about C_min could also be made by our risk score.

Clinicians could make early recognition and prompt intervention about vancomycin-associated AKI if the risk was calculated to be high by our risk score. For example, some antioxidants and renal protectants could be used in advance to prevent the occurrence or aggravation of vancomycin-associated AKI. But whether the routine use of the AKI risk score in the hospitalized patients substantially attenuates AKI risk or even improves patients outcome by better decision making remains to be prospectively ascertained.

Strengths and Limitations

Our study had a number of strengths. We derived a simplified risk score with only three variables which were easily available in patients receiving vancomycin treatment. Compared with mathematical formula or nonparametric DT model, our risk score achieved better application and stronger interpretability in aiding decision making on AKI prevention for clinicians. At variance with previous model designed to predict vancomycin-associated AKI, our model was firstly externally validated and showed comparable predictive accuracy compared with that in the training data.

However, some limitations also existed in our study. Firstly, the discrimination of our risk score just ranged from moderate to good. And this was retrospective study, and some significant predictors, like cystatin and genetic factors, were not available to improve model predictive accuracy. Besides, the role of furosemide dosage was not further illuminated, though concomitant furosemide proved to be an important risk factor. Furthermore, our study excluded patients with a preexisting kidney injury, given that abnormal renal function might influence the determination of AKI outcome. But prediction for AKI risk would be more desirable in those patients. Limited by this, our data failed to indicate the association between baseline kidney function and AKI. Considering these shortcomings, further studies focusing on a more general population, including those with renal dysfunction, were still a matter of urgency. Whether or not our risk score applied to AKI prediction in those patients with renal diseases remained to be revealed. Besides, a well-designed prospective study was warranted to confirm the effect of furosemide on vancomycin-associated AKI, especially the role of furosemide dosage.

Conclusion

In conclusion, we developed a simple, accurate and robust risk score model to predict the probability of vancomycin-associated AKI in Chinese patients. This risk score could help clinicians make informed decision about co-medication and dosage adjustment of vancomycin to reduce the risk of AKI.