Machine Learning Models for Predicting the Risk of Hard-to-Heal Diabetic Foot Ulcers in a Chinese Population

Introduction

According to the most recent data, the global prevalence of diabetes reached 10.5% in 2021 and is projected to reach 12.2% (783,2 million) in 2045.¹ Diabetes is becoming a worldwide epidemic faster than anticipated.² In addition to hyperglycemia, diabetes increases the risk of a number of complications, including diabetic nephropathy, diabetic retinopathy, and diabetic foot ulcers (DFUs). DFUs, which increase the risk of death in diabetic patients by 2.5 times, are one of the most common and devastating consequences.³

DFU is the result of a combination of lower limb vascular obstruction and lower limb nerve disorders. Current treatment often requires multidisciplinary collaboration, with treatment options including glycemic control, wound debridement, lower limb revascularization, and amputation. However, even under standard treatment, a wound may not heal within an appropriate time, increasing the likelihood of infection, prolonged hospitalization, and rising financial burden.^4,5 Early detection, assessment, and intervention are crucial to the outcome of DFUs. According to real-world data from the US Wound Registry (USWR), DFUs as chronic refractory wounds have a healing rate of less than 40% at 12 w.⁶ Early wound area progression helps doctors in the inpatient phase make predictions and decisions as quickly as possible. Retrospective studies with large populations have shown that a percent area reduction (PAR)<50% at week 4 is widely recognized as an indicator of healing failure at 12 weeks or even 20 weeks.^7–12 Therefore, this parameter has also been used as an observational outcome indicator in recent RCT studies.^13,14

In past studies on the prognosis of DFUs, the methods used mainly included untrained general linear regression, including logistic regression and Cox regression.^15–17 Machine learning is a branch of artificial intelligence consisting of a wide range of algorithms for classification research and can be tuned for optimal models, offering considerable advantages for predictions in a variety of diseases, including the prognosis of cancer and the diagnosis of diabetes.^18,19 The most suitable machine learning algorithm depends on the situation in which it is used. Several recent studies on DFUs have demonstrated a high predictive capacity utilizing machine learning algorithms focusing on DFU detection and determination of the need for amputation due to DFUs.^20–25 Nonetheless, no machine learning research is available on healing difficulty, a crucial prognostic factor for DFUs.

In this study, based on data from DFU patients collected from two tertiary hospitals in eastern China, we built models employing six well-known machine learning algorithms to predict the possibility of hard-to-heal DFUs. The six algorithms are as follows: general linear regression (GLM), the naïve Bayesian (NB) model, support vector machine (SVM), random forest (RF), k-nearest neighbor (KNN), and adaptive boosting (AdaBoost).

The performance of these classification models was evaluated and compared to select the optimal model. A corresponding online calculator was created to facilitate use by clinicians. For the screened predictive features, we performed an importance ranking based on the optimal model to analyze the reasons for hard-to-heal DFUs in depth.

Methods

Study Population

Data from all eligible patients with DFUs were collected from January 2018 to December 2019 from Nanjing Drum Tower Hospital, the Affiliated Hospital of Nanjing University Medical School and the Air Force Hospital of Eastern Theater Command, without matching for any variable. These two tertiary hospitals are both major metropolitan tertiary centers with a multidisciplinary team including plastic surgeons, vascular surgeons, endocrinologists, and an experienced nurse team. Our inclusion criteria were patients diagnosed with DFUs between the ages of 18–89 years old. Patients who had tumor-induced ulcers, undergone major amputations, abandoned treatment, or had incomplete information were excluded. Patient demographics, wound characteristics, and laboratory data were collected at the first admission examination. According to the Helsinki Declaration, our study was approved by the Ethics Committee of Nanjing Drum Tower Hospital (No. 2020-10901). The need for informed consent from the participants was waived because of the retrospective study design and the use of anonymized clinical data.

Outcome and Predictors

Outcome

An ulcer with a PAR<50% by week 4 is considered a hard-to-heal ulcer. Healing results were evaluated by an independent clinical evaluator with no knowledge of each patient’s initial examination data in this process. The healing area was measured using a ruler, and the areas of irregular wounds were calculated by ImageJ software after being photographed.

Predictors

We aim to predict whether a DFU patient will experience healing difficulty at the time of initial assessment. Therefore, only data from tests performed at the initial admission were collected. Treatment methods, non-hospital routine tests (e.g., HbA1c level), and features that have not shown predictive value in the previous literature were not included. Finally, a total of 21 clinical characteristics that have been identified to be associated with DFU outcomes were collected as candidate predictors.^16,26 The demographic data included age, sex, current use of insulin, random blood glucose, years with diabetes, diabetic foot ulcer history, drinking history and smoking history; wound characteristics included wound duration, wound area, wound classification (University of Texas, UT) and wound location; comorbidities included hypertension, hyperlipidemia, diabetic retinopathy, cardiovascular disease, peripheral arterial disease and diabetic peripheral neuropathy; and laboratory indicators included serum albumin, serum creatinine and C-reactive protein (CRP) levels. To avoid confusion, we have provided specific explanations of the clinical indicators in Supplementary Table 1.

Model Construction

Statistical analyses were performed using R software (version 4.1.1; https://www.R-project.org).²⁷ Categorical data are expressed as counts (percentages), and measurement data were stratified and are expressed as counts (percentages), with Chi-square tests used to determine χ2 values and p values. Univariate logistic analysis was used to screen factors affecting the outcome variable (p < 0.05). We randomly stratified the data containing these influencing factors at a ratio of 7:3; 70% of the data constituted the training set, and the remaining 30% of the data served as the validation set. Additionally, we selected four single classification algorithms: GLM, a commonly used algorithm that establishes the relationship between the mathematical expectations of labels and the predictor features of a linear combination through a linking function.²⁸ As a special form of GLM, logistic regression was used in this study; KNN is a lazy learning but effective algorithm. It performs the classification by calculating the distance between the test samples and all the training samples to obtain their nearest neighbors; SVM is an algorithm separating the two label classes by maximizing the separating margin. Meanwhile, SVMs can map samples to higher dimensions with different kernel functions.²⁹ In this study, a polynomial kernel was chosen for the SVM; and the NB model is built by solving for the probability of each feature occurring under the conditions in which the labels appear. Two ensemble classification algorithms were also used: RF, a flexible and easy-to-use classifier that uses multiple decision trees to train and predict samples;³⁰ and AdaBoost, an enhanced learning algorithm that weights weak classifiers and averages them to obtain the final classifier.³¹ We used 5-fold cross-verification to filter the optimal parameters of the models.

Model Evaluation

The data in the validation set were used to assess the generalization ability of the classification models that we developed. Each model’s confusion matrix was created separately, and the models’ accuracy, precision, recall rate and F1-score were calculated. These metrics can be calculated from the values in the confusion matrix. The details are provided in Supplementary Table 2. Receiver operating characteristic (ROC) curves, the most common tool for evaluating predictive models, were additionally used to compare the performance of the models. We integrated the ROC curves of each model into one figure for visual comparison and calculated AUC (area under the curve) values for the final model selection.The characteristics of these six algorithms and the compatibility of these algorithms with our data are summarized in Supplementary Table 3.

Implementation of the Web-Based Prognostic Tool

A web tool for hard-to-heal prediction based on Microsoft Azure Web Sites (Microsoft Corporation, USA) was established based on the filtered optimal prediction model. Model building and prediction were carried out on the Microsoft Azure console, resulting in a user-friendly web interface.

Results

The workflow of the study and a flowchart of data allocation are shown in Figure 1.

Figure 1 (A) Workflow of the study; (B) Flowchart of patient selection. Abbreviations: AUC, area under the curve; AdaBoost, adaptive boosting; DFU, diabetic foot ulcer; GLM, generalized linear model; KNN, k-nearest neighbor; NB, naïve Bayes; PAR, percent area reduction; RF, random forest; SVM, support vector machine.

Patient Characteristics

From January 2018 to December 2019, a total of 362 patients with UT3 grade diabetic foot ulcers were collected, including 257 (70.99%) males and 105 (29.01%) females, aged 26–88 years. All the patient data, including demographics and disease and treatment characteristics, grouped by hard-to-heal DFUs,are listed in Table 1.

Table 1 Differences Between Demographic and Clinical Characteristics of Healed and Hard-to-Heal Groups

Univariate Regression Results

Through univariate regression analysis of 21 variables pertaining to demographics, wound features, and laboratory indicators, 6 variables that statistically affected the classification of hard-to-heal DFUs were identified (p < 0.05) as follows: sex, random blood glucose, diabetic retinopathy, peripheral arterial disease, smoking history and CRP. We also incorporated another 4 features—insulin use, wound area, serum albumin, and serum creatinine—that have been shown to have a strong prognostic impact on DFUs based on clinical experience and previous literature.^32–35 These ten characteristics were used to determine the final model. These features and the results of the univariate regression analysis are shown in Table 2.

Table 2 Prediction Factors for Hard-to-Heal in DFUs

Model Establishment and Evaluation

Labels defining whether a DFU is difficult to heal, along with the filtered features, were used to build models based on the six machine learning algorithms. To assess the predictive power of the different models, we patterned the confusion matrix of the models, as shown in Figure 2. The “0” in each confusion matrix identified patients who did not have hard-to-heal DFUs, while “1” identified patients who had hard-to-heal DFUs.

Figure 2 Confusion matrix of the risk prediction models with machine learning algorithms. (A) AdaBoost: adaptive boosting; (B) GLM: general linear regression; (C) KNN: k-nearest neighbor; (D) NB: naïve Bayes; (E) RF: random forest; (F) SVM: support vector machine.

According to the confusion matrix, the four indices of the model’s generalization ability, accuracy, precision, F1-score and recall, were calculated and are displayed in Table 3. Additionally, the ROC curves are visualized in Figure 3. The AUC values of the ROC curves for each model are also listed in Table 3. For comparison purposes, we visualized the five evaluation metrics as bar charts (Figure 4). The NB model clearly obtained the highest AUC (0.864), recall (0.907), and F1-score (0.744) values. SVM achieved the highest scores for accuracy (0.759) and precision (0.66). The NB model was finally selected as the optimal model for detecting hard-to-heal DFUs because it obtained the highest AUC value.

Table 3 The Comparisons of Machine Learning Algorithms

Figure 3 ROC curves for predicting hard-to-heal in DFU patients with machine learning algorithms. Abbreviations: AdaBoost, adaptive boosting; AUC, area under the curve; DFUs, diabetic foot ulcers; GLM, general linear regression; KNN, k-nearest neighbor; NB, naïve Bayesian; RF, random forest; ROC, Receiver operating characteristic; SVM, support vector machine.

Figure 4 The values of evaluation metrics of six machine learning algorithms. Abbreviations: AdaBoost, adaptive boosting; AUC, area under the curve; GLM, general linear regression; KNN, k-nearest neighbor; NB, naïve Bayesian; RF, random forest; SVM, support vector machine.

Web-Based Tool

We built a web calculator on Microsoft Azure Web Sites based on our dataset and the naïve Bayesian algorithm. The web page is https://predicthardtoheal.azurewebsites.net/. This web tool allows the user to select ten variable values to return the probability of predicting hard-to-heal DFUs. We provide an example in Supplementary Figure 1.

Importance Ranking of Characteristic Variables

We explored the relative importance of each feature in the naïve Bayesian model. The feature importance ranking results are shown in Figure 5. The three most important variables were: random blood glucose, which represents poor blood glucose control; CRP, which rises in response to inflammation; and wound area, which is associated with nonhealing ulcers. The remaining variables were ranked in order of importance as follows: smoking history, albumin, peripheral arterial disease, sex, retinopathy, creatinine and insulin use.

Figure 5 Feature importance ranking of the included feature of the naïve Bayesian model. Abbreviations: CRP, C-reactive protein.

Discussion

Early detection of hard-to-heal DFUs will play an important role in preventing prolonged hospitalization and amputation by allowing timely changes to the treatment plan. In this study, data from DFU patients from two Chinese tertiary hospitals were collected, and six popular machine learning algorithms were applied to build predictive hard-to-heal DFU models. In the model evaluation, the NB model showed the most generalization power (AUC 0.864, recall 0.907, and F1-score 0.744). Based on the NB model, an online calculator was created to facilitate its use by clinical staff to assess the probability of hard-to-heal DFUs at the time of initial admission. Finally, we ranked the importance of each feature by contribution based on the NB model.

In the model-building phase of our investigation, the NB algorithm emerged as the optimal algorithm, which is likely because our models were based on a small dataset with stratified statistical characteristics and two categorical outcome variables. Logistic regression in the GLM is a powerful statistical tool; however, it cannot discern nonlinear correlations between features.²⁸ KNN, which utilizes only the k-nearest samples from the training set, has a low bias and high variance. With a limited training set, the KNN algorithm is susceptible to overfitting. SVM and Adaboost are sensitive to noise data, resulting in performance restrictions in particular situations.³⁶ RF has limitations when categorizing data with relatively few features; numerous identical decision trees can be generated in such circumstances, which does not improve classification accuracy and increases the operational burden.³⁷ The NB model has distinct benefits for low-dimensional classification samples. First, the NB model is stable and less vulnerable to noise data, in addition to being quick and requiring no accountable structure. In addition, the NB model is simple, elegant and powerful in that features are independent of each other. The NB model has been the most efficient algorithm in numerous experiments.^38,39 Lastly, because Bayes Theorem generates posterior probabilities based on prior probabilities, NB models can examine previously existing model parameters.⁴⁰ Researchers can blend their own data with existing Bayesian model data for combinatorial statistics without revealing sensitive patient information. This procedure was replicated in a study conducted by Jens Hüsers et al .⁴¹ Our research once again underscores the potential use of the NB model for low-dimensional categorization data from the medical field.

According to the results reported in this study, approximately 53.6% of patients displayed nonhealing wounds after one month, which is slightly more than the 48.3% rate predicted by earlier research.⁴² This finding is acceptable, as Grade III-A hospitals in China typically admit patients with serious ulcers (almost UT grade 3).

Our NB algorithm-based model for predicting hard-to-heal DFUs ranked random blood glucose as the most important factor, indicating that patients with higher than average blood glucose levels are more likely to develop ulcers than those with normal blood glucose levels. This finding can be explained by the fact that poor glucose control in diabetic individuals is one of the causes of DFUs.⁴³ Additionally, hyperglycemia can result in aberrant keratinocyte migration, reduced keratinocyte proliferation and faulty gap junctions, all of which inhibit wound epithelialization.⁴⁴ In prospective studies, hyperglycemia has been demonstrated to be an independent predictor of extended hospitalization, recurrence, and amputation due to DFUs.^45–48 Increases in random blood glucose levels indicate a lack of self-monitoring of glycemic control in diabetic individuals, which shows that providing diabetic patients with health education to enhance adherence and proper psychological support may be the key to minimizing the incidence of hard-to-heal DFUs.

C-reactive protein encompasses a group of proteins (acute proteins) that increase dramatically in plasma when the body is exposed to infection or tissue damage. As a sensitive indicator of infection, C-reactive protein is regarded as a crucial feature in wound healing, recurrence, and a potentially lower risk of limb amputation.^49,50 Recent research has also linked high leukocyte and CRP levels to a poor prognosis and treatment failure in diabetic foot ulcers.⁵¹ Volaco et al observed that elevated CRP levels are a robust predictor of requiring amputation in individuals with diabetic foot ischaemia.⁵²

The wound area is a crucial factor in determining the prognosis and grading of DFUs. In a large multicenter study conducted by David J. Margolis et al, wound size at the initial visit was found to be a predictor of neuropathic foot ulcer healing failure.³² A wound size >2 cm² was a predictor of wound healing failure for 20 weeks.⁵³ Early detection and effective treatment will be crucial to prevent DFUs from developing into nonhealing wounds.⁵⁴

Our research has the following strengths that contribute to its significance: 1. To the best of our knowledge, we are the first to compare machine learning algorithms to identify optimal algorithms for the refractory DFU problem. Our work further verifies the superiority of the NB model in classification issues involving low-dimensional samples. 2. A web calculator was developed using the study’s predictive model. This web-based tool may help doctors with the early detection of hard-to-heal diabetic foot ulcers, leading to a multidisciplinary collaborative or more aggressive treatment that can minimize hospitalization, recurrence, and amputation rates. General practitioners can use these findings as a reference to refer such individuals to a specialist for earlier intervention. 3. Our research indicates that random blood glucose, CRP, and wound area are essential factors influencing whether a DFU will be difficult to heal.

Limitations

Despite these findings, this study has certain limitations. First, the limited amount of data hinders the performance of machine learning. Second, according to data gathered from two tertiary institutions, virtually all patients with diabetic foot ulcers had type 2 diabetes and were UT grade 3, suggesting lesions located as deep as tendons, capsules, and even bone, which causes limitations regarding the application of our prediction model. Nonetheless, our filtered optimal NB technique can integrate current scientific knowledge via a priori distributions; thus, our data can serve as a supplement for future researchers. Last, given the nature of machine learning that can be achieved for large-scale feature filtering, patient therapies and some imaging images can be incorporated into model predictions.

Conclusion

In this study, DFU data from two tertiary hospitals in East China were collected, and six machine learning models for predicting hard-to-heal DFUs were developed and evaluated. In addition, the ranking of the importance of features revealed that random blood glucose, CRP, and wound area were the most significant factors influencing hard-to-heal DFUs. Machine learning algorithms have a black-box nature and are difficult to demonstrate arithmetically. We visualize the naïve Bayesian algorithm model system built in this study as a user-friendly web-based calculator to help clinicians identify refractory DFUs for prompt intervention at initial admission.