Applying Machine Learning Models to Predict Medication Nonadherence in Crohn’s Disease Maintenance Therapy

Introduction

Crohn’s disease (CD) is an inflammatory bowel disease (IBD) that can affect any segment of the gastrointestinal (GI) tract with an etiology combining both genetic predisposition and environmental factors.¹ The disease is characterized by the alternating courses of remission and relapse, and its treatment often involves two consecutive phases of induction and maintenance which plays a critical role in remission. Various studies have found that low medication adherence is closely linked to more frequent relapses and hospitalization, increased mortality, and higher healthcare expenditure.^2,3 Improved medication adherence can, by contrast, lead to well-maintained condition, lowered overall medical cost and better quality of life. It is estimated that at least 30–45% of the patients fall into the category of nonadherence to the maintenance therapy.^4,5

Studies using logistic regression analytics have been performed to identify factors that may influence adherence to medication and/or clinical follow-ups, and the results suggest that age, employment, socioeconomic status, travel time to clinic, C-reactive protein levels, patient perceived stress or anxiety, and duration of the disease are among the predictors for adherence.^6–9 Among them, the psychological impacts of the disease, such as attachment insecurity and impaired mentalization ability, have been particularly placed under the spotlight as they can be both the risk factor to and the outcome of low medication adherence, establishing a vicious triangle of difficulty in disease management, poor prognosis and nonadherence to therapy.^10,11 However, these analyses sometimes generate conflicting results as predictors such as gender, smoking or education level can show strong correlation in one setup yet be irrelevant in another, depending on the study design, patient profile, and data collection. Together these aspects make more difficult the tasks of developing reliable prediction models for adherence and patient interventional programs for adherence enhancement.

With the advance of data processing technologies, machine learning algorithms such as artificial neural network (ANN) and support vector machine (SVM) have shown great potentials in constructing predictive models based on electronic health records to support medical decision making, especially in chronic disease management of diabetes, heart failure and kidney disease.^12–14 Machine learning models are considered having the strengths of including nonlinear associations, less biased auto-learning and higher flexibility to avoid over-fitting when compared with the traditional logistic regression. Three attempts of employing machine learning techniques in CD management have led to the finding of seasonal patterns of disease onset and relapse using ANN, an optimized surgical predictive model using random forest following the comparison of five different algorithms including ANN and SVM, and random forest prediction models for objective remission and nonadherence of IBD/CD patients on thiopurines.^15–17

Although the incidence rate of CD in China is lower than that in the western countries, a fast-growing trend is being observed in parallel with the nation’s rapid industrialization process and adoption of the more westernized dietary habits.¹⁸ Due to the differences of genetic, environmental and socioeconomic backgrounds between the Chinese population and people in the western countries, it is important to develop further understanding of the mechanism underlying nonadherence of CD treatment in China. Hence, in this paper, we aimed to explore and compare machine learning and logistic regression models that can help predict CD patients who may demonstrate nonadherence to azathioprine (AZA), which is the first-line immunosuppressant recommended for CD maintenance therapy. Such model(s) once externally validated in the real-world study can help guide caregivers to prioritize their time and efforts to CD patients according to their risk profiles, and offer proactive and personalized interventions by addressing the relevant risk factors.

Patients and Methods

Patient Recruitment

This cross-sectional study included CD patients who were either hospitalized at or visited the GI Department of Shanghai Ruijin Hospital, Affiliated to Shanghai Jiaotong University School of Medicine (Shanghai, China). A total of 446 consecutive CD patients on AZA maintenance therapy for at least 6 months were recruited between Sep 2005 and Sep 2018. CD diagnosis was confirmed based on clinical, morphological (radiological and/or endoscopic) and pathological evidence, and remission was considered if the Crohn’s Disease Activity Index (CDAI) was less than 150. The maintenance AZA dosage was adjusted according to the side effects and blood tests in a stepwise manner to reach the maximal tolerated dose (1.0–1.5 mg/kg/day). Exclusion criteria were patients with: 1) concomitant treatment using drugs other than AZA, such as corticosteroids, methotrexate or anti-TNF, for maintenance therapy, 2) other accompanied chronic diseases, 3) disease flares yet interrupted AZA treatment, and 4) difficulty understanding the questionnaire. All patients provided written informed consent to participate in the study which was approved by the Ethics Committee of Shanghai Ruijin Hospital in compliance with the Declaration of Helsinki.

Patient Characteristics and Data Extraction

A single-center database of 128 items for CD management was constructed by manually extracting patients’ health information from the hospital electronic medical records (EMR) that captured patients’ profile including demographic characteristics, socioeconomics (education, occupation and income), clinical presentations, laboratory tests and diagnosis, therapeutic regimen, and follow-up records. Questionnaires of AZA adherence, medication beliefs, medication knowledge, and anxiety and depression were other data sources to update and complete this database.

Assessment of AZA Adherence

AZA adherence was assessed using the Medication Adherence Report Scale (MARS) which was designed as a 4-item questionnaire to be provided to patients every time during their hospital visit. Each MARS self-report question had a 5-point scale (where 5 = never, 4 = rarely, 3 = sometimes, 2 = often and 1 = very often) to produce a score between 1 and 5 with the total MARS score between 4 and 20.^19,20 High adherence was indicated by the MARS score of 17–20, approximately ≥80% adherence rate, and nonadherence by <17, ie, <80% adherence rate, according to the previous reports.^21–25

Assessment of AZA Medication Beliefs

Patients’ beliefs about AZA medication were evaluated using the Beliefs about Medicines Questionnaire (BMQs).²⁶ The BMQs consisted of two sections in the 5-point Likert scales: belief of medication necessity and concerns about potential adverse effects. Each section included a 5-item questionnaire with scores ranging from 5 to 25, and was calculated independently. Higher scores (15–25) indicated greater belief or concerns. Medication acceptance was considered in the case of high necessity but low concern scores, suggesting improved medication adherence.

Assessment of AZA Medication Knowledge

Patients’ general knowledge of AZA was evaluated using a self-report questionnaire-the AZA Knowledge Report Scale (AKRS) which we specifically designed for the Chinese patients (see Supplement) considering the uniqueness of the patient profile and the healthcare system in the country. This new instrument had 10 questions regarding the AZA knowledge of treatment indication, dose, cessation, side-effects, surveillance, and pregnancy. The questions were prepared as yes and no responses (yes: 1 point, no: 0 point), and the total AKRS score was 0–10 with higher scores indicating better AZA knowledge.

Assessment of Anxiety and Depression

Patients’ anxiety and depression were evaluated using the Hospital Anxiety and Depression Scale (HADS) which was a 14-item questionnaire (7 for anxiety and 7 for depression).²⁷ Each item was assessed with a 4-point scale (0–3), and higher scores suggested higher levels of anxiety or depression: 0–7 (normal), 8–10 (mild), 11–15 (moderate) and 16–21 (severe).

Data Processing and Feature Selection

We divided all 446 patients into two groups of AZA adherence (MARS 17–20) and nonadherence (MARS < 17), and constructed a database by extracting their information from EMR and questionnaires. Univariate analyses included student’s t-test, Mann–Whitney test or one-way analysis of variance (ANOVA) for continuous variables and Fisher’s exact test or Chi-square test for categorical variables. Statistical analysis was performed using SPSS 22.0 (SPSS, Inc, Chicago, IL) and significance was regarded as P-values <0.05.

For feature selection, we combined two methods of univariate analysis and random forest as random forest variable importance measures have been shown effective in classification tasks such as identifying genetic biomarkers to predict the onset or outcome of certain diseases.²⁸ The importance of each predictor (feature) was evaluated by permutation of out-of-bag (OOB) prediction using the “feature selection” function in MATLAB Statistics and Machine Learning Toolbox.²⁹ As the independent variable data set in this study was heterogeneous, it was not reliable to estimate the variable importance using random forest model developed on classification and regression trees (CART). We hence applied the interaction test instead of CART to grow unbiased trees, and compared the random forest variables with those found in the univariate analysis. The variables in common were then chosen as the feature set for model construction.

Development of Prediction Models of Logistic Regression, Back-Propagation Neural Network and SVM

We first constructed a new dataset using the identified features and classification labels of AZA adherence for all 446 patients, and then randomly divided this new dataset into the training and the testing sets in the ratio of 9:1. The modeling consisted of two steps: 1) the learning process in which the three models of logistic regression, back-propagation neural network and SVM were applied and validated on the training and testing sets, and a stratified 10-fold cross-validation procedure was also employed to the predictive models on both datasets to limit overfitting and selection bias; and 2) the evaluation process in which the five metrics of accuracy, recall, precision, F1 score and AUC were tested and compared (Figure 1).

Figure 1 The flow chart of developing machine learning models. Note: The processed patient data were randomly divided into the training and testing sets in the ratio of 9:1, followed by the learning and evaluation steps. Abbreviations: LR, logistic regression; BPNN, back-propagation neural network; SVM, support vector machine; AUC, area under the receiver operating characteristic curve.

The logistic regression model was developed using SPSS 22.0 (SPSS, Inc, Chicago, IL) and variables with P-value <0.05 were included in the model. We tested the linearity in the logit for continuous variables using the Box-Tidwell Transformation. If any of the resulting statistic terms were significant (P < 0.05), we would translate associated continuous variables into categorical variables to satisfy the linearity assumption.³⁰ The identified variation inflation factor (VIF) for each independent variable was used to check the absence of multicollinearity, and variables with VIF > 10 would be eliminated. Once these assumption tests ensured the use of logistic regression, a stepwise approach was applied to construct the model.

Back-propagation neural network is a machine learning algorithm that learns by adjusting the node connection weights through backward propagating the output error term, and we used MATLAB R2017a in developing the model. As the prediction of patients’ nonadherence may not be linearly separable, we chose back-propagation neural network with two hidden layers that have been found sufficient for creating classification regions of any desired shape.³¹ The number of nodes in each hidden layer would be determined through trial and error. The transfer function between the input/hidden and hidden/output layers was generated using the sigmoid function-logsig, and the train function was achieved using trainlm according to Levenberg-Marquardt optimization.

SVM is a supervised machine learning algorithm commonly used for data classification modeling, and we used the open-source SVM software library-LIBSVM for model development. For both training and testing datasets, values of input features were normalized into the range of 0 to 1, while the classification label of AZA adherence and nonadherence were designated as 1 and −1, in order to meet the format requirement of SVM. As the number of features was small, and the relation between the features and the adherence outcome could be nonlinear, we constructed the SVM model using the kernel of radial basis function (RBF) that included parameter C as the weight between empirical error and generalization error, and parameter γ to control the shape of the separating hyperplane for the training predictive model. Optimization of both C and γ was achieved by grid-search using cross-validation, and the pair of (C & γ) with the highest accuracy would be selected for continuous training steps until the final classifier was produced.³² The classification model would then be applied to the testing dataset for validation.

Performance Evaluation

To assess the performance of each model we included four statistics of accuracy, recall, precision, and F1 score as follows:

• Accuracy = (TP + TN)/(TP + TN +FP + FN)
• Recall = TP/(TP + FN)
• Precision = TP/(TP + FP)
• F1 Score = (2 × Recall × Precision)/(Recall + Precision)

where TP, FP, TN, and FN referred to the number of true positive, false positive, true negative, and false-negative cases, respectively. In this study, true positive indicated the correctly classified case of AZA nonadherence; false-positive indicated a case classified as AZA nonadherence yet was adherent to AZA; true negative indicated the correctly classified case of AZA adherence; and false negative indicated a case classified as AZA adherence yet was of AZA nonadherence. More comprehensive assessment was achieved using AUC which has been proposed as an accurate measure of evaluating the predictive ability of learning algorithms.³³

Results

Patient Characteristics Between Groups of AZA Adherence and Nonadherence

During the study period, 553 CD patients attending the GI Dept. were solicited to the present research. Among them, 34 fell into the exclusion criteria, 50 refused to participate, 21 had missing values in EMR, and 2 were unable to complete the questionnaire. As a result, a total of 446 CD patients on AZA maintenance therapy with an average AZA duration of 34.3 months were recruited and their characteristics are listed in Table 1. Of the study population (male 58.3%, mean age 31.7), 187 (41.9%) patients were of AZA nonadherence (MARS < 17, ie, adherence rate <80%) and demonstrated differences in univariate analysis from the AZA adherence group (MARS 17–20) in age, marital status, education level, alcoholism, psychological distress, medication beliefs and knowledge, P < 0.05, respectively (Table 1). In addition, the incidence rates of moderate-to-severe anxiety and depression were 18.2% (81/446) and 12.1% (54/446), comparable to those in a previous report.³⁴

Table 1 Patient Characteristics Between Groups of AZA Adherence and Nonadherence

Feature Selection by Random Forest and Univariate Analysis

Random forest was used to select SVM model features according to their importance measures. We started with building 100 classification trees based on the original dataset to identify features with the highest predictive accuracy, and the top-10 features were illustrated in descending order in Figure 2. Cross-examination of the random forest features and those of univariate analysis yielded eight common features to be selected for modeling, including: age, education, alcoholism, anxiety, depression, AZA necessity belief, AZA knowledge and AZA concerns belief.

Figure 2 Top 10 features with the highest importance identified by random forest. Note: The importance score on the Y-axis was quantified by computing the OOB error. Abbreviation: OOB, out-of-bag.

Development and Evaluation of the Logistic Regression, Back-Propagation Neural Network and SVM Models

Contribution of each independent variable to the logistic regression model and its statistical significance was listed in Table 2. This multivariate analysis identified that AZA concerns belief (OR: 3.130, 95% CI: 1.673–5.854), education (OR: 2.199, 95% CI: 1.543–3.134), anxiety (OR: 1.549, 95% CI: 1.372–1.749) and depression (OR: 1.190, 95% CI: 1.080–1.312) were risk factors of AZA nonadherence. By contrast, AZA necessity belief (OR: 0.004, 95% CI: 0.0004–0.033) and AZA knowledge (OR: 0.805, 95% CI: 0.679–0.955) were protective factors of adherence.

Table 2 Predictive Factors for AZA Nonadherence in Patients with CD on Maintenance Therapy (Multivariate Analysis)

The performance of the three models was compared using four evaluation measures in Table 3. The average accuracy of the three models was 81.6% for logistic regression, 85.9% for back-propagation neural network, and 87.7% for SVM, respectively. The SVM model scored highest in every other aspect of recall, precision, and F1 score. Moreover, the AUC analysis that can classify patients of nonadherence from adherence also identified that the SVM model had the highest value of 0.930 (high accuracy), as compared with those of logistic regression (0.896, moderate accuracy) and back-propagation neural network (0.912, high accuracy) in Figure 3.³⁵ Taken together, these results indicated that the SVM model is most appropriate in predicting AZA nonadherence in the maintenance therapy among Chinese CD patients.

Table 3 Predictive Performance of LR, BPNN and SVM Models

Figure 3 AUC of LR, BPNN and SVM models for prediction of AZA nonadherence. Abbreviations: AUC, area under the receiver operating characteristic curve; LR, logistic regression; BPNN, back-propagation neural network; SVM, support vector machine; AZA, azathioprine.

Discussion

The present research is the first report of developing machine learning models to predict Chinese CD patients who may undergo nonadherence in AZA maintenance therapy. The connection between nonadherence and poor clinical outcomes has long been established in CD and other chronic conditions.^3,7,8,36 A variety of efforts have been poured into regression analyses to identify predictors that can help profile patients’ adherence so that engagement programs can be developed and implemented to target low adherers.^37–41 To circumvent the common limitation of potential interactions of dependent variables in the process of logistic regression, we applied back-propagation neural network and SVM modeling and compared them with logistic regression on the dataset of CD patients, and found that SVM showed the best prediction performance on AZA nonadherence. These models lay the groundwork of developing a user-friendly digital tool for clinical practitioners to identify CD patients of nonadherence at early stage, allowing appropriate interventions to be provided in time to improve adherence and disease prognosis.

The overall AZA nonadherence rate in the present study was 41.9%, and multivariate analysis revealed results in agreement with the previous findings that education, psychological distress, and medication beliefs and knowledge could be predictors of medication nonadherence.^7,9 The association between education and nonadherence has been postulated that people of higher educational degrees often live a busy professional and social life, which may lead to forgetfulness of taking medicine.³⁸ Disease-related psychological distress is known to be another factor influencing the disease progress and prognosis, as one of the latest studies proposed a potential interplay among the IBD-associated stress, impaired mentalization and attachment insecurity. In this scenario medication, nonadherence could be the cause and the outcome at the same time due to patients’ emotional disturbance, giving adherence improvement a more critical role in CD management.⁴² In addition, we confirmed strong relevance of the following two elements to AZA nonadherence: medication beliefs (necessity and concern) and medication knowledge. These findings are not only in line with others’ works but strongly implies that patient follow-up programs that tackle these factors, such as involving psychiatrists for mental health evaluation and intervention, and healthcare educators for medication-related coaching, shall be developed as a comprehensive solution to improve both mental health and knowledge to achieve higher AZA adherence.

All three models of the present study showed reliable prediction with the minimum accuracy of 81.6% and AUC of 0.896. We believe this generally good performance may attribute to the procedure of feature selection in which random forest and univariate analysis were combined to produce a feature set of eight-dimensional vectors of small dimensionality. This step would not only reduce the time of model development but also help avoid overfitting, leading to enhanced model generalization and better classification. Another reason was the application of the stratified 10-fold cross-validation in the modeling, a step that could identify selection bias or overfitting, shedding further insights into how the model would generalize on an independent dataset. Interestingly, we found SVM outperformed back-propagation neural network and logistic regression in nearly every aspect with an accuracy of 87.7%, recall of 86.2%, precision of 85.6%, F1 score of 0.855, and AUC of 0.930. This result suggests that SVM is an optimal classifier in processing complex clinical data in AZA nonadherence prediction.

An intriguing recent study demonstrated a prediction model on thiopurine nonadherence among IBD/CD patients using random forest modeling but based on mainly lab test data of complete blood count with differential and comprehensive chemistry panel.¹⁷ The goal of this model, which ran on a set of over 20 variables such as the red cell distribution width and mean corpuscular volume, was to predict nonadherence using objective data and therefore to adjust thiopurine dose and regimen during the therapy. It is possible that all these algorithms, regardless of the data processing technology or the source, shall function in complementary to achieve the best performance on classification. Therefore, various real-world studies need to be designed and carried out to evaluate the health outcomes of these models in combination with patient adherence enhancement programs.

There are several limitations. First, AZA was selected as the representative of CD maintenance therapy agent as it was unanimously covered by China’s national payer system and thus most commonly used, while other immunomodulators such as anti-TNF need to be paid out-of-pocket by patients and only account for a small portion of usage. Hence, predictors and models developed in this study may not apply to cases when medications other than AZA are chosen for the maintenance therapy. Second, this was a single-centered study in Shanghai and the patient profile might be biased and not representative of the Chinese as a whole. Data interpretation thus requires caution when extrapolated to the CD patients in general. Last but not the least, application of machine learning prediction models in daily clinical practice remains a challenge, which could result from the much higher regulatory standard for model performance and fidelity when in medical use, or from the lack of causality interpretation when demanded by the practitioner.

Conclusions

In conclusion, the present research introduced three machine learning models to predict Chinese CD patients of AZA nonadherence and proposed an SVM model with better classifier performance than the back-propagation neural network or logistic regression model. This work also reconfirmed that variables including higher educational degrees, enhanced levels of anxiety and depression, and less medication beliefs and knowledge were risk factors for nonadherence and thus can serve as targets to be further addressed by tailored engagement programs. We are in the process of developing a cloud-based solution with built-in SVM model and mobile Apps for caregivers and patients in an effort to integrate the adherence enhancement intervention into daily CD management. Future investigation would focus on clinical validation of this SVM model-enabled cloud solution on health and pharmacoeconomic outcomes, and development of more comprehensive machine learning solutions that aggregate data from multi-centered patient pool and assist in AZA dosage adjustment and life-style coaching.