Machine Learning and Interpretability Study for Predicting 30-Day Unplanned Readmission Risk of Schizophrenia: A Retrospective Study

Introduction

Schizophrenia is an important challenge in the mental health field, affecting about 24 million people worldwide;¹ its incidence rate is one in every 222 adults (0.45%).² The average life expectancy of patients with schizophrenia is about 15 years less than that of general patients and they have a 5–10% lifetime risk of suicide.³ Even after treatment in professional mental health institutions, the relapse rate of patients with schizophrenia remains high. More than 80% of patients relapse within five years and the recurrence rate is as high as 63% within two years.⁴ Up to 86% of patients are readmitted after experiencing a relapse during a 2-year follow-up period,⁵ increasing the risk of unplanned readmission (UPR).

The UPR rate is a focal point for all healthcare systems and decision-makers because it indicates high costs and unfavorable patient outcomes.⁶ Mental illness-related readmission rates range from 5% to 43%,^7,8 which exceed those of the general health condition-related readmission rates of 2.8–38%.⁹ Moreover, schizophrenia has the highest readmission rate among all mental disorders.¹⁰ Numerous studies have attempted to predict the risk of rehospitalization after discharge for patients with schizophrenia, with time windows of concern including 7, 30, 90, and 365 days.^11–13 Hospitals widely use the 30-day readmission rate as a quantifiable quality indicator.¹⁴ Compared with a 7-day window, patients may be more likely to experience medication adjustment and social and environmental changes with a 30-day window, resulting in a better opportunity to capture trends in potential relapse risks. It can also reduce the follow-up time and use of medical resources compared with longer time windows.

The “deinstitutionalization” strategy has limited effectiveness in alleviating the 30-day UPR of Chinese patients with schizophrenia,¹⁵ highlighting the urgency of predicting its risk factors. One systematic evaluation identified a low education level, unemployment status, past psychiatric hospitalization history, and hospital length of more than 7 days as key predictors.¹⁶ A reduction in hospital days and the absence of community care after discharge were found to significantly increase the probability of readmission.¹⁷ Specific demographic characteristics (eg, age, sex, education level, and marital status) and clinical characteristics (eg, initial diagnosis, medical comorbidity index, age at first onset, length of the last hospital stay, and number of previous hospitalizations) were also closely associated with 30-day UPR.^16,18–20 However, few studies have constructed clinically realistic and highly interpretable prediction models based on proven predictive factors to accurately predict the 30-day UPR risk of patients with schizophrenia.

Given the remarkable heterogeneity of schizophrenia and its unique disease development trajectory,²¹ predicted demand for the disease tends toward personalization rather than simple group-level modeling. Risk prediction models are important for promoting the effective allocation of resources for high-risk patients and for preventing readmission. Previous studies have typically used readmission risk assessment models with traditional logistic regression (LR) or Cox regression,^4,22 which are not ideal for processing large-scale, high-dimensional patient data. Machine learning (ML) algorithms have already adapted to handle nonlinear relationships among complex and interrelated sets of variables.²³ For instance, ML has surpassed traditional statistical models for predicting suicide attempts in patients with major depressive disorder, highlighting its superiority.²⁴

Nowadays, ML has been widely used to predict the readmission and prognostic risks of various clinical diseases.^25–27 Through analysis of heterogeneous big data such as electronic medical records (EMRs), ML can find hidden disease rules and risk factors, providing patients with more accurate and personalized risk assessment and intervention suggestions. However, ML models are often called black boxes with internal agnostic properties. In recent years, the SHapley Additive exPlanation (SHAP) method has been used in model interpretation,²⁸ allowing users to compare the correlations between variables and features by visually presenting the influences and relationships of each input variable on the model output. Some studies have evaluated all-cause readmissions after the discharge of patients with schizophrenia,^10,29 but no research has utilized ML models to predict their 30-day UPR. Meanwhile, the SHAP values used to analyze the interpretability of the models remain poorly understood among such patients.

This study identified the key risk factors leading to 30-day UPR by reviewing standardized EMRs data in the Hospital Information System. It subsequently explored the feasibility of using five ML algorithms to predict the 30-day UPR after the discharge of patients with schizophrenia, and the SHAP method was introduced to improve the interpretability of the model to provide a reference for early clinical identification and intervention.

Material and Methods

Study Design and Participants

This retrospective study extracted the EMRs of patients with schizophrenia hospitalized in a university-affiliated tertiary hospital in Wuhan, Hubei province, China between January 1, 2021, and June 30, 2024, from the Hospital Information System. The “index admission” referred to the initial hospital records of patients diagnosed with schizophrenia who underwent at least one admission and subsequent discharge between January 1, 2021, and June 30, 2024. On this basis, 1,131 inpatients were initially identified, covering their 2,564 admission records. Patients were deemed eligible based on the following inclusion criteria: they met the International Classification of Diseases-10 diagnosis criteria for schizophrenia and were aged ≥18 years. The exclusion criteria were as follows: death or transfer during the previous hospitalization, severe amounts of missing data, and planned readmission for various reasons. The study ultimately included a cohort of 1,123 patients of whom we had relatively complete demographic and clinical information during the specified period.

Outcomes

The main outcome was the occurrence of UPR within 30 days of discharge. UPR referred to non-predicted readmission due to the same disease or factors associated with discharge disease, as defined by the National Health Insurance Administration.⁹ For patients with repeat admissions, the first consideration was whether there were two admission records of less than 30 days apart. The attending physician excluded various planned readmissions, as well as patients with multiple admissions records but more than 30 days apart, to obtain case information for outcome events. Patients who experience multiple 30-day UPRs, regardless of the stage of their illness, are categorized as readmission group. The primary goal of this study is to identify key predictors of 30-day UPR. Given that the demographic and clinical information of these patients remains relatively stable across different readmission events, we uniformly retained and analyzed their EMRs from their first 30-day UPR.

Predictor Variables

This study conducted a preliminary selection of the risk factors for 30-day UPR based on prior research and clinical experience. It classified candidate predictors into demographic and clinical variables. For this study, data from the first admission or first UPR were selected for analysis for each predictor variable. First, this study cleaned and sorted the data on patient attributes. It excluded the variables that could not be analyzed, such as names, hospitalization, and discharge dates. Finally, 16 predictors were selected as independent variables to further explore their relationship with 30-day UPR in subsequent analyses.

ML Algorithms

This study selected five ML techniques as the prediction model-based algorithms. LR, a generalized linear regression analysis model suitable for classification problems, estimates the probability of event occurrence based on a given dataset of independent variables.³⁰ The decision tree (DT) classification process can be optimized by recursively selecting features and dividing the training data into data subsets based on these features.³¹ Random forest (RF) is a supervised ML algorithm based on a DT, reducing the one-sidedness and judgment inaccuracy caused by a single DT.³² It can handle both classification regression and dimension reduction problems. The extreme gradient boosting (XGB) algorithm has strong scaling, and it is convenient as well, since it can be constructed in parallel.³³ XGB can also balance prediction accuracy and model complexity. It is used for classification, regression, and ranking problems. Support vector machine (SVM) is a supervised learning algorithm used for classification and regression analysis.³⁴ It implements a classification or regression task by constructing an optimal hyperplane and then uses support vectors to determine its location.

Model Training and Evaluation

This study utilized R 4.3.2 for data analysis, development, and validation. During the data preprocessing phase, this study coded the non-numeric data to ensure compatibility with the algorithm models. In order to deal with the problem of missing data, relevant data in EMRs are used to fill in the missing values. Specifically, for numerical variables, the median is used to fill in the missing values; for categorical variables, the mode is used to fill in the missing values. It processed these data via random downsampling to obtain category-balanced datasets for modeling. It then used the tableone package in R to analyze the demographic information.

The initial feature screening was based on the entire dataset. To identify independent factors capable of predicting 30-day UPR, variables that showed significant differences in univariate logistic regression analysis were included in the multivariate logistic regression analysis. Subsequently, the importance of features was analyzed using SHAP values, and the results were compared with those from the multivariate analysis. The final results were utilized for model construction, with all subsequent modeling steps strictly segregated between the training and test sets.

This study utilized stratified random sampling to divide the dataset into training (70%) and testing (30%) sets, ensuring that the ratio of patients was consistent in both subsets. We constructed ML models by employing the five distinct algorithms: LR, DT, RF, SVM, and XGB. The training set was utilized for both model training and hyperparameter tuning, with the accuracy from 10-fold cross-validation serving as the key metric for identifying the optimal parameters for each model. To implement this, the caret package in R was used to evenly distribute the training set data into 10 groups. Each iteration involved using one group as the validation data and the remaining nine groups for training to assess the prediction stability of the model.

In the model evaluation stage, the mlr3 and tidymodel libraries in R were used to remove the result labels from the test set data and input them into the trained ML model after parameter tuning. The model made judgments and obtained results, and the predicted results were compared with the real result labels using a confusion matrix. The predictive and evaluative performance of the model was evaluated based on the accuracy, precision, recall, F1 score, and area under the receiver operating characteristic curve (AUC). The SHAP tool was used to conduct post hoc interpretation analysis on the most accurate model, determining the importance of the model features in the decision-making process. Finally, the ggplot2 package in R was used to visualize the model data.

Model Interpretation

The universal SHAP method is used to interpret the output of any ML model.²⁸ This innovative approach links game theory to local interpretation and provides the only consistent, locally accurate method for calculating additive features. Its introduction is an important milestone in practical clinical decision support. It enables healthcare professionals to more effectively understand and interpret complex predictions made by ML models and recognize how specific factors drive individual patient predictions, thereby improving the accuracy and reliability of clinical decision support.

Results

Participant Characteristics

This study included 1,123 Chinese adult patients with schizophrenia. Among them, 343 patients experienced unplanned readmissions, while 780 did not (30-day unplanned readmission rate: 30.54%). The 30-day unplanned readmission group displayed sex differences (P<0.001) (70.0% males, 30.0% females). The most represented age group was 45–59 years (54.8%), followed by 18–44 years (22.7%) and ≥60 years (22.4%). Table 1 presents the descriptive statistics.

Table 1 Baseline Characteristics of 30-Day Readmission Vs Non-Readmission Patients (n=1123)

Analysis Of the Influencing Factors of UPR

In the univariate logistic regression analysis, we selected statistically significant variables (P<0.05) to be included in the multivariate logistic regression model for in-depth analysis. Only the variables showing an extremely high significance level were retained (P<0.001) (Table 2). The SHAP method was used for the feature importance analysis (Figure 1). Five common features of the two methods were significant, including in the final risk prediction model: number of somatic comorbid diseases, duration of the disease course, length of the latest hospital stay, drug withdrawal history, and sex.

Table 2 Multivariate Logistic Regression Analysis Results

Figure 1 Feature importance analysis. Notes: The higher the features rank, the more significant its impact on the outcome.

Model Evaluation And Validation

The LR, RF, XGB, SVM, and DT methods were combined with the five statistically significant variables to establish the ML models. The AUC, accuracy, precision, recall, and F1 values were calculated for the five models. The results showed that in the testing set, the AUC values were 0.794, 0.717, 0.823, 0.830, and 0.810 for the LR, DT, RF, XGB, and SVM models, respectively (Table 3 and Figure 2). The XGB model performed the best, with the highest AUC (0.830), accuracy (0.7768), recall (0.7255), and F1 (0.6637) values. Overall, the XGB model outperformed the other models in identifying the risk factors for the 30-day UPR of patients with schizophrenia.

Table 3 Performance Metrics of the Machine Learning Models Based on the 10-Fold Cross-Validation Testing Set

Figure 2 Comparison of the prediction performance of machine learning models based on the 10-fold cross-validation test set.

Model Interpretation

The SHAP is an additive explanatory model inspired by Shapley values. This study performed an overall visualization of the features using the shap.summary_plot method (Figure 3). Among the top five contributing model features, each row represented a feature with an abscissa SHAP value. The dots represented the samples. More yellow indicates larger features, and more purple indicates smaller features. The influencing factors of the XGB model based on SHAP showed that the number of somatic comorbidities was the most important feature affecting 30-day UPR of patients with schizophrenia; the risk of UPR increased with an increase in the number of comorbidities. Long disease duration is a protective factor against 30-day UPR in patients with schizophrenia. Male patients with a history of discontinuation had an increased risk of 30-day UPR. In addition, SHAP can not only analyze the influencing factors of the prediction model at the overall level, but also analyze the influencing factors for individuals.

Figure 3 SHAP summary plot of the 5 risk factors. Notes: Among the top five contributing model features, each row represented a feature with an abscissa SHAP value. The dots represented the samples. More yellow indicates larger features, and more purple indicates smaller features.

As shown in Figure 4A, the SHAP value for Patient A was 0.00163, which was less than the model-predicted base value of 0.305; hence, the model predicted that the patient would not have a 30-day UPR. Patient A was female, with a length of latest hospital stay ≤7 days, duration of disease course ≥30 years, and with a “without drug withdrawal history” value that was blue, indicating that these features reduce the patient’s SHAP value and thus reduce the risk of 30-day UPR. Conversely, patient A with five or more somatic comorbid diseases had the opposite effect on the prediction results compared to other characteristics, indicating that they were at a higher risk of 30-day UPR in the future.

Figure 4 Examples of SHAP interpretation for patients (A and B). Notes: The red bars represent positive significance for the outcome index, and blue bars represent negative significance for the outcome index; Number of somatic comorbid diseases=0 is ≤1, Number of somatic comorbid diseases=1 is 2–4, Number of somatic comorbid diseases=2 is ≥5; Duration of the disease course=0 is <10 years, Duration of the disease course=1 is 10–19 years, Duration of the disease course=2 is 20–29 years, Duration of the disease course=3 is ≥30 years; Length of the latest hospital stay=0 is ≤7 days, Length of the latest hospital stay=1 is 8–29 days, Length of the latest hospital stay=2 is ≥30 days; Drug withdrawal history=0 is No, Drug withdrawal history=1 is Yes; Sex=0 is female,Sex=1 is male.

As shown in Figure 4B, the SHAP value of Patient B was 0.47, which was greater than the base value predicted by the model; hence, the model predicted that the patient would undergo a 30-day UPR. The number of somatic comorbid diseases of Patient B was blue, which played a positive role in reducing the risk of 30-day UPR. This patient was male, had a drug withdrawal history, the latest hospital stay duration of 8–29 days, and a duration of disease course of 10–19 years. These factors placed the patient in a higher category of risk for 30-day UPR. In addition, of all the relevant variables, the duration of the disease course had the widest range and the greatest impact, which should be paid attention to.

Discussion

This study is the first to construct a 30-day UPR model of patients with schizophrenia using multiple predictors extracted from EMRs data via ML, with SHAP values used to interpret the findings. While prior research has used ML to predict the risk of all-cause readmissions or UPR,^25–27 studies predicting the risk of readmission for patients with schizophrenia remain limited.^10,29 Therefore, further exploration is warranted. The results of this study showed that the 30-day UPR rate of patients with schizophrenia was 30.54% (343/1,123), which aligns with the findings from comparative studies; however, this rate was lower than the 52.56% readmission rate in 11 Spanish hospitals (1,611/3,065).^4,29 Variations in rehospitalization rates may be influenced by the specific policies in countries or regions as well as by differences in psychiatric hospital bed turnover and the provision of community care services after discharge.

This study produced AUC values that ranged from 0.717 to 0.830. A previous study utilized generalized linear models via lasso and elastic-net regularization (AUC=0.697) and XGB (AUC=0.738) models to predict the readmission of patients with mental disorders.²⁷ Further, a previous study evaluated six ML algorithms using the hospital EMRs of 10,358 patients aged >65 years undergoing surgery with general anesthesia; they found that all of the models have unremarkable predictive abilities for 30-day UPR rates (AUC: 0.6865–0.8654, precision: 0.2683–0.3977, recall: 0.1550–0.3289, and F1: 0.1957–0.4909).³⁵ A prior study used seven ML models to predict 30-day UPR rates after spinal fusion in 59,145 patients, but only general performance was observed (AUC: 0.63–0.66).³⁶ Overall, the ML models in this study demonstrated promising predictive performance during internal validation, indicating the feasibility of constructing such predictive models. These findings also provide support for future large-scale external validation studies to explore the potential application value of the models in identifying and preventing adverse outcomes in patients with schizophrenia.

This study found that the overall predictive power of the XGB model exceeded that of the other four models. The predictive power of the included features, and the explanatory features between them, is interesting: few studies have considered easily collected features for modeling the 30-day UPR of patients with schizophrenia. In addition, this study utilized multivariate logistic regression and SHAP values to quantify the importance of model features and ultimately identified the top five key features: number of somatic comorbid diseases, duration of the disease course, length of the latest hospital stay, drug withdrawal history, and sex.

Within the dataset, the number of somatic comorbid diseases showed the most pronounced significance. Extant research has highlighted the close connection between physical health status and mental illness.^16,37 Notably, a cross-sectional study revealed an independent correlation between coexisting chronic somatic disease and higher psychiatric readmission rates in patients with schizophrenia after excluding interfering factors,³⁸ reinforcing the argument that somatic comorbid diseases can significantly exacerbate the risk of UPR within 30 days in patients with schizophrenia. Severe somatic comorbidities may complicate treatment and increase the overall vulnerability of the individual.³⁹ Comorbid multiple somatic disorders may require a combination of multiple drugs, which can result in reduced patient tolerance to psychotropic drugs and an increased risk of a treatment interruption.⁴⁰ Therefore, the identification and management of comorbid conditions are crucial to controlling the disease status of schizophrenia patients, preventing readmissions, and improving their quality of life.

Multiple studies have explored the potential association between disease duration and readmission; however, they have not revealed a significant association.^41,42 Conversely, this study found that patients in the intermediate stage of the disease (10–15 years) were at higher risk of 30-day UPR. This finding may relate to attitudes and behavioral changes of the patients at this stage. A meta-analysis noted that the intervention in the early stage of the course of the disease was effective and that the improvement of symptoms after >10 years was relatively limited.⁴³ A prospective study indicated that the average number of patients hospitalized for psychiatric reasons was 2.3, compared with 1.4 for medical reasons over a study period of 12 to 27 years.⁴⁴ In the mid-stage of the disease, some patients may gradually lose confidence in treatment or grow tired of disease management. Therefore, for patients at this stage, hospitals, communities, and families must work together to provide them with comprehensive social support and mental health education to improve their treatment confidence and overall quality of life.

To increase bed turnover and decrease medical costs, medical insurance policies and regulations related to medical institutions tend to encourage minimizing the average hospital stay. However, this study found that patients with schizophrenia with a hospital stay of 8–29 days had an up to a 78.1% higher risk of 30-day UPR. This aligns with a prior study, which found that patients with schizophrenia hospitalized for fewer than 30 days had more frequent readmissions.⁴⁵ Furthermore, a 3-year cohort study observed that a shorter hospital stay was significantly associated with an increased 30-day readmission rate.⁴⁶ There is still a lack of uniform standards for the optimal length of hospitalization that can reduce medical costs without increasing the risk of readmission. Given the unique nature of schizophrenia, patients require more observation time, treatment plan adjustments, and stabilization before discharge to attain better clinical improvement. Consequently, medical staff should better determine the best time to discharge by carefully assessing the patient’s treatment response, disease stability, and post-discharge support measures, thus reducing the risk of UPR and facilitating long-term recovery.

This study revealed that patients with low medication adherence faced a higher risk of 30-day UPR, reinforcing the perspective of previous studies, which not only revealed a strong link between treatment non-adherence and the planned withdrawal of antipsychotics and high readmission rates in schizophrenia,^47,48 but also validated the utility of these predictors in real medical practice. A previous study predicted significant associations between drug non-adherence and treatment outcomes in schizophrenia, based on ML models that account for the complex interactions and mutual influence of multiple factors.⁴⁹ This shows that medical staff must regularly monitor patients’ conditions, observe the effect of drug treatment, and timely adjust treatment and rehabilitation programs when necessary. In the future, ML could be used to dynamically predict the risk of medication non-compliance in patients with schizophrenia,⁵⁰ which would have indispensable value in optimizing management and reducing UPR.

This study found that male had a higher risk of 30-day UPR, consistent with previous findings,¹⁹ and highlighting sex differences in the management of schizophrenia. The data clearly revealed higher prevalence of schizophrenia in male than in female, at a ratio close to 1.4:1.⁵¹ Male patients with schizophrenia often face more severe social distress, coupled with the fact that their treatment compliance is generally lower than that of female patients, aggravating the difficulty of establishing a treatment relationship and resulting in a high involuntary treatment rate, high hospitalization rate, and high suicide rate.⁵² Therefore, medical staff should more finely adjust and optimize the needs of male patients in order to reduce the risk of 30-day UPR and significantly improve both their quality of life and their rehabilitation prospects.

Limitations

This study had several limitations. First, the use of only retrospective EHR data for extracting basic clinical information led to the omission of other potential risk factors that are not documented, thus affecting the comprehensiveness of the prediction. Second, this study was confined to using data from a single tertiary hospital; future studies should use multi-center and multi-region data to enhance the generality and applicability of the results. Third, this study only underwent internal validation. Accordingly, additional external validation is required to establish the credibility of the results.

Conclusion

This study constructed 30-day UPR risk prediction models of patients with schizophrenia using five ML algorithms. After evaluation and comparison, the XGB model demonstrated significantly superior performance compared to other models, achieving an AUC value of 0.830, an accuracy rate of 0.7768, a precision rate of 0.6116, a recall rate of 0.7255, and an F1 score of 0.6637, indicating high robustness. The SHAP method further explained key risk factors, including number of somatic comorbid diseases, duration of the disease course, length of the latest hospital stay, drug withdrawal history, and sex, to support early identification of high-risk patients after discharge and optimization of transitional care plans. This study provides a scientific basis for leveraging ML model to improve the health outcomes and reduce healthcare costs for patients with schizophrenia, which can be further promoted and applied through multi-center external verification and dynamic risk assessment in the future.