Building and Validation of an Acute Event Prediction Model for Severe Mental Disorders

Introduction

Severe mental disorders (SMDs) refer to serious mental illness symptoms that cause severe damage to the social adaptation function of patients. SMD patients cannot fully understand their own health and the objective reality or handle their own affairs of psychiatric disorders,¹ including schizophrenia, bipolar disorder and major depression disorder.^2,3 They usually use antipsychotic medication. Taking drugs is common, which however is associated with serious adverse consequences and leads to disability and premature death to a great extent.⁴ The clinical effect of this kind of individual patient depends on the accurate prediction of results.⁵ It is reported that the prevalence of Chinese SMD patients showed a trend of sustained growth. By 2018, the reported prevalence of SMD patients in China had amounted to 4.3 per thousand.⁶ The risk of acute events in SMDs is higher among the general population.⁷ The ability of SMD patients to identify or control damage often exerts a severe impact on society. Common psychiatric acute events include violence, suicide, self-injury, flight, etc.⁸

Clinical prediction models (CPMs) show good performance, with good diagnosis and calibration, and satisfactory validity and clinical utility.⁹ Based on clinical practice, individualization and automation, the diagnosis of a mental illness risk calculator has developed into the information automatic screening of electronic medical records to help identify individuals at risk of SMDs.¹⁰ However, the risk calculator is not the related data and online research of these patients in the case of acute events.¹¹ SMDs involve some kinds of several serious mental illness, the heterogeneity of their patients is high, and such studies are widely involved worldwide. Studies on its prediction model are as follows: the construction and application of an analytical predictive model of impulsive behavior in hospitalized schizophrenia;¹² the construction and evaluation of a suicide risk prediction model in patients with schizophrenia;¹³ and the suicide prediction of severe mental illness: development and validation of clinical prediction rule (OxMIS).¹⁴ However, most of the above studies are performed with a single or several influencing factors. To bridge this gap, this study intends to analyze the independent influencing factors of acute events in SMD and to construct a predictive model. A prediction model was built for early clinical mental health professionals to identify high-risk patients and take appropriate intervention measures according to their willingness, which maximized safety in the hospital.

Methods

Study Population

SMD patients in a tertiary hospital in Yangzhou were selected as investigation objects. The inclusion criteria are as follows: 1. Patients conformed to the diagnostic criteria of the international classification of diseases (the 10th edition revised; ICD-10);¹⁵ 2. Patients were diagnosed with schizophrenia, bipolar disorder and major depression disorder; 3. Patients offered informed consent, and patients or their family members volunteered to join the study. The exclusion criteria are as follows: 1. Patients were accompanied by mental retardation, dementia, substance abuse or organic brain disorder; 2. Patients were diagnosed with severe body diseases or complications; 3. Patients with language or cognitive impairment cannot complete scales; 4. Patients withdrew informed consent. According to the Logistic regression analysis of the sample size calculation method,¹⁶ 342 cases of this study were eventually included in the sample, given 10% invalid questionnaires. In accordance with the ratio of 7:3, modeling and validation groups contained 239 and 103 examples, respectively. The criteria for acute events during hospitalization are: violence during hospitalization, suicide and self-injury.

After a full explanation of the study, All participants joined voluntarily and signed an informed consent form before participation. The study protocol gained the approval of the Institutional Review Board (IRB) of the Yangzhou Wutai Mountain Hospital, Jiangsu Province (WTSLL2023001).

Definition of Variables

Questionnaire on General Information

With reference to relevant literature and through the discussion of the project design team, general data tables were completed. The following contents were included in tables: gender, age, nationality, occupation, cultural degree, marital status, living environment, medical payment method and religious belief.

Questionnaire on Medical Record Information

The questionnaire completed by investigators according to the medical records of patients included: times of hospitalization in the past year, total times of hospitalization, SMD type, always acute event type, age, course of disease, family history, the length of the environment, with or without escort, relationship with patients and escort patients taking antipsychotic drugs over the past year.

Brief Psychiatric Rating Scale

The Brief Psychiatric Rating Scale (BPRS)^17,18 is used to assess the severity of mental illness patients’ mental symptoms: two negative symptoms including anxiety and depression, and the lack of energy and three positive symptoms including thought disorder, activation and hostile suspicion. The scale utilizes the seven-score rating of 1 = asymptomatic and 7 = very severe, with a total of 18 ~ 126 points. Patients’ symptoms will be more severe when the score is higher. The scale is composed of 18 entries, and Cronbach’s coefficient is 0.787 ~ 0.97.

Hamilton Depression Scale^19,20

Easy to operate, the Hamilton Depression Scale (HAMD) consumes little time, and with good validity. It consists of 24 items. The reaction of 14 items can be divided into five grades from 0 (no symptoms) ~ 4 (extreme). According to the score of 0 ~ 4 points and response, 10 items are divided into three grades from 0 (no symptoms) ~ 2 (moderate). According to the score of 0 ~ 2 points and the possible range of 0 ~ 76 points, the severity of depression was positively related to the score of the scale.²¹ Patients’ symptoms will be more severe when the score is higher. It is generally thought that the total HAMD scores < 8, 8 ~ 20, 21 ~ 35 and > 35 are divided into normal state, underlying depression, depression and severe depression, respectively. Cronbach’s coefficient is 0.766.²²

Hamilton Anxiety Scale^23,24

The Hamilton Anxiety Scale (HAMA), an important tool used by a psychiatrist for the clinical diagnosis and disease degree of patients, can better reflect illness severity. It comprises 14 items. The reaction of patients is classified into five grades from 0 (no symptoms) ~ 4 (extreme). Each component has a score of 0 ~ 4 points within the possible range of 0 ~ 56 points. Higher scores show more severe symptoms. According to the evaluation, the total HAMA scores < 7, 7 ~ 14 and > 14 are divided into no anxiety, underlying anxiety and anxiety or obvious anxiety, respectively. Among patients, 21 of 29 were divided into serious or anxiety. Cronbach’s coefficient is 0.766.

Pittsburgh Sleep Quality Index²⁵

The Pittsburgh Sleep Quality Index (PSQI) covers 19 self-evaluations and five review items. A total of seven entries used for scoring are included: I. sleep quality, II. sleep time, III. sleep duration, IV. sleep efficiency, V. sleep disorders, VI. hypnotic drugs and VII. daytime dysfunction. Each dimension has a score of 0 ~ 3 points, and the total score is 0 ~ 21 points. The total score is the sum of seven components. The score > 7 points indicates a sleep obstacle. The higher the score is, the worse the sleep quality will be. Cronbach’s coefficient is 0.842.

Morisky Medication Adherence Scale-8²⁶

The Morisky Medication Adherence Scale-8 (MMAS-8) is divided into three dimensions and composed of eight items. Items 1 ~ 7 are given the options of “yes” and “no”, with a score of 0 and 1 point. Item 8 includes five-level scoring, namely, 1 = never, 0.75 = sometimes, 0.5 = occasionally, 0.25 = often and 0 = all the time. The scale has a score of 8 points and three levels: ˂ 6, 6 ~ 7 and 8 points for poor, moderate and good medication adherence, respectively. Cronbach’s coefficient is 0.835.²⁷

Social Support-Rating Scale²⁸

The Social Support-Rating Scale (SSRS) is divided into three dimensions and composed of 10 items. Items 1 ~ 4 and 8 ~ 10 choose the options of 1, 2, 3 and 4, respectively, with a score of 1, 2 and 4 points. Item 5 has the options of A, B, C and D. Each option has a score of 1 ~ 4 points from no to full support, respectively. Items 6 ~ 7 have the answers of “no source” for 0 and “the following sources”. Higher scores indicate feelings of higher social support.²⁹ Cronbach’s coefficient is 0.92.³⁰

Insight and Treatment Attitude Questionnaire³¹

Divided into two dimensions and composed of 11 items, the Insight and Treatment Attitude Questionnaire (ITAQ) uses the three-grade scoring method, namely 0, 1 and 2 = no, partial and complete self-knowledge, respectively. The total score is 22 points. The higher the score is, the more complete the self-knowledge will be. The score from the known source can be divided into three levels: 20 points or more, 6 ~ 19 and 5 points or less indicate complete, partial and no self-knowledge, respectively. Cronbach’s coefficient is 0.869.³²

Statistical Analysis

SPSS 25.0 statistical software was used for the statistical analysis of data. Statistics that obeyed normal distribution were expressed by x + s, and those that followed skewness distribution were expressed by the median (quartile). Frequency and composition ratio were used to describe classification variables. A t-test was used to compare the measurement data obeying normal distribution between groups. A rank sum test was used to compare the measurement data without obeying normal distribution between groups. Classification variables between groups were compared using the x2 test. Multivariable Logistic regression analysis was performed to pick out the independent risk factors for the suicidal ideation of patients. A risk prediction model was constructed with R4.1.2 software. A nomogram was drawn to visualize the Logistic regression model. The prediction ability of the receiver operating characteristic curve (ROC) evaluation model and the internal validation of the model were verified by the Bootstrap validation method. P < 0.05 or 0.01 for the difference was of statistical significance.

Results

General Information on Research Objects

Of 342 hospitalized patients with SMDs, 114 (26.5%) in patients reported acute events, including 87 and 27 (28.9% and 21.0%) in modeling and validation groups, respectively. The accident rate between the two groups showed no statistically significant difference. The general data of research objects are shown in Table 1.

Table 1 Patient Clinical Characteristics

Single Factor Analysis of Hospitalized SMD Patients Encountering Acute Events

The single factor analysis results showed that the incidence age, hospitalization time, drug kinds, MMAS-8, SSRS, ITAQ, PSQI, HAMD, disease diagnosis, hospital environment, occupation, marital status, caregivers and childhood bad situation of hospitalized patients with SMDs were statistically significant (P < 0.05). Details are presented in Table 2.

Table 2 Univariate Statistics of Patients

Multivariate Logistic Regression Analysis of the Influence Factors of hospitalized Patients with SMD Acute Incidents

Whether acute events occurred in hospitalized patients with SMDs was used as an independent variable. Values were assigned to multiple classification variables (Table 3). In single factor analysis, variables with P < 0.05 dependent variables for multivariable Logistic regression analysis of variables. Multi-factor analysis results showed that disease diagnosis, marital status, caregivers and hospital environment were independent risk factors for the accurate events of hospitalized patients with SMD. Among them, disease diagnosis and marital status were protective factors. Details are shown in Table 4.

Table 3 Variables Assignment Table of Logistic Stepwise Regression Analysis

Table 4 Multivariate Logistic Regression Analysis of Patients

Building the Nomogram of the Risk Prediction Model for Patients with SMDs

Based on the Logistic regression model: Z = 1.80–0.16 * MMAS-8 −0.90 * disease diagnosis −1.28 * marital status +1.73 * caregivers +5.80 * SSRS +8.43 * the hospital environment, R software was used to build and visualize a nomogram, as shown in Figure 1.

Figure 1 The nomogram for predicting acute event risk of patients. Based on the Logistic regression model: Z = 1.80–0.16 *MMAS-8 −0.90 *Disease diagnosis −1.28 *Marital status +1.73 * caregivers +5.80 *SSRS +8.43 *The hospital environment, R software was used to build and visualize a nomogram.

The specific condition of each risk factor in the nomogram corresponds to corresponding points. The total score was obtained by adding the scores of five indicators to the model. A vertical line was drawn where the total score was obtained, The value corresponding to the intersection position of the vertical line and the “occurrence probability of acute events” was the risk of acute events in hospitalized SMD patients.

Validation of the Risk Prediction Model of Hospitalized SMD Patients Encountering Acute Events

Distinguishing Ability

Prediction variables were used as a test, and hospitalized SMD patients encountering acute events were taken as state variables to draw the ROC curve. The modeling prediction model set of the AUC value is 0.77. When the best cutoff value was taken, the Yoden index was 0.62, and sensitivity and specificity were 0.90 and 0.72, respectively. The AUC value of the prediction model in the validation group was 0.80. When the best cutoff value index was 0.57, sensitivity and specificity were 0.81 and 0.76, respectively. That prediction model showed better performance, as shown in Figure 2.

Figure 2 ROC curve of the hospitalized SMD patients encountering acute events. Prediction variables were used as a test, and hospitalized SMD patients encountering acute events were taken as state variables to draw the ROC curve. The modeling prediction model set of the AUC value is 0.77. When the best cutoff value was taken, the Yoden index was 0.62, and sensitivity and specificity were 0.90 and 0.72, respectively. The AUC value of the prediction model in the validation group was 0.80. When the best cutoff value index was 0.57, sensitivity and specificity were 0.81 and 0.76, respectively.

Calibration Ability

The calibration ability of the risk prediction model for the risk of acute events in hospitalized patients with SMDs was evaluated using the Hosmer-Lemeshow (H-L) goodness-of-fit test. The result showed x² = 8.08, p = 0.43. In the prediction of hospitalized SMD patients encountering acute events, the predicted and actual occurrence probabilities of the model showed no statistical difference, which indicated its better calibration ability, as shown in Figure 3.

Figure 3 The calibration curves of the nomogram for the risk of acute event. The calibration ability of the risk prediction model for the risk of acute events in hospitalized patients with SMDs was evaluated using the Hosmer-Lemeshow (H-L) goodness-of-fit test. The result showed x2 = 8.08, p = 0.43. In the prediction of hospitalized SMD patients encountering acute events, the predicted and actual occurrence probabilities of the model showed no statistical difference.

Clinical Decision Curve Analysis

Clinical decision curve analysis (DCA) is used to evaluate the clinical efficacy of the risk prediction model for the acute events of hospitalized patients with SMDs. It is also used to determine the use of the clinical prediction model to inform whether clinical decisions do more harm than good. It can be seen from Figure 4 that higher clinical utility can be obtained when the probability is 0.01 ~ 0.90.

Figure 4 Decision curve analysis (DCA) for detection of patients. DCA is used to evaluate the clinical efficacy of the risk prediction model for the acute events of hospitalized patients with SMDs. It is also used to determine the use of the clinical prediction model to inform whether clinical decisions do more harm than good. The higher clinical utility can be obtained when the probability is 0.01 ~ 0.90.

Discussion

Present Situation Analysis of Hospitalized SMD Patients Encountering Acute Events

The survey of 342 hospitalized patients with SMDs showed that the incidence of acute events was 18.70%. Wang and Chen^33–37 claimed that “differences exist between the results”. The possible reasons are as follows: 1. Differences were found between research objects, which were related to the different inclusion and exclusion criteria of different studies. The research objects of this study were hospitalized patients diagnosed with SMDs. 2. Differences were found in assessment scales. 3. Events mismatched the standard evaluation, and the evaluation standard of the outcome variable in this research was: accurate events occurred in patients with SMDs during hospitalization.

Risk Factors for Hospitalized SMD Patients Encountering Acute Events

The acute events of patients are a social, psychological and environmental outcome of the combined action of multiple factors.³⁸ Getting familiar with the management of these risk factors of hospitalized patients with SMD acute incidents has important significance for forecasting early warning and intervention.

Social Demographic Factors

In this study, it was found that marital status, caregivers, the hospital environment and social support were independent influence factors for the acute events of SMD patients. Medication adherence and marital status were protective factors. Married, parents’ care, low social support score and living in a closed room were lower risks of acute events in SMD patients. Research by Liu, Zheng et al^39–41 proved the viewpoint of the first three factors. The reasons may be as follows: 1. Families and society may not change various aspects of requirements for married patients in the disease state, which causes psychological stress to be larger than normal. This in turn leads to more moods in patients and the increased occurrence probability of acute events. 2. Parents play an irreplaceable role in the childhood of children. Parents’ care and company as a spiritual pillar can make patients experience a sense of security, alleviate their negative moods and increase the enthusiasm of patients about treatment to maintain the stability of the disease. 3. Items in all closed-end rooms were provided for SMD patients by the hospital during hospitalization. These closed-end rooms were wider in range than semi-enclosed wards, with more stringent schedules. In addition, 24-hour monitoring was performed in the disease’s acute stage. Therefore, the occurrence rate of acute events in totally enclosed wards was lower than that in semi-closed ones. In this study, however, the idea of social support was different from those of Zeng et al.⁴² The reasons may be as follows: 1. The research objects in the hospital environment were arranged in fully and semi-closed wards. The vast majority of patients were in fully enclosed wards without family members during hospitalization. 2. Patients with SMDs were subjected to small psychological pressure due to the lack of good social support, which thus led to the decreasing risk of acute events.

Disease Diagnosis

In this research, it was shown that disease diagnosis was not only an independent influence factor but also a protective factor for the acute events of SMDs, which was similar to the results of V JM, Hj et al^43,44 The reasons are as follows: 1. The symptoms of schizophrenia are one of the most serious SMDs. 2. Schizophrenia can give rise to abnormal behavior and thought disorders in patients. Thus, these patients are prone to various types of acute events. The death rate of schizophrenia is higher than that of other types of SMD. 3. Bipolar disorder is a chronic recurrent SMD.⁴⁵ Thus, the incidence of acute events is lower than those of other types of SMD.

Medication Adherence

In this study, it was proved that the medication compliance of SMD patients, an independent factor of acute events, was similar to the results of Stentzel and Chen et al^46,47 The possible reasons are as follows 1. Medication compliance affects such patients in stable condition. Patients in poor condition are more prone to acute events when the fluctuation in condition is more frequent. 2. The increase in family care burden and economic pressure exerts enormous pressure on patients and family spirit. As a result, patients are prone to urgent events.

Ideal prediction effect of the risk prediction model of SMDs in patients with acute events.

In this research, R software was used to construct a Logistic regression model and draw a diagram. The sensitivity and specificity of modeling and validation groups were high. The AUC values of both groups were greater than 0.75. This shows that the model has an ideal prediction effect and is beneficial for mental health professionals to screen the acute events of high-risk SMD crowds during hospitalization. Meanwhile, calibration and clinical decision curves were used to evaluate the calibration degree and clinical effectiveness of the model. The result shows that the model can be used for clinical diagnosis and treatment activities, with good clinical application value.

Conclusions

To sum up, this study focused on exploring the characteristics of patients with acute events in low- and high-risk SMDs. The results showed that the risk of acute events occurring in patients who were married, with good medication adherence, SMDs diagnosed with bipolar disorder, parents’ care, a lower score of social support and living in totally closed wards during hospitalization in the Psychiatry Department was low. The forecast model of acute events can be applied to the screening and evaluation of potential SMD patients. The acute event prediction model can be used to screen and evaluate SMD patients who encounter acute events. In addition, individualized intervening measures were taken according to different groups. This can more effectively reduce the occurrence and severity of acute events in SMD patients hospitalized in psychiatric medical units, and improve medical efficiency.

The research objects include only patients of this hospital. In the future, researchers should face some problems in clinical research in the field of psychiatry, pay attention to reasonable and strict experimental design, and provide the basis for optimal clinical decisions. In the meantime, multicenter and multi-zone large sample studies can be conducted to constantly adjust to improve accuracy and be more in line with clinical practice for wide application in mental hospital wards.