Effective Analysis of Inpatient Satisfaction: The Random Forest Algorithm

Introduction

The economic development history of many industrialized and post-industrialized countries reflects the rapid transformation of urban and rural structures.¹ In 2000, patient satisfaction was first proposed by World Health Organization (WHO) as an important indicator to measure the quality of medical services in public hospitals.^2,3 Good medical experience not only improves the quality of life of patients but also reduces the disagreements between doctors and patients, which has a positive impact on the whole medical ecology. The patient satisfaction of inpatients can directly reflect the medical technology standards and medical service quality of public hospitals in China.⁴ Therefore, it is vital to improve the quality of medical service and the utilization efficiency of health resources by exploring the risk factors of inpatient satisfaction in Chinese cities.⁵ Many theoretical model studies have assessed the association between a large number of independent variables and patient satisfaction through the use of different statistical methods, including naive Bayesian algorithm, support vector machine, logistic regression, decision tree, and other machine learning models.^6–11 A study in 2019 examined the risk factors of patient satisfaction by using logistic regression analysis.¹² Baek et al used a generalized ordered logistic regression model instead of performing a binary logistic regression analysis, which may yield incorrect results with fewer factors of patient satisfaction.¹³ Liu indicated that the deficiency of ordinal logistic regression in the study of ordered variables is based on the difficulty of analysis in the field of treatment methods.¹⁴ Nusinovici et al found that the predictive power of logistic regression and the machine learning model was consistent in the study of risk prediction and clinical predictors of common chronic diseases.¹⁵ Belciug explored the application of gene expression data sets in cancer research by embedding a logistic regression into relevant mathematical algorithms.¹⁶ Khani selected stable genes with high classification using logistic regression.¹⁷ However, the traditional multiple logistic regression has problems of low evaluation efficiency, multicollinearity, and overfitting. As a linear classifier, logistic regression model requires all features to be linear. When it comes to nonlinear features, it requires a complex transformation, which makes modeling difficult and affects performance. It cannot analyze large sample and multiclassification variables well. Meanwhile, naive Bayes classifier is a classic data mining algorithm, which has been widely used.¹⁸

Yue explored and improved the congenital defects of the naive Bayes classifier.¹⁹ However, the random forest algorithm can identify the hidden and complex endogeneity of original data more effectively than a traditional model. Some studies have shown that the prediction performance of a random forest is always better than that of traditional models, such as super vector machines and logistic models in the economic field.^20,21 Based on its inherent advantages, the random forest model performs well in psychological research with topics such as customer loyalty and winning or losing probabilities.^22,23 Random forest has been widely used in medicine in the last 10 years. It has demonstrated excellent performance in a large number of medical experiments with its high precision, tendency to avoid overfitting, and other advantages. It has gradually become one of the most popular frontier research methods in biomedicine and bioinformatics and has good application prospects. A series of studies by Chinese scholars on disease risk gene identification, risk single-nucleotide polymorphism recognition, protein interaction networks, and other topics have been internationally recognized due to their use of random forest.^24–30 International scholars have also made great progress in gene, tumor, and medical impact research using random forest.^31–34 Machine learning is becoming increasingly valuable and relevant in the field of healthcare.³⁵ There are few studies that explore the impact of different variables on patient satisfaction and the most appropriate evaluation methods of patient satisfaction.

In this study, we introduce machine learning and use the random forest model with the most significant classification effect to predict the influencing factors of patient satisfaction more objectively. Compared with traditional evaluation methods, the random forest model not only has higher evaluation efficiency and accurate dimensionality reduction but also has better accuracy.^36,37 At the same time, the random forest algorithm avoids the need for large sample size and the poor feature selection typical for decision trees by utilizing a double random sampling of samples and variables. At present, many scholars are committed to exploring a better classification method to study patient satisfaction in order to improve and standardize the medical quality control system.³⁸

Therefore, this study attempts to construct an optimal prediction model of the influencing factors of inpatient satisfaction. The study is based on data of inpatient satisfaction of 16 provincial public hospitals in Zhejiang Province, a developed province on the east coast of China. The purpose of this study was to determine a scientific and reliable method of analysis regarding patient satisfaction, thus providing a reference for establishing a standard patient satisfaction evaluation system and improving the quality of medical services.

Patients and Methods

Ethics Statement

The survey content and scheme of this study were approved by Zhejiang Provincial Health Commission, which is the official health management organization in China. All 16 tertiary hospitals had been aware of the survey content and had agreed to participate in the project. The content of the survey does not involve any personal information, such as name, photo, or telephone number. All interviewed patients had given their consent before the start of the investigation and signed informed consent forms. The research process is consistent with the Helsinki Declaration of 2013.

Data Sources and Respondents

For this survey, we used information from 16 public hospitals in 4 categories in Zhejiang Province, including 7 general hospitals, 5 specialized hospitals, 2 traditional Chinese medicine hospitals, and 2 integrated Chinese and Western medicine hospitals. The survey was conducted from January to December 2017. Data of patients discharged in the previous month were collected every month. We collected 4320 questionnaires, and 3888 were completed in full, giving us an effective rate of 90%.

Patients met the inclusion criteria if they were aged 18–70 years, were receiving treatment in the designated hospitals and were hospitalized or had a nonphysical examination.

Method of Investigation

In this study, a questionnaire assessing patient satisfaction was used in a computer-assisted telephone survey. Based on the information of discharged patients, computer-assisted telephone interviews were conducted to ensure the authenticity and reliability of the data.³⁹

The question “How satisfied are you with this hospitalization?” was the outcome variable for inpatient satisfaction and was scored on a 5-point Likert scale: 1 = “very dissatisfied,” 2 = “dissatisfied,” 3 = “neutral,” 4 = “relatively satisfied,” and 5 = “very satisfied.” The responses of “very satisfied” and “relatively satisfied” were defined as “good” (with a value of 1); other responses were defined as “poor” (with a value of 0). For explanatory variables, the response of “very satisfied” and “relatively satisfied” were defined as “good” (with a value of 1); other responses were defined as “poor” (with a value of 0).

Methods of Analysis

Epidata 4.6.0 was used to input and organize the data of the 3888 valid questionnaires. R3.6.1 was used for the data analysis.

Sample Quality Control

An attempt was made to estimate the number of samples required for this study. The sample size estimation formula is as follows:

(1)

Here, n is the sample size, is the overall satisfaction rate, and is the desired level of precision. The overall satisfaction of our survey is 79.73% (n=3888). Meanwhile, we assume 95% confidence and 5% precision. Therefore, the sample size was: . The number of effective samples in this study is 3888, which is much higher than the minimum sample size 248.

Multiple Logistic Regression

First, the patients were divided into two groups according to their satisfaction scores, and the explanatory variables were analyzed using single-factor analysis. Univariate analysis was used because of the number of variables; univariate analysis can effectively eliminate some confounding factors and ensure the reliability of the results in the population study. Then, according to the results of the single factor analysis, the influencing factors with statistical differences (P < 0.05) were included in the multiple logistic regression analysis. The basic expression of the logistic regression model is as follows:⁴⁰

(2)

where P is the probability of good satisfaction with the overall evaluation of inpatient medical services, n is the total number of independent variables, is the logistic regression correlation coefficient of each independent variable, and is an independent variable influencing the satisfaction of inpatients. Here, ε is a random perturbation term.

The probability of good satisfaction with the overall evaluation of inpatient medical services is p (binomial dependent variable y = 1), and the probability of poor satisfaction is 1-p (binomial dependent variable y = 0). The probability prediction model of satisfaction is obtained as follows:

(3)

Random Forest Model Construction

The differences in the importance of influencing factors were further explored through discriminant analysis. First, we divided the dataset into two parts: training set comprising 70% of the data (data for building the model) and test set comprising 30% (data for identifying the best model) to test the effect of the model and test the generalization ability of the model. This process is completed by bootstrap random sampling, in which the missing data are supplemented by the random forest interpolation method.^41,42,43 We used the random forest classifier method to construct different training datasets and finally fit the multiple classifier combination model based on avoiding local maximum.⁴³ The final classification results were defined as follows:

(4)

where is the combined multi-classifier model, I (·) is the indicative function, is the input vector, y is the outcome variable, is the single decision tree model, and is the number of decision trees.

Second, the optimal combination of parameters is determined according to the mtry value when the model error rate is the lowest, and the ntree values range from 1 to 500. The mean square deviation of error rate with the number of decision trees was obtained by comparing and analyzing each outcome variable and comprehensive data in the dataset. The prediction error rate of random forest for out-of-bag samples is called out-of-bag error, and the optimal combination is when the out-of-bag error rate is the minimum^41,43 (Figure 1). The mean square error is defined as

(5)

Figure 1 Tree values and error distribution.

where ρ is the correlation coefficient of the population, and ð is the population standard deviation. It can be seen from the formula that the variance can be reduced by decreasing the ρ or increasing the k, and the variance can be reduced by making each decision tree more independent or increasing the number of decision trees in the forest.

Third, the Gini index method was used to measure the variable importance measurement of the adjusted random forest prediction model, and the satisfaction-related risk factors that significantly affected the satisfaction of inpatients were identified.⁴³ A decrease in average Gini value indicates a decrease in average impurity at the variable partition node of all trees. The higher the value, the greater the importance of the variable. The Gini index can be expressed as follows:⁴⁴

(6)

Here, k represents K categories, and represents the proportion of category K in node M.

Naive Bayesian Analysis

Naive Bayes is a simple multi-class classification algorithm based on the assumption of independence between eigenvalues. In other words, in the one-time transfer of training data, the conditional probability distribution of each feature of a given tag is calculated, and then the conditional probability distribution of a given observation tag is calculated using the Bayesian theorem and used for prediction.⁴⁵ The mathematical expressions are as follows:

(7)

Suppose the characteristics of each dimension B1, B2 Bn-1, and Bn are independent of each other. The prior probability () is calculated according to the training set data.

Identifying the Optimal Model

We attempted to determine the best prediction model through the mutual verification of the logistic regression model, the random forest model, and the naive Bayes model.

First, the prediction efficiency and accuracy of the prediction models were tested, and the accuracy was improved by adjusting the prediction model. Accuracy, classification error rate, precision, recall, and F1_score were the five indicators of score used to evaluate the classification performance. The formulas used were as follows:

(9)

Here, TP (true positive) means that the model correctly judges a positive sample as a positive sample, FP (false positive) means that the model wrongly judges a counter sample as a positive sample, TN (true negative) means that the model correctly judges a counter sample as a positive sample, and FN (false negative) means that the model misjudges a positive example as a counterexample.¹ Accuracy is the ratio of the correct number of predicted values to the total amount of data. Generally, the higher the accuracy of the model, the better its effect. Accuracy is the accuracy of prediction of 0, recall rate refers to the accuracy of true 0, and the harmonic average of accuracy and recall is the F1_ score. In the F1_ score calculation formula, precision is abbreviated as P, and recall is abbreviated as R. The F1_score value ranges from 0 to 1, with 1 indicating the best and 0 the worst.

The AUC is the probability that the positive prediction score is greater than the negative prediction score. In other words, it is the C-index (C statistic of Mann Whitney U-test). The receiver operating characteristic (ROC) curve was drawn according to the predicted model. A larger AUC area, which is defined by the ROC curve and the x-axis, means a better classification ability of the current model. The calculation formula is as follows:

(14)

where M is the number of positive samples, and N is the number of negative samples.

Finally, we analyzed the marginal impact of satisfaction risk factors on the overall satisfaction of inpatient medical services. Marginal influence refers to the influence of an independent variable on the dependent variable while ignoring all other independent variables. We analyzed the detailed relationship between the four most important independent variables and the outcome variables to make the results of this study more objective and complete. The patients were divided into two groups according to their satisfaction scores, and the explanatory variables were tested using single-factor analysis.

Results

Overall Description of the Analysis

The results showed that the overall satisfaction of hospitalized patients was 79.73%, and female patients reported higher overall satisfaction than male patients. The average age of hospitalized patients was 43 years. In terms of education, patients with undergraduate degrees accounted for the largest proportion (39.60%). The average annual family income level was low, showing a left-skewed distribution. Furthermore, the proportion of floating population (migrants who stay in a place without local household registration status) was close to half of the permanent population. In addition, patients’ satisfaction with medical services was low.

Single Factor Analysis Results

The results showed that most of the socio-demographic factors (such as sex, age, and education level) and patient satisfaction had no statistical differences (P > 0.05), which was significantly different from the factors influencing outpatient satisfaction.⁴⁶ There were significant differences in medical insurance and related medical service factors during hospitalization (P < 0.05) as demonstrated in Table 1.

Table 1 Results of Univariate Analysis of Risk Factors of Inpatient Satisfaction

Results of the Multivariate Analysis

The results of the multivariate logistic regression analysis of patient satisfaction and satisfaction risk factors were largely consistent with the overall trend of the random forest model. The results showed that information satisfaction, service satisfaction with medical staff, satisfaction of the right to know, and accuracy of hospital diagnosis were closely associated with the improvement of inpatient satisfaction (P < 0.05) as demonstrated in Table 2. In addition, there were differences in the individual types of medical insurance, and the results of social demographic factors were consistent with the results of the random forest analysis.

Table 2 Multi-Factor Logistic Regression Analysis Results That Affect Inpatient Satisfaction

The Most Appropriate Model of Inpatient Satisfaction

A comparison of the ROC results shows that the prediction effect of the random forest method is the most accurate. The maximum ROC area was 0.932, and the result of the logistic regression (area 0.882) was better than that of the naive Bayes (area 0.836), indicating that the random forest model is better than the logistic regression and naive Bayes models. The ROC curves corresponding to each algorithm are shown in Figure 2.

Figure 2 Receiver operating characteristic curves of random forest, logistic regression, and naive Bayes.

Table 3 shows that all models achieved good results, but the random forest model is better than the logistic regression model and the naive Bayes classification model in each evaluation index. The classification accuracy of random forest model, logistic regression model, and naive Bayesian model were 96.57%, 89.71%, and 83.46%, respectively. In addition, the F1_score was 91.42%, 70.00%, and 65.72%, respectively. The evaluation effect of the random forest model is the highest.

Table 3 Analysis and Comparison of Test Results of Random Forest Model, Logistic Regression Model, and Naive Bayes Classification Model

The random forest model has better practicability and flexibility, and the performance effect is the best in this study. It can not only make high-precision classification decisions but also calculate the importance of each variable.⁴⁷

Importance of Different Influencing Factors

We plotted the importance of risk factors for inpatient satisfaction through a random forest model. Figure 3 shows the top ten variables, with satisfaction with the right to know and timely nursing response in the top two and clarity of procedures in the 10th.

Figure 3 Importance ranking of risk factors of inpatient satisfaction in the random forest model.

Discussion and Conclusion

This study attempts to construct an optimal discriminant model to explore the risk factors related to inpatient satisfaction in public hospitals in megacities of China, and further explore the importance of factors influencing patients. The random forest model is superior to logistic regression and naive Bayes classification models in each evaluation index, with the highest accuracy of 96.57%. This study found that socio-demographic factors had no significant effects on patient satisfaction in urban areas of China. This may be because hospitalized patients are more concerned about the positive role of medical service in improving the quality of life in general, owing to potentially having diseases that are serious. The first five variables (satisfaction with the right to knowledge, timely nursing response, satisfaction of medical staff service, integrity of medical staff, and accuracy of hospital diagnosis) were significantly important. The treatment cost of inpatients in China ranks low globally, indicating that a series of medical reform measures, such as drug “zero plus” policies, diagnosis-related group (DRG) system, use of micro-tools, have significantly reduced the economic burden of patients with major diseases.⁴⁸ In particular, China’s unique social medical insurance provides universal insurance, which reduces treatment costs for critically ill inpatients; however, this is not related to outpatient satisfaction, as China’s social medical insurance does not yet cover outpatient medical expenses. The economic burden of hospitalized patients has been further reduced with the introduction of the Chinese version of DRG-grouping standards and the establishment of key clinical pathways.^49,50 However, the expectation of patient satisfaction may lead to the breakdown of hospitalization system, which can accelerate the adverse selection of patients in public hospitals. Therefore, to improve their motivation at work, medical workers should improve the scale economic benefits of public hospitals by establishing corresponding incentive mechanisms in the implementation of various management tools. Finally, by covering patients’ treatment costs, commercial insurance can indirectly foster positive interactions between doctors and patients, thereby benefitting public hospitals, patients, and insurance companies.

The evaluation and prediction of patient satisfaction is important for many reasons and our study contributes in the following ways. First, modern medical care has been transformed into a social, psychological, physiological, and medical model, from the traditional biological, medical model based on the perspective of patients.⁵¹ Few studies have explored the difference in the importance of influencing factors for inpatient satisfaction, and we did this with an in-depth analysis. Patients’ right to knowledge was ranked first among the factors influencing satisfaction. In this study, as a basic right of patients in the process of disease diagnosis and treatment, we explored patient’s right to knowledge. Relevant research has found that improving patients’ right to knowledge is an effective measure for the rehabilitation of hospitalized patients.⁵² Moreover, further emotional communication and medical information exchange can also significantly improve patient satisfaction by reducing medical disputes. This plays the most important role in effective communication between medical workers and patients in dealing with medical complaints.⁵³ Among the factors, the present role of medical workers needs to be further expanded indirectly. Few studies have explored the specific content of the patient’s right to knowledge, but those that do highlight the patients’ realization of their right to know. Therefore, it is effective to understand the actual needs of patients, improve communication skills, and increase professional training. This will increase the patient’s trust in doctors and nurses, improve treatment and nursing compliance, improve the treatment effect, and increase treatment satisfaction.

Second, the present study showed that the quality of life of patients was significantly improved with timely and effective medical services.^54,55 There is a significant association between nurses’ bedside manner and patient satisfaction, and treatment time can affect patients’ satisfaction with medical experience.^56–59 It is expected that more differentiated and humanized services will be implemented in the healthcare industry. Some exploratory measures have been gradually used in sample hospitals, such as setting up clear signs, simplifying the process of patient treatment, and using the Internet to make appointments, thus reducing patients’ waiting time. The principle of “patient first” is also helpful in improving the efficiency of day-to-day operations and the effectiveness of nursing work.

Third, the service quality of medical staff is facing higher challenges, and there is a continuous increase in patients’ awareness and healthcare demands. A small sample study conducted in Shanghai reported that the overall satisfaction of inpatients during hospitalization was most affected by nurses’ attitude, followed by doctors’ attitude, cost and environment, and patient sex, which is different from the results of this study.⁶⁰ This may be because the best quality medical resources are located in Shanghai, as it is the most developed city in China, and the sample size was small. This indirectly reflects that fact that China’s rapid urbanization has exacerbated the imbalance of medical resources. In addition, the proportion of patients with higher education in large cities is increasing. Due to the internal population flow in China, the main workforce is gradually becoming highly educated and moving to large cities. The psychological satisfaction of highly educated patients is more complex. Regarding the processes of diagnosis and treatment, few large sample studies have explored the actual state of satisfaction with patient privacy. Most of the research on patient privacy is concentrated in special diseases or specific departments.⁶¹ There is still a lack of psychological professionals in China because the discipline started later than in much of the rest of the world. Therefore, public hospitals should strengthen humanistic care for patients by training psychological experts. Public hospitals should improve the sense of belonging of medical and technical personnel by building brand awareness and keeping up with Mayo Hospital.^62,63

Fourth, the accuracy of hospital diagnosis ranked fifth in this study, as one of the key indicators to evaluate patients’ quality of life. This is because public hospitals have become the main places for the treatment of critically ill patients. In China, hospitals benefit from the standardized training of medical practitioners.⁶⁴ China’s unique political system is extremely strict in monitoring the quality of medical services in public hospitals, so people have no doubt about the diagnostic accuracy of public hospitals in China.^65,66 However, China’s health resources, especially doctors, are still scarce.^67,68 Public hospitals undertake a large number of clinical operations but also need to take into account the task of medical technology innovation. Therefore, there is still a gap between China and other developed countries in terms of providing adequate follow-up care for discharged patients.^69,70 China is actively exploring ways to overcome this issue by relaxing the restrictions of running medical services by social forces. It should be noted that the scarcity of human resources may indirectly breed corruption. Medical ethics and style are codes for the conduct of medical staff and the focus of hospital work. Good medical ethics can not only improve patients’ satisfaction but also enhance the understanding and trust between doctors and patients. Therefore, establishing an effective internal supervision mechanism needs more in-depth study.

Finally, although machine learning algorithms are widely used in the medical field, they provide a new research direction for studying patient satisfaction. Metabonomics and bioinformatics use stochastic forest models for disease classification and diagnosis. Many scholars also use machine learning models to screen and classify the risk of a single disease.^71–74 Compared with the traditional multi-factor analysis, the stochastic forest model can avoid the problem of multicollinearity and is stable; thus, it can separate and deeply analyze the influencing factors. This study attempts to fit the optimal prediction model of inpatient satisfaction from the perspective of patients. The random forest model has the highest accuracy and the best prediction ability. In recent years, many scholars are committed to the application of machine learning model in the daily work of the hospital.^75,76 It is based on large sample data in the study of patient satisfaction.^51,56 Therefore, we suggest using the random forest method to explore the influencing factors in future satisfaction-related research.

Limitations of the Study

There are a few limitations of this study. First, the data were obtained from a cross-sectional survey, so it may be different from the real situation due to measurement error. Second, some influencing factors were not included in this study because they could not be measured, such as traffic and other variables. In future research, we will further quantify the indicators that cannot be measured at present and continuously optimize the model to improve the prediction accuracy, based on the current optimal prediction model.