In-Hospital Mortality Prediction Model for Critically Ill Older Adult Patients Transferred from the Emergency Department to the Intensive Care Unit

Introduction

Patients presenting to the emergency department (ED) are heterogeneous, and the severity of their clinical symptoms determines whether they need to receive further care.^1,2 If patients require a transfer from the ED to the intensive care unit (ICU), this indicates that they have life-threatening organ dysfunction or have a risk of death.³ Critically ill older adult patients who are admitted to the ED usually require emergency treatment and monitoring due to emergencies such as acute diseases, trauma, and cardiovascular and cerebrovascular events. Therefore, significant instability and severe physiological dysfunction may have already occurred at the time of admission.^4,5 As the population ages, the number of critically ill older adult patients seeking treatment in the ED is rapidly increasing.^6,7 Because older adult patients often have chronic diseases and organ dysfunction, they tend to have a poor prognosis.⁸ Therefore, the care of critically ill older adult patients is challenging. Establishing early prognosis prediction to identify high-risk critically ill older adult patients is conducive to maximum rational allocation of medical resources and early discussion of end-of-life issues with the patient’s family, if necessary.⁹ However, the ED is a high-need but resource-limited department,¹⁰ and comprehensive disease assessment and diagnosis have not yet been conducted. Therefore, early prognosis prediction after entering the ICU is crucial.

In past studies, many prognostic prediction models have been established for critically ill older adult patients. For example, mortality prediction models have been proposed for older adult patients with severe community acquired pneumonia,¹¹ older adult patients with ischemic stroke,¹² and older adult patients with hip fractures.¹³ However, there is still limited research on critically ill older adult patients transferred from the ED to the ICU.

In this study, we used data from the Medical Information Mart for Intensive Care IV (MIMIC-IV version 2.2) to build an in-hospital mortality prediction model for critically ill older adult patients transferred from the ED to the ICU, and validated this model with an external dataset from the Affiliated Dongyang Hospital of Wenzhou Medical University.

Materials and Methods

Training Cohort

The patient data for the training cohort was sourced from the MIMIC-IV version 2.2 database.¹⁴ The MIMIC-IV database contains de-identified clinical information related to patients admitted to the ICU or ED between 2008 and 2019. This database was reviewed and approved by the Institutional Review Committee of the Beth Israel Deaconess Medical Center. Since this was a retrospective collection of medical data, all patient data were de-identified, and the study did not guide patient diagnosis and treatment; the need for informed consent was waived. One author (Lu) carefully studied the relevant training courses and obtained permission for their use according to the database use agreement (license number: 35953547).

In this study, we included patients admitted from the ED and screened them based on the following exclusion criteria: (1) patients with > 30% missing data, (2) duplicate transfer data except for detailed information on the patient’s first transfer from the ED to the ICU, (3) patients aged ≤ 65 years old, and (4) patients transferred to the ICU more than 24 hours after admission.

External Validation Cohort

The external validation cohort included patients from the Affiliated Dongyang Hospital of Wenzhou Medical University from 2016 to 2023. The exclusion criteria were the same as those for the training cohort. This study was reviewed and approved by the Ethics Committee of the Affiliated Dongyang Hospital of Wenzhou Medical University. As this was a retrospective clinical data analysis, the necessity of informed consent was waived by the ethics committee.

Data Extraction

We used Postgre SQL software to obtain raw data from the MIMIC-IV database using a structured query language. The extracted data included patient demographics, comorbidities (malignant cancer and sepsis), medical treatment and laboratory parameters within the first 24 hours, Glasgow coma scale (GCS) score, patient survival status, and survival time. Medical treatment included mechanical ventilation and vasoactive agents. The laboratory parameters extracted were the first detection values within the first 24 hours, including white blood cells (WBC), hemoglobin, platelets, sodium, chloride, potassium, creatinine, and international normalized ratio. Variables with ≥ 25% missing values were excluded (ratio of missing data for excluded variables: albumin 74.1%, bilirubin 61.4%, lactate 58.7%, prothrombin time 25.2%, partial thromboplastin time 27.5%).

The extracted outcome variable was the all-cause in-hospital mortality of critically ill older adult patients.

Statistical Analysis

The Stata software (version 14.0; Stata Corp., College Station, TX, USA) and R software version 4.2.0 (http://www.r-project.org/) was used for statistical analysis and data visualization.

Continuous variables are expressed as mean ± standard deviation or median [interquartile range (IQR)] according to the normality of their distribution. Categorical variables are expressed as numbers (percentages). Differences in the distribution of continuous variables between the two cohorts were compared using the Student’s t-test or Wilcoxon test. Categorical variables were compared between the groups using the χ² test. When variables had missing values and the percentage of missing values was < 25%, the missing values were processed by multiple interpolation using the “mice” package in R software.¹⁵

The training cohort was used to build a prediction model. The outcome variable is binary, and the incidence of negative events is higher than that of positive events. When the incidence of negative events is five times higher than that of positive events, it is considered to have a significant class imbalance. Class imbalance leads to overfitting issues in the obtained classification model, and accuracy and robustness is also affected.¹⁶ Therefore, class imbalance was addressed using Random Over-Sampling Examples (ROSE). ROSE produces a sample of synthetic data by enlarging the features space of minority and majority class examples. The “ROSE” package in R language can generate synthetic data based on sampling and smooth bootstrap methods.¹⁷ Univariate and multivariate Cox regression analyses were used to screen candidate variables. The final statistically significant variables (P < 0.05) were considered independent predictors and were used to build the prediction models. A visualized nomogram model was built using the “rms” package in R software.

The external validation cohort was used to evaluate the performance of the predictive model. Calibration plots with resampling of 1000 bootstraps were plotted to assess the accuracy of the nomogram model. In addition, the receiver operating characteristic (ROC) curve, clinical utility decision curve analysis (DCA), and clinical impact curve (CIC) were plotted to demonstrate the clinical utility of the nomogram model.

Results

As shown in Figure 1, according to the screening criteria, a total of 10,233 critically ill older adult patients transferred from the ED to the ICU were selected from the MIMIC-IV database as the training cohort, and a total of 277 critically ill older adult patients admitted to Affiliated Dongyang Hospital of Wenzhou Medical University were selected as the external validation cohort.

Figure 1 Screening process of the study population. Abbreviation: ROSE, Random Over-Sampling Examples.

Construction of the Prediction Model

As shown in Supplemental Table 1, the in-hospital mortality rate of the patients in the original training cohort was 17.7%, with a significant class imbalance. Table 1 shows the characteristics of the ROSE-balanced training cohort, where class imbalance was eliminated and the numbers of survivors and non-survivors were 5160 and 5073, respectively. In the ROSE-balanced training cohort, univariate and multivariate Cox regression analyses were performed to identify independent risk factors. The results showed that age, sex, GCS score, malignant cancer, use of mechanical ventilation, use of vasoactive agents, WBC, potassium, and creatinine were independent predictors of in-hospital mortality in critically ill older adult patients (Table 2). According to the results of multivariate Cox regression analysis, we constructed a nomogram to predict the in-hospital mortality of critically-ill older adult patients who needed to be transferred from ED to ICU (Figure 2).

Table 1 Characteristics of the ROSE-Balanced Training Cohort Used for Constructing Prediction Models

Table 2 Univariate and Multivariate Analysis of in-Hospital Mortality of Critically Ill Older Adult Patients in ROSE-Balanced Training Cohort

Figure 2 A nomograph for predicting the in-hospital mortality of critically ill older adult patients who transferred from ED to ICU. When using, draw a vertical line up from each variable to the “point” axis, and then record the corresponding point. Then the total score is obtained by adding each variable corresponding to the in-hospital mortality predicted by the nomogram. Abbreviations: WBC, white blood cells; GCS, Glasgow coma scale.

Validation of the Prediction Model

In the ROSE-balanced training cohort, the nomogram showed good predictive performance (Figure 3). The AUC [95% confidence interval (95% CI)], sensitivity, specificity, positive predictive value, and negative predictive value of the training cohort were 0.792 [0.783–0.801], 0.803, 0.670, 0.705, and 0.775, respectively. The calibration curve shows the accuracy of the nomogram in predicting prognosis, and the good consistency of results demonstrate that there is no significant deviation between the model’s predicted value and the actual value. The DCA and CIC demonstrate that this model improves patient outcomes and has clinical application value.

Figure 3 Performance evaluation of the model in the ROSE-balanced training cohort. (A) Receiver operating characteristic curve to assess the ability of the nomogram to predict in-hospital mortality. (B) Calibration curves of the nomogram model. (C) Decision curve analysis of the nomogram model. The x-axis represents the threshold probability, which is the minimum predicted probability for clinical intervention. The y-axis represents net benefit. When the net benefit of nomogram is higher than that of treat-none and treat-all under the same threshold probability, it indicates that nomogram improves patient outcomes. (D) Clinical impact curve of the nomogram model. Using the nomogram model to predict risk stratification for 1000 individuals. The red curve (number of high risk) represents the number of people classified as positive (high risk) by the nomogram model at each threshold probability. The green curve (number of high risk with event) represents the number of people who are truly positive at each threshold probability.

In addition, the prediction model was validated using the external validation cohort. The predictive variables of patients in the external validation cohort are shown in Supplementary Table 2. The AUC [95% CI], sensitivity, specificity, positive predictive value, and negative predictive value of the external validation cohort were 0.780 [0.727–0.834]), 0.805, 0.639, 0.673, and 0.780, respectively (Figure 4A). In addition, the calibration curve (Figure 4B) shows that the model fits well. The DCA and CIC demonstrate that the predictive model has good clinical application value (Figure 4C and D).

Figure 4 Performance evaluation of the model in the validation cohort. (A) Receiver operating characteristic to assess the ability of nomogram to predict in-hospital mortality. (B) Calibration curves of the nomogram model. (C) Decision curve analysis of the nomogram model. (D) Clinical impact curve of the nomogram model.

Discussion

In this retrospective study, we developed a predictive model for in-hospital mortality of critically ill older adult patients who need to transfer from ED to ICU using a large publicly available MIMIC-IV database, and conducted external validation using patient data from the Affiliated Dongyang Hospital of Wenzhou Medical University. This study identified age, sex, GCS score, malignant cancer, use of mechanical ventilation, use of vasoactive agents, WBC, potassium, and creatinine as independent predictors of in-hospital mortality in critically ill older adult patients, which were then used to construct a nomogram model. Based on the performance evaluation, we found that the nomogram model has good predictive performance and clinical application value.

The ED is one of the main sources of critically ill patients in the ICU. The transfer of older adult patients from the ED to the ICU results in a marked dramatic decline in health status.¹⁸ Studies have found that the challenge of caring for most critically ill older adult patients is not just to prolong survival time, but to reduce unanticipated mortality.^19–21 In addition, with limited resources in the ICU, age may become an important discriminatory factor, and older adult patients are more likely to give up active treatment.^6,22 Therefore, different prognostic tools should be used for critically ill older adult and young patients. Moreover, early prognostic assessment of critically ill patients can help reduce the impact on subsequent medical decisions.²³

Most previous studies on the prognosis of patients admitted to the ED have included hospital mortality and the occurrence of critical illness as joint outcomes. Goodacre et al included age, diagnostic code, malignant tumor, vital signs, and laboratory test results in a prediction model for 7-day mortality in patients admitted to the ED.²⁴ Busch et al predicted the short- and long-term mortality of patients based on age, vital signs, mobility, and physician assessment.²⁵ However, these studies were conducted on unselected emergency patients, therefore the in-hospital mortality rate was < 10%, which may have affected the reliability of the results. Severe sample imbalance issues can lead to a decrease in the accuracy of the model, thereby affecting the reliability of the prediction model.^26,27 In addition, due to the limited resources and time of the ED, there may be execution pressure to select predictive factors obtained during the ED period (eg, detailed physical examination and adequate laboratory testing). The strength of this study is the development of a predictive model for in-hospital mortality in critically ill older adult patients requiring transfer to the ICU, based on a large sample from publicly available databases and an external validation dataset. In addition, prediction variables within the prediction model are easily available within the first 24 hours of entering the ICU. Furthermore, the apparent class imbalance in the data was handled using ROSE, which has been shown in many previous studies to improve the model.^28,29

Nevertheless, our study has some limitations. First, all patient data for the study were obtained from the medical center and the electronic health record system in a retrospective manner. Some variables that could not be included due to an excessive number of missing values, such as lactate and albumin, may be significant. Therefore, further validation is needed through prospective research. Secondly, older adult patients admitted to the emergency department are often delirious, which poses certain difficulties in obtaining baseline medical conditions before admission. Finally, the study considered only critically-ill older adult patients who died in the hospital after admission to the ICU, and did not consider those who were less critically ill and did not require ICU admission, or who died without ICU admission or immediately after discharge.

Conclusion

In this retrospective analysis of critically-ill older adult patients who needed to be transferred from the ED to the ICU, factors of age, sex, GCS score, malignant cancer, use of mechanical ventilation, use of vasoactive agents, WBC, potassium, and creatinine were considered independent risk factors for in-hospital mortality of critically-ill older adult patients and were used to construct a nomogram model. The nomogram model was shown to have good predictive performance using the external validation cohort.