Predicting Short-Term Mortality in Older Patients Discharged from Acute Hospitalizations Lasting Less Than 24 Hours

Introduction

The older population is increasing more quickly than any other age-group and by 2050 one in four persons in Europe and North America could be aged ≥65 years.¹ Currently, older patients already comprise ≥40% of acute emergency department (ED) contacts.^2–4 Thereby, an associated rise in the need for urgent care of older patients over the next few decades is to be expected and will require optimization and development of our current health-care system, eg, through improved preventive care along with fewer and possibly shorter hospitalizations. Shorter length of stay not only leads to lower hospitalization costs but is also associated with a lower risk of many adverse events, such as nosocomial infections, delirium, and complications related to immobility like deep vein thrombosis and loss of muscle mass.^5–8 The average length of stay in hospitals has decreased over the last decade, particularly for older age groups.^9,10 However, although shorter hospitalizations are in many ways to be preferred, concern has been raised that premature discharges among older patients could lead to preventable readmissions and even deaths.^2,9–13 Previous studies have identified several factors associated with short-term mortality following hospital discharge, such as age, frailty, comorbidity burden, number of medications, and laboratory results.^13–16 To our knowledge, no prediction model has been developed that focuses on the short-term mortality of older patients following early discharge. Valid prediction models may assist clinicians in safely planning early discharge for the right group of older patients. Therefore, the aim of this study was to develop a reliable prediction algorithm using random forest to predict the risk of 30-day mortality for acutely hospitalized older patients following early discharge from hospital (≤24 hours). Further, we examined how predictive performance changed with an increasing number of variables.

Methods

Design and Study Population

This was a registry-based cohort study. We included contacts where the patients were permanent residents ≥65 years of age who were discharged alive from acute hospitalizations lasting ≤24 hours between January 1, 2016 and December 1, 2018 in Denmark. To standardize the cohort across, different organizational hospital structures and mimic patients who could be handled in shared-entrance emergency departments, we included only hospitalizations with discharges from emergency, medical, or surgical departments. To consider the entire length of the patients’ hospital stay, we combined sequential Danish National Patient Registry (DNPR) contacts within ≤4 hours, considering contacts with both somatic and psychiatric departments and all types of inpatient/outpatient and acute/elective contacts.⁴ We only included contacts with at least one acute and one inpatient contact. Thereby, patients with a contact with the ED without outright hospitalization were not included. Hospitalizations from the Central Denmark Region were excluded, since we did not have access to their laboratory results. Hospitalizations from the remaining regions were excluded if no laboratory tests were registered during the hospitalization. Hospitalizations with registered laboratory results were excluded if they were still missing one or more laboratory test that was otherwise present in ≥90% of the study population. Hospitalizations where the patient was discharged from psychiatric, ear–nose–throat, dermatology, or ophthalmology departments were excluded, after an initial overview revealed a discrepancy in discharge diagnoses from these departments compared to the other medical, emergency, and surgical departments. Hospitalizations without information on the sociodemographic variables of patients were excluded. Hospitalizations where the patient had registered encounters for palliative care within 5 years prior to and including their index hospitalization were excluded. Finally, all but the first eligible hospitalization for each patient were excluded.

Setting

Denmark has a tax-funded universal health-care system that provides access for every citizen to primary, hospital, and home-care services.¹⁷ Citizens in need of acute hospital care are almost exclusively referred, either by their general practitioners or emergency medical services, and are generally first assessed in the ED, after which they are either discharged or hospitalized.¹⁷ Each citizen has a unique identification number (CPR number), which allows for the continuous tracking of Danish citizens over time as well as linkage among different national registers.^18,19

Data Sources and Predictor Variables

Information on the time and date of admission and discharge, the primary discharge diagnosis, and comorbidities at the time of hospitalization were available through the DNPR. Laboratory results from the index hospitalization were available through the Register of Laboratory Results for Research. Information on prescriptive medication was collected via the Danish National Prescription Registry. Sociodemographic and economic factors were collected via the DNPR, Central Person Register, Income Statistics Register, Building and Housing Register, and Danish Education Register from the end of the calendar year prior to the date of admission.

Diseases from the Charlson Comorbidity Index (CCI) were used individually, resulting in a total of 17 binary categorical comorbidity variables instead of a single CCI score. For descriptive purposes, the CCI score was calculated using the Quan modified CCI score.²⁰ Patients were considered as having a comorbidity if the relevant ICD-10 diagnostic code was registered as a primary or secondary diagnosis up to 5 years prior to their index hospitalization. Living situation was derived from combining the marital status of the patient and the number of people registered living at the same address as the patient. Education was assessed using the International Standard Classification of Education (ISCED) 2011. For descriptive purposes, disposable income was compared to quintiles of the general population stratified by age. Number of prescription drugs was examined using the Anatomical Therapeutic Chemical Classification System (ATC) level 4, collected up to 6 months prior to hospitalization. Since four-digit ICD-10 grouping has been shown to be an inadequate way of categorizing diseases, we conducted our own clinically meaningful grouping of discharge diagnoses for our population (Supplementary Materials; Table S1).^4,21,22 We selected 14 laboratory blood tests to represent a standard ED package, and reference values were drawn directly from the respective laboratories. Initial screening of the frequency of these tests was done: tests missing for >10% of the population were handled as categorical variables with up to four levels: “missing”, “below normal”, “normal”, and “above normal.” Otherwise, the tests were handled as continuous. Continuous laboratory tests were hemoglobin, thrombocytes, leukocytes, C-reactive protein, creatinine, eGFR, sodium ion, and potassium ion. INR, albumin, bilirubin, alanine transaminase, alkaline phosphatase, and glucose were categorical variables. For hospitalizations with repeated laboratory results, the last result prior to discharge was chosen.

Statistics

Data preparation and descriptive analyses were performed using SAS 9.4 (SAS Institute, Cary, NC, US) while R version 4.1.3 (R Core Team, Vienna, Austria), was used for model development and visualizations.^23,24 For descriptive purposes, continuous variables are displayed as medians and quartiles (Q1–Q3), while categorical variables are presented as percentages. Risk differences between proportions are reported using exact differences between proportions with 95% CIs. For continuous variables, the Hodges–Lehmann estimator of location of shift was used to display the median of differences and appertaining 95% CIs. Due to the size of the cohort and complexity of the calculation, this was done on 50,000 patients randomly selected from the cohort.

Random forest was chosen as the classification algorithm.^25,26 Initially, each patient was randomly assigned to either the training (70%), validation (15%), or test dataset (15%) while ensuring an equal distribution of survivors and nonsurvivors between the datasets. The dataset suffered from class imbalance, a common problem when utilizing random forest for classification purposes.²⁷ Therefore, we utilized the random forest quantile classifier during model training and hyperparameter tuning on the training set.²⁸ Hyperparameter optimization was explored in a stepwise fashion through narrowing grid searches for each model level until no further improvement could be achieved based on out-of-bag Brier score. Hyperparameter-optimization was performed on number of trees, tree depth, node size, and number of variables available to choose from at each split. Final testing for each model was done on the assigned test set with no weighing or sampling of survivors/nonsurvivors. The validation set was set aside as a secondary dataset on which to retest the model in the case that the results from testing on the test set showed the need for further model alterations. A cutoff point was selected for each model using the Youden index, and discriminative model performance was measured by ROC (receiver-operating characteristic) curves, AUROCs (area under receiver-operating characteristic) curves, Brier scores, sensitivity, and specificity on the test set. To gain insight into the model’s decision-making process, we utilized variable importance and partial dependence plots.

Variables were selected based on availability at discharge through electronic health records and expected correlation with mortality, as suggested by previous studies.^13–16 To investigate how model performance changed with additional information, variables were gradually added groupwise in an estimated order of most to least easily available to the physician. The order in which variables were added was: level 0 (age, sex), level 1 (level 0 + discharge diagnosis, specialty), level 2 (level 1 + time of day at discharge, day of the week at time of discharge, length of stay), level 3 (level 2 + laboratory results from the index hospitalization, previous hospitalization during the last 30 days prior to index hospitalization), level 4 (level 3 + diseases of Charlson Comorbidity Index, number of different prescription drugs redeemed within 6 months prior to the index hospitalization), and level 5 (level 4 + living situation, marital status, education, yearly disposable income). Finally, we developed a one-step backward-selection model using only variables from the level 5 model with positive importance.

Results

Study Population Baseline Characteristics

We considered 253,534 hospitalizations, of which 108,322 were excluded due to either missing laboratory results from the index hospitalization (including all hospitalizations from Central Denmark Region), discharge from psychiatric, ear–nose–throat, dermatology, or ophthalmology departments, lacking information on socioeconomic variables, or encounters for palliative care registered within 5 years prior to/including index hospitalization. Finally, we excluded all but the first eligible hospitalization for each patient, and the final population was comprised of 107,132 patients. Of these, 3,575 (3.3%) died within 30 days of discharge (Figure 1).

Figure 1 Flowchart of cohort selection process, with inclusion and exclusion criteria.

Descriptive analyses revealed notable differences between nonsurvivors and survivors (Table 1). Nonsurvivors were older than survivors and used more prescription medications. Nonsurvivors were more likely to have been acutely admitted to hospital within 30 days prior to the index hospitalization and were more often single and living alone compared to survivors. Discharge diagnoses also differed between the two groups (Supplementary Materials; Table S1). For example, “Dehydration. Other disorders of fluid, electrolyte, and acid–base balance” was more common among the nonsurvivors, while “Encounters for medical observation for suspected diseases and conditions ruled out” was more common among the survivors. Nonsurvivors had a greater occurrence of all comorbidities than survivors, but notable differences between the two groups included dementia, congestive heart failure, and cancer (Supplementary Materials; Table S2). Notable laboratory tests include higher C-reactive protein and creatinine and lower hemoglobin, albumin, and eGFR for nonsurvivors than survivors (Table 2). Comparison between those included and excluded from the study revealed a slightly lower mortality rate in those excluded from the study compared to those included (Supplementary Materials; Table S3). The excluded group had a larger proportion of men and was also younger than those who were included. There was no difference in Charlson Comorbidity Index scores.

Table 1 Population characteristics of the total cohort and according to survival status

Table 2 Laboratory results from index hospitalization of the total cohort and according to survival status

Predictive Performance

The level 0, 1, and 2 models had AUROCs of 0.723–0.755, similar ROC curves, and Brier scores of 0.0311–0.0313 (Figure 2 and Table 3). The level 3 model achieved an AUROC of 0.835 and a Brier score of 0.0286. The level 4, 5, and backward-selection model (excluding 54 variables with no or negative importance) achieved AUROCs of 0.859–0.861, Brier scores of 0.0284–0.0286, and shared indistinguishable ROC curves. The specificities for the level 0, 1, and 2 models were 0.681–0.710, and their sensitivities were 0.669–0.680. The level 3 model achieved a specificity of 0.771 and a sensitivity of 0.737, while the level 4, 5, and backward-selection model had specificities of 0.726–0.755 and sensitivities of 0.803–0.836. Model performances on the validation set were similar to performances on the test set (Supplementary Materials; Table S4). For the level 0, 1, and 2 models, median risk predictions were 2.2% for survivors and 4.7%–5.3% for nonsurvivors (Supplementary Materials; Table S5). Median risk predictions from the level 3 model were 1.5% for survivors and 8.7% for nonsurvivors. For the level 4, 5, and backward-selection model, median predictions for survivors ranged between 1.3%–1.4% and 10.0%–10.7% for nonsurvivors. The models were well calibrated below predicted probabilities <25% - the range in which most of the predictions were located (Supplementary Materials; Figure S1).

Table 3 Model performance on test set: area under receiver-operating characteristic curve (AUROC), Brier score, specificity, and sensitivity for each model

Figure 2 Receiver-operating characteristics curves and appertaining area under curve for the seven models when evaluated on the test set. Abbreviations: AUROC, area under receiver-operating characteristic curve; BSM, backward- selection model.

Important Variables

Variable importance revealed that from the level 5 model, nine of the 20 most important features were laboratory results, three were related to acute health status, four were related to chronic health status, four were sociodemographic, and none were administrative (Figure 3; Supplementary Materials; Table S6). The partial dependence plots, which show the marginal effect of a variable on predicted outcome, revealed that a comorbidity of dementia resulted in higher mortality predictions, along with older age, higher levels of C-reactive protein, and previous acute admissions 30 days prior, to mention a few (Supplementary Materials, Figure S2).²⁶

Figure 3 Variable importance relative to the most important variable—age. Top 20 ranking variables. Abbreviation: LRTI, lower respiratory tract infections.

Discussion

Using nationally representative data from more than 100,000 patients, we developed random forest models to predict the risk of short-term mortality of older patients after early discharge from acute hospitalizations. Predictive ability improved especially with the addition of laboratory results from hospitalization and information on previous acute admissions up to 30 days prior (level 3), and again with the addition of comorbidities and number of prescription drugs (level 4). Model performance did not improve with the addition of sociodemographic variables, other than age and sex (level 5). The model with all variables (level 5) found the most important variables to be age, a comorbidity of dementia, number of different prescription drugs, C-reactive protein, and eGFR. The observed mortality rate was 3.3%.

Predictions for survivors and nonsurvivors became more distinct with the addition of variables, but when predicted risk of mortality increased, calibration also declined. However, considering that the vast majority of patients were predicted to have a 30-day mortality risk of ≤25%, overall model calibration must be considered satisfactory for the level 3, 4, 5, and backward-selection model. This difficulty in gauging short-term mortality in patients at higher risk of mortality is a known problem for statisticians and clinicians, and is yet to be solved.²⁹ Considering that patients who are discharged soon after acute admissions have been assessed and deemed fit for discharge by a clinician, it seems reasonable that we did not achieve near-perfect discriminative abilities and that median predicted risk of nonsurvivors was 10.5% (level 5 model), especially when considering the heterogeneity of the cohort and lack of clinical information from the hospitalization. The level 0, 2, and 3 models achieved higher specificity than sensitivity, while the level 1, 4, 5, and backward-selection models achieved higher sensitivity than specificity. The level 4 model achieved good discrimination (AUROC 0.860) and was comparable to other models that predict mortality in older patients, though we have not been able to find other models that focus on early discharge.^30–33 However, besides improving predictive performance, introducing comorbidities and number of prescription medications (level 4) makes the model more labor-intensive for the physician, as this information would often require the manual tallying of the patient’s comorbidities and medications. The level 5 model performed comparably to the level 4 model, but with slightly higher specificity and lower sensitivity. Considering the additional work required to attain the sociodemographic information and the minimal difference in performance between the two models, the level 4 model must be considered superior. The backward-selection model also performed comparably to the level 4 model while only requiring half as many variables, and could possibly be more easily implementable in a clinical setting.

Rather than applying imputation, we excluded patients with no laboratory results because of the assumption that a majority of these tests were missing not at random: in Denmark, most patients have a standardized laboratory package taken upon admission to hospital, and it can be assumed that hospitalizations without any tests were of a less severe character. This assumption is supported by the fact that those excluded from the study were younger and had a slightly lower mortality rate than those included. Many older patients have multiple comorbidities and underlying health conditions that can present in many different ways in an acute hospital setting.³⁴ Therefore, rather than only including patients with a specific diagnosis, we included patients from emergency, medical, and surgical departments with many different discharge diagnoses and comorbidities. This approach is concordant with the future treatment of acute patients in Denmark, in which all patients will be seen in the joint emergency department for initial diagnosis and treatment.³⁵ Random forest was chosen for model development as it has been shown to potentially achieve higher precision for large, complex data structures containing many variables with complicated interactions, such as ours, compared to other algorithms.^36–38 As computational capacity and data availability increases, the predictive ability of machine-learning models such as random forest will continue to improve, and these are likely to become more commonly used in the field of medical research.³⁹

A comorbidity of dementia was very high-ranking in terms of variable importance, and is—along with cognitive decline—a known predictor of poor outcomes in older patients.^30,40–42 This high ranking likely partly contributed to the correlation between dementia and age and possibly frailty, which analyses of variable importance do not account for.^43–46 We had no information on where patients were discharged to, which could hold significant value. However, in our population, 3.3% were registered as single, but living with at least one other person, which is concordant with the proportion of people above the age of 65 years living in nursing homes in Denmark.⁴⁷ Though patients were excluded if they were receiving palliative care, it is plausible that some were willingly discharged from hospital despite a poor prognosis, with the intent of dying at home. We did not exclude patients with a comorbidity of cancer, since they comprise a considerable part of older patients. However, cancer and metastatic disease were high-ranking in terms of variable importance, and short-term mortality for patients with a history of cancer and/or metastatic disease is known to differ from patients without.^30,48 Recent acute admission and number of prescription drugs were also important variables, and are both possible indicators of patient morbidity and disease burden. Previous studies found associations between number of medications and mortality, but we could not find studies that examined previous recent admissions as a prognostic factor of mortality.^14,31 A discharge diagnosis of pneumonia and a self-reported feeling of physical fatigue have also previously been shown to be independent prognostic factors of death in older patients.^13,33,49 Previous studies also found a correlation between mortality and routine laboratory tests from hospitalization, such as C-reactive protein, creatinine, sodium, hemoglobin, and glucose.^15,50–52 We have not been able to find studies that have examined eGFR, bilirubin, or alkaline phosphatase as prognostic factors of mortality, though these were all high-ranking variables in our model. Adding administrative information from the hospitalization did not improve predictive ability. Adding sociodemographic information (apart from age and sex) did not improve overall model performance, although marital status and living situation were high-ranking variables. This discrepancy between high variable importance and minimal improvement of model performance can possibly be partly attributed to variable interactions between marital status and living situation and other variables, such as age, though previous studies have shown marital status and living situation to be independent risk factors of mortality.^13,53,54 Although initial testing showed no significant difference in mortality between males and females, sex did have a high importance ranking and male sex was associated with slightly higher mortality predictions than female sex, which is supported by previous findings.^13,31,55 Our mortality rate is concordant with previous studies that reported 30-day mortality rates of 3%–5%, though none of these focused on early discharge.^13,50,56

Strengths and Limitations

This study had good statistical power with the inclusion of more than 100,000 patients through a 3-year period, covering almost the entirety of Denmark. Registry-based studies in Denmark are known to have almost complete follow-up.¹⁸ Furthermore, the models are trained on a highly heterogeneous cohort consisting of hospitalizations from a wide range of medical and surgical specialties. This, coupled with the fact that EDs in Denmark diagnose and treat patients from most medical and surgical specialties, makes the model applicable to most acute clinical settings as a useful tool for physicians prior to discharge.³⁵ We lacked possibly important factors, such as the patient’s vital signs from the index hospitalization (eg, blood pressure, pulse and respiratory rate, temperature, oxygen saturation, Glasgow Coma Scale), presenting symptoms, or information on frailty and functional impairment, which have previously been shown to be strongly correlated with short-term mortality.^16,32,40,57 Future models could include these parameters, possibly improving predictive performance.⁵⁸ Though we only included hospitalizations with a duration of ≤24 hours, we cannot account for randomness in the length of stay, eg, due to inconvenient timing of discharge (nights) or a delay in the discharge process caused by differences in daily workload and patient flow. The department from which patients were discharged is also liable to some randomness due to overcrowding or other administrative difficulties. We excluded hospitalizations from the Central Denmark Region since we did not have access to their laboratory results. However, a previous study of sociodemographic and health-related homogeneity in Denmark found minimal discrepancy among Danish regions, and the cohort should thereby be representative of the entirety of Denmark.⁵⁹ We excluded patients who were missing a specific laboratory test that was present in >90% of the study population, to avoid imputation. Consequently, the model is only applicable to patients with a laboratory package similar to the one in our model. Few of the 66 discharge diagnoses were found to be important in the level 5 model, while none of the broader diagnostic groups, eg, “Other diseases of the respiratory system”, had positive variable importance, indicating that this was an insufficient way of categorizing disease. Due to the sheer number and variety of diagnostic codes in our data, we could not include each patient’s specific discharge diagnosis in our model. Consequently, the new diagnostic groupings may have led to selection bias and affected performance accordingly. Another contributing factor could be misclassification of diagnoses in a busy hospital environment, but the validity of diagnostic registration in Danish medical departments has previously been examined and deemed satisfactory.⁶⁰

Implications

We have developed a model with good predictive ability that can detect patients at risk of short-term mortality following early discharge. The patients in the cohort were assessed by physicians and deemed well enough to be discharged early, and should thus be expected to have a good short-term prognosis. Despite this, the observed 30-day mortality rate was 3.3%. Thus, integration of the model into the Danish electronic health record systems would make this model a useful additional tool for physicians to aid in the safe planning of early discharge for the right group of older patients. Patients at high estimated risk could qualify for interventions such as continued hospitalization, postdischarge follow-up appointments at the hospital or at home, or discharge to rehabilitation facilities instead of their own homes. With limited hospital capacity and an increasing number of acute hospitalizations of older patients, physicians will need to carefully consider which patients will benefit from continued hospitalization. However, even with the best possible treatment, care, and follow-up, certain deaths will remain unavoidable, as it is ultimately impossible to change the natural life course of some patients. This study also shows that random forest prediction modeling is feasible and meaningful for a heterogeneous cohort of elderly ED patients. Before clinical testing and possible implementation, we recommend redeveloping the model with the inclusion of relevant clinical information, such as vital signs, frailty, symptoms, and where the patient is discharged to.

Conclusion

In this registry-based cohort study, we developed random forest models capable of predicting the risk of short-term mortality of older patients following hospitalizations ≤24 hours and examined how an increasing amount of information changed the predictive ability. Model performance improved substantially with the addition of laboratory results and information on previous acute hospitalization, and further with the addition of comorbidities and number of prescription drugs. The addition of sociodemographic variables, apart from age and sex, did not improve predictive ability. Important variables included age, a comorbidity of dementia, number of different prescription drugs, C-reactive protein, and eGFR. Trained on a large and highly heterogeneous dataset, the best model achieved good discriminatory ability (AUROC 0.860) and could be a useful tool for physicians prior to early discharge in most acute clinical settings.