Validation and Comparison of Four Mortality Prediction Models in a Geriatric Ward in China

Introduction

Population aging is a source of concern to health professionals and policymakers worldwide.¹ The disease-oriented approach to healthcare has proven insufficient to meet the healthcare and social needs of a fast-growing cohort of older adults.^2,3 Chronic diseases are not curable, and geriatrics aims to maintain functional capacity, enhance the quality of life, and extend the healthy life expectancy of older adults. Ignoring prognosis may contribute to overuse and low-value care.⁴ It is often difficult to predict life expectancy for older adults only on empirical clinical judgment, muddling the differentiation of those who could benefit from disease screening and medical interventions from those who could benefit from hospice and palliative care.⁵ The benefits-harms balance of clinical interventions has important clinical implications, especially in older adults. Mortality prediction models can help estimate individual prognosis and guide clinical decision-making, enabling physicians to tailor treatment according to the patient’s condition, preferences, and prognosis.

Current mortality prediction models for the older adults include 6-month, 1-year, and more than 1-year mortality prediction models.^6–13 The variables included in these models consist of diseases only, functional status only, diseases combined with functional status, and geriatric syndrome.^6–13 Some of these models have been validated in European and American countries, but there is a lack of studies to confirm their validity for the older adults in China. Based on the data available in this study, the Walter Index, GPI, CCI, and FRAIL Scale models were selected for validation and comparison. The Walter Index prediction variables include diseases and physical function, and is recommended as a 1-year mortality prediction model for older inpatients by the University of San Francisco.⁶ The GPI is the only 3- and 5-year mortality prediction model developed with Korean patients, which centers on comorbidities and the comprehensive geriatric assessment (CGA).¹³ The Charlson Comorbidity Index (CCI) is a 1-year mortality prediction model for older patients with multimorbidity, and high scores predict a significantly increased risk of death.¹⁴ Frailty is a very important geriatric syndrome that significantly increases mortality risk.^15,16 The FRAIL Scale is a simple tool to assess frailty, comprising only five questions that can be assessed in 5 mins.^17,18 The Walter Index and GPI have not been validated in older adults in China, and the CCI and FRAIL Scale have not been compared with the Walter Index and GPI in China; therefore, the purpose of this study was to validate and compare the performance of the Walter Index, GPI, CCI, and FRAIL Scale in predicting 1-year mortality, providing evidence from geriatric inpatients in China.

Materials and Methods

Setting and Patients

This study was conducted in a geriatric assessment and management ward in the Peking Union Medical College, Chinese Academy of Medical Sciences, Peking Union Medical College Hospital, a tertiary care hospital in Beijing. The geriatric ward is a sub-acute management unit staffed by geriatricians, specialists, psychologists, nutritionists, rehabilitation physicians, pharmacists, nurses, and social workers, dedicated to maintaining the functional status of older patients while managing their diseases. We reviewed the records of patients discharged between January 1, 2016, and December 31, 2021. Patients aged 70 years or older with documented CGA were included; patients with a hospital stay ≤24 h or >60 days were excluded. If the patients were hospitalized repeatedly, the first hospitalization was recorded. The data of all these patients were recorded, including demographics (age, sex, marital status, and cigarette use), CGA, length of stay, discharge diagnosis, and clinical laboratory test results (albumin, hemoglobin, and creatinine), to calculate the scores of the four models.

Comprehensive Geriatric Assessment

The CGA of all patients was performed by a trained physician when the disease condition was relatively stable after hospital admission. The functional status assessment was based on the activities of daily living (ADL) and instrumental activities of daily living (IADL). The Physical Self-Maintenance Scale (PSMS) was used to assess ADL by measuring the degree of dependence/independence in daily personal care activities (feeding, toileting, grooming, bathing, and physical ambulation).¹⁹ The Lawton-IADL was used to evaluate IADL (independence for eight activities: ability to use telephone, food preparation, shopping, laundry, housekeeping, responsibility for own medication, mode of transportation, and capacity to handle finances).¹⁹ The Mini-Nutritional Assessment-Short Form (MNA-SF) was used to assess nutritional status and was comprised of mobility, declining food intake, weight loss, psychological stress or acute illness, neuropsychological illness, and body mass index or calf circumference (cm).²⁰ The Mini-Mental State Examination (MMSE) was used to assess time and place orientation, registration, attention, recall, and language.²¹ Mood was evaluated by the Geriatric Depression Scale (GDS-15).²² Muscle strength was evaluated by handgrip strength, with low muscle strength identified as handgrip strength <28 kg for males or <18 kg for females.²³ A Short Physical Performance Battery (SPPB) score ≤9 was considered low physical performance.^23,24 The number of medications (including prescribed medications) was recorded, and ≥ 5 was defined as polypharmacy. The social support network was classified as living alone, living with family, or institutionalized.

Mortality Prediction Models

The Walter Index includes five variables: sex, ADL (a modified version of Katz-ADL) dependence, comorbidities (congestive heart failure, cancer, metastatic cancer), albumin, and creatinine, with a total score of 20.^6,25 The GPI includes eight variables: age, sex, Korean ADL, Korean IADL, Mini-Nutritional Assessment (MNA), MMSE, GDS-15, and CCI, with a total score of 8.^{13,21,22,26–28} The CCI includes 19 diseases; each disease is assigned a different score of 1, 2, 3, or 6.²⁸ The FRAIL Scale consists of 5 variables: fatigue, resistance, ambulation, illness, and weight loss; a score of 0 is strong, 1–2 is pre-frail, and 3–5 is frail.^17,18 Model details of the original studies assessing the Walter Index, CCI, and GPI are summarized in Table S1. The characteristics of the different ADL and IADL scales are shown in Table S2.

Outcome Measures

The primary outcome was all-cause mortality within one year of discharge, which was defined as death from any cause. Data was collected from medical files and telephone interviews conducted between August 2022 and February 2023.

Statistical Analysis

Baseline patient characteristics are presented as medians (interquartile range [IQR]) or numbers (proportion). We report the number (proportion) of participants with missing data for each variable. To maximize statistical power and minimize potential bias by excluding patients with missing data from the analysis, we created 20 imputation datasets using multiple imputations based on the Markov chain Monte Carlo method in the SPSS program (version 26.0).²⁹ To ascertain whether post-imputation data differed significantly from pre-imputation data, we utilized sensitivity comparative analysis. Pre-imputation data were used to report the results of our analyses; the results from the 20 imputation datasets are reported in the Supplementary Materials.

Logistic regression was used to assess the association between the four models and the 1-year risk of all-cause mortality. We adjusted for common covariates as categorical variables, which were not covered in any of the four models, including marital status (married, not married), cigarette use (Never, Former, Current), social support (living alone, living with family, institutionalized), length of stay (≤7d, >7d), number of medications (<5, ≥5), hemoglobin level (≥110 g/L, < 110 g/L), handgrip strength (normal, low), and SPPB (>9, ≤9). To avoid the bias of missing data, multivariate logistic regression results based on the imputation data were combined according to Rubin’s rule.³⁰

We used the calculated model scores as the only independent variable to assess the performance of the models, including the: (1) Brier Score, an overall performance measure, ranging from 0 (excellent overall performance) to 0.25 (indicating a non-informative model)^31,32 and (2) C-statistic, corresponding to the area under the receiver operating characteristic (ROC) curve of the binary dependent variable, which served as the model discrimination measure. The C-statistic shows how well the prediction model differentiates those at higher and lower mortality risk and ranges from 0.5 to 0.59 for poor, 0.6 to 0.69 for moderate, 0.7 to 0.79 for good, 0.8 to 0.89 for very good, and >0.9 for excellent.³³ We performed pairwise comparisons of the ROC curves for each model with the DeLong method.³⁴ (3) The Hosmer-Lemeshow goodness-of-fit-test (HL test) and calibration plots were used to assess the calibration of the four models. The HL test distinguishes between a null hypothesis of an excellent fit, where the model assumes probabilities that are consistent with the actual probabilities, and a generalized alternative hypothesis of a non-perfect fit.³⁵ Calibration curves were plotted based on observed and predicted mortality, using 1000 bootstrap resamples to reduce overfit bias.

Analysis was performed with SPSS, version 26.0 (released 2019, IBM SPSS Statistics for Windows, Armonk, NY), R project, version 4.2.3 (R-project of the statistical calculation, Vienna, Austria) and EmpowerStats software, version 4.1 (www.empowerstats.com, X&Y solutions, Boston, MA, USA). All statistical tests were two-sided, and P < 0.05 was considered statistically significant.

Results

Baseline Characteristics of the Study Population

Of 1275 possible participants, 369 patients with recurrent admissions or that did not complete the CGA and two patients with a length of stay >60 d were excluded from this study. Of the remaining 904 patients, 24 (2.7%) patients’ families refused information on the patients’ condition and 48 (5.3%) patients were lost to follow-up. The study sample included 832 patients, with a median age of 77.0 (74.0–78.0) years (interquartile range, IQR) and 380 men (45.7%). The median (IQR) scores were 2.0 (1.0–4.0), 2.5 (1.5–4.0), 1.0 (0.0–2.0), and 1.0 (0.0–3.0) for the Walter Index, GPI, CCI, and FRAIL Scale, respectively. Within one year, 100 participants (12.0%) died. The baseline characteristics of the participants are shown in Table 1.

Table 1 Baseline Characteristics of Participants

Of the 832 participants, 194 (23.3%) had missing data and two (0.2%) from the Walter Index, 101 (12.1%) from GPI, and 13 (1.6%) from the FRAIL Scale. There were no missing data for CCI (Table 1). Sensitivity comparative analysis results indicated no significant differences between pre- and post-imputation data (Table S3).

Prediction of 1-Year of Mortality Risk

The results of the logistic regression analysis are presented in Table 2. With univariate logistic regression and adjustment for common covariates (length of hospitalization, marital status, social support, number of medications, and hemoglobin levels), the 1- year mortality risk increased with increasing scores in all four models. After adjusting for handgrip strength and SPPB (except for CCI 1–2 scores that were no longer associated with 1-year mortality), and CCI 3–4 and >4 scores, increased scores of the Walter Index, FRAIL Scale, and GPI showed significant associations with 1-year mortality. The results of logistic regression analyses based on the imputed datasets, combined with Rubin’s rules, are shown in Table S4, and indicated no significant differences between pre- and post-imputation.

Table 2 Logistic Regression of Associations Between the 4 Models and 1-Year Mortality Risk

Performance of the Mortality Prognostic Models

According to the Brier scores, the Walter Index (0.07), GPI (0.08), CCI (0.08), and FRAIL Scale (1.00) performed well, with the Walter Index being the best. The C-statistic showed good discriminatory power with the Walter Index 0.88 (95% confidence interval, CI: 0.84–0.91), CCI 0.78 (95% CI: 0.73–0.83), GPI 0.75 (95% CI: 0.69–0.80), and FRAIL Scale 0.74 (0.69–0.78) (Table 3, Figure S1). In the pairwise comparison, the discrimination of the Walter index outperformed those of the GPI, CCI, and FRAIL Scale (all P <0.001). The differences among the GPI, CCI, and FRAIL Scale were not statistically significant (P >0.05) (Table S5). The Brier scores and C-statistics of the four models pre- and post-imputation are shown in Tables S6 and S7 and show no significant changes.

Table 3 The Performance of 4 Prediction Models in Predicting 1-Year Mortality

The P values for the four models in the HL test ranged from 0.66 to 0.98, indicating acceptable calibration (Table 3). The distribution of the calculated risk scores and the relationship between the 1-year observed and predicted mortalities are shown in Figure 1 and Table S8. The calibration curves are shown in Figure 2. The predicted 1-year mortality, based on the Walter Index and FRAIL Scale, for each risk group was approximated to the observed mortality, and the calibration curves almost corresponded to the ideal calibration curve (Figures 1A, 1D and 2). The 1-year mortality risk range predicted by the Walter Index was much greater than that predicted by the FRAIL Scale. GPI scores of 0–5.5 corresponded to predicted mortality risk of 1.6% to 24.6%, with a small difference between predicted and observed mortality risk, with a larger difference for GPI scores of 6–7.5 (Figure 1B). However, the patient sample for GPI scores of 6–7.5 was insufficient. After resampling Bootstrap 1000 repetitions, the calibration curve almost corresponded to the ideal curve for 1-year mortality <20% and deviated slightly from the ideal curve when 1-year mortality increased (Figure 2). The CCI calibration was good when the 1-year mortality rate was below 7% but poorer as the 1-year mortality rate increased (Figure 1C). Despite resampling Bootstrap 1000 repetitions, the calibration curve significantly deviated from the ideal calibration curve (Figure 2). The calibration curves for the pre- and post-imputation data of the Walter Index, GPI, and FRAIL Scale are shown in Figures S2–S4, indicating no significant differences between pre- and post-imputation. For the two prediction models with 1-year mortality data from the original study, the Walter Index and CCI, we further compared the observed 1-year mortality based on the original development cohort with our study population and found that the observed mortality risk per risk strata of the Walter Index and CCI was generally lower in our study than in the development cohorts (Figures S5, S6).

Figure 1 Distribution of calculated risk scores and the relationship between observed and predicted 1-year mortality. Abbreviations: CCI, Charlson Comorbidity Index; GPI, Geriatric Prognostic Index. Notes: The columns show the percentage of patients in different subgroups, and the lines show the 1-year mortality, with the green line is the observed mortality and the orange line is the predicted mortality. Sub-figures A–D represent distribution of calculated risk scores of the Walter Index, GPI, CCI, and FRAIL Scale and the relationship between observed and predicted 1-year mortality, respectively.

Figure 2 Calibration plot for 1-year mortality. Abbreviations: CCI, Charlson Comorbidity Index; GPI, Geriatric Prognostic Index. Notes: Calibration plot with predicted mortality on the x-axis and observed mortality on the y-axis. A perfect prediction corresponds to the 45°dotted line (ideal calibration line). Calibration curves were bias-corrected using bootstrapping (B=1000 repetitions), indicating the performance of the observed models.

Discussion

Prediction models should be validated at various times and locations to better guide clinical practice. For the first time, we provide data from older inpatients in China to validated the Walter Index and GPI and compare them with conventional models of CCI and FRAIL. The 1-year mortality risk in older patients increased progressively with increases in the four model scores. All the models exhibited good discrimination in predicting the 1-year mortality risk, the best of which was the Walter Index. In terms of calibration, the Walter Index also performed the best, followed by the FRAIL Scale and GPI, and the CCI was poor.

Prediction models could provide the determinants of poor prognosis and assist in clinical decision-making and prioritization of care goals. This could help clinicians and patients choose invasive and/or conservative treatments and facilitate shared decision-making between physicians and patients.³⁶ This would be important for older patients with limited life expectancy to avoid useless life-prolonging measures.³⁶ In our study, the Walter Index, GPI, and FRAIL Scale showed good discrimination and calibration, with commonality in the predictive variables used, with both disease and functional status being assessed, compared with CCI, which only assessed disease. Additionally, the Walter Index had better discrimination than the CCI in predicting 1-year mortality in older inpatients in a study from six European countries.³² The results, analyzed from a large European countries database, emphasize that for older patients, poor prognosis is affected by disease patterns, cognition, functional status, and socioeconomic support.² A comprehensive assessment should be conducted to determine different care options and to change the traditional disease-oriented treatment model towards maintaining functional status and improving quality of life.² Therefore, mortality prediction models that simultaneously assess disease and functional status would be more appropriate for older patients.

In our study, the Walter Index was the best-performing 1-year mortality prediction model, with a total score of 20. Analysis of the predictive variables and the weights assigned to each variable of the Walter Index revealed that for comorbidities, it focused more on diseases with high 1-year mortality, such as heart failure, tumors, and metastatic tumors, with a weight of 8 points. It also focused more on disabled patients, with all ADLs dependence assigned five points. However, each variable was assigned the same weight in the GPI and FRAIL Scale, not considering the different effects of each variable on mortality risk. The Walter Index contains laboratory variables, with low serum albumin indicating malnutrition and chronic inflammation, which is an independent predictor of 1-year mortality.^37–39 The Walter Index contains risk factors in four domains: demographics, medical diagnoses, functional status, and laboratory test results, consistent with clinical practice, where the causes of death are multifactorial in older adults.^6,40

The Walter Index showed very good calibration, but the 1-year mortality in our study population was substantially lower than that in the original development cohort, possibly because our study population was younger, less ADL-dependent, had a lower incidence of decreased albumin and increased creatinine levels, and had a lower incidence of comorbidities (other than diabetes and tumors) than that of the Walter index development cohort population (Table S9). The lower mortality may be related to public health developments, such as increased public health funding, improved health care access, decreased health inequities, better educational levels, and improved nutrition, which have contributed to decreased mortality due to chronic diseases and cancer and an increased life expectancy in China.^41–47 For the CCI, we also observed lower mortality for each risk stratum compared to the original development cohort. Further validation of the Walter Index is needed for more accurate prediction of 1-year mortality in older inpatients from different regions of China and other developing countries.

In addition, the GPI and FRAIL Scale were also demonstrated to present good discrimination and calibration in this study, especially for the FRAIL Scale, which can be scored in a short period of time, which is very convenient. In contrast, GPI involves multidimensional assessment, which is relatively time-consuming and laborious, may not be suitable for some scores for very frail patients, such as MMSE and GDS-15. But different interventions can be given according to the results of different dimensions, such as malnutrition, cognitive decline, and depression, which is more beneficial to the comprehensive management of older inpatients. Therefore, clinicians can choose more appropriate scoring methods according to different scenarios.

We evaluated four models using a multidimensional method to provide evidence from older inpatients in China. This study has a few limitations. It was retrospective, with some missing data, thus, the Walter Index, GPI, and FRAIL Scale were not assessed for all participants. We performed multiple imputations, and the results showed no significant differences between the pre- and post-imputation data. Our study was not identical to the scales used to assess functional status (ADL and IADL scales) in the GPI and Walter Index, which may have biased the results; however, no studies comparing the modified Katz ADL, PSMS ADL, and Korean ADL and the Lawton IADL and Korean IADL in predicting poor prognosis differences were found. The content of the different scales is summarized in Table S2. An unauthorized version of the Chinese MMSE was used by the study team without permission and has been rectified with PAR. The MMSE is a copyrighted instrument and may not be used or reproduced in whole or in part, in any form or language, or by any means without written permission of PAR (www.parinc.com).Our study population was single-center and small-sample with low 1-year mortality, and the performance of the four models should be further validated in different regions in China and other developing countries.

Conclusions

The four mortality prediction models exhibited good discriminatory power in predicting the 1-year mortality risk of geriatric inpatients in a geriatric ward in China. However, compared with the CCI, which only assessed disease, the Walter Index, GPI, and FRAIL Scale, which assessed both disease and function, showed better Calibration. Combining the Brier score, discrimination and calibration, the Walter Index was confirmed for the first time to be the best model to predict the 1-year mortality risk of geriatric inpatients in China among the four models.