Systematic Review and Critical Appraisal of Prediction Models for Readmission in Coronary Artery Disease Patients: Assessing Current Efficacy and Future Directions

Introduction

Coronary Artery Disease (CAD), characterized by the accumulation of cholesterol and other substances in the coronary arteries leading to arterial narrowing or blockage,¹ exerts considerable pressure on healthcare systems worldwide.² CAD patients often require repeated readmissions due to inadequate disease control.^3,4 In China, the 30-day unplanned readmission rate for CAD patients stands at 6.3%,⁵ closely paralleling the 7.5% reported in the United States National Database.⁶ Readmissions, defined as hospitalizations within a specific timeframe following discharge, vary in nature. “All-cause readmission” includes any readmission regardless of cause,⁷ while “unplanned readmission” pertains to those unforeseen at the time of discharge.⁸ The latter is often linked to increased mortality and healthcare costs.^9–11 Meanwhile, CAD patients face potential readmissions for multiple reasons, including recurrent myocardial infarction, unstable angina, and heart failure, necessitating a focus on all readmission types due to the chronicity and management complexity of CAD. Hence, precise and early assessment of unplanned readmission risk in CAD patients, coupled with targeted preventive measures against risk factors, is of utmost importance.¹² Prediction models are instrumental in identifying such risks by analyzing relevant factors to estimate rehospitalization probabilities. A foundational study by Engoren et al¹³ in 2013, employing genetic programming, logistic regression, and artificial neural networks, marked a pivotal advancement in predicting post-surgical readmissions for CAD patients. Subsequent models have varied in participants, predictors, and outcomes, facilitating personalized risk assessments.¹⁴ However, their clinical adoption is impeded by insufficient evidence on model performance and utility. Currently, there is an absence of comprehensive systematic reviews assessing these models’ methodological quality, predictive accuracy, and clinical applicability. This systematic review seeks to uncover and detail research focused on the development and/or validation of prediction models for readmission among CAD patients, while also rigorously assessing the potential bias and clinical applicability of these models.

Materials and Methods

This systematic review was conducted following the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).¹⁵ A protocol for this study was registered with PROSPERO (CRD42023429102).

Data Sources and Search Strategy

A systematic literature search was conducted in PubMed, Embase, Web of Science, EBSCOhost, and CNKI (China National Knowledge Infrastructure), targeting publications in Chinese and English, with the search timeframe spanning from the inception of each database/website until September 30, 2023. The electronic search included Mesh terms and entry terms relevant to coronary artery disease, readmission, and prediction models. Search strategies for each database are detailed in Supplementary Table 1. Additionally, reference lists of included studies were manually reviewed for further relevant research.

Inclusion and Exclusion Criteria

Studies qualifying for inclusion met the following criteria:

Population: CAD patients, including patients with acute coronary syndromes (ACS) or acute myocardial infarction (AMI), or who have undergone coronary artery bypass graft surgery (CABG).¹⁶

Intervention: The models were developed using either heuristic or statistical approaches. Heuristic methods relied on expert experience and knowledge for selecting and weighing predictors, illustrated through expert agreement and the Delphi method. Conversely, statistical approaches utilized multivariable analysis and machine learning techniques to detect and rank the significance of predictors. Every study provided at least one measure to assess model efficacy, including the concordance index (c-index), area under the receiver operating characteristic curve (AUC), Brier score, Hosmer–Lemeshow test, calibration plots, and observed-to-expected ratios.

Comparison: Prediction models constructed from various research designs offer complementary and contrasting perspectives, incorporating prospective cohort, retrospective cohort, nested case–control, case-cohort, randomized trial, and cross-sectional design.

Outcome: Readmission includes several types: all-cause readmission, planned all-cause readmission, and unplanned readmission. “All-cause readmission” means a patient returns to the hospital for any reason within a set time after leaving.⁷ “Unplanned readmission” refers to patients readmitted to the same or another hospital for any diagnosis, not including certain planned readmissions.⁸

In addition, in cases of duplicate publications reporting the same model, the one with the most comprehensive description of methods, results, and model specifics was selected for inclusion in our analysis. Studies limited to a single predictor were excluded, as predictive models integrate multiple factors to assess risk. Systematic reviews, conference papers, book chapters, and reports were also not considered.

Study Selection

Study selection was executed by two independent reviewers in three stages. Initially, search results from databases were collated in EndNote X9. Subsequently, titles and abstracts were sifted based on predefined inclusion and exclusion criteria. For studies lacking consensus on exclusion, the full text was examined for a more detailed evaluation. Finally, the full texts of potential studies were examined to determine their suitability for the systematic review. Any disagreements during the process were resolved through discussion or by consulting a third reviewer.

Data Extraction

Data extraction was carried out by two independent reviewers following the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS),¹⁷ along with the details provided in Supplementary Tables 2 and 3, and included: the first author, years of publication participants (category, country), modeling methods, study design, outcome (type of readmission, length of time, readmission rate), model development or validation sample size, validation model methods, c-index (Development/ Validation), predictor numbers, and names of predictors, etc. Disagreements were resolved through discussions, or if necessary, by consulting a third reviewer.

Assessment of Risk of Bias

The risk of bias in the included studies was evaluated independently by 2 investigators using the Prediction model Risk of Bias Assessment Tool (PROBAST).¹⁸ Inconsistencies were resolved by discussion or with the assistance of the corresponding author. The assessment of risk of bias covers 4 domains: participants, predictors, outcomes, and analyses. The assessment of applicability covered 3 areas: participants, predictors, and outcomes. Domain-specific bias risk hinges on responses to signaling questions within each domain. The overall risk of bias is considered high if any one of the four domains is rated as high risk, and it is considered low only if all domains are evaluated as low risk. Any discrepancies in assessment were reconciled through discussion or by consulting a third reviewer. Supplementary Table 4 presents the results of the specific assessment based on PROBAST.¹⁸

Data Synthesis

This study employed a narrative synthesis approach to present its findings. Initially, we summarized the data sources included in the study, covering aspects such as study location, sample size, research design, and participant characteristics. Subsequently, we reported descriptive statistics on model development and validation, including strategies for model development, methods of development and validation, techniques for predictor selection, c-index for assessing model discrimination, final model predictors, predicted outcomes, and methods for handling continuous predictors and missing data.

The c-index, similar to the AUC which measures how well the model can tell different outcomes apart, shows the model’s accuracy; a c-index close to 1 means it’s more accurate.

Results

Search Results

Figure 1 illustrates the outcomes of the literature search and the process of selecting studies. The systematic search produced a total of 12,457 records. This systematic review removed 4816 duplicates and screened out 7587 irrelevant or ineligible records based on title and abstract. The full-text screening was conducted for the remaining 54 records. Of these, 39 records were excluded for the following reasons: (1) no model development (n=21), (2) conference abstract (n=15), (3) duplication data (n=2), (4) and letter to the editor (n=1). Ultimately, 15 studies^5,13,19–31 were included in this systematic review, reporting a total of 30 prediction models for readmission in CAD patients.

Figure 1 Search results and study selection.

Characteristics of Eligible Studies

Among the 15 studies, 7 were conducted in China, 7 in the US, and 1 in Canada. The study populations included CAD (n=3), AMI (n=8), post-CABG (n=3), and ACS (n=1) patients. More than half of the eligible studies employed a retrospective study design (n=10). In all the studies, the first was published in 2013,¹³ with the remaining 14 focused between 2018 and 2023. The temporal cutoffs for the results were as follows: 7 days (n=1), 1 month (n=7), 3 months (n=1), 6 months (n=3), 1 year (n=1), both 1 month and 1 year (n=1), and 5 years (n=1). Due to variations in readmission rates based on CAD type, sample size, and outcome time point, the readmission rates ranged from 4.8% to 45.1%. Further details regarding the characteristics of each study are provided in Supplementary Table 2. The PRISMA checklist is available in Supplementary Table 5.

Prediction Model Approaches and Performance

Supplementary Table 2 presents a comprehensive overview of the characteristics of the 30 prediction models designed for readmissions. In terms of model development, six studies^{20,21,24,25,29,31} utilized logistic regression to identify predictive factors for readmission in CAD patients. Two studies^5,23 employed the Cox proportional hazards model. Two studies^26,27 utilized Bayesian models. One study¹³ employed a combination of logistic regression, genetic algorithms, and artificial neural networks to develop prediction models. Additionally, one study³⁰ applied five machine-learning methods, while another study²⁸ employed six machine-learning algorithms to construct predictive models. Furthermore, one study²⁷ developed five predictive models using a mix of parametric models (elastic net, least absolute shrinkage, selection operator, and ridge regression) and nonparametric models (random forest and gradient boosting). Lastly, one study²² constructed a three-layer stacking-based prediction model.

Model discrimination was frequently assessed using the c-index, which is equivalent to the area under the receiver operating characteristic curve for binary outcomes. All prediction models reported C-index values, which ranged from 0.597 to 0.9149. Among the internally validated models, four studies^13,22,28,29 employed cross-validation for internal validation, while six studies^{19,20,23,26,30,31} used bootstrapping for validation. Only three models demonstrated strong discrimination (c-index > 0.750), with the model developed by Song et al³⁰ for elderly CAD patients achieving the highest discrimination (c-index: 0.9729). Among the externally validated models, two^21,24 used temporal and one²⁷ used geographical validation, the c-index reported by Yang et al²⁴ and Liu et al²¹ study were both pretty, 0.890 and 0.865 respectively. However, three studies^5,21,25 did not conduct internal validation, two studies^22,28 only reported conducting internal validation without reporting specific values, and two^5,25 did not validate the models. Six studies^{19,21,24,25,29,31} were tested for the Hosmer-Lemeshow goodness-of-fit. Four of the studies^19,21,24,31 showed good calibration. Three^27,28,30 used the Brier score to evaluate the accuracy of the model, two^28,30 of the studies had better accuracy (p<0.25), and one²⁷ study specified that the Brier test was performed but did not report the final score.

Predictors for Readmission in CAD Patients

Among the 15 inclusion studies, the number of potential predictors varied from 21 to 293. To identify the ultimate predictors incorporated into the prediction models, multivariable logistic regression was the most commonly used method, with eight studies^{19–21,23,25,27,29,31} utilizing it. One study⁵ chose multifactor Cox regression, two studies^13,24 chose binary logistic regression, Song et al³⁰ chose Lasso regression, Gupta et al²⁸ chose six common machine learning methods, Dreyer et al²⁶ chose Bayesian model averaging, and Zhang et al²² chose Select-From-Model. Research reporting the final included predictors is presented in Supplementary Table 3. Two studies^22,28 did not report predictors for eventual inclusion. The most common predictor was age, followed by length of hospitalization and gender. Predictors can be categorized roughly into first day of hospital admission and full-stay in the hospital. On the first day of admission, predictors such as age, gender, type of insurance, and past history may be of concern. For predictors at discharge, there was more focus on in-hospital complications and the location of post-discharge transfers.

Analysis of Differences in Readmission Prediction Models Among Patient Groups

In analyzing rehospitalization prediction models for CAD patients, we particularly examined the differences between various types of CAD patients. For CABG and AMI patients, the studies focused on different aspects for the one-month post-discharge period. The CABG readmission model included predictors such as “complications” and “deep sternal wound infections”,^13,21,31 while the AMI model incorporated factors related to baseline health, physiological parameters, and medical history.^{5,19,22,23,26–29} For ACS patients, the research centered on a six-month post-discharge model, considering ACS types and treatment methods.²⁰ In contrast, the three studies covering all CAD patients spanned from 7 days to 5 years, showing risk considerations from short to long term, with predictors covering a wide range of clinical and biochemical parameters.^24,25,30 Notably, readmission rates differ among CAD types, with AMI patients (16.3% and 12.9%) generally higher than post-CABG patients (8.2% and 9.1%).

Risk of Bias of Included Studies

Findings from the bias risk and applicability assessments, conducted using the Prediction Model Risk of Bias Assessment Tool (PROBAST), are concisely summarized in Supplementary Table 4. In summary, five studies^{20,21,24,25,29} were considered to have a low concern for applicability, yet all included models carried a high risk of bias. Specifically, two studies were identified as having a high bias risk related to participants, stemming from their methods of selective selection or volunteer recruitment, resulting in samples that were not representative. And in another study,²⁴ the rate of missing persons is as high as 34%. Yang et al²⁴ could not ensure that all study subjects were measured by equally clinically experienced measurers, and Nguyen et al²⁹ lacked information about how the predictor (personality) was measured. In the outcome domain, five^{20,21,24,25,29} studies were unclear risk of bias because they did not clearly describe the definition of readmission. For the analysis part, all studies had a high risk of bias, mainly because the sample sizes were too small and because they did not handle different types of data correctly. Another key reason for the high bias risk in the analysis was the neglect or improper handling of missing data. Multiple imputation is regarded as the optimal method for handling missing data because it enables users to explicitly account for the uncertainty of the actual values of input variables. Only the studies by Matheny et al²⁹ and Dreyer et al²⁶ implemented multiple imputations for missing data. Concurrently, given the numerous potential predictors in the prediction model for unplanned readmissions of CAD patients, it’s challenging for the sample size to meet the criterion where the number of clinical outcome events is 20 times the number of potential predictors. This situation can easily lead to biases stemming from insufficient sample sizes. Among the studies, only Kini et al²³ research met the sample size requirements.

Discussion

The present systematic review aimed to summarize and discuss the evidence of different prediction models, their performance, and their ability to predict readmission in CAD patients. Fifteen studies reporting 30 models were selected from 12,457 studies. Nonetheless, each study was perceived to exhibit significant bias, largely attributed to analytical shortcomings such as a limited sample size, inappropriate treatment of continuous and categorical variables, and insufficient management of missing data. Within the scope of this research, six studies^{19,22,23,26,28,31} were included that encompassed all-cause readmission patients, while only four studies^5,27,29,30 utilized unplanned readmission as the outcome. Additionally, five studies^{13,20,21,24,25} did not explicitly define the type of readmission. Efforts were made during the preparation of this study to contact the authors of these papers for clarification, but no responses were received. Consequently, this research concludes that these five studies with an undefined outcome exhibit limitations in terms of applicability.

Model Employed and Performance

Of the 30 incorporated prediction models, 17 are tailored for AMI patients. It’s evident that healthcare systems and policymakers worldwide are increasingly emphasizing reducing the readmission rates post-AMI discharge.^32,33 Research indicates that between 1990 and 2019, the drop in AMI mortality was the most pronounced among all categories of coronary heart diseases.³⁴ Nguyen et al²⁹ classified predictive factors into “first-day” and “full-stay.” This classification aids clinicians in selecting the most suitable prediction model based on individual circumstances. But the study only underwent internal validation, it achieved c-index scores of 0.70 and 0.73 respectively. Numerous studies have opted for a 1-month post-discharge outcome window, likely due to the sanctions imposed by the Centers for Medicare & Medicaid Services (CMS) on hospitals with high 30-day readmission rates post-AMI discharge since 2012.³⁵ However, this does not diminish the relevance of models targeting alternative durations. For instance, Kini et al²³ underscored a CMS-initiated voluntary episodic payment model (EPM) for AMIs spanning 90 days and subsequently devised a 3-month predictive framework tailored for this EPM. Similarly, Dodson et al¹⁹ posited that a six-month post-discharge period might be more indicative of the challenges faced by the elderly post-AMI.

This study particularly focused on the differences among various types of CAD patients. The findings indicate significant disparities in the risk of rehospitalization and its influencing factors within a specific post-discharge period for different CAD subtypes. It was observed that the rehospitalization rates for AMI patients were generally higher than those for CABG patients. This variance could be attributed to the more complex acute onset, baseline health conditions, and physiological parameters associated with AMI patients. In contrast, the lower readmission rates in CABG patients might reflect the effectiveness of surgical interventions and postoperative care. The CABG’s prediction models primarily focus on specific factors such as surgical complications and deep sternal wound infections, which may have a more direct impact on the risk of rehospitalization in these patients.

Model validation entails assessing performance in distinct populations, encompassing both internal and external validation.¹⁷ Internal validation assesses the reproducibility of the model development process and prevents overfitting of the model. In contrast, external validation focuses more on the model’s portability and generalizability, enhancing its extrapolative capability. External validation was rigorously conducted for only three of the models;^21,24,27 Only the study by Matheny et al conducted a multicenter validation. However, the model’s validation outcomes were suboptimal, with the best performance for the external validation’s c-index only being 0.655, suggesting that the models found were inadequately validated. Internal validation, which involves evaluating the model’s efficacy in the same population from which it was developed, is generally easier and more straightforward than external validation. Nonetheless, three of the studies^21,25,28 included did not undertake this validation. In internal validation, four models^21,24,25,30 demonstrated pretty discrimination with c-indices all exceeding 0.75. The prediction model by Song et al³⁰ exhibited the best distinction of 0.9149, but its limitation is the lack of external validation. Following that, Yang et al²⁴ model, specifically designed for 6-month readmission prediction, had c-indices of 0.890 and 0.875 for internal and temporal validation respectively. However, in Yang et al²⁴ study, the number of patients lost to follow-up exceeded one-third (808/2344) of the total patient cohort.

In conclusion, future clinical research should place greater emphasis on the multi-center and diverse validation of these models, to ensure their effective applicability across various patient groups and clinical settings.

The Predictors Used in Prediction Model

Due to variations in study designs and variables, predictors of readmission for CAD patients vary. However, common predictors have been identified, categorizable as those present at admission and those emerging during hospitalization. Predominantly, models consistently incorporate gender and age as key predictors. Research indicates that female patients, particularly due to a tendency to delay medical care,²⁹ are more likely to develop heart failure and face higher readmission rates.³⁶ Additionally, post-menopausal women show a heightened susceptibility to ACS due to hormonal changes.^37,38 Age is also an independent predictor, correlating with increased CAD prevalence and severity, especially in the elderly who face declining organ functionality and multiple comorbidities, heightening rehospitalization risks.³⁹

The transitional phase post-discharge is of critical importance. Typically, patients are discharged post-acute care without comprehensive plans for rehabilitation and daily care, leading to heightened physiological and psychological vulnerability, especially in older adults.⁴⁰ This transition period, whether returning home or to a facility, presents substantial risks.^9,41

Furthermore, continuous monitoring and management of blood pressure are crucial for CAD patients. Studies have demonstrated that hypertension significantly increases the risk of cardiovascular events and mortality.⁴² For instance, a study targeting individuals at high risk for cardiovascular diseases (CVD) reveals that effective blood pressure management could prevent approximately 2.2 million cases of coronary heart disease among hypertension patients in China over a decade.⁴³ This underscores the importance of blood pressure modulation in secondary prevention strategies for CAD patients.⁴⁴

Clinical staff could integrate these prediction models into clinical warning systems, aiding in the early identification of high-risk patients. Meanwhile, prioritizing early intervention and prevention strategies, focusing on modifiable factors while also considering non-modifiable predictors.

Clinical Practice Recommendations

To effectively translate the research findings of CAD patient readmission prediction models into clinical practice, this study recommends selecting specific prediction models based on the patient’s type (such as CAD, AMI, CABG, ACS). Future clinical research should prioritize model validation across multiple centers and diverse patient populations to enhance the applicability and generalizability of prediction models. It is advised to integrate these models into hospital information systems for automatic risk assessment, enabling early identification and management of potential readmission risk factors. This approach will assist healthcare professionals in creating more personalized discharge plans and follow-up strategies, thereby more effectively reducing readmission rates among CAD patients, improving their quality of life, and alleviating the economic burden on the healthcare system.

Study Limitation

The primary constraint of this systematic review lies in its reliance solely on narrative synthesis, due to the absence of external validation for the included models by independent investigators. The second limitation is excluded grey literatures like conference abstracts and committee reports. Because grey literature often does not undergo rigorous peer review, it typically consists of preliminary research findings presented in brief summaries.⁴⁵

Conclusion

This study reviewed 15 studies, 30 readmission prediction models for CAD patients, finding them generally biased and lacking external validation, with key predictors being age, gender, and hospitalization length. It recommends future research focus on external validation and to use the PROBAST tool to mitigate bias in model development.