Enhanced Risk Stratification in Infective Endocarditis Surgery: A Comprehensive External Validation of All Available Mortality Prediction Scores

Introduction

Infective endocarditis (IE) remains a severe, life-threatening disease with considerable heterogeneity in its clinical presentation, course, and outcomes. It continues to pose a significant challenge to modern cardiovascular medicine despite major advances in diagnostic imaging, microbiological techniques, antimicrobial therapy, and surgical interventions.¹

Epidemiological data indicate a progressive increase in the incidence of IE worldwide.² This trend has been primarily attributed to several factors: the rising number of patients with prosthetic heart valves and cardiovascular implantable electronic devices; the growing prevalence of invasive procedures in high-risk populations; and the aging of the population, with an associated increase in multimorbidity and frailty.

Although early diagnosis and the development of targeted antimicrobial regimens have improved patient outcomes, a substantial proportion of cases ultimately requires surgical intervention. According to large multicenter registries (the ENDO registry), surgery is indicated in up to 50% of patients with IE, primarily for management of severe valvular dysfunction, uncontrolled infection, or prevention of embolic events.³

Given these challenges, current international guidelines strongly recommend that therapeutic strategies for IE be determined by a dedicated multidisciplinary “endocarditis team”, typically comprising cardiologists, cardiac surgeons, infectious disease specialists, microbiologists, neurologists, and radiologists.⁴ Such an approach ensures that treatment decisions—including the timing and modality of surgical intervention—are individualized and evidence-based.

A crucial component of this decision-making process is the accurate assessment of operative risk. Reliable risk stratification is essential not only to guide surgical indications and optimize timing but also to facilitate patient counseling, shared decision-making, and benchmarking of institutional outcomes. Integrating clinical risk scores with individualized patient assessment represents a cornerstone of contemporary management, enabling physicians to balance the potential benefits of surgical intervention against the inherent risks in this highly vulnerable population. Inaccurate or poorly calibrated risk estimation may have direct clinical consequences: overestimation of operative risk can unjustifiably delay or even preclude surgery in patients who would otherwise benefit, while underestimation may lead to premature or overly aggressive surgical intervention in clinically unstable individuals. Moreover, discordant or unreliable risk predictions complicate discussions within the multidisciplinary endocarditis team and may undermine patient counseling, particularly when clinicians attempt to communicate individualized operative risk in the context of complex comorbid profiles.

A wide array of prognostic models have been developed to predict operative mortality in cardiac surgery, with some specifically designed for the IE population. Traditional cardiac surgery scores such as EuroSCORE I and II and the Society of Thoracic Surgeons (STS) score were not originally intended for IE and may not account for disease-specific variables such as pathogen type, embolic phenomena, or valve destruction. More recently, several IE-specific models have emerged: AEPEI,⁵ ANCLA,⁶ APORTEI,⁷ ASSESS-IE,⁸ COSTA,⁹ De Feo,¹⁰ EndoSCORE,¹¹ ICE,¹² LOPEZ,¹³ modified MELD-XI,¹⁴ meta-model,¹⁵ PALSUSE,¹⁶ RISK-E,¹⁷ SHARPEN,¹⁸ Specific ES-I and Specific ES-II,¹⁹ STOP,²⁰ STS-IE,²¹ and one machine learning–based score proposed by Luo L.²²

Despite their promise, these models underwent limited external validation; moreover, most scores were developed without external validation or internal data partitioning (train/test split) and relying only on cross-validation of the overall sample, raising concerns about their generalizability and risk of overfitting.

This lack of robust external testing limits clinicians’ confidence in applying these tools to real-world populations, where comorbidities, microbiological profiles, and disease severity may differ substantially from derivation cohorts. As a result, multidisciplinary teams are often forced to rely on subjective clinical judgment rather than data-driven predictions, increasing variability in practice patterns and potentially affecting the timing and appropriateness of surgical referral.

This study aims to assess and compare the external performance of 18 prognostic models for operative mortality in a real-world cohort of patients with surgically treated IE. Our objective is to provide insights into the models’ utility in contemporary clinical practice and to identify the most accurate tools for risk stratification.

By clarifying which scores perform reliably in a contemporary external cohort, our findings aim to strengthen operative decision-making, improve consistency within endocarditis teams, and ultimately enhance patient counseling and surgical planning.

Methods

A retrospective analysis of 689 consecutive adult patients who underwent cardiac surgery for definite infective endocarditis between January 2010 and April 2025 was conducted. Diagnosis was established according to modified Duke criteria. Clinical, echocardiographic, microbiological, and surgical data were systematically extracted to compute each of the risk scores.

The study protocol received approval from the Romagna Ethics Committee on the 30^th of June 2023 (protocol number 4497/2023 I.5/95). Since the study involved a retrospective analysis of anonymized data with no direct patient interaction or intervention, individual informed consent was waived. All patient data were handled in accordance with strict confidentiality standards, and the study was conducted in full compliance with the principles outlined in the Declaration of Helsinki. Data collection commenced from clinical charts and was systematically collected in a specific registry. Extensive efforts were made to minimize the occurrence of missing information. In instances where data were missing, it was due to the nature of gaps in clinical documentation and assumed to be missing completely at random.

The included models comprised both general cardiac surgery scores (EuroSCORE II) and IE-specific scores (AEPEI, ANCLA, APORTEI, ASSESS-IE, COSTA, De Feo, EndoSCORE, ICE, LOPEZ, modified MELD-XI, meta-model, PALSUSE, RISK-E, SHARPEN, Specific ES-I and Specific ES-II, STOP, STS-IE).

Statistical Analysis

After checking normal distribution with Shapiro–Wilk test, continuous variables were reported as mean and standard deviation (SD) or median and interquartile range (IQR); categorical variables were reported as absolute number and frequencies; the Absolute standardized mean differences (ASMD) were reported in order to quantify the magnitude of differences between groups. Scores were reconstructed based on their original definitions; missing data were imputed using random forest imputation for categorical variables and predictive mean matching for continuous variables. The imputation was performed with 100 trees and 5 iterations, using predictive mean matching with k = 3 to ensure realistic values for continuous variables.

Model performance was assessed by analyzing discrimination using the area under the receiver operating characteristic curve (AUC); the scaled Brier scores were used to assess overall accuracy of the IE scores; firstly, the null Brier score was calculated, which corresponds to the performance of a non-informative model that predicts the same probability for all individuals, equal to the observed death prevalence (p). The null Brier score was computed as Brier_null = p*(1-p)^2 + (1-p)*p^2. The scaled Brier scores, which quantify the relative improvement of the model over the null model, were calculated as Brier_scaled = (1 - (Brier_model/Brier_null))*100%; a value of 100% indicates perfect predictions, while 0% means the model is no better than predicting the prevalence. Calibration was assessed using the Hosmer–Lemeshow test; moreover, calibration intercept and slope were estimated by fitting a logistic regression of the observed outcome on the logit of the predicted probabilities. For some models where the predicted probabilities showed almost no variation, the calibration slope could not be estimated; an intercept equal to 0 reflects perfect average prediction, while above 0 the risk is underestimated and above 0 the risk is overestimated; a slope equal to 1 reflects perfect calibration, while below 1 means that predictions are too extreme (overfitting); conversely, a slope above 1 means that predictions are too conservative, and low and high risks are not well differentiated (underfitting).

All analysis was performed with R 4.5.0 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Among the 689 patients included, 30% were females, with a median age of 69 years (Table 1); more frequent pathogens were Streptococcus (26%), Staphylococcus aureus (18%), Coagulase-negative staphylococci (18%), Enterococcus faecalis (16%) (Table 2). The observed operative mortality was 10.6% (n = 73). Most frequent complications were abscess (28.4%), vegetations (81.7%), leaflet perforation (14.5%), prosthetic valve dehiscence (16.8%); most frequent involved valves were aortic valve (70.1%) and mitral valve (43.8%); previous cardiac surgery was performed on 46.2% of patients; median Log euroscore was 19.9, while patients in Urgency/emergency were 80.7% (Table 3).

Table 1 Scores Characteristics

Table 2 Baseline and Demographic Characteristics

Table 3 Endocarditis and Operative Characteristics

Overall, 56% of the scores had no validation or only internal validation using bootstrap; 17% (3 scores) divided train/test set for the development, while 22% (4 scores) used an external cohort validation; 3 scores (17%) did not involve Infective Endocarditis correlated variables; mortality was assessed in-hospital or at 30 days in 89% of cases; the remaining 11% studied mortality at 6 months (Figure 1).

Figure 1 Scores distribution according to: panel (A) type of validation, panel (B) involvement of IE variables, panel (C) timepoint of mortality assessment.

Age and NYHA class were the most frequently included variables across the models (61% each). Of note, IE-specific predictors were represented in only a minority of scores: pathogen type or structural valve complications were included in just 50% of the models (abscess formation in 50%, and prosthetic valve involvement in 33%) (Figure 2).

Figure 2 Clinical Features Most Frequently Used Across Endocarditis Risk Scores. Abbreviations: CKD, Chronic kidney disease; COPD, Chronic obstructive pulmonary disease; HT, hypertension; IE, infective endocarditis; NYHA, New York Heart Association.

RISK-E demonstrated the highest discriminatory ability (AUC 0.742), followed by APORTEI (0.734) and modified MELD-XI (0.730) (Figure 3).

Figure 3 Scores ROC comparison: scores with AUC ≥0.7 (left) and scores with AUC <0.7 (right).

Brier score was below 0.02 for all scores, showing good accuracy; as a consequence, all scores achieved an 80% reduction in prediction error relative to the null mo scaled Brier scores resulted to be: Costa (87.7), Specific ES-II (87.1), STOP (87.1), ANCLA (86.9), Specific ES-I (86.4), ASSESS IE (85.7), AEPEI (84.7), PALSUSE (84.4), LOPEZ (83.9), ICE (83.5), STS-IE (82.7), De Feo (81.9), meta-model (81.8), EuroSCORE II (81.2), SHARPEN (80.8), APORTEI (80.1), modified MELD-XI (80.1), EndoSCORE (79.9), RISK-E (79.5). Hosmer–Lemeshow test showed bad calibration only for STOP.

Scores with slopes close to 1 (>0.7), indicating good calibration across the risk spectrum include: Specific ES-II, De Feo, PALSUSE, RISK-E, STOP, STS-IE, APORTEI; scores with slopes very low (near 0), indicating extreme overfitting, include: AEPEI, LOPEZ, modified MELD-XI, EndoScore, meta-mo scores for which the slope could not be estimated due to negligible variability in predicted probabilities include: ANCLA, ASSESS IE, Costa, ICE, SHARPEN, Specific ES-I; almost all scores showed intercepts close to 0, indicating good agreement between predicted and observed average risk; scores with extreme negative intercepts, reflecting overestimation of risk, include: EuroSCORE II (−22.2) and APORTEI (−13.9).

Although machine learning-based scores have shown promising predictive performance, they were not included in this study due to their limited explainability, which may limit their clinical applicability; in particular, weights for each variable included in the scores were not reported in the original papers.

Discussion

In this large, single-center external validation study, the performance of 18 operative mortality risk scores in patients undergoing surgery for IE was systematically assessed. Our results confirm that IE-specific models, particularly the RISK-E,¹⁷ offer superior predictive performance compared to generic cardiac surgery scores. These findings underscore the importance of incorporating disease-specific variables in risk stratification tools for IE.

Our results align with previous validation efforts, including the multicenter work by Fernandez-Felix et al,¹⁵ which emphasized the benefit of aggregating models and optimizing them using large-scale registry data. Similarly, Agrawal et al highlighted the underutilization of existing models in clinical practice and the need for tailored tools.²³

Of note, the top-performing scores were based on heterogeneous sets of variables, with no consistent overlap, neither in terms of IE related variables; this lack of overlap was also observed among the lowest-performing scores, suggesting that neither strong nor weak predictive performance is necessarily linked to a specific subset of variables; more specifically, RISK-E,¹⁷ APORTEI⁷ and EndoSCORE¹¹ consider the type of pathogen and valvular complications, while modified MELD-XI¹⁴ and SHARPEN¹⁸ do not consider any IE specific aspects.

General scores such as EuroSCORE II, while widely used, lack of the granularity to capture the complexity of IE. Its tendency to overestimate risk may lead to undue therapeutic futility in high-risk patients. Conversely, well-calibrated scores like RISK-E¹⁷ can facilitate shared decision-making by providing more accurate risk estimates.

With the exception of RISK-E,¹⁷ which included a train/test split, and a few scores that reported external validation (eg, AEPEI,⁵ APORTEI,⁷ ICE¹²), the majority of models were developed on single, undivided datasets without validation. Despite high AUCs reported in some cases (eg, STOP:²⁰ 0.941; De Feo¹⁰ 0.910), these values likely reflect methodological biases, such as overfitting and lack of external testing, rather than true predictive performance.

When recalculated on our external cohort, most scores showed substantially lower AUCs, highlighting their limited generalizability and the importance of rigorous validation procedures in prognostic model development.

It should be emphasized that several prognostic models applied in cardiovascular surgery and critical care (such as AEPEI,⁵ the modified MELD-XI,¹⁴ and SHARPEN¹⁸) do not incorporate variables that are specific to infective endocarditis. While the absence of disease-oriented parameters does not necessarily result in inferior statistical performance, it inevitably reduces their specificity for IE and, consequently, their potential clinical applicability in this complex and heterogeneous condition.

Another important methodological consideration concerns the validation process. For example, the ICE¹² and ASSESS⁸ scores, which were originally derived by splitting training and test cohorts across different time periods, demonstrated unsatisfactory performance in our dataset. This discrepancy most likely reflects the influence of endogenous temporal factors—such as evolving patient demographics, shifting diagnostic strategies, and advances in surgical or antimicrobial therapy—that were not adequately captured by the models. Such findings highlight the inherent limitations of risk scores when applied outside the temporal or institutional context in which they were developed.

Similarly, the STS-IE score²¹ did not report a test-set area under the curve (AUC) in the original publication and also showed suboptimal discriminative ability in our external evaluation. Taken together, these results underscore the need for caution when extrapolating the reported performance of existing risk scores to independent populations. They also highlight the critical importance of robust external validation in determining the true clinical utility and generalizability of such prognostic tools.

Although several publications and narrative reviews^24–26 have summarized the development of IE-related prognostic scores, and some models have undergone partial validation in selected cohorts, to the best of our knowledge, the present work represents the first comprehensive external validation of all 18 prognostic scores currently available for infective endocarditis.

A major strength of our study is its broad scope: in contrast to prior investigations that focused only on a limited subset of IE risk models, we systematically evaluated the entire spectrum of published scores within a large, independent external cohort. This comprehensive approach, not only permits a balanced comparison of predictive performance, but also provides a more nuanced and clinically meaningful assessment of their potential role in guiding decision-making and benchmarking outcomes in patients with IE.

Limitations

Although this study is retrospective and based on a single-center cohort, this characteristic—often viewed as a limitation—does not undermine the validity of our work. In the context of external validation, using data from an institution different from those in which the original scores were developed is an advantage rather than a weakness, as it provides a rigorous test of their true generalizability. Differences in clinical practice, microbiological profiles, and surgical thresholds between centers are expected and, in fact, essential to evaluate whether these widely used scores can be reliable across diverse real-world settings. Additionally, some variables required for model reconstruction were not uniformly available. Another important consideration is that our analysis focused exclusively on operative, short-term mortality, in line with the original works. Long-term outcomes—including recurrent infection, heart failure, late valve-related events, and overall survival—were not assessed. For a disease as complex as infective endocarditis, these longer-term trajectories are highly relevant to patient counseling and therapeutic decision-making, and our results should therefore be interpreted as pertaining only to the immediate perioperative period.

Finally, the study period spans more than a decade, during which diagnostic modalities, antimicrobial strategies, and perioperative management evolved considerably. These temporal changes may have affected the stability of associations between predictors and outcomes and could partly explain the variability in performance of historically derived models. Although such evolution reflects real-world practice and is unavoidable in longitudinal cohorts, it remains a potential source of bias that should be considered when extrapolating our findings to contemporary or future settings.

Conclusions

In this large external validation study, we comprehensively assessed the performance of contemporary IE-specific and general cardiac surgery risk scores in patients undergoing surgery for infective endocarditis. Our findings demonstrate substantial variability in discriminative ability and calibration across models, with IE-specific tools—most notably RISK-E—showing more consistent performance than generic scores. These results highlight the importance of rigorous external validation to determine the true clinical utility and generalizability of prognostic models for IE. Further prospective, multicenter studies are needed to confirm these findings and to support the refinement or recalibration of existing tools.