External Validation of the International IgA Nephropathy Prediction Tool in Older Adult Patients

Introduction

IgAN is the most prevalent primary glomerular disease and one of the leading causes of ESRD, 20–40% of IgAN patients confirmed by biopsy will progress to ESRD within 20 years.¹ Its clinical characteristics, pathological types, and disease progression rates in IgAN patients vary across age groups.^1–3 Besides well-established clinical indicators like eGFR, albuminuria, blood pressure, and pathological indicators such as mesangial hypercellularity, endocapillary hypercellularity, segmental glomerulosclerosis, segmental glomerulosclerosis are linked to disease severity and progression in IgAN, age also serves as a significant risk factor impacting both disease severity and prognosis.^3–5 The prognostic characteristics of older adult patients with IgAN are distinct from those of younger patients. Our previous meta-analysis indicated that disease progression in older adult patients tends to be faster compared to their younger counterparts.⁴ Furthermore, several studies have identified age as an independent risk factor for the progression of end-stage renal disease (ESRD).^6,7 As the older adult population proportion increases, the number of older adult individuals with IgAN also rises. Accurate risk assessment is vital for these patients, yet there is a significant lack of a specialized risk prediction model specifically designed for older adults with IgAN.

In 2019, the IIgANN developed and verified two full prediction models without and with race for IgA nephropathy through a long-term follow-up of a large multi-ethnic cohort.⁸ Both models incorporate standard clinical indicators and the Oxford classification, offering prognostic risk assessments for adult patients during renal biopsy, aiding in patient stratification, and assisting clinicians in making treatment decisions.

While the original study that formulated these models included older adult patients, their representation was minimal. The International IgA Nephropathy Prediction Tool has undergone external validation and affirmed its remarkable prognostic value for Chinese patients, but its specific applicability for older adult patients has not been thoroughly investigated.^9,10 Therefore, the primary objective of our study is to examine these two prediction models’ suitability for older adult patients with IgAN.

Materials and Methods

Study Population

Currently, no established guidelines exist for determining the ideal sample size for validating prognostic models.¹¹ To ensure robustness, we conducted a multicenter retrospective cohort study to evaluate the performance of the full models without and with race in an older adult cohort. All participants were diagnosed with primary IgA nephropathy (IgAN) via renal biopsy and admitted between January 2004 and September 2021. These patients were selected from six medical centers affiliated with the Chinese PLA General Hospital. Their medical records, including outpatient and inpatient files and biopsy reports, were thoroughly analyzed. Our analysis specifically focused on patients aged 60 or older without end-stage renal disease (ESRD) at their biopsy and had accessible data on the estimated glomerular filtration rate (eGFR). Patients with less than one year of follow-up data were excluded, except those who reached the primary outcome within this period. Additionally, we excluded patients with permanent renal failure at biopsy time (eGFR < 15 mL/min/1.73 m²) or those lacking a MEST score. Initially, 231 patients satisfied our eligibility criteria. Nonetheless, after a comprehensive review process, 66 patients were excluded. Consequently, a total of 165 patients were included in the final analysis. Figure 1 visually illustrates the patient selection process. The study was approved by the Ethics Committee of the Chinese PLA General Hospital (approval number S2023-247-01) and informed consent was waived due to the retrospective design, which confirmed compliance with the TRIPOD guidelines for validating risk prediction models.¹²

Figure 1 Flowchart of patients screened, recruited, and included in the final analysis.

Variable Definitions

For this external validation study, all predictors were defined and collected using identical methods to those employed in the original cohort (detailed in Supplementary Table 1). Both full models encompassed the following predictors: age, eGFR, mean arterial pressure, proteinuria, the use of renin-angiotensin-aldosterone system (RAAS) blockers and immunosuppression, and the MEST score. The eGFR was calculated using the Chronic Kidney Disease Epidemiology Collaboration equation.¹³ The mean arterial pressure was derived by adding the diastolic pressure to one-third of the pulse pressure. The utilization of RAAS blockers and immunosuppression was defined as any exposure at or prior to kidney biopsy. Kidney biopsy specimens underwent independent evaluation by two experienced pathologists blinded to clinical data based on the Oxford classification score for IgA nephropathy.¹⁴ In the external validation of the full model with race, we classified our patients as Chinese.

Calculation of Predicted Risk and Risk Groups

To compute the linear predictor and prediction probability for each patient’s primary outcome, we applied the formulas for both full models by Barbour et al.⁸ Consistent with the methodology of the original study, patients were classified into four distinct risk groups based on the centiles of their linear predictors: low risk (< 16th), intermediate risk (16th to 50th), higher risk (50th to 84th), and the extremely highest risk (≥ 84th).¹⁵

Statistical Analyses

All statistical analyses were performed using R version 4.3.1 (R Foundation for Statistical Computing, Vienna, Austria). Continuous data were assessed for normality using the Shapiro–Wilk test. Non-normally distributed data were summarized using medians (interquartile range) and compared using the Kruskal–Wallis test. Categorical data were presented as frequencies (percentages) and compared using the chi-squared test or Fisher’s exact test. In addition, we evaluated the models’ performance through measures of discrimination, calibration, and reclassification.¹⁵ Two-tailed p < 0.05 was considered statistically significant.

For discrimination, we assessed the performance of the models on distinguishing between patients with and without endpoints using the concordance statistic (C statistic): Harrell c index and time-dependent receiver operating characteristic (ROC) curve.^16,17 The primary outcome’s linear predictor (LP) and predictive probabilities for each patient were calculated by fitting two Cox proportional hazard models based on these variables.¹⁵ We first estimated the regression coefficient on the linear predictor coefficient in our data set. If the βlinear predictor≥1, then the discrimination of the models is seen as excellent. Subsequently, Kaplan-Meier analysis was performed based on risk groups, and survival curves were generated.¹⁵ The distinct separation of survival curves for each risk group indicated the model’s strong discriminatory ability. We utilized the Log rank test to assess differences between the survival curves of risk groups, with the low-risk group as the reference. Besides P-values for comparing risk groups, hazard ratios also served as an important metric for confirming model discrimination.

Calibration was assessed by graphically comparing observed and predicted risks of the primary outcome over the follow-up period. Time-specific calibration was reported using calibration plots, which depicted the relationship between observed and predicted risks, categorized by fifths of predicted risk and risk groups derived from linear predictors. Predicted risks were calculated as the mean predicted risks of all individuals or groups, and observed risks were estimated using the Kaplan-Meier method. Additionally, we plotted the predicted survival curve based on predicted risk and compared it with the corresponding Kaplan-Meier survival curve for the same risk groups.

Continuous net reclassification index (NRI) and integrated discrimination improvement (IDI) were employed to assess the reclassification of clinical risk by the prediction models with and without. NRI and IDI values were deemed significant, with 95% confidence intervals (95% CIs) not including 0. Values greater than 0 indicated positive improvement, while values less than 0 suggested negative improvement. Confidence intervals for continuous NRI and IDI were derived from 100 bootstrap samples.

Results

Study Population Characteristics

Table 1 compares participant characteristics between our older adult external validation cohort and the original cohorts. While our older adults cohort showed a similar median follow-up period of approximately 5 years, like the two original cohorts, notable differences were observed in patient demographics. The median age in our cohort was significantly higher (64 years compared to 36 and 35 years, respectively), with a higher proportion of males (60% versus 58% and 49%, respectively). Additionally, the eGFR levels at the time of biopsy were lower (50.8 mL/min/1.73m² compared to 83 mL/min/1.73m² and 90 mL/min/1.73m², respectively), and proteinuria levels were elevated (2.13 g/d versus 1.2 g/d and 1.3 g/d, respectively). Within the older adults cohort, we observed higher rates of M1 lesions (50% versus 38% and 42%, respectively) and T1 lesions (40% versus 25% and 18%, respectively), as well as T2 lesions (19% versus 5% and 11%, respectively), but lower rates of S1 lesions (55% versus 77% and 80%, respectively). Additionally, 32% of the older patients showed a crescent formation at the time of biopsy. Moreover, a higher percentage of patients in our cohort were treated with RAAS blockers (56% versus 32% and 30%, respectively). In comparison, a smaller proportion received immunosuppressive therapy (2% versus 9% and 7%, respectively) at or before the biopsy. Among the 165 patients enrolled in our study, 35 observed major outcome events (50% reduction in eGFR or renal failure), with only 3 (2%) encountering major outcomes within one year.

Table 1 Characteristics of Participants in the Older Adults Validation Cohort, the Original Derivation Cohort, and the Original Validation Cohort

Performance of the International IgAN Prediction Tool

Discrimination and Reclassification

In our older adults cohort, both full models demonstrated good discrimination (as detailed in Table 2). The Harrell c index for the full model with race was 0.788 (95% CI: 0.712–0.865), and for the full model without race 0.790 (95% CI: 0.717–0.863). The area under the ROC curve (AUC) at 3 years (as shown in Figure 2a) were 0.88 and 0.89 for the models with and without race, respectively, and at 5 years (Figure 2b) the AUC values were 0.77 and 0.82. The overall calibration slopes, presented in Table 2, were 0.88 (95% CI: 0.85–0.91) and 0.92 (95% CI: 0.88–0.95) for the respective models. When comparing the full model without race to the one with, no enhancement in risk reclassification for predicting 5-year risk was noted, with NRI of 0.09(95% CI, −0.27 to 0.34) and IDI of 0.003(95% CI, −0.009 to 0.019), respectively. Kaplan-Meier curves and risk ratios between risk subgroups are depicted in Figure 2c and d) and Table 3. The distinct separation of survival curves further confirms the good discriminant function of the full models. While neither model excelled in differentiating between low-risk and intermediate-risk groups, both effectively distinguished between high-risk and highest-risk groups.

Table 2 Concordance Statistics, Calibration Slopes, Net Reclassification Improvements, and Integrated Discrimination Improvements of Full Models in the Older Adults External Validation Cohort

Table 3 Associations of the Risk Groups with the Composite Outcome of 50% Decline in Estimated Glomerular Filtration Rate or Kidney Failure in Patients with Over 5-Year Follow-Up (n=165)

Figure 2 The Time-dependent receiver operating characteristic (ROC) curves and the Kaplan-Meier curves confirmed the excellent distinguishing ability of both full models. Abbreviations: AUC, the area under the receiver operating characteristic (ROC) curve. Notes: (a and b) show the receiver operating curve analysis at 3 years and 5 years for both full models, respectively. (c and d) respectively show the Kaplan-Meier curves of the primary outcome in 4 risk groups distinguished by the Full model with race and the full model without race: the well-separated curves indicate good discriminant ability of both the models. The 4 risk groups were defined as <16th (low risk), 16th −50th (intermediate risk), 50th −84th (higher risk), and >84th (the highest risk) percentiles of the linear predictor from the full model without and with race, respectively.

Calibration

Figure 3 elucidates a comparative analysis between the observed and predicted risks of primary outcomes as delineated by the two distinct models. The full model without race accurately prognoses the risk associated with primary outcomes in the older adults cohort. The full model with race predicted the risk of the primary outcomes more accurately at 3 years and gradually overestimated the risk after 3 years and the longer the follow-up time. Figure 4 depicts the observed versus predicted 3-year and 5-year risks, calculated based on predicted risk and quintiles of risk groups. Both models exhibit some imprecision in predicting primary outcomes for various risk levels in the third year. Advancing to the fifth year, the model inclusive of racial factors markedly overestimates the risk of primary outcomes, and the model without race aligns more closely with actual risk trajectories. While the full model without race slightly underestimates the five-year risk except for the low risk group, it maintains better overall calibration. In contrast, the full model with race minimizes risks in lower risk groups and overestimates in higher ones, resulting in slightly less accuracy than the full model with race.

Figure 3 The full model without race more accurately prognoses the risk associated with primary outcomes in the older adult cohort. Notes: (a) Comparison of observed and predicted risks of the primary outcome during the follow-up period. Predicted risks are mean predicted risk curves (blue and pink solid lines), observed risks were estimated by the Kaplan–Meier method (black solid line), and the grey dashed lines represent 95% confidence intervals of the observed risks. (b and c) Comparison of observed and predicted risks of the primary outcome throughout follow-up. The full prediction models with and without race were analyzed by subgroups based on the percentile of the linear predictor (<16th in yellow, 16th-50th in blue, 50th-84th in green, and >84th in pink) in the older adult cohorts. Predicted risks are mean predicted survival curves (dotted lines), and observed risks are Kaplan-Meier curves (solid lines).

Figure 4 Calibration plots at 3 years for (a and b) the full model with race and (c and d) the full model without race, and at 5 years for (e and f) the full model with race and (g and h) the full model without race. Notes: Plots by risk groups are (a, c, e, and g, and by fifths of predicted risk are b, d, f, and h). The dashed lines indicate perfect calibration, in which predicted risks are the same as the observed risks. Vertical lines in observed groups represent 95% confidence intervals. Risk groups were based on percentiles of the linear predictor (group 1(low risk);<16th, group 2 (intermediate risk); 16th-50th, group 3 (higher risk); 50th-84th, group 4(highest risk: >84th)).

Discussion

In our study, we conducted external validation and performance evaluation of the International IgA Nephropathy Prediction Tool using an older adults cohort from six different centers at the PLA General Hospital.⁸ Both full models exhibited significant discrimination, effectively calibrating the 3-year risk of major outcomes. When forecasting the 5-year risk of these outcomes, the full model without the race variable showed more accurate calibration. Taking into account factors like discrimination, calibration, and reclassification, the full model without the race component outperformed in predicting the 5-year risk for this older adult group.

Given that IgA nephropathy’s progression correlates with aging, accurate risk assessment is imperative for older adult patients.^4,7 However, most existing risk prediction models focus on adult patients, primarily due to IgAN’s peak incidence between 10 and 29 years and its relative rarity in older populations.¹⁸ These models often have limitations, such as small sample sizes, the inclusion of participants from only one ethnic background, or the omission of crucial pathological indicators.^19–23 In contrast, the IIgANN developed two international IgAN prediction models using data from a large, multi-ethnic cohort to predict disease progression and risk stratification in a diverse patient population.⁸ To provide a reliable basis for prognostic assessment in older adult patients, we externally validated the widely acknowledged International IgAN Prediction Tool.

The discrimination depends on the variability of the characteristics of older adult patients in the dataset. Both models show a remarkable ability to distinguish patients within our cohort, with the c statistics for the full models with and without race being 0.788 and 0.790, respectively, which are slightly lower than those of the original cohort.⁸ Additionally, the ROC curves and AUC values indicate that both models excel in predicting 3-year risks (as illustrated in Figure 2). However, the discrimination power of the full model with race slightly diminishes when indicating 5-year risks. In survival analysis, the survival curves for the low and intermediate risk groups did not show significant separation during the 80-month follow-up. However, a clear separation among all four risk groups becomes evident upon extending the observation period to 10 years, as seen in Supplementary Figure 1. This suggests that the models also effectively differentiate between low- and intermediate-risk patients.

In summary, both models are adept at identifying patients at higher risk of progression during renal biopsies. It also indicates that our cohort encompasses diverse risk groups and is thus representative. We also observed that in the older adult patients, the primary outcome was less frequent among low risk and intermediate risk patients, predominantly occurring after 5 years post-diagnosis. In a Chinese adult patient cohort external validation study, the c statistics for both models were 0.889, surpassing those in the original research. There, the full model with race showed superior performance, particularly in improving the 5-year risk reclassification. However, our study reveals that while both models can discern the progressive risk in older adult patients, the full model with race, typically more suitable for Chinese patients, did not exhibit enhanced risk reclassification.

Moreover, the full model without race exhibited superior calibration performance. Notably, this model slightly underestimated the overall predicted risk, aligning with the increased risk of kidney failure in IgAN among Asians.²⁴ Conversely, the full model with race displayed a distinct calibration pattern: it tended to underestimate predicted risks for the first 3 years but significantly overestimated risks beyond this period. The marked rise in risk predicted by the full model with race after 3 years is attributable to the specific coefficients assigned to the Chinese race during the model’s development. An external validation study of 1373 Chinese adults by Zhang et al revealed that while the full model without race often underestimates major outcome risks, the full model with race demonstrates accurate calibration in predicting five-year risks.⁹ However, another study by Zhang et al involving 1275 Asian-Caucasians (91.7% of Chinese patients) found that the full model without race was well-calibrated.¹⁰ Still, race led to an overestimation of risk after three years. Our findings align more closely with the latter observation. This discrepancy underscores the international IgAN prediction tool’s challenge in precisely forecasting risks for older adult patients. Given the model’s development and the previously validated low proportion of older adults in the large Chinese cohort, the tool’s effectiveness in predicting risks for older people remains obscured. This suggests that disease progression in older adult patients differs from that in younger patients. The prognostic characteristics of older people need further investigation through multi-ethnic studies with larger sample sizes. Our study’s outcomes support the clinical practice of using non-race-based full models for predicting risks in older adult patients.

Compared to the broader demographic of IgAN patients diagnosed in China, our cohort was characterized by older age, lower eGFR, higher proteinuria levels, and a greater incidence of T1 and T2 lesions. Yet, there were no increased primary outcome events. Several factors may account for this observation: (1) Some cases may involve IgAN with minimal lesion nephropathy, where IgA deposition could be incidental;^25–27 (2) Kidney aging:²⁸ research indicates that 3% of kidney donors aged 18–29 exhibit glomerulosclerosis (global sclerosis, interstitial fibrosis, and arteriosclerosis), compared to 73% of donors aged 70–77;^29,30 (3) It’s established that eGFR generally declines with age.^31–34 Studies show that approximately 50% of individuals over 70 have an eGFR < 60mL/min/1.73m²;³⁵ (4) The eGFR estimation in this study was based on the CKD-EPI equation 12, which may not accurately assess actual eGFR levels in older adults; (5) The usage rate of renin-angiotensin system blockers (RASB) during a renal biopsy in our study was higher than in the original study (56% vs 32% and 30%), and RASB use has been shown to improve renal outcomes in IgAN.^36–38 Previous studies have identified aging as an independent risk factor for disease progression in IgAN patients, but others suggest a slower progression in older adults.³⁹ Our findings seem to align with the latter, indicating that the impact of age on IgAN prognosis might be less significant than previously thought. Multi-ethnic studies with larger sample sizes are needed to determine the true effect of age on IgAN prognosis more accurately.

This study is the first to evaluate the applicability of IIgAN-PT to older adult patients, and the results show that the full model without race can accurately predict the risk of disease progression in older adult patients. However, it has several limitations. First, while it is a multicenter retrospective study with the largest cohort of older adult IgAN patients to date, the sample size is still relatively small and limited to a single ethnic group, which poses constraints in model validation. Both models warrant further proof with larger, multi-ethnic elderly cohorts. Second, there was significant variability in patient numbers across the different medical centers involved. This discrepancy is attributable to the varying sizes of the nephrology departments, with the First Medical Center being the largest, contributing the most patients. Finally, it is important to note that some data were missing from the study. We chose to analyze only complete datasets to maintain data integrity and reliability. Given that the missing data accounted for only 2% of the total, its potential impact on our findings can be considered negligible.⁴⁰

Conclusion

In conclusion, both full models demonstrated excellent discrimination abilities, with the full model without race variable showing enhanced calibration in predicting the 5-year risk, surpassing the performance of the full model with race. Although the International IgAN Prediction Tool did not achieve perfect results in the older population within this study, its external validation confirms its capability to offer valuable risk assessments for individual patients at early disease stages. To further evaluate the predictive accuracy of the International IgAN Prediction Tool and its impact on clinical decision-making, larger prospective cohort studies incorporating various ethnic groups are necessary.⁴¹