Bias and Reporting Quality of Clinical Prognostic Models for Idiopathic Pulmonary Fibrosis: A Cross-Sectional Study

Introduction

Idiopathic pulmonary fibrosis (IPF) is the most common type of interstitial lung disease and is characterized by dyspnea and progressive deterioration of lung function,^1,2 with median survival of 2 to 3 years from time of diagnosis.^3,4 The acute exacerbations and the associated complications often lead to hospitalization and death,^5–8 resulting in significant economic and health-care burdens.^5,9 At present, there are some challenges in the diagnosis and treatment of IPF, for instance, the complicated diagnosis process, limited and expensive interventions, and corresponding side effects.^1,10 Therefore, to mitigate the risks and burdens of IPF and to improve the perceptions of best practices of care, more efficient prognosis predictions are needed.¹¹

Multivariable prediction models in which multiple characteristics or pieces of information are applied can estimate an individual’s risk of a current condition in future.¹² When a patient’s score passes a certain threshold, an alarm may be sent to the appropriate clinicians for further evaluation and intervention. In recent years, interest in using prediction models has increased and the models are more often recommended in clinical practice guidelines^13,14 for individual prognosis and diagnosis. At present, there are more prognostic models for IPF.^15,16 However, because there is a surplus of IPF prognostic models with widely variable quality, it is important to identify IPF prognostic models of high quality.

Risk of bias (ROB) is usually defined as the presence of a systematic error that may affect the study’s validity. The Prediction model Risk of Bias Assessment Tool (PROBAST)¹⁷ guideline has been developed to assess the ROB of model development and model validation, including updating of a prediction model.¹⁸ A full reporting is essential to evaluate the validation and applicability of a multivariable prediction model;^19,20 therefore, a protocol was developed for a guided, Transparent reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement.²¹ Previous systematic reviews from other research teams showed that there are high ROB and suboptimal reporting quality in prediction models in oral health and preterm birth.^22,23 To date, there has been limited data for evaluating the quality of existing IPF prognostic models; therefore, in this cross-sectional study, we assessed IPF prognostic models with PROBAST and TPRIPOD checklists, aiming to identify the ROB and reporting quality of these studies and highlight the strengths and limitations of the methodologies of IPF prognostic models.

Materials and Methods

Study Design

We conducted a critical cross-sectional appraisal on ROB and reporting quality of IPF prediction models using the PROBAST²⁴ and TRIPOD checklists,²¹ respectively. Although this review was not a typical system review (SR), we did strictly adhere to the guidelines of conducting and reporting a SR,²⁵ the details of reported PRISMA-Checklists are listed in Supplementary Table 1. The review protocol was registered on INPLASY.²⁶

Inclusion and Exclusion Criteria

Studies were included if they met the following criteria: 1) A prognostic multivariable prediction model of IPF describing development, validation, or both, and 2) They should predict events at the probability of future outcomes (prognosis) related to IPF.

Studies were excluded if they 1) examined independent prognostic factors and did not aim to develop a model; 2) combined IPF with other diseases; 3) were not original research (such as review, methodological articles, conference abstracts, protocols); and 4) performed comparison models.

Literature Search

A comprehensive literature search was conducted in PubMed, Embase, and Web of Science between inception to 12 August, 2021. The details of the search strategy can be found in Supplementary Table 2. All relevant articles in the reference list of all included articles were also retrieved.

Study Selection

We created a database in EndNote X9 software. After eliminating duplicates, we read the titles and abstracts for a preliminary screening. Then, we downloaded the full text and filtered it again until all relevant prediction models of IPF were confirmed. Two researchers selected the literature, and if there were discrepancies between them, it was addressed by the discussion with the third researcher.

Data Extraction

We focused our research on study design, outcome measurement, modeling methodology, and validation strategy. Therefore, one researcher extracted the above key information, including author, publication year, population characteristics, follow-up time, etc. Another researcher checked the extracted data, and if there existed different opinions between two researchers, they would refer to the original text and revise it.

Application of Evaluation Tools

We classified each study into model development with or without external validation in the same publication and external validation study of a previously developed model only. Two of us used the PROBAST tool to assess the ROB for each included study. Following four PROBAST domains (Participants, Predictors, Outcome, Analysis), we assessed 20 signaling questions for development models and 17 signaling questions for validation models within each domain with yes/probably yes, no/probably no, or no information. We rated domain-level and applicability assessments using “low risk of bias,” “high risk of bias,” and “unclear risk of bias” according to PROBAST suggestion.¹⁷

Two researchers assessed the reporting quality of the included models according to the TRIPOD statement.²¹ The checklist covered 37 sub-items in 6 domains, including title and abstract (items 1 and 2), introduction (item 3), methods (items 4 to 12), results (items 13 to 17), discussion (items 18 and 19), and other information (items 21 and 22). We rated items as “reported” if the relevant information was fully presented, “unreported” if all relevant information were lacking, and “not applicable” for inappropriate data, as reported by previous studies.^22,27

Statistical Methods

The reporting rates of PROBAST and TRIPOD were calculated in a descriptive manner using proportions (%). The PROBAST items for development studies only (items 4.5, 4.8, 4.9) and the TRIPOD “if done” item (item 5c), the validation items (items 10c, 10e, 13c, 17, 19a) were used as both the numerator and denominator when the overall adherence rate was calculated. In each evaluation, we completed two rounds of pilot evaluation, SPSS 25.0 software was used to calculate the Kappa value of the internal consistency coefficient, and the formal assessment was made when Kappa ≥0.8. Our pre-evaluation results (Kappa = 0.818, P ≤ 0.001) demonstrated a good agreement between our reviewers.

Results

Literature Selection

A total of 1670 records were collected. Of those, 46 eligible full-text articles were reviewed after removal of duplicates and irrelevant articles. We excluded 26 studies for reasons shown in Figure 1. Finally, 20 IPF clinical prognostic model studies^{15,16,28–45} met the inclusion criteria in this process.

Figure 1 Schema of literature selection process.

Overview of Characteristics for IPF Prognostic Model Studies

Of the 20 prognostic model studies, 7 (35%) publications^15,28–33 reported model development and external validation, 8 (40%) publications^34–41 reported model development only, and 5 (25%) publications^16,42–45 reported external validation with one updated.⁴² According to these studies, 20 prognostic models were identified but more than 50% models did not undergo any external validation. A summary of characteristics for the included studies is shown in Table 1.

Table 1 The Characteristics of IPF Prediction Model Studies

ROB of IPF Prediction Model Studies

All models^{15,16,28–45} were at high ROB in development or validation. Eight models^{16,35–37,39,40,42,45} are rated as high ROB in applicability. Figures 2 and 3 show the proportion assessment for each PROBAST item and proportion of studies with potential bias using PROBAST, respectively. Overall, 55% of the studies were identified with at least four sources of bias (sub-items 1.1, 2.2, 4.2, and 4.8).

Figure 2 The proportion assessment for each PROBAST item.

Figure 3 Proportion of studies with potential bias using PROBAST.

As for participants, predictors, outcomes, and analysis domains, there were 12, 12, 6, and 18 studies that had a high ROB, respectively (The “biased” domain, applicability identified in each study is provided in Supplementary Figure 1). Of the included studies, 55.0% resulted in a high risk of bias because of the inclusion of retrospective studies (sub-item 1.1). For predictors of definition (item 2.1) and assessment (item 2.2), one study³⁹ proposed to exclude the predictor of diffusing capacity of lung for carbon monoxide (DLCO), and one study⁴¹ included a question as a predictor. In addition, 45.0% studies may not report any actions to blind assessment of the predictors. However, unreasonable predictor selection methods may limit the use of the model. Most sub-items in outcomes domains have low ROB, especially inconsistency in defining outcome (sub-item 3.4) and time interval between predictor and outcome assessment (sub-item 3.6). This is probably because most studies used outcomes that are easy to assess (eg, death, survival time). In addition, in the analysis domain, continuous variables were converted into dichotomous variables (sub-item 4.2) or there was lack of overfitting consideration (item 4.8) in most studies, which are the main reasons leading to the high ROB in analysis domain.

Bias Related to Applicability

There was high ROB about applicability, indicating that these studies were not well aligned with the search question. This is most commonly due to concerns about applicability in domain 1 (population) for included randomized clinical trials or specific groups. In addition, two^37,40 out of 20 studies’ outcomes included exacerbation, but the definition of acute exacerbation may have varied between studies and may have been expanded to include events for which a trigger can be identified, such as infection.⁴⁶

Reporting Quality of IPF Prediction Model Studies

Of the six domains, the introduction part of TRIPOD with scores of 90.0% is the domain where IPF prediction model studies had the highest mean score reporting rate. The prediction model of IPF had the lowest mean score for the domains of methods and results (46.6% and 44.7%). The mean reporting rate of each domain according to the TRIPOD checklist is less than 65%. The details of reporting rates of each of the TRIPOD domains are shown in Figure 4.

Figure 4 The detail of reporting rates of each TRIPOD domains.

For specific items, eight sub-items had a reporting rate ≥80% adherence to the TRIPOD checklist, and nine sub-items showed a low reporting rate with less than 30%. None of the included IPF prediction model studies reported any actions to blind assessment of the outcome/predictors (items 6b, 7b). In addition, with respect to validation models, few of them identified any difference or showed the comparison from the data between development and validation (items 12, 13c). The details of reporting for each item adherence to the TRIPOD checklist are shown in Figure 5.

Figure 5 The detail of reporting for each item adherence to TRIPOD checklist.

Discussion

Main Findings

In this cross-sectional study of prognostic risk models related to the IPF, we identified and critically appraised 20 studies that were described 20 models. All models reported good to excellent predictive performance, but all of them had high ROB according to PROBAST, and none of them completely adhered to the TRIPOD checklist for reporting, which demonstrates deficiencies in applicability and reporting transparency. First, as for ROB, we identified the mean reporting rate of each domain, which was less than 70%. Similarly, using the TRIPOD standards, we found that the mean reporting rate of each domain was less than 65%, especially the methods and results domains. The two tools suggest that there was a general lack of transparent reporting and identification of bias across the studies of IPF progression models.

The ROB of Included Studies According PROBAST

The main aim of prediction models is to support medical decision-making, and a high ROB implies that these models will probably perform worse in practice than in the studies reported by researchers.¹¹ We identified the reporting rate in participants domain was 45.0% and 67.3% in analysis, which means researchers should pay attention to the ROB of IPF prediction models.

In the participants domain, most studies did not select the appropriate data source, and there were a large number of retrospective studies in our study. The Gender-Age-Physiology (GAP) system, which might help inform decision-making, has been externally validated five times with good discrimination, but the ROB for the external validation studies of the retrospective research data was high. While prospective cohort studies are recommended for model development, there are practical issues for using prospective cohort studies given that IPF is relatively rare. In addition, there are many well-designed prospective national IPF registries that could serve this purpose.

In the analysis domain, continuous predictors should not be dichotomized or categorized, as this will result in loss of information, which in turn may lead to imprecise risk estimates.⁴⁷ In addition, we suggest that continuous predictors should appear as original data in future research on models. Calibration, which can assess the fit of models, is one of the essential features in the assessment of the usability of a predictive scale.⁴⁸ However, few studies considered calibration in our review. Although some studies evaluated this indicator, most of them used the statistical test of Hosmer-Lemeshow (H-L).⁴⁹ However, the H-L test cannot retain the most information on possible miscalibration, so a calibration plot or other evaluation method is recommended.^50,51 Lack of internal validation may lead to overfitting because quantifying the predictive performance of a model on the same data from which the model was developed tends to give optimistic estimates of performance,²² so that internal validation is recommended when there is no external validation.

The Reporting Quality of Included Studies According to TRIPOD

Full reporting of studies facilitates reproducibility of models, appraisal of model validity, and judgment of model generalizability to other clinical settings.²⁰ We found there is room for improvement in the reporting quality of IPF prediction models.

In the methods domain, a large number of these prediction models skipped estimating sample size, mainly because of a lack of consensus in estimating sample size requirements for derivation and validation.⁵² However, a reasonable sample size is necessary, so until a canonical method of calculating sample size is available, we can use 10 or 15 times the flat number of events to initially calculate the sample size. It was brought to our attention that the GAP system has been externally validated several times but has rarely been updated. Models will be more generalizable when the case mix of the new population is within the case mix range of the development population.⁵³ Therefore, in future studies, it is recommended to continuously update the GAP models according to the characteristics of the study population.

In the results domain, the reporting rates of participants’ engagement process and method of using the prediction model are lower than 30%. To make the results of prediction model research more complete and more transparent, we should consider not only the performance of the prediction model but also how to use the developed model for individuals. Therefore, we recommend that items such as how to use the model be reported clearly and directly.

Strengths and Limitations

To our knowledge, this is the first review to assess the ROB and reporting quality of multivariable prognostic models for IPF based on the PROBAST and TRIPOD checklists. We systematically searched the IPF prognostic models to assess the ROB and explore report quality of these studies and more importantly, to summarize the characteristics of the existing IPF prediction model studies to provide necessary references for future research. Before the formal assessment, we conducted a pilot experiment, requiring that the evaluator’s consistency coefficient was higher than 0.8 before the evaluation was carried out to further reduce the subjectivity of the researchers and improve the credibility of the results of our review.

Our study also has several limitations. First, we did not assess the relationship among ROB, reporting rates, and forecast accuracy. In addition, our study only considered the important factor of insufficient reporting, but other factors need to be further explored to promote the scientific implementation of the IPF prediction model in clinical practice.

Conclusion

According to the study, the rates of reporting adhering to PROBAST and TRIPOD checklists are low, especially in the domains of participants and analysis for PROBAST and the domains of methods and results for TRIPOD. Moreover, there is a general lack of identification of bias and transparent reporting, which can decrease the reproducibility rate of IPF models. However, the low rate of reporting, according to the two evaluation tools, does not mean low prediction accuracy, but the reproducibility and transparency can be improved when studies completely adhere to PROBAST and TRIPOD checklists. Therefore, studies adhering to PROBAST and TRIPOD checklists are recommended in the future.