Methodological Quality of Systematic Reviews on Treatments for Liver Cancer: A Cross-sectional Study

Introduction

Liver Cancer (LC) is the sixth most common cancer in the world and the second most deadly cancer.¹ Based on the statistics from the World Cancer Research Fund International, the new cases of LC have reached 900,000 with an 8.7 per 100,000 people mortality and 20.8% 5-year relative survival rate in 2020.² The research estimates that between 2020 and 2040, there would be a 55.0% rise in new instances of LC, with 1.4 million new diagnoses predicted by 2040.³ The burden of LC is greatly increased by the rising trend in the disease’s prevalence and the length of time patients remain unwell before dying, exposing global health and social care systems to a serious threat.

At present, both pharmacological and non-pharmacological therapies are used to treat LC patients. These include surgical therapies like resection, liver transplantation, which is potentially curative therapy but not suitable for the majority, ablation, transarterial embolization and radiotherapy, as well as systemic pharmacological treatments like targeted therapy, immunotherapy, and chemotherapy.^4–8 With the availability of numerous treatment options for LC patients, it is paramount for clinicians to make sound decision based on the best available clinical evidence.

Systematic reviews (SRs) which are thought to offer the best quality of evidence about the effectiveness of specific therapy. As per the Cochrane handbook, a qualified SR with a flawless meta-analysis can identify, critically appraise, and synthesize quantitative evidence from eligible primary studies that comply with predetermined eligibility criteria.⁹ However, not every SR follows critical methodological quality.^10,11 Methodologically flawed SRs have the potential to mislead clinical decision-makers by exaggerating or underestimating treatment effects.¹² Thus legislators, medical professionals, caregivers, and any other evidence users, should critically assess the methodological quality of SRs before implementing the results and conclusions into clinical practice.

So far, no assessment of the methodological quality of SRs in LC treatments has been conducted. Therefore, this cross-sectional study aimed to: (i) describe the bibliographic characteristics of a current sample of SRs on LC treatments; (ii) assess the methodological quality of SRs using the Assessing the Methodological Quality of Systematic Reviews 2 (AMSTAR 2) tool;¹³ and (iii) explore the relationships between methodological quality and bibliographic characteristics. Findings of this study could provide insight on how the methodological quality of the future SRs on treatments for LC could be improved.

Methods

Eligibility Criteria

In this cross-sectional study, SRs only focusing on randomized controlled trials (RCTs) and including one or more meta-analyses that summarize the effectiveness of LC treatments are eligible. SRs published in English or Chinese were taken into consideration for inclusion, and any intervention type for LC was accepted. SRs that focus mainly on the diagnosis, etiology, or risk factors of LC were excluded. Protocols, narrative reviews, network meta-analyses, and overview of SRs were excluded, as well. Only the most recent version of a single SR was included if multiple versions were discovered. We selected the most up-to-date SR with multiple versions with the remaining as supplementary materials.

Literature Search, Literature Selection, Data Extraction

We searched four electronic databases, including MEDLINE (Ovid), Embase (Ovid), PsycINFO (Ovid), and the Cochrane Database of Systematic Reviews to identify potential qualified SRs published between January 2014 and October 2023. Validated search filters for SRs were applied to maximize the specificity of search on MEDLINE and Embase.^14,15 The search strategies are presented in Appendix 1. Two reviewers (ZC and YZ) screened the retrieved titles and abstracts of citations and checked the full text of SRs for determining the eligibility. After they independently collected detailed bibliographic characteristics using a pre-designed data extraction form (Appendix 2), a third reviewer (HW) was consulted in cases with persistent disagreements.

Methodological Quality Assessment

We used AMSTAR 2 tool to evaluate the methodological quality of included SRs.^13,16 According to the published operational guidelines, AMSTAR 2 has seven critical items (items 2, 4, 7, 9, 11, 13, 15) among 16 items.¹³ SRs were assessed as “high”, “moderate”, “low”, or “critically low” in terms of overall methodological quality, depending on how well they performed in each item (Appendix 3).¹³ Two reviewers independently carried out the methodological quality assessment. Any disagreements or discrepancies were settled by senior researcher arbitration or by consensus among the authors.

Data Analysis

Data of the bibliographic characteristics and AMSTAR 2 assessment was presented as frequencies (percentages), and medians (ranges) appropriately. The Kruskal–Wallis rank test was used to compare differences between overall methodological quality of SRs and bibliographic features separately.

Binary logistic regression (for AMSTAR 2 items 1, 3, 5, 6, and 10–16) and multinomial logistic regression (items 2, 4, and 7–9) were applied to explore the associations between bibliographic characteristics and individual AMSTAR 2 items. Drawing from previous methodological research findings, the following seven bibliographical characteristics were selected as independent variables for the logistic regression analyses: (1) whether the SR was a Cochrane review, (2) whether it was an updated SR, (3) the type of treatment that SR focus on, (4) year of publication, (5) impact factor of the journal in the year before the SR was published, (6) number of authors, and (7) location of the corresponding author.^10,11 The model fitting of binary logistic regression was evaluated using the Hosmer–Lemeshow test, and multinomial regression analysis was evaluated using the Pearson deviance test and likelihood ratio test. The associations between AMSTAR 2 ratings and bibliographic characteristics were quantified through calculating the adjusted odds ratios (AORs) and 95% confidence intervals (CIs). Statistical significance was indicated if P <0.05. All data analysis processes were conducted by IBM SPSS Statistics v. 27 software (IBM Corporation, Armonk, NY, USA).

Results

Literature Screening and Selection

We retrieved 9686 records through the literature search. After de-duplication, the remaining 7752 citations were screened based on their titles and abstracts, of which 841 full texts were downloaded for further checks. Finally, 119 SRs that met the eligibility criteria were included. The list of included studies is recorded in Appendix 4. Details on literature selection are presented in Figure 1.

Figure 1 Flow chart of literature search and selection. Abbreviation: RCT, andomized controlled trial.

Bibliographic Characteristics

Overall, this study included 119 SRs (1448 RCTs and 201,713 patients) published between 2014 and 2023 (median: 2019). The medians number of included RCTs and patients were eight (range: 2–65) and 1055 (range: 49–22,113), respectively. Only three (2.5%) SRs were Cochrane reviews, and around one-quarter (24.3%) were updates of previous SRs. Most of the included SRs (84%) were published in English. The impact factor of the journal in the year before the SR published was ranged from 0 to 22.682, with a median of 2.656. The median number of review authors was six (range: 2–17). Most of the corresponding authors (84%) were located at Asia, followed by Europe (9.3%), America (5.0%) and Africa (1.7%). The highest proportion of funding location of SRs was from Asia (39.5%), while nearly one third (31.9%) did not report funding relevant information. Close to two thirds of included SRs focused on pharmacological treatments (66.4%), and over 80% (82.4%) reported the intervention harms. Almost all the SRs (95.8%) included a PRISMA flowchart. All included SRs searched for English databases, but less than half of the SRs (41.2%) searched for non-English databases. More than half of SRs (55.4%) did not restrict the publication language of primary studies. Nearly two-thirds of SRs (64.7%) reported the “year of coverage” by starting and ending years. Around 75% of SRs applied Cochrane risk of bias tool for critical appraisal (74.8%), followed by Jadad scale (15.1%) (Table 1).

Table 1 Bibliographical Characteristics of 119 Included Systematic Reviews

Overall Methodological Quality

Only one SR (0.8%) was appraised as high and moderate methodological quality, respectively. Nine SRs (7.6%) were of low quality while 108 SRs (90.8%) were judged as critically low quality. Kruskal–Wallis test results indicated that Cochrane review (P <0.001), update of previous review (P <0.001), SRs with corresponding author from Africa (P=0.001), SRs receiving fund from institutions or organizations in Europe (P=0.001), and SR reported full search strategy (P=0.025) presented higher overall quality (Table 2). The Spearman’s rank correlation coefficients showed positive association between journal impact factor (rs=0.290; P=0.001) and overall methodological quality.

Table 2 Overall Methodological Quality of the Included Systematic Reviews by Bibliographical Characteristics

Performance of Included SRs on Each Individual AMSTAR 2 Item

Table 3 showed the performances of all included SRs across each AMSTAR 2 item. Explicitly deficient performances were found in three critical items while less than 20% of included SRs met the quality criteria. Only 15 SRs (2.6%) reported a registered protocol (item 2), four SRs (3.4%) used a comprehensive literature search strategy (item 4), and eight SRs (6.8%) provided a list of excluded studies and justify the exclusions reasons (item 7). As well as some of included SRs performed unsatisfactorily on non-critical items 3, 8, 10. Respectively, 23 review authors (19.3%) explained their selection of the study designs for inclusion in the review, 15 (12.7%) described the included studies in adequate detail, and six (5.1%) reported on the sources of funding for the studies included in the review.

Table 3 Results of the AMSTAR 2 Items for the Included 119 Systematic Reviews

The included SRs had excellent performances on non-critical item 1, 5, 6, and 14 for over 80% of the total fulfilling the items’ requirements: (i) all SRs clearly stated PICO (population, intervention, control, outcome) in the research questions and inclusion criteria for the review (item 1); (ii) 118 SRs (99.2%) had study selection and data extraction performed in duplicate (item 5 and 6); (iii) 102 SRs (85.7%) provided a satisfactory explanation for any heterogeneity found in the results of the review. However, none of critical items reached the requirement of outstanding performances.

Factors Associated with Methodological Quality

Binary logistic regression revealed that the more recently that SR published, the better performance it had on following items: explaining inclusion criteria of the study design (AOR: 1.29; 95%CI: 1.02–1.62), applying appropriate methods into statistical combination of meta-analysis results (AOR: 1.29; 95%CI: 1.06–1.57), evaluating and interpreting the potential impact of risk of bias on results (AOR: 1.47; 95%CI: 1.23–1.76), providing a satisfactory explanation for any heterogeneity found in the reviews (AOR: 1.32; 95%CI: 1.05–1.66), and reporting the conflict of interest (AOR: 1.33; 95%CI: 1.03–1.72). Compared with non-updated reviews, updated reviews had better performance on explaining inclusion criteria of the study design (AOR: 6.93, 95%CI: 2.06–23.30). SRs conducted by Asian corresponding authors showed worse performance on using appropriate methods in statistical combination of meta-analysis results (AOR: 0.17; 95%CI: 0.04–0.78) compared to those conducted by European. SRs published in journals with higher impact factors performed better on stating the conflict of interest (AOR: 2.98; 95%CI: 1.67–5.30) (Table 4).

Table 4 Associations Between Bibliographical Characteristics and Methodological Quality of Individual AMSTAR 2 Domain Item: Binary Logistic Regressiona

Multinomial logistic regression indicated that the SRs published more recently were more likely to be rated as “partial yes” (AOR: 1.46; 95%CI: 1.04–2.06) and “yes” (AOR: 1.86; 95%CI: 1.17–2.96) on registering a priori protocols. Compared with non-updated reviews, updated reviews had significantly better performance on describing included studies in details (AOR: 11.65; 95%CI: 2.38–56.92) (Table 5).

Table 5 Associations Between Bibliographical Characteristics and Methodological Quality of Individual AMSTAR 2 Domain Item: Multinomial Logistic Regressiona

Discussion

Summary of Findings

This cross-sectional study appraised the methodological quality of 119 SRs on LC treatments published between 2014 and 2023, involving both pharmacological and non-pharmacological interventions as well as Cochrane and non-Cochrane reviews. Among them, only one (0.8%) SR was evaluated as high overall quality, one (0.8%) with moderate quality, 9 (7.6%) with low quality and more than 90.8% with critically low quality. Our study indicated much improvement for most of the appraised SRs, especially in registering a priori protocols of reviews, explaining the selection of the study designs, using a comprehensive literature search strategy, listing all excluded studies with reasons, describing the included studies in adequate detail, and reporting the sources of funding of primary studies.

For an individual AMSTAR 2 item, SRs published more recently performed well on registering an a priori protocol, explaining the selection of study design, applying appropriate methods into statistical combination of meta-analysis results, evaluating and interpreting the impact of risk of bias in primary studies, providing an explanation for observed heterogeneity, and reporting conflict of interest. Due to the incorporation of the latest methodologies and recent evidence, which enhances the clarity and comprehensiveness of their findings, updated SRs would do better on explaining the selection of study design and describing the included studies in adequate detail.⁹ Compared to the SRs with corresponding author in Europe, SRs with Asian authors tended not to applying appropriate methods into statistical combination of meta-analysis results. Specifically, 87% of SRs with Asian authors inappropriately employed a fixed-effect model despite significant heterogeneity among the studies included in these reviews. SRs with higher journal impact factors had better performance on reporting the conflict of interest.

The progressively stringent demands for methodological quality criteria in journals could potentially account for the observed association between higher impact factors and more recent publication years with better performance on AMSTAR 2 items.^17,18 Along with the continuous progress of scientific research, the majority of journals are increasingly focusing on the methodological quality of literature, aiming to enhance their academic prestige and influence. Consequently, they have set rigorous requirements for the articles they intend to publish in their journals, particularly those with higher impact factors and academic reputations.

Most of English journals required the authors to declare any conflict of interest before publication, while it is not compulsory for most of the Chinese journals.¹⁹ In addition, the impact factors of all 18 publications published in the Chinese language were zero because their journals were not included in the Journal Citation Reports (JCR), which may also be an explanation that SRs with higher impact factors would perform better on reporting the conflict of interest.

Recommendations for Conducting SR in the Future

Registering the Protocol of Systematic Reviews a Priori

The protocol is an essential component for a high quality SR, which presents its research plan and ensure its reproducibility and transparency.²⁰ Review authors can also be benefited from registering a protocol, because it allows them to design and conduct a research plan carefully, reduces duplication on the same research topic, and enables other researchers to judge the validity of reviews.²⁰ Without a protocol registered prior to conducting reviews, the researchers are susceptible to selection bias caused by selective outcome gained from changing research methods on purpose.²¹ The trend shows that more SR protocols were registered on a protocol registration database, like PROSPERO (international prospective register of systematic reviews), which may support our findings that the SR was more likely to register a protocol if it was published recently.^22,23 In Cochrane reviews it is compulsory to provide an a priori protocol, it is highly recommended for non-Cochrane researchers to register their protocols.²⁴

Providing a Rigorous Explanation for the Selection of the Study Designs

A randomized controlled trial (RCT) is proved the best primary study design for assessing the effects of treatments, because randomization can minimize the confound sourced from different groups characteristics.²⁵ Although reviewers may have a short description about the eligibility criteria of an SR, it is necessary to provide a rigorous explanation for the selection of the study designs, even if it limited the eligibility criteria of an SR to RCT only. The possibility exists that a review restricted to RCTs would give an incomplete summary of the effects of a treatment if there were no relevant RCTs or missing outcomes in available RCTs.²⁶ In our findings, only 19.3% SRs justified the study design selection with reasonable strategy. SR authors are advised to improve the rigor of the SR in terms of explaining study design.

Conducting a Comprehensive Literature Search

Conducting a comprehensive literature search is an essential process of a SR to acquire an unbiased sample of primary studies. The rigor and results of a SR would be affected by the publication bias, if review authors did not provide their search strategy with supplement, or search gray literature in some cases.²⁷ The studies with positive results of one treatment are more likely to get published than those with no significant results regardless of their scientific rigor. However, unpublished studies, including gray literature, and those consulting from experts in the field or from the reference list of published reviews, can explore new evidence and that can be helpful for reducing publication bias.²⁸ In our study, we found only four (3.4%) SRs that conducted a comprehensive literature search, while 112 SRs (94.1%) did not take into account gray literature or consulting experts. Future reviewers should allocate sufficient resources to ensure the availability of gray literature searches and access to field experts.

Listing All Excluded Studies and Justifying Their Exclusions

The list of excluded studies with rationales for exclusion could minimize the subjective influences in the literature selection, improving the transparency and reproducibility of a SR in the review process, and preventing the risk of exclusion error.¹³ This approach may alleviate issues related to restrictive eligibility criteria in certain systematic reviews, enabling evidence users to access pertinent studies that are deemed valuable for addressing specific clinical questions. In our findings, only 6.8% of SRs offered a list of excluded studies with excluded reasons. This indicates that a significant proportion of included SRs are at high risk of being exposed to conclusion bias, which could lead to exaggeration or underestimation of results.⁹ Although it is not compulsory for non-Cochrane authors, we still highly recommend that future SR authors, editors, and peer reviewers take this issue into consideration.

Describing the Included Studies in Adequate Detail and Reporting on Their Resources of Funding

While describing the primary studies included in the SR, reviewers are supposed to present their study subjects, interventions, controls, outcomes, design, analysis, settings, and funding resources of the studies. It can help evidence users evaluate external validity and applicability of the findings, facilitate the exploration of clinical heterogeneity, and judge whether the financial interest exists.¹³ Multiple investigations have also demonstrated that studies funded by commercial sponsors are more prone to producing results that favor the sponsors’ products.²⁹ Unfortunately, only 12.7% SRs offered adequate details of included studies, and 5.1% SRs reported funding resources. Therefore, there is much room for improvement.

Comparisons with Similar Studies

Our results showed that the proportion of SR with high methodological quality (0.8%) is similar to those focused on osteoporosis treatments (1.0%) and atopic dermatitis treatments (1.9%), but much lower than those focused on asthma treatment (8.8%) and Alzheimer’s disease (AD) treatments (3.9%).^10,11,30,31 In terms of evaluating performance of an individual AMSTAR 2 item, our sample had better results on reporting PICO in the research questions and inclusion criteria, conducting study selection and data extraction in duplicate, and providing an explanation for heterogeneity in the results, compared with SRs focused on AD treatments, osteoporosis treatments, atopic dermatitis treatments. However, similar to SRs focused on atopic dermatitis and osteoporosis treatments, no critical item showed satisfactory performances.

Strengths and Limitations

The strength of this study lies in using AMSTAR 2 for assessing the most up-to-date sample of SRs on LC treatments. AMSTAR 2 has been proved to be a robust tool widely applied into evaluating the methodological quality of SRs, and we utilized it to conduct critical appraisal and reported the performance of SRs on each of the AMSTAR 2 items offering recommendations in specific domains.^16,32 Furthermore, we performed a comprehensive search to identify eligible SRs focused on LC treatments, covering pharmacological treatments, non-pharmacological interventions.

These limitations should be noted. First, appraisal results may be affected by insufficient information reported by published SR papers, which could limit the accuracy of our assessment. Second, we only included SRs published in English or Chinese, which may reduce the representativeness of our sample. Third, given the small number of SRs addressed with African corresponding authors (n=2), the finding of SRs with corresponding author from Africa presented higher overall quality may be very unstable However, this study has tried to search literature within the latest 10 years, but only identified two eligible SRs from African corresponding authors. If manpower and material resources are sufficient, further studies are suggested to extend the search time, which might identify more relevant studies to validate our results. Fourth, this study only considered seven confounders in the regression model. We did not include other potential confounders, such as language barriers and resource differences due to time and manpower limitations. Future studies should address this by incorporating language variables and regional resource indicators to refine the associations. This cross-sectional design did not assess the concrete impact of methodological deficiencies on reported efficacy estimates. Future research specifically comparing effect sizes across SRs of varying methodological quality, or re-analyzing data accounting for deficiencies, is needed to quantify the potential distortion these deficiencies cause.

Implications

The majority of the SRs on LC treatments were of critically low methodological quality in our study. The flaws existed in the SRs would mislead readers into significantly overestimating or underestimating the effects of LC treatments. Therefore, politicians, medical professionals, and other evidence users should carefully evaluate the methodological quality of SRs before implementing the evidence, in order to guarantee well-informed policy and clinical decisions. The Cochrane Handbook and AMSTAR 2 guidelines are highly recommended to be followed by authors, editors, and reviewers to promote the scientific integrity of SRs for publication.

Conclusions

The methodological quality of SRs on LC treatments was explicitly substandard. Future SR authors are supposed to improve quality of SRs through registering the protocol of systematic reviews a priori, providing a rigorous explanation for the selection of the study designs, conducting a comprehensive literature search strategy, listing all excluded studies and justifying their exclusion reasons, describing the included studies in adequate detail, and reporting their funding resources.