Methodological Quality of PROMs in Psychosocial Consequences of Colorectal Cancer Screening: A Systematic Review

Plain Language Summary

Previous research has found that cancer screening is associated with psychosocial consequences, such as anxiety. Measuring psychosocial consequences can be difficult and requires at least a valid questionnaire, so-called Patient-Reported Outcome Measures (PROMs). A PROM should be developed in collaboration with people from the target population and relevant experts. This is important to make sure that: 1) the PROM adequately covers the potential psychosocial consequences, and 2) the PROM is relevant and understandable for the respondent. Also, valid measurement requires that PROMs are statistically tested in accordance with measurement theory. However, many PROMs that are used to measure the psychosocial consequences of cancer screening lack both elements. This results in low-quality evidence and leaves us uncertain about the true magnitude of psychosocial consequences. This review analyzes the quality of the PROMs used in studies that measure the psychosocial consequences of colorectal cancer screening. Twenty-nine out of thirty PROMs included in this review lacked proper patient involvement and had inadequate measurement properties in a screening context. This means that we cannot trust the results of the studies that use these PROMs. Future studies should use PROMs with adequate patient involvement and proper psychometric measurement properties, and existing evidence should be critically evaluated considering potential biases.

Introduction

Patient-Reported Outcome Measures (PROMs) are outcomes reported by patients often in the form of standardized questionnaires.¹ PROMS are commonly used in health research to measure latent traits in patients, for example, opinions, behavior, or psychological states, which can then be used to compare or evaluate interventions.^1,2 PROMs have previously been used in screening contexts to measure psychosocial consequences of screening.^3–7 In a screening setting, the word patient in PROM should be read as apparently healthy persons. Any PROM should be rigorously developed and its measurement properties should be assessed to ensure that the PROM validly measures the latent trait and reliably assess changes over time. When this is not done, it is unclear what is measured.

Another aspect of PROMs is that they can be condition-specific or generic. Generic PROMs can be used in general populations to measure broad aspects of latent traits.^2,8,9 Generic PROMs presumably have higher generalizability but at the cost of containing items that are irrelevant for specific conditions (content relevance) and vice versa lack items related to aspects of the construct relevant for the specific condition (content coverage). Generic PROMs might therefore have low content validity in settings for specific conditions. Condition-specific PROMs are developed for a specific group or condition and capture elements of traits that are relevant in the specific context. When condition-specific PROMs are used outside of their intended context or generic PROMs are used without proper pre-testing, the validity and reliability of measurement can potentially become compromised.⁹ The PROMs will then measure inaccurately or have low power to detect the specific trait, and thus findings are questionable.¹⁰ Despite these concerns, the use of generic PROMs for specific conditions and condition-specific PROMs outside of their validated context is pervasive.^9,11,12 Arguably, many factors potentially drive this practice: 1) the ease of using available PROMs, thus bypassing the extensive work required for development and psychometric testing, 2) the comparability of findings to research that uses the same PROM, and 3) a spiral effect, where PROMs are used so frequently to assess specific traits in clinical and research settings, that it becomes a dogma that the PROM is valid and reliable even though it has never been tested in a relevant target population. Another widespread practice is the use of shortened versions or subscales of frequently used PROMs which also expose measures to the risk of poor validity and reliability.¹¹

This research tendency or methodological unawareness have both scientific and practical implications. Scientifically, studies that use PROMs with inadequate measurement properties can produce biased effect estimates and hence evidence of low quality. Further, reviews and meta-analyses that are based on studies that use inadequate PROMs will not provide a higher rank of evidence. The scientific implications may in turn lead to practical implications. For instance, systematic reviews and meta-analyses are important for policymaking and an essential component of the practice of evidence-based medicine.¹³ Systematic reviews and meta-analyses that build on studies using inadequate PROMs might cause practices or policies to be implemented or changed based on biased effect estimates. Consequently, such evidence syntheses, can have negative consequences for patients, care providers, and society, and result in ineffective and harmful policies and interventions. From an ethical viewpoint, this is especially important to keep in mind in regard to screening as it involves the general population and not a group of patients.¹⁴ As a countermeasure to these concerns, different guidelines and checklists have been developed to assess the quality of PROMs, thus promoting valid and reliable measurement in research.^15–19 These guidelines and checklists define the quality of a PROM, which should then be defining for the trustworthiness of the results.

The aim of this study was to systematically assess studies that measure the psychosocial consequences of colorectal cancer screening using PROMs and review their methodological quality.

Methods

Prior to the initial search, we uploaded a protocol for this systematic review to the International Prospective Register of Systematic Reviews (PROSPERO) on November 17, 2016 (Registration number CRD42016051608).²⁰ The conduct and reporting of this systematic review have followed the Cochrane Handbook,² the PRISMA checklist,²¹ and relevant methodological literature.^13,16

Approach and Eligibility

We included all empirical research that studied aspects of psychosocial consequences of colorectal cancer screening.²⁰ Studies were included regardless of the PROMs that was used to assess the outcome (Appendix 1). Studies that only reported, for example, anxiety as a single item were excluded. We restricted the inclusion to studies that reported on an average screening population, in other words, adults (+18) that did not have any known risk factors of colorectal cancer. Eligible study designs included randomized controlled trials, cohort studies, case-control studies, prognostic studies, and qualitative development or validation studies. We did not perform any restrictions regarding study groups, screening settings, follow-up time, or language. Peer review was not required.

Search Process

Three authors and a librarian scientist developed the literature search. We performed the search in four databases: MEDLINE, CINAHL, PsycINFO, and EMBASE (including PubMed) all on August 22, 2016. We initially developed the search for PubMed and subsequently adapted it to the databases (Appendix 2). We updated the search twice: in August 21, 2019 and November 23, 2021.

Two authors independently screened studies at title, abstract, and full-text levels according to the pre-defined eligibility criteria (Appendix 1). ²⁰ Discrepancies were resolved by discussion. The last author was consulted if consensus could not be reached.

Two authors independently assessed the reference lists of included studies to identify additional studies not found in the systematic search (Snowballing). We kept track of systematic reviews and likewise scrutinized reference lists for relevant literature.

Data Extraction and Synthesis

Data extraction was pre-specified in the protocol.²⁰ Data extraction included study design, setting and population, content validity, statistical psychometric measurement properties, and information about the PROM. Two authors extracted the data independently, and discrepancies were resolved by discussion. When consensus could not be reached, the last author was consulted. Authors of the included publications were contacted when necessary, for example, if the methodology or use of the PROM were unclear or in case of missing data.

The COSMIN Checklist

We used the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) Risk of Bias checklist to systematically evaluate the quality of measurement properties in PROMs.^15,22 Risk of bias refers to whether the results of the study are trustworthy in regard to methodological quality.^15,16 In accordance with the COSMIN checklist, two authors independently assessed the PROMs.^15,16 Each domain was assessed by several items rated as “Very good”, “Adequate”, “Doubtful”, or “Inadequate”. Some items could only be graded dichotomously: “Very good” or “Doubtful”.¹⁵ For some standards, “Not applicable” (N) was also an option. The COSMIN checklist grades PROMs according to the principle of “worst score counts” because poor methodological aspects cannot be compensated by good aspects.²³ Any disagreements were resolved through discussion, and in case of non-consensus, the last author was consulted. The COSMIN checklist consists of 10 domains and respective subdomains (Table 1). ¹⁵

Table 1 The COSMIN Checklist

The first domain covers the development of the PROM, which includes an overall assessment of the construct and appropriateness of cognitive interviews or pilot testing. The second domain covers the content validity and the degree of patient and professional involvement in the PROM development. This should be evaluated for the context in which the PROM is used. This domain was evaluated based on three aspects: content relevance, content coverage (comprehensiveness), and understandability (comprehensibility). These two domains have been emphasized as the most important properties of a PROM.^{15,16,23–26}

The rest of the domains and subdomains were evaluated based on design requirements, use of specific statistical methods, and an assessment of design and methodological flaws. Regarding the dimensionality of the factors, the COSMIN checklist considers whether the factor structure is validated by exploratory factor analysis (EFA), confirmatory factor analysis (CFA), or item response theory (IRT) models, as well as the adequacy of the sample size. The COSMIN checklist assesses internal consistency through the calculation of Cronbach’s alpha or omega. Measurement invariance is confirmed if the researchers have performed differential item function (DIF) analysis or CFA.

We evaluated PROMs based on all the studies that used them and based on their respective method sections and references. If no references were provided, we searched PubMed for original development or validation studies. This approach gave the grading of the PROMs the benefit of the doubt, as we are aware that poor methodological reporting does not always equals poor quality.

The COSMIN checklist is a modular tool and the boxes can be used separately. If a review only focuses on elements of PROM quality or if not all measurement properties are assessed for the respective PROM, it is not necessary to complete the whole checklist.^15,16 We categorized PROMs as condition-specific if they were either developed in the context of cancer or screening. If the PROM was condition-specific, then we used box 1. If not, this box was skipped, and the PROM was immediately rated “Inadequate” in regard to PROM development (Table 1). We continued the grading if the overall score was “Adequate” or “Very good”.¹⁵ If not, the grading was concluded. This approach applied for each of the ten domains. According to the COSMIN taxonomy, if the study did not report on a domain or measurement property, the respective boxes were skipped (Table 1).^15,23

Data Synthesis

As pre-specified in the protocol, we anticipated a wide spectrum of PROMs and thus limited scope for meta-analyses. If several PROMs had adequate measurement properties and comparable study designs or subgroups, we would perform meta-analyses. The COSMIN guideline for systematic review of PROMs recommends that evidence is graded according to the GRADE approach if studies’ results are analyzed or pooled.¹⁶

Results

After the removal of duplicates, we overall identified 13687 unique publications whereof 33 were included for review (Figure 1). We excluded 68 studies at full-text level, mostly due to wrong outcome or design (Appendix 3).

Figure 1 PRISMA flow diagram.

Study Characteristics

The majority of the included studies were observational studies (88.6%) and used a control group (74.3%). All studies were conducted in high-income countries: one study was conducted in Taiwan (2.9%), while the rest was from European countries (77.2%), Australia (11.4%), or The United States (8.5%). Most studies reported on adults aged 50–80 (94.3%), despite two studies that included all adults older than 20 or 40 years, respectively (Table 2). Most studies reported on the impact of a positive FOBT (42.9%) or invitation to screening (25.7%) (Table 2).

Table 2 Study Characteristics

Most studies defined their primary outcome as psychological consequences (69.7%), the rest aimed to measure psychosocial consequences (12.1%), quality of life (3.0%), or both quality of life and psychological impact (15.2%) (Table 2). The 33 included studies used 30 different PROMs. Eleven of the PROMs (36.7%) were condition-specific (counting all versions of Psychological Consequences Questionnaire (PCQ)), and 16 of the 33 studies (48.5%) used one of these. Consequences Of Screening – ColoRectal Cancer (COS-CRC) and Cancer Worry Variables (CWV) were the only two measures (6.7%) developed in both a colorectal cancer and screening setting. The studies generally used multiple PROMs with an average of 2.1 (Range 1–4).

Because of the wide spectrum of outcome measures and study designs, we argue that there was no scope for meta-analyses. Therefore, we conducted descriptive data synthesis with focus on the methodological aspects of included PROMs.

The COSMIN-Checklist Grading

The COSMIN grading is presented in the two tables below (Tables 3 and 4). Inter-rater reliability of COSMIN grading was 93%.

Table 3 Quality of the PROM Development

Table 4 Quality of the Content Validity

The quality of PROM development is presented in Table 3. Most condition-specific PROMs had a clear description of the construct of interest they aimed to assess and of the target population. Only one PROM, COS-CRC, used appropriate construct theory (origin of construct); all other PROMs received the lowest possible score “Doubtful” on this item. PROMs generally received low scores due to a lack of appropriate qualitative data-collection methods to identify relevant items and use of cognitive interviews. For example, both COS-CRC and CWV were developed in the context of screening and colorectal cancer, yet only COS-CRC was qualitatively tested in the context of colorectal cancer screening.

Most studies had sparse methodological reporting and did not reference any development- or validation studies and were thus graded “Inadequate”. Except for Worry Variables, all condition-specific PROMs had one or more scores of “Very good”. All PROMs, except COS-CRC, received the lowest possible score for the total assessment. COS-CRC was overall graded “Doubtful” because two researchers did not code the qualitative data independently (Table 3).

We assessed the content validity for all PROMs whether they were condition-specific or not (Table 4).

The majority of the PROMs had inadequate involvement of patients and professionals in the development phase. While COS-CRC and CWV were the only two measures developed for both colorectal cancer and screening setting, CWV did not have any patient involvement in the development phase. Three PROMs were overall graded “Doubtful”: Health Assessment Questionnaire (HAQ) short-form, EuroQual 5 Domains (EQ-5D), and COS-CRC. COS-CRC was the only PROM to ever receive the best score “Very good” in three out of five subdomains. COS-CRC was overall graded “Doubtful” due to a lack of proper involvement of professionals (Table 4).

No PROMs were assessed beyond this domain as none were overall rated as “Adequate” or “Very good”. The low grading was mainly due to the fact that studies had not sufficiently adapted the PROM to a colorectal cancer screening context.

Discussion

Summary of Findings

This review included 33 studies that all together used 30 different PROMs to measure psychosocial consequences of colorectal cancer screening. Only eleven of these PROMs (36.7%) were developed in the context of either cancer or screening, and only two in both (6.7%). Studies generally used multiple PROMs, while less than half included one that was condition-specific. According to the COSMIN checklist, 29 out of 30 studies (96.7%) had inadequate PROM development. PROMs generally lacked proper cognitive interview and pilot studies. Across all PROMs, three (10%) had doubtful and 27 (90%) had inadequate measurement properties, due to lack of patient and professionals involvement in the development phase.

According to the COSMIN manual, we chose not to conduct meta-analyses, as PROMs generally had inadequate measurement properties and pooling results in one analysis assumes high content and construct validity. By extension, we neither graded the evidence according to GRADE or COSMIN criteria.^61,62

Comparison to Existing Literature

Using PROMs with inadequate or unknown measurement properties or outside of their intended context has been criticized before.^8,12,63–66 Already in 2004, Brodersen et al argued that the General Health Questionnaire (GHQ), the State-Trait-Anxiety-Inventory (STAI), and the Hospital Anxiety and Depression Scale (HADS) were not adequate in any cancer screening context.¹² Despite these critiques, this questionable use of PROMs remains partly unchanged. A 2002 study illustrates these concerns; post-hoc analyses showed that the generic Short-Form 36 (SF-36) had limited validity as an outcome measure of health status after stroke.⁶⁷ The authors highlighted the importance of testing scale assumptions before applying outcome measures to new populations. This study also highlights another concern; that this issue is not unique to cancer screening. For example, a review on PROMs used in sports science showed that about one-third of included studies used PROMs in other contexts than they were intended for.¹¹ The same was found in a review on PROMs used in Randomized Controlled Trials (RCTs) in sports medicine.⁸ Even though a PROM is validated in one context, there is no guarantee that it is valid in another context. Previous studies have reported a misalignment between patients’ interpretation, and hence response, and the PROM’s intended meaning.^68,69 This misalignment could stem from a lack of proper cognitive interviews or pilot studies.²⁴ A scoping review found that only 6.7% of included PROMs had proper patient involvement in the development phase and suggests that future research should base their choice of PROM on the level of patient involvement.²⁵ A finding compatible with our findings. One of the reviews found that when PROMs are used to evaluate conditions that they were not developed for, estimates are biased toward null.⁸ This is potentially due to inaccuracy of measurement or low power; lack of content validity and thereby responsiveness. This might also be the case in the included studies of this review, and their respective effect estimates should be evaluated with these biases in mind.

In this review, the grading of PROMs was hindered by poor methodological reporting. Poor methodological reporting on the use of PROMs was also the case in reviews on shared decision-making,²⁴ pain,⁶⁵ and PROMs used in RCTs.⁸ This lack of emphasis on methodological reporting also speaks to the lack of attention on PROM development and validity.

Previous reviews on psychosocial consequences in colorectal cancer screening have not assessed the adequacy of measurement properties or quality of PROMs.^70,71 However, based on included studies, van der Velde et al conclude that no psychological impact was sustained three months after a false-positive colorectal cancer screening.⁷⁰ This interpretation is questionable as it relies on the quality of the PROMs, which was not assessed and estimates are likely to be biased towards null when based on generic or non-specific PROMs.⁸ A review by Selva et al reported that only 7 out of 75 identified PROMs used to measure experience or satisfaction with colorectal cancer screening were (self-reported) validated.⁷²

Strengths and Limitations

Initially, we aimed to synthesize the psychosocial consequences of colorectal cancer screening. As we did the formal screening of results, we found that this was not feasible due to number of different PROMs and the varying quality of these. Therefore, we chose to change the main outcomes of our review to the methodological quality of PROMs instead of the psychosocial harms themselves as specified in the original protocol (Appendix 4). Another change was made from the original protocol concerning methods, because the COSMIN checklist was published in the meantime and we wanted to conform to best available methods (Appendix 4).

In this study, a PROM was considered condition-specific if it was developed within a context of either cancer or screening. Our definition was very inclusive to give semi-condition-specific PROMs the benefit of the doubt regarding COSMIN grading. However, what is relevant in the context of cancer patients will not always be relevant for apparently healthy citizens participating in colorectal cancer screening – and will most likely not cover all aspects of psychosocial consequences.^42,73 We did not assess the PROM development for PROMs that were considered non-specific (Box 1). A generic or non-condition-specific PROM can indeed be well-developed, yet that does not mean that it is adequate in a context of colorectal cancer screening. If these non-condition-specific PROMs were evaluated in box 1, it would seem that they were more adequate in the context than they are. Nevertheless, If PROMs had at least adequate content validity, the quality of the PROM development could be relevant for the use of the PROM. Ideally, if researchers wish to use a non-condition-specific PROM, they should conduct a content-validity study in the population of interest. However, if such a study is not conducted or if such information is not available, the PROM development could have some value.

Although the COSMIN checklist is one of the most rigorous and widespread tools for evaluating the adequacy of PROM measurement in health research, the approach has some limitations. First, the COSMIN checklist is a standardized checklist which means that it does not allow for subjective assessment of the specific PROM. The standardized checklist assumes that the domains weigh equally in the overall grading of the PROM. For example, in the context of psychosocial consequences of colorectal cancer screening, we would argue that patient involvement is far more important than the involvement of professionals. The majority of PROMs included in this review failed to involve a proper number of professionals from relevant disciplines, but the importance of this specific item should potentially be downgraded. The COSMIN checklist requires ≥7 professionals for qualitative studies and ≥50 for quantitative studies to reach the grade “Very good” and ≥4 or 30 professionals, respectively, for “Doubtful”. This quality-quantification of the qualitative development might falsely grade PROMs better or worse than they actually are. Acknowledging these problems, the COSMIN group has amended a number of standards since the first introduction of the checklist and recommends that the checklist is used as guidance.^15,16

Limitations of the COSMIN checklist regarding assessment of the dimensionality has previously been discussed in Heiberg Agerbeck et al 2021 and McKenna and Heaney 2021.^26,65 Researchers have also argued for the importance of construct theories in the development of PROMs, which the COSMIN checklist only sparsely emphasizes.^26,74 Other researchers have argued that the COSMIN checklist does not take into account that quality in the development phase can differ across health outcomes.²⁶

We graded according to the principle of benefit of the doubt, but we cannot rule out that PROMs might be of better or worse quality than what is graded here. Conclusively, we encourage research that use the COSMIN checklist, to evaluate the risk of bias beyond the checklist, for example, in regard to composite measures, unidimensionality, and biases relevant for the specific health outcome.

Additional Findings

While conducting this review, we noticed additional aspects of the included PROMs beyond the outcomes defined in the protocol.

Only seven articles (21.2%) across three author groups discussed the limitations of the PROMs they used.^42–46,59 However, this reporting was generally deficient and did not focus on the implications for results.

Almost half of the studies (42.2%) used at least one shortened form of a PROM (Table 2). To use only a part of a questionnaire threatens content coverage and construct validity. Using a short-form has an underlying, often implicit, assumption that the short-form can be a surrogate for the full measure. A statement that should be tested qualitatively or statistically. However, none of the studies that used short-forms had tested whether this was valid in their respective population.

We also noticed a tendency to produce composite outcomes. Summating scales into one composite measure can conceal true changes in the outcome. Individuals might score higher on one domain, lower on another compared to baseline, and thus these changes will be balanced out when scores are summated across domains. McKenna and Heaney argue that composite scales, by principle, should be considered low quality.²⁶

Implications for Research and Practice

This review sheds light on the trustworthiness of studies that use inadequate outcome measures and calls for reevaluating the existing evidence on the psychosocial consequences of colorectal cancer screening. The magnitude of measurement bias should be evaluated for each PROM individually.

PROMs should be tested in the population of interest prior to measurement. If the resources are not available, then researchers will have to explore alternative methodologies. Patient involvement is crucial in PROM development to ensure concordance between patients’ interpretation and intention, and to gain high content validity. Therefore, this part cannot be left out in proper PROM design. Future grading of PROMs should account for the bias beyond the COSMIN checklist, for example, in regard to composite measures and unidimensionality.

For existing evidence, policymakers, researchers, and clinicians will need to beware of the poor quality and resulting potential biases, so that real-life practices are not affected accordingly. If policies or medical practices are changed as a consequence of biased research it might have unintended harmful implications that can affect patients as well as professionals. Scientific journals should preferably not publish studies that use inadequate outcome measures or should at least demand disclosure of the limitations when doing so.

Unanswered Questions and Future Research

This review focused on the methodological quality of the use of PROMs to measure psychosocial consequences in a colorectal cancer screening context. Future reviews could focus on the overall quality of evidence of the studies’ designs and how the quality of measurement affects estimates of psychosocial consequences. Such a review should follow the COSMIN manual for systematic reviews of PROMs and use the GRADE approach and give overall quality ratings of PROMs.

Conclusion

This review included 33 studies that used 30 different PROMs. Studies generally used multiple PROMs, yet less than half included one that was condition-specific. According to the COSMIN checklist, 29 out of 30 PROMs had inadequate PROM development and 27 had inadequate measurement properties. Conclusively, the majority of PROMs used to study psychosocial consequences of colorectal cancer screening have no proven content validity in this context. This grading of methodological quality should be used in the overall grading of the evidence. Evidence of methodological quality should as well be a defining factor for the trustworthiness of studies that report on the psychosocial consequences of colorectal cancer screening.