A Systematic Analysis of Reviews Exploring the Scope, Validity, and Reporting of Patient-Reported Outcomes Measures of Medication Adherence in Type 2 Diabetes

Introduction

Medication adherence (MA) can be broadly defined as the extent to which patients take their medicines as prescribed in line with agreed recommendations by a healthcare professional.¹ Medicines are the most widely used healthcare intervention globally, yet the World Health Organisation (WHO) estimates approximately only 50% of individuals living with a chronic condition are adherent to their treatment.¹ As a result, significant economic costs are incurred to healthcare providers with estimates ranging in excess of €1.25 billion to $290b annually across parts of Europe and the USA, respectively.^1,2 One condition of particular interest with respect to non-adherence is type 2 diabetes.

The most recent prevalence data estimates approximately 8.5% of the global population are affected by diabetes, with the vast majority (>95%) reporting a type 2 diagnosis.³ Despite the availability of a broad range of therapeutic agents to manage the condition, including oral antiglycaemic (OA) medicines and insulin, diabetes was the ninth leading cause of death globally in 2019.^4,5 Numerous studies have specifically implicated poor adherence to type 2 diabetes with higher hospital admission rates, increased length of hospital stay as well as poorer outcomes, including mortality, during hospital admission.^5–7

To address the outcomes associated with non-adherence, it is important to quantify and define a patient’s medication-taking behaviour. Several techniques have been designed to capture MA behaviour, however one particular method of interest comes with the development of Patient-Reported Outcome Measures (PROMs). PROMs are tools or surveys designed to capture patient-reported outcomes using a self-report approach to data collection.⁸ They have seen increasing popularity over the last two decades owing to their often-simple design and ease of implementation that has led to extensive application across fields, such as health service research, pharmaceutical development, and surgical/medical outcome assessments.^9,10

The Morisky Medication Adherence Scale (MMAS-4/8),¹¹ the Summary of Diabetes Self-Care Activities (SDSCA),¹² and the Medication Adherence Rating Scale (MARS-5/10)¹³ are but a few examples of validated PROMs specifically designed to assess MA that have demonstrated strong psychometric properties. However, it should be noted that PROMs are also subject to moderately high variability when comparing their ability to measure MA even within the same population, potentially limiting their reliability.^14,15 Furthermore, as Nguyen et al¹⁶ highlight, there are few conditions where a clear link between specific levels of MA and clinical outcomes has been established despite the available evidence to demonstrate the psychometric validity of PROMs. In fact, several of the available and widely implemented PROMs of MA have not been validated against either clinical outcomes or other direct methods of measuring MA.^16,17 This suggests a potential lack of standardization in terms of validation and reporting within the literature, a finding similarly discussed in a recent 2021 review exploring the trends and issues associated with PROMs.¹⁷

Churruca et al¹⁷ identified that the phrase “patient-reported outcome” is a somewhat novel term used in the description of PROMs and related measures, hence using this taxonomy to capture relevant PROMs can be challenging. Furthermore, the review highlights the significant disparities between studies in what can be considered the minimum threshold for determining the validity of a PROM based on various criteria, such as face, content, and construct validity as three such examples. Methods such as the COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN),¹⁸ introduced in 2010, have been developed with a view to provide a framework for addressing the methodological quality of PROM studies, but also as an assessment criterion for systematic reviews (SRs) exploring PROMs and their validity. The COSMIN framework has seen increasing use since its conception in 2010; however, the extent to which it has been implemented for PROMs or SRs focused on MA has yet to be explored.

Previous SRs have sought to capture the range of PROMs or evaluate specific PROMs used in type 2 diabetes, however there is little to no commentary focused on the standardization of how these PROMs are reported or assessed for validity.^19,20 Therefore, to broaden the scope of data collection, this review opted to conduct an analysis of SRs exploring PROMs of MA in type 2 diabetes. This approach was taken to capture the widest range of PROMs whilst aiming to overcome previously cited concerns around the impact of taxonomy on reporting. This SR also set out to explore the use of validity criteria as part of PROM reporting, including a review of COSMIN implementation throughout the literature.

Materials and Methods

Literature Review

This SR was conducted in January 2022 in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) 2020 guidelines.²¹ To identify all relevant SRs prior to analysis, a search was conducted using the following databases: PubMed, EMBASE, CINAHL, Cochrane Database of Systematic Reviews and Web of Science (WOS). A sample search strategy was adapted for each database, including MeSH terms where applicable (Supplementary File 1) to identify SRs reporting on PROMs of MA in people living with type 2 diabetes. The filter “Review” or “Systematic Review” was applied to extract the intended type of the literature as part of the screening procedure from each database where appropriate. Any peer-reviewed SR that met the sample search criteria, was published in English and available up to and including December 2021, was considered eligible for abstract screening. References to the included SRs were reviewed for incidental inclusion of other eligible studies as part of the screening process.

Selection of Systematic Reviews

Several additional inclusion criteria were applied to the SR selection process. SRs that included additional comorbidities, including type 1 diabetes, were considered eligible if a distinct breakdown of study populations was provided so that the use of a PROM could be identified in a specific cohort, eg the number of participants diagnosed with type 2 diabetes that used a PROM within a study was clearly described if more than one comorbidity cohort was included. Studies involving only paediatric populations were excluded; however, the same principle was applied to SRs that included studies with participants both over and under the age of 18; in this example only SRs that clearly delineated studies between adult and paediatric populations were included. Studies examining MA to OA and/or insulin or related pharmacological agents were reviewed. SRs that included studies not published in English that met the above inclusion criteria were considered eligible. Moreover, only SRs that clearly attributed an eligible PROM to a specific included study were considered relevant for systematic analysis. Protocols were not considered eligible for inclusion.

Two researchers (JW and PC) conducted independent title and abstract screening as per the outlined inclusion and exclusion criteria. Full-text publications were reviewed using the same method. Discrepancies at either stage of the review were presented to a third researcher (RK) who made a final decision on inclusion of the SR.

Systematic Analysis

Several criteria, in addition to broad descriptions of the included SRs and their objectives, were assessed as part of the systematic analysis and broken down into three major domains that covered the types of PROMs included in each review, assessment of validity and a reflection on the reporting of eligible PROMs within each SR (Figure 1).

Figure 1 Flow chart of the review assessment framework. Abbreviations: PROM, patient-reported outcome measure; MA, medication adherence; SR, systematic review; COSMIN, COnsensus-based Standards for the selection of health status Measurement Instruments.

Individual studies were examined as part of the analysis, with only those specifically reporting PROMs of MA in people living with type 2 diabetes considered eligible as part of data synthesis. Furthermore, only PROMs that directly explored quantifiable differences in MA in terms of medication-taking behaviour were considered valid, eg relevant items may include “I forget to take my medication” or “rate how many days in a week you take your medicine”. PROMs that explored correlated factors of MA, such as treatment satisfaction or empowerment as two examples, but without items directly assessing MA were not considered eligible as part of the systematic analysis. Identified PROMs were then synthesized to reflect the scope of tools used across a range of included SRs.

The assessment of PROM validity was explored for each SR as well as for each individual study within the included SRs. Validity reporting was considered relevant if at least one of the following factors had been explored as part of the SR: internal consistency, reliability, measurement error, content validity (including face validity), construct validity (including structural validity, hypotheses testing and cross-cultural validity), criterion validity, responsiveness, and interpretability. In addition, except for one, all included SRs were published after 2010, the year in which the COSMIN checklist was published.¹⁸ Therefore, reporting of the COSMIN checklist across the included SRs was also examined to determine uptake and implementation. It should be noted that not all SRs included in the analysis had either a primary or secondary objective relating to the evaluation of measurement properties or specific selection of a PROM and hence the COSMIN checklist may not apply. However, this finding was deemed relevant as part of this review’s commentary given the value that standardized research frameworks can provide in the way research is both conducted and reported to an international audience. For each individual study referenced within an included SR, an additional assessment was conducted to examine the implementation and validity of the reported PROM. Data were derived from this process to reflect the overall assessment of PROM validity within each SR as a percentage of eligible individual studies.

The final domain assessed as part of the systematic analysis focused on the approach taken to reporting of PROMs. Although tools such as the COSMIN checklist exist, there is still limited evidence to assess the extent of PROM reporting standardization within the literature. Therefore, prior to the systematic analysis, several criteria were determined as part of a rudimentary framework to assess the reporting of PROMs in this review and are described below:

1. Taxonomy
   • Has the language of the PROM been defined if translated or used in a target language different from the original PROM?
   • Has the correct nomenclature of the PROM been reported in the SR?
   • If more than one variation of the reported PROM exists, has a specific version of the PROM been distinctly defined when reported in the SR?
2. Accuracy
   • Has the PROM reported in the SR been used at all in the referenced study?
   • If a PROM reported in the SR has been used in the referenced study, was it the correct PROM that was reported?
   • If the study provided evidence of self-reported adherence data, was the PROM used to collect the data declared?

As with the validity assessment, data were derived to reflect the overall approach to PROM reporting within each SR as a percentage of eligible individual studies using the rudimentary framework developed for the analysis.

Where applicable, data were derived from the SCImago Journal database to collate quartile rankings for each SR included in this study. The non-parametric correlations were assessed using Spearman’s Rho (r) to explore any associations between impact factor and the prevalence of validity or reporting issues. Values of p<0.05 were considered to be significant.

Results

A total of 2074 records were identified in the initial review phase. After the removal of duplicates (n = 289), abstract and title screening was conducted, leading to the removal of 1863 records prior to full-text screening (n = 75). An additional five eligible records were identified for full-text screening via citation searches of previously reviewed records, leading to a total of 80 full-text records to assess. A total of 19 SRs were identified as eligible for inclusion and systematic analysis. Figure 2 provides a breakdown of the PRISMA diagram screening process including details of reasons for exclusion.

Figure 2 PRISMA 2020 flow diagram. Notes: *From: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi: 10.1136/bmj.n71. For more information, visit: http://www.prisma-statement.org/.

Across the 19 included SRs, a total of 1283 studies were reported from which 241 were identified as eligible for analysis post-screening (Table 1). Within the 241 studies, a total of 228 eligible instances of PROM reporting were identified. It should be noted that although 241 studies met the inclusion criteria for the broad validity and taxonomy analyses, not all reports provided sufficient data for individual PROM analysis; hence, the number of studies and number of PROMs are incongruent. Furthermore, some studies reported >1 eligible PROM for analysis. The number of reported eligible PROMs was reduced when excluding duplicates through consolidation of collated data across the 19 SRs to produce a list of 104 unique PROMs – language variations of the same tool were treated as distinct PROMs as part of the analysis. Almost half (n=9/19, 47.4%) of the SRs were published in the last 5 years and included studies dating from 1990 to 2020 providing a 30-year scope of the literature. The qualitative characteristics of the SRs have been defined in Table 2. The objectives of SRs varied; however, they were predominantly split between two groups, firstly those assessing factors that impact MA (n=6/19, 31.6%) and secondly studies evaluating the impact of MA interventions (n=6). Only two (n=2/19, 10.5%) SRs specifically excluded studies with patients using insulin.

Table 1 Quantitative Characteristics of Included Systematic Reviews

Table 2 Qualitative Characteristics of Included Systematic Reviews

Table 3 Reporting Assessment of PROMs

Of the 104 identified unique PROMs (Supplementary File 2) eligible for analysis, the MMAS-4 was the most widely implemented throughout the eligible reports (n=35/228, 15.4%). Notably, both the MMAS-4 and MMAS-8 were the most widely translated with 17 confirmed adaptations for each PROM, respectively (Supplementary File 3).

At the SR level, there were three studies^20,22,23 (n=3/18, 16.7%) that implemented the COSMIN checklist and only four studies^19,20,22,23 that reported PROM validity criteria (n=4/19, 21.1%) (Figure 3). When evaluating each of the individually reported PROMs, a quarter (n=58/228, 25.4%) of studies reported PROMs without adequate evidence of validity as defined by this review (Supplementary File 2). Reporting on non-validated PROMs was identified in 63.2% (n=12/19) of the included SRs (Figure 3). The most common reason for the lack of validity was the translation of a PROM into a new target language. Translation was considered valid at a minimum if the PROM had been forward and backward translated with the support of native speakers from the target language(s) and at least one relevant validation criterion had been reported, eg internal consistency. Alternatively, if a translated tool was used, a reference pertaining to a previous study where the minimum validity evidence had been demonstrated was required. The analysis identified 47 (n=47/58, 81.0%) instances where a PROM had been translated without the minimum evidence threshold for validity being met. PROMs that had been validated in other target conditions were used in seven studies (12.1%). Similarly to the assessment of PROM translation, the application of the PROM within a type 2 diabetes cohort was only considered valid if at least one validation criterion was reported as evidence for cross-cultural validity if previously validated in a different target condition. Finally, the remaining four studies (6.9%) assessed as lacking validity were those that provided no evidence of target language for the study population whilst using a PROM that did not have a translated adaptation relevant to that population, eg MMAS-8 (originally designed in English) was used amongst a population in Ethiopia, however no evidence of translation or cohort ability to speak English was defined to assess the validity of the PROM.²⁴

Figure 3 Systematic review assessment criteria. Notes: The COSMIN criteria were introduced in 2010, which excluded one systematic review from the analysis due to its publication prior to 2010, hence n = 18 for this sample. A total sample of n = 19 was used to assess the other criteria included in this figure.

When assessing the overall eligible studies against the pre-determined framework for reporting of PROMs defined in this review, almost half (n=114/241, 47.3%) of the included studies were identified with issues (Table 3). Only two SRs^22,23 (n=2/19, 10.5%) met the reporting framework standards for all of their included studies (Figure 3). Taxonomical inconsistencies relating to the target language or distinction of PROM adaptations were the most prevalent issues identified when assessed against the reporting framework (n=84/114, 73.7%). Of those studies identified with issues, almost 10% (n=11/114, 9.6%) were reported using an incorrect PROM compared to the primary literature used in the SR. For example, at least three SRs^25–27 reported the use of the SDSCA as a measure of MA for studies identified in their reviews, however on closer examination of the source literature, the MA items of the SDSCA had been omitted in each study and alternative PROMs had been implemented, such as the Brief Medication Questionnaire (BMQ).²⁷ In addition to the reporting of incorrect PROMs, a further seven studies (n=7/114, 6.1%) failed to define and declare the use of validated PROMs despite having a methodology and available evidence to do so. In some instances, despite reporting on PROMs at the SR level, some individual studies were described as using “self-report” measures of MA, when in fact a PROM such as MMAS-4 had been used as a valid MA measure in the source literature.²⁸

Quartile data for publication quality were available for 89.5% (n=17/19) of the included SRs ranging from 1 to 3. When compared with the prevalence of studies identified with validity issues, a significant moderate correlation was observed (r=0.44, p=0.04) indicating that an increase in reported validity issues was associated with a higher quartile ranking and conversely lower impact factor. No significant association was observed between quartile ranking and the prevalence of reporting issues identified in the systematic analysis (p>0.05).

Discussion

To our knowledge, this is the first study to systematically analyse SRs reporting on PROMs of MA in type 2 diabetes with an additional focus on understanding the validity of included PROMs and the reporting of PROM studies within the literature. By taking a broad approach to inclusion of SRs, the collation and synthesis of data provides a wide scope of evidence spanning the last three decades. Furthermore, although only SRs published in English were included, individual studies within each SR were included irrespective of the target language, hence this SR reflects literature at a global scale including low- and middle-income countries, which are often disaggregated for specific studies and under-represented in health research.²⁹

Previous work by Kwan et al²³ provides arguably the most comprehensive SR currently available in the literature, identifying 121 unique PROMs of MA across all clinical conditions. Although the SR did provide reference to different languages as characteristics of the PROMs identified in the appendix, these were not considered as independent tools. Conversely, this systematic analysis did independently classify PROMs and adaptations, producing a total of 104 tools used specifically in type 2 diabetes. Though neither approach is necessarily designated as the “gold standard” for reporting, the results of this analysis identified the vast majority (>80%) of studies with validity concerns that were related to translated adaptations. It is evident that construct equivalence between translations was assumed in some cases of reporting. This approach, which was identified in over 60% of the included SRs, may jeopardize PROM validity. As Hawkins et al³⁰ highlight, confirmation of equivalence or invariance is often assessed post-translation using appropriate statistical methods described in previously published guidelines for cross-cultural adaptation (CCA).³¹ It should be noted that a 2015 review³² found no consensus between guidelines for CCA, however it emphasised the need for more evidence of CCA, particularly methodological strategies for translation and assessment of psychometric properties. To this extent, this systematic analysis emphasizes the importance and potential impact of describing and distinguishing translated adaptations of PROMs in both the primary literature and SRs to improve transparency of CCA, or lack thereof, and identify potential validity issues when reporting PROMs.

Beyond validity, issues such as taxonomy and inaccurate interpretation of PROMs captured as part of the systematic analysis also demonstrate the need for clearer guidance or standardization with respect to reporting. A recurring example includes misinterpretation of the SDSCA. The original tool was developed in 1994 by Toobert¹² and Glasgow, followed by a review of the SDSCA literature and revision to the scale in 2000³³ that notably recommended removal of items pertaining to MA due to significant ceiling effects. Despite this, the reference for the revised scale was cited across several SRs^{22,23,25,26,34–38} and often reported generically as the SDSCA, when in fact various iterations of the PROM were used including a single medication subscale (SDSCA-MS), a 5-item adaptation (SDSCA-5), as well as the original tool (SDSCA) and revised version (SDSCA-Revised). Furthermore, the PROM was subject to language adaptations with and without validity being determined, as well as three SRs^25–27 where the SDSCA was cited as the PROM to record MA, however closer inspection of the primary literature demonstrated that the MA items were not included. This scenario presents several issues. In line with recommendations drawn from the COSMIN¹⁸ criteria, individual subscales are in fact independent PROMs that measure defined constructs with discrete psychometric properties. Studies that therefore implement adaptations and interpret their results (often devoid of reliability or validity data) without due care may fail to address the high variability associated with PROM outputs.^14,15 Furthermore, without consistent reporting or identification of PROMs, future reviews may fail to adequately collate literature pertaining to a specific measure(s) to synthesise aggregated data. This was a specific issue identified as part of this analysis where despite the incredibly broad range of included studies from both a geographical and historical perspective, a number of relevant PROMs were omitted across all SRs. A notable example includes the German adaptation of the SDSCA (SDSCA-G).³⁹ The study provides strong evidence of the psychometric properties of the PROM and hence would have met the inclusion criteria as a valid measure of MA in type 2 diabetes as defined by this systematic analysis.

Previous work by Wee et al²⁰ succinctly states that the breadth of data is itself a limitation with respect to identifying and assessing the numerous applications and iterations of PROMs reported in the literature. Standardised methodologies could certainly help to tackle the variation in practice, yet it was interesting to note that only three of those studies published after 2010 referred to the COSMIN criteria.¹⁸ Not all SRs specifically reported on PROM measurement outcomes; however, there was a high prevalence of validity/reporting issues and a significant correlation between quartile ranking and validity concerns. Bibliometrics, such as quartile ranking, should not be used as proxy indicators of research quality in isolation, but the authors recognize that the moderate relationship (>0.4)⁴⁰ identified in bivariate correlation testing was of interest in the wider context of validity reporting in this systematic analysis and warrants further investigation in future work.⁴¹ It seems relevant to emphasize that improved implementation of COSMIN and/or related guidelines may help to address the issues identified in this review by providing a standardized framework for PROM validity assessment and reporting. To this effect, the authors propose three recommendations based on the findings of this systematic analysis that they believe would benefit PROM research at both an individual study and SR level:

1. Adapted translations of PROMs should be distinctly categorized from the original PROM and reported as such. Construct equivalence should not be assumed unless a clear methodology for translation is described, and relevant post-translation psychometric properties are reported. If the PROM was translated into a previous study, the validity of the methods used to translate the PROM should be assessed using these criteria prior to citation of the study.
2. Adaptations of PROMs that affect subscales and/or items should be distinctly categorized from the original PROM and reported as such. Relevant psychometric properties of the adaptation should be reported unless the analyses were conducted in a previous study, whereby the these methods should be assessed to determine the validity of the adaptation using these criteria prior to citation of the study.
3. Where relevant and available, reference to an appropriate guideline for reporting or determining psychometric properties of PROMs, such as the COSMIN criteria,¹⁹ should be clearly cited in the study or SR. If no relevant guideline or framework is followed, a statement to justify exclusion may be appropriate to support the interpretations of the study/SR results and their reliability for external audiences

The scope of this SR review may be limited due to the exclusion of SRs not published in English. In addition, the baseline criteria for validity reporting were quite low meaning that perhaps further validity issues would have been identified if a more stringent approach had been taken. This SR only focused on PROMs of MA that focused directly on medicine-taking behaviour and not other determinants of MA behaviour such as treatment satisfaction or beliefs about medicines, hence future work should look to include these types of PROMs as part of the evaluation.

Conclusion

This systematic analysis provides an extensive insight into the broad current landscape of PROMs of MA used in patients living with type 2 diabetes.The authors hope that researchers will look to access this review when considering the implementation and suitability of PROMs in future studies in addition to considering the methodological issues highlighted in the analysis when developing PROM-related study protocols. To our knowledge, this is the first SR to take such a granular approach to PROM classification based on language and scale adaptations, which was then used as part of a novel framework to evaluate validity and reporting of PROMs. Moreover, this review has provided a centralised quantitative evaluation of translations and adaptations of some of the most widely used PROMs available for assessing MA, which included most notably the MMAS-4 and MMAS-8 with 17 translations identified for each adaptation, respectively. The systematic analysis also identified a high prevalence of issues relating to PROM validity and reporting that were largely related to CCA. Future research may therefore benefit from standardized methodologies to assess and report PROM validity with three clear recommendations provided by the authors within this SR. The significant relationship between journal quartile ranking and the number of observed validity issues among published studies was another notable finding that warrants further exploration with a larger sample to establish more conclusive evidence of a correlation. Furthermore, to our knowledge, this is the first review to quantify and identify the poor uptake of the COSMIN criteria with respect to PROMs used in patients living with type 2 diabetes. Further studies should look to address PROMs that explore other drivers of MA as well as investigate alternative clinical conditions, such as COPD, cancer or mental health diagnoses.