Comparing a Conceptual Framework and Factor Analysis to Achieve Survey Item Reduction in Predicting Medication Non-Persistence

Introduction

An immense body of research has uncovered numerous possible risk factors for medication non-adherence.^1,2 Triangulating all these factors into a single prediction model is challenging. Researchers must account for complex domains such as beliefs,³ literacy,⁴ self-efficacy,⁵ socioeconomic status,⁶ and treatment satisfaction.⁷ Each of these domains often demands multiple survey questions for their valid measurement. As a result, adherence surveys can be lengthy^8–11 and item reduction strategies are often employed to address power limitations or examine specific themes during analysis. Two main strategies have been used to guide item reduction: conceptual approaches or factor analysis. In conceptual approaches, items are categorized into pre-defined domains.^12,13 Medication adherence domains are well recognized in the literature. The World Health Organization (WHO) has identified five dimensions of adherence (ie, Social/economic, Therapy-related, Patient-related, Condition-related, and Health system/health care team factors).¹

Factor analysis (FA) is an alternative approach to categorizing survey items into domains.¹⁴ The key difference with FA (versus conceptual approach) is that it identifies domains exclusively from data correlations rather than pre-existing paradigms. Both conceptual approaches and FA can be used for the same purpose, yet they have the potential to produce vastly different groupings from the same set of survey items. To our knowledge, the extent to which domains identified with FA categorization align with pre-defined categories by the WHO is unknown for medication adherence surveys. Furthermore, if items are categorized differently with FA, it is unclear downstream prediction models are affected. From a single adherence survey dataset, we performed item reduction using both approaches (ie, conceptual approach vs FA) and compared logistic regression models from each approach on the outcome of non-persistence.

Materials and Methods

This study used a cross-sectional adherence questionnaire entitled Major Determinants of Non-adherence in Saskatchewan (MDNAS) developed by our research group.^8,15 The paper questionnaire, consisting of 58 questions (51 adherence items), was mailed by post to new medication users across the province of Saskatchewan between September 16, 2019, and February 28, 2020. Details about the survey have been previously published.^8,15 Almost 4,000 surveys were completed and returned; previous studies from the MDNAS survey have explored the interactions between side effects, expectations, and beliefs on the risk for self-reported non-persistence.^8,15 We defined non-persistence as an answer of “no” to the question, “Are you still taking the new medicine prescribed to you?”.

For the conceptual approach, the research team organized items under domains from the World Health Organization (WHO) framework for adherence. Some WHO domains were subdivided to help identify coherent groups of items (eg, a subcategory of “beliefs” was created under “patient factors”). A Cronbach’s alpha threshold of 0.6 or higher was used to confirm domains.^16,17 Items not categorized in a domain were retained and analyzed individually.

The second approach employed exploratory factor analysis (EFA) to group survey items based on their underlying relationships, independent of pre-existing frameworks.¹⁸ To verify the dataset’s suitability for EFA, we conducted the Kaiser–Meyer–Olkin (KMO) test and Bartlett’s test of sphericity, with the number of respondents (ie, 3,029) well above the rule of thumb power requirement (N>1000).¹⁹ We employed the Principal Factors method with squared multiple correlations (SMC) as prior communality estimates and applied varimax rotation to optimize the clarity of the factors extracted. This process allowed us to identify distinct domains by focusing on patterns of high correlations among items. Domains were considered significant and retained for further analysis if their eigenvalues exceeded 1.0.²⁰ We further assessed the relevance of each item to its corresponding domain through factor loadings, prioritizing items that demonstrated strong and unique loadings (≥0.4).¹⁸ Next, a confirmatory factor analysis was conducted to validate the identified domains and assess the relationships between the latent factors and their observed variables. Model fit was assessed using various indices, including the Adjusted Goodness of Fit Index (AGFI), Comparative Fit Index (CFI), Standardized Root Mean Square Residual (SRMR), and Root Mean Square Error of Approximation (RMSEA). As above, items not categorized were retained for individual analysis in the subsequent prediction model.

Domains identified from each approach were pooled into a single score (ie, Likert scales were scored from 1 to 4 or 1 to 5 depending on the question). All the summary scores were highly skewed making it impossible to model the raw (ie, continuous) values. Thus, we converted each domain into a binary variable using either the 25% or 50% percentile as boundaries. We originally aimed to apply the 25^th percentile threshold for all summary scores; however, some scores were so highly clustered in the top 50^th percentile it was illogical to divide them. Next, all variables (ie, individual items and pooled) were tested individually against non-persistence and were retained in the final logistic regression model if they: a) demonstrated a significant association with non-persistence on univariable analysis (p-value < 0.05); and b) increased the predictive accuracy of the final model indicated by the Integrated Discrimination Improvement (IDI) value.²¹ Age and sex were forced in both models to ensure their potential impact was quantified. The c-statistic (ie, area under the receiver operator curve)²² was calculated for each model and compared using the “DeLong test”.²³

Survey participants provided written, informed consent and the study received ethics approval from the Research Ethics Board of the University of Saskatchewan (BEH 14–120). The study complied with the Declaration of Helsinki.

Results

Baseline characteristics of MDNAS respondents have been previously reported.^8,15 Half of respondents were female (49.5%), 55.7% were under 65 years old and most of the respondents were white (89.9%). Both approaches identified three domains from the 51 adherence items. Ten items were categorized by the conceptual approach, while 15 were categorized by the FA approach. Only five items were grouped by both approaches [Table 1].

Table 1 Comparison of Two Models Predicting Non-Persistence Using Data Organized by a Conceptual Approach or Factor Analysis

Domains created from the conceptual approach represented the following three themes: Healthcare Provider Support (HCPS), Patient Knowledge, and Patient Beliefs. Cronbach’s alpha scores were 0.80, 0.82, and 0.67, respectively. In contrast, one of the FA domains contained at least one item from all domains of the conceptual approach: two items representing “beliefs” (ie, “importance of medication for health” and “concern the medication will do more harm than good”), one item from “health care provider support” (ie, overall, you trust the doctor), and one item from knowledge (ie, you know what this medicine is used for). [Table 2] Model fit indices indicated good to moderate fit; the Adjusted Goodness of Fit Index (AGFI = 0.88), comparative fit index (CFI = 0.85), standardized root mean square residual (SRMR = 0.06), root mean square error of approximation (RMSEA = 0.09) and the chi-square statistic for the model was 2017.54 (df = 73, p < 0.0001). Cronbach’s alpha scores for the FA domains were 0.79, 0.66, and 0.53.

Table 2 Coefficient of Factor Loading, and Total Variance Explained in Principal Component Analysis

All three domains from the conceptual approach met inclusion criteria for the final model whereas two (out of three) FA domains were included. Both approaches captured four of the same individual items: Age and sex (forced into both models), and two additional items: “Did you experience side effects from your new medicine?” and “You expected to get side effects from this new medicine before you started taking it”. Their associations with non-persistence were similar in both models, including a significant interaction between them. In addition, the FA model retained another item (You expected this new medicine to be difficult to take exactly as prescribed by the doctor’). However, this additional item was not associated with non-persistence after multivariable adjustment. Both models demonstrated good predictive performance with c-statistics of 0.84 (conceptual model) and 0.82 (FA model, p=0.060).

Discussion

We created two prediction models for non-persistence from the same survey dataset; one model contained covariates organized by a conceptual framework,¹ while the other model contained covariates organized by FA.¹⁴ The conceptual approach categorized fewer items into domains, likely due to conceptual boundaries preventing some items from being tested together. However, more effective item reduction with FA is potentially counterproductive if the research goal is to produce a highly effective prediction model. In other words, more grouped variables mean fewer variables accounting for outcome variance. In addition, EFA produced groups of items (ie, domains) that did not align with our conceptual framework. This outcome was felt to be a critical limitation, given the widespread recognition of domains identified from previous medication adherence research.¹

Surveys to assess risk factors for medication adherence continue to provide valuable knowledge to the field. Zhang and colleagues published a study examining risk factors for antihypertensive medication non-adherence among individuals living in Xinjiang, China.²⁴ They identified 9 domains identified using EFA. Although the items contained in these domains crossed traditional categories of non-adherence determinants,¹ the model explained a high percentage of variance and the risk factors identified corresponded to conventional knowledge. We conducted a very similar study in Saskatchewan, Canada. However, we followed a conceptual approach (rather than FA) to identify domains from our lengthy survey.¹⁵ The conceptual approach was critical for our study because we intended to test the construct of patient “beliefs” specifically; we could not “allow” EFA to mix other items within that domain. At the same time, we could not find any information to understand if the conceptual approach would negatively impact the performance of downstream modeling or the identification of individual risk factors. This preliminary study suggests model performance may not be negatively impacted. Also, the most powerful risk factors were identified by both approaches.

Several limitations of our study must be acknowledged. Replication of our study using other survey data would be needed to confirm if our observations are generalizable. Also, our prediction of non-persistence was based on a self-reported outcome; therefore, generalizability of the high discrimination remains to be proven with respect to real-world practice. Bias is inherent in self-reported survey data and poses challenges to accurately capturing participants’ true experiences and behaviors.^25,26 Similarly, constructs such as beliefs and knowledge were measured using tools that lacked external validation. These limitations could impact the reliability, comparability, and generalizability of the findings. Also, our EFA procedure identified a domain with very low internal consistency (ie, Cronbach’s alpha of 0.53). It appears that conventional fit indices did not always produce domains with high internal consistency.^27,28 Similarly, we allowed a domain with a Cronbach’s alpha <0.7 (ie, 0.67) using the conventional approach. Because the primary goal of this study was to compare methods rather than develop a validated tool, we chose to retain these domains to avoid losing potentially important information. It may be imperfect from a statistical standpoint but remains highly relevant to the outcome of non-persistence. Despite these limitations, both models produced c-statistic values above 0.8.

Conclusions

In the area of medication adherence, decades of research and clinical insights have helped to create theoretical paradigms such as the WHO framework.^1,29,30 Organizing items within these pre-existing themes allows focused testing of hypotheses and can produce a strong prediction model. On the other hand, the FA process provides greater opportunity for item reduction, but that may limit prediction performance of downstream models.