Validity Of The Self-Reported Domains Of Subjective Extent Of Nonadherence (DOSE-Nonadherence) Scale In Comparison With Electronically Monitored Adherence To Cardiovascular Medications

Introduction

Medication nonadherence represents a significant challenge to the management of chronic disease. It is both highly prevalent and associated with a wide range of negative outcomes, including increased healthcare spending, hospitalizations, and mortality.^1–11 Tools are needed to measure medication nonadherence in research and clinical settings. Indeed, a recent systematic review concluded that first measuring nonadherence and then targeting adherence interventions at nonadherent patients is a key component of successful interventions to improve medication adherence.¹²

Self-report measures are an attractive option for measuring nonadherence and identifying nonadherent patients, given low cost and feasibility of administering these assessments. It is also possible that patients who are willing to endorse nonadherence may be more amenable to interventions. Yet, no “gold standard” self-report measure exists.¹³ Many of these measures have limited reliability and/or validity,^14,15 in part because they measure both extent of nonadherence and reasons for nonadherence as a unidimensional construct, even though these are separate (but related) considerations.¹⁴

To address the limitations of many self-reported adherence measures, Voils et al^14,16 developed a 2-domain measure comprising 3 items to assess the extent of nonadherence to medications, along with a separate, disease-specific method of capturing reasons for nonadherence, the Domains of Subjective Extent of Nonadherence (DOSE-Nonadherence) scale. In initial work in patients prescribed antihypertensive medications, the extent of nonadherence measure demonstrated good internal consistency reliability (α=0.84)¹⁴ and remained reliable over a 12-month span (αs=0.75–0.90).¹⁷ The extent of nonadherence was also associated with higher blood pressure (rs=0.27, ps<0.001).¹⁴ Additionally, the extent of nonadherence measure demonstrated a strong correlation with the widely used Morisky Medication Adherence Scale (8-item MMAS),¹⁸ another self-report measure of nonadherence. In the study that compared the DOSE-Nonadherence scale and MMAS, only the DOSE-Nonadherence scale was associated with blood pressure.¹⁴ More recently, the DOSE-Nonadherence scale showed high reliability across repeated assessments in a sample of patients taking oral medications for hepatitis C.¹⁹ Furthermore, endorsement of any nonadherence to medications prescribed for dyslipidemia was shown to be related to higher serum total and LDL cholesterol levels,^17,20 and changes in self-reported extent of nonadherence exhibited predictive validity for changes in serum total cholesterol over time.¹⁷ These results are promising for both reliability and validity of the scale.

Thus far, the DOSE-Nonadherence scale has not been compared to other approaches to measuring nonadherence to daily medications (i.e. missed doses). The present study is intended to assess the convergent validity of the DOSE-Nonadherence scale by comparing self-reported nonadherence, in terms of daily missed or skipped doses, to electronically monitored daily adherence to cardiovascular medications in a sample of patients recently evaluated for a suspected acute coronary syndrome (ACS). Electronic monitoring is widely regarded as a gold standard measure of adherence to daily medications because it is more objective than self-report (which may be impacted by, e.g. recall bias or social desirability bias) and because it has shown little-to-no reactivity (i.e. does not impact medication-taking behaviors).^2,7,21 Electronic monitoring may also be ideally suited to evaluating the concurrent validity of the DOSE-Nonadherence scale because, even though the extent of medication nonadherence scale measures ingestion of daily medications and electronic monitoring measures daily bottle openings, both measurement approaches evaluate day-to-day medication-taking behavior (commonly known as regimen implementation).²²

Methods

Design

This paper presents secondary analyses from an observational cohort study (REactions to Acute Care and Hospitalization; REACH) that examines the development and impact of cardiac-induced posttraumatic stress disorder among patients presenting to the emergency department (ED) with a suspected ACS.²³ For the present study, we report cross-sectional analyses comparing self-reported and electronically monitored adherence to cardiovascular medication for a 7-day period that occurred approximately 30 days post-discharge.

Participants

Inclusion criteria for REACH included presentation to the ED of a tertiary care hospital (Columbia-New York Presbyterian Hospital, New York, NY) with a suspected ACS (unstable angina or non-ST elevation myocardial infarction by the treating ED physicians). Exclusion criteria were non-English- or Spanish-speaking and any cognitive or psychiatric impairment that would impede study participation (for further details on REACH design, see Birk et al²³). Participants (N=165) were included in the present study if they completed the self-reported nonadherence scale and if they completed electronic medication adherence monitoring substudy (i.e. participants had to be prescribed aspirin or another cardiovascular medication²³) in the first month after the index hospitalization (the period corresponding with the self-report adherence assessment).

Procedures

Potentially eligible participants were identified via treating emergency medicine physicians and approached by a research assistant in the ED to confirm eligibility, gauge interest, and obtain informed consent. A brief assessment took place in-ED (e.g. demographics) and again a median of 3 days post-ED-discharge (either in-person while inpatient or via telephone at home). At the post-ED follow-up visit, participants prescribed at least one cardiovascular medication were invited to utilize an electronic pill cap (eCAP, Information Mediary Corp., Ottawa, Canada) to monitor their medication adherence. The eCAP, which looks similar to a conventional pill bottle cap and fits on a standard size medication bottle, records the date and time when the cap is opened. Those who agreed were mailed an eCAP-covered medication bottle after discharge from the hospital. They were instructed to place one of their cardiovascular medications, preferably aspirin, in the bottle, and to take the medication in the manner prescribed by their doctor. No other counseling about medication adherence was provided.

Participants completed a telephone interview again at a minimum of 1 month post-discharge. This assessment included the measure of self-reported medication nonadherence with a recall period of the past 7 days. During this phone call, participants were asked to return the eCAP in a stamped envelope that they would soon receive in the mail. Participants did not receive any advice about how to take their cardiovascular medication during this phone call. The study protocol conforms to the ethical guidelines of the 1975 Declaration of Helsinki, and all materials were approved by the Columbia University Irving Medical Center Institutional Review Board (IRB). All participants provided written informed consent before completing study procedures.

Measures

Extent Of Nonadherence: Self-Report

The extent of nonadherence to cardiovascular medication over the past 7 days was measured using the DOSE-Nonadherence scale, which comprises three items scored from 1, “None of the time,” to 5, “All of the time.” Items included: “In the past 7 days, how often did you … miss a dose of your heart medication?” “[…] skip a dose of your heart medications?” and “[…] not take your heart medication as prescribed by your doctor?”^14,16,19 For those using an eCAP, research coordinators substituted “heart medications” with the name of the cardiovascular medication being tracked by the eCAP when administering the extent of nonadherence questionnaire. The DOSE-Nonadherence scale was modified by the study in two ways. The third item added “as prescribed by your doctor” to the end of the question; this change occurred after multiple Spanish-speaking participants requested clarification of the third item during user testing in our study population. The response option “all of the time” was modified from “every time” for consistency with other response scales in the survey.

The questionnaire was translated into Spanish in an iterative process. First, measures were translated from English to Spanish by an IRB-certified translator. Next, translations were reviewed and user tested with a group of experienced bilingual research assistants representing a diverse range of dialects to maximize understanding in all Spanish-speaking populations (e.g. Spain, Dominican Republic, Cuba, Puerto Rico, Colombia, Ecuador). During this review, a final Spanish version was agreed upon through consensus.

The mean of the 3 items was computed to form a continuous assessment of the extent of nonadherence with a range of scores from 1 to 5 (for English-speaking participants, α=0.95; for Spanish-speaking participants, α=0.87). The score was also dichotomized to indicate 0, no endorsement of nonadherence, and 1, any nonadherence (i.e. a response of 2 or greater on any of the 3 items).

Extent Of Nonadherence: Correct Dosing Adherence Via Electronic Monitor

Participants were asked to track one of their daily cardiovascular medications – preferably aspirin for ease of between-patient comparisons – using eCAPs. They were otherwise told to take the medication as prescribed by their doctor. The prescribed frequency of dosing was determined based on a review of the discharge medication list in the electronic medical record. Adherence was calculated as the percent of days a participant opened the eCAPs bottle the correct number of times. For example, if a medication was prescribed once a day and that individual opened the bottle once, then they were considered adherent on that day. In contrast, if a participant opened the bottle twice that day or not at all, then they were considered nonadherent on that day. Data were censored if the participant was in the hospital or had an explained gap (e.g. doctor advised participant to stop taking medication).

Percent adherence for the 7 days prior to the self-report was computed. For example, if a participant opened their bottle the prescribed number of times, as determined by eCAPs, on 7 of 7 days, they were assigned a score of 1.00 (100% taken as prescribed). If a participant opened their bottle the prescribed number of times on only 2 of 7 days, they were assigned a score of 0.29 (28.57% taken as prescribed). Note that participants with any data were included (e.g. if eCAPs had been started only 5 days prior, such that 5 days of data were available, then 5 was the denominator for the percent adherence calculation). Continuous percent adherence scores were also dichotomized such that 0 indicated perfect adherence (i.e. 100% taken as prescribed on the days they were monitored), whereas 1 indicated any nonadherence during the monitoring period (i.e. any percent adherence ranging from 0% to less than 100%). For use in sensitivity analyses, we additionally computed adherence as opening the eCAP bottle at least the prescribed number of times (e.g. opening a bottle twice for a once-per-day medication was considered adherent). “At least” was chosen to correspond to the fact that the self-report measure assesses missed doses only, whereas correct dosing adherence penalizes both missed doses and extra doses.

Data Analysis Strategy

Extent of nonadherence (self-reported and as measured by eCAPs; continuous and dichotomous) was examined using descriptive statistics. For correlational analyses, self-reported nonadherence was reverse coded so that a 1 represented the lowest adherence and a 5 indicated the highest adherence. This was done to facilitate comparisons with the eCAPs measure of adherence. Associations between self-report and eCAPs measures were tested using Spearman correlations due to high levels of skewness in the data.

The sensitivity and specificity of the self-report measure for detecting nonadherence were calculated using electronically monitored adherence of less than 100% as the reference standard. A cutpoint of 100% was selected as it best corresponded to the self-report response options, where an answer of 1 on each item was suggestive of perfect adherence, but any endorsement of a response option 2–5 indicated at least some nonadherence. Sensitivity indicates the ability of the self-report measure to identify “true positives” (i.e. of those who were nonadherent on eCAPS, what proportion screened as nonadherent on the self-report measure?). Specificity indicates the ability of the self-report measure to identify “true negatives” (i.e. of those who were adherent on eCAPs, what proportion screened adherent on the self-report measure?). Positive and negative likelihood ratios (LRs) were also calculated. LRs indicate how a positive (nonadherence indicated) or negative (no nonadherence indicated) screen impacts the likelihood that a patient is nonadherent. Positive LRs (LR+) are calculated as sensitivity/(1-specificity), and negative LRs (LR-) as specificity/(1-sensitivity). A LR+ of more than 10 is considered to have a large effect on posttest probability of being a “true positive” if a patient screens positive on self-reported nonadherence.²⁴ A LR+ of 5 to 10 indicates a moderate effect, and 2 to 5 a small effect (1 indicates no change). Similarly, a LR- of less than 0.1 is considered to have a large effect on post-test probability of being a “true negative” if a patient does not report nonadherence, 0.1 to 0.2 a moderate effect, and 0.2 to 0.5 a small effect.²⁵ Analyses were conducted separately for English- and Spanish-speaking patients.

Results

Demographic characteristics for the N=165 included in the present study can be seen in Table 1. Participants were a mean of 61.87 years old (SD=11.16), 54.55% were male, and 41.21% spoke English as a first language. The most commonly electronically monitored medication was aspirin (70.91%), and the most common dose frequency was once per day (92.12%; the rest were twice per day).

Table 1 Characteristics Of Study Participants

Mean self-reported nonadherence during the 7 days prior to the 1-month assessment was low (M=1.27, SD=0.62). In other words, participant tended to report that they “missed,” “skipped,” or “did not take medications as prescribed” none of the time (recall that 1=“none of the time”). Only 47 participants (28.48%) reported any nonadherence. Mean continuous adherence as measured by eCAPS during the 7 days prior to the 1-month assessment (i.e. percent bottle openings, as prescribed) was 66.56% (SD=36.29%). In other words, the eCAPs bottle was opened the correct number of times on 66.56% of days measured. More than half of the participants (61.82%) were considered nonadherent (i.e. incorrect number of openings on at least 1 day).

Continuous Measures Of Adherence

The continuous measure of self-reported nonadherence to cardiovascular medications was correlated with the eCAP-measured adherence. Recall that self-report was reverse coded for these correlations such that a positive correlation indicates convergence between the measures. For the 68 English-speaking participants, this correlation was, r=0.24, p=0.046, and for the 97 Spanish-speaking participants it was, r=0.18, p=0.071.

Sensitivity Analysis

When substituting the correct dosing adherence measure with an electronic measure in which opening the bottle “at least” the prescribed number of times was adherent (i.e. not counting days on which participants took extra doses as nonadherent days), the correlation of self-report with eCAPs was slightly lower for English-speaking participants, r=0.22, p=0.067, but slightly higher for Spanish-speaking participants, r=0.24, p=0.016.

Dichotomous Measure Of Adherence

Proportion of participants detected as 100% adherent by eCAPs stratified by self-reported adherence can be seen in Figure 1. Details regarding sensitivity, specificity, and positive and negative LRs can be seen in Table 2. Participants who self-reported at least some nonadherence were more likely to be nonadherent by electronic monitoring than those who self-reported being adherent. Sensitivity was lower than specificity in both English-speaking participants (0.47 versus 0.77) and Spanish-speaking participants (0.28 versus 0.88).

Figure 1 Comparison of adherence classification by the 3-item extent of nonadherence questionnaire with electronically monitored adherence, stratified by participant language. Note: Medications were electronically monitored using eCAPs (English speaking, n=68; Spanish speaking, n=97).

Table 2 Sensitivity, Specificity, And Likelihood Ratios (LRs) Of The Extent Of Nonadherence Questionnaire For Detecting Medication Nonadherence

Sensitivity Analysis

When substituting the correct dosing adherence measure with an electronic measure in which opening the bottle “at least” the prescribed number of times was adherent, sensitivity and specificity were 0.48 versus 0.74 for English-speaking participants, respectively, and 0.33 versus 0.89 for Spanish-speaking participants.

Discussion

The present study provided modest support for the convergent validity of the DOSE-Nonadherence scale with electronically monitored adherence to daily cardiovascular medications as the reference standard in a cohort of patients with suspected ACS. The self-report measure had low sensitivity and moderate specificity for identifying nonadherent patients per electronic monitoring.

The DOSE-Nonadherence scale was developed to address the limited reliability and/or validity of self-report measures that often conflate reasons for nonadherence with the behavior of nonadherence.^14,15 A recent systematic review of these previously developed self-report adherence questionnaires (14 standardized questionnaires, 11 unnamed) found comparable correlations between self-reported and electronically monitored medication adherence in diverse patient populations.²⁶ Also similar to other self-report questionnaires, participants were more likely to report being adherent by self-report as compared to electronic-measurement; only 29% reported at least some nonadherence, whereas 62% were found to have at least some nonadherence by eCAPs. Considering the present study alongside prior research on this self-report scale, which provided evidence for predictive validity regarding health outcomes and response to treatment,¹⁷ the DOSE-Nonadherence scale has demonstrated good statistical properties for research. The fact that this is a brief, 3-item scale that is inexpensive and easy to implement with acceptable specificity may make it desirable to incorporate into research settings.

The sensitivity of the dichotomized DOSE-Nonadherence scale was low (i.e. ability to detect “true positives” from the entire population of nonadherent patients). This is not necessarily surprising given a general tendency of patients to over-report adherence across self-reported medication adherence questionnaires. Furthermore, eCAPs assessed bottle openings, not actual ingestion of medications, and so correlations could be attenuated by the lack of conceptual overlap regarding the behavior being measured (i.e. self-reported ingestion vs the behavior of bottle openings). There may also be other factors contributing to the lack of correspondence, such as social desirability bias, inability to recall behavior (particularly if cognitive load is high), or any other number of cognitive processes.

Although the scale was poor at identifying all participants who were nonadherent by electronic monitoring, 72.0% of English-speaking participants and 81.8% of Spanish-speaking participants who endorsed any nonadherence were also nonadherent per the electronic monitor. Thus, this scale may be useful for identifying nonadherent patients in clinical settings. In addition, the fact that these patients disclosed nonadherence may make them more amenable to adherence interventions. Pairing the extent of nonadherence scale items with items to assess reasons for nonadherence,^14,27 as the DOSE-Nonadherence scale was designed,²⁷ may identify targets for adherence interventions.

There were a number of strengths of our analyses. We rigorously measured electronic adherence in a diverse patient population inclusive of those with low socioeconomic status. We additionally computed a measure of electronic adherence for use in sensitivity analyses that did not count patients as nonadherent if they took extra doses. There were also some limitations. eCAPs measure bottle openings, and it is possible that participants did not actually ingest the medication when bottles were opened. Alternatively, participants may have taken doses without using the eCAP. Results may not generalize to patients prescribed other types of medications, to regimens that comprise multiple cardiovascular medications, or to patients recruited outside of urban ED settings. Future studies should explore whether self-report scales are best used to measure adherence to a single medication or to regimen adherence more broadly.²⁸ The DOSE-Nonadherence scale focuses on missed doses; extra doses or wrong timing of doses may be important as part of adherence measurement for other self-report adherence questionnaires. It is possible that 30-day adherence more accurately reflects long-term adherence, however, in cognitive interviews conducted for scale construction and item generation, participants felt that asking about adherence over the past 7 days led to more accurate recall and greater sensitivity to nonadherence.¹⁴ To measure long-term patterns, ideally, the DOSE-Nonadherence scale should be assessed repeatedly. The scale was developed using English speakers and the Spanish version may have been interpreted differently by participants. The DOSE-Nonadherence scale had lower reliability and validity in the Spanish version of the present study. Future research is needed to refine and further validate this scale in Spanish-speaking populations. The sample size for both samples was modest, which limited power for precise determination of diagnostic utility. The DOSE-Nonadherence scale was modified slightly from its original version in response to questions from study participants during user testing in our study population; rather than asking about “not taking” doses according to the original version, participants were asked about not taking doses “as prescribed by your doctor.” Thus, extrapolation of the current findings to use of the original questionnaire should be done cautiously. However, although “as prescribed” could refer to numerous aspects of dose administration (e.g. time of day, food instructions, etc.), the cardiovascular medications assessed (such as aspirin) come with few dosing requirements. Furthermore, this wording is in parallel to our eCAPs measure of correct dosing adherence, and results did not change in sensitivity analyses when patients were not penalized for taking extra doses.

Conclusion

Our analyses demonstrate that the extent of nonadherence domain of the DOSE-Nonadherence scale has modest convergent validity with a behavioral measure of medication adherence, providing support for the usefulness of this tool as a measure of medication adherence in research settings. Future studies should assess the convergent validity of this measure versus behavioral measures of regimen implementation in other patient populations and should evaluate the usefulness of this tool in clinical settings in larger samples.