Back to Journals » Substance Abuse and Rehabilitation » Volume 10

Measuring recovery in opioid use disorder: clinical utility and psychometric properties of the Treatment Effectiveness Assessment

Authors Ling W , Nadipelli VR , Solem CT , Farabee D , Ronquest NA , Perrochet B, Learned SM, Deshpande CG, Heidbreder C

Received 15 December 2018

Accepted for publication 6 May 2019

Published 5 June 2019 Volume 2019:10 Pages 13—21

DOI https://doi.org/10.2147/SAR.S198361

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Professor Li-Tzy Wu



Walter Ling,1 Vijay R Nadipelli,2 Caitlyn T Solem,3 David Farabee,1,4 Naoko A Ronquest,2 Brian Perrochet,1,5 Susan M Learned,6 Chinmay G Deshpande,3 Christian Heidbreder7

1UCLA Department of Family Medicine, Center for Behavioral and Addiction Medicine, Los Angeles, CA, USA; 2Global Health Economics and Outcomes Research, Indivior Inc, Richmond, VA, USA; 3Patient-centered outcomes Center of Excellence, Pharmerit International, Bethesda, ML, USA; 4Department of Population Health, School of Medicine,, New York, NY, USA; 5Marron Institute of Urban Management, Marron Institute, New York University, New York, NY, USA; 6Global Medicines Development, Indivior Inc, Richmond, VA, USA; 7Global Research and Development, Indivior Inc, Richmond, VA, USA

Purpose: The Treatment Effectiveness Assessment (TEA) is a patient-centered instrument for evaluating treatment progress and recovery from substance use disorders, including opioid use disorder (OUD). We assessed the TEA’s reliability and validity and determined minimal clinically important differences (MIDs) in participants with moderate to severe OUD.
Patients and methods: The TEA measures change in four single-item domains (substance use, health, lifestyle, community involvement) from treatment initiation across the duration of a treatment program. Self-reported responses range from 1 (“none or not much”) to 10 (“much better”) with items summed to a total score ranging from 4–40. We assessed floor and ceiling effects, internal consistency, test-retest reliability, known-groups validity (ANOVA stratified by current health status [36-Item Short Form Health Survey item 1]), convergent/divergent validity, and MIDs using data from a phase 3, open-label clinical trial of buprenorphine extended-release monthly injection for subcutaneous use (BUP-XR). Participants with OUD completed the TEA at screening and before monthly injections for up to 12 months.
Results: Among 410 participants (mean age 38 years; 64% male), the mean baseline (pre-injection 1) TEA total score was 25.4 (SD 9.7), with <10% of participants at the measure floor and 10%–20% at the ceiling across domains. Internal consistency was high (Cronbach’s α=0.90), with marginal test-retest reliability (intraclass correlation coefficient =0.69). Mean TEA total score consistently increased from baseline (n=410; mean 25.4 [SD 9.7]) to end of study (n=337; 35.0 [6.7]) and differentiated between current health status groups (P<0.001); it was weakly correlated with other measures of health-related quality of life/severity. MIDs ranged from 5–8 for the TEA total score across anchor- and distribution-based approaches.
Conclusion: The TEA exhibited acceptable reliability and validity in a cohort of participants with moderate to severe OUD treated with BUP-XR. Given its brevity and psychometric properties, the TEA is a promising tool for use in clinical practice and research.

Keywords: Treatment Effectiveness Assessment, TEA, opioid use disorder, patient-reported outcomes, addiction, psychometrics

Introduction

Most studies evaluating the treatment of substance use disorders (SUDs), including opioid use disorder (OUD), focus on measuring retention and abstinence, paying little attention to recovery. Retention and abstinence alone may not adequately describe a person’s recovery journey toward regaining and maintaining a meaningful and useful life.1,2 Tools clinicians can use to easily measure gains in life activities that promote and help sustain comprehensive recovery in people with SUDs are lacking.

The Treatment Effectiveness Assessment (TEA),3 designed to address this need, is a brief instrument used to assess recovery outcomes in people with SUDs receiving treatment. The TEA assesses outcomes through the use of questions within four major dimensions involved in recovery: substance use (eg, drugs, alcohol, tobacco), health (eg, physical and emotional health), lifestyle (eg, housing or living situation, family, employment, relationships), and community (eg, obeying laws and becoming a responsible member of society).

The TEA was included as a patient-centered measure of recovery from OUD in a 12-month open-label safety study (NCT02510014)4 evaluating buprenorphine extended-release for monthly subcutaneous injection (CIII) (BUP-XR, SUBLOCADETM, Indivior Inc), which was shown to be efficacious and safe for the treatment of moderate or severe OUD.5 As this was the first use of the TEA in an OUD phase 3 clinical trial to our knowledge, we assessed the reliability, validity, and responsiveness of the TEA in people receiving BUP-XR for moderate to severe OUD. Additionally, we determined the minimal clinically important differences (MIDs) in TEA total scores for reference in future studies.

Patients and methods

Study design

This multicenter, phase 3, open-label, long-term safety study (NCT02510014)4 was conducted in accordance with International Council for Harmonisation Good Clinical Practice guidelines and US Food and Drug Administration (FDA) regulations governing clinical study conduct. An institutional review board reviewed and approved the study protocol, amendments, informed consent form, and all other appropriate study-related information. All patients provided written informed consent.

Following study enrollment (screening), participants began a 3-day induction period with buprenorphine/naloxone sublingual film (2–24 mg); buprenorphine/naloxone doses were adjusted for up to 11 days, followed by a 48-week BUP-XR treatment period. On day 1 of the open-label study, participants received BUP-XR 300 mg. Subsequent doses (300 mg or 100 mg at investigator's discretion) were administered every 28 (−2/+4) days. All participants also received manual-guided individual behavioral therapy. While the long-term safety study itself included participants who rolled over from the randomized placebo-controlled trial of BUP-XR, the current analysis only focused on those participants who were newly enrolled (de novo) within the safety trial.

Measures

The TEA measures improvement associated with treatment along four single-item domains: substance use, health, lifestyle, and community. Scores range from 1 (“none or not much”) to 10 (“much better”), resulting in a TEA total score ranging from 4–40. The TEA allows respondents to consider the characteristics within the domains that are most important to them in evaluating how their lives have changed since initiating treatment. Within the trial, the TEA was completed at initial screening and monthly before BUP-XR injections 1 (ie, BUP-XR baseline) through 12 and at the end-of-study visit. As the majority of analyses required that the participant completed the baseline assessment, only participants who completed the TEA at baseline were included in the TEA analysis for psychometric assessment.

Other key measures used in the study to monitor clinical status and symptoms included the opioid craving (OC) visual analog scale (VAS) and the Addiction Severity Index-Lite (ASI-Lite).69 Measures of health status and health-related quality of life (HRQoL) included the 5-level EuroQol 5 dimensions instrument (EQ-5D-5L),10 which is a generic measure of health status, and the 36-Item Short Form Health Survey, Version 2 (SF-36).11

Statistical analysis

Statistical analyses were undertaken to assess the psychometric properties of the TEA. The proportions of participants indicating each level of response were calculated to assess floor and ceiling effects12 at baseline for TEA single-item domains.

Tests of reliability included an assessment of internal consistency (the extent to which items constituting a scale measure the same concept) as well as test-retest reliability (a measure of whether the instrument would give consistent scores over time). Internal consistency of the TEA total score was assessed at baseline using Cronbach’s alpha. Test-retest reliability for TEA single-item domains and total score was examined using intraclass correlations between responses at BUP-XR injection 12 and end of study (approximately 1 month apart) among people who had stable urine drug screen results between the two visits (ie, either both opioid positive or both opioid negative). These time point visits were chosen based on the expectation that most improvement related to pharmacotherapy should occur early on and then remain more stable. A threshold of 0.7 for Cronbach’s alpha13 and intraclass correlation coefficients (ICCs)14 was used to evaluate reliability.

Also assessed were known-groups validity (ie, the extent to which scores can distinguish among groups hypothesized a priori to be different), convergent validity (ie, whether two different scores measuring similar concepts were associated), and divergent validity (ie, whether different concepts were not associated). Known-groups validity at baseline was assessed using 1-way ANOVA to determine whether the TEA total score and/or domains were able to distinguish between participants who rated their current health status as excellent, very good, good, fair, or poor based on SF-36 item 1 (current health status). Convergent and divergent validity for baseline TEA domain and total scores were assessed by correlating these scores with EQ-5D-5L index and VAS scores, SF-36 mental and physical component scores, ASI-Lite problem areas, and OC VAS scores at baseline. We anticipated moderate negative correlations between the TEA substance use domain and the ASI-Lite alcohol use and drug use problem areas and moderate positive correlations with OC VAS; moderate positive correlations between the TEA health domain and EQ-5D-5L index and VAS scores and SF-36 scores; and negative correlations between the TEA health domain and ASI-Lite medical and psychiatric status problem areas; moderate negative correlations between the TEA lifestyle domain and ASI-Lite social/family problem areas; and moderate negative correlations between the TEA community domain and the ASI-Lite legal status and community problem areas.

The TEA’s ability to detect change was evaluated by comparing change in TEA scores to change in SF-36 item 1 (current health status), evaluated from screening to baseline. The rationale for this window was that previous analyses of the same population15 had observed the most sizeable treatment effect on the SF-36 in this window, and it was anticipated to be similarly associated with improvement in effectiveness as measured by the TEA. The participants were grouped as those who improved by ≥1 category in their current health status, had no change in their current health status (ie, same health status at screening and baseline), or declined by ≥1 category in their current health status based on SF-36 item 1. The corresponding score on the TEA total scale was evaluated to assess whether the TEA measure would be sensitive to changes in current health status.

Additionally, the MID estimates for the TEA total score were calculated using multiple techniques to understand the range of potentially important differences. Distributional approaches included calculating half the SD at baseline (0.5 SD) and the standard error of measurement (computed as SD × √ [1-reliability]) as estimates of the MID. The change from baseline on TEA score was calculated among participants in whom ≥80% of their weekly urine drug screens and self-reports of opioid use were both negative between BUP-XR injection 2 (study week 5) and end of study as an additional indication of MID.1618 Treatment success was a key secondary efficacy endpoint of the pivotal phase 3 trial and was defined as ≥80% opioid abstinence during weeks 5 through 24.5 This endpoint was included because patients can be clinically successful despite occasional lapses in abstinence. The MID was also evaluated using ROC curves that assess what change in TEA total score from baseline to end of study would most discriminate between participants who achieved the same ≥80% threshold of negative opioid use, balancing sensitivity and specificity.19

Results

Among 412 de novo participants who were enrolled into the open-label safety study, 410 (99.5%) completed the baseline TEA assessment and formed the cohort for this analysis. The mean age was 38.4 years; 63.7% of participants were male and 71.5% indicated their race as white (Table 1).

Table 1 Baseline descriptive characteristics

Item distribution

The mean (SD) TEA total score before BUP-XR injection 1 (following up to 14 days of sublingual buprenorphine) was 25.4 (9.7). At this time point, less than 10% of participants were at the floor of any domain (ie, responded with 1 [“none or not much”]), and 15%–25% were at the ceiling of the measure (ie, 10 [“much better”]) across domains (Figure 1). All response levels were used in all domains.

Figure 1 Item distribution and floor-ceiling effects at baselinea. (A) TEA Item 1: substance use. (B) TEA Item 2: health. (C) TEA Item 3: lifestyle. (D) TEA Item 4: community. Note: aGiven that there are ten response options for each item, an item may be subject to a floor or ceiling effect if more than 10% (=100/the number of response options) of responses are in the highest or lowest response categories. Abbreviation: TEA, Treatment Effectiveness Assessment.

Reliability

The TEA exhibited strong internal consistency (Cronbach’s α=0.90), which was retained when each item was removed and internal consistency was re-calculated (Table 2). When considering the test-retest reliability among the 177 participants who had stable urine drug screen results at the last 2 visits and a TEA collected at these time points, the ICC for the total score approached 0.7, with CIs overlapping this threshold (ICC 0.69; 95% CI 0.60–0.76; Table 2).

Table 2 Measures of Treatment Effectiveness Assessment (TEA) reliability and validity

Validity

The TEA total score demonstrated differentiation between known groups as described by current health status (SF-36 item 1; ANOVA-based P<0.001; Table 2). For convergent and divergent validity, TEA correlations with ASI-Lite domains, SF-36 physical and mental component scores, and EQ-5D-5L index and VAS scores were uncorrelated or weakly correlated, but in the anticipated directions (Table 2).

Ability to detect change

The TEA instrument was sensitive to change in current health status (SF-36 item 1; Table 2). Participants whose health status had improved had a 13.5-point increase in TEA total score compared to an 11.5-point increase in those indicating no change in health status and an 8.7-point increase in those who reported a decline in health status. Similar trends were observed at a domain level.

MIDs

MIDs for the TEA total score were highly dependent on the method used. When using the distribution-based method and ROC analysis, MIDs ranged from 4.8–8.0. Among participants who reached 80% or more abstinence as an anchor, the mean MID for the TEA total score was 6.8 (Table 3).

Table 3 Summary of MID estimates for TEA total score

Discussion

The US FDA supports including the patient voice within medication development, as evidenced by the inclusion of patient-reported outcomes in a 2018 draft guidance document describing endpoints demonstrating effectiveness in people receiving medication-assisted treatment. The TEA is an easy-to-use clinical tool that clinicians can use to monitor a patient’s recovery during treatment for an SUD. Results from our analysis demonstrate that the TEA has acceptable levels of reliability, validity, and ability to detect changes in a cohort of people with moderate to severe OUD treated with BUP-XR. While we did not do a formal testing of factor structure in this particular analysis, the relatively high Cronbach’s alpha across the items would point to the high inter-relationship between the single-item domains. The purpose of the TEA is to provide a simple tool in which each area can be measured over time. Additionally, we determined the MID for change in the TEA total score, further informing health care providers on what constitutes a clinically meaningful change for this instrument.

Our results broadly support using the TEA to evaluate improvements associated with OUD treatment. Floor and ceiling effects were consistent with expectations; items exhibited strong internal consistency, illustrating that these items work together; and scores were able to discriminate between participants self-reporting excellent, very good, good, or fair/poor health. Furthermore, we observed that participants self-reporting improvement in health status showed greater improvements in TEA total scores as compared to participants who had no change in their health status or with declined health status. However, in interpreting these results it should be noted that while the SF-36 current health status variable would be expected to be associated with changes in the TEA health domain (and therefore TEA total score as well), this anchor does not capture global improvement as it may relate to treatment effectiveness in SUDs.

A challenge in evaluating measurement properties within this cohort was the lack of a clear anchor for change. This affected our ability to anchor measures of meaningful change and identify a group of clinically stable participants for test-retest reliability. Because of this, we used change in health status (during a window when health change was expected) as a proxy for assessing the ability to detect meaningful change. However, this anchor was imperfect as treatment effectiveness and recovery from OUD go far beyond health. This was evidenced by an improvement in participants’ TEA total scores, including improvement in participants with declining health status, consistent with expectations, as this measurement occurred when participants were first inducted onto pharmacotherapy treatment and improvement was expected.

Our test-retest reliability results overlapped with the 0.7 threshold for reliability; however, the point estimate ranged from 0.609 (community domain) to 0.708 (health domain) despite selecting a time period when participants would be expected to be more stable and limiting measurement to those with stable urine drug screens between the two visits. For assessing MIDs, data obtained from participants achieving at least 80% abstinence were used as an anchor; this may be a high threshold for establishing meaningful change in the TEA and changes in TEA total scores under 7 points may still be meaningful. Confirming these results in future studies with a global indicator of change is warranted.

When assessing convergent and divergent validity, the direction of correlations between the TEA and other outcomes was consistent with expectations, but the correlations were low. It is likely that the TEA is capturing different constructs than measures of the EQ-5D-5L and SF-36, which focus on HRQoL. The TEA may also differ from the ASI-Lite, which focuses on disease severity, whereas the TEA measures person-specific measures of treatment effectiveness. These differences should be considered when weighing the comparative evidence for the TEA's validity. Future research may also be warranted to explore correlations with other patient-centered measures in SUD, particularly those that move beyond a simple abstinence-based definition and encompass a more complete picture of the recovery process.2022 The TEA was developed to implicitly incorporate patient perspectives, as patients are solely responsible for determining which characteristics within the domains are most important to them in evaluating how their lives have been affected by treatment. However, if administered during a first treatment visit, the TEA results must be judiciously interpreted because the instrument asks about changes for the better since beginning a treatment program. Similarly, care should be taken when the possibility exists that a given program or treatment modality may cause decline instead of improvement, as the tool is worded to measure changes for the better. Future studies may be needed to explore the effects of widening this scale to include a version for assessment of baseline status, and/or providing wording specific to measuring change from a fixed time point.

Additional limitations of this analysis include that it was a secondary analysis of participants within a long-term open-label clinical trial and was not a psychometric study on its own; this may impact the generalizability of stated results. Future comparisons of the TEA with other patient-centered instruments could help elucidate aspects of the instrument’s utility in measuring recovery from OUD and other SUDs.

Conclusion

Pragmatic tools such as the TEA are needed to quantify the patient experience in clinical practice and translate it into a format that provides meaningful, evaluable data that can benchmark person-specific improvement over time in clinical research and in real-world treatment settings. Equally important is that participants are reminded that life changes, as concretely assessed by the TEA, are critical steps in their pathway to recovery. Results from this study document the reliability and validity of the TEA in a cohort of participants with moderate to severe OUD and provide preliminary benchmarks for assessing meaningful changes in TEA scores. Further research is needed to evaluate the reliability, validity, and responsiveness of the TEA in real-world settings, and to identify person-centered recovery-related outcomes that may be strongly correlated with TEA scores.

Acknowledgments

The authors wish to thank all participants and investigators who took part in the long-term safety study for BUP-XR (NCT02510014). We wish to thank Beth Lesher for her medical writing support. This study was funded by Indivior Inc.

Disclosure

Walter Ling is a consultant for Indivior Inc, Braeburn, Opiant, and Titan Pharma. David Farabee is a consultant for Indivior Inc and has received study medications from Alkermes. Vijay R Nadipelli, Susan M Learned, and Christian Heidbreder are employees of Indivior Inc. Naoko A Ronquest was an employee of Indivior Inc when this work was developed. Caitlyn T Solem is an employee of Pharmerit International and is a consultant for Indivior Inc. Chinmay G Deshpande was an employee of Pharmerit International at the time the research was conducted. Vijay R Nadipelli reports personal fees from Indivior Inc, outside the submitted work. Caitlyn T Solem reports grants from Indivior, personal fees from Pharmerit, during the conduct of the study. Christian Heidbreder reports personal fees from Indivior Inc., during the conduct of the study, personal fees from Indivior Inc., outside the submitted work. The authors report no other conflicts of interest in this work.

References

1. Betty Ford Institute Consensus Panel. What is recovery? A working definition from the Betty Ford Institute. J Subst Abuse Treat. 2007;33(3):221–228. doi:10.1016/j.jsat.2007.06.001

2. Substance Abuse and Mental Health Services Administration. SAMHSA’s working definition of recovery. Substance Abuse and Mental Health Services Administration 2012; Available from: https://store.samhsa.gov/shin/content/PEP12-RECDEF/PEP12-RECDEF.pdf. Accessed July 27, 2018.

3. Ling W, Farabee D, Liepa D, Wu LT. The Treatment Effectiveness Assessment (TEA): an efficient, patient-centered instrument for evaluating progress in recovery from addiction. Subst Abuse Rehabil. 2012;3(1):129–136. doi:10.2147/SAR.S38902

4. Indivior Inc. Safety and tolerability study of depot buprenorphine in treatment seeking subjects with opioid use disorder. March 29, 2018; Available from: https://clinicaltrials.gov/ct2/show/NCT02510014. Accessed April 16, 2019.

5. Haight BR, Learned SM, Laffont CM, et al. Efficacy and safety of a monthly buprenorphine depot injection for opioid use disorder: a multicentre, randomised, double-blind, placebo-controlled, phase 3 trial. Lancet. 2019;393(10173):778–790. doi:10.1016/S0140-6736(18)32259-1

6. Cacciola JS, Alterman AI, McLellan AT, Lin YT, Lynch KG. Initial evidence for the reliability and validity of a “Lite” version of the Addiction Severity Index. Drug Alcohol Depend. 2007;87(2–3):297–302. doi:10.1016/j.drugalcdep.2006.09.002

7. McMillan DE, Gilmore-Thomas K. Stability of opioid craving over time as measured by visual analog scales. Drug Alcohol Depend. 1996;40(3):235–239. doi:10.1016/0376-8716(96)01218-5

8. Tompkins DA, Bigelow GE, Harrison JA, Johnson RE, Fudala PJ, Strain EC. Concurrent validation of the Clinical Opiate Withdrawal Scale (COWS) and single-item indices against the Clinical Institute Narcotic Assessment (CINA) opioid withdrawal instrument. Drug Alcohol Depend. 2009;105(1–2):154–159. doi:10.1016/j.drugalcdep.2009.07.001

9. Handelsman L, Cochrane KJ, Aronson MJ, Ness R, Rubinstein KJ, Kanof PD. Two new rating scales for opiate withdrawal. Am J Drug Alcohol Abuse. 1987;13(3):293–308. doi:10.3109/00952998709001515

10. Herdman M, Gudex C, Lloyd A, et al. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res. 2011;20(10):1727–1736. doi:10.1007/s11136-011-9903-x

11. Ware JE, Kosinski M, Bjorner JB, Turner-Bowker DM, Gandek B, Maruish ME. User’s Manual for the SF-36v2 Health Survey. Lincoln, RI: Quality Metric; 2008.

12. O’Mahony PG, Rodgers H, Thomson RG, Dobson R, James OF. Is the SF-36 suitable for assessing health status of older stroke patients? Age Ageing. 1998;27(1):19–22.

13. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297–334. doi:10.1007/BF02310555

14. Oliva TA, Oliver RL, MacMillan IC. A catastrophe model for developing service satisfaction strategies. J Mark. 1992;56(3):83–95. doi:10.1177/002224299205600306

15. Ling W, Nadipelli VR, Solem CT, et al. Impact of RBP-6000 (Once-Monthly Depot Buprenorphine) on patient-reported outcomes: a long-term study. Poster presented at: The American Society of Addiction Medicine 49th Annual Conference; April 12–15; 2018; San Diego.

16. Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10(4):407–415.

17. Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care. 2003;41(5):582–592. doi:10.1097/01.MLR.0000062554.74615.4C

18. Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008;61(2):102–109. doi:10.1016/j.jclinepi.2007.03.012

19. Froud R, Abel G. Using ROC curves to choose minimally important change thresholds when sensitivity and specificity are valued equally: the forgotten lesson of pythagoras. Theoretical considerations and an example application of change in health status. PLoS One. 2014;9(12):e114468. doi:10.1371/journal.pone.0114468

20. Kelly JF, Abry AW, Milligan CM, Bergman BG, Hoeppner BB. On being “in recovery”: a national study of prevalence and correlates of adopting or not adopting a recovery identity among individuals resolving drug and alcohol problems. Psychol Addict Behav. 2018;32(6):595–604. doi:10.1037/adb0000386

21. Neale J, Nettleton S, Pickering L. What is the role of harm reduction when drug users say they want abstinence? Int J Drug Policy. 2011;22(3):189–193. doi:10.1016/j.drugpo.2010.09.007

22. Neale J, Vitoratou S, Finch E, et al. Development and validation of ‘sure’: a patient reported outcome measure (PROM) for recovery from drug and alcohol dependence. Drug Alcohol Depend. 2016;165:159–167. doi:10.1016/j.drugalcdep.2016.06.006

Supplementary material

Treatment Effectiveness Assessment instrument

Creative Commons License © 2019 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.