Stability of the alcohol use disorders identification test in practical service settings

Introduction

Nearly 17.6 million American adults fit a diagnosis of alcohol use disorder (AUD), while only 6% of those Americans receive treatment.¹ The alcohol use disorders identification test (AUDIT)² is extensively used to screen for AUD across the globe.³ Since 2004, millions of administrations have been given and over 400 studies published; yet, proper AUDIT cutoff score validity has not been determined, with suggestions ranging between 3 and 24 depending on severity.³ Thus, an initial analysis of AUDIT stability in a service setting, at the suggested cutoff of eight signaling brief intervention,² would contribute to the vast literature concerning the AUDIT’s clinical utility.

Currently, health care practitioner referrals to substance use treatment rank among the lowest between all referring agencies.^4,5 In an effort to address the problem and heavy drinking identification in primary care settings, the Iowa Department of Public Health (IDPH) and Substance Abuse and Mental Health Services Administration (SAMHSA) deployed a comprehensive public health approach to screening, intervening, and effectively referring patients with problem substance use in primary health care facilities. The Screening, Brief Intervention, and Referral to Treatment (SBIRT) program uses motivational interviewing, brief intervention, and referral, which may be helpful in improving outcomes in health care settings.⁶ SBIRT is funded in state medical cooperatives and universities throughout the USA.⁷

While SBIRT has been employed in many sites, the evidence-based scientific research has revealed gaps in need of analysis.⁸ Early analyses of SBIRT demonstrated inconclusive results possibly due to design issues and collection error.^6,9,10 Using observational service data, Madras et al⁶ found alcohol use decreased from baseline to follow-up in six of six SBIRT sites using the AUDIT. Conversely, in a systematic review of randomized controlled trials, Saitz et al⁹ found SBIRT brief interventions were not associated with treatment utilization or reduced drinking. While observational and randomized controlled trials cannot be directly compared, findings such as these do not account for the variability in the natural course of drinking behaviors. Thus, further research with AUDIT stability findings accounting for variability of drinking behaviors would contribute to understanding whether the AUDIT is an appropriate instrument for SBIRT. Drinking behavior is a variable trait when involved in substance use intervention. For example, a patient’s drinking behavior will fluctuate based on mood, coping, or intervention. Measures of quality of life (e.g., unhealthy drinking) are subject to the patient’s appraisal, which is less variable in short time frames.¹¹ The test–retest reliability of a measure assumes stability in a behavior. However, when conceptualizing behavior variability, those with high adherence to a trait will demonstrate much greater variability than those demonstrating low trait adherence.¹² Therefore, the stability of a measure, such as the AUDIT, in a real-world uncontrolled setting marks an important contribution to the changing drinking behaviors. Because instrument stability is equal to test–retest reliability at worst,¹³ investigating the stability of the AUDIT would help better understand patient behavior variability in context. Furthermore, instrument stability would confirm the AUDIT as an appropriate measure determining intervention level in SBIRT.

The purpose of the present study is to explore the stability of the AUDIT in a clinical setting, by comparing prescreening heavy drinking questions and AUDIT scores over time. Findings will contribute to the SBIRT literature by demonstrating how patients’ AUDIT scores change from Visit 1 to Visit 2. It is important to know the stability of the AUDIT if it is to be used in SBIRT programming.

Methods

Subjects

SAMHSA has developed a public health approach for screening and treating substance use disorders (SUD) in primary care settings.¹⁴ SAMHSA introduced a large-scale grant program entailing screening, intervening, and referring patients with substance use problems to appropriate treatment centers. Primary care settings utilizing SAMHSA’s SBIRT program use compulsory prescreening to identify problem substance use, and motivational interviewing and referrals to treat SUD.⁶ SBIRT operates in medical facilities receiving federal funding, known as federally qualified health centers (FQHCs). SBIRT grants are in use at various residency training cooperative agreements, state cooperative agreements, and universities across the USA. Some military reserve bases utilize SBIRT as part of separate contracts with the US National Guard. The IDPH received an SBIRT grant. SBIRT IOWA uses brief intervention with motivational interviewing, brief treatment (5–12 sessions) using motivational enhancement and cognitive behavioral techniques aligned with the American Society of Addiction Medicine criteria for early intervention, and referral to treatment to Iowa substance use treatment facilities.¹⁵ Figure 1 illustrates a flow chart of the SBIRT program implementation.

Figure 1 Flowchart of SBIRT IOWA implementation with the AUDIT. Percentages represent the full sample (N=17,699). Note that not all patients receiving positive prescreenings at Visits 1 and 2 received AUDITs. Note: Data from Iowa Department of Public Health.15 Abbreviations: AUDIT, alcohol use disorders identification test; SBIRT, Screening, Brief Intervention, and Referral to Treatment.

Data represent de-identified data from an SBIRT project in Iowa from October 2012 to July 7, 2015. Sites included four FQHCs and Iowa’s Army National Guard. Individuals aged 18 and over receiving medical services at the FQHCs could be prescreened/screened more than once in the SBIRT project. The original dataset contained 85,277 records representing 66,036 individuals. From the full dataset, 17,717 had more than one entry. Excluding 18 records with duplications on the same day, the full sample included 17,699 individuals for analysis. Only their first and second screening information was used. Data for this study’s analyses were de-identified, and all personal health information variables were removed.

Measures

On visiting the FQHC or during an annual physical examination, male participants were asked a prescreen question, “How many times in the past year have you had five or more drinks in a day?” Females and those aged over 65 years were asked about four or more drinks in a day. A positive answer to this question (i.e., >0) resulted in administration of the ten-item AUDIT.^2,16 The screening criteria for SBIRT IOWA AUDIT scoring are 0–7=encouragement and education, 8–15=brief intervention, 16–19=brief treatment, 20+=referral to outside treatment.¹⁵ The AUDIT has been validated in many international primary care samples at a cutoff score of 8. These studies demonstrate high average sensitivity (mid-0.90 s) and specificity (0.80 s).² In addition, the test–retest reliability in the general population samples was typically high, measured at least 1 year apart (Kolmogorov–Smirnov=0.80 s).¹⁷ A few cases in our sample received an AUDIT despite a negative prescreen, and individuals could refuse to take the AUDIT.

Statistical analysis

The first step in the analytic plan was to assess stability of the participants’ prescreening scores between Visits 1 and 2 in the full sample (N=17,699). Prescreening answers were represented both as the raw responses, number of times the participants drank five or more drinks in a day (four for women), as well as the binary 0 (negative prescreen) versus 1 or more (positive prescreen). Next, the AUDIT scores were analyzed with a subsample of patients completing an AUDIT at both Visits 1 and 2 (n=1,099), both as a scale score (range 0–40) and as ordered categories using the standard cutoffs: 0–7 Low Risk, 8–15 Risky, 16–19 Harmful, 20+ Probable Dependence.² We used Spearman correlations and McNemar’s test to assess Visit 1 to Visit 2 stability for the prescreen responses and the AUDIT raw scores and the four ordered severity categories. Kendall’s tau_b assessed stability in the binary responses. A final scoring process was used to include all participants, making use of the prescreen and AUDIT scores. Individuals who prescreened negative were assigned to the Low Risk category. Additionally, the analyses included fixed model intraclass correlations (ICCs) and weighted Kappa statistics.

An exploratory follow-up analysis was conducted with the subsample (n=1,099). To understand the stability analyses better, we regressed potential covariates on the absolute value of each individual’s difference in AUDIT score (dependent variable). Multiple regression models included age, sex, the time between the two visits, the ordinal categories of the first AUDIT, and interactions (independent variables). Analyses were conducted using Stata 13.1 (StataCorp LP, College Station, TX, USA). Because these data represent de-identified pre-existing service data collected by IDPH, there was no informed consent and the University of Iowa Human Subjects Office Institutional Review Board exempted this study from review.

Results

The full sample (N=17,699) included more females than males (59.4% versus 40.6%) and was predominantly white (79.2%). African Americans made up 15.9%, 4.1% were Asian, and no other racial group accounted for more than 1% of the sample. Regarding ethnicity, the sample contained 18.0% Hispanic/Latinos. Patients identified their race and ethnicity separately. Mean age at the first visit was 46.9 years (standard deviation [SD]=15.26, ranging 18–100). The median time between visits was 16 months (mean=16.4, SD=4.67). Demographic information is provided in Table 1.

Table 1 Demographics of patients in the full sample and subsample Note: Percentages may not sum to 100 due to rounding. Abbreviation: SD, standard deviation.

Prescreening change from Visit 1 to Visit 2

Analyses with prescreening were based on the full sample (N=17,699). At prescreening, 75.7% scored negative for a full screening at both visits and 8.4% were positive on both visits. From first to second prescreen, 11.1% went from positive to negative, while 7.8% went from negative to positive. McNemar’s test indicated significantly more positive to negative switches (χ² (1)=105.31, p<0.0001). Visit 1/Visit 2 stability was moderate (tau_b=0.42, 95% confidence interval [CI]: 0.40–0.44), with the kappa yielding the same values. Using raw responses of the prescreen, the Spearman correlation was positive and moderate in strength (r=0.44, 95% CI: 0.42–0.45; ICC=0.24, 95% CI: 0.22–0.25).

AUDIT stability analysis

After the Visit 1 prescreen, 2,534 individuals filled out an AUDIT. On the second visit, 2,169 received an AUDIT score, and 1,099 filled out an AUDIT score both times. AUDIT score descriptive statistics are provided in Table 2. The following stability analyses were computed based on a subsample (n=1,099) completing a full screening to include an AUDIT assessment. In the primary stability analysis, the Spearman correlation and ICC for the two AUDIT raw scores were positive and moderate in strength (r=0.49, 95% CI: 0.43–0.53; ICC was 0.56, 95% CI: 0.52–0.60).

Table 2 Total completed and subsample AUDIT scores (n=1,099) Abbreviations: AUDIT, alcohol use disorders identification test; CI, confidence interval; ICC, intraclass correlation; Max., maximum; Min., minimum; SD, standard deviation.

Further descriptive analyses recoding the AUDIT scores to Low Risk, Risky, Harmful, and Probable Dependence and considering all prescreen negative visits as Low Risk provided a positive, yet weak correlation (r=0.20, 95% CI: 0.19–0.22; weighted kappa=0.19, 95% CI: 0.1 5–0.21). The sample overall showed a positive and moderate correlation between Visit 1 and Visit 2 AUDIT scores, both in raw and in ordinal severity groups. When examining the correlations within the AUDIT categories, there were differences. For those in the Low Risk group (n=953), the Visit 1/Visit 2 correlation was positive and moderate/weak in strength (r=0.33, 95% CI: 0.27–0.39), and for those in the Risky group (n=178), the correlation was positive and moderately strong (r=0.45, 95% CI: 32–56). The number of patients in the Harmful and Probable Dependence groups was low (22 and 46, respectively). However, the correlations were weak (r=−0.13 and 0.19, respectively). Moreover, the CIs were wide, approximately ±0.40 for the Harmful and ±0.25 for the Probable Dependence group.

We also examined the correlation between the two visits as a function of time between the visits. Categorizing individuals by the number of months between visits, the effect of time span (months) was not significantly correlated with the magnitude of the stability coefficient (within month Visit 1 to Visit 2 Spearman correlation; r=−0.14, p>0.45). The correlation between the absolute differences between the Visit 1 and Visit 2 AUDIT scores was significant, but weak with the time difference between the two visits (r=0.07, p<0.03).

Exploratory follow-up regression analysis

An exploratory follow-up analysis with the subsample (n=1,099) attempted to identify when the AUDIT scores were most stable, controlling for the suggested risk category and the time between visits. Table 3 shows the results from the exploratory multiple regression, predicting the (absolute) difference between the two AUDIT scores. Sex and all interactions were nonsignificant. However, the participants’ age was highly significant, t(1,098)=6.23, p<0.001. Participants were grouped according to age quintile, and the Visit 1/Visit 2 AUDIT correlations are shown in Figure 2. Younger participants clearly showed less stability than older ones. In addition, the lower risk groups (i.e., modality level) showed the largest change.

Table 3 Exploratory multiple regression analysis predicting the absolute difference between the two AUDIT scores from Visit 1 to Visit 2 (n=1,099) Notes: Modality refers to the four screening groups as determined by AUDIT scores. Modality = screen negative (0), brief intervention indicated (1), brief treatment indicated (2) and referral to treatment indicated (3). Timespan = the number of days between Visit 1 and Visit 2. Abbreviations: AUDIT, alcohol use disorders identification test; SE, standard error.

Figure 2 Stability of the AUDIT between Visit 1 and Visit 2 by age quintitles. Error bars represent 95% confidence intervals. Abbreviation: AUDIT, alcohol use disorders identification test.

Discussion

In this uncontrolled real-world application, the prescreening question as well as the AUDIT scores appeared moderately stable over time. Based on classical test theory,¹³ the reliability of an instrument is equal to or greater than the stability of the instrument. Therefore, screening for unhealthy drinking, or risky to dependent drinking (i.e., cutoff of 8),² with the AUDIT in SBIRT settings is at worst moderately reliable in the present setting. This finding adds to the literature base demonstrating high test–retest validity.¹⁷ Two factors may be associated with the moderate stability demonstrated in the present study. One factor is the potential instability of the measure itself and the other is the instability of patients’ drinking patterns. Individual drinking patterns vary over time and the variability is associated with the younger of the legal-drinking-age groups.^18,19

If patients received a positive prescreen, they were then screened for AUD using the AUDIT. Comparing prescreening results at Visit 1 and Visit 2 revealed that the majority of patients (75.7%) reported no need for intervention. Of those who were prescreened positive at Visit 1, just over half were negative at Visit 2, indicating patients were reporting a change in drinking behavior. In addition, there were significantly more switches from negative to positive prescreening between Visit 1 and Visit 2. While there appears to be a positive change between prescreening from Visit 1 to Visit 2, assumptions cannot be certain why the changes occurred (i.e., reporting bias, patient maturation, SBIRT effectiveness).

Patients may be reacting to the brief interventions provided by SBIRT, resulting in reduced drinking, thus producing moderate stability. While we noted a statistically significant reduction in the number of people prescreening positive between Visit 1 and Visit 2, the effect was relatively small. At prescreening, 11.1% went from prescreen positive at Visit 1 to prescreen negative at Visit 2. However, 7.8% went from negative to positive, for a net change of 3.3 percentage points. Respondents who scored 8 or higher on the AUDIT at the initial visit were suggested for brief intervention, brief treatment, or given a referral. Thus, higher scoring individuals may have had an intervention that affected their second visit’s score. Consequently, it seems unlikely that the SBIRT interventions were responsible for the modest stability in AUDIT scores based on the small percentage of positive switches. Unfortunately, it is unclear which patients actually received an intervention or attended treatment, as there is no mandate and only referrals are given. Furthermore, reporting of brief interventions and motivational interviewing was not well documented in the dataset. Future longitudinal research following all interventions received by each participant and the accompanying AUDIT scores would improve the understanding of both reliability and stability of the AUDIT.

Most notably, age appears to be a factor in drinking behavior change, suggesting patient drinking variability is more likely than AUDIT instability. In this sample, the stability was markedly reduced for younger people. The present finding corresponds with extant literature demonstrating the increase and high use of drinking, as well as unstable drinking patterns in youth.^18,20,21 Recent work suggests that hazardous drinkers reduce their consumption over time in the absence of treatment, particularly younger drinkers, supporting the presence of maturation effects.¹⁹ Variability in drinking behaviors in an SBIRT sample was previously demonstrated, with 4.2% of patients referred to brief treatment actually attending the treatment.²² The present study found age and its related drinking variability may possibly be responsible for the shifting AUDIT scores. Patients’ ages should be considered in conjunction with AUDIT scores and health care professionals’ clinical decision making.

Limitations

The AUDIT is a well-studied instrument. However, the AUDIT cutoff of eight may need to be adjusted for general SBIRT applications.²³ Further, it is unclear if patients constituting the sample received an intervention or attended treatment. All we do know is that a health care provider screened and administered an AUDIT to the sample on two separate occasions for unspecified visits. In addition, in this real-world setting, the fidelity of reporting and scoring is unknown. The data source was based on an uncontrolled observational study of SBIRT service data, and the analysis was based on a smaller subsample of patients with an AUDIT scores on both visits (i.e., selection bias). Because of the use of service data, internal validity may be low in terms of the study conclusions, robustness of the associations, and lacks causality of SBIRT-related drinking outcomes. Madras et al⁶ suggest service data is likely to experience greater missing data and implementation variation. In addition, the retrospective observational nature of this study means results may not be generalizable as the sample consisted of primarily White people and all from FQHCs. However, the strengths may outweigh the limitations in that service data present a realistic reflection of clinical practice with many opportunities to improve upon weaknesses. The AUDIT is a past year measure of alcohol use. Patients in the present sample were seen at Visit 2 both within and beyond 1 year’s time. Thus, the AUDIT should be stable for those retested within a year because their answers should be the same as at Visits 1 and 2. A time analysis demonstrated that the quarterly correlation in AUDIT scores at Visits 1 and 2 decreased each quarter, demonstrating appropriate change within and beyond a year. Finally, only the total AUDIT scores are available. Collecting item-level AUDIT information would allow for analysis of validity and reliability, in addition to stability.

Conclusion

The present study revealed that the AUDIT has moderate stability in a clinical setting. We also found the stability is somewhat dependent on age, as the variability in drinking behaviors may be subject to maturation effects of the patient. The present findings contribute to the SBIRT literature by demonstrating the AUDIT changes are moderately reliable from Visit 1 to Visit 2 while taking into account patient drinking behavior variability. It is important to know the stability of the AUDIT for continued use in SBIRT programming. Despite only finding moderate stability in the AUDIT, the prescreening outcomes between Visit 1 and Visit 2 demonstrated a high percent of negative screens. SBIRT staff and researchers could benefit from understanding that younger patients demonstrate less stable scores, which supports that maturational effects may be at play in both behavioral drinking stability and measurement stability. However, sample size limitations may be associated with the moderate stability findings. A larger sample may demonstrate better stability. The present study examined one instrument, in particular, the AUDIT. Given the above, we conclude the stability of the AUDIT in a real-world setting is sufficient as a screening instrument for applications such as SBIRT. The practical applicability of the AUDIT should be reinforced. This is an uncontrolled clinical setting, rather than a controlled experimental study. The AUDIT is currently in use across multiple state SBIRT programs in the USA. The present study has confirmed that in one state, Iowa, the AUDIT is effective in identifying variable drinking behaviors in a stable manner, in an uncontrolled clinical setting.

Acknowledgments

This research was funded by a grant from the Iowa Department of Public Health (Grant #588 6 YM50). The Consortium is a program evaluator for SBIRT IOWA. The Consortium is funded by the Iowa Department of Public Health to perform the SBIRT program evaluation, and separately, to conduct academic research on SBIRT. SBIRT IOWA is operated by the Iowa Department of Public Health, Division of Behavioral Health and is funded by the Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment.

Author contributions

ES, DAL, and SA conceived, wrote, and edited the manuscript, made substantial contributions to data/statistical interpretation, and provided administrative support for the manuscript. ES and SA took responsibility for data integrity and had full access to the data. SA made substantial contributions to conception and design, analysis, and interpretation of the data.

Disclosure

All authors are employees of the Iowa Consortium for Substance Abuse Research and Evaluation (Consortium). The authors report no conflicts of interest in this work.

References

1.		Grant BF, Stinson FS, Dawson DA, et al. Prevalence and co-occurrence of substance use disorders and independent mood and anxiety disorders: results from the national epidemiologic survey on alcohol and relatedconditions. Arch Gen Psychiatry. 2004;61(8):807–816.
2.		Babor TF, Higgins-Biddle JC, Saunders JB, Monteiro MG. The Alcohol Use Disorders Identification Test: Guidelines for Use in Primary Care (WHO/MSD/MSB/01.6a). Vol 2. Geneva, Switzerland; 2001. Available from: http://whqlibdoc.who.int/hq/2001/WHO_MSD_MSB_01.6a.pdf. Accessed March 10, 2016.
3.		Babor TF, Robaina K. The Alcohol Use Disorders Identification Test (AUDIT): a review of graded severity algorithms and national adaptations. Int J Alcohol Drug Res. 2016;5(2):17–24.
4.		Sahker E, Toussaint MN, Ramirez M, Ali SR, Arndt S. Evaluating racial disparity in referral source and successful completion of substance abuse treatment. Addict Behav. 2015;49:25–29.
5.		St. Marie B, Sahker E, Arndt S. Referrals and treatment completion for prescription opioid admissions: five years of national data. J Subst Abuse Treat. 2015;59:109–114.
6.		Madras BK, Compton WM, Avula D, Stegbauer T, Stein JB, Clark HW. Screening, brief interventions, referral to treatment (SBIRT) for illicit drug and alcohol use at multiple healthcare sites: comparison at intake and 6 months later. Drug Alcohol Depend. 2009;99(1–3):280–295.
7.		SAMHSA. Systems-Level Implementation of Screening, Brief Intervention, and Referral to Treatment. Technical Assistance Publication (TAP) Series 33. HHS Publication No. (SMA) 13–4741. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2013. Available from: www.samhsa.gov. Accessed March 10, 2016.
8.		Bernstein E, Bernstein JA, Stein JB, Saitz R. SBIRT in emergency care settings: are we ready to take it to scale? Acad Emerg Med. 2009;16(11):1072–1077.
9.		Saitz R, Palfai TP, Cheng DM, et al. Brief intervention for medical inpatients with unhealthy alcohol use. Ann Intern Med. 2007;146(3):167–176.
10.		Saitz R. Alcohol screening and brief intervention in primary care: absence of evidence for efficacy in people with dependence or very heavy drinking. Drug Alcohol Rev. 2010;29(6):631–640.
11.		Schwartz CE, Rapkin BD. Reconsidering the psychometrics of quality of life assessment in light of response shift and appraisal. Health Qual Life Outcomes. 2004;2(1):16.
12.		Fraley RC, Waller NG, Brennan KA. An item response theory analysis of self-report measures of adult attachment. J Pers Soc Psychol. 2000;78(2):350–365.
13.		Guilford JP. Psychometric Methods. New York, NY: McGraw-Hill; 1954.
14.		SAMHSA. SBIRT implementation: the Iowa Army National Guard Program. 2014. Available from: www.idph.state.ia.us/IDPHChannelsService/file.ashx?file=02C1A989-06E9-4993-B7F8-5371C6111959. Accessed March 10, 2016.
15.		Iowa Department of Public Health. SBIRT Iowa Policy Manual. Des Moines, IA: Iowa Department of Public Health; 2012. Available from: www.idph.state.ia.us. Accessed March 10, 2016.
16.		Skinner HA. The drug abuse screening test. Addict Behav. 1982;7(4):363–371.
17.		Reinert DF, Allen JP. The alcohol use disorders identification test: an update of research findings. Alcohol Clin Exp Res. 2007;31(2):185–199.
18.		Britton A, Ben-Shlomo Y, Benzeval M, Kuh D, Bell S. Life course trajectories of alcohol consumption in the United Kingdom using longitudinal data from nine cohort studies. BMC Med. 2015;13:47.
19.		Dearing RL, Witkiewitz K, Connors GJ, Walitzer KS. Prospective changes in alcohol use among hazardous drinkers in the absence of treatment. Psychol Addict Behav. 2013;27(1):52–61.
20.		Johnstone BM, Leino EV, Ager CR, Ferrer H, Fillmore KM. Determinants of life-course variation in the frequency of alcohol consumption: meta-analysis of studies from the collaborative alcohol-related longitudinal project. J Stud Alcohol. 1996;57(5):494–506.
21.		Syden L, Wennberg P, Forsell Y, Romelsjo A. Stability and change in alcohol habits of different socio-demographic subgroups – a cohort study. BMC Public Health. 2014;14:525.
22.		White K, Arndt S. SBIRT-to-Treatment Analysis (Iowa Department of Public Health Contract #5885YM50). Iowa City, IA: University of Iowa; 2014. Available from: http://iconsortium.subst-abuse.uiowa.edu/downloads/IDPH/SBIRT-to-Treatment Oct 2014.pdf. Accessed March 10, 2016.
23.		Johnson JA, Lee A, Vinson D, Seale JP. Use of AUDIT-based measures to identify unhealthy alcohol use and alcohol dependence in primary care: a validation study. Alcohol Clin Exp Res. 2013;37(Suppl 1):E253–E259.