Psychometrics of the Concise Health Risk Tracking Self-Report (CHRT-SR₁₆) Assessment of Suicidality in a Sample of Adults with Moderate to Severe Methamphetamine Use Disorder: Findings from the ADAPT-2 Randomized Trial

Introduction

Substance use disorders and suicidal ideation are highly prevalent and concerning public health problems; nearly 20 million Americans aged 12 or older had a substance use disorder in 2017,¹ and in 2019, 47,511 Americans died by suicide.² Suicide rates have also been steadily rising from 1999 to 2018,³ and individuals with substance use disorders are at increased risk for suicide.^4,5 Beyond outcomes like suicide attempts and completion, specific components of substance misuse are also associated with suicide risk factors. Despair, hopelessness/helplessness, and lack of social support may be risk factors for suicidality,^6,7 and may be more common in those with substance abuse disorders.⁸ Impulsivity and substance use disorders are strongly associated,⁹ and impulsivity more than doubles the odds of a suicide attempt.¹⁰ Irritability is also associated with substance use and withdrawal,^11,12 and there is some evidence that irritability may be a risk factor for suicidality, though this relationship may vary by patient population.^13,14 Recent work found irritability to be predictive of suicidal ideation in adults in residential treatment for stimulant use disorder, and this relationship persisted when controlling for depression.¹⁵ Thus, it is clear that studying the relationship between suicidal ideation and risk factors among those with primary substance use disorders is vital.

Given the clear public health problem represented by suicidal ideation and behaviors, several self-report and clinician-rated measures have been developed to measure suicidal ideation, behaviors, and risk factors. Perhaps the most widely used suicide assessment is the Columbia Suicide Severity Rating Scale (CSSRS),¹⁶ a clinician rated measure that classifies possibly suicidal events into the Columbia Classification Algorithm for Suicide Assessment (C-CASA) categories. Another popular and validated measure has been the self-report Concise Health Risk Tracking Index (CHRT-SR).¹⁷ Unlike the CSSRS, the CHRT-SR is a self-report measure and includes not only suicidal ideation and behaviors, but also other factors associated with increased risk of suicide. Prior research has validated multiple versions of the CHRT-SR for both unipolar and bipolar depression (the 7-item, 12-item, and 14-item CHRT-SR₇, CHRT-SR₁₂, and CHRT-SR₁₄ respectively),^18–20 for use in adolescents (CHRT-SR₁₄),²¹ and for use in treatment studies (CHRT-SR₁₂).^22–24 Specifically, one paper²⁵ investigated the psychometrics of the CHRT-SR₁₂ in a sample of participants who were primarily receiving an education or exercise-based intervention to treat stimulant use disorder, and found that the psychometric properties of the CHRT-SR₁₂ were consistent with prior work (ie, Propensity and Suicidal Thoughts factors, and four domains within the Propensity subscale), and that the scale demonstrated strong evidence of reliability and convergent validity.

This research contributes to the literature by examining a 16-item version of the CHRT-SR (CHRT-SR₁₆), which adds four additional items that assess impulsivity and irritability to the original CHRT-SR₁₂. It is worth noting that prior work found that the inclusion of the two impulsivity items to the CHRT-SR₁₂ (creating the CHRT-SR₁₄) slightly worsened the psychometric properties of the scale in a sample of individuals with substance use disorder.²⁵ However, assessment of non-depressive risk factors, such as impulsivity and irritability, is a unique feature of the CHRT-SR₁₆, and there may be value in including these items to assess these factors in people with methamphetamine use disorder.

We conducted psychometric analyses of the CHRT-SR₁₆ in a sample of adult participants receiving an intervention for moderate-to-severe methamphetamine use disorder, with a goal to assess this measure’s factor structure, reliability, and convergent validity among individuals with substance use disorders. Specifically, this study investigated the following questions:

1. What is the overall factor structure of the CHRT-SR₁₆ among a sample of participants receiving an intervention for a primary methamphetamine use disorder and how does this compare to the factor structure observed in other samples?
2. What is the test-retest reliability (from baseline to week 1) and internal consistency/coefficient alpha values for the domains of the CHRT-SR₁₆?
3. What is the convergent validity as measured by correlations between the CHRT-SR₁₆ total score (and domains) and the nine-item Patient Health Questionnaire (PHQ-9)²⁶ total score and the score on the PHQ-9 suicidal ideation item?

Materials and Methods

Design

This study utilized data from a previously published Accelerated Development of Additive Treatment for Methamphetamine Disorder (ADAPT-2) clinical trial, a 12-week randomized, double-blind, sequential parallel comparison design trial evaluating the efficacy of combination injectable naltrexone and oral bupropion versus placebo in outpatient adults with moderate-to-severe methamphetamine use disorder.²⁷ The analyses utilized baseline ADAPT-2 data to evaluate the psychometric properties of the CHRT-SR₁₆, including internal consistency and convergent validity. Baseline and week 1 data (average 6.5 days between visits) were used to assess test-retest reliability of the scale.

Participants

The detailed inclusion and exclusion criteria for the ADAPT-2 study are described elsewhere.²⁷ The baseline sample consisted of 403 adults between 18 and 65 years of age who used methamphetamine, while complete baseline and week 1 data (for test-retest reliability) were available for 277 participants. Randomized participants met criteria for moderate-to-severe methamphetamine use disorder based on the 5th edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5),¹² and reported methamphetamine use on at least 18 out of the last 30 days before the consent. Participants undergoing substance use treatment, needing opioid medications during the study, or having other contraindicated medical conditions were excluded from the study. In addition, those with suicidal or homicidal ideation requiring immediate attention at baseline were excluded and referred to appropriate psychiatric care. The ADAPT-2 trial was carried out in accordance with the Declaration of Helsinki. All participants provided written informed consent approved by a central institutional review board and each site’s local institutional review board prior to initiating any study procedures.

Measures

The CHRT-SR₁₆ is a Likert-type ordinal scale with item-level responses ranging from 0 (“Strongly Disagree”) to 4 (“Strongly Agree”) to each item/statement when considering the past week. Total score ranges from 0 to 64, with higher CHRT scores indicative of greater suicidality risk and higher frequency of suicidal thoughts. Baseline and Week 1 CHRT-SR₁₆ data were utilized in the analysis.

The self-report Patient Health Questionnaire (PHQ-9) consists of nine items corresponding to the degree of frequency of being bothered by the nine symptoms of major depressive disorder over the last two weeks, where item-level responses are measured on a scale from 0 (“not at all”) to 3 (“nearly every day”) and the total score ranges from 0 to 27. Higher scores on the PHQ-9 are indicative of greater depressive symptom severity. Item 9 of the scale assesses suicidal ideation.

Statistical Analyses

A series of confirmatory factor analyses (CFA) were performed to evaluate the factor structure of the CHRT-SR₁₆. The following four models were tested with factors identified a priori based on prior literature:¹⁶ (1) a single-factor model with 16 items loading on one factor of Suicidal Risk (CHRT-SR₁₆ total); (2) a two-correlated-factor model with items 1–13 loading on the Propensity factor and items 14–16 loading on the Suicidal Thoughts factor; (3) a seven-correlated-factor model with items 1–13 loading on the Pessimism, Helplessness, Social Support, Despair, Impulsivity, and Irritability factors and items 14–16 loading on the Suicidal Thoughts factor; and (4) we tested the hypothesis that the 16-items on the CHRT-SR would load on seven first-order factors (Pessimism, Helplessness, Social Support, Despair, Impulsivity, Irritability, and Suicidal Thoughts), with six factors (Pessimism through Irritability) loading on a second-order latent factor of Propensity.

All CFA models were fitted using diagonally weighted least squares mean and variance adjusted (WLSMV) estimators with robust standard errors. WLSMV does not assume multivariate normality among observed indicators, resulting in less biased estimates of factor loadings compared to maximum and robust maximum likelihood estimators, particularly among large sample sizes and ordinal-level responses.^28,29 The following fit indices and cutoffs were used as indicators of satisfactory model fit: root mean square error of approximation (RMSEA) ≤ 0.06 (90% CI ≤ 0.08), standardized root mean square residual (SRMR) ≤ 0.08, comparative fit index (CFI) ≥ 0.95, Tucker-Lewis Index (TLI) ≥ 0.95,^30,31 and the weighted root mean square residual (WRMR) ≤ 1.00.³²

The internal consistency reliability of the CHRT-SR₁₆ was examined for the entire scale, two main factors (Propensity and Suicidal Thoughts), and six Propensity subscales (Pessimism, Helplessness, Social Support, Despair, Impulsivity, and Irritability) using coefficients alpha (α) and omega (ω). Acceptable internal consistency was indicated by coefficient values ≥ 0.70.³³ Coefficient ω does not assume equal factor loadings (essential tau equivalence), making it a more general estimator of reliability.³⁴

The test-retest reliability was assessed between baseline and week 1 (mean 6.5 days apart) using the intraclass correlation coefficient (ICC) as a relative measure of reliability and the standard error of measurement (SEM) as an absolute reliability index. Specifically, we used a two-way mixed effects ICC measurement, with subjects as random effects and visits as fixed effects.^35,36 The SEM formula was computed as follows: .³⁶ ICC and SEM scores were obtained for CHRT-SR₁₆ overall scale, the second-order Propensity factor and each of the first-order factors. Convergent validity was evaluated by assessing the Spearman ρ correlation coefficient between PHQ-9 total score and the CHRT-SR₁₆ total score, including the first- and second-order factors.

All statistical analyses were conducted using R statistical software (version 4.1.1), with CFA models fitted using package lavaan,³⁷ and reliability and validity estimates obtained using package psych.³⁸

Results

The baseline sample consisted of 403 participants randomized into the study. Full descriptive and demographic information, including substance use history, is presented in Table 1, and has also been previously published.²⁷ The participants were primarily male (68.7%), middle-aged (M = 41.0 years; SD = 10.1), and White (71.2%), with 55 (13.6%) and 48 (11.9%) participants of Hispanic or Latino ethnicity and identifying as Black, respectively. All participants met DSM-5 criteria for moderate-to-severe methamphetamine use disorder, with an average of 26.7 days (SD = 4.1) of methamphetamine use in the 30 days prior to consent.

Table 1 Baseline Characteristics of the Sample (N = 403)

CHRT-SR₁₆ descriptive statistics, including means and SDs at baseline, are included in Table 1. Most participants scored 0 (“strongly disagree”) or 1 (“disagree”) to items related to Suicidal Thoughts and the subfactors of pessimism, helplessness, and despair. Particularly for item 16, an item regarding having a suicide plan, 382 individuals (94.8%) reported that they “strongly disagree[d]” (79.2%) or “disagree[d]” (15.6%) with the statement. The median Suicidal Thoughts factor total score was 0, with 59.8% of participants having a total score of 0 (out of a possible 12) across three suicidality items. Items of impulsivity (M = 4.4, SD = 2.2) and irritability (M = 4.0, SD = 2.4) were more frequently endorsed.

CHRT-SR₁₆ Fit Indices

The one-factor (Model 1; CHRT-SR₁₆) model, the two-factor (Model 2; Propensity latent factor and Suicidal Thoughts factor) model, and model 4 (Propensity latent factor as a second-order factor determined from the correlation among the factors) did not adequately fit the observed data. Only model 3 (7-factor structure) exhibited acceptable model fit. Table 2 shows fit indices across all models, and Figure 1 shows Model 3 factor structure.

Table 2 Goodness-of-Fit Indicators (N = 403)

Figure 1 CHRT-SR16 Confirmatory Factor Analysis of Model 3 where a seven-correlated-factor model with items 1–13 loading on the Pessimism, Helplessness, Social Support, Despair, Impulsivity, and Irritability factors and items 14–16 loading on the Suicidal Thoughts factor. Notes: Fitted using diagonally weighted least squares mean and variance adjusted (WLSMV) estimators with robust standard errors. The figure shows standardized factor loadings. Covariance coefficients are not shown, all coefficients were statistically significant at p < 0.001.

CHRT-SR₁₆ Standardized Factor Loadings

In Model 3, the majority of items loaded strongly on their underlying factors and exhibited strong inter-item correlation. Items 7 and 16 had the weakest loading on their respective factors of Despair (0.61) and Suicidal Thoughts (0.57). Item 7 also showed a moderate correlation with Items 8 (ρ = 0.44) and 9 (ρ = 0.46) – the other items in the Despair factor. In comparison, Items 8 and 9 correlated strongly with each other at ρ = 0.77. For the Suicidal Thoughts factor, Items 14 and 15 had excellent factor loadings of 0.92 and 0.96, respectively, while Item 16 had a weak factor loading of 0.57. Item 16 also showed lower correlation with other items on the Suicidal Thoughts factor (ρ = 0.63 and ρ = 0.67 with Items 14 and 15) compared to the correlation between Items 14 and 15 (ρ = 0.84). In general, Item 16 correlated weekly with most other items on CHRT-SR₁₆, with inter-item correlations ranging between 0.10 (Item 5) to 0.46 (Item 8). The standardized factor loadings for Model 3 are presented in Table 3.

Table 3 Standardized Factor Loadings for Model 3 (N = 403)

CHRT-SR₁₆ Internal Consistency

The internal consistency for CHRT-SR₁₆ was 0.89 for both coefficients α and ω, suggesting acceptable internal consistency. The internal consistency for individual factors was also acceptable, with the exception of the Social Support factor (both α and ω = 0.66), which was slightly lower than the rest of the factors. Both Propensity and Suicidal Thoughts demonstrated excellent internal consistency at Baseline, with α = 0.89 and ω = 0.89 for Propensity, and α = 0.86 and ω = 0.90 for Suicidal Thoughts. The correlations between six first-order factors (Pessimism, Helplessness, Social Support, Despair, Impulsivity, and Irritability) loading on a second-order latent Propensity factor ranged from 0.63 (Impulsivity) to 0.80 (Helplessness) and were all statistically significant at p < 0.001. The lowest first-order inter-correlations were observed between the Impulsivity factor and the other first-order factors (correlation coefficients ranging between 0.29 and 0.42). Additionally, the individual items showed strong and statistically significant correlations with underlying factors, with all item-factor correlations greater than 0.70. Table 4 and Table 5 provide a summary of inter-factor and inter-item correlations for the CHRT-SR₁₆.

Table 4 Inter-Subscale Correlations for CHRT-SR16 Total and Subscale Scores (ρ, Spearman’s Rho Rank Order Coefficient Test)

Table 5 Item-Subscale Correlations for CHRT-SR16 (ρ, Spearman’s Rho Rank Order Coefficient Test)

CHRT-SR₁₆ Test-Retest Reliability

The CHRT-SR₁₆ total score and the Propensity factor had acceptable test-retest reliability between Baseline and week 1 (ICC = 0.78 for both). Among the first-order factors, the ICC ranged from 0.64 (Helplessness) to 0.74 (Despair). The lowest ICC was observed for the Suicidal Thoughts factor (ICC = 0.49). Table 6 presents the ICC and SEM scores for the scale’s total score and individual factors.

Table 6 CHRT-SR16 Internal Consistency and Test-Retest Reliability Between Baseline and Visit 1 for CHRT-SR16

CHRT-SR₁₆ Convergent Validity

The association between the CHRT-SR₁₆ total score and PHQ-9 total score was moderate-to-high (ρ = 0.62, p < 0.001), as was the association between the Propensity factor and the PHQ-9 total score (ρ = 0.63, p < 0.001). The Suicidal Thoughts factor had the weakest correlation with the PHQ-9 total score, but the relationship was still statistically significant (ρ = 0.31, p < 0.001). Among the first-order factors that load on Propensity, the correlation coefficients (ρ) with PHQ-9 total score ranged from 0.32 (Social Support) to 0.53 (Despair) and all were statistically significant at p < 0.001. Similar to the CHRT-SR₁₆ Suicidal Thoughts factor (items 14–16), the distribution of the suicidality item (#9) on the PHQ-9 was positively skewed, with 80.9% of participants endorsing “not at all” (no thoughts that they would be better off dead or hurting themselves) over the past week. The suicidality item on the PHQ-9 exhibited the strongest correlation with the Suicidal Thoughts factor on the CHRT-SR₁₆ (ρ = 0.51, p < 0.001). Table 7 shows the CHRT-SR₁₆ factor correlations with the PHQ-9 total score and the suicidality item (item 9).

Table 7 Convergent Validity (Spearman ρ Coefficients)

Discussion

With a goal of examining a more comprehensive, 16-item version of the self-report Concise Health Risk Tracking Scale, we validated this measure’s psychometric properties and suitability for use in people seeking treatment for moderate-to-severe methamphetamine use disorder. Specifically, confirmatory factor analysis determined that all 16 items loaded onto a series of factors and subfactors encapsulating the propensity for suicide risk as well as a second factor of Suicidal Thoughts. Test-retest reliability and internal consistency for the factors and subfactors were generally highly acceptable. Convergent validity between the CHRT-SR₁₆ and the suicidal ideation item from the PHQ-9 was also established.

Testing four potential models of factor structure, one model was deemed acceptable which established six first-order factors loading on a second-order latent factor of Propensity, along with a separate Suicidal Thoughts factor. This model aligns with prior CHRT-SR psychometric research and theory on how the factors of pessimism, helplessness, social support, despair, impulsivity, and irritability are described as suicide propensity.¹⁷ Although the current study tested four models based upon theory and prior research, other models may exist that fit the data just as well. Thus, the good model fit demonstrated by one of our models does not indicate that it is “correct or true”, only plausible and well supported.

Across several metrics in our sample, the three-item Suicidal Thoughts factor and its individual items did not perform as well as other CHRT items in this sample. Low base-rate prevalence may mostly explain these findings, as nearly 60% of participants responded “strongly disagree” to all three items on the Suicidal Thoughts factor, and only 19.6% responded in the affirmative on all three items, with the majority endorsing 0 or 1 responses. In addition, the final item of this factor regarding a current suicidal plan was especially infrequently endorsed, as 95% of participants reported “strongly disagree” or “disagree” to this item at baseline, and only 0.2% and 0.7% reported “agree” or “strongly agree” respectively. These low base rates are to be expected given study eligibility criteria that excluded severe current suicidality at screening. It is therefore not surprising that this item performed relatively poorly in intercorrelation with all other items. In addition, variability across time on all three suicide thoughts items and overall factor was relatively low, with very infrequent endorsement. Low variability among items and a small number of time points can contribute to low ICC,³⁹ and both issues are present in our current study.

Study strengths include a large sample size and the inclusion of individuals with a primary stimulant use disorder – a relatively understudied population (compared to participants with a primary mood disorder) with respect to suicidality, but a critical one from a public health perspective, given the association between substance use disorders and suicide mortality.⁴

Study limitations include low endorsement of several items and factors, particularly those regarding Suicidal Thoughts and planning. The sample also had a relatively low proportion of women, who are more likely to experience suicidal thoughts and plans as compared to men.^40–42 Inclusion of a greater proportion of women may have permitted a more robust and accurate test of the performance of the Suicidal Thoughts factor in individuals with substance use disorders. This work also did not assess for this scale’s predictive validity for future suicide attempts or suicidal behaviors among those with a primary stimulant use disorder, although this predictive validity has been established with another version of the CHRT-SR among adolescents treated for suicidality in an intensive outpatient program.⁴³

Nonetheless, our research adds to prior literature by confirming the psychometric factor structure, reliability, and convergent validity of the CHRT-SR₁₆ among a sample of participants receiving treatment for a primary stimulant use disorder. To improve our understanding of the Suicidal Thoughts factor, future research should examine the reliability, validity, and psychometrics of the Suicidal Thoughts factor and its constituent items among a sample of participants where suicidal thoughts are more frequent, and where data are obtained at more than two timepoints. In addition, determining the CHRT’s predictive validity of suicidal events (ie, suicide attempts, hospitalizations, and deaths by suicide) among a sample of participants with a substance use disorder is an important future direction which will require larger sample sizes. Finally, investigating the extent to which suicidal ideation as assessed by the CHRT-SR₁₆ predicts substance use overdoses is especially important given substance users’ frequent reports of suicidal intent before overdosing.⁴⁴

Conclusion

The results of this study offer evidence for strong reliability and convergent validity of the CHRT-SR₁₆, among the first validated instruments to measure Suicidal Thoughts and behavior in a population that is disproportionately experiencing overdose and death due to methamphetamine use.⁴⁵ The CHRT-SR₁₆ builds upon prior work with the CHRT-SR,^17,25 and allows for the novel assessment of impulsivity and irritability as possible risk factors for suicidal behavior.