Construct Validity and Differential Item Functioning of the PHQ-9 Among Health Care Workers: Rasch Analysis Approach

Introduction

Depression is considered the most common mental disorder and a major cause of disability in Thailand^1–3 and globally.⁴ A number of studies have shown that health care workers, defined by WHO as all personnel involved in actions whose principal intent is to promote health,⁵ are exposed to psychological distress related to their occupation. They have heavy workloads, night work or shift work. These occupational stress factors can lead to burnout, anxiety, sleep problems, psychiatric disorders or even depression.^6,7 Health systems and healthcare workers worldwide are encountering tremendous stress because of the growing Coronavirus Disease 2019 (COVID-19) pandemic,⁸ especially those on the frontline, migrant workers, and workers in contact with the public.⁹ Reports have shown that screening for depression among these healthcare workers is increasing,¹⁰ and healthcare workers significantly experience anxiety and depression symptoms.¹¹

In Thailand, one study among health care workers revealed a high rate of depression at 21.5%.¹² Consequences of depression among health care workers could produce medical errors and affect treatment outcome.^13–15 Therefore, screening and grading depression symptom severity are important approaches to early detect and manage depression among health care workers.

Several tools have been used for screening depression including the Hospital Anxiety and Depression Scale (HADS),¹⁶ Center for Epidemiologic Studies Depression Scale (CES-D),¹⁷ self-rating depression Scale (SDS), Depression, Anxiety and Stress Scale-21 items (DASS-21).¹⁸ While some have included anxiety, CES-D and The Patient Health Questionnaire (PHQ-9) measure only depressive symptoms. Besides, the different number of items used for each scale, the outcome as the prevalence of depression may differ across measurements. In comparison among tools, a study conducted among patients with cancer found that HADS had the widest measurement range, whereas the DASS-D had the narrowest. Based on the cutoff, PHQ-9 was easier to meet the criteria for mild depression than CES-D and DASS-21.¹⁹

PHQ-9, one of the most used tools, has screened for depression in various settings, demonstrating good sensitivity and specificity for depressive disorders.²⁰ It comprises nine items of diagnostic symptom criteria based on the DSM-IV with Likert scale responses, ie, 0 (not at all) to 4 (nearly every day).²¹ The PHQ-9 has been translated into many languages including Thai.²²

Even though the PHQ-9 has been widely used because of its brevity; related literature has shown the PHQ-9 exhibited some problems that led to modification when tested for its construct validity. Problems may have stemmed from the sample regarding the scale’s unidimensionality (how well items support the same underlying construct of depression), problems of low response endorsement regarding some items, and problems involving response bias despite that the severity of depression was the same (differential item functioning-DIF).^23–25

PHQ-9 has been investigated for psychometric properties both by traditional classical test theory (CTT) and by item response theory, such as Rasch model analysis.^23,26–29 Most studies reported that all nine items could form a unidimensional construct of overall depressive severity.

Rasch analysis is the science of developing, examining, and analyzing the performance and quality of measurement instruments that are completed by individuals.³⁰ The Rasch model offers procedures for constructing and revising instruments and documenting measurement properties of instruments (eg, reliability, construct validity). Contrasting with traditional classical test theory, the Rasch model allows investigators to make critical corrections when using raw test score data by transforming raw data (nonlinear, ordinal level data) to interval-level data, which then can be evaluated through the use of parametric tests.^31–33

One way that the Rasch model assesses the quality of a measurement instrument is to evaluate the “fit” of items to the Rasch model based on the fact that items at the more difficult end of the variable should be harder to correctly answer than items at the easy end of the continuum. In addition, the unique person-item (Wright) map created by Rasch analysis displays the linear scale graphically, to suggest good test-item targeting.³⁴

In addition, Rasch analysis was used to evaluate differential item functioning or item bias related to subpopulations. Most studies found that the items of PHQ-9 were free of DIF with regard to age, education, and employment status; however, DIF related to visual impairment was found with “‘trouble falling asleep’” item,²⁶ and DIF related to sex among elderly populations, which was found with “sleep” and “self-blame” items.²⁵

The PHQ-9 has been investigated for DIF in many subpopulations. While some found that DIF was unrelated to ethnic background,³⁵ language,³⁶ race,³⁷ sex^37,38 and level of education,²⁹ it found DIF related to age,^38,39 cognitive impairment,⁴⁰ a history of mania⁴¹ and visual impairment.²⁶ In addition, age group plays a role concerning DIF. For instance, DIF related to sex was detected in an elderly population.²⁵

The psychometric quality of the PHQ-9, as evidence shows, varied depending on the studied population. For health care workers, a report of Rasch validation of PHQ-9 is lacking. In addition, one common clinical characteristic among health care workers is alcohol assumption.^42,43 Alcohol users tend to have irrational beliefs,⁴⁴ as well as other psychiatric symptoms, such as anxiety or depression and cognitive impairment.⁴⁰ This casts doubt that alcohol users may view or assess depressive symptoms differently from nonalcohol users due to their distorted beliefs.⁴⁵ Whether alcohol consumption is related to DIF has, nevertheless, has yet to be examined.

The present study aimed to investigate the psychometric properties of PHQ-9 among health care workers using the Rasch model. We investigated 1) to see whether the PHQ-9 data in this sample fit the Rasch measurement model, 2) to see how appropriate the PHQ-9 was for this population, ie, whether item difficulty levels of depressive symptoms sufficiently covered the whole range of an individual’s depression and 3) to examine whether DIF due to age, sex, education and condition of alcohol consumption existed.

Patients and Methods

Participants and Procedures

Health care workers were classified into three groups. The first group consisted of doctors, dentists, nurses and pharmacists (42.1%). The second group was “other health professionals” and other health-related positions (19.4%). The last group was “nonhealth professionals” and mainly consisted of workers (38.5%). This cross-sectional health care workers’ study, conducted at Chiang Mai University (CMU), was approved by the Ethics Review Committee for Research in Human Subjects, Faculty of Medicine, Chiang Mai University Hospital. Ethics approval was obtained from the Faculty of Medicine, Chiang Mai University (No. 069/2012). It was a single site study from a survey of health care workers from Chiang Mai University Hospital. The first recruitment period was from January to February 2013 where about 56.3% of all health care workers responded to the survey. The second enrollment period was from March to May 2013. Additional recruitment strategies included posters and turning the desktop wallpaper of all computer-operated computers to provide information about the survey. The second recruitment period helped bring the proportion of overall respondents to 75%, and 3532 (65.8%) consented to participate in the study. In the end, 3204 participants (59.7% response rate) completed the self-rating online questionnaires concerning PHQ-9 as well as their demographic information. This comprised age, sex, education level and alcohol consumption. A Thai National ID number was used to assign a study identification number and to ensure no duplicate responses. A detailed description of the study has been published.⁴⁶

Measure

Patient Health Questionnaire (PHQ-9)

PHQ-9 is a self-report tool, consisting of nine questions regarding depressive symptoms based on the DSM-IV criteria for a major depressive episode (Kroenke et al, 2001). The questions included the symptoms of: lack of interest, depressed mood, sleeping difficulties, tiredness, appetite problems, concentration problems, psychomotor agitation/retardation, negative feelings about self and suicidal ideation. The respondent was asked how many symptoms he/she experienced during past the two weeks. Items were administered on a 4-point Likert scale with the response options: 0 “not at all”, 1 “several days”, 2 “more than one half of the days”, and 3 “nearly every day”. The Thai version of the PHQ-9 was shown to have acceptable psychometric properties to screen for major depression in the primary care setting.²²

Statistical Analysis

Demographic data were described using mean, SD and frequency. The Rasch rating scale model was used to verify the construct validity of the PHQ-9.

Rasch analysis is a mathematical method to calibrate linear logit measures of item difficulty and person ability from ordinal data. To examine the PHQ-9 construct, a firmly established calibration of item measures was needed to make inference about the construct. According to the Rasch model, the probability of an individual’s response counts on both “person ability” and “item difficulty”.⁴⁷ Herein, “person ability” refers to as the extent to which the participants experience depression and “item difficulty” refers to the severity of depression expressed by the item. The response probabilities of each person to each of the individual items, according to the Rasch model, are modeled as a logistic function of the latent depression trait. This model yields person and item depression estimates, as well as estimates of a set between response category thresholds common to all items. Item estimates below 0 (mean) are considered easy for the person to endorse, comparable to a person with a lower level of depression. The opposite meaning is applied when item and person estimates are above 0.

To test whether the data could fit the Rasch model, fit statistics, eg, information-weighted fit statistics (infit) mean square (MnSq) and outlier-sensitive fit statistics (outfit) MnSq were used. An item with infit or outfit MnSq out of the 0.7–1.5 range was considered a misfit.⁴⁸ The performance of the scale was examined using Rasch fit statistics, and the dimensionality of the scale was examined using principal component analysis (PCA) of the standardized residuals. To indicate unidimensionality, there should be an absence of any meaningful pattern in the residuals. The first residual dimension is usually expected to have a value smaller than 2.0, which has been shown to happen entirely due to random variation.⁴⁹ In addition, fit statistics <0.6 indicate items overfit the model, usually because they share some components of meaning with other items.³³

Local dependency, referring to the items containing a latent trait other than depression, was tested using the correlation (r) of the Rasch residuals between each pair of items; r ≤0.3 was considered acceptable.⁵⁰

Item ordering, indicating that a higher severity of a symptom should score a higher category, was examined using the category function. The threshold estimates for a 4-category response option were examined to verify whether participants discriminated between the available ordered response categories. The disordering threshold could be examined in two ways: first) by considering infit and outfit MnSq within 0.7 and 1.3 and second) by the ordering of the “observed averages; acceptable response scores should monotonically increase average difficulties (average measure) and step difficulties (step measure).

We used Wright map to plot item difficulty and the individual’s abilities along its continuum on the same axis of the logits allowing the evaluation of the fit of the item difficulties matched to the abilities of the individuals. We examine to what extent the item positions match the person positions (targeting) using the Wright map. The best targeting of a measurement is when the mean items are at the same measure as the mean persons. Researchers suggest the difference between the mean value of the mean person measure should be within one logit.³⁰ Floor or ceiling effects could also be visualized using this map.

We tested the differential item functioning (DIF) across sex, age, education and alcohol consumption. Both statistical test and DIF contrast were used, and a DIF contrast >0.64 indicated a substantial DIF.⁵¹

Finally, reliability was evaluated using the person separation index (comparable to Cronbach’s alpha). Person separation index denotes how well the test is able to differentiate among groups of respondents with different levels of depression. An acceptable value for separation is at least 2. Item reliability was assessed using the item separation index. Separation value was less than 3 and the item reliability was less than 0.9, implying that the sample is not large enough to endorse construct validity or a difficulty exists with the item hierarchy of the instrument.⁵¹

All analyses were conducted using IBM SPSS for Windows, Version 22 (Chicago, IL, USA), STATA, Version 14 (College Station, TX: StataCorp LP) and Rasch models using WINSTEPS, Version 4.5.4 (Winsteps^® Rasch Measurement, 2017).

Results

Table 1 shows that the majority of the population was female. Up to 31% consumed alcohol; however, most (80.7%) were considered at low harmful risk. For PHQ-9, over 94% (n = 3005) scored below the cut-off point for clinical depression (a sum score of 10 or over).

Table 1 Sociodemographic Characteristics of the Subjects

Table 2 shows the proportion of each category and fit statistics. All items were shown to have fit statistics in the required range, 0.7–1.5. As suggested by Linacre, the standardized statistics is basically overly sensitive to misfit a large sample. However, Zstd >2.00 could be ignored, when mean-squares were acceptable. The overall fit of the data to the model was good, indicating that overall, the 9-item scale formed a valid measure.

Table 2 Item Fit for PHQ-9

“Trouble sleeping” was the most endorsed item (logit= −1.70), while “Better off dead” was the least endorsed (logit = 3.91).

In terms of dimensionality, the analysis showed that the raw unexplained variance was 48.2%, while the eigenvalue of the unexplained variance was in 1st contrast = 1.58 which was less than 2.0 indicating the PHQ-9 was less likely to have another dimension. In addition, we found the Pearson’s correlation was 1.000 for item clusters 1–2, 1–3, and 2–3 indicating no different dimension. No pair of items had a residual correlation >0.2. The most were items 2, “Feeling down” and 1, “Little interest” (r = 0.12). This lacked substantial residual correlations between items, indicating the criteria of local independence was met.

PHQ-9 was free of DIF. However, one item showed a minimal level of DIF by sex, with male participants rating item #6 (Feeling bad about self) as easier than female participants (0.66 and 1.08, respectively). Participants who had alcohol consumption rated the item “Feeling bad about self as 0.70 logits more easily than those who did not consume alcohol (1.06 logits). The DIF contrast for all were, however, less than the cut-off 0.64 logits.

Figure 1 shows the person-item map for the PHQ-9. The person-item map indicated items for persons with lower ability estimates were missing. (3.31 logits was the highest person ability estimate) and presented evidence for a floor effect of the PHQ-9. The mean person ability was −3.83 logits (SD 1.85). The PHQ-9 had a wider range but was not a better match for the sample. Most individual’s level of depression did not match any item of the PHQ-9. Specifically, no items were available to accurately measure individuals with abilities between −2.22 and −5.85 logits. This implied that the items were too difficult for the abilities of the respondents. In addition, redundancy of the items was observed, that is, pairs of items, eg, “Feeling tired” and “Feeling down”, were shown on the map to be located at the same difficulty level, indicating that they exhibited a similar level of depression.

Figure 1 Person-item Wright Map. Notes: The persons are on the left of the dashed line, and items are located on the right of the dashed line. More able (depressed) persons are located at the top of the map. More difficult (severe) items are located at the top of the map. Each “#” represents 33 persons. Each “.” represents 1–32 persons (M = mean; S=1 standard deviation from the mean; T = 2 standard deviations from the mean).

The three most difficult items were, “Better off dead,” “Moving slowly,” and “Feeling bad about self” Conversely, the three least difficult items were “Trouble sleeping”, “Little interest,” and “Poor appetite”.

Table 3 shows the summary statistics for the 4 rating-scale categories. The “frequency of use” of categories 0 and 1 were much more than any of categories 2 and 3. This implied that most were “less able” persons (low level of depression), consistent with the fact that mean person ability (−3.83) was lower than mean item difficulty (0). The observed person measures that increased from category 0 to category 3 represented low (−4.80) to high ability (0.60) denoting that no collapse of rating categories was necessary. The outfit MnSq for category 3 was slightly high (1.52), indicating an idiosyncratic use of category 3. An adjustment of the description for “category 3 (nearly every day)” would probably improve the functioning of the entire rating scale.

Table 3 Summary Statistics for the 4-Rating Scale Categories of PHQ- 9

Figure 2 shows the category probability curves for an item of the PHQ-9. No evidence of disordered thresholds with the 4-category response was observed. The person separation was 1.54 and the reliability, 0.70. The internal consistency was good when used (Cronbach Alpha = 0.80). The item separation was 31.24 and item reliability = 1.00.

Figure 2 Categorical probability curves for PHQ-9 with no threshold disorder. Notes: The curves for the PHQ-9 illustrating the range over which each of the 4 categories is most likely to be chosen. The red, blue, pink and blank curves on the graph represent the 0, 1, 2 and 3 and 4 PHQ-9 rating categories.

Discussion

The present study aimed to evaluate the validity of the PHQ-9 in a large sample of health care workers using Rasch analysis, as well as possible item bias due to sex, age, education and particularly alcohol consumption. Our results were in line with related studies that the PHQ-9 fitted the assumption of the Rasch measurement model, ie, unidimensionality and local independence, indicating that all items contributed to the same depression underlying construct,^23,39,52 but contrasted with other related studies.^25,27,28 This inconsistency may have contributed to the different characteristics of the studied sample, especially since none of the studies had been conducted before among a health care worker sample.

In terms of item hierarchy, our findings concurred with related studies. The easy items were, “Trouble sleeping”, “Little interest” and “Poor appetite”, while the most difficult item was “Better off dead”. Basically, suicidal ideation was related to severe depression.^22,53 It appeared that this item was difficult to be endorsed in the general population including among health care workers because health care workers experienced mild levels of depression compared with a clinical subject with depression who is intended to be the real target for the PHQ-9. The item ordering may vary from sample to sample, especially among clinical subjects.^23,26,54 In line with related studies, the present results showed that items, “Feeling tired” and “Feeling down” appeared to be redundant and one could be removed when the same accuracy of the reduced version is warranted.^27,28

Notably, item 9 is loosely related to latent dimension depression, leaving the question whether or not incorporating suicidality is useful in such a screening scale for depression.^25,28,55 Not only did it receive the lowest endorsement, one study showed that item 9 illustrated both item misfit and disordered threshold.⁵⁶ From this reason, item 9 was removed, and PHQ-8 became adopted for a screening tool of depression.^57,58 In terms of category, most participants tended to endorse “0 (not at all)” or “1 (several days)” due to their low level of depression, category “3 (nearly every day)” was thus less endorsed to the extent that it created a mild misfit category. Despite that; however, this can be ignored as the effect is not detrimental to the scale.⁴⁸

That PHQ-9 does not target well in a sample who generally have low-level depression was expected and consistent with one related study in a nonclinical population.²⁵ Because the PHQ-9 adopted symptoms originating from DSM, the items are designed for depressed people. Well-targeting is more usually found in studies in a clinical sample than a nonclinical population.²⁷ This is to confirm that PHQ-9 functions as intended among clinical but not general people including health care workers. Therefore, PHQ-9 can still be used as a screening tool for major depression but is not to be applied to measure depression (as outcome measure) among health care workers. The big gap in the Wright map from Rasch analysis results suggested that easier items are needed if the PHQ-9 is used to assess depression level. Depression is developed by the influence of biopsychosocial factor. Clinically, it may form at least part of a continuum of affective disorder, rather than a discrete disorder.⁵⁹ A mild form of depression precedes moderate and may become a severe form of depression when an individual’s coping strategy fails. A screening that detects a mild form of depression is useful, but PHQ-9 is deemed unable to capture well enough among this population.

Other measurements, containing more items, for example, the 20-item Center of Epidemiological Study of Depression scale (CES-D) and 21-item Beck Depression Inventory (BDI-II), were found to have better targeting in adult populations, whereas the 15-item geriatric depression rating scale showed better targeting in elderly populations.^19,25 This could be because those scales have more items that are able to cover the broader latent construct of a subject’s depressive severity.

In terms of reliability, the person separation was not excellent, albeit acceptable (0.74), while the Cronbach alpha was 0.80. This may have contributed to the number of items being relatively low – more items may be needed.⁵¹ Pearson separation is basically lower than Cronbach’s alpha because the Rasch-based reliability of separation statistics is based on a linear, interval-level scale when a good model-data fit is observed, whereas alpha is based only on the assumption of linear measures.⁶⁰ Here again, it could be suggested that more items are needed for PHQ-9 to be used as a measuring tool for this population.

The present study found no significant DIF due to sex, age, education and alcohol consumption indicating that the PHQ-9 could be used in this population without modifying. This may have contributed to the no to low level of depression for the whole sample. DIF is usually found in some specific demographic or clinical sample, eg, visual impairment, the elderly, primary care with depression or ethnic background.^{25,26,28,29,52} However, to emphasize no DIF was due to alcohol consumption, that is, whether or not, a participant used alcohol, revealed no bias concerning the PHQ-9 items.

Strengths and Limitations

Our study revealed some limitations. First, we did not use other measurements to concurrently validate the PHQ-9. Second, avoiding response bias was difficult. Because the respondents were health workers, some might underreport their real symptoms for fear of stigma. Our study, however, indicated some strengths. To the best of our knowledge, this was the first study to report the validity of the PHQ-9 using Rasch analysis with this substantial sample size of health workers. Thus, it would be likely to conclude that no DIF was observed among this population, which was somewhat comparable to the general population.

Conclusion

The PHQ-9 was, demonstrated by Rasch measurement model, shown to be a unidimensional structure with ordered response categories evaluating a single construct of depression with sufficient person and item reproducibility. However, the low person separation value and poor targeting showing on the person-person-item map suggested PHQ-9 might not be appropriate to measure depressive level among health care workers who seem to have low levels of depression. Rasch analysis provides an opportunity to improve the measurement so that it would have better matched with the target population. PHQ-9 might not be appropriate enough to assess the depressive symptoms among nonclinical subjects such as healthcare workers. Other screening tools for depression, especially those with more items (eg, CES-D, DASS-21) should be trialed in further study.