Psychometric evaluation and linking of the PHQ-9, QIDS-C, and VQIDS-C in a real-world population with major depressive disorder

Introduction

Major depressive disorder (MDD) is a leading cause of disability worldwide, with an estimated lifetime prevalence in the United States (US) of 20%.¹ Accurate and reliable assessment of MDD is crucial for effective treatment and management. The Patient Health Questionnaire-9 (PHQ-9; self-report) and the clinician rated Quick Inventory of Depressive Symptomatology (QIDS-C)² are widely used and validated tools to assess the prevalence and severity of depressive symptoms in patients with MDD. These scales are used both in clinical research and practice. The PHQ-9 assesses the prevalence over the past two weeks of the nine criterion depressive symptom domains that define a major depressive episode based on the Diagnostic and Statistical Manual of Mental Disorders (DSM-5).³ It is a self-administered questionnaire that can be completed in a few minutes, making it a convenient tool for clinicians and patients. The PHQ-9 has demonstrated good psychometric properties in various populations, including adults^4,5 and adolescents,⁶ and has been translated into multiple languages.^7–13

The QIDS-C consists of 16 items that cover the same nine DSM-5 criterion symptom domains; it has clinician report and self-report versions.^2,14 The QIDS-C has demonstrated good psychometric properties, including high internal consistency, test-retest reliability, and strong convergent validity with other depression measures.¹⁵ In recent years, the 5-item Very Quick Inventory of Depressive Symptomatology (VQIDS-C)¹⁶ was developed as a more convenient method for rapid clinical assessment and is effective at evaluating the core features of depression. The self-report version of the QIDS and PHQ-9 have been compared in Asian primary care settings and shown good internal consistency.¹⁷ However, these measures have not been evaluated together in a large real-world US cohort. Previous research has demonstrated the value of linking depression scales such as the PHQ-9 and Hamilton Rating Scale for Depression (HAM-D)¹⁸ and Montgomery-Åsberg Depression Rating Scale (MÅDRS), Self Rated Scale (SRS), PHQ-9 and The Beck Depression Inventory-II (BDI-II).¹⁹ Given the high prevalence of MDD and the need for an accurate and reliable assessment of this disorder, validating the PHQ-9, QIDS-C, and VQIDS-C using real-world data is paramount. In this study we aimed to answer the following research questions: 1) How do the individual symptom domains perform in relation to the overall trait when evaluating the QIDS-C, VQIDS-C, and PHQ-9 in the same patients?; and 2) Can conversion tables be developed to estimate total scores among the three rating scales?

Methods

Participants

This study leveraged EHR derived, de-identified data from the NeuroBlu Database (Version 23R1) a longitudinal behavioural health real-world database comprising both structured and unstructured patient-level clinical data.²⁰ At the time of this study, the NeuroBlu Database included data from over 30 behavioural health centres across the US spanning over 20 years, consisting of data from over 1.4 million patients.

Patients were selected from the NeuroBlu Database if they had at least one recorded non-psychotic MDD diagnosis and did not have a record of schizophrenia, schizoaffective disorder, and bipolar disorder at any time (Supplemental Table 1). Patients were also required to have a record within the EHR of both the QIDS-C and the PHQ-9 (after MDD diagnosis) on the same day.

Institutional review board (IRB) approval of the study protocol, including a waiver of Health Insurance Portability and Accountability Act authorization, was obtained prior to study conduct, and covers data originating from all sites represented. Approval was granted by the WCG IRB. (The Holmusk Real-World Evidence Parent Protocol; IRB registration number 1–1470336-1; Protocol ID HolmuskRWE_1.0).

Measures

Demographic characteristics and clinical features were based on documentation at the time of entry into the EHR (sex, race, ethnicity) or baseline (the first instance in which QIDS-C and PHQ-9 are recorded on the same day).

PHQ-9 is a 9-item self-report instrument designed to assess the prevalence of depressive symptoms in patients over the last two weeks. The items on the scale correspond to the nine diagnostic criteria for major depressive disorder from DSM-5 (eg depressed mood, low interest, sleep disturbance, fatigue, appetite changes, etc.).³ The prevalence of each symptom item is scored on a four-point Likert scale ranging from 0 (not at all) to 3 (nearly every day). The total score ranges from 0 to 27, with higher scores indicating greater presence of depressive symptoms.

The QIDS-C is a 16-item clinician-rated scale developed to measure the severity of depressive symptoms in both research and clinical settings.^2,14 The 16 items cover the same nine DSM-5 criterion symptom domains of a major depressive episode as the PHQ-9. Each item is rated on a four-point Likert scale, ranging from 0 to 3 with responses largely aimed at defining the overall severity of the symptom over the prior 7 days. Items 1–4, 6–9, and 15–16 assess sleep, appetite/weight, and psychomotor symptoms, respectively. For these three domains, only the highest score is counted towards the total QIDS-C score. The total score, therefore, ranges from 0 to 27, with higher scores indicating more severe symptoms of depression.

The VQIDS-C consists of five items selected from the longer 16-item QIDS-C^16,21,22 to reflect core depressive symptoms (sad mood, self-outlook, involvement, fatigue, and psychomotor changes) modelled after the 6-item Hamilton rating scale for depression.²³ The brevity of the scale allows for smart phone use. VQIDS-C scores range from 0 to 15, with higher scores indicating more severe depressive symptoms. This tool is sensitive to changes in adults¹⁶ and has been validated through linking to other depression scales.²²

Removal of Invalid Data

When linking psychometric scales, it is essential to ensure that the conversion system relies on valid data, the removal of invalid data prior to linking scales is a practice recommended by previous studies.^24,25 The validity of data can be assessed using the similarity of scores among patients on comparable scales. For instance, if a patient’s overall score on the PHQ-9 indicates severe depressive symptomatology, a similar score should be reflected on the QIDS-C. Invalid data was therefore eliminated by plotting the total PHQ-9 score against the total QIDS-C score for each patient. The patients with a more than 10-point difference between the scores, which is indicative of a two-severity category difference were then removed, this process ensures the reliability and accuracy of the link between the scales. This threshold was based on the clinical opinion that it is unlikely that a patient would move two severity categories in the space of a week, as the PHQ-9 captures a two-week period compared to the QIDS-C which is 7 days.

Data Analysis

Descriptive statistics were reported for demographic characteristics and baseline clinical variables. Continuous variables were summarised using mean with standard deviation (SD) or median with interquartile range (IQR), as appropriate. Categorical and ordinal variables were summarised using frequencies and percentages.

Classical Test Theory (CTT) was used to evaluate the internal consistency of the PHQ-9, QIDS-C, and VQIDS-C. CTT tests the reliability and validity of a scale based on its items. CTT assumes that each observed score is a combination of an underlying true score representing the latent trait (depressive symptomology) and unsystematic error. Cronbach’s alpha and domain-total correlations were calculated to determine the extent to which domains on the scales were interrelated and consistently measured the underlying latent trait (depressive symptomology). Values of Cronbach’s alpha range from 0 to 1, with values greater than 0.70 considered acceptable. For domain-total correlation: values greater than 0.3 indicate that a domain discriminates well.

Item Response Theory (IRT)²⁶ is a statistical framework that models the relationship between individual test domains and an underlying latent trait or construct. Unlike CTT, it does not assume that all domains contribute equally to the underlying latent trait or construct, but it does assume the unidimensionality of each scale. Unidimensionality was assessed using parallel analysis.²⁷ A graded response IRT model was used in the current study as previously described.²⁸

IRT estimates discrimination (a) and difficulty (b) parameters for each domain in each scale. The discrimination parameter measures how effectively a domain can differentiate between individuals with different levels of depression. The difficulty parameters represent the level of depression at which an individual has a particular probability of endorsing a specific response category for a domain. In this context, b0 indicates the depression level at which a subject would be equally likely to score either “0” compared with “1”, “2”, or “3”. Similarly, b1 represents an equal likelihood of scoring “0” or “1” versus “2” or “3”. Lastly, b2 indicates an equal likelihood of scoring “0”, “1”, or “2” versus “3”. IRT was also used to calculate the Test Information Function (TIF) which is calculated by the addition of the information of every domain in a scale. The information refers to the precision at which the latent trait can be estimated by each domain.

IRT methods as described by previous literature^29,30 were used to link the total scores for each scale. An individual’s IRT latent trait score was estimated by multiplication of the item response category characteristic curve (IRCCC) corresponding to patients’ response patterns to represent the posterior distribution, and an average of the distribution was taken. This IRT latent trait score measures depressive symptomology in standard deviation units where 0 represents the average. This IRT latent trait score is then averaged for every individual with a specific summary score total for each scale, this is then linked to a corresponding summary score for a different scale depending on which score has the closest IRT latent trait score, creating a conversion table.

The linking of scales was then validated using multiple methods (Correlation, Root Mean Square Error (RMSE), and Kappa) to compare the score estimated by the conversion with the actual score reported for that patient. Firstly, the correlation coefficient between the estimated scores and the actual score reported for that patient was calculated, 0.7–0.9 represents a high correlation. Secondly, RMSE, which represents the average difference between the actual and estimated scores was calculated. As RMSE is scale-dependent, the interpretation differs by scale, but a lower RMSE is better. Finally, the Kappa was calculated to represent the level of agreement between the estimated severity category (none, mild, moderate, severe, very severe) and the actual score severity category. Kappa ranges from −1 to 1, where 1 indicates perfect agreement, 0 indicates agreement due to chance, 0.21–0.40 represents a fair agreement and 0.41–0.60 represents a moderate agreement.³¹

Data Privacy

The data ingestion and aggregation process comply with HIPAA guidelines, including applying the Safe Harbor definition for de-identified data.³²

Software

The following programs were used for the various analysis: ltm version 1.2-0 for the IRT graded response model, mirt version 1.38.1 for the confirmatory factor analysis, psych version 2.3.3 for Cohen’s kappa and exploratory factor analysis and paran version 1.5.2 for the parallel analysis.

Results

Demographics

Of the 2,231 study eligible patients, 75 had at least a 10-point difference between scores and were removed bringing the final cohort to 2,156. A comparison of demographic features between the final cohort and those that were excluded is shown in Supplemental Tables 2 and 3.

Table 1 shows the features of the analytic sample (n = 2,156) with a mean age of 36.4 years and included nearly 60% females. Most patients were white or black or African American. Nearly a third endorsed Latino or Hispanic ethnicity. Overall depression severity was moderate based on the PHQ-9 and QIDS-C.

Table 1 Demographic characteristics at first entry into the electronic health record and measurements after first MDD diagnosis

Classical Test Theory

CTT was used to evaluate the psychometric properties of three scales: PHQ-9, QIDS-C, and VQIDS-C. The internal consistency of the scales was examined using Cronbach’s alpha and domain-to-total score correlation, the coefficients for PHQ-9, QIDS-C, and VQIDS-C were 0.9, 0.8, and 0.7, respectively, with the domain-to-total score correlation ranging from 0.3 to 0.8 (Supplemental Table 4). All three scales therefore demonstrated acceptable internal consistency.

Unidimensionality

Parallel analysis was used to assess the unidimensionality of PHQ-9, QIDS-C, and VQIDS-C. For all three scales, the first eigenvalues from the sample data were much larger than that of the randomly generated data (PHQ-9 – 3.4 vs 0.1, QIDS-C – 2.4 vs 0.1 and VQIDS-C – 1.6 vs 0.1). The second eigenvalues from the sample data were much lower than the first values and were similar to or lower than those of the randomly generated data (PHQ-9 – 0.1 vs 0.1, QIDS-C – 0.1 vs 0.1 and VQIDS-C - −0.003 vs 0.1). These results provide evidence of unidimensionality.

Item Response Theory

The IRT parameter estimates for the questions within each scale are shown in Table 2. Questions 1, 2, and 4 of the PHQ-9 pertaining to little interest or pleasure, feeling down, depressed, or hopeless, and feeling tired or having little energy were found to be the most sensitive in distinguishing depression. Question 9 of the PHQ-9, relating to suicidal thoughts, required higher levels of depression to be endorsed. Question 13 of the QIDS-C assessed general interest and was the most sensitive at discriminating depression, this was also found for the VQIDS-C. Both question 12 relating to suicidal thoughts, and questions 15 and 16, pertaining to psychomotor symptoms, required higher depression for endorsement. The TIF for PHQ-9, QIDS-C, and VQIDS-C were 30.3, 25.8, and 17.8, respectively, suggesting that PHQ-9 provided the most accurate measure of depressive symptomatology (Figure 1). Table 3 and Figure 2 links the scores between PHQ-9 and QIDS-C, and VQIDS-C. The validation of the estimated scores compared with the true scores is shown in Table 4.

Table 2 Item Response Theory parameter estimates for PHQ-9, QIDS-C, and VQIDS-C (N = 2,156)

Table 3 Linking of the Summary Total Scores of PHQ-9, QIDS-C, and VQIDS-C

Table 4 Validation of estimated scores vs actual reported scores

Figure 1 Test Information Functions for the PHQ-9, QIDS-C, and VQIDS-C. Abbreviations: PHQ-9, Patient Health Questionnaire-9; QIDS-C, clinician rated Quick Inventory of Depressive Symptomatology; VQIDS-C, v5-item Very Quick Inventory of Depressive Symptomatology.

Figure 2 Linking the total PHQ-9, QIDS-C, and VQIDS-C scores based on the IRT latent trait score. Abbreviations: IRT, Item Response Theory; PHQ-9, Patient Health Questionnaire-9; QIDS-C, clinician rated Quick Inventory of Depressive Symptomatology; VQIDS-C, 5-item Very Quick Inventory of Depressive Symptomatology.

Discussion

The current study assessed the convergent validity, reliability, and interconversion of the PHQ-9, QIDS-C, and VQIDS-C in a large real-world cohort of adults with a non-psychotic MDD diagnosis. Results showed that all the scales were unidimensional and demonstrated good internal consistency. This is consistent with findings from two previous analyses in adult populations, the first in 400 Singaporean primary care patients (Cronbach’s alpha: PHQ-9 = 0.87 and QIDS-SR = 0.79)¹⁷ and the second in 297 inpatients in China with a diagnosis of depression (Cronbach’s alpha: PHQ-9 = 0.88 and QIDS-SR = 0.83).³³ Our findings are also consistent with a recent evaluation of PHQ-9, QIDS-SR, and VQIDS-SR in adolescents with depression (Cronbach’s alpha: PHQ-A (adapted for adolescents) = 0.86, QIDS-SR = 0.80 and VQIDS-SR = 0.76).³⁴

All the scale items related to “lack of interest in things” and “feeling tired” had a high level of discrimination. This finding reflects previous research that found “lack of interest in things” was the most sensitive at distinguishing levels of depression for PHQ-9, and “feeling tired” was the most sensitive for QIDS-SR and VQIDS-SR.³² For the PHQ-9, “feelings of sadness” also highly discriminated levels of depression. This item was less prominent in the QIDS-C and VQIDS-C, with the “view of self” as a highly discriminating factor for these scales. Items related to “suicidality” or “psychomotor symptoms” required a higher degree of depression to endorse.

In alignment with our findings, the symptoms associated with MDD exhibit a notable convergence across various dimensions. This convergence is observed regardless of the assessment method, whether through prevalence assessment or evaluation of symptom severity, and irrespective of whether self-reported (PHQ-9) or assessed by clinician rating (QIDS-C). Specifically, the symptoms harmonize around five pivotal dimensions, encompassing mood, interest, concentration, energy, and self-view. In contrast, psychomotor changes and suicidal ideation demonstrate weaker associations, as evidenced by their relatively lower correlation coefficients. The literature supports the idea that suicidal ideation is not universally present in all patients,³⁵ and its inclusion as a marker of overall depression severity could be questioned due to its cross-diagnostic nature and sensitivity to contextual variables.³⁶ Similarly, research highlights that psychomotor changes necessitate a higher level of severity for recognition.^37,38 Particularly intriguing is the inclusion of four out of the five core dimensions within the VQIDS-C. This observation prompts consideration for the replacement of the psychomotor aspect derived from the six-item Hamilton with the dimension of concentration, thereby contributing to a more comprehensive representation of the core features of depression.

This study represents the first effort to link PHQ-9, QIDS-C, and VQIDS-C in an adult population using real-world EHR data, resulting in valuable conversion tables. It is noteworthy to highlight that the conversion from QIDS-C and VQIDS-C to the PHQ-9 appears to exhibit lower reliability compared to the conversion of PHQ-9 scores to QIDS-C and VQIDS-C scores. This observation is underscored by the higher RMSE of 4 in the former case. However, a closer examination of the TIF supports this outcome and suggests that the slightly increased RMSE aligns with the underlying psychometric structure. Notably, our findings also underscore the viability of employing the abbreviated VQIDS-C if there is insufficient time to perform the full QIDS-C, given its considerably reduced administration time while still providing a reasonably accurate approximation of the QIDS-C total score. In addition, the viability of the VQIDS-C conversion to provide an approximation of the QIDS-C could be used to overcome issues of data missingness within EHR based datasets to increase the measures available for the analysis of real-world data. Clinicians and researchers can utilize the conversion tables to replace missing scores of patients with the corresponding scores, thereby maximizing the available data for patients or specific groups of interest. This enables more comprehensive descriptive and inferential analyses, enhancing the depth of understanding and insights gained from the data. It is important to utilize our conversion tables while keeping in mind that the agreement between any two instruments, irrespective of the statistical method used, can be influenced by variations in the population being studied.³⁹

There are several limitations of the current study. First, due to the real-world nature of the data source, the administration of the scales was not randomized, so that order effects were not controlled for. Order effects can manifest as primacy effects (early items having a more significant impact) or recency effects (later items having a greater impact).⁴⁰ The current data source omits details on the timing of the administration of the scales, however, if these data were available the impact of such order effects could have been more thoroughly investigated. Additionally, the real-world data used in this study were derived from community mental health centres and thus findings may not be generalizable to other settings, such as primary care or population studies. Second, the scales being equated were completed from different perspectives. The PHQ-9 was a patient self-report, whereas the QIDS-C was clinician reported. Third, there was a lack of complete scale range across all scale scores. For example, no patients scored 0 or 26 on the QIDS-C. Fourth, there are inherent differences in the scales such as the period that the scales correspond to, PHQ-9 – 2 weeks and QIDS-C – 7 days or severity versus prevalence. Fifth, limitations of IRT include sample dependence, assumption that item responses are independent, and the assumption that item parameters are invariant across time. These factors should be carefully considered when interpreting the current findings. Sixth, the current study considers only scales recorded at one moment in time, however, previous research has demonstrated the value of linking improvement or worsening from a baseline score.⁴¹ Future research using the scales in this study could consider such a longitudinal perspective. Seventh, although the removal of patients with a 10-point difference between PHQ-9 and QIDS-C was a study requirement, this criterion may have disregarded the discrepancy between the subjective and objective experience of depressive symptomatology. Indeed, previous research has shown that patient and clinician reported depression scale do not always demonstrate agreement,^42–44 and highlight the importance of capturing both viewpoints. Finally, the generalizability and applicability of the conversion tables should be considered. This sample was sourced from specialty mental health settings, where higher levels of comorbidity and more severe depression might be found compared with primary care settings or epidemiological samples.

Conclusion

PHQ-9, QIDS-C, and VQIDS-C were equated using real-world data from the EHRs of patients diagnosed with non-psychotic MDD. Establishing a link between these scales has the potential to open up new opportunities for measurement integration, research collaboration, and improved patient care. For clinicians and researchers, the conversion tables developed in this study may have several uses, including to address missing data in a patient’s clinical history and to harness data from multiple sources to produce larger more representative cohorts for research studies. Despite the potential benefits of linking these scales it is important not to overlook the value of capturing multiple dimensions when measuring psychiatric constructs such as depression. We have demonstrated that variation in the perspective of the individual completing the scale, in addition to prevalence and severity of symptoms may deferentially contribute to depressive symptomology.