The Chinese Version of the Perceived Stress Questionnaire-13: Psychometric Properties and Measurement Invariance for Medical Students

Introduction

Ever since Hans Selye, known as the founder of the stress theory, brought the concept of “stress” into the psychometric field,^1–4 researchers have tried to develop methods to measure this “nonspecific response” of the body to any demand. One of the most prevailing methods⁵ to evaluate stress is the psychological approach based on the transactional model. The transactional model focuses on measuring a person’s subjective evaluation of his or her abilities to cope with specific events or changes and the assessment of subjects’ emotional response to the experiences.⁶ Rather than a plain outcome of an individual lacking balance between objective demands and coping capability, through the psychological approach, stress was conceptualized as a complicated combination.^7,8 Then, “the feelings or thoughts that an individual has about how much stress they are under at a given point in time or over a given period”⁸ was concluded as the conception of so-called “perceived stress”.^7,9 As yet, the direct relationships between stress and study-life conflicts, alertness promotion, symptoms of depression or anxiety, and even sleep disorders are well documented.^10–16 Furthermore, the indirect mediating role and moderating effect of stress between various health outcomes are gradually understood.^17–21 It has also been reported that stress may relate to a greater risk of infectious disease, tumors, immune disease, cardiovascular disease, and chronic disorders such as gastroenteritis, diabetes, and asthma.^22–25 Thus, how to measure this perception is increasingly getting researchers’ attention.

The Perceived Stress Questionnaire (PSQ)²⁴ is used internationally to measure perceived stress and has shown dependable reliability and validity. It aims at measuring the level of instinctively perceived and withstood current stress at the cognitive layer during the last four weeks (one-month recent form, PSQ-R) or for the last two years (two-year general form, PSQ-G). The PSQ is a self-report instrument that consists of 30 items on a four-point scale, and it measures seven factors (harassment, overload, irritability, lack of joy, fatigue, worries, and tension). The PSQ-30 has shown good reliability with Cronbach’s alpha values 0.900–0.920 and good validity as it significantly correlates with other reliable scales or symptoms (ie, the Perceived Stress Scale [PSS], the State-Trait Anxiety Inventory [STAI], the Center for Epidemiologic Studies Depression Scale [CES-D]; r = 0.400–0.750). Short forms of the PSQ-30 have been proposed consisting of 8, 13, 16, 20, 21, 24 items and tested in American, Chinese, German, Swedish, and Spanish samples.^{11,12,23,26–28}

As is well known, the two most prevalent approaches applied in shortening scales are classical test theory (CTT) and item response theory (IRT).^29,30 Nevertheless, CTT has a commonly unrealistic assumption: the obtained score is seen as absolutely true with no measurement error.³¹ What is more, the measurement properties obtained under CTT are dependent on sample types, which indicates that these obtained properties need to be multiply tested in other cultures or populations.³¹ Conversely, IRT is independent of samples and centers mainly on a relationship between an individual’s trait level and how they respond to items. Thus, IRT has shown its strength by selecting items with more information about the targeted trait.³² Among those short PSQ versions, only the 13-item PSQ was accordingly translated into Chinese, tested in the Chinese population, shortened under IRT, and validated multiple measurement properties.^28,33

The 13-item PSQ might provide a more efficient and reliable way to screen individuals with high-level stress, further assess other mental conditions, and discover the possible association between stress and health disorders. Considering that the validated Chinese version of the Perceived Stress Questionnaire-13 (PSQ-C-13) was shortened under IRT with a single-measure survey, the measurement properties of the PSQ-C-13 need to be assessed with repeated measures. We hypothesized that the measurement invariance across multiple groups and time points can be supported. Therefore, the current study is to further evaluate the psychometric properties and measurement invariance of the PSQ-C-13 based on the previous study.²⁸

Materials and Methods

Procedures

Medical students of various grades in a university in Hangzhou, China, were recruited for this study from September until December 2021 using a convenient sampling method. All procedures followed relevant ethical principles for medical research involving human subjects in the Declaration of Helsinki.³⁴ The research protocol was approved by the Institutional Review Board of School of Public Health, Hangzhou Normal University (Reference No. 20210014). All research participants provided informed consent. After conducting the paper-and-pencil survey twice, student ID was used to match the same participant between tests. The total number of valid questionnaires included is 309, which has reached the minimum recommendations: 1) the sample size should be included from 3 to 20 times the number of items in the scale;^35–37 2) based on the ratio of items-to-factors in our study is approximate to 6, the sample size should be higher than 200.³⁸

Measures

Socio-Demographic

Demographic data were collected on gender, age, home location (urban, rural, suburban), single-child status (yes, no), monthly households’ income (unit: CNY [1 CNY ≈ 0.160 US dollars]; < 5000, 5000–9999, 10,000–14,999, 15,000–19,999, 20,000–24,999, 25,000–29,999, 30,000–34,999, 35,000–39,999, ≥ 40,000), and part-time status (yes, no), for further analysis.

Perceived Stress Questionnaire (Chinese Version)

The validated Chinese version of the Perceived Stress Questionnaire-13 was tested with medical students.²⁸ A previous study showed that the PSQ-C-13 with two factors (constraint and imbalance) had satisfactory fit based on Confirmatory Factor Analysis (CFA; Comparative Fit Index [CFI] = 0.968, Tucker Lewis Index [TLI] = 0.961, Root Mean Square Error of Approximation [RMSEA] = 0.070).²⁸ The PSQ-C-13 has also shown good internal consistency (Cronbach’s α = 0.878, Guttman’s λ-2 = 0.880, and McDonald’s ω = 0.880) and test–retest stability (Intraclass correlation coefficient = 0.805, 2-day interval).²⁸ Moreover, the PSQ-C-13 correlates moderately to strongly with the PSS-C-10 (r = 0.777), the Short Form-8 Health Survey (r = −0.595), and the Goldberg Anxiety and Depression Scale (r = 0.584).²⁸

Perceived Stress Scale (Chinese Version)

The Chinese version of the Perceived Stress Scale-10 (PSS-C-10) is a five-point Likert scale with two components (positive and negative).^39–41 It is designed to measure the degree to which individuals have encountered “unpredictable, uncontrollable, and overloaded situations in the previous month”. The PSS-C-10 has shown sound validity and reliability with a CFI^36,41 of 0.980, an RMSEA^41,42 of 0.048, and a Cronbach’s alpha⁴¹ of 0.860. Consequently, the PSS-C-10 was used in this study to test convergent validity.

Data Analysis

The database was built using EpiData (version 3.1) software. R (version 4.1.2) software was adopted to perform all statistical analyses. Descriptive statistics were used to describe the characteristics of included participants. Absolute numbers and relative numbers were calculated under each demographic variable for both tests. The multivariate normality test of scores was performed using “MVN” package.⁴³ Measurement properties were assessed guided by the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) methodology user manual and taxonomy of measurement properties.^44–46

Structural Validity

Structural validity was assessed by applying CFA with a Weighted Least Squares Mean and Variance adjusted (WLSMV) method to accommodate ordinal indicators.^47,48 Using R package “lavaan”,⁴⁹ the goodness-of-fit of the PSQ-C-13 factor structures under IRT was assessed by means of CFI,³⁶ TLI,³⁶ and RMSEA.⁴² All the goodness-of-fit indices were reported scaled, considering they are more exact than unscaled indices.⁵⁰ For these indices, models with CFI close to or greater than 0.900, TLI close to or greater than 0.900, and RMSEA close to or smaller than 0.080 indicate good model fit.⁵¹

Convergent Validity

To assess convergent validity, we hypothesized that the PSQ-C-13 has a:

Hypothesis 1: Strong positive correlation (≥ 0.500) with the PSS-C-10, given the fact that both instruments are supposed to measure related constructs (perceived stress).⁵²

Convergent validity was considered sufficient if at least 75% of the correlations were as expected.⁴⁶

Measurement Invariance

Using the “semTools” package in R, configural, thresholds, metric, scalar, and strict measurement invariance was examined across multiple groups and across time.^48,53,54 First, in the configural invariance model, all the parameters were free, to test whether the proposed factor structure fits equally across groups. Second, the thresholds were constrained to test equality in each group in the thresholds invariance model. Third, in the metric invariance model, the thresholds and the factor loadings were constrained to be equal in each group. In the fourth scalar invariance model, the thresholds, the factor loadings, and observed variable intercepts were constrained to test whether they were equal in the groups. Last, in the strict invariance model, the thresholds, the factor loadings, intercepts, and the residuals were constrained to be equal across groups. The following scaled indices of goodness-of-fit were used as outcomes of the multi-group and longitudinal measurement invariance: CFI,³⁶ TLI,³⁶ and RMSEA.⁴² And we evaluated changes of fit-indices (Δ) within CFI, TLI, and RMSEA: models with ΔCFI lesser than 0.010, ΔTLI lesser than 0.010, and ΔRMSEA lesser than 0.015 were favored.^55–57 We also compared Chi-square (χ²) and Chi-square change (Δχ²) between models. However, because of the dependency of Chi-square on sample size,⁵⁵ we mainly focused on evaluating fit with ΔCFI, ΔTLI, and ΔRMSEA.

Internal Consistency

The internal consistency of the PSQ-C-13 was determined with the “ufs” package after conducting a CFA.^58–60 Ordinal forms of Cronbach’s alpha and McDonald’s Omega were used to access the internal consistency, where values equal to or higher than 0.700 were considered good.^56,57

Test–Retest Reliability

Intraclass correlation coefficient (ICC) was computed to assess the test–retest reliability using the “irr” package.^61,62 Divided by the total variation, ICC centers on the interindividual variation in the population expressed as a ratio between 0 and 1.⁶³ An ICC value close to or higher than 0.700 was considered sufficient.^62,63

Standard error of measurement (SEM),⁶² which helps appraise the variability of measurement errors and determine measurement precision, was also computed as “standard deviation × sqrt (1-ICC)”. SEM will be assessed as a supplement indicator in test–retest reliability appraisal.

Results

Participants

The participants comprised medical students between ages from 17 to 23, and the average age is 19.611 ± 1.367 (mean ± standard deviation) years (Table 1). Average time interval of two measurements between baseline and follow-up was 7 days + 10.850 hours. Most item scores of the PSQ-C-13 did not obey in multivariate normal distribution, as assessed by checking skewness, kurtosis, and P values (P < 0.001) (Supplementary Material Table S1).

Table 1 Characteristics of Participants (N = 309)

Measurement Properties

Structural Validity

Concerning the previous evidence of IRT analysis, CFAs were conducted to examine one-factor, two-factor, and second-order models (Table 2). Based on the fit indices of the models, the two-factor models showed relatively better fit. Therefore, a two-factor model with constraint and imbalance was confirmed for further analysis.

Table 2 Confirmatory Factor Analysis of Alternative Factorial Solutions of the PSQ-C-13 (N = 309)

Convergent Validity

The correlation matrix of the PSQ-C-13 and the PSS-C-10 is depicted in Figure 1. The left part of the figure shows the inter-factor correlations, whereas the right part shows convergent validity correlations. The inter-factor correlations ranged from 0.602 to 0.952, indicating moderate to high correlations. Subscales and total scale of the PSQ-C-13 moderately correlated with the PSS-C-10 and its subscales, especially negative feelings correlated higher than positive feelings. Overall, convergent validity of the PSQ-C-13 and its subscales was relatively satisfactory.

Figure 1 Inter-factor, factor-total, and convergent validity correlations between the PSQ-C-13 and the PSS-C-10. Abbreviations: Con, constraint; Imb: imbalance; PSQ-13, Perceived Stress Questionnaire-13; Neg, negative; Pos, positive; PSS-10, Perceived Stress Scale-10; “T” after each scale, T-scores; T1, time 1; T2, time 2. Notes: Spearman correlations. Color gradient represents correlation strength. Red represents positive correlation. T-scores were standardizedly calculated as “T index = (total raw score - scale items)/scale items × (item points - 1); T scores = T index × total raw score”.

Measurement Invariance

The two-factor model was chosen to conduct measurement invariance tests across subgroups and time points. The fit indices of most comparisons across subgroups were decent. Changes of the CFI, TLI, and RMSEA did not fall outside cutoff values and all remained in an acceptable range. As for the Δχ², comparisons across groups based on gender, home location, single-child, income, and part-time were found to be not significant. Despite the chi-square difference test was significant in the case of the scalar invariance, the other fit indices supported the invariance across age groups (Table 3).

Table 3 Tests of Measurement Invariances of the PSQ-C-13 Across Subgroups (N = 309)

As indicated by all fit indices, longitudinal measurement invariance was supported (CFI = 0.964–0.968, TLI = 0.958–0.968, REMSA = 0.049–0.056). The model comparison of the change in fit indices indicates that all models were supported, since ΔCFI, ΔTLI, and ΔRMSEA did not fall outside the proposed cut-off points. Δχ² showed a significant difference in the metric invariance (Table 4).

Table 4 Test of Measurement Invariance of the PSQ-C-13 Across Time (N = 309)

Internal Consistency

Table 5 illustrates results of internal consistency of the scales across the two time points. Internal consistency of the measurements was sufficient, according to Cronbach’s alpha (PSQ-C-13 = 0.830–0.922, PSS-C-10 = 0.744–0.906), and McDonald’s omegas (PSQ-C-13 = 0.828–0.923, and PSS-C-10 = 0.754–0.906).

Table 5 Internal Consistency and Test–Retest Reliability of the PSQ-C-13 (N = 309)

Test–Retest Reliability

The ICCs of the PSQ-C-13 for two time points met the threshold (ICCs equal to 0.802, 0.761, and 0.834, for constraint, imbalance, and total score, respectively). The SEMs were 2.598 (baseline) and 2.581 (follow-up) vs. 2.378 (baseline) and 2.527 (follow-up) in the PSQ-C-13 and the PSS-C-10, respectively. Overall, the reproducibility over time of the PSQ-C-13 was comparably good (Table 5).

Discussion

The present study supported the good psychometric evidence on the 13-item Chinese version of the Perceived Stress Questionnaire. This 13-item Chinese version was shown to be valid, reliable, and efficient, offering an alternative to the full form in a sample of medical students.

Structural Validity

The PSQ was previously developed and shortened as a seven-factor model,²⁴ six-factor model,¹⁰ and four-factor model.²³ The CFA fit results supported a two-factor construct (constraint and imbalance) of the PSQ using 13 items, which are complementary evidence for the previous study.²⁸ The similarities of confirmation on two-factor for the PSQ-C-13 illustrated a better fit than one general factor. Overall, compared to the seven-, six-, and four-factor model, the structure of the PSQ-C-13 may availably measure constructs (constraint and imbalance) of perceived stress.

Convergent Validity

Regarding convergent validity, our two hypotheses were supported. Spearman correlation between the PSQ-C-13 and the PSS-C-10 revealed that the two scales and their subscales covary significantly and share considerable amounts of variance.³⁰

The good convergent validity of this short form renders the PSQ-13 a suitable instrument for use in clinical settings, where time and respondent burden are important considerations. In addition, the good validity of both the general index (total score) and the two subdimensions showed implies that separate aspects of the stress experience can be captured relatively easily and concisely.

Measurement Invariance

The multi-group CFA fit-values showed that almost all were good, and only the Δχ² difference test was not supported in the case of age. As older age might face the problem of clinical internship, graduate, and post-graduate examination, this could be the reason that age group did not support scalar invariance during the first Δχ² test. However, considering Δχ² difference was mostly influenced by the sample size, we may still conclude that analysis of differences is feasible between groups and time points, according to changes in goodness-of-fit values.⁵⁵ Consequently, future studies are still needed to be applied in larger samples to affirm that these groups and time points are comparable or non-comparable.

Internal Consistency

Even using a shortened form of the Chinese PSQ, two forms of internal consistency reliability still displayed exceptional reliability. The sufficient results of internal consistency could be attributed to the IRT method used in the previous study; compared to the CTT method that assumes all scores were true with no bias, IRT approach may produce a relatively more reliable result in item-reduction.^29,32 Cronbach’s alpha and McDonald’s omega values were all higher than 0.900 for the PSQ-C-13. Even though the subscales of the PSQ-C-13 were lower than 0.900, all the values were still higher than 0.800. Collectively, we may conclude that the PSQ-C-13 has reliable internal consistency.

Test–Retest Reliability

Although test–retest reliability is commonly measured with Pearson’s or Spearman correlation, we applied the bias-sensitive intraclass correlation, which was above 0.750 in our study, indicating appropriate level of both stability and responsiveness to change over time.⁶⁴ Briefly, the relatively dependable internal consistency (alphas, omegas) and responsiveness (ICCs, SEMs) values showed decent reliability.

Strengths and Limitations

The first strength of the study is that we used the shortened PSQ-C under IRT, given that those other shortened forms based on CTT might ignore the systematic errors to biased results. The second strength is the follow-up test we performed, which was less done by previous researchers who intended to improve the performance of the PSQ.^{10–12,23,33} The last strength is the comprehensive measurement invariance (ie, multi groups and time points) offered a relatively new perspective compared to the previous studies that have not examined this.

However, this research contains some limitations. The first limitation of this study is that even if the sample size has met the minimum recommendations, it might still be relatively small. Further study on a larger sample of participants is warranted. Second, non-probability sampling method has the known drawback of making it difficult for us to determine if the population is adequately represented. Another limitation, also considered a major limitation, is that the population was not diverse: only in medical students of a university; this made the distribution of gender unbalanced and age group only falls between 17 and 23 years. Thus, selection bias may be produced for recruiting participants from one university; yet recruiting participants from one discipline may introduce information bias concerning medical students may report more negative mental symptoms.^65,66 This flaw also could be the reason that measurement invariance of subgroups was not fully upheld. Fourth, discriminant validity assessment has not yet been done due to time and resource constraints. Last, the information bias could exist since students might over- or under-report their symptoms and perceptions.

Future Directions

Taking all the conditions into consideration, future studies using the PSQ-C-13: 1) need to evaluate various populations with a larger sample size; 2) could be expanded to clinical trials, community nursing, and even psychometric screening; 3) could be applied as an efficient stress indicator in the multi-wave survey.

Conclusion

To conclude, the PSQ-C-13 contained valid and reliable measurement properties that allow research practitioners to assess perceived stress. Two dimensions including constraint and imbalance may availably reflect the degree of stress on participants. In consideration of its low respondent burden and sound properties, the PSQ-C-13 is an instrument with the potential for economically and efficiently assessing stress levels.