Validation of the 4-Item and 10-Item Uncertainty Stress Scale in a Community-Based Sample of Chinese Adults

Introduction

Uncertainty is an increasingly common feature in modern society, where individuals often lacked of sureness about something (like life-threatening and fearful COVID-19, especially at the starting stage) in their daily lives.^1,2 A considerable amount of empirical evidence supports the notion that uncertainty is a powerful stressor that is strongly related to physical, behavioral, and mental health.^2–5 Therefore, it is of great importance to appropriately measure uncertainty in order to better understand this psychological construct and to develop efficient interventions (eg, mindfulness and exercise) to effectively cope with its associated negative consequence.^3,6

Figure 1 ROC curve analysis for predicting severe perceived stress (>3) (A) ROC curve for USS-10; (B) ROC curve for USS-4.

The literature dealing with instruments to operationalize and measure uncertainty has mainly focused on the clinical context of uncertainty in illness or assessed personality traits related to the intolerance of uncertainty.^3,7–11 For clinical research, the Mishel Uncertainty in Illness Scale (MUIS) was developed to measure illness uncertainty in hospitalized individuals; the original scale was later modified to measure uncertainty associated with living with chronic illness and included subscales evaluating ambiguity, complexity, unpredictability, and lack of information across the disease trajectory.^9,12 For non-clinical research, the Intolerance of Uncertainty Scale (IUS) was initially developed in the French language to measure an individual’s emotional, cognitive, and behavioral reactions to ambiguous and uncertain situations and consisted of 27 items with a five-factor structure.¹³ A stable 12-item IUS was later created which consisted of two factors: prospective anxiety and inhibitory anxiety.^7,14–16 In addition, the Uncertainty Response Scale (URS) is another psychometric instrument used to assess emotional uncertainty, desire for change, and cognitive uncertainty.⁸ Taken together, there are several instruments to quantify the construct of uncertainty that primarily focus on the cognitive aspects of uncertainty (ie, MUIS), one’s ability to tolerate it (ie, IUS), or a combination of the former two factors (ie, URS). However, existing instruments do not explicitly assess stress stemming from uncertainty, restricting our ability to assess and intervene in a key clinical and theoretical outcome of living with uncertainty.

The assessment of uncertainty stress is of vital importance since it is associated with negative clinical and social outcomes. Considering that uncertainty is recognized as a major stressor, uncertainty stress has been defined as “perceived stress generated by uncertain events or situations”.^17–19 According to previous studies, uncertainty stress is associated with deliberate self-harm,⁴ unintentional injuries,²⁰ suicidal intentions,²¹ mental disorders,²² problematic alcohol use,²³ internet addiction,²⁴ late bedtimes, worse self-rated health, and poor academic performance²⁵ among college students. Moreover, uncertainty stress was observed to be positively associated with disease fear and negatively associated with self-efficacy and prevention behaviors during the COVID-19 pandemic among the general community population.²⁶ Considering the above-mentioned findings, research on uncertainty stress and its appropriate measurement is of great interest and significance for public health. Nevertheless, very few studies directly assessed perceived uncertainty from the perspective of stress tolerance. An exception is the Hilton Uncertainty Stress Scale, which was developed to assess the stress, threat, and positive feelings caused by an uncertain state.²⁷ However, this scale focusses on the assessment of uncertainty in illness-related situations.

Nevertheless, a specific tool to quantify uncertainty stress in non-clinical situations rather than illness-related situations was proposed in 2002.¹⁷ Based on the pervasiveness of uncertainty in the general society, Yang et al designed and developed the original 4-item and 10-item Chinese versions of the Uncertainty Stress Scale.^19–21 The Chinese version of the USS is designed as a measure of uncertainty stress in the social context and thus supplements available instruments that have been designed and evaluated to measure uncertainty stress in clinical situations such as acute and chronic illnesses. The USS-4 comprises four items measuring current life uncertainty, social change uncertainty, goal achievement uncertainty, and social value uncertainty. The original version of the USS-4 shows good reliability and validity and has been successfully and widely used in previous studies.^4,21,26,28 Given that the short item scale might influence the stability of the application, Yang et al expanded the USS to 10 items assessing the level of uncertainty in social situations and the abilities to cope with such situations.¹⁹ This USS-10 has already been applied in a study.²⁹

Although the reliability and validity of the USS were initially tested, the validity of the USS-10 has only been examined in a Chinese student population, and thus it remains unknown whether these findings can be generalized to other populations. To address this gap in the literature, we conducted a comprehensive validation for both the USS-4 and the USS-10 in a large cohort of community residents (n = 904) covering a wide age range (18 to 72 years). In this context, we hypothesized that (1) the USS has good psychometric properties in the community-dwelling sample, and (2) the USS-4 and USS-10 have comparable indices of reliability and validity.

Methods

Participants and Sampling Procedure

A cross-sectional survey utilizing a convenience sampling method was conducted online in February 2020. Twenty psychology students from one university in the Guangdong province were recruited as research assistants to distribute the survey links among the adults in their communities using social media (eg, WeChat groups, QQ groups, and other major social networking platforms). The survey was based on the Wenjuanxing Platform (https://www.wjx.cn/app/survey.aspx). In this study, adults older than 18 years of age who lived in the local community for the past 12 months were included. Of 1148 community residents who were contacted online, 1093 (95.2%) agreed to participate, and 904 (78.7%) of them completed the questionnaires. Participants took approximately 10 minutes to complete the questionnaires. The study protocol was approved by the Ethics Committee of Shenzhen University (Ethics Approval Number 2020009), and complies with ethical principles stated in the latest version of the Declaration of Helsinki. All participants provided written consent prior to the administration of the questionnaires.

Measures

Demographic Characteristics

Demographic data including the date of birth, gender, place of residence, ethnicity, marital status, educational attainment, occupation, and per capita annual family income were collected.

Uncertainty Stress Scale

Uncertainty stress was measured by the 4-item and 10-item Chinese Uncertainty Stress Scale (USS), which was developed by Yang et al.^4,17–19 The USS assesses the perception of stress in response to uncertain situations (eg, “life is impalpable, and fate is unpredictable”; “unexpected things often happen in life”; and “the world is changing too fast and I cannot keep up”). Items are rated on a 5-point Likert-type scale with scores ranging from 1 (very little stress) to 5 (extremely stressful). Items are summed to obtain a total score, with higher scores indicating a greater level of uncertainty stress. The Cronbach’s α coefficient for the USS-4 and USS-10 in this sample was 0.941 and 0.851, respectively.

Perceived Stress Scale, Chinese Version (CPSS)

Perceived stress was measured by the Chinese version of the 8-item Perceived Stress Scale (CPSS).^18,30,31 This scale consists of eight items that assessed the perception of stress during the month prior to the survey (eg, “How often have you been upset because of something that happened unexpectedly?”; “How often have you felt that you were unable to control the important things in your life?”; and “How often have you felt nervous and stressed?”). The items were rated on a 5-point Likert type scale and ranged from 1 (never) to 5 (very often). The item scores were summed to yield a total stress score. A higher total score reflects a greater level of perceived stress.^18,30,31 The internal reliability of the CPSS in this sample, measured by Cronbach’s α, was 0.805.

Data Analysis

Data were entered into Microsoft Excel and then imported into SPSS (version 22.0) for statistical analysis. A descriptive analysis was used to determine the sample distribution. The total score and item scores of the scales were described as the mean and standard deviation (SD). t-tests and one-way analysis of variance (ANOVA) were used to examine the difference in uncertainty stress by comparing the demographic subgroups. To determine reliability, internal consistency was examined using Cronbach’s alpha coefficients. Alpha values were described as excellent (0.93–0.94), strong (0.91–0.93), reliable (0.84–0.90), and robust (0.81–0.83).³² An alpha value of 0.70 or above indicated an acceptable reliability.^32,33 To test for validity, the factor structure of the two USS versions (USS-4 and USS-10) was examined by confirmatory factor analysis (CFA) of the 904 community subjects who completed both versions of the USS. To validate that the USS-4 and USS-10 items were inter-correlated and applicable to factor analysis procedures, Bartlett’s test of sphericity was conducted. Principal component analysis (PCA) was performed to explore the factor structure. To assess the fit of the one-factor CFA model, five indicators from the Structural Equation Model analysis were calculated, including the comparative fit index (CFI), the Tucker–Lewis index (TLI), the goodness-of-fit index (GFI), the root mean square error of approximation (RMSEA), and the standardized root-mean-square residual (SRMR). A CFI or TLI score of 0.97 or more indicated good fit and above 0.95 indicated acceptable fit.³⁴ A GFI score above 0.95 indicated a good fit and above 0.90 indicated an acceptable fit. Both RMSEA and SRMR scores below 0.05 indicated a good fit, and an RMSEA below 0.08, as well as an SRMR below 0.10 indicated an acceptable fit.³⁴ To assess the criterion validity and to predict perceived stress by the USS, correlation analysis and receiver operating characteristic curve (ROC Curve) analysis were performed. The area under the ROC curve (AUC) is widely recognized as the measure of a diagnostic test’s discriminatory power. An AUC value of 0.5 or above indicates that it has a sufficient discriminative value.³⁵

Results

Sample Characteristics

The 904 community-based residents who participated in the current study included 41.7% males and 58.3% females. The mean age was 32.71 (SD: 10.99) years. More detailed information on the sociodemographic characteristics of the participants is presented in Table 1.

Table 1 Overview of the Demographic Characteristics of the Sample

Both the USS-10 and USS-4 had similar distributions across sociodemographic factors. Omnibus tests revealed significant differences in several demographic characteristics (age, place of residence, marital status, occupation, and household annual income). Uncertainty stress was highest among participants who were younger, rural, divorced or widowed, work in commerce/service/operations, or earn 60,000 RMB or less annually. Males had a higher uncertainty stress score than females on the USS-10 but not on the USS-4.

Reliability

In this study, the internal consistency of the two USS versions can be rated as good, based on Cronbach’s alpha coefficients of 0.941 in USS-10 and 0.851 in USS-4. For the USS-10, the Cronbach’s alpha values ranged from 0.934 to 0.937 when any one of the items was removed. For the USS-4, the Cronbach’s alpha values ranged from 0.795 to 0.819 if any one of the items was deleted. The composite reliability (CR) was also estimated to reflect the internal responsiveness. CR for the USS-10 and USS-4 were 0.950 and 0.900, respectively.

The total score and item mean scores for USS-10 and USS-4 are displayed in Table 2. Intercorrelations between the USS-10 items ranged from 0.548 to 0.783 suggesting acceptable homogeneity reliability (shown in Table 3). In Table 4, we present item-total correlations ranging from 0.768 to 0.836 in USS-10 and ranging from 0.816 to 0.852 in USS-4.

Table 2 Overview of the Results for Validity and Reliability Analysis for USS-4 and USS-10

Table 3 Overview of the Intercorrelations Between Uncertainty Stress Scale Items

Table 4 Overview of the Factor Loadings and Reliability Analysis for the 10-Item Uncertainty Stress Scale (USS-10)

Construct Validity

The one-factor structure of the USS-4 and USS-10 was supported by the PCA. Prior to the PCA, the Kaiser–Meyer–Olkin measure of sampling adequacy was 0.954 and 0.815 in USS-10 and USS-4. The Bartlett’s test of sphericity was significant for both USS-10 and USS-4 (P < 0.001), suggesting that the items were correlated and fit for factor analysis.

Convergent Validity

Table 4 demonstrates the loadings of ten items on a single factor, with loading values ranging from 0.767 to 0.840. Table 5 shows the factor loading for each item of USS-4 ranged from 0.817 to 0.856. The average variance extracted (AVE) was used to evaluate the convergent validity. The one-single factor in USS-4 accounted for 69.2% of the variance, and the value of AVE in USS-10 was 0.655, which suggested that the one-single factor accounted for 65.5% of the total variance.

Table 5 Overview of the Factor Loadings and Reliability Analysis for 4-Item Uncertainty Stress Scale (USS-4)

Factor Structure Model Fit

As shown in Table 6, the unidimensional model fits the data quite well. Goodness-of-fit results indicated a good to excellent model fit for both USS-4 and USS-10. Specifically, CFI (0.99), TLI (0.98), and GFI (0.99) for USS-4 were all higher than those of USS-10 (0.96, 0.95, and 0.94, respectively). RMSEA (0.08) and SRMR (0.03) values for USS-4 were lower than those of USS-10 (0.09 and 0.02, respectively).

Table 6 Overview of the Goodness-of-Fit Indices of USS-10 and USS-4

Criterion Validity

External Responsiveness

As shown in Table 7, Spearman correlations indicated that USS-10 and USS-4 scores are both significantly correlated with CPSS-8 scores, with correlation coefficients of 0.497 and 0.475, respectively.

Table 7 Overview of the Results of the Correlation Analysis Between Different Scale Scores

ROC Analysis

The area under the ROC curve in Figure 1 ranged from 0.725 to 0.738, indicating that both versions of the USS had similar effectiveness as the perceived stress screening tool. To determine how well the USS performed to identify individuals with high uncertainty stress, sensitivity and specificity were further evaluated by ROC curve analysis. The sensitivity and specificity of USS-10 and USS-4 for predicting severe perceived stress are shown in Table 8.

Table 8 Overview of the Results of ROC Curve Analysis

Discussion

The results suggest that the USS-10 and USS-4 have an acceptable reliability, construct validity, and criterion validity which provides convincing evidence that both versions of the USS are reliable and valid self-report measures to assess uncertainty stress in community-based samples of Chinese adults.

It is worth noting that the mean score of uncertainty stress measured by the USS-4 among Chinese urban residents in 2002 was 2.04 (SD: 0.85) and that it was 2.74 (SD: 0.89) when assessed by the USS-10 in 2016 among college students.^18,19 The mean scores of uncertainty stress observed in the current study in the community population from the USS-10 and the USS-4 were 2.71 (SD: 0.94) and 2.73 (SD: 0.97), respectively. Since the original USS-4 was developed approximately two decades ago, the current findings of our study suggest that among the general population the level of uncertainty stress has considerably increased. These data implied that uncertainty pervades our daily lives during the transformation period of the contemporary society and supported the notion that severe uncertainty stress may become a major and global public health issue. Whether this increase is caused by a general rise in uncertainty stress, is specifically related to the current COVID-19 pandemic, or is a result of both should be assessed in more detail in future studies. However, there is some evidence that the COVID-19 pandemic has increased the level of uncertainty stress, and, in consequence, has fostered and/or exacerbated the occurrence of negative health events. For instance, one study reported that during the COVID-19 pandemic, healthcare workers with higher levels of health-related uncertainty suffered from more pronounced symptoms of anxiety, depression, and loneliness.³⁶ In addition, the COVID-19-related increase in the level of uncertainty stress has also been associated with poorer sleep status.³⁷ Although the above-presented evidence suggests that the COVID-19 pandemic is a major driver of uncertainty stress worsening physical and mental health,^1,6,36,37 future longitudinal studies assessing how uncertainty stress changes over time, accounting for potential future waves of COVID-19 spread, are needed to investigate this phenomenon in more detail.

Moreover, the findings of the current study highlighted that uncertainty stress is greater among younger adults, rural residents, divorced/widowed or unmarried adults, and adults with a lower socioeconomic level. Based on the negative health consequences of uncertainty stress, more attention should be paid to these vulnerable and socially disadvantaged populations. Given that the availability and psychometric evaluation of instruments to assess uncertainty stress is currently limited, the development, validation, and cross-cultural application of the USS are urgent for both research and practical application.

Reliability

Internal consistency Cronbach’s alpha values were greater than 0.80, which is considered as a good reliability for research purposes.³⁸ Cronbach’s alpha coefficients for the USS-4 and the USS-10 originally measured by Yang et al were 0.821 and 0.925, respectively.^18,19 With a sample of 4446 medical students, the internal consistency of the USS-4 was 0.81.^4,21 For a sample of 11,942 Chinese college students, Cronbach’s alpha value for the USS-4 was 0.79.^5,23,28,39 A panel study with 102 Chinese residents also exhibited a Cronbach’s alpha of 0.89 for USS-4.²⁶

Similar to the previous studies conducted in China, the Cronbach’s alpha values of the USS-10 and the USS-4 in this study were both greater than 0.85. Thus, our findings complement the available literature as they suggest that the USS-10 and USS-4 have a high internal consistency for the community population as well. Specifically, the alpha coefficient of the USS-10 was 0.941, suggesting excellent reliability which is better than that of the USS-4. Moreover, composite reliability with a value of 0.95 for USS-10 and 0.90 for USS-4 was higher than 0.70, which is generally considered excellent in behavioral research.⁴⁰ Given the alpha coefficient increases with the instrument’s length,³⁴ the USS-4 can still be considered a reliable tool to assess uncertainty stress.

Validity

Construct Validity

The findings of the CFA in this study paralleled the original studies during the development of the USS.^18,19 Specifically, the results for all factor loading values were greater than 0.75, revealing a unidimensional structure with strong loadings. The AVEs for both the USS-10 and the USS-4 were higher than 0.65 indicating a good convergent validity. Furthermore, values of CFI and TLI above 0.97, GFI above 0.95, and RMSEA and SRMR below 0.05 indicated a good fit of the data to the one-factor model.³⁴ RMSEA for the USS-10 (0.09) was slightly greater than the recommended acceptable fit cutoff (0.08), indicating that it was at the threshold of acceptability. Considering the results from SEM, the single-factor structure for the two versions of the USS was confirmed, whereas the USS-4 has a superior model fit as compared to the USS-10. Similar to previous studies, a shortened version of the scale would exhibit superior factorial stability while maintaining the original measure’s high reliability and construct validity.¹⁴ To our knowledge, this study is the first to investigate the goodness-of-fit of the USS in a large community-based sample of Chinese adults, which provides some evidence for the applicability of the USS in the general population.

Criterion Validity

In comparison with the perceived stress scale (PSS-8), both the USS-4 and the USS-10 showed acceptable sensitivity and specificity. Notably, the USS-10 demonstrated a higher diagnostic accuracy for perceived stress, with an AUC value of 0.738 as compared to the USS-4. The PSS is one of the most frequently used instruments to assess stress in chronic conditions and situations by asking whether people’s lives seem to be unpredictable and uncontrollable.³⁰ The PSS suggested an acceptable criterion standard for screening the uncertainty stress. Furthermore, both versions of the USS score were significantly positively correlated with the CPSS-8 score, indicating that they had good concurrent validity.

Implications

The findings from the current study have some implications for research and policy. Considering the findings of our study in conjunction with the existing literature, our study highlights that both the USS-4 and USS-10 are reliable and valid instruments that can be used in a variety of populations and settings (ie, from college students to the general community setting). Although the USS-10 has better internal consistency and criterion validity as compared to the USS-4, the data of reliability and construct validity supported that the overall performance of the USS-4 is almost equivalent to that of the USS-10. The shorter version (USS-4) provides the advantage that it is extremely easy to administer, simple to understand, and requires a minimum amount of time to be completed making it well situated for assessments in which time is a critical factor (eg, large-scale epidemiological studies with a large test battery). Thus, our findings support the notion that the USS-4 is an economical alternative to the USS-10 and can be used when the circumstances require such an application.

With respect to policy, we advocate that policymakers pay more attention to the phenomenon of uncertainty stress as it is linked to negative health consequences. More specifically, a regular monitoring by appropriate measurement instruments and the education for uncertainty stress as well as potential coping methods need to be placed on the agenda. In this context, we hope that both the USS-4 and USS-10 are seen as a useful tool for researchers and practitioners interested in the mental health promotion concerning the uncertainty stress.

Limitations

The study has some limitations that need to be acknowledged. Firstly, as we used a convenience sampling method, the generalizability of our findings is somewhat limited. However, the large sample size and geographic diversity of our community-based sample which consists of individuals with different sociodemographic backgrounds mirror, at least partly, the general Chinese adult population. Secondly, self-reported measures can be confounded by the different sources of bias such as the social desirability bias and the recall bias which, in turn, might have influenced the current findings. Thirdly, both the USS-4 and USS-10 were only applied and evaluated in adults living in mainland China but not in overseas Chinese populations. In addition, the validation of the Chinese version of the USS-4 and USS-8 did not comprise individuals of younger age groups such as children and adolescents. The previous points might limit the generalizability of our findings to some extent. Fourthly, there is no non-self-reported (objective) instrument available to directly quantify the level of uncertainty stress. Thus, future research is needed to investigate the relationships between subjective markers (ie, USS-10 and USS-4) and objective markers of uncertainty stress. Finally, because this survey was cross-sectional, test–retest reliability and sensitivity to change over time were not assessed.

Conclusion

Based on the findings from our community-based study which includes a large sample of Chinese adults and which indicates that both scales (ie, USS-10 and USS-4) have appropriate psychometric properties, the current study provided robust evidence that both the USS-10 and USS-4 are well suited to study uncertainty stress in the general and adult Chinese population.