Development and Application of Global Health Events-Mental Stress Scale for Assessment of Medical Staff’s Acute Mental Stress Responses

Introduction

A large number of people worldwide are annually infected and killed by epidemic diseases.^1–5 Due to the tremendous pressure of the epidemic, medical workers fighting the epidemic are prone to negative impacts of physical deterioration and fatigue, including physical exhaustion and mental disorders. On the psychological level, such medical workers face traumatic reactions, interpersonal difficulties, communication problems, loss of faith in fairness and kindness, and severe mental disorders (eg, depression). These challenges further reveal the link among interpersonal difficulties, traumatic reactions, and negative clinical outcome such as suicidal behavior.⁶ In a previous study, measures of affective temperament-types were independently and more strongly associated with lifetime suicide attempts than was diagnosis of a major affective disorder or other variables.⁷

The development of public health strategies largely depends on social and behavioral changes. Therefore, it is crucial to evaluate and compare the behavioral and stress responses to public health events.⁸ Although several investigations have been concentrated on the psychological anxiety and sleep quality assessment for medical workers,^9–15 currently, no standardized, officially recognized, unified scale exists for the measurement of medical workers’ mental stress responses after experiencing public health events. Most people experience distress after their exposure to an extreme event. In case of good psychosocial resilience and access to social support, the distress of such people may be relatively transient, and adaptation processes can soon be initiated.¹⁶ However, in a minority of patients, a mental health disorder is developed from one to six months after a traumatic event; nevertheless, treatment after one year of such an encounter can often provide satisfactory therapeutic results.¹⁷ Therefore, the development of a mental stress scale for medical workers, especially after experiencing public health events, is essential for early diagnosis and timely treatment of mental disorders.¹⁸

The Impact of Event Scale-Revision (IES-R)¹⁹ is a tool utilized to evaluate post-traumatic stress disorder (PTSD). The original IES-R requires subjects to respond to their psychological conditions in a specific stressful environment in the past week. In the present study, we assumed that after experiencing public health events, medical workers would prefer the use of self-test scales specifically developed for the medical population rather than ones employed for assessment in the general population. Here, we propose GHE-MSS as an addition to the existing established measures of medical workers’ acute mental disorder rather than a replacement.

Methods

Ethics Statement

The present study was approved by the Ethics Committee of Shantou Central Hospital (Shantou, China; Approval No. 2020-Ke-Yan (011)). This research was conducted in compliance with the Declaration of Helsinki. Informed consent was obtained from all the participants.

Development of the Scale

Design of the Survey

Modifications to the original IES-R questionnaire were performed accordingly. We modified the original questionnaire as follows: we changed “Below is a list of difficulties that people sometimes experience after stressful events” to “The following questions relates to thoughts, feelings, and behaviors after participating in the treatment of public epidemic diseases”. In addition, we replaced “it” in the items 1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 17, 19, 20, and 22 with “the treatment experience”. We also added “After providing treatment” to items 2, 4, 10, 15, 18, and 21, and to change “that time” to “the treatment time” in item 14.

The Delphi Survey

Two rounds of the Delphi survey were undertaken to evaluate 22 items and validate the content. The first step of the Delphi survey was to determine the number of experts based on the specific research content. Usually, 17–30 experts are selected for exploring underlying information or assumptions. However, 4–16 experts could be selected for consultations to obtain satisfactory answers for an ordinary topic, such as slight modifications of an existing psychological scale.²⁰ In the present study, a total number of ten experts were selected by the convenience sampling method: three psychologists, five neuropsychologists, and two psychiatrists invited from a psychological services clinic, a neurology clinic, and a mental health center of two tertiary hospitals. In addition, experts could access the questionnaire through software, namely “questionnaire star and email”. The questionnaires that were not filled out in accordance with the instructions were treated as invalid and were not included in the data processing. The data collected in the first round, were directly displayed in the second round after statistical analysis. The second round of expert consultation questionnaires was issued, and data were collected for subsequent analysis. Finally, the experts made judgments based on the importance of each indicator. The newly developed scale, GHE-MSS, is a revised version of IES-R.¹⁹ The revised Delphi survey was previously applied to the Chinese version of the IES-R.²¹ The content of Chinese version of GHE-MSS was further verified by back-translation until the meaning of each item was matched with the original item of GHE-MSS. Finally, the Chinese and English GHE-MSS versions were released. The agreement of each item was evaluated using the five-point Likert’s scale (five = strongly agree, four = agree, three = neutral, two = disagree, and one = strongly disagree). The quality control of the Delphi survey was based on the following criteria: (1) The return rate of the questionnaire used to measure the positive degree marks given by experts on this questionnaire;²² (2) The authoritative coefficient (Cr) was composed of the basis of expert judgment (Ca) and the expert familiarity with the questions (Cs). The formula applied was Cr = (Ca + Cs)/2. The values of Ca and Cs were obtained through expert self-evaluation.²³ The mean and coefficient of variation were used to describe experts’ opinions. The screening and revision of the items were carried out on the basis of mean > 3.5 and coefficient of variation < 0.25.²⁴ The Kendall coefficient was implemented to indicate the level of consensus among experts’ opinions, where Kendall coefficient > 0.7 indicated credibility.²⁵ The authority degree of the experts was expressed by the authority coefficient, and an authority coefficient > 0.7 showed high authority of the experts.

Testing of the Proposed Scale

Participants were asked to complete an online socio-demographic questionnaire (eg, data on sex, identity, and technical titles) via WeChat and Wen-juan Star software. All personal information including demographic was anonymized to maintain and protect confidentiality. Then, the subjects were asked to fill the questionnaire after familiarizing with the introductory part attached to the form to better understand the aim of the research. Next, the participants were prompted to answer questions and to complete the GHE-MSS.

Here, the target subjects were medical workers who had participated in the frontline treatment of COVID-19 since its outbreak in December 2019. The inclusion criteria applied were as follows: (1) medical workers’ age ≥ 18 years; (2) medical workers with valid practicing certificates; (3) participation in the prevention, diagnosis, and treatment of COVID-19 for at least two weeks; and (4) voluntary participation in this research. Hospital administrative workers, professional researchers, non-clinical nurses, and non-doctors were excluded from the study cohort. The sample size in the primary survey was set to be at least five-fold higher than the number of the entries, and the sample size in the formal survey was set to be at least ten-fold higher than the number of the entries²⁶ (Figure 1).

1. The IES-R¹⁹ is a 22-item self-report measure that assesses subjective distress caused by traumatic events, but it is not appropriate for PTSD diagnosis. Items are rated on a five-point scale, ranging from zero (“not at all”) to four (“extremely”). The IES-R yields a total score (ranging from zero to 88), and intrusion, avoidance, and hyperarousal subscale scores can also be calculated. Scores exceeding a value of 24 are quite meaningful. Those with such high scores, who do not have full PTSD, would have partial PTSD or at least some of the symptoms.²⁷
2. PTSD Checklist‐civilian Version (PCL‐C). The PCL is a standardized self‐report rating scale for PTSD comprising 17 items that correspond to the key symptoms of PTSD. Currently, two versions of the PCL exist: 1) PCL-M that is specific to PTSD caused by military experience and 2) PCL-C that is applied to any traumatic events. PCL‐C was utilized in this study based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM‐IV). Scores for each question ranged from zero (not at all) to five (extremely). A total symptom severity score (range=17–85) can be obtained by summing the scores for each of the 17 items. The best PCL-C cut-off score is used to screen PTSD.^28–30 This approach for diagnosing PTSD has shown reasonable sensitivity (0.60) and a high specificity (0.99).³¹ The PCL‐C was used for the external validity of GHE-MSS in the present study.

Figure 1 Flow chart of the study process.

Primary Survey

Primary survey was conducted based on the results of the Delphi survey. For this purpose, 115 medical workers from the Emergency Department, Intensive Care Unit (ICU), and the Infection Department of a tertiary hospital in Guangdong Province (China) were enrolled in the primary survey conducted in March 2020. Principal component analysis was employed to evaluate the primary survey results. The reliability of the primary survey was analyzed by Cronbach’s alpha coefficient.³² The common-method variance was utilized to investigate the validity of the primary survey.

Formal Survey

The reliability and validity of the formal survey were assessed via the participation of 300 medical workers from the Emergency Departments, ICUs, Infection Departments, Respiratory Departments, and the Clinical Laboratories of five tertiary and secondary hospitals located in Guangdong Province (China). The formal survey was conducted in April 2020. The Cronbach’s alpha coefficient was employed to evaluate internal consistency, and the reliability and validity of the formal survey were confirmed. Bartlett’s test and the Kaiser-Meyer-Olkin (KMO) measure were used for the determination of the exploratory factor analysis adequacy. The KMO statistics ranged from zero to one, with values closer to one denoting greater adequacy of the factor analysis (KMO ≥ 0.6 (low adequacy), KMO ≥ 0.7 (medium adequacy), KMO ≥ 0.8 (high adequacy), and KMO ≥ 0.9 (very high adequacy)).^33,34

Clinical Evaluation

Here, 566 medical workers from the Emergency Departments, ICUs, Infection Departments, Respiratory Departments, and the Clinical Laboratories of five tertiary and secondary hospitals in Guangdong Province were surveyed in September 2020. We used the PCL-C to assess the consistency and validity of PTSD occurrence. The Pearson’s correlation coefficient (PCC) was employed to calculate the correlation coefficient between each item score and the total score. Confirmatory factor analysis (CFA) was next used to establish the validity of the scale structure.³⁵ A standard load factor > 0.7 was considered to indicate a strong correlation, whereas a factor < 0.4 represented a weak correlation between the item and the factor. Criterion-related validity was utilized to test the correlation between scores of the new scale and the standard scale. The correlation coefficient was within the range 0.40–0.80; the validity was considered to be confirmed at P<0.05.

Statistical Analysis

Data were statistically analyzed with SPSSAU 2016–2021 (Qing-Si Technology Ltd, Beijing, China) and GraphPad Prism 5.0 (GraphPad Prism Software Inc., San Diego, CA, USA) software. CFA was conducted using SPSSAU. Kline’s references³⁶ were utilized to analyze univariate normality (skew index ≤ 3.0; kurtosis index ≤ 10.0). Good model fit cut-off values adopted were those proposed by Hair et al³⁷ and Schumacker and Lomax³⁸ (samples with N > 250 and number of observed variables between 12 and 30): Chi-square test of the model fit (χ2/df) values less than 5 indicated a good model fit; comparative fit index (CFI) values close to 0.90 or 0.95 represented a good model fit; goodness-of-fit index (GFI) values close to 0.90 or 0.95 indicated a good model fit; standardized root mean square residual (SRMR) and root mean square error of approximation (RMSEA) values below 0.05 revealed good fit, and values between 0.05 to 0.08 with a CFI of 0.92 or higher indicated close model fit.³⁹ Convergent validity was established by the average variance extracted (AVE ≥ 0.50)³⁷ and composite reliability (CR ≥ 0.70).⁴⁰ In addition, we calculated the heterotrait-monotrait (HTMT) ratio of correlation through a multitrait-multimethod matrix among the GHE-MSS factors by using a value lower than 0.85 to indicate adequate discriminant validity.^40,41

Results

The Delphi Survey

Approximately 50% of the study participants were aged >40 years. Besides, 80% of the participants had a Bachelor’s degree or above, and 40% of them had work experience>15 years. Experts’ research fields were psychotherapy education or research, clinical psychotherapy, and psychological care (Table 1). The authority coefficient was within the range 0.63–0.93, and the overall authority coefficient was above 0.76, indicating that the experts had high authority. Two rounds of the Delphi survey were carried out for each group of experts. Ten questionnaires were sent out in each of the both rounds, which were then recovered. The recovery rate of the two rounds of the Delphi was 100%, indicating that the experts were highly concerned about the questionnaire. The Kendall coefficient in the first round of the Delphi survey was equal to 0.47(P<0.01). The mean number of entries in the first Delphi round was 4.73, with a coefficient of variation of 0.11. After the modification of experts’ opinions, the mean number of the entries in the second round of the Delphi was 4.85, with a coefficient of variation of 0.08. The Kendall coefficient in the second round was equal to 0.74(P<0.01). Therefore, experts’ opinions were relatively consistent and the survey results were accordingly reliable (Table 2).

Table 1 Experts’ Demographic Characteristics

Table 2 Recovery of the Questionnaire and Suggestions Offered by Delphi Consultation

Proposed Scale Testing

The number of participants in the primary survey was 115 (Table 3). Based on the results of the primary survey, a Cronbach’s alpha coefficient of 0.98 was determined. The range of common-method variance was 0.71–0.86. The sample size of the formal survey was 300 (Table 3). KMO value was 0.97, and the Bartlett’s test of sphericity showed that degree-of-freedom value was equal to 231 (P<0.01). The percentages of the variance of the three measures were 40.71%, 23.77%, and 10.73%, respectively, with cumulative percentages of variance after the rotation of 75.21%, which showed a good validity. The Cronbach’s alpha coefficient value in the formal survey was 0.98, which indicated that the proposed scale had a promising internal consistency.

Table 3 Medical Workers’ Demographic Characteristics in the Three Main Stages of the Survey

Clinical Evaluation

Reliability, Validity, and Relevance

In the current survey, 566 subjects participated in the clinical evaluation (Table 3). Based on the analysis of univariate normality, the score of each scale basically showed normal distribution (Supplementary Table 1). The Cronbach’s alpha and the Spearman-Brown coefficients of the GHE-MSS were 0.88 and 0.87, respectively, indicating a credible internal consistency of the scale. The results of the analysis of the retest reliability revealed that the reliability of the three measures and the total scale were both above 0.9, confirming a high stability. The correlation coefficient between the relative item and the score of the avoidance measure was within the range 0.76–0.83 (P<0.01). The Pearson’s correlation coefficients of the dimensions of the PTSD’s hyperarousal and intrusion symptoms were 0.78–0.84 and 0.81–0.90, correspondingly (P<0.01) (Table 4).

Table 4 Reliability, Validity, and Relevance Analyzed by Clinical Evaluation of GHE-MSS

Correlation analysis was conducted to assess the correlation between the GHE-MSS and the PCL-C, for which a correlation coefficient of 0.68 (P<0.01) was determined, indicating that the correlation validity of the scale was relatively high (Figure 2). CFA showed that the standard load factor was within the range 0.73–0.89 (Table 5). The fitting index met the requirements for the model fitting adequacy comparative fit index (CFI) = 0.947; the standardized root mean square residual (SRMR) = 0.038; the root mean square error of approximation (RMSEA) = 0.067, representing a reasonable goodness-of-fit (Table 6). Good convergent and discriminant validity indices were established and evaluated (Supplementary Tables 2 and 3).

Table 5 Factor Loadings of the Scale Items (Pattern Matrix)

Table 6 Confirmatory Factor Analysis of the Oblique Bi-Factorial Model

Figure 2 Correlations between GHE-MSS and PCL-C. Abbreviations: GHE-MSS, Global Health Events-Mental Stress Scale; PCL-C, PTSD Checklist-Civilian Version.

Assessment Results

The clinical evaluation of the scale was performed six months after conducting the formal survey. We found that the PTSD incidence was from 30.00% to 19.79% lower than that in the formal survey (Figure 3). Furthermore, the incidence of PTSD in PCL-C was 8.13%. Therefore, the incidence of PTSD in GHE-MSS was markedly higher than that in PCL-C (Figure 4).

Figure 3 Estimation of the incidence of PTSD using the GHE-MSS in the stages of scale testing and clinical evaluation. Abbreviations: GHE-MSS, Global Health Events-Mental Stress Scale; PTSD, post-traumatic stress disorder.

Figure 4 The rate of PTSD symptoms assessed by GHE-MSS and PCL-C in the stage of clinical evaluation. Abbreviations: GHE-MSS, Global Health Events-Mental Stress Scale; PTSD, post-traumatic stress disorder; PCL-C, The PTSD Checklist-Civilian Version; ICU, Intensive Care Unit.

Discussion

GHE-MSS was developed that included three measures: avoidance, PTSD’s hyperarousal, and PTSD’s intrusion symptoms. Each measure contained eight, six, and eight items, accounting for a total number of 22 items. Each of the items included in the questionnaire was measured by the five-point Likert’s scale. The GHE-MSS was composed of the Delphi consultation, primary survey, formal survey, and clinical evaluation. After clinical evaluation, the Cronbach’s alpha coefficient (0.88) showed a good internal consistency. In addition, the items of the scale were slightly modified based on the IES-R via two rounds of Delphi consultation, which enhanced the content validity. Exploratory factor analysis and CFA were implemented to verify all the indicators of the scale, which confirmed that the scale had a promising validity. GHE-MSS was developed to accurately and effectively evaluate the occurrence of mental stress responses in medical workers who experienced public health events. To date, research has been concentrated mainly on indicators, such as acute anxiety, depression, and acute mental stress. In spite of the importance of public health events, few investigations have been performed to determine and assess the incidence of chronic stress among medical workers. Hence, the present study presents a revised version of the IES-R for assessment of chronic stress incidence among medical workers. The proposed scale has the potential to evaluate medical workers’ mental stress responses after stressful public health events within six months.

Mental stress response is delayed and thus often fails to attract quickly people’s attention. Therefore, a targeted scale should be developed for early screening. Early intervention in people experiencing a major trauma was highly, meaningful in several cases.^42,43 The initial IES is a 15-item self-reported scale assessing subjective distress caused by traumatic events. The IES has been revised to IES-R via the addition of seven items related to excessive arousal symptoms of PTSD. Earlier studies showed that a high IES score within one week after a traumatic event was associated with the appearance of PTSD symptoms six months later, with a sensitivity of 92%.^44,45 Therefore, the development of the scale in the present study included a primary survey, a formal survey in the early stage of the COVID-19 outbreak, and a clinical evaluation of the stress response six months later. The PCL-C was employed as the standard for accuracy and validity assessment. The results showed that the Cronbach’s alpha coefficient and the Spearman-Brown coefficient of the scale both had values above 0.85. The CFA revealed that the modified model fitted well, suggesting that the scale has a promising reliability and stability. Each measure was characterized by good internal consistency and stability. However, the incidence of PTSD in the GHE-MSS in the clinical evaluation was significantly higher than that in the PCL-C, which may be due to the differences in the reliability and validity of the mentioned scales. As previously reported, IES-R is not appropriate for PTSD diagnosis.¹⁹ Previous evidence showed that PCL-C had a reasonable sensitivity and a higher specificity for PTSD diagnosis.³¹ Generally, a positive correlation exists in the incidence of PTSD between GHE-MSS and PCL-C, confirming that GHE-MSS can be applied to assess the mental status of medical workers experiencing public health events. In the present study, GHE-MSS was developed on the basis of IES-R. We found that the GHE-MSS was more promising for the assessment of acute and chronic stress mental responses of medical workers involved in public health events rather than the diagnosis of PTSD symptoms.

In addition to the assessment of the acute stress mental response, the early identification of subgroups of the population of medical workers exposed to a higher risk of mental stress could provide guidance on the timely application of therapeutic measures after major adverse health events. In this regard, our study has several strengths. To the best of our knowledge, this is the first self-rating scale for the assessment of the mental response of medical workers experiencing major adverse health events, specifically an infectious disease outbreak. Nevertheless, further research is still urgently needed. As known, infectious disease outbreaks have a profound psychological impact on healthcare workers. In this respect, Tam et al established that healthcare workers experienced acute stress reactions during the Severe Acute Respiratory Syndrome (SARS) outbreak in 2003.^46,47 Recently, several reports have confirmed the substantial psychological stress as well as physical symptoms experienced by health care professionals dealing with the COVID-19 outbreak.^48,49 In addition, this study was commenced during the initial wave of COVID-19 pandemic, emphasizing the acute mental response of medical workers. With the long-expected entry into the post-COVID-19 era, local and national mental health institutions have conducted evaluations and provided support for the general public, including health care professionals, such as online mental health services.⁵⁰ Therefore, increasingly more healthcare facilities and governmental efforts are likely to focus on the psychological aspect of severe infectious disease outbreaks such as that of COVID-19. It is hence necessary that medical workers exposed to significant work pressure should perform appropriate mental disorder self-tests. Furthermore, this study assessed mental stress in both tertiary and secondary hospitals in different areas, which is often a missing element of research exploring stress in healthcare workers.

This study provides initial evidence for the validity and reliability of scores derived for GHE-MSS. Earlier findings support the view that GHE-MSS can be used to measure mental distress, especially in medical workers experiencing public health events. For example, Han et al⁵¹ reported that 34.6% of the medical staff in China had depressive symptoms from February 17 to 24, 2020. Another study also showed the combined prevalence of anxiety, depression, and insomnia of Hubei medical staff, which reached as high as 38%.⁵² Additionally, Kheradmand et al reported that healthcare workers in Tehran developed mild (41.4%) to moderate (31.5%) impact mental issues during the initial phase of the pandemic.⁵³ The results of these earlier studies are similar to our findings on the incidence of PTSD among medical workers in the early stages of COVID-19. Hopefully, GHE-MSS can be assessed and validated in other countries to further deepen this evidence. We plan to do a large-sample, cross-sectional study by applying this scale in national hospitals through cooperation with the related government departments and psychological support centers. In this future study, we intend collect and assess more clinical evidence on the application of GHE-MSS. Further studies could also facilitate the development of effective strategies for the prevention of mental disorders, which could not only allow early intervention and identification of high-risk medical individuals, but also provide the supportive environment for medical workers seeking help.

Limitations

The present study contains several limitations. First, this is a cross-sectional study, which might have limited the possibility for establishing temporality between exposure and health outcome, increasing the risk for reporting bias. For instance, reporting bias can occur when the same person reports both exposure and health effects as in the case of questionnaire studies and when there is a public awareness of the problem.⁵⁴ In such a situation, a higher risk might exist for systematic over-reporting of mental disorder-associated problems in diseased subjects when compared with healthy ones and/or a risk for over-reporting of symptoms in subjects after experiencing public health events when compared with people without such experience. Such reporting bias does not occur to the same extent in longitudinal studies, and, therefore, further research is required. Second, the developed scale was more appropriate for the assessment of mental stress, particularly associated with public health events, whereas the participants were medical workers who participated in COVID-19 treatment in China. We failed to include medical workers who participated in other health events (eg, influenza A infection outbreaks). Therefore, this scale still needs to be validated in research on other infectious diseases. In addition, the Chinese version of GHE-MSS was verified by back-translation with the original item of GHE-MSS. The targeted population of our study consisted of Chinese medical workers, which may decrease the generalizability of our results because of the differences between languages and the inhomogeneous distribution of mental response across the globe. It is unknown whether the results of this study are applicable to other medical populations in China or other countries. Hopefully, this self-rating scale can be assessed and validated in other countries in the future. The results of the convenience sampling cannot be generalized to the target population because of the potential bias of the sampling technique. Young medical workers and people experiencing heavy mental stress that are familiar with social media were more likely to participate in our research. However, senior doctors and nurses were reluctant to participate in this study. Therefore, the degree of mental stress reaction in our study might have been overestimated. Meanwhile, we could not assess how many people had been invited to participate in the research, and the response rate could not be, therefore, calculated. Our study included medical workers from only five tertiary and secondary hospitals in Guangdong Province. Hence, the source and size of the samples were limited. Thus, further research with a larger sample size is warranted to eliminate the aforementioned shortcomings and to confirm our findings.

Conclusion

In conclusion, here, we propose a revised version of the IES-R, called GHE-MSS. Our findings revealed that the proposed scale has a good internal consistency and validity. GHE-MSS may facilitate the screening of mental stress responses in medical workers in the early stage of their occurrence after experiencing public health events. Therefore, the identification of subgroups of medical workers with higher susceptibility to PTSD may be of significance. The proposed scale is also promising for developing targeted therapies for individuals who are at high-risk of mental stress exposure due to their participation in the treatment of pandemics-affected subjects.