Item Response Theory Analysis of the Fear of COVID-19 Scale (FCV-19S): A Systematic Review

Introduction

The coronavirus disease 2019 (COVID-19) epidemic has affected (and continues to affect) all nations, continents, races, and socioeconomic groups. Early responses to the pandemic included quarantining entire communities, closing schools and colleges, implementing social isolation, and staying at home, which suddenly changed people’s daily lives.^1,2 The sudden onset of COVID-19 (ie, its epidemic nature with very high rates of infection and relatively high mortality) caused individuals to naturally become anxious and afraid of becoming infected with COVID-19.^3,4

One of the characteristics of infectious diseases is fear, which is directly related to the rate of transmission and its medium (fast and invisible) as well as its complications and mortality. This leads to other psychosocial challenges such as stigma, discrimination, and loss.⁵ The threats of new COVID-19 variants are also known to have effects on psychological responses.^6–8 Such psychological reactions, especially anxiety and fear, are important factors for individuals’ consequent behaviors.^9,10 Low levels of anxiety may lead to rational adherence to preventative behaviors (eg, hand washing) and high levels of anxiety may lead to socially destructive behaviors (eg, fear of shopping).¹¹ High levels of fear can cause individuals not thinking clearly and rationally when reacting to COVID-19, resulting in increased damage from the disease.¹²

The psychological effects of the COVID-19 epidemic have demonstrated the need for healthcare providers having validated tools to assess individuals’ psychological responses specifically to COVID-19 during the epidemic.¹¹ However, healthcare providers should not only care for COVID-19-related symptoms (including psychological reactions). Instead, they still need to pay attention to the “old diseases”.^13,14 Given the importance of the issue of COVID-19 fear, the Fear of COVID-19 Scale (FCV-19S) has been developed to help healthcare providers to design and implement appropriate interventions to reduce fear.¹² The FCV-19S was developed by a panel of experts after an extensive literature review. Moreover, the FCV-19S items assessed using classical test theory and the Rasch model were shown to be satisfactory among the general public and other populations.^12,15–17

The total score of the FCV-19S can be used to help diagnose, evaluate, and monitor the severity of COVID-19 fear-related anxiety. Using total score is a fast and simple application that quickly shows the severity of individuals’ COVID-19 fears.¹² However, the simple summation of raw scores ignores the difference between items and the information that the response pattern can provide. Therefore, it may lead to incorrect estimation.¹⁸ Items vary in one measurement in different ways. First, some may be more difficult than others. For example, for most individuals, repeating a noun will be more difficult than remembering a phrase or a list of words. Second, some cases may be more sensitive to the early stages of the disease and others to the later stages of the disease. Third, items vary in sensitivity to clinical changes. Finally, some items may offer no significant variation in measurement. These can be eliminated to reduce the temporal burden on assessments.¹⁹

Additionally, the same total score can be obtained through different response patterns. For example, two individuals who received a score of 20 on the same scale may have answered in completely different patterns. Similarly, an individual who achieves the same total score before and after treatment will be considered as not changing status. However, the intervention may have changed the pattern of answering questions.¹⁹ Therefore, there is a need to examine the pattern of response to items, of which can be overcome using the statistical method “Item Response Theory” (IRT).²⁰

IRT is a statistical method based on the probability of an individual achieving a specific score in a test that the result of that individual’s status lies in the structure.²¹ As this situation changes, so does the probability of achieving a specific score from the individual, and the accuracy of the measurement varies with the level of the situation.²² IRT can show two useful features of each test, namely difficulty and discrimination (or differentiation).²³ Discrimination is an indicator that shows how an item can differentiate between individuals with different intensities. The more distinct the item, the steeper the slope, the better they can differentiate between individuals within the range of latent features.²⁴ The next feature is the difficulty of the items which vary according to the diagnosis or by country/region or different translations. Awareness of the order of difficulty helps physicians to design and implement their assessments according to the severity of the situation.²⁴

IRT can check the sensitivity of items on a scale via the item characteristic curve (ICC). Items with a higher ICC slope are where the item is most differentiated, distinguishing well between different degrees of disorder and making them more sensitive to change. Determining the differentiation of items can indicate which items are most likely to best reflect changes in status.²⁵ Examining the case curves in relation to each other provides useful information about the measurement scope of a tool. The IRT can also identify key items that provide valuable information or whether the items are on an additional scale (ie, items with similar ICCs).²²

Using IRT techniques to assess COVID-19 fear can have far-reaching implications for physicians and researchers, leading to advances in screening evaluation and diagnosis, charting disease, and measuring changes in disease progression and response to treatment. IRT has been used to analyze clinical practice in several different contexts including schizophrenia,²⁶ depression,²⁷ attachment,²⁸ and quality of life.²⁹ Given that many studies have translated and examined the psychometric properties of the FCV-19S, IRT methods have also been applied. However, no synthesized evidence regarding the IRT-assessed properties of the FCV-19S has been published. Therefore, the aim of the present systematic review was to synthesize the psychometric evidence of the FCV-19S on its IRT findings, including these studies’ methodological quality, FCV-19S item difficulties, FCV-19S item discriminations, and differential item functioning (DIF) across age and gender.

Methods

Design and Protocol Registration

The present systematic review was designed to assess the measurement properties evaluated using IRT for the FCV-19S. The project was registered (ID number CRD42020188890) in the International Prospective Register of Systematic Reviews (PROSPERO) website.³⁰ The study’s findings are reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.³¹ The present systematic literature search was done using four academic databases (Scopus, PubMed Central, ProQuest, and ISI Web of Knowledge). Relevant studies were abstracted and their methodological quality was assessed using the COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN) checklist.^31,32 Findings were synthesized using a narrative approach.

Search Strategy

Scopus, PubMed Central, ProQuest, and ISI Web of Knowledge were searched systematically from December 2019 to end of August 2021. “Fear of COVID-19 scale” and its abbreviation of “FCV-19S” together with IRT related terms (eg, item response theory, modern test theory, Rasch) were used to customized search syntax for the aforementioned academic databases.

Study Screening and Selection

The titles and abstracts of all retrieved papers were screened based on eligibility criteria. All peer-reviewed observational studies were considered eligible if psychometric properties of FCV-19 Scale were assessed using an IRT method. No limitation was exerted regarding participants’ characteristics. After screening for potentially eligible papers, their full texts were downloaded and reviewed for final selection.

Quality Assessment

The methodological quality of the included papers was assessed using the COSMIN checklist. Generally, the COSMIN methodology is used to assess measurement properties of a single outcome measurement instrument or make a systematic evaluation of the most suitable tool in assessing a specific construct.^32,33 This checklist can be used to evaluate the methodological quality based on 10 measurement properties including outcome measure tool development, content validity, structural validity, internal consistency, cross-cultural validity/measurement invariance, reliability, measurement error, criterion validity, hypotheses testing for construct validity, and responsiveness.³³ Each of the aforementioned items have a clear guideline with several items for assessment. An overall score for the methodological quality of each measurement property for each study is determined by taking the lowest rating of any of the items according to the guideline.³⁴ The present systematic review specifically focused on the properties of internal consistency, structural validity, cross-cultural validity, criterion validity, and IRT properties. Therefore, these properties were selected and assessed through the included studies. Each item was rated using a Yes/No/Not applicable (NA) rating scale. No studies in the present review were excluded on the basis of poor methodological quality.

Data Extraction

Data extracted included the first author’s name, publication date, title of the study, research area as country, occupation of participants, sample size, study design, IRT approach and their measures, COSMIN score (ie, methodological quality), and main findings of the study. Disagreements were resolved via discussion between research team members. To synthesize the data, a qualitative synthesis of findings was performed due to small number of included studies and the heterogeneity of the assessed outcomes.

Data Synthesis

To synthesize the data, a qualitative synthesis of findings was performed to report findings of included studies based on analytical method.

Results

Study Screening and Selection Process

The initial search in the four databases resulted in 552 papers. After removing duplicates, 479 papers were screened based on titles and abstracts. After removing irrelevant studies using the information from titles and abstracts, 118 papers were further evaluated using their full texts. Finally, 16 papers were deemed to be eligible and their full-texts were reviewed. Figure 1 shows the search process based on the PRISMA flowchart.

Figure 1 PRISMA flowchart. Notes: Adapted from Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of Internal Medicine. 2009;151(4):264–269.31 Creative Commons.

Study Description

The 16 studies comprised 32,624 participants from 21 different countries (Argentina, Bangladesh, Brazil, China, Cuba, Ethiopia, France, India, Iran, Italy, Japan, Jordan, Korea, Malaysia, Mozambique, New Zealand, Pakistan, Romania, Taiwan, Turkey, United Kingdom) and were included for final synthesis. The mean age of participants varied from 18 years to 52 years. Approximately half of overall participants were females (52.83%). Almost all studies recruited general population as the target participants, except for one study surveying adult patients with type 2 diabetes. Two main IRT methods were used: Graded Response Model (n=8 studies) and Rasch analysis (n=8 studies). The summary results of measurement properties are reported based on the IRT method used in the included studies. Table 1 provides the summary characteristics of all included studies. Table 1 also reports the link functions used in each study given that there are two types of functions (ie, logit or probit) in the IRT.³⁵

Table 1 Summary Characteristics of Included Studies

Methodological Quality of Included Studies

Each item regarding the methodological quality of assessed psychometric properties (ie, internal consistency, structural validity, cross-cultural validity, criterion validity, and IRT properties) was rated on a Yes/No/Not applicable (NA) rating scale. The results of methodological assessment are presented in Figures 2–Figure 6. There were no serious flaws regarding the design or methods of the included studies in all aspects. However, most studies did not describe the percentage of missing items and how missing items were handled. IRT tests were not performed with the aspect of factor structure (ie, whether the FCV-19S is unidimensional or multidimensional) in most of studies. Moreover, most studies that evaluated the psychometric properties of translated FCV-19S were not reviewed by the original developers, the sample used in the pre-test among the translated FCV-19S was not adequately described, and no adequate descriptions were provided for the translated FCV-19S about how differences between the original and translated versions were resolved.

Figure 2 Results of methodological quality assessment regarding Box A. Internal consistency.

Figure 3 Results of methodological quality assessment regarding Box E – structural validity.

Figure 4 Results of methodological quality assessment regarding Box G. Cross-cultural validity.

Figure 5 Results of methodological quality assessment regarding Box H. Criterion validity.

Figure 6 Results of methodological quality assessment regarding Box IRT.

Summary Results of Rasch Analysis

Eight studies used Rasch analysis, a type of IRT analyses that require the discrimination parameters to be the same across all elements, to assess the measurement properties of FCV-19S. Table 2 provides the results, including information-weighted fit (infit) mean square (MnSq), outlier-sensitive (outfit) MnSq, and difficulty for each item. Infit MnSq varied from 0.73 to 1.38 (Both for FCV 3); Outfit MnSq varies from 0.68 to 1.05 (for FCV 3 and FCV 6 respectively); and difficulty of tem varied from −1.5 to 1.94 (for FCV 2 and FCV 6 respectively).

Table 2 Summary Results of Rasch Analysis Properties

Summary Results of Graded Response Method Analysis

Eight studies used Graded Response Method IRT, a type of IRT analyses that allow element discrimination to be varied, to assess the measurement properties of FCV-19S. Table S1 provides the results for each item. In graded response analysis, a coefficient varied from 0.03 to 5.11 (for FCV5 and FCV6 respectively); b1 coefficient varied from− 5.21 to 1.19 (for FCV1 and FCV3 respectively); b2 coefficient varied from 0 to 1.86 (for FCV2 and FCV 3 respectively); b3 coefficient varied from 0.1 to 2.51 (for FCV5 and FCV3 respectively); b4 coefficient varied from 0.11 to 5.1 (for FCV5 and FCV6 respectively).

Summary Results of Differential Item Functioning (DIF) Contrast Across Age and Gender Group

Seven studies reported DIF contrast of FCV-19S items across age and gender groups. Table 3 provides the results for each item. DIF contrasts across age were ranged between 0.05 and 0.67 (FCV1); between 0 and 0.49 (FCV2); between 0 and 0.35 (FCV3); between 0 and 1.05 (FCV4); between 0 and 0.49 (FCV5); between 0.08 and 0.73 (FCV6); and between 0 and 0.76 (FCV7). DIF contrasts across gender were ranged between 0 and 0.2 (FCV1); between 0 and 0.33 (FCV2); between 0 and 0.29 (FCV3); between 0.06 and 0.33 (FCV4); between 0.04 and 0.48 (FCV5); between 0 and 0.24 (FCV6); and between 0.02 and 0.22 (FCV7). Therefore, FCV2, FCV3, FCV5 can be concluded as no substantial DIF across age; all FCV items as no substantial DIF across gender.

Table 3 Summary of Items DIF Based on Age and Gender

Discussion

The present systematic review evaluated the methodological quality of 16 studies regarding their IRT-based findings in relation to the Fear of COVID-19 Scale. Moreover, the IRT findings of the 16 studies are summarized (Tables 2,3) regarding the item properties, item discrimination, item difficulty, and DIF. Almost all the IRT properties assessed in the 16 reviewed papers were satisfactory. In addition, the IRT evidence was reported across different language versions, including Arabic (Jordan),³⁶ Portuguese,^37–39 English,^39–41 Amharic,⁴² Spanish,^39,43 Turkish,⁴⁴ Korean,⁴⁵ Chinese,^39,46 Japanese,³⁹ Italian,³⁹ Urdu,³⁹ French,³⁹ Malay,⁴⁷ Bangla,^48,49 Persian,¹² and Romanian.⁵⁰ Therefore, the FCV-19S is verified by the findings of the present systematic review regarding its psychometric properties across different language versions.

In addition to the good properties shown in the CTT findings (eg, good internal consistency, test-retest reliability, and concurrent validity),^12,36–50 the present systematic review demonstrated the good properties were also shown on the IRT testing. From the findings of reviewed papers,^12,36–50 the seven items in the FCV-19S had satisfactory infit and outfit MnSq (ie, between 0.5 and 1.5),^51,52 indicating that every item in the FCV-19S is essential and fits in the concept of fear of COVID-19. Moreover, the item difficulties were acceptable (Table S1), which indicates that the seven items in the FCV-19S together explain a wide rage levels of fear of COVID-19.²³ Additionally, the item discriminations were good for the seven items. Therefore, the FCV-19S can efficiently and sharply differentiate low fear and high fear.²³ Another important feature in the IRT findings for the FCV-19S is that the instrument does not display substantial DIF across age and gender groups (Table 3). This indicates that the participants in different ages and different genders interpret the FCV-19S in a similar way.⁵³

Although the present systematic review showed that the seven items of the FCV-19S are essential for assessing the concept of fear of COVID-19, there are some studies suggesting shortening the FCV-19S.^37,54,55 For example, Mercado-Lara et al⁵⁴ and Campo-Arias et al⁵⁵ recommended shortening the FCV-19S to a five-item scale, and de Medeiros et al³⁷ suggested a four-item version of the FCV-19S. However, given the synthesized findings from the present systematic review, it is apparent that shortening the FCV-19S may jeopardize the psychometric properties of the FCV-19S. Indeed, the seven items of the FCV-19S were all fully evaluated regarding their content and appropriateness for the fear concept when the FCV-19S developers generated this instrument.^12,56 Therefore, together with the synthesized findings reported here, it is strongly recommended that researchers and practitioners use the seven-item FCV-19S for assessing fear of COVID-19 rather than the briefer versions.

Strengths and Limitations

The present systematic review has a number of strengths. First, with the use of four important databases and following the PRISMA guidelines,³¹ the present systematic review conducted a comprehensive search strategy. Consequently, the papers on FCV-19S properties were found and evaluated. Second, the systematic review used standardized international guidelines (ie, COSMIN checklist)^32,33 to evaluate the methodological quality for each included study and ensured that the included studies achieved an accepted set of quality criteria. This indicates that the evidence reviewed in the present systematic review is trustworthy and reliable. In particular, all the studies reviewed in the present systematic review had very good sample sizes (ranging between 228 and 15,653), which are far beyond the suggestion recommended by the COSMIN risk of bias checklist (ie, N > 100).^32,33 With the large sample sizes, the present systematic review is confident in the psychometric evidence synthesized from the studies reviewed because large sample sizes can somewhat decrease the biases in the summarized psychometric properties of FCV-19S.

Despite the aforementioned strengths, the present systematic review has a few limitations. First, the present systematic review did not use meta-analysis to quantify the findings of psychometric properties for the FCV-19S. Given that the reviewed papers used different methods in IRT (eight in Rasch and eight in graded response method), such psychometric information is hard to quantify using the meta-analysis method. Moreover, to the best of the present authors’ knowledge, no proper meta-analysis methods can be used to integrate the IRT findings for the present systematic review. Therefore, the present systematic review can only provide qualitatively synthesized results for the psychometric properties of the FCV-19S. Second, responsiveness, an important psychometric property for healthcare providers to know if an instrument can be used to evaluate treatment effects,⁵⁷ was not examined in all the papers reviewed in the present systematic review. Without the information of “responsiveness”, researchers cannot be confident that the FCV-19S can capture the treatment effect on fear of COVID-19 reduction. Therefore, further studies should be based on the good properties of FCV-19S found in the present systematic review to understand how much improvement shown in the FCV-19S indicates treatment effects on fear reduction. Third, the present systematic review primarily relied on academic databases to search the papers. Therefore, nonacademic databases, such as Google Scholar database, was not used in the search. Given that such a nonacademic database may provide more comprehensive findings (though the nonacademic database may be hard to screen because it contains a large portion of nonacademic articles in the search output) for scholars to supplement the literature search,⁵⁸ the present systematic review may have missed some IRT psychometric studies concerning the FCV-19S. However, the present authors believe that there will not be many because the four academic databases are well known for their coverage in health-related scientific papers. Fourth, as compared to classical test theory, IRT has the disadvantages that (i) it is more complex and most healthcare providers were unfamiliar with, including the software analyzing IRT models,^59,60 (ii) it requires large sample sizes to estimate accurate parameters⁶¹ and some studies analyzed in the present systematic review seemed not to have sufficient sample size for IRT models, (iii) it needs strict assumptions and most studies analyzed in the present systematic review did not check these assumptions. Fifth, the FCV-19S, as with other psychological assessment tests,^62,63 has been modulated by cultural aspects; for example, Spanish adaptation in the FCV-19S is the Latin American adaptation from Argentina and Cuba; there are important cultural differences with the Spanish of Spain.

Conclusion

Based on the present review’s findings, the seven FCV-19S items seem to be unidimensional in the assessment of fear, indicating that all items are necessary in the FCV-19S. More specifically, the psychometric properties of the FCV-19S, especially in relation to IRT properties, were systematically reviewed, summarized, and synthesized in the present paper. Apart from the IRT properties, some commonly used psychometric properties such as internal consistency and criterion validity were also reviewed. Almost all evidence reviewed in the present paper indicates that the FCV-19S is a promising instrument for assessing the fear of COVID-19. The seven items in the FCV-19S appear to be unidimensional in assessing COVID-19-related fear. Age and gender were not significant factors that affect individuals in interpreting any item in the FCV-19S, and diverse difficulties (eg, the item difficulties ranging between −5.21 and 5.11 logit) have been reported across the seven items in the FCV-19S that ensure different levels of fear can be captured by the FCV-19S items. However, current evidence does not indicate whether the FCV-19S has good responsiveness, ie, whether a treatment program regarding fear of COVID-19 reduction can be effectively evaluated using the FCV-19S. Future studies are therefore needed to address this knowledge gap.