Development and Validation of Questionnaires to Assess Workplace Violence Risk Factors (QAWRF): A Tripartite Perspective of Worksite-Specific Determinants in Healthcare Settings

Introduction

The National Institute for Occupational Safety and Health (NIOSH) defines workplace violence (WPV) as the act or threat of violence, ranging from verbal abuse to physical assaults directed toward persons at work or on duty.¹ Among occupations, healthcare workers (HCW) face a significant risk of WPV, having the closest contact with patients and their relatives and being on the healthcare system frontlines.² Indeed, the rate of WPV among this group of workers is reported to be 20% overall higher than all other workers.³ According to the Injury Prevention Research Center, University of Iowa, there are four types of WPV: 1) Type 1 is WPV with criminal intent; 2) Type 2 is client-to-worker WPV; 3) Type 3 is worker-to-worker WPV; and 4) Type 4 is WPV related to personal relationships.⁴ Type 2 WPV is the commonest type of WPV experienced by HCW and is perpetrated mainly by patients or their dissatisfied family members.^5–7

A meta-analysis of 253 studies published globally revealed that HCW’ reported exposure to any WPV was 61.9%.⁸ For Type 2 WPV experienced by HCW, a recent meta-analysis found that the pooled one-year prevalence was 19.3%,⁹ whereas a systematic review reported prevalence ranging from 9.5% to 74.6%.¹⁰ Indeed, the International Labour Organisation (ILO) reported that about every second HCW worldwide has been affected by WPV at least once during their work lives,¹¹ whereas the United States Bureau of Labor Statistics indicated that there were 8590 cases of nonfatal intentional injuries by another person among HCW in 2020.¹² Despite the above, these figures might be less than actual figures, as multiple studies have shown that WPV is usually underreported.^13–15 Among the common reasons given for not reporting WPV were the perceived uselessness of reporting, perceived insignificance of the violence perpetrated, fear of consequences, and the impression that WPV is “part of the job”.^15–17 Some HCW also complained that the formal reporting system is problematic due to it being difficult and time-consuming to use.¹⁷

The high prevalence of WPV among HCW are concerning as WPV can affect workers’ physical, mental, and social health. In addition, WPV incidences can also affect the quality of care and relationship with patients, work functioning, and financial and social aspects of workers.^6,18–22 Among the most commonly reported consequences of HCW exposure to WPV are psychological effects such as anxiety, fear, posttraumatic stress disorder, and depression, as well as effects on work functioning such as prolonged work absences, poor job performances, job dissatisfaction, and burnout.¹⁸

Given the significance of WPV among HCW, multiple interventions for WPV have been developed globally. However, there was only moderate evidence of the interventions’ effectiveness.²³ This may be because though the risk factors for WPV are well known, they are generalized to all workplace settings. Therefore, a worksite-specific assessment of risk factors needs to be conducted to provide the essential inputs for intervention.²³ Moreover, current suggestions to tackle WPV include engaging stakeholders at the regional or local level and considering their collective perspectives to facilitate better policymaking, as the risk factors of WPV can be unique from one setting to another.²⁴ This is even more pertinent given that the prevalence of WPV among HCW varies depending on the level of awareness and cultural norms in particular countries or organisations.²⁵ However, existing survey questionnaires mainly explore workers’ perspectives and rarely consider clients’ or administrators’ perspectives on the WPV incident.

In addition, workplaces currently collect information to identify possible WPV risk factors through records analysis, job hazard analysis, and specified WPV program checklist.³ Although these conventional WPV risk factors identification methods are useful, they require training and may be time-consuming.²⁶ On the other hand, most employee or client survey questionnaires use open-ended questions to explore employees’ or clients’ perspectives or feedback, but this may be limited by inadequate responses due to respondents’ difficulty in articulating the answers, partial responses according to varying dissatisfaction levels, and disinterest in the topic.²⁷ They would also require more time for data collection and subsequent analysis.¹⁴ At present, there is no validated questionnaire for policymakers to conveniently collect quantitative data and subsequently determine the level of WPV risk factors existing in their healthcare settings based on the perceptions of all their important stakeholders, to enable them to tailor WPV preventive measures based on the risk factors rated as prevalent at the setting. Due to these reasons, our study aimed to develop, validate, and determine the reliability for a set of structured survey questionnaires to assess WPV risk factors in healthcare settings based on the tripartite perspectives of key stakeholders in healthcare, which are the administrators, workers, and clients.

Materials and Methods

Prior to the conduct of this study, ethical clearance was obtained from the Medical Research & Ethics Committee [NMMR-21-537-58890 (IIR)] and Universiti Teknologi MARA Research Ethics Committee [REC/07/2021 (MR/629)].

Development of the Questionnaires to Assess Workplace Violence Risk Factors (QAWRF)

Three questionnaires were developed for each group of stakeholders, namely the administrators, workers, and clients, in view that each group’s perception of WPV risk factors is different. Administrators are those involved in administrative work at each healthcare department, such as the head of department, the head medical officer, the head of paramedics, and the head nurse. On the other hand, workers are those involved clinically with clients, such as doctors, paramedics, and nurses, whereas clients are patients and those accompanying patient to a healthcare setting.

All questionnaires were developed based on the Chappell and Di Martino’s Interactive Model of Workplace Violence,¹⁹ a WPV model accepted by the ILO. In this model, the risk factors for WPV are categorized into the victim’s risk factors, the perpetrator’s risk factors, the workplace risk factors, and the wider contextual and societal risk factors that indirectly affect WPV incidences. These factors were used to form the domains of the questionnaires.

In the Chappell and Di Martino’s WPV model, the risk factors listed under each domain in the model are described in general, and they need to be adjusted according to the setting the users wish to apply the model.²⁸ Therefore, as this study was contextualized to healthcare settings, the risk factors that were used as items in QAWRF were identified by conducting a systematic literature review and the items were placed under the suitable domains according to the description of items in the model. This is appropriate because the risk factors of WPV in healthcare are well known,²³ and most literature grouped the risk factors identically to the model used in this study. The criteria for items inclusion were 1) risk factors that are related to WPV in healthcare settings, 2) risk factors that are related to WPV perpetrated by patients or visitors of healthcare settings, the most common type of WPV in healthcare settings, 3) risk factors that are suitable to be assessed using a structured questionnaire, and 4) risk factors that are related to the local setting. Databases such as Web of Science, Scopus, Cochrane, PubMed, and Google Scholar were used for the systematic literature review. The search was conducted using the keywords “workplace violence” or “workplace aggression”, “risk factors” or “predictors”, and “healthcare settings”. Risk factors extracted from primary studies, systematic reviews, and meta-analyses published since 2013 were listed and grouped under the suitable domain as suggested by the Chappell and Di Martino’s WPV model.¹⁹

The suitability of domains and items to be included in each questionnaire were then discussed by a three-member committee comprising an investigator studying WPV, an experienced occupational and safety committee member, and an occupational health specialist. The decision to include or remove any domain or item in each questionnaire was based on the committee’s judgement of the perceived ability of each target population to recognize the presence of each WPV risk factor generated. Domains or items judged as impossible for the target population of the questionnaire to detect or recognize were excluded from the respective questionnaire. The committee also assessed the wording used in the questionnaires, as the questionnaires were developed in the Malay language to ensure a better understanding of the questionnaires among the target population. A 5-point Likert scale was used (1 = the level of risk factor existence is very low at the facility, 5 = the level of risk factor existence is very high at the facility) for participants to rate the presence of each risk factor in their respective healthcare settings. Modifications to each questionnaire (elimination of constructs or items, change of wordings or fonts, order of items, and improvement on instructions given to participants) were made iteratively until committee members indicated that they were satisfied with the overall content and appearance of the questionnaires.²⁹

Content Validation of QAWRF

The three component questionnaires, ie QAWRF-administrator, QAWRF-worker, and QAWRF-client were then sent to a panel of experts to empirically determine the content validity of QAWRF.³⁰ The number of raters required for content validation ranged from two to 10 experts.^30,31 Therefore, six experts with experience in managing WPV in healthcare settings and handling questionnaires for content validation were recruited for this study. They included 1) a public health specialist, 2) an occupational health specialist, 3) a senior assistant director of the occupational health unit in the Ministry of Health (MOH), 4) an assistant director of the human resource unit in the Department of Safety and Health (DOSH), 5) the head of the Occupational and Safety department in one of Malaysia’s main hospitals, and 6) a questionnaire expert.

The content validation was conducted through email because the national Standard Operation Procedure for Coronavirus Disease 2019 (COVID-19) during the validation process did not allow face-to-face meeting sessions. The experts were asked to rate the relevance of each item and provide further comments or suggest any changes. A 4-point Likert scale was used for scoring, where a score of one indicated that the item was not relevant, whereas a score of four suggested that the item was highly relevant.³¹ The expert panel was given two weeks to rate the questionnaires, and a reminder email was sent after the first week.

The collected data were compiled in Microsoft Excel, where the relevance rating given by the experts were dichotomized as 1 (agreed) if the experts rated items as 3 or 4 and 0 (disagreed) if the expert rated items as 1 or 2.^30,31 Indices used to determine the content validity of each questionnaire included the Item Level Content Validity Index (I-CVI) and Scale Level Content Validity Index (S-CVI), which were calculated independently for each questionnaire. I-CVI for each item was calculated as the number of experts who agreed with the given items divided by the total number of experts [I-CVI = (agreed item) / (number of expert)].^30–32 Items with I-CVI of more than 0.790 were considered relevant, items with I-CVI between 0.700 and 0.790 were revised according to the panel’s suggestions, and any item with I-CVI less than 0.700 was eliminated.^30,32 Next, S-CVIs were then calculated based on the average method where the sum of I-CVIs for all items was divided by the total number of items on the questionnaire (S-CVI/Ave) [S-CVI/Ave = (sum of I-CVI scores) / (number of item)].^30,32 An S-CVI/Ave score of more than 0.900 was considered to indicate excellent content validity.^30,33

Face Validation of QAWRF

A quantitative method was used for face validation of the QAWRF, where respondents rated the clarity and comprehension of individual items in the QAWRF using the face validity form.³⁴ Respondents rated each item based on a 4-point Likert scale, with 1 being not clear and understandable and 4 being very clear and understandable. An open-ended commentsection was also provided at the end of the form for raters to write any commentto further improve the QAWRF.

The minimum number of raters for the quantitative method of face validity was 10.^34,35 Therefore, 12 raters for each component questionnaire and 36 total raters for QAWRF who fit the inclusion and exclusion criteria of the study (Table 1) were recruited to complete the survey. In a face-to-face session, raters were given the drafted questionnaire, the face validity form, and a pen. They were first instructed to complete the drafted questionnaire and then fill out the face validity form based on their experience when answering the drafted questionnaire.

Table 1 Inclusion and Exclusion Criteria for the Face Validation and Pilot Survey

The collected data were entered into Microsoft Excel, where ratings of 3 or 4 were recoded as 1 (clear), and ratings of 1 or 2 were recoded as 0 (not clear).³⁴ The Item Level Face Validity Index (I-FVI), which is the proportion of rater scoring the items as clear was calculated [I-FVI = (agreed item) / (number of rater)].³⁴ Then, the average of the total I-FVIs score on each questionnaire (S-FVI/Ave) [S-FVI/Ave = (sum of I-FVI scores) / (number of item)], and the proportion of items on the questionnaire that reached “Universal Agreement (UA)” where the item achieved 100% experts in agreement (S-FVI/UA) [S-FVI/UA = (sum of UA scores) / (number of item)] were calculated.³⁴ The acceptable value for I-FVI, S-FVI/Ave and S-FVI/UA was 0.83.^31,35 Modifications were made to QAWRF based on the scores and comments given by raters.

Pilot Survey and Reliability Testing of QAWRF

After modifications were made to the QAWRF based on the face validation findings, a pilot survey was conducted to ascertain the usability of QAWRF.^36–38 In addition, Cronbach’s alpha was calculated to examine the internal consistency reliability of QAWRF questionnaires.^39,40 Thirty respondents based on the pre-determined criteria (Table 1) were recruited to answer each questionnaire.^39,41 The respondents were recruited from University Teknologi MARA (UiTM) Specialist Private Centre and Hospital Al-Sultan Abdullah, Selangor, Malaysia. Respondents were given a Participant Information Sheet, the self-administered questionnaire, and were asked to complete and return the questionnaire. The investigator recorded the time taken for respondents to answer the QAWRF and identified any logistic needs for a full-scale study. Meanwhile, data obtained from the pilot study were entered into SPSS Version 28 where Cronbach’s alpha was calculated for each questionnaire. A Cronbach’s alpha value of more than 0.7 was considered as acceptable.^39,42,43

Results

Four domains were created by adapting Chappell and Di Martino’s Interactive Model of Workplace Violence. The domains include the client’s, worker’s, workplace, and social risk factors; the latter combines both broader contextual and societal risk factors. Twenty-eight studies were used for item generations (Table 2, Supplementary Table 1). A total of 65 items that fulfilled the item’s inclusion criteria were generated.

Table 2 Summary of Literature Used for Items Generation

The QAWRF-administrator was developed by including all four domains and a total of 48 items. After content validation, three items with I-CVI score of less than 0.700 were eliminated and nine items were revised based on the comments given by the experts. No item was eliminated or revised during the face validation and pilot survey. The indices for the QAWRF-administrator were 0.920 for S-CVI/Ave (Table 3), 0.983 for S-FVI/Ave, and 0.844 for S-FVI/UA (Table 4). All domains in the QAWRF-administrator had acceptable Cronbach’s alpha values, which ranged from 0.809 to 0.930 (Table 5). The time taken to respond to the QAWRF-administrator ranged from 9 to 16 minutes.

Table 3 Content Validity Indices for QAWRF

Table 4 Face Validity Indices for QAWRF

Table 5 Internal Consistency Reliability of QAWRF

The QAWRF-worker started with four domains and a total of 47 items. After content validation, four items with I-CVI score of less than 0.700 were eliminated and nine items were revised based on the comments given by the experts. No item was eliminated or revised during the subsequent face validation and pilot survey. The indices for the QAWRF-worker were 0.922 for S-CVI/Ave (Table 3), 0.981 for S-FVI/Ave, and 0.837 for S-FVI/UA (Table 4). The Cronbach’s alpha values for the QAWRF-worker were acceptable and ranged from 0.892 to 0.949 (Table 5). The time taken to respond to the QAWRF-worker ranged from 8 to 15 minutes.

The QAWRF-client was developed using three domains with 22 items. After content validation, the societal domain, which contained six items, was eliminated following an expert comment. One item with I-CVI score of less than 0.700 from the worker domain was also eliminated. Meanwhile, three items were revised according to the experts’ comments. During the face validation, no item was eliminated, but one item was revised according to raters’ comments. Indices for the QAWRF-client were 0.917 for S-CVI/Ave (Table 3), 0.978 for S-FVI/Ave, and 0.867 for S-FVI/UA (Table 4). During the pilot survey, one item from the worker domain was eliminated because it contributed to low internal consistency (Cronbach’s alpha 0.653). The final Cronbach’s alpha values for the worker domain after the item removal improved to 0.866, whereas Cronbach’s alpha for the workplace domain was 0.770, which were acceptable (Table 5). The time taken to respond to the QAWRF-client ranged from 8 to 15 minutes.

In addition, several face validation raters commented that they had some confusion during the rating of items because of the end descriptors of the Likert scale used in QAWRF. The initial Likert scale end-descriptors used were 1 “the level of risk factor existence is very low at the facility” and 5 “the level of risk factor existence is very high at the facility”. Therefore, the study committee decided to change the scale’s end-descriptors to 1 “highly not apparent at the facility” and 5 “highly apparent at the facility” for users to rate the level of risk factor present at respective healthcare facilities. After the change was made, no further commentwas received for the Likert scale’s end descriptors.

Discussion

This study developed, validated, and determined the reliability of QAWRF, a set of structured survey questionnaires that can be used to assess WPV risk factors in healthcare settings based on a tripartite perspective of key stakeholders. Three different questionnaires were developed separately for each target group, namely administrators, workers, and clients. The separation of questionnaires was deemed essential because the perception of WPV risk factors differs among the three target population. For example, the administrators of departments, who are focused more on the department’s policies, staffing, and environments rather than delivering clinical services,⁶⁵ might view WPV risk factors differently compared to workers. On the other hand, workers who work directly on the ground are likely more familiar with process activities, facility operations, and potential threats.³ This is supported by previous studies which demonstrate that divergent perceptions often exist between healthcare administrators and workers on different issues.^66–68 Moreover, clinical workers were suggested to often report to other clinical workers in their department rather than to the administrators, which can create a gap in the perception of WPV risk factors between workers and administrators.⁶⁸ Meanwhile, clients who spend the least time at a particular healthcare department compared to the other two target population may only recognize certain risk factors, and their knowledge regarding the presence of risk factors may also be influenced by incidences that might or might not happen during their visit to the healthcare setting. Therefore, three separate questionnaires for each key stakeholders are needed to ensure accuracy in determining WPV risk factors in healthcare settings and avoiding information bias.^69,70 The separation also enables QAWRF users to collectively assess the risk factors of WPV at a particular healthcare setting through three different perspectives simultaneously, thus following the suggestion to involve all stakeholders at the local level for better policymaking.²⁴ Ultimately, users can tailor interventions for their WPV prevention programs based on the findings of QAWRF.

The items for each component questionnaire were generated from the same pool of WPV risk factors. However, the instruction wordings, the items, and the number of items included differed. Moreover, since QAWRF is intended to be self-administered, each component questionnaire was designed specifically for each target population to ensure that the respondents can answer the questionnaire without the researcher’s or surveyor’s assistance.⁷¹ The component questionnaires were also developed separately to avoid contingency or filter questions, which can be confusing, counterintuitive to respondents, and lead to data error because only a subset of respondents are required to answer specific questions.^72,73

A risk assessment approach was used for QAWRF, where respondents are expected to quantitively rate the existence of the risk factors available at a particular health department or facility based on their perception. This approach was selected because WPV incidences are difficult to predict, highlighting the need for WPV risk assessment or predictive tools.⁷⁴ The quantitative design of QAWRF enable users to independently assess WPV risk factors without the need for interviews or walkthrough surveys. It might also provide better insight into WPV risk factors due to respondents’ anonymity and flexibility in answering the questions.^73,75,76

For content validation, the less conservative method, ie, the S-CVI/average method, was chosen.^30,32,77 Another method that can be used to calculate S-CVI is by calculating the value based on Universal Agreement among experts (S-CVI/UA). However, S-CVI/UA method will only yield a good score if all experts can 100% agree (Universal Agreement) with most items in a questionnaire.³¹ A Universal Agreement among experts on an item could not have been achieved during the content validation process which was conducted during the COVID-19 pandemic, because physical meetings were prohibited in Malaysia, and the experts had to rate the component questionnaires independently. Nevertheless, the S-CVI/Ave index indicated excellent content validity for QAWRF, suggesting that it can measure WPV risk factors adequately.

The QAWRF-administrator was developed using 48 items with one extra item related to funding for WPV prevention compared to the QAWRF-worker. After content validation, the items eliminated from the QAWRF-administrator include risk factors like “low educational status of the client”, “client with a history of violence”, and “unprepared worker”. The two items related to the client were eliminated because administrators are unlikely to know about the client’s educational status and violence history. Meanwhile, “unprepared worker” was eliminated because experts believed workers scheduled for work at the healthcare department should be fully prepared physically and mentally and should otherwise be on medical or annual leave.

For the QAWRF-worker, the four items eliminated include “low educational status of the client”, “client with a history of violence”, “worker’s having low self-esteem”, and “high unemployment in nearby communities”. The first two items were eliminated due to the workers being unlikely to know about the client’s educational and violence history as well. “Worker having low self-esteem” item was eliminated because the experts argued that a worker may not be able to evaluate another worker’s self-esteem. “High unemployment in nearby communities” was eliminated because the experts felt that workers might not know about the unemployment rate in the communities where they work.

The QAWRF-client was developed using only three domains where the client’s risk factors domain was excluded. The exclusion was made because clients who attend a healthcare department are only exposed to characteristics of other clients for a minimal time, and thus are not suited to rate the risk factors related to other patients. After content validation, the societal domain was also eliminated from the questionnaire because some clients who visit a particular healthcare department do not live near the facility. In these cases, they might have travelled far to get treatment at a particular healthcare department which may cause them to have poor knowledge regarding the area’s societal risk factors for WPV. In addition, two items were eliminated; one during the content validation whereas another was eliminated during the pilot survey. The item “unprepared worker” was eliminated because experts felt that clients are unlikely to know about a worker’s physical or mental state of preparation at their workplace, whereas “worker probably having high workload” was eliminated because of the initial low Cronbach’s alpha value for the worker domain (Table 5). The low Cronbach’s alpha value might reflect poor inter-relatedness between the eliminated item and other items within the domain,⁴³ which may be due to clients’ difficulties in estimating the workload of HCW, thus causing random responses. Therefore, the committee agreed to eliminate the item from the questionnaire.

The Likert scale’s end descriptors for QAWRF were modified following comments from the face validation, where raters commented that they were confused when using the scale for rating the questionnaire. This supports the suggestion that respondents’ communicative and cognitive processes when answering a questionnaire can be affected by the design of the rating scale.^78–80 For QAWRF, the study committee noted that this issue existed because the end descriptors were too wordy. In addition, the QAWRF items were based on risk factors which had to be negatively worded (eg, “workers not knowing workplace violence prevention guidelines at their workplace”), where upon using the initial scale’s end descriptors, some raters reported being confused as to whether the risk factors should be rated as 1 or 5 if the negatively worded items were in a good or desirable condition. Therefore, to improve the understanding of the rating scale among respondents, the scale’s end descriptors were revised to “highly apparent” and “highly not apparent”, which are less wordy and the word “apparent” itself is better understood in Malay compared to the word “existence”.

Some studies recommend avoiding negative statements in a questionnaire as they might affect the construct validity and reliability of a questionnaire.^79,81 However, this may not apply to QAWRF as all items included are WPV risk factors despite some being stated in non-negative wording (eg, “overcrowding of patient’s area”). Respondents are required to rate the presence of the WPV risk factors unidirectionally without the need to reverse their cognitive understanding on the rating scale, which might occur if protective factors are also included in QAWRF. This is supported by good internal consistencies of QAWRF as reflected by the Cronbach’s alpha indices.^79,81

There are a few limitations to this study. Firstly, QAWRF potentially lacks generalizability because it was developed in Malay, and the risk factors selected were specific to Type 2 WPV in healthcare. Secondly, the administrator and QAWRF-worker were quite lengthy because there were multiple WPV risk factors that had to be included to ensure the robustness of the questionnaire. Nevertheless, efforts were made to use short wording for each item to reduce respondent fatigue.⁸² Thirdly, as aforementioned, the questionnaires were validated without any physical meeting between expert panels, which may have limited the discussion regarding the inclusion of items in the questionnaires. Nevertheless, the psychometric indices for QAWRF remained adequate. Future studies should include assessment of the construct validity and test–retest reliability of QAWRF, as well as the translation and validation of the English version of QAWRF to strengthen its usability and validity.

Conclusion

This study has developed, validated, and determined the reliability of a set of questionnaires, QAWRF, that can be used to assess risk factors of WPV in healthcare facilities based on the perspectives of three key healthcare stakeholders. In terms of theoretical implications, the QAWRF would be a valid and convenient tool for researchers to collect data on the prevalence of different WPV risk factors in any healthcare setting. In terms of practical implications, the findings from QAWRF will be valuable for policymakers developing WPV prevention policies and programs at a departmental or organizational level. These worksite-specific WPV interventions are expected to be resource efficient and more effective than a general WPV intervention.