Testing of Reliability and Validity of the Peninsula Health Falls Risk Assessment Tool (PHFRAT) in Acute Care: A Cross-Sectional Study

Background

Patient falls are a significant problem in acute care hospitals.^1,2 They are the most commonly reported accidental events in hospitals, and about 25–50% of inpatient fallers suffer injuries.^1,3 Inpatient fall rates in acute care range from approximately 1 to 9 falls per 1000 patient days.^1,3–5

Falls can cause morbidity, social and personal burdens, an increased need for long-term care, and mortality. In addition, falls increase health care costs by additional examinations and prolonged stays in the hospital.^1,6,7 Costs will be generated even if the fall does not cause injury to the patient because imaging services are typically used.^8,9

Prevention of falls is a part of quality management systems, and falls are used as an indicator of the quality of care. Healthcare quality guidelines and health services recommend the use of risk assessment tools to identify risk patients.^10,11 A fall risk assessment provides an objective evaluation of a patient’s risk of falling.¹⁰

Several tools have been developed for the detection of fall risk.^12,13 However, it is essential that the risk assessment is verified to be suitable for the environment in which it is used. There is no single suitable fall risk assessment for use in all environments and all patient populations.¹⁴

The fall can be defined as an event of a patient that is an unexpected, unintentional descent coming to rest on the floor or against some other surface, with or without injury.¹⁵ This includes slips, trips, being lowered, loss of balance, and legs giving way.^15,16 The number of falls is often reported as the total falls per 1000 patient days. Patient days can be most accurately calculated over the entire treatment period in actual hours.¹⁶

Falls are generally multifactorial, with synergism among various individual risk factors. Successful fall prevention requires systematic and effective identification of risk patients and individualized interventions to prevent falls of the high-risk patients. Given that patients are acutely ill and stay in the hospital for only a few days, a fall risk assessment needs to be quickly conducted and be effective.^17,18

Generally, two types of tools are used for a fall risk assessment. First, risk screening tools typically consist of several items, and a total score is then generated from the risk points in the sub-areas. The screening tools aim to predict the patient’s risk of falling based on a numerical score. Secondly, fall risk assessment tools assess/evaluate factors that contribute to the patient’s risk of falling. Each risk factor should be addressed to prevent falls. Typically, these tools include both fall risk factors and a scoring mechanism.^19,20

Fall risk assessment tools are often developed and tested for a certain type or age group of patients. Therefore, a tool tested on one patient group may not be valid or reliable in another group.^13,21 The validity of fall risk assessment tools varies between studies and depends on the unit type and the medical specialty. For certain risk assessment tools, evident limitations have been identified, and some of the reviews do not recommend the use of any tool due to their low reliability.^12,13,22 Instead of developing new tools, there should be good validation of existing tools. The focus should be on good validation rather than the development of new tools.²³

The aim of this study is to explore the validity and reliability of the Peninsula Health Falls Risk Assessment Tool (PHFRAT) in acute care. The numerical risk prediction tool was developed in Australia Peninsula Health (PH) for subacute and residential aged care in 1999. Further validity of the tool has been studied in the Stapleton et al’s study in residential age care.²⁴

The Finnish Institute for Health and Welfare (THL) translated the PHFRAT from English into Finnish in 2012. Since then, the PHFRAT has been introduced in the study organization in accordance with the national recommendation. The tool has not been previously studied in acute care. It is therefore important to obtain evidence of the validity and reliability of the PHFRAT from the different specialties of acute care.

Materials and Methods

This study with a retrospective and a prospective arm was conducted in southern Finland at a large university hospital. The acute care hospital system has approx. 2500 hospital beds. Every year, 92,000 surgical procedures are performed in the hospital system, and treatment is given to approximately 614,000 individual patients. The average length of stay (LOS) is 4.0 days (20.7 days in psychiatry). The total number of personnel is 24,900, of which 14,300 are nurses.²⁵

The study hospital system has implemented a fall prevention practice in which all patients are pre-screened for the risk of falling. All patients over 65 years or having previous falls will also be evaluated with the PHFRAT. The assessment is carried out by the nursing staff as soon as possible when a patient arrives at the facility. Assessment data and falls are registered in electronic health records (EHRs) in a structured format.

The purpose of the retrospective study arm was to determine the predictive validity of the PHFRAT in different medical specialties of acute care. The study data was formed from all adult inpatients who were evaluated for the risk of falling between the years 2014 and 2016. Demographic, medical, and fall risk assessment data were collected from the hospital system’s data warehouse. Figure 1 shows the formation of the data.

Figure 1 Retrospective medical data 2014–2016 for predictive validity study.

The prospective study arm concerned the reliability of the PHFRAT in different medical specialties in acute care. The inter-rater data was collected by nurses from September to December 2016 from 12 inpatient units: two psychogeriatric, three surgical (vascular surgery, orthopedics, plastic surgery), two neurological, and five medical (pulmonary disease, cardiology, general internal medicine) units.

The patients fulfilling the following inclusion criteria were recruited to the reliability study: Age 18 years or over, admitted as an inpatient to one of the study units, and signed their informed consent to participate in the study. The patient was assessed twice with the PHFRAT during the same shift so that the patient’s condition did not change between the measurements. Two nurses did the PHFRAT assessment without knowing each other’s assessment results. One of the nurses documented the information in the patient’s EHR, and the other nurse used a paper form. The nurse who filled out the paper form also responded to the questionnaire on the usefulness of the PHFRAT.

The PHFRAT used in this study was developed based on the literature. Nine fall risk factors were studied in subacute and elderly care. In the analysis, the four most predictive factors were identified and selected for the tool: Recent falls, psychological status, medications, and cognition. Total scores range from 5 to 20, and patients are classified into three risk groups: Low-risk (5–11), medium-risk (12–15), and high-risk (16–20). Table 1 describes the PHFRAT scoring by risk factors. The same test study found that the PHFRAT provided a sensitivity (correctly identifying high-risk patients) of 70% and specificity (correctly identifying those patients not at risk) of 68%, the cut-off point was 12, and good reliability (intraclass correlation coefficient, ICC = 0.79).²⁴ In Barker et al’s study, the accuracy was worse. Sensitivity was 52% and specificity was 66%, but the cut-off used was at 16 points. The PHFRAT’s reliability was found to be good, test-retest agreement Kappa was 0.68, and interrater agreement Kappa was 0.84.²⁶

Table 1 PHFRAT – Assessment Tool

Ethical Consideration

The study was conducted in accordance with the Declaration of Helsinki. The study was approved by the Helsinki University Hospital ethics committee in 2016 (80/13/03/00/2016) before the study was initiated and study permission was given by the hospital authorities (HUS/53/2016 and HUS/466/2019). Patient information for the validity study was obtained from patient registers and consent to participate in the study was not needed. Confidentiality of the participants was maintained because no identifying information was recorded. All participants in the reliability study were informed by a nurse about the purpose of the study. All the patients were asked to give their written consent to participate in the study, and participation in the study was entirely voluntary. Personal identification information was removed from the data before the analysis.

Data Analysis

Statistical analyses were conducted using IBM SPSS Statistics 25 (IBM Corporation, USA) and Microsoft Excel (Microsoft, USA).

Predictive Data

Descriptive statistics were calculated for demographic data including all evaluated patients. Cross-tabulation and a Pearson’s chi (χ²) test were used to analyze connections between falls and the PHFRAT. Logistic regression was conducted between falls (dichotomous category yes/no) and the risk level of the PHFRAT. No other independent variables were used in the analysis. The level of significance was set at p≤0.05. An ROC (receiver operating characteristic) curve was used to provide an illustrative and accurate picture of the test classification capability. The ROC curve is a graph generated where the true positive rate (sensitivity) is on the y-axis, and the false positive rate (the complement of specificity) is on the x-axis and calculated for each possible threshold value. The area under the ROC curve (AUC) measures the functionality of the test. The AUCs can range from 0.5 to 1.0, where 0.5 results have no predictive ability. The target value is >0.7, and >0.9 is excellent.^27,28 Only those falls that were connected to the PHFRAT assessment seven days before the fall were included in the analysis (n = 155). The validation of the predictive power of the PHFRAT was calculated by the Poisson regression model (PRM), which can also be used to calculate rare occurrences.²⁹ Poisson regression were used to estimate relative risk (RR) with 95% CI of risk levels of the PHFRAT with the number of falls and length of stay in the hospital. In the first analysis, only the risk levels of the risk assessment were used as an explanatory variable. In the second analysis, risk factors (recent falls, medications, psychological status, and cognitive status) for each risk level were used as explanatory variables.

Inter-Rater Reliability and Content Validity Data

Inter-rater agreement data was calculated using Fleiss’ Kappa (K) with 95% confidence intervals (CI) that incorporate the distribution of disagreement. Fleiss’ Kappa is used in ordinal data with several evaluators. The quantitative content validity index (CVI) was calculated for the item level (I-CVI) and the scale level (S-CVI). I-CVI is calculated as the number of evaluations giving a rating of positive responses and divided by the total number of evaluators. An I-CVI of 0.78 or higher is considered to be evidence of good content validity.³⁰ For S-CVI, a criterion of 0.80 is considered as the lower limit of acceptability.³¹ Nurses evaluated the usability of the items of the PHFRAT to identify the risk of falling. A 4-level Likert scale ranging from very useful to not at all useful was used in the evaluation.

Results

Predictive Validity

Between 2014 and 2016, 22,700 inpatients were assessed with the PHFRAT. The typical patient was almost seventy years of age, had a low fall risk, and was being treated for an internal medicine specialty for an average of six days. Over 80% of the patients were at low risk (Table 2). A total of 353 falls occurred for the 22,700 patients, which is 2.6 falls per 1000 hospital days, varying by medical specialty: medical 2.8 (n = 180), surgical 2.8 (n = 54), obstetrics and gynecology 2.2 (n = 3), ophthalmology and otolaryngology 2.3 (n = 3), oncology 4.0 (n = 22), psychiatry 2.4 (n = 24), neurology 2.0 (n = 61), and general medicine 3.9 (n = 6).

Table 2 The Demographic Data for Validity Study

The data was divided into two groups by somatic and psychiatric data. In the χ²-test, no dependencies were found in the psychiatric data, while a significant association between falls (yes/no) and risk level was found in the somatic data (p < 0.01). A logistic binary regression model was performed with the somatic data, in which the dependence between risk level and falls was found: low-risk to medium-risk odds ratio (OR) 4.19 (95% CI, 2.46–7.12; p <0.01) and low to high OR 2.40 (95% CI, 1.30–4.41; p < 0.05). Table 3 shows the results of the ROC analyses by medical specialties. In all somatic data, the area under the curve was 0.668 (CI 95%, 0.63–0.71; p < 0.01) and the optimal cut-off point was 9, while sensitivity was 0.71 and specificity was 0.59.

Table 3 Summary of ROC Analyses for Each Medical Speciality

Poisson regression was used to predict the falls at each risk level in both subdata, but the psychiatric data had no evidence of an association between the score and the fall risk. In the somatic data, the change of the risk level from the low-risk to the high-risk level increased the probability of falls by a factor of relative risk (RR) 2.88 (95% CI, 1.68 to 4.94; p <0.01). There was no difference between the low-risk and moderate-risk groups. Table 4 shows the probability of falls by medical specialty.

Table 4 Poisson Regression Results and Falls per 1000/Hospital Days by the Risk Levels of the PHFRAT in Somatic Data (Obstetrics and Gynecology/Ophthalmology and Otolaryngology Data Were Not Analyzed Due to Incomplete Data)

In addition, the separate items of the PHFRAT were examined, in which the recent falls and the psychological status were statistically significant and predicted falls. Change in the recent falls items increased the relative risk by a factor of 1.08–2.1, depending on the level of the parameter. A change in the item of psychological status increased the relative risk by a factor of 1.5–2.0, depending on the parameter level. There was no statistical significance in the items of cognitive status or the factor of medication. All the examined item results of the tool are presented in Table 5.

Table 5 Poisson Regression Results and Falls per 1000/Hospital Days by Risk Factors of the PHFRAT in Somatic Data n=22,407

Inter-Rater Reliability and Content Validity

There were 270 beds in the study units. In all, 30,992 patient days and 6582 patients were treated during the study period. The mean LOS was 4.3 days. A total of 292 nurses were employed in the study units. In all, 143 falls were recorded in the study units in 2016, signifying 1.65 falls per 1000 patient days.

The inter-rater and content validity assessment was performed by the nurses in the study units. The majority of the assessors were registered nurses (77.4%) and the rest were licensed practical nurses (22.6%). An assessment was made for a total of 448 patients who gave their consent to participate in the study. Data on 359 patients were included in the analysis because only those were evaluated by the PHFRAT twice during the same shift. The participants were divided into four different specialties. Table 6 shows characteristics of the patients and results of the PHFRAT by specialty.

Table 6 The Characteristics of the Study Patients and Results of the PHFRATa by Medical Specialties

In the total sample, Fleiss’ kappa was at risk level 0.68 (95% CI, 0.56–0.77). The results of the Fleiss’ kappa are described by specialty in Table 7. I-CVI, the content validity index for factors, was in all data over 0.95, and in specialties, it varied between 0.86 and 1.0. S-CVI, the content validity index for scale level, was 0.96 for the whole data, and it varied by specialty from 0.89–0.98.

Table 7 Inter-Rater Reliability Coefficients by Fleiss’ Kappa for the PHFRATa in the All Data and Specialties

Discussion

This study aims at addressing the feasibility and predictability of the PHFRAT in acute care. The tool has not been previously studied in acute care, and therefore no comparative data are available for the results. In the retrospective data, there were 2.6 falls per 1000 patient days, which is a moderate number and slightly lower than in many other studies.^3,4 The variation in occurrence was based on the risk level of the PHFRAT, so that in the low-risk level, there were 2.1 falls per 1000 patient days, and in the high-risk level, there were 7.4 falls per 1000 patient days. Comparable findings have not been reported in previous studies.

In the somatic data, the PHFRAT was found to predict falls. Binary regression analysis showed that falls are more likely in medium- and high-risk-level patients compared to those at a low-risk level. Poisson analysis showed that the PHFRAT does predict the increase in the probability of a fall when the change in the risk level is from low risk to high risk. However, no difference was found to be statistically significant between the risk levels of low risk and medium risk. Previous studies have not evaluated the predictive power of the PHFRAT.

In addition, the ability of the separate items of the PHFRAT to predict falls was studied, and previous falls predicted the most significant patient falls during the hospital stay. In the development of the PHFRAT, previous falls were most significantly related to falls.²⁴ This has been found in other studies on fall risk assessment tools as well.^32–34 A number of studies have identified a significant link between medications and the risk of falling.^35–37 However, no statistical significance was found in the medication item of the tool. Psychological status was associated with an increased risk of falling, which is also a parallel result with other studies.³⁸ In the present study, a patient’s cognitive status did not increase the risk of falling, although in other studies, it has been identified as a significant risk factor.^34,39 In the future, the adequacy of the medication category in the PHFRAT in acute care needs to be tested. Other fall risk assessment scales, such as the St. Thomas Risk Assessment Tool (STRATIFY) and the Morse Falls Scale, are in wide clinical use and although they have been studied the knowledge of their psychometric properties also remain deficient. Unlike PHFRAT, they include item about persons mobility.²⁰ Therefore it would be of interest to study whether adding mobility item would have the effect of the predictive power of PHFRAT.

In this study, the optimum cut-off point of the PHFRAT was 9 points when the sensitivity (72%) and specificity (59%) were reasonable. However, it should be noted that sensitivity varied by specialty, and it was lowest in surgical units (57%). Similarly, there was variation in specificity so that medical specialty had the lowest value (56%). In a previous study, sensitivity and specificity were better, but the study was conducted in a sub-acute environment and targeted to older people.²⁴ The sensitivity and specificity of the PHFRAT remain somewhat lower compared to more commonly used fall risk assessment tools.^13,40 In a review that evaluated the validity of the fall risk tools in an acute care setting, validity was calculated for the Hendrich II Fall Risk Model (HRM II; sensitivity 76%, specificity 60%) and the STRATIFY (sensitivity 89%, specificity 67%).¹³ In a study of three different fall risk tools (the Morse Fall Scale (MFS), HRM II, and STRATIFY), the sensitivity varied from 55–75%, and the specificity varied from 61–76%.⁴¹ If the PHFRAT cut-off point is set to the optimum 9 points, this would mean that 6227 (27%) patients in this study would not have been at risk of falling, and 16,473 (73%) would have been at risk of falling. This would change the practice because in the present study, only 3% of the patients were at high risk (16–20 points) and 14% were at medium risk (12–15 points) of falling.

The ROC curve combines specificity and sensitivity into one measure of the resolution of the gauge. In the present study, the resolution of the entire material is moderate (AUC = 0.67), which is just below the target value. The value is a bit lower compared to the MFS, STRATIFY, and HRM II tools, with the ROC curve AUC ranging from 0.7 to 0.80.⁴¹ Based on these results, the PHFRAT predicts moderately. The sensitivity of the tool is reasonable, but the specificity is poor, and the ROC curve AUC values of both are just about acceptable. The validity of the fall risk tools should always be tested depending on the environment. In this study, validity was also studied by specialty. In some specialties, due to the low number of falls (obstetrics and gynecology, ophthalmology and otolaryngology, and psychiatry), no reliable analysis was obtained. The challenge is that in the study organization, patients from all medical specialties are treated, and therefore—especially in psychiatry and in surgical specialties such as orthopedic and plastic surgery, and also in specialties of internal medicine, such as cardiac diseases—more research is needed.

The reliability of the PHFRAT was examined by assessments of patients, where two nurses assessed the same patient. The inter-rater agreements level of risk was at the moderate agreement level (k = 0.68). The result was exactly the same (k = 0.68) when the PHFRAT was researched in subacute health care.²⁶ Compared to the MFS, STRATIFY, and HRM II tools, the result is worse. For those tools, the values of Kappa were higher than 0.8.⁴¹ In the factors of the PHFRAT, Kappa varied from 0.56 to 0.66, which means moderate agreement or weak agreement. In addition, the content validity of the tool was examined, and excellent results were obtained. The overall content validity index of the assessment using a universal agreement approach was almost perfect agreement (I-CVI >0.95, S-CVI >0.96). In this study in psycho-geriatric units, Fleiss’s kappa was perfect (k = 1), and in surgical units, it was weak (k = 0.55). The mean LOS for patients assessed in psycho-geriatric units was 43 days and in surgical patients’ units six days, respectively. In psycho-geriatric units, the nurse gets to know patients better because of the longer care relationship, and this could partly explain the differences in kappa values in this study. In addition, the low kappa value may be explained by the fact that in this study, the risk assessments were not performed by the same designated nurses. The assessment was performed in the normal course of work, and the assessor may have had little experience using the PHFRAT. For patients with acute medical conditions, the challenge is that fall risk may rapidly change in the course of their hospital stay. Therefore there is need for short easy-to-use and reliable tool for fall risk assessment.

Limitations

The present study depicts a real-life situation where nurses carry out a fall risk assessment as part of their routine care. This may affect the results to some extent when compared to studies where the fall risk assessment has been conducted by research nurses. Additionally, in other studies, the assessment has often been carried out in different environments and populations.

The most relevant limitation of the present research is related to the prospective prospective word is wrong, change: retrospective data. The hospital has an inpatient fall prevention program and instructions. However, fall risk and falls can be interpreted and documented in many ways among care workers, thus affecting the validity of the data. In addition, a variation between the hospital units in the use of the PHFRAT and systematic documentation may exist. Falls are likely being underreported, thus affecting the analysis and reliability of the study.^42,43

Although the initial data was extensive, a modest group of patients had been assessed with the PHFRAT before the fall, limiting the calculation of the prediction of the tool in all subgroups or specialties. For ethical reasons, it was not possible to carry out a controlled clinical trial in which the patients’ risk of falling would have been assessed and identified but no preventive interventions were carried out. These also potentially affect the predictive validity.

Conclusion

The purpose of the study was to explore the validity and reliability of the PHFRAT in acute care and in various medical specialties. The PHFRAT proved to be moderately suitable for detecting the risk of falling for adult patients admitted to somatic units in acute care. The item of previous falls in the PHFRAT best predicted the future fall events. However, sensitivity and specificity were only moderate, and the optimal cut-off point was found to be lower than determined in the PHFRAT. Reliability settled on a moderate agreement level, and the content validity was at a good level. The results of the study can be used in somatic acute care and especially in the specialty of internal medicine. However, results indicate the need to study the PHFRAT more broadly in psychiatric as well as some specialties in somatic care. In addition, the effect of suggested amendments to the tool on its predictability is still to be tested.