External Validation of the Saga Fall-Related Injury Risk Model and Exploration of Common Factors in Multiple Hospitals: A Retrospective Observational Study

Introduction

While 88% of patients who fall during hospitalization do not sustain injuries requiring additional treatment,¹ some suffer from limited activities of daily living (ADLs) and long-term sequelae.^2,3 Moreover, fall-related injuries during hospitalization increase healthcare costs owing to longer hospital stays and additional treatment. Additionally, fall-related injuries lead to medical lawsuits, causing stress for healthcare providers.^4–10 The rates of fall-related injuries for inpatients have been reported as 23–63% in acute-care hospitals, and 8.3–38% in chronic care, psychiatric, and rehabilitation hospitals.^4,11–16 Therefore, prediction and prevention of fall-related injuries is an important and urgent issue in any hospital setting.

Many healthcare organizations use tools such as bed alarms and low beds to prevent falls and fall-related injuries among inpatients.^17,18 In addition, recent years have seen the emergence of fall prevention strategies utilizing technologies such as wearable sensors and robotic rehabilitation devices.^19–21 However, due to cost issues,²² storage space constraints, and a shortage of care personnel, it is difficult to introduce and operate these tools without restrictions in medical facilities. Although several predictive models for falls in hospitalized patients have been developed,^23,24 there is only one predictive model for fall-related injuries: the Saga Fall-related Injury Risk Model (SFIRM). SFIRM was developed using six factors (age, sex, emergency transport, medical referral letter, history of falls, and bedriddenness ranks) that were collectible at the time of admission.^25–30 SFIRM showed high discrimination and statistical fitting, with an area under the curve (AUC) of 0.772 and a shrinkage coefficient of 0.945 on internal validation.³⁰ Furthermore, externally validated results from 2 hospitals (one acute care hospital where the model was developed and another chronic-care hospital) showed good discrimination with an AUC of 0.721 (95% confidence interval [CI]: 0.662–0.781).³¹ In addition, we have previously reported the external validation of SFIRM using data from three chronic care hospitals included in the present study. That study also showed good discrimination, with an AUC of 0.856 (95% CI: 0.820–0.891), supporting the model’s applicability in chronic care settings.³² These findings suggest that SFIRM has the potential to be widely implemented in various healthcare institutions.

Building on these prior studies, this study aimed to evaluate the usefulness and generalizability of SFIRM by expanding the targeted hospitals through external validation of SFIRM at eight hospitals in Japan. For this, chronic care hospitals, acute care hospitals, and tertiary care emergency hospitals were included. Furthermore, as most studies on factors associated with fall-related injuries have been conducted at single institutions, this study aimed to explore the factors associated with fall-related injuries that are common in diverse hospitals.^1,30,33–36

Materials and Methods

Ethical Statement

This study complied with the Declaration of Helsinki and the “Ethical Guidelines for Medical and Health Sciences Research Involving Human Subjects” of the Ministry of Health, Labour and Welfare and the Ministry of Education, Culture, Sports, Science and Technology. This study was approved by the Saga University Clinical Research Review Board (no. 2021–07-SCR-07) and registered with the University Hospital Medical Information Network at www.umin.ac.jp (ID: UMIN000045420). Consent was obtained from all the patients at Hospital A using the comprehensive agreement method. Additionally, the opt-out method was used at all hospitals, including Hospital A, with detailed explanations of the study published on each hospital’s website. Patients had the opportunity to refuse participation by contacting the information provided on the website before their personal information was separated from the collected data. Patient data were gathered from electronic or paper medical records at each hospital. Before submission to Saga University Hospital, patients’ personal information was anonymized and separated from the data.

Study Design and Population

This was a multicenter retrospective observational study. Eight hospitals in several regions of Japan were included in the study, including tertiary acute care, acute care, and chronic care hospitals. Although the target hospitals included an acute-care hospital where the SFIRM was developed, the data in this study were collected from inpatients admitted during different time periods; thus, no data overlapped with the dataset used for model development. Additionally, data from three of the chronic care hospitals included in this study were also used in our previous study to validate the SFIRM in chronic care settings.³² In this study, the eight hospitals were denoted as hospitals A–H, and the characteristics of each hospital are listed in Table S1. The study population consisted of consecutive patients aged 20 years or older who were admitted to eligible hospitals between April 1, 2018, and March 31, 2021. Since the SFIRM employed in prior research was developed using datasets comprising individuals aged 20 years and older,^23,24,28,30 the present study also targeted patients from this age group. Given that “age” is incorporated as a predictor variable in the model, its applicability extends to the entire adult population, thereby obviating the need for population-specific adjustments during implementation. Nevertheless, considering that fall risk profiles differ in developmental stages from adult stage, individuals under the age of 20 were excluded from the analysis.

Variables and Sources of Information

All data were collected from paper or electronic medical records. In addition to 6 SFIRM factors, the following 41 items were investigated for all hospitalized patients based on previously reported factors associated with falls or fall-related injuries: data at admission; date of admission (date), patient age (years), sex, department (internal medicine, neurosurgery, others), type of admission (planned, unplanned), emergency transport, medical referral letter, history of falls, hypnotics usage, antipsychotic drug usage, walking aids usage, disturbance of consciousness, visual impairment, ophthalmological disease (cataract and glaucoma), parkinsonism, permanent damage by stroke, paraplegia and hemiplegia, alcoholism, diabetes mellitus, frequent urination (present or absent), ADLs (eating, grooming, dressing, swallowing, toileting, bladder control, bowel control, bathing, taking prescription drugs, transferring from wheelchair to bed, mobility on level surfaces, walking up and down the stairs) (independent, requiring assistance), BMI (<18.5, ≧18.5 and <24, ≧25), the easy-to-use official Japanese scales of bedriddenness ranks (normal, J, A, B, C) and cognitive function score (normal, 1, 2, 3, 4, M), fall protection measures (implemented, not implemented).³⁷ During the hospitalization there were also surveyed the following data: rehabilitation, surgery (Implemented, not implemented), falls (present, absent), fall-related injuries (present, absent), degree of fall-related injuries (Those that do not involve a disability, With disabilities 1, With disabilities 2, With disabilities 3).³⁸ Data at discharge; date of discharge (date).

Definition of Fall and Fall-Related Injury

Information on falls and fall-related injuries was collected from incident/accident reports. Usually, when a patient falls in a hospital, the nurse in charge of the patient or the one who discovers the fall records the details in an incident/accident report. A fall was defined as an unexpected fall from any height or position, including stairs, chairs, and beds, while walking, sitting, and lying down, with or without injuries caused by a fall, and included falls and suspected falls. Patients whose falls were reported in incident/accident reports were defined as those who fell, and their level of fall-related injuries was investigated. In this study, the severities of injuries caused by falls were classified per a part of “Reasons for Falls and Falls and Injury Severity Input Criteria” in the acute care indicators in the Maryland Hospital Association Center for Performance Sciences’ Quality Indicator Project.³⁸ Patients who sustained fall-related injuries that were equivalent to or more severe than level 1, defined as “Injury with no residual disability and no prolonged hospital stay”, were classified into the “With fall-related injury” group. All other patients, including those who fell but did not sustain injuries and those who did not fall, were classified into the “Without fall-related injury” group.

Definitions of Variables

The variables were based on the data assessed and entered into the patient’s medical records by the nurse in charge of the patient during routine medical practice. Bedriddenness ranks and cognitive function score are official ADLs indices proposed by the Ministry of Health, Labour and Welfare and used in the Japanese medical care system. There are five major categories for bedriddenness ranks (normal, J: independence, A: house-bound, B: chair-bound, or C: bed-bound) (Figure S1). There are six categories for cognitive function score (normal, 1: independence, 2: attention needed, 3: intermittent care needed, 4: continuous care needed, M: specialized treatment needed) (Figure S2).^1,37,39 Some hospitals classify bedriddenness rank J into two categories: J1 (those who have some disability and go out using public transportation) and J2 (those who have some disability and go out in the neighborhood). Data from Hospital A did not include patients of “normal bedriddenness rank”, which was inconsistent with reality and the data from other hospitals. It was considered a misclassification due to the difference in bedriddenness ranks usage. Therefore, J1 in Hospital A was analyzed as normal bedriddenness rank and J2 as J. In the data from hospital D, patients with a history of femoral neck fractures were classified as having a history of falls because they had no information on the history of falls.^40,41 Consciousness disorders were defined as Japan Coma Scale > 0, Glasgow coma scale < 15, or the presence of consciousness disorders in their health records. Hypnotics were defined as benzodiazepines and all non-benzodiazepine hypnotics, including melatonin receptor agonists and orexin receptor antagonists.⁴² Visual impairment was defined as a visual abnormality that limited the patient’s ADLs. Preventive measures against falls during hospitalization were assessed and entered by nurses based on an assessment of the patients’ fall risk at each hospital using their own methods.

The following variables, Parkinsonism, permanent stroke, paraplegia and hemiplegia, ophthalmologic diseases (cataract and glaucoma), alcoholism, and diabetes mellitus, were defined as the presence of the respective disease in their health records. This was also used for billing payments to insurers under the Japanese health insurance system.

Sample Size

Referring to the AUC of 0.794 for the SFIRM by internal validation reported in a previous study, the AUC for external validation in this study was assumed to be 0.700. With an alpha error of 0.05, a beta error of 0.2, and an effect size of 0.20 (null hypothesis AUC 0.5), the required sample size for each group was 15.7 persons. Given that the incidence of fall-related injuries in the previous study was 0.66%, the total sample size for the combined population of the 2 groups was 2374 persons.

Statistical Analysis

The required number of patients was randomly selected from the total sample for analysis. For each survey item in the with/without fall-related injury groups, the median (interquartile range) or absolute number (percentage) of continuous and categorical variables was reported for the two groups. Missing data for each item were treated as an independent category labeled “missing data”. Univariate analyses were performed using the Mann–Whitney test for comparison with continuous variables and the chi-square test for nominal variables. The AUC of SFIRM scores with a 95% CI and shrinkage coefficients were calculated using with fall-related injury as the outcome. The calculation formula for SFIRM is as follows: Score = −8.672+0.034 × (age) + (sex: female = −0.641) + (emergency transport: presence = −0.887, missing = −0.551) + (medical referral letter: presence = 0.399, missing = 1.189) + (history of falls: presence = 0.542) + (Bedriddenness rank: J = 1.597, A = 2.188, B = 1.986, C = 1.510, others = 0.154). The patients were divided into deciles based on the SFIRM scores, and the predicted and actual rates of fall-related injuries were calculated and compared visually.

Subsequently, the factors associated with fall-related injuries were examined in all patients. Variables that were available at admission were considered covariates in the multivariate regression analysis. When two factors exhibited high correlations (Spearman’s r > 0.7), one was selected as a candidate and the other was excluded owing to collinearity. A logistic regression analysis employing the forced entry method was conducted using the selected covariates for each hospital. The analyses of bedriddeness ranks were conducted using “normal” as the reference category, except for hospital D. As patients from hospitals C and D did not have a normal bedriddenness rank in the medical records, which is consistent with actual practice, bedriddenness rank J was used as the reference category. All analyses were conducted using SPSS version 22 (SPSS, Inc., Chicago, IL, USA), with the significance level set at p < 0.05.

Results

Characteristics of Patients and Incidence of Fall-Related Injury

A total of 144,777 adult inpatients were included in the analysis. A subset of 2376 patients was randomly selected for validation of the SFIRM (Figure 1). The median age of the validation population was 70 years (interquartile range: 58–78 years), with 1280 (54%) male patients, and a median hospital stay of 9 days (4–16 days). The incidence of falls was 1.6 per 1000 patient-days, whereas that of fall-related injuries was 1.1/1000 patient-days. A total of 51 patients (2.1%) were hospitalized, and 35 patients (1.5%) had fall-related injuries. In the fall-related injury group, the median age was 76 years (66–86 years), With 19 males (54%). The results of univariate analyses of all variables for all patients are shown in Table S2. The characteristics of the sampled population were similar to those of all patients.

Figure 1 Data flow diagram. In total, 162,177 patients were initially screened for inclusion in this study. After excluding 11,899 patients aged under 20 years, 10 patients with missing data on injury level, and 5491 patients with missing data on fall history, 144,777 patients were retained for multivariable analysis across hospitals. From this cohort, a random sample of 2376 patients was selected for evaluation of Area Under the Curve for the SFIRM.

Univariable Analysis

Table 1 presents the results of the univariate analysis of the sampled validation population. The fall-related injury group was significantly older (76 vs 70 years) and had a longer hospital stay (26 vs 8 days). At admission, the fall-related injury group was likely to be hospitalized in an emergency (66% vs 37%), have a history of falls (29% vs 7%); take antipsychotics (20% vs 6%); require assistance in eating (37% vs 18%), grooming (40% vs 18%), dressing (49% vs 23%), swallowing (23% vs 3%), toileting (51% vs 34%), bladder control (40% vs 16%), bowel control (43% vs 17%), bathing (51% vs 20%), taking prescription drugs (20% vs 5%), transferring from wheelchair to bed (60% vs 36%) and mobility on level surfaces (54% vs 22%), and; be using a walking aid (23% vs 7%); be obese with BMI ≥ 25 (34% vs 23%); have impaired consciousness (29% vs 9%); and receive rehabilitation during hospitalization (23% vs 9%). The two groups had significant differences in the distribution of the bedriddenness ranks and cognitive function score. The fall-related injury group had a significantly higher percentage of patients with bedriddenness rank A (23% vs 17%) and C (43% vs 16%) and cognitive function score of 1 (9% vs 3%), 2 (3% vs 2%), and 3 (26% vs 3%), respectively.

Table 1 Univariate Analysis for Patients Selected by Random Sampling

Performance of Predictive Models

The AUC for the SFIRM in the sampled validation population was 0.617 (95% CI, 0.534–0.701) (Figure 2). Figure 3 shows the predicted and observed rates of fall-related injuries in each group divided by the deciles of the SFIRM scores. The differences between the predicted and measured rates were observed, with a shrinkage coefficient of −0.050. In a hospital-specific analysis, the respective AUCs for SFIRM across hospitals with sample sizes exceeding 2376 cases (Hospitals A to E) exhibited a wide range, from 0.614 to 0.719 (Figure 2).

Figure 2 Receiver operating characteristics (ROCs) and AUC by randomly sampled group and hospitals. In this Figure, ROCs were shown for hospitals A-E that met the required sample size of 2376 cases.

Figure 3 Rates of predicted and actual fall-related injuries by score for each group. Scores were calculated by SFIRM and divided into 10 groups by quartiles. The higher the group number, the higher the risk of falls predicted by the model.

Selection of Covariates and Multivariate Analyses for Each Hospital

In total, 5 variables were excluded from the 38 collected variables for the analyses: falls and fall-related injuries were excluded as they were outcomes; independence in taking prescribed drugs and use of walking aids were excluded owing to a low percentage of available data (less than 40%); and implementation of surgery, rehabilitation, and length of hospital stay were excluded owing to unavailability of the data at the time of admission.

The following 23 variables were selected from the 33 variables as covariates in consideration of their collinearity: age, sex, department, type of admission, emergency transport, medical referral letter, history of falls, use of hypnotic drugs, use of antipsychotic drugs, disturbance of consciousness, visual impairment, ophthalmological disease, Parkinsonism, permanent damage due to stroke, paraplegia and hemiplegia, alcoholism, diabetes mellitus, frequent urination, eating, swallowing, BMI, bedriddenness ranks, and cognitive function score.

Table S2 presents the binary logistic regression analysis results employing the forced-entry method across hospitals. Seven factors showed significant differences between groups With and Without fall-related injuries (Table 2). Table S3 presents the results for all factors across hospitals. Among the 23 factors analyzed, age (hospitals A, B, C, D, and F) and bedriddenness ranks (hospitals A, B, C, D, and E) were identified as significant common factors associated with fall-related injuries across the 5 hospitals. Male sex (hospitals B, D, E, and F), history of falls (hospitals A, B, C, and D), and diabetes mellitus (hospitals A, B, D, and F) were identified as significant common factors associated with fall-related injuries across all four hospitals.

Table 2 Multivariate Analysis by Hospital, Selecting 7 Factors That Were Significantly Different at More Than 1 Hospital as Factors Associated with Fall-Related Injuries

Discussion

This study was a multicenter, retrospective, external validation of the ability of the SFIRM to predict fall-related injuries during hospitalization. The SFIRM requires the assessment of six factors upon admission. Unexpectedly, the discernibility and calibrability of the SFIRM in different backgrounds were lower. The results of this study reveal that the SFIRM needs to be adjusted for use in medical hospitals in different settings. Multivariate analyses across hospitals detected five significant factors associated with fall-related injuries across multiple hospitals: age, bedriddenness ranks, male sex, history of falls, and diabetes mellitus. These factors could be the leading predictors in fall-related injury risk models across multiple centers. We describe the reasons for the low AUC of the SFIRM in this study and the validity of five common factors associated with fall-related injuries as new model predictors across multiple centers.

Quality of Data

One reason for the decreased accuracy of the SFIRM in external validation using retrospective data from diverse backgrounds is the quality of the data for the factors comprising the SFIRM. For example, bedriddenness rank “normal” is defined as a state with no disabilities of any kind, whereas rank “J” is defined as a state with some disabilities, where the individual is able to go out into the neighborhood alone. There is a clear difference in ADLs between the “normal” and “J” ranks.³⁰ However, data from Hospitals A, C, and D showed no or few patients with bedriddenness rank “normal”. Therefore, the definitions of A or J in this study were partially changed, as described in the Methods section. The coefficient for bedriddenness rank “J” in SFIRM was higher at 1.597 compared to rank “normal”. Consequently, the actual state of bedriddenness rank “normal”, which was treated as rank “J1” in analysis, would be predicted to have a higher fall-related injury rate than the actual rate. An accurate assessment of bedriddenness ranks is essential for preserving the discriminative power of SFIRM.

The quality of the data on the presence or absence of a medical referral letter could be another reason for the low accuracy of the SFIRM in this study. In hospital B (31.1% of the total), all cases of medical referral letters were categorized as “unknown”. Although the category “unknown” included missing data and unassessed statuses, most cases were expected to be known based on data from other hospitals in this study and the data used to develop SFIRM. In this study, the actual status of the presence or absence of a medical referral letter at hospital B was treated as “unknown”. The coefficient of missing data on medical referral letters in SFIRM is the highest at 1.189, compared to those in the “normal” and “presence” categories. Consequently, rates of fall-related injury would have been overestimated compared to the actual status owing to categorizing all cases into the “unknown” category. Furthermore, the presence or absence of a medical referral letter was not a significant factor associated with fall-related injuries across hospitals. The relationship between medical referral letters and falls or fall-related injuries has not been reported anywhere except in our previous study. The presence of a medical referral letter is a fall-related factor specific to acute care hospitals.

Properties of SFIRM Factors Would Differ Across Hospitals

The highest AUC for the SFIRM was 0.719 for hospital E, where the model was developed. In contrast, a lower AUC for the SFIRM was observed in other hospitals, ranging from 0.614–0.647. The key difference between hospital E and the other hospitals is that the former is an acute-care hospital in a rural area, while the latter are tertiary acute-care or chronic-care hospitals. Tertiary acute care hospitals typically have emergency and intensive care units to which critically ill patients are referred or transported emergently. Chronic care hospitals typically have many referred or emergently transferred patients for long-term care. Therefore, the properties of some factors comprising SFIRM, such as medical referral letters and emergency transport, would differ across hospitals. Although SFIRM maintains good discrimination in acute-care settings, the model should be modified for use across hospitals with different backgrounds.

Common Factors Associated with Fall-Related Injuries Across Hospitals

Multivariate analysis using data from all patients across hospitals identified several common factors significantly associated with fall-related injuries, including age, bedridden ranks, male sex, history of falls, and diabetes mellitus. Age has consistently been reported as a predictor of fall-related injuries.^1,30,43,44 Bedriddenness ranks is a Japan-original ADL index, which demonstrated good inter-rater reliability and criterion-related validity with the Katz Index and Barthel Index in a single-center study.^1,30,37 Furthermore, the bedriddenness rank A, B, and C were reported to be associated with fall-related injuries and falls, respectively. Females have been reported to be associated with fall-related injuries, whereas males are associated with falls.^{13,27,31,42,45,46} As the data included patients who did not fall, males were significantly associated with falls, which is a prerequisite for fall-related injuries. A history of falls has been reported to be associated with both falls and fall-related injuries.⁴⁷ Diabetes has been reported to be associated with falls, which may have influenced the association with fall-related injuries in this study. Additionally, diabetes significantly increases the risk of fractures, indicating that it is also associated with fall-related injuries.^47–50 Therefore, the common factors associated with fall-related injuries across hospitals in this study were consistent with previously reported factors associated with falls or fall-related injuries. These factors could be useful in developing a prediction model for fall-related injuries that can be used in multiple hospitals.

Limitations

This study has some limitations. As this was a retrospective, multicenter study, there may have been problems with data quality, aside from the bedriddenness ranks and medical referral letters. Standardizing the data quality can increase the discrimination of the model. Additionally, fall prevention measures were implemented at each hospital based on their own fall risk assessments. Therefore, the true fall injury rates might vary between hospitals. Moreover, this study utilized data collected from 2018 to 2021, which may not fully reflect the current conditions. Future research should consider using more recent datasets to validate the findings. Moreover, this study could not conduct analyses that were adjusted for the presence or absence of preventive measures. Addressing these limitations requires conducting prospective studies that standardize data quality and clarify the criteria for implementing fall prevention measures in each facility. Finally, a portion of the dataset used in this study, specifically from three chronic care hospitals, was also analyzed in our previous study. Although the objectives and scope of the two studies differ, this overlap may limit the complete independence of the current validation.

Conclusion

External validation in populations with diverse backgrounds showed that the SFIRM had low discrimination. Age, bedriddenness ranks, sex, history of falls, and diabetes mellitus were common factors associated with fall-related injuries across multiple centers. Development and validation of a fall-related injury prediction model using the five factors identified in this study as predictors is desired.