Validation of the Nottingham Hip Fracture Score (NHFS) in Patients with Hip Fracture: A Prospective Cohort Study in the Netherlands

Introduction

Hip fractures are associated with high postoperative 30-day mortality of 7%–13%.^1–4 It is of great importance to identify patients most at risk for 30-day mortality. A 30-day mortality-prediction tool should have adequate and be extensively validated before usage in daily practice. A reliable prediction tool can help to provide appropriate informed consent for patients and families, assist clinicians in determining optimum timing of surgery, and point to intervention possibilities for perioperative management. Several mortality-prediction tools have been developed to help identify patients at high risk of mortality. Among these are the Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity, the Charlson Comorbidity Index, and the Nottingham Hip Fracture Score (NHFS).^3,5–9 The NHFS has shown the most promising results in predicting 30-day mortality in hip-fracture patients.¹⁰

The NHFS was originally developed in 2008 by Maxwell et al as a prediction tool for 30-day mortality in patients with a hip fracture (Supplementary Table 1 and Supplementary Figure 1).¹¹ After its original development, the NHFS was validated by Moppett et al in 2012.⁵ They found overestimation of 30-day mortality for higher and lower scores. A revised equation was thus developed, which has been calibrated in three hip-fracture cohorts (ie, Peterborough, Brighton and Nottingham).⁵ Rushton et al thenvalidated this modified NHFS in a retrospective cohort of patients who had had surgical treatment for a hip fracture.¹² Simultaneously with Rushton et al, the NHFS was validated by Marufu et al in a different cohort.¹³ This simultaneous recalibration and validation of the NHFS emphasizes the clinical demand for a simple but accurate prediction tool for 30-day mortality in patients with hip fractures. Development of the NHFS also demonstrates that such a prediction tool is subject to evolution of patients and care; therefore ongoing research and validations are necessary to keep the prediction tool uptodate.¹³

In the literature, the NHFS has seemed to overestimate the risk of 30-day mortality after a hip fracture and has thus been adjusted over time.^5,13 With epidemiological changes in the hip-fracture population, ongoing research and validations are necessary to keep the prediction tool uptodate. Furthermore, the extent of applicability of the NHFS in patient populations outside the UK has to be examined more thoroughly. This study aimed to prospectively investigate the predictive value of the modified NHFS for 30-day mortality in a large (Dutch) cohort of patients with hip fractures.

Methods

NHFS

The NHFS is based on age (≤65 years, 66–85 years, ≥86 years), sex (male), hemoglobin concentration at admission (≤10 g/dL), living in an institution, number of comorbidities (greater than or equal to two), presence of malignancy in the last 20 years, and abbreviated mental test score (AMTSof ≤6 out of 10). The AMTS is used to detect cognitive dysfunction at admission. A cutoff ≤6 is used to indicate moderate–severe cognitive dysfunction associated with dementia.¹¹ The AMTS was not available in the participating study centers. Geriatric assessment of cognitive dysfunction or prior diagnosis of cognitive dysfunction was used as a proxy for the AMTS in calculating the NHFS. The NHFS can be calculated by combining the scores of the aforementioned variables (Supplementary Table 2). NHFS scores range between 0 and 10, and based on the score, 30-day mortality can be predicted. Predicted mortality for patients scoring 0 is 0.7% and increases to 45% in patients scoring 10.⁵

Patient Inclusion

Data were extracted from a prospective hip-fracture database (FAMMI trial, NL8313). Consecutively admitted patients between January 1, 2018 and January 11, 2021 in two level II trauma teaching hospitals in Rotterdam, the Netherlands were screened for eligibility. Inclusion criteria were age ≥18 years and a hip fracture, ie, femoral neck fracture, intertrochanteric fracture, or subtrochanteric fracture. Patients were included regardless of the chosen treatment (surgical or nonsurgical). Surgical treatment included placement of cannulated screw fixation, a dynamic hip screw, a femoral neck system, an intramedullary fixation (gamma nail), hip hemiarthroplasty, or total hip arthroplasty. In cases of nonsurgical treatment, patients underwent a fascia iliaca compartment nerve block or received adequate pain medication. Patients were treated nonsurgically if after careful consideration among trauma surgeon, geriatrician, patient, and family, it was estimated that a significant decrease in quality of life or death would be inevitable despite surgery. Patients were excluded for missing baseline characteristics needed to calculate the NHFS.Minimum follow-up after surgery was 30 days. The Office of National Statistics (Basisregistratie Personen) was consulted after a minimum of 30 days to extract death dates if patients were deceased. These dates were cross-checked with hospital and nursing-home records.

Patient Variables

Baseline characteristics comprised patients’ age, sex, residence type, potential use of walking aids, American Society of Anesthesiologists score, medical history (malignancies and ongoing chemotherapeutic treatment, diabetes mellitus, rheumatoid arthritis, Parkinson’s disease, chronic obstructive pulmonary disease, transient ischemic attack, cerebrovascular accidents, myocardial infarction and other cardiovascular diseases, and hip-fracture surgery), glomerular filtration rate (mL/min/1.73m²), hemoglobin level at admission, weight and height to calculate body-mass index (kg/m²), and medication use. A cutoff of NHFS ≥5 was used to identify patients at high risk of 30-day mortality, as in previous literature.⁴

Statistical Analysis

Data were stored using a cloud-based clinical data–management platform, (Castor EDC, Amsterdam, Netherlands) and analyzed using Stata 14.0 (StataCorp, College Station, TX, USA). The 30-day mortality observed in the study cohort was compared to that predicted by the NHFS. Predicted mortality was calculated using the formula for the modified NHFS: 100/(1+exp [5.012NHFS0.481]).⁵ All statistical tests were two-sided with a significance level of P<0.05. Pearson’s ?² was used to assess the overall goodnessoffit, using the range of possible NHFS as natural categories. Mann–Whitney U values were calculated to find potential miscalibration. Discrimination evaluation was performed using the concordance statistic as an equivalent to the area under the receiver-operating curve: 0.70–0.79 is considered to represent acceptable discrimination of a model and 0.80–0.89 excellent discrimination.¹⁴

Ethics Statement

The local medical research ethics committee (Maasstad Hospital, Rotterdam, Netherlands) approved the study (trail registration NL8313). All protocols were in compliance with the Declaration of Helsinki. Because of the high percentage of cognitive dysfunction and no changes made to the standard practice of care, the local ethics committee decided that patient consent to review their medical records was not required. No external funding was received for this trial, and no conflicts of interest need to be reported.

Results

In total, 2,481 patients were screened for eligibility, of which 23 (0.9%) were excluded due to missing data needed to calculate the NHFS (Figure 1). Data were missing due to absent laboratory tests at admission. Baseline characteristics of the included patients are shown in Table 1. Mean age was 80±12 years, 66% were women (n=1,631), and 69% had an American Society of Anesthesiologists score ≥3 (n=1,698). Median NHFS was 5 (IQR 4–6). Observed overall 30-day mortality was 7.9% (n=195).

Table 1 Baseline characteristics of study patients

Figure 1 Flowchart of patient inclusion.

Observed and predicted 30-day mortality for each NHFS are presented in Table 2. Observed 30-day mortality increased with higher NHFS. In total, 1,294 (53%) patients had an NHFS ≥5 (with a 30-day mortality of 12.4%) and were identified as being at high risk of 30-day mortality based on previous literature.^4,12 Details of the NHFS and observed mortality are presented in Table 3.

Table 2 Mortality for each group compared with predicted NHFS

Table 3 Proportion of patients considered to be at higher risk of 30-day mortality

Overall goodness of fit was tested with Pearson’s ?² (11.8, df 10; P=0.297; Figure 2). No statistically significant signs of miscalibration were found (P=0.08). Discrimination was tested with area under the receiver-operating curve, which was 72.1% (95% CI 68.7%–75.4%). Results of validation of the modified NHFS in the original validation cohorts used by Moppett et al are presented in Supplementary Figure 2 as a visual comparison.⁵

Figure 2 MortalityOobserved in Rotterdam cohort compared to modified NHFS.

Discussion

This prospective cohort study aimed to externally validate the NHFS with sufficient power. The NHFS was a reasonable predictor of 30-day mortality for the patients in this study cohort. No statistically significant signs of miscalibration were found (ie, no systematically deviating predictions of 30-day mortality were detected, P=0.08). The NHFS showed acceptable discrimination in predicting 30-day mortality, with an area under the receiver-perating curve of 72.1% (95% CI 68.7%–75.4%). For validation of the NHFS, the outcome measure of interest is 30-day mortality. As stated by Karres et al, compared to other prediction tools, such as the Charlson Comorbidity Index, Orthopaedic Physiologic and Operative Severity Score for the Enumeration of Mortality and Morbidity, and Estimation of Physiologic Ability and Surgical Stress, the NHFS shows the most promising results in predicting 30-day mortality rates.¹⁰

For external validation of a model, a number of statistical principles apply: validation is done in a fully different setting than that in which the development was done,¹⁵ the minimum number of observed events in the study cohort is 100;^15,16, and the variable of interest has similar incidence or prevalence in both development and validation cohorts.¹⁷ Three validation studies have been in conflict with these statistical principles: by Kau et al, Tilkeridis et al, and Karres et al.^10,18,19 These studies were not compliant with the minimum required observed events, reporting 15 (7.3% mortality in a sample of 212), 33 (9.5% mortality in a sample of 349), and 89 events (8.2% mortality in a sample of 1,050), respectively. Kau et al, however, found that the NHFS accurately predicted 30-day mortality for surgically treated hip-fracture patients.¹⁸

In Greece, most elderly adults live at home, supported by family members.^2,11,19 For this reason, Tilkeridis et al replaced the factor of institutionalization with the New Mobility Score to calculate the NHFS; however, this version of the NHFS was not validated externally.¹⁹ Similar adjustments to the NHFS have been made to develop new prediction tools, such as the Almelo Hip Fracture Score proposed by Nijmeijer et al.³ However, their also had observed events included in the model (64 events in a population of 850 patients).

Another retrospective validation study was performed by Rushton et al.¹² In their cohort, neither patients aged <60 years nor those not treated surgically were included in statistical analysis to validate the NHFS. The original cohorts used by Maxwell et al and Moppett et al included nonsurgically treated patients to develop and adjust the NHFS. Patients that do not undergo surgery for a hip fracture are naturally the frailest patients with the highest NHFS. Identifying patients for whom it is estimated that significant decrease in quality of life or death would be inevitable despite surgery is helpful in the process of well-informed shared decision-making and choosing the most appropriate treatment for hip-fracture patients. Excluding these patients from statistical analysis contributes to the finding that the NHFS has less accuracy in predicting 30-day mortality for the higher NHFS in the study by Rushton et al. This less accurate prediction can be explained by the limited number of included patients (n=42) in these higher NHFS (ie, level 8 and 9) derived by Rushton et al.¹² Observed mortality in those with an NHFS of 8 and 9 in our study cohort was nevertheless comparable to that observed by Rushton et al. The mortality observed in these two cohorts could thus be more representative than the 30-day mortality rates predicted by the NHFS.¹² Since one of the variables of the NHFS is age, there should be no need to exclude patients according to age limits, as was done by Rushton et al. Also, a limited number of events were observed (n=79) in this cohort, which makes it uncompliant with the principles of external validation, as described previously.

The NHFS was developed for all types of hip fractures. Some of the literature, however, indicates that intracapsular hip fractures can significantly increase the risk of mortality compared to extracapsular fractures. Since fracture type is not included in the NHFS, de Jong et al performed a validation study of the modified NHFS in a cohort of patients with femoral neck fractures treated with hip hemiarthroplasty.² They found that the NHFS underpredicted 30-day mortality in patients aged >70 years in this cohort, as the observed 30-day mortality rate was higher. The potential role of type of surgery could explain this underprediction. Smith et al identified preoperative indicators of mortality following hip-fracture surgery. One of the factors affecting 12-month mortality was the presence of an intracapsular fracture. Compared to extracapsular fractures, the relative risk of mortality was 77% (95% CI 0.63%–0.95%),²⁰ possibly because a hip hemiarthroplasty is a more extensive operation than intramedullary fixation of a hip fracture with prolonged recovery.²⁰

Lastly, a study by Doherty et al was published recently describing the predictive value of the NHFS for mortality, physical function, length of stay, and postoperative complications.²¹ Compared to our multicenter study design with an inclusion period of 3 years, this single-center study had an inclusion period of about 10 years, limiting its generalizability. Also, its results would be subject to changes in care over the period in which enrolment of the patients was performed.

To make the NHFS more usable in daily practice, a subdivision into low- and high risk patients has been described.^4,10 Previous literature suggested an NHFS of ≥5 as a cut-off-point between low- and high risk patients.⁴ In our cohort, the observed 30-day mortality is 3% for patients with an NHFS ≤4 (n=1164). However, an NHFS ≥5 identified 53% (n=1294) of the patients in our cohort as patients at high risk of 30-day mortality with an observed mortality of 12.4%. Rushton et al report the same finding and opt for an NHFS ≥6 as cut-off-point to identify patients at high risk.¹² We on the other hand opt for an NHFS ≥8 as cut-off-point for high risk patients, since for this cut-off-point one in three patients deceased within 30 days (n=17, observed mortality 30.9%). Also, for the low risk patients with an NHFS ≤7 the observed mortality is lower than the overall observed mortality in the entire study cohort (n=178, observed mortality 7.4%).

Strengths and Limitations

Our study differs from the aaforementioned studies in that it is the first validation to prospectively include all types of hip fractures in a large adult-patient cohort. The number of deceased patients in our population (n=195) was sufficient to externally validate the NHFS as per the previously mentioned principles for external validation.¹⁶ Since similar occurrence of the variable of interest (ie, 30-day mortality) in both development and validation cohorts is desirable, overall 30-day mortality in our cohort was compared to the developmental and validation cohorts, and was 7.9%, ranging from 6.6% to 8.0% in the two cohorts.^5,11 Since these percentages were comparable to the outcome variable, our cohort was suitable to validate the NHFS. Another strength of our study is the large sample combined with the limited percentage of excluded patients (n=23, 0.9%). The real-time registration of patients during admission and follow-up prevented selection bias or observational bias.

There was, however, a difference in baseline characteristics on cognitive dysfunction compared to the cohorts described by Maxwell et al and Moppett et al.^5,11 In the participating study centers, all patients were assessed by a geriatrician at admission to distinguish cognitive dysfunction from frequently present delirium.^22–25 In these centers, it is not standard to perform such cognitive tests as the AMTS at admission, since these tests overestimate the prevalence of cognitive dysfunction when delirium is present. Assessment by a geriatrician to detect cognitive dysfunction is thus at least as reliable as the AMTS. This explains the difference in prevalence of cognitive dysfunction between our study and those of Maxwell et al and Moppett et al (24% versus 33%–45%, respectively).^5,11 These studies used an AMTS ≤6 as a cutoff for cognitive dysfunction, but when using this cutoff, mild cognitive dysfunction or delirium can be detected.^5,11 Cognitive dysfunction, as expected in dementia, is detected in cases of an AMTS ≤8.²⁶ This might also be the explanation for the difference in mortality in the higher NHFS found in our cohort. When patients in our cohort had a higher NHFS and score on cognitive dysfunction, they were likely to be frailer than those who had an AMTS ≤6, since sincere cognitive dysfunction was present.

Conclusion

This study demonstrates that the NHFS is an accurate predictor of 30-day mortality for adult Dutch patients with a hip fracture. The NHFS is based on a limited number of objective criteria which makes it manageable in daily practice. However, clinically relevant variables, such as fracture type and mobility status, are not accounted for in the NHFS in its current form. This knowledge, combined with the lack of accuracy for higher NHFS values, suggests a need to develop a more precise tool to predict 30-day mortality in patients with a hip fracture.