Use of risk-adjusted CUSUM charts to monitor 30-day mortality in Danish hospitals

Introduction

Monitoring hospital outcomes and clinical processes has become an essential part of modern health care.^1–5 Hospital-wide mortality measures may be monitored as an indicator of the clinical quality. Compared with metrics based on a single diagnosis, hospital-wide mortality can potentially uncover cross-cutting concerns such as infection control, handoff of patient care between shifts, and care coordination. However, only a proportion of hospital-related deaths will be preventable and changes in hospital mortality are in general weakly associated with hospital performance and offer no explanation of the underlying cause.¹ Nonetheless, monitoring hospital mortality using risk-adjusted cumulative sum (CUSUM) charts^6,7 may still provide valuable insight at a low cost for the hospital. When a risk-adjusted CUSUM chart signals significant changes in hospital mortality, additional descriptive analyses can be used to compare mortality in patient subgroups before and after the alarm, which can be used to guide further in-depth analyses directed specifically at the relevant time period and patient group.

The risk-adjusted CUSUM chart is a sequential monitoring scheme, which excels at detecting small persistent changes in a process parameter of interest, while adjusting for case-mix.⁸ The method has been implemented in many different medical settings, eg, to monitor the performance of surgeons and general practitioners, to analyze surgery learning curves, and to monitor in-hospital outcomes.^2,6,9 The risk-adjusted CUSUM chart is a well-established method,^6,7,10,11 but the literature lacks detailed examples on how the method has been implemented in practice in hospital settings. Here, we describe how risk-adjusted CUSUM charts were implemented in Denmark to monitor 30-day mortality after admission at a public hospital, using 2015 as an example. In addition, we provide an example on how an alarm of increased hospital mortality from the charts could guide further in-depth analyses at the hospital.

Materials and methods

Materials

In this study, we used routinely collected data from Danish population-based nationwide registries. Denmark has ~5.6 million citizens.¹² The Danish National Health Service provides universal tax-supported health care, guaranteeing unfettered access to general practitioners and hospitals for all residents.¹³ Unambiguous linkage of all Danish registries at individual level is possible using the unique 10-digit Central Personal Register number assigned to each Danish citizen at birth and to residents upon immigration.¹² Each month, we obtained data from the Danish National Patient Registry (DNPR)¹⁴ and the Danish Civil Registration System (DCRS)¹² to get information on all inpatient hospital admissions.

Danish National Patient Registry

The DNPR is a nationwide administrative registry that has collected data from all Danish hospitals since 1977 with complete nationwide coverage since 1978.¹⁴ In 1995, the DNPR was expanded to cover outpatient, emergency room, and psychiatric hospital contacts, and in 1999, hospital examinations and treatments were included. Reporting to the DNPR at least once a month is required by law, but in practice, most hospitals do so weekly or, in some cases, daily. We obtained data on type of admission (acute/elective), hospital of admission, admission and discharge dates, and primary and secondary diagnoses as recorded at discharge (coded according to the International Classification of Diseases [ICD]-10 since 1994) from the DNPR. The primary diagnosis was categorized into groups determined by the first three digits of the ICD-10 code. If a patient was discharged and readmitted to another hospital/department within 24 hours, the readmission was interpreted as a hospital/department transfer and the two admissions were combined into one. Similarly, if the patient was discharged and readmitted to the same hospital department in <24 hours, the two admissions were combined into one. All public hospitals in Denmark were grouped into 24 units corresponding to administrative hospital units in Denmark. We ascribed admissions to the last hospital unit of the combined admission if the combined stay was <30 days and to the hospital unit where the patient was hospitalized at day 30, in case of hospitalizations of ≥30 days. We classified comorbidity according to the Charlson Comorbidity Index¹⁵ (CCI) using data from all public hospital admissions since 1977 (visits at emergency departments excluded). The CCI was categorized into the following three groups: 0 (no comorbidity), 1–2 (medium), and 3+ (high). Denmark consists of five geographical regions. An interregional transferal was defined as an admission with a transfer between hospital units in different regions during the first 30 days of admission.

Danish Civil Registration System

The DCRS is a nationwide administrative registry established in 1968 that includes individual-level information on all persons residing in Denmark. We obtained data on vital status, date of birth, sex, and marital status from the DCRS. Information on vital status is updated on a daily basis, and the data are virtually complete and have high accuracy.¹² We determined vital status 30 days after hospital admission, and both in and out of hospital deaths were considered. Age at hospital admission was categorized into 10-year age groups, with patients older than 90 years concatenated into one category. We defined marital status at admission as married, never married, divorced, or widowed. If marital status was unknown at the time of admission, we excluded the admission (~10 admissions nationwide each month).

Implementation of risk-adjusted CUSUM charts

The risk-adjusted CUSUM chart plots the function:

where W_t is a weight assigned to each value of t. In this study, the risk-adjusted CUSUM charts were updated on a patient-to-patient basis, ie, each value of t corresponds to a new admitted patient. Consequently, the weights W_t are given by

Here, Y_t is the outcome of patient t (death within 30 days after admission yes/no) and pt is the expected probability of the outcome estimated from a prediction model based on data from a reference period. Finally, RA > 1 is a specified OR increase in the outcome rate, as compared to the reference period, that the risk-adjusted CUSUM chart is set to detect. The weight W_t is positive if the patient has the outcome, and negative if not, and the absolute value of the weight is large if the outcome is unexpected. Thus, in our study, if more patients died than predicted, the CUSUM function would increase. If the CUSUM function crosses a specified control limit h, sufficient evidence has accumulated that the outcome rate has changed by an OR of at least RA as compared to the outcome rate in the reference period. In this study, the control limits were determined using simulations so that false alarms of increased 30-day hospital mortality should only happen once every 25 years, on average, for each hospital (Supplementary materials). We used RA = 1.5 in this study. This choice was motivated as a compromise between the desire of being able to detect clinically relevant increases in 30-day mortality, at the same time avoiding that the CUSUM charts would signal an alarm in case of small temporary changes in the mortality rate. After the control limits had been determined, we did further simulations to estimate the average time it would take the risk-adjusted CUSUM charts to signal an alarm when the mortality rate had increased by an OR of RA = 1.5 (Supplementary materials). Table S1 illustrates the CUSUM method calculations using a dummy data set.

Factors such as diagnosis coding and referral patterns are ever-changing in hospitals. To reduce the potential problem that predictions made from past data would become inappropriate due to changes in these factors, we used a recent period, 2014, as the reference period from which we used hospital admissions to build our prediction model. Using all inpatient hospital admissions to the 24 hospital units in 2014, we determined the number of admissions leading to death within 30 days of admission for each primary diagnosis (three-digit ICD-10 codes) and ranked them according to the number of deaths. The diagnoses with the largest number of deaths accounting for ~80% of all deaths were determined, and only admissions with one of these diagnoses were used to build the prediction model and included in the risk-adjusted CUSUM charts in 2015.³ In the process of identifying these diagnoses, we excluded hospital admissions with ICD-10 codes Z99 (no specified cause of death) and Z38 (live births). We will henceforth refer to admissions with one of these diagnoses as “CUSUM eligible admissions”.

To estimate the risk of 30-day mortality for CUSUM eligible hospital admissions in 2015, we used a logistic regression model based on the CUSUM eligible hospital admissions from 2014 including the following covariates: hospital code, admission type (acute/elective), primary diagnosis (three-digit ICD-10 code), quarter of admission, interregional transfer during the first 30 days of hospitalization (yes/no), CCI score (0, 1–2, and 3+), sex, age group (10-year age groups, 90+ in one category), and marital status (single, divorced, married, and widow). The prediction model was validated on all CUSUM eligible hospital admissions from 2013. The discriminative ability of the model was assessed using the c-statistic, and calibration of the prediction model was assessed by the calibration-in-the-large and the calibration slope from the framework as originally proposed by Cox.^16–18

When using a risk-adjusted CUSUM chart, a starting point X₀ of the CUSUM function has to be chosen (the chart needs to be initialized). Using the starting point X₀ = 0 implicitly assumes that the process being monitored is under control when the monitoring begins, ie, that the outcome rate at the beginning of the monitoring period corresponds to the rate in the chosen reference period. It was uncertain if this was a reasonable assumption in our study. Consequently, we used an alternative approach when initializing the risk-adjusted CUSUM charts; to initialize a risk-adjusted CUSUM chart for a hospital on January 1, 2015, the CUSUM function was initialized at X₀ = 0 on January 1, 2014, and then ran through the hospital-specific CUSUM eligible admissions from that year. The value of the CUSUM function on December 31, 2014, was then used as the initialization value of the risk-adjusted CUSUM chart on January 1, 2015.

While all risk-adjusted CUSUM functions were updated on a patient-to-patient basis, plotting a CUSUM function graphically on a patient-to-patient basis was judged infeasible due to the high number of daily inpatient admissions to the hospitals. Consequently, we only plotted the value of the risk-adjusted CUSUM function after the last admission each day. The risk-adjusted CUSUM charts were distributed to hospitals once each calendar month, with ~3 months delay. The 3-month delay allowed time for the outcomes to be observed and to acquire and analyze the data. When a risk-adjusted CUSUM chart signaled an alarm of increased mortality in a hospital, we conducted a descriptive analysis using data from the period giving rise to the alarm. For comparison, we also did a corresponding descriptive analysis in an earlier period with no alarms. In the descriptive analyses, we tabulated patient characteristics and listed the diagnosis groups with the highest differences between observed and expected mortality and compared the results for the two periods. Depending on the results, additional subgroup analyses were performed. The descriptive analyses were conducted in collaboration with hospital representatives. Further in-depth analysis, such as medical record reviews, was decided exclusively by the respective hospital and performed without any obligation to report their findings. The risk-adjusted CUSUM chart was not reset after the alarm (or after the analysis of the alarm).

All data management and statistical analyses were performed using the SAS software, Version 9.2, with the exception of control limit simulations, which were performed on a dedicated computer using the SAS software, Version 9.4.

According to Danish legislation, no approvals from Institutional Review Board or ethics committees are needed for purely registry-based research. The study was approved by the Danish Data Protection Agency (Jr Number: AU 392 2007-41-0357).

Results

Among 1,091,660 admissions to the 24 hospital units in 2014, 77 diagnoses accounted for ~80% of all deaths within 30 days of hospital admission (Table S2). There were 437,647 admissions in 2014 with one of the 77 included diagnoses as the primary diagnosis, and we used these admissions to build the prediction model. Of the 1,090,461 admissions in 2013, 425,697 admissions had one of the 77 selected diagnoses as the primary diagnosis and we used these admissions to validate the prediction model. Among 1,085,576 admissions in 2015, 441,352 were CUSUM eligible admissions and were included in the risk-adjusted CUSUM charts. Patient characteristics of CUSUM eligible admissions for each year are shown in Table S3.

On the validation data, the prediction model had a c-statistic of 0.846. The calibration-in-the-large was 0.005 (95% CI, -0.025;0.035), and the calibration slope was 1.029 (95% CI, 1.016;1.042).

The size of the hospital units varied from 3,396 (of the 7,382 annual admissions) to 38,940 (of the 98,188 annual admissions) CUSUM eligible admissions in 2015. When considering only CUSUM eligible admissions, the crude 30-day mortality varied between 3.60% and 7.54% in 2015 among hospital units. Risk-adjusted CUSUM control charts for all the hospital units covering all of 2015 are shown in Figure 1. In 2015, the risk-adjusted CUSUM charts yielded a total of eight alarms of increased mortality. The median of the hospitals’ estimated average time to detect a 50% increase in 30-day mortality was 50 days (interquartile interval, 43;54), with a minimum and maximum of 40 and 92 days, respectively.

Figure 1 Risk-adjusted CUSUM charts for all 24 hospital units (A-X) in 2015. Notes: The horizontal line is the control limit. The x-axis is the date, and the plotted values of the chart are the values of the risk-adjusted CUSUM function after the last admission each day. Abbreviation: CUSUM, cumulative sum.

Management of an alarm—an example

In late February 2015, the risk-adjusted CUSUM chart for hospital unit H signaled an increase in 30-day mortality (Figure 1). The alarm was analyzed in June when data from March became available. The chart signaled in February, and after the signal, additional evidence kept accumulating for about a month. Afterward, the chart quickly dropped below the control limit, indicating that the increase in the mortality rate was temporary and followed by a period with a decrease in the mortality rate as compared to the mortality rate in 2014. For the initial descriptive analysis, we compared two periods of admission data from the hospital. We identified the period February 10 to March 7, 2015, as the period with possible performance problems, corresponding to the period when the CUSUM function was rapidly increasing. The CUSUM function seemed stable in the period before this, indicating a mortality rate comparable to that in 2014, and thus, we used the period January 13 to February 7, 2015, as the comparison period for the descriptive analysis, a period with the same distribution of weekdays. The observed and expected 30-day mortality risks among CUSUM eligible admissions in the first period were 8.11% and 6.74%, respectively, and those in the second period were 9.64% and 7.36%, respectively, showing that there was an overall increase in both observed and expected 30-day mortalities in the period up to the time of the alarm. In both time periods, we noticed excess mortality among the specific diagnoses “pneumonia, unspecified organism” (J18), “other chronic obstructive pulmonary disease (COPD)” (J44), and “fracture of femur” (S72) (Table 1). We interpreted this as an indication of a general quality problem among these patient groups. Later, in 2016, other measures of clinical quality similarly indicated that 30-day mortality following hip fracture surgery and 30-day mortality following a COPD diagnosis were higher than in other hospitals in 2015 and actions such as medical record reviews were undertaken.

Table 1 Descriptive analysis of the alarm from hospital unit H Notes: Top 10 diagnoses with increased mortality in the following two different periods: a time period where the mortality rate caused the risk-adjusted CUSUM chart to signal an alarm (February 10, 2015 to March 7, 2015) and a previous time period (January 13, 2015 to February 7, 2015) where the mortality rate did not result in an alarm. Abbreviation: CUSUM, cumulative sum.

Discussion

Routinely collected medical data can be used to create risk-adjusted CUSUM charts monitoring hospital mortality. Although an alarm from the risk-adjusted CUSUM chart does not explain the cause of the change in the mortality rate in itself, it allows the identification of a time interval in which the change occurred, which together with descriptive analyses may guide further in-depth analyses at the hospital. Under optimal conditions where the risk-adjusted CUSUM charts are updated as soon as 30-day mortality can be assessed, the presented implementation of risk-adjusted CUSUM charts in this study can detect significant increases in 30-day mortality within 2 months, on average, in most Danish hospitals. Together with descriptive analyses, it was possible to use an alarm from a risk-adjusted CUSUM chart to identify potential performance problems well in advance of other already implemented monitoring systems measuring selected clinical indicators.

In a previous study describing the implementation of risk-adjusted CUSUM charts to monitor hospital-wide outcomes in England,² one of the key strengths of the implementation was that it incorporated an interactive presentation of the results via a web-based front end, which enabled users to monitor the CUSUM charts and analyze alarms themselves. A limitation of our implementation was that no such tool was available to the users and instead they had to rely on collaboration with the provider to obtain information on alarms related to their hospital. In connection with this, another limitation of our implementation was that the risk-adjusted CUSUM charts were only distributed on a monthly basis and with delay (in addition to the 30 days, it took to assess the 30-day mortality outcome). Using only the hospital’s own data to predict 30-day mortality would ensure that differences in diagnosis coding and so on between hospitals would not influence the predictions. As such, it is a limitation in our implementation that we built a single combined prediction model using data from all the different hospitals, instead of separate prediction models for each hospital, since this result in predictions being based on an “average” hospital, not the hospital itself. In contrast, having a combined prediction model also had its advantages, since it eliminated sparse data problems, and facilitated the practical implementation. The prediction model was judged to be well calibrated and have good discrimination. As such, we expect that the prediction model performed well on hospitals with “average” hospital coding, referral patterns, and so on, but we cannot rule out that predictions from hospitals with differences in these factors might be inappropriate. A strength of our implementation was that it used routinely collected administrative data from Danish population-based registries. Information on vital status from the DCRS is updated on a daily basis, the data are virtually complete and have high accuracy, and many diagnosis codes from the DNPR have been shown to have acceptable predictive values.^12,14

Commonly, clinical indicators on hospital-wide mortality are reported quarterly or yearly.^3,19,20 Our choice of implementation of risk-adjusted CUSUM charts made it possible to detect significant increases in 30-day mortality in ~2 months, on average, for most hospitals in Denmark. As shown in our example, the information on an alarm from a risk-adjusted CUSUM chart also has the potential of being more informative than traditional clinical indications of an increase in hospital-wide mortality over a long time period, since the descriptive analyses might reveal that the increased mortality is concentrated on specific subgroup of patients in a specific time period.

Monitoring decreases in hospital mortality may be important as well, since a decrease in hospital-wide mortality can be an indicator of improvements in hospital performance. For simplicity, we have focused on the implementation of risk-adjusted CUSUM charts to monitor increases in 30-day mortality in this study, but separate risk-adjusted CUSUM charts monitoring decreases in 30-day mortality were also implemented.⁷

A nonstandard method was used to initialize the risk-adjusted CUSUM charts. A more conventional method to initialize a risk-adjusted CUSUM chart would be to initiate the charts halfway between zero and the chosen control limit or to simply initiate the chart at zero.⁶ The method we used was purely heuristic, and using the more conventional methods may be considered in other settings. We also chose not to reset the charts after an alarm because of the time lag between the actual time of the alarm and the time it was discovered and analyzed. Resetting risk-adjusted CUSUM charts makes more sense in the traditional use of the CUSUM methods where an alarm is investigated with little or no time lag.

The importance of different aspects of the implementation of risk-adjusted CUSUM charts depends on the monitoring period. In our implementation, the decision not to reset the risk-adjusted CUSUM charts after an alarm and the method with which we initialized the charts were influenced by the fact that we planned to update the prediction model and reset the charts each January. When the monitoring period is short, proper initialization is important, or alarms in the start of the monitoring period might be delayed or missed entirely. In contrast, resetting the charts after an alarm is not that important in our implementation, since large increases in hospital mortality are likely to be temporary. When the monitoring period is long, the method with which the charts are initialized are probably of little importance, but resetting the charts after an alarm might be appropriate, if the aim is to detect a lower increase in mortality.

Conclusion

The presented implementation of risk-adjusted CUSUM charts has the potential to detect significant increases in 30-day mortality within 2 months, on average, in most Danish hospitals. Together with descriptive analyses, alarms from risk-adjusted CUSUM charts may identify potential performance problems in advance of other previously implemented clinical quality indicators.

Acknowledgments

We thank Anders Kjærsgaard and Johnny Kahlert for helpful feedback on an earlier draft of the article. This article was funded by the Program for Clinical Research Infrastructure (PROCRIN) established by the Lundbeck Foundation and the Novo Nordisk Foundation and administered by the Danish Regions.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet. 2004;363(9415):1147–1154.
2.		Bottle A, Aylin P. Intelligent information: a national system for monitoring clinical performance. Health Serv Res. 2008;43(1):10–31.
3.		Jarman B, Gault S, Alves B, et al. Explaining differences in English hospital death rates using routinely collected data. BMJ. 1999;318(7197):1515–1520.
4.		Heijink R, Koolman X, Pieter D, van der Veen A, Jarman B, Westert G. Measuring and explaining mortality in Dutch hospitals: the hospital standardized mortality rate between 2003 and 2005. BMC Health Serv Res. 2008;8:73.
5.		Miyata H, Hashimoto H, Horiguchi H, Matsuda S, Motomura N, Takamoto S. Performance of in-hospital mortality prediction models for acute hospitalization: hospital Standardized Mortality Ratio in Japan. BMC Health Serv Res. 2008;8:229.
6.		Grigg OA, Farewell VT, Spiegelhalter DJ. Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Methods Med Res. 2003;12(2):147–170.
7.		Steiner SH, Cook RJ, Farewell VT, Treasure T. Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics. 2000;1(4):441–452.
8.		Montgomery DC. Introduction to Statistical Quality Control. 6th ed. Hoboken: John Wiley & Sons; 2009.
9.		Wang MJ, Meng LW, Cai YQ, et al. Learning curve for laparoscopic pancreaticoduodenectomy: a CUSUM analysis. J Gastrointest Surg. 2016;20(5):924–935.
10.		Page ES. Continuous inspection schemes. Biometrika. 1954;41(1–2):100–115.
11.		Moustakides GV. Optimal stopping-times for detecting changes in distributions. Ann Stat. 1986;14(4):1379–1387.
12.		Schmidt M, Pedersen L, Sorensen HT. The Danish Civil Registration System as a tool in epidemiology. Eur J Epidemiol. 2014;29(8):541–549.
13.		Nordic Medico-Statistical Committee. Health Statistics for the Nordic Countries. København: Nordic Medico-Statistical Committee; 2015.
14.		Schmidt M, Schmidt SA, Sandegaard JL, Ehrenstein V, Pedersen L, Sorensen HT. The Danish National Patient Registry: a review of content, data quality, and research potential. Clin Epidemiol. 2015;7:449–490.
15.		Charlson ME, Pompei P, Ales KL, Mackenzie CR. A new method of classifying prognostic co-morbidity in longitudinal-studies—development and validation. J Chronic Dis. 1987;40(5):373–383.
16.		Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–138.
17.		Miller ME, Langefeld CD, Tierney WM, Hui SL, McDonald CJ. Validation of probabilistic predictions. Med Decis Making. 1993;13(1):49–58.
18.		Cox DR. Two further applications of a model for binary regression. Biometrika. 1958;45:562–565.
19.		Information Services Division NHS. Hospital Standardised Mortaliy Ratio Quarterly Release. Edinburgh: Information Services Division NHS; 2017.
20.		Canadian Institute for Health Information. HSMR: A New Approach for Measuring Hospital Mortality Trends in Canada. Ottawa: Canadian Institute for Health Information; 2007.

Supplementary materials

Simulation of control limits

For each risk-adjusted CUSUM chart, we determined control limits from simulations corresponding to an average run length under control (ARL₀) of 25 years, ie, we determined control limits so that each hospital should only get a false alarm every 25 years, on average. After the control limit corresponding to the desired ARL₀ had been determined for a hospital, we used that control limit to simulate the corresponding average run length when the process was not in control (ARL_A), ie, the average time it would take for the risk-adjusted CUSUM chart to signal an alarm when the mortality rate had increased with an OR of R_A = 1.5.

Estimation of the control limit for each hospital corresponding to an ARL₀ of 25 years, and the corresponding ARL_A, was done using the CUSUM eligible admission data from 2014, restricted to admissions assigned to that specific hospital. Since the risk-adjusted CUSUM charts in our study is updated on a patient-to-patient basis, ARLs are not naturally expressed in terms of time but in number of admitted patients. Therefore, the hospital-specific admissions were used to estimate the average number of daily CUSUM eligible admissions in each hospital and we used this as the number of simulated admissions that corresponded to a time period of 1 day.

We performed simulations of ARLs using the approach described in Figure S1.¹ Patient covariate patterns were drawn at random, with replacement, from the specific hospital’s CUSUM eligible admissions from 2014 in step 1. When estimating the ARL₀, the monitored process is assumed to be under control and estimation of 30-day mortality in step 2 can be done using the prediction model. We simulated 500 years of hospital admissions for each hospital in step 4 to ensure that the risk-adjusted CUSUM chart would signal an alarm before running out of simulated data, regardless of the value of the control limit that was considered (all values from 3.0 to 9.0 with 0.1 steps). We simulated the 500 years of data 1,000 times in step 5. The control limit corresponding to an ARL₀ of 25 years was then estimated by linear interpolation of the two control limits corresponding to ARL₀ values just >25 and <25 years (step 6). Having estimated a control limit corresponding to the desired ARL₀, we repeated steps 1–5 in the simulation scheme, using only that specific control limit, to estimate the corresponding ARLA. When estimating the ARLA, the monitored process is assumed to be out of control, and risk estimation of 30-day mortality in step 2 can be modified using the following formula:¹

where p_t is the estimated probability of 30-day mortality using the prediction model.

Figure S1 Simulation scheme for estimating average run lengths. Abbreviation: CUSUM, cumulative sum.

In real life, 30-day mortality cannot be assessed before 30 days have passed. This unavoidable delay can potentially make the estimate of the ARL_A misleading. Because of this, we added 30 days to the estimated ARL_A to get an estimate of the average time it would take a CUSUM chart to detect an increase in mortality under optimal conditions in a real-life setting where data can be acquired and processed immediately when available.

Table S1 Illustration of the CUSUM method calculations using a dummy data set Note: In this example, the CUSUM chart has been initialized at X0 = 0. Abbreviation: CUSUM, cumulative sum.

Table S2 Primary hospital diagnoses in 2014 with the highest number of deaths Abbreviations: CKD, chronic kidney disease; NSTEMI, non-ST elevation myocardial infarction; STEMI, ST elevation myocardial infarction.

Table S3 Patient characteristics of all CUSUM eligible admissions in 2013 (used to validate the prediction model), 2014 (used to build the prediction model), and 2015 (used in the risk-adjusted CUSUM charts) Abbreviation: CUSUM, cumulative sum.

Reference

1.

Webster RA, Pettitt AN. Stability of approximations of average run length of risk-adjusted CUSUM schemes using the Markov approach: comparing two methods of calculating transition probabilities. Commun Stat. 2007;36(3):471–482.