Validating administrative data for the detection of adverse events in older hospitalized patients

Introduction

Every year, millions of older adults (≥65 years) receive safe and effective health care, yet there is clear evidence that some patients experience unintended harm from their care.^1–3 Older hospitalized patients have a particularly high risk of experiencing adverse events. The elevated risk has been attributed to a number of factors, including the increased likelihood of more complicated illness and complex interventions, the presence of co-morbid conditions, and higher rates of hospitalization.^1,2,4 An adverse event has been defined as an unintended injury or complication that is caused by health care management, rather than by the patient’s underlying disease.³ Adverse drug events, fall-related injuries, and pressure ulcers are examples of adverse events that older patients may experience while in hospital.^1,2,4

Medications are the most common and cost-effective therapeutic intervention that older adults will obtain when they seek health care.⁵ It is estimated that more than 80% of older adults take at least one medication per day,⁶ and close to 40% take five or more per day, placing them at an increased risk of developing a drug-related problem.^6,7 It has been reported that hospitalized older adults receive an average of eight or nine different medications during their hospital stay, including many “routine” medications, such as sedatives to promote sleep.^8,9 Adverse drug events have been consistently identified in patient safety studies involving acute care.^1–3,10,11 An adverse drug event is an injury that is caused by a drug, and can include injuries that result if an indicated drug is not taken as prescribed.^12,13 Evidence from a report commissioned by the American Association of Retired Persons (AARP) to synthesize the literature on “medical injury” in older adults suggests that drug complications are two-and-a-half times more common in older hospitalized patients than in those less than 65 years of age.¹ Certain drugs have also been associated with an increased risk of falls in this population.¹⁴

Unintentional falls comprise a significant proportion of injury-related morbidity and mortality in older adults.^15,16 Approximately one-third of older adults will fall each year; of those, one-half will sustain a minor injury and up to 25% will sustain a more serious injury.^14,16 Fall-related injuries can accelerate functional decline and nursing home admission.^17,18 Hospitalization poses unique risks of falling in older patients. In addition to the consequences of immobilization, the unfamiliar environment, factors arising from the acute illness, and the effects of treatment may all place older hospitalized patients at higher risk of a fall.^4,19 Rates cited in the literature for in-hospital falls for patients of all ages range from 2.2 to 7/1,000 bed days in acute care hospitals.^19,20 However, it has been reported that older patients are nine times more likely to fall in hospital than younger patients.¹

Pressure ulcers are another common problem in patients admitted to hospital. The incidence of pressure ulcers has been reported in the range of 0.4% to 38% in acute care, with up to 60% of new pressure ulcers developing while a person is in hospital.^21,22 Older patients are a high-risk group for developing pressure ulcers, due in part to age-related changes in skin integrity and mobility limitations.⁴ The presence of comorbid conditions such as peripheral vascular disease or nutritional deficiencies exacerbates the effects of sustained pressure from immobility to increase the risk of developing pressure ulcers.^23,24 Within hours of immobilization, some patients are at risk of developing pressure ulcers.⁴ In addition, their presence may adversely affect functional status, which, in turn, affects discharge disposition of the patient.²¹ Thus, on an organizational level, some adverse events may have an unintended impact on bed availability and patient flow.

Methods to detect adverse events

The first step in understanding the impact of adverse events and how to prevent them is the development and validation of methods to identify when they have occurred. However, one of the challenges in a plan to monitor and manage adverse events in an organization is the lack of readily accessible information on their occurrence. The presence of an adverse event may be documented on individual health records, but data abstraction from this source is resource intensive.²⁵

Administrative data sources have been used to develop patient safety indicators and performance measures, as well as to enhance surveillance capability for adverse events.^26–31 Using a public health approach to the problem, US investigators from Wisconsin developed and validated surveillance criteria for injuries related to medical and surgical care for use with administrative data.²⁸ The Wisconsin surveillance criteria differ from other measures of adverse events in that they focus on harm to a patient as identified by an injury coded in the administrative data. Early studies of adverse events focused on medical error and negligence.^10,11 Unlike some indicators and performance measures, the Wisconsin surveillance criteria are not limited to subsets of hospitalized patients. Their classification scheme uses a combination of diagnosis and external-cause-of-injury codes from the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) to identify adverse events or harm to patients. The diagnosis code describes the nature of the problem (eg, rash) and the external-cause codes identify the mechanism (eg, adverse effects of antibiotics in therapeutic use).²⁸ A validation study examined the performance characteristics of the surveillance criteria when compared with the “gold standard” of the major epidemiologic studies in patient safety – the health record review. The surveillance criteria had an overall sensitivity of 59.9% (95% confidence interval [CI]: 42.8–75.0) and a specificity of 97.4% (95% CI: 94.1–98.8).²⁸ The Wisconsin surveillance criteria have not been validated using the tenth revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10).³²

Use of routinely collected diagnostic data has been underutilized as an ongoing decision-support resource despite its advantages over other methods for identifying adverse events: it is less costly; it captures new information on all patients admitted to hospital without having to rely on active reporting by health care professionals; the data are readily available in national and international jurisdictions; and the approach to event-finding can complement what is known from other sources and thus provide a more comprehensive view of the problem. The need for robust monitoring is increasingly important, as the relatively at-risk aging population seeking health care is rapidly growing. There is a research gap in using administrative diagnostic data specifically to inform patient safety improvements in the care of older patients.

Study objective

The objective of this retrospective cross-sectional study was to validate diagnostic codes for pressure ulcers, fall-related injuries, and adverse drug events found in routinely collected hospitalization data.

Methods

Setting

The validation study was conducted at the Queen Elizabeth II Health Sciences Centre (QEII HSC) in Halifax, Nova Scotia, Canada. The QEII HSC is the largest teaching hospital and academic health sciences center in Atlantic Canada, with approximately 1,100 beds and 31,000 admissions per year.

Subject selection and data source

Patients (≥65 years) discharged between April 1, 2009 and March 31, 2011 from the QEII HSC were eligible for inclusion in the validation study. Patients who spent any time in non-acute care during their hospital stay were excluded (eg, patients waiting for a nursing home bed) because it was anticipated that these patients would have multiple comorbidities and that there would be other factors contributing to the risk of adverse events. For patients with more than one hospitalization during the study period, only the first hospitalization was included. For each of the three types of adverse events, a random sample of 50 patients whose records were positive and 50 patients whose records were not positive for an adverse event was sought for review in the validation study (n=300 records in total). In Canada, the national standards for assigning diagnostic codes in administrative data are such that conditions are only coded if they have an impact on length of stay or management in hospital.³³ Coding standards may vary in different countries.

Adverse events were identified by the diagnostic codes selected from ICD, Tenth Revision, Canada (ICD-10-CA) and found in the administrative hospitalization Discharge Abstract Database. The diagnostic codes for adverse drug events were derived from the Wisconsin surveillance criteria, as these have been previously used in our organization and they have strong performance characteristics compared with health record review.^28,34 To determine diagnostic codes for pressure ulcers and fall-related injuries, the multidisciplinary research team systematically examined published sources of codes for these adverse events using explicit criteria established a priori (eg, yield, availability, coding rules, etc) to select candidate codes.^26–28,35 Table S1 describes the list of diagnostic codes used in the validation study. The diagnosis type was also examined for those records in which adverse events were coded. The diagnosis type is used in the Discharge Abstract Database to identify preexisting conditions present at the time of admission (Type 1) and those that develop during the hospital admission (Type 2).³⁶ Diagnosis types are critical for identifying harm to patients that occurs during hospitalization.

The adverse events identified by the administrative Discharge Abstract Database were compared with those identified by the gold standard of a structured health record review.^3,10,11,28 Two health care providers (a nurse practitioner and a pharmacist) experienced in geriatrics independently reviewed each patient record for the occurrence of an adverse event without any knowledge of the coding in the administrative data. The reviewers used a standard data abstraction form and a structured review process.³⁷ There was a brief education session about the review process and more detailed written instructions were provided to ensure that each reviewer went through a uniform process. The written instructions included items such as confirming the patient name, correct hospital admission, and the order in which different parts of the record were to be reviewed. Reviewers were asked to identify if there was documentation about the presence of a pressure ulcer, fall-related injury, and/or adverse drug event. Reviewers were asked to note the location in the health record where this information was documented. The form also included a section for free text where a reviewer could identify documentation that indicated a possible adverse event. The reviewers were not asked to assess potential preventability. Using the same structured review process, a physician reviewed 20 records from each sample reviewed by the health care providers (n=40) to establish interrater agreement. The study was reviewed and approved by the Capital District Health Authority Research Ethics Board.

Data analysis

The sensitivity, specificity, and 95% CIs were calculated for each type of adverse event. For pressure ulcers and fall-related injuries, diagnostic codes were compared with the results of the health record review by the nurse practitioner. For adverse drug events, diagnostic codes were compared with results of the review by the pharmacist. The interrater agreement was calculated for each type of adverse event based on the subset of records also reviewed by the physician.

Results

The nurse practitioner identified a total of 39 pressure ulcers and 56 fall-related injuries in the 185 complete records that were reviewed. Of these, 34 pressure ulcers and 54 fall-related injuries were identified in administrative data. The pharmacist reviewer identified a total of 69 adverse drug events in the 99 complete records that were reviewed. Of these, 47 were also identified in administrative data. Table 1 describes the sensitivity and specificity of the diagnostic coding for each type of adverse event as compared with the expert health record review. The physician reviewed 20 of the health records reviewed by each of the other reviewers. Interrater agreement was 100% for pressure ulcers and 87.5% for fall-related injuries, and ranged from 81.0%–95.2% for adverse drug events, depending on whether or not events coded as “possible” adverse drug events were included in the calculation. Although we randomly selected 300 records for review, 15 (5.0% of 300) records were missing one or more key documents needed to definitively confirm the presence or absence of a documented adverse event (eg, assessment note). One of the records reviewed by the nurse practitioner and the pharmacist was considered to be incomplete for the pharmacist review as it was missing key documents for determination of an adverse drug event.

Table 1 Performance characteristics of diagnostic codes for adverse events compared with expert health record reviews Notes: aNurse practitioner review; bpharmacist review. Abbreviation: CI, confidence interval.

Five pressure ulcers and two fall-related injuries were identified by the nurse practitioner, but were not coded in administrative data. Twenty-two adverse drug events were identified by the pharmacist, but were not coded in administrative data. The adverse events may not have been coded in the administrative data for reasons related to the coding standards (ie, coding restricted to conditions that have an impact on length of stay or management in the hospital). This limitation became apparent during the case selection phase of the validation study, as there were fewer than 50 records coded for in-hospital pressure ulcers and fall-related injuries during the first fiscal year. To address this limitation, the random selection process was applied to hospitalization data for an additional fiscal year (2010–2011) to identify sufficient records with diagnostic codes indicative of a pressure ulcer or fall-related injury.

Discussion

Valid epidemiologic data are central to the identification of risk factors and development of effective prevention strategies for adverse events. In this validation study, a total of 39 pressure ulcers, 56 fall-related injuries, and 69 adverse drug events were identified through health record review. Of these, 34 pressure ulcers, 54 fall-related injuries, and 47 adverse drug events were also identified in administrative data. Overall, the diagnostic codes for adverse events had a sensitivity and specificity exceeding 0.67 (95% CI: 0.56–0.99) and 0.89 (95% CI: 0.72–0.99), respectively. Different sources of information on adverse events provide varying degrees of depth and breadth with respect to an adverse event. Use of multiple sources of information will provide a more complete picture of the nature and extent of adverse events in an organization.^38,39

One of the most common approaches to collecting information about adverse events is through spontaneous reporting using an institutional or other centralized incident-reporting system. Although reporting provides valuable detail about an event, one consistent finding is that adverse events are grossly underreported – by as much as 50%–96%.^25,38,40 The biases associated with potential underreporting make it difficult to accurately determine the organization-wide impact of those adverse events. Accordingly, there are challenges with effectively measuring the results of interventions aimed at prevention and management of adverse events across an entire organization.

Use of administrative data to support system-level monitoring of adverse events has garnered increased attention.^{26–31,41–44} The reported sensitivity and specificity of diagnostic codes in administrative data used to identify adverse events is variable. The performance characteristics of the diagnostic codes in our validation study compare favorably with those of the Wisconsin surveillance criteria. The latter have a reported overall sensitivity of 59.9% (95% CI: 42.8–75.0) and specificity of 97.4% (95% CI: 94.1–98.8). The Wisconsin surveillance criteria include both the diagnosis code and external-cause-of-injury codes. The relative utility of the codes used in the Wisconsin validation study varied depending on the type of adverse event. The external cause of injury codes identified 98.5% of adverse drug events, which suggests that this is an important type of code to include.²⁸ Other studies^41,42 have reported substantially poorer performance characteristics. The variability is partially explained by differences in study methodology, case definitions, and care setting. A systematic review of ICD-10 codes used to identify adverse drug events in administrative data found 827 ICD-10 codes for adverse drug events that had been reported in the literature. Of the 41 papers included in the review, only two reported the sensitivity of their codes and both involved emergency department data.⁴⁴ For instance, Hohl et al used a combination of prospectively collected data on adverse drug events linked with administrative data from two Canadian emergency departments. Their reported sensitivity ranged from 6.8% (95% CI: 4.0–11.2) to 28.1% (95% CI: 22.3–34.6), depending on the level of certainty that the event was attributable to a medication.⁴²

There is scant literature on the performance characteristics of diagnostic codes in administrative data for identifying pressure ulcers and fall-related injuries. Verelst et al validated diagnostic codes to screen for five adverse events using the Belgian Hospital Discharge Dataset. The adverse events were drawn from the Agency for Healthcare Research and Quality Patient Safety Indicators and used ICD-9-CM diagnostic codes. They determined that the positive predictive value for diagnostic codes to identify pressure ulcers ranged from 54.3 to 74.5 for those that were hospital-acquired and present on admission, respectively.⁴³ The positive predictive value is considerably higher in our study population (97.1 [95% CI: 83.3–1.0]). This may be related to the exclusive focus on older patients. Not only is this population at higher risk of developing a pressure ulcer, it may be that documentation is better because of the increased potential for complications in older patients. Although this study focused on three common types of adverse events in a high-risk population, the approach can be applied to patients from all age groups and in any acute care facility that collects routine administrative diagnostic data. The Wisconsin surveillance criteria have been used in pediatric and adult populations. Their criteria include validated diagnostic codes for other types of adverse events, such as those related to procedures or devices.²⁸

Administrative data have important limitations with respect to adverse event surveillance. Administrative data are generally collected for purposes other than surveillance or research, so the quality and completeness of the data may be limited. There is often insufficient clinical detail to perform rigorous risk adjustment using administrative data.^28,31,43 As described earlier, coding standards are such that conditions are only coded if they have contributed to an increased length of stay or change in management of care. In addition, coding standards typically require a documented physician diagnosis in order to be coded in administrative data. There is evidence that physician documentation about some types of adverse events (eg, pressure ulcers) is poor.^45,46 This limitation became apparent during our validation study, as there were insufficient cases of pressure ulcers and fall-related injuries identified during 1 fiscal year. In a large validation study of patient safety indicators using administrative hospitalization data, similar challenges in reaching the target sample size for some indicators were reported.⁴¹ Fifteen (5.0%) of the 300 randomly selected records were missing one or more key documents. These limitations in documentation underscore the importance of using multiple sources of data on adverse events to understand the full scope of the problem.^39,40 The use of diagnostic data from administrative sources will likely result in an underestimate of the number of adverse events, yet it is anticipated that it will yield important information using fewer resources than other methods of event-finding.

Use of administrative data for monitoring permits population-based analyses that can be used to identify risk factors, impact on hospital length of stay, disposition, and costs associated with adverse events. Administrative data also provide a feasible, relatively low-cost mechanism for evaluating the impact of prevention strategies on these same measures. These attributes offer important tools for organization-level monitoring that will augment data collected from other sources, such as incident reporting systems. Once high-risk patients and/or settings are identified using administrative data, more detailed methods such as quality reviews or case studies can be used to understand the contributing factors and system-, patient-, and/or provider-level issues so that prevention strategies can be appropriately tailored. As with all approaches to monitoring adverse events, an essential element includes one or more mechanisms by which to broadly share the information to maximize learning and the potential for making safety improvements to reduce harm to patients.

Conclusion

Patient safety has become a critical focus for health care organizations and accreditation bodies in the last decade. Prevention of adverse events is very relevant to a growing number of older patients seeking care, many of whom are especially vulnerable to the unintended harm that may occur in health care. Fall-related injuries, pressure ulcers, and adverse drug events are common and costly examples of potentially preventable adverse events in older hospitalized patients. The results of this validation study demonstrate that it is feasible and valid to identify adverse events in hospitalized patients using routinely collected administrative data. Although the approach likely underestimates the extent of the problem, the information is relatively inexpensive and easy to access, with no impact on clinical staff. Administrative data can provide a valuable tool for ongoing monitoring of the incidence and impact of adverse events as well as the identification of potentially modifiable risk factors upon which targeted prevention strategies can be developed and evaluated.

Acknowledgments

The authors would like to gratefully acknowledge Ms Lisa Dillman from Decision Support and the staff of Health Information Services at Capital District Health Authority for their assistance during the study, as well as the professionals who conducted the health record review: Ms Nikki Kelly (nurse practitioner), Mr Lucas Thorne-Humphrey (pharmacist), and Dr Jock Murray (physician). The study was funded by the Canadian Institutes of Health Research with a Pilot Projects in Aging Catalyst Grant.

Author contributions

SAS conceived and designed the study and acquired, analyzed, and interpreted data. SAS prepared the first draft of the manuscript and revised according to feedback. SB and LG contributed to the design of the study and interpretation of the data. All authors reviewed and approved the manuscript to be published and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.	Rothschild JM, Leape LL. The Nature and Extent of Medical Injury in Older Patients. Washington, DC: AARP; 2000. Available from: http://assets.aarp.org/rgcenter/health/2000_17_injury.pdf. Accessed March 12, 2014.
2.	Long SJ, Brown KF, Ames D, Vincent C. What is known about adverse events in older medical hospital inpatients? A systematic review of the literature. Int J Qual Health Care. 2013;25(5):542–554.
3.	Baker GR, Norton PG, Flintoft V, et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ. 2004;170(11):1678–1686.
4.	Creditor MC. Hazards of hospitalization of the elderly. Ann Intern Med. 1993;118:219–223.
5.	Abrams WB, Beers MH. Clinical pharmacology in an aging population. Clin Pharmacol Ther. 1998;63:281–284.
6.	Snyder L, Connolly S, Becker B. Pharmacotherapy and adverse drug-related events in elders treated in the emergency department. In: Meldon SW, Ma OJ, Woolard RH, editors. Geriatric Emergency Medicine. New York, NY: McGraw-Hill Medical Publishing Divisions; 2004; Chapter 3; pages 13–21.
7.	Gu Q, Dillon CF, Burt VL. Prescription Drug use Continues to Increase: US Prescription Drug Data for 2007–2008. NCHS Data Brief No 42. Hyattsville, MD: National Center for Health Statistics; 2010. Available from: http://www.cdc.gov/nchs/data/databriefs/db42.pdf. Accessed March 12, 2014.
8.	Gorbien MJ, Bishop J, Beers MH, Norman D, Osterweil D, Rubenstein LZ. Iatrogenic illness in hospitalized elderly people. J Am Geriatr Soc. 1992;40:1031–1042.
9.	Stolley JM, Buckwalter KC, Fjordbak B, Bush S. Iatrogenesis in the elderly. Drug-related problems. J Gerontol Nurs. 1991;17:12–17.
10.	Leape LL, Brennan TA, Laird N et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. NEJM. 1991;324(6):377–384.
11.	Thomas EJ, Studdert DM, Burstin HR et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38(3):261–271.
12.	Bates DW, Boyle DL, Vander Vliet MB, Schneider J, Leape LL. Relationship between medication errors and adverse drug events. J Gen Intern Med. 1995;10:199–205.
13.	Ackroyd-Stolarz S, Hartnell N, Mackinnon NJ. Demystifying medication safety: making sense of the terminology. Res Social Admin Pharm. 2006;2:280–289.
14.	Finlayson ML, Peterson EW. Falls, aging, and disability. Phys Med Rehabil Clin N Am. 2010;21(2):357–373.
15.	Centers for Disease Control and Prevention (CDC). Fatalities and injuries from falls among older adults – United States, 1993–2003 and 2001–2005. MMWR Morb Mortal Wkly Rep. 2006;55:1221–1224.
16.	Scott VJ, Dukeshire S, Gallagher EM, et al. A best practices guide for the prevention of falls among seniors living in the community. 2001. Ottawa, ON: Health Canada, Division of Aging and Seniors. Available from: http://www.publications.gc.ca/site/eng/103922/publication.html. Last accessed July 14, 2014.
17.	Tinetti ME, Williams CS. Falls, injuries due to falls, and the risk of admission to a nursing home. N Engl J Med. 1997;337:1279–1284.
18.	Tinetti ME, Williams CS. The effect of falls and fall injuries on functioning in community-dwelling older persons. J Gerontol A Biol Sci Med Sci. 1998;53:M112–M119.
19.	Morse JM. Enhancing the safety of hospitalization by reducing patient falls. Am J Infect Control. 2002;30(6):376–380.
20.	Hitcho EB, Krauss MJ, Birge S, et al. Characteristics and circumstances of falls in a hospital setting: a prospective analysis. J Gen Intern Med. 2004;19:732–739.
21.	Reddy M, Gill SS, Rochon PA. Preventing pressure ulcers: a systematic review. JAMA. 2006;296(8):974–984.
22.	Bennett G, Dealey C, Posnett J. The cost of pressure ulcers in the UK. Age Ageing. 2004;33(3):230–235.
23.	Mobily PR, Skemp Kelly L. Iatrogenesis in the elderly. Factors of immobility. J Gerontol Nurs. 1991;17:5–11.
24.	Skemp Kelly L, Mobily PR. Iatrogenesis in the elderly. Impaired skin integrity. J Gerontol Nurs. 1991;17:24–29.
25.	Battles JB, Lilford RJ. Organizing patient safety research to identify risks and hazards. Qual Saf Health Care. 2003;12 Suppl 2:ii2–ii7.
26.	Patient Safety Indicators – Detailed Definitions: PSI Technical Specifications, Version 3.2, 2008. Rockville, MD: Agency for Healthcare Research and Quality; 2008. Available from: http://www.qualityindicators.ahrq.gov/downloads/modules/psi/v32/psi_technical_specs_v32.pdf. Accessed July 14, 2014.
27.	Canadian Hospital Reporting Project (CHRP) [webpage on the Internet]. Ottawa, ON: Canadian Institute for Health Information. Available from: http://www.cihi.ca/CIHI-ext-portal/internet/EN/TabbedContent/health+system+performance/indicators/performance/cihi010657. Accessed February 28, 2014.
28.	Layde PM, Meurer LN, Guse CE, et al. Medical Injury Identification Using Hospital Discharge Data. Available from: http://www.ncbi.nlm.nih.gov/books/NBK20501/pdf/ch10.pdf. Accessed March 11, 2014.
29.	Stausberg J, Hasford J. Drug-related admissions and hospital-acquired adverse drug events in Germany: a longitudinal analysis from 2003 to 2007 of ICD-10-coded routine data. BMC Health Serv Res. 2011;11:134.
30.	Van den Heede K, Sermeus W, Diya L, Lesaffre E, Vleugels A. Adverse outcomes in Belgian acute hospitals: retrospective analysis of the national hospital discharge dataset. Int J Qual Health Care. 2006;18(3):211–219.
31.	Hougland P, Nebeker J, Pickard S, et al. Using ICD-9-CM Codes in Hospital Claims Data to Detect Adverse Events in Patient Safety Surveillance. Available from: http://www.ahrq.gov/downloads/pub/advances2/vol1/Advances-Hougland_26.pdf. Accessed March 11, 2014.
32.	ICD-10-CA [webpage on the Internet]. Ottawa, ON: Canadian Institute for Health Information. Available from: http://www.cihi.ca/cihi-ext-portal/internet/en/document/standards+and+data+submission/standards/classification+and+coding/codingclass_icd10. Accessed February 28, 2014.
33.	Canadian Institute for Health Information, Canadian Coding Standards for Version 2012 ICD-10-CA and CCI, Revised September 2012 (Ottawa, Ontario: CIHI, 2012). Available from: https://secure.cihi.ca/free_products/canadian_coding_standards_2012_e.pdf. Accessed July 15, 2014.
34.	Ackroyd-Stolarz S, Read Guernsey J, Mackinnon NJ, Kovacs G. The association between a prolonged stay in the emergency department and adverse events in older patients admitted to hospital: a retrospective cohort study. BMJ Qual Saf. 2011;20(7):564–569.
35.	Hospital e-Scorecard Report 2008: Acute Care – Clinical Utilization and outcomes technical summary. Toronto, ON: Health System Performance Research Network. Available from: www.hsprn.ca/?p=32. Accessed March 14, 2014.
36.	Appendix C: diagnosis typing. In: Canadian Hospital Reporting Project Technical Notes – Clinical Indicators. Ottawa, ON: Canadian Institute for Health Information; 2013:89. Available from: http://www.cihi.ca/CIHI-ext-portal/pdf/internet/CHRP_TNCI_PDF_EN. Accessed February 28, 2014.
37.	Worster A, Haines T. Advanced statistics: understanding medical record review (MRR) studies. Acad Emerg Med. 2004;11(2):187–192.
38.	Barach P, Small SD. Reporting and preventing medical mishaps: lessons from non-medical near miss reporting systems. BMJ. 2000;320:759–763.
39.	Naessens JM, Campbell CR, Huddleston JM, et al. A comparison of hospital adverse events identified by three widely used detection methods. Int J Qual Health Care. 2009;21(4):301–307.
40.	Hogan H, Olsen S, Scobie S, et al. What can we learn about patient safety from information sources within an acute hospital: a step on the ladder of integrated risk management? Qual Saf Health Care. 2008;17:209–215.
41.	Quan H, Eastwood C, Cunningham CT, et al; IMECCHI investigators. Validity of AHRQ patient safety indicators derived from ICD-1- hospital discharge abstract data (chart review study). BMJ Open. 2013;3(10):e003716.
42.	Hohl CM, Kuramoto L, Yu E, Rogula B, Stausberg J, Sobolev B. Evaluating adverse drug event reporting in administrative data from emergency departments: a validation study. BMC Health Serv Res. 2013;13:473.
43.	Verelst S, Jacques J, Van den Heede K, et al. Validation of Hospital Administrative Dataset for adverse event screening. Qual Saf Health Care. 2010;19:e25.
44.	Hohl CM, Karpov A, Reddekopp L, Stausberg J. ICD-10 codes used to identify adverse drug events in administrative data: a systematic review. J Am Med Inform Assoc. 2014;21:547–557.
45.	Thoroddsen A, Sigurjónsdóttir G, Ehnfors M, Ehrenberg A. Accuracy, completeness and comprehensiveness of information on pressure ulcers recorded in the patient record. Scand J Caring Sci. 2012;27(1):84–91.
46.	Berlowitz DR, Brand HK, Perkins C. Geriatric syndromes as outcome measures of hospital care: can administrative data be used? J Am Geriatr Soc. 1999;47(6):692–696.

---

Supplementary materials

Table S1 Summary of ICD-10-CA codes for adverse events

---

References

1.	Layde PM, Meurer LN, Guse CE, et al. Medical Injury Identification Using Hospital Discharge Data. Available from: http://www.ncbi.nlm.nih.gov/books/NBK20501/pdf/ch10.pdf. Accessed March 11, 2014.
2.	Hospital e-Scorecard Report 2008: Acute Care – Clinical Utilization and outcomes technical summary. Toronto, ON: Health System Performance Research Network. Available from: www.hsprn.ca/?p=32. Accessed March 14, 2014.
3.	Canadian Hospital Reporting Project (CHRP) [webpage on the Internet]. Ottawa, ON: Canadian Institute for Health Information. Available from: http://www.cihi.ca/CIHI-ext-portal/internet/EN/TabbedContent/health+system+performance/indicators/performance/cihi010657. Accessed February 28, 2014.
4.	ICD-10-CA [webpage on the Internet]. Ottawa, ON: Canadian Institute for Health Information. Available from: http://www.cihi.ca/cihi-ext-portal/internet/en/document/standards+and+data+submission/standards/classification+and+coding/codingclass_icd10. Accessed February 28, 2014.