Venous thromboembolism risk assessments on trauma patients has suboptimal interobserver reliability among inexperienced clinicians (fourth-year medical students)

Introduction

Venous thromboembolism (VTE) is a significant cause of mortality and morbidity in hospitalized patients worldwide. Fatal pulmonary embolism (PE) is the most common preventable cause of hospital death. The incidence of clinically symptomatic VTE in the USA is ~250,000–300,000 new cases each year.^1,2 PE occurs in 0.2%–0.4% of hospitalized patients and causes ~200,000 deaths annually.³ It is estimated that the USA spends ~US$7–10 billion in direct medical costs on VTE each year.⁴ Each hospital-acquired deep vein thrombosis (DVT) represents an incremental inpatient cost of US$10,000, while each PE represents ~US$20,000 in additional cost. Without prophylaxis, the incidence of DVT in general surgery patients is about 15%–40%, while it is 40%–80% in major trauma.⁵ Despite guidelines being regularly published by the American College of Chest Physicians (ACCP), VTE prophylaxis remains underutilized. Current estimates suggest that only 30%–60% of patients at risk receive appropriate VTE prophylaxis.⁶

Established clinical risk factors include presence of inherited conditions, such as factor V Leiden mutation, prothrombin gene mutation, and protein S or C deficiency, or acquired risk factors, such as major surgery involving the abdomen, pelvis, and lower extremities; active malignancy and its treatment; immobilization; trauma, especially involving the spinal cord injuries or spinal fractures; fracture of pelvis, hip, and leg; presence of central venous catheter; pregnancy; drugs; congestive heart failure; chronic renal disease; the antiphospholipid syndrome; obesity; smoking; older age; and history of thromboembolism.

Trauma is the leading cause of death in people of age <40 years, and the care of patients with trauma accounts for more hospital days per year than the care of patients with heart disease or cancer. DVT and PE are common complications of major trauma.⁷ The 2016 Annual Report of the National Trauma Data Bank (NTDB) reports DVT/thrombophlebitis as one of the top 10 posttrauma complications, with incidence of 0.60%. The incidence of PE is reported as 0.27% (NTDB).

The Eastern Association for the Surgery of Trauma has developed evidence-based practice management guidelines for the prevention of VTE in trauma patients.⁸

The process of providing VTE prophylaxis begins with assessment of each patient’s VTE risk. Assessment forms are often completed by first-year residents on admission or arrival to the floor; based on the relative inexperience of the interns, we hypothesize that these VTE risk assessments may be inconsistent. The objective of this study is to compare interobserver reliability for qualitative and quantitative VTE risk assessments in hospitalized trauma patients.

Methods

Study design

We conducted a retrospective medical record review of 40 randomly selected trauma patients (data only) admitted to a 448-bed urban level-I American College of Surgeons-verified adult trauma center from January 2013 to January 2014. Random selection was by the closed-envelope draw technique. Accordingly, 500 envelopes were made for patients seen and randomly selected. The sample was equally distributed between patients admitted to the floor and the intensive care unit. As this study had deidentified data that were used for study/review, the institutional review board (IRB) at SUNY Downstate exempted this study from full IRB review. Because of this, the IRB of SUNY Downstate Medical Center deemed patient consent also unnecessary.

Data analysis

Eight fourth-year medical students on surgical rotation performed the risk assessments using the deidentified medical records. Two forms for risk assessment were used, a qualitative screening and a quantitative screening. Students were instructed to carry out the following: 1) risk score assessment; 2) risk stratification – low, medium, and high risk; 3) identification of contraindications to pharmacological prophylaxis, whether absolute or relative; and 4) VTE prophylaxis plan. The qualitative screening comprised the point-based VTE protocol currently followed in our hospital (not an established risk assessment), wherein the students had to identify the medical conditions associated with risk for VTE, contraindications to pharmacological prophylaxis, and, based on their assessment, choose the appropriate prophylaxis from the order set according to the risk (Figure 1). The quantitative method was a modified version of the Caprini score; students had to identify the risk factors, which in turn were given a particular score to be added to estimate the cumulative score. Patients were then classified to be at low (<2), medium (2–5), and high (>5) risk (Figure 2). Prophylaxis included early ambulation, enoxaparin, unfractionated heparin, and warfarin. Students were instructed on how to fill the assessment forms. Statistical analysis was performed by calculation of interclass correlation coefficient (ICC) based on a two-way mixed model. The composite of intraobserver and interobserver variabilities was determined.

Figure 1 VTE prophylaxis screening and ordering form. Abbreviations: VTE, venous thromboembolism; SCr, serum creatinine; GFR, glomerular filtration rate; GI/GU, gastrointestinal/urogenital; PLT, platelet; HIT, heparin-induced thrombocytopenia; OR, operating room; wt, weight INR, international normalized ratio; SQ, subcutaneous; PO, per os; IPC, intermittent pneumatic compression; GCS, graduated compression stockings; UFH, unfractionated heparin;q, every.

Figure 2 VTE risk assessment form. Abbreviations: VTE, venous thromboembolism; CRP, C-reactive protein; COPD, chronic obstructive pulmonary disease; OC, oral contraceptive; HRT, hormone replacement therapy; DVT/PE, deep vein thrombosis/pulmonary embolism; BMI, body mass index; CHF, congestive heart failure.

Results

We excluded incomplete charts from the final sample. We also excluded one observer due to multiple incomplete assessments. The final sample thereby consisted of 35 patients assessed by seven observers for qualitative assessment and 37 charts assessed by eight observers for quantitative assessment.

Qualitative risk assessment

The final sample had 35 charts assessed by seven observers. Cronbach’s alpha was measured to be 0.644. The ICC for this group was 0.845 (95% CI 0.753–0.911) (Figure 3).

Figure 3 Qualitative VTE risk assessment. Abbreviations: VTE, venous thromboembolism; df, degrees of freedom.

Quantitative risk assessment

The final sample had 37 patients assessed by eight observers. Cronbach’s alpha was determined to be 0.869 for this group, and ICC was determined to be 0.628 (95% CI 0.413–0.787) (Figure 4).

Figure 4 Quantitative VTE risk assessment. Abbreviation: VTE, venous thromboembolism.

Discussion

PE resulting from DVT, collectively referred to as VTE, is the most common preventable cause of hospital death.⁹ Despite the reality that hospitalized medical and surgical patients routinely have multiple risk factors for VTE, making the risk for VTE nearly universal among inpatients, large prospective studies continue to demonstrate that these preventive methods are significantly underutilized.^6,10,11–13 The Agency for Healthcare Research and Quality^9,15 calls thromboprophylaxis the “number one patient safety practice’’ against VTE. Current estimates suggest that only 30%–60% of patients at risk receive appropriate VTE prophylaxis. Prevention requires a reliable tool for risk stratification for the development of VTE, screening strategies, and effective prophylaxis to significantly reduce mortality in trauma patients. A study has shown that implementation of a mandatory computerized provider order entry-based clinical decision support tool significantly improved compliance with VTE prophylaxis guidelines in hospitalized adult trauma patients.¹⁴ Many risk assessment tools are available for estimating thromboembolism risk.¹⁵ Examples are the Caprini score,^16,17 Padua score,¹⁸ Wells score,^19,20 the risk assessment profile developed by Greenfield et al,²¹ the revised Geneva score, and the simplified revised Geneva score.²² No single risk assessment has been prospectively validated as being superior to others. Using an individualized, point-based protocol in the assessment process is a complex task and might contribute to variability in VTE prophylaxis-prescribing behavior. There are no published data on how reliably medical residents can perform risk assessment and prophylaxis using a point-based VTE risk assessment tool.²³ The relative inexperience of interns might lead to inconsistency and underutilization of VTE prevention strategies. It is necessary that a risk assessment tool that would reduce inconsistency among residents be utilized, thereby enabling them to order an effective prophylactic measure. Studies have suggested that even with repetitive use of a VTE risk assessment tool, the aggregate resident interrater reliability is moderate for risk stratification and contraindications, with reliability being only fair for VTE plan.²⁴ Some have suggested that risk-appropriate VTE prophylaxis prescription varied dramatically among residents and that a substantial number of residents may habitually provide suboptimal care. Residents may be a more reliable target for quality improvement efforts.²⁵ Providing personal clinical effectiveness feedback, including data and peer-to-peer coaching, improves resident performance and results in significant reduction in harm for patients.²⁶

In our study, we sought to compare interobserver reliability for qualitative and quantitative assessment in trauma patients and to determine whether qualitative or quantitative risk assessment is a better tool for residents to evaluate VTE risk, as these forms are often completed by first-year residents.

Limitations of this study include that we presumed that fourth-year medical students are equivalent to incoming first-year residents in terms of clinical experience, and risk assessment forms were completed by them. It would have been ideal if the observer pool consisted of incoming residents. Another limitation is that performance of the residents should have been compared to that of senior residents to determine the effect of formal training on resident performance. As most of the history and physical examinations are done by junior residents, and hence the DVT risk assessment, ideally, the study should track incoming resident performance and evaluate which type of risk assessment has better patient outcomes. However, our aim was to see whether quantitative assessment gave better assessment of VTE risk and whether the interobserver reliability was optimal.

Conclusion

Quantitative VTE risk assessment has been shown to be more reliable than qualitative assessment, yet our findings suggest that quantitative interobserver reliability is suboptimal among fourth-year medical students to first-year residents. To optimize the accuracy of VTE risk stratification and appropriate prophylaxis, inexperienced clinicians should be formally trained to perform quantitative assessments. Qualitative VTE risk assessment is more challenging to use, and institutes should consider using the quantitative method for VTE risk assessment.

Acknowledgment

A version of this work was published in Critical Care Medicine, December 2014, Volume 42, Issue 12, p A1623.

Author contributions

All authors contributed toward data analysis, drafting, and revising the paper, and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Raskob GE, Silverstein R, Bratzler DW, Heit JA, White RH. Surveillance for deep vein thrombosis and pulmonary embolism. Am J Prev Med. 2010;38(4 suppl):S502–S509.
2.		Office of the Surgeon General (US), National Heart, Lung, and Blood Institute (US) [webpage on the Internet]. The Surgeon General’s Call to Action to Prevent Deep Vein Thrombosis and Pulmonary Embolism. Rockville, MD: Office of the Surgeon General (US); 2008. Available from: https://www.ncbi.nlm.nih.gov/books/NBK44178/. Accessed April 20, 2018.
3.		Maynard G, Stein J. Designing and implementing effective venous thromboembolism prevention protocols: lessons from collaborative efforts. J Thromb Thrombolysis. 2010;29(2):159–166.
4.		Grosse SD, Nelson RE, Nyarko KA, Richardson LC, Raskob GE. The economic burden of incident venous thromboembolism in the United States: a review of estimated attributable healthcare costs. Thromb Res. 2016;137:3–10.
5.		Geerts WH, Pineo GF, Heit JA, et al. Prevention of venous thromboembolism: the seventh ACCP conference on antithrombotic and thrombolytic therapy. Chest. 2004;126(3 suppl):338s–400s.
6.		Tapson VF, Decousus H, Pini M, et al. Venous thromboembolism prophylaxis in acutely ill hospitalized medical patients: findings from the international medical prevention registry on venous thromboembolism. Chest. 2007;132(3):936–945.
7.		Geerts WH, Code KI, Jay RM, Chen E, Szalai JP. A prospective study of venous thromboembolism after major trauma. N Engl J Med. 1994;331(24):1601–1606.
8.		Rogers FB, Cipolle MD, Velmahos G, Rozycki G, Luchette FA. Practice management guidelines for the prevention of venous thromboembolism in trauma patients: the EAST practice management guidelines work group. J Trauma. 2002;53(1):142–164.
9.		Maynard G. Preventing Hospital-Associated Venous Thromboembolism: A Guide for Effective Quality Improvement. 2nd ed. Rockville, MD: Agency for Healthcare Research and Quality; 2015. AHRQ Publication No. 16-0001-EF.
10.		Goldhaber SZ, Tapson VF; DVT FREE Steering Committee. A prospective registry of 5,451 patients with ultrasound-confirmed deep vein thrombosis. Am J Cardiol. 2004;93(2):259–262.
11.		Monreal M, Falgá C, Valle R, et al; RIETE Investigators. Venous thromboembolism in patients with renal insufficiency: findings from the RIETE registry. Am J Med. 2006;119(12):1073–1079.
12.		Kahn SR, Panju A, Geerts W, et al. Multicenter evaluation of the use of venous thromboembolism prophylaxis in acutely ill medical patients in Canada. Thromb Res. 2007;119(2):145–155.
13.		Bergmann JF, Lloret-Linares C, Rami A, et al. [Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): results obtained in France]. Presse Med. 2011;40(12 pt 1): e528–e537. French.
14.		Haut ER, Lau BD, Kraenzlin FS, et al. Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerized clinical decision support tool for prophylaxis for venous thromboembolism in trauma. Arch Surg. 2012;147(10):901–907.
15.		Appendix B. Risk Assessment Models, Protocols, and Order Sets (Continued). Rockville, MD: Agency for Healthcare Research and Quality; 2016. Available from: http://www.ahrq.gov/professionals/quality-patient-safety/patient-safety-resources/resources/vtguide/appendixb.html.
16.		Caprini JA. Thrombosis risk assessment as a guide to quality patient care. Dis Mon. 2005;51(2):70–78.
17.		Caprini JA, Hyers TM. Compliance with antithrombotic guidelines. Manag Care. 2006;15(9):49–50, 53–60, 66.
18.		Barbar S, Noventa F, Rossetto V, et al. A risk assessment model for the identification of hospitalized medical patients at risk for venous thromboembolism: the Padua Prediction Score. J Thromb Haemost. 2010;8(11):2450–2457.
19.		Modi S, Deisler R, Gozel K, et al. Wells criteria for DVT is a reliable clinical tool to assess the risk of deep venous thrombosis in trauma patients. World J Emerg Surg. 2016;11:24.
20.		Wells PS, Anderson DR, Bormanis J, et al. Value of assessment of pretest probability of deep-vein thrombosis in clinical management. Lancet. 1997;350(9094):1795–1798.
21.		Greenfield LJ, Proctor MC, Rodriguez JL, Luchette FA, Cipolle MD, Cho J. Posttrauma thromboembolism prophylaxis. J Trauma. 1997;42(1):100–103.
22.		Klok FA, Mos IC, Nijkeuter M, et al. Simplification of the revised Geneva score for assessing clinical probability of pulmonary embolism. Arch Intern Med. 2008;168(19):2131–2136.
23.		Beck MJ, Haidet P, Todoric K, Lehman E, Sciamanna C. Reliability of a point-based VTE risk assessment tool in the hands of medical residents. J Hosp Med. 2011;6(4):195–201.
24.		Moheimani F, Jackson DE. Venous thromboembolism: classification, risk factors, diagnosis, and management. ISRN Hematol. 2011;2011: 124610.
25.		Lau BD, Streiff MB, Pronovost PJ, Haider AH, Efron DT, Haut ER. Attending physician performance measure scores and resident physicians’ ordering practices. JAMA Surg. 2015;150(8):813–814.
26.		Lau BD, Arnaoutakis GJ, Streiff MB, et al. Individualized performance feedback to surgical residents improves appropriate venous thromboembolism prophylaxis prescription and reduces potentially preventable VTE: a prospective cohort study. Ann Surg. 2016;264(6):1181–1187.