The Effects of Using Checklists on Electrocardiogram Interpretation: A Cross- Sectional Study on Medical Interns

Introduction

Heart disease is the second most common cause of death in the modern world. In the United States, heart disease accounts for one-third of death.¹ Accordingly, heart disease is one of the most important areas of medical science that needs to be improved in diagnostic processes.

Electrocardiography (ECG) has a special status among all the methods and tools that help physicians to diagnose heart disease. Because that this diagnostic tool is completely dependent on the person for the interpretation of findings, one of the most important challenges for using this tool is to upgrade the ability of the responsible persons to detect and reduce their error rate.

Considering the dual classification of decision-making processes (object-oriented and component-oriented systems),^2,3 the major system used in ECG interpretation by the health team members in the decision-making process of “System 1”. The problem in the object-oriented process (System 1) is the requirement of full individual mastery of the subject. This thinking system is inefficient in cases where the conditions are very complex or when the thinker has no prior experience of the subject.⁴ In contrast, the component-oriented process (System 2) has a high and very powerful concentration, the ability for reconstruction and a tendency for the abstraction of things, but this system has a limited angle of view, is incapable of recognizing the orientation, and needs time to solve problems. Therefore, top-level managers and lawyers are generally successful in using this method, and this decision-making system is very suitable in cases with modest changes and enough time to address complexities. In other words, when the analytical criteria are constant, such as referral to legal laws during an attorney or stock market forecast, System 2 will be the best option.⁴ One of the most important tools used to activate System 2 is a checklist.⁵ By definition, the checklist is a predefined list of questions or instructions that must be implemented to achieve a specific goal. The guide checklist is a standardized form that helps administrators make a decision and assists executors to take action and ensures that attention is given to all information. In fact, it is used to compensate for memory-related defects or reduced attention. Given the very high importance of System 2 activation in improving the diagnostic course in processes with multiple measurable factors (like ECG interpretation) and considering the lack of studies on this, especially among medical interns who are at the forefront of disease detection, we decided to design a study in this group via the checklist as a tool for using System 2 to examine the impact of this tool on improving the process of ECG interpretation in this group.

Methods

Study Design and Setting

The present cross-sectional study was conducted on medical interns of Mashhad University of Medical Sciences in the first semester of the academic year 2014–2015.

Sample Size

The sample size was estimated at 40 subjects in the present study according to similar studies with sample sizes less than 30.

Participants

The interns, who at any time expressed their willingness to participate in the project, were randomly divided into two groups. First, were determined and recorded the demographic and background profiles including the gender, the grade of the pre-internship exam, the number of months spent in the internship period, and the status of the medical intern in terms of whether the four-week course of heart disease had already been completed.

Data Gathering

According to an example entitled “Ezra’s ECG interpretation checklist” an ECG interpretation checklist was prepared for this study, revised by a cardiologist and three emergency medicine specialists, and ultimately adjusted in nine sections each with two to eight episodes.⁶ Nine standard 12-lead terraces were selected from the Cardiology website.⁷ These strips had different diagnoses, including infarction, metabolic disorders, pericarditis, atrioventricular block, and several types of ventricular and supraventricular arrhythmias. In terms of the degree of difficulty for interpretation by the interns, they were equally divided into three categories: easy, medium, and difficult. The number and scale of tracings were verified by Department of Cardiology professors at Mashhad University of Medical Sciences (Supplementary Figure 1). Then, the selected terraces were given to both groups. The participants in the first group were given a checklist in addition to terraces, and the interns were asked to complete a checklist for each of the terraces and finally to detect the pathology in the ECG according to the completed checklist, in fact, to express their ECG interpretation.

For the second group of interns, the design was performed in two steps. In the first stage, only terraces, which were identical with the terraces of the first group in terms of content, and arrangement, were given to the participants, who were asked to take notes their detection in the answer sheet. In the second step, the checklist was added to the terraces and the interns were asked to complete a checklist for each terrace, and finally state the ECG interpretation according to items in the completed checklist.

After collecting the ECG interpretation checklists, and answer sheets, the checklist was scored and the interpretations were corrected. A score of 0–9 was calculated for each of the participants in each part for the checklist, depending on the existence of the nine parts and the items that must be answered in each part. In the case of recorded interpretations, unanswered, and incorrect or incomplete responses were considered unacceptable, and only the correct responses were considered as acceptable. For each of the difficulty levels of terraces, a maximum of 3 correct responses and a total of 9 correct responses were considered. Considering the probability of a randomly correct response in each of the easy, medium and difficult terraces, the participants who submitted a maximum of 3 correct responses in the interpretation of the terraces were placed in the sub-group of “undesirable response”, and those who had in total more than 3 correct responses were placed in the group of “desirable response”, with the assumption that they have an acceptable ability to interpret the ECG findings. The Chi-Square test was used to compare the subjects of the first group - those who interpreted terraces after completing the checklist - and subjects of the second group - those who responded terraces without a checklist. To determine the effect of the use of the checklist on increasing the interns’ accuracy of revision in the reinterpretation of terraces, the same steps were performed for the participants in the second group before and after completing the checklist.

Statistical Analysis

Data analysis was done using descriptive tests (mean, standard deviation, and percentage) and inferential statistical tests (chi-squared test, Fisher’s exact test, and independent t-test) were carried out using SPSS v23.0. The significance level was P<0.05.

Ethical Considerations

• Before beginning the studies, written informed consent forms was taken from all medical interns.
• Participation in the research was voluntary and free.
• Participants could stay anonymous when filling out the checklists.
• The research was conducted in accordance with privacy principles.
• This study approved by the Ethics Committee of Mashhad University of Medical Sciences, Iran in March12, 2014 with reference number 98/374,839.

Results

A total of 40 students (in two groups of 20) entered the study. Table 1 shows that there is no statistically significant difference between the two groups in terms of baseline variables.

Table 1 Comparison of Baseline Characteristics in the Studied Groups

The first group identified 41.6% of the terraces correctly using the checklist. The second group, without using the checklist, correctly identified 25.5% of the terraces; and after using the checklist, this indicator increased to 32.7% in this group. As shown in Table 2, most of the correct responses of the first group were related to the easy terraces (Figure 1). In the second group, the use of the checklist has led to an improvement in responsiveness to both easy and medium terraces (Figure 2).

Table 2 Comparison of the Number of Correct Responses in the Interpretation of ECG Terraces in the Studied Groups

Figure 1 Comparison of the frequency of correct responses to the difficulty level of terraces in the first group.

Figure 2 Comparison of frequency of correct responses to the difficulty level of terraces in the second group before and after using the checklist.

The mean score obtained for completing the checklist criteria in three levels of easy, moderate and difficult terraces was 45 ± 7.21, 42.05 ± 7.64 and 43.81 ± 5.37 respectively in the first group and 42.83 ± 6.86, 43.08 ± 4.95 and 43.11 ± 5 respectively in the second group. There was no significant difference between the two groups (p = 0.338, p = 0.394, p = 0.672, respectively).

Given that a total of 9 responses were correct for all terraces, considering the probability of a random response in each of the easy, medium and difficult terraces, the participants who submitted up to 3 correct responses in the interpretation of terraces were placed in the subgroup of “undesirable response,” and those who submitted more than 3 correct responses in the group of “desirable response ” with the assumption that they have an acceptable ability to interpret the ECG findings.

Accordingly, it was found that 50% of the subjects in the first group (using the checklist) (n=10) and only 15% (n=3) of the second group (without using the checklist) had desirable responses, but not statistically significant (p = 0.531). On the other hand, the comparison of responses before and after the use of the checklist in the second group showed a significant improvement in the number of desirable responses (15% (n=3) with desirable response before using the checklist and 25% (n=5) with desirable response after using the checklist, p = 0.009) (Table 3). The results of Table 3 also show that using checklists was effective in the reviewing stage and it had a positive effect on the reviewing skill of interns during the reinterpretation of tracings. In other words, using the checklists after the first interpretation increased the correct answers of the second group participants). (Table 4).

Table 3 Comparison of Favorable and Unfavorable Results Between First Group (with Checklists) and Second Group (Without Checklists)

Table 4 Number of Second Group Participants with Favorable and Unfavorable Answers Before and After Using Checklists

Discussion

We showed that the findings of the ECG interpretation performed by the interns of the second group before the completion of the checklist were not statistically different from the findings of the ECG interpretation based on System 2 or the analysis performed by the interns of the first group using the checklist. In other words, the use of the checklist did not have much effect on the correct ECG interpretation compared to not using the checklist. This may be due to the limitations of using the checklist, the most common of which is the phenomenon called Cognitive Load Increase. Some believe that using a checklist may result in cognitive load and exclude the ability to use personal experiences from skilled people, which is referred to as Expertise reversal.^8,9 This is especially evident in the stage of interpretation in various studies.^8–10 Various studies have shown that the use of System 2, including the use of a checklist, has other potential disadvantages. For example, Eva Regehr indicated that compulsory use of System 2 is also accompanied by error.¹⁰ In other words, the presence of structured patterns in the minds of experienced people, which are formed due to exposure in similar situations, acts as a checklist when thinking and deciding, and it creates a coherent, but intrinsic, structure in individual thinking. Considering the evidence in previous studies, it might be possible to explain the results of the present study based on the lack of improvement in the detection process using a checklist. It is possible to design and conduct a similar study on a target group with less experienced and less clinical skill, such as apprentices, to determine the validity of this claim. Similarly, a study examined the importance and impact of the use of the checklist in the ECG interpretation by fifteen experienced individuals. Matthew Sibbald and Anique BH de Bruin, professors at the Toronto Western Hospital in Canada, as well as G van Merrienboer, professor at the Maastricht University in the Netherlands, knew that the use of an abstract-oriented System 2 decision-making process among experienced individuals would increase the cognitive load and so the expertise reversal, that is a phenomenon leading to fatigue mind and slow decision-making instead of improving the speed and accuracy of decision-making with the reverse process and because of the accumulation of massive amounts of trivial content. These professors also knew that the checklist as a simple yet powerful tool was able to demonstrate its effectiveness in reducing human errors in numerous studies. The fact that the checklist activates System 2 and undoubtedly raises the possibility of expertise reversal led them to think that its use under certain conditions could avoid its adverse outcomes. Since there has been no evidence on the confirmation of the checklist efficacy in the verification phase earlier, they assumed that the use of the checklist in the revision phase, rather than in the initial interpretation, could prevent the expertise reversal because it would not stop System 1. The researchers, in a project conducted in February and March 2012, gave 18 difficult terraces, selected by two cardiologists from cardiology reference books, for fifteen cardiac fellows who experienced between 10 and eight years of ECG interpretation and asked to interpret these terraces in four stages and revise their detection. These 18 people interpreted terraces 1–12 first using their usual method in the first step and revised the terraces 1–6 without a checklist in the second step and then terraces 7–12 in the third step using the checklist. Finally, they interpreted terraces 13–15 in the fourth step initially with a checklist and subsequently re-evaluated their own interpretation. Both the number of items referred to in the interpretations and the time dedicated for reading and reassessed terraces were recorded and scored. Two skilled and expert individuals who were unaware of the inquiry process scored the interpretations in terms of the number of correct and incorrect cases. Comparison of Stages 3 and four was considered as an Expertise Reversal evaluation, and comparison of Stages 2 and three could indicate the potential positive impact of using the checklist at the time of verification. Statistical analysis showed that using Expert checklist causes cognitive load in the Interpretation stage, but it increases the speed and accuracy of the decision-making process in the Verification stage.¹¹ One of the main limitations of our study is the small sample size. It seems that doing a study using larger sample size and repeating in different groups of medical students, from trainees to specialist assistants, can more accurately assess the role of using a checklist to reduce diagnostic errors when reading the ECG findings. It also seems that the use of a checklist can be measured in a further study on the decision of therapeutic intervention to evaluate its value in the final decision.

Conclusions

It can be said that despite the effectiveness of the use of the checklist, it has the greatest impact on cases leading to a final or summative decision, such as summarizing the findings for a proper treatment decision. The checklist has no much application for the skilled person in diagnostic cases given that much of the subjective process leading to a diagnosis is implemented when collecting information and that the individual is simultaneously collecting, analyzing, classifying, and even evaluating the data in most cases, and then diminishes or eliminates parallel diagnoses using evidence. However, the checklist may be helpful for unskilled people to avoid forgetting important data.