Do Checklist-Induced Behavioral Changes Improve Self-Confidence in Fundoscopic Examination? A Mixed-Methods Study

Introduction

Fundoscopic exams are conducted during outpatient consultations to assess intracranial disease, ocular symptoms caused by systemic diseases, and complications from chronic diseases.^1,2 Medical students, medical interns, and medical home care specialists are expected to attain the skill to execute fundoscopic exams.^3–5

However, implementation frequency is low even for patients suggested for fundoscopic exams. A Japanese study of medical interns and attending physicians found that more than 80% of them conducted fundoscopic exams only once every 2–3 months.⁶ The low frequency of implementation has been reported to be caused by the low level of confidence among medical interns and attending physicians in their fundoscopic examination skills.^2,7,8 Our preliminary interviews clearly showed that medical practitioners did not conduct fundoscopic exams because they forgot to do so.

Research has not yet identified specific measures through which the healthcare system may increase the implementation of fundoscopic exams, nor a qualitative process that enables physicians to gain confidence in their fundoscopic exam skills. In this study, we quantitatively verified increases in exam implementation for patients suggested for fundoscopic exams, as well as improvements in physicians’ confidence in fundoscopic exam procedures, by introducing a checklist verifying exam implementation. We also conducted a qualitative survey on the psychological backgrounds of novice medical practitioners who have had little practice with fundoscopic exams in order to clarify the process by which they gain confidence in examination procedures, using questionnaires and interviews.

Materials and Methods

Design

This study was a before-and-after study within an embedded-experimental mixed-methods design⁹ in which qualitative data were collected following the intervention and analyzed after the quantitative analysis.

Participants

The subjects of this study satisfy two conditions. The first condition was that they are physicians who can independently conduct fundoscopic exams and gain confidence in the procedure. The second condition was that the subjects were novice medical practitioners (they had graduated less than 10 years prior) who had completed their first 2 years of medical residency. The subjects were selected so that the study could track them as their experience with fundoscopic exams increased. We needed to exclude subjects with high confidence or experience.

In line with these conditions, we uniformly sampled 15 doctors, who were beginners in fundoscopy, assigned to initial outpatient consultations at the Department of General Medicine at Chiba University Hospital. They were not residents, but fellows in this department. We excluded physicians who did not undertake any initial consultations during the research period, who did not conduct medical examinations of patients indicated for the fundoscopic exam, and who were partially absent due to training in other hospitals. For physicians who met the conditions more than once across multiple time periods, we only sampled the first record and excluded all subsequent ones.

Procedure

Figure 1 shows the design flow of this study. Phase 1 (target period) spans April to May 2016, April to May 2017, and October to November 2017. Phase 2 (intervention period) spans the two months following each target period. Phase 3 (post-intervention period) was set as August to September 2016 only. The three varying periods were set because the physician assigned to the initial outpatient consultation changed over time. In Phase 1, we measured the implementation rates for fundoscopic exams without interventions. In Phase 2, we carried out an intervention as described below. At the end of Phase 1 and Phase 2, we administered a self-completed questionnaire and semi-structured interview. Introductory teaching sessions, including hands-on training in ophthalmoscopy, were provided before Phase 1.

Figure 1 The design flow of this study.

Intervention

The intervention is the introduction of a checklist that verifies whether the fundoscopic exam was implemented, included a structured multiple-choice question addressing possible reasons why the exam was not implemented (multiple answers were allowed). Principal investigator (PI) (DY) and the research collaborator (KS) prepared and test the checklist (Supplement Figure 1). The checklist was printed and presented at the start of the consultation. Participants completed the checklist immediately following the medical consultation. They were required to discuss all cases with the attending physician. They can complete the checklist within one minute. The attending physician signed the checklist during consultation in order to avoid any omissions by participants. The attending physician did not make any recommendations to participants about whether to perform fundoscopy or not. Participants have the opportunity to receive feedback on fundoscopic findings during outpatient consulting, and in regular practice, they almost always receive it from the attending physicians. Participants could refer patients to the department of ophthalmology in the same hospital for close examination of fundus abnormalities. They submitted the checklist at the end of the consultation to a designated box. PI collected a checklist from that box periodically and tabulated them.

Data Collection

Quantitative Data Collection

We investigated the ratio of the fundoscopic exam to the total number of indication cases during the target and intervention periods. Between April to September 2016, we measured monthly implementation ratios for Phases 1, 2, and 3 to measure the residual effects of the checklist. Based on a preliminary interview, we defined the patients indicated for fundoscopic exams as possessing “primary symptoms such as a headache or ocular symptoms” or “having a medical history of hypertension or diabetes.”

In the self-completed questionnaire conducted at the end of Phases 1 and 2, we measured and compared VAS (Visual Analog Scale, 0–100 mm) values for the question, “Are you confident in the procedure? (confidence in procedure).” VAS is used to evaluate the degree of confidence in other contexts such as confidence and importance felt for health class exercises¹⁰ and confidence in cardiopulmonary resuscitation.¹¹ We selected VAS for this study for the purpose of assessing subtle changes in awareness, such as confidence.

We measured and compared VAS values pertaining to the survey items in the self-completed questionnaire conducted at the ends of Phases 1 and 2: “Do you feel it is indispensable for diagnosis? (usefulness for diagnosis),” “Do you feel it is indispensable in the management of chronic diseases? (usefulness with chronic diseases),” “Do you think it will benefit the patient? (patient benefit),” “Do you wish to teach this procedure to subordinates and students? (enthusiasm for education).”

We used a checklist to tally the reasons for which physicians did not perform fundoscopic exams during consultations with each patient. The reasons were as follows: “Not a patient indicated for the fundoscopic exam,” “Not useful for differential diagnosis,” “Cost and social factors,” “Patient-related factors (ex. lack of consent),” “Other (please specify).” In addition, in the questionnaires conducted at the end of Phases 1 and 2, we presented the respondents with multiple-choice options for why the fundoscopic exam was not implemented in each phase.

We have based our sample size estimate on the results of the initial study. In our preliminary study, we calculated the number of fundoscopy attempts and the number of patients who would need fundoscopy. Detecting an implementation ratio difference between the two phases with a one-sided significance level of 5% and power of 80% would require 95 patients in each phase. Detecting a mean difference of confidence in fundoscopy with a one-sided significance level of 5% and power of 80% would require 10 pairs of participants.

Qualitative Data Collection

We obtained qualitative data from individual interviews using semi-structured questions. Supplement Table 1 shows the interview guide, which included questions about participants’ demographics. We carried out these details and participants were asked to respond freely. PI (DY) and the research collaborator (KS) conducted on one-on-one Interviews with the participants immediately before and after the intervention period; a private room was used to ensure privacy. We recorded the interview as audio data with permission from the research subjects, and transcribed and collected the data as verbatim records. The interview was conducted in Japanese. The data transcribed into Japanese and coded and then translated into English. DY and KS are colleagues of the interviewees. The research collaborator is an attending physician who received their training and master’s degree in Health Professions Education and is proficient in qualitative and mixed-methods research. Both are responsible for educating medical students and interns, and are in a position to host lectures about fundoscopic exams.

Data Analysis

Of the quantitative data, a χ² test was used on the implementation ratio, ratio of indication cases, and the responses of the self-completed questionnaires conducted at the end of Phases 1 and 2. We used Fisher’s exact test when more than 20% of the expected values were less than 5. We quantified the VAS values in a range of [0–10.0] and analyzed them using the paired t-test. Statistical analyses were performed using SPSS Statistics for Windows 22.0 (IBM Corp. Armonk, NY).

To analyze the qualitative data, we used the Modified Grounded Theory Approach (M-GTA).^12,13 We assumed that the implementation of the fundoscopic exam by a novice medical practitioner possesses reciprocity such as physician factors, exam characteristics, patient factors, and the healthcare system. We also assumed that the attainment of confidence is procedural; thus, we used M-GTA as the methodology to perform the analysis.

Based on the M-GTA analysis, we set the focus of analysis to be novice medical practitioners implementing the fundoscopic exam, and the theme of analysis to be the reasons for implementation or non-implementation of the fundoscopic exam, as well as the psychological processes behind the attainment of confidence. We highlighted the focus of analysis themes in the verbatim interview, identified relevant parts, recorded them as specific examples, and attempted to create concepts that resembled these examples. An analysis worksheet was used to generate the concepts; we then entered the names, definitions, specific examples, and theoretical memos of the concepts. Similarly, we proceeded with the analysis of the sentences, each time generating a new concept to create an analysis sheet. With the generated concepts, we verified similar examples between the concepts and compared the opposite examples. We entered such relationships between concepts into the theoretical memo to prevent the list of concepts from becoming arbitrarily biased. We created categories based on these inter-conceptual relationships, highlighted the analysis themes, and examined whether they could explain the entire process.

We conducted a pre-intervention interview and analyses, then conducted a post-intervention interview and added an analysis in each subsequent period. After verifying that the data across 15 participants (30 interviews) possessed no excess or deficiencies in interpretation, we determined theoretical saturation and created categories, story lines, and result diagrams.

Ethics

The study protocol was approved by the Ethics Committee of Chiba University School of Medicine (Chiba, Japan), and was conducted in accordance with the Declaration of Helsinki. A detailed explanation of the study was given to all participants, who confirmed that they fully understood the information before voluntarily giving informed consent to participate. The trial was registered with the University hospital Medical Information Network Clinical Trials Registry Clinical Trial (Unique trial number: UMIN 000032714). The data that support the findings of this study are available from the corresponding author on reasonable request.

Results

Participants’ Baseline Characteristics

Figure 2 shows the flow of participants; 15 participants were recruited to the study. Four participants conducted medical consultations in varying intervention periods, and we thus excluded all periods after the first.

Figure 2 Participants flow.

Table 1 shows the characteristics of participants. The median number of years after graduation was 3 years (range: 3–10), and the median age was 29 years (range: 26–41). The median number of years after graduation excludes work absences such as maternity leave.

Table 1 Baseline Characteristics

Quantitative results

Primary Outcome: Implementation Ratio of Fundoscopy

Supplement Table 2 shows the breakdown of indications. There were 284 patients in Phase 1 and 311 patients in Phase 2 that received initial outpatient consultations by the physicians targeted. There was only one instance in which a response was missed on the checklist.

There was a significant increase in the implementation ratio of fundoscopy for applications to clinical cases (19.2% (29/151 cases) vs 64.8% (105/162 cases), p<0.001) between the two phases.

With respect to the residual effect, there was no significant difference in the implementation ratios between Phase 1 (April–May 2016) and Phase 3 (August – September 2016) (18/78 cases (23％) vs 14/70 cases (20％) (p=0.65)).

Primary Measure: Changes in Confidence in Fundoscopy

In the self-completed questionnaire, we observed a significant difference before and after the intervention period for the survey item, confidence in procedure (1.9 mm vs 4.7 mm (p<0.001)) (Table 2).

Table 2 Changes in Attitude Toward the Fundoscopy

Secondary Measure: Changes in Feelings About Fundoscopy Except Confidence

In the self-completed questionnaire, we observed a significant difference before and after the intervention period for the survey item, enthusiasm for education (5.1 mm vs 6.3 mm (p=0.028)) (Table 2).

Secondary Measure: Reasons for Not Conducting Fundoscopy

According to the checklist questions (multiple selections allowed), the reasons cited for omitting the fundoscopic exam for patients who were not indicated for the exam during Phase 2 were as follows: “Not a patient indicated for the fundoscopic exam (92 cases, 61.7%),” “Not useful for differential diagnosis (48 cases, 32.2％),” “Cost and social factors (27 cases, 18.1%),” and “Patient-related factors (9 cases, 6.0%).” In five cases (3.4%), “Forgetting” was written in as a response.

Supplement Table 3 shows the reasons cited in the self-completed questionnaire for omitting the fundoscopic exam. Before and after the intervention period, the response, “I’m not confident in how to use the fundoscopy” decreased due to the intervention (12/15 participants vs 1/15 participants (p <0.001)).

Qualitative Results

The 15 participants received 30 interviews: one interview before the intervention and one after. The average interview time was 10 minutes 54 seconds ± 55 seconds. Analysis of the interviews produced five themes from the initial 632 codes, 31 concepts, and 8 categories.

For each of the eight categories, we identified the categories, [Positive motivation: exam characteristics] and [Positive motivation: physician factors] as reasons to conduct the fundoscopic exam, and the categories, [Negative motivation: exam characteristics] and [Negative motivation: physician factors] as reasons to not conduct the fundoscopic exam. Exam characteristics refer to the properties and characteristics of the fundoscopic exam itself, while physician factors refer to the views and procedures of the participating doctors. [Improvement plan] refers to the specific ways that each factor may be improved. Positive and negative motivations have opposite properties, and their magnitudes determined the implementation or [Non-implementation] of the fundoscopic exam. Implementation was divided into two categories representing a successfully executed exam ([Successful implementation]) or unsuccessfully executed exam ([Unsuccessful implementation]). Each concept is described in more detail in the Discussion and presented in Supplement with supporting codes. We indicate categories with [], concepts with <>, variations with “”, and supplementary explanations with (). Figure 3 shows the conceptual diagram of the categories.

Figure 3 Conceptual diagram of categories.

Discussion

This study used a quantitative survey to assess changes in the implementation ratio of fundoscopic exams effected by a checklist intervention, as well as a qualitative survey to assess the psychological backgrounds of subject to affect the implementation ratio. We found that the introduction of the checklist elevated the implementation ratio of the fundoscopic exam, and post-intervention, there was an elevated level of confidence and educational initiative.

Reasons for Implementing the Exam

Despite the reported demand to learn fundoscopy techniques among medical students and interns,^6,7 as well as doctors who have completed training,¹⁴ there are limited opportunities to learn the skills needed to implement fundoscopic exams. The qualitative results of this study – <Self-directed learning opportunity> and <Desire to learn procedure> – have also shown a desire among physicians.

The fundoscopic exam possesses significance as <Significance for evaluating complications from chronic diseases>, <Significance for evaluating retinal diseases>, and <Significance for evaluating increased brain pressure>. Insufficient understanding of these concepts (<Lack of understanding significance>) and the <Lack of knowledge> of the fundoscopic exam itself became negative motivations in the category of [Negative motivation: physician factors].

Among the questionnaire results, there was no significant difference pre- and post-intervention in the survey items, “Do you feel it is indispensable for diagnosis?”; “Do you feel it is indispensable in the management of chronic diseases?”; and “Do you think it will benefit the patient?” Even in the interviews, the participants understood the significance of fundoscopic exams, but their responses remained formulaic. Specifically, participants had little experience with the procedure being clinically useful (<lack of successful experiences in detecting abnormalities>); thus, the weighting of the motivation to use the fundoscopy appeared small. In addition, participants were aware of the disadvantages to patients of not performing fundoscopy as the <Opportunity cost of patient benefits>.

Reasons for Not Implementing the Exam

In the self-completed questionnaire and the interviews, many participants cited reasons for not implementing fundoscopic exams as follows: <Lack of understanding significance>, <Lack of confidence>, <Inexperience in procedure>, <Time performance>, and <Forgetting>. Questionnaires in prior studies¹⁴ reveal barriers to the fundoscopic exam as “ophthalmoscope not working,” “no ophthalmoscope available,” “unable to see,” “not attempted,” and “not allowed.” Similar to this study, the inability to see through the fundoscopy was a reason not to implement the exam.

In the self-completed questionnaire administered post-intervention, the survey item, “I am not confident in how to use the fundoscopy’’ had a statistically significant decrease; the survey item, “Even though I always use the fundoscopy, I can’t conduct medical exams well with it” decreased as well, although it was not a significant difference. The results showed that the intervention using the checklist could potentially resolve a <Lack of confidence> or <Inexperience in the procedure>. We believe that these two concepts have a large weighting with respect to the motivations driving the decision not to implement the fundoscopic exam.

Because the fundoscopic exam is a simple exam that can be performed by a primary care physician (ie, not an ophthalmologist), physicians felt the exam to be <Inferior to ophthalmology>, <Inferior to alternative testing> and <False positive/negative findings>. These concepts acted negatively during motive formation and were associated with the concepts <Inexperience in procedure>, <Lack of knowledge> and <Lack of confidence>.

Decrease of <Forgetting> Through Use of the Checklist

An independent negative motivation was <Forgetting>. Opportunities are present in internal medicine, primary care, and emergency medicine to use the fundoscopy, but fundoscopic exams are often not performed unless there are obvious ocular symptoms.¹⁵

Checklists are useful in reducing postoperative mortality and major surgical complications,^16,17 reducing human error during extracorporeal circulation,¹⁸ and providing reminders to mitigate missed collections of outpatient guidance and management fees.¹⁹ In the qualitative survey, the use of a checklist served as a reminder to reduce <Forgetting> and led to the motivation to implement the fundoscopic exam (<usefulness of the checklist>).

However, in the survey inquiring the reason for not performing the fundoscopic exam, there was no significant difference in the number of people who responded, “I forgot,” despite the checklist. In short, the <Forgetting> also occurred in post-intervention interviews. We speculate that the elimination of < Forgetting> factors will require not only the checklist but a system to remind the physicians during the consultation.

Impact of Frequency on Confidence

The quantitative results revealed that the checklist intervention increased the implementation ratio and the confidence to implement fundoscopic exams. The qualitative survey revealed that a <Lack of successful experiences in detecting abnormalities> resulted in a <Lack of confidence>, which became a reason for non-implementation. These results are similar to those of prior studies.^7,20 Sixty-three percent of medical students responded that they had experience with an abnormal fundus less than 3 times, while 23% had never seen one. Medical students who had had fewer opportunities to observe abnormal findings were more likely than those who had observed many abnormalities to report a lack of confidence in identifying optic disc edemas and proliferative retinopathy.²⁰

On the contrary, this study confirmed the process of <successful experiences and confidence building> through [Successful implementation]. Participants gained confidence by having more practice in the procedure, which led to the <Progress of skill>. The following strategies have been proposed to increase confidence: having the participants perform a task,¹ hosting a teaching session,²¹ having the participants practice a lot,⁷ and daily use of the fundoscopy.²⁰

Impact of Confidence on Frequency

Studies have shown that increased confidence in fundoscopic exams leads to skilled use.¹ In this study, we found that <Lack of confidence> is also related to the concepts <inferior to ophthalmology> and <False positive/negative findings> due to a synergistic effect with <Inexperience in technique>; we were able to verify the process in which these became the motivation for non-implementation. We believe that apart from <Environmental pressures>, <Successful experiences and confidence building> reduced the <Lack of confidence>, which led to an increase in exam implementation. From this, we believed an important strategy to increase exam implementation frequency would be to reduce negative motivations in addition to increasing positive motivations.

Increased Aggressiveness to Education

Qualitative feedback through instruction is said to increase confidence in implementing fundoscopic exams.^7,20 Confidence is lost from continued lack of exposure,²⁰ and it is possible that ongoing extracurricular instruction may lead to an increase and/or maintenance of confidence. Our qualitative survey also found that <Lack of confidence> is related to the <Lack of educational initiative>, and conversely, gaining confidence increases <Educational initiative>.

Residual Effect

We expected that the number of implementations would have been maintained if the checklist intervention maintained its reductive effects on [Negative motivation: physician factors] after completion. However, two months post-intervention, the implementation ratio had returned to the same level as two months pre-intervention. The residual effects of education on healthcare professionals are proportional to the time of training received and the time engaged in related tasks.²² It is possible that a checklist intervention over a long period of time could change physicians’ actions.

Limitations

Potential limitation of this study is that the design of the quantitative study is a before-and-after comparative study. While we had considered the possibility that the increase in exam implementation and confidence might have been impacted by habituation, we believe habituation did not have any effect, since we verified the post-intervention disappearance of the residual effect.

The relationship to the researchers and the participants is cause for concern in the methodological rigor foundation of this study. Given the teaching/instruction position of the researchers and the novice classification of the participants, there may be a power hierarchy. It is likely that the participants provided favorable information about using the checklist because they know the researchers and do not want to disappoint them (Supplemental Table 4 on Qualitative coding, theme <Environmental Pressures>). The fact that the attending physician had to sign each checklist received from the participant may have put pressure on the participant. Participants may have felt obligated to comply and responded positively, which may have affected the quantitative study as a result.

We did not measure indicators of whether or not participants improved their skills. However, the qualitative study revealed that participants were aware of their improved skills as a result of the increased number of fundoscopies performed:

The procedure itself became smoother as my skills improved in seeing the fundus. It takes a shorter time to obtain findings now. I can obtain them accurately and in a short amount of time.

Also, we did not investigate whether patient outcomes improved. Even when participants could see the fundus, they rarely seemed to detect an abnormality there or experience anything that would change their subsequent course of action. This is evident from the qualitative study results, as participants also made statements about the <Lack of successful experiences in detecting abnormalities> in the post-intervention interviews.

Conclusion

This study found that factors such as <Forgetting>, <Lack of confidence>, and <Inexperience in procedure> became reasons for not performing fundoscopic exams. In the qualitative survey, the presentation of a checklist served as an intervention to prevent <Forgetting> of implementation and provide <Environmental pressures>. The intervention increased the implementation ratio, thereby increasing successful experiences and confidence among physicians, and ultimately promoting the <Progress of skills>. In addition, the growth of confidence boosted motivations to implement fundoscopic exams and may have further increased the implementation ratio. These processes could be comprehensively understood by the mixed-methods research.