Teaching Medical Students About Kawasaki Disease: The Development and Evaluation of a Digital Educational Resource

Introduction

Medical educators increasingly utilize digital educational resources to disseminate content and knowledge in their teaching.¹ These resources are used either as a standalone-learning tool or as part of “flipped learning”; a pedagogical approach that allows students to acquire foundational knowledge during independent study, which can then be applied and expanded on in the classroom.² In line with adult learning theory, this shifts the focus from educator to learner-centered learning,³ and research shows the benefits of the approach for improving knowledge retention and learner engagement. Indeed, evidence suggests that engagement with such resources is an efficient way to acquire knowledge and skills compared to traditional educator-led teaching, which in turn can lead to increased motivation among learners.⁴

Within pediatric education, digital educational resources covering a wide range of essential skills are available. While some resources are designed for the purposes of providing quick access to high-yield information such as the Peds Cardiology Handbook app,⁵ others target a wider audience and take the form of written, audio and video content, as discussed in Martiniuk et al’s study.⁶ Some resources provide training in areas of pediatrics that traditionally take place face-to-face, such as pediatric resuscitation⁷ and communication.⁸ Both studies found this mode of teaching to be effective for knowledge acquisition and feelings of competence among healthcare professionals. Digital educational resources are not always developed by expert content creators; for example, Peds Cases is a highly successful student-driven, peer-reviewed interactive website comprising quizzes, cases and clinical videos that is aligned to the pediatric undergraduate medical education curriculum at the University of Alberta.⁹ Therefore, it is possible for medical educators with initiative and sound educational principles to create content that is enriching and beneficial for learners across the globe.

Tools to create digital resources are readily available and accessible for teaching clinicians; an example being Xerte, which is a free, open-source software.¹⁰ Several Xerte e-learning resources, primarily focusing on design and development, have been published in the literature by medical educators.^11,12 However given the proliferation of digital educational resources in the last couple of decades,¹³ there remains a need to focus attention on the evaluation of such tools to enhance e-learning quality and user satisfaction. This paper attempts to contribute to the body of literature on the development and evaluation of digital educational resources. It outlines how a Xerte revision aid for medical students on Kawasaki Disease was developed and evaluated and the way in which this was informed by robust instructional frameworks and educational theory (https://xerte.cardiff.ac.uk/play_14952).

Resource Development

The digital resource was developed by following the Analysis, Design, Development, Implementation and Evaluation (ADDIE) instructional model.¹⁴ To begin with, a People, Activities, Contexts and Technologies (PACT) analysis was conducted on the target audience to explore learners’ needs and requirements of the resource. This took place through informal discussions with medical students via common social media platforms. Analysis revealed that students were in favor of utilizing digital resources for revision purposes, as accessing educational content via hand-held devices, such as mobile phones and tablets offers great flexibility. The interactivity of a digital resource was highlighted as an appealing feature; the overriding opinion was that this would serve as an engaging and enjoyable alternative to revising from textbooks. This is in keeping with literature, which suggests that many of the “generation Y” students prefer e-learning packages that have an interactive component to them over textbooks.¹⁵ Students unanimously agreed that the topic for the digital resource should be both important and relevant to their learning, and identified a need to learn about Kawasaki Disease.

Kawasaki Disease is an acute, self-limited vasculitis of childhood that is the most common cause of cardiovascular disease in childhood. Prompt diagnosis is crucial as a delay in treatment with high dose intravenous immunoglobulins can lead to one in five children developing coronary artery aneurysms.¹⁶ Owing to nonspecific clinical signs and the lack of a definitive diagnostic test, it can be a difficult condition to diagnose. However, there is a criteria established by Tomisaku Kawasaki¹⁷ that clinicians refer to, and which medical students are expected to learn about through independent study and pediatric clinical attachments. Given that the incidence of Kawasaki Disease stands at around 8 in 100,000 in the United Kingdom (UK),¹⁸ the chances of medical students encountering a patient with this condition are scarce. Further, during the COVID-19 pandemic, clinical placements came to a halt, which left students with no option but to turn to books and alternative resources to supplement their learning. This digital resource was therefore developed as an interactive revision aid that students could undertake in their own time and at their own pace. Following the PACT analysis, specific learning outcomes were developed in the Design phase using measurable verbs such as “explain” and “describe” (see Box 1). These learning outcomes were based upon the lower-order cognitive levels in Bloom’s taxonomy.¹⁹

Box 1 Learning Outcomes of the Digital Educational Resource

In the Development stage, Mayer’s multimedia principles were applied.²⁰ Based on cognitive load theory, the underpinning assumption was that an individual has limited cognitive processing capacity at any given time. Thus, as an asynchronous resource, it provided students with the flexibility and autonomy to work through the resource independently and no training from instructors was required. In line with Mayer’s Pre-Training principle, an “Orientation” page was included at the start to provide learners simple guidance on how to navigate through the resource.²¹ This was followed by a short pre-course quiz that tested learners’ existing understanding of Kawasaki Disease and identified gaps in their knowledge. The resource was designed to take the learner systematically through key aspects of the conditions such as epidemiology, etiology, diagnostic criteria and treatment.

Interactive activities were weaved into the resource to consolidate learning and to keep learners engaged; an example of this was the “drag and drop” exercise as shown in Figure 1. The Coherence Principle states that all non-essential and extraneous material should be eliminated as this can burden the cognitive load. While colorful images and sound effects can be attention-grabbing and appealing, research from controlled laboratory settings has found that excluding unnecessary graphics and text leads to greatest knowledge transfer and retention.²² The way in which this principle was applied is demonstrated in Figure 2, where only salient clinical points on “Incomplete Kawasaki Disease” were included.

Figure 1 An example of an interactive activity.

Figure 2 Coherence principle applied in the resource.

The use of relevant pictures with words can enhance learning and lead to greater learner recall as demonstrated by Mayer’s Multimedia Principle.^20,23 This is demonstrated in Figure 3, where it shows a diagram of a young boy with the diagnostic clinical features of Kawasaki Disease. Learners interacted with the resource by clicking the arrow beneath the diagram, where a more in-depth description of each clinical feature was displayed with the corresponding abnormality lighting up on the body of the child. A final quiz at the end of the resource assessed learning and whether the learning outcomes have been met.

Figure 3 An example of multimedia principle applied in the resource.

Methods

Implementation and Evaluation Methods

The digital resource was implemented by sending the Xerte link via email to a group of medical students of all year groups who had expressed prior interest in undertaking the e-learning tutorial. Students were provided with the opportunity to evaluate the resource and give feedback upon completion.

Sample

Seven undergraduate medical students from a UK university who attended a clinical placement in Pediatrics completed and evaluated the digital resource. To protect the anonymity of participants, no demographic details such as level of study or gender, were gathered as the potential sample was small and such information may have been classed as identifiable. With research showing that 95% of usability problems can be identified in a sample of five to six people,²⁴ a sample size of seven is considered adequate, although a bigger sample size would be required in evaluation research.

Materials and Procedure

Based on the assumption that shorter surveys yield greater response and completion rates, we used a brief evaluation survey that included five questions (see Box 2).²⁵ The questions were informed by Zaharias and Poylymenakou’s²⁶ usability evaluation method, which demonstrated reliability and validity. This method goes beyond merely the cognitive and task-oriented paradigm by focusing on the holistic experience of the user by factoring in affective-emotional aspects of interaction. Questions 1–2 were based on instructional design parameters of “navigation” and “visual design” while questions 3–5 fell into the category of “intrinsic motivation to learn” that can be characterized as an inner drive to carry out an activity, of which the reward is enjoyment from the activity itself.²⁷ Students were asked to rate the extent to which they agreed with these five statements on a scale from 1 to 5 (1 = strongly disagree to 5 = strongly agree).

Box 2 Evaluation Questions

Results

Evaluation of the digital resource was overwhelmingly positive, and all participants agreed or strongly agreed with the five statements. As can be seen in Figure 4, nearly all participants agreed that the resource was easy to navigate, and all were highly satisfied with the way it was presented. Most participants also found the interactive component of the resource to be interesting and beneficial to learning, and that the content was relevant for their level of study. In response to the question as to whether they prefer using such a resource or books and reading on the topic, all showed a preference for the digital resource. The comments left in the free-text mainly related to the interactivity and visual aspect of the resource. Participants commented that the “drag and drop” feature was unclear and suggested that more pictures should be included in the slides.

Figure 4 Results of student feedback.

Discussion

This paper described how instructional models and principle informed the design and development of an e-learning resource for teaching Kawasaki Disease to undergraduate medical students. Preliminary evaluation showed that students who completed the resource found it beneficial to their learning and were highly satisfied with the way it was presented. However, it is important to note some limitations to this small-scale evaluation in that the questions only addressed a few cherry-picked surface level aspects of this digital resource. Thus, further in-depth and robust evaluation should be conducted to ensure quality assurance and improvement.¹³ At its core, evaluation refers to making a value judgment or determining the worth or merit of something.^28,29 In the context of digital education resources, it is essential that medical educators create resources that are educationally beneficial and meaningful for learners.³⁰

Given the nature and purpose of this resource, we argue that further evaluation should not be over complicated or resource intensive.¹³ As a relatively low-stake educational tool, it was designed to aid revision, and to supplement, not replace, existing educational activities within the curriculum (eg, clinical placement, lectures, books). Further evaluation should therefore focus on whether the resource was effective in meeting its intended outcomes (revision), whether it met learners’ needs (eg, learning and confidence) and to establish the suitability of the educational context and pedagogy.³¹

Evaluating the Ability of the Resource to Meet Its Intended Outcomes

Evaluation of learning is typically done via a pre- and post-knowledge test, put together by the resource developers in consultation with subject experts. Evaluation and further development will take into account not just the learners’ opinions and perspectives, but that of other stakeholders such as medical educators, expert clinicians and technology experts.²⁸ Affective elements will be examined through measures of self-efficacy,³² confidence and intrinsic motivation.²⁶ These are well-understood psychological constructs that can be measured through pre-existing measures, adapted for the context. For example, Keller’s³³ Instructional Material Motivation Survey and Motivated Strategies for Learning Questionnaire³⁴ have been used in previous studies. Behavior will be evaluated by asking learners how and when they have used the resource through a range of open-ended questions.

Evaluating the Ability of the Resource to Meet Learners’ Needs

The resource is designed with undergraduate students in mind at all levels of study. However, as the demographics of the responders was unknown, it is possible that certain year groups were overrepresented. Further evaluation will therefore consider the views of other stakeholders regarding the suitability of learning outcomes, content, and activities and if they are likely to meet the needs of learners at all levels of study, or whether the resource is more suitable for learners who have already attended a clinical placement in pediatrics.

Evaluating the Educational Context

The educational context was digital and for this reason evaluation focused on criteria related to design, content, usability and functionality. Further evaluation will be more comprehensive, yet straightforward, by using pre-existing models such as the Learning Object Review Instrument (LORI), which comprises nine items: content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, reusability and standards compliance.¹³ While ADDIE was used in a linear fashion in the production of this resource, some favor a more iterative approach by placing the evaluation component at the centre of the model and collecting regular formative evaluation after each step in the process. An advantage of this is that it allows the designer to identify strengths and weaknesses and make amendments throughout the process, which can lead to continuous improvement of the educational tool.³⁵ There is no standard number of formative evaluations that should take place and evaluation will be ongoing to ensure it continues to meet learners’ needs.³⁶