Back to Journals » Medical Devices: Evidence and Research » Volume 11

Design and preliminary validation of a mobile application-based expert system to facilitate repair of medical equipment in resource-limited health settings

Authors Wong AL , Lacob KM, Wilson MG, Zwolski SM, Acharya S

Received 17 January 2018

Accepted for publication 1 March 2018

Published 16 May 2018 Volume 2018:11 Pages 157—169

DOI https://doi.org/10.2147/MDER.S162854

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 4

Editor who approved publication: Dr Scott Fraser



Alison L Wong,1,2 Kelly M Lacob,1 Madeline G Wilson,1 Stacie M Zwolski,1 Soumyadipta Acharya1

1Center for Bioengineering, Innovation and Design, Johns Hopkins University, Baltimore, MD, USA; 2Division of Plastic Surgery, Dalhousie University, Halifax, NS, Canada

Background: One of the greatest barriers to safe surgery is the availability of functional biomedical equipment. Biomedical technicians play a major role in ensuring that equipment is functional. Following in-field observations and an online survey, a mobile application was developed to aid technicians in troubleshooting biomedical equipment. It was hypothesized that this application could be used to aid technicians in equipment repair, as modeled by repair of a pulse oximeter.
Methods: To identify specific barriers to equipment repair and maintenance for biomedical technicians, an online survey was conducted to determine current practices and challenges. These findings were used to guide the development of a mobile application system that guides technicians through maintenance and repair tasks. A convenience sample of technicians in Ethiopia tested the application using a broken pulse oximeter task and following this completed usability and content validity surveys.
Results: Fifty-three technicians from 13 countries responded to the initial survey. The results of the survey showed that technicians find equipment manuals most useful, but these are not easily accessible. Many do not know how to or are uncomfortable reaching out to human resources. Thirty-three technicians completed the broken pulse oximeter task using the application. All were able to appropriately identify and repair the equipment, and post-task surveys of usability and content validity demonstrated highly positive scores (Agree to Strongly Agree) on both scales.
Discussion: This research demonstrates the need for improved access to resources for technicians and shows that a mobile application can be used to address a gap in the access to knowledge and resources in low- and middle-income countries. Further research will include prospective studies to determine the impact of an application on the availability of functional equipment in a hospital and the effect on the provision and safety of surgical care.

Keywords: medical apps, global health, mHealth, repair system, pulse oximeter

Plain language summary

Surgery cannot be performed without the necessary equipment. Biomedical technicians are the workforce behind maintenance and repair of this equipment, but they are lacking in numbers and resources in most low- and middle-income countries. This research investigated whether a mobile application could be used to help biomedical technicians troubleshoot and solve equipment malfunctions by providing guides and connecting them to information. The application design and platform were based on a survey of technicians. The application was then tested by a group of technicians to identify and fix a common piece of medical equipment (a portable oxygen monitor). Technicians were able to successfully use the application and found it helpful. Development is in progress, and prospective studies are being planned to see what effect such an application could have on the availability of equipment and the delivery of surgery.

Introduction

The World Health Organization (WHO) estimates that 50%–70% of all biomedical equipment in low- and middle-income countries (LMICs) is partially or completely nonfunctional.1,2 This challenge is one of the underlying problems contributing to the lack of access to surgical care; it has been reported that 1.5 million deaths could be prevented annually at existing hospitals by providing essential surgical care which is currently not available.35 Unavailability of biomedical equipment results in delays of surgery due to fewer available operating rooms and referrals to higher-level hospitals. Even when the necessary equipment is present, it might be only partially functioning and lead to a higher risk of complications or death.69

The lack of functional biomedical equipment is due to a number of factors, including that much of the equipment in developing countries is donated, arriving without manuals or service contracts; most is designed for Western markets and not well-suited to environmental conditions in low-resource settings; and there is a lack of well-trained biomedical technicians in developing countries to address these problems.6,1012 Most of the hospital-affiliated technicians in LMICs do not have formal training in biomedical equipment repair; often, they have high school level education, and some experience repairing bikes, air conditioners, or other mechanical, nonmedical apparatuses, but attempt to self-train to cater to hospitals’ needs.10,13 Research from Engineering World Health (EWH), a nonprofit organization that mobilizes the biomedical engineering community to improve health care in the developing world, has demonstrated that 66% of all equipment failures can be fixed with simple, generalizable technician skills, without the need for spare parts.1,2,13 The underlying issue is that many of these technicians have not received training on a systematic approach to problem solving.1,2 Further, EWH’s research shows that having access to manuals or noncomplicated flowcharts significantly increases successful work order completion rates by up to 66%,1 as does having another contact that the technician feels comfortable calling for an on-the-phone consult.2

To address these gaps, we developed a mobile application-based expert system to aid biomedical technicians in troubleshooting, repairing, and maintaining biomedical equipment. We designed the application based on in-field observations and an online survey of current technician practices and resources. A prototype of the application was then brought to a group of biomedical technicians to use to complete a standardized broken pulse oximeter repair. It was hypothesized that this application could be used to aid in equipment repair, with appropriate usability and content validity.

Methods

Preliminary survey of technicians and user needs

To determine existing practices and gaps in technician knowledge and resources, an electronic survey regarding day-to-day practices, resources, and comfort with various repairs was constructed using an online survey building tool (Qualtrics, Provo, UT, USA). Questions were chosen based on gaps identified from research and refined through a process of debate and consensus among the research team. The survey was edited by local technicians and engineers to ensure appropriate wording and content. The final survey consisted of 20 items with a mixture of question types (Figure S1A).

Distribution to biomedical technicians and engineers in different countries was done through convenience sampling through contacts established through the School of Engineering, School of Medicine, and School of Public Health and Johns Hopkins University. Responses were collected during a 2-month period. All participants were over the age of 18, and written informed consent was obtained electronically prior to completing any survey questions. The survey was approved by Johns Hopkins University Institutional Review Board.

Identifying user needs

Based on ethnographic research, literature review, and results from the survey (presented in detail in the “Results” section), we identified the following high-level user needs that should guide the design of any solution to this problem: 1) On-the-job, contextual guidance during troubleshooting and repairing; 2) Assistance with a wide variety of equipment of different types, and from different manufacturers; 3) Ability to consult with live experts in biomedical equipment repair when needed; 4) Skill training; 5) Wide scope of resources; 6) Data collection and analysis; and 7) No additional investment in additional telecommunication and computational equipment.

Application design

Based on the user needs, our team developed a mobile application-based expert system to address these gaps. The application was developed using the Android Java development kit and can be operated on any Android device with operating system 4.0 or above. The initial prototype application included troubleshooting guides and resource libraries of general equipment documents from the WHO, EWH, and Frank’s Hospital Workshop. The pieces of equipment chosen were the 16 pieces of biomedical equipment that have been deemed “essential” by the WHO, and/or “important and commonly broken” by EWH.1,8,14 The troubleshooting guides were adapted from EWH’s device troubleshooting flowcharts into simple, discrete steps. When a user encounters a piece of equipment that is nonfunctional, he or she can open the application and select the device from a menu. The user is then navigated through a decision tree via “yes/no” answers. The flowcharts contain links to skill tutorials, where appropriate, and a function that connects the user with a select network of skilled individuals who would be able to guide the technician through a more advanced repair (Figure 1). All resources and experts are peer-reviewed and rated, and the system does not require Internet access to work, only to sync with new updates. Each session is logged so that metrics regarding equipment and repairs can be aggregated and further analyzed.

Figure 1 System architecture of our mobile application-based expert system.

Notes: When the user encounters a piece of equipment that is in need of maintenance or repair, he or she can use the mobile application that contains troubleshooting guides and general manuals, all available without the need for Internet connection. Working through the guides generates a report that is saved both locally and, when connected to data, synced with a database accessible by the hospital and Ministry of Health. If the user is unable to repair the piece of equipment with the guide alone, he or she may connect to an expert. This expert is chosen by matching the problem with documented skill sets. The initial report is shared with the expert and updated to include the additional steps that the local user is guided through. In cases where there is a centralized repository for replacement parts, the repository can be notified about the need for spare parts directly from the application.

Usability testing and content validity of the application

For pilot usability testing, a prototype was developed that included the decision tree guide for repair of pulse oximeters, basic skill tutorials, and links to generic equipment manuals. This application was brought to Ethiopia where testing was conducted with participants at hospitals and technician training programs through convenience sampling with help from our in-country partner organization (Jhpiego). Inclusion criteria were position as a technician, engineer, technician student, or technician instructor; exclusion criteria were inability to speak or read English. We used a combination of modified preverified quantitative tools to test formative usability, as well as a selection of qualitative, open-ended questions to probe reactions to the application features. All participants were given a brief introduction and were then given 10 minutes to familiarize themselves with the mobile application. The amount of guidance to users when they interacted with our application’s platform was minimized to reduce bias. Participants were then given a pulse oximeter that had either 1) an occluded light sensor or 2) a frayed wire in the probe cable. Using the application, they were guided through troubleshooting the problem. Following this exercise, they completed surveys on the usability of the application and its content and shared qualitative feedback.

For usability testing, we used a modified version of Usability.gov’s tool designed for usability testing of an electronic system and applications.15 This tool uses a combination of positive and negative statements to evaluate the user’s high-level perceptions of the application, which are strongly indicative of the user’s willingness to engage further.16 Each statement was scored on a 5-point scale ranging from Strongly Disagree to Strongly Agree (Figure S1B).

Content validity (the user’s perception on the accuracy and usefulness of the content) was assessed using a tool adapted from a review study by Hoehle and Venkatesh.17 The Content Validity Check examines slightly more granular, actionable elements, surrounding the various features of the application and their perceived value. Each statement was scored on a 5-point scale ranging from Strongly Disagree to Strongly Agree (Figure S1C).

Statistical analysis

For the online survey, descriptive statistics of the responses were calculated. For ease of interpretation of the usability and content validity test responses, the total percentage of “agree” and “strongly agree” responses were combined, as were the “disagree” and “strongly disagree”. The responses to resources to repair equipment were compared using a Kruskall–Wallace test. Trends in positive and negative responses to the usability and content validity survey were determined by a Fisher’s exact test. All statistical analysis was done using MatLab (MathWorks Inc., Natick, MA, USA).

Results

Online survey of technicians

Demographics

A total of 53 respondents completed the survey. A majority (37) of these were from Ethiopia. Other countries included Bangladesh (1), Belize (1), Ecuador (1), Ghana (3), Haiti (1), Ireland (1), Kenya (1), Nigeria (1), Oman (1), Rwanda (3), Uganda (1), and the United States (1). Of the respondents, 38% were engineers, 28% technicians, 21% students, 11% teachers, and 4% medical equipment managers. One respondent self-identified as both a technician and teacher. The students and teachers were primarily from biomedical colleges or institutions in Ethiopia that were recently instituted to train future technicians or engineers. The engineers in this study are analogous to a high-level technician. Most participants (85%) received a college education. A majority (89%) have a mobile phone, with Android being the most common type. Most (72%) also reported that their hospital has reliable Internet connectivity.

Current work as a technician

Respondents were most comfortable diagnosing equipment problems and least comfortable validating that the problem had been fixed (Figure 2). Diagnosing the problem was also found to take the most time, followed by finding resources, and implementing the repair. Interestingly, validating that the problem was fixed was reported as taking the least amount of time (Figure 2).

Figure 2 Current work as a technician.

Notes: (A) Relative comfort level performing common tasks; (B) relative time needed for common tasks; (C) pieces of equipment that technicians are not comfortable maintaining and repairing.

Abbreviation: ECG, electrocardiogram.

Resources to repair equipment

There was a significant difference in the resources commonly used by respondents (Figure 3; P<0.05). Post hoc analysis showed that equipment manuals and the Internet were more commonly used than any of the other resources (P<0.05).

Figure 3 Resources most commonly used to repair equipment.

Notes: There were significant differences between what resources were commonly used (P<0.05). Normalized responses to the survey questions showing median responses and 95% confidence intervals. Different letters signify significant differences (P<0.05).

Motivation to repair equipment

Seventy-five percent of technicians responded that they feel their job is important to improve patient outcomes. With regard to compensation, a majority (58%) are paid on a fixed salary.

Application prototype testing

Usability and content validity

Thirty-four participants (17 biomedical engineers, 6 technicians, and 11 technician students) completed the broken pulse oximeter task, and all were able to appropriately identify and repair the equipment using the application (Figure 5). The post-task usability survey was completed by all 34 individuals, and the content validity survey was completed by 33 (16 biomedical engineers, 6 technicians, and 10 technician students), 1 engineer did not complete the questionnaire as he/she had to leave to attend to a job. All participants were from Ethiopia. Figure 4 shows a sample use case of the application. Fisher’s exact test showed that there were significantly more positive than negative responses for both usability and content validity (P<0.05).

Figure 4 Representative screen shots of the pulse oximeter troubleshooting guide, buttons to access resources contained within the application or call for external help, and the troubleshooting report.

Figure 5 Application testing (A) usability and (B) content validity surveys.

Notes: Asterisks denote statistical significance (P<0.05) between strongly agree/agree responses and strongly disagree/disagree responses as determined by a Fisher’s exact test.

During open-ended discussion, users reported that the primary benefits of the application would be time saved identifying the problem and fixing equipment, decreased likelihood of introducing new problems during repair, increased confidence, and an increased number of pieces of equipment that can be repaired without requiring spare parts.

Discussion

A large proportion of biomedical equipment in LMICs is completely or partially nonfunctional and is one of the factors leading to compromised delivery of surgical care.13,1820 We hypothesized that a mobile application could be used to help troubleshoot and repair equipment.

The development was guided by an online survey of technicians and engineers. It was designed to help users identify the equipment problem by guiding them through the troubleshooting process step-by-step, using simplified language, pictures, and diagrams. If the user is still unable to resolve the issue, the application can connect the user in real time with an expert. While performing these services, our application generates reports, and this data can be aggregated and analyzed for use in equipment-related decision-making. Preliminary testing of a prototype of the application supported our hypothesis, as users could use it to identify and repair a standardized broken pulse oximeter.

The WHO and the Lancet Commission have both identified the lack of functional biomedical equipment and the skill capacity of local biomedical technicians as barriers to safe surgery, but to date there has been little research to elucidate the underlying issues and opportunities for innovation.3,19 While many countries are starting to recognize the importance of biomedical technicians and the need for formal training institutions,2,10,13,21 our survey revealed the need to also increase access to on-the-job support and facilitate interactions between technicians. Our survey also indicated that access to manuals is considered the most important resource that aids in equipment repair, but these are difficult to access. Therefore, there is a need to increase ease of access to preverified materials.

The mobile platform of our application was chosen as the majority of technicians who responded to the survey stated that they have a mobile phone, with the predominant operating system being Android. Information is stored locally on the phone, eliminating the need for consistent Internet access, but is synced with a database when a connection is available. The troubleshooting guides focus on not only identifying and fixing the problem with equipment, but validating the repair, as the results from our survey showed that technicians are least comfortable with these steps and that they are the most time consuming.

Initial testing of the application prototype revealed that users believed that it would speed up the diagnosis process by helping them perform recommended checks for common problems, and that it would decrease the chance that they could make errors and introduce new problems in the process. Nearly all users claimed they would use such a tool frequently. Users particularly liked the easy access to preverified resources, since it decreased the amount of time spent trying to find applicable aids. Another positively reviewed feature was how the application creates a record of the repair steps because it both facilitates collaboration with colleagues and can be used as evidence of effort.

Limitations

Due to the quasi-experimental nature of our study, there are several limitations. First, the sample size is relatively small and predominantly from a single geographic location. Also, the application is currently only available in English; therefore, we had to limit study inclusion eligibility to English-speaking respondents. This meant we could not include many respondents who were older, from more rural backgrounds, or less formally trained. It is possible that these end-users would also have struggled more with use of a mobile application, and it is possible that they may have had different perceptions toward content features. There are inherent problems with survey research, which can lead to result biases, and this may have been further complicated by the design of our usability study, since we provided a certain amount of guidance to users when they interacted with our application’s platform and used it to repair the broken pulse oximeters. Using pulse oximeters as our initial test case allowed us to test general perceptions of such a system’s usefulness, but further research is needed to know if the usability is scalable to other pieces of equipment.

Potential limitations to the application are that it is currently only available as a mobile application on Android devices. Most survey respondents stated that they had a phone with this operating system and agreed that a mobile application increases portability, but a web-based platform would ensure the greatest access as not all end-users have smartphones, and a larger screen would also facilitate clearer viewing. Another limitation to the application is that it only includes generic equipment guides.2 In the future, there is the possibility to expand to more specific guides and manuals if a clear need is demonstrated.

Future directions

Aside from the front-end user interface, our application’s back-end data analytics have the potential to provide unique and powerful information regarding the availability of biomedical equipment. Ministries of Health, hospitals, and international organizations frequently cite the lack of available data on current practice as a significant barrier to the objective decision-making and implementation of new initiatives.3 There is currently a paucity of data regarding the status of donated and purchased biomedical equipment and how this affects the delivery of health care.3 Since our application is designed to capture inventory and work order data through the initial input screens, and passively collects data about skills and resources utilized and work order outcomes throughout use sessions, it will help fill this information gap. Data can be used to inform policy decisions on equipment management, to prioritize equipment and spare part procurement decisions, or to suggest needed training curriculums.

Currently, only a limited number of troubleshooting guides have been developed, but the system uses a simple decision tree algorithm that can be easily scaled. Further development and curation of guides is planned to proceed as part of an open-source movement with both academic and industry partners. Our application could also be an affordable, practical service alternative for medical device companies that have sold or donated equipment to hospitals in developing countries but do not have sufficient technical support on-the-ground, while also providing them access to data about their local market shares and customer experiences, among other benefits. Ideally, the application could be included with new equipment with a physical card for reminding users and technicians to use it for troubleshooting.

Development and refinement of the mobile application and expert system are ongoing. Though our initial application was designed on an Android platform, we plan to expand it to other platforms. Our goal is to conduct a prospective study to directly assess how our application can help technicians and lay-people work through problems with broken equipment, compared to traditional practice. We will also aim to definitively capture how the availability of functional equipment relates to the number of surgeries canceled, delayed, or referred, and how the safety of surgery is impacted.

Moving forward, we wish to share the application widely, but currently it is not yet ready for beta release. For more information, please contact the senior author (Dr. Acharya).

Acknowledgment

The authors thank Dr. Tigistu Adamu and colleagues from Jhpiego and Jhpiego Ethiopia; Felege Hiwot, Debre Markos; Finot Selam, St. Paul’s, Black Lion, and Menelik Hospitals; Debre Markos Biomedical College and Addis Ababa Institute of Technology; the Ethiopian Federal Ministry of Health; Engineering World Health; Katie Kloss; Dr. James Cobey; Dr. Lew Schon; and Dr. Barclay Stewart.

This research was funded in part by GE foundation (“Base of Pyramid Technology for Saving Mothers and Children” #116873).

Disclosure

The authors report no conflicts of interest in this work.

References

1.

Malkin R, Keane A. Evidence-based approach to the maintenance of laboratory and medical equipment in resource-poor settings. Med Biol Eng Comput. 2010;48(7):721–726.

2.

Malkin RA, Whittle C. Biomedical equipment technician capacity building using a unique evidence-based curriculum improves healthcare. J Clin Engineering. 2014;39(1):37–44.

3.

Meara JG, Leather AJM, Hagander L, et al. Global Surgery 2030: evidence and solutions for achieving health, welfare, and economic development. Lancet. 2015;386(9993):569–624.

4.

Ulisubisya MM. The critical condition of anaesthesia provision in low-income and middle-income countries. Lancet Glob Health. 2016;4(9):e597–e598.

5.

Stewart BT, Quansah R, Gyedu A, Ankomah J, Donkor P, Mock C. Strategic assessment of trauma care capacity in Ghana. World J Surg. 2015;39(10):2428–2440.

6.

Spiegel DA, Choo S, Cherian M, et al. Quantifying surgical and anesthetic availability at primary health facilities in Mongolia. World J Surg. 2010;35(2):272–279.

7.

Wong EG, Gupta S, Deckelbaum DL, Razek T, Kushner AL. Prioritizing injury care: a review of trauma capacity in low and middle-income countries. J Surg Res. 2015;193(1):217–222.

8.

Wong EG, Gupta S, Deckelbaum DL, et al. The International Assessment of Capacity for Trauma (INTACT): an index for trauma capacity in low-income countries. J Surg Res. 2014;190(2):522–527.

9.

Lynch PK. Is “old school” equipment testing the way to go in developing countries? Biomed Instrum Technol. 2014;48(2):139–141.

10.

Mullally S, Bbuku T, Musonda G. Medical equipment maintenance personnel and training in Zambia. IFMBE Proceed. 2013;39(Chapter 197):750–753.

11.

Wesson HK, Stevens KA, Bachani AM, et al. Trauma systems in Kenya: a qualitative analysis at the district level. Qual Health Res. 2015;25(5):589–599.

12.

O’Hara NN. Is safe surgery possible when resources are scarce? BMJ Qual Saf. 2015;24(7):432–434.

13.

Whittle C, Malkin R. Capacity building for medical equipment technicians improves healthcare. J Clin Engineering. 2014;39(3):142–147.

14.

WHO. WHO Generic Essential Emergency Equipment List. Geneva; 2012:1–2.

15.

U.S. Department of Health & Human Services. System Usability Scale (SUS). Usability.gov. 2013. Available from: http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html. Accessed April 6, 2016.

16.

Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4(3):114–123.

17.

Hoehle H, Venkatesh V. Mobile application usability: conceptualization and instrument development. Mis Quarterly. 2015;39(2):435–472.

18.

Meara JG, Greenberg SLM. The Lancet Commission on Global Surgery Global surgery 2030: evidence and solutions for achieving health, welfare and economic development. Surgery. 2015;157(5):834–835.

19.

WHO | WHA reaches agreement on polio, International Health Regulations and strengthening surgical care. Geneva: WHO; 2015. Available from: http://www.who.int/mediacentre/news/releases/2015/wha-22-may-2015/en/. Accessed July 20, 2016.

20.

Debas HT, Donkor P, Gawande A, Jamison DT, Kruk ME, Mock CN, editors. Disease Control Priorities. Vol 1. 3rd ed. Washington, DC: World Bank Publications; 2015.

21.

Howitt P, Darzi A, Yang GZ, et al. Technologies for global health. Lancet. 2012;380(9840):507–535.

Figure S1 Surveys used in this research.

Notes: (A) Biomedical technician practices and resources; (B) usability of application; and (C) content validity of application.

Creative Commons License © 2018 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.