Google Glass-Supported Cooperative Training for Health Professionals: A Case Study Based on Using Remote Desktop Virtual Support

Introduction

Health professionals’ ability to collaborate with other experts over a network has become more important than ever to improve quality of care and maintain high standard practice.¹ Expert knowledge is needed to develop an optimal solution for problems and prevent unnecessary errors, especially in care environments, where a high level of clinical expertise is costly and rare.² Transfer of knowledge to a large proportion of inexperienced health providers is key to successful remote collaboration. An efficient approach to achieve this goal is to improve both the availability and accessibility of experts to trainees, at scale.

Systems for remote collaboration enable remote workers to work together as if they are present at the site. Recently, people in the industry have been interested in synchronous remote collaboration systems that support collaboration between remote experts and local workers.³ There is a system that monitors local workers’ real-time videos, facial expressions, and gazes with the help of a remote expert.⁴ Few companies provide commercial tools through which real-time videos of workers with remote expertise can be shared over a high-speed network.⁵

Currently, partly due to the COVID-19 pandemic, advancements in computing processors, graphic processors, form factors, and battery life to overcome existing barriers related to remote collaboration, are being accelerated. A previous study suggests that wearable devices can help health professionals solve problems more quickly and safely via knowledge transfer using direct verbal and visual information.⁶ The first commercial smart glass was Google Glass. It was created for the general public, but due to functional limitations and lack of public acceptance of smart glasses, it is now being developed for specialized target groups for business, industrial, medical, and military uses. We can increase the efficiency of work by using the display, communication module, and computational power of smart glasses.

There are many different types of smart glasses such the Glass Enterprise Edition 2 (Glass EE2), Vuzix blade, Echo frame, HoloLense, Magic Leap One, Nreal Light, BT-300, and SmartEyeglass, however, their features vary. Though the Glass EE2 and Vuzix Blade are both equipped with a monocular display, the Glass EE2, has a small display on the top of the right-side eyeglasses, so that the screen does not obstruct the view, however, the Blade’s display is on the eyeglass’ glass. Realwear-HMT1 is a video-see-through Head Mounted Display (VST-HMD), which is specialized for manufacturing fields such as factories, has dustproof and waterproof functions, and operates entirely with voice recognition. The Nreal Light, BT-300, and SmartEyeglass have binocular displays. HoloLens 2 and Magic Leap One recognize space and visualize information in a three-dimensional space, unlike the aforementioned products. Additionally, it is capable of motion and voice recognition and is the most advanced augmented reality display device. The Echo Frame released by Amazon is unique as it does not have a display and is characterized by using various functions by communicating with Alexa through voice recognition. Details are summarized in (Appendix 1).

The usability and feasibility of smart glasses in education and health care areas, such as reading health data, telemonitoring, video recording, documentation, and education, have been extensively examined.^7,8 Google Glass, in particular, has been widely adopted in surgical operations, and previous reviews have identified its applicability as a videography device, monitoring device, and navigation display.^9,10 Smart glasses adapted to medical settings have been used to offer a first-person point-of-view, mostly in motionless environments such as surgical settings. Particularly, using video streaming is appropriate not only for telemedicine but also for medical documentation, patient recognition, and training students.^11,12 Along with state-of-art technology to deliver precise information to smart glasses, remote experts need to consider clinical circumstances. Compared to the traditional in-person observation, smart glasses have greater efficacy in clinical usage due to the autonomic and unobstructive data collection.¹³ A correct understanding of the clinical situation can facilitate seamless interactions among professionals, and the need for high-end technology for uninterrupted interactions has never been higher, especially since the outbreak of COVID-19.¹⁴ Smart glasses are small computers, and a video camera records what the wearer is seeing, thereby offering video streaming to remote viewers.¹⁵ This video streaming feature is suitable for clinical settings where supervision from more experienced experts is closely related to patient safety. Furthermore, the previous qualitative interview revealed health professionals’ expectations regarding smart glasses for effective real-time information sharing, so they can discuss patients.¹⁶ However, the usefulness of smart glasses in the context of telemedicine is not well established. Audio and video streaming among health providers enables remote collaboration, and it is important to identify the factors influencing the process of information exchange.

Despite the benefits of streaming real-time videos, such as reduction of time and cost for remote observation or supervision, concerns regarding its actual usability in clinical practice remain. Previous studies have indicated that display resolution, field of view (FOV), internet connection, and image distortion determine the quality of the video streamed from smart glasses.¹⁷ In particular, wearers have focused on minimizing head movements to reduce motion blur in videos.¹⁷ However, this cannot be the case for certain clinical practices such as emergency units; thus, a comprehensive system evaluation would ensure the active application of smart glasses in the clinical environment.

Objectives

The present study aimed to assess the feasibility of a desktop user interface to monitor remote collaboration systems using the latest Google Glass (Glass EE 2) and to determine whether real-time video and audio provided via Glass EE2 is helpful, informative, and provides adequate information needed in emergency care settings. This study evaluated the usability of virtual support on remote desktops. The participants were the remote supporters who did not wear Google Glasses but watched and interpreted video streams, on a desktop program, that were generated by the trainee’s Glass EE2 and provided audio feedback to the trainee in real-time.

Materials and Methods

Setup

Hereinafter, we refer to the expert who sits in front of a desktop monitor as the “supporter” and the worker who works in an emergency room wearing a Glass EE2 as the “trainee.” Our system was comprised a supporter-side desktop system, trainee-side wearable system, and network server. The supporter-side desktop system consisted of a typical desktop computer (ie, with a monitor, a mouse, and a keyboard), a headset (audio input/output), and software tools. The software tools were a remote video conferencing application program based on the App-RTC, to monitor the trainee, and an image and text message transmission program for the trainee-side wearable system (Figure 1).

Figure 1 Real-time video captured via the Google Glass EE2 is delivered to the remote supporters’ desktop screen.

The trainer-side wearable system comprised Glass EE 2, Bluetooth earphones, and a small mirror that is driven by a remote app we made. Bluetooth earphones were used because Google Glasses have built-in speakers, due to which, the sound from another person’s Glass EE2 could confuse the trainees. A software for the video and audio communication, based on the App-RTC, and for receiving images and text messages was implemented. The supporter-side desktop system (display resolution 1920×1080) received the trainee’s video and voice and sent voice commands, images, and text messages for help in real-time. The trainee-side wearable system received voice commands, images, and text messages from the supporter, and sent video and audio in real-time.

The trainee-side wearable system consisted of the Glass EE 2, Bluetooth earphones, and a small mirror attached to the Glass EE 2. The Glass EE 2 has a front camera (up to 1080p, 30f/s), Bluetooth-enabled audio input/output, and a touchpad. The trainee-side wearable system transmitted real-time video captured from the Glass EE 2 camera and audio through Glass EE 2’s microphone and received audio and image files transmitted from the supporter-side system. The Glass EE 2, which runs on Android Oreo 8.1 (API Level 27), with the firmware version OPM1.200625.001 was used. Android Studio and Android SDK8.1 (API 27) were used as development tools.

We also developed a software tool that can transmit images and texts that are required for training. We implemented one-to-one video communication by accessing the web RTC server using Google Chrome. Google provides the App-RTC (https://appr.tc/) server for free with an android sample source code (https://developers.google.com/glass-enterprise/samples/code-samples). The App-RTC supports video calls with 640×480 video resolution on Android apps using Google’s Chrome browser. Text and images were transmitted by a direct connection between the desktop and Glass EE2 without the data going through the server.

We developed a software for delivering images and text messages from supporter-side system to trainee-side system. We used the Unity3D for developing a desktop-side application as shown in Figure 2. The application shows image arrays and the image that the supporter clicks goes to the trainee’s display. Additionally, there is a text box for messages to be sent from the supporter to the trainee. This application was developed using UDP (User Datagram Protocol). On the trainee’s side, the images and text messages were displayed on the Glass EE2.

Figure 2 A mirror attached to the Google Glass to increase the FOV.

An issue with the Glass EE2 is that a blind spot occurs because the camera faces the front, which makes it impossible to capture an image from the bottom. In general, nurses stand and handle devices on a desk, however, since the camera faces the front, it is impossible to capture a video of things on the desk. To solve this problem, images were transmitted to the navigator by attaching a mirror to the Glass EE2, which transmitted both the front side and lower part simultaneously. As shown in Figure 2, the mirror attached to the Glass EE2 was made of a light plastic material. The direction of the mirror was adjusted by the trainee by rotating it. The navigator received a video and the top half of the video shows the bottom side and the bottom half of the video shows the front side view, which allowed them to observe the trainee’s working situation.

Sample and Setting

Data were collected from 31 participants who attended supporter-trainee simulations and used two Glass EE2s to take part in the simulation program as both bedside trainees (glass wearers) and remote supporters. All participants completed a 15-hour simulation based on a scenario of patients with arrhythmia in an emergency unit. During the simulation, students were required to accomplish tasks related to each patient’s condition, including patient monitoring, EKG lead application, medication, and cardioversion (Figure 3). Students were required to take turns being bedside trainees and remote supporters. The inclusion criteria for study participants were 1) nursing students who participated in team-based simulation of an emergency nursing scenario involving interprofessional interaction, 2) provided informed consent to participate, and 3) who had taken on the role of desktop remote supporters within the scenario. This study obtained the ethics approval from the Institutional Review Board of Mokpo National University in Korea (No. MNUIRB-201,006-SB-011-02) in accordance with the Declaration of Helsinki.

Figure 3 Tasks that were required by the wearers and their range of movement during simulation.

Data Collection

A structured questionnaire comprising the following eight essay questions was used: 1) Were you able to figure out the clinical situation in general? If not, how can it be improved? 2) Were you able to determine the condition of the patients? If not, how can it be improved? 3) Were you able to figure out the trainees’ hand movements? If not, how can it be improved? 4) Were you able to determine what the trainees were doing? If not, how can it be improved? 5) Was the mirror-reflected display helpful to figure out what was happening in the field? 6) Did you have any difficulties because of the unsteady camera? 7) How did you find the display resolution? 8) How was the sound transmission? Was it sufficient to facilitate flawless communication?

Statistical Analysis

SPSS (version 25.0) was used to compute descriptive statistics, and values were expressed as numbers, percentages, means, and standard deviations.

Results

Study Participants

The participants’ mean age was 23.94 years and approximately 74% of them were female. Most participants were candidates for graduation (93.5%), and all the participants had previous experience with simulation-based training. Table 1 presents the general characteristics of the participants.

Table 1 General Characteristics of the Study Participants (n=31)

Identification of the Clinical Situation

Table 2 provides a summary of responses from participants to the essay questions. Twenty-seven (out of 32; 84%) reported that it was not easy to identify the overall situation due to narrow FOV (47%), motion blur (22%), and mirror mode of display (16%) (Figure 4).

Table 2 Summary of Responses from Participants

Figure 4 Ease of the overall situation identification.

Figure 5 shows examples of restrictions on situation identification during video streaming. These included limited information and a narrow FOV due to the short-distanced filming. This provided a partial view because of which the video captured only a small part of the manikin. In the application of devices, the mirror mode caused confusion, resulting in inaccurate directions by the supervisors to trainees wearing the smart glasses. Lastly, motion blur was a predominant problem; frequent relocation of smart glasses wearers and an unstable network made it difficult to identify the situation.

Figure 5 Examples of restrictions related to video streaming: limited information, narrow field of view, and mirror mode.

Monitoring of Patients

Less than one-third of the participants (n=9, 28%) responded easily to the system for monitoring the patient’s condition. Limited FOV (n=20, 63%) and network issues (n=3, 9%) were requested to be considered for further improvement (Figure 6).

Figure 6 Ease of identifying patients’ conditions in general.

Monitoring of Trainees

Seven participants (22%) expressed the usefulness of mirror-reflected screens. Twenty-two participants (69%) perceived that monitoring trainees was not easy due to limited FOV and motion blur (Figure 7), and most participants experienced difficulty identifying trainees’ hand movements (n=30, 94%).

Figure 7 Ease of monitoring smart glass users’ performance.

Figure 8 shows an example of the systemic limitation of desktop-side video streaming. The short distance filming could capture only one hand of the trainee, which made it difficult to identify the performance of the trainees. In addition, the mirror mode increased confusion, hence advice for the decision-making process was only tentative.

Figure 8 Examples of mirror mode and limited FOV.

Quality of Image (Screen Resolution)

The majority of the participants expressed satisfaction with the screen resolution; 78% (n=25) responded that they had fair to good experience with video streaming provided via a desktop monitor (Figure 9).

Figure 9 Perceived screen resolution quality of the desktop computer.

The participants also reported that the unstable network led to low screen resolution, which made it difficult to identify patient conditions in the scenario situation.

Motion blur was experienced by 84% (n=27) of the participants, and 25% (n=5) reported irritation caused by unstable video streaming (Figure 10). Participants also reported that motion blur worsened with trainees’ moving speed and activity intensity, especially their head motions.

Figure 10 Difficulties during physical operation and on-the-go.

Auditory Communication

The study participants reported that the audio system did not work smoothly at times, and further improvements were requested for Bluetooth (n=11, 34%) and earphones (n=9, 28%) (Figure 11).

Figure 11 Perceived stability of sound transmission.

Discussion

The present study evaluated the usability of virtual support on remote desktops for collaboration and investigated whether video streaming via Glass EE2 could assist remote supporters to monitor clinical situations. Most monocular optical see-through smart glasses or AR glasses have similar hardware specifications (refer to Appendix 1), produce comparable video streams, and technical limitations, reported in this study that was conducted with Glass EE2.

Our study specifically aimed to evaluate cooperative training, supported by Glass EE2, on the remote supporter’s end (ie, a desktop program user, rather than an app on the Google Glass). Boillat et al’s¹⁸ work reviewed studies and used cases and scenarios regarding those wore smart-glasses and used an app on it. Ponce et al,¹⁹ Hashimoto et al,²⁰ and Datta et al’s²¹ studies are similar to our cooperative training, since they included trainees who wore smart-glasses and remote supporters who watched what the trainees were viewing. However, these studies had some limitations related to using the Google Glass, such as limited FOV and viewing angles, these were revisited by our study, using the latest Google Glass. In an attempt to provide more usable user interface and comprehensible video streams on the remote supporter’s end, we attached a customized mirror to the Glass EE2.

Our remote collaboration system’s current configuration demonstrated some deficiencies regarding streaming performance and video quality, in terms of both resolution and continuity, for the remote supporters. This study’s findings, however, identified a few limitations in the current model of Google Glass related to its clinical use. First, the videos generated by Google Glasses, even before it is streamed, are not of sufficient quality to be used for clinical purposes. In many cases, due to Google Glasses’ limited FOV, essential parts of the clinical scene that remote supporters need to interpret are either cut-off or not recorded. The mediocre quality videos are further degraded due to the wearer’s rapid head movements (ie, motion blur in the video), therefore, interpreting such videos would require the experts to meticulously analyze them. Second, the streaming performance does not meet the remote supporter’s expectation (ie, delay, buffering, frozen frames, and dropped frames).

We intentionally designed interactions of the trainees who wear Glass EE2 to be minimal, non-obtrusive and hands-free. Trainees (ie, nursing students) could focus on their training without worrying about operating the Glass EE2. With a pre-configured app created for this study, trainees could share their view and receive vocal feedback without learning how to operate the device or requiring technical expertise. Remote supporter’s instructions were shown on the trainee’s Glass EE2 display. The videos that were streamed were not shown on the Glass EE2 display, because they overlap what the trainees’ were viewing directly.

Remote supporters reported difficulties in monitoring and identifying patient conditions because of partially available information through a limited FOV. The limited FOV is due to the camera installed on trainees’ smart glasses. To provide a wider FOV and higher resolution images, advanced cameras, such as 360 cameras, and ultra-wide fisheye lenses can be used. However, high-resolution videos of greater size have the risk of reducing streaming performance by requiring higher network bandwidth (ie, buffering, and frame drops).^22,23 Additionally, the remote supporters would require a high-resolution display to match the wearable camera’s resolution and a large screen monitor to enlarge objects of interest. Ideally, the mirror capturing a patient should be firmly fixed (ie, attached to the ceiling or on the corners) to minimize motion blurs in the transferred video. This approach is suitable for hospitals that can satisfy steady and controllable infrastructure requirements.

Our approach of using a trainee’s wearable camera is applicable to emergency care scenarios outside well-prepared facilities. Moreover, the trainee’s wearable camera can provide focused eye- and first-person views as needed to mitigate any blind spots in the fixed cameras.²⁴ The videos captured via the trainee’s wearable camera can be stabilized using an additional gimbal and software-based image processing, which would increase equipment cost. On the remote supporter’s side, advanced controls of the received video, such as pausing the live video stream to capture a moment of interest on a separate window and the zoom-in function to take a closer look at a region of interest, would make the process more efficient.

There is another issue with video streaming: always-on cameras could be a violation of privacy.²⁵ However, several strategies can be designed to protect the privacy of patients. For example, patient faces in the video can be blurred for anonymity, and a wide view mode can be used optionally or the remote supporters can authorize the use of 360 cameras. The actual use and acceptance of smart glasses in clinical settings are unknown, and there should be other issues requiring crucial consideration. The acceptance (due to the perceived ease of use) of new technology system could differ among health professionals who are not digital natives. Additionally, there are concerns regarding regulatory requirements for its approval as a medical device. Moreover, the perceived resolution may not be sufficient for an actual clinical setting where the complexity of the disease condition could be far greater than that in a simulation setting. For example, the scenarios used in this study did not involve physical symptoms such as bruising or bleeding. Given the large body of evidence to support the suitability of smart glasses,^9,10 the potential is high. Thus, future studies with device reinforcement are warranted to facilitate remote collaboration for wearers outside the simulation.

The remote supporters appreciated the multiple sources of information. When the real-time video did not provide sufficient information, an auditory communication system was used to compensate for the visual restraints. This is consistent with previous studies which showed that multimodal communication improves the clarity and accuracy of message delivery.^26,27 In a dynamic hospital environment, remote assistance would be helpful only when the supporter grasps the situation comprehensively. Although diverse modes facilitate interprofessional communication, concerns remain as to whether multimodal communication is without side effects. Continuous attempts of remote supporters requesting additional information could impede the clinical performance of wearers.

Our current configuration relies on Bluetooth wireless earphones for audio communication between the trainee and remote supporters. Our choice of using Bluetooth should be compared and validated against other alternatives such as currently available audio-video communication smartphone apps²⁸ and high-end Bluetooth sets.

The findings of this study revealed the constraints of using newly developed digital interprofessional interaction methods. Since remote supporters’ high intervention could impede the clinical performance of wearers, a hierarchical guideline should be designed to provide no excessive multimodal communication at a given time. In the high-risk and time-restraint environments of hospitals, especially in emergency units, effective communication is key to quality care.²⁹ Further considerations for developing guidelines for digital interprofessional interactions are needed. For example, simple pointing and capture of a patient would suffice for most cases, whereas assessing a patient’s moan is rarely required.

In this study, a customized mirror was attached to get a better view of the wearers’ performance; however, the outcome was not satisfactory because only a few users expressed that the glasses assisted their work. In fact, users reported that half and half screens reduced the already narrow FOV, and the mirror mode caused confusion from time to time, particularly regarding the regions of the body. According to the results, remote supporters would benefit from incorporating a sufficiently large monitor to correctly interpret the transferred video of high resolution and a wide FOV. As discussed earlier, a wide FOV and higher resolution video would fit better on a larger display to provide more usable UI and effective user experience. The transferred video can be clearly segmented into mirror mode and non-mirror mode. To reduce confusion among remote supporters, the mirror-mode area should be flipped vertically using image processing.

Limitations

There are limitations to this study that warrant caution when interpreting the findings. First, the small sample size of 31 may not have well represented all potential users in clinical settings. Second, considering that the participants were young (mean age = 23.94), the perceived usefulness and ease of use could significantly differ for the older people in the clinical settings,³⁰ thus, there is a high probability that important issues that need to be addressed, were not. Additionally, although the advancement of ICT technology can enforce a greater working interaction, it might cause technostress among older individuals. Perceived potential of mastery and optimism could predict the active adaptation of current system among healthcare practitioners in clinical settings,³¹ thus, a different approach (reinforcing the necessity of devices for improved safety) would ensure positive attitude toward smart glass-based interactions. Third, strong empirical evidence in support of the current system is required, for it to be certified as a medical device. To be used in clinical settings for interactions between health professionals, the current system including smart glasses with the additional mirror accessory need to be certified as medical device.^32,33 Since it is of relatively low risk and has a purpose, gaining regulatory approval would ensure that the current model/version is safe for clinical use. Lastly, since we used essay questionnaires alone, we could not quantify the user experience. This may restrict future studies from having comparative outcomes. We focused on Google Glass-supported cooperative training for remote supporters who received visual information through the trainee’s wearable camera. In the future, remote collaboration from both sides should be assessed to determine the nature of interaction and feedback between the trainee and remote supporter bi-directionally.

Conclusion

The present study investigated the initial experience of a Google Glass-based video streaming system for remote collaboration. The clinical situation and trainees’ performance were streamed to develop a work environment where timely information and guidance were made available via the Google Glass. Using the Google Glass-based video streaming system, the remote supporters experienced some difficulties, especially regarding identifying clinical situations, patients’ conditions, and the trainees’ performance. The findings of this study could contribute to the design and implementation of cooperative training systems using smart glasses by identifying barriers to its implementation. Future studies should make further mechanical progress and upgrade the user interface to address the current limitations, technical issues, and unmet user requirements.

Acknowledgement

This research was supported by a grant (20012234) of Regional Customized Disaster-Safety R&D Program funded by the Ministry of Interior and Safety (MOIS, Korea).