Back to Journals » Advances in Medical Education and Practice » Volume 5

An integrable, web-based solution for easy assessment of video-recorded performances

Authors Subhi Y , Todsen T, Konge L

Received 11 February 2014

Accepted for publication 4 March 2014

Published 3 May 2014 Volume 2014:5 Pages 103—105

DOI https://doi.org/10.2147/AMEP.S62277

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 4



Yousif Subhi,1,2,3 Tobias Todsen,1,4 Lars Konge1,2

1
Centre for Clinical Education, Centre for HR, The Capital Region of Denmark, Copenhagen, Denmark; 2University of Copenhagen, Copenhagen, Denmark; 3Clinical Eye Research Unit, Department of Ophthalmology, Copenhagen University Hospital Roskilde, Roskilde, Denmark; 4Department of Internal Medicine, Queen Ingrid's Hospital, Nuuk, Greenland

Abstract: Assessment of clinical competencies by direct observation is problematic for two main reasons: the identity of the examinee influences the assessment scores, and direct observation demands experts at the exact location and the exact time. Recording the performance can overcome these problems; however, managing video recordings and assessment sheets is troublesome and may lead to missing or incorrect data. Currently, no existing software solution can provide a local solution for the management of videos and assessments but this is necessary as assessment scores are confidential information, and access to this information should be restricted to select personnel. A local software solution may also ease the need for customization to local needs and integration into existing user databases or project management software. We developed an integrable web-based solution for easy assessment of video-recorded performances (ISEA).

Keywords: education assessment, assessment software, video-based assessment

Background

Clinical performance can be assessed by expert observation using assessment sheets, but this is problematic for two main reasons. Firstly, the identity of the examinee can influence the assessment.1,2 This can be due to personal relations and prejudices,1 which can be a large problem in smaller countries or within smaller medical specialties. Age and job title have also been shown to influence assessment scores.2 Secondly, assessments are time-consuming and need experts to be at a specific location and at a specific time. A feasible way of addressing these problems is by video recording the clinical performance.

Recording videos can be unproblematic if designed carefully.3 However, the assessment procedure can easily become problematic and lead to logistic problems. Typically, hard drives or flash drives with videos and assessment sheets are sent to all examiners. When these are returned back to the data manager, some sheets may be incomplete and in some cases, may even be forgotten, requiring that the data manager go through a tiresome process to obtain all assessment scores. Finally, when all the cases are fully assessed, paper sheets are transcribed into a database in duplicate to avoid any typing error. This nonautomated assessment procedure is costly, time-consuming, and error-prone.

Existing software enable an automated process4,5 but are not optimized for video-recorded assessments and are problematic since data are stored at external institutions; instead, data should be stored locally as assessment scores are confidential information, and access to such should be restricted to course personnel (or study personnel, in the case of research).6,7 We developed an integrable web-based solution for easy assessment of video-recorded performances (ISEA).

Design

The design of ISEA is based on two user interfaces: one for the administrator and one for the examiner. The administrator manages projects, examiners, videos, and assessment sheets. A “project” is defined as a group of examiners who assess videos based on the same assessment sheets. The assessment sheet is composed of questions, each having a number of scoring items. Within a project, examiners are assigned one or more videos that, if necessary, may differ between the examiners. Results from all assignments are available from the administrator interface. User profiles and results are stored in a local database. Depending on the legal requirements in each project, videos can either be uploaded to an external video storage platform, such as Vimeo,8 or stored internally within an institution, and ISEA is then provided with a link to the uploaded video. Examiners have their own personal page with a list of assigned videos. An assigned video is seen side-by-side with the assessment sheet, to enable simultaneous assessment (Figure 1). When the assignment is completed, it is locked and scores cannot be changed. The results are available for the administrator and can be exported in a semicolon-separated or a comma-separated file format (.csv).

Figure 1 Simultaneous view of the video and the assessment sheet, as seen by the examiner.

We used PHP: Hypertext Preprocessor (The PHP Group) and My Structured Query Language (MySQL) as the server-side scripting language and database, respectively, as these are the most popular and commonly used9,10 and may facilitate an easier implementation and integration to existing software. We wanted the examiners to be able to use our web solution in any digital device anywhere, so the interfaces were developed using very simple Hyper Text Markup Language (HTML) and Cascading Style Sheets (CSS) that are compatible with a large range of internet browsers, including smart phones and tablets. No add-on software is needed for ISEA. This clean and simple interface may ease integration to existing platforms. Adobe Dreamweaver CS5 (Adobe Systems, Inc., San Jose, CA, USA) was used for development. A download link for ISEA is available in the references.11

Discussion

Video-based assessment of clinical performance may help eliminate bias related to the identity of the examinee by masking the identity, which can be achieved by adjusting the camera-angle or by editing the video afterwards.1,12,13 However, the identity is rarely perfectly hidden – the face may be blurred and the voice may be distorted, but the identity may still be guessed if the examiner is familiar with the examinee and is aware of his or her specific characteristics, such as manner of speech, preferences of clothes, or characteristic body figure or posture. Also, in some cases, facial expression, eye contact, or the finer aspects of the voice, such as the tone, may be relevant for the assessment itself. A step toward better blinding and the reduction of biases emerging from personal relations may therefore be achieved by, not only use of videos but also, collaboration with examiners in other institutions and other countries. Video-recorded assessments provide flexibility for the examiners as they no longer need to be at a specific location at a specific time but may be at any location and assess the clinical performance at any time.12 Implementing video-based assessments is, theoretically, not a problem, but practically, this is a tiresome process with potential pitfalls. Few studies have compared direct observation with video-based assessments.13,14 Ma et al assessed central venous catheterization using both direct observation and video recordings and found no differences in ratings, and they reported that importance should be given to practical aspects.13 Kneebone et al explored the feasibility of a video-based assessment procedure with remotely placed examiners, and they also reported that a future improvement would be an automated approach for easy assessment of video-recorded performances.14 We developed ISEA, an integrable web-based solution for easy assessment of video-recorded performances that provides a format that can be stored locally, is customizable into existing systems, does not require any add-on software, and that is compatible with a large range of web browsers. ISEA may ease the assessment process of video-recorded performances and collaboration with examiners from other institutions. ISEA was used in the assessment process of a recent project,15 and the general comment from the examiners was that ISEA was easy to use. Also, ISEA can be customized to local systems, for integration to an existing project or user management systems that are used to administer courses and research. A few limitations should be noted. ISEA is an integrable application and not an “out of the box”, stand-alone application, and the implementation and customization may thus require some technical assistance. Customization, however, can be advantageous and even necessary for some types of education and research projects.7 We did not describe the implementation process in this paper as the implementation process is more technical and may vary greatly between institutions, depending on the software used. In its simplest form and without much effort, ISEA can be implemented using only a user database for the administrator and the examiners. In our experience, implementation has been unproblematic, and the limited technical assistance needed to implement ISEA is a small price to pay for a smooth and easy assessment process.15

Acknowledgment

We thank the University of Copenhagen for an education research grant (Undervisningskvalitetspuljen) for covering our publication costs.

Disclosure

The authors report no conflicts of interest in this work.


References

1.

Vogt VY, Givens VM, Keathley CA, Lipscomb GH, Summitt RL. Is a resident’s score on a videotaped objective structured assessment of technical skills affected by revealing the resident’s identity? Am J Obstet Gynecol. 2003;189(3):688–691.

2.

Konge L, Vilmann P, Clementsen P, Annema JT, Ringsted C. Reliable and valid assessment of competence in endoscopic ultrasonography and fine-needle aspiration for mediastinal staging of non-small cell lung cancer. Endoscopy. 2012;44(10):928–933.

3.

Cohen L, Manion L, Morrison K. Research Methods in Education. 7th ed. Abingdon: Routledge; 2011.

4.

Cheng A, Nadkarni V, Hunt EA, Qayumi K; EXPRESS Investigators. A multifunctional online research portal for facilitation of simulation-based research: a report from the EXPRESS pediatric simulation research collaborative. Simul Healthc. 2011;6(4):239–243.

5.

Leskovský P, Harders M, Székely G. A web-based repository of surgical simulator projects. Stud Health Technol Inform. 2006;119:311–315.

6.

Lallas CD, Preminger GM, Pearle MS, et al. Internet based multi-institutional clinical research: a convenient and secure option. J Urol. 2004;171(5):1880–1885.

7.

Durkalski V, Wenle Zhao, Dillon C, Kim J. A web-based clinical trial management system for a sham-controlled multicenter clinical trial in depression. Clin Trials. 2010;7(2):174–182.

8.

vimeo.com [homepage on the Internet]. Your videos belong here. Vimeo; 2014 [cited February 2, 2014]. Available from: http://www.vimeo.com/. Accessed March 21, 2014.

9.

w3techs.com [homepage on the Internet]. Usage of server-side programming languages for websites. W3Techs Web Technology Surveys; 2014 [cited February 2, 2014]. Available from: http://w3techs.com/technologies/overview/programming_language/all/. Accessed March 21, 2014.

10.

db-engines.com [homepage on the Internet]. DB-engines ranking. DB-Engines; 2014 [updated March 2014; cited February 2, 2014]. Available from: http://db-engines.com/en/ranking/. Accessed March 21, 2014.

11.

ISEA. Available from: http://isea.cekukurser.dk/download.zip. Accessed March 21, 2014.

12.

Beckmann CR, Lipscomb GH, Ling FW, Beckmann CA, Johnson H, Barton L. Computer-assisted video evaluation of surgical skills. Obstet Gynecol. 1995;85(6):1039–1041.

13.

Ma IW, Zalunardo N, Brindle ME, Hatala R, McLaughlin K. Notes from the field: direct observation versus rating by videos for the assessment of central venous catheterization skills. Eval Health Prof. Epub January 12, 2014.

14.

Kneebone R, Nestel D, Yadollahi F, et al. Assessing procedural skills in context: Exploring the feasibility of an Integrated Procedural Performance Instrument (IPPI). Med Educ. 2006;40(11):1105–1114.

15.

Todsen T, Tolsgaard MG, Olsen BH, et al. Reliable and valid assessment of point-of-care ultrasonography. Ann Surg. Epub February 6, 2014.

Creative Commons License © 2014 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.