Back to Journals » Open Access Emergency Medicine » Volume 9

Empirical research on the utility of a preparation manual for a disaster medical response drill

Authors Arai T, Ohta S, Onishi M , Taniguchi M, Tsurukiri J, Kumasaka K, Nagata K, Suzuki K, Harigae K, Yukioka T

Received 13 January 2017

Accepted for publication 13 April 2017

Published 14 September 2017 Volume 2017:9 Pages 81—88


Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Hans-Christoph Pape

Takao Arai,1 Shoichi Ohta,1 Masaki Onishi,2 Miyu Taniguchi,1 Junya Tsurukiri,1 Kenichiro Kumasaka,1 Katsuhiro Nagata,1 Kensuke Suzuki,3 Ken Harigae,4 Tetsuo Yukioka1

1Department of Emergency and Critical Care Medicine, Trauma and Emergency Center, Hachioji Medical Center, Tokyo Medical University, Tokyo, Japan; 2National Institute of Advanced Industrial Science and Technology, Tokyo, Japan; 3Faculty of Medical Science, Nippon Sport Science University, Yokohama, Kanagawa, Japan; 4Emergency Department, Hino Municipal Hospital, Hino, Tokyo, Japan

Purpose: It is difficult for emergency physicians to plan and execute a disaster medical response drill while conducting their daily work activities. Readily available drill preparation manuals are therefore essential, alongside assessment methods to ensure quality. Here, we propose email text analysis as a manual assessment method, and investigate its validity.
Methods: The preparation status of two similar large-scale disaster medical response drills were compared. All email texts exchanged during the preparation stage were analyzed, and frequently appearing words (quality element) and word counts (quantity element) were compared between Drill 1, which was organized without a manual, and Drill 2, organized with a manual.
Results: Word frequency analysis revealed that the key components of the manual (visualization of necessary work, preparation of documents in a certain format, and clarification of aims of the drill) contributed to the effectiveness of the preparation process for Drill 2. Furthermore, work volume during the preparation for Drill 2 was decreased by 41.9% from that during the preparation for Drill 1.
Conclusion: Preparation of a high-quality manual is crucial so that emergency physicians can plan and execute a disaster medical response drill. Email text analysis can serve as an objective method assessing the quality of manuals.

Keywords: disaster medicine, textual analysis, database, quality assessment



Fostering experts with accumulated experience in disaster medicine is difficult1,2 and highlights the importance of disaster medical response training. However, emergency physicians cannot easily attend such drills in daily clinical practice. In particular, it is a great burden for emergency physicians to participate in the planning and operation of disaster medical response drills38 because the details of training phases and scenes are so varied and different types of training must be thoroughly understood, including actual drills, map-based drills, and tabletop drills. Nevertheless, because of the close relationship between emergency medicine and disaster medicine, emergency physicians have a central role to play in the planning and implementation of disaster drills. Thus, readily available drill preparation manuals are essential. If a system allowing emergency physicians to have and view a manual for a specific drill is available, a considerable volume of unnecessary work will be avoided while preparing for the drill. We previously organized a disaster medical response drill for our medical district in Tokyo Metropolis and realized that the preparation process was extremely cumbersome. Therefore, we developed a drill manual and passed it on to the next organizers.

The validity of manuals is generally assessed based on an analysis of users’ opinions obtained through interviews or questionnaires. However, these approaches are not highly objective. To provide high-quality manuals, development of a new method which assesses manual contents in a more objective manner is needed.

To this end, we propose email text analysis as a new objective method for the assessment of manuals. In this study, we employed email text analysis to assess a drill manual which we developed, and then evaluated the validity of email text analysis for the objective assessment of manuals.

Material and methods

Tokyo disaster medical response simulation drill

After the Great East Japan Earthquake, Tokyo Metropolis established a disaster medical response system in preparation for a large-scale disaster and has continued to improve its systems in order to ensure smooth medical operations in the event of a disaster. As part of the system, the Tokyo Disaster Medical Response Simulation Drill started in 2013. Twelve medical districts in Tokyo Metropolis participate in the drill in turns. As of November 2016, the drill has been implemented in ten of the 12 medical districts. This is a simulation drill aimed at evaluating and improving the functions of the medical information and communications system.

The Disaster Countermeasures Basic Act of Japan states that, at the time of a disaster, the Office of Disaster Response and Emergency Management (hereinafter, the Office) will be set up in prefectures and municipalities, with the head Office and local Offices forming a hierarchical structure, as shown in Figures 1 and 2. The general role of these Offices will be to collect disaster-related information and implement disaster emergency measures.911 Disaster medical services are also provided in line with this basic framework, and disaster-related information is transferred from the Offices supervising small local areas to Offices supervising wider areas and then to the head Office governing the entire area. In turn, the head Office disseminates information gathered from the entire area and specific instructions to local Offices. In this hierarchical system, the Offices at each level must provide the right information to each Office that needs it. For this, information must be collected, screened, and disseminated promptly. The functions of the Offices of Disaster Response and Emergency Management are shown in Figure 2.9,1214

Figure 1 Hierarchical structure formed by the Offices of Disaster Response and Emergency Management.

Figure 2 Functions of the Offices of Disaster Response and Emergency Management.

In Tokyo Metropolis, the Offices form a three-layer structure composed of 1) those in individual municipalities, 2) those in the 12 medical districts, and 3) one in the Tokyo Metropolitan Government Building. Tokyo Metropolis sponsors and implements the Tokyo Disaster Medical Response Simulation Drill in each of the 12 medical districts to evaluate information exchange between the Offices.

Planning of the disaster medical response simulation drill

Tokyo Metropolis sponsors the Tokyo Disaster Medical Response Simulation Drill, but commissions the planning and implementation of the drill to regional hospitals designated as core disaster medicine centers and where regional disaster medical/health coordinators for the medical district work. The coordinators play a central role in planning and implementing the drill, in cooperation with hospital staff and Tokyo Metropolitan Government staff who are assigned to the drill. All regional disaster medical/health coordinators are emergency physicians who play an active role in the community and have been appointed to serve as coordinators by Tokyo Metropolis. The procedures followed were in accordance with the ethical standards of our institutional committee and with the Helsinki Declaration, as revised in 2004.

The first drill (Drill 1) conducted in the Minami-Tama medical district was planned and organized mostly by the authors, who were appointed as regional disaster medical/health coordinators. Many staff, such as drill instructors and simulation coordinators (collectively referred to as drill staff) were deployed to simulate actual disaster conditions and increase the realism of the disaster experience for the participants (hereinafter, players).15,16 On the day of the drill, Offices were set up for Tokyo Metropolis, each medical district, and each municipality in accordance with the Disaster Countermeasures Basic Act of Japan (Figure 3). Then, players were assigned a profession to play, such as administrator or health care professional. When the drill began, simulation coordinators provided players with a heavy, continuous stream of previously prepared information regarding the disaster and patients. Each Office processed information as it came in and arranged for information dissemination and patient transfer.

Figure 3 Deployment of different disaster drills.

The drill was retrospectively evaluated and verified. Specifically, the data obtained during the drill were objectively evaluated by government research staff at a meeting held on a different day from the drill. Based on the verification results, Tokyo Metropolis is currently improving the disaster medical response system.

Creating the disaster drill manual

We previously organized a disaster medical response drill for our medical district in Tokyo Metropolis and realized that the preparation process was extremely cumbersome, involving many complex and detailed activities, including planning the details of the drill, arranging the drill staff, venues, supplies and equipment, and budgeting, among various other administrative procedures. Also, a lack of previous experience meant there were many unclear points about individual preparation activities, making the entire preparation process ineffective. We therefore created a drill preparation manual and passed it on to the next organizers.

We learned from the numerous problems we encountered in Drill 1 (Table 1) that organizing a drill from scratch is unrealistic, so looking back at our experiences, we created a disaster drill manual to streamline preparation for future drills. This manual includes a time schedule of tasks to complete, necessary staff members, detailed supply and equipment lists, and formatted documents/forms. In this study, we assessed the effectiveness of this drill manual by comparing preparation procedures for Drill 1 (before creation of the drill manual) with those of Drill 2 (in which we used the drill manual for drill preparation). Drill 2 was our third drill, conducted in the western medical district.

Table 1 Problematic events

Assessment of the drill manual content

For a drill of this scale, planners come from multiple institutions, which makes frequent face-to-face meetings difficult. Therefore, most of the drill preparation activities were handled by email, resulting in large numbers of email communications. The contents of these emails were discussions between drill planners about various preparation processes, such as developing drill scenarios, creating documents/forms and formats, communicating with Tokyo Metropolitan Government, and making requests to drill staff. We decided to use these emails exchanged during the preparation process as a quantitative and qualitative indicator of the drill preparation process, comparing the results for Drill 2 against those of the earlier Drill 1 as an indication of the effectiveness of the drill manual created.

Specifically, to elucidate the scale and details of Drills 1 and 2, we analyzed the number of drill staff, number of players, number of booths placed, and ratings of players. To quantify the preparation process for Drills 1 and 2, we next analyzed the preparation period, total number of emails, total number of characters in each email, and duration of face-to-face meetings (excluding time spent on email). In addition, to analyze the drill preparation processes qualitatively, frequently occurring words were extracted from emails, and the word frequencies were compared between the drills. We then evaluated words showing a large difference in frequency between the drills to show a change in the quality of the drill preparation process.

Note that we excluded words that do not have meaning by themselves, such as “this” and “and”, when extracting frequently occurring words. In addition, we manually examined words that were automatically extracted and counted those with a specific meaning when appearing in sequence as one term; for example, “action” directly followed by “card” was regarded as the single term “action card”.

Frequently occurring words were analyzed using the following equation to calculate the difference in the frequencies of appearance between Drills 1 and 2.

Number of occurrences in Drill 1 emails / total number of characters in Drill 1 emails – Number of occurrences in Drill 2 emails / total number of characters in Drill 2 emails.

To eliminate observer and respondent bias, text analysis was performed by research staff different from those involved in the organization/execution of drills or the development of the drill preparation manual.


There were 45 drill staff in both Drill 1 and Drill 2, four of whom participated in both drills as drill instructors (Table 2). In Drills 1 and 2, there were 31 and 58 players and six and eight booths, respectively (with booths representing Offices and hospitals). The two drills did not differ significantly in any factors (Tables 2 and 3). In addition, drill results were very similar between the drills (Table 4).

Table 2 General data for each simulation drill

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

Table 3 Diagnoses of red tag players

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual. All red tag players were assigned a diagnosis randomly in each simulation.

Abbreviation: TBSA, total body surface area.

Table 4 Performance indicator results in the first and second simulation drills

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

The number of emails exchanged for drill preparation was 797 for Drill 1 and 502 for Drill 2, a 37.0% reduction. The corresponding total number of email characters was 204,360 and 118,692, a 41.9% reduction. The approximate preparation period was 7 months and 2.5 months (Table 5), a 64.3% reduction. The total amount of time spent in meetings was approximately 6 h for both drills. Email text analysis identified 72 frequently occurring words, 50 of which appeared less frequently in Drill 2 emails than in Drill 1 emails (Table 6), whereas 23 words appeared more frequently in Drill 2 emails (Table 7).

Table 5 Quantitative comparison of drill preparation

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

Table 6 Words that appeared less frequently in Drill 2 than in Drill 1 emails

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

Table 7 Words that appeared more frequently in Drill 2 than in Drill 1

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.


Less preparation was involved in Drill 2 than in Drill 1. Moreover, in Drill 2, the length of the drill preparation showed a larger reduction than the total number of emails or email characters, suggesting that many emails were exchanged over a shorter period of time when preparing for Drill 2. There were also fewer preparation tasks overall and each process was performed more efficiently when preparing for Drill 2, which we attribute to use of the drill preparation manual.

The analysis of frequently occurring words revealed that a wider variety of word types were used in preparation emails for Drill 1 than for Drill 2, suggesting that discussion was more tightly focused in the preparation for Drill 2. When preparing for Drill 1, we had to start off with a discussion of what to prepare because we had no drill manual to follow, whereas the discussion for Drill 2 was more focused because we were using the manual.

The most frequently occurring words were “medical care”, then “disaster”, and “Medical Association” in Drill 1 preparation emails, whereas these were “Office”, “hospital”, and “conference” in Drill 2 preparation emails. This suggests that players needed to start discussions from the most fundamental issue – the nature of disaster medicine – in Drill 1 emails, whereas their discussions in Drill 2 emails were about the specifics of the drill itself.

Interestingly, the words “EMERGO” and “triage”, both of which appeared frequently in Drill 1 emails, were not factors incorporated into either Drill 1 or Drill 2, suggesting that while we continuously debated what type of drill to perform in Drill 1, such discussion was unnecessary in Drill 2 because of the presence of the manual. Furthermore, because the Drill 1 planners needed to create various documents and forms from scratch, words such as “planning documents”, “paper reports”, and “action card” appeared often in Drill 1 emails.

On the other hand, words that frequently appeared in the Drill 2 emails were “cooperation”, “information”, “communication”, “request”, and “explanation”, all of which were associated with information sharing and thus matched with the theme of the drill, which is to verify the medical information and communications system between the Offices. In addition, words representing people and roles, such as “player”, “administrative staff”, and “person in charge”, appeared frequently, suggesting that discussions on Drill 2 were more specific to the process.

Based on the above findings, we believe the drill preparation manual enabled the following:

  1. Visualization of necessary work, making the preparation efficient.
  2. Use of formatted documents/forms, making the preparation easy.
  3. Easy decisions on the specifics of the drill because the goal of the drill was clear.

This also means that the problematic events in Drill 1 (Table 1) were adequately addressed in Drill 2 because of the manual, objectively showing the utility of the manual.

The planning of various types of drills can be made more effective by creating manuals, similar to ours, for different hospital scales, administrative levels (prefecture, city, smaller municipality, etc.), and drill patterns (tabletop, map-based, and full-scale drill). In addition, we plan to revise the manual over time, improving it based on the results of using it to plan future drills and making it more effective.

The formulation of high-quality preparation manuals is necessary to help emergency physicians plan and execute disaster medical response drills while also performing their routine duties in the clinical setting. Manuals can be assessed using subjective assessment methods, such as questionnaires for users, or objective text analysis methods developed for qualitative and quantitative investigation in various fields including cultural anthropology.17,18 In addition, text analysis and text mining of emails are performed routinely in research as well as business.1921 Therefore, in this study, we proposed email text analysis as an objective alternative and then tested it for the assessment of an actual manual. We found that examining the attributes of frequently appearing words during the preparation of the drill, enabled quantitative assessment of the manual as well as qualitative assessment of reasons to demonstrate the validity of the manual. Repeat studies examining a larger number of drills would be valuable.

This study has some limitations. It is a retrospective study comparing preparation status between two drills only. Also, assessment of qualitative and quantitative effects of the manual on the preparation of a drill entirely relies on text analysis without taking other factors into account.


It takes enormous time and effort to plan disaster drills. However, the findings of this study suggest that the use of a drill manual helps those involved in the preparation of drills visualize the tasks needed for planning, and then streamline the preparation process to reduce wasted time and resources. This study suggests that email text analysis is a useful objective method for assessing the effectiveness of such drill manuals.


The authors report no conflicts of interest in this work.



Steward D, Wan TT. The role of simulation and modeling in disaster management. J Med Syst. 2007;31(2):125–130.


Franc-Law JM, Bullard M, Della Corte F. Simulation of a hospital disaster plan: a virtual, live exercise. Prehosp Disaster Med. 2008;23(4):346–353.


Perry RW. Disaster exercise outcomes for professional emergency personnel and citizen volunteers. Journal of Contingencies and Crisis Management. 2004;12:64–75.


Peterson DM, Perry RW. The impacts of disaster exercises on participants. Disaster Prev Manag. 1999;8(4):241–255.


Okada Y, Haruki Y, Ogushi Y. Disaster drills and continuous medical education using satellite-based Internet. Methods Inf Med. 2000;39(4–5)343–347.


Nagami K, Nakajima I, Juzoji H, Igarashi K, Tanaka K. Satellite communications for supporting medical care in the aftermath of disasters. J Telemed Telecare. 2006;12(6):274–275.


FitzGerald DJ, Sztajnkrycer MD, Crocco TJ. Chemical weapon functional exercise--Cincinnati: observations and lessons learned from a “typical medium-sized” city’s response to simulated terrorism utilizing Weapons of Mass Destruction. Public Health Rep. 2003;118(3):205–214.


Klein KR, Brandenburg DC, Atas JG, Maher A. The use of trained observers as an evaluation tool for a multi-hospital bioterrorism exercise. Prehosp Disaster Med. 2005;20(3):159–163.


Nilsson H, Jonson CO, Vikström T, et al. Simulation-assisted burn disaster planning. Burns. 2013;39(6):1122–1130.


Nilsson H, Ruter A. Management of resources at major incidents and disasters in relation to patient outcome: a pilot study of an educational model. Eur J Emerg Med. 2008;15(3):162–165.


McEntire DA, Myers A. Preparing communities for disasters: issues and processes for government readiness. Disaster Prev Manag. 2004;13(2):140–142.


Christie PM, Levary RR. The use of simulation in planning the transportation of patients to hospitals following a disaster. J Med Syst. 1998;22(5):289–300.


Aylwin CJ, König TC, Brennan NW, Shirley PJ, Davies G, Walsh MS, Brohi K. Reduction in critical mortality in urban mass causality incidents: analysis of triage, surge, and resource use after the London bombings on July 7, 2005. Lancet. 2006;368(9554):2219–2225.


Lewis B, Swarup S, Bisset K, Eubank S, Marathe M, Barrett C. A simulation environment for the dynamic evaluation of disaster preparedness policies and interventions. J Public Health Manag Pract. 2013;19 Suppl 2:S42–48.


Gillett B, Peckler B, Sinert R, et al. Simulation in a disaster drill: comparison of high-fidelity simulators versus trained actors. Acad Emerg Med. 2008;15(11):1144–1451.


Kaji AH, Bair A, Okuda Y, Kobayashi L, Khare R, Vozenilek J. Defining systems expertise: effective simulation at the organizational level–implications for patient safety, disaster surge capacity, and facilitating the systems interface. Acad Emerg Med. 2008;15(11):1098–1103.


Baker P. Using Corpora in Discourse Analysis. London: Continuum; 2006.


Liu B. Web Data Mining-Exploring Hyperlinks, Contents, and Usage Data. Berlin: Springer; 2007.


Hanai T, Oguchi T. [The appearance patterns of emotional expressions in e-mail exchanges: A text-mining analysis]. The Japanese Society of Social Psychology. 2008;24(2):131–139. Japanese.


Moriwaki H. 2013 Text mining techniques for analyzing big data, UNISYS Technology Review, 115. Available from: Accessed June 30, 2017.


Kumar H. Exploratory Data Analysis of Enron Emails. University of California-Berkeley; 2015. Available from: Accessed June 9, 2017.

Creative Commons License This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.