Failure mode and effects analysis drastically reduced potential risks in clinical trial conduct

Introduction

Quality of clinical trials depends on the way that they are conducted, which is indispensable for ensuring the credibility of the results to determine the efficacy and safety of a diagnostic, prognostic, or therapeutic product under testing.¹ Furthermore, ethical conduct of clinical trials is critical to protect the rights and safety of human research subjects, which contributes to the quality of clinical trials. However, many quality risks can happen during the entire course of clinical trials, undermining the validity of the clinical trial results and threatening the rights and safety of human research subjects. These quality risks include, but are not limited to, protocol deviations, poor conduct, inadvertent errors, inadequate record keeping, unethical treatment of clinical trial participants and so on.^1,2 Recognizing those quality risks in clinical trials, the regulatory agencies such as the US Food and Drug Administration and European Medicines Agency have recently published guidelines that emphasize the importance of maintaining high quality throughout clinical trials.^3,4 These initiatives by the regulatory agencies have made clinical trial investigators and site managers realize the importance of a suitable risk management tool applied in a pre-emptive manner not only to improve the quality of the clinical trials, but also to avoid serious official actions such as the disqualified/restricted/assurance list for clinical investigators⁵ when they fail to meet the regulatory standards.

Several quality risk management tools have been proposed for industry and regulators such as basic risk management facilitation methods (ie, flowcharts and check sheets), failure mode and effects analysis (FMEA), fault tree analysis, hazard analysis and critical control points, hazard operability analysis, and preliminary hazard analysis.⁶ Of these tools, FMEA² is a systematic, proactive and teamwork-based method aimed to identify and assess the causes and effects of potential failures in a system, thereby preventing them from happening beforehand.^7,8 Typically, an FMEA is performed by completing the following 5 steps: selection of processes to be assessed, construction of a multidisciplinary team, collection and classification of risk scores from each process, conduct of a risk analysis, and implementation of remedial actions and reanalysis to see if those actions are effective.⁹ The multidisciplinary team performs brainstorming to assign a risk priority number (RPN) to each process of potential failure. RPN is the product of scores representing the 3 aspects of a risk: severity (ie, how severe is the consequence if the process fails?), occurrence (ie, how frequently can a process fail?), and detection (ie, how easily can a failure be detected if the process fails?).⁷

FMEA has been employed in the health care industry since 1990s, including a Health Failure Mode and Effect Analysis suggested by The Veterans Affairs National Center for Patient Safety.⁸ Moreover, the Joint Commission on Accreditation of Healthcare Organization recommended that FMEA be conducted for risk assessment at least once a year.⁹ Since then, many clinical departments in the hospital such as radiology,¹⁰ cardiology,¹¹ dialysis unit,¹² and intensive care unit⁸ have actively accepted FMEA as a critical quality improvement tool. For example, FMEA enabled Mercy Hospital Springfield to reduce the incidence of surgery site infections, resulting in a huge cost saving, shortened length of stay, and fewer readmissions.¹³ Likewise, FMEA reduced the median door-to-balloon time from 146 to 32 min in patients with ST-segment elevated myocardial infarction, which contributed to a lower risk of reinfarction and mortality.¹¹

Although FEMA has been successfully used to increase the quality of various clinical procedures and processes in the hospital system and the utility of FEMA in mitigating risks for human subjects in clinical research was suggested,¹⁴ its application to the clinical trials system has been infrequent. Based on this understanding, the objectives of the present study were 1) to identify, assess and prioritize quality risks in clinical trials using FMEA in a tertiary care university hospital setting and 2) to evaluate the impact of remedial actions following the FMEA assessment.

Methods

Setting

Seoul National University Hospital (SNUH) Clinical Trials Center (CTC) was established in the year 1995 as the first fully-dedicated university hospital-affiliated clinical trial support unit in Korea. Since its establishment, SNUH CTC has developed 254 standard operating procedures (SOPs) at the time of preparing this manuscript that covered various activities in clinical trials in accordance with the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) – Good Clinical Practice and other international guidelines. These SOPs have been regularly reviewed and updated to correctly reflect the current best practices. SNUH CTC has also constantly upgraded the quality of clinical trials through diverse in-house online and offline education and training programs for clinical research personnel. Thanks to the state-of-the-art facility and quality operation system along with the advanced infrastructure, the number of clinical trials conducted at SNUH CTC has rapidly increased for the past 20 years. For example, SNUH CTC manages >253 active clinical trial protocols as of December 2015. Furthermore, SNUH CTC has closely worked with other research institutes and pharmaceutical companies both inside and outside of Korea to develop new medical goods. The present study was exempt from the Seoul National University Hospital Institutional Review Board review because its research activities did not constitute any human subject and it involved only the collection of existing data, documents, or records, available such that subjects could not be identified, directly or through personal identifiers linked to the subject.

Pfizer’s investigator networks, site partnerships and infrastructure for research excellence (INSPIRE) project

From November 2012 to September 2014, the Investigator Site Development Lead Group of Pfizer introduced FMEA methodology to their INSPIRE sites, one of which was SNUH CTC, to contribute to solidifying the clinical research capabilities and enhancing the quality and competitiveness in clinical trial operations of the sites. INSPIRE sites were highly interested in quality improvement, had resources and system for quality management, and were willing to participate in this FMEA project. It was anticipated that FMEA would help the sites proactively identify the most serious risks that could fail in clinical trial operations, thereby preventing them from happening in the first place. Furthermore, FMEA was expected to assist the sites to become sensible about possible concerns that were not, in reality, serious or likely enough to warrant special attention.

Baseline FMEA

To assess the baseline risk profile, an FMEA was performed from January 2013 to June 2013 by completing the following steps.

Creation of an FMEA focus group

An FMEA focus group was formed, which consisted of 6 people collectively representing the quality improvement group, clinical research coordinators (CRCs), and clinical investigators of SNUH CTC. Additionally, an investigator Site Development Lead from Pfizer also joined the SNUH FMEA focus group to guide and assist the onsite FMEA processes. The focus group was able to consult with other clinical trial experts such as Phase I unit staff to complement its expertise.

Development of process maps and identification of potential failure modes

The FMEA focus group first enumerated the overall clinical trial processes of SNUH CTC. After several rounds of internal discussion and process evaluations, the focus group selected 6 core processes that were deemed most prone to risk failure based on the previous audits and regulatory inspections at SNUH CTC. Then, the focus group identified a total of 114 potential failure modes based on brainstorming followed by iterative reviews. The 6 selected core processes (number of potential failure modes in each core process) included: obtaining informed consent (32), screening and patient identification/enrollment (16), site personnel training (8), document management, including source documents and essential documents (23), safety management such as reporting serious adverse event and suspected unexpected serious adverse reaction (25), and inspection readiness (10) (Table 1). The final list was reviewed and concurred by those who operated or were in charge of the selected clinical trial processes. Then, the FMEA focus group reviewed all those 6 core processes and 114 potential failure modes in detail to create a process map for each process as shown in Figure 1 for obtaining informed consent as an example (Table S1). Furthermore, potential causes for each failure mode were listed and the impact of the failure modes was described in the process worksheet.

Table 1 Summary of the severity, occurrence, and detection scores and RPN by six selected core processes Note: Mean ± standard deviation is displayed. Abbreviations: RPN, risk priority number; SAE, serious adverse event; SUSAR, suspected unexpected serious adverse reaction.

Figure 1 Example of a process map for FMEA (obtaining the initial informed consent). Abbreviation: FMEA, failure mode and effects analysis.

Scoring with RPNs

The FMEA focus group assigned a score to each identified failure mode in terms of its severity (S), occurrence (O), and detection (D). These scores ranged from 1 to 10 based on the degree of impact, frequency, and the probability of detection of each failure mode (Table 2). Any number between the 2 adjacent scores in Table 2 was assigned if the FMEA focus group assessed the score somewhere in the middle of them. The Phase I unit staff independently cross-checked the assigned S, O, and D scores for concurrence, and any differences between them and the focus group were discussed until an agreement was reached. Then, the RPN was defined as the product of the S, O, and D scores. An RPN of <40, 40–160, and >160 was considered low, moderate, and high risk, respectively.

Table 2 Rating scale of the severity occurrence, and detection scores for FMEA Abbreviation: FMEA, failure mode and effects analysis.

Follow-up FMEA

Based on the results of the baseline FMEA, the focus group developed a list of action plans to preempt high-risk (ie, RPN >160) failure modes by reducing any of the S, O, and D scores. These action plans also included further steps, responsibility, and timeline to address the identified potential high-risk failure modes that were approved by SNUH CTC Director. SNUH CTC then began a center-wide quality improvement initiative based on the action plans. To assess the effectiveness of this initiative ~1 year later (ie, September 2014), the FMEA focus group conducted a follow-up RPN scoring for the high-risk failure modes identified during the baseline FMEA.

Results

Baseline FMEA and RPN Scoring

RPNs for 114 potential failure modes identified by the FMEA focus team varied widely, ranging from 3 to 315 with a median value of 66.5 (Figure 2). Although most of the 114 potential failure modes had an RPN of low (n=28, 24.6%) or moderate risk (n=72, 63.2%), 14 failure modes (12.3%) were considered high risk (ie, RPN >160) (Figure 2). These 14 high-risk failure modes belonged to the following 3 core processes: site personnel training (n=5), document management, including source documents and essential documents (n=4), and obtaining informed consent (n=5) (Table 3 for more details). The mean RPNs for these high-risk core processes were also greater than those for other core processes (81.24–195.25 vs 37.52–69.04, Table 1). The highest score was always assessed in severity, followed by detection and occurrence (6.92 vs 4.16 vs 2.88, Table 1).

Figure 2 Histogram of the distribution of the RPNs for 114 potential failure modes (baseline FMEA). Abbreviations: FMEA, failure mode and effects analysis; RPN, risk priority number.

Table 3 Comparison of RPNs for14 high-risk failure modes before and after the quality improvement initiative was implemented Notes: *Percent change in RPN from baseline. (−) denotes reduction in RPN. Abbreviations: CRC, clinical research coordinator; D, detection score; ICF, informed consent form; IRB, institutional review board; O, occurrence score; QI, quality improvement; RPN, risk priority number; S, severity score; SMF, site master file.

The failure modes with the 2 highest RPNs were 1) using outdated informed consent form (ICF) (RPN =378) and 2) not updating training records (RPN =315) (Table 3). The 14 high-risk failure modes (ie, RPN >160) had an S score of >7 except for one failure mode (Table 3). Furthermore, the RPN of the 14 high-risk failure modes was mainly driven by the S score (mean ± SD, 7.50±1.16), which was significantly or marginally higher than that of the D score (5.43±1.09, p=0.002, analysis of variance [ANOVA]) or the O score (6.43±1.40, p=0.0651, ANOVA) (Table 3).

Remedial actions for the high-risk failure modes

For each high-risk failure mode, a remedial action plan was devised such as granting CRCs an authority to get access to the clinical trials management system in case “CRCs use an outdated version of patient information and the informed consent form” (Table 3). Of these 14 remedial action plans, “developing a system by which clinical trial participants can consent and sign the ICF on an individual basis such as making a room available for them where they can ask any clinical trial-related question for investigators and get those answered” was thought to be able to simultaneously address 3 high-risk failure modes (Table 3). However, this remedial action was not implemented due to space limitation at the time of devising remedial action plans. Instead, it was decided to put this action plan into practice during the next round of CTC renovation and expansion, which was scheduled in the first half of 2017.

Implementing the remedial actions for the remaining 11 high-risk failure modes dramatically reduced the mean RPN from 257.67±48.60 to 49.33±20.33 (90.5% reduction from baseline, Table 3). Furthermore, 9 and 2 high-risk failure modes were assigned an RPN of low and moderate risks, respectively (Table 3). Interestingly, those decreases in RPN were mainly driven by reductions in the O and D scores (6.43±1.40 to 3.14±2.21 in O score and 5.43±1.09 to 3.14±0.66 in D score).

Discussion

We demonstrated that FMEA is useful to systematically and proactively address quality risks in clinical trials, as previously suggested that integrating FMEA techniques with research governance could reduce human risks in clinical research.¹⁴ After a baseline FMEA, we identified 114 potential failure modes in 6 core clinical trial processes, 14 of which were high risk (Tables 1 and 3). Furthermore, FMEA was effective in improving the quality of clinical trial performance. For example, we devised a set of remedial action plans based on the results of the baseline FMEA, which led to a significant decrease in RPN by 80% (Table 3). To the best of our knowledge, the present study showed for the first time that FMEA can identify vulnerable risk areas in clinical trials and contribute to quality improvements in clinical trial process, similar to those reported previously in other areas.^8,10–13,15

We selected 14 high-risk failure modes with an RPN >160 and developed action plans to rectify them. Although the S score was significantly or marginally higher than the O or D scores in high-risk failure modes, the reduction in RPN after remedial actions was mainly driven by the decreases in the O and D scores (Table 3). Previous studies reported similar findings; remedial actions mostly reduced the O and D scores rather than the S score.^12,16,17 Because the S score is associated with the nature of a failure mode, it is understandable that there is hardly an action plan to directly affect the S score. However, FMEA is still effective in mitigating risks as clearly shown in our results. This indicates that preemption of risk occurrence based on FMEA, along with increased detectability, can successfully address a potential failure mode, no matter how severe it can be when it happens.

Our FMEA focus team newly developed a double-checking procedure for several high-risk failure modes. This remedial action was effective to prevent occurrence or to increase the likelihood to detect it as we showed in the present study. However, the remedial action could have been more effective if it was strengthened by a formal checklist. For example, a previous FMEA for gynecologic high-dose-rate brachytherapy used a checklist to improve quality that listed a timestamp for entire brachytherapy to keep a steady and non-missing workflow.¹⁸ Checklist has been commonly used to improve the perception of teamwork and safety climate¹⁹ or to assure quality.¹⁸ Because clinical trials are a very complex process involving many parties and workforces with diverse credential and training backgrounds, team communication is of utmost importance. The World Health Organization (WHO) also developed a WHO surgical safety checklist to decrease errors and adverse events during surgery through improved teamwork and team communication. Thus, checklists would have been additionally beneficial to improve quality of clinical trial, which can be tested in further FMEA.

Scoring for failure modes is subjective in any FMEA.²⁰ To minimize subjectivity in risk scoring, we created a multidisciplinary FMEA focus group consisting of collective expertise in various areas of clinical trials and ability to identify potential problems that could arise in the conduct of a clinical trial. Furthermore, we developed an objective scoring system to reduce bias while increasing consistency (Table 2). Likewise, the final scores for potential failure modes were independently assessed by another team for concurrence and any difference was discussed until a consensus was reached.^21,22 Therefore, risk scoring in our results was of less bias and better reliability.

Meaning of RPN implicated by itself in FMEA is controversial. However, the decreased RPN supports the feasibility of the FMEA approach and its utility for aiding the adoption of technologies because scoring process before and after remedial actions in our results is consistent as the same FMEA focus group scored RPN for high-risk failure modes. Therefore, the decreased RPN by remedial actions offers probability that FMEA can be applied to understand processes and probably also for identifying risks worth addressing in clinical trial center as a useful quality control tool.

Limitations

The present study had 3 major limitations. First, only 6 core processes were selected for FMEA in this study, which could not represent the whole aspect of a clinical trial. However, quality in 1 area of clinical trials is closely linked with that in other areas given the complex and concerted nature of the clinical trial. Those 6 core processes were deemed most risky based on the previous audits and regulatory inspections that SNUH CTC had undergone. Therefore, the ability and utility of FMEA to improve the quality in clinical trials, particularly for areas most prone to risk failure as we showed in the present study, can be readily applied to other less risky areas. Second, our results may not be directly applicable to smaller clinical trial centers or private investigational sites where resources for FMEA are likely to be limited. Theoretically, individual clinical scientists can perform an FMEA due to its simple nature,¹⁴ but we do not see this as a reality. Based on our experience, it is assumed that at least 1 full-time quality person is required to prepare, initiate, manage, analyze, and follow-up an FMEA project. In fact, considering the resource required to perform an FMEA, it is recommended to conduct it sequentially around several important SOPs every year or so even for large health care organizations.⁹ Finally, like other FMEAs, our analysis could be subjective, and depended on the experience and expertise of the FMEA focus group. Therefore, future studies are warranted to quantitatively assess the effect of the remedial actions, particularly in clinical trial settings.

Conclusion

FMEA is a powerful tool to improve quality in clinical trials by proactively and systematically identifying potential quality risks. SNUH CTC is expanding its FMEA capability to other core clinical trial processes.

Acknowledgments

We are indebted to all the staff members at SNUH CTC and the FMEA focus team. Pfizer’s Investigator Networks Site Partnerships and Infrastructure for Research Excellence (INSPIRE) program, along with its Investigator Site Development Lead Group, has been instrumental in completing the work described in this paper. We also thank Prof Yung-Jue Bang, former Director to SNUH CTC and President of the Biomedical Research Institute and Prof In-Jin Jang, current Director to SNUH CTC, for their support for FMEA at the institute. This work was supported by a grant from the Korea Health Technology R&D Project through the Korea Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI14C1060).

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.		Bhatt A. Quality of clinical trials: a moving target. Perspect Clin Res. 2011;2(4):124–128.
2.		Macri M. Managing clinical trial risk: it’s a tough job, but one person has to do it. inVentiv Health Clinical. 2014:1–4.
3.		European Medicines Agency. Reflection paper on risk based quality management in clinical trials; 2013. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2013/11/WC500155491.pdf. Accessed September 11, 2017.
4.		U. S. Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER). Guidance for Industry. Oversight of Clinical Investigations – A Risk-Based Approach to Monitoring; 2013. Available from: https://www.fda.gov/downloads/Drugs/Guidances/UCM269919.pdf. Accessed September 11, 2017.
5.		U.S. Food and Drug Administration. Clinical Investigators – Disqualification Proceedings. Available from: http://www.accessdata.fda.gov/scripts/SDA/sdNavigation.cfm?sd=clinicalinvestigatorsdisqualificationproceedings. Accessed 26 Dec 2016.
6.		U. S. Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER). Guidance for Industry. Q9 Quality Risk Management; 2006. Available from: https://www.fda.gov/downloads/Drugs/Guidances/ucm073511.pdf. Accessed September 11, 2017.
7.		Liu HC, Liu L, Liu N. Risk evaluation approaches in failure mode and effects analysis: a literature review. Expert Syst Appl. 2013;40(2):828–838.
8.		Asefzadeh S, Yarmohammadian MH, Nikpey A, Atighechian G. Clinical risk assessment in intensive care unit. Int J Prev Med. 2013;4(5):592–598.
9.		Chiozza ML, Ponzetti C. FMEA: a model for reducing medical errors. Clin Chim Acta. 2009;404(1):75–78.
10.		Scorsetti M, Signori C, Lattuada P, et al. Applying failure mode effects and criticality analysis in radiotherapy: lessons learned and perspectives of enhancement. Radiother Oncol. 2010;94(3):367–374.
11.		Kuo FY, Huang WC, Chiou KR, et al. The effect of failure mode and effect analysis on reducing percutaneous coronary intervention hospital door-to-balloon time and mortality in ST segment elevation myocardial infarction. BMJ Qual Saf. 2013;22(8):626–638.
12.		Bonfant G, Belfanti P, Paternoster G, et al. Clinical risk analysis with failure mode and effect analysis (FMEA) model in a dialysis unit. J Nephrol. 2010;23(1):111–118.
13.		Hover AR, Sistrunk WW, Cavagnol RM, et al. Effectiveness and cost of failure mode and effects analysis methodology to reduce neurosurgical site infections. Am J Med Qual. 2014;29(6):517–521.
14.		Cody RJ. Anticipating risk for human subjects participating in clinical research: application of failure mode and effects analysis. Cancer Invest. 2006;24(2):209–214.
15.		Veronese I, De Martin E, Martinotti AS, et al. Multi-institutional application of failure mode and effects analysis (FMEA) to cyberKnife stereotactic body radiation therapy (SBRT). Radiat Oncol. 2015;10:132.
16.		Rodriguez-Gonzalez CG, Martin-Barbero ML, Herranz-Alonso A, et al. Use of failure mode, effect and criticality analysis to improve safety in the medication administration process. J Eval Clin Pract. 2015;21(4):549–559.
17.		Sayler E, Eldredge-Hindy H, Dinome J, Lockamy V, Harrison AS. Clinical implementation and failure mode and effects analysis of HDR skin brachytherapy using Valencia and Leipzig surface applicators. Brachytherapy. 2015;14(2):293–299.
18.		Mayadev J, Dieterich S, Harse R, et al. A failure modes and effects analysis study for gynecologic high-dose-rate brachytherapy. Brachytherapy. 2015;14(6):866–875.
19.		Haynes AB, Weiser TG, Berry WR, et al. Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist-based surgical safety intervention. BMJ Qual Saf. 2011;20(1):102–107.
20.		Ashley L, Armitage G. Failure mode and effects analysis: an empirical comparison of failure mode scoring procedures. J Patient Saf. 2010;6(4):210–215.
21.		Wilkinson DA, Kolar MD. Failure modes and effects analysis applied to high-dose-rate brachytherapy treatment planning. Brachytherapy. 2013;12(4):382–386.
22.		DeRosier J, Stalhandske E, Bagian JP, Nudell T. Using health care failure mode and effect analysis: the VA national center for patient safety’s prospective risk analysis system. Jt Comm J Qual Improv. 2002;28(5):248–267, 209.