Challenges Associated with the Safety Signal Detection Process for Medical Devices

Introduction

Signal detection is defined by the International Medical Device Regulators Forum (IMDRF) as “The process of determining patterns of association or unexpected occurrences that have the potential to impact patient management decisions and/or alter the known benefit-risk profile of a device.^1”

The aim of safety signal detection for medical devices is to promptly identify risks associated with the use of a product.² Signals can be production related (eg, a defective batch or a released series of batches) or linked to the design and/or use. Signals can be identified during the pre-market surveillance phase using clinical trial data, or during the post-market surveillance phase using post-market data sources. The decision of whether a finding represents a “signal” and whether such finding is subject to further investigation can be challenging.

For medicinal products, quantitative safety signal detection is followed by a signal validation process during which the signal is verified to be real or not. This process is often performed through careful case evaluation. Thereafter, signal prioritization is completed depending on the strength of the signal, whether or not the signal represents a new finding, the clinical importance and potential public health implications, and the availability of preventive measures to mitigate the adverse public health impact.³

After prioritization, the marketing authorization holder together with the regulators has to decide whether additional risk minimization measures are needed to address this safety issue.²

Although the signal management for medicinal products and medical devices are conceptually equivalent, the legislation requirements are better described in the pharmaceutical regulation^4–6 than in the medical devices regulations. For the latter, the guidelines defining requirements and giving practical advice on signal management are yet to be developed.

Recent safety issues involving medical devices have highlighted the need to improve signal detection.⁷ Various European Union (EU) national competent authorities have started to focus on strengthening the analysis of vigilance data of medical devices. As a consequence, the new EU medical device regulation was published; namely, article 90 that states that the European Commission shall put in place systems and processes to actively monitor the data available in order to identify trends, patterns or signals that may reveal new risks or safety concerns.⁸

In this paper, we aim to describe aspects that influence signal detection of safety issues related to medical devices in order to identify gaps and provide recommendations for optimizing signal detection approaches.

Methods

We performed a systematic literature review to identify articles describing different aspects associated with safety signal detection for medical devices (see Table 1). We searched Embase, Medline, Cochrane, Web of science, and Google scholar using terms that included “medical device” and terms attributable to safety (“signal detection” or ”post-marketing surveillance” or “risk management”), following Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines for systematic reviews. Appendix A outlines the strings that have been used.

Table 1 Systematic Literature Review Methodology

Due to the high number of articles that resulted from the initial search, we decided to implement a restrictive query search on the abstracts from the list of selected articles in order to narrow down the selection. This restrictive query consisted of: any abstract with date range year 2004–2017 (Jan-2004 through Jan-2017) containing any of the following terms: “Signal” OR “Adverse reaction” OR “Adverse event” OR “Injury” OR “Malfunction” OR “Product Problem”.

Review of Articles

Following the query, all remaining abstracts were reviewed. Articles were excluded if the abstract review did not include any of the following items: “post-market safety data sources in medical devices”, “signal detection methodologies for medical devices” or “medical device event coding dictionaries”. Articles containing the latter terms were included in the further study. Subsequently, a full-text review was conducted for each of the remaining articles. Articles were excluded if they did not include any information related to “medical device Post-Market Surveillance (PMS) data sources” OR “Methodologies used for signal detection for medical devices” OR “Coding dictionaries for medical devices”.

All remaining full-text articles were subject to a formal evaluation to extract information on the following items (articles that did not contain at least one of the following items were excluded):

1. Type of PMS data sources on medical devices:-Spontaneous reporting systems (SRS) are reactive systems that contain reports on patient harms and product problems collected from healthcare professionals, patients, healthcare authorities and manufacturers whether reported directly or through published articles.-A medical device registry is defined by the IMDRF as an “Organized system with as primary aim to increase the knowledge on medical devices contributing to improve the quality of patient care that continuously collects relevant data, evaluates meaningful outcomes and comprehensively covers the population defined by exposure to particular device(s) at a reasonably generalizable scale. (eg international, national, regional, and health system)”¹

2. Methodologies used for signal detection for medical devices.

3. Coding dictionaries for medical devices.

Each article was scored 1, 2 or 3 points depending on its content. The total score represents the sum of all three topics; 1 point being attributed to articles containing one of the three topics, 2 points being attributed to articles containing two of three topics, and finally, 3 points being attributed to articles containing all three topics. This score serves as a measure of the articles’ completeness.

Following the author’s full-text review, KV conducted a second review of the full-text articles. KV agreed with the initial selection of the 24 articles, and the assigned score based on the described inclusion and exclusion criteria.

Results

Our initial search strategy identified a total of 20,819 articles (10,199 Embase, 8,374 Medline Ovid, 1,501 Web of Science, 545 Cochrane and 200 Google Scholar). After applying specific search restrictions, a total of 996 abstracts were identified. During the abstract review, 951 articles were excluded, due to lack of information on any of the three key contents: post-market data sources for medical devices, signal detection methodologies for medical devices and coding dictionaries for medical devices. A total of 45 articles were included for full-text review. During the review cycle, 21 articles were excluded due to the lack of information on any of the three key topics. Details of the 24 remaining articles are shown in Table 2.

Table 2 Characteristics of the 24 Selected Articles

Papers were categorized into two different categories: 11 review articles, and 13 studies (12 retrospective studies and 1 prospective study). Completeness scoring yielded 13 articles with a score of 1, 11 with a 2 point score, whereas no article scored a 3 point rating. Twenty-one articles included information on post-market data sources of medical devices, 10 articles included information on signal detection methodologies for medical devices and 4 articles included information on coding dictionaries for medical devices (Figure 1).

PMS Data Sources

Of the 21 articles including post-market data sources, 12 articles discussed SRS (one of the articles also included information on medical device registries), 9 articles discussed medical device registries (one of the articles also included information on SRS) and 1 article described a Non-Standard Data Source.

Of the 12 articles including different SRS, the following SRS were discussed: FDA MAUDE database (US), TGA DAEN database (Australia), the future European Databank on Medical devices (Eudamed) (EU), MHRA database (UK), MEDSUN database (US), Adverse Event Triggered Reporting for Devices (ASTER-D) (US), MEdical DEvices VIgilance and Patient Safety (MEDEVIPAS) (Greece), and the National Electronic Injury Surveillance System (NEISS) (US)^9–20 (see Table 2).

Of the nine articles including registries, the following were discussed: American College of Cardiology’s National Cardiovascular Registry (US), Massachusetts Angioplasty Registry (US), Kaiser Permanente Orthopedic Implant registries (US), National Cardiovascular Data Registry (NCDR) (US), database of Sprint Fidelis and Quattro Secure implantable cardioverter defibrillator leads (US), Swedish Coronary Angiography and Angioplasty Registry (SCAAR) (Sweden), European Registry of Quality Outcomes for Cataract and Refractive Surgery (EUREQUO) (EU), Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR) (Australia), Data Extraction and Longitudinal Trend Analysis (DELTA) Registry (US), and Medicare database (US claims database constituting a person-specific registry of medical histories recording the use of all hospital services that are eligible for payment, including use of medical devices)^14,21–28 (see Table 2). One article described a non-standard data source, namely, an online social networking community of people with diabetes and their caregivers or family members. This diabetes safety network captured data entered by patients in apps (see Table 2) and contained patient case reports of medical device events.²⁹

Signal Detection Methodologies

Ten articles described safety signal detection methodologies for medical devices, four articles discussed signal detection methodologies applied to SRS,^9,12,19,20 four articles discussed signal detection methodologies applied to registries,^22–24,26 and two articles discussed optimal signal detection methodologies for medical devices without applying the methodology to a specific type of PMS data source.^30,31 No articles associated with signal detection methodologies applied to non-standard data sources were identified.

Of the four articles using signal detection methodologies applied to SRS, two articles discussed disproportionate analysis (DPA) methodologies (Frequentist and Bayesian),^12,19 and two articles discussed multivariate methods (change point analysis and entity matching algorithm).^9,20 Of the four articles that included signal detection methodologies applied to registries, all four articles discussed methodologies associated with the Data Extraction and Longitudinal Trend Analysis (DELTA) network.^22–24,26

Coding Dictionaries

Of the four articles that included information on coding dictionaries for medical devices, different dictionaries and nomenclatures were used, namely, FDA codes and International Organization for Standardization [ISO], IMDRF codes for product problems and investigation results, and Systematized Nomenclature of Medicine-Clinical Terms [SNOMED CT], MedDRA_, International Classification of Diseases [ICD], FDA Patient Problem and IMDRF Patient Codes for patient outcomes.^14,25,27,32

Discussion

From this review, it is clear that spontaneous reporting systems and registries are primarily used for the medical device safety signal detection. Coding is remarkably diverse, no agreement on the preferred methods for signal detection currently exists, and no gold standard for signal detection has been established thus far.

The main publicly available SRS are the FDA MAUDE (US), TGA DAEN (Australia) and the future EU Eudamed (EU)³³ (see Table 3; available PMS data sources for medical devices). The organization and content of each SRS varies, some are based on voluntary reporting and others on mandatory reporting, and usually track suspected medical devices, suspected patient harms or product problems, and patient data collected in a centralized and structured format.¹³

Table 3 Available PMS Data Sources for Medical Devices

Per our literature review, the identified SRS are organized based on the relationship between medical devices and events. The data is available for assessment and located in a repository or database.^{11,12,16,17,21} Nevertheless, SRS suffer from different limitations including: lack of harmonized global standard data set for reporting which makes integration of data from different databases challenging,^14,34 difficulty to determine root causes for individual events conclusively due to limited information and no access to the actual device, with a large part of investigation results being inconclusive,³⁵ missing and incomplete data that impacts the evaluation of the case, underreporting due to different reasons including lack of time, uncertainty about the medical device causing the adverse event, difficulty in accessing reporting forms, lack of awareness of the requirements for reporting, and lack of understanding of the purpose of SRS,³⁶ and over-reporting where medical devices with well-known adverse event/product problems are more likely to be reported based on influence from media coverage – the so-called notoriety bias.³⁷

Despite SRS being a standard and required source for signal detection, we identified that medical device registries are important for signal detection as well and may also be used for risk quantification. Registries typically contain valuable information such as medical device information, diagnoses, medications, medical narratives and surgical interventions. Unlike spontaneous reports, medical device registries are not restricted to patients experiencing medical device product problems or patient harms. Therefore, medical device registries data provide some advantages that can be used to complement the more traditional PMS data sources (SRS), particularly the possibility to perform active PMS. In our literature search, we found that some retrospective studies have demonstrated the feasibility of an early warning detection system using medical device registries. For example, it has been demonstrated that the fracture of the Fidelis implantable cardioverter defibrillator (ICD) leads that caused inappropriate ICD shocks could have been detected much earlier if a medical device registry would have been created.^19,22

Our literature review identified different types of methodologies (depending on the type PMS data source used) that can be applied to calculate reporting associations for all medical device-event combinations. Disproportionality analysis (DPA) was used as the main signal detection method for SRS – some used frequentist and other Bayesian approaches.^12,19 These methods are well established for signal detection in drug safety. For complex types of SRS analysis, multivariate approaches have been proposed: change point analysis²⁰ or entity matching algorithm.⁹ These are not yet used for medical device safety signal detection. Methods applied to medical device registries can be categorized into those based on modified DPA ported from spontaneous reporting, and those based on the DELTA network methodology.²² Signal detection methods applied to medical device registries based on the Data Extraction and Longitudinal Trend Analysis (DELTA) network are considered automated safety surveillance tools that can competently support the detection of new potential post-market safety issues,¹⁵ complementing existing signal detection strategies and providing an additional tool to evaluate the safety of marketed medical devices.²⁶

Our literature review demonstrated that different types of coding dictionaries used for medical device signal detection currently exist; for patient harms, product problems and evaluation/investigation codes (methods, results and conclusion of the investigation). These coding systems are very heterogeneous. We conclude that there is a lack of standardization of medical device event coding across different jurisdictions. Furthermore, no mapping between some of the coding dictionaries currently exists. This issue could eventually delay the timely generation of safety signals associated with a medical device event reported in jurisdictions using different event coding dictionaries without an appropriate event code mapping.

Recommendations

Based on the analysis of the current literature on safety signal detection for medical devices and their limitations we have developed some recommendations (see Table 4):

Table 4 Limitations and Recommendations on Improving the Signal Detection for Medical Devices

PMS Data Sources: Defragmentation and Harmonization

Currently different national SRS exist; however, no global database to access spontaneous reports on medical devices has been introduced. The inconsistency in post-market reporting requirements between regions leads to different levels of completeness that makes a comparison between different SRS databases challenging. The most significant one is adverse event reporting exemption applications in the EU (amongst other exemptions, expected side-effects are not reportable in EU but subject to event trending,⁸ in Canada³⁸ and Australia,¹⁸ however, no exemptions are applicable in the US³⁹). Together with the standardization of SRS databases, harmonization of reporting criteria is needed. Otherwise, it will not be possible to identify signals from the National Competent Authority (NCA) SRS databases globally when some datasets completely exclude certain types of spontaneous reports. A multicomponent global database including reporting by manufacturers, clinicians and patients collecting spontaneous reports on adverse events related to medical devices, should be established for the assembly of all medical device reports from all National Spontaneous reports databases. The analysis of the collected data can then be performed by region or by country.

In addition, harmonization of medical device registries databases is recommended. This harmonization could be established by using the international Coordinated Registry Network (iCRNs) to maximize the potential of information collected in the international registries.¹ The International Consortium of Orthopedic Registries (ICOR) is a good example of the effective use of a distributed safety data system with harmonized data definitions and data extraction followed by evaluating the data using innovative methodology across multiple national orthopedic registries. This decentralized structure system helps overcome issues related to security, operations, legal, and those related to patient privacy.⁴⁰

In order to link and potentially merge the data received from the different PMS data sources (SRS or registries), it is also recommended that guidance on common data elements and a common standardized data set for reporting individual device cases are developed (both for SRS and registries), and adopted by the different stakeholders involved in the process of collection and extraction of safety data for signal detection purposes.

Agreement on Signal Detection Methodologies

At this time, there is no agreement on the preferred methods for medical device signal detection for each of the different PMS data sources (SRS, registries and nonstandard data sources), and thus no gold standard has yet been established. IMDRF and Health Authorities should work together to develop guidance on gold standard methodologies that should be used to mine data from the different types of PMS data sources (SRS, registries and nonstandard data sources).

Standardization of Coding Dictionaries

To ensure more efficient signal detection we recommend the global adoption of the IMDRF coding dictionary by all Health Authorities. All the existing coding dictionaries will need to be mapped to the IMDRF coding dictionary to allow for an efficient system of signal detection for medical devices. Taking into consideration that IMDRF codes are very high level with many events falling in the “no code available” category, we recommend IMDRF to develop a more granular level of coding developing additional IMDRF codes to increase specificity, when appropriate. In order to ensure consistency when selecting the appropriate codes, we also recommend the development of IMDRF coding guidelines classified by therapeutic area. IMDRF needs to ensure the maintenance of the coding dictionary, and establish the right balance between having meaningful event code categories while avoiding too much granularity.

To ensure successful implementation of this new system, a global harmonized system for Unique Device Identifiers (UDIs) needs to be implemented, the UDIs need to be consistently assembled within PMS data, and all stakeholders need to ensure a continual use of the SRS and registries, including UDIs. The establishment of a global medical device identification database will aid in accomplishing this objective. The identification of devices during the signal detection process will continue to be a hurdle until the UDI is standardized and widely utilized for some time. This is a long-term goal because it involves significant policy change. Active collaboration and support from all stakeholders will ultimately lead to the success of these recommendations.

Developments in Recent Years

PMS Data Sources

Recent research emphasizes that the underlying data received from the PMS data sources need to meet high-quality standards to ensure a timely safety signal generation. The authors of a recent case study describe PMS data as one of the main important publicly available SRS for medical device safety signal detection: FDA MAUDE.³⁵ This research outlines that the completeness and the quality of the spontaneous reports in FDA MAUDE can be improved. The authors, furthermore, highlight the difficulty to determine root causes conclusively for individual events due to limited information, and no access to the actual medical device, with a large part of investigation results being inconclusive. Amongst others, it is recommended to address these challenges by considering the possibility of enriching FDA MAUDE PMS data with data from active PMS data sources such as medical device registries. In order to be able to link the registry data with the spontaneous reports, common standardized dataset including UDI should be created.

Signal Detection Methodologies

The developments regarding the applicability of new methods to the safety signal detection of medical devices have been a wide research topic over the past few years. The research in the area of passive safety surveillance (the data-mining methods used for disproportionality analysis of medical device–adverse event combinations from SRS) has become a main research focus area.

Recent developments associated with the signal detection methodologies used for medicinal products have been applied to medical device signal detection;⁴¹ for example, the likelihood ratio test (LRT) method that is applied to perform passive safety surveillance of medicines has now been successfully used to perform passive safety surveillance of medical devices. LRT is a frequentist method based on multiple 2x2 tables. It compares the reporting rate of different adverse events for a given drug or medical device of interest. The LRT method has successfully been applied for safety signal detection purposes to medical device SRS, and can also be used as spatial-cluster signal detection for an adverse event of interest from medical device registries and other databases that have patient-level geographical information. Moreover, the LRT method was compared to other frequentists and Bayesian methods, and found to be the most conservative method when evaluating the total number of detected safety signals, given its ability to control for false-positive safety signals.^42,43

A big effort has been made in developing signal detection methods for medical device safety signal in passive safety surveillance. However, challenges still exist for the development of new active surveillance methods (statistical signal detection methods for medical device registries, and other longitudinal databases) for monitoring the safety of new medical devices over time. In medicines, this effort is currently being undertaken by the Observational Medical Outcomes Partnership (OMOP) and FDA Sentinel Initiative:

-OMOP: The OMOP is a public–private partnership involving the FDA, multiple pharmaceutical companies and healthcare providers. OMOP conducts methodological research on active drug safety surveillance by evaluating the performance of safety signal methods and their ability to identify true drug-adverse event associations. OMOP established a common infrastructure to collect different types of observational data from post-market data sources around the world, and successfully developed and implemented a large-scale signal detection methodology applied to medicines.⁴⁴

-FDA Sentinel: The FDA Sentinel is an active surveillance program that was established in the US with the long-term objective to create a national electronic system for PMS of FDA-regulated medical products (drugs, vaccines, biologics and medical devices). Over time, Sentinel has developed the largest multisite distributed database in the world dedicated to medical product safety. This new approach can help public health officers (who depend on passive surveillance tools lacking in denominator information, ie, patient exposure data) in detecting safety signals related to medicines and medical devices, and therefore aid in the accurate comparative assessments of safety risks.⁴⁵

The application of these methods in medical device safety signal detection may have the ability to address some of the challenges associated with active safety surveillance of medical devices. Further research is required to evaluate the potential applicability of these two initiatives to active safety surveillance of medical devices.

Coding Dictionaries

There have also been some initiatives to address some of the challenges associated with adverse event coding for medical devices. An IMDRF project has worked on linking IMDRF codes and MedDRA codes.⁴⁶ Moreover, IMDRF has also developed the IMDRF adverse event terminology maintenance plan; a document describing how to add, modify or delete adverse event terms to the IMDRF coding dictionary.⁴⁷ Although these projects have the potential to address some of the identified challenges, some work still needs to be completed. The development of IMDRF coding guidelines by therapeutic area, and the creation of additional IMDRF codes to increase the granularity of the IMDRF coding dictionary are crucial to enhance the current adverse event coding for medical devices.

Conclusions

We have shown that a wide range of PMS data sources, coding dictionaries and signal detection approaches are available for the detection and assessment of medical device problems and patient harms. Each of them offers unique opportunities that together can contribute to developing standards for robust, consistent and improved signal detection for medical devices.

New detection methodologies have been developed to utilize data that has not been used in the past, allowing for the introduction of new proactive models of medical device surveillance. Despite the increasing evidence of the benefits of medical device registries for the purpose of signal detection, spontaneous reports will remain a key data source of post-market device data and therefore a relevant source of potential signals. Standardized methods applied to similar data sources will be required. Data quality and coding harmonization will need to be improved and the UDI system will need to be fully implemented to benefit from the potential of proactive systems for the safety evaluation of medical devices. In order to succeed, all stakeholders involved in the PMS system must actively support each other and collaborate. This system will use real-world evidence to support regulatory decision-making across all jurisdictions.

Figure 1 PRISMA flow diagram outlining all steps for the inclusion of articles in the review.