Real world data: an opportunity to supplement existing evidence for the use of long-established medicines in health care decision making

Background

New therapeutic products are typically approved following many years of rigorous randomized controlled trial (RCT) evaluation to ensure their efficacy and safety. The foundation of this evidence comes from carefully designed, adequately powered RCTs, which seek to evaluate the efficacy and safety of medications in a specifically enrolled and well-controlled target population. Post approval, medications are used in the real world setting as well as being evaluated in post approval studies. In clinical practice, data on medication use are routinely captured in different formats, including information on efficacy and tolerability in populations that extend beyond those enrolled in clinical trials. Real world data (RWD) refer to data associated with patient health, collected from sources other than RCTs; these data may be used for decision-making purposes.^1,2 RWD can be analyzed to produce real world evidence (RWE), that is, evidence from RWD on the usage and/or benefits and risks of a medication or a medical product.^1,3 Not only do RWD support clinical interpretation of how products act in more diverse patient populations, but they may also point to additional therapeutic benefits or uses beyond those originally studied in RCTs.^4,5 This article critically considers the current and future role of RWD in the regulatory approval process and discusses important key issues surrounding the use of these data.

Evolution of clinical trials and regulatory processes

Clinical trials in various formats have been conducted for millennia, but the first controlled clinical trial is attributed to James Lind in 1747, who found that citrus fruits could prevent and cure scurvy; the first RCT was conducted for streptomycin in 1946. Despite the long history of clinical trials, it was not until 1947, following the atrocities committed during World War II, that the Nuremburg Code was formulated; it provided the first international guidance for ethical human experimentation.⁶ In 1964, the World Medical Association developed the Declaration of Helsinki, “as a statement of ethical principles for medical research involving human subjects, including research on identifiable human material and data”. It has undergone several revisions over the years and now addresses issues such as posttrial access to interventions, research registration, and the publication and dissemination of results.⁷

The development and evolution of ethical and regulatory guidelines in the 20th century have allowed and enabled clinical trials to evolve, introducing scientific rigor in establishing efficacy and safety of medications through introduction of controls, randomization, placebos, and double-blind assessments. Consequently, RCTs have become the gold standard in study trial design.⁸

Types of clinical evidence: RCTs versus RWD

In traditional evidence-based medicine, hierarchies have been established to rate the relative strength of evidence of a particular clinical study methodology. Different levels of hierarchies exist, with RCTs considered to provide higher-quality evidence than observational studies, because of their scientifically rigorous and robust methodology and randomized setting that eliminates both measured and unmeasured confounding. Typically, RCTs measure the short- or long-term efficacy, and/or tolerability, of a medical intervention versus a known comparator (or placebo), using a carefully selected and narrowly defined homogenous patient population. These trials are then conducted under standardized and controlled conditions in order to minimize bias and potential confounders (Table 1).^9,10 Meta-analyses and systematic reviews of RCTs are largely thought to provide the next highest-quality evidence (Figure 1),^11–15 but they can suffer from limitations that do not exist in individual RCTs, for example, heterogeneity of results, inclusion of poor-quality data, and publication bias.^12,15,16 Indeed, while a high-quality RCT holds a “higher” position in the level-of-evidence hierarchy compared with observational studies due to a greater capacity to attribute treatment effect, a poor-quality RCT with biased results will inherently lower the quality of any given meta-analysis through its inclusion. Evidence hierarchies provide useful guidance; however, in the process of clinical decision making, all available evidence should be critically appraised.¹⁶

Table 1 Overview of the strengths and weaknesses of randomized controlled trials9,10

Figure 1 General hierarchy of study types.11–15

Most institutes now have transitioned to electronic health records (EHRs) resulting in a rise in the prevalence of collecting and reporting clinical data. In addition, the breadth and quality of the RWD being collected are improving incrementally. These data can now allow scrutiny at a granular level and have the potential to inform dosing, the types of patients who are prescribed a particular medication, and the impact of the use of a pharmaceutical intervention on real world patient outcomes. In addition to clinical outcomes, RWD databases also enable assessment of health care utilization and associated costs, thus further improving understanding of the broader impact of the use of a particular medication. RWD may also allow for longer-term follow-up of patients being evaluated, compared with RCTs.

The level of evidence is particularly important when guidelines are developed by international scientific organizations.^17,18 Despite the various advantages of RWE, only a few guidelines currently review RWD, and few actually use RWE to guide clinical practice recommendations.^1,2,5,19 As the robustness of the information available from RWD increases, scientific organizations and regulatory authorities should consider elevating the status of RWE. An integral part of this process will be to understand and address the barriers that currently challenge the application and utility of RWD.²⁰

RCTs have high internal validity and, for this reason, remain the gold standard for informing both pre- and postmarket regulatory decision making. For example, data from RCTs are used in initial registration of a medication, expansion of a product label to include a novel indication, postauthorization safety surveillance, and development and support of treatment guidelines.²¹ However, RCTs have limited external validity since the results from a study conducted in a specially selected homogeneous patient subpopulation, and carried out under ideal circumstances, may not necessarily extend well to the general (heterogeneous) patient population with complex care needs, multiple comorbidities, or concomitant medications.^4,21 Indeed, in a recent review discussing RWE from a China perspective, Sun et al reflected on RWE versus RCT data, and proposed that “there is a continuum in the study features of traditional RCTs and real world studies, with external validity increasing as more real world features are included in the design”.²² As RCT data are easier to combine into a meta-analysis than observational data, the latter are often considered as supportive evidence; that is, they are not recognized as causal evidence for guidelines or as providing key support in regulatory approval. However, within the past decade there are early indications from international regulators to suggest that RWD are starting to be recognized as a strong repository, although specific guidance needs to be developed regarding how data can be optimally collected and analyzed.^2,5,20 Observational studies can address important clinical questions that RCTs cannot answer, such as the expanded efficacy or tolerability of therapies in nontarget populations. Hence, observational studies may extend our knowledge of products to provide a broader view of a medication’s place in clinical practice, rather than in a niche, controlled population. This knowledge will come from a wide range of potential sources of RWD, each with its associated strengths and limitations (Table 2).^2,23–25 These sources include EHRs, patient/disease registries, routinely collected administrative and insurance claims data, primary and secondary patient-level data collection (prospective and retrospective), and population health surveys, as well as emerging sources, such as social media and data collected via mobile devices and apps.²¹

Table 2 Overview of the strengths and limitations of different types of real world data sources2,23–25 Abbreviations: EHR, electronic health record; RCT, randomized controlled trial; RWD, real world data.

Potential sources of RWD

In essence, RWD originate from a variety of sources associated with, or used in, routine clinical settings (Table 2).^2,23–25 The majority of sources are available electronically and, combined with recent technological advances that make systematic data collection increasingly easy, integration of a wealth of data from different sources is becoming possible. The significant increase in the use of EHRs within clinical practice, combined with the surge in technology applications that record health information, means that the amount of information available to track the use of medication products in the real world has never been greater.

A number of databases have emerged that collate the growing volume of RWD (eg, Optum^®, Premier Healthcare Database, Truven Health MarketScan^® [IBM Watson Health], Kantar Health’s National Health and Wellness Survey, and IQVIA™). These global databases store data relating to large numbers of patients, including physician and patient survey data, costs, medications and procedures, laboratory results, and outcomes, alongside demographic and socioeconomic status. This information can be analyzed by health care professionals to generate RWE for research hypotheses that cannot be addressed by RCTs. An institution perfectly placed to contribute a wealth of RWD is the National Health Service (NHS) in the UK. The NHS provides health care from birth to death for the entire UK population, and therefore holds a vast repository of RWD. However, a problem up until now has been that the data are held in many different databases, and therefore have been an underutilized resource. Initiatives designed to collect and link these data are underway, such as the Clinical Practice Research Datalink, which is the world’s largest database of anonymized, longitudinal primary care medical records and is also linked to secondary care datasets.²⁶

There is also a growing trend for hospitals and primary practices to merge under the umbrella of integrated delivery networks and accountable-care organizations. This merging of multiple institutes requires alignment of their processes and treatment algorithms, as well as their ability to track data. This necessitates a common, unified, highly sensitive, and detailed EHR system. These large databases can quickly become resources for analyzing outcomes and impacts of various interventions.²⁷

The increasing volume of RWD needs to be balanced against data integrity and usability. Indeed, IBM classify big data, that is, large and complex datasets, into four categories (four Vs): volume, velocity, variety, and veracity.²⁸ The complexity and robustness of data will vary for each source. This underscores why some health care providers and guidance committees have long-standing concerns about using RWD when making key regulatory decisions. Observational studies of RWD, such as cross-sectional studies or surveys, case–control studies, and cohort studies,¹⁵ are often conducted in larger patient populations than would be enrolled in an RCT, and are not subject to the recruitment biases found with RCTs.⁴ However, this strength in validity is achieved at the expense of the robustness of interpreting the findings. In addition to selecting the most appropriate source of RWD, understanding the nuances of each data source is critical for researchers using RWD to supplement data from RCTs.²⁹ For example, although claims data provide useful information on baseline characteristics and comorbidities of patients taking medications, thereby facilitating a group-wise comparison, they provide only a “snapshot” rather than a longitudinal assessment regarding cause and effect.³⁰ In comparison, EHR data can be more complex and provide an assessment of cause and effect through longitudinal assessment of patient care, but sophisticated analysis tools are required to decipher the large amount of data captured by EHRs.³¹ Only with the most appropriate methods of analysis, capable of extracting meaningful patterns of information from the best sources of RWD, will acceptance of RWE move forward in the opinion of many key decision makers, to become useful beyond observational studies.

Potential uses of RWD

RWD are increasingly recognized as an important contributor to the evidence originating from traditional RCTs; however, application stops short of having a real influence in decision making.^2,32,33 A major driver for the integration of RWE into the clinical decision-making process is understanding that RWD may provide valuable information on both short- and long-term medication safety and effectiveness in patient populations that are not well represented in RCTs. RWD may also be used to provide insights into other aspects of medication use in clinical practice, including how a medication compares with other therapies in terms of surrogate outcomes, for example, biomarkers, clinical uptake, safety, and cost.³⁴ Through this application, RWD plays a critical role in bridging the gaps in information that cannot be met by RCTs alone. In resource-scarce health care environments, RWD may increasingly be used to generate more cost- and time-efficient postmarketing evidence than RCTs.

The use of RWD may also change throughout the lifecycle of a product, particularly postauthorization (Figure 2). For example, RWD requirements are already mandatory in some European Union (EU) countries, with positive evidence being essential to maintain access to medications in countries such as France, Italy, the Netherlands, Belgium, and the Nordic countries.³⁵ Health technology assessment (HTA) agencies in the EU use RWD to provide evidence for relative effectiveness assessment of medications in the settings of pharmacoeconomic analyses, initial reimbursement discussions, and conditional reimbursement schemes. However, a recent study found that policies for the use of RWD in relative effectiveness assessment vary across agencies, which presents challenges when using these data for HTA.³⁶ Therefore, in order to fully exploit the potential of RWD, agencies and pharmaceutical companies need to collaborate in order to provide guidance on data acquisition, management, and use across a range of settings.³⁷

Figure 2 Schematic illustration of the utilization of randomized controlled trial data and real world data through the lifecycle of a medical intervention.

One area where RWE may be particularly useful is for medications that have shown beneficial effects in indications for which they are not approved. In such cases, real world use of the medication may provide adequate effectiveness and safety evidence to meet the requirements of regulatory authorities to grant approval for the expanded indications. Indeed, the US Food and Drug Administration (FDA) has suggested that RCTs using RWD may have a role in supporting regulatory approval of new indications or label expansion for existing medications.³⁸ This may be especially pertinent in situations where, due to the wide availability of data in many different types of patients, a medication has already demonstrated efficacy “off label”, but no, or only small-scale, RCT observations can support its approval.³⁹

Some professional bodies and organizations involved with coverage and reimbursement decisions are seeking to use RWD to help inform health care policies.^2,40 Regulatory decision makers, such as the FDA and the European Medicines Agency (EMA), are already accepting RWE in the form of postauthorization drug safety surveillance studies.^23,41 The FDA has suggested that they may start to consider RWD alongside clinical RCTs when evaluating new therapies, devices, or indications of existing products, but particularly in the evaluation of medical devices in real world clinical practice.^1,42 Other attempts to review evidence systematically have also gained momentum, including those by private and public health plans.^2,43 Stakeholders stand to gain from the development of an RWD-inclusive evidence strategy in health care, including physicians, patients, guidelines, committees, regulators, researchers, reimbursement committees and payers. The key to using RWD more effectively to guide decision making may lie in the validity of observations surrounding the risk–benefit ratio.

Role of RWD in determining the risk–benefit profile of a medicine

Establishing a risk–benefit profile is a pivotal part of the regulatory process for assessing the safety of medicines. As a result, regulatory agencies are increasingly viewing safety through the lens of the risk–benefit ratio alongside issues of cost; analysis of RWD can help in assessment of both. For example, both the FDA and EMA have implemented a proactive approach toward safety surveillance and overall risk assessment.⁴⁴ These regulators have similar data needs and are driven by common objectives with regard to the identification, monitoring, and minimization of risk.

A united approach within the international pharmaceutical community in assessing the risks and benefits of medications will be essential, as any significant differences would inevitably lead to increased pre- and postmarketing approval costs. In today’s global market environment, robust and generalizable data on risk–benefit assessment also facilitate the exchange of information between major regulators. There is a growing appreciation of the benefit of an expanded approach to gathering information and providing evidence on risk–benefit assessment to patients. For example, the EMA has recently advocated an “adaptive pathway” approach in order to improve timely access for patients to new medicines; this includes assessment of risk–benefit.45 This approach builds on regulatory processes already in place within the existing EU legal framework, developed mainly for conditional approval of medicines addressing life-threatening illnesses.⁴⁵ These patient registries, in combination with other pharmacovigilance tools, allow collection and development of risk-management plans to augment RCT data. Utilizing a broader set of information for assessing the risk–benefit profile, the adaptive pathway seeks a balance between shortening approval times for promising medicines while ensuring there is sufficient information on the safety and benefits of a particular medicine.⁴⁵ Adoption of this progressive approach by other international bodies would enhance use of RWE from their own health care settings.^2,41,46 The challenge in maximizing uptake of RWE will be finding which data sources provide the most appropriate observations to support specific regulatory or guideline changes.

Maximizing the applicability of RWE to inform aspects of health care and decision making

With the current focus on maximizing the utility of RWE, publications have aimed to provide guidance on conducting real world studies in order to maximize their credibility and usefulness within the regulatory approval setting.^20,24,47,48 From a regulatory perspective, RWE has the potential to inform regulators on many fronts (Figure 3), but concerns raised often surround accuracy, reproducibility, and validity of the records, as well as reproducibility of analyses, and the handling of missing data.³² For example, claims data are used for different purposes than EHRs and could present biased data by their purpose in supporting reimbursements.^2,30 Alternatively, EHRs may supply information on prescribing practices, but not on the filling or dispensing of medications. Therefore, maximizing the applicability of RWE to inform decision makers will depend on well-designed post hoc analyses, using the most appropriate data sources and analytics.⁴⁸ Bringing together the restrictions around RCT methodology and the breadth and strengths of RWD, it may not be until an RCT can be conducted within an RWD source that we see RWE being routinely used in decision making beyond that of rare diseases, or expansion of existing product labels. Indeed, Hernán and Robins suggest that large observational databases – the so-called big data – can be used to emulate an RCT (the target trial) when the latter is not feasible.⁴⁹ Moreover, the authors describe a framework for comparative effectiveness research using observational studies, analyzing how the protocol components of a target trial may be considered in the design and analysis of observational studies to maximize emulation while acknowledging the inherent limitations.⁴⁹

Figure 3 Interconnected platform for maximizing the use of real world evidence.

Despite key advantages and the wealth of RWD already available, limitations in its application are well known. Key stakeholders need to ensure that the terminology surrounding different RWE is clearly defined and used consistently. In particular, the distinction between RWD and RWE, efficacy and effectiveness, and the strengths and weaknesses of each data source should be clearly defined and understood (Table 2).^2,23–25 Concerns surrounding informed consent privacy and data integration also need to be addressed, and protocols developed to maintain data integrity. Methodical challenges will need to be worked out to understand what RWD can and, importantly, cannot provide. In order to provide robust and reliable evidence, analytical experts should develop reproducible, standard analytical approaches within the most appropriate data source.^5,49 Missing data are a known limitation; however, accuracy of data records will always be a potential confounder when using any database or other RWD source, where the personnel capturing the data may not all be following the same protocols. Finally, regulatory bodies need to have a clear alignment on policies for the use of RWD,³⁶ provide direction on the evidence they require in order to evaluate innovative or existing medicines appropriately, and disseminate information on how data sharing can work to the benefit of all. Considering these challenges, the onus will be on health care providers and payers to become more experienced in assessing the quality of and outcomes from RWE, and associated data sources.

Conclusion

Although RWD are generally accepted as an adjunct to RCTs, more work must be done to clarify which types of RWD and RWE are robust enough to provide information on aspects such as risk–benefit assessment and cost-effectiveness analysis to support guideline and regulatory decisions (Figure 3). While the importance of data from RCTs is well known for establishing the efficacy and safety of new medical interventions, comprehensive assessment of therapeutic strategies requires evaluation of efficacy under optimal conditions (high internal validity) as well as effectiveness in real-life populations and situations (high external validity). RWE therefore has the potential to support, improve, and potentially accelerate the delivery of safe and cost-effective therapeutic interventions, or to support the expansion of approved therapeutics into a new indication.

Acknowledgment

Medical writing support was provided by Karen Burrows, MPhil, and Radhika Bhatia, PhD (Engage Scientific Solutions, Horsham, UK) and was funded by Pfizer.

Author contributions

All authors were involved in the design, drafting and critical revision of the manuscript, approved the final version, and agree to be accountable for all aspects of the work.

Disclosure

The authors are full-time employees of Pfizer and report no other conflicts of interest in this work.

References

1.		Food and Drug Administration (FDA). Use of real-world evidence to support regulatory decision-making for medical devices. Available from: https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm513027.pdf. Accessed April 5, 2017.
2.		Garrison LP Jr, Neumann PJ, Erickson P, et al. Using real-world data for coverage and payment decisions: the ISPOR Real-world Data Task Force report. Value Health. 2007;10(5):326–335.
3.		US Congress [webpage on the Internet]. H.R.34 – 21st Century Cures Act. 114th Congress (2015–2016). Available from: https://www.congress.gov/bill/114th-congress/house-bill/34/. Accessed August 28, 2017.
4.		Cohen A, Goto S, Schreiber K, et al. Why do we need observational studies of everyday patients in the real-life setting? Eur Heart J Suppl. 2015;17(Suppl D):D2–D8.
5.		Berger M, Daniel G, Frank K, et al. A framework for regulatory use of real-world evidence. Available from: https://healthpolicy.duke.edu/sites/default/files/atoms/files/rwe_white_paper_2017.09.06.pdf. Accessed August 24, 2017.
6.		Bhatt A. Evolution of clinical research: a history before and beyond James Lind. Perspect Clin Res. 2010;1(1):6–10.
7.		World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191–2194.
8.		Davies MJ, Kermani F. FDA and clinical drug trials: a short history. In: Davies MJ, Kermani F, editors. A Quick Guide to Clinical Trials. 1st ed. Rockville, MD: BioPlan Associates; 2008.
9.		Booth CM, Tannock IF. Randomised controlled trials and population-based observational research: partners in the evolution of medical evidence. Br J Cancer. 2014;110(3):551–555.
10.		Nallamothu BK, Hayward RA, Bates ER. Beyond the randomized clinical trial: the role of effectiveness studies in evaluating cardiovascular therapies. Circulation. 2008;118(12):1294–1303.
11.		Guyatt GH, Sackett DL, Sinclair JC, et al. Users’ guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. JAMA. 1995;274(22):1800–1804.
12.		Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg. 2011;128(1):305–310.
13.		Price D, Chisholm A, van der Molen T, et al. Reassessing the evidence hierarchy in asthma: evaluating comparative effectiveness. Curr Allergy Asthma Rep. 2011;11(6):526–538.
14.		Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–926.
15.		Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nurs. 2003;12(1):77–84.
16.		Barton S. Which clinical studies provide the best evidence? The best RCT still trumps the best observational study. BMJ. 2000;321(7256):255–256.
17.		National Institute for Health and Clinical Excellence [webpage on the Internet]. Developing NICE guidelines: the manual. Available from: https://www.nice.org.uk/process/pmg20/chapter/introduction-and-overview. Accessed May 11, 2018.
18.		Lis Y, Guo JJ, Roberts MH, Kamble S, Raisch DW. A comparison of US Food and Drug Administration and European Medicines Agency regulations for pharmaceutical risk management: report of the International Society for Pharmacoeconomic and Outcomes Research Risk Management Working Group. Available from: https://www.ispor.org/sigs/riskbenefitmanagement/A-comparison-of-US-Food-and-Drug-Administration-and-European-Medicines-Agency.pdf. Accessed May 11, 2018.
19.		Bell H, Wailoo AJ, Hernandez M, et al. The use of real world data for the estimation of treatment effects in NICE decision making. Available from: http://nicedsu.org.uk/wp-content/uploads/2018/05/RWD-DSU-REPORT-Updated-DECEMBER-2016.pdf. Accessed June 1, 2018.
20.		Forum on Drug Discovery, Development, and Translation; Board on Health Sciences Policy; Health and Medicine Division; National Academies of Sciences, Engineering and Medicine [webpage on the Internet]. Examining the impact of real-world evidence on medical product development: I. Incentives: proceedings of a workshop—in brief. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29431954. Accessed March 3, 2018.
21.		Ho PM, Peterson PN, Masoudi FA. Evaluating the evidence: is there a rigid hierarchy? Circulation. 2008;118(16):1675–1684.
22.		Sun X, Tan J, Tang L, et al. Real-world evidence: experience and lessons from China. BMJ. 2018;360:j5262.
23.		European Medicines Agency. Update on real world evidence data collection. Available from: http://ec.europa.eu/health/files/committee/stamp/2016-03_stamp4/4_real_world_evidence_ema_presentation.pdf. Accessed April 5, 2017.
24.		Miani C, Robin E, Veronika Horvath V, et al [webpage on the Internet]. Health and Healthcare: Assessing the real-world data policy landscape in Europe. Available from: https://www.rand.org/pubs/research_reports/RR544.html. Accessed May 11, 2018.
25.		Sedgwick P. Prospective cohort studies: advantages and disadvantages. BMJ. 2013;347:f6726.
26.		Medicines and Healthcare Products Regulatory Agency (MHRA) [webpage on the Internet]. The Clinical Practice Research Datalink (CPRD). Available from: https://www.cprd.com/home/. Accessed August 2, 2017.
27.		Cooney A [webpage on the Internet]. Examining the importance of IDNs: where their influence matters and where it doesn’t. Available from: https://www.pm360online.com/examining-the-importance-of-idns-where-their-influence-matters-and-where-it-doesnt/. Accessed November 5, 2017.
28.		IBM Big Data and Analytics Hub. The four V’s of big data. Available from: http://www.ibmbigdatahub.com/infographic/four-vs-big-data. Accessed November 3, 2017.
29.		Schaumberg DA, McDonald L, Shah S, et al. Evaluation of comparative effectiveness research: a practical tool. J Comp Eff Res. Epub 2018 Feb 21.
30.		Birnbaum HG, Cremieux PY, Greenberg PE, et al. Using healthcare claims data for outcomes research and pharmacoeconomic analyses. Pharmacoeconomics. 1999;16(1):1–8.
31.		Daniel RM, Cousens SN, De Stavola BL, et al. Methods for dealing with time-dependent confounding. Stat Med. 2013;32(9):1584–1618.
32.		Annemans L, Aristides M, Kubin M [webpage on the Internet]. Real-life data: a growing need. Available from: https://www.ispor.org/news/articles/oct07/rld.asp. Accessed May 11, 2018.
33.		Chatterjee S, Davies MJ, Khunti K. What have we learnt from “real world” data, observational studies and meta-analyses. Diabetes Obes Metab. 2018;20 Suppl 1:47–58.
34.		Bipartisan Policy Center. Using real-world evidence to accelerate safe and effective cures. Available from: https://bipartisanpolicy.org/wp-content/uploads/2016/06/BPC-Health-Innovation-Safe-Effective-Cures.pdf. Accessed June 1, 2018.
35.		Baird J. Real-world evidence in Europe. Available from: http://fgcasal.org/LibroRWD/Libro_RWD_Jesammy_Baird.pdf. Accessed June 6, 2018.
36.		Makady A, van Veelen A, Jonsson P, et al. Using real-world data in health technology assessment (HTA) practice: a comparative study of five HTA agencies. Pharmacoeconomics. 2018;36(3):359–368.
37.		Wise J, Moller A, Christie D, et al. The positive impacts of real-world data on the challenges facing the evolution of biopharma. Drug Discov Today. 2018;23(4):788–801.
38.		National Academies of Sciences, Engineering, and Medicine. 2017. Real-World Evidence Generation and Evaluation of Therapeutics: Proceedings of a Workshop. Washington, DC: The National Academies Press. https://doi.org/10.17226/24685. Accessed May 11, 2018.
39.		European Voice. Fast-tracking medicines innovation: a question of uncertainties. Available from: http://www.politico.eu/wp-content/uploads/2015/03/Fast-tracking-medicines-PDF.pdf. Accessed April 5, 2017.
40.		Alemayehu D, Cappelleri JC, Emir B, Zou KH. Statistical Topics in Health Economics and Outcomes Research. Boca Raton, FL: CRC Press, Taylor and Francis Group; 2017.
41.		The Network for Excellence in Health Innovation. Real-world evidence: a new era for health care innovation. Available from: http://www.nehi.net/writable/publication_files/file/rwe_issue_brief_final.pdf. Accessed May 11, 2018.
42.		Resnic FS, Matheny ME. Medical devices in the real world. N Engl J Med. 2018;378(7):595–597.
43.		ABPI. The vision for real world data – harnessing the opportunities in the UK – demonstrating value with real world data. Available from: http://www.abpi.org.uk/media/1378/vision-for-real-world-data.pdf. Accessed May 11, 2018.
44.		Lis Y, Roberts MH, Kamble S, et al. Comparisons of Food and Drug Administration and European Medicines Agency risk management implementation for recent pharmaceutical approvals: report of the International Society for Pharmacoeconomics and outcomes research risk benefit management working group. Value Health. 2012;15(8):1108–1118.
45.		European Medicines Agency [webpage on the Internet]. Adaptive pathways. Available from: http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/general/general_content_000601.jsp. Accessed April 6, 2017.
46.		Holtorf AP, Watkins JB, Mullins CD, et al. Incorporating observational data into the formulary decision-making process – summary of a roundtable discussion. J Manag Care Pharm. 2008;14(3):302–308.
47.		Roche N, Reddel H, Martin R, et al. Quality standards for real-world research. Focus on observational database studies of comparative effectiveness. Ann Am Thorac Soc. 2014;11 Suppl 2:S99–S104.
48.		Willke RJ, Mullins CD. “Ten commandments” for conducting comparative effectiveness research using “real-world data”. J Manag Care Pharm. 2011;17(9 Suppl A):S10–S15.
49.		Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol. 2016;183(8):758–764.