Worldwide Trends in Registering Real-World Studies at ClinicalTrials.gov: A Cross-Sectional Analysis

Introduction

Real-world data (RWD) are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.¹ Real-world evidence (RWE) is clinical evidence regarding the usage and potential benefits or risks of a medical product derived from the analysis of RWD.¹ A real-world study (RWS) collects RWD in a real-world environment and obtains RWE of the use value and potential benefits or risks of medical products through analysis.² The coronavirus disease 2019 (COVID-19) pandemic has led to the disruption of traditional clinical trials and the activation of a large number of clinical studies. In recent years, the Food and Drug Administration (FDA) has issued numerous RWE-related guidelines to support regulatory decisions.³ A number of previous studies have assessed the characteristics of COVID-19 studies registered in the American Clinical Trials Registry (ClinialTrials.gov).^4,5 Previous studies have reviewed in detail the types of clinical research for COVID-19, particularly with regard to issues of data collection and sharing in terms of real-world data.⁶ However, the characteristics of RWSs registered worldwide for different diseases are still unknown. Efforts have been made to standardize RWD/RWE for some time. In guiding researchers to design and implement RWD research, some standards have been put forth, such as guidelines,^7,8 processes,^9–11 and templates.^12,13 Although RWSs differ greatly from traditional clinical trials in terms of research characteristics, researchers still adopt the habit of writing traditional randomized controlled trial (RCT) research protocols. Since there is still a large gap in the writing of standardized RWS research protocols, it is necessary to analyze the problems in RWS registry research to provide insights for the improvement of follow-up guidelines. Therefore, the purpose of this study was to comprehensively evaluate RWSs registered at ClinicalTrials.gov to help investigators better conduct relevant research in clinical practice.

Methods

Search Strategy

On February 28, 2023, we retrieved data using the ClinicalTrials.gov application programming interface (API) (version 1.01.05). The API contained 328 research fields (see Supplementary File). After filtering out fields with a large number of missing values that were not important, we screened only 97 valuable research fields (see Supplementary File). ClinicalTrials.gov limits API queries to 1000 records and 20 fields per use. Using the code loop, all data can be obtained within minutes. The search expression we used to build the query Uniform Resource Locator (URL) was “real-world data” OR “real-world study” OR “real-world evidence” OR “real-world research”. Python (version 3.9.12, Python Software Foundation, Fredericksburg, Virginia, USA) was used to send requests and parse returned query data. Retrieved data were manually checked to exclude clinical trials. The entire process was completed independently by two authors, and finally, the results were compared. The Python code for scraping data from ClinicalTrials.gov using its API is provided in our online Supplementary Material (see Supplementary File). This code is publicly available and can be used to make data requests with any search term.

Data Summarization and Visualization

Python was used to perform statistical analysis. Descriptive statistics were used to analyze the data. Sankey diagrams were used to show the flow between two or more categories, in which the width of each individual element was proportional to the flow rate. To ensure that only the most relevant flows were included in the Sankey diagram, we applied a threshold screening approach. This approach allowed us to visualize the most relevant flows and avoid cluttering the diagram with nonsignificant flows. We showed Sankey diagrams for conditions and countries, as well as Sankey diagrams for multicenter research numbers and countries. For data requested from the API, the field “LocationCountry” recorded the country in which each research center was located in a long string. To avoid double counting, only the country of the study sponsor was included as the country of registration for each study in this analysis. We used the same method for counts from studies involving multiple countries.

Results

Selection of Eligible Studies

A total of 960 RWSs were registered at ClinialTrials.gov. Sixteen studies were excluded due to ineligibility. Finally, a total of 944 studies were included. The flow diagram of the study screening process is shown in Figure 1.

Figure 1 Study selection flow diagram.

General Characteristics

Table 1 presents the general characteristics of the included studies. We divided the studies into two groups by status, including completed and other statuses. In 63.2% (597) of the studies, the observational model was designed as a cohort. A total of 46.8% (442) of the studies were prospective. Regarding intervention type, 42.4% (400) of the studies involved drugs, and only 9.1% (86) of the studies involved devices. In 40.7% (384) of the studies, the lead sponsor type was industry. More than one-third (37.6%, 355) of the studies were in the recruiting stage, and 27.4% (259) of the studies were completed; therefore, approximately one-third of the studies had not yet recruited participants. Only 15.1% (143) of studies were supervised by data monitoring committees. A total of 87.3% (824) of the studies were observational. The median (interquartile range) estimated sample size was 481 (158, 2000). In 14% (132) of the studies, the sample size was less than 100 participants. A total of 49.4% (466) of studies had a sample size of 500 participants and above. Overall, 63% (595) of the studies were single-center studies. Only 8.5% (80) of the studies mentioned both the detailed study design type and data source in the “Brief Summary”. A total of 22% (208) of studies defined their specific study types or data sources.

Table 1 General Characteristics of Real-World Registration Studies

Study Conditions

“Condition” and “ConditionMeshTerm” were included in the study fields provided by the ClinicalTrials.gov API. We manually compared the two fields and kept the main standardized condition names. A total of 213 conditions were covered in the included studies. Gastrointestinal neoplasms were the leading condition in the registered studies (8.6%, 81), followed by lung neoplasms (8.4%, 79) and breast neoplasms (6%, 57). Overall, one-third of the studies (32.7%, 309) involved neoplasms (or tumors). The top 20 conditions are shown in Figure 2. The top 10 conditions in China and the United States are shown in Figure 3. Among the registered studies in the United States that included tumors, 22.2% (6/27) of the United States studies did not specify particular tumor types, whereas only 3.8% (7/182) of the studies in China did not restrict specific tumor types. Out of all the disease registries that contained information on tumors, 18.5% (5/27) of the research data in the United States came from tumor registry databases. In contrast, only 1.1% (2/182) of research data in China came from tumor registry databases.

Figure 2 Number of registered studies by condition. Only the top 20 conditions are shown.

Figure 3 Number of registered studies in China and the United States by condition. Only the top 10 conditions are shown.

Global Distribution

The included studies involved a total of 48 countries. China was the leading country in terms of the total number of registered studies (37.9%, 358), followed by the United States (19.7%, 186). The sum of the number of studies in China and the United States was 57.6%, which was more than half of the total number of studies. The detailed global distribution of the number of included studies is shown in Figure 4. The top 20 countries are shown in Figure 5.

Figure 4 Number of real-world registration studies in each area.

Figure 5 Number of registered studies in the top 20 countries.

Trends in Study Registration

We counted the number of new studies registered on the ClinicalTrials.gov website and displayed it in a line graph. Figure 6 contains two subgraphs, one for the overall trend and the other grouped by country. In the country grouping, we separated out China and the United States and merged all remaining countries into an “other countries” category. From 2020 to 2022, the total number of registered studies was 60% (568).

Figure 6 Number of new studies registered at ClinialTrials.gov each year and by country. Notes: The 2023 numbers might be low because the search was performed on February 28, 2023, rather than the reduction being a true representation of the data.

Relationship Between Countries and Conditions

We used a Sankey diagram to explore countries that contributed to major condition studies. We excluded flows that represented less than four of the total number of studies flowing from a given country or to a given condition. The filtering thresholds were manually defined based on the complexity of the Sankey diagram. The Sankey diagram is shown in Figure 7.

Figure 7 Sankey diagram from country to condition. The numbers after the country and condition names indicate the total number of outflow and inflow studies, respectively. For example, “China - 195” means that a total of 195 studies from China flowed to different condition nodes on the right side of the figure. Note that since the remaining nodes are filtered out, these numbers only represent the statistical association between the nodes displayed in the Sankey diagram.

Relationship Between Countries and Multicenter Research Numbers

We excluded flows that represented less than ten of the total number of studies flowing from a given country or to a given multicenter research number. Only 3% (29) of the studies had 50 or more research centers. Therefore, the category “≥50” in the study research number classification is not shown in the Sankey diagram (Figure 8).

Figure 8 Sankey diagram from country to research center classification. Numerical research center variables are discretized into categorical variables with different intervals. The study centers were classified into five categories: “1”, “2~4”, “5~9”, “10~49”, and “≥50”. The category “≥50” is not displayed in the plot, as lines with node counts less than 10 are filtered out.

Discussion

The passage of the 21st Century Cures Act in 2016 provided a milestone for the FDA in the promotion of RWE. This can also be seen from the line chart in Figure 6, which shows 2016 as an important time inflection point. The COVID-19 outbreak that began in 2020 has boosted researchers’ interest and acceptance of RWSs. As seen in Figure 3, China and the United States are very different regarding the study of different conditions. The number of registered RWSs shows great differences in tumor diseases. By 2019, a total of 574 cancer registries had been established in China, covering a population of 438 million (accounting for 31.5% of the total population of China).¹⁴ The United States has several national cancer registries that collect data on cancer incidence, mortality, and survival rates.¹³ Some of the major cancer registries in the United States include the National Cancer Institute’s Surveillance, Epidemiology, and End Results (SEER) program¹⁵ and the National Program of Cancer Registries (NPCR). The reason behind this may be related to the differences in disease distributions in the two countries. The incidence rate and mortality rate of most digestive system malignancies in China are higher than those in the United States.¹⁴ Another important reason is that China’s national tumor registration platform has not yet been made public, which makes RWE-based oncology research largely dependent on routine diagnosis and treatment data from electronic medical records (EMRs). China’s EMR system cannot link the data of multiple medical institutions, unlike the American electronic health record (EHR) system. The number of cancer-related registration studies in China far exceeded that in the United States, mainly due to differences in disease distribution and the acquisition of data sources. Although the number of cancer-related registration studies in the United States is relatively small, it is easier to cover multiple tumor types and convene multiple research centers with the help of tumor registry platforms. The Sankey diagram from country to condition can provide researchers with ideas for selecting diseases when registering multinational multicenter studies.

When talking about RWE or RWD, it is easy to become confused by the terms that describe research design. While the term RWS is not mentioned in the FDA guidelines, the term is derived from the China National Medical Products Administration (NMPA) guidelines on RWE.² These three terms are not entirely newly proposed concepts, as data sources and study design types have not fundamentally changed. Furthermore, these terms do not define the study design. The recommended strategy is to define the interventional or noninterventional study design, the primary collection or secondary use of data in the study design, the characteristics of the comparison groups, and the assessment of prognostic certainty for the corresponding causal associations.¹⁶ The dichotomy of RCTs and observational studies does not capture the full picture of the study design. Depending on the definition of the term, it is important to note that RWD differs from data explicitly collected for research purposes in a health care setting. The degree of reliance on RWD varies with different types of study designs. Therefore, when an RCT is incorporated into RWD, RWE can also be generated. The review by Baumfeld Andre et al outlined the hybrid trial methodology in detail to encompass studies with RCT characteristics and RWD.¹⁷ The study by Taur et al introduced the observational design of the RWS and enumerated the advantages and disadvantages of common study types.¹⁸ From this perspective, for studies registered at ClinicalTrials.gov, the description of the study design in the study field “Brief Summary” is very confusing. Therefore, special attention needs to be given to the correct and comprehensive description of the study design in the Brief Summary, thereby promoting communication and understanding.

Compared to the strict research registration process of RCTs, there are currently no necessary requirements for the registration of RWSs. When interpreting the results, it must be noted that due to the lagged nature of self-reporting on ClinicalTrials.gov, the characteristics of some studies may be biased from the actual conditions of the groups that registered the studies. Some common characteristics, such as study execution status or enrollment numbers, are neglected when research teams fail to make timely updates to registries. Previous studies have pointed out the shortcomings of registered studies on ClinicalTrials.gov.¹⁹ It has been reported that in RCTs,²⁰ the discrepancy between study registration results and published results is still relatively high due to reasons such as the research team neglecting to update. Similar studies have also raised the need for improvements in the quality of registry data and the reporting of studies for pragmatic trials.²¹ Our research similarly reveals numerous deficiencies in RWS registries.

Regulators and academics are working hard to promote the writing of RWS proposals. The 2013 Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT 2013) provide a good reference for the writing of clinical trial protocols.²² The main applicable target of SPIRIT 2013 is RCTs. RWSs have great variability in design and analysis choices, and the currently used SPIRIT 2013 does not reflect the characteristics of RWSs very well. The National Evaluation System for Health Technology Coordinating Center in the United States released two frameworks related to data quality and research methods in 2020 as part of its work to enable and support the use of RWE to better understand certain medical devices.²³ This methodological framework outlines the structure of a device study protocol and the importance of prespecification to support the rigorous design and execution of RWSs. A novel RWE framework tool was developed and pilot tested by researchers from the pharmaceutical industry to support the interactive visualization of study designs.²⁴ A structured template, Structured template and reporting tool for real world evidence (STaRT-RWE), developed by a public‒private consortium serves as a guiding tool for designing and conducting reproducible RWSs, facilitating reproducibility, validity assessment, and evidence synthesis.²⁵ This research provided a STaRT-RWE example library containing 4 common use cases, which is also of great reference value for the writing of research proposals.²⁵ An extended research framework examining long-term safety and efficacy using RWD was recently reported in a study accredited by the International Society for Pharmacoepidemiology.²⁶ Although the COVID-19 pandemic has provided new opportunities for RWSs, the rigor of scientific research still needs to be emphasized.^27,28 Compared with the retraction of previous clinical studies, the fierce publishing competition during the COVID-19 pandemic has produced a series of side effects.^29–31 The pandemic has also led to a degree of abuse in promoting the widespread use of RWD.³²

The three main components of RWSs (the triad of research question, design, and data) are important when evaluating research validity.³³ Therefore, a visual framework is recommended for demonstrating study designs and data observability.³⁴ In addition, there have been many efforts to improve the transparency and reporting standards of RWSs.^35–40 Regarding the challenge of integrating data from multiple sources, the FDA Sentinel Innovation Center recently presented a roadmap and outlined plans and visions for the future.^41,42 The Center for Drug Evaluation of the NMPA recently drafted the Guidelines for Design and Protocol Frameworks for Real-World Drug Studies (draft open for comments).⁴³ The guidelines mention that the source of research data should be clarified, including the research center from which they came, the start and end time of data collection, and the system and record format of data storage. If the data are derived from previous research, the form of recording and storage of the original data should be described to ensure the traceability of the research data.⁴³

Our study has the following limitations. First, the registry studies we included were only from ClinicalTrials.gov. Second, we did not search for published RWSs more broadly to aggregate this information for a more in-depth analysis. Finally, this study is of interest to researchers who are new to RWSs. Therefore, we did not explain the details of the various study designs from a statistical point of view. We focused on providing readers with an analysis of research hotspots in RWSs, as well as reducing misunderstandings and summarizing more comprehensive research design tools and resources.

Conclusion

Although the pandemic has provided new opportunities for RWSs, the rigor of scientific research still needs to be emphasized. Special attention needs to be given to the correct and comprehensive description of the study design in the Brief Summary of registered studies, thereby promoting communication and understanding. In addition, deficiencies in ClinicalTrials.gov registration data remain prominent. Although most of the registered studies describing the 3 most commonly mentioned diseases were from China, the reasons behind this were related to the source of the data and the distribution of the diseases.