Harmonizing Case Validation for Vaccine Safety in Real-World Data: The VAC4EU Experience

Introduction

Vaccines prevent millions of deaths and diseases annually.¹ Monitoring their benefit-risk balance is essential throughout the product lifecycle.² As vaccines are administered to generally healthy individuals, the tolerance for safety issues is much lower compared with therapeutic products.³ In this context, real-world evidence (RWE) derived from real-world data (RWD) plays a crucial role in evaluating the safety and effectiveness of vaccines in the post-licensure phase.

Health authorities such as the European Medicines Agency (EMA) and the US Food and Drug Administration (FDA) now widely support the use of RWD in vaccine safety studies, provided that outcomes are rigorously validated to ensure data reliability, relevance, and traceability.⁴ In particular, the FDA’s recent guidance on RWD and RWE underscores the critical role of validation processes in regulatory decision-making, particularly when using electronic health records (EHRs) and medical claims data.⁴ The guidance emphasizes the importance of data reliability, defined as the accuracy and consistency of data over time; relevance, referring to how well the data elements reflect the concepts of interest; and traceability, which allows verification of the data source and transformations. It prioritizes comprehensive data capture, mitigation of missing information, and effective data linkage. The guidance also recommends evaluating whether data sources adequately represent study populations, exposures, and outcomes. Suggested validation approaches include complete verification or performance assessments through sampling strategies, tailored to specific study contexts.⁴

The European Network of Centers in Pharmacoepidemiology & Pharmacovigilance (ENCePP) Methods Guide highlights the potential impact of misclassification of exposure, outcomes and covariates on study results and advocates for estimating sensitivity, specificity, and positive predictive values.⁵ Misclassification can introduce bias into effect estimates and validation of outcomes can provide an estimate of the error and bias, manual validation against medical charts remains the gold standard for outcome validation; however, algorithm-based or indirect approaches may be acceptable alternatives when chart review is not feasible.^6–8

The EMA’s Guideline on Good Pharmacovigilance Practices (GVP) emphasizes the use of routinely collected healthcare data, such as electronic health records, disease registries, hospital discharge databases, or insurance claims, for large-scale studies, despite challenges like incomplete data and limited follow-up. The EMA stresses selecting data sources that balance validity and efficiency while adhering to privacy regulations.⁹

The Brighton Collaboration (BC) aims to harmonize vaccine safety data collection and has created more than 60 case definitions for various outcomes.¹⁰ The BC case definitions aim to provide an ordinal Level of Certainty (LOC) scale that classifies cases by diagnostic certainty based on the strength of available evidence. They are well recognized by regulators and are intended to be applicable in different types of data collections: prospectively in clinical trials, during passive reporting, and retrospectively in surveillance systems or RWD.¹¹ BC case definitions are typically developed by clinical experts with a prospective mindset, and implementation for retrospective use in heterogeneous RWD is more challenging because the required clinical details are inconsistently recorded, moreover the level of recording and access to source data varies largely across different RWD systems. Although the BC criteria are explicit, absence of information in RWD may be interpreted differently by different abstractors. During the H1N1 pandemic vaccine safety studies, application of BC case definitions across countries yielded high variability in LOC for Guillain-Barré syndrome (GBS) and narcolepsy cases, across settings.^12–14 A Danish study also described that traditional trial-based definitions may not directly fit real-world data.¹⁵

The Vaccine Monitoring Collaboration for Europe (VAC4EU) is a non-for-profit association of institutions that aim to generate high-quality evidence on vaccines to support regulatory decision-making worldwide. The VAC4EU validation pipeline was set up because of the need of a harmonized approach to validation in the context of five multi-site Post-Authorization Safety Studies (PASS) on COVID-19 vaccines. This paper describes how we adapted BC case definitions for RWD, and trained abstractors for outcome validation using dummy case descriptions for each outcome. It does not show the results of the validation of cases that were identified in the actual studies, as these will be reported in the study reports.

Methods

Setting

VAC4EU facilitates the harmonized implementation of vaccine studies with RWD in a distributed manner across different organizations who may have different roles. Roles include lead scientific center (LSC), lead operating center and data expert and access providers (DEAPs). The validation pipeline was designed based on 5 regulatory required PASS studies on COVID-19 vaccines which were registered in the EU PAS registry (EUPAS numbers 41623, 47708, 45362, 105009, and 43556) which were conducted by different LSCs and lead operating centers. A total of 8 different DEAPs were contracted to conduct validation as part of one or more studies and each of them designated one or more clinical abstractors to validate identified cases in a retrospective manner. The validation task was coordinated by a VAC4EU validation task force composed of principal investigators of the different studies and medically trained experts. VAC4EU uses the ConcePTION common data model (CDM) for common analytics across sites and DEAPs need to transform their data to this CDM.¹⁶ DEAPs process the data in accordance with the local implementation of the General Data Protection Regulation (GDPR) and data stay local.¹⁷ Each study site was responsible for obtaining all required ethics committee or Institutional Review Board (IRB) approvals for the validation activity, which was monitored by the LOC.

The VAC4EU validation task force developed a stepwise process to allow for implementation of harmonized validation of the identified potential cases during 1 April 2023 and 30 October 2024.

The step-by-step workflow of this pipeline is illustrated in Figure 1 and described below.

Figure 1 VAC4EU Case Validation Workflow. Abbreviations: BC, Brighton Collaboration; RWD, real-world data; REDCap, Research Electronic Data Capture; eDCF, electronic data collection form; VAC4EU, Vaccine Monitoring Collaboration for Europe.

Definitions of Outcome Variables

The first step in the validation process is to identify case definitions for outcomes that require validation. VAC4EU adopted the principle of using BC case definitions, because of their regulatory adoption. If BC definitions were not available, the VAC4EU validation task force created definitions based on a literature review. A total of 16 outcomes required validation in one or more of the PASS, for 13 of these safety outcomes, a BC definition was available. Outcomes included GBS, narcolepsy, idiopathic thrombocytopenic purpura (ITP), thrombosis with thrombocytopenia syndrome (TTS), thrombocytopenia with bleeding, myocarditis, pericarditis, transverse myelitis, anaphylaxis, major congenital anomalies, encephalopathy (including acute disseminated encephalomyelitis, ADEM), deep vein thrombosis (DVT), pulmonary embolism (PE), hemorrhagic stroke, non-hemorrhagic stroke, and cerebral venous sinus thrombosis (CVST).

Review and Adaptation of Data Extraction Forms

The VAC4EU validation task force conducted a comprehensive review of publicly available data extraction forms from the BC case definition and adapted them, where necessary, to better align with data likely to be available in an RWD setting (supplemental material 1). A key challenge addressed by the task force during this phase was to clarify how abstractors should handle the absence of information. When clinical records are accessed retrospectively, absence of information can mean either absence of the condition or that the presence of the condition has not been recorded. In general, in RWD, there is information on which procedures, such as diagnostic tests, imaging, or treatments, were done and which diagnoses were made, but limited information on which conditions were actively excluded, or which procedures were not performed. To reduce heterogeneity in interpretation by different abstractors, the VAC4EU validation task force aimed to make the data extraction questionnaires as unambiguous as possible. Simplified question formats were used to minimize complexity, and specific guidance documents were provided to assist abstractors.

REDCap Data Collection Tool

VAC4EU uses the REDCap system for data collection to deploy a common data collection structure while keeping data local.^18,19 The platform was centrally configured by Vall d’Hebron Hospital, which served as the VAC4EU REDCap managing center during the development and testing phases. Dedicated REDCap projects with outcome-specific electronic data collection forms (eDCFs) were developed and subsequently cloned for local implementation at participating sites. A quality review of the eDCF was conducted by the VAC4EU validation task force using the dummy cases.

Dummy Cases

The VAC4EU validation task force created a set of five dummy case narratives for each outcome to train the abstractors and collect feedback on the eDCFs. These cases were adapted from published case reports and modified to ensure blinding to vaccination status. Each case was designed to represent a different LOC, allowing full coverage of all LOC categories during training.

Training of Abstractors

The VAC4EU validation coordinator conducted training sessions to train validation abstractors from each DEAP on how to use the data collection forms and guidelines, discuss complex items, and resolve questions through examples. Each abstractor was required to extract the dummy cases and enter the information in the eDCF. The VAC4EU validation coordinator (AD) reviewed the results and discrepancies in the assigned LOCs were discussed.

Individual training certificates were issued after successfully completing the training. A follow-up meeting was held to discuss any doubts or challenges encountered during the dummy cases validation, particularly for cases where discrepancies were observed between the abstractors’ results and the expected outcomes defined by the VAC4EU validation task force.

Finalization of eDCF

Based on insights gained during the training, the VAC4EU validation task force reviewed the abstractors’ feedback. Any fields in the eDCF or logic that caused confusion or produced varied responses were discussed, and necessary adjustments were made to improve clarity and accuracy.

After incorporating these refinements, the VAC4EU validation task force released the final eDCF for use in the PASS case validation.

Support and Quality Control During Study Case Validation

During the study-specific case validation, the VAC4EU validation task force provides ongoing support to abstractors, addressing their questions and uncertainties if they arise. To maintain consistency and ensure alignment across DEAPs, the task force has periodic meetings to discuss common challenges and review emerging issues.

Analysis

Based on the analysis of the abstraction of the dummy cases by new abstractors we assessed the agreement between abstractors (inter-rater reliability) across all outcomes using Fleiss’ Kappa,²⁰ suitable in settings with more than two abstractors per outcome. For each outcome, a total of 4 to 22 abstractors were involved across all participating DEAPs combined.

The observed agreement (P_o) was calculated for each case as the proportion of abstractors who selected the correct LOC classification. The overall observed agreement for each dataset was then computed as the average of the observed agreements across all cases.

The expected agreement (P_e) was calculated across all abstractors and cases as follows:

The Fleiss’ Kappa statistic was then computed using the following formula:

A higher Fleiss’ Kappa indicated stronger agreement between abstractors than expected by chance. Kappa values were categorized as ≤ 0 (no agreement), 0.01–0.20 (slight agreement), 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost high agreement.²¹

To estimate the precision of the Fleiss’ Kappa statistic, the standard error (SE) was calculated using the formula:

Where P_o is the observed agreement and N is the number of cases in the dataset. The 95% confidence interval (CI) for Fleiss’ Kappa was then calculated by multiplying the standard error by the Z-score (Z) corresponding to a 95% confidence level (1.96) and adding this value from the Kappa estimate:

All calculations were performed using R (version 4.3.0) Fleiss’ κ was calculated with the “irr” package (version 0.84.1) in R, using an unweighted approach (treating LOC categories as nominal).^21,22

Results

Identification of Case Definitions

For GBS, narcolepsy, ITP, TTS, myocarditis, pericarditis, transverse myelitis, anaphylaxis, major congenital anomalies, encephalopathy (including ADEM), DVT, PE, and CVST, a BC definition was available. For the remaining 3 outcomes (hemorrhagic stroke, non-hemorrhagic stroke, and thrombocytopenia with bleeding), data collection forms were developed based on literature review.

Review of eDCF: Table 1 summarizes how BC definitions were adapted for use in the VAC4EU eDCFs. For each outcome, the total number of criteria implemented reflects both original (unmodified) BC criteria and those either newly introduced or adapted to suit RWD settings. The “Adapted BC Criteria” column indicates how many original BC items required changes for clarity, relevance, or feasibility in RWD abstraction, while the “New Criteria” column captures additional items created by the task force. One such example is the criterion “Outcome reported by a specialist, but without accompanying clinical details”, which the task force added to some case definitions (eg, Myocarditis/pericarditis, GBS and TTS) to capture situations where a diagnosis is documented by a specialist in the record but without the expected clinical documentation to fully assess it. Most importantly, the task force introduced a distinction in level 4 between diagnoses made by a specialist without further details and cases where there was insufficient information to meet the case definition. This was achieved by adding a subgroup to level 4, labeling it as 4a, and classifying cases with insufficient information to meet the case definition as 4b.

Table 1 Adaptations to Case Definitions, Introduction of New Criteria by the VAC4EU Validation Task Force

Dummy Cases

For the 16 different outcomes, 78 dummy case narratives were created. Case narratives and their LOC are available in the supplemental materials and are also publicly available on Zenodo.³³

Training of Abstractors

During the study period 33 abstractors were trained across eight DEAPs. Most abstractors were trained only on a subset of outcomes depending on study contracts and requirements. Consequently, the number of abstractors that were trained per outcome ranged between 4 to 22 (Table 2). Altogether, 747 individual case abstractions were entered in REDCAP by the abstractors on the dummy cases in the training set. Discrepancies between the abstractors’ results and the expected LOC defined by the VAC4EU task force were observed for 93 dummy cases (12.40%). The overall inter-rater agreement was moderate (κ = 0.55). This κ represents concordance across the entire pooled sample of dummy cases and abstractors. Table 2 presents the per-outcome Fleiss’ κ values with their 95% CIs. The highest agreement among abstractors (Fleiss’ Kappa = 1.00) was observed for transverse myelitis, thrombocytopenia with bleeding, pulmonary embolism, and non-hemorrhagic stroke. In contrast, the highest number of discrepancies was found for the TTS dummy cases, with 53.75% discrepancies and a Kappa of −0.05 (95% CI: −0.48 to 0.38), indicating no agreement. Also the GBS dummy case abstraction showed 21.50% discrepancies, with a Kappa of 0.36 (95% CI: 0.00 to 0.72), suggesting only moderate agreement amongst abstractors. The remaining cases, such as narcolepsy, myocarditis, pericarditis, ITP, encephalitis including ADEM, hemorrhagic stroke, CVST, anaphylaxis, and major congenital anomalies, showed moderate to fair agreement.

Table 2 The Distribution of Abstractors Across Different Outcomes with the Distribution of Discrepancies Across Abstracted Dummy Cases in the Training

Figure 2 illustrates the distribution of type of discrepancies across five major criteria categories: clinical symptoms, laboratory data, imaging evidence, other criteria, and alternative diagnosis. The most substantial number of discrepancies was observed in the laboratory data category, predominantly driven by TTS cases, where laboratory-related errors accounted for 17.60% of total discrepancies. These discrepancies were often due to inconsistent interpretation of laboratory values, for example, variation in D-dimer reporting units such as fibrinogen equivalent units (FEU) or milligrams per milliliter.

Figure 2 Distribution of validation discrepancies across criteria for different outcomes. Abbreviations: TTS, thrombosis with thrombocytopenia syndrome; GBS, Guillain-Barré syndrome; CVST, cerebral venous sinus thrombosis; ITP, idiopathic thrombocytopenic purpura; DVT, deep vein thrombosis; ADEM, acute disseminated encephalomyelitis.

Errors in the alternative diagnosis and other criteria categories were more prominent in myocarditis, pericarditis, and GBS cases. Notably, myocarditis and pericarditis each had over five discrepancies attributed to alternative diagnoses. The GBS cases also had a high contribution from alternative diagnosis errors, totaling 12.00% of discrepancies for that event. Discrepancies related to clinical symptoms were generally lower, but showed some variation, especially in TTS, Narcolepsy, and Anaphylaxis. For instance, Anaphylaxis showed 8.00% discrepancies due to errors in identifying clinical features. Additionally, Major Congenital Anomalies had 8.30% discrepancies, primarily attributed to misclassification under the “other criteria” category, such as incorrect assignment of the event date or misclassification of the anomaly type, including internal, external, or functional anomalies.

Increasing Harmonization Based on Training Results

For cases with low or no concordance, such as in TTS, additional training meetings were scheduled to address the issues identified during the dummy case abstraction. These sessions included a review of misclassified cases, clarification of complex criteria, for instance, interpretation of D-dimer values or alternative diagnoses, and discussion of recurring misunderstandings. Minor discrepancies, such as overlooking one of several required clinical symptoms or entering an incorrect event date due to a typographical error, were communicated via Email with specific guidance to reduce errors. In total, three abstractors were not certified due to repeated errors and insufficient medical knowledge.

Real Case Validation Outcomes

Following dummy-based training and reliability assessment, abstractors were certified for validation of real identified cases across five PASS. At this moment a total of 153 cases of GBS, 83 narcolepsy, 297 ITP, 243 TTS, 1570 myocarditis, 2250 pericarditis, 41 transverse myelitis, 16 cases of acute disseminated encephalomyelitis including ADEM, 50 DVT and PE, 39 hemorrhagic stroke, 50 non-hemorrhagic stroke, 12 CVST, 115 thrombocytopenia with bleeding, 66 anaphylaxis, and 237 cases of major congenital anomalies have been entered in the REDCAP systems.

Discussion

This study described how a validation process of outcomes was implemented in the VAC4EU network, with the aim to reduce heterogeneity across sites and studies. A key finding is that harmonization through guidance documents, and substantive training is needed because initial inter-rater agreement (overall κ ~0.55) based on a standard set of dummy cases was moderate. It is important to note that this inter-rater agreement reflects reliability, the consistency among abstractors given the same information, and does not by itself indicate diagnostic validity. While BC case definitions and companion guides offer a required tool for harmonized vaccine safety assessment this alone does not eliminate heterogeneity when applied in RWD. BC definitions are very clinical and rely on detailed physical examination findings, specific laboratory parameters, and imaging results, which are often unavailable, inconsistently recorded, or entirely absent in some RWD sources. This mismatch affects abstractors’ ability to apply case definitions uniformly, especially when evaluating critical diagnostic elements such as laboratory values (eg, variation in D-dimer units) or imaging findings. Our results also provide insight into which outcomes were more challenging for abstractors and why. TTS had the highest level of disagreement, with a low Kappa value, indicating no agreement between abstractors. Overall, the distribution of discrepancies suggests that alternative diagnosis and other criteria introduce the greatest challenges for abstractors.

Examination of the errors showed that application of the TTS definition requires complex judgments about laboratory data, imaging evidence, and especially about ruling out alternative causes. Abstractors varied in how they interpreted these criteria in the dummy cases. To ensure more concordance in validation of the cases in the PASS we organized additional focused training for TTS, clarifying definitions of key laboratory thresholds and what constituted an alternate explanatory diagnosis. Also, GBS had only moderate agreement between abstractors, partly because abstractors differed on interpreting certain neurological exam details in the narratives. To reduce that heterogeneity, we provided supplemental guidance on those points that could be used in the PASS. In contrast, other outcomes (transverse myelitis, thrombocytopenia with bleeding, PE, non-hemorrhagic stroke) achieved perfect agreement (κ = 1.0) in dummy cases.

The training on dummy cases and review of errors was very instructive and showed that certain types of errors were common. In particular, interpretation of laboratory results (eg, how to handle a slightly ambiguous D-dimer value for TTS) or determining whether a potential alternative diagnosis explained the case’s symptoms (especially in outcomes like TTS and GBS), created differences. These patterns highlight where additional guidance was needed. The moderate to poor agreement observed for some outcomes reflect the challenges in achieving harmonized results in situations where clinical judgment and alternative diagnoses play a significant role. These insights provide important recommendations also for other large distributed research networks that implement case validation; for example, the Vaccine Safety Datalink (VSD) also highlights the need for thorough clinical review to ensure the accuracy of diagnoses, especially in cases where automated data may not fully capture complex clinical histories and uses BC definitions.³⁴ The Global Vaccine Data Network also validates cases, using BC definitions, and deployed training on dummy cases, based on VAC4EU recommendations.³⁵ Based on this experience VAC4EU provides structured feedback to the Brighton Collaboration as part of a living lab.

Strengths and Limitations

A major strength of this study is the application of a harmonized and scalable validation framework across a large, multinational network using RWD. The use of standardized training materials, structured data collection tools, and systematic adjudication processes will increase consistency and comparability across diverse settings.

However, several limitations should be noted. The dummy cases used for training were narrative-based and may not fully reflect the structure of actual data extracts encountered during study implementation, this means that the heterogeneity between abstractors my actually be greater. Furthermore, differences in data availability, language, and healthcare practices across DEAPs may require further local feedback.

We designed the validation process to be broadly applicable across different data sources and settings. Training materials and forms were developed in English to support broad accessibility across language settings, but minor interpretation differences may have arisen for non-native English abstractors. Differences in data availability may further create heterogeneity. For instance, not all DEAPs collect the same laboratory details (some may not record a troponin level needed for a myocarditis definition; others might not have imaging reports readily accessible). This variation highlights the importance of clear data abstraction tools for RWD.

The validation process required considerable effort into creation REDCap forms and verification of the logic. Making such forms and logic available in a digitized format, whether through the Brighton Collaboration or other networks, could reduce redundancy across networks, and improve consistency in applying the definitions. Moreover, incorporating insights from field implementation, including training feedback and validation challenges, would enhance the practical applicability of the BC tools in RWD settings.

The reported process and training highlight several opportunities for further improvement. First, it will be important to capture the learning curve among abstractors, for example, did agreement rates improve after the initial round of dummy case training, or with additional practice and feedback? Quantifying any improvement over successive training iterations could provide insight into how much training is “enough” and which outcomes needed extra reinforcement (we suspect outcomes like TTS would show significant improvement from round 1 to round 2 of training). Second, we can use alternative agreement metrics such as Gwet’s AC1 in addition to Fleiss’ κ. Gwet’s AC1 which is more stable in situations with very high or very low prevalence of a category. Third, this study does not capture information on the heterogeneity of actual validation in PASS, further validation of this within the boundaries of the GDPR should be explored.

Conclusion

In conclusion, the VAC4EU validation pipeline offers a scalable and adaptable implementation model for harmonized case validation in distributed RWD studies. VAC4EU has set a standard for implementation of validation according to Brighton Collaboration criteria in retrospective use of RWD.

By adapting BC case definitions to the realities and constraints of RWD, and by implementing comprehensive abstractor training with ongoing feedback, this approach has the potential to improve the consistency and reliability of outcome adjudication across multiple sites and countries. Overall, this validation process supports more harmonized robust post-marketing vaccine safety monitoring and may serve as a replicable model for future multinational collaborative studies leveraging heterogeneous RWD sources.