Recalibrating SARS-CoV-2 Antigen Rapid Lateral Flow Test Relative Sensitivity from Validation Studies to Absolute Sensitivity for Indicating Individuals Shedding Transmissible Virus

Introduction

Testing for SARS-CoV-2 in most countries has, until recently, focused on COVID-19 diagnosis among symptomatic individuals using quantitative reverse transcriptase polymerase chain reaction (PCR) tests. Recently, however, SARS-CoV-2 antigen rapid lateral flow tests (LFT) have been rolled out in several countries for testing asymptomatic individuals in public health programmes and for providing a more rapid, low-cost alternative to PCR in specific contexts.¹

In contrast to PCR, LFT are primarily being used to identify likely infectious individuals by detecting SARS-CoV-2 antigen from people who are shedding virus but who may not have classical COVID-19 symptoms, or at least do not use symptomatic testing centres. It is well recognised that pre-symptomatic transmission is a key driver of spread, with many individuals being infectious without displaying classic symptoms - around a third of transmission may be accounted for by this group.^2,3 Validation of LFT for asymptomatic testing has employed several cross-sectional studies and reported conventional diagnostic test accuracy statistics. However, given PCR is testing for any sign of SARS-CoV-2 nucleic acid (fragments), which most of the time will be material left over after the individual has stopped being infectious, and that LFTs test for SARS-CoV-2 antigen which is most abundant when people are shedding viable virus and are most infectious, these two tests are not comparable.⁴ Therefore, LFT validated against PCR is not reporting diagnostic accuracy statistics against a gold standard of equivalent measurements. Instead, these validation studies have reported relative performance statistics that need recalibrating to the purpose for which LFT is being used.

In most of the validation studies, individuals were tested simultaneously with LFT and PCR, with PCR used as a gold standard, i.e., a marker of SARS-CoV-2 infectiousness, when it is actually a marker of having been infected at some point within a certain window of time. Sensitivity was thus evaluated as the ability of LFT to identify the same positive cases as the PCR and the specificity as the ability to identify the same negative cases as the PCR.⁵ In these studies, it often appears that the LFTs have a low sensitivity, but high specificity. For example, in a pilot study of asymptomatic testing in Liverpool the sensitivity and specificity of the LFT was estimated to be 40.0% (28.5%–52.4%; 28/70) and 99.9% (99.8%–99.99%; 5431/5434), respectively.⁶ In a Danish study, the estimates were 69.7% (63.2%–75.7%; 154/221) and 99.5% (99.2%–99.7%; 4567/4590), respectively.⁷ In a Spanish study of close contacts the estimate of sensitivity among asymptomatic individuals was 60% (95% CI, 40.7%–76.6%) and 80.2% (95% CI, 70.9%–87.1%) among symptomatic individuals.⁸ An evaluation of a rapid antigen test done by the Centers for Disease Control and Prevention (CDC) found the sensitivity was 41% (95% CI, 18.4%–67.1%) and 80% (95% CI, 64.4%–90.9%) among asymptomatic and symptomatic individuals, respectively⁹ and another evaluation by CDC found the sensitivity, relative to PCR, was 36% and 64% amongst asymptomatic and symptomatic populations, respectively. However, when measured against specimens with a positive viral culture, sensitivity rose to 78.6% (95% CI, 59.1%–91.7%) and to 92.6% (95% CI, 83.7%–97.6%) in asymptomatic and symptomatic populations, respectively.¹⁰

These diverging figures have provoked debate about the sensitivity of the LFTs and concerns have been raised about their utility in the context of testing asymptomatic individuals.^11–13

To assist policymakers, we have further investigated the reported sensitivities of LFTs as tests of presence of SARS-CoV-2 antigen, and we derive a formula for recalibrating these LFT-to-PCR relative performance statistics into absolute sensitivity of LFT for detecting individuals who are shedding SARS-CoV-2 antigen, indicative of individuals who are shedding transmissible virus.

The Difference Between PCR and LFTs

Before evaluating the results of the validation studies, it is important to understand the biology of SARS-CoV-2 and recognise that the two tests reflect different properties of the infection and have different testing utilities – clinical versus public health. The tests involve taking a swab from the throat and nose. These can be done in test centres by trained staff or by the individuals themselves at home, or while supervised at testing centres.

The PCR test is an established laboratory technique which can be used to identify the presence of SARS-CoV-2 by reverse transcription of RNA and then repeated thermal cyclic amplification of target viral nucleic acid present in a sample. At each cycle, the nucleic acid doubles and so the test is exquisitely sensitive, potentially able to detect a single RNA fragment in a sample. The more cycles it takes to detect an infection the less RNA there is in a sample. Cycle threshold (Ct) is the point in the early exponential phase of the process at which fluorescence from amplified genetic material becomes detectable. Beyond a mathematically defined limit, the genetic material is considered undetected. ¹⁴ So, lower Ct values reflect higher levels of virus RNA. Due to the amplification step, PCR can detect RNA in quantities much lower than the limits of detection by virus culture, which is typically recognised as the proxy for infectiousness.¹⁵ As such, while PCR has high analytical sensitivity, it is poorly specific for infectiousness, the fundamental characteristic of a communicable disease and a critical consideration in the control of SARS-CoV-2 transmission. It is important to note that PCR is still the gold standard for diagnostic testing – if the test is conducted shortly after an individual develops symptoms, it is highly likely a positive PCR result equates to an infectious case as most people who develop symptoms are infectious just before and for a median 5 days after symptom onset.¹⁶ A large systematic review on the duration of viral shedding and infectiousness found the mean duration of SARS-CoV-2 RNA shedding in upper respiratory tract is 17 days (95% CI 15.5–18.6).¹⁷ However, there is large variation, and many studies did not take into account that shedding may have started several days before individuals received their first positive test result. Therefore, in many people nucleic acid  fragments may be detectable for 3 weeks or more. Most people infected with SARS-CoV-2 are infectious for 4–8 days and no studies have cultured viable virus beyond day 9 of illness.¹⁷ Because of the prolonged presence of residual RNA and relatively short infectious time window, we will expect only to find 50% or less in an asymptomatic sample to be within the infectious time window when they test positive on PCR. This discrepancy is also recognised by Public Health England guidance recommending individuals not to be tested with PCR tests within 90 days of exiting isolation because they may remain positive for a long time.¹⁸

LFTs are an established technology, for example used in pregnancy test kits, that has been repurposed for detecting SARS-CoV-2 surface proteins (antigens). The tests do not require laboratory processing and results are provided within 10–30 minutes.¹⁹ LTFs have a lower analytical sensitivity than RT-PCR (and thus are unlikely to provide positive results long after the infectious period), and a high specificity.^1,20

When performing cross-sectional testing in a population, the ratio of currently infectious to post-infectious cases is expected to change with the phase of the epidemic curve.²¹ This ratio is higher when the epidemic is surging and lower when it is shrinking or in steady state because the time course of infectiousness within infected individuals is asymmetrical (front-loaded). Where there is sustained community transmission there will be a significant proportion of individuals who are beyond the infectious period, but still shedding RNA which will be detected by PCR, but unlikely to be detected by the LFTs.²¹

Calibrated Sensitivity of LFTs

Based on the information about the biology of SARS-CoV-2, the dynamic of the epidemic and data on the performance of the PCR and LFTs, we illustrate how the reported sensitivity estimates in cross-sectional validation studies likely underestimate the sensitivity of the LFTs in terms of detecting individuals who are shedding transmissible SARS-CoV-2 at the time of the test.

We derive a formula for the apparent relative sensitivity¹ of the LFT from a cross-sectional validation study when using the PCR as the reference test, which we then re-arrange for calculating a calibrated absolute sensitivity² of the LFT for detecting individuals who are shedding SARS-CoV-2 antigens.

Let D denote an individual’s infection status, with D = 0 for never infected or previously infected but with no detectable viral material, D = 1 for infected and shedding SARS-CoV-2 antigens and D = 2 for post-infectious (still with detectable RNA virus, but no longer shedding SARS-CoV-2 antigens). Let T_PCR and T_LFT denote the PCR and LFT outcomes for an individual, with + and – for positive and negative results. We assume the two test results depend only on the individual’s true infection status D. To simplify this further we assume the PCR test never tests positive in those who have never been infected (D = 0) and PCR always tests positive in those with D = 1 or D = 2. We additionally assume that LFT never tests positive in post-infectious (D = 2) individuals. Under these assumptions we can derive the following formula for the apparent relative sensitivity of the LFT from a cross-sectional validation study when using the PCR as the reference test:

where denotes the proportion of individuals who are shedding SARS-CoV-2 antigens among those who harbour viral RNA (detectable by PCR) (D = 1 or D = 2). This shows the sensitivity relative to PCR will appear markedly lower than its sensitivity for detecting those currently shedding SARS-CoV-2 antigens (D = 1). Re-arranging the formula shows the calibrated absolute sensitivity of LFTs for detecting those individuals shedding SARS-CoV-2 antigens is:

Limitations

We note that PCR is likely to identify potential infectious individuals slightly earlier than LFTs, but due to the exponential growth of the infection this time window is quite short (perhaps less than 24 hours) and therefore is not included in our calibration. We recommend further longitudinal series of LFT and PCR to determine the time difference between turning positive on PCR and LFT, and how this relates to symptom onset and transmission.

Likewise, we acknowledge that the sensitivity of the tests may be affected by sampling error and experience of the person performing the sampling and the test. These uncertainties are not taken into account in our calibrations, but they are discussed in the reports from Liverpool and by Peto and colleagues.^20,22

Calibrated Absolute Sensitivity for Detecting Individuals with SARS-CoV-2 antigens in Swab Samples

Table 1 illustrates the relationship between 1) the proportion of individuals who are shedding SARS-CoV-2 antigens among those who harbour viral RNA (detectable by PCR), 2) the apparent sensitivity and 3) calibrated sensitivity of LFTs.

Table 1 Calibrated Sensitivity (%) of LFT to detect SARS-CoV-2 antigen in nose/throat swab samples by different values of and Apparent Sensitivity Estimates. - Indicates that this apparent sensitivity value is not possible for the given value of π, under our assumptions

From Table 1 it can be seen that the upper limit (100%) of the calibrated absolute sensitivity is reached when and apparent relative sensitivity are equivalent. For example, in a sample where under half of the individuals are still shedding SARS-CoV-2 antigens a validation study with PCR test as the reference test can never reach an apparent sensitivity of the LFTs of more than 50%. On the other hand, in a study which includes symptomatic individuals the proportion of individuals who are shedding SARS-CoV-2 antigens among those who harbour viral RNA in the sample is likely to be much higher and thus we would expect the apparent sensitivity to be higher. This is what was observed in the validation studies in Denmark, Spain and the US.^7–9 As mentioned above the proportion of individuals who are shedding SARS-CoV-2 antigens in a sample also varies over time and across locations. Thus, we may find the apparent validity of the same type of LFTs also varies substantially between studies carried out in different locations and at different times. Hence, a large variation of the apparent sensitivity has been observed in empirical studies carried out so far.⁵

One outstanding question remains; what is the actual sensitivity of the LFTs in terms of identifying individuals who are shedding SARS-CoV-2 antigens? With knowledge of the biology of the virus and information about the local developments of the pandemic we can calculate calibrated sensitivities of the test. For example, using the data from the Liverpool validation study²² we would expect to be smaller than 0.5 based on our knowledge of the virus. It may have been even smaller at the time of the study as the epidemic was shrinking in Liverpool by December 2020. This suggests that the apparent relative sensitivity of 40% found in Liverpool may result in a calibrated absolute sensitivity above 80%. As we will never know the exact value of in these validation studies we cannot provide an exact value for the calibrated absolute sensitivity.

Further Validation of LFTs

As LFTs are becoming more widely used in schools, workplaces and for admittance to venues such as those used for large events, it is important that health professionals and the public have clear information about the operating characteristics of the tests. We have demonstrated that the calibrated absolute sensitivity to detect SARS-CoV-2 antigens is likely high with LFT. To improve our understanding of their characteristics, longitudinal studies where individuals, ideally contacts of cases, are tested daily by LFT and PCR would help to further understand false-negatives (and false-positives) and, importantly, time differences between turning PCR positive, LFT positive, and symptom onset. Further validation of the LFTs could be performed against viable viral  cultures, as it was done by Prince-Guerra¹⁰ and Pickering et al.,²³ or, ideally, against a gold-standard antigen test with proven accuracy for detection of not only SARS-CoV-2 antigens, but also infectious individuals. Some validation studies have sought to use specific Ct cut-offs when comparing the performance of PCR and LFTs,²⁰ but differently calibrated PCR systems mean that Ct levels cannot easily be compared between studies, and the values do not always indicate the same level of virus in a sample between laboratories.^4,13

Criticisms of LFT for apparent low sensitivity have failed to take the viral biology and epidemiology into account and we believe have reached the wrong conclusions.^11,12,24 This has confused policymaking and damaged public trust in LFTs, despite the need for better tools to control transmission of SARS-CoV-2.⁴ It is our hope that recalibrated absolute sensitivity statistics will assist policymaking and help to build public confidence in LFT as a tool to aid COVID-19 resilience and recovery.

Conclusion

In this study we investigated validation studies of LFT and showed the pitfalls of reporting sensitivity values relative to PCR as if they were absolute values measured against a gold-standard test. In most samples of asymptomatic individuals, we would expect less than half of PCR positive individuals to be shedding SARS-CoV-2 antigens. A well-performing test for detecting those shedding SARS-CoV-2 antigens would therefore have an apparent relative (to PCR) sensitivity never exceeding 50%. Recalibrating an apparent relative sensitivity of 50%, on average we would expect an approximate absolute sensitivity of over 80% in testing for individuals shedding SARS-CoV-2 antigens. Future studies might improve this calibration further using series of daily repeated PCR and LFT among substantial cohorts drawn from the general population, and viral cultures from the samples.

Formula derivation

To derive the calibrated sensitivity formula, we have:

where in the first line we use our assumption that PCR never tests positive in D = 0 individuals and always tests positive in D = 1 or D = 2 individuals, such that testing positive by PCR is equivalent to the event D = 1 or D = 2 and we use the fact that D = 1 and D = 2 are mutually exclusive.