Recalibrating SARS-CoV-2 Antigen Rapid Lateral Flow Test Relative Sensitivity from Validation Studies to Absolute Sensitivity for Indicating Individuals Shedding Transmissible Virus
Received 20 March 2021
Accepted for publication 20 May 2021
Published 14 October 2021 Volume 2021:13 Pages 935—940
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 3
Editor who approved publication: Dr Eyal Cohen
Irene Petersen, 1 Alexander Crozier, 2 Iain Buchan, 3 Michael J Mina, 4, 5 Jonathan W Bartlett 6
1Department of Primary Care & Population Health, University College London, London, UK; 2Division of Biosciences, University College London, London, UK; 3Institute of Population Health, University of Liverpool, Liverpool, UK; 4Department of Epidemiology, Department of Immunology and Infectious Diseases, Harvard T.H. Chan School of Public Health, Boston, MA, USA; 5Department of Pathology, Clinical Microbiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA; 6Department of Mathematical Sciences, University of Bath, Bath, UK
Correspondence: Irene Petersen Email [email protected]
ABSTRACT: Testing for SARS-CoV-2 internationally has focused on COVID-19 diagnosis among symptomatic individuals using reverse transcriptase polymerase chain reaction (PCR) tests. Recently, however, SARS-CoV-2 antigen rapid lateral flow tests (LFT) have been rolled out in several countries for testing asymptomatic individuals in public health programmes. Validation studies for LFT have been largely cross-sectional, reporting sensitivity, specificity and predictive values of LFT relative to PCR. However, because PCR detects genetic material left behind for a long period when the individual is no longer infectious, these statistics can under-represent the sensitivity of LFT for detecting infectious individuals, especially when sampling asymptomatic populations. LFTs (intended to detect individuals shedding SARS-CoV-2 antigens) validated against PCR (intended to diagnose infection) are not reporting against a gold standard of equivalent measurements. Instead, these validation studies have reported relative performance statistics that need recalibrating to the purpose for which LFT is being used. We present an approach to this recalibration. We derive a formula for recalibrating relative performance statistics from LFT vs PCR validation studies to give likely absolute sensitivity of LFT for detecting individuals who are shedding shedding SARS-CoV-2 antigens. We contrast widely reported apparent sensitivities of LFT with recalibrated absolute sensitivity for detecting individuals shedding SARS-CoV-2 antigens. After accounting for within-individual viral kinetics and epidemic dynamics within asymptomatic populations we show that a highly performant test for SARS-CoV-2 antigen should show LFT-to-PCR relative sensitivity of less than 50% in conventional validation studies, which after re-calibration would be an absolute sensitivity of more than 80%. Further studies are needed to ascertain the absolute sensitivity of LFT as a test of infectiousness in COVID-19 responses. These studies should include longitudinal series of LFT and PCR, ideally in cohorts sampled from both contacts of cases and the general population.
Keywords: rapid test, PCR, validation, recalibration, lateral flow tests
Testing for SARS-CoV-2 in most countries has, until recently, focused on COVID-19 diagnosis among symptomatic individuals using quantitative reverse transcriptase polymerase chain reaction (PCR) tests. Recently, however, SARS-CoV-2 antigen rapid lateral flow tests (LFT) have been rolled out in several countries for testing asymptomatic individuals in public health programmes and for providing a more rapid, low-cost alternative to PCR in specific contexts.1
In contrast to PCR, LFT are primarily being used to identify likely infectious individuals by detecting SARS-CoV-2 antigen from people who are shedding virus but who may not have classical COVID-19 symptoms, or at least do not use symptomatic testing centres. It is well recognised that pre-symptomatic transmission is a key driver of spread, with many individuals being infectious without displaying classic symptoms - around a third of transmission may be accounted for by this group.2,3 Validation of LFT for asymptomatic testing has employed several cross-sectional studies and reported conventional diagnostic test accuracy statistics. However, given PCR is testing for any sign of SARS-CoV-2 nucleic acid (fragments), which most of the time will be material left over after the individual has stopped being infectious, and that LFTs test for SARS-CoV-2 antigen which is most abundant when people are shedding viable virus and are most infectious, these two tests are not comparable.4 Therefore, LFT validated against PCR is not reporting diagnostic accuracy statistics against a gold standard of equivalent measurements. Instead, these validation studies have reported relative performance statistics that need recalibrating to the purpose for which LFT is being used.
In most of the validation studies, individuals were tested simultaneously with LFT and PCR, with PCR used as a gold standard, i.e., a marker of SARS-CoV-2 infectiousness, when it is actually a marker of having been infected at some point within a certain window of time. Sensitivity was thus evaluated as the ability of LFT to identify the same positive cases as the PCR and the specificity as the ability to identify the same negative cases as the PCR.5 In these studies, it often appears that the LFTs have a low sensitivity, but high specificity. For example, in a pilot study of asymptomatic testing in Liverpool the sensitivity and specificity of the LFT was estimated to be 40.0% (28.5%–52.4%; 28/70) and 99.9% (99.8%–99.99%; 5431/5434), respectively.6 In a Danish study, the estimates were 69.7% (63.2%–75.7%; 154/221) and 99.5% (99.2%–99.7%; 4567/4590), respectively.7 In a Spanish study of close contacts the estimate of sensitivity among asymptomatic individuals was 60% (95% CI, 40.7%–76.6%) and 80.2% (95% CI, 70.9%–87.1%) among symptomatic individuals.8 An evaluation of a rapid antigen test done by the Centers for Disease Control and Prevention (CDC) found the sensitivity was 41% (95% CI, 18.4%–67.1%) and 80% (95% CI, 64.4%–90.9%) among asymptomatic and symptomatic individuals, respectively9 and another evaluation by CDC found the sensitivity, relative to PCR, was 36% and 64% amongst asymptomatic and symptomatic populations, respectively. However, when measured against specimens with a positive viral culture, sensitivity rose to 78.6% (95% CI, 59.1%–91.7%) and to 92.6% (95% CI, 83.7%–97.6%) in asymptomatic and symptomatic populations, respectively.10
These diverging figures have provoked debate about the sensitivity of the LFTs and concerns have been raised about their utility in the context of testing asymptomatic individuals.11–13
To assist policymakers, we have further investigated the reported sensitivities of LFTs as tests of presence of SARS-CoV-2 antigen, and we derive a formula for recalibrating these LFT-to-PCR relative performance statistics into absolute sensitivity of LFT for detecting individuals who are shedding SARS-CoV-2 antigen, indicative of individuals who are shedding transmissible virus.
The Difference Between PCR and LFTs
Before evaluating the results of the validation studies, it is important to understand the biology of SARS-CoV-2 and recognise that the two tests reflect different properties of the infection and have different testing utilities – clinical versus public health. The tests involve taking a swab from the throat and nose. These can be done in test centres by trained staff or by the individuals themselves at home, or while supervised at testing centres.
The PCR test is an established laboratory technique which can be used to identify the presence of SARS-CoV-2 by reverse transcription of RNA and then repeated thermal cyclic amplification of target viral nucleic acid present in a sample. At each cycle, the nucleic acid doubles and so the test is exquisitely sensitive, potentially able to detect a single RNA fragment in a sample. The more cycles it takes to detect an infection the less RNA there is in a sample. Cycle threshold (Ct) is the point in the early exponential phase of the process at which fluorescence from amplified genetic material becomes detectable. Beyond a mathematically defined limit, the genetic material is considered undetected. 14 So, lower Ct values reflect higher levels of virus RNA. Due to the amplification step, PCR can detect RNA in quantities much lower than the limits of detection by virus culture, which is typically recognised as the proxy for infectiousness.15 As such, while PCR has high analytical sensitivity, it is poorly specific for infectiousness, the fundamental characteristic of a communicable disease and a critical consideration in the control of SARS-CoV-2 transmission. It is important to note that PCR is still the gold standard for diagnostic testing – if the test is conducted shortly after an individual develops symptoms, it is highly likely a positive PCR result equates to an infectious case as most people who develop symptoms are infectious just before and for a median 5 days after symptom onset.16 A large systematic review on the duration of viral shedding and infectiousness found the mean duration of SARS-CoV-2 RNA shedding in upper respiratory tract is 17 days (95% CI 15.5–18.6).17 However, there is large variation, and many studies did not take into account that shedding may have started several days before individuals received their first positive test result. Therefore, in many people nucleic acid fragments may be detectable for 3 weeks or more. Most people infected with SARS-CoV-2 are infectious for 4–8 days and no studies have cultured viable virus beyond day 9 of illness.17 Because of the prolonged presence of residual RNA and relatively short infectious time window, we will expect only to find 50% or less in an asymptomatic sample to be within the infectious time window when they test positive on PCR. This discrepancy is also recognised by Public Health England guidance recommending individuals not to be tested with PCR tests within 90 days of exiting isolation because they may remain positive for a long time.18
LFTs are an established technology, for example used in pregnancy test kits, that has been repurposed for detecting SARS-CoV-2 surface proteins (antigens). The tests do not require laboratory processing and results are provided within 10–30 minutes.19 LTFs have a lower analytical sensitivity than RT-PCR (and thus are unlikely to provide positive results long after the infectious period), and a high specificity.1,20
When performing cross-sectional testing in a population, the ratio of currently infectious to post-infectious cases is expected to change with the phase of the epidemic curve.21 This ratio is higher when the epidemic is surging and lower when it is shrinking or in steady state because the time course of infectiousness within infected individuals is asymmetrical (front-loaded). Where there is sustained community transmission there will be a significant proportion of individuals who are beyond the infectious period, but still shedding RNA which will be detected by PCR, but unlikely to be detected by the LFTs.21
Calibrated Sensitivity of LFTs
Based on the information about the biology of SARS-CoV-2, the dynamic of the epidemic and data on the performance of the PCR and LFTs, we illustrate how the reported sensitivity estimates in cross-sectional validation studies likely underestimate the sensitivity of the LFTs in terms of detecting individuals who are shedding transmissible SARS-CoV-2 at the time of the test.
We derive a formula for the apparent relative sensitivity1 of the LFT from a cross-sectional validation study when using the PCR as the reference test, which we then re-arrange for calculating a calibrated absolute sensitivity2 of the LFT for detecting individuals who are shedding SARS-CoV-2 antigens.
Let D denote an individual’s infection status, with D = 0 for never infected or previously infected but with no detectable viral material, D = 1 for infected and shedding SARS-CoV-2 antigens and D = 2 for post-infectious (still with detectable RNA virus, but no longer shedding SARS-CoV-2 antigens). Let TPCR and TLFT denote the PCR and LFT outcomes for an individual, with + and – for positive and negative results. We assume the two test results depend only on the individual’s true infection status D. To simplify this further we assume the PCR test never tests positive in those who have never been infected (D = 0) and PCR always tests positive in those with D = 1 or D = 2. We additionally assume that LFT never tests positive in post-infectious (D = 2) individuals. Under these assumptions we can derive the following formula for the apparent relative sensitivity of the LFT from a cross-sectional validation study when using the PCR as the reference test:
where denotes the proportion of individuals who are shedding SARS-CoV-2 antigens among those who harbour viral RNA (detectable by PCR) (D = 1 or D = 2). This shows the sensitivity relative to PCR will appear markedly lower than its sensitivity for detecting those currently shedding SARS-CoV-2 antigens (D = 1). Re-arranging the formula shows the calibrated absolute sensitivity of LFTs for detecting those individuals shedding SARS-CoV-2 antigens is:
We note that PCR is likely to identify potential infectious individuals slightly earlier than LFTs, but due to the exponential growth of the infection this time window is quite short (perhaps less than 24 hours) and therefore is not included in our calibration. We recommend further longitudinal series of LFT and PCR to determine the time difference between turning positive on PCR and LFT, and how this relates to symptom onset and transmission.
Likewise, we acknowledge that the sensitivity of the tests may be affected by sampling error and experience of the person performing the sampling and the test. These uncertainties are not taken into account in our calibrations, but they are discussed in the reports from Liverpool and by Peto and colleagues.20,22
Calibrated Absolute Sensitivity for Detecting Individuals with SARS-CoV-2 antigens in Swab Samples
Table 1 illustrates the relationship between 1) the proportion of individuals who are shedding SARS-CoV-2 antigens among those who harbour viral RNA (detectable by PCR), 2) the apparent sensitivity and 3) calibrated sensitivity of LFTs.
From Table 1 it can be seen that the upper limit (100%) of the calibrated absolute sensitivity is reached when and apparent relative sensitivity are equivalent. For example, in a sample where under half of the individuals are still shedding SARS-CoV-2 antigens a validation study with PCR test as the reference test can never reach an apparent sensitivity of the LFTs of more than 50%. On the other hand, in a study which includes symptomatic individuals the proportion of individuals who are shedding SARS-CoV-2 antigens among those who harbour viral RNA in the sample is likely to be much higher and thus we would expect the apparent sensitivity to be higher. This is what was observed in the validation studies in Denmark, Spain and the US.7–9 As mentioned above the proportion of individuals who are shedding SARS-CoV-2 antigens in a sample also varies over time and across locations. Thus, we may find the apparent validity of the same type of LFTs also varies substantially between studies carried out in different locations and at different times. Hence, a large variation of the apparent sensitivity has been observed in empirical studies carried out so far.5
One outstanding question remains; what is the actual sensitivity of the LFTs in terms of identifying individuals who are shedding SARS-CoV-2 antigens? With knowledge of the biology of the virus and information about the local developments of the pandemic we can calculate calibrated sensitivities of the test. For example, using the data from the Liverpool validation study22 we would expect to be smaller than 0.5 based on our knowledge of the virus. It may have been even smaller at the time of the study as the epidemic was shrinking in Liverpool by December 2020. This suggests that the apparent relative sensitivity of 40% found in Liverpool may result in a calibrated absolute sensitivity above 80%. As we will never know the exact value of in these validation studies we cannot provide an exact value for the calibrated absolute sensitivity.
Further Validation of LFTs
As LFTs are becoming more widely used in schools, workplaces and for admittance to venues such as those used for large events, it is important that health professionals and the public have clear information about the operating characteristics of the tests. We have demonstrated that the calibrated absolute sensitivity to detect SARS-CoV-2 antigens is likely high with LFT. To improve our understanding of their characteristics, longitudinal studies where individuals, ideally contacts of cases, are tested daily by LFT and PCR would help to further understand false-negatives (and false-positives) and, importantly, time differences between turning PCR positive, LFT positive, and symptom onset. Further validation of the LFTs could be performed against viable viral cultures, as it was done by Prince-Guerra10 and Pickering et al.,23 or, ideally, against a gold-standard antigen test with proven accuracy for detection of not only SARS-CoV-2 antigens, but also infectious individuals. Some validation studies have sought to use specific Ct cut-offs when comparing the performance of PCR and LFTs,20 but differently calibrated PCR systems mean that Ct levels cannot easily be compared between studies, and the values do not always indicate the same level of virus in a sample between laboratories.4,13
Criticisms of LFT for apparent low sensitivity have failed to take the viral biology and epidemiology into account and we believe have reached the wrong conclusions.11,12,24 This has confused policymaking and damaged public trust in LFTs, despite the need for better tools to control transmission of SARS-CoV-2.4 It is our hope that recalibrated absolute sensitivity statistics will assist policymaking and help to build public confidence in LFT as a tool to aid COVID-19 resilience and recovery.
In this study we investigated validation studies of LFT and showed the pitfalls of reporting sensitivity values relative to PCR as if they were absolute values measured against a gold-standard test. In most samples of asymptomatic individuals, we would expect less than half of PCR positive individuals to be shedding SARS-CoV-2 antigens. A well-performing test for detecting those shedding SARS-CoV-2 antigens would therefore have an apparent relative (to PCR) sensitivity never exceeding 50%. Recalibrating an apparent relative sensitivity of 50%, on average we would expect an approximate absolute sensitivity of over 80% in testing for individuals shedding SARS-CoV-2 antigens. Future studies might improve this calibration further using series of daily repeated PCR and LFT among substantial cohorts drawn from the general population, and viral cultures from the samples.
where in the first line we use our assumption that PCR never tests positive in D = 0 individuals and always tests positive in D = 1 or D = 2 individuals, such that testing positive by PCR is equivalent to the event D = 1 or D = 2 and we use the fact that D = 1 and D = 2 are mutually exclusive.
Professor Iain Buchan reports grants as the senior investigator from NIHR, during the conduct of the study; receives personal fees as the Chief Data Scientist Advisor from AstraZeneca, outside the submitted work. In addition, Professor Buchan also reports that Innova Medical Group gave a gift to the Pandemic Institute in Liverpool, which is chaired by Liverpool City Council and involves the University of Liverpool. It has not funded any of his work to date, which was supported only by Department of Health & Social Care and NIHR for studies into lateral flow SARS-CoV-2 rapid antigen testing. Dr Michael J Mina reports personal fees from Detect, during the conduct of the study. Dr Jonathan W Bartlett reports personal fees from Bayer for consultancy on statistical methods for clinical trials and consultancy to University of Bath on statistical methods for clinical trials for Bayer, AstraZeneca, Novartis, and Roche, outside the submitted work. The authors report no other conflicts of interest in this work.
1. Crozier A, Rajan S, Buchan I, McKee M. Put to the test: use of rapid testing technologies for covid-19. BMJ. 2021;3(372):n208. doi:10.1136/bmj.n208
2. Petersen I, Phillips A. Three Quarters of People with SARS-CoV-2 infection are asymptomatic: analysis of english household survey data. Clin Epidemiol. 2020;12:1039–1043.
3. Buitrago-Garcia D, Egli-Gany D, Counotte MJ, et al. Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: a living systematic review and meta-analysis. Ford N, editor. PLoS Med. 2020;17(9):e1003346.
4. Mina MJ, Peto TE, García-Fiñana M, Semple MG, Buchan IE. Clarifying the evidence on SARS-CoV-2 antigen rapid tests in public health responses to COVID-19. The Lancet. 2021;S0140673621004256.
5. Dinnes J, Deeks JJ, Berhane S, et al. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Infectious Diseases Group, editor. Cochrane Database Syst Rev. 2021.
6. Council LC. Symptom-Free Testing [Internet]. Liverpool City Council; 2021. Available from: https://www.liverpool.ac.uk/media/livacuk/coronavirus/Liverpool,Community,Testing,Pilot,Interim,Evaluation.pdf.
7. Jakobsen KK, Jensen JS, Todsen T, et al. Detection of SARS-CoV-2 infection by rapid antigen test in comparison with RT-PCR in a public setting. medRxiv. 2021;2021.
8. Torres I, Poujois S, Albert E, Álvarez G, Colomina J, Navarro D. Point-of-care evaluation of a rapid antigen test (CLINITEST® Rapid COVID-19 Antigen Test) for diagnosis of SARS-CoV-2 infection in symptomatic and asymptomatic individuals. medRxiv. 2021.
9. Pray IW. Performance of an antigen-based test for asymptomatic and symptomatic SARS-CoV-2 testing at two university campuses — wisconsin, September–October 2020. MMWR Morb Mortal Wkly Rep. 2021;69.
10. Prince-Guerra JL. Evaluation of Abbott BinaxNOW rapid antigen test for SARS-CoV-2 infection at two community-based testing sites — pima County, Arizona, November 3–17, 2020. MMWR Morb Mortal Wkly Rep. 2021;70.
11. Covid-19: government must urgently rethink lateral flow test roll out [Internet]. THE BMJ. 2021.
12. Deeks JJ, Raffle AE. Lateral flow tests cannot rule out SARS-CoV-2 infection. BMJ. 2020;11:m4787.
13. Guglielmi G. Rapid coronavirus tests: a guide for the perplexed. Nature. 2021;590(7845):202–205.
14. Understanding cycle threshold (Ct) in SARS-CoV-2 RT-PCR.: 12.
15. Singanayagam A, Patel M, Charlett A, et al. Duration of infectiousness and correlation with RT-PCR cycle threshold values in cases of COVID-19, England, January to May 2020. Eurosurveillance. 2020;25(32):2001483.
16. Cevik M, Kuppalli K, Kindrachuk J, Peiris M. Virology, transmission, and pathogenesis of SARS-CoV-2. BMJ. 2020;m3862.
17. Cevik M, Tate M, Lloyd O, Maraolo AE, Schafers J, Ho A. SARS-CoV-2, SARS-CoV, and MERS-CoV viral load dynamics, duration of viral shedding, and infectiousness: a systematic review and meta-analysis. Lancet Microbe. 2020;S2666524720301725.
18. COVID-19: management of staff and exposed patients or residents in health and social care settings [Internet]. GOV.UK. Available from: https://www.gov.uk/government/publications/covid-19-management-of-exposed-healthcare-workers-and-patients-in-hospital-settings/covid-19-management-of-exposed-healthcare-workers-and-patients-in-hospital-settings.
19. Bunn S Mass testing for COVID-19: january update on lateral flow tests. 2021. Available from: https://post.parliament.uk/mass-testing-for-covid-19-january-update-on-lateral-flow-tests/.
20. Peto T. Team UC-19 LFO. COVID-19: rapid Antigen detection for SARS-CoV-2 by lateral flow assay: a national systematic evaluation for mass-testing. medRxiv. 2021;2021.
21. Hay JA, Kennedy-Shaffer L, Kanjilal S, Lipsitch M, Mina MJ. Estimating epidemiologic dynamics from single cross-sectional viral load distributions. medRxiv. 2020;13:2020.
22. Council LC. Symptom-Free Testing [Internet]. Liverpool City Council. Available from: https://liverpool.gov.uk:443/communities-and-safety/emergency-planning/coronavirus/how-to-get-tested/symptom-free-testing/.
23. Pickering S, Batra R, Snell LB, et al. Comparative performance of SARS CoV-2 lateral flow antigen tests demonstrates their utility for high sensitivity detection of infectious virus in clinical specimens [Internet]. Infect Dis. 2021.
24. Kmietowicz Z. Covid-19: controversial rapid test policy divides doctors and scientists. BMJ. 2021;12(372):n81.
© 2021 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.