Validation of autism spectrum disorder diagnoses recorded in the Clinical Practice Research Datalink, 1990–2014

Plain language summary

This study was done to assess the quality of autism spectrum disorder (ASD) diagnoses recorded in an electronic medical database, the Clinical Practice Research Datalink (CPRD). The results of this study indicate that 91.9% of ASD diagnoses recorded in the CPRD are valid. These results suggest that ASD diagnoses in the CPRD are of high quality and can be used for research.

Introduction

Autism spectrum disorder (ASD) is a complex, broad-range of neurodevelopmental disorders characterized by deficits in social function and communication with the presence of stereotyped behavior, interests, or activities, which are typically noted in early childhood and impact daily activities.^1,2 Diagnostic criteria used to identify children with ASD have changed over time.^3,4 Diagnostic and Statistical Manual of Mental Disorders (DSM), Fourth Edition (DSM-IV) criteria for autism, first published in 1980 and revised in 1994 and 2000, describe separate disorders: autistic disorder, Asperger’s disorder, or pervasive developmental disorder (PDD) not otherwise specified. These disorders share core features that appear in early childhood, including social dysfunction, communicative difficulties, and behavioral problems (aggressive behavior, hyperactivity, increased repetitive/restricted behaviors and interests). DSM Fifth Edition (DSM-5), published in 2013, captured these separate subtypes under the umbrella diagnosis of “Autism Spectrum Disorders” to reflect a continuum of severity. Another notable change in DSM-5 was that the three core features of ASD were reduced to two: social/communicative deficits and restricted repetitive behaviors. The prevalence of ASD is currently estimated to be approximately 1–2%.^2,5,6 The etiologies of these disorders are not well understood; however, both genetic and environmental factors are likely to play an important role.

Over the last two decades, large electronic data sources have been used in epidemiologic studies seeking to identify potential risk factors for ASD. One potential concern with use of these existing data sources is uncertainty about the validity of the ASD cases identified and the availability of clinical details. The United Kingdom’s Clinical Practice Research Datalink (CPRD) is a large population-based electronic medical record database that has been used in investigations of risk factors of ASD. Previously published studies have indicated that ASD diagnoses recorded in the CPRD were of high validity;^7,8 however, the children described in these studies were born in the 1990s and medical knowledge, ASD diagnostic criteria, and screening practices have changed extensively since the last of those studies was published. In this study, we sought to provide an updated assessment of the quality of ASD diagnoses recorded in the CPRD between 1990 and 2014. We conducted a validation study to assess the quality of ASD diagnoses recorded in the CPRD and calculated the positive predictive value (PPV) of ASD diagnoses recorded in the CPRD compared to original medical records. We also described characteristics of children with ASD and changes in the recording of clinical diagnoses and treatments related to ASD in the electronic data over time.

Methods

Data Source and Source Population

We validated ASD diagnoses recorded in the UK CPRD, a large, longitudinal, population-based electronic medical record database.^9,10 The UK National Health Service provides universal health coverage; therefore, no segment of the population is excluded from the CPRD and the age and sex distributions are representative of the UK population.⁹ Participating general practitioners (GPs) contribute data in an anonymous format, including demographic information, medical diagnoses, symptoms, referrals, and details of hospital stays and specialist visits, recorded using the Read coding system.^9,10

The children with ASD described in this study were identified for a study that evaluated the association between maternal depression and antidepressant use during pregnancy and risk of ASD in offspring, the protocol of which included a validation component. The source population included live-born, singleton infants born between 1990 and 2011 who had at least 3 years of follow-up in their medical record after birth and who were followed from birth.

ASD Case Identification

We identified children with at least one Read code for an ASD, including diagnoses of autism, Asperger’s syndrome, and PDD. See Table 1 for the complete list of Read codes used to identify children with ASD. The patients with ASD identified in the CPRD, a primary care data source in a country with national health care, represent a broad range of ASD severities seen in clinical practice. These diagnoses are not restricted to the most severe cases that would be identified using secondary care or specialty care data sources.

Table 1 Read codes indicating ASD Notes: **Indicates Read codes most commonly used by GPs to indicate ASD in the CPRD. Abbreviations: ASD, autism spectrum disorder; CPRD, Clinical Practice Research Datalink; GP, general practitioner; ICD, International Classification of Diseases; NOS, not otherwise specified.

Characteristics of interest described in this study include child sex, birth cohort (categorized as 1990–1999, 2000–2004, 2005–2009, 2010–2011), age at first ASD diagnosis, birth order, type of ASD Read code recorded (autism spectrum disorder, Asperger’s or PDD), and the number of ASD codes recorded in the child’s electronic record. We also reviewed electronic medical records of children with an ASD code to identify Read codes that supported the ASD diagnosis. These codes included those indicating presence of the core signs of ASD (e.g., developmental delay, speech delay, behavioral problems), as well as codes that indicated referrals and visits to specialists, letters from specialists, receipt of specialized therapy, or suspected autism. Finally, we noted the presence of codes for other conditions known to be associated with ASD, such as repetitive movements, gait abnormalities, parental concern, school problems, sleep problems, attention deficit hyperactivity disorder (ADHD) diagnoses or pharmaceutical treatment, and epilepsy.

Validation

To assess the validity of ASD diagnoses recorded in the CPRD, we compared the information present in the electronic medical record to information from de-identified original medical records, which included detailed hospital clinic letters, consultant reports, speech and language assessments, and/or specialist reports. We received and reviewed the original records for a random sample of 37 ASD cases. If the diagnosis recorded in the electronic data was confirmed by the original medical record, then we considered the child to be a confirmed case of ASD. Otherwise, we considered the child to not have ASD.

Statistical Analysis

We described the characteristics of ASD cases by sex and birth cohort (1990–1999, 2000–2004, 2005–2009, 2010–2011) to evaluate changes over time in the recording of core signs and supporting clinical codes and other characteristics recorded in the electronic medical records by GPs. We calculated PPVs as a measure of diagnostic validity of ASD diagnoses recorded in the CPRD. PPVs were calculated as the number of true ASD cases confirmed by review of original medical records divided by the total number of children with an ASD diagnosis recorded in the electronic record sampled.

All analyses were performed using SAS statistical software version 9.3 (SAS Institute, Cary, NC, USA). The protocol for this study was reviewed and approved by the Independent Scientific Advisory Committee of the CPRD (protocol number 15_256).

Results

We identified 2154 children with a Read code for ASD in our study population (Table 2). The majority of cases (84%) were male and 50.7% were first born. The peak age at first ASD diagnosis was 3 years, and more than half of the cases were diagnosed by age 6 years. Higher functioning individuals (i.e., those diagnosed with Asperger’s) were diagnosed at a later age. The vast majority of cases (78.4%) had only 1 ASD diagnosis. Most cases (79.3%) had a Read code indicating autism or ASD, whereas 20.1% had a code for Asperger’s and 0.6% had a code for PDD. The majority of cases identified (72.5%) were first diagnosed with ASD between 2000 and 2011 corresponding to the time after DSM-IV guidelines were updated (2000), while 2.7% were diagnosed between 1990 and 1999 using DSM-IV guidelines updated in 1994, and 24.8% were diagnosed at the end of the study period (2013–2014) after the DSM-5 guidelines were published.

Table 2 Characteristics of ASD cases, overall and by sex Abbreviations: ADHD, attention deficit hyperactivity disorder; ASD, autism spectrum disorder; PDD, pervasive developmental disorder.

Approximately half of the cases had codes for at least one of the core signs of ASD (e.g., developmental delay, behavioral problem, or speech delay). Approximately one-quarter of cases had codes for referral to or visit with a specialist, 10.3% had codes for receipt of occupational or speech therapy, and 17% had codes indicating that the GP received a letter from a specialist on the same date as the ASD diagnosis. There were proportionally more cases with no supporting clinical codes (25.8%) among those with a diagnosis of Asperger’s than among cases who had records for core signs of ASD or codes that indicated they received treatments or referrals. Proportionally more ASD cases that had supporting clinical codes in their record also had codes for gait abnormality, sleep problems, ADHD diagnosis or medications prescribed to treat ADHD compared to cases who had no supporting evidence.

We present characteristics of ASD cases by sex in Table 2. A higher proportion of male cases had Read codes indicating behavioral problems, whereas a higher proportion of female cases had codes indicating developmental delay. Male cases had more diagnoses of gait abnormalities and ADHD or received prescriptions for medications to treat ADHD.

We present characteristics of ASD cases by birth cohort in Table 3. The proportion of children diagnosed at age 3 years was higher among children born in 2005–2009 (23.9%) compared to those born in 1990–1999 (8.9%) and 2000–2004 (13.7%). Among cases born in 1990–1999, GPs used ASD codes for 64.8% and Asperger’s codes for 34.4% of all cases, whereas GPs used ASD codes for over 90% of cases born in 2005–2011, reflecting changes in ASD definitions and terminology over the study period (Figure 1). The proportion of cases with only 1 ASD code increased over the study period. The number of cases with a code indicating the presence of one or more of the core ASD signs (behavioral problem, developmental problem, or speech delay) declined from 55.3% of those born between 1990 and 1999 to 46.5% of those born in 2005–2011. This was mainly due to a decline in the proportion of cases with a code for behavioral problem, whereas there was a slight increase in codes indicating speech delay among those born in later birth cohorts. There was an increase in the proportion of patients born in 2005–2011 with codes indicating specialist referrals or visits and letters from specialists on the same date as the ASD diagnosis compared to those born in 1990–1999.

Figure 1 General practitioner recorded ASD diagnosis subtype by birth year. Abbreviations: ASD, autism spectrum disorder; DSM-IV, Diagnostic and Statistical Manual of Mental Disorders Fourth Edition; DSM-5, Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition.

Table 3 Characteristics of ASD cases, by birth cohort Abbreviations: ADHD, attention deficit hyperactivity disorder; ASD, autism spectrum disorder; n/a, not applicable; PDD, pervasive developmental disorder.

An ASD diagnosis was present in the original medical record for 34 out of the 37 sampled patients with an ASD diagnosis recorded in the electronic record. Based on this, the PPV of ASD diagnoses recorded in the CPRD was 91.9%.

Discussion

The results of this study indicate that the PPV of ASD diagnoses recorded in the CPRD is high (greater than 90%). The characteristics of ASD cases in this study were consistent with those of published studies in other populations of children with ASD, including the high proportion of males, age at diagnosis, epilepsy, and birth order.^11–13 The changes in Read codes used by GPs during the study period reflect changes in the diagnostic criteria and terminology that removed the distinction between the different ASD subtypes (autism, Asperger’s, PDDs) into one category (autism spectrum disorder) over this same time period. The finding that males with ASD in our study had more codes indicating behavioral problems, whereas females had more codes indicating developmental delays is consistent with published studies that report differences in the presentation of autistic traits by sex.¹¹ Finally, the recording of clinical codes that support the diagnosis of ASD in the electronic data has declined over time, specifically those for the core signs of ASD such as behavioral problems, developmental delays, and speech delays. However, there was a rise in the proportion of children with codes for referrals and visits to specialists, which may reflect that ASD diagnosis and care is being conducted outside of the GP offices.

Two previous studies have evaluated the validity of ASD diagnoses in the General Practice Research Database (now CPRD). In the first study, autism was confirmed in 80% of 83 patients with a diagnosis of autism recorded between 1988 and 1999.⁷ In a second study published in 2004, PDDs, which included patients with a diagnosis of autism, Asperger’s, or PDD, was confirmed in 92.5% of cases born between 1973 and 1997.⁸ In our study, the overall PPV was 91.9%. These three studies provide evidence that ASD diagnoses recorded in the CPRD are reliable and can be used with confidence to study ASD.

Our study results indicate that there has been a decrease in the recording of codes for core signs of ASD (behavioral problems, developmental delays, and speech delays) by GPs in the electronic medical records over time, whereas details about these core signs were present in the original medical records over the full study period. However, there was an increase in the recording of codes indicating specialist referral or visit over the study period, which may reflect that diagnosis and care for ASD are occurring outside the GPs office. From a research point of view, it is unfortunate that use of these additional supporting codes is declining in the electronic data because, in the absence of original medical records, these codes provided important details of each case. In addition, these types of supporting clinical details can be useful in assessing the quality of diagnoses recorded in electronic data sources so that case misclassification can be minimized. Despite the decline in these supporting codes, the PPV has remained high, indicating that ASD diagnoses recorded in the CPRD are reliable and can be used to study ASD with confidence.

It is important to note that ASD diagnostic criteria and terminology referring to ASD have changed multiple times since the data in the CPRD were first collected in 1987. As a result, the Read codes used by GPs to indicate ASD have changed over time. For example, GPs used codes indicating ASD in 64.8% and Asperger’s in 34.3% of the cases born in 1990–1999, whereas GPs used codes for ASD in over 90% of cases born in 2005–2011 while only 2% of cases born in these later years had a code for Asperger’s. These changes emphasize the need for researchers to take care to select all appropriate Read codes used by GPs over the time period under study to avoid missing cases.

A strength of our study is that we describe the experiences of children with ASD from birth through the end of their electronic record. The majority (72.5%) of the ASD cases described in this study were diagnosed after the DSM-IV criteria for autism were revised in 2000. Although approximately 25% of the cases were diagnosed after the DSM-5 revision was published in 2013, the NICE quality standards for assessment and diagnosis of ASD use DSM-IV criteria;¹⁴ therefore, it is not clear whether GPs or specialists had begun using the new criteria outlined in DSM-5. Finally, policy changes around access to original medical records were implemented during the course of this study, and as a result we did not receive all of the original medical records that we requested. Thus, our PPV estimates are based on a smaller sample than originally planned.

Conclusion

The results of this study suggest that ASD diagnoses recorded in the CPRD are reliable and can be used with confidence to study ASD despite changes in ASD diagnostic criteria and screening practices over time.

Acknowledgments

Research reported in this publication was supported by the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health under Award Number R01HD084680. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. All authors had independence from the funding source.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Park HR, Lee JM, Moon HE, et al. A short review on the current understanding of autism spectrum disorders. Exp Neurobiol. 2016; 25(1):1–13.
2.		Baxter AJ, Brugha TS, Erskine HE, Scheurer RW, Vos T, Scott JG. The epidemiology and global burden of autism spectrum disorders. Psychol Med. 2015;45(3):601–613.
3.		Volkmar FR, Reichow B. Autism in DSM-5: progress and challenges. Mol Autism. 2013;4(1):13.
4.		King BH, Navot N, Bernier R, Webb SJ. Update on diagnostic classification in autism. Curr Opin Psychiatry. 2014;27(2):105–109.
5.		Mattila ML, Kielinen M, Linna SL, et al. Autism spectrum disorders according to DSM-IV-TR and comparison with DSM-5 draft criteria: an epidemiological study. J Am Acad Child Adolesc Psychiatry. 2011;50(6):583–592.
6.		Kim YS, Leventhal BL, Koh YJ, et al. Prevalence of autism spectrum disorders in a total population sample. Am J Psychiatry. 2011;168(9):904–912.
7.		Black C, Kaye JA, Jick H. Relation of childhood gastrointestinal disorders to autism: nested case-control study using data from the UK General Practice Research Database. BMJ. 2002;325(7361):419–421.
8.		Fombonne E, Heavey L, Smeeth L, et al. Validation of the diagnosis of autism in general practitioner records. BMC Public Health. 2004;4:5.
9.		Lawson DH, Sherman V, Hollowell J. The general practice research database: scientific and ethical advisory group. QJM. 1998;91(6):445–452.
10.		Herrett E, Gallagher AM, Bhaskaran K, et al. Data resource profile: clinical practice research datalink (CPRD). Int J Epidemiol. 2015;44(3):827–836.
11.		Werling DM, Geschwind DH. Sex differences in autism spectrum disorders. Curr Opin Neurol. 2013;26(2):146–153.
12.		Durkin MS, Maenner MJ, Newschaffer CJ, et al. Advanced parental age and the risk of autism spectrum disorder. Am J Epidemiol. 2008;168(11):1268–1276.
13.		Gardener H, Spiegelman D, Buka SL. Prenatal risk factors for autism: comprehensive meta-analysis. Br J Psychiatry. 2009;195(1):7–14.
14.		National Institute for Health and Care Excellence (NICE). Autism spectrum disorder in under 19s: recognition, referral and diagnosis [published Septemeber 2011]. Available from: https://www.nice.org.uk/guidance/cg128. Accessed June 14, 2017.