Validation Study of Mild Traumatic Brain Injury Case-Identifying Algorithms in French Emergency Departments: High Performance in Children but Limited in Adults

Introduction

Of the estimated 50 to 60 million traumatic brain injuries worldwide each year,¹ approximately 90% are mild traumatic brain injuries (mTBI).² mTBI are defined as transient disruptions of brain function caused by an external force (eg, head impact, violent acceleration/deceleration movement of the head). They can occur in a variety of contexts (eg, sporting or leisure activities, road traffic accidents, falls at home, assaults).

Despite being described as mild, mTBI are not benign. Several studies have shown that 30% to 40% of mTBI patients have persistent symptoms 3 to 6 months after their injury occurs, and 5% to 20% still present symptoms at 12 months.^3–6 These physical, cognitive, and emotional symptoms have a major impact on an individual’s personal and professional lives.⁷ Moreover, there is a strong suspicion that mTBI are linked to subsequent psychiatric disorders.^8,9

Despite being an important public health issue, France does not have a dedicated epidemiological surveillance system for mTBI. A large proportion of mTBI victims in the country are treated in emergency departments (ED).¹⁰ Data on ED visits are collected by a national system known as OSCOUR (Organisation de la surveillance coordonnée des urgences).¹¹ These data are collected daily and are near-exhaustive. Given this context, data from OSCOUR could be useful for setting up a dedicated mTBI surveillance system.

In OSCOUR, disease and injuries can be identified using single codes or a combination of codes from the International Classification of Diseases 10th revision (ICD-10). However, the performance of potential algorithms based on ICD-10 codes for identifying cases of mTBI in OSCOUR has not been assessed.

Due to potential coding inaccuracies in routinely collected data, it is essential to conduct a validation study before applying any algorithm—regardless of the pathology or injury being identified and regardless of the routine database in which the algorithm will be applied.¹⁰ More specifically regarding mTBI, several factors may affect coding accuracy. While there is a specific ICD-10 code for mTBI—S06.0 “concussion”—uncertain or suspected cases of mTBI may not be coded, as there is no explicit ICD-10 code for suspected mTBI to our knowledge. These suspected cases represent a significant proportion of emergency department visits. Additionally, patients with mTBI often present with other injuries or conditions, which may lead to mTBI being overlooked in favor of coding more urgent or obvious diagnoses. Furthermore, coding inaccuracies by emergency physicians may occur due to the high workload and limited time available to select the most appropriate diagnostic code in the emergency setting.

We found only two validation study of performance indicators of mTBI algorithms in the literature. Conducted in the United States, and based on ICD-9 codes, it found that potential case-identifying algorithms performed poorly.^12,13 While the results of these validation studies are interesting, they cannot be directly transposed to OSCOUR data. In OSCOUR, diagnoses are coded using ICD-10 codes. The type of classification system used to encode diagnoses is a key factor that can significantly impact coding quality. Notably, ICD-9 and ICD-10 are fundamentally different in structure, and ICD-10 contains 12,420 codes, compared to « only » 6,969 in ICD-9, making it a far more detailed classification system.¹⁴ Other studies which validated algorithms for traumatic brain injuries of all severities (not only mTBI) based on ICD-10 codes found contrasting results.¹⁵

These contrasting findings, together with the abovementioned risk of classification bias, underline the importance of conducting a validation study before applying algorithms to identify mTBI cases in OSCOUR for routine monitoring.

The objective of the present validation study was to measure the performance of various potential algorithms, based on ICD-10 codes, for identifying mTBI ED visits in the OSCOUR database.

Methods and Analysis

Study Design

We performed a retrospective multicenter validation study of potential algorithms for identifying mTBI based on ICD-10 codes using the OSCOUR database. We calculated performance indicators of the various algorithms by using medical charts from ED visits as a reference source. The protocol for this study has been published previously.¹⁶

Study Population and Data Sources

Our study population was composed of a random sample of patients of all ages presenting to one of the four emergency departments (EDs) within the OSCOUR^® network in France in 2019, regardless of the reason for their visit. We selected a broad and inclusive study population, comprising all emergency department (ED) visits regardless of the reason for attendance. This approach aligns with best-practice recommendations for validation studies, as it allows for the comprehensive estimation of all standard performance indicators, including: Sensitivity, Specificity, Positive Predictive Value (PPV), and Negative Predictive Value (NPV). A detailed description of these metrics, is provided later in the Methods section.

Three of these ED were randomly selected while the fourth participated in a pilot study which we conducted prior to the present study. Two ED were located in university hospitals and two in general hospitals. The ED were located in four different regions in France: Auvergne-Rhône-Alpes, Centre-Val de Loire, Grand-Est, and the Ile-de France (which covers greater Paris).

For each ED visit, medical information such as primary diagnosis (PD) and up to 10 associated diagnoses (AD) are recorded in OSCOUR. Other information including chief complaint, demographic data (gender, age, residential zip code), ED date and time of arrival and of discharge, and orientation on discharge (home, hospital ward, etc) are also included. Doctors or specific administrative staff code the PD and AD according to the ICD-10.¹⁷ In 2019, the OSCOUR network included 680 ED and covered 93% of all ED visits, including French overseas regions (except Martinique). On average, 56,700 ED visits per day were recorded. PD was recorded for 77% of the visits while AD was recorded in less than 10%. Data are pseudonymized in the OSCOUR database; accordingly patients cannot be directly identified.

We randomly selected 12,000 ED visits (3,000 ED per site) occurred between 01/01/2019 and 31/12/2019 in OSCOUR. The rational for the sample size was described in detail in the study protocol published earlier.¹⁶ The study period was chosen as it was the last year with consolidated data on ED visits before the onset of the COVID-19 pandemic.

Identification of mTBI Cases in Medical Charts (Reference Source)

Medical charts from ED visits were used as the reference source to validate the mTBI cases identified with the algorithms tested in the OSCOUR database (see below).

Medical charts for each ED visit include data on clinical examination, and the attending physician’s conclusions regarding diagnosis and medical management. This information is stored on each ED’s hospital server.

Medical charts corresponding to the ED visits which we randomly selected from OSCOUR database were extracted from the corresponding ED’s hospital server.

No unique identifier existed for each patient allowing matching information between the OSCOUR database and the medical chart, in order to identify medical charts corresponding to selected ED visits in OSCOUR database, we had to cross-link several variables (gender, residential postal code, date of birth, ED date and time of arrival and of discharge, and orientation upon discharge).

Once extracted, two epidemiologists separately analysed the charts and categorised patients as “certain”, “probable”, “possible” or “non-case” mTBI cases (see definitions and classification criteria below). Chart classification was blinded: neither epidemiologist knew the ICD-10 codes detailed in the OSCOUR database. Cases which were difficult to classify were reviewed and classified in collaboration with Dr Lorton and two other mTBI expert physicians in order to guarantee the best possible classification of cases.

A total of 250 medical charts were separately reviewed by the two epidemiologists in order to assess classification agreement between both and the Kappa statistic was calculated.

Initially, for feasibility reasons, we had planned to use a “saturation process” approach to reduce the number of medical charts requiring manual review. This process is fully described in the protocol article.¹⁶ The general principle of this approach was to first identify all possible ICD-10 codes used to classify mild traumatic brain injury (mTBI) cases in the OSCOUR database for each emergency department. Subsequently, we would have reviewed only the medical charts corresponding to ED visits with a potential mTBI-related ICD-10 code. Medical charts from ED visits without any potential mTBI-related ICD-10 codes in the OSCOUR database would have been automatically classified as non-mTBI cases without further review. Unfortunately, due to the significant heterogeneity in ICD-10 codes associated with mTBI cases, the saturation process could not be successfully completed, and it was not possible to reduce the number of medical records requiring manual review.

Criteria for the Classification of Medical Charts

mTBI diagnosis is complicated for various reasons. First, no reference tests exist; diagnosis is based solely on the identification of clinical signs, which may be very brief and therefore difficult to identify (by the patient, their relatives or witnesses). It is also sometimes difficult to understand whether other factors may explain the observed clinical signs. Consequently, many mTBI cases are suspected/uncertain.

Given this great diagnostic uncertainty, it does not seem appropriate for epidemiological surveillance to define mTBI using a single definition. It seems more appropriate to use several definitions, as has already been done in the surveillance of mTBI.¹⁸

Thus, we used three definitions of mTBI that have a nested structure.

A narrow definition which included only certain cases of mTBI

A moderately broad definition that included certain and probable cases of mTBI

A broad definition that included certain, probable and possible cases of mTBI.

The medical charts were classified, as certain cases, probable cases, possible cases and non-cases.

“Certain” mTBI cases were defined on the basis of the diagnostic criteria proposed by the World Health Organisation (WHO)¹⁹ which are widely accepted and are frequently used in epidemiological studies.

Cases classified as “probable” included i) cases of head trauma with signs of cerebral dysfunction which could possibly have been explained by another cause, and ii) head trauma cases presenting only post-concussion symptoms without any clear clinical signs of altered cerebral dysfunction. We used the list of post-concussion symptoms recently proposed by the American Congress of Rehabilitation Medicine (ACRM).²⁰ We also included children under five years old who had behavioural changes following a head trauma among the probable cases. This is because it is more difficult to identify post-concussion symptoms in children.

Finally, the “possible” group included cases with simple head injuries who did not meet any of the WHO criteria, and who had no post-concussion symptoms or behavioral changes.

The choice to create this “possible” group was based on the fact that the absence of WHO criteria, post-concussion symptoms or behavioural changes in case medical charts does not necessarily mean that these signs or symptoms were absent. There are at least three reasons why this may occur: first, signs or symptoms may not have been reported by the patient (eg, because they were very brief) or were too difficult to identify (eg, in children under 5 years of age). Second, it is possible that signs were reported by the patient or loved ones, but were not recorded by the attending physican in the patient’s medical chart. Third, some patients may have experienced post-concussion symptoms after being released from the ED.²⁰

The diagnostic criteria used to classify cases as certain, probable, or possible are presented in Table 1. For each criterion, it is specified whether it is a necessary criterion, a sufficient criterion (in combination with others), for classifying the case as certain, probable, or possible.

Table 1 Criteria for Certain, Probable, and Possible Mild Traumatic Brain Injury Cases

Definitions for each criterion are provided in Box 1 and final definition criteria are summarized in Figure 1.

Box 1 Definition of Criteria for Certain, Probable, and Possible mTBI Cases

Figure 1 Presentation of the various stages involved in classifying cases from medical charts as “certain”, “probable”, “possible” and “non-cases” of mTBI.

Identification of mTBI Cases in the OSCOUR Database Using Algorithms Based on the ICD-10

The literature review we conducted highlighted that various ICD-10 codes were used to identify cases of mTBI in routinely collected databases, and that there is no consensual algorithm. The most frequently used algorithm included only the ICD-10 code S06.0 (concussion).

We tested a first algorithm based on the S06.0 “concussion” ICD-10 code only, which we named the “S06.0 algorithm”. We developed other algorithms based on combinations of the S06.0 ICD-10 code and other codes identified in the literature review. We named these algorithms “Broad algorithms”. It is important to specify that all codes identified in the literature were clinically relevant supporting their inclusion in our algorithms.

The S06.0 algorithm and the Broad algorithms fall under the umbrella of “traditional algorithms” (ICD-10 codes previously used in the literature).

The ICD-10 codes composing the traditional algorithms are presented in Table 2, and the different traditional algorithms are presented in Table 3.

Table 2 ICD-10 Codes Included in Algorithms and Their Description

Table 3 Definition of “Traditional” ICD-10 Algorithms Tested

ED visits with at least one ICD-10 coded medical diagnosis (PD or AD) included in our algorithms (S06.0 or Broad) were considered “cases” in the OSCOUR database. ED visits without an ICD-10 coded medical diagnosis included in our algorithms were considered “non-cases”.

Following this initial analysis, we sought to improve the performance of our best traditional algorithms by adding codes associated with the false negatives cases. By simplifying, we will named those codes “false negative ICD-10 codes”.

To do this, we listed all the false negative ICD-10 codes associated with the best performing traditional algorithm (performance criteria are defined below in the statistical analysis subsection) according to each of our three definitions (narrow, moderately broad, broad). Among these codes we selected only those which were clinically relevant: describing injuries frequently associated with mTBI. Each of the latter was then added individually to the best performing relevant traditional algorithm. ICD-10 codes that improved the performance of the relevant algorithm were retained, and then based on these codes, all possible combinations of codes were tested. Given the potentially large number of false negative ICD-10 codes and their potential low frequency, we chose to group codes from the same ICD-10 sub-chapter (codes beginning with the same three characters). As our “moderate” mTBI definition also encompassed the narrow definition and the broad definition covered all three possibilities, we included all false negative codes that improved the performance of the best “narrow” algorithm in both the “moderate” algorithms and in the “broad” algorithms. Similarly, false negative codes that improved the performance of the best “moderate” algorithm were included in the “braod” algorithm. We called these algorithms, which included traditional codes and new ICD-10 codes, “optimized algorithms”.

Statistical Analysis

First, we described characteristics of patients (gender, age). For categorical variables, both numbers and percentages were reported. For continuous variables the median with inter quartile range (IQR) were reported.

We evaluated the performance of the ICD-10 code algorithms according to our three mTBI definitions (see above). The following performance indicators were calculated: sensitivity, specificity, predictive positive value (PPV), and predictive negative value (NPV) with their 95% percent Clopper-Pearson exact confidence intervals. These indicators were first calculated for traditional algorithms and then for the optimized algorithms. The formulas for calculating the algorithms’ performance indicators criteria are described in Table 4.

Table 4 Formulas for Calculating Sensitivity, Specificity, PPV, and NPV of mTBI Case Identification Algorithms in OSCOUR

We expected the proportion of ED visits for mTBI defined according to our three definitions to be low. Accordingly, we considered that the best performing algorithm was that with the highest PPV and an acceptable degree of sensitivity. This strategy has been used in several studies validating algorithms of low-prevalence diseases.^21–23 There is no consensus for an acceptable threshold for sensitivity or PPV in the literature. We considered that both should be at least 80%, in accordance with several studies for these two indicators.^15,24 Assuming that the PPV could reach values in excess of 90%, for several algorithms according to a given mTBI definition (narrow, moderate, broad)), we considered that the best algorithm was the one with the greatest sensitivity.

Finally, we hypothesized that mTBI cases were less accurately coded in certain contexts, specifically when patients used psychoactive substances prior to their trauma, or when patients presented other pathologies or other lesions in addition to mTBI. Furthermore, we hypothesized that this inaccurate coding varied in particular according to the age and gender of the patients. Consequently, we conducted an analysis stratified by age group and gender to check whether the performance of our best algorithms varied with these two variables. Thanks to sufficient case numbers, we evaluated our top-performing algorithms for broad mTBI detection across six age groups: 0–4, 5–17, 18–24, 25–64, 65–74, and ≥75 years. The factors we assume could affect coding vary across these age groups. However, for algorithms detecting restrictive and moderately broad mTBI, limited sample sizes necessitated broader age categories: 0–17, 18–64, and ≥65 years. For comparability, we also evaluated our top-performing algorithms for broad mTBI detection algorithm within these three groups.

All statistical analyses were performed using SAS Enterprise Guide 7.4.

Results

Characteristics of the Study Sample

For feasibility reasons, of the 12,000 medical charts from ED visits initially randomly selected, 5,185 medical charts from ED visits were reviewed by the two epidemiologists. Of these, 1272 (24.5%) were from the Auvergne-Rhône-Alpes region, 1607 (31%) from the Centre-Val de Loire region, 1049 (20.2%) from the Grand-Est region, and 1257 (24.2%) from the Ile-de France region. Of all 5,185 charts, 44 (0.8%) were classified as certain cases, 106 (2.0%) as probable cases, and 378 (7.3%) as possible cases. Another 4,577 (88.3%) were classified as non-cases. A further 80 (1.5%) charts were unclassifiable (empty, or too little information), and were classified as non-cases. Therefore, a total of 4657 (89.8%) charts were classified as non-cases.

The demographic characteristics of the mTBI cases (all types) and non-cases according to the medical charts are presented in Table 5.

Table 5 Charateristics of mTBI Cases (Narrow, Moderate and Broad Definition) and Non-mTBI Cases Identified in Medical Charts Review

Of the 250 medical charts separately reviewed by the two epidemiologists, the Kappa statistic was 0.88 [0.79–0.98], reflecting very good classification agreement.

Performance of mTBI Case-Identifying Algorithms Based on ICD-10 Codes Previously Used in the Literature

For ease of reading, hereafter we will refer to “narrow”, “moderate”, and “broad” mTBI cases when referring to mTBI cases according to their definition.

The performances of the best mTBI case-identifying algorithms based on the traditional ICD-10 codes are presented in Table 6.

Table 6 Performance of the Best Traditional Algorithms for Identifying mTBI According to Each Definition (Narrow, Moderate, Broad)

The best traditional algorithm for identifying narrow mTBI cases was algorithm BA6 which combined the following codes: S06.0, S02.9, S06.8. Sensitivity, specificity, PPV and NPV were 38.6%, 97.5%, 11.5% and 99.5%, respectively.

The best traditional algorithm for identifying moderately broad mTBI cases was BA17, which combined the following codes S06.0, S02.9, S06.8, S09.9. Sensitivity, specificity, PPV and NPV were 36%, 98.1%, 35.8% and 98.1%, respectively.

The best traditional algorithm for identifying broad mTBI cases was BA31, which combined the following codes S06.0, S02.9, S06.8, S06.9 S09.8, S09.9. Sensitivity, specificity, PPV and NPV were 32.2%, 99.7%, 92.9% and 92.8%, respectively.

The performance of all traditional algorithms for identifying these various mTBI are presented in appendices 1,2 and 3 respectively.

Description of False Negatives From Algorithms

BA6, the best traditional algorithm for identifying narrow mTBI, did not identify 27 true (according to the medical charts) mTBI in OSCOUR (it returned 27 false-negatives). The most frequent ICD-10 codes among these 27 false negative cases were: S00.0: Superficial injury of scalp (n=4, 15%), S00.7: Multiple superficial injuries of head (n=2, 7%) and S06.1: Traumatic cerebral oedema (n=2, 7%).

BA17, the best traditional algorithm for identifying moderately broad mTBI cases, did not identify 96 true mTBI. Among these false-negatives, the most frequent ICD-10 codes were: F10.0: Mental and behavioural disorders due to use of alcohol (n=11, 11%), S00.0: Superficial injury of scalp (n=6, 6%), and S06.9: Intracranial injury, unspecified (n=5, 5%).

BA31, the best traditional algorithm for identifying broad mTBI cases, did not identify 358 true mTBI. Among these false-negatives, the most frequent ICD-10 codes were S01.8: Open wound of other parts of head (n=39, 11%), S01.0: Open wound of scalp (n=26, 7%) and F10.0: Mental and behavioural disorders due to use of alcohol (n=20, 6%).

The complete list and frequency of false negative codes associated with the BA6, BA17 and BA31 algorithms are presented in appendices 4, 5 and 6, respectively.

Performance of Optimised mTBI Case-Identifying Algorithms

The best performing optimized algorithms (after adding the false negative codes) are shown in Table 7.

Table 7 Performance of the Best Optimised Algorithms for Identifying mTBI Cases

The best optimised algorithm for identifying narrow mTBI cases was BA6_OPT4, which combined all the ICD-10 codes of BA6 (see above) and the following codes: S00.0, S00.7, S00.9, S06.1, S06.5 and S06.6. The code labels are detailed in appendix 4. Sensitivity, specificity, PPV and NPV were 63.6%, 97.1%, 15.8% and 99.7%, respectively.

The best optimised algorithm for identifying moderately broad mTBI cases was BA17_OPT27, which combined all the ICD-10 codes of BA17 (see above) and the following codes: S00.0, S00.7, S00.9, S02.2, S02.3, S06.1, S06.5, S06.6, S12.9. The code labels are detailed in appendix 5. Sensitivity, specificity, PPV and NPV were 48.7%, 97.7%, 38.2% and 98.5%, respectively.

The best optimised algorithm for identifying broad mTBI was BA31_OPT1, which combined all the ICD-10 codes of BA31 (see above) and the following codes: S00.0, S00.1, S00.3, S00.5, S00.7, S00.8, S00.9,S01, S01.0, S01.1, S01.2, S01.3, S01.4, S01.5, S01.7, S01.8, S01.9, S02.2, S02.3, S02.5, S02.6, S02.8, S03.1, S05.1, S06.1, S06.3, S06.5, S06.6, S09.2, S10.8, S11.9, S12.9, S13.4, S16. The code labels are detailed in appendix 6.

Sensitivity, specificity, PPV and NPV were 70.1%, 99.2%, 91.4% and 96.7%, respectively.

The definitions of all the algorithms tested during the optimization process for identifying mild traumatic brain injury (mTBI) cases are presented in appendices 7, 8, and 9. The performance of all the algorithms, including the addition of various “false negative” ICD-10 codes, is detailed in appendices 10, 11, and 12. Finally, the performance of all the optimized algorithms is presented in appendices 13, 14, and 15, respectively.

The performance of the three best performing optimized algorithms varied according to gender and age (Table 8). BA6_OPT4 (narrow mTBI), did not meet the acceptable threshold of PPV≥80% and sensitivity ≥80% for any of the sub-groups considered. Similarly, BA17_OPT27 (moderate mTBI) did not meet the minimum performance thresholds for any of the sub-groups considered. BA31_OPT1 (broad mTBI) exceeded the acceptability threshold but only for the 0–4 years age group. Sensitivity, specificity, PPV and NPV were 90,7%, 99.3%, 95.1% and 98,7%, respectively. For the 5–17 age group, sensitivity (79,4%) nearly reaches the acceptability threshold, while the PPV is 95,3%, well above the threshold. Across the entire 0–17 age group, sensitivity and PPV both exceeded acceptability thresholds, at 85.2% and 95.2%, respectively.

Table 8 Performance of the Best Optimised Algorithms in Identifying mTBI Cases for Different Age Groups and Sex

Discussion

To our knowledge it is the first study to validate potential algorithms for identifying cases of mTBI based on French ED data from the OSCOUR network. It is also the first such study based on ICD-10 codes. Algorithm performance varied according to population characteristics, and none of the algorithms tested for the identification of mTBI cases achieved the minimum performance requirements (sensitivity and PPV ≥ 80%) over all age or sex groups. However, sub-group analyses highlighted that one algorithm (BA31_OPT1) had acceptable performance for identifying mTBI according to our “broad” definition for people under 18 years old. Sensitivity, specificity, PPV and NPV for this algorithm were 85.2%, 99.4%, 95.2% and 98%, respectively.

We used three definitions to classify mTBI cases in patient medical charts (our reference source for validating our algorithms): a narrow definition covering only certain cases, a moderate (moderately broad) definition covering certain and probable cases, and a broad definition covering certain, probable and possible cases. The number and characteristics of the cases selected varied greatly depending on the definition used, from just under 1% of ED visits for the narrow definition to 10% for the broad definition.

An Australian study estimated the proportion of mTBI among all ED visits by retrospectively reviewing medical charts in a single emergency department between June 2015 and February 2016. Using a narrow definition similar to ours, the study reported a proportion of 1.2% (351/30,479) in the 18–65 age group. In our study, the corresponding proportion was 0.6% (14/2,465) for the same age group. While cross-country comparisons of such frequency indicators are complex—due to differing epidemiological contexts, healthcare-seeking behaviors, and health system organizations—it is noteworthy that our findings fall within a comparable range.¹⁸

Regarding case characteristics, our study shows that mTBI cases defined using the broad definition were younger than those defined using the two more conservative definitions. Possible cases (included in the mTBI broad definition) correspond to minor head injuries with no signs of altered brain function, post-concussion symptoms or behavioural changes. Consistent with the literature, these minor injuries are particularly frequent in children under 5 years old, in our study (data not shown); this explains the much larger number of cases and the younger age of the cases selected with the broad mTBI definition.^25,26 Furthermore, few very young children fell under the narrow and moderate definitions because the criteria used for both were not adapted to the identification of mTBI in this population. Indeed, the signs of altered cerebral function and post-concussion symptoms are complicated to identify in the youngest children.²⁷

The performance of our algorithms for identifying “narrow” and ‘moderate’mTBI was particularly weak. First, the PPVs were very low for these algorithms. This can be explained by the fact that the same ICD-10 codes were most frequently used to code “certain”, “probable” and “possible” cases (data not shown) in OSCOUR. Consequently, these three case types cannot be distinguished on the basis of the ICD-10 codes used and, by extension, it is not possible to identify narrow or moderate mTBI.The imprecise use of code S06.0 (concussion) contributed significantly to the low PPV of the narrow and moderate algorithms. More specifically, as it is the only code explicitly describing mTBI in the ICD-10 classification, it should in theory be used only to code certain cases. However, in our study, it was the most used to code certain, probable and possible cases. The absence of a specific code for probable cases could explain why probable cases are frequently coded as S06.0. Moreover, the lack of differentiation in coding between certain and possible cases (the latter describing “simple” minor head trauma without clinical signs or symptoms) may reflect a semantic ambiguity. It has been shown that doctors do not always make a precise distinction, at least in the terms used to describe them, between mTBI and minor head injuries without transient disruption of brain function: some genuine mTBI cases are incorrectly labeled as minor head injuries despite meeting diagnostic criteria for mTBI and conversely simple minor head injury without any alteration of brain function are often inappropriately classified as mTBI.¹⁸

Then, sensitivity was below the acceptability threshold for our best “narrow” and “moderate” mTBI algorithms. mTBI cases are coded using highly heterogeneous ICD-10 codes in OSCOUR. Our best traditional narrow and moderately broad algorithms, based on the limited ICD-10 code sets identified in the literature for mTBI detection, failed to capture all relevant mTBI cases coded in OSCOUR. The optimization phase’s strategy of incorporating false-negative codes to boost sensitivity was unsuccessful, encountering two major obstacles. First, a large proportion of false negatives involved head trauma codes. While their inclusion should theoretically have enhanced sensitivity, these codes lacked specificity: none distinguished between certain, probable, or possible mTBI cases. Their addition would have improved case detection but at the expense of severely degrading the PPV. As a result, most of these codes could not be integrated into either our best narrow algorithm (restricted to certain cases) or our moderately broad algorithm (covering certain and probable cases).

Second, a substantial proportion of false negatives involved ICD-10 codes that lack clinical relevance to mTBI—such as those describing non-specific symptoms, unrelated pathologies, or traumas other than head injuries—and were therefore not included in either our best narrow or moderately broad algorithms. For example, the code F10.0 (“Mental and behavioural disorders due to use of alcohol”) was frequently used in isolation for intoxicated mTBI patients, without any accompanying head trauma code.

Results for our best “broad” algorithm were slightly different: the PPV and sensitivity was far higher than that of the best “narrow” and “moderate” algorithms. To capture the highly heterogeneous nature of broad mTBI cases (which encompass certain, probable, and possible mTBI cases), our best “broad” algorithm incorporates a very large number of ICD-10 codes, including many head trauma codes and a few neck trauma codes. This algorithm includes all ICD-10 codes identified in the literature, as well as numerous additional codes incorporated during the optimization phase. The best-performing algorithm includes a very large number of ICD-10 codes yet maintains a remarkably high PPV (exceeding the 80% acceptability threshold). This is explained by the fact that the numerous ICD-10 codes for head or neck trauma in our best-performing algorithm are almost exclusively reserved for coding true broad mTBI cases in the OSCOUR database, rather than other injuries or pathologies. However, this broad approach remains insufficient, as sensitivity stays below the 80% acceptability threshold. As observed with narrow or moderately broad mTBI algorithms, a substantial proportion of broad mTBI cases are coded with non-specific codes: codes describing alcohol-related behavioral disorders, symptomatic presentations, various pathologies, or non-head traumas. Overall, the performance of our algorithm remains insufficient: while the identified cases are indeed true broad mTBI cases, the algorithm fails to capture all broad mTBI cases effectively.

Two exceptions emerge in the 0–4 age group, where our broad algorithm demonstrates remarkably strong performance, achieving a PPV of 95.1% and sensitivity of 90.7%, and in the 5–17 age group, where PPV was 95.3% and sensitivity was 79.4%, nearly meeting the 80% acceptability threshold. The high PPV aligns with the patterns observed across all age groups, but the markedly elevated sensitivity—significantly higher than that observed in older populations—can be attributed to the unique clinical profile of children (0–17 age group). In children and particularly in the 0–4 age group, alcohol intoxication and comorbid conditions are much less frequent than in the adult population. The far lower frequency of these confounding factors—which frequently complicate mTBI coding in older age groups—likely explains the higher sensitivity observed in children.

Two previous studies assessing algorithm based on ICD-9 codes were conducted in the United States.^12,13 These studies found that algorithms performed poorly (Sensitivity: 45.9%, Specificity: 97.8%, PPV: 23.7%, NPV: 99.2% for the first study published in 2006 and Sensitivity: 22.1%, Specificity: 97.1%, PPV: 70.9%, NPV: 56.3% for the second study published in 2013).

A more recent study from Australia (which is not technically a validation study), showed that it was difficult to capture all mTBI cases in routinely collected databases. Specifically, it found that only 10.7% of cases identified retrospectively in patient records and defined using a similar definition to our “moderate” definition, were selected on the basis of SNOMED codes (a classification other than ICD-10) describing mTBI.¹⁸

Our study has several strengths. First, we followed best practice recommendations to conduct validation studies.²⁸ Second, three of the four participating ED were selected randomly. Selecting all the ED on a voluntary basis could have led to possible bias linked to the participation of “good coders”. Second, the study was based on a random selection of ED visits irrespective of the reasons for consultation. This enabled us to calculate the four indicators classically measured in validation studies (Sensitivity, Specificity, PPV and NPV) in order to have the most comprehensive view possible of the performance of our various algorithms. Third, we used three definitions for the mTBI in our reference source in order to take into account the difficulty of diagnosing mTBI in the absence of a specific diagnostic test (biological or radiological). Fourth, we included patients of all ages; this helped us to observe whether there were significant variations in algorithms’ performance across age groups.

Finally, we reviewed the classification of uncertain cases with Dr Lorton and two other mTBI expert physicians in order to guarantee the best possible classification of cases.

Our study also has several limitations. First, one important limitation is the small sample size. The aim was to analyse 12,000 medical charts, but this target could not be meet for feasibility reasons. The “saturation process”, described in detail in the protocol article, which should have enabled us to reduce strongly the number of medical charts to be analysed, was not achieved.¹⁶ As a result, all the medical charts had to be analysed manually. This activity is very time-consuming, and technical problems in some centres slowed down the review of medical charts so that only 5,185 medical charts could be reviewed effectively. The small number of mTBI cases involved affected the performance indicators of the “narrow” and “moderate” algorithm definitions, which were relatively imprecise. These small numbers also meant that we had to use very broad age groups for the age-group analyses. Another limitation is that this study was only conducted in four ED, whereas the OSCOUR network comprised 690 ED in 2019. Consequently, our results cannot be considered as representative of the accuracy of coding in all ED. Having said that, the four ED chosen were heterogenous (two university and two non-university hospitals in four different regions). This diversity should have reduced this possible lack of representativeness.

Then, as no specific biological or radiological tests exist, diagnosis of mTBI is often uncertain, and is largely based on what the patient or those around him or her report (circumstances, signs and symptoms). Therefore, it is possible that reported information is not subsequently recorded by the attending physician. As a result, we cannot rule out the possibility that some medical charts of true mTBI cases may have been misclassified.

Another limitation of our study is that, for feasibility reasons, we analyzed data from a single year. While we cannot entirely rule out the possibility that coding quality may vary across years—potentially leading to misinterpretations of temporal trends—we did not identify any factors likely to explain changes in coding practices outside the COVID-19 period, which uniquely disrupted hospital systems and probably coding practices.

This study provides important information for mTBI surveillance in France, we showed that it is possible to identify cases precisely in OSCOUR according to a “broad” mTBI definition in the paediatric population (0–17 years). Being able to produce surveillance indicators for this population is particularly important, as this is one of the age groups most affected by mTBI. The consequences of mTBI in young children (<5 years), whose brains are still developing, can be particularly serious.^29,30 Despite this positive outcome, our findings underscore the necessity of improving mTBI coding to establish a comprehensive surveillance system across all age groups. This is essential to reliably quantify and characterize cases of mTBI. Such data are crucial for justifying the need for preventive measures, effectively targeting these interventions, and tailoring healthcare services to the population’s actual needs.

This need is particularly acute among older adults. Our best algorithm for detecting broad mTBI significantly underestimates the true burden in older adults. With only 55% sensitivity in the 65+ group, it fails to identify nearly half of all mTBI cases in this population. There is a risk that this age group will not emerge as a priority target for preventive measures. Yet this age group undeniably represents a critical public health challenge: not only does it exhibit high and rising mTBI incidence rates—as consistently demonstrated in recent literature³¹ —but this trend is poised to accelerate with demographic aging. Improving coding is equally critical to enable the identification of mTBI cases under different definitions, given the diagnostic uncertainties associated with this condition.

There are several possible ways to do this. First, coding recommendations could be introduced so that specific codes are used for certain, probable, and possible mTBI cases. Certain cases could be coded using S06.0, the only code specifically describing a mTBI in ICD-10. The code S06.9, which describes unspecified intracranial injury could be used to code probable cases. Possible cases could be coded using head injury codes starting with S00 or S01, or with S16 (cervicalgia) for mTBI resulting from acceleration-deceleration mechanisms without head injury. Additionally, codes for signs of cerebral dysfunction (eg, loss of consciousness, amnesia, confusion) could optionally be used for confirmed cases, while codes for post-concussion symptoms (eg, headaches, nausea, photophobia) could be applied to probable cases.

Integrating pre-formatted, structured interviews—based on the ACRM’s 2023 operational criteria, which are adapted for all age groups and reflect the latest research on mild traumatic brain injury (mTBI)—into emergency department software could be another solution. These interviews, designed as checklist-style questionnaires, would enable emergency physicians to automatically classify cases as certain, probable, possible, or non-cases, while simultaneously assigning the appropriate ICD-10 codes. This approach will enable systematic coding of frequently overlooked cases, such as those involving: alcohol use, elderly patients with femoral fractures or other underlying conditions. Ideally, these questionnaires should automatically appear whenever the reasons for the emergency visit (such as falls, accidents, assault, alcohol intoxication, fainting) suggest a potential mild traumatic brain injury (mTBI).

Another solution could involve using generative artificial intelligence (AI) models to automate mTBI coding by analyzing clinical texts of patient medical charts. Like pre-formatted structured interviews, AI could systematize coding for often-missed mTBI cases and standardize ICD-10 assignment according to diagnostic certainty (certain/probable/possible). However, the essential prerequisite for implementing such methods is that patient medical charts must be accurately completed. A large field of research is waiting to be developed on the question of improving the quality of information collected in medical charts.

Secondly, although the results are not directly transposable, several of the findings of this study may be useful for other studies using algorithms based on ICD-10 codes. Despite the poor performance of most of our algorithms, we found that adding a cervical lesion code improved their performance. Such codes have never been used in previous studies to identify cases of mTBI in data collected routinely. The addition of this code makes it possible to include cases of mTBI occurring as a result of acceleration/deceleration mechanisms, without a direct impact to the head (eg, road accident). The potential value of including this code in case-identifying algorithms for the identification of mTBI cases in other data sources requires further investigation. More generally, the results of this study serve as a reminder of the need to conduct algorithm validation studies before using routinely collected databases, whatever the pathology or lesion of interest.

Conclusion

This is the first validation study of case-identifying algorithms for mTBI based on ICD-10 codes in the French OSCOUR database; it showed that most mTBI case identification algorithms performed poorly in identifying mTBI cases of all ages. Nevertheless, it was possible to identify cases defined according to a “broad” mTBI definition (including certain, probable and possible cases) in the paediatric population (0–17 years). Accordingly, it could be possible to set up surveillance at a national level in the paediatric population. The implementation of comprehensive surveillance of mTBI in ED for all ages and using different definitions of mTBI requires improvements in coding.