Usability of Norm Values on Results from a Simulator Device and Cognitive Tests in Traffic Medicine

Introduction

It is well established that safe driving, an important activity for daily living, successful occupational function, social participation, and well-being, is heavily reliant on functioning cognitive processes.¹ However, there is a lack of evidence in how to predict driving capacity after brain injury/disease^2,3 and most studies conclude that no single assessment tool is sufficient to decide on driving fitness.^3–7 Most studies recommend to collect information from different areas of cognitive ability by analyzing test results from several cognitive assessment tools and preferably in addition observe behavior in simulated or real traffic activities.⁸ To improve the ability to interpret results from cognitive tests, age-related norm values might be of supplementary value, especially because age is known to affect most cognitive functions.⁹ Another aspect of age that might affect the decision from a traffic medicine evaluation, is that the elderly often have more driving experience than younger persons, meaning that actual driving tasks can often happen automatically, which might compensate for some cognitive impairment.¹⁰

Focused, sustained and divided attention, processing speed, executive function, and memory are described as important functions for safe driving.^11–13 To assess the relevant cognitive functions, several tests, such as Useful Field of View (UFOV),^14,15 Trail Making Test (TMT),¹⁶ and the Nordic version of Stroke Driver Screening Assessment (NorSDSA)¹⁷ are often used in the Nordic countries in decisions on continued driving. These assessment tools are available to occupational therapists, who are often engaged in traffic medical assessments.

A simulator device might contribute with information on cognitive areas other than cognitive tests and could therefore be of value in decision based on assessments made in clinical practice.^7,18,19 A new type of a simulator tool has recently been developed and tested at five different traffic medicine units in Sweden. The device (CyberSiM) has shown good internal consistency and promising results in adding important information to the decision discriminating patients considered fit or unfit to drive.⁶ The simulator device includes cognitive assessment of attention, processing speed, visuospatial function, simultaneous capacity and executive function including inhibition.⁷ Because age is known to have an impact on cognition,^20,21 norm values have been produced for the CyberSiM device as well as for some other cognitive tests.^6,22

The aim was to study the usability of cut-off values, based on recently developed norm values for a driving simulator device (CyberSiM), as well as cognitive tests, for patients already considered fit-to-drive after a standardized traffic medical investigation.

Methods

Design

A retrospective case-control study was done based on existing data from two other studies.^6,7 A cut-off for comparing results for patients already considered fit-to drive with a norm population, was set at ±2 standard deviations (SD) from the mean.^16,22

Participants

Healthy individuals (n=129) were used as controls. Cases included patients referred to four different traffic medicine units in Sweden (n=126) for an assessment of continued driving ability after a brain injury/disease (B license).

Data from five different cognitive assessment tools, as well as results from a simulator device, were used to compare the results for the control and patient groups. Based on expected age differences²⁰ and the limited number of participants, the two groups were divided into two according to age: ≤59 years versus ≥60 years.

Data Collection

The study-specific protocol has been used in two former studies.^6,7 The protocol included descriptive information on age, gender, educational level (≤9 years, 10–12 years, >12 years) and estimated driving/year in Swedish miles (<1000, 1000–2000, >2000). Occupational therapists at each unit collected all patient data, and a psychology student collected all norm data. All data related to one participant were collected by the same therapist on one occasion.

Assessment Tools

CyberSiM: the Simulator Test

CyberSiM consists of a software which is installed on a laptop and attached to a steering wheel.⁶ Paddles located on the left- and right-hand side of the steering wheel are used to respond to visual stimuli on the screen (Figure 1), no pedals are included. The participants are asked to steer the car on the simulated road at a constant speed (40 km/h) and at the same time react to stimuli shown on the left or right or both sides of the screen. Before each subtest, a short exercise is used to check that the participants understand each subtest. Participants are asked to steer the car on the right side of the road and avoid obstacles at the side of the road. The examiner follows the session via a separate computer (Figure 1).

Figure 1 CyberSiM. The steering wheel with two paddles (www.Cybersim.com).

Each time a visual stimulus (an arrow pointing to the left or right) pops up on the screen, participants are asked to respond, by pressing the correct paddle as fast as possible (Figure 1). Three subtests (I, II and III) are included, representing increasing cognitive challenges. The time-frame for the entire session is 30–40 minutes.

Subtest I: A simple stimuli-response task is presented and the participant is expected to press the right paddle whenever an arrow appears on the screen.

Subtest II: An arrow pointing to the left or right on the left or right side of the screen is presented and the participant responds by pressing the corresponding paddle depending on the arrow direction.

Subtest III: Two arrows are presented at the same time, one on each side of the screen. When the arrows are pointing in opposite directions, the participant should not respond (inhibited response); when both arrows point in the same direction, the participant should press the paddle representing the direction of the two arrows.

Mean reaction time (in milliseconds), missed and wrong responses are reported. A wobbling algorithm is calculated to illustrate how well the participant manage to keep the car on the right side of the road and within road markings. The algorithm is based on the number of the intersection of the road markings and how long.

Useful Field of View

UFOV 6.1.4 is a computer-based screening assessment for fitness to drive,^14,15 consisting of three subtests assessing the precision of visual processing under increasingly complex tasks. Briefly presented targets has to be detected, identified and localized by the participant by responding on a touch screen or by using a mouse. UFOV adjusts the duration of stimuli presentation depending on the participant’s responses.¹⁴ The process of tracking the perceptual threshold goes on until a stable estimate of 75% correct is calculated.

Subtest 1: Processing speed is required for this subtest. The task is to detect, identify, and localize briefly presented vehicles centrally on the screen.

Subtest 2: Divided attention is required for this subtest. The task is to simultaneously identify a centrally located vehicle as well as a peripherally located vehicle.

Subtest 3: Selective attention is required for this subtest. The task is identical to subtest 2 except that the vehicle in the periphery is embedded in 47 triangles.

Results are reported in milliseconds. A total score as well as scores for every subtests are provided. A high validity (Cohen’s d = 0.945) when UFOV was compared with driving measures was shown in a meta-analysis study.²³ In addition, UFOV has been shown to have a high level of prediction of crash risk.^24–26

Trail Making Test

The TMT version used in this study is a paper-based test consisting of two subtests.¹⁶ Results are presented as the time taken to complete each subtest.

Subtest 1: TMT-A The participant is asked to draw lines sequentially connecting 25 numbers as quickly as possible without lifting the pencil.

Subtest 2: TMT-B The participant is asked to draw lines alternating between 13 numbers and 12 letters in two sequences.²⁷

TMT is considered to assess visual searching, scanning, processing speed, mental flexibility, and executive functions.¹⁶ TMT-A is considered to require visuoperceptual function and TMT-B primarily working memory and secondarily task-switching ability.²⁸

Nordic Stroke Driver Screening Assessment

The NorSDSA¹⁷ is based on the British Stroke Driver Screening Assessment (SDSA).²⁹ The Nordic version is adapted for right-hand traffic, and some road signs were replaced.

The NorSDSA is a paper-based test including four subtests:

Subtest 1: This dot-cancellation test includes dots lined up in groups of three, four, or five and the participant is asked to mark all groups with four dots. Results are expressed as time taken in seconds, missed groups of four, and crossed out groups of three or five, so-called false results.

Subtest 2: This direction test consists of a 4×4 square matrix and 16 stimulus cards showing a lorry (represented by a large arrow) and a car (represented by a small arrow) traveling in different directions. Directional arrows are placed along the side and top of the matrix. The participant then places the stimulus cards in the square that corresponds correctly to one small and one large arrow. Maximum time is 5 minutes and results are expressed in points, one point given for each correctly placed vehicle.

Subtest 3: This compass test includes arrows representing eight compass directions and two cars leaving or driving onto a roundabout on two of eight different roads. The participant is asked to correctly place 16 out of 28 cards. Maximum time is 5 minutes and results are expressed in points, one point given for each correctly placed card.

Subtest 4: This road sign recognition test includes 12 cards depicting different traffic situations. The participant is asked to correctly place 12 out of 20 cards with traffic signs, on top of the corresponding traffic situation. One point is given for each correctly placed card.

According to the NorSDSA manual, dot cancellation requires visual scanning and visual perception, sustained and selective attention as well as speed. The direction test requires visuospatial functions, and the compass test scanning and processing of visual material. Road sign recognition requires visual scanning and processing as well as verbal and visuospatial memory. NorSDSA provides a weighted total score based on results for dot cancellation, compass and road sign recognition subtests. Higher score represent better result.¹⁷

Some studies have shown that SDSA as well as NorSDSA correctly classifies a sufficient number of unsuitable drivers,³⁰ but have been reported inaccurate for neurological conditions other than stroke.^31,32

On-Road Observation: Patient Group

An on-road observation with specially educated occupational therapists was performed after the cognitive tests, when additional information was needed and not in cases when decision considering fitness for driving was obvious without this observation. The patients undergoing on-road assessment (67%) drove on a standardized road trip, lasting for about 60 minutes. Different traffic situations and demands were included in the on-road observation. Risk situations and misbehavior in relation to actions taken were documented.

Final Decision

The final decision on whether a patient should be considered fit or unfit to continue car-driving was always made in a team consensus discussion, among a specialist physician, an occupational therapist and a psychologist. Cognitive test results as well as results from an on-road assessment (when performed) were the basis for this discussion. Norm data adjusted for a Nordic population were not available for any of the included tests at the time for data collection or the final decision. The on-road observation was performed and included in the final decision whenever additional or confirmatory information was needed. The simulator test results were not included in the final decision.

Statistics

The Statistical Package for the Social Sciences (SPSS) version 23.0 was used for statistical analyses. Descriptive statistics were used for the sociodemographic data. Parametric and non-parametric statistical methods were used to analyze data based on the results from normal distribution analyses and type of variables analyzed. The accepted level of significance was p < 0.05. Based on a number of studies describing a wide range of variability in test results, a cut-off for using norm values for comparison was set at ±2 standard deviations (SD) from the mean.^16,22,33

Results

A comparison of demographic data for the norm versus the patient group showed significant differences for gender, age, level of education, and estimated driving per year (Table 1). A within-group analysis taking gender and educational level into account showed no differences in the test results (norm group, p > 0.05; patient group, p > 0.05), thus this was not taken into further account.

Table 1 Demographic Data for the Participants

Test results for the patient group based on age and results for the final team assessment (fit/unfit to drive) were compared showing significant differences between groups for all tests except TMT-A for the oldest group and for some of the results for CyberSiM wobbling, misses, and faults (Table 2).

Table 2 Comparison Between Scores for Patients Who are Fit to Drive versus Unfit to Drive

A comparison between age-related norm values versus patients who were considered unfit to drive showed significant differences for all test results and for both age groups (p < 0.05).

Results for the norm groups versus patients fit to drive showed no differences in the results for CyberSiM II and III for the oldest age group; other test results differed significantly. Test results on the age groups for norms, fit to drive, and unfit to drive are illustrated with medians, 1st and 3rd quartiles, minimums, and maximums in Figures 2–5.

Figure 2 UFOV total score: norm data, fit to drive, unfit to drive.

Figure 3 (A and B) TMT-A and TMT-B: norm data, fit to drive, unfit to drive.

Figure 4 NorSDSA total score: norm data, fit to drive, unfit to drive.

Figure 5 (A–C) CyberSiM I–III: norm data, fit to drive, unfit to drive. * =outlier.

Cut-off values based on norm values (mean ± 2SD) were applied to study the extent, to which patients who were evaluated as fit to drive were within set limits, expressed as proportion of patients. Results varied between the tests (Table 3). For the oldest age group, UFOV total and CyberSiM I–III had the highest conformity with the norms. For patients <59 years of age, CyberSiM I–III had the highest conformity. The highest proportion of results within the set cut-off limits for both age groups was identified for the CyberSiM reaction time (75–95%). For CyberSiM wobbling and misses, conformity with norms was 83–100% (Table 3). TMT-A had the lowest proportion of patients fit to drive within the defined interval (±2SD): 46% and 55% respectively for the different age groups.

Table 3 The Proportion of Patients Considered Fit to Drive and Within the Cut-Off Limits Based on the Mean ± 2SD (Norm)

Discussion

Several studies report the value of using standardized cognitive tests in clinical practice on traffic medicine issues.³⁴ The aim for this study was to analyze the usability of cut-off values, based on recently developed norm values for a driving simulator device (CyberSiM), as well as cognitive tests, for patients already considered fit-to-drive after a standardized traffic medical investigation. The comparison between norms (mean ± 2SD) and patients considered fit to drive showed that 75–95% of patients performed within the stipulated limits for all three subtests of CyberSiM (reaction time) and for both age groups.

The proportion of patients who were within the cut-off limit for UFOV total was 82% for the oldest patient group and 71% for the younger group. For TMT-B, the proportion was about 70%. TMT-A and NorSDSA were less compliant compared with norm values (between 46% and 65%). NorSDSA has been shown to be inaccurate for neurologic conditions other than stroke, which might explain the low compliance for the study group, which included a diverse range of diagnoses.^31,32

UFOV total score represents a summary of the results from the three different subtests, which together aim to address three different types of attention (processing speed, divided attention and selective attention). Another form of attention described as being of utmost importance in driving is sustained attention.¹³ Sustained attention is not addressed in any of the cognitive tests included here. The relatively short period of time it takes to perform, for example, the UFOV assessment, and the fact that the test person is well aware of the different stimuli and the response needed, and is thus well prepared and focused, must be taken into consideration. CyberSiM takes longer to perform, and thus demands sustained attention. In addition, it includes different “driving scenarios” and unexpected events as well as inhibited responses, all of which might be more in concordance with demands on attention in a real traffic context. An interesting finding was the lack of differences in the results based on age group for CyberSiM reaction time (Figures 2–5). This indicates that CyberSiM is not as sensitive to age as the other tests. The reason for this lack of difference could be related to the fact that sustained attention is less sensitive to age than other forms of attention³⁵ and/or that the activity is equally familiar to everyone regardless of age.

Much of what we do when driving is repetitive and hence predictable, and thus might be expected to be an automated skill when it comes to the actual driving task.¹³ Thus the actual driving skill, is seldom the problem after a brain injury/disease unless a muscular weakness or paralysis is involved.

Referrals on cognitive conditions for continued driving after a brain injury/or disease often come from physicians to occupational therapists or psychologists. Because psychologists are significantly fewer and often have other responsibilities within health care, most driving assessments are performed by occupational therapists. Immediate response to a stimuli repeated for a short period of time can be captured using traditional tests such as UFOV. However, driving puts demands on sustained attention; driving in real life means that unpredictable events might arise at any time and thus the driver must make decisions on action or no action. To capture this, we need tests that last over time and with unforeseen elements and elements that require both action and no action.

UFOV could be used as a basic attention test, but the results should be handled with care and not as a stand-alone test for making conclusions on driving. A simulator assessment such as the CyberSiM could contribute with information about attention, and especially sustained attention, in a more complex context. In addition, a simulator device might contribute with observed aspects of behavior, multitasking, and active multiple responses in terms of reaction and inhibition for different stimuli.^6,7

Although age-related norms might contribute to the evaluation of continued driving after a brain injury/disease, one should pay attention to the fact that age in itself is not the best indicator of driving. Irrespective of age, there are differences in, for example functional ability and driving experience that affect driving competence.³⁴

Strengths and Limitations

All data were collected by well-informed and experienced personnel, which should be seen as a strength together with the standardized procedures used. The simulator device was not included in the final decisions for the patient group, and no norm values were available at the time of the study.

A limitation was the small number of participants and a wide spread in the results, especially in the oldest groups. Using ±2SD as the cut-off limit could be seen as an arbitrary decision, however a lot of calculated alternatives were performed and rejected, finally resulting in this permissive approach, which seems to capture the team decisions in a reasonable manner. The generalizability is limited due to the relatively small patient group and the large spread in results. More studies are needed to verify the results of the study.

Conclusions

The use of different standardized assessment tools in evaluating continued car-driving in patients after brain injury/disease is important and recommended. The availability of norm values when interpreting the results of different assessments could guide decision-making, but should be handled with care. For CyberSiM and UFOV, a comparison with norm values (cut-off limit of ±2SD) showed that a large proportion of patients considered fit to drive were within those limits. This could indicate the value of including norms in the discussion and evaluation.