Exploring a Prediction Model Based on Patient Record Data in Carotid Stenosis Risk Assessment

Introduction

Atherosclerotic diseases represent a large group of conditions, where demographic changes impact both development and progression. In the same disease group, diagnostics and treatment options are developing quickly, and keeping track of all aspects can be challenging. Improved prevention strategies could slow down progression of the chronic disease, and prevent or postpone life-threatening acute events such as heart attack, stroke or critical limb ischemia. Still, evidence-based tailoring of therapy is necessary for safe and cost-effective prophylaxis, and minimised impact of adverse treatment effects.

Surgical or endovascular treatment of carotid stenosis in stroke prevention is well known for its inevitable risk of periprocedural strokes. Although the stroke risk in association to treatment is reported to be slightly lower today than 20 years ago,¹ the risk of a procedure-related stroke is still considerable for both endovascular procedures and open endarterectomy.

Risk prediction tools could provide an aid in determining who are likely to benefit the most, and thereby tailor treatment recommendations for each individual patient. Several risk calculators have been developed,^2–4 but none has yet succeeded in significantly challenging the established guidelines for carotid disease.⁵ In the past, such risk prediction tools have generally been based on the same data material as current guidelines, and the additional value in decision making seems sparse. Risk prediction tools based on randomized controlled trials may be representative at the time of analysis, but necessitate repeated studies to account for demographic changes and improvements in treatment options.

Artificial intelligence (AI) is technology with a broad range of applications in medicine.⁶ The virtual AI branch includes deep learning, which enables data extraction from free text. This could enable real-time risk prediction by allowing continuous updates with current data, thereby facilitating continuous adjustment to evolving demographic characteristics. Risk prediction models based on free text patient record data still lack verification for this purpose. In this study we aim to evaluate the performance of free text patient record data in risk prediction.

We present a new method for evaluating the risk of stroke in patients with carotid stenosis, based on manually extracted patient record data. An extended regression model is applied, to include repeat event analysis and time dependency of key variables. The intention is to explore this data source for risk prediction by applying a suitable statistical model. In this article we will describe the background for the model choice, and the method applied in its development. To our knowledge, the suggested extended regression model has not previously been applied on carotid stenosis data.

The study complies with the Declaration of Helsinki, and has been assessed and approved by the Regional Committee for Medical Research Ethics Northern Norway (REC).

Methods

Data Material

This analysis is performed in order to evaluate the model potential, where a retrospective approach to risk prediction is carried out. The test data set was manually established by the use of “all comer” patient record data from the University Hospital of North Norway. 1428 patients registered with the ICD-10 diagnosis “I65.2 Occlusion and stenosis of carotid artery” since the introduction of an electronic patient record system in 2003, were invited. Both symptomatic and asymptomatic patients were included. Most of the asymptomatic patients had been referred to the out-patient clinic from the Tromsø Study. This is a large population study that has been taking place since 1974, with seven separate studies performed to date. The study consists of questionnaires for all participants, and various examination studies performed on a smaller selection. In the studies Tromsø 4 (1994–95), Tromsø 5 (2001) and Tromsø 6 (2007–2008), a selection of participants were examined by ultrasound for carotid stenosis by an ultrasound technician. Those who had carotid atherosclerosis, were referred to the neurological out-patient clinic for a more detailed ultrasound evaluation. Accordingly, information in relation to this consultation has been documented in the hospital patient record, including ultrasound findings, ongoing medical therapy, smoking history, relevant past medical history, and usually measurement of vital signs. Advice was given in relation to treatment and follow up, and most of these asymptomatic patients had standard medical therapy prescribed. For most patients with a stenosis of more than 50%, follow-up examination to evaluate disease progression was performed.

Inclusion by Consent

All participants referred from the Tromsø Study had previously given a broad consent to the use of Tromsø Study data. The Regional Ethical Committee (REC) advised inclusion of these patients without new consent. For all deceased individuals, REC also advised exemption from consent to access patient record data. Written consent was obtained to permit access to patient record data for all other participants.

Parameters of Consideration

The results from previous carotid studies were used to target patient properties associated with disease progression and/or ischemic events.^2,7–13 We registered previous medical or surgical treatment at the start of follow up, and separately registered therapy initiated at the time of diagnosis. Antihypertensive agents, antiplatelets and other anticoagulation therapy, and lipid lowering drugs were registered by type but not dose. If carotid endarterectomy (CEA) or stenting (CAS) was performed at any time during follow-up, preceding events, timing of treatment, method and complications, including postoperative strokes, were registered.

Cerebral Ischemic Events

The primary endpoint was defined as ischemic strokes. All new ischemic cerebral events were registered, until death or censoring at the end of follow up. Transient ischemic attacks (TIAs) were defined as ischemic events with full recovery within 24 hours, where no haemorrhagic or ischemic lesion was detected on CT or MRI. TIAs were treated in the analysis as risk predictors rather than endpoints, as no permanent injury to brain tissue will follow, and because these events are known to frequently precede a stroke. TIAs were the presentation of symptomatic carotid disease in 156 patients, and by that triggered the indication for surgical treatment in 65 of these.

The remaining symptomatic patients had already suffered an ischemic stroke at the time of presentation. For these patients, stroke severity was registered by use of the Modified Rankin Scale (mRS).

Endpoint Classification

Less than 20% of all patients with ischemic strokes are found to have carotid stenosis, and the condition is seen as the cause of the event in only about half of these.¹⁴ As the stroke risk is shown to depend on the degree of stenosis,⁸ it is assumed to increase in the presence of traditional risk factors for atherosclerotic progression if the observation time is long enough. A previous study with a median of 13 years of follow-up still fails to prove this.¹² Stroke by other causes, such as cardiac thromboembolism, is perceived as noise in regards to carotid stenosis risk. Thus, endpoint subclassification is desirable, to pinpoint the risk of strokes caused by the carotid stenosis. A clear cause of cerebral ischemia can be difficult to define, and we aimed for a systematic approach where the subclassification was reproducible and standardised. We have chosen the ASCOD score¹⁵ for this purpose.

Ischemic strokes were registered when a new lesion was detected by CT or MRI, or when symptoms persisted for more than 24 hours and haemorrhage was ruled out. We applied the ASCOD phenotyping to subtype the events.¹⁵ The ASCOD phenotype A1-1, when S, C, O and D are 0 or 3, strongly indicates the carotid stenosis as causative of the event. In the case of A1-2 or −4, or A2-1, where SCOD grade is 0, the causation is also likely. The phenotype A0 makes other causes of the ischemic event more likely. This also applies to A2 and A3 if S, C, O or D grade is 1. When no ASCOD 1 grade is defined, or two or more grade 1 coexist within the phenotypes, the cause is uncertain. In these cases, the most likely cause is evaluated individually for each patient.

The potential source of error due to a degree of objective endpoint evaluation, was explored by performing separate analyses for the endpoints “carotid stroke”, “any ischemic stroke”, and “any ischemic stroke or death”.

Recurrent Events in Survival Analysis

In evaluation of the effect of prophylactic therapy, a long observation time is desirable, as ideally, we would like to reveal the lifetime risk. During a decade or more of follow up, some individuals will experience more than one stroke. These patients could potentially have properties making them particularly vulnerable to new events. Therefore, all new events were registered and included in the model. Multiple endpoint analysis is rarely performed, and existing datasets from previous carotid stenosis studies generally only include time to the first event or exclusion.

Recurrent event models in survival analysis offer statistical methods for evaluating the risk of an event that can occur more than once. Different models exist, and the choice of model depends on whether or not the events are seen as different processes, and on the dependency of risk on the time from entry into the study to the first and recurrent events. In the analysis of stroke, each event is seen as the same process. The time to the first event is assumed to impact the time to a potential recurrent event. The Prentice, Williams, and Peterson Total Time Model (PWP-TT)¹⁶ accounts for this order of events, and is our model of choice.

Time Dependent Variables

Some key variables are known to have a strong impact on the risk of new events, namely the degree of carotid stenosis and new TIAs. We have chosen to include these variables with time dependency to explore the impact of disease progression over time on stroke risk. We do not know of previous datasets with time dependent registrations of carotid stenosis progression and all new TIAs. Treatment of the carotid stenosis is given time dependency by means of a reduction of the stenosis to zero on the treated side, although in some cases this is an assumption, as not all patients received a postoperative control ultrasound.

By the use of an extended Cox model, the effect of a change in a variable after a given time interval can be incorporated in the risk prediction model. It is possible to include both time dependent and time independent variables in the same analysis. The time-independent variables will not change value during the observation time, and is obviously applied for gender. Several of our registered variables in relation to general cardiovascular risk, such as smoking, blood pressure and antihypertensive treatment, could be seen as time dependent. However, for practical reasons, only the predictors expected to have a strong impact on stroke risk were included with time dependency in the test dataset.

The Complexity of Sidedness

Ischemic events can occur in two cerebral hemispheres, each supplied by a separate carotid artery. Intracerebral communications through the Circle of Willis serve as security across the hemispheres, to maintain perfusion if a precerebral artery is compromised. The risk of an embolic stroke affecting one side depends on the degree of carotid stenosis on the ipsilateral side.^8,17 Still, stroke by hypoperfusion can occur in both hemispheres if the total perfusion to watershed areas of the brain is insufficient due to severe stenosis or occlusion of the precerebral arteries. Each side is therefore considered to carry a separate stroke risk, where the atherosclerotic status of both carotid arteries may contribute to the risk on each side.

If an ischemic event occurs, surgical or endovascular treatment is usually performed on the ipsilateral side. If there is ipsilateral carotid occlusion, treatment might be performed on the contralateral side to prevent hypoperfusion, although there is controversy about this indication for therapy.¹⁸ In our data material, the time and side of treatment is registered, and the degree of stenosis on this side drops to zero after successful recanalization.

As previously described, TIAs are included in the model as time dependent variables. The TIA variable is also subdivided into right and left sides, or unknown if the symptoms are not side specific. These variables are allowed to “count” if several TIAs occur, which means that the excess risk assumed to be associated with having more than one TIA on the same side can be estimated.¹⁹

Software for Data Analysis

Redcap was used for data registration, and data was then exported to SPSS. SPSS was used for simple data analysis and some graphics. Data preparation for a matrix set-up with time intervals to new events and/or changes in time dependent variables was also performed in SPSS, and the matrix dataset was then exported to the SAS software 9.4, SAS Institute Inc., Cary, NC, USA for analysis with time dependent variables and time dependency of endpoints.

Results

1428 patients were registered with the ICD10 diagnosis I65.2 between 2003 and 2019. Of these, 494 were dead at the time of inclusion, and these were included without consent as advocated by REC. Consent was received from 440 patients, either through the Tromsø Study, or by returning a signed consent form. For the individuals where permission to access patient records was granted, the ICD10 diagnosis was individually verified through patient record data. Patients found to have an incorrect diagnosis of carotid stenosis were excluded. A total of 863 patients, 392 symptomatic and 471 asymptomatic, were included in the analysis.

The baseline characteristics were comparable to those seen in previous studies, as shown in Table 1.

Table 1 Baseline characteristics

Distribution of Carotid Stenosis

At the time of presentation, symptomatic patients had a distribution of carotid disease heavily skewed towards high degree ipsilateral stenosis. The distribution of ipsilateral and contralateral carotid stenosis is shown in Figures 1 and 2.

Figure 1 Distribution of ipsilateral carotid stenosis severity in patients included after a cerebral ischemic event.

Figure 2 Distribution of contralateral carotid stenosis severity in patients included after a cerebral ischemic event.

For the asymptomatic patients, the distribution of right and left sided carotid stenosis was quite similar for the two sides, as shown in Figures 3 and 4.

Figure 3 Distribution of right sided carotid stenosis severity in asymptomatic patients.

Figure 4 Distribution of left sided carotid stenosis severity in asymptomatic patients.

Effect of Surgical Treatment

We would like to explore the statistical model with time dependent variables and multiple endpoints by comparing the results to those of a traditional Cox regression analysis of the same data set. The standard Cox regression analysis was performed with survival time to the first ipsilateral stroke for symptomatic patients, and time to first event for asymptomatic patients. Among symptomatic patients, a total of 65 new ipsilateral strokes occurred, of which 49 were attributed to carotid stenosis. Including recurrent events, a total of 76 events occurred, of which 61 were ipsilateral to the side of the initial symptoms. Among asymptomatic patients, a total of 87 strokes occurred, of which 56 were attributed to the stenotic carotid artery. Including recurrent events, a total of 74 attributable events occurred.

As shown in Table 2, very little impact on the risk of stroke is seen by individual variables, but a strong risk improvement occurred in patients who had carotid surgery performed after the index event. This is illustrated for the symptomatic group in Figure 5.

Table 2 Stroke risk by traditional Cox regression analysis

Figure 5 Survival time at mean of covariates in days to the first ipsilateral stroke in symptomatic patients for those who had surgery performed after the index event and those who did not, when the stroke event is suggested most likely to be caused by the carotid stenosis after ASCOD classification.

A similar result is seen if the endpoint is defined as any ipsilateral stroke (ie ischemic stroke from any cause), but with a weaker protective effect of the carotid surgery, as illustrated in Figure 6.

Figure 6 Survival time at mean of covariates in days to the first ischemic ipsilateral stroke in symptomatic patients for those who had surgery performed after the index event and those who did not, when the cause of the stroke is not specified.

If the endpoint is defined as any stroke (any side, any cause) or death, the protective effect of surgery after the index stroke is still evident, but less pronounced, as shown in Figure 7.

Figure 7 Survival time at mean of covariates in days to stroke or death from any reason in symptomatic patients for those who had surgery performed after the index event and those who did not, when neither the side or cause of the stroke is specified.

When patient record data are analysed, the estimated protective effect of CEA is suspected to be higher than in analyses of RCT data, as patients selected for surgery are believed to have fewer comorbidities, better life expectancy, and lower stroke severity. To illustrate this, we explore the mRS after the index event for those with an index stroke (ie Not TIA), as shown in Figure 8.

Figure 8 Analysis of stroke severity by mRS for symptomatic patients who had an ischemic stroke as the index event, comparing those who were selected for surgery to those who were not.

This shows that a larger percentage of those who did not undergo carotid surgery had a major index stroke with mRS >2.

Surgical Risk

In the 392 patients who were symptomatic at presentation, 194 received surgical treatment for the carotid stenosis. Of these, 10 suffered a postoperative ipsilateral stroke (4,9%).

In the 471 patients who were asymptomatic at the time of diagnosis of the carotid stenosis, 40 underwent surgery for asymptomatic disease, of which four had bilateral surgery. In 44 procedures, two postoperative strokes occurred, one ipsilateral and one contralateral (4,5%, or 2,3% if only ipsilateral strokes are considered). The patient with a contralateral stroke had 80% ICA stenosis and vertebral artery occlusion on the non-operated side.

60 patients that were asymptomatic at the time of diagnosis of the carotid disease, received treatment for symptomatic disease which occurred at a later stage. Of these, 56 had one procedure, three had two procedures and one had three procedures, of which the last was a CAS. In a total of 65 procedures, seven strokes occurred (10,8%). This incidence is higher than the expected stroke risk in association to the procedure. The CAS for a second procedure on the same side resulted in a stroke, but even with this procedure excluded, the stroke incidence after surgery is 9,4% which is surprisingly high. The reason for this is not known, although the small number of procedures could account for a random outcome without true statistical significance. In a pooled analysis of both patients that were symptomatic at the time of registration and patients who became symptomatic during follow-up, the total number of procedures for symptomatic disease was 258 (excluding the CAS redo). This resulted in a total of 16 postoperative strokes (6,2%) occurring after treatment of symptomatic disease.

The described patient journal data is processed in order to allow analysis of multiple endpoints and time dependent variables, and results are shown in Table 3. When compared with the traditional Cox analysis in Table 2, the introduction of the extended model seems to impact individual risk predictors. Interestingly, the carotid stenosis on the side of the event is not significantly associated with a recurrent event for symptomatic patients in a traditional Cox analysis, but shows significant association when the extended model is applied, with a Hazard ratio (HR) of 2.25 for ipsilateral stenosis of 70–99%. The same applies to the asymptomatic patients, where a high degree right sided carotid stenosis is significantly associated with a right sided event, HR 4.98, and a high degree left sided carotid stenosis is significantly associated with left sided events, HR 5.23. This is not clearly significant in the traditional Cox analysis, where the according HRs are 2.40 and 2.08, respectively.

Table 3 Survival analysis by an Extended Cox Regression Model

This could imply an additional value in risk prediction, but further exploration is considered necessary to evaluate the results. The results have been supported by model tests, which are described in a separate article.

Discussion

In the establishment of new risk prediction tools, a large data source is desirable in order to develop a reliable and stable model. The lower the event frequency and the higher the number of variables, the larger the need for power. Our moderate data material is only intended for model exploration, and is not yet applicable in risk prediction.

With the standard use of digitalised patient record systems, data derived from this gold mine of information could play an important role in future prognostic models. Extraction of data from free text information by the use of machine learning could make anonymised data collection possible. Regardless, even for anonymised data, ethical considerations about ownership, data protection and confidentiality represent a controversy for the use of this information in research. The European Data Protection Regulation (GDPR) of 2018 serves to protect the data subject, and by that intensifies the responsibility of the processors. By article 11 of GDPR, processing which does not require identification can still be performed as long as adherence to these regulations is fulfilled, and it seems likely that anonymised data collection could become possible in future research. If so, evaluating whether patient record data are suitable for risk prediction is considered a first step. Evaluating manually extracted data allows for a controlled and verified analysis of this material prior to potential digital data extraction.

Selection bias is a potential source of error in our current data material for individuals included by consent. The Tromso study has a randomised approach to invitation, and randomly assigns participants to the “special studies”, where the carotid ultrasound was performed. There will still be a selection bias in regards to volunteering participants. The bias could be reduced by all-comer analysis, which could be performed if the proposed anonymized free text data extraction is made possible.

In free text patient records, the lack of uniformity of information and tests represent a challenge. Spot parameter measurements, operator dependency, language variability and subjective evaluation could all generate a potential “free text error”. If the prognostic model is to be applied in a clinical setting, the patient record derived data could still be postulated to resemble that of real-life observations in a representative way. Missing data will naturally occur in a real-life dataset, and we have intentionally allowed this to be omitted.

The various causes of cerebral ischemia represent an issue of uncertainty when working with stroke risk. Numerous carotid stenosis studies still present their results without commenting on their endpoint management. The easiest approach would be to include all ischemic strokes as endpoints, and assume an even distribution of strokes from other causes. This seems counterintuitive in cases where the stroke is obviously caused by i.e. cardiac embolization, such as in the presence of bilateral shower embolies. Knowing from previous studies that a stroke is caused by the carotid stenosis in only about half of those who have the condition,¹⁴ at the very least an opinion on stroke subclassification is considered appropriate. Our emphasis is a best effort in standardising stroke subclassification, to predict the actual risk of thromboembolic strokes from carotid lesions, but to complement this by additionally all cause ischemic stroke analyses. Even after ASCOD classification, there is some uncertainty in relation to the causality of the carotid disease when an ischemic stroke occurs. We consider this a challenge which is not yet overcome in stroke research.

The decision to perform CEA is expected to be partly subjective, depending on the surgeon’s assessment of patient frailty and the neurologist’s evaluation of the cause of the preceding event. This is a key rationale for applying a more suitable statistical model that incorporates recurrent event analysis, as recurrent events while awaiting treatment and in relation to the procedure can be captured, and the long-term effects of successful treatment can be assessed.

Incorporating recurrent event analysis and time dependent variables in the model is considered to be a valuable supplement to traditional survival analyses in cardiovascular risk prediction. In evaluation of any prophylactic treatment, sufficient observation time to estimate the long term effect of the prevention strategy is considered desirable. For any cardiovascular disease, repeat events of a specific condition in the same individual are relatively frequent, and expected to be encountered particularly in data material where the observation time is long. Still, the application of recurrent event analysis in cardiovascular research is quite uncommon. The same applies for time dependent variables, which are also expected to impact risk in relation to recurrent events, i.e. increase in stroke risk with progressive carotid stenosis. We have not found any other studies describing the combination of time dependent variables and recurrent event analysis in the same model. Recurrent event analysis is known to carry a risk of type I error, where a false positive result occurs due to overestimation of risk. It is not known if this can be corrected by the introduction of time dependent variables. Thorough model testing is considered necessary to evaluate model performance, and has been performed in a separate study.

We have selected only a few risk predictors for time dependent analysis for practical reasons, but consider it an interesting next step to include other parameters, such as smoking, biochemical markers, alterations in drug therapy, and plaque property alterations. This data extraction would yield large and complex datasets that could be managed using machine learning methods.

Conclusion

We consider the free text data material as representative when compared to baseline properties from previous carotid studies. The dataset appears sufficient for a study of the performance of a novel risk prediction model, although a larger sample size would be desirable given the relatively low event frequency. The introduction of multiple endpoint analysis and time dependent variables seems to impact individual risk predictors, implying a potential value in risk prediction. Model tests are still necessary to confirm this. This has been performed in a separately published study.