Article Text
Abstract
Background Carotid endarterectomy (CEA) results in fewer perioperative strokes, but more myocardial infarctions (MI) than carotid artery stenting (CAS). We explored a combined modelling approach that stratifies patients by baseline stroke and MI.
Methods Baseline registry-based risk models for perioperative stroke and MI were identified via literature search. We then selected treatment risk models in the Carotid Revascularisation Stenting versus Endarterectomy (CREST) trial by serially adding covariates (baseline risk, treatment (CEA vs CAS), treatment-risk interaction and age-treatment interaction terms). Treatment risk models were externally validated using data from the Society for Vascular Surgery (SVS) Vascular Quality Initiative (VQI) CEA and carotid stenting registries and treatment models were recalibrated to the SVS-VQI population. Predicted net benefit was estimated by summing the predicted stroke and MI risk differences with CEA versus CAS.
Results Perioperative treatment models had moderate predictiveness (c-statistic 0.69 for stroke and 0.68 for MI) and reasonable calibration across the risk spectrum for both stroke and MI within CREST. On external validation in SVS-VQI, predictiveness was substantially reduced (c-statistic 0.61 for stroke and 0.54 for MI) and models substantially overpredicted risk.
Most patients (86.7%) were predicted to have net benefit from CEA in CREST (97.0% of symptomatic patients vs 75% of asymptomatic patients).
Discussion A combined modelling approach that separates risk elements has potential to inform optimal treatment. However, our current approach is not ready for clinical application. These data support guidelines that suggest that CEA should be the preferred revascularisation modality in most patients with symptomatic carotid stenosis.
- technique
- stroke
Data availability statement
Data are available in a public, open access repository. Data may be obtained from a third party and are not publicly available. CREST data are available from NINDS. SQS-VQI data are available via application to SVS-VQI.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Two primary revascularisation strategies exist for patients with carotid artery stenosis—carotid endarterectomy (CEA) and carotid artery stenting (CAS). Most patients with CAS could be treated with either procedure and, thus, numerous trials have compared these techniques, generally slightly favouring CEA.1 The largest single trial, which also used the most rigorous proceduralist credentialing strategy, was The Carotid Revascularisation Endarterectomy versus Stenting Trial (CREST).2 3 It found that CAS was non-inferior to CEA on a composite endpoint including stroke and periprocedural myocardial infarction (MI) and that younger patients may benefit more from CAS than CEA. Reflecting this evidence, guidelines have generally suggested that while there is more evidence for CEA than CAS, that both procedures can be considered as treatment options for individual patients.4
Yet, the question of how to select the best approach for an individual patient is unclear. In part, based on the results of the Stenting and Angioplasty with Protection in Patients at High Risk for Endarterectom (SAPPHIRE) trial,5 CAS is generally recommended for high risk patients, although the precise definition of high risk is uncertain.4 Similarly, the subgroup effect in CREST towards greater benefit with CAS in younger patients could be used to inform individual decisions.2 In addition to these strategies, the observation that CAS is generally associated with a higher perioperative risk of stroke while CEA is generally associated with a higher risk of periprocedural MI1 suggests a possible approach to selecting the optimal treatment for individual patients. To the extent that risk for stroke and MI risk can be accurately predicted and these risks can be separated, treating patients at relatively high stroke risk with CEA and patients with relatively high MI risk with CAS may be a feasible strategy. This strategy is particularly promising because prior work suggests that the risk of stroke in patients with symptomatic carotid stenosis can be effectively stratified with straight-forward clinical models.6–9
In this study, we describe the development and external validation of a combined modelling approach to predict the optimal revascularisation technique in individuals undergoing carotid revascularisation using data from the CREST trial. The goal of this modelling approach is to inform individual-level risk with CEA and CAS by accounting for all known factors that predict outcomes with either treatment.
Methods
We used a 3-step approach to develop models to estimate and compare individual-level untreated and treated risk of MI and stroke among patients who are candidates for CEA versus CAS (figure 1). First, we identified baseline risk models for perioperative stroke and MI based on clinical factors measured prior to undergoing a procedure. We focused on perioperative stroke risk because most strokes occurred in the perioperative period and the stroke risk after the perioperative period is similar in CEA and CAS in CREST. (online supplemental figure 1) Second, we developed treatment models to predict stroke/MI using individual level data from the CREST trial. Third, we externally validated these models in the Society for Vascular Surgery (SVS) Vascular Quality Initiative (VQI) CEA and carotid stenting registries.
Supplemental material
Datasets
The CREST randomised 2502 patients with carotid stenosis (47% asymptomatic) to receive either CAS or CEA. The primary inclusion criterion was greater than 50% carotid stenosis by angiography or 70% stenosis on non-invasive imaging. A distinguishing feature of CREST, relative to other CEA versus CAS trials, was the requirement that both surgeons and interventionalists with must be qualified via both volume and outcome criteria prior to performing CEA or CAS in the trial. This feature may have contributed to the relatively low event rates in both arms compared with other CEA versus CAS trials.10 The primary composite outcome in CREST was a combination of periprocedural stroke, death or MI or ipsilateral stroke within the next 4 years. Fifty per cent of patients were allocated to each treatment arm. Mortality was the same in both arms (0.2%), while stroke was more common in CAS (4.1%) vs CEA (2.3%) and MI was more common in CEA (2.3%) vs CAS (1.1%).2
The SVS VQI registries contain detailed clinical self-reported data on hundreds of thousands of patients undergoing vascular surgery in more than 500 centres in the USA and Canada. The registry records preprocedure baseline clinical status for the patient, details of the indication for the procedure, procedural details, complications and outcomes. For patients at participating institutions, all individual receiving a vascular procedure at the institution are included in the registries.11 12 Short-term mortality and MI measures have been shown to correlate well between SVS VQI and other registries,13 but less is known about the validity of stroke outcomes reported in the registries.
Identifying baseline risk models for MI and stroke
Numerous models have been developed to predict the risk of perioperative MI and stroke in patients undergoing CEA versus CAS. For perioperative MI, we used the Revised Cardiac Risk Index14 because the data necessary to stratify patients with this widely used model were readily available in CREST and prior work has found that this model has reasonable discrimination and calibration across a wide variety of patient populations.15 For perioperative stroke, no single model has been widely used and a number of risk models have been developed.16–20 Most of these models use a series of binary risk predictors (eg, baseline demographics, vascular risk factors, comorbidities, procedural circumstances) to stratify baseline risk,16–19 while one model also included preprocedure blood pressure.20 To select a model for application in this analysis, we mapped the variables included in these models to the CREST trial and SVS registries as best as possible, (necessitating that some risk factors were omitted) and measured their discrimination, accounting for omissions. Generally, discrimination was modest (c-statistics ranging from 0.60 to 0.64) and similar across models. However, modest discrimination has been shown to be adequate to reveal clinically important heterogeneity of treatment effect.21 Consequently, we selected the Goodney et al 19 model as our Primary Baseline Stroke Model as it was the model that was most completely mappable to the CREST and SVS VQI datasets (age >70, contralateral occlusion, heart failure, qualifying event Stroke/transient ischemic attack (TIA)). In a sensitivity analysis, we applied the Rothwell and Warlow20 model as a Sensitivity Baseline Stroke Model as it included baseline blood pressure (in addition to sex and whether the qualifying event was a stroke/TIA) as it was the model most qualitatively dissimilar from the other models. For both models, we determined their discrimination in CREST by calculating c-statistics and explored their clinical utility by determining how event rates varied for CEA and CAS across baseline risk quintiles.
Developing models to predict MI, stroke with CEA/CAS treatment in CREST
To inform the risks of treatment for individual patients, we built models incorporating the relevant baseline risk indices (eg, Primary Stroke Treatment Model incorporates the Primary Baseline Stroke Model) to separately predict the risk of stroke and MI in CREST. Initial treatment models predicted each outcome (MI, perioperative stroke/death) separately using logistic regression models that included the baseline risk index (linear predictor of risk using each baseline risk score), symptomatic status and treatment (CEA vs CAS) as independent variables. For all treatment models multi-collinearity of variables was assessed using variance inflation factors. For all baseline risk and treatment models alpha values of 0.05 were applied, calibration was assessed by plotting observed (proportion of population with a given outcome) versus expected (mean predicted outcome probability from a given model) across quintiles of baseline risk. Analyses were executed using Stata (V.15) (StataCorp: Release 15).
To assess the accuracy of model specification, we explored whether treatment interaction terms should be retained in the final model. This was done by exploring alternate model specifications that included interaction terms with treatment status. Interactions were added to the model based on our judgement of their prior theoretical/empirical evidence and retained in the final model if the interaction term was statistically significant or if the model with the interaction reduced the model’s Akaike information criterion.22 Interactions were explored in the order: baseline risk, age (given the presence of an age-treatment interaction in CREST), symptomatic status, and then all individual baseline risk factors included in each individual risk index. Models are summarised in online supplemental table 1.
Supplemental material
To assess for overfitting, models were internally validated with bootstrapping and shrinkage coefficients were estimated for each model.23 After final internally validated treatment models were developed for both stroke and MI, overall model discrimination was measured using c-statistics. Model calibration for stroke, MI and total vascular events was assessed (predicted probability of both individual models) by plotting actual event risk versus estimated event risk across event quintiles.
To inform the overall net benefits of CEA versus CAS, we first explored the correlation between risks estimated from individual models using Pearson correlation coefficients. We then used model-based estimates to predict the net number of events for each individual in CREST under the assumption they received CEA versus CAS in the overall trial and separately in the symptomatic and asymptomatic populations.
External validation of treatment models
To approximate the CREST population, all patients were included in SVS-VQI external validation except those that had: (1) prior neck radiation, (2) CEA or CAS as a reoperation on a previously treated carotid, (3) unstable angina or (4) a MI within the prior 6 months. For the final stroke and MI models, we externally validated the models in SVS VQI by measuring c-statistics for the linear predictors in the combined SVS VQI CEA and CAS registries and measured calibration by comparing observed to expected event rates. As summarised in results, the primary model overpredicted risk in SVS-VQI. To account for this, we estimated models that recalibrated relative to the SVS-VQI population. Specifically, we estimated a model with a recalibrated intercept by calculating the linear predictor of the treatment model in SVS-VQI and estimating a model with a single parameter, the intercept in SVS-VQI. We also estimated a model with recalibrated slope and intercept by fitting logistic regression models in SVS-VQI using a single predictor variable with the linear predictor from the CREST-based treatment models. Finally, to estimate the potential real world utility of the combined modelling approach, we compared estimated stroke and MI rates (using intercept and slope recalibrated models) in patients assigned to optimal model assigned treatment compared with the currently assigned treatment and strategies of treating all patients with CAS or CEA in SVS-VQI. Final models and recalibrated models were integrated into an online risk calculator, available at: (https://carotid-risk-calculator.herokuapp.com/riskCalculator/).
Results
Baseline risk index performance in crest
Our Primary Baseline Stroke Model predicted stroke with a c-statistic of 0.64 in CREST, while the Sensitivity Baseline Stroke Model was slightly less predictive with a c-statistic of 0.62. Conversely, the Sensitivity Baseline Stroke Model had slightly better calibration compared with the Primary Baseline Stroke Model which underestimates risk in the highest risk quintile. The Primary Baseline MI Model had a c-statistic of 0.62 with reasonable calibration across the risk spectrum. Predicted baseline MI and stroke risks were correlated. Correlation for predicted risk from the Primary Baseline Stroke Model and the Primary Baseline MI Model was slightly higher (r=0.28, variance explained=0.08) than for the Sensitivity Baseline Stroke Model and the Primary Baseline MI Model (r=0.19, variance explained=0.04). Primary and sensitivity baseline stroke risk models were moderately strongly correlated. (r=0.55, variance explained=0.30).
Treatment model performance
The Primary Stroke Treatment Model did not include an interaction between baseline risk and treatment status, but did retain an age-treatment status interaction (as observed in the baseline CREST data) (c-statistic 0.69). The Sensitivity Stroke Treatment Model did retain a baseline risk-treatment interaction (c-statistic 0.68). Coefficients in both treatment models are summarised in table 1. Calibration for the Primary Stroke Treatment Model was reasonable across the risk spectrum. (online supplemental figure 2A). The Primary MI Treatment Model also had moderate predictiveness (0.68) and reasonable calibration (online supplemental figure 2B). There was little evidence of overfitting, mean shrinkage coefficient for the Primary Stroke Treatment Model 1.00 (95% CI 0.99 to 1.01) and the Primary MI Treatment Model : 1.00 (95% CI 1.00 to 1.00). Predicted stroke and MI risk (estimated from the primary treatment models) were correlated, (correlation coefficient=0.28, variance explained=0.08), but the magnitude of the correlation was modest (online supplemental figure 3).
Most patients (86.7%) under the Primary Stroke and MI Treatment Models were predicted to have net benefit (change in stroke risk from CEA-CAS—change in MI risk from CAS-CEA) from CEA (median net benefit 0.8% (IQR 1.8%)). The retention of an interaction term in the model in the Sensitivity Stroke Treatment Models substantially reduced this proportion such that only 54.0% were anticipated to benefit from CEA, although the magnitude of the expected benefit was larger (median net benefit 2.1% (IQR 3.1%)). The distribution of net benefit for both models is illustrated in figure 2. These proportions varied markedly by symptomatic status with 97% of symptomatic patients with predicted net benefit from CEA in the Primary Stroke and MI Treatment Models (median net benefit 2.0%, IQR 0.9%) vs 83% in the Sensitivity Stroke Treatment Model/Primary MI Treatment Model (median net benefit 2.0%, IQR 3.4%). For asymptomatic patients, 75% were predicted to have net benefit from CEA in the Primary Stroke and MI Treatment Models (median net benefit 0.2%, IQR 0.6%) vs 22% in the Sensitivity Stroke Treatment Model/Primary MI Treatment Model (median net benefit −0.8%, IQR 1.3%).
External validation in SVS-VQI
In the combined SVS-VQI dataset, (n=76 269), 14.4% of patients received CAS. The rate of perioperative stroke or death was 1.4% and the rate of perioperative MI was 0.7%, the SVS-VQI population is described in online supplemental table 2. Both Primary Stroke and MI Treatment Models had reduced discrimination in SVS VQI compared with CREST (c-statistic 0.60 for stroke model and 0.54 for MI model). Both models overpredicted risk compared with self-reported event rates in SVS VQI and the magnitude of over prediction was more extreme at the high end of the risk spectrum for both. Recalibrating the model intercept in SVS-VQI attenuated this over prediction, but some over prediction persisted among high-risk patients for both stroke and MI. Models that recalibrated both the slope and intercept had good calibration across the risk spectrum (figure 3).
The optimal predicted treatment was CEA for 89% vs 11% for CAS in the SVS-VQI sample. The median age among patients for whom CAS was predicted optimal treatment is 53 (IQR 7) vs 72 (IQR 8) for CEA. CEA was the predicted optimal treatment for 84% of patients receiving CAS. Conversely, CAS was the predicted optimal treatment for 10% of patients currently receiving CEA.
The model-based optimal treatment strategy resulted in the lowest overall event rates (2.04%), but the difference between the as-treated population (2.12%) was quite small and was almost identical to the predicted overall event rate if all patients were treated with CEA (2.05%) . Conversely, the predicted overall event rate was higher for a strategy of treating all patients with CAS (2.63%) (table 2).
Discussion
We found that a combined modelling strategy that stratifies stroke and MI risks while accounting for other factors that predict response to treatment and outcomes has promise to identify the best carotid treatment modality for individual patients. However, for this approach to result in meaningful practical gains, more discriminating and better calibrated tools would be needed to justify the increased complexity.
In terms of selecting the optimal carotid revascularisation technique our analyses, based on limited imperfect baseline risk models, support current guidelines that generally favour CEA over CAS as initial treatment for most symptomatic patients. Concordant with prior work, we found that CEA results in consistently lower stroke rates and while it is associated with higher MI rates, the magnitude of the effect is small and that across both baseline risk models that more patients would benefit from CEA. Moreover, for the subset where CAS was favoured among symptomatic patients, the magnitude of projected benefit for CAS was quite small both in CREST and SVS-VQI. As a consequence, we found that a treatment policy that offered CEA as a first line treatment to all patients in SVS-VQI would be likely to result in similar overall event rates (and slightly lower stroke rates) as applying the optimal model based prediction. The fact that a simple strategy such as CEA for all would perform almost as well as a policy of is in many ways the strongest argument against applying our current complex model-based strategy clinically. At the same time, our data suggest that for most symptomatic patients that the magnitude of benefit with CEA over CAS is not large—with a risk reduction of about 2 percentage points for CEA for CAS. Consequently, if clinical factors are present that would substantially increase the risk of CEA (but not CAS) by more than this margin, CAS could be considered as an alternative treatment. Given that most existing risk prediction models include factors that likely increase the risk of both CEA and CAS among patient with no anatomic contraindications to open surgery (eg, increased age, medications, qualifying event as a stroke), it is likely that such patients are relatively rare. However, it is not implausible that models with better discrimination might identify a meaningful subset of such patients.
Our data provide less insight into the optimal treatment for asymptomatic carotid stenosis. Markedly different optimal treatment decisions emerged depending on which baseline risk model was applied. When applying the primary baseline risk model—75% were predicted to benefit from CEA, whereas under the secondary/blood pressure model, 78% were expected to benefit from CAS. Although, under both baseline risk models, the absolute difference in the predicted benefit of CEA versus CAS was fairly small—less than one percentage point for most patients under both baseline risk models. The ongoing CREST2 trial should provide strong data not only on which is the optimal revascularisation modality in this population, but about which patients, if any, should receive revascularisation at all compared with modern medical management.24
Our experience with these models highlights two issues which may be relevant to future treatment decision modelling: 1. selecting the optimal baseline risk model and 2. challenges in calibration and external validation. Numerous different models have been previously published to predict the baseline risk of perioperative stroke—most based on reasonably high-quality data and with reasonable modelling strategies. When building a treatment model, then, which baseline risk model should be chosen? Our approach to model selection was to pick the model that fit our data best and, practically, could best be applied to the available data. Theoretical considerations led us to also explore a sensitivity analysis model—a trial-derived model that included a continuous physiologic measure, baseline blood pressure, as opposed to the types of binary risk factors included in the other mostly registry-based models. Yet, as described, treatment models based on our primary and secondary baseline risk models arrived at divergent conclusions for asymptomatic patients, even though the baseline risk models were moderately correlated. This divergence is largely due to the retention of a baseline risk-treatment interaction term in the secondary treatment model, leading to a lower predicted stroke risk in low baseline risk patients in the secondary model compared with the primary model. In retrospect, we do not see an obviously superior strategy for selecting a baseline risk model. Instead, we would generally argue that the best approach would be to apply the model that predicts most accurately in the target population. In this case, unfortunately, the absence of a baseline blood pressure measures in SVS-VQI, precludes us from comparing model performance in our target population. More predictive baseline risk models, incorporating both physiologic, anatomic (eg, plaque morphology, high risk features) and risk factor data, are likely needed to improve the accuracy and consistency of predictions of the optimal carotid revascularisation strategy.
The problem of adequately calibrating risk models relative to their target population is common. As a particularly salient example, the atherosclerotic cardiovascular disease risk score that is guideline-supported to target statin therapy in primary prevention has varying calibration depending on the study population.25 In this analysis, we found that our trial-based treatment risk models predict considerably higher risk than what was observed in the SVS-VQI registry. While this may occur for many reasons, the two most plausible reasons are that the CREST population was higher risk than the SVS-VQI population or that outcomes are under-reported in SVS-VQI. It is unlikely that the CREST population was at markedly higher baseline risk than the SVS-VQI population. First, CREST did not have particularly restrictive inclusion criteria, but to the extent that it did, it likely biased the population to a lower risk population (ie, excluding patient with prior disabling stroke).26 Moreover, clinical trial populations generally select for somewhat lower risk populations than in the general population,27 28 and in prior carotid trials, those selected into the trials likely had lower risk than the treated populations at those centres.29 Second, optimal medical management is likely an important factor in limiting stroke risk in this population and was explicitly attended to in the CREST trial.2 Conversely, it is quite plausible that the self-reported outcomes in SVS-VQI under-report some events. While CREST’s evaluation of stroke risk included a systematic examination and screening instrument applied to all patients, no specific evaluation strategy is specified in SVS-VQI. What is reported, then, is likely a reflection only of events that are clinically diagnosed and likely underdiagnoses some mild strokes. Regardless, neither interpretation is necessarily a fundamental barrier to the application of our carotid models, nor of risk models more generally. Based on the features of one’s practice environment, one could reasonably choose to apply either the CREST-based or SVS-VQI based risk estimates. The optimal approach would likely be to recalibrate trial-derived models in the target population, to account for differences between the trial and target populations and intervention characteristics, in an approach that accounts for measurement error within the target-population ascertained outcomes.30
Conclusion
Our data suggest that decomposing individual elements of risk (eg, stroke and MI) could potentially be a valuable approach for individualising the CEA versus CAS treatment decision, but would require risk prediction tools that are somewhat more discriminant and locally calibrated. For selection of carotid revascularisation, our data support guideline-based recommendations to favour CEA over CAS, and strongly support this idea for symptomatic patients.
Data availability statement
Data are available in a public, open access repository. Data may be obtained from a third party and are not publicly available. CREST data are available from NINDS. SQS-VQI data are available via application to SVS-VQI.
Ethics statements
Patient consent for publication
Ethics approval
The study protocol was reviewed by the University of Michigan Institutional Review Board and consider exempt as it relied on deidentified or limited datasets.
References
Supplementary material
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors JFB conceived the research, acquired data, executed analyses and drafted the manuscript. RAH and LBM conceived the research, supervised and provided detailed feedback on the manuscript and analyses. NHO participated in data acquisition, study design and provided detailed feedback on the manuscript.
Funding This work was funded by the National Institute of Health (NINDS, K08 NS082597).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.