Article Text
PDF
Abstract
Background and purpose Triage tools to identify candidates for thrombectomy are of utmost importance in acute stroke. No prognostic tool has yet gained any widespread use. We compared the predictive value of various models based on National Institutes of Health Stroke Scale (NIHSS) subitems, ranging from simple to more complex models, for predicting large artery occlusion (LAO) in anterior circulation stroke.
Methods Patients registered in the SITS international Stroke Register with available NIHSS and radiological arterial occlusion data were analysed. We compared 2042 patients harbouring an LAO with 2881 patients having no/distal occlusions. Using binary logistic regression, we developed models ranging from simple 1 NIHSS-subitem to full NIHSS-subitems models. Sensitivities and specificities of the models for predicting LAO were examined.
Results The model with highest predictive value included all NIHSS subitems for predicting LAO (area under the curve (AUC) 0.77), yielding a sensitivity and specificity of 69% and 76%, respectively. The second most predictive model (AUC 0.76) included 4-NIHSS-subitems (level of consciousness commands, gaze, facial and arm motor function) yielding a sensitivity and specificity of 67% and 75%, respectively. The simplest model included only deficits in arm motor-function (AUC 0.72) for predicting LAO, yielding a sensitivity and specificity of 67% and 72%, respectively.
Conclusions Although increasingly more complex models yield a higher discriminative performance for predicting LAO, differences between models are not large. Assessing grade of arm dysfunction along with an established stroke-diagnosis model may serve as a surrogate measure of arterial occlusion-status, thereby assisting in triage decisions.
- thrombolysis
- thrombectomy
- stroke
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Stroke is one of the leading causes of death and disability worldwide, afflicting approximately 15 million people each year, of which 10 million patients either die or suffer from permanent disability.1 For many years, intravenous thrombolysis with recombinant tissue plasminogen activator (r-tPA) alteplase was the only available pharmacological treatment for acute ischaemic stroke, improving long-term functional outcome after stroke.2 Recently, several randomised controlled trials have proven the benefit of endovascular thrombectomy using stent-retrievers in large artery occlusion (LAO) for anterior circulation stroke within 6 hours of symptom onset and in selected cases up to 24 hours after symptom onset.3–5 These results have revolutionised the management of acute ischaemic stroke. However, the beneficial effects of endovascular thrombectomy have been shown to be time-dependent, decreasing with increased time from symptom onset to reperfusion.6 7 The need of prehospital tools rapidly identifying patients likely harbouring an LAO amenable to endovascular treatment have sparked the publication of several scales.8–15 These scales aim at directing patients with a high probability of LAO to comprehensive stroke centres with endovascular capabilities, potentially reducing the time to reperfusion.8–15 None of these scales have so far gained widespread use, and only one of the scales has been validated prospectively in a prehospital setting.13 All scales have incorporated all or subitems of the National Institutes of Health Stroke Scale (NIHSS), as stroke severity has been shown to be an important predictor of LAO.16–18 Using the full NIHSS in the prehospital setting is time-consuming, and the need for simple prediction rules is evident.
The aim of the current study is to compare the performance of various models for predicting LAO based on NIHSS subitems, ranging from simple to more complex models.
Methods and materials
The study was based on patients registered in the Safe Implementation of Treatments in Stroke (SITS)-International Stroke Treatment Register (SITS-ISTR) between 1 December 2012 and 23 October 2015. Only patients with available CT angiography/MR angiography (CTA/MRA) and baseline NIHSS data treated with intravenous alteplase and/or endovascular thrombectomy were considered. The time period was chosen based on the time of implementation in the database of additional variables for registering detailed information on arterial occlusion site using baseline CTA/MRA. Data for the current study were contributed by hospitals which reported specified arterial occlusion sites in at least 20 patients in whom any occlusion was reported as present. This selection of experienced centres was done in order to ensure a high quality registration of arterial occlusion data.
The SITS-ISTR is a multinational, prospective, observational monitoring register documenting data for centres using various treatments in acute ischaemic stroke. The aims of the register, collection of data and structure of the database have been described previously.19 Baseline and demographic characteristics, stroke severity per the NIHSS, arterial occlusion status assessed using CTA/MRA, risk factors, time logistics, medication history, imaging data and 3-month outcome data were registered.
Ethical approval and data monitoring
Ethical approval or patient consent for participation in SITS-ISTR was obtained in countries that required this; other countries approved the register for conduct as an anonymised audit. The SITS-MOST II study was approved by the Stockholm Regional Ethics Committee. The SITS International Coordination Office performed regular online monitoring of the SITS-ISTR data online and checked individual patient data on a regular basis to identify errors or inconsistencies.
Outcome measurements
Arterial occlusion was defined as a complete occlusion or less than 50% filling of the affected vascular territory on CTA/MRA. LAO was defined as occlusion of any segment of the internal carotid artery, carotid terminus, M1 branch of the middle cerebral artery or basilar artery. In cases where more than one arterial segment was reported as being occluded, the most proximal vessel was registered for the analysis. The control population was defined as patients without visible occlusions or with more distal occlusions (M2, M3, distal to M3; A1, A2, distal to A2, P1, P2 and P3).
All assessments of imaging studies were done according to clinical routine at centres participating in the SITS-ISTR, and no central assessment of imaging data was routinely performed.
Statistical analysis
For baseline and clinical data, univariate analysis was performed comparing patients with and without LAO. Proportions were calculated for categorical variables, dividing the number of events with the total number excluding missing/unknown cases. For continuous variables, medians were calculated. Statistically significant differences between proportions were assessed using the χ² method, and the Mann–Whitney U-test was used for medians.
The most highly predictive models, exploratively identified using highest area under the curve (AUC) by receiver operating characteristic (ROC) analysis and varying from single-variable models to more complex models, were developed using binary logistic regression analysis.
We assessed for differences between the models using both comparisons of the area under the ROC curves (likelihood ratio tests and Wald tests) as well as IDI (Integrated discrimination improvement) and NRI (net reclassification improvement) statistics to judge the value of added items. Sensitivities and specificities for predicting LAO were determined at three distinct cut-offs (definition see below). The various models were solely based on NIHSS subitems registered in a continuous fashion separating different degrees of dysfunction.
Using binary logistic regression, we derived the most predictive models for predicting LAO based on subitems of the NIHSS. We investigated 1-item, 2-items, 3-items, 4-items models as well as a model including all NIHSS subitems. For each model, whether a 1-item, 2-items, 3-items, 4-items or full NIHSS subitem model, we investigated which combination of items yielded the highest AUC and subsequently chose these models for further comparison. Within each type of model (1-item, 2-items, 3-items or 4-items model), the AUCs of the different subitem combinations were compared using likelihood ratio tests. In this way, we finally chose one specific item-combination for each 1-item, 2-items, 3-items and 4-items model based on maximising the AUC for LAO prediction. For each model, we assessed the sensitivity and specificity for LAO at three different cut points: at an optimal cut-point maximising the Youden index, at a cut-point maximising the specificity (as close to 80% as possible) and at a cut-point maximising the sensitivity (as close to 80% as possible). In addition to investigating the above-mentioned models, we decided to specifically include a hemiparesis model including the arm and leg NIHSS-subitems (NIHSS item 5 and NIHSS item 6) as well as a model solely based on the total NIHSS score, in contrast to the full-items model which included each NIHSS subitems as a separate predictor. In order to better appreciate the performance of our models, we also performed a comparison with selected pre-existing LAO prediction instruments (PASS, RACE, G-FAST and CPSSS). These scales were computed on our study-base in accordance with the descriptions in the respective original articles.10 12 13 20
Results
Between 1 December 2012 and 23 October 2015, 5112 patients with available CTA/MRA data from large-volume centres were registered in the database. Baseline NIHSS (and complete NIHSS subitem data) was available in 96% of the study group (4923/5112). Of these, 2042 patients (41%) had a LAO, while 2881 patients (59%) had either no occlusion or more distal occlusions. Table 1 summarises the baseline characteristics, comparing patients with LAO with patients without LAO, with complete NIHSS baseline data.
Baseline clinical characteristics of patients with and without LAO
Patients with and without LAO differed somewhat regarding several baseline variables, however not to any larger extent although being statistically significant, see table 1. The most notable difference between patients with and without LAO was in the total baseline NIHSS score at 16 vs 8, respectively, as well as higher systolic blood pressure in patients without LAO and higher prevalence of atrial fibrillation in patients with LAO.
Models for predicting LAO
The most predictive models with increasing number of included subitems are given below. The sensitivity and specificity of the models at the optimum Youden cut-off, at the maximum specificity cut-off and at the maximum sensitivity cut-off are given in table 2. Figure 1 depicts the AUC for LAO comparing the most predictive 1-item, 4-items and full-items models. The AUCs for the models given below were all statistically significantly different from each other at the 5% level (assessed using likelihood ratio tests and Wald tests). An online supplementary table e-1, displays the proportion of patients being detected using the various models and cut-offs as well as positive and negative predictive values (PPVs and NPVs). Online supplementary table e-2 displays the IDI and NRI statistics, assessing the additional value of adding extra subitems to increasingly complex models.
Supplemental material
AUC for predicting LAO, comparing the most predictive NIHSS 1-item, 4-items and full-items models. The 1-item model included NIHSS subitem 5 (arm function), the 4-items model included subitems 1C, 2, 4 and 5 (level of consciousness commands, gaze, facial and arm motor function. The full-items model included all NIHSS subitems. AUC, area under the curve; LAO, large artery occlusion; NIHSS, National Institutes of Health Stroke Scale.
Sensitivity and specificity for predicting LAO for various models at optimum cut-off, high specificity cut-off and high sensitivity cut-off
1-item model
The 1-item model associated with the highest AUC (0.72) including only NIHSS subitem 5, arm motor function. The optimum Youden cut-off (NIHSS item 5≥3) resulted in a sensitivity and specificity of 67% and 72%, respectively. At the high-specificity cut-off (NIHSS item 5=4) a sensitivity and specificity of 53% and 81%, respectively, was seen. The high sensitivity cut-off (NIHSS item 5≥2) resulted in a sensitivity and specificity of 77% and 57%, respectively.
A 1-item model including only NIHSS subitem 6, leg motor function, yielded an AUC of 0.71.
2-items model
The 2-items model associated with the highest AUC (0.75) included NIHSS subitems 5 and 2, arm motor function and gaze, respectively. The optimum Youden cut-off resulted in a sensitivity and specificity of 68% and 73%, respectively, at the high-specificity cut-off a sensitivity and specificity of 64% and 76% was seen, and the high sensitivity cut-off resulted in a sensitivity and specificity of 82% and 53%, respectively.
3-items model
The 3-items model associated with the highest AUC (0.755) included NIHSS subitems 5, 2 and 4, arm motor function, gaze and facial motor function, respectively. At the optimum Youden cut-off a sensitivity and specificity of 70% and 72% was seen, the high-specificity cut-off resulted in a sensitivity and specificity of 61% and 80%, and at the high sensitivity cut-off a sensitivity and specificity of 80% and 59% was seen.
4-items model
The 4-items model associated with the highest AUC (0.76) included NIHSS subitems 5, 2, 4 and 1C, arm motor function, gaze, facial motor function and level of consciousness commands, respectively. The optimum Youden cut-off resulted in a sensitivity and specificity of 67% and 75%, respectively, at the high-specificity cut-off a sensitivity and specificity of 61% and 80% was seen, and the high sensitivity cut-off resulted in a sensitivity and specificity of 79% and 59%.
Full-items model
The full-items model resulted in an AUC of 0.78. The optimum Youden cut-off yielded a sensitivity and specificity of 69% and 76%, at the high-specificity cut-off a sensitivity and specificity of 62% and 80% was seen, and the high sensitivity cut-off resulted in a sensitivity and specificity of 81% and 59%.
Hemiparesis model
The hemiparesis model, including arm and leg function (NIHSS subitems 5 and 6), was associated with a AUC of 0.73. At the optimum Youden cut-off a sensitivity and specificity of 65% and 74% was seen, the high-specificity cut-off resulted in a sensitivity and specificity of 53% and 82%, and at the high sensitivity cut-off a sensitivity and specificity of 79% and 52% was seen.
Total NIHSS score model
The model including the total NIHSS score as a predictive model was associated with a AUC of 0.76. The optimum Youden cut-off (NIHSS ≥13) score resulted in a sensitivity and specificity of 69% and 73%, at the high-specificity cut-off (NIHSS ≥14) a sensitivity and specificity of 60% and 80% was seen, and the high sensitivity cut-off (NIHSS ≥9) yielded a sensitivity and specificity of 80% and 57%.
Sensitivity analysis including M2/A1
When including M2, A1 and P1 occlusions in the definition of LAO, AUC of the 1-item model including arm paresis was 0.71, the AUC of the 4-items model including NIHSS subitems 5, 2, 4 and 1C was 0.76 and the AUC of the full-items NIHSS 0.77.
Sensitivity analysis excluding basilar artery occlusions
When excluding basilar artery occlusions from the definition of LAO, AUC of the 1-item model including arm paresis was 0.72, the AUC of the 4-items model including NIHSS subitems 5, 2, 4 and 1C was 0.76 and the AUC of the full-items NIHSS 0.77.
Pre-existing LAO models
The results of the comparison of the arm paresis model with the four selected pre-existing LAO prediction scales can be seen in online supplementary table e-3.
Discussion
In the era of endovascular thrombectomy in acute ischaemic stroke, reducing time to reperfusion in patients harbouring LAO is of the utmost importance.7 Although baseline stroke severity measured by the NIHSS score predicts the presence of arterial occlusions,8 16–18 21 22 the full NIHSS is time-consuming and probably too complicated in the prehospital setting. Several attempts at developing simplified scores for use in the prehospital setting have been done; however, none of these has to our knowledge gained any widespread use.8–10 12–15
In this study, we show that although increasing numbers of included NIHSS subitems in LAO prediction models leads to better prediction, the differences do not seem to be clinically relevant. A simple predictive model only including the arm motor function item of the NIHSS shows both sensitivities, specificities, PPVs and NPVs (67%, 72%, 63% and 75%, respectively) quite close to the values for a model including the full range of NIHSS subitems (69%, 76%, 67% and 77%, respectively). To our knowledge, no study has investigated such a simple model previously.
Different settings and situations may need different trade-offs between sensitivity and specificity. In a recent article from the SITS collaborators, two different clinical scenarios were described.23 In the first scenario, a patient assessed by paramedics at an early time-point or with a short transfer-time to a comprehensive stroke centre with endovascular capabilities was considered. In such a patient, a predictive model with a high sensitivity would be desirable. Using an unadjusted cut-off NIH item 5 (arm function)≥2 gave a sensitivity, specificity, PPV and NPV of 77%, 57%, 57% and 78%, quite similar to the adjusted full-items NIH model, which gave a sensitivity, specificity, PPV and NPV of 81%, 59%, 58% and 81%, respectively. This corresponds to the high-sensitivity cut-off of the total NIHSS score ≥9. In the second scenario, a patient evaluated by paramedics close to the end of the IVT time-window or with a long transfer-time to a comprehensive stroke centre was considered. A high specificity for LAO would be desirable in this situation, in order to reduce the number of futile transfers. Using a cut-off NIH item 5 (arm function)=4 gave a specificity, sensitivity, PPV and NPV of 81%, 53%, 66% and 71%, not very far from the adjusted full-items NIH model which gave a specificity, sensitivity, PPV and NPV of 80%, 62%, 69% and 75%. This corresponds to the high-specificity cut-off of the total NIHSS score ≥14.
In summary, although the more complex models perform statistically significantly better than our simple arm dysfunction model (see table 2 and online supplementary table e-2), differences are not large and the simple model performs reasonably well in both clinical scenarios. We performed a comparison of the simple arm paresis model with four pre-existing LAO prediction instruments (CPSSS, PASS, G-FAST and RACE scores), see online supplementary table e-3. The AUCs of three of the pre-existing models differed significantly from the arm-paresis model, CPSSS and RACE performing better and PASS worse. G-FAST did not differ significantly. However, the AUCs of all five models were similar, suggesting a similar predictive performance.
Prediction of LAO without vessel imaging will always lead to either missing patients harbouring a treatable arterial occlusion or unnecessary transfer of patients without treatable occlusions to comprehensive stroke centres. However, simple prediction rules may improve patient selection and potentially lead to improved onset-to-reperfusion times. Using the high-specificity cut-off (NIH item 5=4) as a triage tool to endovascular centres would identify 33% of our study population as possible LAO candidates (see online supplementary table e-1), the PPV amounting to 66%, ie, nearly 7/10 successful transfers. Using the high-sensitivity cut-off (NIH item 5≥2) as a triage tool would identify roughly 60% of our study population (see online supplementary table e-1), the PPV amounting to 57%, ie, nearly 6/10 successful transfers (see online supplementary table e-1). The choice of cut-off depends on the clinical situation and local circumstances as outlined above. We suggest that a thorough screening for severity of arm motor function can be used as an LAO screening step after confirmation of likely stroke diagnosis has been performed using a good stroke recognition score at the primary screening step like the FAST score.24 Previous studies have shown an excellent inter-rater reliability of the motor subitems of the NIHSS score as well as the possibility of its use by non-physicians undergoing a brief training period.25 26 We therefore believe that our simple arm paresis model would suit the prehospital setting.
This study bears with it certain limitations inherent to the observational design. Our definition of LAO did not include M2 occlusions which may be amenable to treatment; however, a sensitivity analysis including M2 occlusions in the LAO definition showed very similar results to those presented in the main results. In addition, a subanalysis excluding basilar occlusions from the definition of LAO also presented very similar results. Although we extend our results to a prehospital setting, assessments of the NIHSS registered in the database were performed in a hospital-setting. Also, our study is based on patients treated with intravenous-tPA and/or endovascular thrombectomy and may not be valid for unselected patients with a prehospital suspicion of stroke. However, previous studies on unselected ischaemic stroke cohorts have shown LAO proportions similar to our cohort.27 Our selection of patients with documented ischaemic stroke is a potential source of bias, as both haemorrhagic stroke and stroke mimics will be a part of the assessed population in the prehospital phase. Despite these limitations, we believe that our findings are of interest.
Conclusion
Although increasingly more complex clinical models yield a higher discriminative performance for predicting LAO, the difference compared with extremely simple models is not very large. Balancing predictive performance against ease of use and practicality, simply assessing the grade of arm dysfunction may serve as a surrogate measure of LAO status, and in conjunction with a validated stroke recognition instrument, may assist in decision making in guiding high-risk patients to comprehensive stroke centres with endovascular treatment capabilities. These findings need to be confirmed in a future prospective study on prehospital triage.
References
Footnotes
Contributors CC, NW, MVM and NA designed the study. CC and MB performed all statistical analysis. CC prepared the first draft of the manuscript under the active supervision of NA. MN was one of the leading local recruiters of patients in SITS-ISTR. RM and DT were the Scientific Committee members and top national coordinators of leading recruiting countries. MN, RM, DT, JFS, AHA-R, AK, MM and TPM contributed with important interpretation of the draft manuscript and subsequent revisions.
Funding SITS is financed directly and indirectly by grants from Karolinska Institutet, Stockholm County Council, the Swedish Heart-Lung Foundation, the Swedish Order of St. John, Friends of Karolinska Institutet and private donors as well as from an unrestricted sponsorship from Boehringer-Ingelheim. SITS has previously received grants from the European Union Framework 7, the European Union Public Health Authority and Ferrer Internacional. SITS is currently conducting studies supported by Boehringer-Ingelheim and EVER Pharma as well as in collaboration with Karolinska Institutet, supported by Stryker, Covidien and Phenox. RM has been supported by the project no. LQ1605 from the National Program of Sustainability II (MEYS CR) and by the project FNUSA-ICRC no. CZ.1.05/1.1.00/02.0123 (OP VaVpI). JFS is participant in the Charité Clinical Scientist Program funded by the Charité Universitätsmedizin Berlin and the Berlin Institute of Health.
Competing interests CC was supported by the Stockholm County Council (combined clinical residency and PhD training program). MVM is an employee of SITS International, which receives a grant from Boehringer Ingelheim for the SITS-ISTR. MVM is also supported by the Stockholm County Council (clinical postdoctoral appointment). TPM has received travel and speaker grants from Boehringer Ingelheim for the SITS-ISTR and research grants from Cerevast Inc. NW has received expenses from Boehringer Ingelheim for his role as member of the Steering Committee in relation to the ECASS III trial with alteplase and served as a consultant to Thrombogenics as chairman of the Data and Safety Monitoring Board. SITS International (chaired by NW) receives a grant from Boehringer Ingelheim for the SITS-ISTR. His institution has also received grant support towards administrative expenses for coordination of the ECASS III trial. NW has also received lecture fees from Boehringer Ingelheim and Ferrer. NW is also supported by grants provided by the Stockholm County Council (ALF project). NA is an employee of SITS International, which receives a grant from Boehringer Ingelheim for the SITS-ISTR. NA is also supported by grants provided by the Stockholm County Council (ALF project) and Swedish Heart-lung foundation.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Any additional unpublished data from the study are available to corresponding author CC.