Validity and Reliability of Simple Questions in Assessing Short- and Long-Term Outcome in Norwegian Stroke Patients

Transcription

1 Original Paper Cerebrovasc Dis 2001;11: Received: April 13, 2000 Accepted: June 22, 2000 Validity and Reliability of Simple Questions in Assessing Short- and Long-Term Outcome in Norwegian Stroke Patients Eivind Berge a Hild Fjærtoft b Bent Indredavik b Per Morten Sandset a a Haematological Research Laboratory, Department of Haematology, Ullevål University Hospital, Oslo, and b Stroke Unit, Department of Medicine, University Hospital of Trondheim, Norway Key Words Outcome stroke W Recovery stroke W Simple questions, validity and reliability W Barthel ADL Index W Modified Rankin Scale W Validity of simple questions W Reliability of simple questions Abstract The utility of simple questions for the assessment of stroke outcome in large-scale international studies has generally been approved, but their validity and reliability have not been evaluated in different cultures or at different intervals after a stroke. The study comprised 150 stroke patients who had been admitted consecutively to a stroke unit 6 weeks or 6 months earlier. Two weeks before the visit the patient received a postal questionnaire containing the simple dependency question: In the last 2 weeks, did you require help from another person for everyday activities? and the simple recovery question: Do you feel that you have made a complete recovery from your stroke?. The visit was performed by trained personnel unaware of the patient s or his carer s replies, and comprised the same 2 questions administered by the personnel, the Barthel ADL Index (BI) and the modified Rankin Scale (mrs). The patients functional status was categorised as good or bad according to the chosen cutoff levels on BI and mrs. At 6 months the dependency question had an accuracy of 83 and 82% in identifying patients with good or bad outcome, defined as BI 695 or!95 and mrs!3 or 63, respectively, whereas the recovery question had an accuracy of 86% when compared with mrs = 0 or 10. There was no difference in accuracy of the simple questions at 6 weeks compared with 6 months, and there was no clinically important difference between responses from patients and carers. The agreement between the responses to the questionnaire and the interview was good to moderate (Î = 0.62 for the dependency question, and 0.55 for the recovery question). We conclude that the simple questions seem to be valid and reliable measures of stroke outcome when tested in Norwegian patients after 6 weeks or 6 months, which supports their continued use in large-scale multinational stroke studies at different intervals after stroke. Introduction Copyright 2001 S. Karger AG, Basel Two simple questions on dependency and recovery are found to be a valid and reliable, as well as relevant and practical way of measuring long-term stroke outcome in large numbers of patients [1 3], and their utility has been shown in a number of recent stroke trials [4 8]. Despite their widespread use, the evaluation of the 2 questions has been confined to British stroke survivors 6 months after the stroke. Further testing may therefore be needed to confirm their validity and reliability in different cultures as well as at different time intervals after stroke [3]. The former evaluation of the simple questions has disclosed that the validity and reliability are sensitive to ABC Fax karger@karger.ch S. Karger AG, Basel /01/ $17.50/0 Accessible online at: E. Berge Haematological Research Laboratory Department of Haematology, Ullevål University Hospital N 0407 Oslo (Norway) Tel , Fax , eivind.berge@ulleval.no

2 small modifications of their phrasing [3]. Small differences in their exact meaning or possible ambiguities may significantly reduce their usefulness, and translation into other languages for use in other cultures may introduce even larger errors. Another concern relates to the application of the 2 questions shortly (within weeks) after stroke onset. The meaning of help from another person (dependency question) and complete recovery (recovery question) may be different for patients shortly after the stroke, as compared with 6 months later. In the present study, our aim was to provide some additional data on the concurrent validity and inter-rater reliability of the simple questions, to clarify their potential usefulness as measures of stroke outcome in different cultures and at different time intervals after stroke. Methods During a period of 9 months we consecutively contacted all surviving patients from the city of Trondheim who had been admitted with acute stroke to the stroke unit of the University Hospital of Trondheim 6 weeks or 6 months earlier. The Questionnaire Two weeks before the scheduled study visit a postal questionnaire was sent to the patients address. The questionnaire contained 2 simple questions which had earlier been evaluated in Great Britain [1 3], translated into Norwegian. The question, In the last 2 weeks did you require help from another person for everyday activites?, had been designed to capture the important qualitative difference between patients with good and bad functional outcome. The question, Do you feel that you have made a complete recovery from your stroke?, was aimed at assessing recovery. The exact Norwegian phrasing of the questions was: Har du i løpet av de to siste ukene hatt behov for hjelp for å klare dine personlige gjøremål?, and Føler du at du er kommet deg helt etter hjerneslaget?. The patients or their carers were asked to complete the questionnaire and to keep it in an enclosed, sealed envelope until the visit. The Interview The visits at 6 weeks and 6 months were performed by trained personnel in the patient s place of residence, and was part of the stroke unit s follow-up programme, which included the Scandinavian Stroke Scale and tests for various aspects of outcome. The interview was performed blind to the patient s replies to the questionnaire, and comprised the patient s replies to the simple questions asked by the visiting personnel, and the assessment of the Barthel ADL Index (BI) [9] and the modified Rankin Scale (mrs) [10]. To avoid interactions with BI and mrs the simple questions were asked at the beginning of the interview. Definitions of Good and Bad Functional Outcome, and Good and Bad Recovery The patients functional status were classified as good or bad dependent on whether they scored above or below the chosen cutoff levels on BI and mrs, representing the gold standards. The patients recoveries were similarly classified as good or bad according to the mrs. Some might argue that the patient (or his carer) provides the only true answer as to whether the patient has made a full recovery from his/her stroke, since there is no accepted scale to measure outcome relative to the patient s pre-stroke status. However, to provide a comparison with a standard outcome scale, we compared the answer to the recovery question with the mrs obtained at the interview. As the BI is the composite of selected activities of daily living (ADL) scores, and not a measure of overall outcome, it was not considered appropriate as a criterion for recovery. Patients with maximal score on the BI may still not have recovered completely, and this ceiling effect of the BI would have resulted in a low accuracy of the recovery question [11, 12]. Type and Measures of Validity We assessed the concurrent validity of the simple questions, defining the BI and mrs as gold standards [13]. The measures of sensitivity, specificity and accuracy were used to indicate the validity of the simple questions compared with the accepted standard scales [14]. Accuracy indicates the ability of the simple questions to predict whether a patient will score above or below a given cutoff on the BI and mrs. It is defined as the sum of the true-positive and true-negative test results divided by the total number of test results (fig. 1). In individual patients, the accuracy of the replies to simple questions may be defined as a binary variable with values 0 or 1, dependent on whether the reply agrees with the outcome category of BI or mrs. Planned Analyses and Statistical Methods Concurrent Validity. The answers to the 2 questions in the questionnaire were compared with outcome according to BI and mrs. Their accuracies in predicting good or bad outcome were determined for different cutoff levels on BI and mrs, to find the cutoff level that yielded the optimal accuracy. The data were entered in 2!2 contingency tables, and the sensitivity, specificity and accuracy estimated by a standardised method [14]. The answers to the questions obtained during the interview were validated against BI and mrs obtained at the same interview, as described for the questionnaire simple questions. To test whether the time of assessment (6 weeks or 6 months) influenced the accuracies the proportions of accurate answers at 6 weeks and 6 months were compared, using a test of paired proportions (McNemars test) [15]. Inter-Rater Reliability. The accuracy of the simple questions may be influenced by who completes the questionnaire (patient or carer). The functional status of the patient (good or bad) may also influence accuracy, as well as who completes the questionnaire (fig. 2). To control for the potential confounding effect of the patient s functional status, a stratified analysis was carried out, and the effect of the two factors on the accuracy were further explored in a logistic regression analysis [16]. In these analyses accuracy was defined as a binary variable (values 0 or 1 in individual patients), as described above. The agreement between the answers obtained by the interview and the questionnaire methods was determined using the Î statistics [17]. All analyses were performed by the statistical programme SPSS (version 9.0) or EPI Info (version 6.04c). 306 Cerebrovasc Dis 2001;11: Berge/Fjærtoft/Indredavik/Sandset

3 Fig. 1. Definition of sensitivity, specificity and accuracy. Fig. 2. Interrelationships determining the accuracy of the replies to the simple questions. A reply is accurate if the outcome category as determined by the question agrees with the criteria set by BI or mrs. Table 1. Sensitivity, specificity (with 95% confidence intervals) and accuracy of the response to the dependency question at six months compared with BI and mrs outcome categories Dependency question Barthel Index bad (! 95) good (695) Modified Rankin Scale bad (63) good (!3) Yes No sensitivity 85% (78 92) specificity 82% (74 89) accuracy 83% sensitivity 86% (79 93) specificity 79% (71 87) accuracy 82% Results Patients and Completeness of Data A total of 160 patients were contacted, but 10 of these had not filled in the questionnaire by the time of the interview and were not included in the study. Of the 150 patients who were included in the study, 49 were contacted at 6 weeks, 45 at 6 months, and 56 at both 6 weeks and 6 months. The mean age at stroke onset was 74 (range 45 96) years, and the estimated median mrs before the stroke was 0 (10 90%, percentiles 0 1). Median BI and mrs at both 6 weeks and 6 months were 95 (25 75%, percentiles ) and 2 (1 3), respectively. The questionnaires and interviews were 100% complete at both 6 weeks and 6 months. In all patients the functional status was stable without new events during the intermission period from receipt of the questionnaire until the interview visit. Concurrent Validity Questionnaire Method The results for the dependency question at 6 months are shown in table 1. The highest levels of accuracy were found by dichotomising BI at a score of 695 and mrs at! 3, and the obtained accuracies were high regardless of the chosen outcome scale. At 6 weeks the corresponding accuracies were 87 and 86% relative to BI and mrs, respectively. For the recovery question an optimal accuracy at 6 months was observed by dichotomising mrs at score 10 (table 2). It discriminated well between good and bad recovery, with accuracies of 86% at both 6 months and 6 weeks. Patients who could not complete the questionnaire themselves, or patients who no longer lived in their own homes were likely to have a bad outcome (data not shown). Interview Method The answers to the simple questions obtained during the interview at 6 months were validated against BI and mrs obtained at the same interview. The accuracies of the dependency question was 90 and 88%, according to BI (dichotomised at score 695) and mrs (score! 3), respectively. The optimal accuracy of the recovery question was 96% according to mrs (score 10). Simple Questions for Assessment of Stroke Outcome Cerebrovasc Dis 2001;11:

4 Table 2. Sensitivity, specificity (with 95% confidence intervals) and accuracy of the response to the recovery question at six months compared with mrs outcome categories Table 3. Accuracy of the dependency question at six months for patients or carers, compared to BI, in patients with good outcome (BI 695) Recovery question Modified Rankin Scale bad (10) good ( = 0) Accuracy of dependency question 1 (= present) 0 (= absent) Total No 75 2 Yes sensitivity 86% (80 93) specificity 83% (76 91) accuracy 86% Patient Carer Total OR = 5.25, 95 % CI Comparison of Validity of Questionnaire at 6 Months and 6 Weeks We compared the accuracy at 6 months and 6 weeks for the dependency question relative to BI (dichotomised at score 695), and the recovery question relative to mrs (score 10). There was no difference in the accuracy of the simple questions at 6 weeks compared with 6 months (p = 0.77 and p = 1.0 for the dependency and recovery questions, respectively). Inter-Rater Reliability Patients vs. Carers The accuracies of the simple questions depended on who completed the questionnaire (patient or carer) and the functional state of the patient (good or bad, as classified by BI or mrs). We also found that the functional state of the patient influenced who completed the questionnaires. To control for the potential confounding effect of this variable (fig. 2), a stratified analysis was performed, which showed that the effect of who completed the questionnaire was significantly modified by the patient s functional status ( 2 square test for evaluation of interaction, p = 0.01). Therefore, the results are described for good and bad functional groups separately. Patients with Good Outcome (i.e., BI 695 or mrs = 0, for the Dependency and Recovery Question, Respectively). We determined the accuracy of the dependency question (compared with the BI) and the recovery question (compared with the mrs) for carers and patients separately, and present a summary of the data for the dependency question to illustrate how the analysis was performed (table 3). Patients with a good outcome were more likely to complete the questionnaire themselves (50/54 in group with BI 695). The self-completed questionnaires were more accurate than those completed by carers (OR = Table 4. Accuracy of the dependency question at six months for patients or carers, compared to BI, in patients with bad outcome (BI! 95) Accuracy of dependency question 1 (= present) 0 (= absent) Carer Patient Total OR = 4.46, 95% CI Total 5.25, 95% CI ). As few carers answered for this category of patients, the overall effect of this difference was small. For example, the accuracy for carers answering the dependency questions for those with a good functional outcome was 50%. However, because carers only rarely answered for patients with good outcome, there was an overall accuracy of 81% in this group. For the recovery question, the difference in accuracy between patients and carers was even more clear (data not shown). Patients with Bad Outcome (i.e., BI!95 or mrs 10, for the Dependency and Recovery Question, Respectively). In this group the carers were more likely to complete the questionnaires (27/46; table 4). The stratified analysis showed that the carer-completed questionnaires were more accurate than those provided by the patients (OR = 4.46, 95% CI ). The effect of this difference was again small because the majority of these patients had the questionnaire completed by their carer. A similar difference was found for the recovery question (data not shown). 308 Cerebrovasc Dis 2001;11: Berge/Fjærtoft/Indredavik/Sandset

5 Questionnaire vs. Interview Method The agreement at 6 months between the answers to the dependency question obtained by the interview method and the questionnaire method was good, with a Î of 0.62 (95% CI ). For the recovery question, Î was 0.55 (95% CI ). Discussion The present study provides evidence that the simple dependency and recovery questions seem to be valid and reliable measures of stroke outcome in Norwegian patients tested 6 weeks or 6 months after the stroke. This finding is consistent with the evaluation performed among British stroke survivors after 6 months [1 3], and supports the continued use of these questions in different cultures and at different time intervals after stroke. Both evaluations have focused on concurrent validity, which involves comparison of the response to the simple questions with outcome defined by accepted gold standards (BI and mrs, or the equivalent Oxford Handicap Scale). Taken together, the studies show that the dependency and recovery questions, whether administered by questionnaire or interview, accurately predicts good or bad outcome as defined by BI or mrs, and good or bad recovery as defined by the mrs, respectively. For the assessment of recovery a high accuracy was achieved, even though the mrs is a measure of overall functional status, including disability not related to the stroke (e.g., due to arthritis or angina). The reason may be that nonstroke symptoms were a relatively unimportant cause of disability in our patients, as 120/150 (80%) patients had a pre-stroke mrs score estimate of 0, and 135/150 (90%) had a score estimate of 0 or 1. Alternatively, the patients answers to the recovery question may actually reflect their subjective judgement of their overall functional status, and not only the recovery relative to their pre-stroke status. For the validation of the dependency question the BI and mrs intuitively seem more relevant, and BI was originally developed as a measure of dependency [11]. The simple questions have previously been evaluated only after 6 months, and extrapolating the results to other stages of stroke rehabilitation may introduce bias that makes the simple questions inappropriate or even useless. The meaning of help from another person and complete recovery may be different for patients shortly after the stroke, as compared with 6 months later. In the present study there was no evidence that the accuracy is different at 6 weeks as compared with 6 months, which adds confidence in the simple questions as a widely applicable measure of stroke outcome. Our study confirmed the previous finding that the accuracies of the patients and his carers answers were dependent of the patient s outcome [1], although the differences in our study were smaller and statistically nonsignificant. Patients were more accurate than carers when they had a good outcome, carers more accurate than patients when the patient had a bad outcome. One explanation may be that help from another person (dependency question) and complete recovery (recovery question) have different interpretations for patients and carers, dependent on the functional state of the patient. Patients with a bad outcome may neglect their functional incapacity, i.e., underestimate the need for help and overestimate their recovery, or they may have cognitive impairment, in which case the carer will provide the most reliable answer [11, 18, 19]. On the other hand, patients with good outcome may have carers who are focused on the patients restrictions and have an idealised memory of the patients pre-stroke status, while the patients themselves give the most accurate answer [20]. However, the overall effect of the difference in accuracy between patients and carers was small, as patients with good outcome tended to answer themselves, while carers tended to answer on behalf of patients with bad outcome. In both outcome groups the majority of the patients were assessed by the most accurate method. In addition, an acceptable interrater reliability was also demonstrated by the level of agreement between the replies obtained in the questionnaire and in the interview. In conclusion, our study shows that the simple questions seem to be valid and reliable measures of stroke outcome in Norwegian patients tested after 6 weeks or 6 months, which supports their continued use in large-scale multinational stroke studies at different time intervals after stroke. Acknowledgements The study was executed with grants from the Norwegian Council on Cardiovascular Diseases and the Norwegian Research Council. We are grateful to R. Lindley, M. Dennis, and P. Sandercock for their advice during the design stage of the study. We are especially grateful to T. Bruun Wyller for his valuable suggestions during the preparation of the manuscript. Simple Questions for Assessment of Stroke Outcome Cerebrovasc Dis 2001;11:

6 References 1 Lindley RI, Waddell F, Livingstone M, Sandercock P, Dennis M, Slattery J, Smith B, Warlow C: Can simple questions assess outcome after stroke? Cerebrovasc Dis 1994;4: Dennis M, Wellwood I, Warlow C: Are simple questions a valid measure of outcome after stroke? Cerebrovasc Dis 1997;7: Dennis M, Wellwood I, O Rourke S, Machale S, Warlow C: How reliable are simple questions in assessing outcome after stroke? Cerebrovasc Dis 1997;7: International Stroke Trial Collaborative Group: The International Stroke Trial (IST): A randomised trial of aspirin, subcutaneous heparin, both, or neither among 19,435 patients with acute ischaemic stroke. Lancet 1997;349: CAST (Chinese Acute Stroke Trial) Collaborative Group: CAST: Randomised placebo-controlled trial of early aspirin use in 20,000 patients with acute ischaemic stroke. Lancet 1997;349: Kay R, Wong KS, Yu YL, Chan YW, Tsoi TH, Ahuja AT, Chan FL, Fong KY, Law CB, Wong A, et al: Low-molecular-weight heparin for the treatment of acute ischemic stroke. N Engl J Med 1995;333: Hommell M: Fraxiparine in Ischaemic Stroke Study (FISS bis) (abstract). Cerebrovasc Dis 1998;8(suppl 4):19. 8 Berge E, Abdelnoor M, Nakstad PH, Sandset PM: Low molecular-weight heparin versus aspirin in patients with acute ischaemic stroke and atrial fibrillation: A double-blind randomised study. Lancet 2000;355: Mahoney FI, Barthel DW: Functional evaluation: The Barthel index. Md State Med J 1965; 14: van Swieten JC, Koudstaal PJ, Visser MC, Schouten HJ, van Gijn J: Interobserver agreement for the assessment of handicap in stroke patients. Stroke 1988;19: Wade DT: Measurement in Neurological Rehabilitation. New York, Oxford University Press, Wellwood I, Dennis MS, Warlow CP: A comparison of the Barthel Index and the OPCS disability instrument used to measure outcome after acute stroke. Age Ageing 1995;24: Hobart JC, Lamping DL, Thompson AJ: Evaluating neurological outcome measures: The bare essentials. J Neurol Neurosurg Psychiatry 1996;60: Fletcher RH, Fletcher SW, Wagner EH: Clinical Epidemiology. The Essentials, ed 3. Baltimore, Williams & Wilkins, Altman DG: Practical statistics for medical research, ed 1. London, Chapman & Hall, Kleinbaum DG, Kupper LL, Morgenstern H: Epidemiologic Research. Principles and Quantitative Methods. New York, Van Nostrand Reinhold, Brennan P, Silman A: Statistical methods for assessing observer variability in clinical measures. BMJ 1992;304: Ranhoff AH, Laake K: The Barthel ADL Index: Scoring by the physician from patient interview is not reliable. Age Ageing 1993;22: Wyller TB, Sveen U, Bautz-Holter E: The Frenchay Activities Index in stroke patients: Agreement between scores by patients and by relatives. Disabil Rehabil 1996;18: Magaziner J, Bassett SS, Hebel JR, Gruber- Baldini A: Use of proxies to measure health and functional status in epidemiologic studies of community-dwelling women aged 65 years and older. Am J Epidemiol 1996;143: Cerebrovasc Dis 2001;11: Berge/Fjærtoft/Indredavik/Sandset