Original Article 315 Diagnostic Performance of Short Portable Mental Status Questionnaire for Screening Dementia Among Patients Attending Cognitive Assessment Clinics in Singapore Chetna Malhotra, 1 MBBS, MD, MPH, Angelique Chan, 1 PhD, David Matchar, 1 MD, MPH, Dennis Seow, 2 MRCP, FRCP, FAMS, Adeline Chuo, 3 MBBS, MRCP, FAMS, Young Kyung Do, 1 MD, MPH, PhD Abstract Introduction: The Short Portable Mental Status Questionnaire (SPMSQ) is a brief cognitive screening instrument, which is easy to use by a healthcare worker with little training. However, the validity of this instrument has not been established in Singapore. Thus, the primary aim of this study was to determine the diagnostic performance of SPMSQ for screening dementia among patients attending outpatient cognitive assessment clinics and to assess whether the appropriate cut-off score varies by patient s age and education. A secondary aim of the study was to map the SPMSQ scores with Mini-Mental State Examination (MMSE) scores. Materials and Methods: SPMSQ and MMSE were administered by a trained interviewer to 127 patients visiting outpatient cognitive assessment clinics at the Singapore General Hospital, Changi General Hospital and Tan Tock Seng Hospital. The geriatricians at these clinics then diagnosed these patients with dementia or no dementia (reference standard). Sensitivity and specificity of SPMSQ with different cut-off points (number of errors) were calculated and compared to the reference standard using the Receiver Operator Characteristic (ROC) analysis. Correlation coefficient was also calculated between MMSE and SPMSQ scores. Results: Based on the ROC analysis and a balance of sensitivity and specificity, the appropriate cut-off for SPMSQ was found to be 5 or more errors (sensitivity 78%, specificity 75%). The cut-off varied by education, but not by patient s age. There was a high correlation between SPMSQ and MMSE scores (r = 0.814, P <0.0001). Conclusion: Despite the advantage of being a brief screening instrument for dementia, the use of SPMSQ is limited by its low sensitivity and specificity, especially among patients with less than 6 years of education. Ann Acad Med Singapore 2013;42:315-9 Key words: Cognitive, Dementia, SPMSQ, Validation Introduction Healthcare workers and researchers have widely acknowledged the need for a brief cognitive screening instrument for the diagnosis of dementia. While many physicians use the Mini-Mental State Examination (MMSE), it is considered too long and too difficult to interpret, especially by junior healthcare workers. The Short Portable Mental Status Questionnaire (SPMSQ) is a widely used 10-item cognitive screening instrument, whose items test orientation to time and place, memory, current event information (date, day of the week, name of this place, telephone number, date of birth, age, name of current prime minister and previous prime minister, mother s maiden name), and calculation (subtract 3s starting with number 20). The total number of errors is computed and it ranges from 0 to 10. 1 It was first validated by Pfeiffer in a sample of 141 clinic patients in which SPMSQ scores were found to be correlated with clinical diagnosis of organic brain. 1 The validity and reliability of SPMSQ was also subsequently established in other studies. 2-6 These studies have reported a high specificity, but lower sensitivity of SPMSQ. For instance, Erkinjuntti et al found the sensitivity and specificity of SPMSQ to be 86.2% and 99.0%, respectively, among community and medical inpatient samples of Finnish elderly, using the cut-off point of 3 or more errors. 2 A study by Fillenbaum reported 55% sensitivity and 96% specificity for SPMSQ, 5 while another by Albert et al, found the sensitivity and specificity of 1 Program in Health Services and Systems Research, Duke-NUS Graduate Medical School 2 Geriatric Medicine, Singapore General Hospital 3 Geriatric Medicine, Changi General Hospital Address for Correspondence: Dr Young Kyung Do, Program in Health Services and Systems Research, Duke-NUS Graduate Medical School, 8 College Road, Singapore 169857. Email: young.do@duke-nus.edu.sg
316 SPMSQ to be 34.4% and 94.3%, respectively. 4 A Spanish language validation of SPMSQ reported its sensitivity and specificity as 85.7% and 79.3% respectively, with a cut-off of 3 or more errors. 6 In contrast, in a study by Dalton et al, SPMSQ was not found to be significantly related to either clinical or neuropsychological diagnosis. 3 While the validity of SPMSQ has been established in many settings, it has never been used in Singapore. The majority of Singapore s population is Chinese (74.1%), with a smaller proportion of Malays (13.4%) and Indians (9.2%). 7 It has a rapidly ageing population, with the proportion of older adults 60 years being 8% and projected to increase to 19% by 2030. 8 The prevalence of dementia among older adults aged 60 years is currently 5.2% and is likely to increase as the population ages. 9 While a brief instrument such as SPMSQ for screening cognitive secondary to dementia among people of different ethnic groups will be useful, its scores will be considered meaningful only if its diagnostic performance has been systematically evaluated against standard clinical diagnosis. Thus, the primary aim of this study was to determine the diagnostic performance of SPMSQ for screening dementia among patients attending outpatient cognitive assessment clinics and assess whether the appropriate cut-off score varies by patient s age and education. A secondary aim of the study was to map the SPMSQ scores with MMSE scores; this could be useful as it would enhance our ability to compare results across studies. Materials and Methods Data Collection Data collection was conducted between December 2010 and November 2011. All patients attending cognitive assessment clinics were invited by their treating physicians to participate in this study. A total of 127 patients visiting outpatient cognitive assessment clinics at the Singapore General Hospital (n = 42), Changi General Hospital (n = 44) and Tan Tock Seng Hospital (n = 41), consented to participate and were administered the SPMSQ and MMSE by a trained interviewer. SPMSQ was administered in English or Chinese based on participants preference. The Chinese version was back-translated to ensure accuracy. The geriatricians at these clinics then diagnosed these patients with dementia or without dementia after a thorough work up which included taking patient history, physical examination, psychometric assessment, laboratory and neuroimaging investigations; this clinical diagnosis served as the reference standard. The geriatrician and clinic nurse were blinded to the SPMSQ scores. Statistical Analysis Unpaired t-test was used to compare the number of errors on SPMSQ among patients with and without a clinical diagnosis of dementia. Then, sensitivity and specificity of SPMSQ with different cut-off points (number of errors) were calculated and compared to the reference standard. Receiver Operating Characteristic (ROC) analysis was conducted and area under the curve (AUC) was calculated. During the ROC analysis, we compared the sensitivity and specificity of SPMSQ at different cut-off points. The cut-off point chosen was the one that provided the best balance of sensitivity and specificity. The ROC analysis was then repeated for subgroups of patients in the 2 education categories (<6 years vs 6 years), to determine whether the optimal cut-off value for SPMSQ changed by education. The correlation coefficient was also calculated between the MMSE and SPMSQ scores. The pre-defined categories of SPMSQ (intact, <3 errors; mild, 3 to 4 errors; moderate and severe, 5 or more errors) 1 and MMSE scores (intact, score 24; mild, score 18 to 23; severe, score 17) 10 were cross-tabulated to assess the extent of agreement between the cognitive categories determined using the 2 scoring scales. Results Demographics The majority of the patients were 75 years of age (78.7%), female (69.3%), had <6 years of education (59.8%), were Chinese (90.6%) and had dementia (81.1%) (Table 1). The distribution of SPMSQ scores for patients without dementia (cognitively intact/ mild cognitive ) and with Table 1. Demographic Characteristics of Study Participants (n = 127) Characteristics n (%) Age (60 to 94 years) 60 to 74 27 (21.3) 75 100 (78.7) Gender Male 39 (30.7) Female 88 (69.3) Education <6 years 76 (59.8) 6 years 51 (40.2) Ethnicity Chinese 115 (90.6) Malays, Indians, and Others 12 (9.4) Diagnosis Intact cognition 9 (7.1) 15 (11.8) Dementia 103 (81.1) Annals Academy of Medicine
317 Table 2. Distribution of Short Portable Mental Status Questionnaire Scores Total sample Education Age <6 years 6 to 12 years 60 to 74 years old 75 years old Intact cognition/ Mean number of errors ± SD 3.2 ± 2.5 (n = 24) 5.4 ± 2.8 (n = 7) 2.2 ± 1.8 (n = 17) 3.5 ± 4.0 (n = 4) 3.1 ± 2.3 (n = 20) Dementia Mean number of errors ± SD 6.3 ± 2.4 (n = 103) 6.8 ± 2.4 (n = 69) 5.4 ± 2.3 (n = 34) 6.3 ± 2.9 (n = 23) 6.3 ± 2.3 (n = 80) P values <0.0001 0.73 dementia (mild/moderate/severe) is shown in Table 2. Table 2 also shows that SPMSQ scores varied significantly by level of education (<6 years vs 6 years), but not by age (60 to 74 years vs 75 years). Optimal Cut-off Values The AUC for the overall sample was 0.81. For those with <6 years of education, it was 0.64 and for those with 6 years of education, it was 0.87. Based on the ROC analysis and a balance of sensitivity and specificity, the appropriate cut-off point was found to be 5 or more errors (78% sensitivity, 75% specificity) (Fig. 1). For those with <6 years of education, the appropriate balanced cut-off was 6 or more errors (72% sensitivity, 43% specificity). Among those with 6 years of education, the appropriate balanced cut-off was 4 or more errors (79% sensitivity, 76% specificity) (Figures not shown). Mapping with MMSE We found a high correlation between SPMSQ and MMSE scores (r = 0.814, P <0.0001). The mean number of errors on SPMSQ among patients categorised as having intact cognition with MMSE (score 24) was 2.3 ± 2.0, among those defined as having mild cognitive (MMSE score 18 to 23) was 4.6 ± 1.9 and among those with severe cognitive (MMSE score 17) was 7.5 ± 1.9. Cross-tabulation of cognitive categories using MMSE (intact, mild, severe) and original SPMSQ categories (intact, <3 errors; mild, 3 to 4 errors; moderate and severe, 5 or more errors) showed an overall 70% agreement between the categories defined by the 2 scores. However, 21 of the 41 patients (51.2%) determined as having mild cognitive using MMSE, were classified as having moderate/severe cognitive on SPMSQ (Table 3). The kappa statistic (i.e. the agreement between 2 scales above and beyond that expected by chance alone) was 0.47 (P <0.0001), suggesting moderate agreement. Discussion Previous studies have reported a high specificity and sensitivity of SPMSQ using a cut-off of 3 or more errors for classifying cognitive /dementia. 2,6 Using a similar cut-off for our study, we get a high sensitivity (95.1%), but a much lower specificity (45.8%). On the other hand, a cut-off of 5 or more errors as recommended by Pfeiffer for moderate to severe cognitive Fig. 1. Receiver Operating Characteristic (ROC) curve for the overall sample. Table 3. Cross-tabulation of Mini-Mental Status Examination and Short Portable Mental Status Questionnaire Categories Short Portable Mental Status Questionnaire Intact cognition (n = 16) (n = 25) Moderate/ severe cognitive (n = 86) Intact cognition (n = 21) Mini-Mental Status Examination (n = 41) Severe cognitive (n = 65) 12 4 0 5 16 4 4 21 61
318 provides a good balance of sensitivity (78%) and specificity (75%), which indicates that this value is the more balanced cut-off point for screening cognitive secondary to dementia. 1 The low sensitivity and specificity of SPMSQ at a cut-off point of 5 may be because SPMSQ is limited in the cognitive domains assessed. Thus, while SPMSQ has the advantage of being a brief screening instrument for dementia, the low sensitivity and specificity may limit its ability to be used alone in certain clinical settings, where a more comprehensive test assessing multiple cognitive domains may be more useful. On the other hand, in situations where screening will be followed by a more detailed cognitive assessment, a cut-off point (e.g. 3 or more errors) with high sensitivity, but low specificity may also be acceptable. In addition, the study results demonstrate that optimal number of errors on SPMSQ for screening dementia varies by education. Several previous studies have also shown that low education level increases the likelihood of cognitively intact people being misclassified as having dementia (i.e. false positives). On the other hand, there are likely false negative cases among those with high education level. 11,12 Thus, educational level needs to be taken into account while using SPMSQ. Similar to the original SPMSQ scoring, 1 we propose that those with less than 6 years of education should be allowed one more error, and those with 6 or more years of education should be allowed one less error. The low specificity of SPMSQ at the proposed cut-off point among those with less than 6 years of education, suggests that it should be used with caution in this group of population and if possible, limit its use among those with 6 or more years of education. SPMSQ s correlation with education thus limits its use in current clinical settings where most patients have lower levels of education, although its potential as a screening tool may improve with a better-educated ageing population. We also found SPMSQ to be correlated with MMSE scores, possibly because of several common items across the 2 scales. However, the cross-tabulation of MMSE and SPMSQ scores suggested that SPMSQ is likely to assign about 51% of the individuals with mild cognitive by MMSE as having moderate/severe cognitive. This should be kept in mind while comparing prevalence estimates from SPMSQ and MMSE across different studies/settings. The main strength of this study is that it is a multicentre study conducted in 3 cognitive assessment clinics in Singapore. We included dementias of any aetiology and of all severities (mild, moderate, severe) in this study because a screening instrument with good validity should be able to pick up dementias with mixed aetiology and severity. A main limitation of this study is that it has been conducted only among patients visiting cognitive assessment clinics. As a result, there were only a few people with intact cognition in the sample. Future attempts to validate this instrument should include a cognitively intact sample from the community. Moreover, the majority of our patients were Chinese, thus limiting the external validity of the instrument for dementia screening among other ethnic groups, notably Malays and Indians. No information was available for patients who did not consent to participate. Thus, we are unable to say whether those patients who participated were different from those who did not. Conclusion In conclusion, SPMSQ can distinguish between patients with dementia and those without, especially among those with 6 or more years of education. To maximise sensitivity and specificity, a cut-off point of 5 or more errors is suggested for screening patients with dementia. However, the cut-off needs to be adjusted depending on the educational level of the patient. Acknowledgements The study was funded by a grant through the National University of Singapore Virtual Institute of the Study of Ageing by the Ministry of Community Development Youth and Sports (Grant number: VG3). We acknowledge Dr Mei Sian Chong from Cognition and Memory Disorders Services, Tan Tock Seng Hospital for her contribution to the study. We thank Ms Karen Seah for conducting the data collection for this study. Annals Academy of Medicine
319 REFERENCES 1. Pfeiffer E. A short portable mental status questionnaire for the assessment of organic brain deficit in elderly patients. J Am Geriatr Soc 1975;23:433-41. 2. Erkinjuntti T, Sulkava R, Wikstrom J, Autio L. Short Portable Mental Status Questionnaire as a screening test for dementia and delirium among the elderly. J Am Geriatr Soc 1987;35:412-6. 3. Dalton JE, Pederson SL, Blom BE, Holmes NR. Diagnostic errors using the Short Portable Mental Status Questionnaire with a mixed clinical population. J Gerontol 1987;42:512-4. 4. Albert M, Smith LA, Scherr PA, Taylor JO, Evans DA, Funkenstein HH. Use of brief cognitive tests to identify individuals in the community with clinically diagnosed Alzheimer's disease. Int J Neurosci 1991;57:167-78. 5. Fillenbaum GG, Landerman LR, Simonsick EM. Equivalence of two screens of cognitive functioning: the Short Portable Mental Status Questionnaire and the Orientation-Memory-Concentration test. J Am Geriatr Soc 1998;46:1512-8. 6. Martinez de la Iglesia J, Duenas Herrero R, Onis Vilches MC, Aguado Taberne C, Albert Colomer C, Luque Luque R. [Spanish language adaptation and validation of the Pfeiffer's questionnaire (SPMSQ) to detect cognitive deterioration in people over 65 years of age]. Med Clin (Barc) 2001;117:129-34. 7. Department of Statistics Singapore. Monthly digest of Statistics Singpore, February 2012. 2012. Available at: http://www.singstat.gov.sg/stats/ themes/people/demo.html. Accessed 6 March 2012. 8. Ministry of Community Development Youth and Sports. Committee on Ageing Issues: Report on the Ageing Population 2006. 9. Chiam PC, Ng TP, Tan LL, Ong PS, Ang A, Kua EH. Prevalence of dementia in Singapore--results of the National Mental Health Survey of the Elderly 2003. Ann Acad Med Singapore 2004;33(5 Suppl):S14-15. 10. Tombaugh TN, McIntyre NJ. The mini-mental state examination: a comprehensive review. J Am Geriatr Soc 1992;40:922-35. 11. Sahadevan S, Lim PP, Tan NJ, Chan SP. Diagnostic performance of two mental status tests in the older chinese: influence of education and age on cut-off values. Int J Geriatr Psychiatry 2000;15:234-41. 12. Lorentz WJ, Scanlan JM, Borson S. Brief screening tests for dementia. Can J Psychiatry 2002;47:723-33.