The Psychometric Qualities of Four Observational Pain Tools (OPTs) for the Assessment of Pain in Elderly People with Osteoarthritic Pain

Transcription

1 582 Journal of Pain and Symptom Management Vol. 40 No. 4 October 2010 Original Article The Psychometric Qualities of Four Observational Pain Tools (OPTs) for the Assessment of Pain in Elderly People with Osteoarthritic Pain Justina Y.W. Liu, RN, PhD, Michelle Briggs, PhD, and S. José Closs, PhD School of Nursing (J.Y.W.L.), The Hong Kong Polytechnic University, Hung Hum, Hong Kong; and School of Healthcare (S.J.C., M.B.), University of Leeds, Leeds, United Kingdom Abstract Context. Pain in cognitively impaired elderly people (CIEP) often goes unrecognized. Observational pain tools (OPTs) have been designed, but with limited evidence to support their psychometric qualities. Objectives. This study compared four OPTs (the Pain Assessment IN Advanced Dementia [PAINAD], Abbey Pain Scale [Abbey PS], Pain Assessment Checklist for Seniors with Limited Ability to Communicate [PACSLAC], and Discomfort ScaledDementia of Alzheimer Type [DS-DAT]), two self-report scales, and two proxy-report scales in assessing osteoarthritic (OA) pain among CIEP. Methods. Participants (n ¼ 124) were divided into two groups: cognitively intact and impaired. They were observed by two raters simultaneously at rest and during a standardized exercise program. Besides reliabilities, the correlation between the OPTs and the self-report/proxy-report scores was evaluated. The OPT scores collected during different activity levels were compared to establish the convergent and discriminant validity. Confirmatory factor analysis was used to evaluate the construct validity. Results. Similar and accepted patterns of reliability/validity were obtained for all OPTs, in which better levels of psychometric properties were consistently obtained during exercise. However, a single construct (OA pain) appeared only in the PAINAD and Abbey PS after deletion of the breathing and physiological change indicators, respectively. This showed that OPTs were better used to detect OA pain when pain was triggered by movement (i.e., an exercise program). Conclusion. The PAINAD and Abbey PS appeared to be more reliable and valid for assessing OA pain while using an exercise program among elderly people, regardless of their cognitive ability. J Pain Symptom Manage 2010;40:582e598. Ó 2010 U.S. Cancer Pain Relief Committee. Published by Elsevier Inc. All rights reserved. Address correspondence to: Justina Y.W. Liu, RN, PhD, School of Nursing, The Hong Kong Polytechnic University, Hung Hum, Kowloon, Hong Kong. hsjustina@inet.polyu.edu.hk Ó 2010 U.S. Cancer Pain Relief Committee Published by Elsevier Inc. All rights reserved. Accepted for publication: February 9, /$ - see front matter doi: /j.jpainsymman

2 Vol. 40 No. 4 October 2010 The Psychometric Qualities of Four Observational Pain Tools 583 Key Words Pain behavior, observational pain tool, cognitive impairment, elderly, pain assessment, psychometric, nursing homes Introduction Because of the complex and subjective nature of pain, accurate pain assessment for elderly people, especially for those who are cognitively impaired, is a major obstacle to successful pain management. During the last decade, many researchers have developed observational methods of assessing pain among cognitively impaired elderly people (CIEP). 1,2 However, self-reporting and proxy-reporting methods are still the most common assessment methods for CIEP. 3e5 Completion rates of CIEP on self-report scales have been varied, ranging from 7% to 100% 6e10 and decreasing with participants impaired levels of cognition. Among different pain scales, the verbal rating scale (VRS) has consistently had the highest completion rates, whereas the visual analog scale (VAS) has consistently had the lowest. However, it cannot be assumed that CIEP can correctly use the scales purely based on the scale completion rates. When correlation coefficients (r) between different pain scales were calculated, strong and significant correlations (r ¼ 0.5e0.77) could be identified among pain scales when they were used by elderly people with no to moderate cognitive impairment. However, there was no correlation among pain scales when they were used by those with severe cognitive impairment. 7 The significant correlations among pain scales suggested that the pain scales were measuring the same construct when used among those with no to mild impairment. In contrast, the lack of correlation among pain scales used by severely CIEP suggests that this group of people is unable to comprehend the concept of quantifying pain to a scale item, leading to poor consistency between pain scales. Hence, steps should be included to explore CIEP s comprehension of the selfreporting scales before using these scales for pain assessment. Although many CIEP can give pain scores, it has been shown that almost one-fifth of severely CIEP were unable to use any scale. 8 An alternative to self-report for CIEP is commonly proxy-report by a caregiver. These include different types of pain intensity scales that were originally designed for self-report. Literature has shown that proxy ratings may be of some value when dealing with noncommunicative patients, although health care professionals tend to underestimate the presence of pain and there may be poor interrater reliability. 9 Although there may be a reasonable level of agreement (70%) between nursing staff and patient ratings in identifying the presence of pain, estimates of pain intensity may be poor when compared with the selfreport. 9,10 Because CIEPs pain experiences may not be fully reflected by self- or proxy-report, observational pain tools (OPTs) have recently been developed for this special group of people. Their structure relies on common pain-related behavioral indicators, for example, facial expressions, body movements, verbalizations, vocalizations, physiological changes, emotional changes, and changing patterns of activities of daily living (ADL). By observing for the presence or absence of the behavioral indicators, an observer identifies the likely presence of pain in a patient. Appraisals of OPTs have been based mainly on their psychometric properties, rather than comparing them directly in a clinical setting. 11e15 It is unclear which OPTs should be used for pain assessment among CIEP, because all appraisals point out that the OPTs are still under development and show only moderate psychometric qualities. Therefore, further validation of the OPTs in different clinical settings is necessary. The limitations of previous validating studies include insufficient sample size to represent the target group, 16 unsatisfactory methodology to avoid expectation bias from raters, 17 incomplete psychometric properties, and a lack of effort to investigate how various levels of cognitive impairment affect the psychometric properties of the OPTs. Additionally, most validating studies have not clearly

3 584 Liu et al. Vol. 40 No. 4 October 2010 defined the types of pain that the OPTs are intended to measure. It is argued that most OPTs poorly defined measure construct leads to inadequate strategies to control for samples type of pain. This may explain the relatively low OPT scores reported by previous studies. 16 This study, therefore, aimed to evaluate the psychometric properties of four OPTs (the Pain Assessment IN Advanced Dementia [PAI- NAD] Scale, the Abbey Pain Scale [Abbey PS], the Pain Assessment Checklist for Seniors with Limited Ability to Communicate [PACSLAC], and the Discomfort ScaledDementia of Alzheimer Type [DS-DAT]) in a real clinical setting among nursing home residents with osteoarthritic (OA) pain, using a standardized exercise program to trigger OA pain. Research questions included: 1. Are the four OPTs reliable in terms of internal consistency and interrater reliability? 2. Are the four OPTs able to discriminate between a nonpain event (rest) and OA pain triggered by a standardized exercise program? 3. Do the four OPTs correlate well with one another, the two self-report pain scales and the two proxy-report pain scales? 4. What factors are identified in the four OPTs when evaluated by confirmatory factor analysis (CFA)? 5. Are the psychometric qualities of the four OPTs similar when they are used to assess OA pain among elderly people with or without cognitive impairment? 6. To what extent do participants with and without cognitive impairment understand the self-reported pain scales? Methods An observational study was used to answer the research questions. Raters A, B, and C observed participants behavioral responses by using the four OPTs simultaneously and independently during two conditions: at rest and during a standardized exercise program. Observation of rest lasted for approximately five minutes, whereas standardized exercise lasted approximately 20e30 minutes. Both rest and exercise were observed in the same morning by three raters. Pain rating on the OPTs was performed after the completion of each observation. Rater A completed all four OPTs for rest and exercise, whereas Rater B completed two of four OPTs for rest and the remaining two OPTs for exercise. Rater C performed a similar procedure to Rater B but with a reverse order of the OPTs. The OPTs used by Raters B and C were selected randomly. The sequence of the administration of the OPTs was selected randomly for all the raters to minimize the possible bias because of order effects. This special arrangement was implemented to minimize possible raters expectation bias that might have been produced by the same rater completing the same OPTs for rest and exercise. Thus, the interrater reliability was established by comparing the OPT scores collected by Rater A to those of either Rater B or C depending on the sequence in which the OPTs were completed. This study hypothesized that levels of pain during the standardized exercise program would be higher than when participants were at rest, and that they should exhibit more pain-induced behaviors during exercise. Rater A, the principle investigator of this study, was the only rater aware of the hypothesized higher pain levels during the exercise program. To avoid the possible expectation bias produced by Rater A, the hypothesis test was calculated based on OPT scores obtained by Raters B and C during rest and exercise. These two raters were unaware of the hypothesis and thus, together with the special arrangements for pain score comparison, the expectation bias by the raters was minimized. Another research assistant, who was not involved in the administration of the OPTs, asked the participants to rate their current pain on two self-reporting scales (i.e., a fivepoint VRS and the Wong-Baker FACES Pain Scale) after each observational rating. This research assistant also asked the participants direct contact nurses to rate the pain levels of their clients on two proxy-reported pain scales (i.e., the proxy-vrs and the proxy- FACES). Standardized Exercise Program to Trigger OA Pain All participants were confirmed as having OA pain by screening the medical notes. OA pain is often associated with joint activity and relieved by rest. 18e20 Following literature review, a set of passive range of motion (ROM) exercises was designed, which included all

4 Vol. 40 No. 4 October 2010 The Psychometric Qualities of Four Observational Pain Tools 585 major joints. The participants were then requested to do a defined set of active motions that progressed from lying to sitting, to standing, and then to a walking exercise. These active motions have been shown to be successful in triggering pain-related behavioral responses. 21,22 Participants A convenience sample of 124 residents was recruited from 14 nursing homes in Hong Kong (HK). Residents had to be 65þ years of age, of either gender, and living in the nursing home for more than three months before data collection. They must have been diagnosed as having osteoarthritis for at least six months to ensure that this was a chronic condition. Residents were excluded if: 1) they had any conditions known to influence OA pain sensation, for example, cancer pain or acute wound pain; 2) they had severely impaired hearing and sight that may have affected their ability to use the self-report scales; and 3) they were experiencing any distressing social circumstances, for example, the recent death of a close relative. Residents who met all the inclusion criteria but who had other kinds of chronic illnesses were still recruited into this study, providing the illnesses were neither acute nor likely to affect their pain sensation. Measurement A data sheet was used to collect demographic information. The medical history, such as medications and dementia diagnosis, was obtained from participants medical and nursing records. Participants cognitive status was evaluated using the Cantonese version of the Mini-Mental State Examination (C-MMSE). 23 The C-MMSE was adapted from the original MMSE 24 and was found to have good psychometric qualities in the identification of cognitive impairment in Chinese patients in HK. 23 In this study, a C-MMSE score of 18 or less was used to identify cognitive impairment. Pain was assessed with four English-version OPTs: the PAINAD, the Abbey PS, the PAC- SLAC, and the DS-DAT (Table 1). The selection of these four OPTs was based on the critical appraisal of the 12 OPTs, which is described in detail elsewhere. 25 These four OPTs received high scores in the appraisal and were chosen for direct comparison in a clinical setting. All original OPT creators gave permission to use the OPTs and were consulted about the practical use of each OPT. The PAINAD consists of five pain behaviors: breathing, negative vocalization, facial expressions, body language, and consolability. Each behavior is rated from 0 to 2 according to the surgery of behavior exhibited, thus giving a total score of The PAINAD was tested on different groups of patients, including posthip surgery elderly patients 26 and nursing home residents. 16,27,28 The findings showed that the PAINAD could distinguish pain from other pain-free conditions. 16 Its correlation with other pain tools varied from 0.34 to 0.91, but consistently strong interrater reliability and internal consistency was obtained in various studies. 16,26e28 Generally, preliminary data provide support for validity and reliability in long-term care and acute settings. However, Table 1 Behavioral Items of Each OPT Behavioral Items PAINAD Abbey PS PACSLAC DS-DAT Facial expressions (e.g., Grimacing, frowning, etc.) 4 (e.g., Sad, frowning, etc.) Body movements (e.g., Fidgeting, flinching, etc.) 3 (e.g., Fidgeting, tense) Vocalizations and (e.g., Mumbling, grunting, etc.) 1 verbalizations Physiological changes 1 (Breathing 1 6 (e.g., Pale face, flushed) 1 (Noisy breathing) patterns) Emotional changes 15 (e.g., Angry, upset, etc.) Changing patterns of ADL 1 2 (e.g., Change in sleep, change in appetite) Other items Consolability Physical changes Total items

5 586 Liu et al. Vol. 40 No. 4 October 2010 research is still needed to determine whether similar results can be obtained from other settings. The Abbey PS includes six pain behaviors, each rated on a 0e3 point scale for the severity of the behavior, with a total score ranging from 0 to 18. The total score is interpreted as severity of pain: 0e2 ¼ no pain; 3e7 ¼ mild pain; 8e13 ¼ moderate pain; and 14þ ¼severe pain. 17 However, this interpretation has been criticized as lacking the support of empirical evidence. 12 In the original design, the rater is asked to indicate which type of pain the elderly person has: chronic, acute, or acute on chronic. This item was deleted, as this study recruited only participants with OA pain. The Abbey PS was able to detect the change of pain level before and after pain-relieving interventions and had a moderate level of correlation with nurses proxy-pain scores. 17 With the exception of internal consistency (a ¼ 0.84), other types of reliability have not yet been tested. 17 The Abbey PS is an undervalidated OPT with limited available psychometric findings, although it is recommended by the Australian Pain Society 29 and the British Geriatrics Society. 30 Thus, its psychometric qualities warrant further exploration. The PACSLAC is a checklist with 60 behavioral indicators divided into four subscales (i.e., facial expression; activity/body movements; social/personality/mood; and physiological indicators/eating and sleeping changes/vocal behavior). Each indicator is scored on a dichotomous scale. Subscale scores are added to arrive at a total score ranging from 0 to 60. In a retrospective study, the PACSLAC was able to distinguish pain events from distressing or calm events. It was moderately correlated with a proxy 10-point scale. High levels of internal consistency were found for the total scores, although the Cronbach s alpha for the subscales was moderate to lower. Therefore, the OPT creators recommended use of the total score only. 31 In a prospective study, the PACSLAC showed good levels of interrater and intrarater reliability. It correlated well with proxy-vas and self-reported VRS scores. 28 These correlations were better than those collected in the retrospective study. There are contrasting opinions about the appropriateness of the length of the PACSLAC. The creator of this OPT suggested that a long OPT was better than a short one, as a comprehensive tool is more sensitive in detecting pain, especially when pain behaviors are diverse and individualized. Thus, a short OPT may be unable to identify pain among people exhibiting specific pain-related behavior. 2 However, another study reported that 28 indicators in the PACSLAC were not observed for over 90% of the participants. 28 This shows that some indicators may be redundant and refinement may be needed. The DS-DAT is a nine-item observational scale for assessing general discomfort. Besides common pain-related behaviors, it also includes two items with negative meanings regarding discomfort (i.e., content facial expression and relaxed body language ). A rater scores each item on a scale of 0e3, based on the frequency, intensity, and duration of each scale item. Scores are summed, resulting in a possible range from 0 to When febrile episodes were defined as discomfort situations, this OPT was able to detect significant differences between discomfort and nondiscomfort situations. It also demonstrated acceptable levels of interrater reliability, test-retest reliability, and internal constancy. 32e34 Although, conceptually, discomfort encompasses pain, discomfort also includes symptoms that may not be an expression of pain, such as fever. In many circumstances, the terms pain and discomfort are used interchangeably. However, no conclusive result as to whether the DS-DAT could retain its psychometric quality was used for chronic pain assessment. It is worth evaluating the appropriateness of using the DS-DAT to identify pain alone. Two pain scales commonly used in local clinical settings were chosen for assessing participants self-reported pain levels. Studies have suggested that CIEP find self-report scales using written words easiest. 7,8 Thus, a 5-point VRS was selected. However, it is estimated that 31% of local people aged 65 years and older have no schooling and may be illiterate. 34 Therefore, the Wong-Baker FACES Pain Scale was also used in this study. There was a high correlation between the FACES and the VAS, in which the correlation coefficients (r) ranged from 0.88 to 0.92, 35,36 suggesting that the FACES was a valid self-report scale for elderly people. It cannot be assumed that self-reported pain scores are accurate simply because they have

6 Vol. 40 No. 4 October 2010 The Psychometric Qualities of Four Observational Pain Tools 587 high completion rates among the CIEP. The ability of participants to translate their subjective pain experience into an objective pain score on both the VRS and the FACES was assessed by the following questions/instructions: 1) Which descriptor/facial expression describes the worst pain? 2) Which descriptor/facial expression describes the least pain? 3) Put the descriptors/facial expressions in sequence from no pain to the worst possible pain. A correct response on one question was equivalent to one point. For instance, if participants could put all five items in the 5-point VRS in the correct sequence, they obtained a comprehension score of 5. If they could also correctly identify the scale items that described the worst and the least pain, they could obtain two more comprehension points, making a total comprehension score of 7 for the 5-point VRS. Similarly, the total comprehension score was 8 for the 6-point FACES. The level of understanding on using the pain scales between participants with and without cognitive impairment was compared. This assessment was conducted during the predata collection interview. Two proxy-pain scales (i.e., the proxy-vrs and the proxy-faces) were used in this study as an additional source in an attempt to capture information about participants, especially those unable to complete the self-report scales. The designs of these two pain scales were identical to the original self-report versions. In previous studies, proxy-reported pain scales have demonstrated a reasonable agreement (70%) between nursing staff and patients ratings in identifying the presence of pain, but estimates of pain intensity were quite poor when compared with those of self-reporting scales. 9,10 In this study, the participants direct contact nurses were asked to score on the proxy-rating scales based on their overall perceptions of the pain levels of their clients during the previous week. We reminded the nurses to recall the pain levels that the elderly residents might have during activities such as morning exercise, bathing, transferring, etc. We used this method to collect the proxy-pain scores, as this is a common practice. We intended to correlate these with the OPT scores. Continuous Rater Training Three trained raters, JYWL and two research assistants, used the English-version OPTs to assess participants pain levels. The use of English documents is common for health professionals in HK clinical settings. Training sessions were organized to make sure that all raters were able to use all OPTs in a correct and consistent manner. The objective of the training was to learn the techniques of delivering and scoring the OPTs. Raters practiced using the OPTs with various clinical vignettes and real nursing home residents. Discussion sessions were held after each practice to clarify the selection of pain-related indicators. The rater training was continued throughout the period of data collection to ensure the correct standard of use of the OPTs. Ethical Considerations This study was given ethical approval by the School of Nursing of the Hong Kong Polytechnic University. Permission to conduct the study also was obtained from the managing directors of the nursing homes. Informed consent from either participants and/or their close relatives was obtained before data collection. Statistical Analysis The data were analyzed using the Statistical Package for the Social Sciences 14.0 (SPSS Inc., Chicago, IL) for most of the analyses. Descriptive statistics were used to evaluate the demographic data of the participant, the pain scale comprehension scores, and the pain scores obtained in different conditions by different pain scales. For the inferential analysis, a value of P > 0.05 was considered statistically significant when testing for associations. CFA was undertaken using the LISREL 8.7 (Scientific Software International, Inc., Lincolnwood, IL). 37e39 Results Participants Sixty-two residents with C-MMSE scores above 18 were assigned to the cognitively intact group, whereas another 62 residents with scores of 18 or below were assigned to the cognitively impaired group. The mean age for the entire group was (range 70e102 years), of whom 119 (96.8%) were female and 4 (3.2%) were male. Demographic data are shown in Table 2.

7 588 Liu et al. Vol. 40 No. 4 October 2010 Table 2 Descriptive Information on Demographic Data for All the Participants Participant Characteristics Intact Group (n ¼ 62) Impaired Group (n ¼ 62) General demographic data Age a (mean SD) C-MMSE score a (mean SD) Years of education a (mean SD) Length of stay in institutions (mean SD) Female:male ratio 60:2 60:2 History of OA Years of suffering from OA (mean SD) Number of OA joints (mean SD) Prescription of analgesics (n (%)) 40 (64.5%) 36 (58.1%) a Mann-Whitney U test showed significant difference, with P < A two-tailed Mann-Whitney U test indicated significant differences between the cognitively intact and impaired groups according to age (z ¼ 3.15, P < 0.001), C-MMSE score (z ¼ 9.57, P < 0.001), and years of education (z ¼ 5.30, P < 0.001). An unpaired t-test (t ¼ 0.25, P ¼ 0.91) showed no significant difference between these two groups in the length of participants institutionalization. In general, the characteristics of the cognitively intact group were different from those of the cognitively impaired group. Except for the length of stay in institutions, there were significant differences in all other items in the general demographic data. However, they had similar histories of OA pain. The period of suffering from OA pain and the number of affected joints showed no significant difference between these two groups when compared with the Mann-Whitney U test (z ¼ 1.45, P ¼ 0.15 and z ¼ 0.38, P ¼ 0.70, respectively). The cognitively intact group appeared to have been prescribed slightly more analgesics than the cognitively impaired group but, on testing, no significant difference could be identified (c 2 ¼ 0.31, P ¼ 0.58). About 80% of analgesics were prescribed as as needed medications for all participants. Throughout the data collection period, only one participant from the intact group dropped out. The reason for this was that the participant felt unwell during data collection. Comprehension of Self-Reported Pain Scales Fifty-five (88.7%) and 17 (27.4%) participants in the intact group and the impaired group, respectively, were able to complete the VRS assessment. The mean score of the VRS comprehension was and out of a maximum of seven in the intact and impaired groups, respectively. The Mann- Whitney U test showed a significant difference between the groups in their ability to quantify their subjective pain experience in objective pain descriptors (z ¼ 4.13, P < [twotailed]). The completion rates for both groups improved for the FACES comprehension assessment, in which all participants in the intact group and 48 (77.4%) participants in the impaired group could complete the assessment. The mean scores for the FACES comprehension were and in the intact and impaired groups, respectively, out of a maximum of eight. There was no significant difference between these two groups in understanding the FACES scale (z ¼ 1.24, P ¼ 0.215). Internal Consistency For each of the OPTs and the subscales of the PACSLAC, the Cronbach s alpha was calculated to examine the homogeneity of the tool items. The calculations were based on the entire group, the intact group and the impaired group OPT scores collected by Rater A vs. Raters B/C at different activity levels (Table 3). The lowest alphas were found for the physiological/eating/sleeping/vocal behaviors (ranging from 0.20 to 0.49) and social/personality/mood (ranging from 0.42 to 0.65) subscales of the PACSLAC. Fortunately, the Cronbach s alpha for the total PACSLAC scale was above 0.7. This finding is similar to the initial validating study of the PACSLAC, and it is, therefore,

8 Vol. 40 No. 4 October 2010 The Psychometric Qualities of Four Observational Pain Tools 589 Table 3 Internal Consistency (Cronbach s a) of PAINAD, Abbey PS, PACSLAC, and DS-DAT Rest Exercise Total Group (n ¼ 123) Intact Group (n ¼ 61) Impaired Group (n ¼ 62) Total Group (n ¼ 123) Intact Group (n ¼ 61) Impaired Group (n ¼ 62) Scores taken by Rater A PAINAD total Abbey PS total PACSLAC total Facial expressions (n ¼ 13 items) Activity/body movement (n ¼ 20 items) Social/personality/mood (n ¼ 12 items) Physiological/eating/sleeping/ vocal behaviors (n ¼ 15) DS-DAT total Scores taken by Raters B/C PAINAD total Abbey PS total PACSLAC total Facial expressions (n ¼ 13 items) Activity/body movement (n ¼ 20 items) Social/personality/mood (n ¼ 12 items) Physiological/eating/sleeping/ vocal behaviors (n ¼ 15) DS-DAT total recommended to use only the total PACSLAC score. 31 Most of the Cronbach s alphas calculated based on total OPT scores for the entire group were above 0.7, which indicated that the OPTs had adequate levels of internal consistency. In other words, all items within an OPT were measuring a similar characteristic. Interrater Reliability In general, moderate to good levels of interrater reliability were obtained when using all four OPTs to measure pain among the participants, regardless of their cognition (Table 4). The values of the interclass correlation coefficient (ICC) collected during exercise ranged from 0.78 to 0.87 and from 0.82 to 0.90 for the intact and impaired groups, respectively. These ICCs were slightly higher than those collected at rest (i.e., ICC ¼ 0.66e0.67 for the intact group; ICC ¼ 0.80e0.87 for the impaired group). Capability of the OPTs to Distinguish Pain from the Nonpain Condition The mean pain scores of all pain scales collected at different activity levels among the intact and impaired groups are listed in Table 5. Pain scores obtained during the exercise program were consistently higher than those obtained when the participants were at rest. Based on the OPT scores collected by Raters B/C, who were unaware of the hypothesis of this study, the results of the Wilcoxon signed-rank test indicated that all pain scales (i.e., four OPTs and the two self-reported pain scales) produced statistically significantly higher pain scores during the exercise program than when participants were at rest for both groups. This is evidence that the OPTs were able to distinguish pain from the nonpain condition. As there was only one type of Table 4 Interrater Reliability of PAINAD, Abbey PS, PACSLAC, and DS-DAT Total Group (n ¼ 123) Intact Group (n ¼ 61) Impaired Group (n ¼ 62) OPTs Rest Exercise Rest Exercise Rest Exercise PAINAD Abbey PS PACSLAC DS-DAT

9 590 Liu et al. Vol. 40 No. 4 October 2010 Table 5 Mean (SD) of Pain Scores from All Pain Scales and Wilcoxon Signed-Rank Test Compared Pain Scores Obtained During Rest and Exercise Intact Group Impaired Group OPTs Rest, Mean (SD) Exercise, Mean (SD) z Rest, Mean (SD) Exercise, Mean (SD) z PAINAD 1.00 (1.04) 3.06 (1.87) 5.96 a 0.76 (1.37) 5.43 (2.00) 6.70 a Abbey PS 2.05 (1.23) 4.61 (2.56) 5.63 a 2.82 (1.68) 7.69 (3.21) 6.70 a PACSLAC 1.85 (1.74) 6.52 (4.16) 5.86 a 3.34 (2.82) (6.02) 6.71 a DS-DAT 0.55 (3.27) 4.82 (3.95) 5.78 a 0.40 (5.25) 9.98 (3.81) 6.74 a VRS 0.63 (0.70) 1.88 (0.86) 5.46 a 0.86 (0.77) 2.13 (0.83) 2.91 b wn ¼ 55 wn ¼ 17 FACES 1.05 (1.06) 2.65 (1.12) 6.31 a 1.51 (1.33) 3.08 (1.19) 4.69 a wn ¼ 61 wn ¼ 48 SD ¼ standard deviation; wn ¼ sample size of the analysis. a Significance level at P < (two-tailed). b Significance level at P < 0.05 (two-tailed). proxy-reported pain score (i.e., the general pain level of the participants rated by their direct nurses), proxy-report pain scales were not included in this analysis. Convergent Validity The Spearman correlations among the four OPTs were established according to different raters, participants cognitive impairment levels and different activity levels. As expected, the results of r s showed that all OPTs were significantly correlated with each other under different conditions (Table 6). This is evidence of convergent validity, showing that all OPTs were measuring similar domains. Concurrent Validity When comparing the OPT scores to the selfreport scores collected when participants were at rest, the majority of the r s reflected poor to fair levels of correlation between pain scores collected during rest (Table 7). During the exercise program, correlations were higher, where r s ranged from 0.38 to 0.59 and from 0.35 to 0.71 when the VRS was used in the cognitively intact group and the impaired group, respectively. The r s ranged from 0.51 to 0.64 and from 0.60 to 0.82 when the FACES was used in the cognitively intact group and impaired group, respectively. Among the two self-reported scales, the FACES generally correlated better with the OPTs. As a limited number of participants were able to use the VRS (n ¼ 55 and 17 for the intact and impaired groups, respectively), the small sample size reduced the statistical power in detecting the significance of a data set. Therefore, the VRS tended to have lower r s with the OPTs when compared with the r s obtained from the FACES. For individual OPTs, the Abbey PS correlated well with the FACES when used in the impaired group during the exercise program (r s ¼ 0.74e0.82, P < 0.01). The r s between the other three OPTs and the FACES was slightly lower during exercise. However, all r s were above 0.6, which showed good concurrent validity between these pain assessment methods Table 6 Correlation Coefficient Among OPTs Rest Exercise Abbey PS PACSLAC DS-DAT Abbey PS PACSLAC DS-DAT Intact group PAINAD 0.72e e e e e e0.80 Abbey PS e e e0.69 PACSLAC 0.61e e0.72 Impaired group PAINAD 0.75e e e e e e0.82 Abbey PS 0.56e e e e0.79 PACSLAC 0.63e e0.86 All correlations are significant at the 0.01 level (one-tailed).

10 Vol. 40 No. 4 October 2010 The Psychometric Qualities of Four Observational Pain Tools 591 Table 7 Correlation Coefficients (r s ) of OPT Scores to Self-Reported Scores At Rest Exercise Program Intact Group Impaired Group Intact Group Impaired Group VRS (n ¼ 55) FACES (n ¼ 61) VRS (n ¼ 17) FACES (n ¼ 48) VRS (n ¼ 55) FACES (n ¼ 61) VRS (n ¼ 17) FACES (n ¼ 48) OPTs Rater A Rater B Rater A Rater B Rater A Rater B Rater A Rater B Rater A Rater B Rater A Rater B Rater A Rater B Rater A Rater B PAINAD a b 0.55 b b b b b b Abbey PS b b a b 0.59 b 0.59 b b b PACSLAC b b 0.50 a 0.23 b 0.36 a b 0.62 b 0.62 b b b DS-DAT 0.37 b 0.32 a 0.27 a 0.23 b a b b b b a P < 0.05 level (one-tailed). b P < 0.01 level (one-tailed). when using them to assess pain during exercise among cognitively impaired participants. In the cognitively intact group, comparable levels of r s were identified between all OPTs and the FACES, which were 0.61e0.64, 0.59, 0.62, and 0.51e0.64 for the PAINAD, the Abbey PS, the PACSLAC, and the DS-DAT, respectively. Correlation Between OPTs and Proxy-Reported Pain Scales The proxy-reported scores were compared with the OPT scores obtained during different levels of activity. When compared with the OPT scores collected when participants were at rest, the proxy-reported scores varied between zero and fair levels of correlation. Although the proxy-reported pain scales had slightly higher values of Spearman rank correlation coefficient (r s ) than the OPTs scores collected during the exercise program, most of r s clustered around fair levels of correlation (Table 8), which indicated a poor correlation between these two types of pain assessment method. Confirmatory Factor Analysis CFA was used to identify a pattern of linkage between the observed scores and underlying factors. It was hypothesized that OA pain was more detectable by the OPTs during the exercise program. This assumption was supported by the concurrent validity, in which acceptable levels of r s were only consistently obtained among pain scores collected during exercise. Thus, the CFA was generated according to the observed scores from each OPT collected during exercise by different raters for estimating the factor structure (i.e., OA). If the OPTs were validly constructed, all items in an OPT should contribute toward the construct (i.e., OA pain). For calculating the CFA, initially all items in an OPT were included in the first CFA (Model 1) to see whether a fit model could be identified (see the footnote in Table 8 of all goodness-of-fit indices and their acceptable levels). A fit model was identified only in the PAINAD and the Abbey PS after deletion of the breathing and physiological change items, respectively. Table 9 summarizes the goodness-of-fit indices of the PAINAD and the Abbey PS used by Raters A and B/C during exercise. For the

11 592 Liu et al. Vol. 40 No. 4 October 2010 Table 8 Correlation Coefficients (r s ) of OPT Scores to Proxy-Reported Scores Exercise Program Intact Group Impaired Group Proxy-VRS (n ¼ 61) Proxy-FACES (n ¼ 62) Proxy-VRS (n ¼ 61) Proxy-FACES (n ¼ 61) OPTs Rater A Rater B Rater A Rater B Rater A Rater B Rater A Rater B PAINAD 0.34 a 0.26 b 0.32 b 0.53 a 0.41 b 0.44 a 0.41 a 0.36 b Abbey PS 0.29 b a 0.57 a 0.39 a 0.30 b 0.28 b 0.41 a PACSLAC 0.49 a 0.4 a 0.47 a 0.45 a 0.4 b 0.44 a 0.36 b 0.37 a DS-DAT a.43 a a 0.34 b 0.24 b 0.38 a a P < 0.01 level (one-tailed). b P < 0.05 level (one-tailed). PAINAD, Model 1 for the first CFA included all tool items and showed an unfit model in Raters B/C, with Root Mean Square Error of Approximation (RMSEA) ¼ Additionally, a relatively low factor loading (0.33) from the item breathing was identified by Rater A during exercise. Hence, Model 2, with the exclusion of the breathing item, was used to run the second CFA. Finally, Model 2 fit all conditions with acceptable levels of all goodnessof-fit indices and item loadings (0.45e0.90). Additionally, the Cronbach s alphas were 0.72 for both Rater A and Raters B/C during exercise (Table 3). After deletion of the item breathing, the Cronbach s alpha values had slightly improved to 0.75 and 0.76 for Rater A and Raters B/C, respectively. These findings echoed the results in the CFA. For the Abbey PS, Model 1 for the first CFA included all tool items and showed an unfit model in Rater A and Raters B/C during exercise, with RMSEA ¼ 0.13 and 0.12, respectively. Additionally, an unacceptably high factor loading for the item physiological change was identified in Raters B/C. In view of this, Model 2 with the exclusion of this item was used to run the second CFA. Eventually, Model 2 fit all conditions with acceptable levels of all goodness-of-fit indices and item loadings (0.65e0.92). The Cronbach s alphas were 0.75 for Raters A and B/C during exercise. After excluding the item physical changes, the Cronbach s alpha was 0.77 and 0.76 for Rater A and Raters B/C, respectively. No consistency of model could be identified in the DS-DAT. The items noisy breathing, contented facial expression, and relaxed body language continued to be problematic, with low factor loadings toward various combinations of model. The CFA data of the DS-DAT are available from the first author on request. Ten participants for every free parameter estimated is a widely accepted guideline for estimating the sample size of the CFA. 39 Thus, the sample size was sufficient for running the CFA for most of the OPTs in this Table 9 Goodness-of-Fit Indices of CFA of the PAINAD and the Abbey PS OPTs Conditions Model c 2 (P-value) RMSEA (90% CI) GFI AGFI ECVI PAINAD Rater A exercise (P ¼ 0.68) 0.0 (0, 0.10) (P ¼ 0.55) 0.0 (0.0, 0.16) Raters B/C exercise (P ¼ 0.02) N0.12 (0.04, 0.20) (P ¼ 0.16) 0.08 (0.0, 0.22) Abbey PS Rater A Exercise (P ¼ 0.002) N0.13 (0.074; 0.18) (P ¼ 0.02) (0.043, 0.20) Raters B/C Exercise (P ¼ 0.003) N0.12 (0.067, 0.18) (P ¼ 0.53) 0.0 (0.0, 0.11) Model 1 includes all tool items. Model 2 excludes tool item breathing and physiological change in the PAINAD and the Abbey PS, respectively. c 2 (Minimum fit function Chi-square) with no significant difference at P-values > 0.05 ¼ a good fit of the model. RMSEA with value < 0.05 ¼ good fit; between 0.05 and 0.08 ¼ reasonable fit; >0.08 ¼ poor fit. GFI (Goodness-of-Fit Index) and AGFI (Adjusted Goodness-of-Fit Index) with GFI > 0.90 and AGFI > 0.80 ¼ a good fit of the model. ECVI (Expected Cross-Validation Index), the smaller the better, used for comparing independence model. N ¼ unacceptable level of GFI (i.e., evidence of unfit model). CI ¼ confidence interval.

12 Vol. 40 No. 4 October 2010 The Psychometric Qualities of Four Observational Pain Tools 593 study, except for the PACSLAC, which contains 60 items. To avoid bias because of an insufficient sample size, the PACSLAC was not examined with CFA. Discussion Our main conclusions from the findings are as follows: Besides the CFA, other psychometric analyses findings for the four OPTs show similar patterns regardless of participants level of cognition. Better results of various analyses were consistently identified from the OPTs scores obtained during the exercise program. These findings suggest that the OPTs should only be used to measure OA pain during the exercise program. In the CFA, a single model was identified to fit the PAINAD and the Abbey PS during the exercise program, which was achieved after deletion of the items breathing and physiological change in the PAINAD and the Abbey PS, respectively. These two OPTs seemed to be relatively more valid and reliable for assessing OA pain during the exercise program for nursing home residents regardless of their cognition levels. Thus, these two OPTs are recommended for use under similar conditions. The proxy-pain scales correlated poorly with all OPTs in both conditions, indicating that these two types of pain assessments may be measuring pain in slightly different ways. Besides cognitive impairment, the level of literacy also may affect elderly people s ability to use self-reported pain scales, especially the VRS. It is suggested that an individual elderly person s comprehension of self-reported scales should be assessed so that the most appropriate pain scale can be chosen for them. Similar Levels of Reliability Among Four OPTs Our findings showed adequate levels of internal consistency for all OPTs (total scores) when they were used at rest or during the exercise program, although the alpha values of two subscales of the PACSLAC were relatively low. This indicated that all pain indicators within an OPT, to a certain extent, were truly measuring the same characteristic (i.e., OA pain in this study). Hence, the other psychometric properties of the OPTs were evaluated based on the total OPT scores. Our findings also showed good interrater reliability for all OPTs. Closer examination revealed that many OPT scores were close to zero when pain was observed at rest, which may have influenced the agreement between the raters. As a result, the interrater reliability tended to be lower at rest than during exercise. This is evident to show that it is harder to estimate a person s pain at rest by using OPTs when stimulation such as an exercise program is absent. Similar Levels of Validity Among Four OPTs This study hypothesized that participants should exhibit more pain behaviors during exercise as they should be experiencing higher levels of OA pain. Wilcoxon s signed-rank test showed that all OPTs produced significantly higher scores during the exercise program for all the participants regardless of their cognition level. This evidence supports the notion that OPTs are more sensitive in detecting OA pain during the exercise program. By this means, OPTs are able to discriminate between pain and nonpain conditions. This is in accordance with the initial validating studies for all the four OPTs. 16,17,31,32 Additionally, our analysis of convergent validity found that all OPTs correlated well with one another, which illustrated that they were measuring a similar construct. The findings regarding concurrent validity also suggested that all OPTs should be used to measure OA pain during the exercise program. Generally, all OPTs were moderately to highly correlated with self-reported pain scales during the exercise program among all participants. However, when OPT scores were obtained when participants were at rest, they had no to fair correlation with the selfreported scales. Thus, acceptable levels of concurrent validity among different OPTs were only obtained when the OA pain was assessed during the exercise program. When comparing the two self-reported pain scales, the OPTs tended to correlate better with the FACES. As expected, the number of

13 594 Liu et al. Vol. 40 No. 4 October 2010 residents who were able to use the selfreported scales declined with increased levels of cognitive impairment. However, the number of residents who could use the FACES was obviously higher than those who could use the VRS in both groups. The lower correlations between the OPTs and the VRS seemed to be affected by the small sample size, leading to low statistical power to detect the significance of the data. However, this also may have been caused by the poor literacy levels of the participants, which would have affected their understanding of the VRS. Comprehension on Self-Reported Scales in Elderly People The strength of this study is that we examined the participants comprehension on both self-reported scales. The findings in the Mann-Whitney U test showed that both groups had a similar understanding of the FACES, but the cognitively intact group seemed to have a better understanding of the VRS than the impaired group. If both methods (i.e., selfreported and observational) are measuring pain, a better correlation between these two methods should be obtained in the group demonstrating better understanding of the self-reported scales, as was found in the previous study. The PAINAD and the self-reported numeric rating scale (NRS) were used to assess post-hip surgery pain during the likely pain conditions in 12 cognitively impaired and 13 cognitively intact patients. That study reported that the r s was in the intact group, noticeably higher than the r s ¼ in the impaired group when the correlation between the PAIN- AD and the self-reported NRS was examined. 26 In contrast, this study found that higher levels of correlation were obtained in the impaired group, which had a poor understanding of the VRS. This may have been caused by the potential problems of using the VRS among participants with lower literacy levels, regardless of their cognition levels. Comprehension of the VRS is not only affected by one s cognition but also by one s literacy. In Hong Kong, illiteracy is very common among elderly people, especially women. 34 Among all participants, 38.7% and 83.9% in the cognitively intact and impaired groups, respectively, had no formal education and were illiterate. In this study, comprehension of the self-reported scales was conducted on a one-to-one basis in a relaxed manner before data collection. Whenever a resident was illiterate, the interviewer explained each VRS item in detail. Great efforts were made to minimize the effect of illiteracy affecting participants ability to use the VRS. Thus, the participants without cognitive impairment showed good comprehension of the VRS because of these detailed explanations. Therefore, participants in the intact group, with higher cognition, could achieve significantly higher scores to reflect their comprehension of the VRS. Thus, the percentages of participants who were unable to use the VRS were 11.3% and 72.6% for the cognitively intact and impaired groups, respectively, differing slightly from the percentages of illiteracy. During the real data collection procedure, the explanation was not provided in as much detail as during the interview because of time constraints. It was likely that the participants ability to use the VRS was hindered by their literacy levels during the real data collection periods. The cognitively intact participants might have pretended that they understood the VRS and attempted to provide a score. The cognitively impaired participants might not have bothered to pretend but simply did not use the VRS, leading to only 27.4% of subjects in the impaired group scoring on the VRS. This is a possible explanation for the diverse correlations between the OPTs and the VRS obtained in this study. Similarly, inconsistent correlations were identified from the previous studies. A high correlation (r ¼ 0.81, P < 0.01) between the selfreported VRS and the PAINAD was reported by Zwakhalen et al. 28 By contrast, a fair correlation (r ¼ 0.304, P # 0.005) between these two scales was reported by Leong et al. 27 These studies did not evaluate their participants comprehension of the VRS or their literacy levels. Illiteracy may not be common in Western societies, so the VRS has consistently been recommended as a suitable pain scale for assessing pain among elderly people. The current results raise concern about using the VRS among elderly people with lower literacy levels. Besides cognition and literacy, many factors affect elderly people s comprehension of self-reported scales, such as time spent on explanation or elderly people s health status and their