Point-of-service questionnaires can reliably assess patients experiences

Transcription

1 European Journal for Person Centered Healthcare Vol 2 Issue pp ARTICLE Point-of-service questionnaires can reliably assess patients experiences Stephen D. Gill PhD Head of Safety and Quality Improvement, Measurement and Research, Barwon Health, Geelong, Australia Abstract Rationale, aims and objectives: Point-of-service patient experience measures are increasingly used to evaluate health services, but their psychometric properties have not been reported. The current study determined the test-retest reliability of a point-of-service patient experience questionnaire used in an inpatient rehabilitation centre. Method: Patient experience data were collected from inpatients by 2 community volunteers using an electronic hand-held 5-item Likert-style questionnaire. Eighty-one participants completed the questionnaire at Test 1 and Test 2 (2 to hours apart). Twenty three participants completed the questionnaire at Test 1 and Test 3 (one week apart). Results: There were small and non-significant differences between test and retest scores for individual questions and the 5- item total score. Kappa and intra-class correlations coefficients indicated moderate to good agreement. Subgroup analysis indicated that participants who had previously completed the questionnaire during their inpatient stay gave more consistent answers between Test 1 and Test 2 than first responders. Conclusion: Patients participating in sub-acute inpatient rehabilitation can provide reliable answers to a point-of-service patient experience questionnaire. The study is advanced as an important contribution to the person-centered healthcare literature. Keywords Patient experience, person-centered healthcare, questionnaires, reliability, service evaluation, validity Correspondence address Dr. Stephen Gill, Safety and Quality Unit, Barwon Health, PO Box 281, Geelong, 3220, Australia. STEVEG@BarwonHealth.org.au Accepted for publication: 20 January 201 Introduction Patients are increasing their participation in healthcare beyond the consumption of goods and services and are wanting to and being asked to, evaluate health services. Patient-reported experience measures are an increasingly important part of service evaluation and are considered a key performance indicator [1,2]. Point-of-service feedback is one popular method of evaluating patient experience, where feedback from patients is collected during or immediately after care [2-]. A key advantage of point-of-service feedback is increased participation rates and more rapid feedback when compared to traditional feedback methods such as mailed out questionnaires []. To be useful, point-of-service questionnaires need to be reliable and valid. Despite their prevalence, patient satisfaction questionnaires have been criticized for failing to consider and report the measurements validity and reliability which undermines the integrity and usefulness of the assessment findings [5]. Reliability considers a measurement s precision, or the amount of error [6,7]. Test-retest reliability is a statistical technique that repeats the measurement process on the same subjects under similar conditions and compares observations [6]. A highly reliable instrument will give the same or very similar responses for each test. No published data exist on the reliability of point-ofservice questionnaires.the reliability of point-of-service questionnaires is especially important when measurements are used to assess the effectiveness of interventions that are designed to improve the patient experience. More reliable measures detect real change, rather than measurement error [8]. The current study aimed to determine the test-retest reliability of a point-of-service patient-reported experience questionnaire used at an inpatient rehabilitation centre as a contribution to the person-centered healthcare literature. Methods Study setting The study was conducted at a government funded, 100-bed sub-acute inpatient rehabilitation facility in regional Victoria, Australia. The Inpatient Rehabilitation Centre 85

2 Gill Reliability of point-of-service patient experience questionnaires (IRC) is staffed by an interdisciplinary team including medical, nursing and allied health professionals. The IRC consists of 3 units (south, central and north) and provides inpatient rehabilitation to main patient groups: 1. Orthopaedic patients, including people with fractures and joint replacements (south unit) 2. Amputee patients (south unit) 3. Neurology patients, mostly following a stroke (central unit). Geriatric patients who are debilitated following events such as cardiac or respiratory illness (central and north unit). Patients are at least 18 years of age with more than 80% aged 65 years and older. Approximately 75 rehabilitation patients are discharged home per month from the IRC. Data from state-wide patient satisfaction surveys indicated that the IRC performs above the average for inpatient rehabilitation centres and that previous patients were very satisfied with most aspects of their stay [9]. Patient reported experience measure Each week since 2010, 2 community volunteers have collected patient experience data using a hand-held electronic questionnaire [10] that contains 5 Likert-style questions (see Table 1 for the questionnaire). Questions were developed by: (1) examining existing questionnaires, (2) asking IRC patients to describe their needs and (3) discussing patients needs with staff. Questions were tested and modified with patient input to ensure they were understood and interpreted correctly by patients. Volunteers invite patients to complete the questionnaire by explaining that: (1) the IRC would like their feedback, (2) that participation is voluntary, (3) that responses are anonymous and () that results are collated and sent to the IRC leadership group for inspection and action as required. Prior to data collection, the volunteers speak with the head nurse to identify appropriate patients for that week s survey. Inappropriate patients (typically less than 5 each week) are those too unwell to complete the survey or have inadequate communication or cognitive skills. Data from each week s survey are transferred electronically to an external business provider [10] which the IRC employs to collate the results and provide weekly reports. To assist data interpretation the Likert-style scale is scored from 0 to 100 in 25 point increments: seldom or poor is scored as 0, somes or fair is scored as 25, etc. (see Table 1 for questions and response options). Mean scores for each question are calculated and presented in the weekly report. A total average score for all 5 questions is also calculated. Participants Participants in the reliability study were IRC patients. Patients were excluded if they lacked adequate English language skills to communicate verbally or read words or if they had a cognitive impairment that prevented them from understanding the purpose of the study and the questionnaire. The data collector identified ineligible patients through discussions with IRC s head nurses. Participation in the study was voluntary. Data collection Two community volunteers independently collected patient experience data on 3 separate occasions: baseline (Test 1), 120 to 20 minutes later (Test 2) and one week later (Test 3). The volunteers were experienced in collecting patient experience data, having done so each week for at least one year prior to the study. Participants were invited to complete the questionnaire in their rooms consistent with usual practice. Participants could complete the questionnaire by themselves or, if they needed help, the volunteer would read the questions and enter the participant s responses. It was the patient s preference whether to complete the questionnaire by themselves or receive assistance. With the exception of clarifying the meaning of questions, volunteers were instructed not to influence participant s answers. The relatively short duration between Test 1 and Test 2 was chosen to reduce the likelihood of the patient s experience and hence their answers, changing between each test. The longer period between Test 1 and Test 3 was chosen to reduce the likelihood of memory affecting re-test scores, but increased the possibility that the patient s experience and hence their answers, would change. Data were collected between May and August Data analysis Individual question data were analysed as ordinal data [11]. Agreement between Test 1, 2 and 3 scores was assessed using Wilcoxon s signed ranks test, percentage agreement and weighted Kappa [12]. Total score data (the average of the 5 questions) were assumed to approximate an interval scale [11]. Changes in mean scores between Test 1, 2 and 3 were assessed using t- tests and intra-class correlation coefficients (ICCs (2,1)) [13,1]. For both Kappa and ICC, a score of less than 0.2 was considered poor agreement, 0.2 to 0. fair agreement, 0. to 0.6 moderate agreement, 0.6 to 0.8 good agreement and 0.8 to 1 very good agreement [15]. 86

3 European Journal for Person Centered Healthcare Table 1 Patient experience questionnaire Response options (numerical score) Good 1. Did each staff member treat you with dignity and respect? 2. Did each staff member listen and communicate in a way that met your needs? 3. Were all staff attentive and quick to respond to your individual needs?. Were you involved in decisions about your care as much as you wanted to be? 5. Overall, how would you rate your experience at the rehabilitation centre? Excellent Most of the Most of the Most of the Most of the Very good Somes Somes Somes Somes Fair Poor Because the rehabilitation centre receives and evaluates group-level data, score variability at an individual level, such as typical error [8], was not determined. Data were analysed using SPSS statistical software version 21 [16] and R statistical computing [17]. Subgroup analysis Subgroup analysis investigated whether reliability varied according to whether: 1. Participants were first responders or had completed the questionnaire before. 2. Participants read the questions and entered responses themselves or whether the volunteer read the questions to them and entered the paticipants verbal responses. Given the relatively small number of participants who completed Test 3, subgroup analysis was only conducted for Test 1 and Test 2 scores using total scores and 2-way repeated measures ANOVA s and ICC s (2,1). Sample size According to Walter, Eliza and Donna [18], for 2 assessors, an alpha of 0.05, power of 0.80 and an intraclass correlation of , a minimum of 19 participants were required for each subgroup in the study. Ethical considerations Barwon Health Human Ethics Research Committee approved the study. All participants received written information about the study and signed a consent form. Results Participants Ninety-nine participants were assessed at Test 1. Of these, 81 were assessed Test 2 and 23 at Test 3. Drop outs between Test 1 and Test 2 occurred because participants were out of their rooms participating in rehabilitation therapy. Drop outs between Test 1 and Test 3 occurred because participants had been discharged home or were in therapy. Of the 81 participants assessed at both Test 1 and 2, 1 were in North Unit, 31 in Central Unit and 36 in South Unit. Participants average age was 73 years (SD = 15 years, range 18 to 96 years) and 9 (60%) were female. At the of Test 1 data collection, participants had been in the rehabilitation centre for an average of 13 days (SD = 13 days, range 1 to 62 days). Scatterplots revealed that the data were negatively skewed with most participants scoring questions near the top of each scale (see Table 2 for score distribution). The Shapiro - Wilk Test indicated that the data were not normally distributed. Individual questions Wilcoxon s signed ranks tests found no significant differences between Test 1 and Test 2 scores or between Test 1 and Test 3 scores for each question (p > 0.05). Differences between mean scores at Test 1 and Test 2 and between Test 1 and Test 3 scores were less than 5% of mean Test 1 scores (see Table 2 for mean scores). Participants gave the same answers at Test 1 and Test 2 on 72% (Question 2) to 89% of occasions (Question 1) (see Table 2). When discrepancies occurred between Test 1 and Test 2 scores, 85% of these discrepancies were one response category either side of the Test 1 score. Thirtyeight (7%) participants gave the same answer at Test 1 and Test 2 score for all 5 questions. 87

4 Gill Reliability of point-of-service patient experience questionnaires Table 2 Test 1, Test 2 and Test 3 scores and levels of agreement for individual questions Frequency distribution of scores, n, (%) Question 1 Test Test (21%) 22 (27%) Test (13%) Question 2 Test (5%) Test (%) 19 (2%) 31 (38%) Test (17%) Question 3 Test (8%) Test (11%) Test (%) Question Test 1 2 (2%) Test 2 2 (2%) 0 5 (6%) 1 Test (%) Question 5 Test Test (2%) (5%) 2 (9%) 11 (1%) 6 (8%) 29 (36%) 33 (1%) (18%) 19 (2%) 23 (29%) (17%) 30 (37%) 31 (38%) 63 (78%) 58 (72%) 20 (87%) 57 (70%) 7 (58%) 19 (83%) (5%) 39 (8%) 18 (78%) 55 (68%) 51 (63%) 16 (70%) 39 (8%) 2 (52%) Mean score (SD) 9.1 (11.) 92.6 (12.1) 96.7 (8.6) 90.7 (16.0) 88.6 (1.3) 95.6 (9.7) 85.2 (19.7) 8.3 (17.0) 93.5 (13.5) 88.6 (20.6) 87.0 (21.3) 88.0 (21.1) 83.0 (18.9) 8.9 (18.) % agreement a 2 (n=81) 89% 72% 77% 78% 75% % agreement a 3 (n=23) 91% 82% 78% 70% 83% Kappa (95% CI) 2 (n=81) 0.66 (0.51 to 0.82) 0.6 (0.28 to 0.65) 0.58 (0.37 to 0.78) 0.57 (0.38 to 0.76) 0.6 (0.3 to 0.85) Kappa (95%CI) 3 (n=23) 0.62 (0.39 to 0.8) 0.0 (0.13 to 0.66) 0.3 (0.11 to 0.75) 0.5 (0.12 to 0.78) 0.58 (0.22 to 0.9) a Percentage of participants who gave the same score at each test Table 3 Test 1, Test 2 and Test 3 scores and levels of agreement for total scores Baseline mean score (SD) Follow up mean score (SD) Difference between means* (95% CI) ICC (95% CI) Test 1v Test 2 (n=81) 88.3 (13.2) 87.5 (12.9) -0.8 (3.3 to 1.5) 0.65 (0.51 to 0.76) 3 (n=23) 92. (8.9) 91.5 (10.5) -0.9 (-3.3 to 5.1) 0.50 (0.12 to 0.75) *paired t-test Percentage agreement for Test 1 and Test 3 scores ranged between 70% (Question ) and 91% (Question 1). When discrepancies occurred between Test 1 and Test 3 scores, 86% of these discrepancies were one response category either side of the Test 1 score. Thirteen (56%) participants gave the same answer at Test 1 and Test 3 score for all 5 questions. Kappa varied between 0.0 and 0.66 for individual questions indicating moderate to good agreement between test and re-test scores (see Table 2). Total scores Differences in mean total scores between Test 1 and Test 2 and between Test 1 and Test 3 scores were small (less than 1% of the average mean baseline score) and not significantly different (see Table 3). The ICC for Test 1 and Test 2 scores indicated good agreement (ICC = 0.65, 95% CI 0.51 to 0.76) and moderate agreement for Test 1 and Test 3 scores (ICC = 0.50, 95% 0.12 to 0.75). 88

5 European Journal for Person Centered Healthcare Subgroup analysis Forty-one participants had the volunteer read the questions to them and enter the responses. Forty participants read the questions themselves and entered their responses. There were no significant differences in total scores at Test 1 or Test 2 regardless of whether the patient or the volunteer read or inputted the answers (p > 0.05) (data not shown). When questions were read to the participant, the ICC for Test 1 and Test 2 scores was moderate (ICC = 0.62, 95% CI 0.39 to 0.78). When the participant read the questions, the ICC for Test 1 and Test 2 scores was also moderate (ICC = 0.70, 95% CI 0.50 to 0.83). Twenty participants had completed the questionnaire prior to participating in the study and 61 participants had not. There were no significant differences in total scores at Test 1 or Test 2 for participants who had completed the questionnaire before compared to those who had not (p > 0.05) (data not shown). When the participant had completed the questionnaire prior to the study, the ICC for Test 1 and Test 2 scores was very good (ICC = 0.9, 95% CI 0.86 to 0.98, n=20). When the participant had not previously completed the questionnaire the ICC for Test 1 and Test 2 scores was moderate (ICC = 0.58, 95% CI 0.38 to 0.72, n=61). Discussion Point-of-service patient experience questionnaires are relatively new and potentially useful for evaluating health services from the patient s perspective and represent important tools for increasing the person-centeredness of clinical care. The measurement properties of point-ofservice patient experience measures have not been reported. The results of the current study indicate that patients participating in inpatient rehabilitation can provide consistent answers to patient experience questions. There were small and non-significant variations in group-level patient experience scores with repeated testing. Reliability coefficients showed moderate to good reliability. Subgroup analysis found that reliability improved when participants had previously completed the questionnaire. In healthcare, questionnaires are frequently implemented without understanding or reporting their psychometric properties [5,19]. Sitzia [5] assessed 195 published patient satisfaction questionnaires and found study authors exhibited poor understanding of the importance of reliability and validity and that instruments demonstrated little evidence of these properties. The current study makes an initial step towards understanding the measurement properties of point-of-service patient experience questionnaires in order to contribute to the person-centered healthcare literature. There is contention regarding the best statistical methods to assess reliability and how to interpret the results [8,13]. The current study employed a variety of common tests and interpreted the results using popular guidelines [12,13,15]. Likert data were converted to numerical data and although the appropriateness of recoding ordinal data to numerical values to create mean scores has been questioned, it is common practice and acceptable alternatives are hard to find [20]. Mean scores changed by less than 1% and 5% for total and individual questions respectively. Staff within the rehabilitation centre can use the reliability estimates when interpreting patient experience data to appreciate the measurement s precision and to consider whether real change has occurred over. Using a composite score for the 5 questions appears more reliable and sensitive to detecting real change than using individual questions. Describing and evaluating one s healthcare experiences can be difficult and complex [21] and involves an interaction between external circumstances and a person s internal world of expectations, memories, motivations and other cognitive abilities. The participants in the current study were recovering from illness and injury and might have been affected by pain, medication or the distractions of their external environment. Despite these challenges most patients gave consistent responses to repeated questioning. Scores from participants who completed the questionnaire themselves had similar reliability to scores from participants who had the volunteer read the questions to them and input the answers. The volunteer s presence, together with allowing participants to choose how if they or the volunteer inputted the answers, could have increased the reliability of the questionnaire. Patients with reading difficulties, physical impairments or anxiety using the electronic questionnaire received assistance from the volunteer. The volunteer also clarified the meaning of questions should participants be uncertain. The questionnaire appeared to have a small practice effect. Participants who had completed the questionnaire prior to the study had more stable scores than those who had not. The rehabilitation centre s usual practice is to collect weekly patient experience data. Given that patients often stay at the rehabilitation centre for several weeks, repeated testing should increase the reliability of the measurements. The opportunity for repeated testing and increased reliability is an advantage of inpatient point-ofservice patient experience questionnaires that is not realized by once-off mailed out questionnaires. The reliability estimates presented relate to the questionnaire and data collection methods used in the current study and might not generalize to other settings or patient groups. However, the questions and concepts assessed by the current study s questionnaire are commonly included in patient experience measures [1,22], such that the results might be indicative of other similar point-of-service patient experience questionnaires. Also, choosing the optimal interval for test-re-test reliability studies can be difficult [23]. The re-test should be soon enough that the patient s experience has not changed, yet sufficiently delayed so that re-test responses are simply not memories of earlier answers [23]. The current study retested at 2 intervals. The relatively short period between Test 1 and Test 2 reduced the chance of a patient s experience changing between tests, but increased the likelihood of memory effects. Test 1-2 reliability estimates are more likely to overestimate reliability. 89

6 Gill Reliability of point-of-service patient experience questionnaires Participants completing Test 1 and Test 3 were less likely to remember previous answers, but it is more likely that their experience changed between assessments. Test 1-3 reliability estimates are more likely to under-estimate reliability. Internal consistency, or intra-scale reliability, is often used when the measured construct might change between assessments. We preferred to use test-re-test reliability to assess reliability, because the questionnaire contained only 5 questions that assessed different aspects of a patient s experience. The small sample size for Test 1 and Test 3 comparisons reduced the precision of the reliability estimates which should be interpreted cautiously. Greater confidence can be placed in the accuracy of the Test 1-Test 2 reliability estimates. Reliability is one aspect of a measurement s psychometric properties. Although beyond the scope of the current study, validity, including responsiveness to change, is an important attribute that must be established to be confident that a measurement tool assesses what it is intended to measure and is sensitive to detecting change in the underlying construct. Patient acceptability is also required. Conclusion The use of point-of-service patient experience questionnaires is increasing throughout the world, but the measurement properties of these tools have not been reported. The current study provides reliability estimates for a patient experience questionnaire used in an inpatient rehabilitation centre and demonstrates that patients can provide consistent answers over. Measurement error does exist and, though small, clinicians and administrators should consider this when interpreting questionnaire results. Despite their limitations, point-of-service patient experience questionnaires represent an important mechanism for increasing the person-centered of clinical care. Acknowledgements and Conflicts of Interest I am indebted to the study s participants and the volunteers who collected the data. I wish to acknowledge the support of Jo Bourke, Director of Safety and Quality Unit, Barwon Health; Robyn Hayles, Executive Director of Community Health and Rehabilitation Services, Barwon Health and Stephen Lane, Statistician, Barwon Health. I declare no conflicts of interest. References [1] Centers for Medicare and Medicaid Services. (2012). CAHPS hospital survey (HCAHPS) quality assurance guidelines, version 7. [2] Coulter, A., Fitzpatrick, R. & Cornwell, J. (2009). Point of care. Measures of patients experience in hospital: Purpose, methods and uses. London: The Kings Fund. [3] Dirocco, D.N. & Day, S.C. (2011). Obtaining patient feedback at point of service using electronic kiosks. American Journal of Managed Care 17 (7) e [] Jensen, H.I., Ammentorp, J. & Kofoed, P-E. (2010). User satisfaction is influenced by the interval between a health care service and the assessment of the service. Social Science & Medicine 70, [5] Sitzia, J. (1999). How valid and reliable are patient satisfaction data? An analysis of 195 studies. International Journal for Quality in Health Care 11 () [6] Lavrakas, P.J. (2008). Test-retest reliability. Thousand Oaks, CA: SAGE Publications. [7] Kottner, J., Audige, L., Brorson, S., Donner, A., Gajewski, B.J., Hrobjartsson, A., Roberts, C., Shoukri, M. & Streiner, D.L. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology 6, [8] Hopkins, W.G. (2000). Measures of reliability in sports medicine and science. Sports Medicine 30 (1) [9] Victorian Department of Health. (2010). Victorian patient satisfaction monitor. Wave 18 report for McKellar centre. Melbourne, Victoria: Department of Health. [10] Customer Feedback Systems. (2006). Customer Feedback Systems Australasia. Pymble, NSW: Customer Feedback Systems Australasia. Available from: [11] Boone, H.N. & Boone, D.A. (2012). Analyzing likert data. Journal of Extension 50 (2) 2TOT2. [12] Sim, J. & Wright, C.C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy 85, [13] Atkinson, G. & Nevill, A.M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Medicine 26 () [1] Shrout, P.E. & Fleiss, J.L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 86 (2) [15] Landis, J.R. & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics 33, [16] IBM SPSS statistics 21. IBM; [17] The r project for statistical computing; Available from: [18] Walter, S.D., Eliasziw, M. & Donner, A. (1998). Sample size and optimal designs for reliability studies. Statistics in Medicine 17 (1) [19] Brink, Y. & Louw, Q.A. (2012). Clinical instruments: Reliability and validity critical appraisal. Journal of Evaluation in Clinical Practice 18, [20] Jakobsson, U. (200). Statistical presentation and analysis of ordinal data in nursing research. Scandinavian Journal of Caring Sciences 18, [21] Staniszewska, S.H. & Henderson, L. (2005). Patients' evaluations of the quality of care: Influencing factors and the importance of engagement. Journal of Advanced Nursing 9 (5)

7 European Journal for Person Centered Healthcare [22] Jenkinson, C., Coulter, A. & Bruster, S. (2002). The picker patient experience questionnaire: Development and validation using data from in-patient surveys in five countries. International Journal for Quality in Health Care 1 (5) [23] Frost, M.H., Reeve, B.B., Liepa, A.M., Stauffer, J.W. & Hays, R.D. (2007). What is sufficient evidence for the reliability and validity of patient-reported outcome measures? Value in Health 10 (2) S9-S