An evaluation of the QSP and the QPP: two methods for measuring patient satisfaction

Transcription

1 International Journal for Quality in Health Care 2001; Volume 13, Number 3: pp An evaluation of the QSP and the QPP: two methods for measuring patient satisfaction JÖRGEN NATHORST-BÖÖS 1, INGRID M. E. MUNCK 2, INGEMAR ECKERLUND 3 AND CAROLINA EKFELDT-SANDBERG 4 1 Department of Obstetrics and Gynaecology, Karolinska Institute, Karolinska Hospital, 2 The Swedish Agency for Administrative Development, 3 National Board of Health and Welfare and 4 SPRI, Box 70487, Stockholm, Sweden Abstract Background. Patient satisfaction is a function of several variables addressing reasons why it is important to use methods in which these different factors can be isolated and their importance analysed. Objective. In this project, two methods using this approach were used: the Quality from the Patient s Perspective and the Quality, Satisfaction, Performance models. The aim of the present study is to evaluate these two different methods with respect to application, strengths and weaknesses. Design. In the Quality from the Patient s Perspective model, the patient judges the different domains in two dimensions: perceived reality and subjective importance. The Quality, Satisfaction, Performance model uses a multivariate analysis to capture the patient s priorities. Four hundred and sixty forms for each model were distributed to a random sample of patients at the Department of Obstetrics and Gynecology at Karolinska Hospital. Main measures. The quality factors treatment by the nurse, participation, information, environment and accessibility were measured. Results. On both forms, medical care, treatment by the doctor and access to nursing treatment received high scores in perceived reality while accessibility and participation received low scores. Subjective importance measured directly and indirectly, respectively, in the two models showed high values for medical care and treatment by the doctor. Conclusion. The advantages of the Quality from the Patient s Perspective model are that it has a comprehensive and solid question bank. The Quality, Satisfaction, Performance model s advantage is its immediate usefulness and its clear graphic presentation. An integration and further development of these two approaches may prove useful. Keywords: patient satisfaction, patient questionnaire, quality from the patient s perspective, quality satisfaction performance find it easier to follow medical instructions thereby eliminating unnecessary medical visits. Continuous measurement of our clients opinions about products and services given by the medical community also provides a basis for quality assurance and different forms of benchmarking. There are, however, several problems concerning method- ological issues in these studies. In many standardized surveys a high satisfaction rate of 75 90% is often found [1]. Many authors have suggested that the high level of patient satisfaction found is due to measurement errors causing a different kind of bias, such as social desirability, reluctance In the past decade, quality issues in health care have gained increasing interest. Several tools have emerged to continuously monitor hospital care processes and to improve and control different areas of care. To date, most studies have focused on medical and economic criteria, but attempts have also been made to include the customer s or patient s judgement about the care that health services supply. Beside democratic and ethical reasons for assessing patient s views on health care, the findings from such studies can lead to better use of allocated resources. There are other important medical priorities such as that satisfied and well-informed patients Address reprint requests to J. Nathorst-Böös, Department of Obstetrics and Gynaecology, Karolinska Hospital, Box 140, Stockholm, Sweden. jorgen.nathorst-boos@ks.se International Society for Quality in Health Care and Oxford University Press 257

2 J. Nathorst-Böös et al. Figure 1 Spectrum of influences on patient satisfaction. to express a negative opinion, the wording of questions and non-specific questions [2 5]. The lack of differentiation seen seems to indicate that patient satisfaction is not easily operationalized as a discrimination measure [1]. Patient satisfaction is a function of several variables (Figure 1) and it is therefore of great importance to use multivariate methods in which the different factors can be isolated. These validation problems have been approached by com- municating with patients to identify the issues that concern them most acutely and to determine how they perceive and interpret the services they receive. This is often done with focus groups or by one-on-one interviews with patients prior to measuring their satisfaction. In this way, questions that are important to patients can be elucidated and used in questionnaires. After identification of the patients concerns, these can be grouped into specific domains, such as medical, physical, emotional etc. [6 10]. In order to reduce the number of questions and to detect the structure in the relationships between the variables, the traditional factor analysis or the more general statistical methodology, structural equation mod- elling, may be used [8]. Another methodology problem in these studies concerns the relative importance of the different categories. It is often implied that it is those aspects of satisfaction with which patients are least satisfied that should be considered as priorities for improvement, i.e. patient satisfaction surveys seldom take into account patients priorities among different variables. Deciding what is a priority by choosing the domain with the lowest score also ignores the possibility of improving satisfaction generally. It is therefore important that the method used captures the patient s judgement of the different domains incorporated [11]. In this project, two methods using this approach have been used. In the Quality from the Patient s Perspective (QPP), the patient judges the different domains in two dimensions: perceived reality and subjective importance. The Quality, Satisfaction, Performance (QSP) model uses a multivariate analysis to capture the patient s priorities. The two models are described in more detail in the Materials and methods section. The aim of the present study was to evaluate the two different methods with respect to application, strengths and weaknesses. Materials and Methods The QPP model The QPP model is based on the premise that the quality of care can be understood by evaluating two aspects of care provision: the resource structure of the care organization and the patient s preferences. The resource structure of the care organization consists of the personnel and the qualities of the physical and administrative environment while the patient s preferences have a rational and a human aspect [10]. The QPP questionnaire was developed from the findings of a study using personal interviews [12]. In this instrument, each question is judged in two dimensions; perceived reality and subjective importance. The question bank in the QPP form originates from interviews where factors of importance for patient satisfaction with care have been identified [10]. In this study, 35 interviews were conducted consisting of open- ended and individually adapted follow-up questions covering the following themes: (i) Issues of importance with regard to the care the patient received; (ii) what the patient perceived as positive or negative in connection with the care; (iii) whether the patient felt that anything was lacking during the period of care; and (iv) whether the patient wished to change anything regarding the care. The interviews lasted approximately minutes and were tape-recorded and later transcribed verbatim for con- secutive analysis. Nine hundred indicators were sorted into 27 categories. The model was then operationalized using factor-analysis into a questionnaire consisting of 56 questions. The point of departure for measurement are these quality factors: (i) medical care; (ii) treatment by the doctor; (iii) treatment by the nurse; (iv) participation; (v) information; (vi) environment; and (vii) accessibility. We designed two main QPP forms for outpatient assessment: a long and a short form. In this study the long form was used. In the QPP approach, each question was judged in two ways: (i) perceived reality on a four-point scale (quite correct, almost correct, not quite correct, not correct at all, not applicable) and (ii) subjective importance on a four-point scale (of utmost importance, of great importance, of little importance, of no importance, not applicable). An example of a QPP question is given in Table

3 An evaluation of QSP and QPP: two methods measuring patient satisfaction Table 1 Example of a question from the QPP model Waiting time at clinic... I... was... Perceived reality Subjective importance... treated without any delay Quite correct 4 Of utmost importance 4 during my visit at the clinic Almost correct 3 Of great importance 3 Not quite correct 2 Of little importance 2 Not correct at all 1 Of no importance 1 Not applicable X Not applicable X Table 2 Example of a question from the QSP model How... satisfied are you with the waiting time at the clinic? Not satisfied Very Do not at all satisfied know Φ Φ Φ Φ Φ Φ Φ Φ Φ Φ Φ A personal quality index (PQI) was calculated for each question by placing the response scores into the following formula: PQI=subjective importance (2 perceived reality subjective importance). The formula gives an index that can vary between 8 and 16. High values indicate good patient satisfaction and low values suggest that action should be taken in order to improve satisfaction [12]. Figure 2 The relationship between the three concepts quality factors, patient satisfaction and goals in the QSP model. The QSP model The QSP model originates from a method used to assess customer satisfaction. The model has been developed and adapted for assessing patient satisfaction. Studies have been The statistical QSP model consists of two parts: performed in eye clinics and dermatological clinics [13]. The (i) Part I. A structural model describes how the latent QSP model consists of three parts that are later combined variables patient satisfaction and the different quality to give one score for each respondent: factors and goals are related to each other. The relationship between the three concepts quality fac- (i) Patient satisfaction is rated on a global, standardized tors, patient satisfaction, and goals is described in scale and assessed by means of three questions. This Figure 2. This model is used for the statistical forms the Patient Satisfaction Index (PSI) which is analysis. estimated from the weighted average of the three (ii) Part II. Each concept in the model is measured with questions. a latent variable built up as a factor related to several (ii) Experienced quality of different dimensions or facquestions and this constitutes the measurement part tors of quality are a priori regarded as important in of the model. The relationships between the different explaining patient satisfaction. Examples of such quality factors and patient satisfaction are expressed factors are information, waiting-time and satisfaction as equations. This means that a regression coefficient with medical care. Three or four questions were is calculated describing the importance of each factor formulated to cover each factor. on the PSI. The regression factor can be interpreted (iii) Experienced quality of factors is defined as the in terms of the importance of the factor studied clinic s goals. Examples of these goals are increased while at the same time controlling for the other confidence and willingness to come back. factors, i.e. keeping the rest of the factors constant. The results can be presented in a coordinate system An example of a question from the QSP questionnaire is with the factor under study on the y-axis and its given in Table 2. All questions were judged on a 10-point scale. impact on the PSI shown on the x-axis. 259

4 J. Nathorst-Böös et al. The statistical method used is a multivariate analysis based Table 3 The eight domains and 37 questions used in the on the principles used in the Partial Least Squares (PLS) QPP and QSP questionnaires method [14]. The study was conducted at the Department of Obstetrics Domain Question area and Gynecology at Karolinska Hospital, a university out- Medical care Examination patient clinic with approximately 4000 visits each year. The Medical treatment catchment area of the clinic is the northern part of Stockholm, Doctor s skill serving approximately inhabitants. Apart from the Nurse s skill problems often observed at gynecological clinics, such as Waiting time On the telephone bleeding disorders, Pap smear tests and HRT treatment, most During the appointment of the clinic patients are healthy. Treatment by the doctor Showed you respect Gave you sincere answers Main measures Treated you in a positive way The first step was to perform a pilot study with open-ended Listened to you questions in order to see what issues were regarded as Showed commitment and important by the patients. A simple questionnaire form was care given to 100 patients who were asked to indicate three things Treatment by the nurse Showed you respect they considered good and three things they considered bad. Gave you sincere answers From this assessment the most important domains and Treated you in a positive way questions were identified. Suitable questions were then se- Listened to you lected from the QPP questionnaire. This process was carried Showed commitment and out in conjunction with the nurses and doctors at the clinic. care It was considered important to make the questionnaire as Information Examinations simple as possible and to exclude questions which seemed Blood sampling inappropriate to the outpatient setting. The questions were Treatment formulated in a similar way and arranged in the same order Drugs as in the QPP and QSP questionnaires. The 37 questions Results of tests finally chosen covered eight different factors. The questions Where to turn with questions and factors are presented in Table 3. Participation Choose time for appointment An additional 19 questions from the QSP form covered Choose doctor Goals and General satisfaction. The questionnaires were Discuss alternative treatments handed out by the receptionist to all women attending the Choose if students should be outpatient clinic and were alternated among the patients. present or not during Every other patient received a QPP questionnaire and the examination remaining patients received a QSP questionnaire. The women Environment Finding your way to the clinic were requested to fill in the forms during their visit but those Tidiness at clinic who for various reasons (lack of time being the most frequent Cosiness at clinic excuse) chose to fill in the forms after their visit received an Comfort at clinic stamped addressed envelope so that they could return their Atmosphere completed form by post. The number of forms distributed Possibility of speaking with to each group was 460. Two hundred and sixty (63%) of the the doctor in private QSP questionnaires and 247 (54%) QPP questionnaires were Accessibility Telephone hours for making returned. After sorting out the forms with incomplete answers, an appointment it was found that 60% of the QSP questionnaires and 52% Reaching the nurse for an of the QPP questionnaires had been returned satisfactorily. appointment Doctor s telephone hours Reaching the doctor by Statistical analysis phone The PQI was calculated for the QPP from the formula the clinic s opening hours described earlier. The QSP was analysed using the PLS method. Furthermore, a Structural Equation Modelling (SEM) was performed for both methods using STREAMS 1.7 (Structural Equation Modelling Made Simple) [15]. By use of the SEM technique, different features of the application of the quality factor were calculated [3]. Regression coefficients for two methods could be evaluated. The aspects brought up in the relationships between the quality factors and patient this study were: (i) confirmatory one-factor models for the satisfaction were estimated with standard errors taking quality factors were tested for goodness-of-fit and (ii) reliability estimates for the sum-index of variables for each quality factors into measurement errors in the 56 questions underlying the seven account. 260

5 An evaluation of QSP and QPP: two methods measuring patient satisfaction Table 4 Background data of the QSP and QPP questionnaire respondents QSP QPP Respondents... n=260 n=247 Median age (age range) 42 (14 80) 41 (11 80) Swedish as native language 86% 90% What level of education do you have? Elementary school, primary school or similar 22% 20% Middle school, junior-high or similar 24% 23% High school 21% 20% University or college 34% 37% Was this your first visit to the clinic? Yes 26% 20% No, I have been here before occasionally 21% 30% No, I have been here several times before 53% 50% How would you rate your present state of health? Very good 30% 27% Rather good 48% 52% Neither good nor bad 16% 14% Rather bad 5% 6% Very bad 1% 1% Results Patients background data including age, native language, education, present state of health and previous visits to the clinic are presented in Table 4. The women s ratings of the different factors in the QPP and QSP questionnaires are presented in Table 5. In the QPP model, the results regarding perceived reality, subjective importance, personal quality index are presented using both the PLS and the SEM-analysis techniques. Treatment by the doctor and medical care received the highest scores in the perceived reality domain. Accessibility received the lowest score. The PQI was calculated according to the formula given above, and the scores ranged between 3.1 and The patients placed highest value on treatment by the doctor and medical care but accessibility and participation received low scores. In the QSP, the total PSI received a relatively high score of 81.8 out of a possible 100. Also in the QSP model treatment by the doctor and medical care received high scores while participation and accessibility received low scores. As previously, a calculation was performed for every factor studied in order to explain the variance in the PSI. This means that a high value indicates high importance on the PSI and vice versa. As a comparison, the regression coefficient was also calculated for the QSP data using the SEM-technique (Table 5). In Figure 3 the mean values of the factors are plotted on the y-axis and their impact on the PSI is shown on the x- axis as described above. The results can be illustrated in a coordinate system with the ranking value on the y-axis and its impact on the total PSI on the x-axis. Factors with high scores and large impact on the total PSI are placed in quadrant I, while factors with high scores but low impact are placed in quadrant IV. In the same way, factors with low scores but large impact are placed in quadrant II, while factors with low scores and low impact are placed in quadrant III. Treatment by the doctor and medical care were placed in quadrant I, which consequently means that these factors had both high impact and ranking. Also environment turned out to have a relatively high impact on the PSI while its ranking was moderate. Accessibility was ranked as the lowest factor but had a significant impact on the PSI. The three remaining factors displayed no significant impact on the PSI. The variance in the PSI explained by the seven quality factors was as high as 79%. This indicates that about 62% of the PSI variance was captured by the QSP questionnaire. The alternative, SEM method was a two-step procedure: (i) testing and adjusting a SEM model to the QSP data, and (ii) estimation of regression coefficients with the so called Max- imum Likelihood Method (MLM) taking measurement errors into account [16]. The results are presented in Table 5 and can be compared with the PLS regression coefficients. As seen from the figures, results were found to be similar to the PLS data. Conclusion Measuring patient satisfaction has become increasingly popular. It is important to discuss how these results can be integrated into the health care process in order to increase patient satisfaction. Early in the planning of a study it is therefore important to integrate in what context the results will be used. This should, if possible, include the headings: goals, relevant questions, interpretation and intervention. The aim of the present study was to perform an evaluation of the two different methods with respect to the proposed 261

6 J. Nathorst-Böös et al. Table 5 The women s judgement of the different factors in the QPP and QSP questionnaires QSP SEM QPP QPP QSP Importance Importance Perceived reality Subjective importance QPP Experienced quality PLS regression SEM regression Quality... factors (mean±95% CI) (mean±95% CI) PQI (mean±95% CI) coefficient coefficient (mean±95% CI) (mean±95% CI) Medical care 3.64 ( ) 3.76 ( ) ( ) 2.5 ( ) 4.5 ( ) Treatment by the doctor 3.68 ( ) 3.83 ( ) ( ) 3.2 ( ) 1.8 ( ) Treatment by the nurse 3.45 ( ) 3.51 ( ) ( ) 0.5± n.s. 0.4 (0 1.2) Information 3.33 ( ) 3.72 ( ) ( ) 0.5± n.s. 0.8±n.s. Participation 2.68 ( ) 3.27 ( ) ( ) 0.6 (0 1.4) 0.9 (0 3.1) Environment 3.23 ( ) 3.20 ( ) ( ) 2.7 ( ) 2.4 ( ) Accessibility 2.53 ( ) 3.23 ( ) ( ) 1.4 ( ) 1.1 ( ) n.s., not significant. 262

7 An evaluation of QSP and QPP: two methods measuring patient satisfaction Figure 3 The mean value of the different quality factors in the QSP model and their impact on the PSI. field of application, and the methods strengths and weak- nesses. Sixty-three per cent of the QSP and 54% of the QPP forms were returned indicating that the QPP form was somewhat more time consuming to complete, resulting in a higher dropout rate. Regarding the SEM analysis of the References dimensions of quality; that is of the seven quality factors, the two methods generated the same factor structure. As ex- 1. Carr-Hill R. The measurement of patient satisfaction. J Public pected, a lower reliability index was found for the four-point Health Med 1992; 14: QPP scale than for the 10-point QSP scale. This means that 2. Raphael W. Do we know what patients think? A survey comfor this group of women patients a better measure is achieved paring the views of patients, staff and committee members. Int if the more differentiated 10-point scale is used. For other J Nursing Studies 1967; 4: patients, for example elderly people, the four-point response scale may be preferred, as it is simpler and quicker to answer 3. Locker D, Dunt D. Theoretical and methodological issues in sociological studies of consumer satisfaction with medical care. and generates the same answer pattern, although it may have Soc Sci Med 1978; 12: a somewhat higher measurement error rate. The ease of interpreting the results and developing a basis 4. French K. Methodological considerations in hospital patient of action is different for each model. In the QSP, this is opinion surveys. Int J Nursing Studies 1981; 18: incorporated directly into the method of questioning which 5. Carr-Hill R, Dixon P, Thompson A. Too simple for words. is built upon a structural equation model that shows how Health Serv J 1989; 99: quality factors, patient satisfaction and the clinic s goals are related to each other. The equation results in a series of 6. Meredith P, Wood C. Aspects of patient satisfaction with coefficients of importance indicating the different factors communication in surgical care: confirming qualitative feedback through quantitative methods. Int J Qual Health Care 1996; 8: relative impact on patient satisfaction. These relationships can subsequently be presented graphically and the presentation describes the mutual relationship of the factors involved and 7. Ware JE, Sherbourne CD. The MOS 36-Item short form health gives a good indication for how intervention can be made. survey (SF-36): 1 Conceptual framework and item selection. The high level of explained variance in the PSI indicates that Med Care 1992; 30: the modelling of the QSP data has given reliable results. 8. Hays RD, Larson C, Nelson EC, Batalden PB. Hospital quality In this study the same questions concerning experienced trends: a short form patient based measure. Med Care 1991; 29: reality were used in both the QSP and the QPP. The QPP form also included the subjective judgement of importance. and from which suitable questions can be selected. This firststep procedure is important; it addresses the issue of validating the questionnaire and is not included in the QSP model. In the QPP model, a PQI is directly calculated on the basis of the patient s estimation of subjective importance and experienced reality. The properties of the formula resulting in this index are, however, inconsistent and the same index can be reached by means of several combinations of importance and reality factors, which makes interpretation difficult. The index gives no information on how patient satisfaction changes if quality improvement is made. As a contrast between these two methods, analysis of these additional data specific to the QPP was set as a secondary priority, mainly because this method showed a tendency towards estimating most subjective importance factors as very important or important. In summary, the advantages of the QPP model are that it has a comprehensive and solid question bank. This consists of 56 questions extracted by means of multiple regression analysis from 900 indicators. The QSP model s advantage is its immediate usefulness and its clear graphic presentation. The importance of the different factors are grounded in the patients values but can be extracted from the model. An integration and further development of these two approaches could prove useful. The QPP model has its basis in orally conducted interviews aimed at investigating what issues in health care are considered important by patients. This process has resulted in a question bank where questions are grouped into specific categories 9. Tarlov A, Ware JE, Greenfield S et al. The Medical Outcomes Study: An application of methods for monitoring the results of medical care. J Am Med Assoc 1989; 262: Wilde B, Starrin B, Larsson G, Larsson M. Quality of care from 263

8 J. Nathorst-Böös et al. a patient perspective a grounded theory study. Scand J Caring Sci 1993; 7: departments of ophthalmology. Int J Health Care Quality Assurance 1997; 10: Fornell C, Cha J. Partial Least Squares. In Bagozzi RP, ed., 11. Scott A, Smith RD. Keeping the customer satisfied: Issues in Advanced Methods of Marketing Research. Cambridge: Blackwell, the interpretation and use of patient satisfaction surveys. Int J 1994: pp Qual Health Care 1994; 16: Gustaffson HE, Stahl PA. STREAMS User s Guide Version Wilde B, Larsson G, Larsson M, Starrin B. Quality of care. Gothenburg: Multivariate Ware, Development of a patient-centered questionnaire based on a 16. Hox JJ, Becger TM. An Introduction to Structural Equation grounded theory model. Scand J Caring Sci 1994; 8: Modelling. Family Science Review 1998; 11: Eckerlund I, Jönson B, Tambour M, Westlund AH. Change oriented patient questionnaires testing a new method at three Accepted for publication 23 February