Consumer Satisfaction at a Child and Adolescent State Psychiatric Hospital Stuart Kaplan, M.D. Joan Busner, Ph.D. John Chibnall, Ph.D. Glara Kang, B.A. Objective: This study examined satisfaction with services among patients in a child psychiatric hospital and their parents, and assessed the relationship between consumer satisfaction and the perception of improvement in the problem that led to hospitalization. Methods: A consumer satisfaction survey developed by the investigators was administered to three sampling waves of child and adolescent psychiatric inpatients (N=157) and their parents or guardians (N=111). Ninety-five percent of patients contacted and 97 percent of their parents or guardians agreed to participate in the study. The survey provided data about the children s and parents satisfaction with inpatient care and their perceptions of the children s clinical improvement. Results: Most parents and children reported high satisfaction with patient care. Twenty-eight percent of children and 21 percent of parents reported some form of abuse by the staff during the hospital stay. Those who reported abusive behavior were significantly less satisfied with the hospital experience than those who did not report abuse. The participants perception of clinical improvement was only weakly related to their satisfaction. Conclusions: Most child psychiatric patients and their parents will participate in consumer satisfaction surveys about inpatient care. Consumers are critical of a hospital if specific prompts in the survey are provided. An unexpectedly high level of consumer-reported abuse was found. Consumer-perceived clinical improvement was only weakly related to satisfaction. (Psychiatric Services 52:202 206, 2001) Consumer satisfaction is an important outcome measure of child mental health service delivery. With relative ease, child psychiatrists can document consumer satisfaction and use it as a measure of quality of service delivery for regulatory agencies and insurance companies. This study illuminates a number of methodological and conceptual issues in measuring consumer satisfaction in a children s psychiatric inpatient setting. Young and colleagues (1) have reviewed the studies on outpatient child mental health consumer satisfaction Dr. Kaplan is professor of psychiatry and director of child and adolescent psychiatry and Dr. Busner is associate professor of psychiatry and director of psychiatric clinical trials at the Penn State College of Medicine Hershey Medical Center. Dr. Chibnall is associate professor of psychiatry at Saint Louis University School of Medicine, where Ms. Kang is a medical student. Address correspondence to Dr. Kaplan at the Hershey Medical Center, Department of Psychiatry, MC H073, 500 University Drive, Hershey, Pennsylvania 17033 (e-mail, slk26@psu.edu). This paper was presented at the American Psychiatric Association Psychiatric Services Institute, held October 29 to November 2, 1999, in New Orleans. surveys. Methodological problems identified by Young and other investigators include potential sampling bias because of a low response rate (1 4); the increased complexity of child surveys compared with adult surveys (1,4,5), since both patients and their parents must be surveyed; and cognitive immaturity of child patients (1,4,6). They also found a dearth of psychometric data on the instruments used in consumer satisfaction surveys (1 3,6,7). Studies of consumer satisfaction with the delivery of outpatient child mental health services have tended to use more rigorous methodology than studies of inpatient services. Although this area of research is receiving increasing attention (4 11), a number of methodological problems remain unsolved. In the only published study that we found of consumer satisfaction with child psychiatric inpatient services (12), the parents and therapists of 42 child psychiatric inpatients were interviewed four to 17 months after the patients had been discharged from a ten-bed acute psychiatric unit in a pediatric hospital. Seventy percent of the parents reported that the treatment in the hospital had been very helpful ; six percent found it very unhelpful. Only the parents were interviewed, and there was no attempt to study the psychometrics of the interview instrument. One goal of our study was to assess the overall satisfaction of patients in a child psychiatric inpatient hospital and their parents. Another goal was to examine the relationship between consumer satisfaction and parents and children s perception of improvement in the problem that had led to 202
hospitalization. We hypothesized that the greater the perception that the problem had been resolved, the greater the satisfaction. Methods Subjects Of the 175 patients contacted, 166 (95 percent) agreed to be interviewed. A parent or caretaker was available for 114 of the 166 patients (67 percent), and of those, 111 (97 percent) agreed to participate in the study. The patients were unselected consecutive child psychiatric inpatients at an 80-bed freestanding state children s psychiatric hospital in New York. The patients characteristics are summarized in Table 1. Instrument We developed a four-section, 28-item questionnaire to cover our areas of interest. All ratings were made on 5- point Likert-type scales. In the first section, respondents made an overall evaluation of the extent to which the primary problem the diagnosis primarily responsible for hospitalization changed as a result of treatment at the hospital. In sections 2 and 3, respondents rated their satisfaction with treatment experiences and other characteristics of the hospital for example, types of therapy, medications, activities, staff, and the physical plant. The fourth section used a yesor-no format to inquire about abuse of patients by the staff and drug use by any staff member or patient at the hospital. For children younger than 13 years of age, several of the items employed a smiling face or frowning face format rather than a numeric format. Parents made a final overall evaluation of the likelihood that they would return to the facility if their child needed to be hospitalized again. Procedure Three waves of patients and parents were sampled, from March to April 1994, May to July 1995, and October to December 1995. A target sample of 50 patients was set for each wave. Patients were contacted by a social worker within two weeks preceding their discharge and asked to answer a questionnaire concerning satisfaction or dissatisfaction with the hospital. Table 1 Characteristics of 166 patients in a children s psychiatric hospital who participated in a consumer satisfaction survey 1 Characteristic N % Mean SD Gender Male 108 65 Female 58 35 Age (years) All patients 12.8 2.8 Six to 12 62 37 9.7 1.8 13 to 17 104 63 14.6 1.3 Length of stay (days) 52.8 56 1 Information about race was not obtained. Consecutive patients were contacted until the target sample number was met. The total number of patients surveyed slightly exceeded the target number for all three waves, for a total of 166. The questionnaires were completed in the presence of a social worker, who was available to answer any questions. Data analysis To reduce the number of variables and to create conceptually meaningful composite scores for analysis, the multiple items measuring satisfaction (sections 2 and 3 of the questionnaire) were first subjected to principal components analysis, for child and parent responses separately. For each analysis, principal components were extracted, and factors with eigenvalues exceeding 1 were rotated with a Varimax rotation. Factor scores were then calculated for each factor by using the regression approach. This approach weights standardized versions of the original variables by the factor coefficients to yield a single factor score for each case for each factor. When combined with principal components analysis, the regression method yields factor score distributions with means of 0 and standard deviations of 1. Multiple regression was then used to predict overall evaluation ratings from the satisfaction factor scores. A standard regression procedure was used whereby all predictors were entered into the regression equation simultaneously. Entry into the equation required a given predictor to explain a significant proportion of the variance remaining after the variance explained by all other variables in the model was accounted for. The equation was first evaluated for its ability to explain a significant amount of variance in the criterion, as determined by R 2. For equations with a significant R 2, individual predictors were evaluated by examining the standardized beta weights associated with each. Weights significantly greater than zero as determined by a t test were considered significant predictors. Residual analysis was conducted to evaluate violations of assumptions. When serious assumption violations occurred, logistic regression was used on dichotomized criterion variables, with the odds ratio as the effect size indicator. Data were analyzed for differences between the three sampling wave groups as well as between parents and children, between children aged 13 years and older and other participants, and between children aged 12 years and younger and other participants. Because no differences were found between the sampling wave groups or between the children s groups, we were able to collapse the sampling wave groups into one and the children s age groups into one. Thus we retained parents and children of all ages as the two grouping categories. Results The overall ratings across the groups from the three sampling waves reflected high levels of satisfaction with services received, a high degree of perceived change in the problem that led to hospitalization, and a high likeli- 203
Table 2 Satisfaction ratings by patients in a children s psychiatric hospital and their parents hood of using the facility again. Table 2 presents the participants ratings. Perception of abuse Disturbingly, 21 percent of parents and 28 percent of children reported that a staff member had verbally abused a child and 5 percent of parents and 10 percent of children reported physical abuse by a staff member. Less than 1 percent of parents and 1 percent of children reported sexual abuse by a staff member. We categorized the child and parent samples into those who did and those who did not report either worrying about abuse or actual abuse. Reports of sexual abuse were excluded because the prevalence was too low for statistical comparisons. Comparisons were then made on ratings of satisfaction, perceived improvement of the problem, and the likelihood of using the facility again. As shown in Table 3, children liked the staff less if they were worried about abuse or if they reported abuse. Among parents, Parent (N=111 ) Child (N=166 ) Categories evaluated Mean SD Mean SD Individual category 1 Handling of medical problems 4.1 1.1 Handling of complaints 4.0 1.1 Parent s ability to reach child by phone or visit 4.4.9 Handling of child s money 4.6.7 Staff 4.6.7 Understanding of child s problem 3.8 1.2 Medication 4.0 1.1 3.9 1.1 Family therapy 4.4 1.1 4.1 1.1 Therapist 4.4.8 Behavior modification program 4.0 1.0 Group therapy 4.0 1.0 Activities 4.1.9 Trips 4.4.9 Pool 4.1 1.1 Cottages 3.6 1.3 Food 2.5 1.4 School program 3.9 1.0 Being with staff 3.6 1.2 Talking to staff 3.9 1.1 Overall evaluation Amount main problem improved 2 4.1.8 4.4.7 Likelihood of using facility again 3 4.1 1.3 Overall Ns; minor variations in item response Ns 1 Rated on 5-point scale, with 1 indicating least satisfaction and 5 most satisfaction 2 Rated on 5-point scale, with 1 indicating got much worse and 5 got much better 3 Rated on 5-point scale, with 1 indicating least likely and 5 most likely satisfaction with staff and the likelihood of returning were significantly lower among those who reported that their child had experienced abuse. By contrast, parents or children s perceptions about problem improvement were not associated with whether or not they reported abuse. Problem improvement and likelihood of return A principal components analysis of parents satisfaction ratings yielded three factors. The first factor accounted for 35.4 percent of the variance and reflected satisfaction with the staff s understanding of the child s problem, the staff s handling of complaints, and the parents involvement in therapy. This factor, labeled parent needs, had an internal consistency reliability (alpha) coefficient of.73. The second factor accounted for 14.6 percent of the variance and reflected satisfaction with medication use and attention to medical problems. This factor, labeled medical services, had an alpha of.69. The third factor accounted for 13.3 percent of the variance and reflected satisfaction with staff and procedures such as those governing visitation. This factor, labeled staff-facility, and had an alpha of.62. Factor scores were used to predict perceived problem improvement and likelihood of using the facility again as rated by parents. As shown in Table 4, the three factors explained significant amounts of variance in both items (perceived problem improvement, R 2 =.13, p<.001; likelihood of returning, R 2 =.41, p<.001). Medical services was the only unique predictor of perceived problem improvement, while parent needs and staff-facility predicted the likelihood of using the facility again. The scatterplots of predicted and residual values for prediction of perceived problem improvement showed no evidence of significant violations of normality, homoscedasticity, or linearity. The scatterplot for prediction of the likelihood of return showed evidence of nonnormality, manifested by negative skewness of residuals at higher predicted values. Transformations of the data did not correct this problem. A simultaneous logistic regression was then performed. The likelihood of using the facility again was dichotomized by use of a median split (median=4.11), resulting in one group of 49 cases (29.5 percent) and one group of 117 cases (70.5 percent). The resulting logistic model was statistically reliable (χ 2 =45.7, df=3, p<.001; classification accuracy=82.5 percent). However, only parent needs emerged as a significant predictor of return likelihood (odds ratio=2.3, 95 percent confidence interval=1.5 to 3.5). Both medical services and stafffacility yielded nonsignificant odds ratios. The principal components analysis of the child satisfaction ratings yielded four factors. The first factor accounted for 31.3 percent of the variance and reflected satisfaction with the facility and its amenities for example, activities, trips, and cottages. It was labeled facility-amenities, and it had an alpha of.75. The second factor accounted for 9.5 percent of the variance and reflected satisfaction with staff. It was labeled staff, and it had an 204
alpha of.77. The third factor accounted for 8.8 percent of the variance and reflected satisfaction with medications and individual and group therapy. It was labeled medication-therapy and had a relatively low alpha of.54. Only one item satisfaction with family therapy loaded on the final factor, which accounted for 8 percent of the variance. Regression was used to predict the children s perceived improvement of their problem from these four factors, and a statistically significant equation emerged (R 2 =.16, p<.001). Only satisfaction with medication or therapy uniquely predicted perceived problem improvement, as shown in Table 4. The scatterplot of predicted and residual values revealed linearity, relatively normally distributed residuals, and no evidence of heteroscedasticity. Discussion and conclusions Both parents and children reported high satisfaction with the services provided by the hospital, and they reported that the problem that had led to hospitalization had been greatly helped. However, it is characteristic of consumer satisfaction surveys that patients and other consumers rate themselves as highly satisfied, regardless of the quality of services provided (2,13). An often-used theoretical explanation for this phenomenon is drawn from cognitive dissonance theory (14) that is, to the extent that a consumer believes he or she has freely chosen a service, acknowledging that the service is poor will be discrepant (dissonant) with the consumer s self-perception as a wise chooser of services. A drive to avoid dissonance in this scenario, then, would tend to push the consumer s attitude toward service in a more favorable direction. The high abuse rate reported by the parents and children in our sample indicates that despite the high rates of satisfaction reported, the patients and their parents were able to criticize the hospital. To elicit information about untoward events, specific inquiries seemed to be required. We hypothesized that parents and children s ratings of problem improvement would be strongly related to ratings of overall satisfaction, and Table 3 Satisfaction ratings by patients in a children s psychiatric hospital and their parents as a function of abuse reports that those who had obtained the help they had sought would be inclined to value the service highly. An important issue raised by our study is the interpretation of the parental report that Likelihood Perceived Satisfaction of using problem with staff 1,2 facility again 1,3 improvement 4 Respondent and category Mean SD Mean SD Mean SD Parent (N=111 ) Worried about physical abuse Yes 3.6 1.9 3.0 1.8 4.0.7 No 4.6.8 4.3 1.1 4.3.8 Verbal abuse occurred Yes 4.1 1.5 3.5 1.6 4.0.7 No 4.7.7 4.3 1.0 4.2.8 Physical abuse occurred Yes 3.0 1.9 2.6 1.8 4.0 1.0 No 4.6.8 4.2 1.1 4.1.8 Child (N=166 ) Worried about physical abuse Yes 3.1 1.4 4.4.8 No 3.7 1.1 4.4.8 Verbal abuse occurred Yes 3.1 1.3 4.3.7 No 3.9 1.0 4.4.8 Physical abuse occurred Yes 3.0 1.3 4.5.5 No 3.7 1.1 4.3.8 1 All yes-no comparisons were significant by independent t test (p.05). 2 Rated on 5-point scale, with 1 indicating least satisfaction and 5 most satisfaction 3 Rated on 5-point scale, with 1 indicating least likely and 5 most likely 4 Perceived improvement rated on 5-point scale, with 1 indicating got much worse and 5 got much better. All yes-no comparisons were not significant by independent t test (p.05). Overall N; minor variations in item response Ns Table 4 the child s problem had improved: Is such a report a valid judgment of the child s clinical improvement, or is it just another global measure of consumer satisfaction and largely unre- Prediction of overall evaluations from satisfaction factor scores on a survey of patients in a children s psychiatric hospital and their parents Perceived problem improvement Likelihood of using the facility again Predictors R 2 Beta weight R 2 Beta weight Parent.13.41 Parent needs.18.47 Medical services.29.13 Staff and facility.11.22 Child.16 Facility and amenities.14 Staff.08 Medication and therapy.29 Family therapy.15 p.05, multiple linear regression p.05, multiple linear regression only; effect nonsignificant with more conservative logistic regression analysis 205
lated to any change in the child s clinical condition? Bickman and colleagues (13) have provided the best study of this problem to date in the Fort Bragg experiment. In a confirmatory factor analysis of two mental health systems, they concluded that parent ratings of problem improvement were simply another measure of consumer satisfaction. The data from our study provide an additional opportunity to examine the relationship between parents ratings of children s improvement and parents satisfaction with services. We factor analyzed the parent s satisfaction ratings and used the factors as independent variables in two multiple linear regression equations. In the linear regression equation in which the willingness to come back to the hospital was the dependent variable, the four factors of consumer satisfaction together explained 41 percent (R=.64) of the variance in willingness to return to the hospital. Willingness to come back is a critical measure of consumer satisfaction, because, from an economic perspective, all other indexes of consumer satisfaction are directed to achieving an affirmative response to this query. The identified parental satisfaction factors explained only 13 percent (R=.36) of the variance of the dependent variable, problem improvement. We concluded from these analyses that parents satisfaction is significantly but only weakly related to parent-rated problem improvement and much more strongly related to another measure of consumer satisfaction, willingness to return to the hospital. That parents assessment of the children s improvement reflects something largely distinct from consumer satisfaction is most clearly demonstrated in the analyses comparing measures of consumer satisfaction and problem improvement ratings among parents who did and did not complain of abusive staff behavior. Both groups reported substantial problem improvement, but the parents who complained of abusive staff behavior were much less satisfied with the hospital than those who did not complain. Parents are not the only ones who can discriminate between problem 206 improvement and satisfaction with the hospital; child patients make similar discriminations. Children who complained of staff abuse reported a level of problem improvement that was statistically similar to that reported by children who did not complain of abuse, yet they reported significantly less satisfaction with the staff than did their counterparts. The amount of variance in perceived problem improvement explained by consumer satisfaction factors among children (16 percent) was almost identical to the amount of variance in perceived problem improvement explained by the parent factors of consumer satisfaction (13 percent). This finding suggests that for both children and parents the perception of problem improvement was largely independent of overall satisfaction. Our study suggests that consumer satisfaction can be measured with satisfactory levels of response when consumers are individually approached and when a trained interviewer is present. Clinically, we were keenly interested in using a consumer satisfaction survey to identify intrainstitutional issues, and the survey did prove useful in identifying these areas for further administrative and clinical intervention. There may be clinical utility in performing surveys at multiple time points, as problems identified can be addressed and solutions later evaluated. Consumer satisfaction surveys that assess perceived problem improvement should include independent measures of clinical improvement along with parent- and child-rated global measures of improvement. References 1. Young SC, Nicholson J, Davis M: An overview of issues in research on consumer satisfaction with child and adolescent mental health services. Journal of Child and Correction Family Studies 4:219 238, 1995 2. Lebow JL: Client satisfaction with mental health treatment: methodological considerations in assessment. Evaluation Review 7:729 752, 1983 3. Lebow JL: Research assessing consumer satisfaction with mental health treatment: a review of findings. Evaluation and Program Planning 6:211 236, 1983 4. Godley S, Fiedler E, Funk R: Consumer satisfaction of parents and their children with child/adolescent mental health services. Evaluation and Program Planning 21: 31 45, 1998 5. Cournoyer DE, Johnson HC: Measuring parents perceptions of mental health professionals. Research on Social Work Practice 1: 399 415, 1991 6. Shapiro JP, Welker CJ, Jacobson BJ: The Youth Client Satisfaction Questionnaire: development, construct validation, and factor structure. Journal of Child Clinical Psychology 26:87 98, 1997 7. Stallard P: Validity and reliability of the parent satisfaction questionnaire. British Psychological Society 35:311 318, 1996 8. Lambert W, Salzer MS, Bickman L: Clinical outcome, consumer satisfaction, and ad hoc ratings of improvement in children s mental health. Journal of Consulting and Clinical Psychology 66:270 279, 1998 9. Kiser LJ, Millsap PA, Hickerson S, et al: Results of treatment one year later: child and adolescent partial hospitalization. Journal of the American Academy of Child Adolescent Psychiatry 35:1,81 90, 1996 10. Loff CD, Trigg LJ, Cassels C: An evaluation of consumer satisfaction in a child psychiatric service: viewpoints of patients and parents. American Journal of Orthopsychiatry 57:132 134, 1987 11. Hernandez M, Hodges S, Cascardi M: The ecology outcomes: system accountability in children s mental health. Journal of Behavioral Health Services and Research 25:2, 137 150, 1998 12. Bradley EJ, Clark BS: Patients characteristics and consumer satisfaction on an inpatient child psychiatric unit. Canadian Journal of Psychiatry 38:175 180, 1993 13. Bickman L, Gutherie PR, Foster EM, et al: Evaluating Managed Mental Health Services: The Fort Bragg Experiment. New York, Plenum, 1995 14. Festinger L: A Theory of Cognitive Dissonance. Stanford, California, Stanford University Press, 1957 The Web site address of the Center for Psychiatric Rehabilitation at Boston University was given incorrectly in Web Sites Worth Watching in the October 2000 issue (page 1241). The correct address is www.bu.edu/sarpsych.