Range Ment~l Health Center Chi1dren s Summer Day Treatment Program Program Evaluation Prepared by: Melissa Wells Project Supervisor: Mike Raschi~k, Ph.D.
.. \. Introduction This study evaluates the summer component of a rural Minnesota day treatment program for children with mental health needs. This school-based program includes both traditional "in-office" mental health services and a "therapeutic recreation program." Participant and staff ratings were utilized in measuring changes in participants' level of self-esteem and their behavioral cooperation. The ADAPT (Accept Differences and Pull Together) children's day treatment program at the Range Mental Health Center (RMHC), began serving clients in the fall of 1993 through its Hibbing, Minnesota office. In the winter of 1994 the Virginia Range Mental Health Center office began the program. The school-based day treatment program provides mental health services to children ages 8 to 13. The program is designed as a preventative program for children with mental health needs. The objective of ADAPT is to provide support for children at risk of out of home placement. Description of the ADAPT Program In order to be eligible for the program, children must.. meet several criteria. Children may be referred for services by teachers, parents, and-or mental health professionals. Upon referral, the child is seen individually, as well as with parents-guardians for a diagnostic assessment.
2 In order to participate, children must be diagnosed as having a disorder of early childhood or adolescence as defined by the Diagnostic and Statistical Manual of Mental Disorders, third edition, revised (American Psychiatric Association, 1987)~ Children referred to the program most frequently have been diagnosed with conduct disorder, oppositional deviant disorder, reactive attachment disorder, attention-deficit hyperactivity disorder, and other disorders of early childhood and adolescence. The program falls under Rule 47 (the section on day treatment) of the Minnesota Department of Human Services Medical Assistance Providers Manual (1992), and as such specific criteria for program eligibility are mandated by the state of Medical Assistance guidelines. At this time, participants must be recipients of medical assistance in order to be involved in the program. The Medical Assistance guidelines provide specifics regarding program planning, children's needs and other criteria specific to children's day treatment (Medical ~ssistance Provider Manual, 1992). Range Mental Health Center currently offers the program in Virginia, Hibbing, Eveleth, and Gilbert schools. In addition to the services provided during the school year, RMHC provides a summer version of the ADAPT program for participants, a service mandated by the Medical Assistance guidelines (1992). In the summer program component, group therapy is the primary treatment modality and the goals include soc~al skill development and self-esteem enhancement. The summer
3 program has available a designated classroom site in a school building, although the majority of the activities, interventions, and experiences in the summer session.take place outdoors. A day long canoeing adventure in which the group is responsible for organizing appropriate gear, planning a paddling course, and actually following through with the plan they design is an example of this type of intervention. Prior to the canoeing experience, group facilitators provide paddling instruction and review group safety. During the activity, facilitators offer suggestions for working more efficiently or effectively, and assist group members in problem solving related to their particular situation. After the experience has been completed, staff encourage participants to reflect on what happened during the activity. Nadler (1992) stresses the importance of processing and application in adventure experience. In the ADAPT summer program, staff facilitate participant discussion of how they were behaving, thinking and feeling during the experience. This processing component may include debriefing questions such as: Joe, you and Sarah seemed to be having a difficult time heading straight ahead at the beginning of our paddle, but I noticed that by the end, you were zipping right along as straight as can be. What kinds
4 of things happened between you and Sarah to make that possible? This type of activity often allows children to see in a concrete way, positive results of their cooperative behavior. Questions may be also posed in a way that encourages group problem-solving: Mary and Jack stated that they yelled at each other for most of our adventure because they were so angry with each other for not keeping the boat straight. Can any of you help them to think of ways that they could have worked out this problem in a way that did not involve yelling at each other for the entire trip? Both during the activity and at the completion of the adventure, facilitators encourage cooperative behaviors and communication between group members. At the end of the activity, staff challenge participants to see connections between the adventure experience and real life situations. During the processing session, a participant may observe that by communicating with her canoeing partner they were able to learn the best way to paddle togethe-r. The facilitators encourage the participants to consider how this realization may apply to their family, school, or social life (Nadler, 1992). During the summer program, outdoor activities such as hiking, camping, swimming, cooperative games, canoeing, and kayaking are utilized as a means through which therapeutic
goals are met. 5 In addition to these activities, the program curriculum includes a daily verbal group, weekly "group topics" (such as non-violence, family systems, and chemical use/abuse), and individualized staff/client interventions when appropriate. At the beginning of the program, a comprehensive intervention plan is developed by the staff for each child, with input from both the child and the parent/guardian. The intervention plan includes specific target/problem behaviors, objectives to be met during the program, and tasks which relate to the identified objectives. For example, for a participant diagnosed as having a conduct disorder, a target/problem behavior may be aggression toward peers. The identified objective may be to reduce the number of fights with peers per group session initiated by participant by fifty-percent by the end of the third week of the program. Tasks related to this objective may include participant completion of problem-solving exercises, and daily check-in with staff regarding behavior in group. Although the majority of activities during the summer have a group focus, staff evaluate individual client treatment plans on a daily basis. Change in target behaviors identified in the intervention plan (such as aggression toward peers) are monitored and recorded. This research study evaluates this summer component of the ADAPT program. The evaluation examines what impact the program had on the self-esteem, cooperative skills, and target/problem behaviors of participants.
6 Literature review Since the ADAPT Program is a day treatment program that is based on a therapeutic recreation approach, I will begin by reviewing literature relating to day treatment for children and therapeutic recreation programs. I will then review literature relating to the two dependent variables I will be measuring. Day treatment program effectiveness for children with disruptive behavior problems Day treatment facilities offer supportive services for children with disruptive behavior problems. Day treatment programs differ from residential treatment environments in that they offer the "advantages of community location and preservation of links to family and peer group" (Grizenko, et al., 1993, p. 130). Children participate in behavior management and academic/task groups during day-time hours, and return to their family/peer system after the program. Day treatment has been shown to produce improvement in behavior and self-perception in children with disruptive behavior problems such as attention-deficit hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), and conduct disorder (CD) (Grizenko, el al., 1993). The behaviors these children present, such as loudness, impulsiveness, and defiance, are often disruptive to others.
7 Grizenko (1993) observes that: "As a result, they are at odds with their social environment in school, in the family, and at play, leading others to react punitively or to break contact" ( p. 131). Day treatment, in particular school-based models, offers opportunities for children to coordinate different components of their environment. The importance of mental health services which place children in the least restrictive treatment environment possible and that strive for a "match 11 between the child's problem and the approach to treatment has been stressed in the literature (Grizenko, 1993; Lecroy, 1992). Lecroy (1992) suggests that effective treatment "focuses on teaching children how to respond effetively to new situations, which produces more positive consequences than did past behaviors used in similar situations 11 ( p. 2 2 6) Therapeutic Recreation Theory and Practice Programming of the type described in this study appears in the literature under the descriptors therapeutic recreation, adventure programming, and wilderness therapy. The core concept of these models is the integration of experiential teaching of wilderness-based adventure with psycho-social counseling (Marx, 1988). Marx (1988) stresses that in therapeutic recreation, therapy is the goal, and outdoor adventure is the means. Marx further defines programs similar to the type identified in this study as an integration of "the experiential
8 teaching of a wilderness-based adventure program with the psychosocial counseling of a community-based human service organization" {p. 518). The Outward Bound literature suggests that personal growth and change do occur within the outdoor adventure curriculum (Godfrey, 1977). The changes that have been most documented have been in the areas of self-esteem, self-awareness, and acceptance of others (Burton, 1981). Outward Bound programs, however, are geared in large part toward high functioning individuals in search of a personal challenge (Godfrey, 1977). Much of the data available on Outward Bound programs is anecdotal in nature. The empirical research which has been done on the program has shown global outcome results rather than any systematic examination of the components of the Outward Bound experience (Burton, 1981). In a 1989 study, Clagett (1989) presented a study examining a wilderness program as treatment for delinquent youth. This study showed a significant reduction in recidivism rate (Clagett, 1989). This study, however, did not directly address the needs of children and adolescents referred to mental health practitioners. Kaplan (1974) studied the effect of an outdoor adventure program had on a group of adolescent clients of a community mental health center. This study suggested that after participation in the program, the clients reported more positive self-esteem, concern for others, and a more realistic view of their strengths and weaknesses (Kaplan,
1974). 9 Davis-Berman & Berman (1989) have designed therapeutic adventure programs as a component of their private practice. Evaluative studies of the The Wilderness Therapy Program have shown a number of significant effects. The program has been used with both inpatient and outpatient adolescent clients diagnosed with a number of psychiatric. diagnoses (Berman & Anton, 1988; Davis-Berman & Berman, 1989). These studies showed significant change in such measures as reduction of behavioral symptoms, self-esteem, and self-efficacy (Berman & Anton, 1988; Davis-Berman & Berman, 1989) Therapeutic recreation has been used with such diverse populations as traumatic head injured adults, bulemic adolescents, adolescents in psychiatric treatment (Fazio & Gralish, 1988; Kaufman, 1988; Witman, 1987). The meanings and significance of studying self-esteem and cooperation The significance of self-esteem for psychological and physical health, job satisfaction, academic achievement, and success in life is becoming increasingly evident (Reasoner, 1983). Burnett (1983) identifies self-esteem development as an "important goal for all children'' {p. 101). Reasoner (1983) defines self-esteem as a process of establishing a sense of; (a) security, (b) identity, (c) belonging, (d) purpose, and (e) personal competence. Security is gained through well defined limits and a sense
10 of trust; identity is established through encouragement of feelings of self-efficacy and an understanding of personal strengths; belonging relates to feelings of social acceptance; a sense of purpose is achieved through learning to set realistic goals; and personal competence is developed through feelings of being able to cope with problems or meet goals. A study by Engle & Raine (1963) asked third grade students to identify the main factors related to self-esteem and self-concept. The two main factors identified were: (1) quality of interpersonal relationships with peers and adults, and (2) personal attributes such as bravery, strength, and attractiveness. The rationale for including adventure programs in psychiatric treatment programs as a means of increasing self-esteem and self-efficacy has been documented in previous studies (Berman & Anton, 1988; Davis-Berma~ & Berman, 1989; Witman, 1987). Witman (1987) states that "completion of adventure activities deemed as challenging will lead to greater confidence and sense of self-empowerment" (p. 101) Cooperation has been defined as "working together to accomplish shared goals where individuals seek outcomes that are beneficial to themselves and beneficial to other group members" (Johnson et al., 1993, p. 67). Johnson et al. (1993) indicate a correlation between working cooperatively with peers and greater psychological health.
11 Witman (1987) identifies a number of manifestations of a cooperative attitude. He lists, among others; positive attitudes toward others, sharing, comforting peers, advice-giving, and enhanced interpersonal interaction. Therapeutic recreation programs have shown improvement in social skills. Witman (1987) states that social skills such as group cooperation and trust are "enhanced through the sharing and helping required by the cooperative problem-solving and 'spotting' for one another inherent in most adventure activities." Therapy in a wilderness setting often leads to the development and enhancement of "prosocial behaviors" such as cooperation, effective communication, and mutual support (Berman & Davis-Berman, 1989). Siqnif icance of topic to the social work profession Lecroy (1992) stresses the importance of the person-in-environment perspective of social work in in the children's mental health field. He states that "emotionally disturbed children and youth are one of the most underserved and disabled populations in the United States" and stresses their dependence on the environment. The program identified in this study addresses this ecological view of mental health treatment. The approach, combining the familiar school environment with the opportunities of wilderness adventure, offer possibilities for creative person-in-environment approaches. In a
12 natural setting, clients are often directly responsible for their behavior,attitudes, and feelings (Berman & Davis-Berman, 1989). In a wilderness setting, immediate feedback often results. Studies undertaken to identify skills relevant to adventure based counseling are very much applicable to the social work field. Study findings confirmed that group counseling skills used in traditional group counseling settings are also relevant to adventure based counseling. Skills identified included relational skills, empowering clients, and understanding of group dynamics (Gerstein, 1992). Adventure therapy is not traditional clinical work by any means. Berman & Davis-Berman (1989) stress that workers must be committed to this alternative type of treatment in order for programs to be successful. Planning involves traditional planning of group therapy with the additional organization of activity specific gear (such as food and camping gear), as well related knowledge of relevant skills (camping skills, knowledge of wilderness first aid). The typical outdoor adventure therapy program differs significantly from a traditional fifty minute therapy session. The intensity of the client-worker relationship and duration of the intervention is markedly different (Berman & Davis-Berman, 1989). Programming often lasts for a number of hours and in some cases days. The results of this study will be useful in developing day treatment programs with an emphasis on
self-esteem and cooperative behavior development. 13 It suggests a direction for day treatment programs with a "therapeutic recreation" focus wishing to focus on these variables. The study provides offers some insight into the effect of this type of program on participant self-esteem and cooperative skills and behaviors. Research questions and/or hypotheses This research was designed to test five hypotheses in regard to the Range Mental Health Center's ADAPT program. Hypotheses were addressed using data collected from the participants themselves, as well as by the program facilitators. The five hypotheses are: 1) Program participants self-esteem increases to a significant degree over the course of the program. 2). Participants' cooperative behaviors and cooperative attitudes positively change to a significant degree over the course of the ADAPT summer program. 3) The program is more effective with certain target behaviors in improving participant self-esteem. 4) The program is more effective with certain target behaviors in improving participant cooperative
" ' behaviors and attitudes. 14 5) Participants mean level of improvement is higher with certain target behaviors. Methodology Population and Sample The sample consists of 30 elementary-age students who participated in the Virginia and Hibbing ADAPT program summer session during the summer of 1994. All program participants who completed the program were included in the study. All information regarding subjects was provided to the researcher by Range Mental Health Center Staff. Participant names and any identifying data with the exception of designation of sex were removed prior to the beginning of the study. Subjects were coded by number. Research Design The study consists of secondary data analysis of results from three pre-test-post-test instruments and one post-test only evaluation. Two of the instruments were completed by program participants and two were completed by the program facilitators. The first instrument completed by participants at preand post-program, the Feeling Good about Me: Developing Self-Esteem inventory, measures individual ideas, beliefs, attitudes, or feelings about self (Appendix A). The
15 instrument measures self-perceptions using a Likert-type scale on a continuum ranging from 1 to 5. A rating of 1 represents "never"; a rating of 5 represents "always." The second instrument is a Self Concept Questionnaire developed by Range Mental Health Center staff (Appendix B). The questionnaire is designed to measure concepts such as ability to make friends, cooperative ability, and feelings about self. Students were asked to "make an X" through the face which shows the way they feel about nine questions. The faces are labeled, and presented Likert-style, with the face furthest to the left being "awful" and that furthest right "very good." The program facilitators completed the Stress Response Scale at pre/post program for each participant (Appendix C). This rating form, developed by Psychological Assessment Resources, Inc. consists of a scale of 40 items to be rated. Facilitators are to check off the score which "best describes the behavior of the child being rated." Rating is Likert-type, with a score of 0 indicating "never" and a score of 5 indicating "always." In examining degree of change in primary target behaviors of participants, the facilitators completed an inve~tory for each child in the program at post-test (Appendix D). This survey tool, developed by the researcher, allowed facilitators to identify one primary target behavior for each participant and to rate the degree of change in that target behavior over course of the summer session. Using a Likert-type scale, with a score of 1
16 indicating "greatly improved" and a score of 10 indicating "significantly worse", facilitators circled the number most representative of the degree of change for each participant. Dependent Variables Self-esteem was conceptualized as a sense of belonging (relating to feelings of social acceptance), personal competence (ability to cope with problems or meet goals), and purpose (learning to set realistic goals)(reasoner, 1983). In addition, self-esteem in this study was related to children's self-perception of quality of interpersonal relationships with peers and adults, and of personal attributes such as attractiveness (Engle & Raine, 1963). Children's overall perception of self, in addition to the facilitators' perceptions, was measured through the use of questionnaires (specific questionnaires are discussed in data collection section). Cooperative skills was conceptualized as the ability to interact interpersonally; evidence of attributes such as sharing; comforting peers; and participation in group activities (Johnson et al., 1993; Witman, 1987). This dependent variable was measured through the use of questionnaires (specific questionnaires are discussed in data collection section). Target behavior was identified as the one key behavior identified at intake as the primary focus of treatment. The rating of degree of change of target behavior was measured by staff on a scale from "greatly improved" to
"significantly worse." 17 Independent Variab1e The primary independent variable is participation in the ADAPT program summer session. The program consists of thirty three hour group sessions related to problem solving abilities, risk taking, and self-esteem. Activities were developed with a therapeutic recreation foc~s whenever appropriate. Brochure describing program is included in Appendix (Appendix E). Operationa1 Definitions and Data Co11ection Data was collected using materials supplied by Range Mental Health Center staff. RMHC staff forwarded original pre/post test instruments to researcher for study. All participant names were deleted prior to beginning of research. In analyzing the results of the Stress Response Scale completed by facilitators, items relating directly to the dependent variables were highlighted (Psychological Assessment Resource, 1982). The items examined related to self-esteem are as follows: #06. helpless (relates to personal competence [Reasoner, 1983]). # 27. self-confident (relates to personal competence [Resoner, 1983]). The items related to cooperative ski11s are as follows: #21. participation (relates to enhanced interpersonal
interaction [Witman, 1987]). 18 #39. Cooperative (relates to cooperative behaviors [Witman, 1987]). In analyzing the results from the "Feeling Good about Me" instrument, questions which relate directly to the dependent variables under study were highlighted. Questions analyzed in relation to evaluation of participant self-esteem are as follows: In Self Concept Questionnaire; "How you look" (personal attributes) "How you like yourself" (social acceptance) In Feeling Good About Me: Developing Self-Esteem; #1. I feel good about myself (belonging, personal competence as identified by Reasoner [1983]) #2. I feel as though I fit in at school (belonging, quality of interpersonal relationships as identified by Engle & Raine [1963]) #3. I feel as though I am a successful person so far in my life (personal competence [Reasoner, 1983]) Questions to be analyzed in relation to evaluation of participant cooperative skills are as follows: In Self Concept Questionnaire; "How well you take turns" (cooperative participation as described by Witman [1987]). "How you are at sharing things" (sharing as described by Witman [1987]).
19 "How well you take turns" (cooperative participation as described by Witman [1987]). "How you are at sharing things" (sharing as described by Witman [1987]). "How you are at joining in an participating in a group" (enhancing interpersonal interaction as identified by Witman [1987]). In Fee1ing Good about Me questionnaire: #7. Other kids want me to be their friend (enhancement of interpersonal interaction [Witman, 1987]). Participant responses to these questions were examined at pre and post-test for any significant change. Data Ana1ysis Data obtained in this study is quantitative in nature. In addressing the first and second research hypotheses, questions were initially clustered into two categories-those measuring self-esteem and those measuring cooperative behaviors. Changes in participant's self-esteem were determined through a three-step process. First, pre-test scores of each participant on each of the seven questions measuring self-esteem were subtracted from their respective post-test scores. This yielded changes scores for each question ranging from -2 to +2 (e.g., a decline of 2 points from a participant's pre-test to post-test scores on a particular self-esteem question would be recorded as -2; whereas if the pre- and post-test scores were the same, the
result would be O). 20 Then, in order to get an overall self-esteem change score for each respondent the total number of their positive change scores on all of the self-esteem questions was subtracted from the total number of negative change scores. For instance, if participant "X" had self-esteem change scores of -1, -2, O, +2, +2 and +2, her overall self-esteem scale score would be +l, since he had three positive scores and 2 negative ones. [This procedure was followed versus merely subtracting the total positive from the total negative scores--in the above case 6-3--due to the data being at the ordinal instead of interval level]. Finally, the scale scores for each individual respondent were added to determine how many respondents "stayed the same", etc. The results are summarized in table form, indicating frequencies of scale scores which: (1) declined, (2) stayed the same, and (3) improved for each question as well as cumulative frequencies for each category. These results were analyzed through Chi-square, comparing observed and expected values for "declined" and "improved". A similar procedure was followed to determine the patterns of responses versus respondents, e.g., if all 22 respondents answered a particular question, perhaps 7 showed a "decline, 7 an "increase", and 8 "stayed the same". In determining participants' cooperation, pre-test scores of each participant on the six questions measuring cooperation were subtracted from their respective post-test
scores. 21 The analysis then followed the process indicated for self-esteem determination. Respondents' self-esteem and cooperation scale scores obtained in evaluating the first two hypotheses were utilized in addressing the third and forth hypotheses. Net total scale scores for self-esteem and cooperation were analyzed in terms of participant target behavior. For each of the six target behaviors, the number of participants showing "net decline", "stayed the same" and "net improvement" were tabulated and summarized in table form. Then the net scores for each individual target behavior were added to determine the total number of participants "stayed the same", etc. In addressing the fifth hypothesis, the mean level of improvement for each target behavior was calculated and presented in table form. An ANOVA test was utilized to determine if the program is more effective with certain target behaviors. It should be noted that in evaluating the program's effectiveness in relation to target behaviors, the target behavior "elective mutism" identified by the program staff was eliminated. This target behavior applied to only one participant and for purposes of statistical analysis, the number of cases for each target behavior needed to be two or greater than two. The participant whose target behavior was identified as "elective mutism" was not included in data analysis for hypotheses #3, #4, and #5.
,.. III. Resu1ts 22 Hypothesis #1: Program participants' self-esteem increases to a significant degree over the course of the program. \ Table #1 describes the frequencies of change in participant self-esteem as measured by self-esteem scales between pre and post-test. The total number of respondents whose self-esteem scores declined was eleven (fifty percent), seven participants {thirty-two percent) stayed the same, and four participants (eighteen percent) improved over the course of the program. The results showed an overall decline from pre- to.,,. post-testf ~Chi-square analysis was done comparing observed and expected values. The resulting probability of.0707 indicates that the null hypothesis can not be rejected. Table #1 Respondent Self-esteem Scores (both programs) Chi-square 3.2667 DF =I Prob.. 0707 Total includes Case #2 responses to survey questions relating to self-esteem. In evaluating overall responses, self-esteem declined in 35% of responses, stayed the same in 41% of responses, and improved in 24% of responses. The results of the Chi-square test in evaluating the total number of responses indicating improvement and declined in self-esteem compared with
. 23 expected values indicate a probability of.4913. This value is not in the region of rejection for alpha =.05, thus the null hypotheses of statistical independence can not be reiected. Table #2 Responses to Self-esteem Questions (both programs) Chi-square 2.92208 DF = 1 Prob.. 0874 Due to~the lack of improvement in respondents'., 1..' self-esteem and lack of improvement in self-esteem responses, Hibbing and Virginia results were split to see if one program improved. Tables #2A and #2B describe frequency of change in participant self-esteem as measured by self-esteem scales between pre- and post-test in Hibbing and Virginia respectively. These results seem to indicate that program location is not a crucial variable in improving participant self-esteem. The results indicate that in the Hibbing program, participants' responses showed a significant decline in self-esteem. Seventy-five percent of participants showed decline in self-esteem and seventeen percent showed an increase. A Chi-square test run on actual and expected
24 change in participants' 11 declined 11 and "improved" response scores indicated a probability of 5413, thus the null hypothesis can not be rejected. The Chi-square test on actual and expected change in respondents "declined" and "improved" scores indicated a probability of.0348, indicating that the decline in respondent self-esteem in not due to chance. Table #2A Self-esteem Scores (Hibbing only) * Chi-Square.373134 DF = 1 Prob. 5413 ** Chi-Square 4.45455 DF = 1 Prob 0348 In the Virginia program, twenty percent of participants declined and twenty percent improved in self-esteem. The Chi-square results for both responses and respondents were not in the region of rejection for alpha =.05, thus the null hypothesis can not be rejected.
25 Table #2B Self-esteem Scores (Virginia only) * Chi-Square 10 DF = 1 Prob.. 6547 ** Chi-Square 0 DF = 1 Prob. 1.0 Hypothesis #2: Participants' cooperative behaviors and attitudes positively change to a significant degr~~ over the course of the ADAPT summer program. :.. The results showed an overall tendency to decline from pre- to post-test in cooperative behaviors and attitudes. Table #3 describes the frequency of change in respondent cooperative behaviors and attitudes over the course of the program as measured by cooperation scale. The total number of respondents whose cooperation scores declined was ten (forty-five percent), five stayed the same (twenty-three percent), and seven (thirty-two percent) showed improvement. The results cf the Chi-square test in evaluating the total number of respondents whose cooperation scores improved and declined compared with expected values indicate a probability of.4669. This value is not in the region of
_.. 26 rejection for alpha =.05~ thu~ the null hypothesis can not be rejected. Table #3 Respondent Cooperation Scores (both programs) Chi-square.529412 DF = 1 Prob.. 4669 *Total includes Case #2 Table #4 describes the frequency of change in responses to cooperation questions for both programs. In examining overall participant responses to survey questions, comparison between pre and post-test scores indicate that cooperative behaviors and attitudes declined in 24% of responses, stayed the same in 51% of responses, and improved in 25% of responses. The r~~ults of the Chi-square test in evaluating the :~ total numbe''x;-:-. of responses that declined and improved compared with expected values indicate a probability of.8946. The resulting value indicates that the null hypothesis can not be rejected. Table #4 Responses to Cooperation Questions'(both programs) Chi-Square.0175439 DF = I Prob.. 8946
27 Due to the lack of improvement in respondents cooperation and lack of improvement in cooperation responses, Hibbing and Virginia results were split to see if one program improved. Tables #4A and #4B describe frequency of change in participant cooperation as measured by. ' cooperation scales between pre- and post-test in Hibbing and Virginia respectively. These results seem to indicate that program location is not a crucial variable in improving participant cooperation. The results indicate a decline in cooperation scores in both programs. In Hibbing, fifty percent of participants declined and forty-two percent improved. The Chi-square analysis of responses indicated a probability of.0411, suggesting ~hat :.. decline in participant responses was not due to chance. The Chi-square test on respondents indicated a value of.7630, thus the null hypothesis can not be rejected. Table #4A Cooperation Scores (Hibbing Only) *Chi-square 4.17 DF =I Prob.. 0411 **Chi-square.0909 DF = 1 Prob.. 7630 In the Virginia program, forty percent of participants declined and twenty percent improved. The Chi-square
,... 28 analysis of responses indicated a probability of.4913, suggesting that decline in participant responses was due to chance. The Chi-square test on respondents indicated a value of.4142, thus the null hypothesis can not be rejected. Table #4B Cooperation Scores (Virginia Only) *Chi-square.473684 DF 1 Prob.. 4913 **Chi-square.6667 DF = 1 Prob.. 4142 Hypothesis #3: The program is more effective with certain target behaviors in improving participant I :.. self-esteem. Table #5 describes the program's effectiveness in improving self-esteem in relation to participant target problem behavior. Net improvement in self-esteem occurred among those participants whose target behaviors were anger control (2 cases), withdrawn (1 case), and low self-esteem (1 case). Four participants scores reflected net improvement in self-esteem. There may be a tendency for self-esteem to decrease in those participants whose target behavior is low self-esteem. Net decline in self-esteem occured in five cases whose target behavior was low self-esteem. Decline
' ' 29 in self-esteem also resulted in participants whose target behaviors were anger control (2), oppositional behavior (1), lack of social skills (1), and depressed mood (2). Self-esteem stayed the same for target behaviors anger control (1), oppositional behavior (2), withdrawn (1), and lack of social skills (2). Table 5 Net Self-esteem scores. ':'- ' Hypoth~sis #4: The program is more effective with certain target behaviors in improving participant cooperative behaviors and attitudes. Table #6 describes the program's effectiveness in improving cooperative behaviors and attitudes in relation to participant target problem behavior. Net improvement in cooperation was present in those participants whose target behaviors were anger control (1 case), withdrawn (1 case), lack of social skills (1 case), low self-esteem (3 cases), and depressed mood (1 case). Seven participants' scores indicated a net improvement in cooperation. Although the numbers in this study are small, it
30 appears that cooperation declined in those participants (4 cases) whose target behavior was anger control. Net decline in cooperation also occurred in participants whose target behaviors were oppositional behavior (1), lack of social skills (1), low self-esteem (2), and depressed mood (1). Nine participants' scores indicated a net decline in cooperation. Cooperation stayed the same in participants with target behaviors oppositional behavior (2), withdrawn (1), lack of social skills (1), and low self-esteem (1). ~ Table 6 Net Cooperation Scores Hypothesis #5: Participants' mean level of improvement is higher with certain target behaviors. Table #7 describes the mean level of improvement for each target behavior as rated by program facilitators. The rating scale ranges from 1 (greatly improved) to 10 (significantly worse). Any score above 5.5 indicates a decline in target behavior. The results indicate that the program was most effective in improving three target
behaviors. 31 Participants identified as having low self-esteem showed a mean improvement score of 2.2; the mean improvement score of those whose target behavior was anger control was 2.8; and those participants whose target behavior was identified as oppositional behavior had a mean improvement ' score of 3. Of thirty participants, two (7%) were rated as declining in target behavior and twenty-eight (93%) were rated as improved. The ANOVA score noted under table #5 (.0538) indicates that there is statistical significance between target behavior and participant level of improvement. Table #7 Target Behavior Mean Level of Improvement 1 = greatly improved 10 = significantly worse Anova F 2.69698 Prob.0538 Discussion The results seem to indicate that the ADAPT summer program does not significantly affect participant self-esteem and cooperative behaviors and attitudes. While the program facilitators rated participants as improving in
32 overall target behaviors, the results of this study do not indicate any significant increase in self-esteem or cooperative behaviors and attitudes over the course of the program. When participants were tested through scales used in the ADAPT program, results universally indicated that there was no improvement and that in some instances there was a decline in self-esteem and cooperation. Dividing the results by program location indicated a difference between cities, but still no significant improvement in either program. Although the results as indicated by the scales used to evaluate this program showed no improvement in self-esteem and/or cooperation, the program coordinators rated the majority of participants as improving in target behavior. Coordinators rated more than ninety percent of participants as having improved in target behavior. This observed improvement in target behaviors is significant in that the program is designed as day treatment for mental health needs related to these target behaviors. Improvement of these behaviors is a primary goal of the program. This study indicates, however, that the program is not attaining its goal of developing participant self-esteem and cooperation. While these findings may be related to the validity of the instruments used in the study, they may also indicate that in focusing on improving target behaviors, important concepts such as self-esteem and cooperative skills are overlooked.
33 One limitation of this study relates to the reliabililty and validity of the instruments used. The two questionnaires completed by participants have not been tested to evaluate either reliability or validity. One was developed and selected to evaluate agency specific programs. The instruments may not have been sensitive enough to identify self-esteem and cooperative behaviors and attitudes as dependent variables. Program facilitators completed the target behaviors instrument (Appendix D). The high levels of improvement indicated by Table #5 could indicate differences in perceived improvement between participants and program staff. These results could also be influenced by bias on the part of the observers. Staff indicated that ninety-three percent of participants demonstrated improvement in target behavior. These results could be influenced by the sensitivity of the instruments to self-esteem and cooperation. Although all instruments addressed these variables, it is possible that they were not sensitive enough for the purposes of this study. The Stress Response Scale for example, is designed to look at a variety of behaviors, including self-esteem and cooperation variables. Range Mental Health Staff may choose to utilize a more powerful instrument to isolate self-esteem as a dependent variable, such as the Tennessee Self-Concept Scale or the Piers-Harris Self-Concept Scale. The results may also be influenced by the participants
response patterns. 34 It appeared that on some surveys, participants choose one answer and continued answer the same way for the duration of the survey. In addition, no effort was made by program staff to evaluate the cultural sensitivity of the evaluation instruments used. All program staff were of European American heritage, and although cultural diversity was included in curriculum design, no cultural competence assessment of the program was done. The lack of a control group is a limitation as well. The use of a control group in the study, possibly of elementary age children who have been diagnosed as being in need of mental health services, but not currently participating in the program would have increased the validity of the study. The study did not in any way isolate therapeutic recreation as a independent variable in this study. In future program evaluations, the agency may consider isolating variables which relate specifically to therapeutic recreation. The internal validity of the study may have been affected by experimental mortality as post-test results were not available for eight participants. This loss of subjects is due to a number of issues including family vacations during the last week of the program, lack of pr~gram completion due to participant choice to drop-out, and program staff requesting individuals' withdrawal from program.
35 A final limitation of this study relates to the use of secondary data in research. The nature of the survey instruments utilized by Range Mental Health Center (a mix of interval and ordinal surveys) made it difficult to evaluate statistical significance in this program evaluation. Implications This study offers a number of possible directions for the existing programs in Virginia and Hibbing as well as for proposed programs in other agencies offering summer day treatment programs. In addition, it suggests possible directions for future research. The program evaluation results seem to indicate that while program facilitators rated participants as improving overall in target behaviors, there is no significant improvement in self-esteem or cooperative behaviors. As prqgram faciliators have indicated that these variables are part of the program focus, Range Mental Health staff may choose to incorporate these results in their program development and planning in several ways. Program staff could strengthen the emphasis on activities which are designed to encourage self-esteem development and cooperative skills. Specific activities which stress these variables are well documented in the literature (Johnson et al., 1993; Reasoner, 1983). In addition, Range Mental Health Staff could specifically include development of self-esteem and
36 cooperative skills in individual treatment plans of each program participant. These variables could be indicated as treatment goals at the beginning of the program and change could be evaluated on a regular basis. In performing program evaluations, Range Mental Health staff may consider using different measurement instruments. An instruments such as the Tennessee Self-Concept Scale may be a better indicator of change in self-esteem over the course of the program. In program planning, emphasis could be placed on the cooperative nature of the recreational focus of the program. The "therapeutic recreation'' component of this program has been documented in the literature as promoting significant change in self-esteem and peer interactions with this population (Kaplan, 1974; Davis-Berman & Berman, 1989). Additional research should be completed before any conclusions are made regarding the effectiveness of this modality in the Range Mental Health ADAPT program. Further research could isolate the therapeutic recreation aspects of the program as an independent variable. A study of this type could include a similar day treatment program without a recreational component as a comparison group. Conc1usion This study evaluates the summer component of the Range Mental Health Center children's day treatment program. The
' 37 study identified an improvement in participants' target behavior over the course of the program. The findings do not indicate that participant self-esteem or cooperation improved as a result of the programs. The results provide opportunities for program staff to target self-esteem and cooperative skill development in program planning and participant treatment plans. Instrumentation and several other limitations of this study were cited, as well as possibilities for future research.
References American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders (3rd ed.). (1987). Washington, DC: American Psychiatric Association. Berman, D. s. & Anton, M. T. (1988). A wilderness therapy program as an alternative to adolescent psychiatric hospitalization. Residential Treatment for Children & Youth, 5(3), 41-53. Berman, D. s. & Davis-Berman, J. L. (1989). Wilderness therapy: A therapeutic adventure for adolescents. Journal of Independent Social Work, 3(3), 65-76. Burton, L. M. (1981). A critical analysis and review of the research on outward bound and related programs. Dissertation Abstracts International, 42, 1581. Clagett, A. (1989). Effective therapeutic Wilderness Camp programs for rehabilitating emotionally-disturbed, problem teenagers and delinquents. Journal of Offender Counseling, 14(1), 79-97. Davis-Berman, J. & Berman, D. s. (1989). The Wilderness Therapy Program: An empirical study of its effects with adolescents in an outpatient setting. Journal of Contemporary Psychotherapy, 19, 271-281. Engel, M. & Raine, W. J. (1963). A method for the measurement of the self-concept of children in the third grade. Journal of Genetic Psychology, 102, 125-137. Fazio, s. M. & Fralish, K. B. (1988). A survey of
leisure and recreation programs offered by agencies serving traumatic head injured adults. Therapeutic Recreation Journal, 22, 46-54. Gerstein, J. s. (1992) The adventure based counselor. In Coalition for Education in the Outdoors Research Symposium Proceedings. Indiana, Bradford Hills. Gillis, H. L. & Gass, M. A. (1993). Bringing adventure into marriage and family therapy: An innovative experiential approach. Journal of Marital and Family Therapy, 19, 273-286. Godfrey, R. (1980). Outward Bound: Schools of the possible. Garden City, New York: Anchor Press. Grizenko, N.; Papineau, M.; & Sayegh, L. (1993). Effectiveness of a multimodal day treatment program for children with disruptive behavior problems. Journal of the American Academy of Child & Adolescent Psychiatry, 32,(1), 127-134. Johnson, D. W.; Johnson, R. T.; Holubec, E. J. (1993). Circles of Learning: Cooperation in the Classroom. Edina, MN: Interaction Book Company. Kaufman, J., McBride, L. G., Hultsman, J. T., Black, D. R. (1988). Perceptions of leisure and an eating disorder: An exploratory study of bulemia. Therapeutic Recreation Journal, 22, 55-63. Lecroy, c. w. (1992). Enhancing the delivery of effective mental health services to children. Social Work, ~ 225-231. Marx, J. D. (1988). An outdoor adventure counseling
program for adolescents. Social Work 33,(6), 517-520. Medical Assistance Provider Manual. (1992). Minnesota Department of Human Services. Nadler, R. S. (1992). Processing the Adventure Experience. Dubuqe, Iowa: Kendall/Hunt Publishing Company. Reasoner, R. W. (1983). Enhancement of self-esteem in children and adolescents. Family & Community Health, 6 (2), 51-64. Priest, s. & Ewert, A. (1992). Report for the Individual/Personal Development group. In Coalition for Education in the Outdoors Research Symposium Proceedings. Bradford Woods, Indiana. Publication Manual of the American Psychological Association (4th ed.). (1994). Washington, DC: Amerian Psychological Association. Witman, J. P. (1987). The efficacy of adventure programming in the development of cooperation and trust with adolescents in treatment. Therapeutic Recreation Journal, Third Quarter.
APPEND[X A Name Date Feeling Good about Me: D eveloping Self-esteem Instructions: Each of the statements below concerns your ideas, beliefs, attitudes, or feelings about yourself. After each statement is a response you could choose. Circle the response that is how you think or feel now. Scale 1 - never 2 - hax:_9ly_ev.er 3-so~times 4 - most of the time 5 - always Example: I like to choose my own clothes. You always like to choose your awn clothes. 1 2 3 4 1. I feel good about myself. 2. I understand my feelings. 3. I know what others *ink of me. 4. I am a good student. 5. My teachers seem to like me. 6. My 'parents are pleased with how I am doing in school. 7. Other kids want me to be their friend. 8. I feel as though I fit in at school. 9. I feel as though I am a successful person so far in my life. 10. I believe I have a good future ahead of me. 1 1 1 1 1 1 1 1 I I 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5...
APPENDIX B ~... To be filled out by the child at the beginning of the program and again upon his or her completion of the program. SELF CONCEPT QUESTIONNAIRE Directions: Please make an "X" across the face which shows the way you feel about the question G) A\./FUL I NOT SO GOOD How well you make new friends. 0. K.? GOOD VERY GOOD G) @ @ AWFUL NOT SO GOOD O.K. GOOD VERY GOOD ~--~~~~~~~--~~~----~~-----~~~------ How well you do school work ~~--~~~----~~~----~----~--~----- AWFUL NOT SO GOOD O.K. GOOD VERY GOOD O.K. GOOD VERY GOOD How-you are at sharing things ~ ~~~~~~ -.-, ~~~~~-:...--,~--~~~-~-u- ~~--- AWFUL NOT SO GOOD o. K. GOOD VERY GOOD How you are at following rules in games (g) _ _ _!.:- --- A\JFUL NOT SO GOOD 0. K. GOOD VERY GOOD How you are at keeping your temper @ AWFUL NOT SO GOOD O.K. GOOD VERY GOOD How you like yourself ----------------~------------------~----~-- @ @. A\./FUL tlot SO GOOD O.K. GOOD VERY GOOD --------------~------~~--------~------~----~-------- How you are at joining in and participating in a group @ @ A\./FUL NOT SO GOOD O.K. GOOD VERY GOOD --------~----~------------
APPENDIX C SRS I I I Child's Name Q Boy Q Girl Age Grade School/Clinic For office use only ID# Directions: This scale consists of 40 items. For each item, mark a (..J) in the column which best describes the behavior of the child being rated. Please answer all items carefully. The scale usually takes about 10 minutes to complete. 0 Never 1 Almost Never 2 Sometimes 3 Often 4 Almost Always. 5 Always FOR OFFICE USE 01. Worries.... 02. Daydreams..... 03. Easily excited.... 1 2 3 04. Easily distracted.... 05. Demanding.... 06. Helpless..... 4 5 6 07. Underachiever.... 08. Quiet, withdrawn 7 8 09. Selfish..... 9 10. Passive.... 10. 11. Temper outbursts.... 12. Immature speech..... 13. Procrastinates, puts things off... 1-1. Restless, overactive... 15. Poor attitude toward school... 16. Giddy, silly behavior... 11 12 13 14 15 16 COMPLETE REVERSE SIDE PAR Psychological Assessment Resources, Inc. P.O. Box 998/0dessa, FL 33556 Copyright 1982, 1993 by Psychological Assessment Resources, Inc. All rights reserved. May not be reproduced in whole or in part in any form or by any means without written permission of Psychological Assessment Resources, Inc. This form is printed in green ink on white paper. Any other-version is unauthorized. 987654321 Reorder #R0-2337 TOLL-FREE 1-800-331-TEST Printed in the U.S.A... ~---
17. Defiant... 18. Sensitive, easily hurt... 19. Playful... 0 Never 1 Almost Never 2 Some times 3 Often 4 5 Almost Always Always FOR OFFICE USE 17 --- 18 --- 19 _ 20. Pays attention..... (l:t-j \ a. V II 0...,.fffapai~... : E' 20 ---.,,... 22. Talkative.... 22 23. 24..,- L.:). 26. stzjj:. 28. 29. ""l ~- 31. 32. 33. 34. Cares about schoolwork... Declining school grades... Picks on other children... Impulsive... Self.Corifidenf ~..-... ~... Willful... Figl1ts... Shy... Completes assignments... Nervous, jumpy... Easily upset... Detached, out of touch... Afraid of new situations... 23 24 25 --- 26 --- 27:... 28 29 30 --- 31 --- 32 --- 33 34 35... 36. 37. Independent.... Miscllievous.... 38. Able to take criticism.... C"-/- -~.J9. Cooperative.... '.../ '"5'"-1 40. Stubborn.... 36 37 --- ~ 39 - ~
APPENDIX D Participant number: Primary target behavior/problem area: Please circle the number which best indicates the degree of change in participant target behavior over course of summer session: greatly improved significantly worse 1 2 3 4 5 6 7 8 9 10