Determining Program Success: The Importance of Employing Experimental Research Designs

Transcription

1 Determining Program Success: The Importance of Employing Experimental Research Designs Franklyn W. Dunford Data from the San Diego Nay) Experiment are used to illustrafe howfindingsjrom nonexperimental evaluafions of programs designed to treat men who abuse their cohabitant partners may lead to erroneous conclusions. After reviewing 1 batterer treatment progams, Edleson and Tolman (1 992) observed, Taken as a whole, the studies that have been done to date indicate that the majority of men stop their physically abusive behavior for some period of time subsequent to intervention. Percentages of successful outcome ranged from 53% to 85%. This surprising convergence of successful reports on different programs, with different structures and different evaluation methods, holds promise for group interventions for men who batter. (p. 86, italics added) A variety of questions are raised by statements such as these. What, for example, constitutes a successful intervention formen who batter? What is the litmus test for successful interventions? Is the complete elimination of all forms of continued abuse in all men receiving treatment required for an intervention to be classfied as successful? If not, what proportion of those receiving treatment must not recidivate for an intervention to be classified as successful 95%, 85%, 75%. or 5%? Following reviews of the literature and asking these questions of a large number of therapists and others who worlc with batterers and in batterer treatment progams, I found no consensus concerning these issues. There is no rule ofthumb or generally accepted standard for success. FRANKLYN W. DUNFORD: Senior Rcscarch Associate. Institutc of Behavioral Sciencc. Un~vcrsity of Colorado at Boulder. The pant suppomng this article was awardcd and administered by the Nat~onal Institutc of Mental Ilcalth (NIMH) (Grant No. 2 ROlMH458). All funds were provided by thc U.S. Navy through an interagency agreement with the NTMN. Thc contents of [his articlc rcflect the views of thc author and [lot necessarily the position of cithcr the U.S. Navy or the CRIME d DELMQUENCY Vol. 46 No. 3. July Sage Publicnt~ons. Inc.

2 426 CRIME & DELINQUENCY I JULY ZOO In the absence of a widely held standard to determine program success, how are interventions for batterers to be judged? A look at findings from evaluations ofbatterertreatment programs conducted in the recentpast illustrates the problems in evaluating program effectiveness. Richard Tolman and Larry Bennett (199) reported the results of their review of 22 evaluations of spouse abuse treatment programs, 77% of which involved cognitive behavioral treatment approaches. They defined program success as the percent of participants who "stopped the physical abuse" of cohabitants. The mean followup period for the risk for new abuse for the 22 programs they reviewed was approximately 9 months, and the mode was approximately 6 months. The range of the followup risk periods for recidivism across the program evaluations was 7 weeks to 3 years. The prevalence of recidivism ranged from 15% to 41% for male perpetrator reports of the continued abuse of cohabitant partners and from 13% to 47% as determined by female victim reports. The mean prevalence rate formale perpetratorrecidivism for the physical abuse of female victims was 35% as reported by male perpetrators and 3% as reported by female victims. The mean prevalence of arrest recidivism for male perpetrators was 13% (ranging from 4% to 19%). It is important to note that, of the 22 programs cited, only 5 were found to have prevalence rates of the continued abuse of women less than 2% (1 5%, 16%, 13%, 15%, and 16%). Sample sizes were, for the most part, small (ranging from 9 to 244), averaging 8 1 cases. None of the studies employed a randomized experimental grouppcontrol group research design. Barry Rosenfeld (1992) reviewed 25 evaluations of interventions for men who were court ordered to treatment for abusing their cohabitant partners. All but 9 of those programs were reviewed by Tolman and Bennett (1 99). Seven of the 9 that were not reviewed by Tolman and Bennett had samples of fewer than 5 cases, and none of the program evaluations employed an experimental groupcontrol group design. The aftertreatment followup periods for the 9 studies ranged from 6 months to 3 years, with an average of about 12 months. The mean prevalence of continued male perpetrator violence against women for these studies was 18.5% as reported by male perpetrators and female victims. Male perpetrator reports ranged from 1 1 % to 19%, and female victim reports ranged from 25% to 27%. Kevin Hamberger and James Hastings (1993) reviewed 73 studies and presentations that evaluated treatment outcome for domestic violence from 1984 to 199. Fifteen of the evaluations overlapped those reported in the Tolman and Bennett (199) and Rosenfeld (1992) reviews. Of the 8 remaining program evaluations, 3 failed to provide specific outcome data. The prevalence of male perpetrator violence against women for the rest of the studies as

3 Dunford / DETERMINING PROGRAM SUCCESS 427 reported by male perpetrators ranged from % to 3%. Followup risk periods ranged from (outcome assessed at the end of treatment) to 1 year. None of these studies employed an experimental groupcontrol group design. Summaries of evaluations of male offender treatment programs reviewed by Saunders and Azar (1989) and Eisil<ovits and Edleson (1989) involved most of the studies already cited. In instances where they differed. the findings did not depart significantly from those that have been noted. Because of the differences in sample sizes, riskperiods, definitions of violence, outcome sources, and so forth, of the nonexperimental evaluations of the treatment programs reviewed by all ofthese authors, it is difficult to accurately summarize the findings. However, when all of the imperfections and methodological problems of the evaluations reviewed are overlooked, two interesting observations emerge. First, the only programs that reported having 1% success in limiting continued violence in the men treated were programs in which outcome was measured at the end of treatment (i.e., without posttreatment followup periods). All of the evaluations involving posttreatment followup periods reported continued abuse, suggesting that findings based on endoftreatment assessments for recidivism should be viewed with caution. Second, the outcome findings of the studies that were reviewed indicate that the most successful programs were those in which approximately 15% to 25% of perpetratorslvictims reported new instances of violence against female partners. That is, success rates (defined as the prevalence of no new violence) of 75% to 85% are among the best found in the literature. These data suggest that a reasonable rule of thumb or standard for assessing the success of batterer treatment programs might be set at 75%. With such a standard in place, one might assume that any program in which at least 75% of the perpetrator clients did not, for some reasonable time period after treatment, engage in new instances of violence against female partners would be defined as successful. It should be clear, however, that summaries of the kind described here do not provide the kind of information necessary to select a criterion for defining program success. The question is not whether a program successfully treats a given percentage of participants over some specified time period. Rather, it is whether persons in treatment do better than those receiving no treatment over identical followup periods. Also, it is whether a particular program does more good than harm and whether those who participate (or who are required to participate) in batterer treatment programs are better off than those who do not participate. The answers to these kinds of questions are best found through the use of randomized experimental research designs.

4 428 CRIME & DELINQUENCY 1 JULY 2 ASSESSING PROGRAM EFFECTIVENESS WITH AN EXPERIMENTAL DESIGN The assertion that experimental research designs are best suited to assess pro,gam effectiveness is supported by fmdings from the San DiegoNavy Experiment (Dunford, in press), as shown in the following sections. Backgroundfor the San Diego Navy Experiment The San Diego Navy Experiment involved 861 married Navy couples in which husbands were reported (and reports subsequently substantiated) as havingphysically assaulted their wives. Each case was randomly assigned to one offour goups: a men's group (MG), a conjoint group (CG), a rigorously monitored group (RM), and a noservices control group (Cont.). The interventions for the men's and conjoint goups were based on a cognitive behavioral model in which clients met in groups conducted weekly for 6 months followed by 6 monthly meetings (U.S. Navy 1993a. 1993b). Approximately 12 men were assigned to each of the men's groups and seven couples to each of the conjoint groups. Sample sizes were as follows: MG = 218, CG = 216, RM = 213, and Cont. = 214. All group sessions were led by two cotherapists (a man and a woman) and were audiotaped and reviewed monthly for program adherence and therapist competency. The interventions for the rigorous monitoring group involved monthly meetings with case managers in which client progress was monitored and counseling services delivered. The point of the rigorous monitoring intervention was to make visible to those in authority over the perpetrator clients all new incidents of abuse. Consequently, the results of regular police record searches and monthly interviews with victims conducted to identify new instances of abuse were sent to client commanders each month. informing them of client progress. Copies of these reports were also sent to the perpetrators to ensure that they were aware that those in authority over them were notified monthly of new abuses. The men in the control group were assigned to receive no treatment services. Victim wives, regardless of the experimental group to which their husbands were assigned, typically received a onetime ~ntervention from the Family Advocacy Center (FAC), the Navy agency responsible for handling cases of domestic violence among Navy couples. This intervention was called Stabilization and Safety Planning. FAC case managers contacted victims as soon as possible after the presenting incident to ensure that victims were not in immediate danger of new abuse and then provided them with safety planning information.

5 Dunford / DETERMINING PROGRAM SUCCESS 429 Both victim wives and perpetrator husbands were interviewed, separately and usually in their homes, by the nonnavy research staff four times over the duration of the experiment: prior to the start of treatment, at the end of the weekly treatment period (approximately 6 months after the start of treatment), 6 months after the end of treatment, and 12 months after the end of treatment. A portion of each of the followup interviews focused on new instances of abuse. Continued abuse was measured using a number of assessment tools, two of which are discussed here: measures of episodic abuse obtained from victims and official arrest records. With regard to the former, respondents were asked to report instances in which they felt in danger of being physically hurt, episodes in which they were pushed, hit, had hands laid on them, or were beaten up, and episodes in which they were physically injured. Official arrest recidivism was based on the arrests of male perpetrators for any offense in which their female cohabitant partners were the victims. Findings of the San Diego Navy Experiment To illustrate the need for randomized experimental groupcontrol group research designs, the results of the San Diego Navy Experiment are examined with and without the use of a randomized control group. The conclusions reached without the use of a control group are much different than are those reached when data from the control group are included in the analyses. Consider, for example, female victim responses to questions about episodic injury. Table 1 summarizes victim reports of new incidences of all three types ofepisodic abuse during the 1 year period following the first 6 months of treatment. Comparisons are limited to the three treatment interventions used in the San Diego Navy Experiment (i.e., data from the control group are not included). The data indicate that there were no statistically significant differences among the three treatment groups on any ofthe measures ofnew abuse.' Of special interest are the victim reports of new episodes of physical injury, because they most closely parallel the violence measures used in the program evaluations cited earlier. The prevalence rates for new instances of physical injury for each of the three interventions of the San Diego experiment (MG. CG, RM) fell within the range (1 5% to 25%) of the best of the nonexperimental evaluations found in the literature, clearly meeting the success standard postulated earlier.' The data in Table 2 tell a similar story. When the three interventions of the San Diego Navy Experiment were compared with official arrest recidivism, no statistically significant differences among the treatments were found, and the prevalence of recidivism reported for the I year followup period (4%) was about as good as any found in the literature.

6 TI 7J. ; s a e m e nż C " 'D P,a m.o 1 2 L.'.. g %. P 9"cq *NbV]Nb mmmb m d Zwwmmm ~ 9 " awmm 7 7 cq N Ummmdm ammm 7 7 ' 3 q T Zi.mmj m m m w * * pwm7b Zcubbrm v Ir! u! g u s. i m (U I= Z R U) $ R r _ LT VL >. s.g ~ b m o m w=+l$ 2 m E; "2 7 a 4 g.a a?? T m m a o w wwr: 7 S " Y p Ou,rd+ y tmgwr T * r umrmam mmmwm 7 U c q? S wwrn.r (Dmoww 7

7 Dunford / DETERMINING PROGRAM SUCCESS 431 TABLE 2: Arrest Recidivism by Treatment Group at IYear FollowUp (control group not included) Number of Cases Number of Arrests MG (%) CG (%) RM (%) Total (%) 162 (96.4) 149 (97.4) 163 (94.2) 474 (96.) 1 6 (3.6) 4 (2.6) 1 (5.8) 2 (4.) Total NOTE: MG = men's group. CG =conjoint group. RM = rigorous monitoring group.x'' (2, N = 494) = 2.244, p =.33. Given these findings, it would not be unreasonable to do as others have done with similar nonexperimental outcome data (or to use the hypothesized rule of thumb cited earlier) and define all three of the interventions of the San Diego Navy Experiment as successful. The interpretation of the data of Tables 1 and 2, using such standards, would suggest that the Navy could continue to use any one of the three interventions evaluated because all three of the interventions had been shown to be equally effective. Furthermore, the Navy could assume that the needs of clients would be met by using any one of the interventions tested. However, when the outcome data obtained from the control group are added to Tables 1 and 2, the conclusions drawn from those tables change. The data in Tables 3 and 4 indicate that the men in the control group recidivated at a rate about equal to that of the men in the three interventions groups. No statistically significant differences were found among the men assigned to the three treatment groups and the control group on the prevalence or frequency with which victims reported being fearful ofbeing injured, being pushed, hit, shoved, or physically injured during the Iyear followup period following the first 6 months of treatment. Norwere there differences among the four groups on the prevalence or frequency of arrests for the continued abuse of wives. These findings dramatically alter the conclusions that would have been reached in the absence of an experimental groupcontro~ group research design. Furthermore, analyses reported elsewhere (Dunford, in press) confirm the results presented here. No statistically significant differences were found among the four experimental goups when compared on a set of Modified Conflict Tactic Scales. Survival curves also did not differ, further supporting the null hypothesis. In other words, the three interventions used in San Diego by the Navy at the time of the evaluation were not meeting the needs of clients nor thenavy, at least with respect to reducing the continued abuse ofwives.

8 TABLE 3: Wives' Reports of Husbands' Abuse by Treatment Group at 1Year FollowUp (control group included) Victim Felt Endangered?a Viclirn Pushed or it? Victim Physically lnjured? Outcome MG CG RM Con! Total MG CG RM Cont Total MG CG RM Cont Total Yes No % Total Frequency of victimizalion 1year ,47, followup SD NOTE: MG = men's group. CG = conjoin1 group. RM = rigorous moniloring group. Con! = control group. 2 a. x (3, N= 622) = 2.725, p=.44. ANOVA F= 1.185, p=.31. b. r2 (3, N= 622) = 2.365, p=.5. ANOVA F = 1.274, p=.28. c. X2 (3, N = 622) = 1.852, p=.6. ANOVA F= 1.55, p=.2.

9

10