INCREASING THE USE OF PRAISE BY DIRECT LINE STAFF CONDUCTING ABA THERAPY: A COMPONENT ANALYSIS OF GOAL SETTING AND FEEDBACK

Transcription

1 INCREASING THE USE OF PRAISE BY DIRECT LINE STAFF CONDUCTING ABA THERAPY: A COMPONENT ANALYSIS OF GOAL SETTING AND FEEDBACK A Thesis Presented to the Faculty of California State University, Stanislaus In Partial Fulfillment of the Requirements for the Degree of Master of Science in Psychology By Amanda Nicole Hurlock November 2016

2 CERTIFICATION OF APPROVAL INCREASING THE USE OF PRAISE BY DIRECT LINE STAFF CONDUCTING ABA THERAPY: A COMPONENT ANALYSIS OF GOAL SETTING AND FEEDBACK by Amanda Nicole Hurlock Signed Certification of Approval page is on file with the University Library Dr. Bruce Hesse Professor of Psychology Date Dr. William Potter Professor of Psychology Date Dr. Jessica Lambert Assistant Professor of Psychology Date

3 2016 Amanda Nicole Hurlock ALL RIGHTS RESERVED

4 DEDICATION This thesis is dedicated to my entire family. I am so grateful to the family members who have always blessed my life and to those who I have been fortunate enough to add to my life over the years. Your unwavering support keeps me motivated and persevering toward my goals. I love you all. iv

5 ACKNOWLEDGEMENTS Thank you to my professors, classmates, and coworkers who helped me complete my thesis and my degree. I am thankful for all of the support and advice provided and for the editing and data collection done on my behalf. I am truly grateful for your dedicated participation in this process. v

6 TABLE OF CONTENTS PAGE Dedication... Acknowledgements... iv v List of Figures... viii Abstract... ix Introduction... 1 Antecedent and Consequence Based Interventions... 1 Manipulation of Establishing Operations Through Goal Setting... 5 Feedback Component... 9 Component Analyses as a Potential Benefit in Research... 9 Direct Line Staff Performance in ABA Treatment Purpose of Present Study Methods Setting and Participants Dependent and Independent Variables Procedures Inter-Observer Agreement Treatment Integrity Results Participant Participant Participant Participant Discussion Use of Multiple Interventions to Improve Staff Performance Feedback Combined with Goal Setting Staff Performance Impact on Client Performance in ABA Treatment Feedback as a Conditioned Reinforcer Goal Setting as a Conditioned Establishing Operation Manipulation vi

7 Goal Setting as a Conditioned Abolishing Operation Manipulation History of Reinforcement Experimental Limitations Environmental Limitations Recommendations for Future Research Conclusion References Appendices A.... Recruitment Flyer B.... Informed Consent C.... Data Collection Sheet D.... Interobserver Agreement Data Sheet E.... Feedback Note F.... Goal Setting Script G.... Treatment Fidelity Data Collection Sheet vii

8 LIST OF FIGURES FIGURE PAGE 1. Rate of Behavior Specific Praise Statements for Participants 1 and Percentage of Behavior Specific Praise Statements for Participant 1 during sample observations across baseline and two intervention conditions Percentage of Behavior Specific Praise Statements for Participant 2 during sample observations across baseline and two intervention conditions Rate of Behavior Specific Praise Statements for Participants 3 and Percentage of Behavior Specific Praise Statements for Participant 3 during sample observations across baseline and two intervention conditions Percentage of Behavior Specific Praise Statements for Participant 4 during sample observations across baseline and two intervention conditions viii

9 ABSTRACT A staff performance intervention was introduced in a non-public behavior analytic agency with four behavior technicians working with children with autism to analyze the effects of goal setting and feedback on use of behavior specific praise statements. Results indicated that for some individuals feedback alone may act as a conditioned reinforcer and may improve staff performance. Results further indicated that goal setting may act as a conditioned establishing operation manipulation for the value of feedback to improve staff performance. Learning history may impact the extent to which feedback acts as a conditioned reinforcer and to which goal setting acts as a conditioned establishing operation manipulation. Considerations for future research creating stronger verification of results and reducing potential aversive qualities of goal setting are discussed. ix

10 INTRODUCTION There is much research in the Organizational Behavior Management (OBM) literature about staff performance. Members of the OBM community evaluate and manipulate staff performance across a variety of sectors including medical services (Stephens & Burroughs, 1978), higher education (Wilk & Redmon, 1998), food services (Downing Jr. & Geller, 2012), and retail (Newby & Robinson, 1983). OBM and applied behavior analysis (ABA) researchers evaluate staff performance in human service agencies (Nepo, 2010). Individual staff performance is of interest because of the impact it has on the organization as a whole. Staff performance influences the organization s ability to meet goals (Selden & Sowa, 2011), decrease theft and fraud (Downing Jr. & Geller, 2012), improve client performance (Howard & DiGennaro Reed, 2014), and increase customer enrollment (Wilk & Redmon, 1998). Researchers in the OBM and ABA literature use manipulation of antecedents, consequences, and establishing operations to change staff behavior and improve work performance. Antecedent and Consequence Based Interventions Antecedent-based and consequence-based interventions are commonly used either in isolation or in combination to improve staff performance. Antecedent-based interventions in the form of training techniques are used to change staff behavior in a variety of work settings. Catania, Almeida, Liu-Constant, and Digennaro Reed (2009) used video modeling with three direct-care staff members newly employed at a 1

11 2 private school for children and young adults with autism. Discrete trial training session accuracy improved by as much as 86% after video modeling was introduced. In another study, supervisors at a non-governmental organization were given task clarification and additional training to improve delivery of feedback to animal trainers. Supervisor performance and animal trainer performance improved as a result of task clarification and training (Durgin, Mahoney, Cox, Weetjens, & Poling, 2014). Another study implemented a train-the-trainer antecedent intervention focusing on role-play of common problem scenarios with nursing staff on medical-surge units. The intervention resulted in increased teamwork, decreased missed care, and improved employee satisfaction with and knowledge of organizational principles (Kalisch, Xie, & Ronis, 2013). Manipulation of consequences to improve staff performance is more common than antecedent interventions in the behavior analysis and organizational behavior management literature. In example, staff absenteeism was reduced using a monetary reward system in a hospital setting (Stephens & Burroughs, 1978) and in a manufacturing/distribution facility (Pedalino & Gamboa, 1974). As well, a lottery system was used to positively reinforce improved work performance by staff of a residential facility (Iwata, Bailey, Brown, Fosher, & Alpern, 1976) and to reinforce special education staff s treatment fidelity of discrete trial training procedures implemented with classroom students (Miller, Carlson, & Sigurdsson, 2014). Miller et al. s (2014) lottery system reinforcement package involved monetary incentive and feedback. The participants were scored for correct use of a prompt

12 3 hierarchy, accuracy of data collection, and completeness of activity prescriptions for the next day s activities. The participants were given visual graphed feedback on their performance during the first day of baseline recording. On subsequent sessions, the participants were given verbal feedback about the prior session before the start of the next session. If a staff member reached 80% or higher accuracy during a minimum of three sessions each week, he or she was entered into a weekly drawing for a $25 gift certificate to a retailer of his/her choice. This intervention resulted in an increase in mean percentage of steps correctly completed by all three participants. In another study, Burroughs and Richardson (1975) provided planned time off as reinforcement for hospital staff s effective performance. A group contingency was introduced to eleven venipuncture technicians in the hospital. Increments of time-off were given to each staff member if the entire shift had less than a set amount of unfilled requests of blood samples. The amount of unfilled blood sample requests reduced significantly following implementation of the group contingency. Across studies, positive reinforcement of staff performance in the form of monetary reward and time-off has been effectively utilized in a variety of workplaces. Researchers often use a combination of antecedent and consequence manipulations to improve staff performance. Howard and DiGennaro Reed (2014) assessed the effects of different training packages on the performance of trainers who provided obedience training to dogs. Written instruction, video modeling, feedback, and in-vivo modeling were sequentially introduced to trainers to teach a 9-step training procedure. Data were collected on correct implementation of the procedure

13 4 after each phase of intervention. The use of feedback and modeling resulted in the highest increase in percentage of correctly implemented steps of the procedure. This suggests that a combination of antecedent and consequence manipulations was most effective in improving trainer behavior. A training package that involved role-playing and feedback was used to teach new behavior technicians how to implement a preference assessment with over 80% accuracy (Roscoe & Fisher, 2008). In another study, Sarokoff and Sturmey (2004) used antecedent and consequent interventions with teachers working one-on-one with a disabled child. They utilized a combination of written instruction, rehearsal, modeling, and verbal and visual feedback. The intervention resulted in improved implementation of discrete trial training. As well, Koegel, Russo, and Rincover (1977) used a training package with teachers using behavior modification techniques with children with autism. During baseline sessions, all eleven teachers in the study demonstrated low percentages of the use of behavior modification procedures. Ten of the eleven teachers were consistently below 58% and one teacher ranged from 63-91% correct usage of the procedures. During intervention, the participants were given a training package that involved written information, videos, feedback, and modeling of procedures. After training, teacher implementation of teaching techniques increased; all eleven teachers scored between 90 and 100% for correct use of the techniques during 25 out of 26 sessions. Student performance also increased after implementation of teacher training. Students demonstrated low rates of responding during baseline and an increase in

14 5 correct responding during intervention. Thus, the use of antecedent and consequencebased interventions can be used to improve both staff and client performance. Manipulation of Establishing Operations Through Goal Setting In addition to antecedent and consequence-based interventions, goal-setting interventions combined with some form of feedback are commonly used to manage organizational behavior (Downing Jr. & Geller, 2012). Goal setting can be conceptualized as an intervention that effectively manipulates motivating operations (MO), specifically establishing operations (EO), of those individuals for which the goals are set. Motivating operations are defined as antecedent events that (a) change the value of the consequence, or, (b) along with the immediate discriminative stimulus, may alter the behavior or, said more parsimoniously, an event that alters the reinforcing or punishing value of a stimulus, and increases or decreases the rates of behavior that produce that consequence (Mayer, Sulzer-Azaroff, & Wallace, 2012, p. 701). Establishing operations are defined as A motivating operation that establishes (increases) the effectiveness of some stimulus, object, or event as a reinforcer (Cooper, Heron, & Heward, 2007, p. 695). Establishing operations are further categorized into unconditioned and conditioned. Unconditioned establishing operations (UEO s) are EO s that are unlearned. For example, humans are born with the capacity for food deprivation to increase the reinforcing value of food. In contrast, conditioned establishing operations (CEO s) are EO s that increase the reinforcing value of a stimulus as a result of the organism s learning history (Cooper et al., 2007).

15 6 Goal setting is most accurately conceptualized as a conditioned establishing operation manipulation. Goals are introduced in a variety of staff performance interventions. In one study, a combination of group goal setting and individual verbal feedback was effectively used with grocery store clerks to increase identification-checking behavior (Downing Jr. & Gellar, 2012). Wilk and Redmon (1998) utilized supervisory created goals combined with either verbal or graphic feedback to increase the number of routine clerical tasks completed by employees in a university admissions department. The highest increase in productivity resulted from a combination of goal setting and graphic feedback. In another study, DiGennaro, Martens, and Kleinmann (2007) manipulated establishing operations in combination with feedback and a negative reinforcement contingency to increase staff performance. Four special education teachers working with students who exhibited disruptive behavior participated in the study. The goal of the intervention was to increase correct implementation of behavior intervention plans by the special education teachers. The experiment consisted of six phases (a) pretraining baseline, (b) training, (c) post-training baseline, (d) goal setting and student performance feedback, (e) teacher performance feedback and directed rehearsal with meeting cancellation, and (f) fading. After a basic staff-training package was provided with baseline levels taken before and after, two staff members were placed in condition (d) goal setting and student performance feedback then (e) teacher performance feedback and directed rehearsal with meeting cancelation, while the

16 7 other two staff members experienced (e) teacher performance feedback and directed rehearsal with meeting cancelation then (d) goal setting and student performance feedback. In condition (d) staff members were given a goal to reduce student problem behavior to 50% below baseline. Staff were given written and visual feedback about student performance daily. During this phase of the intervention, two of the participants increased levels of treatment integrity while the other two participants demonstrated similar levels of treatment integrity in comparison with post-training baseline levels. In phase (e), performance feedback plus directed rehearsal with meeting cancellation was implemented. During this phase, a scheduled meeting meant to provide time for practice of missed steps was cancelled following target performance of the treatment plan. Three of the four participants exhibited improvement in this condition. Finally, once all four participants demonstrated improvement in implementation of treatment, the daily feedback schedule was eventually thinned out to once every two weeks in phase (f). High levels of treatment integrity (91-98%) for all participants were maintained during this thinned schedule. Removal of additional feedback as a result of the meeting cancellation contingency and thinning of the feedback schedule resulted in the highest levels of treatment integrity. This suggests that a rich schedule of feedback may not be necessary to maintain staff behavior changes over time. The study also suggests that goal setting may not be necessary in combination with feedback in order to increase performance. The many conditions of

17 8 this design make it difficult to sort out the parts that most effectively impacted treatment integrity levels. In a similar study, Martens, Hiralall, and Bradley (1997) demonstrated that goal setting combined with minimal feedback, in the form of a written note, can effectively improve teacher performance. Martens et al. (1997) recorded three categories of appropriate student behavior: (a) schoolwork, (b) attending to instruction, and (c) responding aloud; and one category of teacher behavior: (d) teacher praise. During intervention, the classroom teacher was first informed of how often she praised the specified student during baseline data collection. The teacher was then asked to create a goal related to the amount of praise statements she wanted to use during a thirty-minute observation period. The teacher was observed for thirtyminutes and data were recorded on the student and teacher behavioral targets. Prior to the next observation session, the teacher received a feedback note stating whether or not she met her goal during the previous session. The goal-setting plus written feedback intervention resulted in an increase in the teacher s use of praise statements and an increase in appropriate behavior for both students in the study. It is unclear whether the goal setting was necessary in combination with the feedback to improve participant performance. More research is needed to determine whether goal setting effectively manipulates staff member EOs and increases the reinforcing value of performance feedback.

18 9 Feedback Component Performance feedback is a common component of many staff interventions and is commonly used in conjunction with an additional component. When feedback is utilized as one of many interventions in a multi-element training package, its need for producing behavior change is unclear. Miller et al. s (2014) staff performance improvement package used verbal feedback in combination with positive reinforcement in the form of monetary gains to increase staff performance, without separating the two consequences. Howard and Digennaro Reed (2014) found the highest level of treatment integrity during the intervention phase where feedback and modeling were provided. However, it is unclear which component of this phase was most effective in changing staff behavior. It is also unclear whether repeated exposure to the training steps from previous phases affected the results. Sarokoff and Sturmey s (2004) training package involved a combination of instructions, feedback, rehearsal, and modeling that produced an increase in treatment integrity. As noted by the authors, we do not know which aspect of the training package was most effective in improving staff delivery of discrete trial training. Downing Jr. and Gellar (2012), Wilk and Redmon (1998), and Martens et al. (1997) utilized a combination of feedback and goal setting without examining the effects of either component in isolation. Component Analyses as a Potential Benefit in Research Many researchers use a combination of antecedent, consequence, and EO manipulation interventions with multiple components to change staff behavior. The

19 10 use of more than one technique leads to unanswered questions about critical controlling variables. For example, which parts of the training packages are the crucial components for improving staff performance? There are few studies that have investigated the steps of typical staff training packages to understand which pieces of the package most significantly impact staff behavior change. Since using multiple elements in successful staff-training packages is so common, a component analysis of the different elements may prove useful in future training. This may help to determine which intervention elements are critical to change staff behavior and which elements are unnecessary. This information could prove useful to many businesses, as efficiently improving staff performance is a common objective among a variety of employers. Direct Line Staff Performance in ABA Treatment Staff performance is a concern in virtually all industries. Thus, staff performance research is conducted across different organizations and varying areas of work. Staff performance in human service agencies not only affects the agency itself, but the performance of the consumers receiving the service. In human service agencies that conduct ABA therapy, staff training is of particular interest. In many of these agencies, 50% of the employees are direct-line staff. These staff work directly with clients and generally control the contingencies involved in ABA therapy. Direct line staff have the most direct contact with consumers yet the least amount of education and experience.

20 11 There are several reported issues with direct line staff s implementation of treatment procedures. These issues include spending little time interacting with residents (Harmatz, 1973), spending 20-40% of work time engaging in leisure activities (Bensberg & Barnett, 1966), and engaging in interactions that are conducive to maintaining inappropriate behavior (Warren & Mondy, 1971). There is potential benefit in research to increase the integrity of behavior change interventions. Failure of some ABA interventions may not be due to the design of the intervention, but to the way staff implemented it. Evaluating and attempting to change the behavior of direct line staff can, in turn, improve treatment integrity. The ability of direct-line staff to correctly implement procedures impacts client improvement (Iwata, Bailey, Brown, Fosher, & Alpern, 1976). Koegel, Russo, and Rincover (1977) demonstrated that by training special education teachers to implement behavior modification techniques, the performance of the children receiving their instruction improved markedly. Specifically, the teachers working with children with autism increased their use of behavior-modification procedures by an average of 32% after training. All children involved in the study demonstrated an increased percentage of correct responses from baseline levels. Similarly, Howard and DiGennaro Reed s (2014) intervention with dog trainers resulted in an increase in dog obedience. DiGennaro et al. (2007) also found a correlation between high treatment integrity and treatment effectiveness. As a result of their intervention, teacher participants increased levels of treatment integrity and decreased targeted maladaptive behavior of all students involved in the study. Likewise, in Martens et

21 12 al. s (1997) study, increased praise statements used by the teacher was followed by an increase in appropriate behavior displayed by the students being praised. There is a relationship between increases in accurate treatment implementation and improved performance of individuals receiving treatment. These results provide a compelling reason to continue the search for strategies that promote accurate treatment implementation by direct line staff. Purpose of Present Study Improving staff performance is a popular topic in the OBM literature. Treatment integrity is a common research subject in applied behavior analysis literature. Staff training packages that utilize antecedent, consequence, or EO manipulation, or more commonly some combination of the three, are used to improve staff performance and increase treatment integrity in human service organizations. Few studies have attempted to improve staff behavior in human service agencies by analyzing the individual components of an intervention package. When multiple elements are used in an intervention package, there could be interaction effects from two or more components that outweigh any single component effect. It may prove useful to study one component of a treatment package at a time in order to determine which component plays the largest role in creating staff behavior change, particularly in the ABA treatment industry. Martens et al. (1997) used both goal setting and feedback to effectively improve teacher performance and in turn increase desired student behavior. The authors did not determine which of the intervention components was most influential

22 13 in changing behavior. The current study replicated some components of Martens et al. s (1997) study. Martens et al. s (1997) study utilized a combination of goal setting and feedback to increase teacher use of praise statements. The current study also attempted to increase the use of praise statements, but in a way that the component effects were assessed. The intervention was introduced in a non-public agency with behavior technicians who implement discrete trial training during one-to-one instruction with children with autism. Similar to Martens et al. s (1997) study, goal setting and feedback were used. In the current study, two conditions were used to isolate the components of the intervention. A feedback alone condition and a goal setting plus feedback condition were used to determine whether the use of goal setting as an EO manipulation for the reinforcing value of feedback has a larger influence on staff behavior change than feedback alone. In the current study two main questions were asked in relation to increasing staff praise statements during discrete trial training sessions with children with autism. First, in comparison to baseline data, does the introduction of performance feedback alone increase the rate of praise statements? Second, does the use of goal setting in addition to performance feedback increase the amount of praise statements more than the feedback only condition?

23 METHODS Setting and Participants The present study was conducted at a non-public agency that provides intensive applied behavior analytic (ABA) therapy to children diagnosed with autism spectrum disorder (ASD). Behavior technicians at the non-public agency work directly with clients. Behavior technicians are a minimum of 18-years-old, have at least a high school diploma, and receive 35 to 40 hours of initial training. Clients of the non-public behavior analytic agency are children diagnosed with ASD and range in ages from one and a half to twenty-one years old. Clients receive one-hour to fourhour ABA therapy sessions for one to seven days per week. ABA therapy sessions are conducted at the center, in the client s home, or in community settings. Four behavior technicians at the non-public behavior analytic agency participated in the present study. Participants were recruited via recruitment flyer posted on a staff message board at the agency (Appendix A). To participate, the behavior technicians must have passed the beginning minimum competency evaluation (BMC) and the Teaching Procedures Skill Performance Early Evaluation Document (SPEED) at least one month prior to the start of data collection. Staff members who participated in the study worked with clients receiving a minimum of three hours of ABA therapy per day a minimum of three days per week at the nonpublic agency. All participants in the study worked with clients receiving 2 therapy sessions per day, hours per session, for 5 days per week. Prior to inclusion in 14

24 15 the study, participants received a consent form that outlined the risks and benefits expected from participation in the study (Appendix B). Participant 1 was 27 years old and began working at the agency 7 months prior to the start of the study. She passed her BMC and Teaching Procedures SPEED 6 months prior to the start of the study. She worked with a 5-year-old female diagnosed with ASD. Participant 2 was 26 years old and began working at the agency 2 years and 8 months prior to the start of the study. She passed her BMC and Teaching Procedures SPEED 2 years and 7 months prior to the start of the study. She worked with the same client as participant 1; a 5-year-old female diagnosed with ASD. Participant 3 was 22 years old and began working at the agency 1 year and 10 months prior to the start of the study. She passed her BMC and Teaching Procedures SPEED 1 year and 7 months prior to the start of the study. She worked with a 5-yearold male diagnosed with ASD. Participant 4 was 19 years old and began working at the agency 2.5 months prior to the start of the study. She passed her BMC and Teaching Procedures SPEED 2 months prior to the start of the study. She worked with a 4-year-old male diagnosed with ASD. Dependent and Independent Variables The dependent variable was the rate of behavior-specific praise statements made toward a client during ABA therapy. During training at the agency, behavior technicians are taught to use behavior specific praise statements as a form of

25 16 reinforcement for clients appropriate behavior. Behavior specific praise statements were counted during three 5-minute time samples during the ABA therapy session. Behavior specific praise statements were defined as statements that follow a two-step formula involving praise and an observable behavior. The behavior specific praise statement was a word or phrase with a positive connotation that is generally defined as praise (e.g., good, great, awesome, fantastic job) paired with the name of the appropriate behavior demonstrated by the child (e.g., walking, talking, looking, following directions, hands down). The statement had to be made in a tone that ranged from neutral to excited (i.e., sarcasm or a demeaning tone was not considered praise). Tacting of observable behaviors in isolation or praise statements used to describe unobservable behavior were not included. For example, good hands down was considered a behavior specific praise statement, whereas hands down or good spoken separately or good trying did not qualify. Independent variables included feedback and goal setting with feedback. Independent variables were presented in two conditions. One condition consisted of delivery of feedback after the therapy session, and the other included delivery of a pre-set goal for number of praise statements prior to the start of the therapy session accompanied by feedback after the session. Feedback was provided in a written note. The note was handed to the participant after completion of the therapy session. The note included feedback on the average rate of behavior specific praise statements used per 5-minute period; for example, You used an average of 5 behavior specific praise statements every 5 minutes when working with Johnny B. on 5/5/16. The researcher

26 17 verbally provided the goal to the participant prior to the start of the ABA therapy session using succinct language. For example, Hi Sally, your goal is to use an average of 10 behavior specific praise statements every 5 minutes when you work with Johnny B. today. Procedures The study employed a single-subject design in the form of a multiple baseline across participants with multiple interventions counter-balanced across pairs of participants. The researcher observed the participants while working with the clients in the clients workstations during the ABA therapy sessions. Therapy sessions consist of discrete trial training at the table in the clients designated stations, known as work time; play activities in various play areas outside of the clients stations, known as play time; bathroom breaks, and specialized activities (e.g., Fokes training, group instruction) conducted in work areas outside of the clients stations. Therapy sessions were 2.5 to 3.5 hours long. Observations were conducted during work time when participants provided discrete trial training to the clients at the table in the clients stations. The tasks completed during discrete trial training were individualized for each client. The nature of the tasks completed for a single client was roughly equal across time of data collection. Data were collected during 2 to 3 therapy sessions per week. During observation periods, the researcher recorded the frequency of behavior specific praise statements made to the client (Appendix C). Observation periods were five minutes long and occurred three times during the hour ABA therapy session, for a

27 18 total of 15 minutes of data collection per therapy session. The five-minute observation periods were divided into 10, 30-second segments on the data sheet. During observation periods, the researcher video recorded the participant and the client and kept observation time on an ipad. Each observation period within a therapy session had a minimum of thirty minutes between the end of the prior observation period and the beginning of the next observation period. Data were collected during the second work time after the end of the thirty-minute waiting period. Data were collected on behavior specific praise statements, which were conditional upon the participants clients engaging in behavior that the participants had previously been trained to define as praise-worthy (i.e., work readiness behaviors). If the client engaged in aberrant behavior during the observation period (i.e., crying, aggression, refusal to respond) for any duration longer than half of the observation period (more than 2.5 minutes), the observation and data collection ceased. The incomplete data from the cancelled observation was discarded, and a new data collection period occurred during the subsequent work time. Data collection was stopped due to client engagement in maladaptive behavior one time in the feedback only condition during the 5 th session for participant 1, and one time in the feedback plus goal setting condition during the 13 th session for participant 2. For participant 4, data collection was stopped due to client engagement in maladaptive behavior once during sessions 1 and 2 and twice during session 3 of baseline, and once each during sessions 14 and 18 in the feedback only condition.

28 19 During the initial baseline phase praise statement data were collected. The researcher did not talk to participants during this phase. Intervention began once baseline data were stable for a minimum of three sessions. In the feedback only condition, after completion of the hour ABA therapy session, the participant received a feedback note stating the average rate of behavior specific praise statements used per five minutes, based on the researcher s observations. Total amount of praise statements recorded across observations was divided by total amount of five-minute observation periods to obtain an average rate of behavior specific praise statements per five minutes. The goal setting plus feedback condition consisted of a meeting between the researcher and the participant prior to the start of the ABA therapy session. The researcher verbally provided a goal for rate of behavior specific praise statements per five minutes that was determined based on the participant s performance during the previous ABA therapy session. The goal was provided one time prior to the hour therapy session, and was not changed or re-stated between the three 5-minute observations that occurred during the course of the therapy session. Goal selection was calculated, prior to the start of the therapy session, in one of three ways. If goal setting was implemented immediately following baseline or immediately following the feedback only condition, then the goal given to the participant was one higher than the average rate of behavior specific praise statements made per five minutes from the observations of the previous therapy session. During subsequent therapy sessions and after the initial implementation of the goal setting procedure, if the

29 20 previously provided goal was met, then the new goal was one higher than the average rate per five minutes from the observations of the previous therapy session. If the previously provided goal was not met, then the goal remained the same for the subsequent therapy session. At the end of the ABA therapy session, identical to the feedback only condition, the participant received a feedback note stating their average rate of behavior specific praise statements per five minutes during the session. Interobserver Agreement The dependent measure of the study was rate of behavior specific praise statements used by the participant. Behavior specific praise statements were defined as statements that followed a two-step formula involving praise and an observable behavior. A second observer reviewed video recordings and collected data on behavior specific praise for 50% of five-minute observations. Five-minute observation periods were divided into 10, 30-second segments. Interobserver agreement was scored using an exact-count-per-interval calculation. The 30-second segment was scored as an agreement if the researcher and observer recorded the same number of praise statements. The 30-second segment was scored as a disagreement if the researcher and observer recorded a different number of praise statements. Interobserver agreement was calculated by dividing the total number of researcher/observer agreements by the number of agreements plus disagreements and then multiplying by 100 to yield a percentage (Appendix D). Using exact-count-perinterval calculation, interobserver agreement for rate of behavior specific praise statements was 77% for Participant 1, 69% for Participant 2, 73% for Participant 3,

30 21 and 88% for Participant 4. When using mean-count-per-interval calculation, a less conservative form of interobserver agreement calculation, agreement for rate of behavior specific praise statements was 84% for Participant 1, 88% for Participant 2, 87% for Participant 3, and 92% for Participant 4. Treatment Integrity Prior to the beginning of the study, a feedback note template (Appendix E) and a goal script (Appendix F) were created. The researcher made copies of each feedback note prior to delivering the note to participants. The researcher video recorded all presentations of goals. All feedback notes and video recordings of goal presentations were viewed and evaluated for treatment integrity. An independent observer scored each feedback note as correct or incorrect according to the template and the amount of behavior specific praise statements recorded by the researcher compared to the amount written on the note. The amount of correctly written feedback notes was divided by the total number of feedback notes and then multiplied by 100 to yield a percentage. The experimenter correctly completed feedback notes for each session with 100% treatment integrity across participants. An independent observer scored each recording of goal presentation as correct or incorrect according to the goal presentation script and the previous session s rate of praise statements. The amount of correctly presented goals was divided by the total number of goal presentations and then multiplied by 100 to yield a percentage. The

31 experimenter correctly presented goals for each session with 100% treatment integrity across participants (Appendix G). 22

32 RESULTS Participants were randomly assigned to condition order. Participants 1 and 2 were first provided feedback only and then goal setting in addition to feedback. Participants 3 and 4 initially received goal setting in addition to feedback, with goal setting removed during the second phase of intervention. Participants 2 and 4 began the first session of baseline during participants 1 and 3 s 3 rd baseline sessions. Data points that are aligned on the graph are from observations during therapy sessions that occurred within 3 days of one another. Behavior specific praise statements are reported in rate per 15 minutes to most accurately reflect participant performance during total observation time. An additional analysis was completed on sample observations to compare amount of praise statements used by participants with amount of praise-worthy behaviors displayed by the client with whom the participant worked. Data were collected on four praise worthy behaviors displayed by the client: eye contact, sitting, correct responding to a discriminative stimulus (SD) presented by the participant, and spontaneous language. Data were collected on two randomly selected observation samples from each condition of the study for each participant. Amount of behavior specific praise statements used by the participant was divided by amount of praiseworthy behaviors demonstrated by the client in order to obtain a percentage of praise statements used per opportunity during the sample observations. Study results as well as results from the additional analysis are reported below. 23

33 24 Participant 1 During baseline, Participant 1 ranged 7-9 behavior specific praise statements per 15 minutes. In the feedback only condition, Participant 1 s average rate of behavior specific praise ranged from 5-31 per 15 minutes. The data trend was gradually increasing. In the goal setting plus feedback condition, praise statements ranged from statements per 15 minutes. The data trend was rapidly increasing. There were clear level changes between the intervention conditions (See Figure 1). Results of the additional analysis indicate that Participant 1 used behavior specific praise statements for 8% of 51 opportunities and 18% of 22 opportunities during sessions 1 and 3 of baseline. Participant 1 s praise statements increased to 32% of 28 opportunities during session 6 and were at 31% of 42 opportunities during session 10 of the feedback only condition. These increased to 62% of 52 opportunities during session 13 and 67% of 54 opportunities during session 16 of the feedback plus goal setting condition (See Figure 2). Participant 2 During baseline, Participant 2 ranged 9-12 behavior specific praise statements per 15 minutes. In the feedback only condition, Participant 2 s rate of behavior specific praise statements ranged The data trend was rapidly increasing. In the goal setting plus feedback condition, praise statements ranged per 15 minutes. The data trend was gradually increasing with some variability. There were clear level changes between the intervention conditions (See Figure 1).

34 25 Results of the additional analysis show that Participant 2 used behavior specific praise statements for 9% of 23 and 22 opportunities during sessions 2 and 4 of baseline. Participant 2 s praise statements increased to 21% of 39 opportunities during session 8 and increased to 55% of 31 opportunities during session 11 of the feedback only condition. These increased slightly to 56% of 41 opportunities during session 14 and increased to 84% of 38 opportunities during session 18 of the feedback plus goal setting condition (See Figure 3).

35 26 Rate of Behavior Specific Praise Statements in 15 minutes Figure 1. Rate of Behavior Specific Praise for Participant 1 and Participant 2 across baseline and two intervention conditions.

36 27 Figure 2. Percentage of Behavior Specific Praise Statements for Participant 1 during sample observations across baseline and two intervention conditions. Figure 3. Percentage of Behavior Specific Praise Statements for Participant 2 during sample observations across baseline and two intervention conditions.

37 28 Participant 3 During baseline, Participant 3 ranged 9-17 behavior specific praise statements per 15 minutes. During the goal setting plus feedback only condition, rate of behavior specific praise statements ranged from 34 to 66 per 15 minutes. The data trend was variable and increasing. In the feedback only condition, data trended upward rapidly, with a subsequent gradual downward trend to close to the previous phase. Praise statements per 15 minutes ranged from (See Figure 4). Results of the additional analysis show that Participant 3 used behavior specific praise statements for 11% of 57 opportunities and 4% of 45 opportunities during sessions 2 and 3 of baseline. Participant 3 s praise statements increased to 35% of 48 opportunities during session 5 and increased again to 41% of 66 opportunities during session 9 of the feedback plus goal setting condition. These increased to 56% of 61 opportunities during session 12 and were 37% of 59 opportunities during session 17 of the feedback only condition (See Figure 5). Participant 4 During baseline, Participant 4 ranged behavior specific praise statements per 15 minutes. During the goal setting plus feedback only condition, the data pattern was variable with a gradually increasing upward trend, and a range of behavior specific praise statements per 15 minutes. In the feedback only condition, data gradually increased and was slightly variable, ranging praise statements per 15 minutes (See Figure 4).

38 29 Results of the additional analysis indicate that Participant 4 used behavior specific praise statements for 21% of 39 opportunities and 12% of 51 opportunities during sessions 1 and 3 of baseline. Participant 4 used praise statements 25% of 40 opportunities during session 5 and 15% of 41 opportunities during session 10 in the feedback plus goal setting condition. Participant 4 used behavior specific praise 19% of 67 opportunities during session 14 and 20% of 64 opportunities during session 16 of the feedback only condition (See Figure 6).

39 30 Rate of Behavior Specific Praise Statements in 15 minutes Figure 4. Rate of Behavior Specific Praise for Participant 3 and Participant 4 across baseline and two intervention conditions.

40 31 Figure 5. Percentage of Behavior Specific Praise Statements for Participant 3 during sample observations across baseline and two intervention conditions. Figure 6. Percentage of Behavior Specific Praise Statements for Participant 4 during sample observations across baseline and two intervention conditions.

41 DISCUSSION The present study sought to examine the effects of feedback and goal setting on the behavior of staff members conducting ABA therapy with children with ASD. More specifically, the study examined whether the implementation of feedback alone or the implementation of feedback in addition to a set goal would increase the amount of behavior specific praise statements used by behavior technicians during an ABA therapy session. Results from the current study suggested that performance feedback acted as a reinforcer for the use of praise statements and that goal setting acted as an effective establishing operation manipulation for the value of feedback for three out of the four participants. Individual learning history may have influenced how reinforcing the feedback was, and how much the addition of a goal increased the value of the performance feedback. Data patterns, contribution to current research, and considerations for future research will be discussed in the context of behavioral phenomena observed, as well as confounding variables and limitations of the present study. Use of Multiple Interventions to Improve Staff Performance Catania, Almeida, Liu-Constant, and Digennaro Reed (2009), Durgin, Mahoney, Cox, Weetjens, and Poling (2014), and Kalisch, Xie, and Ronis (2013) used antecedent interventions in the form of training packages that utilized either task clarification, video modeling, or role-playing in order to improve staff performance. Howard and DiGennaro Reed (2014), Sarokoff and Sturmey (2004), Koegel, Russo, 32

42 33 and Rincover (1977), and Roscoe and Fisher (2008) implemented staff training procedures that utilized some combination of instruction, modeling, and role-playing in addition to feedback. Other staff performance intervention packages modified staff behavior through the use of tangible reinforcement. Stephens and Burroughs (1978), Pedalino and Gamboa (1974), and Miller, Carlson, and Sigurdsson (2014) utilized a monetary reward and Burroughs and Richardson (1975) provided time off to reinforce staff behavior. The current study provided only feedback or feedback paired with a goal. Goal setting and feedback were not paired with additional training or tangible reinforcers. The results suggested that implementation of additional training procedures or providing an additional reinforcer may not be necessary to improve staff performance, and that feedback alone is effective in producing changes in staff behavior. This is useful to consider when evaluating the cost effectiveness and time restraints of staff improvement interventions. In order to provide additional training to staff members, an employer must dedicate valuable time and resources in the form of a trainer, time outside of usual work duties, and training materials. When implementing reinforcement in the form of monetary or time-off rewards the employer loses money and staff availability. Providing feedback and a goal requires very little time and few physical resources. Thus, it is an efficient alternative to more complex training packages. The current study s data supported findings of previous researchers who also implemented

43 34 feedback and goal setting without additional training or reinforcement. In previous studies that utilized goals with feedback, the necessity of the goal was not evaluated. Feedback Combined with Goal Setting Downing Jr. and Gellar (2012), DiGennaro, Martens, and Kleinmann (2007), Wilk and Redmon (1998), and Martens, Hiralall, and Bradley (1997) all successfully improved staff behavior using feedback and goal setting. The current study replicated many components of Martens et al. s (1997) study while analyzing the effect of goal setting as an establishing operation manipulation to increase the value of feedback. The results of the current study contributed to research on the value of feedback and the role of goal setting in manipulating the reinforcing value of feedback. The present study s results suggested that feedback alone can effectively reinforce staff behavior, and that the addition of a goal can increase the reinforcing value of feedback. The results of the current study additionally suggested that goal setting may not effectively change the value of feedback for all individuals, and may not be necessary to obtain meaningful behavior change. Thus, goal setting may be beneficial but not necessary when implementing a feedback intervention. It may prove useful to evaluate factors that influence the effectiveness of feedback as a reinforcer and goal setting as an establishing operation based manipulation prior to introducing a staff performance intervention that includes feedback and goal setting. Staff Performance Impact on Client Performance in ABA Treatment Koegel, Russo, and Rincover (1977), Howard and DiGennaro Reed (2014), DiGennaro et al. (2007), and Martens, et al. (1997) demonstrated correlations