New Mexico Striving Toward Excellence Program (NM STEP), The Data Scholars Initiative for Child Welfare

Transcription

1 2015 New Mexic Striving Tward Excellence Prgram (NM STEP), The Data Schlars Initiative fr Child Welfare YEAR ONE ANNUAL EVALUATION REPORT 2015 LARICIA LONGWORTH-REED THE BUTLER INSTITUTE FOR FAMILIES University f Denver

2 Acknwledgements This publicatin represents a cllabrative effrt by the Butler Institute fr Families, University f Denver; New Mexic Children Yuth and Families Department, Prtective Services Research, Assessment and Data Bureau (RAD); New Mexic Striving Tward Excellence Prgram (NM STEP) Data Schlars Initiative; and Safety+Success, the develpers f the STEP learning management system. This wrk is funded by Casey Family Prgrams. Principal Authr: Laricia Lngwrth-Reed, Research Assciate Recmmended citatin: Butler Institute fr Families, University f Denver, (2015). New Mexic Striving Twards Excellence Prgram (NMSTEP), The Data Schlars Initiative fr Child Welfare: Year One Annual Reprt, Denver, Clrad. Fr mre infrmatin: Fr mre infrmatin regarding this evaluatin, cntact Laricia Lngwrth-Reed, Laricia.Lngwrth- Reed@du.edu r (303) Please visit the Butler Institute fr Families website at Fr mre infrmatin n STEP in New Mexic, cntact Katherine O. Curtney, PhD, Bureau Chief fr Prtective Services Research Assessment and Data Bureau at Katherine.curtney@state.nm.us r (505) Fr mre infrmatin n the STEP learning management system, cntact Dminic Cappell at Safety+Success, dm@safetyandsuccess.cm r (505) The cntents f this dcument are slely the respnsibility f the Butler Institute fr Families. 1

3 This dcument may be reprduced in whle r part withut restrictin as lng as the Butler Institute fr Families is credited fr the wrk. Upn request, the cntents f this dcument will be made available in alternate frmats t serve accessibility needs f persns with disabilities. Table f Cntents Acknwledgements... 1 Executive Summary... 3 Evaluatin Highlights... 3 Intrductin... 4 Initiative Overview... 4 Prgram Mdel... 5 This Reprt... 6 Methdlgy... 8 Evaluatin Design... 8 Data Cllectin Results New Mexic Statewide Baseline NMSTEP Participants Results Qualitative Findings Cnclusin Opprtunities fr Further Explratin Appendices Table f Tables Table 1. Evaluatin Design Summary... 8 Table 2. Summary f Statewide Baseline Survey and Participants Pre- and Pst-Training Surveys Table 3. Means f Statewide Baseline (n=350) Table 4. Means and Standard Deviatins fr Training Satisfactin Items Table 5. Means and Standard Deviatins fr Learning Transfer Factr Items Table f Figures Figure 1. Prgram Missin, Gal and Visin... 4 Figure 2. Cre Prgram Mdel Cmpnents... 5 Figure 3. Cmparisn f STEP Data Participant Means and Statewide Means Figure 4. Cmparisn f STEP Data Participant Means frm Baseline t Fllw-up, *p< Figure 5. Cmparisn f STEP Cmpetencies Befre and After Training

4 Executive Summary The New Mexic Striving Tward Excellence Prgram (NM STEP) seeks t educate and empwer New Mexic Prtective Services (PS) emplyees with prfessinal develpment pprtunities in areas surrunding evaluatin and research t benefit children and families. The gal f NM STEP is t incrprate data-infrmed appraches int practice and increase PS wrkfrce skills in data analysis, research and evaluatin. Over the curse f nine mnths, PS emplyees participate in NM STEP thrugh web-based learning, caching sessins, and cnducting their wn experiments. The Butler Institute fr Families cnducted a mixed methds evaluatin f NM STEP fr the first chrt f NM STEP participants frm January t September f Data were cllected frm participants thrugh pre- and pst-training web-based surveys and tw fcus grups held during the 9- mnth training. Evaluatin Highlights Assessment f participants perceptin f Wrkplace Supprts, Readiness fr Change, CQI Attitudes and Utilizatin, Learning Culture, Shared Visin, Jb Satisfactin, Supervisin, and Self- Efficacy shwed a psitive and statistically significant change fr all NM STEP participants fllwing training. Participants reprted increased cnfidence with the NM STEP Training Cmpetencies frm pret pst-training acrss all cmpetencies. Assessment f the participants cnfidence with the STEP cmpetencies ver time shwed a psitive and statistically significant change fr all NM STEP training cmpetencies. Participants expressed high levels f training satisfactin with the training, trainers and teaching styles and caching sessins. Participants demnstrated high levels f mtivatin t transfer their learning. This indicates that supprts prvided thrugh NM STEP facilitate participants incrprating their learning int practice. Training challenges revealed that participants felt they wuld benefit frm mre technlgical supprts such as access t Wi-Fi as well as supprt at the agency level in rder t recgnize the time cmmitment, especially fr nline cntent that is cmpleted ut f the ffice. 3

5 NM STEP is future-fcused, as it seeks t engage New Mexic PS emplyees in data-driven practice. The results frm this pilt year demnstrate the ptential fr the NMSTEP and its impact n the child welfare wrkfrce in New Mexic. Intrductin In 2015, the New Mexic Children, Yuth and Families Department Prtective Services Research Assessment and Data Bureau, in cllabratin with Casey Family Prgrams, launched NM STEP, a 9- mnth blended-learning prfessinal develpment prgram. It was designed t build the capacity f New Mexic s PS wrkfrce t prmte data-driven and results-riented practice within the cntext f Cntinuus Quality Imprvement t imprve child safety, permanency, and well-being. The current reprt prvides infrmatin abut the prgram s prgress and utcmes during year ne (2015) f implementatin. Initiative Overview Missin Prmte a data-driven and results-riented prtective services prcess t supprt children and strengthen families. Gal The prgram is firmly grunded in the values f the child welfare agency with a wrkfrce cmmitted t the gals f helping children and their families. We are cmmitted t finding slutins t sme f ur mst persistent prblems. Visin We envisin a statewide netwrk f child welfare prfessinals skilled in data analysis wrking tgether, sharing strategies, and vercming challenges with the capacity t best serve their cmmunities. Figure 1. Prgram Missin, Gal and Visin 4

6 The nine-mnth prgram is designed arund the Cntinuus Quality Imprvement (CQI) framewrk, which includes assessment, planning, actin, and evaluatin. The initiative builds participants skills in data mining, analysis, and research. The NM STEP uses the CQI framewrk and skillbuilding t guideparticipants thrugh the implementatin f lcal experiments/prjects in rder t slve lcal challenges. Cmpnents include classrm experiences, caching and web-based lessns, surveys, multimedia, and links t data, research, and resurces. NM STEP seeks t prepare the PS wrkfrce with the skills, resurces, and supprt t strengthen CQI thrugh data mining, analysis, research, cmmunicatin, and brainstrming, in rder t accmplish the prgram missin, visins and gals (Figure 1). The verall missin f the prgram is t educate and empwer lcal PS ffices, as well as central ffice staff, t utilize CQI t increase psitive utcmes fr children and families in New Mexic. Prgram Mdel Multimedia presentatins plus large and small grup discussins t prvide pprtunities t learn, share, debate, rle play and dialgue. Technical assistance in data analysis, research, sftware, technlgy, capacity-building, grup facilitatin and cmmunicatin. Classrm Instructin Caching Lcal Sharing + Experiments Web-Based Lessns + Resurces Lcal experiments give participants the pprtunity t apply CQI with the supprt f instructrs. Participants share all aspects f the prgram with their lcal ffice. Figure 2. Cre Prgram Mdel Cmpnents A learning mangement system husing lessns, links t data, research, and all the tls and virtual training needed t supprt a lcal data-driven, results-riented prcess. 5

7 Over 9 mnths, participants receive tw days per mnth f face-t-face training and supprt frm the NM STEP training team. The NM STEP training team is staffed by the Bureau Chief f RAD, the training specialist frm Safety+Success, and the develpers f the STEP learning management system. The first day f training is cmmitted t classrm instructin, fllwed by a day f caching by RAD staff and experts frm Casey Family Prgrams, fcused n lcal ffice challenges. Classrm instructin and caching are cmplemented by nline learning mdules cmpleted frm the lcal ffice via webbased lessns thrughut the 9 mnths f training. Additinally, each lcal ffice team chses a lcal challenge t address with nging supprt frm the instructrs. The curse fcuses n experiment areas designed t achieve strategic gals identified in New Mexic s Child and Family Services Plan (CFSP) and aligned with the state s Child and Family Services Review (CFSR). As a result, the curse addressed the fllwing lng-term gals: Gal 1: Imprve placement stability fr children in PS custdy Gal 2: Decrease shrt stays Gal 3: Decrease repeat maltreatment Gal 4: Imprve assessment f parents' and legal guardians' needs, including absent and incarcerated parents r guardians, and address the lack f prvisin f services t meet thse needs Gal 5: Increase recruitment and retentin f Prtective Services field staff Finally, knwledge-sharing and knwledge transfer are crucial t NM STEP, as participants are tasked with sharing knwledge and resurces frm the training with lcal ffice peers t facilitate a peer-t-peer learning netwrk that allws fr the diffusin f the training cntent t the lcal wrkfrce. Additinally, NM STEP serves as ne f New Mexic s tls fr implementing the state s CFSR Perfrmance Imprvement Plan (PIP). This Reprt The Butler Institute fr Families at the University f Denver is cnducting an evaluatin f the NMSTEP t assess the initiative prgress twards achieving the NMSTEP gals. This reprt evaluates prgress during the first chrt f implementatin and answers the fllwing questins. 1. D participants exhibit cmpetency and knwledge gain as a result f training? 2. T what extent are participants satisfied with training? 3. T what extent is there evidence f transfer f learning? 6

8 7

9 Methdlgy Evaluatin Design The evaluatin f NM STEP uses a mixed-methds design t cllect bth qualitative and quantitative data t answer key evaluatin questins. The evaluatin was designed with a three-year prjectin t examine shrt-term participant utcmes; intermediate lcal agency utcmes; and, eventually, lng-term statewide utcmes. The NM STEP three-year evaluatin plan can be reviewed in detail in Appendix D. The current evaluatin cvers the first year f prgram implementatin, where shrt-term utcmes are the mst likely t be impacted. The current evaluatin fcuses n the participant-level utcmes in the initial year f prgram implementatin. Table 1 prvides a summary f the evaluatin questins that were explred during the first year f evaluatin, the targeted utcmes, surces f data, and methds used in the evaluatin design. Table 1. Evaluatin Design Summary Evaluatin Questin Outcmes Data Surces Methds Shrt Term/Participant Outcmes 1. T what extent des NMSTEP build the capacity f New Mexic Prtective Services (PS) managers and supervisrs t t: implement CQI and prmte data driven decisin making? 2. D participants exhibit cmpetency and knwledge gain as a result f training? 3. T what extent are participants satisfied with training? 4. T what extent is there evidence f Imprved cmpetency gain frm pre-pst Imprved knwledge and skills f STEP participants Use data t identify lcal prblems Mine and analyze data Assess wrkflw Research evidence infrmed slutins Implement CQI Increased use f STEP tls and resurces Transfer f learning f CQI skills t the jb thrugh applicatin and use f ROM system Imprved cmmunicatin and facilitatin skills f STEP participants Participant Selfreprt pre and pst surveys Embedded dcumentatin f lcal experiments Participant interviews and Fcus Grups Training and Caching Observatin Online Survey administered at pre-pst training t STEP participants t assess: CQI Cmpetency and Learning Objectives selfassessment Transfer f learning factrs Self-efficacy Mtivatin Cmmunicatin and Facilitatin Technlgy Supprts Training Satisfactin Review f dcumentatin fr lcal experiments, and reflectin papers. First pilt grup interviews and fcus grups at 3 and 9 8

10 Evaluatin Questin Outcmes Data Surces Methds transfer f learning? 5. Hw d additinal supprts fr supervisrs and managers increase the likelihd that supervisrs and managers will successfully implement CQI? Imprved ability f STEP participants t identify and address lcal issues thrugh the use f structured experiments that use data t infrm decisins Imprved jb self-efficacy amng STEP participants Successful implementatin f lcal experiments mnths int training and pst training. STEP Training and caching bservatin cmpleted by Butler researchers at 3 mnths. 6. T what extent d participants successfully utilize STEP technlgy? 9

11 Data Cllectin Measures In rder t examine the impact f STEP participatin n participant utcmes, the NM STEP survey was created t cllect quantitative indicatrs f participants wrkplace climate, facilitative factrs fr utilizing CQI, and individual characteristics that might be impacted by NMSTEP participatin. Table 2 prvides a detailed summary f each subscale that cmpses the battery, the scale type, and number f items, and prvides a general descriptin and sample items. Table 2. Summary f Statewide Baseline Survey and Participants Pre- and Pst-Training Survey Measure (Subscales) Authrs Scale Type # f Items Demgraphics n/a Mix multiple chice and penended items Learning The Butler Institute Agreement Culture Scale fr Families scale Readiness fr Change Self-Efficacy The Butler Institute fr Families TCU Institute f Behaviral Research, Frt Wrth Agreement scale Agreement scale General Descriptin and Sample Items 10 Jb characteristics (psitin, department/unit, wrk histry, etc.) 11 Child welfare staff s perceptin f hw the rganizatin fsters and supprts prfessinal learning activities. Staff take the time t reflect abut the wrk. Staff strategize ways t imprve practice. 10 Child welfare staff s perceptin abut rganizatinal practices that prmte an envirnment suitable fr change. Management clearly cmmunicates hw changes will affect ur practice. The reasns fr the changes are clear. 5 Child welfare staff s perceptin f their wn ability t perfrm their wrk. I cnsistently plan ahead and then carry ut my plans. 10

12 Measure Authrs Scale Type # f General Descriptin and Sample Items (Subscales) Items Shared Visin Ellett, A. J. (2009). Intentins t remain emplyed in child welfare: the rle f human caring, self-efficacy Agreement scale 4 Child welfare staff s perceptin f their unit s chesin in terms f rganizatinal visin. C-wrkers in my unit clearly understand the agency visin fr child welfare prgrams. beliefs, and prfessinal rganizatinal culture. Children and Yuth Services Review, 31(1), Jb Ellet and Agreement 6 Child welfare staff s jb satisfactin. Satisfactin TCU Institute f Behaviral Scale In my wrk, I have a feeling f success and accmplishment. Research, Frt Wrth All in all, I am satisfied with my jb. Supervisin (Knwledge, The Butler Institute fr Families Agreement scale 18 Child welfare staff s perceptin f supervisin prvided t them. Supprt & Skill) My supervisr knws hw t assess safety and risk. My supervisr values my pinin in case decisin-making. My supervisr is able t gather relevant case infrmatin frm me. Time Pressure The Butler Institute fr Families Agreement Scale 5 Child welfare staff s perceptin f the availability f time t cmplete their wrk. I dn't have enugh time t d my Wrkplace Supprts CQI Attitudes and Utilizatin The Butler Institute fr Families The Butler Institute fr Families Agreement Scale Agreement Scale jb effectively. 6 Child welfare staff s perceptin f rganizatinal supprts fr new initiatives. We lk fr new ways t imprve ur practice. We seek creative slutins t challenges. 5 Child welfare staff s perceptin f CQI. My agency actively prmtes CQI. 11

13 Measure (Subscales) Authrs Scale Type # f Items General Descriptin and Sample Items Training Participants Only Training Cmpetencies New Mexic STEP Data Schlars Agreement Scale 6 Participants cnfidence in utilizing CQI skills cvered in training, including: Initiative Data mining + analysis Research + analysis f evidenceinfrmed practice fr PS Open cmmunicatin fcused n challenges and slutins Develpment f experiments Implementatin f evidence infrmed practice Data-driven decisin-making Technlgy Supprt The Butler Institute fr Families Agreement Scale 6 Whether technlgy supprts are in place t facilitate a blended apprach t learning. Technlgy at my agency is current and up t date. I am eager t learn t use new technlgy that might imprve my wrk. Training Participants Only at 9 Mnths Training Satisfactin Transfer f Learning STEP Cmmunicatin The Butler Institute fr Families Natinal Child Welfare Wrkfrce Institute The Butler Institute fr Families Satisfactin Scale Agreement Scale Agreement Scale 5 Participants verall satisfactin with STEP 28 Participant s perceptins n facilitatrs, barriers, and mtivatin fr transferring skills gained in training t their jb 4 Participants cnfidence in cmmunicating and facilitating lcal agency presentatins n STEP and their lcal experiments. Survey Administratin In January 2015, staff frm the Butler Institute fr Families administered the NM STEP Statewide Survey t 800 staff f the New Mexic Children, Yuth and Families Department. The web-based survey examined numerus agency climate and culture dmains, including supervisin, jb satisfactin, and learning culture, as well as several additinal scales. In August 2015, staff frm the Butler Institute fr Families administered the NM STEP Pst-Training Survey t the graduates f the 2015 NM STEP pilt chrt. The web-based survey examined the same cnstructs that were administered at baseline as well 12

14 as additinal reprts f participants NM STEP cmpetencies, training satisfactin, and learning transfer factrs. Fcus Grups In March and August f 2015, the third and ninth mnths f the training, NM STEP participants jined in cnversatins t discuss their experiences f the NM STEP pilt training. All participants were asked t cntribute their feedback in a fcus grup. Each grup was held fr apprximately ne hur, and participants were asked t share thughts n their experiences, expectatins, and challenges while participating in STEP. Results The current reprt presents bth quantitative and qualitative results frm the surveys f statewide New Mexic Prtective Services staff, and pre-pst surveys and fcus grups f NM STEP participants. Data were analyzed by calculating mean scres and frequencies f the survey respnses, and cnducting thematic cntent analyses f qualitative data gathered thrugh interviews and fcus grups. Additinally, statistical analyses were cnducted t determine changes in key areas ver time. Due t small sample sizes, the current findings are preliminary and shuld be interpreted with cautin. New Mexic Statewide Baseline Table 3 presents means, standard deviatins and categrizatin f each area as a challenge, watch area, r strength area frm the survey f statewide Prtective Services staff. At baseline, statewide staff reprted high levels f self-efficacy. Hwever, all ther areas fell int a watch r challenge area. Time pressure was the mst significant challenge that emerged frm the survey results. At this time, these results are purely descriptive and are intended t prvide a picture f the statewide results fr measures f interest. 13

15 Table 3. Means f Statewide Baseline (n=350) Area and Example Item Self-Efficacy Supervisin Jb Satisfactin Shared Visin Learning Culture CQI Attitudes and Utilizatin Mean Scre & Standard Deviatin (SD) Challenge Watch Area Strength (1.05) 4.65 (0.90) 4.42 (0.98) 4.26 (0.99) 4.23 (1.10) 5.14 (.613) Readiness fr Change 3.76 (1.19) Wrkplace Supprt 3.48 Time Pressure 2.80 (0.81) (1.30) 14

16 NMSTEP Participants Results NM STEP participants were als assessed prir t training n the same areas examined thrugh the statewide survey f New Mexic PS staff. Figure 3 shws the scres n each area fr NM STEP participants cmpared t the statewide scres. Due t a large difference in the sample sizes f the tw grups, n statistical cmparisns can be made. Hwever, descriptively, we see a trend f NM STEP participants having higher scres at baseline n almst all dmains cmpared t the statewide sample. The ne exceptin is the area f self-efficacy, where participants scres are nly slightly lwer than the statewide average. Again, these results are purely descriptive at this time and are intended t prvide a picture f participant perfrmance at baseline cmpared t statewide averages. STEP Data Schlars (n=22) Statewide Baseline (n=317) Time Pressure Wrkplace Supprt Readiness fr Change CQI Attitudes and Utilizatin Learning Culture Shared Visin Jb Satisfactin Supervisin Self-Efficacy Figure 3. Cmparisn f NM STEP Participant Means and Statewide Means 15

17 Pre- and Pst-Training After attending NM STEP, participants were asked t cmplete a fllw-up survey t again measure areas examined at baseline, in rder t examine any changes ver time. Figure 4 shws the scres n each area fr MM STEP participants at baseline and at fllw-up. Scres increased fr all areas frm baseline t fllw-up except time pressure. Additinally, all increases were statistically significant. Data Schlars Baseline (n=22) Fllw-up (n=19) Time Pressure Wrkplace Supprt* Readiness fr Change* CQI Attitudes and Utilizatin* Learning Culture* Shared Visin* Jb Satisfactin* Supervisin* Self-Efficacy* Figure 4. Cmparisn f NM STEP Participant Means frm Baseline t Fllw-up, *p<.05 Pre- and Pst-Training Cmpetencies Befre and after attending NM STEP, participants were asked t rate their cmpetency level n the critical cmpetency dmains and learning bjectives f NM STEP. Participants reprted increased cnfidence with the NM STEP training cmpetencies frm pre- t pst-tests acrss all cmpetencies (Figure 5). A paired-samples t-test was cnducted t cmpare pre- and pst-training means fr the whle NM STEP Training Cmpetencies scale. Thugh the sample is small, preliminary results indicate a statistically significant change in the participants cnfidence with NM STEP cmpetencies, with an increase in cnfidence rating frm befre training (M=2.13) t after training (M=3.57), (t (17) = 10.19, p<.000). 16

18 Befre After Data-driven decisin-making Implementatin f evidence-infrmed practice Experiment Develpment Open cmmunicatin/brainstrming fcused n challenges and slutins Research+Analysis f evidence-infrmed health/safety strategies fr cmmunity partners Research + Analysis f evidence-infrmed practice fr PS Data mining+analysis Figure 5. Cmparisn f NM STEP Cmpetencies Befre and After Training Training Satisfactin After cmpleting the training, participants were asked t reprt (n a 6-pint scale) the degree t which they were satisfied with the training cntent and frmat as well as the trainers. As a whle, participants expressed high levels f satisfactin with the training. Mean scres indicate that participants were highly satisfied with caching (M = 5.74), training discussin (M = 5.79), peer netwrking (M = 5.79), and the supprt frm the STEP Initiative (M = 5.74). Training satisfactin items and means fr each item are presented in Table 4. Table 4. Means and Standard Deviatins fr Training Satisfactin Items Statement Mean Item Standard Deviatin Scre I have received the caching I need frm the STEP training I have clleagues frm the STEP training I can turn t when I need t talk thrugh challenges. During training, it was useful t discuss ideas with ther peple Thrugh the STEP training I have cnnected with peple wh understand the difficulties I face. I am satisfied with the supprt I have received frm the STEP Initiative

19 Learning Transfer Factrs The Learning Transfer Factrs Scale rated aspects f participants wrkplace envirnment and training experiences knwn t influence the degree t which new learnings may be transferred t the wrkplace. Fr this reasn, the scale was part f the pst-training survey nly. On average, participants indicated high levels f agreement that factrs such as pre-training preparatin, training applicability, mtivatin, NM STEP supprts, and wrkplace envirnment and supprts wuld enable participants t successfully transfer their learning n the jb. The verall mean fr the scale was Learning Transfer items and means fr each item are presented in Table 5. Table 5. Means and Standard Deviatins fr Learning Transfer Factr Items Statement Mean Item Scre Standard Deviatin As a result f the training, I substantially increased my knwledge n this tpic. As a result f the training, I have develped new skills I am mtivated t use what I've learned in this training I am excited t try new things that I learned in this training My skills will imprve as a result f this training My supervisr expects me t use this training n the jb I will meet with my supervisr t discuss applicatin f this training n the jb. My supervisr values prfessinal develpment fr all staff The training was relevant t my jb The infrmatin I received frm this training can definitely be used in my wrk. I am very cnfident that I will use the training n the jb I will have sufficient pprtunities t practice the new ideas/skills/techniques n the jb. During the training, I was thinking f ways I culd apply the training cntent t the jb. As a result f the training, I will be mre effective in my wrk I have already started planning with a c-wrker t use this training My c-wrkers will supprt my attempts t use the training n the jb. I feel supprted by c-wrkers in develping my CQI skills I vluntarily attended this training Prir t the training, I was mtivated t attend My supervisr helped prepare me fr this training by discussing my learning needs and ptential applicatins. Prir t attending, I heard that this training was wrthwhile /valuable

20 The trainer prvided sme practical ideas that can be used n the jb. This training cntent is cnsistent with my agency s missin, philsphy, and gals. This training cntent is cnsistent with my agency s plicies and my individual respnsibilities Qualitative Findings Overall, fcus grups at bth three mnths and nine mnths f trainings shwed that NM STEP participants were engaged and mtivated in their NM STEP participatin. Participants mtivatin, engagement and satisfactin with their experience was driven by: NM STEP s innvative training frmat Engaging trainers Hands-n learning pprtunities Caching At three mnths and at nine mnths f training the challenges facing NMSTEP included: A need fr supprt at the lcal agency level t recgnize the time cmmitment required fr the I lved diving int the data and really learning hw t design an experiment and use the data t test whether what we are ding is wrking. - NMSTEP Graduate prgram, including time utside f classrm instructin fr cmpleting nline cntent A need fr technlgical supprts, specifically access t Wi-Fi, in lcal ffices in rder fr recmmended technlgy, such as tablets, t be utilized fr nline cntent Overall, participants expressed enthusiasm and great satisfactin with NM STEP. They felt that their time was efficiently utilized and that the quality f the curse cntent and the effectiveness f the trainers were excellent. At nine mnths, participants indicated they were cnfident in their ability t transfer their learning t the wrkplace and t cntinue implementatin f their lcal experiments. 19

21 Cnclusin The year ne evaluatin fr the NM STEP explred fur key questins, which are discussed in mre detail belw. 1. T what extent d participants perceptins f individual, unit and agency factrs change after participating in training? Assessment f participants perceptin f Wrkplace Supprts, Readiness fr Change, CQI Attitudes and Utilizatin, Learning Culture, Shared Visin, Jb Satisfactin, Supervisin, and Self-Efficacy shwed a psitive and statistically significant change fr all NM STEP participants fllwing training. 2. D NM STEP participants exhibit cmpetency and knwledge gain because f training? While current findings are preliminary, initial results are prmising. Assessment f the participants cnfidence with the NM STEP cmpetencies ver time shwed a psitive and statistically significant change fr all cmpetencies. T truly understand the impact, it will be imprtant fr future evaluatin t cntinue t track participants ver time and measure cmpetencies pst-training. Additinally, due t the small study size, further evaluatin shuld capitalize n additinal chrts, t increase the pwer t detect the effects f the prgramand t increase the ability t generalize these findings. 3. T what extent are NM STEP participants satisfied with training? As a whle, participants expressed high levels f satisfactin with the training. This was supprted bth thrugh quantitative self-reprts frm training surveys and thrugh emerging themes gathered thrugh the qualitative fcus grups. Participants reprted high levels f engagement and mtivatin during the training and they fund the training useful. 4. T what extent is there evidence f transfer f learning? On average, participants indicated high levels f endrsement fr transfer f learning factrs, such as pre-training preparatin, training applicability, mtivatin, NM STEP supprts, and wrkplace envirnment and supprts that will allw them t successfully transfer their learning n the jb. 20

22 Opprtunities fr Further Explratin NM STEP is envisined as an institutinalized prcess, ffering training and technical assistance yearly t strengthen CQI and supprt lcal experiments. Year ne was seen as a pilt year t test a training prcess and assess whether participants felt engaged, infrmed and empwered. As NM STEP mves frward, new areas fr evaluatin will present themselves, especially f NM STEP n Prtective Services capacity t imprve utcmes fr children and families (which is nted in the prpsed Year 3 evaluatin plan). Future evaluatins shuld cnsider the fllwing as the prgram cntinues. 1. Impact n utcmes: T what extent are the participants ffice staff members supprting CQI, and what impact is this having n utcmes fr children and families? 2. Impact n cmmunity partners: T what extent are the participants able t transfer learning abut CQI t lcal cmmunity partners (utside child welfare) and what is the impact n reaching shared health and safety utcmes fr children and families? 3. Transfer f learning between cunty ffices r lcal ffices and central ffice: NM STEP brings tgether participants frm acrss the state t a unique learning envirnment, wrking n the lcal and state levels. Hw can we measure the impact f sharing strategies acrss the state and between management and field wrkers? 4. Opprtunities fr transfer f learning between states and agencies: As STEP is implemented in ther states, hw can we measure the sharing f successful prtcls, prgrams, and plicies that can impact psitive utcmes fr staff and clients? 5. NM STEP s ability t address CFSR gals as a PIP tl: NM STEP has been identified a tl fr facilitating New Mexic s PIP in respnse t the state s CFSR. Future evaluatins shuld cnsider explre the impact f NM STEP n the Perfrmance Imprvement Plan. 6. NM STEP s impact n career advancement within child welfare: Fllwing participatin in NM STEP, 20% f participants applied fr and received prmtins. All f these participants indicated that they had referenced STEP n their resumes and during interviews. Hw des NM STEP cntribute t participants prfessinal develpment and prepare them fr career advancement? 7. NM STEP fidelity. Prgram fidelity is crucial t btaining similar utcmes fr future participants. As NM STEP cntinues t be ffered, fidelity shuld be mnitred and any mdificatins nted, as these may affect participant satisfactin and utcmes. 21

23 Appendices A. Detailed Descriptive fr New Mexic Statewide Baseline N Minimum Maximum Mean Std. Deviatin Self-Efficacy Supervisin Jb Satisfactin Shared Visin Learning Culture CQI Attitudes and Utilizatin Readiness fr Change Wrkplace Supprt Time Pressure

24 B. Detailed Descriptive fr New Mexic STEP Participants Baseline N Minimum Maximum Mean Std. Deviatin Self-Efficacy Supervisin Jb Satisfactin Shared Visin Learning Culture CQI Attitudes and Utilizatin Readiness fr Change Wrkplace Supprt Time Pressure

25 C. Detailed Descriptive fr New Mexic STEP Participants at Fllw-up N Minimum Maximum Mean Std. Deviatin Self-Efficacy Supervisin Jb Satisfactin Shared Visin Learning Culture CQI Attitudes and Utilizatin Readiness fr Change Wrkplace Supprt Time Pressure

26 D. STEP 3 Year Evaluatin Plan Evaluatin Questin Outcme Data Surce Methd Shrt Term/Participant Outcmes 1. T what extent des NMSTEP build the capacity f New Mexic Prtective Services (PS) managers and supervisrs t implement CQI and prmte data driven decisin making? 2. D participants exhibit cmpetency and knwledge gain as a result f training? 3. T what extent are participants satisfied with training? 4. T what extent is there evidence f transfer f learning? 5. Hw d additinal supprts fr supervisrs and managers increase the likelihd that supervisrs and managers will successfully implement CQI? 6. T what extent d participants successfully utilize STEP technlgy? Imprved cmpetency gain frm pre-pst Imprved knwledge and skills f STEP participants t: Use data t identify lcal prblems Mine and analyze data Assess wrkflw Research evidence infrmed slutins Implement CQI Increased use f STEP tls and resurces Transfer f learning f CQI skills t the jb thrugh applicatin and use f ROM system Imprved cmmunicatin and facilitatin skills f STEP participants Imprved ability f STEP participants t identify and address lcal issues thrugh the use f structured experiments that use data t infrm decisins Imprved jb self-efficacy amng STEP participants Successful implementatin f lcal experiments Participant Selfreprt pre and pst surveys Embedded dcumentatin f lcal experiments Participant interviews and Fcus Grups Training and Caching Observatin Online Survey administered at pre-pst training t STEP participants t assess: CQI Cmpetency and Learning Objectives selfassessment Transfer f learning factrs Self-efficacy Mtivatin Cmmunicatin and Facilitatin Technlgy Supprts Training Satisfactin Review f dcumentatin fr lcal experiments, and reflectin papers. First pilt grup interviews and fcus grups at 3 and 9 mnths int training and pst training. STEP Training and caching bservatin cmpleted by Butler researchers at 3 mnths. Intermediate/Lcal Office Outcmes 7. T what extent Increased knwledge and des the PS skills f PS wrkfrce abut wrkfrce in lcal the imprtance f CQI and cunty ffices exhibit increased use f data knwledge and skill in CQI and the use f data? Self-Reprt survey f lcal cunty ffice PS staff Embedded lcal ffice PS staff surveys Lcal ffice PS staff interviews and fcus grups Online Survey administered pre -pst training delivery t all staff at lcal ffices participating in STEP and t a cmparisn grup f lcal ffices matched n cnstituent demgraphics, t assess: 25

27 Evaluatin Questin Outcme Data Surce Methd 8. Is there evidence f imprved service utcmes in lcal cunty ffices? 9. T what extent is there an increase in efficient utilizatin f STEP tls and resurces such as the ROM data system? Increased use f STEP tls and resurces by PS wrkfrce Effective use f ROM system by all levels f PS wrkfrce t identify, address and slve practice issues Lcal experiment dcumentatin Cunty level indicatrs frm ROM system and state data systems Knwledge f CQI and data use Self-efficacy Mtivatin Lcal Experiments Feedback Onging lcal ffice surveys administered by STEP participants t lcal acrss 9 mnths f training. Interviews with cunty ffice managers and fcus grups with staff frm randm sample f cunty ffices that have sent staff t STEP training pst training delivery. Dcumentatin review fr lcal experiments. Lcal experiment identificatin Supprting Data Lcal Experiment Plans Training Pster Presentatins Access ROM System usage indicatrs fr participating cunties. Access cunty level indicatrs targeted in lcal experiments frm state data system. Lng-term/Statewide Outcmes 10. T what extent des STEP prmte diffusin f learning frm STEP participants t PS wrkfrce t strengthen all aspects f CQI Institutinalize statewide Data Participants prgram t implement CQI. Statewide netwrk f PS wrkfrce skilled in data mining and analysis Self-Reprt survey f all cunty ffice PS staff State level indicatrs frm ROM system and state data systems Online Survey administered pre STEP delivery and annually acrss the state t all cunty ffices t assess: Knwledge f STEP Data Participants prgram 26

28 Evaluatin Questin Outcme Data Surce Methd (Cntinuus Quality Imprvement) Practice acrss all cunty ffices t imprve services delivery t families 11. Hw des NMSTEP build the capacity f NMPS wrkfrce t prmte data driven and results riented practice t imprve child safety, permanency, and well-being. Imprved capacity f PS wrkfrce t prmte datadriven and results-riented practice t supprt children and strengthen families Imprved safety, permanency and well-being f children and families CQI Implementatin Mtivatin Knwledge f CQI and data use Access statewide practice indicatrs frm state data system. Access ROM System usage indicatrs fr the state 27