THE DEFENSE INTELLIGENCE AGENCY LEADER EVALUATION of DEVELOPMENT and INVESTMENT DIALED IN FY 2014 ANNUAL MANAGEMENT REPORT A report designed to aid in making informed evidence-based decisions when recommending improvements to the Leader Development Program. Academy for Defense Intelligence DIA Leader Advantage Webinar Results.
DIA LEADER DEVELOPMENT PROGRAMS... 1 SUMMARY...1 RECOMMENDATIONS...1 LEADER ADVANTAGE WEBINAR SURVEY OVERVIEW and OBSERVATIONS... 2 OVERVIEW OF LEADER ADVANTAGE WEBINAR SURVEY TOOL...2 QUICK-LOOK SURVEY QUESTIONS... 2 WEBINAR AVERAGE BASED ON PARTICIPANT FEEDBACK, USING THE SCALE:... 2 OFFERED... 2 ATTENDED... 2 REGISTERED... 2 NO SHOWS... 2 CANCELLATION... 2 SURVEYS... 2 MOMENT of LEARNING NEED... 2 WEBINARS by TOPIC...3 WEBINARS by ITERATION...4 WEBINAR VALUE as DELIVERED by FACILITATOR / PRODUCER...5 1
DIA LEADER DEVELOPMENT PROGRAMS SUMMARY The collective survey data from participants in DIA s Leader Development Webinars events agreed or strongly agreed they found value and quality in DIA s Leader Development curriculum. As rated, the Primary Indicators of Value (PIV) and Primary Indicators of Quality (PIQ) suggest shared success across the development continuum. The FY2014 Defense Intelligence Agency Leader Evaluation of Development and Investment (DIALED IN) Management Report displays consolidated survey data drawn from Questionmark Perception (QMP). Managers can use this report to help with: Evidence-based decisions, review of individual and collective course survey results, and comparisons of iterative data against the averages for focused improvement. This FY2014 report shows survey results for Leader Advantage Webinars, using Quick-Look Micro-Metrics (QL2M) to measure Primary Indicators of Value (PIV) in the leader competencies delivered within the semiformal learning venue of the collaborative virtual environment. The QL2M survey helps evaluate each Webinar categorized by both topic and iteration in support of the DIA Leader Advantage Program. The survey also includes questions on the learner s moment of learning need. QL2M measures Webinar PIV in: Value, relevancy, and benefit. RECOMMENDATIONS Leader Advantage Manager: As Webinar NO SHOW rate continue to trend upward, attempt to determine if the same employees register yet do not attend. Though currently not an issue, NO SHOWS will become a challenge with the data showing a steady increase in registration numbers. This trend will require registration quotas in the future. 1
LEADER ADVANTAGE WEBINAR SURVEY OVERVIEW and OBSERVATIONS OCTOBER 1, 2013 - SEPTEMBER 30, 2014 OVERVIEW OF LEADER ADVANTAGE WEBINAR SURVEY TOOL Participants in Leader Development Webinars were surveyed using Quick-Look Micro-Metrics (QL2M) to measure Primary Indicators of Value (PIV) in leader competency delivered within the semi-formal learning venue of the collaborative virtual environment. The metrics and measures used in the survey: QUICK-LOOK SURVEY QUESTIONS 1. The Webinar helped me see more clearly how this topic is relevant to my job. 2. The Webinar provided me with some practical suggestions and/or best practices. 3. The Webinar will help me discuss this topic with others in the workplace. 4. The knowledge I gained through interactions with others during the Webinar will help me improve my performance as a leader. WEBINAR AVERAGE BASED ON PARTICIPANT FEEDBACK, USING THE SCALE: 1 - STRONGLY DISAGREE 2 - DISAGREE 3 - NEITHER AGREE NOR DISAGREE 4 - AGREE 5 - STRONGLY AGREE OFFERED Number of times the Webinar was delivered. ATTENDED Total participants who attended the topic; average attended per topic. REGISTERED Total employees who registered to attend; average registration per topic. NO SHOWS Total expected on day of event that did not attend; rate of no shows per topic [NO SHOWs divided by (REGISTERED minus CANCELLATION)] CANCELLATION Total participants who cancelled; rate of cancellation per total registrants. SURVEYS Surveys completed; survey rate completed per attendees. MOMENT of LEARNING NEED Shown in % of Participants that attended the Webinar to: 1. Learn about this subject for the first time. 2. Learn more about this subject. 3. Refresh my memory to apply this subject. 4. Find a solution to a specific problem related to this subject. 5. Change my approach to the subject because things have changed. 2
WEBINARS by TOPIC OBSERVATION: The 462 surveys collected from 530 participants represents a highly acceptable 87% survey completion rate. These 462 data points provided the consolidated value-based survey data from 56 Webinar iterations within 24 Webinar topics offered between 1 October 2013 and 30 September 2014. Participants Agreed or Strongly Agreed they found value, relevancy, and benefit in the Webinar topics, rating the Overall Value (OV) in topic average at 4.38 on a scale of 5. With a rating of 4.41 and 4.40, participants indicate they found the most value in 1) practical suggestions and/or best practices provided in the Webinar topics and 2) the relevancy of the topic to their job. The Moment of Learning Need shows 57% of participants attended Webinars to learn more about a particular topic, 17% attended as a refresher, and 18% attended Webinars to learn about the subjects for the first time. FY2014 LEADER ADVANTAGE WEBINARS by TOPIC # OFFERED SURVEY QUESTION TOPIC AVERAGE Attended OV AVG 1 2 3 4 Total per Total AVG per Total Rate Total Rate Total Rate 1 2 3 4 5 Accountability at Work 2 4.23 4.00 4.10 4.23 4.14 9 5 20 10 6 40% 5 25% 9 100% 13% 78% 10% 0% 0% Building Trust Managing Rapid Change 2 4.59 4.25 4.50 4.46 4.45 9 5 9 5 0 0% 0 0% 7 78% 0% 46% 42% 13% 0% Business Execution Culture 3 4.26 4.19 4.33 4.12 4.23 27 9 46 15 10 27% 9 20% 24 89% 42% 43% 11% 0% 3% Communication Effectiveness 3 4.18 4.43 4.44 4.26 4.33 24 8 33 11 10 30% 0 0% 20 83% 15% 57% 28% 0% 0% Considering Conflict 4 4.23 4.27 4.23 4.22 4.24 40 10 65 16 17 30% 8 12% 34 85% 21% 46% 31% 0% 3% Controlling Projects 1 4.62 4.75 4.62 4.62 4.65 10 10 26 26 9 47% 7 27% 8 80% 13% 75% 13% 0% 0% Developing and Motivating People 1 4.46 4.46 4.31 4.46 4.42 14 14 34 34 9 36% 9 26% 13 93% 23% 77% 0% 0% 0% Diversity Expectations 2 4.68 4.55 4.58 4.45 4.56 11 6 21 11 5 33% 6 29% 9 82% 10% 43% 25% 10% 13% Finance and Your Role 1 4.22 4.33 4.22 4.22 4.25 9 9 14 14 4 31% 1 7% 9 100% 33% 44% 25% 0% 0% Giving and Receiving Feedback 2 4.60 4.78 4.77 4.65 4.70 18 9 33 17 9 33% 6 18% 16 89% 52% 31% 17% 0% 0% Influence and Persuasion 1 4.31 4.54 4.46 4.31 4.41 14 14 35 35 16 53% 5 14% 13 93% 0% 62% 38% 0% 0% Leading Effective Meetings 4 4.50 4.68 4.52 4.50 4.55 48 12 108 27 27 36% 33 31% 46 96% 12% 55% 15% 9% 9% Leading Successful Teams 6 4.41 4.35 4.33 4.33 4.35 90 15 124 21 17 16% 19 15% 74 82% 16% 48% 26% 4% 7% Listening to the Unspoken 2 4.55 4.62 4.47 4.54 4.54 21 11 39 20 8 22% 3 8% 18 86% 21% 56% 5% 12% 7% Motivational Leader 3 4.65 4.50 4.35 4.27 4.44 18 6 27 9 10 40% 2 7% 15 83% 8% 44% 19% 13% 15% Need for Perseverance and Flexibility 1 4.17 4.08 4.08 4.08 4.10 12 12 26 26 11 48% 3 12% 12 100% 8% 67% 17% 8% 0% Negotiating Technique 1 4.45 4.36 4.27 4.09 4.29 11 11 18 18 3 21% 4 22% 11 100% 18% 55% 27% 0% 0% Planning to Communicate Vision 3 4.39 4.36 4.53 4.39 4.42 16 5 27 9 9 36% 2 7% 14 88% 22% 53% 25% 0% 0% Problem Solving Process 2 4.27 4.37 4.17 4.10 4.22 26 13 44 22 13 33% 5 11% 21 81% 4% 66% 14% 7% 11% Removing Barriers to Change 4 4.17 4.17 4.10 3.90 4.09 33 8 38 10 5 14% 1 3% 25 76% 16% 46% 23% 10% 8% Removing Obstacles to Innovation 1 4.40 4.40 4.40 4.40 4.40 6 6 23 23 12 67% 5 22% 5 83% 0% 100% 0% 0% 0% Strategic Thinking 1 4.09 4.27 4.27 4.18 4.20 13 13 20 20 5 28% 2 10% 11 85% 45% 55% 0% 0% 0% Successfully Managing Up 3 4.34 4.45 4.49 4.37 4.41 38 13 72 24 18 32% 16 22% 35 92% 11% 51% 16% 9% 11% What to Delegate 3 4.77 4.69 4.39 4.61 4.62 13 4 29 10 11 46% 5 17% 13 100% 8% 65% 17% 5% 5% TOTALS, AVERAGES, RATES, and PERCENTAGES 56 4.40 4.41 4.37 4.32 4.38 530 9 931 17 244 32% 156 17% 462 87% 17% 57% 18% 4% 4% Registered No Shows Cancellations PARTICIPANTS Surveys MOMENT of LEARNING NEED 3
WEBINARS by ITERATION OBSERVATION: Participants Agreed or Strongly Agreed they found value in each Webinar, rating the Overall Value (OV) by iterative average at 4.37. The annual average attendance of 9 participants per Webinar steadily increased from an average of 6 attendees per session in Q1, to 8 in Q2, to 11 in Q3, to 12 in Q4. The average 'No Show' rate of 32% shows a steady increase from 26% in Q1 to a 38% in Q4. An indicator of effective advertising and word-of-mouth marketing is shown in the steady increase of registration average per Webinar, up 300% from 8 in Q1 to 24 in Q4. FY2014 OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEPT WEBINAR ITERATION TIME DELIVERE PARTICIPANTS QUESTIONS OV No Shows Cancellations Surveys MOMENT of LEARNING NEED Attended Registered 1 2 3 4 Total Rate Total Rate Total Rate 1 2 3 4 5 Building Trust Managing Rapid Change 0800 4.67 5.00 5.00 4.67 4.84 4 4 0 0% 3 75% 0% 67% 33% 0% 0% Removing Barriers to Change 1400 4.00 3.80 4.20 3.40 3.85 9 9 0 0% 5 56% 0% 60% 20% 20% 0% Motivational Leader 1600 5.00 5.00 4.75 4.75 4.88 4 4 0 0% 4 100% 25% 50% 0% 25% 0% Leading Successful Teams 0800 4.80 4.60 4.60 4.60 4.65 5 5 2 40% 5 100% 20% 40% 40% 0% 0% Building Trust Managing Rapid Change 1000 4.50 3.50 4.00 4.25 4.06 5 5 0 0% NOT 4 80% 0% 25% 50% 25% 0% Leading Successful Teams 1200 4.78 4.22 4.22 4.67 4.47 10 10 1 10% COLLECTED 9 90% 11% 67% 11% 11% 0% Communication Effectiveness 1400 4.17 4.33 4.67 4.33 4.38 8 8 1 13% FY14Q1 6 75% 17% 50% 33% 0% 0% Listening to the Unspoken 1600 4.60 4.80 4.36 4.64 4.60 12 15 5 33% 11 92% 27% 55% 9% 9% 0% What to Delegate 0800 5.00 4.50 4.50 4.50 4.63 2 6 4 67% 2 100% 0% 50% 50% 0% 0% Communication Effectiveness 1200 4.00 4.33 4.00 4.00 4.08 3 8 5 63% 3 100% 0% 67% 33% 0% 0% Motivational Leader 1400 4.33 4.00 3.67 3.67 3.92 6 9 6 67% 3 50% 0% 33% 33% 0% 33% Communication Effectiveness 0800 4.36 4.64 4.64 4.45 4.52 13 17 4 24% 0 0% 11 85% 27% 55% 18% 0% 0% What to Delegate 1000 4.75 5.00 4.25 4.75 4.69 4 5 1 20% 0 0% 4 100% 25% 75% 0% 0% 0% Planning to Communicate Vision 1200 4.17 4.33 4.33 4.17 4.25 7 11 3 30% 1 9% 6 86% 17% 83% 0% 0% 0% Removing Barriers to Change 1400 4.22 4.22 3.89 3.89 4.06 9 10 1 10% 0 0% 9 100% 22% 33% 33% 0% 11% Accountability at Work 1600 4.20 4.00 4.20 4.20 4.15 5 6 0 0% 1 17% 5 100% 0% 80% 20% 0% 0% Finance and Your Role 0800 4.22 4.33 4.22 4.22 4.25 9 14 4 31% 1 7% 9 100% 33% 44% 25% 0% 0% Considering Confllict 1000 4.50 4.50 4.50 4.50 4.50 4 13 5 56% 4 31% 4 100% 0% 25% 75% 0% 0% Planning to Communicate Vision 1200 4.50 4.25 4.75 4.50 4.50 5 8 3 38% 0 0% 4 80% 25% 50% 25% 0% 0% Removing Barriers to Change 1400 4.67 4.67 4.50 4.50 4.59 7 9 2 25% 1 11% 6 86% 33% 50% 17% 0% 0% Removing Barriers to Change 0900 3.80 4.00 3.80 3.80 3.85 8 10 2 20% 0 0% 5 63% 10% 40% 20% 20% 20% Leading Successful Teams 0930 4.33 4.20 4.20 4.07 4.20 18 19 1 5% 0 0% 15 83% 7% 40% 47% 7% 0% Planning to Communicate Vision 0800 4.50 4.50 4.50 4.50 4.50 4 8 3 43% 1 13% 4 100% 25% 25% 50% 0% 0% Strategic Thinking 1000 4.09 4.27 4.27 4.18 4.20 13 20 5 28% 2 10% 11 85% 45% 55% 0% 0% 0% Leading Effective Meetings 1200 4.40 4.60 4.40 4.50 4.48 10 23 5 33% 8 35% 10 100% 10% 70% 10% 10% 0% Motivational Leader 1400 4.62 4.50 4.62 4.38 4.53 8 14 4 33% 2 14% 8 100% 0% 50% 25% 13% 13% Considering Confllict 0800 4.17 4.00 4.00 4.17 4.09 10 17 5 33% 2 12% 4 40% 0% 67% 33% 0% 0% Leading Successful Teams 1200 4.12 4.44 4.19 4.19 4.24 17 22 1 6% 4 18% 16 94% 18% 38% 19% 0% 25% Business Execution Culture 1400 3.90 3.80 4.12 3.60 3.86 13 23 3 19% 7 30% 11 85% 20% 70% 0% 0% 10% Leading Effective Meetings 0800 4.46 4.62 4.38 4.38 4.46 13 36 6 32% 17 47% 13 100% 31% 38% 15% 15% 0% Leading Successful Teams 1000 4.18 4.18 4.40 4.27 4.26 14 27 5 26% 8 30% 11 79% 18% 46% 27% 0% 9% Leading Effective Meetings 1200 4.45 4.82 4.64 4.45 4.59 12 26 6 33% 8 31% 11 92% 0% 55% 27% 9% 9% Business Execution Culture 1400 4.20 4.10 4.20 4.10 4.15 11 18 5 31% 2 11% 10 91% 40% 60% 0% 0% 0% Giving and Receiving Feedback 1600 4.33 4.56 4.67 4.44 4.50 11 19 3 21% 5 26% 9 82% 33% 33% 33% 0% 0% Leading Effective Meetings 0800 4.67 4.67 4.67 4.67 4.67 13 23 10 43% 0 0% 12 92% 8% 58% 8% 0% 25% Diversity Expectations 0800 4.75 4.50 4.75 4.50 4.63 4 9 2 33% 3 33% 4 100% 0% 25% 50% 0% 25% Considering Confllict 1000 4.20 4.40 4.30 4.20 4.28 10 19 7 41% 2 11% 10 100% 20% 60% 10% 0% 10% Problem Solving Process 1200 4.15 4.23 4.08 4.08 4.14 14 25 8 36% 3 12% 13 93% 8% 69% 15% 0% 8% Business Execution Culture 1400 4.67 4.67 4.67 4.67 4.67 3 5 2 40% 0 0% 3 100% 67% 0% 33% 0% 0% Accountability at Work 1600 4.25 4.00 4.00 4.25 4.13 4 14 6 60% 4 29% 4 100% 25% 75% 0% 0% 0% Considering Confllict 0930 4.06 4.19 4.13 4.00 4.10 16 16 0 0% 0 0% 16 100% 63% 31% 6% 0% 0% Successfully Managing Up 0800 4.31 4.38 4.50 4.19 4.35 18 35 9 33% 8 23% 16 89% 19% 38% 6% 13% 25% Diversity Expectations 1000 4.60 4.60 4.40 4.40 4.50 7 12 3 33% 3 25% 5 71% 20% 60% 0% 20% 0% Leading Successful Teams 1200 4.22 4.44 4.39 4.17 4.31 26 41 7 21% 7 17% 18 69% 22% 56% 11% 6% 6% Developing and Motivating People 1400 4.46 4.46 4.31 4.46 4.42 14 34 9 36% 9 26% 13 93% 23% 77% 0% 0% 0% Listening to the Unspoken 1600 4.50 4.43 4.57 4.43 4.48 9 24 3 14% 3 13% 7 78% 14% 57% 0% 14% 14% What to Delegate 0800 4.57 4.57 4.43 4.57 4.54 7 18 6 46% 5 28% 7 100% 0% 71% 0% 14% 14% Negotiating Technique 1000 4.45 4.36 4.27 4.09 4.29 11 18 3 21% 4 22% 11 100% 18% 55% 27% 0% 0% Successfully Managing Up 1200 4.40 4.60 4.60 4.60 4.55 5 18 6 55% 7 39% 5 100% 0% 80% 20% 0% 0% Controlling Projects 1400 4.62 4.75 4.62 4.62 4.65 10 26 9 47% 7 27% 8 80% 13% 75% 13% 0% 0% Giving and Receiving Feedback 1600 4.86 5.00 4.86 4.86 4.90 7 14 6 46% 1 7% 7 100% 71% 29% 0% 0% 0% Influence and Persuasion 0800 4.31 4.54 4.46 4.31 4.41 14 35 16 53% 5 14% 13 93% 0% 62% 38% 0% 0% Problem Solving Process 1000 4.38 4.50 4.25 4.12 4.31 12 19 5 29% 2 11% 8 67% 0% 63% 13% 13% 13% Successfully Managing Up 1200 4.31 4.38 4.38 4.31 4.35 15 19 3 17% 1 5% 14 93% 14% 46% 21% 14% 7% Need for Perseverance and Flexibility 1400 4.17 4.08 4.08 4.08 4.10 12 26 11 48% 3 12% 12 100% 8% 67% 17% 8% 0% Removing Obstacles to Innovation 1600 4.40 4.40 4.40 4.40 4.40 6 23 12 67% 5 22% 5 83% 0% 100% 0% 0% 0% TOTALS, AVERAGES, RATES, and PERCENTAGES 4.40 4.41 4.37 4.32 4.37 530 931 244 32% 156 17% 462 87% 17% 54% 20% 5% 5% 4
WEBINAR VALUE as DELIVERED by FACILITATOR / PRODUCER OBSERVATION: Shows facilitator/producer Webinar iteration ratings against their own average. The Overall Value (OV) rating per iteration was transposed to each facilitator/producer that delivered the session. This table does not intend to, nor should it be used to compare ratings between individual facilitators/producers. Evidence exists of a positive correlation between the OV rating of Webinar topics and the facilitator/producer involved in delivering it. 2014 OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEPT WEBINAR NAME TIME FACILITATOR PRODUCER 1 2 3 4 5 6 7 8 9 10 11 12 12 Building Trust Managing Rapid Change 0800 4.84 4.84 Removing Barriers to Change 1400 3.85 3.85 Motivational Leader 1600 4.88 4.88 Leading Successful Teams 0800 4.65 Building Trust Managing Rapid Change 1000 4.06 4.06 Leading Successful Teams 1200 4.47 Communication Effectiveness 1400 4.38 4.38 Listening to the Unspoken 1600 4.60 4.60 What to Delegate 0800 4.63 4.63 Communication Effectiveness 1200 4.08 4.08 Motivational Leader 1400 3.92 3.92 Communications Effectiveness 0800 4.52 4.52 What to Delegate 1000 4.69 4.69 Planning to Communicate Vision 1200 4.25 4.25 Removing Barriers to Change 1400 4.06 4.06 Accountability at Work 1600 4.15 4.15 Finance and Your Role 0800 4.25 4.25 Considering Confllict 1000 4.50 4.50 Planning to Communicate Vision 1200 4.50 4.50 Removing Barriers to Change 1400 4.59 4.59 Removing Barriers to Change 0900 3.85 3.85 Leading Successful Teams 0930 4.20 4.20 Planning to Communicate Vision 0800 4.50 4.50 Strategic Thinking 1000 4.20 4.20 Leading Effective Meetings 1200 4.48 4.48 Motivational Leader 1400 4.53 4.53 Considering Confllict 0800 4.09 4.09 Developing Talent (FPM 4/15) 0900 4.57 4.57 Leading Successful Teams 1200 4.24 4.24 Business Execution Culture 1400 3.86 3.86 Leading Effective Meetings 0800 4.46 4.46 Leading Successful Teams 1000 4.26 4.26 Leading Effective Meetings 1200 4.59 4.59 Business Execution Culture 1400 4.15 4.15 Giving and Receiving Feedback 1600 4.50 4.50 Leading Effective Meetings 0800 4.67 4.67 Diversity Expectations 0800 4.63 4.63 Considering Confllict 1000 4.28 4.28 Problem Solving Process 1200 4.14 4.14 Business Execution Culture 1400 4.67 4.67 Accountability at Work 1600 4.13 4.13 Considering Confllict 0930 4.10 4.10 Successfully Managing Up 0800 4.35 4.35 Diversity Expectations 1000 4.50 4.50 Leading Successful Teams 1200 4.31 4.31 Developing and Motivating People 1400 4.42 4.42 Listening to the Unspoken 1600 4.48 4.48 What to Delegate 0800 4.54 4.54 Negotiating Technique 1000 4.29 4.29 Successfully Managing Up 1200 4.55 4.55 Controlling Projects 1400 4.65 4.65 Giving and Receiving Feedback 1600 4.90 4.90 Managing Performance (FPM 9/4) 0930 4.74 4.74 Influence and Persuasion 0800 4.41 4.41 Problem Solving Process 1000 4.31 4.31 Successfully Managing Up 1200 4.35 4.35 Need for Perseverance and Flexibility 1400 4.10 4.10 Removing Obstacles to Innovation 1600 4.40 4.40 WEBINAR AVERAGE WHEN FACILITATOR BY WEBINARS FACILATATED/ PRODUCED 4.39 4.45 4.46 4.36 4.36 4.25 4.38 4.20 4.29 4.56 4.44 4.29 4.44 22 6 2 14 6 1 3 2 9 8 3 19 19 5