Institute of Nuclear Power Operations So You Want to do a Survey? G. Kenneth Koves Ph.D. NAECP Conference, Austin TX 2015 September 29 1 Who am I? Ph.D. in Industrial/Organizational Psychology from Georgia Tech Sprint 7.5 years INPO 10 years Done lots of work in using, creating, and analyzing surveys Part of the IAEA safety culture survey development panel 2 Agenda Answer some FAQs Open it up to your questions (If you have questions in the mean time, go ahead and ask) Review other prepared questions if we have time 3 1
Should I administer a survey? Will you do anything with it? How will it be communicated? Are you expecting actionable results? 4 Should I buy a survey or make a survey? It all depends Construct of interest? Applicability? Specificity? Cost? Norms? Length? 5 What is a scale? Response Scale Example: a 7 point Likert response scale with Don t know/not applicable 1=Strongly disagree 2=Disagree 3=Somewhat disagree 4=Neither agree nor disagree 5=Somewhat agree 6=Agree 7=Strongly agree Don t know/not applicable 6 2
What is a scale? Factors A factor (sometimes called a scale) is a group of questions that have been statistically shown to be related to each other and the construct of interest (e.g., Questioning Attitude) 7 Why so many similar questions? The picture of reliability Reliability of a single item ~.2 But a good scale will be.8 or.9 Takes at least three items for acceptable reliability (typically 4-6 items) Questioning Attitude Item #1 8 Psychometric Basics Reliability High reliability is when the scale measures a particular construct the same way every time Higher reliability implies the survey will be more immune to transient events Measured 0 to 1 higher value is more likely to provide a similar outcome at different times Fundamental to quality and validity Validity Degree to which the test measures what it is supposed to measure Example: a weight scale 9 3
Is it Statistically Significant? Without going into what statistical significance means, what people really what to know is How confident can I be in the results? or How confident can I be that the results are relatively accurate? I won t go into confidence intervals, confidence levels, etc. 10 Sample size vs. total population (Confidence Level = 95%, Margin of Error = 5% 11 Is it Statistically Significant? Confidence, to a large degree, is a function of sample size and the population size of the smallest group of interest Typically, Ns of 300-400 adequately cover all population sizes if you are looking at the high-level results The critical question is, How much confidence can I have in the results for the smallest group/population of interest? When can you have an N (sample) of 10 and have complete confidence in the results? 12 4
What is a good response rate? It is the number of responses relative to the population that gives you confidence that the results are relatively accurate Sorry, there is no definitive number If is a function of the factors that drive confidence Population size Sample size Smallest population of interest 13 Institute of Nuclear Power Operations Survey Fallacies 14 Let s force people to decide! Sometimes managers want to eliminate the middle, or neutral, point in an odd numbered scale Research shows that the proportions of favorable and unfavorable remain about the same so you gain nothing while losing information which can affect your interpretation An extreme example to show the point: 15 5
Do you want battered, deep-fried broccoli at the company picnic? Results of a question with three-point survey response scale Favorable Neutral Unfavorable 2 297 1 What is your interpretation? How will the results look with a two-point scale? Favorable Unfavorable 200 100 Now what is your interpretation? By eliminating the neutral point you lose information and gain nothing! 16 Straightlining The term straightline comes from the days when survey respondents completed bubble sheets and might draw a straight line down the responses without reading the questions Managers are often concerned about the number of people who straightline negative answers and drive the scores lower than where management think is appropriate 17 17 Straightlining Bias Data From a representative sample of stations From six years of data; three cycles 7 point scale 1= Strongly disagree; 7= Strongly agree The vast majority of straightliners are positive Sometimes there are more at larger plants, but not always There can still be true variance in straightlining What patterns do you notice? 18 6
Reverse coding to identify bias Primary question: Are respondents paying attentions to the way they are responding to the items? Similar to straightlining Problems with reverse coding If they are identical then you are adding extra questions that add little to no value Unless the items are identical you don t know if differences are real or not Negatively worded questions often group together confusing the factor structure Our research and experience does not indicate a issue If it was an issue you would probably see more straightlining 19 Survey Fatigue Survey Fatigue is a function of multiple variables not just the length of a survey or number of surveys administered per year 1. What is done with the survey results? 2. How well are results communicated back to participants? 3. How difficult are the items? Long? Wordy? Convoluted? Multiple clauses? 4. Do the response scales keep changing? Number of options? Changing anchors? 5. How many items are there? 6. How many surveys administered? (Related to #1) 20 What are some really bad questions? I clearly differentiate to decision makers assumptions from facts and engineering judgment from empirical analysis to ensure the limitations of technical analysis are fully understood When presenting design options I separate facts from assumptions I demonstrate a deep personal commitment and obligation to ensure plant conditions and proposed changes are appropriately bounded by requirements of plant design and emergency response strategies, as well as applicable operating, design, and safety margins When proposing design changes I ensure the changes meet all requirements 21 7
What Range for a Rating Scale? If variance is king then increase the number of rating scale options For example scales ranging from 1 to 10 or 1 to 100 People treat 1-100 scales like 1-10 scales Respondents tend to clip off the ends of larger scales (e.g. 10 point scales) In general, the greatest levels of variance occur with five point or seven point scales The sweet spot for variance 22 Correlation does not imply causation Simply because a correlation is strong does not mean that the two variables are causal. For example, ice cream sales are strongly correlated with crime rate; however, they do not directly relate to each other. They both simply occur at a higher rate in the summer time. So when you find a strong correlation, you need to question the nature of the relationship, e.g., does one variable relate to another, or are both variables influenced by something else, such as a contextual variable that was not measured (in the example, this was the season). But causation does imply correlation. 23 Institute of Nuclear Power Operations Your survey questions If we run out of questions, I have additional questions that I have been asked 24 8
Additional questions about surveys Is it preferable to have one long survey covering as many aspects as possible or several shorter surveys focused on narrower areas of interest? What is, based on experience, the optimum number of questions / the optimum time needed to complete a survey? Is it necessary or advisable to collect "demographic" information such as: work group / area of expertise; number of years of experience in the organization; manager or worker; university studies or just high school; age; gender; etc.? Does the time of conducting the survey influence the results? If yes, how? Does it matter whether it is summer or winter, holiday season or high workload, before or after the salary is paid, before or after organizational changes, etc.? How to motivate people to participate in surveys? What are the potential causes of low level of participation in surveys? 25 Additional questions about surveys How do people like surveys: do they see them as an opportunity to express an opinion or as some additional work to be done with no benefit to themselves? Can the designer / developer of the survey influence the respondents? (i.e. because of the way in which questions are formulated) How to avoid such influence so that the survey provides objective results? What are the most common uses of surveys in nuclear organizations? You have the results of a survey: how do you interpret them? Do you look for patterns? Do you compare them to some pre-defined expectations or to the results from previous surveys? Is there specific training needed to interpret the results of surveys? Should the results of surveys be made available to all those who have taken part in answering or should they be used only by the managers for their information? Is the Likert scale particularly useful? Is it the most common? Is it the best? Is there such thing as the best tool for quantifying the results of surveys? 26 Additional questions about surveys People may interpret questions in surveys differently. What can be done to ensure they all understand the questions in the same way? What are the differences between safety culture and safety climate surveys? Should surveys include questions / statements like: I am... I know... I feel... etc. or they should better have questions / statements like: The workers... The managers... My colleagues... The organization... Should surveys aim to determine how an employee sees himself or herself in relation to the organization or just to get his / her opinion of the organization? What is happening internationally regarding culture surveys? 27 9
Institute of Nuclear Power Operations Final Questions? 28 10