5/20/ Administration & Analysis of Surveys

Transcription

1 5/20/ Administration & Analysis of Surveys

2 Goals Considerations when Administering Surveys Statistical Analysis of Surveys Examples of Reporting Survey Data 5/20/2015-2

3 Accuracy of Survey Data is Affected by Many Factors Controlled Factors Test Factors & Levels Survey Data Analysis Effectiveness & Suitability Potential factors: Tasks Operating conditions Systems Individuals Groups Order of events Uncontrolled Factors Factor + Error = Analysis Types of error Problem Random error Statistical power Uncontrolled factors Specious effects 5/20/2015-3

4 Administration Procedures = Controlled Factor Surveys = Measure of thoughts Thoughts altered by context (time, place, abilities, etc ) Feelings fade quickly Ensure consistency Same time Same environment Same instructions Same administrator Ensure participants represent population Range of skills & abilities in test; NOT just best individuals Ensure participant s opinion & not others Little interaction between respondents & test team (contractor, PM, etc ) Administrator presents as if no opinion on survey 5/20/2015-4

5 Considerations for Data Exploration and Analysis Data collected from surveys are bound by the same principles as data collected through other means Follow best practices for analyzing data in DOE context Exploratory Data Analysis (EDA)» Detect anomalies» Test model assumptions and maximize insights into data Statistical Assumptions» Regression, ANOVA, Chi-Square, etc. Select the appropriate analysis or descriptive statistics for your data scenario and study goals Emphasis on the relationship between demographic variables and survey outcomes 5/20/2015-5

6 Survey Design & Test Goal Determine Appropriate Statistic 1: What measurement scale? 2: Descriptive or inferential? 3: What is question being asked of data? Survey Design Sample & Data Test Goal Measurement Scale Descriptive Statistics Inferential Statistics Describe the data collected Infer from sample to the population Categorical Continuous 5/20/ Nominal Ordinal Interval/ Ratio Mode Frequency distribution Median Mean Standard deviation Percentiles Chi square All of the above and Sign test All of the above and Correlation Regression t test ANOVA

7 Check the assumptions Inferential Analysis Could Aid Interpretation NO Normally Distributed Data NO Report descriptive statistic/investigate # of Participants Strongly (1) Somewhat (2) MEAN = 3.4 Slightly (3) MEDIAN = 3.5 MODE = 4 Slightly Agree (4) Question 1 Somewhat Agree (5) Strongly Agree (6) 5/20/2015-7

8 Inferential analysis could aid interpretation Normally distributed data NO 5/20/ # of Participants Strongly (1) Check the assumptions MODE = 2 Somewhat (2) MEAN = 3.8 MEDIAN = 3.5 Slightly (3) Slightly Agree (4) Question 1 Yes ½ agree & ½ disagree Somewhat Agree (5) Check demographics. Nonparametric test may be appropriate (e.g. sign test) Strongly Agree (6)

9 Inferential analysis could aid interpretation Normally distributed data NO 5/20/ # of Participants Strongly (1) Check the assumptions Inexperienced Operators MODE = 2 Somewhat (2) MEAN = 3.8 MEDIAN = 3.5 Slightly (3) Slightly Agree (4) Question 1 Yes Somewhat Agree (5) Check demographics. Nonparametric test may be appropriate (e.g. sign test) Experienced Operators Strongly Agree (6)

10 Check the assumptions Inferential analysis could aid interpretation Normally distributed data Yes # of Participants Strongly (1) Somewhat (2) MODE = 3 MEDIAN = 3 MEAN = 3.43 MEAN = 3.4 Slightly (3) MEDIAN = 3.5 MODE = 4 Slightly Agree (4) Question 1 Yes Somewhat Agree (5) Common Inferential Statistics may be appropriate (e.g. regression) Strongly Agree (6) 5/20/

11 Selecting Statistics Step 1: What measurement scale? Step 2: Descriptive or inferential? Step 3: What is question being asked of data? Survey Design Data & Sample Test Goal Mode What is the Most Common Response? Nominal Descriptive Median What is the 50 th Percentile Response? Ordinal Descriptive Mean What is the Average Response? Interval Descriptive Std. Deviation How Variable is the Data? Interval Descriptive Chi Square Is Frequency Distribution Different from Expected? Nominal Inferential Sign Test Is RV Different from a Threshold/Standard? Ordinal Inferential Correlation Are 2 Factors Correlated? Interval Inferential Regression Can a RV Be Predicted from Other Factors? Interval Inferential t test Is RV Different from a Threshold/Standard? Interval Inferential ANOVA Is RV Different Under Different Levels of Factor(s)? Interval Inferential 5/20/

12 GBS Survey Results Survey ratings are generally positive, but technical orders were insufficient to configure the legacy receive suite for the GBS DECC broadcast, requiring GBSOC help 7-point scale used with rating levels 1-3 defined as unacceptable and 4-7 as sufficient or better Rating Topic Receive suite operator ratings of overall acceptability of GBS DECC products Receive Suite operator ratings of the overall acceptability of the process to develop a GMR GBSOC operator ratings of the GBS DECC for planning broadcast missions Number of Respondents Mode Number of Answers Receive suite operator ratings of the GBSOC Help Desk function GBSOC operator ratings of GBS DECC safety GBSOC and receive suite operator ratings of GBS DECC documentation GBSOC operator ratings of training GBSOC operator ratings of human factors topics GBSOC operator ratings of the overall acceptability of the GBS DECC planning software 5/20/ Acronyms this slide: Defense Enterprise Computing Center (DECC); Global Broadcast Service (GBS); GBS Operations Center (GBSOC)

13 Frequency Distributions for Categorical Survey Data # of Respondents3 Years of Experience < 5 yrs > 5 yrs Grouped Bar Chart: Presents the response frequencies of different sub groups. Stacked Bar Chart: Presents the Cumulative Frequency ( running total ) of responses. (ordinal data) Worst Very Bad Bad Good Very Good Best Question 4 Question 3 Question 2 Question 1 5/20/ % 25% 50% 75% 100% % Respondents

14 Frequency Distributions for Continuous Survey Data % of Respondents Histogram: Presents the frequency distribution for intervals of responses System Usability Scale 100 Scatter Plot: Presents the relationship between a continuous response variable & a continuous factor Workload System 1 System Yrs Experience 5/20/

15 Using SUS to compare versions: DSL self install 95% Success in the lab 90% Install ethernet card New modems introduced Kortum, P., Grier, R. & Sullivan, M. (2009). DSL Self-installation: From Impossibility to Ubiquity. Interfaces, 80, /20/

16 Using NASA TLX to compare versions: value of multi-modal system to C 2 Grier, R.A., Parasuraman, R., Entin, E., Bailey, N., & Stelzer, E. (2008). A test of intra- versus inter-modality interference as a function of time pressure in a warfighting simulation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting in New York City. 5/20/

17 5/20/ QUESTIONS?