Stat Wk 8: Continuous Data

Transcription

1 Stat Wk 8: Continuous Data proc iml Loading and saving to datasets proc means proc univariate proc sgplot proc corr Stat 342 Notes. Week 3, Page 1 / 71

2 PROC IML - Reading other datasets. If you want to use an existing dataset as a matrix, rather than setting your own manually, you can do this with the 'use' and 'close' commands, combined with the 'read' command. 'Use' is used to tell the system to bring a certain dataset into memory. 'Close' is to wipe it from memory (useful for keeping things running smooth). Read is used to take that dataset in memory, and save it into a matrix. Stat 342 Notes. Week 3, Page 2 / 71

3 The general syntax is use <DATASET>; read all var <VARIABLE NAMES> into <MATRIX NAME>; close <DATASET>; Stat 342 Notes. Week 3, Page 3 / 71

4 Use (and close) use the same syntax for specifying a dataset as other SAS procedures. The dataset is specificed by libname.dataset, and addition options like (obs = ) can be used to specify which rows to load into memory. use ds1; use work.ds1; use somelib.cars; use somelib.cars(obs = 10); Stat 342 Notes. Week 3, Page 4 / 71

5 Close is the same, except that the options are not necessary in the close statement. close ds1; close work.ds1; close somelib.cars; close somelib.cars; Stat 342 Notes. Week 3, Page 5 / 71

6 With the read command, references to entire classes of variable names still work, including _ALL_, _NUM_, and _CHAR_ for all variables, numeric ones, and character ones, respectively. proc iml; use SASHELP.Cars(OBS=5); read all var _NUM_ into m1; read all var _NUM_ into m2[colname=numericnames]; close SASHELP.Cars; print m1, m2; Stat 342 Notes. Week 3, Page 6 / 71

7 Matrices are saved as SAS datasets with the create command and the append command. 'Create' makes a new SAS data set of whatever library and name you specify. However, the create command alone only makes a blank dataset (with whatever formatting you specify). 'Append' takes data from some matrix in the IML environment and adds (appends) it to the particular dataset, such as the one you just made with 'create'. Stat 342 Notes. Week 3, Page 7 / 71

8 The general syntax is... create <DATASET> from <MATRIX>; append from <MATRIX>; close <DATASET>; Like when loading, it's a good idea to close your datasets after you're done with them. Stat 342 Notes. Week 3, Page 8 / 71

9 Example of saving data, and seeing with with a proc print; proc iml; m = {1 2 3., }; m = t(m); create newds from m[colname={"x","y"}]; append from m; close work.newds; quit; proc print (data=newds); Stat 342 Notes. Week 3, Page 9 / 71 run;

10 Readings on proc iml Textbook source: Pages of 'SAS and R'. Additional source for interest: SAS Whitepaper Getting started with the SAS/IML Language, by Rick Wilkin pdf Stat 342 Notes. Week 3, Page 10 / 71

11 I hope IML isn't too fuzzy from two weeks ago Stat 342 Notes. Week 3, Page 11 / 71

12 Proc means and proc univariate are used to describe individual variables on their own. Their capabilities are very similar, but They have different defaults and slightly different syntaxes. 2. Proc univariate can create histograms and boxplots. Stat 342 Notes. Week 3, Page 12 / 71

13 Proc summary also operates similarly to proc means and proc univariate, but since Proc summary can't produce output to the screen (only to another file or dataset) and its syntax is less intuitive, we won't be covering it. Stat 342 Notes. Week 3, Page 13 / 71

14 The general syntax for proc means is... proc means data=input_ds <table options> <statistics of interest>; <class/by varnames>; var <numeric variables>; output <output_ds> <statistic = varnames>; run; Stat 342 Notes. Week 3, Page 14 / 71

15 The complete list for <table options> is found here: HTML/default/viewer.htm#a htm Notables include: missing When summarizing by groups using a categorical variable, this makes 'missing data' count as a group (shown in document camera example). Stat 342 Notes. Week 3, Page 15 / 71

16 Alpha= <'0.05'> The significance level. Used for confidence interval statistics. The default value is 0.05, which will provide 95% confidence intervals. Maxdec= <2> The maximum digits of precision after the decimal point. 0 means everything is rounded to the nearest whole number, 1 means everything is rounded to the nearest 0.1, and so on. Print / noprint Produce or ignore on-screen output. Stat 342 Notes. Week 3, Page 16 / 71

17 Example 1: What does this do? proc means data=input_ds alpha=0.10 maxdec=3 missing noprint;...; run; Stat 342 Notes. Week 3, Page 17 / 71

18 <statistics of interest> is where we list the statistics we wish to use to describe the variables. proc means data=input_ds <table options> <statistics of interest>; var <numeric variables>; <class/by varnames>; output <output_ds> <statistic = varnames>; run; Stat 342 Notes. Week 3, Page 18 / 71

19 By default the stats computed are n mean std min max n The number of observations. std The standard deviation of the observations. Other notables include sum stderr var and... nmiss The number of values missing clm Confidence limits of the mean. Affected by the alpha level determined earlier, but defaults to the 95% confidence bounds/limits. Stat 342 Notes. Week 3, Page 19 / 71

20 Q1 Q3 The first (lower) and third (upper) quartiles of the data, where 25% and 75% is below these points, respectively. qrange Interquartile range, the difference between Q1 and Q3. In other words, the size of the middle half of the data. Skewness skew Skewness, or skew. The central third moment. If this is positive, then you have more extreme Stat 342 Notes. Week 3, Page 20 / 71

21 data values above the mean than below. In other terms, the distribution stretches or 'skews' in the positive direction. kurtosis The fourth moment, used to describe how peak- like (positive kurtosis), or flat/plateau-like a unimodal distribution like the normal distribution is. P1 P5 P10 P90 P95 P99 Percentiles. The points at which 1%. 5%, 10%, 90%, 95%, and 99% of the (observed) values for a given variable. See also quartiles, and the median. Stat 342 Notes. Week 3, Page 21 / 71

22 This is the complete list Stat 342 Notes. Week 3, Page 22 / 71

23 var <numeric variables> defines the variables that will be summarized. proc means data=input_ds <table options> <statistics of interest>; var <numeric variables>; <class/by varnames>; output <output_ds> <statistic = varnames>; run; Stat 342 Notes. Week 3, Page 23 / 71

24 Example 2: What does this do? proc means data=input clm std P1 P5 Q1 median Q3 P95 P99;...; run; Stat 342 Notes. Week 3, Page 24 / 71

25 Now you can be the meanest of the mean. Stat 342 Notes. Week 3, Page 25 / 71

26 var <numeric variables> defines the variables for which summary statistics are required. By default, EVERY numeric variable is included in the summary, even ones that wouldn't make sense to take statistics of, like observation numbers, IDs, and phone numbers. This is just like if you had specified _numeric_ in the list of variables. Stat 342 Notes. Week 3, Page 26 / 71

27 The batting dataset, found in your s and on the website of the auithor of SAS and R includes year-by-year records of many different Major League Baseball players. It's more than 90,000 rows long, and includes thousands of players. We are interested in finding and confirming year-toyear patterns like 'have strike-outs become more common in recent years'. A dataset like this is too large to feasibly print to screen. SAS University Edition will only show 100 rows at a time. Stat 342 Notes. Week 3, Page 27 / 71

28 However, we CAN use a procedure like proc means to effectively characterize the data. (Apologies to everyone that doesn't care about baseball, but its discrete events make it the easiest to analyze of popular sports in North America) (Go Cubs!) Stat 342 Notes. Week 3, Page 28 / 71

29 Loading the batting dataset proc import datafile='dir_location\baseball.csv' out=ds dbms=csv; delimiter=','; getnames=yes; run; Stat 342 Notes. Week 3, Page 29 / 71

30 In the batting dataset, the variables that make the most sense would be games batted (G_batting), and the number of at-bats (AB), runs (R), hits (H), home-runs (HR) walks (BB), intentional walks (IBB), and strike outs (SO). Therefore, the variable specification line would look like this: var G_batting AB R H HR BB IBB SO; Stat 342 Notes. Week 3, Page 30 / 71

31 Output <out_ds> defines the dataset that the output will be saved to (if any). <statistic = varnames> defines the variable names for the statistics that you produce. proc means data=input_ds <table options> <statistics of interest>; <class/by varnames>; var <numeric variables>; output <out_ds> <statistic = varnames>; run; Stat 342 Notes. Week 3, Page 31 / 71

32 The format of the statistics and varnames is quite rigid. These work as expected, producing the sum of AB and BB for the relevant set of observations: output out = out_ds sum(ab) = Atbats sum(bb) = Walks; However, this causes a compiling error: output out = out_ds sum(bb, IBB) = Walks2; Stat 342 Notes. Week 3, Page 32 / 71

33 Finally, <class/by varnames> is what you use to specify how you wish to group each set of observations. Without it, you will get the specified statistics of every observation put together. proc means data=input_ds <table options> <statistics of interest>; <class/by varnames>; var <numeric variables>; output <out_ds> <statistic = varnames>; run; Stat 342 Notes. Week 3, Page 33 / 71

34 This code would give you the summary stats of (yearid, AB, SO) for all the data combined. You'll get the total at-bats and strikeouts of ALL observations in the data set, and your output will be a single row. proc means data=batting sum; var AB SO; run; Stat 342 Notes. Week 3, Page 34 / 71

35 This code, however, will give you the total number of at-bats and strikeouts for EACH year. proc means data=batting sum; var AB SO; class yearid; run; Stat 342 Notes. Week 3, Page 35 / 71

36 Classes are sorted in ascending alphabetical / numeric order. Proc means also gives the number of observations for each class. No class variable. Stat 342 Notes. Week 3, Page 36 / 71 Class yearid.

37 More that one variable can be used as a class variable. If this is the case, observations will be grouped by each unique combination of the class variables, and aggregated within that group. Class (and its pre-sorted version, 'by') may take a VERY long time to do this for a large dataset. It is HIGHLY recommended to test multiple class variables by some smaller 'toy' dataset first to avoid locking up your SAS terminal for hours. Stat 342 Notes. Week 3, Page 37 / 71

38 class yearid lgid; A variable can be one of the variables summarized AND be a class variable at the same time. The results may be confusing though (or obvious in the case of mean/median). proc means data=batting sum mean; var yearid AB SO; class yearid lgid; run; Stat 342 Notes. Week 3, Page 38 / 71

39 The output will look like this as a table. Stat 342 Notes. Week 3, Page 39 / 71

40 ...and this if you use the default dataset output. output out=ds_name;... it looks like this. Stat 342 Notes. Week 3, Page 40 / 71

41 Next in this series, hypothesis tests! Stat 342 Notes. Week 3, Page 41 / 71

42 Before we continue, let's create some derived variables for the batting average, strike-out average, and walking average of each player per year. Each of these variables is a proportion of the at-bats that a given player experiences in a given year. data batting; set batting; batavg = H / AB; so_avg = SO / AB; walk_avg = sum(bb, IBB) / AB; run; Stat 342 Notes. Week 3, Page 42 / 71

43 Also, let's limit our dataset to years and players in which at least 300 at-bats were observed. This way our derived variables are proportions of a large number of observations. data batting_300ab; set batting; if AB ge 300 then output; run; Stat 342 Notes. Week 3, Page 43 / 71

44 Consider the distributions of the these three derived variables. Intuitively, they should be normally distributed. Is that the case? Let's check the histograms with the UNIVARIATE procedure. Including the 'histogram' statement in 'proc univariate' tells SAS to make histograms of the variables you specify. If you don't specify any variables, it uses every variable that was listed in the 'var' statement. Including '/normal' at the end of the histogram statement adds a fitted normal curve to the histograms created. Stat 342 Notes. Week 3, Page 44 / 71

45 The 'var' statement will include every numeric variable in the dataset unless you specify otherwise. So this code will produce summary statistics and histograms for the three derived variables we made earlier, with fitted normal curves included proc univariate data=batting_300ab; var batavg so_avg walk_avg; histogram run; Stat 342 Notes. Week 3, Page 45 / 71 / normal;

46 Stat 342 Notes. Week 3, Page 46 / 71

47 Stat 342 Notes. Week 3, Page 47 / 71

48 Stat 342 Notes. Week 3, Page 48 / 71

49 We can also check the quantile-quantile (vs normal distribution) plots for these variables, and see how well they fit. In a quantile-quantile plot, the observations of a variable are sorted and compared to some theoretical quantiles. For example: Say we have 1000 observations, and we want to compare the distribution of these observations to a normal distribution. The quantile of the normal distribution is 1.96 standard deviations below the mean. If our data is normal, then the 25 th lowest data point should also be 1.96 sd below the mean. Stat 342 Notes. Week 3, Page 49 / 71

50 Quantile-quantile plots can be produced with the 'qqplot' statement in the univariate procedure. The '/ normal' specifies the distribution. Normal is the default. Other options include 'beta', 'exponential', 'gamma', and 'Weibull'. proc univariate data=batting_300ab; var batavg so_avg walk_avg; qqplot / normal; run; Stat 342 Notes. Week 3, Page 50 / 71

51 If the selected distribution fits the data, we should see a straight line. (It doesn't have to be perfect at the extreme ends) Stat 342 Notes. Week 3, Page 51 / 71

52 Skewed distributions will demonstrate a single bend (vs normal). Stat 342 Notes. Week 3, Page 52 / 71

53 More complicated deviations from will exhibit complex behaviour, such as s-curves or extreme values. Stat 342 Notes. Week 3, Page 53 / 71

54 Finally, we can conduct formal tests for normality by using the normal option right after we specify the input dataset. proc univariate data=batting_300ab normal; var batavg so_avg walk_avg; run; Some normality tests have minimum / maximum sample sizes. Only the appropriate tests will be performed. (Shapiro-Wilks test, for example, won't show for N > 2000) Stat 342 Notes. Week 3, Page 54 / 71

55 Batting averages Strikeout averages Walking averages Stat 342 Notes. Week 3, Page 55 / 71

56 Let's go exploring for bivariate trends. Stat 342 Notes. Week 3, Page 56 / 71

57 For continuous data, we can look at two variables at a time in several ways. - Correlation - Scatterplots - Regression For scatterplots, we can use the SGPLOT procedure, which standard for Statistical Graphic PLOT. Stat 342 Notes. Week 3, Page 57 / 71

58 Proc sgplot is the basic procedure for a wide variety of graphing options. These include: Scatterplots with the scatter statement. Bar plots with the hbar and vbar statements. Box plots with the hbox and vbox statements. Fitted regression lines with the reg statement. Locally fitted spline-smoothed fit with the loess statement. Kernel-based density curves with the density statement. Stat 342 Notes. Week 3, Page 58 / 71

59 For a scatterplot between strikeout average and batting (hitting) average, the code is: proc sgplot data=batting_300ab; scatter x=batavg y=so_avg; run; Stat 342 Notes. Week 3, Page 59 / 71

60 Stat 342 Notes. Week 3, Page 60 / 71

61 We can draw a 99% prediction ellipse around these data points with the 'ellipse' statement. The default is a 95% level for confidence/prediction, including for the ellipse, so we decrease alpha in the settings in order to make an ellipse that's farther out and more visible. proc sgplot data=batting_300ab; scatter x=batavg y=so_avg; ellipse x=batavg y=so_avg / alpha = 0.01; run; Stat 342 Notes. Week 3, Page 61 / 71

62 Stat 342 Notes. Week 3, Page 62 / 71

63 There are tons of data points (about 19,700), so it's hard to get a sense of the relative density of data points. We can fix that with some graphical options. proc sgplot data=batting_300ab; scatter x=batavg y=so_avg /transparency = 0.9; ellipse x=batavg y=so_avg / alpha = 0.01; run; Stat 342 Notes. Week 3, Page 63 / 71

64 Stat 342 Notes. Week 3, Page 64 / 71

65 We can do something more formal with the correlation procedure, PROC CORR. For example, we can get the simple pairwise (Pearson) correlation between all three of our derived variables with the following: proc corr data=batting_300ab; var batavg so_avg walk_avg; run; Stat 342 Notes. Week 3, Page 65 / 71

66 This will produce some simple summary stats, and the correlation coefficient, p-value against 0 correlation (two-tailed), and number of observation pairs for each pair of variables. Stat 342 Notes. Week 3, Page 66 / 71

67 We can look much deeper into this with alternative correlation measures, like Spearman's Rank Sum (ideal for non-linear, monotonic relationships), or Kendall's Tau. proc corr data=batting_300ab Pearson Spearman Kendall; var batavg so_avg walk_avg; run; Stat 342 Notes. Week 3, Page 67 / 71

68 This will produce equivalent matrices for the other measures. If you still want the Pearson correlation as well, you have to specify it. Stat 342 Notes. Week 3, Page 68 / 71

69 We can also produce scatterplots and histograms of each variable and pair of variables in a neatly-arranged matrix. For large datasets, you may need to get around the default limit of the number of data points to draw with the maxpoints option in the plot() settings. proc corr data=batting_550ab plots=matrix(histogram) plots(maxpoints = 50000); var batavg so_avg walk_avg; run; Stat 342 Notes. Week 3, Page 69 / 71

70 Stat 342 Notes. Week 3, Page 70 / 71

71 Readings on week 8 material Textbook source: Chapter 5 'SAS and R'. Sources for your interest: The Essential Meaning of PROC MEANS Readings on week 9 material (next week) Textbook source: Chapter 6 'SAS and R'. Stat 342 Notes. Week 3, Page 71 / 71