Matt Weber Christopher Malone

Size: px

Start display at page:

Download "Matt Weber Christopher Malone"

Nicholas Cole
6 years ago
Views:

1 WStatistical Investigation of Methods to Calculate Area Under the Curve Introduction: Matt Weber Christopher Malone A study conducted by S.E. Landin, et. al, (2010, Effect of relaxation breathing on glycemic response in healthy humans., FASEB J. 2010; 24s:783.7.) looked at the effect of relaxation breathing on glucose levels. The subjects were split into two groups with one getting relaxation breathing and the other the control. The glucose measurements were taken at times 0, 30, 60, and 90. In this study, the researchers could utilize an area under the curve (AUC) approach to simplify and make comparisons between the relaxation and control group the results of the study. There are multiple ways to calculate an AUC statistic when using this approach. This research is being done to investigate the behavior and accuracy of the different methods. Example: Below is data from the example comparing two different methods.

Both methods yielded statistically significant results (trapezoidal method: p value=0.0195 and Fit Method: p value=0.0215).

2 Both methods yielded statistically significant results (trapezoidal method: p value= and Fit Method: p value=0.0215). Although both methods produced the same outcome there are different p values. So it is possible for another study to produce different outcomes given the same data using the different methods of analysis. Methods: There are four methods that were investigated. The Trapezoidal Method connects the points to make trapezoids and then calculates the area of each and adds them together (see Formula and Diagram). [(y i+1 +y i )/2](x i+1 x i ) The second method was the Trapezoidal Baseline Method. This method is calculated the same way as the Trapezoidal Method, but it subtracts the area below the first data point. This may be useful because in the example people have different baseline levels of glucose. So this method would adjust for that so two people who reacted the same to the treatment, but had different glucose level baselines would not be statistically different from each other. [(y i+1 +y i )/2](x i+1 x i ) y i *(x n x 1 )

The third method is the Fit Method. This method would calculate the AUC statistic by fitting a simple quadratic model, using the method of least squares, to the points.

3 The third method is the Fit Method. This method would calculate the AUC statistic by fitting a simple quadratic model, using the method of least squares, to the points. Then the AUC would be calculated by taking an integral of the fitted curve. The final method is the Fit Baseline Method. This is the same as the Fit Method, but with the baseline area subtracted off. This method may be useful for the same reasons as the Trapezoidal Baseline Method.

4 Simulation: To compare these four methods, a simulation in R was used. The R function is below. The function will create samples of size n1 and n2. Then using the model parameters (a, b, c, and error), a curve is produced and an observation is created. Then for each observation, all four AUC methods are conducted. Finally a t test is conducted for each method comparing the difference of the two samples. The formula then returns an iter amount of p values for each t test (Note: running iter=10000, takes about seven minutes and iter=2000 takes a minute to two minutes). In the appendix is also another function that will also calculate and return the Standard Error for each method comparison. Capstone=function(a1=.1,b1=10,c1=100,n1=15, a2=.1,b2=10,c2=100, n2=15, error1=1,error2=1, iter=10000){ x=c(0,30,60,90) TrapAreaA1=rep(0,n1) TrapAreaA2=rep(0,n2) TrapAreaB1=rep(0,n1) TrapAreaB2=rep(0,n2) TrapAreaC1=rep(0,n1) TrapAreaC2=rep(0,n2) TotalTrapArea1=rep(0,n1) TotalTrapArea2=rep(0,n2) TotalTrapBaseArea1=rep(0,n1) TotalTrapBaseArea2=rep(0,n2) Fit1=rep(0,n1) Fit2=rep(0,n2) FitAreaA1=rep(0,n1) FitAreaA2=rep(0,n2) FitAreaB1=rep(0,n1) FitAreaB2=rep(0,n2) FitAreaC1=rep(0,n1) FitAreaC2=rep(0,n2) TotalFitArea1=rep(0,n1) TotalFitArea2=rep(0,n2) TotalFitBaseArea1=rep(0,n1) TotalFitBaseArea2=rep(0,n2) TTAtest=rep(0,iter) TTBAtest=rep(0,iter) TFAtest=rep(0,iter) TFBAtest=rep(0,iter)

5 for (i in 1:iter){ for (j in 1:n1){ err1=rnorm(4,0,error1) y1= a1*x^2+b1*x+c1+err1 TrapAreaA1[j]=((y1[2]+y1[1])/2)*(x[2] x[1]) TrapAreaB1[j]=((y1[3]+y1[2])/2)*(x[3] x[2]) TrapAreaC1[j]=((y1[4]+y1[3])/2)*(x[4] x[3]) TotalTrapArea1[j]=TrapAreaA1[j]+TrapAreaB1[j]+TrapAreaC1[j] TotalTrapBaseArea1[j]=TotalTrapArea1[j] y1[1]*(x[4] x[1]) Fit1=lm(y1~poly(x,2,raw=T)) FitAreaA1[j]=(Fit1$coeff[3]/3)*x[2]^3+(Fit1$coeff[2]/2)*x[2]^2+Fit1$coeff[1]*x[2] FitAreaB1[j]=(Fit1$coeff[3]/3)*x[3]^3+(Fit1$coeff[2]/2)*x[3]^2+Fit1$coeff[1]*x[3] ((Fit1$coeff[3]/3)*x[2]^3+(Fit1$coeff[2]/2)*x[2]^2+Fit1$coeff[1]*x[2]) FitAreaC1[j]=(Fit1$coeff[3]/3)*x[4]^3+(Fit1$coeff[2]/2)*x[4]^2+Fit1$coeff[1]*x[4] ((Fit1$coeff[3]/3)*x[3]^3+(Fit1$coeff[2]/2)*x[3]^2+Fit1$coeff[1]*x[3]) TotalFitArea1[j]=FitAreaA1[j]+FitAreaB1[j]+FitAreaC1[j] TotalFitBaseArea1[j]=TotalFitArea1[j] y1[1]*(x[4] x[1]) } for (k in 1:n2){ err2=rnorm(4,0,error2) y2= a2*x^2+b2*x+c2+err2 TrapAreaA2[k]=((y2[2]+y2[1])/2)*(x[2] x[1]) TrapAreaB2[k]=((y2[3]+y2[2])/2)*(x[3] x[2]) TrapAreaC2[k]=((y2[4]+y2[3])/2)*(x[4] x[3]) TotalTrapArea2[k]=TrapAreaA2[k]+TrapAreaB2[k]+TrapAreaC2[k] TotalTrapBaseArea2[k]=TotalTrapArea2[k] y2[1]*(x[4] x[1]) Fit2=lm(y2~poly(x,2,raw=T)) FitAreaA2[k]=(Fit2$coeff[3]/3)*x[2]^3+(Fit2$coeff[2]/2)*x[2]^2+Fit2$coeff[1]*x[2] FitAreaB2[k]=(Fit2$coeff[3]/3)*x[3]^3+(Fit2$coeff[2]/2)*x[3]^2+Fit2$coeff[1]*x[3] ((Fit2$coeff[3]/3)*x[2]^3+(Fit2$coeff[2]/2)*x[2]^2+Fit2$coeff[1]*x[2]) FitAreaC2[k]=(Fit2$coeff[3]/3)*x[4]^3+(Fit2$coeff[2]/2)*x[4]^2+Fit2$coeff[1]*x[4] ((Fit2$coeff[3]/3)*x[3]^3+(Fit2$coeff[2]/2)*x[3]^2+Fit2$coeff[1]*x[3]) TotalFitArea2[k]=FitAreaA2[k]+FitAreaB2[k]+FitAreaC2[k] TotalFitBaseArea2[k]=TotalFitArea2[k] y2[1]*(x[4] x[1]) }

6 Ttest=t.test(TotalTrapArea1,TotalTrapArea2,alternativee="two.sided") TTAtest[i]=Ttest$p.value Btest=t.test(TotalTrapBaseArea1,TotalTrapBaseArea2,alternative="two.sided") TTBAtest[i]=Btest$p.value Ftest=t.test(TotalFitArea1,TotalFitArea2,alternative="two.sided") TFAtest[i]=Ftest$p.value FBtest=t.test(TotalFitBaseArea1,TotalFitBaseArea2,alternative="two.sided") TFBAtest[i]=FBtest$p.value } data.frame(ttat=ttatest,ttbat=ttbatest,tfat=tfatest,tfbat=tfbatest)} Simulation Outcomes: There were three main things that were of interest. 1. How the methods compared when everything was equal. 2. How the methods compared when the variance differed. 3. How the methods compared when the area is increased by a Standard Error or two. 1. All Equal: The results here show there was a statistical difference of the two samples approximately 5% of the time. This is results in a nice Type I Error. 2. Unequal Vairance: Below the error terms were set at two, five, and ten times higher than original error. 2 times

5 times 10 times The results show that the unequal variance has little effect on the t tests. There were slightly more p values below 0.05.

7 5 times 10 times The results show that the unequal variance has little effect on the t tests. There were slightly more p values below Something of interested is that there were more p values below 0.05 in the 5 times than in the 10 times. 3. Average Difference: In the appendix is the code for capstone2 that includes the calcuation of Standard Errors for each test. Here are the results of the average standard error for each test. Interestingly there is a higher standard error for the methods that subtract the base then the ones that don t. For the first two, I found two different coefficients that increased the area by one and two Standard Errors. The Trap, TrapBase, and Fit Base seem to correctly test the difference between the two samples. However, the Fit method only rejected 30% and 65% of the time. 1SE

8 2SE Another way to compare these was to increase only the c coefficient. This still increases the total area by a Standard Error. Interestingly the test picks up the difference in the Trap and Fit methods, but in the Base methods it doesn t. This is because we are subtracting off everything the area was just increased by. c c + 5

9 Conclusion: The results of this statistical simulation of methods to calculate area under the curve were mostly what one would expect to happen. The one curious thing, that perhaps needs more investigation, is when the data is pulled from two different models, why does the Fit method have far less significant p values than the other methods? The other tests yielded expected results. Further investigation can be done in altering the variables or comparing to another method of peak times. Appendix: I. Code for second function with Standard Error calculation. capstone2=function(a1=.1,b1=10,c1=100,n1=15, a2=.1,b2=10,c2=100, n2=15, error1=1,error2=1, iter=100){ x=c(0,30,60,90) TrapAreaA1=rep(0,n1) TrapAreaA2=rep(0,n2) TrapAreaB1=rep(0,n1) TrapAreaB2=rep(0,n2) TrapAreaC1=rep(0,n1) TrapAreaC2=rep(0,n2) TotalTrapArea1=rep(0,n1) TotalTrapArea2=rep(0,n2) TotalTrapBaseArea1=rep(0,n1) TotalTrapBaseArea2=rep(0,n2) Fit1=rep(0,n1) Fit2=rep(0,n2) FitAreaA1=rep(0,n1) FitAreaA2=rep(0,n2) FitAreaB1=rep(0,n1) FitAreaB2=rep(0,n2) FitAreaC1=rep(0,n1) FitAreaC2=rep(0,n2) SDTTA1=rep(0,iter) SDTTA2=rep(0,iter) SDTTBA1=rep(0,iter) SDTTBA2=rep(0,iter) SDTFA1=rep(0,iter) SDTFA2=rep(0,iter) SDTFBA1=rep(0,iter) SDTFBA2=rep(0,iter) Sp.TTA=rep(0,iter) Sp.TTBA=rep(0,iter) Sp.TFA=rep(0,iter)

10 Sp.TFBA=rep(0,iter) SE.TTA=rep(0,iter) SE.TTBA=rep(0,iter) SE.TFA=rep(0,iter) SE.TFBA=rep(0,iter) TotalFitArea1=rep(0,n1) TotalFitArea2=rep(0,n2) TotalFitBaseArea1=rep(0,n1) TotalFitBaseArea2=rep(0,n2) TTAtest=rep(0,iter) TTBAtest=rep(0,iter) TFAtest=rep(0,iter) TFBAtest=rep(0,iter) for (i in 1:iter){ for (j in 1:n1){ err1=rnorm(4,0,error1) y1= a1*x^2+b1*x+c1+err1 TrapAreaA1[j]=((y1[2]+y1[1])/2)*(x[2] x[1]) TrapAreaB1[j]=((y1[3]+y1[2])/2)*(x[3] x[2]) TrapAreaC1[j]=((y1[4]+y1[3])/2)*(x[4] x[3]) TotalTrapArea1[j]=TrapAreaA1[j]+TrapAreaB1[j]+TrapAreaC1[j] TotalTrapBaseArea1[j]=TotalTrapArea1[j] y1[1]*(x[4] x[1]) Fit1=lm(y1~poly(x,2,raw=T)) FitAreaA1[j]=(Fit1$coeff[3]/3)*x[2]^3+(Fit1$coeff[2]/2)*x[2]^2+Fit1$coeff[1]*x[2] FitAreaB1[j]=(Fit1$coeff[3]/3)*x[3]^3+(Fit1$coeff[2]/2)*x[3]^2+Fit1$coeff[1]*x[3] ((Fit1$coeff[3]/3)*x[2]^3+(Fit1$coeff[2]/2)*x[2]^2+Fit1$coeff[1]*x[2]) FitAreaC1[j]=(Fit1$coeff[3]/3)*x[4]^3+(Fit1$coeff[2]/2)*x[4]^2+Fit1$coeff[1]*x[4] ((Fit1$coeff[3]/3)*x[3]^3+(Fit1$coeff[2]/2)*x[3]^2+Fit1$coeff[1]*x[3]) TotalFitArea1[j]=FitAreaA1[j]+FitAreaB1[j]+FitAreaC1[j] TotalFitBaseArea1[j]=TotalFitArea1[j] y1[1]*(x[4] x[1]) } for (k in 1:n2){ err2=rnorm(4,0,error2) y2= a2*x^2+b2*x+c2+err2 TrapAreaA2[k]=((y2[2]+y2[1])/2)*(x[2] x[1]) TrapAreaB2[k]=((y2[3]+y2[2])/2)*(x[3] x[2]) TrapAreaC2[k]=((y2[4]+y2[3])/2)*(x[4] x[3]) TotalTrapArea2[k]=TrapAreaA2[k]+TrapAreaB2[k]+TrapAreaC2[k] TotalTrapBaseArea2[k]=TotalTrapArea2[k] y2[1]*(x[4] x[1]) Fit2=lm(y2~poly(x,2,raw=T)) FitAreaA2[k]=(Fit2$coeff[3]/3)*x[2]^3+(Fit2$coeff[2]/2)*x[2]^2+Fit2$coeff[1]*x[2]

11 FitAreaB2[k]=(Fit2$coeff[3]/3)*x[3]^3+(Fit2$coeff[2]/2)*x[3]^2+Fit2$coeff[1]*x[3] ((Fit2$coeff[3]/3)*x[2]^3+(Fit2$coeff[2]/2)*x[2]^2+Fit2$coeff[1]*x[2]) FitAreaC2[k]=(Fit2$coeff[3]/3)*x[4]^3+(Fit2$coeff[2]/2)*x[4]^2+Fit2$coeff[1]*x[4] ((Fit2$coeff[3]/3)*x[3]^3+(Fit2$coeff[2]/2)*x[3]^2+Fit2$coeff[1]*x[3]) TotalFitArea2[k]=FitAreaA2[k]+FitAreaB2[k]+FitAreaC2[k] TotalFitBaseArea2[k]=TotalFitArea2[k] y2[1]*(x[4] x[1]) } SDTTA1[i]=sd(TotalTrapArea1) SDTTA2[i]=sd(TotalTrapArea2) SDTTBA1[i]=sd(TotalTrapBaseArea1) SDTTBA2[i]=sd(TotalTrapBaseArea2) SDTFA1[i]=sd(TotalFitArea1) SDTFA2[i]=sd(TotalFitArea2) SDTFBA1[i]=sd(TotalFitBaseArea1) SDTFBA2[i]=sd(TotalFitBaseArea2) Sp.TTA[i]=((n1 1)*SDTTA1[i]+(n2 1)*SDTTA2[i])/(n1+n2 2) Sp.TTBA[i]=((n1 1)*SDTTBA1[i]+(n2 1)*SDTTBA2[i])/(n1+n2 2) Sp.TFA[i]=((n1 1)*SDTFA1[i]+(n2 1)*SDTFA2[i])/(n1+n2 2) Sp.TFBA[i]=((n1 1)*SDTFBA1[i]+(n2 1)*SDTFBA2[i])/(n1+n2 2) SE.TTA[i]=sqrt(Sp.TTA[i]^2/n1+Sp.TTA[i]/n2) SE.TTBA[i]=sqrt(Sp.TTBA[i]^2/n1+Sp.TTBA[i]/n2) SE.TFA[i]=sqrt(Sp.TFA[i]^2/n1+Sp.TFA[i]/n2) SE.TFBA[i]=sqrt(Sp.TFBA[i]^2/n1+Sp.TFBA[i]/n2) Ttest=t.test(TotalTrapArea1,TotalTrapArea2,alternativee="two.sided") TTAtest[i]=Ttest$p.value Btest=t.test(TotalTrapBaseArea1,TotalTrapBaseArea2,alternative="two.sided") TTBAtest[i]=Btest$p.value Ftest=t.test(TotalFitArea1,TotalFitArea2,alternative="two.sided") TFAtest[i]=Ftest$p.value FBtest=t.test(TotalFitBaseArea1,TotalFitBaseArea2,alternative="two.sided") TFBAtest[i]=FBtest$p.value } data.frame(ttat=ttatest,se.tta=se.tta,ttbat=ttbatest,se.ttba=se.ttba,tfat=tfatest,se.tfa=se.tf A,TFBAt=TFBAtest,SE.TFBA=SE.TFBA) } II. Calculating the proportion of p values below 0.05 for each method. Trap=cap$TTAt sum(ifelse(trap<= 0.05, 1, 0))/2000 TrapB=cap$TTBAt sum(ifelse(trapb<= 0.05, 1, 0))/2000 Fit=cap$TFAt sum(ifelse(fit<= 0.05, 1, 0))/2000

12 FitB=cap$TFBAt sum(ifelse(fitb<= 0.05, 1, 0))/2000 III. Calculating the mean of each Standard Error mean(cap2$se.tta) mean(cap2$se.ttba) mean(cap2$se.tfa) mean(cap2$se.tfba)

Inferential Statistics

Inferential Statistics and t - tests ScWk 242 Session 9 Slides Inferential Statistics Ø Inferential statistics are used to test hypotheses about the relationship between the independent and the dependent