Comparison of cross-validation and bagging for building a seasonal runoff forecast model

Transcription

1 Comparison of cross-validation and bagging for building a seasonal runoff forecast Simon Schick, Ole Rössler, Rolf Weingartner University of Bern, Switzerland Institute of Geography Oeschger Centre for Climate Change Research 1/12

2 Why resampling? training selection testing (Hastie et al., 2009) 2/12

3 Why resampling? training selection testing (Hastie et al., 2009) small sample sizes 2/12

4 Why resampling? training selection testing (Hastie et al., 2009) small sample sizes weak relationships and large pool of candidate predictors 2/12

5 Why resampling? training selection testing (Hastie et al., 2009) small sample sizes weak relationships and large pool of candidate predictors Can we benefit from the s out of resampling? 2/12

6 Comparison of 2 approaches best guess (BGS) use all data points testing by leave-one-out cross-validation (LOO) bagged (BAG) bagging: bootstrap aggregating (Breiman, 1996) testing by out of bag predictions (OOB) 3/12

7 Regression Y = β 0 + β 1 x Q + β 2 x P + β 3 x T + ε Y i,j mean runoff for i = 30,60,90 days, starting with the 1 st and 16 th of every month (j = 1,...,24) centered and scaled 4/12

8 Regression Y = β 0 + β 1 x Q + β 2 x P + β 3 x T + ε Y i,j mean runoff for i = 30,60,90 days, starting with the 1 st and 16 th of every month (j = 1,...,24) centered and scaled x initial conditions, parametrized by runoff Q, precipitation P, and temperature T univariate screening for 10, 20,..., 720 days 4/12

9 Regression Y = β 0 + β 1 x Q + β 2 x P + β 3 x T + ε Y i,j mean runoff for i = 30,60,90 days, starting with the 1 st and 16 th of every month (j = 1,...,24) centered and scaled x initial conditions, parametrized by runoff Q, precipitation P, and temperature T univariate screening for 10, 20,..., 720 days β estimation by partial least squares at least the first PLS direction is selected ε residuals 4/12

10 (32 years) 66 catchments (no nesting) P and T out of E-OBS gridded data set (Haylock et al., 2008) 5/12

11 Hindcast experiment training selection testing (Hastie et al., 2009) 6/12

12 Hindcast experiment training selection testing leave one out predictor screening cross validation partial least squares best guess (BGS) testing: LOO 6/12

13 Hindcast experiment training selection testing leave one out predictor screening cross validation partial least squares best guess (BGS) testing: LOO bootstrap predictor screening cross validation partial least squares bagged (BAG) testing: OOB 6/12

14 Hindcast experiment training selection testing leave one out predictor screening cross validation partial least squares best guess (BGS) testing: LOO bootstrap predictor screening cross validation partial least squares bagged (BAG) testing: OOB leave one out empirical mean seasonal regime (SRG) testing: LOO 6/12

15 Hindcast experiment training selection testing leave one out predictor screening cross validation partial least squares best guess (BGS) testing: LOO bootstrap predictor screening cross validation partial least squares bagged (BAG) testing: OOB leave one out empirical mean seasonal regime (SRG) testing: LOO outer cross validation (8 folds) 6/12

16 Mean squared error of prediction (n=66) 7/12

17 Mean squared error of prediction H0: The less complex performs equal or even better than the more complex. H1: The more complex performs better. Table: p-values of right-sided t-test using paired differences of Ê MSP (only outer cross-validation) Y 30 Y 60 Y 90 SRG - BGS 0.21 >0.99 >0.99 SRG - BAG < BGS - BAG <0.01 <0.01 <0.01 8/12

18 Mean squared error of prediction How much reduces bagging Ê MSP? Table: Ê MSP and reduction (only outer cross-validation) Y 30 Y 60 Y 90 BGS BAG reduction % /12

19 LOO and OOB provide on average accurate estimates of prediction error. 10/12

20 LOO and OOB provide on average accurate estimates of prediction error. SRG outperforms BGS and BAG in many catchments. 10/12

21 LOO and OOB provide on average accurate estimates of prediction error. SRG outperforms BGS and BAG in many catchments. Most likely, BAG outperforms BGS on average. 10/12

22 LOO and OOB provide on average accurate estimates of prediction error. SRG outperforms BGS and BAG in many catchments. Most likely, BAG outperforms BGS on average. Error reduction is strongest, when BGS has low or no skill. 10/12

23 LOO and OOB provide on average accurate estimates of prediction error. SRG outperforms BGS and BAG in many catchments. Most likely, BAG outperforms BGS on average. Error reduction is strongest, when BGS has low or no skill. Suboptimal: Comparison rests on resampling. 10/12

24 runoff series and catchment boundaries: Landesanstalt für Umwelt, Messungen und Naturschutz Baden-Württemberg; Bayerisches Landesamt für Umwelt; Land Vorarlberg (data.vorarlberg.gv.at); Bundesministerium für Land- und Forstwirtschaft, Umwelt und Wasserwirtschaft Österreich; Schweizerisches Bundesamt für Umwelt precipitation and temperature series: E-OBS data set (EU-FP6 project ENSEMBLE, ensembles-eu.metoffice.com, and ECA&D, project ecad.eu) digital elevation : EU-DEM, produced using Copernicus data and information funded by the European Union Bernhard Wehren made available additional runoff data for the river Kander at Hondrich 11/12

25 Breiman, L.: Bagging Predictors. Machine Learning 24.2, , Garen, D. C.: Improved techniques in regression-based streamflow volume forecasting. Journal of Water Resources Planning and Management, 118.6, , Hastie, T., Tibshirani, R., and Friedman, J.: The Elements of Statistical Learning. Mining, Inference, and Prediction. Second Edition. Springer New York Inc., Haylock, M. R., Hofstra, N., Klein Tank, A. M. G., Klok, E. J., Jones, P. D., and New, M.: A European daily high-resolution gridded data set of surface temperature and precipitation for Journal of Geophysical Research: Atmospheres, 113, D20119, Mevik, B.-H., and Wehrens, R.: The pls Package: Principal Component and Partial Least Squares Regression in R. Journal of Statistical Software, 18, 2, experimental forecasts: 12/12

26 Ê MSP : mean squared error of prediction H0: The less complex performs equal or even better than the more complex H1: The more complex performs better Table: p-values of right-sided t-test (nonparametric bootstrap) using paired differences of Ê MSP (only outer cross-validation) Y 30 Y 60 Y 90 SRG - BGS 0.21 (0.21) >0.99 (>0.99) >0.99 (>0.99) SRG - BAG <0.01 (<0.01) 0.03 (0.02) 0.67 (0.69) BGS - BAG <0.01 (<0.01) <0.01 (<0.01) <0.01 (<0.01) 12/12

27 Ê MAP : mean absolute error of prediction (n=66) 12/12

28 Ê MAP : mean absolute error of prediction H0: The less complex performs equal or even better than the more complex H1: The more complex performs better Table: p-values of right-sided t-test (nonparametric bootstrap) using paired differences of Ê MAP (only outer cross-validation) Y 30 Y 60 Y 90 SRG - BGS 0.02 (0.01) 0.92 (0.92) >0.99 (>0.99) SRG - BAG <0.01 (<0.01) <0.01 (<0.01) 0.25 (0.25) BGS - BAG <0.01 (<0.01) <0.01 (<0.01) <0.01 (<0.01) 12/12

29 Nash-Sutcliffe Efficiency (n=66; six outliers in [-3.7,-1.8] are not shown for readability) 12/12

30 12/12

31 12/12

32 12/12

33 12/12

34 alpine: Landwasser, Davos (Y 60 ) 12/12

35 lake: Aabach, Hitzkirch (Y 60 ) 12/12

36 lowland: Töss, Neftenbach (Y 60 ) 12/12

37 regulated: Julia, Tiefencastel (Y 60 ) 12/12