Ensemble size: How suboptimal is less than infinity?

Transcription

1 Ensemble size: How suboptimal is less than infinity? Martin Leutbecher 13 September 2017 Acknowledgements: Zied Ben Bouallègue, Chris Ferro, Sarah-Jane Lock, David Richardson M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

2 Ensemble size at ECMWF M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

3 Ensemble size at ECMWF M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

4 Ensemble size at ECMWF 50 member since Dec 1996 Why 50? M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

5 Ensemble size at ECMWF 50 member since Dec 1996 Why 50? Are the benefits of more than 50 members marginal? M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

6 Talagrand, Vautard and Strauss (1997) (adapted from their Fig. 4.5) M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

7 Talagrand, Vautard and Strauss (1997) According to Talagrand et al (1997) not more than 30 members are needed. (adapted from their Fig. 4.5) M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

8 Buizza and Palmer (1998) Comparison of 2, 4, 8, 16, 32 members Verified against analysis Verified against member (adapted from their Fig. 11) Z500 in NH; T63L19 model, initial uncertainty represented with singular vectors, no representation of model uncertainty Careful conclusions that do not rule out increases in skill beyond 32 members. M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

9 Buizza et al. (1998) Impact of resolution and ensemble size In December 1996, resolution was increased from T63 (quadratic grid) to TL159 (linear grid) and ensemble size was increased from 32 to 50 members. M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

10 Miyoshi et al (2014) identical twin EnKF using SPEEDY model Horizontal correlations of mid-tropospheric specific humidity with yellow star location (from their Fig. 3) M. Leutbecher (from their Fig. 4) Ensemble size: How suboptimal is less than infinity? 13 September

11 Chua s circuit Machete and Smith (2016) ensemble forecasts of electronic circuit (from Wikipedia) (from their Fig. 12; colour corresponds to lead time; competitive advantage is similar to a probabilistic skill score) M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

12 Guess the ensemble size i.i.d. members; pdfs for 20 realisations; ensemble size fixed in each panel M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

13 Guess the ensemble size i.i.d. members; pdfs for 20 realisations; ensemble size fixed in each panel k 4k 16k M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

14 Ensemble size at ECMWF M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

15 Experiments with IFS ensembles IFS cycle 41r2 as operational ensemble but lower resolution: TCo members June-July-August 2016 (92 cases) probabilistic skill evaluated with continuous ranked probability score (CRPS): mean squared error of cumulative distribution M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

16 CRPS Impact of ensemble size on CRPS M=8 M=20 M=50 M=100 M= hpa zonal wind in northern extratropics M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

17 CRPS Impact of ensemble size on CRPS Predictions of CRPS for infinite ensemble size M=8 M=20 M=50 M=100 M= hpa zonal wind in northern extratropics M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

18 Where does increased skill come from? Sampling uncertainty of Z500 ensemble mean at D7 8 member 50 member M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

19 CRPS and ensemble size: What to expect? Kernel representation of CRPS kernel representation of CRPS CRPS(x j, y) = 1 M x j y 1 M 2M 2 j=1 M j=1 k=1 M x j x k With exchangeability of members, the expected CRPS is E x CRPS(x j, y) = E x x y M 1 2M E x,x x x For an infinite size ensemble we get E x CRPS(x j, y) = E x x y 1 2 E x,x x x M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

20 How can CRPS for infinite ensemble size be predicted with a finite ensemble? The fair CRPS is a modified version of the CRPS that removes the bias in the score due to the finite ensemble size (see Chris Ferro s talk) From the kernel representation, one can see easily that the CRPS for infinite ensemble size is obtained by the estimator CRPS (x j, y) = CRPS(x j, y) 1 2M 2 (M 1) The correction term is a measure of ensemble spread. M j=1 k=1 M x j x k M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

21 Analytic result for statistically consistent ensembles When members are statistically consistent (iid) draws from same distribution as observation (perfectly reliable ensemble), the CRPS for an m-member ensemble satisfies ( CRPS M = 1 M 1 ) E x x ( = ) CRPS 2M M Eqns. (8) and (9) in Richardson (2001) show that the Brier score also satisfies BS M = (1 + M 1 ) BS. M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

22 Analytic result for statistically consistent ensembles When members are statistically consistent (iid) draws from same distribution as observation (perfectly reliable ensemble), the CRPS for an m-member ensemble satisfies ( CRPS M = 1 M 1 ) E x x ( = ) CRPS 2M M Eqns. (8) and (9) in Richardson (2001) show that the Brier score also satisfies BS M = (1 + M 1 ) BS. Extreme events? Relationship for BS implies that for any weighting in the twcrps (Gneiting and Ranjan, 2011) we also have ( twcrps M = ) twcrps M M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

23 Actual convergence with ensemble size from right to left 2, 4, 8, 20, 50, 100 and 200 members 1.5 CRPS(M)/fairCRPS(M) /M Data from 200 member TCo399 IFS experiment, JJA data points for each ensemble size 15 lead times 4 variables (z500, T850, u850, u200) 2 regions (NH and SH extratropics) 50 and 200 members are 2% and 0.5% worse than, respectively M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

24 from right to left 2, 4, 8, 20, 50, 100 and 200 members Actual convergence with ensemble size zoom 20, 50, 100 and 200 members CRPS(M)/fairCRPS(M) CRPS(M)/fairCRPS(M) /M /M Data from 200 member TCo399 IFS experiment, JJA data points for each ensemble size 15 lead times 4 variables (z500, T850, u850, u200) 2 regions (NH and SH extratropics) 50 and 200 members are 2% and 0.5% worse than, respectively M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

25 Quantile score and CRPS QS α α = 0.5 α = 0.1 QS α (q, y) = 2 (I {y < q} α) (q y) y α = 0.9 q with indicator function I(true) = 1 and I(false) = 0, quantile q and observation y; α (0, 1) denotes the probability level CRPS(F, y) = 1 0 QS α ( F 1 (α), y ) dα where the quantile q for cumulative distribution F is F 1 (α) M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

26 QS M / QS Quantile score for a standard Gaussian Simulations with M = 20 to 1000 members quantile probability level For QS of q.98, 50 and 200 members are 7% and 2% worse than, respectively M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

27 Ensemble size at ECMWF M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

28 Research and development What is a good ensemble size? Large ensemble size can delay progress in R&D It would be most efficient to use the smallest ensemble size that is sufficient to estimate impact for operational ensemble size Using proper scores with small ensembles can mislead though M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

29 0 Ensemble configurations R, O and N CRPS for 850 hpa temperature in northern extratropics 50 member 4 member CRPS CRPS N R, M= R, M= O, M=50 N, M= O, M=4 N, M=4 M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

30 0 Ensemble configurations R, O and N CRPS for 850 hpa temperature in northern extratropics 50 member 4 member CRPS CRPS N R, M= R, M= O, M=50 N, M= O, M=4 N, M= R, M= R, M=4 fair CRPS O, M= O, M= N, M= N, M=4 M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

31 Ensemble configurations R, O and N CRPS for 850 hpa zonal wind in tropics 50 member 4 member R, M=50 O, M=50 N, M= CRPS CRPS N R, M=4 O, M=4 N, M=4 M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

32 Ensemble configurations R, O and N CRPS for 850 hpa zonal wind in tropics 50 member 4 member 0 R, M=50 O, M=50 N, M= CRPS CRPS N R, M=4 O, M=4 N, M=4-0.1 R, M= R, M=4 fair CRPS -0.2 O, M=50 O, M=4 N, M=50 N, M=4 M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

33 0 Ensemble configurations R, O and N CRPS for 850 hpa zonal wind in tropics 50 member 2 member (1,3) CRPS CRPS N R, M= R, M=2-0.2 O, M= O, M=2 0 N, M= N, M=2-0.1 R, M= R, M=2 fair CRPS -0.2 O, M=50 O, M=2 N, M=50 N, M=2 M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

34 0 Ensemble configurations R, O and N CRPS for 850 hpa zonal wind in tropics 50 member 2 member (1,2) CRPS CRPS N R, M= R, M=2-0.2 O, M= O, M=2 0 N, M= N, M=2-0.1 R, M= R, M=2 fair CRPS -0.2 O, M=50 O, M=2 N, M= N, M=2 M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

35 Small ensemble sizes Can be used for R&D if evaluation uses fair scores Can be used in reforecasts for estimating skill Applicability of fair scores is linked to ensemble generation Current ensemble generation at ECMWF not fully consistent with exchangeability required for fair scores Benefits for R&D faster turnaround time more configurations can be explored scope for increasing statistical significance by using less members but more start dates M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

36 Conclusions How suboptimal is less than infinity? Three possible answers: M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

37 Conclusions How suboptimal is less than infinity? Three possible answers: A bit or maybe a lot, tell me the score and your ensemble size... M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

38 Conclusions How suboptimal is less than infinity? Three possible answers: A bit or maybe a lot, tell me the score and your ensemble size... Operational ensemble forecasts: 50 members are too few let s increase the ensemble size to... M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

39 Conclusions How suboptimal is less than infinity? Three possible answers: A bit or maybe a lot, tell me the score and your ensemble size... Operational ensemble forecasts: 50 members are too few let s increase the ensemble size to... Research & Development: Small ensembles are highly efficient. Two to four members may be enough for standard evaluations (provided exchangeability in the ensemble generation and use of fair scores) M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September

40 Discussion How can we increase ensemble size when we need to increase resolution too? Different users will have different needs, how to obtain a good compromise for all of them? How to increase ensemble size in a computationally efficient way for all forecast ranges from medium-range to extended-range? What is an adequate ensemble size for the reforecasts? Which other proper scores permit the construction of an associated fair score? M. Leutbecher Ensemble size: How suboptimal is less than infinity? 13 September