ONLINE TECHNICAL APPENDIX TO: UK ALCOHOL INDUSTRY S BILLION UNITS PLEDGE : INTERIM EVALUATION FLAWED

Size: px

Start display at page:

Download "ONLINE TECHNICAL APPENDIX TO: UK ALCOHOL INDUSTRY S BILLION UNITS PLEDGE : INTERIM EVALUATION FLAWED"

Charity Baker
5 years ago
Views:

1 ONLINE TECHNICAL APPENDIX TO: UK ALCOHOL INDUSTRY S BILLION UNITS PLEDGE : INTERIM EVALUATION FLAWED This appendix provides additional technical critiques of the UK Department of Health s second interim report evaluating the alcohol industry s pledge to remove one billion units from the alcohol market. It covers three points: 1. A discontinuity in the HMRC beer data used in the report 2. The sensitivity of the method to discrepancies between HMRC and market research data 3. Counter-intuitive findings for different consumer response scenarios. 1. Discontinuity in HMRC beer data The main finding of the DoH report, that 1.3 billion units have been removed from the market, is primarily due to a reduction in the average %ABV of beer (see Table 2, main paper). The DoH report notes that a revision to HMRC s beer data up to and including 2011 decreased the volume of beer sold in 2011 by 5%, but did not substantially affect the estimated number of units of alcohol in beer sold and thus increased the estimated %ABV of that beer from 4.21% to 4.42%. As 2011 is the baseline for the analysis and the %ABV of beverages is the key metric by which success under the pledge is measured, this change has a large impact on the evaluation of the pledge. Using the unrevised data available from HMRC, 1 the estimated units removed from the market using DoH s method falls from 1.3 billion to 470 million. As can be seen in Figure A1, the revision to the HMRC data created a step-change in the average %ABV of beer released for sale in the UK between August and September Figure A1 also shows that the step-change in %ABV of beer appears to only affect the beer released for sale and did not affect beer produced in the UK. We are unaware of any reason why this one-month step-change should occur or why it should affect beer released for sale but not beer production. The unrevised data (also shown) suggest the production and sales data previously tracked each other closely. Taken at face value, it appears a substantial change occurred in a single month in the average strength of either imported or exported beer. Although we cannot rule this possibility out, we are unaware of any supporting evidence that this occurred. The %ABV of several leading products was changed as part of the Billion Unit pledge; however, these changes were staggered over time and many of the affected beers are produced in the UK. We asked HMRC if they could explain the anomaly; however their response was unclear and indicated administrative processes had been changed to accommodate the new high and low strength beer tax bands introduced in October Requests for further clarification were not answered. Although introducing new beer tax bands likely affected the market over time, it remains unclear why such a large change in the %ABV of beer sold would occur in a single month and why a commensurate change did not occur in the %ABV of beer produced. In the absence of a credible alternative, our view of the most likely explanation is that HMRC administrative changes introduced a methodological discontinuity into the data series. Irrespective of its source, this discontinuity occurs during the evaluation period and should be controlled for in the DoH analysis. The failure by DoH to do so reduces confidence in the evaluation, particularly given that the claimed success of the pledge hinges entirely on this point.

2 Figure A1: Average %ABV of beer

3 2. Sensitivity of the method to discrepancies between HMRC and market research data The results of DoH s analysis vary hugely when small changes are made to the input data. For example, increasing the estimated number of units of beer sold in 2011 by 1% increases the estimated number of units removed from the market by 187 million (or 14% of the overall impact of the pledge). This means precise data are required to produce reliable estimates of policy impact. However, substantial measurement error which is unstable over time is clearly present in the data used as HMRC and Nielsen/CGA (who provide the supplementary market research data) differ in their estimate of total volume of alcoholic drinks sold by 9% in 2011 and 14% in The DoH analysis uses the average %ABV for cider, wine, spirits and RTDs from Nielsen/CGA data (see Table 1, main paper). The degree to which these average %ABVs are inaccurately estimated and to which this changes over time has not been quantified by DoH or others. The impact will only be negligible if the difference in %ABV between products included and excluded from the market research data does not change over time. This is unlikely to be the case as a key reason for the discrepancy is that Nielsen s data do not include budget retailers. These retailers have increased their share of the alcohol market over time and, given their focus on low prices, it is unlikely that the average %ABV of products sold by these retailers is the same as that sold by other retailers. Therefore, to test the sensitivity of DoH s results to the increasing inaccuracy of Nielsen/CGA data, we reduced the estimated 2011 %ABV of all products by 0.1 percentage points except beer (as all beer data come from HMRC). Under this sensitivity analysis, the estimated number of units removed from the market fell by approximately 270 million or 20% of the claimed effect of the pledge. a Therefore, the results appear highly sensitive to modest inaccuracies in the %ABVs taken from Nielsen/CGA data. In the absence of a robust estimate of the change over time in accuracy of these data, caution is required in drawing conclusions from the DoH analysis. 3. Counter-intuitive findings for different consumer response scenarios In the main paper, we discuss how DoH assume only one possible consumer response to the Billion Unit pledge: that consumers drink the same volume of each beverage but at a lower % ABV. This analytical approach ignores a wide range of alternative consumer responses which may affect either the volume of beverage sold or the %ABV of beverages sold. Below we demonstrate how this may lead to counter-intuitive conclusions being reached. DoH s main assumption is actually comprised of a number of lower-level assumptions. We summarise four of these below: 1. Using %ABV changes as the key indicator of policy impact assumes all changes in average %ABVs for each beverage type are wholly caused by pledge activities; 2. Holding volumes of beverage sold at 2011 levels assumes changes in these volumes are unrelated to pledge activities; 3. Estimating impacts of the pledge using changes in beverage-specific %ABVs assumes a reduction in the number of units in the market will not lead to %ABVs rising within beverage categories; aa Exact figure not provided as rounding of numbers in the DoH report prevents perfect replication of their analysis.

4 4. Estimating units removed from the market based on %ABV falls assumes all decreases in average %ABVs are the result of a reduction in units consumed whereas units consumed may increase when %ABVs fall. Below and in Table A1 to A3 we use three hypothetical examples to illustrate why these assumptions are incorrect: Scenario 1 (illustrating assumption 1): A 20% reduction in units of high strength beer sold occurs and the equivalent number of units are now sold as spirits. They are distributed across types of spirits in a way that leaves the average %ABV of the category unchanged (i.e. they are proportionately distributed across spirits of different strengths). The total number of units sold across all beverages remains the same, but average %ABV for beer falls. DoH would estimate that 266 million units have been removed from the market despite consumption remaining the same and no pledge-related actions occurring. Scenario 2 (illustrating assumptions 2 and 3): The ABV of a beer is reduced under the pledge but it remains above average strength. Some white wine drinkers switch to this beer, reducing the total volume of wine sold by 0.6m hectolitres and increasing the volume of beer sold by the same amount. As the %ABV of white wine is less than the average for wine, the average wine %ABV would increase. The average %ABV of beer would also increase as more beer with above average %ABV is being sold. Despite consumers switching to a reduced strength product and despite this switch being from wine to beer which means fewer units of alcohol are consumed, DoH would estimate that 163 million extra units are being sold as the average %ABV of both beverage types increased. Scenario 3 (illustrating assumption 4): Some abstainers begin to drink low-strength beer following an effective marketing campaign, doubling the volume sold. More alcohol would be consumed overall but the average ABV of beer would decrease and it would be estimated that 119 million units have been removed from the market. Each of the above scenarios focuses on only one consumer response. In reality, a wide range of consumer responses are likely to occur and the DoH approach does not separate those which it is and is not designed to measure. Estimates of the effect of pledge-related activities which are clearly incorrect will be mixed with plausible estimates to generate an overall estimate with little validity.

5 Table A1: Numerical data for Scenario 1 High strength beer All beer Spirits Total 2011 Volume of product (hectolitres) 5.0m 42.5m 2.9m 45.5m Average ABV 7.00% 4.40% 36.90% - Volume of pure alcohol (units sold) 3.5bn 18.7bn 10.9bn 29.6bn 2013 Volume of product (hectolitres) 4.0m 41.5m 3.1m 44.7m Average ABV 7.00% 4.34% 36.90% - Volume of pure alcohol (units sold) 2.8bn 18.0bn 11.6bn 29.6bn Change in units sold Actual -700m -700m 700m 0 Department of Health method 0-266m 0-266m Table A2: Numerical data for Scenario 2 White wine All wine All beer Total 2011 Volume of product (hectolitres) 4.0m 13.3m 42.5m 55.8m Average ABV 11.00% 12.92% 4.40% - Volume of pure alcohol (units sold) 4.4bn 17.2bn 18.7bn 35.9bn 2013 Volume of product (hectolitres) 3.4m 12.7m 43.1m 55.8m Average ABV 11.00% 13.01% 4.41% - Volume of pure alcohol (units sold) 3.7bn 16.5bn 19.0bn 35.6bn Change in units sold Actual -660m -660m +307m -353m Department of Health method m +43m +163m Table A3: Numerical data for Scenario 3 Low strength beer All beer Other beverages 2011 Volume of product (hectolitres) 0.5m 42.5m 27.0m 69.6m Average ABV 2.00% 4.40% 12.38% - Volume of pure alcohol (units sold) 0.1bn 18.7bn 33.5bn 52.2bn 2013 Volume of product (hectolitres) 1.0m 43.0m 27.0m 70.1m Average ABV 2.00% 4.37% 12.38% - Volume of pure alcohol (units sold) 0.2bn 18.8bn 33.5bn 52.3bn Change in units sold Actual +100m +100m m Department of Health method 0-119m 0-119m Total

6 Reference List (1) HM Revenue and Customs. Alcohol duty archive ame=alcohol%20duty%20archive, accessed 2 Feb (2) Department of Health. Responsibility Deal: Monitoring the number of units of alcohol sold: Second interim report, 2013 data ond_interim_report.pdf, accessed 2 Feb. 2015

This is a repository copy of UK alcohol industry's "billion units pledge": interim evaluation flawed.

This is a repository copy of UK alcohol industry's "billion units pledge": interim evaluation flawed. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/87742/ Version: Accepted