Department for Transport SoNA 2014 Peer Review Final Report Prepared by Dr Hannah Devine-Wright Placewise Ltd & Stephen Turner, MA, MSc, HonFIOA Stephen Turner Acoustics Limited Department for Transport Contract: SO 16241-2 Report Number: PW/STA/03 Date: 26 January 2017 Page 1 of 13
CONTENTS Section Page 1.0 Introduction 3 2.0 Overview of the review process 4 3.0 Specialist review 5 4.0 Aims of SoNA 2014 10 5.0 Conclusions 13 Page 2 of 13
1.0 INTRODUCTION 1.1 Placewise Limited and Stephen Turner Acoustics Limited (STA) were separately appointed to undertake an expert peer review of the study entitled Survey of Noise Attitudes 2014 (SoNA 2014) carried out by the Department for Transport. Dr Hannah Devine-Wright of Placewise Limited was tasked with focusing particularly on reviewing the social survey and analytical aspects of the survey;; Stephen Turner of STA was asked to review, in particular, the acoustics aspects. Despite the different emphases of their scope, both reviewers contributed comments and observations on all aspects of the survey report and it is their combined views that are presented in this document. 1.2 It is understood that the main objective of SoNA 2014 was to allow the Department for Transport and other government departments to understand people s attitudes to noise from various sources and specifically, for this study, the impact of noise from fixed wing civil aircraft. Furthermore, the study investigated how attitudes to aircraft noise related to the aircraft noise exposure experienced by the respondents. 1.3 This peer review report has five sections. Section 2 provides an overview of the review process. Section 3 of this report provides a response and commentary on various aspects of the study, as required by the peer review scope. 1.4 Section 4 examines the extent to which the results of the survey as presented in the final study report met the five aims of the project. Comment is also provided on the robustness of the conclusions drawn. Brief conclusions are given in Section 5. Page 3 of 13
2.0 OVERVIEW OF THE REVIEW PROCESS 2.1 The peer review inception meeting took place on 29 th February 2016. At that meeting the peer reviewers were informed that the first draft analysis report, prepared by the Civil Aviation Authority (CAA) would be provided the following day. 2.2 As requested, the peer reviewers undertook an evaluation of that report and provided feedback in time for a project board meeting held on the 14 th March 2016. 2.3 That feedback highlighted a number of issues regarding the presentation of the information and the analysis that had been carried out. The reviewers concluded that it was not clear from this first draft the extent to which the study had met the declared aims. 2.4 No fundamental issues were identified in terms of survey methodology, but it was felt that there would be merit in considering again the narrative structure of the report and in undertaking further analysis of certain elements. To that end, the reviewers provided a suggested report outline and indicated the further analysis that they thought would be helpful. 2.5 It was agreed that an iterative process would be adopted so that the reviewers would receive the various report chapters as they were revised and, in turn, the reviewers would provide detailed written and verbal feedback to the report authors. This occurred throughout the summer of 2016. 2.6 Overall co-ordination of the review process was managed by CAA, who also arranged teleconference calls and face-to-face meetings with the peer reviewers and members of the project board in September and November 2016. 2.7 Co-ordination between the acoustics and social peer reviewers was achieved through regular telephone conversations and/or email exchanges. 2.8 A preliminary final report was received by the peer reviewers on 7 December 2016. A Statement of Assurance (Report PW/STA/02, dated 14 December 2016) was compiled based on the text of that report. A final version of the report was received on 19 January 2017 and forms the basis for this peer review. Page 4 of 13
3.0 SPECIALIST REVIEW 3.1 According to the peer review scope, the acoustician reviewer was required to comment, in particular, on the following: 1. Whether the methodology used for the noise modelling, and the construct of the various metrics was consistent with current standards/best practice. 2. Whether the use of noise data in the statistical analysis was robust. 3. Comment on the conclusions drawn on the appropriateness of the LAeq metric to quantify health impacts. 3.2 The social science reviewer was required to comment on the following: 4. The robustness of the sampling approach. 5. The quality of the survey method. 6. The appropriateness of the interviewing procedure. 7: The robustness of data analysis and the interpretation of data. Outcome of acoustician review Noise modelling methodology and the determination of the various metrics 3.3 Standard noise modelling methodology was used to determine the exposure at the respondents locations. The principle of using the 2013 exposure to determine the sampling strategy was appropriate, as was the use of the 2014 summer (or annual) exposure to determine the values to associate with the survey responses. 3.4 The inevitable, but generally small, year on year variation in noise exposure did result in 79 respondents experiencing a noise exposure below the previously pre-determined cut-off of 51 db, L Aeq,16h. (Paragraph 5.6) It was reasonable to keep these samples in the survey. 3.5 It is noted that data availability meant that the noise exposure for 2012 had to be used for East Midlands, London City and Newcastle airports when determining the sampling strategy (Paragraph 3.16). It is also noted that, for Luton Airport, 2014 data were not available and that 2013 data had to be used to associate with the survey responses. (Paragraph 3.23). No other information was available from which it might have been possible to evaluate the extent of any differences in noise exposure between those two years. It is noted, however, that there were only 21 respondents in the study associated with Luton. Consequently, it is concluded that having to use 2013 data for the Page 5 of 13
respondents associated with Luton Airport would have little bearing on the overall conclusions drawn from the study. 3.6 Given the increasing use of the L den indicator as a result of the Environmental Noise Directive and the interest in the use of the N70 indicator, it was appropriate to examine how well these indicators were associated with the survey responses. Given that nearly 28% of the sample experienced no events above 70 db, L Amax, it was also sensible to investigate N65. 3.7 Apart from L den which is usually calculated for an annual average day, the primary analysis considered the average summer day value for L Aeq,16h, N70 and N65. 3.8 Given previous views that, despite the way the question is couched, people may actually answer survey questions in terms of their exposure just prior to the survey, it was good that the analysis also considered the exposure for the 7 days and 30 days prior to the date of the survey. (Paragraph 3.26). 3.9 Furthermore, the use of single mode has been advocated for many years as a means of explaining more clearly the day to day impact that is experienced by someone affected by aircraft noise. It was therefore helpful to examine the association between that metric and annoyance. 3.10 In all cases, the derivation of these various indicators was felt to be robust and provided a good basis for determining the exposure the respondents experienced. The consistency of the methodology for the noise modelling with current standards/best practice. 3.11 It is understood that SoNA 2014 was specifically designed to make use of noise exposure information that was readily available, rather than obtaining such information specifically for the study. Given that many airports routinely produce noise exposure data, this was a sensible and, probably, cost effective approach. 3.12 That method did mean that the exposure information had to be gathered from more than one source. As set out in Paragraph 3.16, data for 5 of the airports covered in the study came from the CAA and for four of the airports, from the company, Bickerdike Allen and Partners (BAP). Paragraph 3.22 confirmed that the CAA data was produced from their ANCON model, whereas the BAP data came from INM. 3.13 That paragraph also confirmed that although two different software platforms were used, they both conformed with the relevant international standard for aircraft noise modelling (as set out in the ECAC.CEAC Document 29). STA is familiar with both ANCON and INM and is satisfied that for the same input data, both models would produce comparable results. 3.14 Consequently, the noise modelling used for the study was consistent with current standards and best practice. Page 6 of 13
The robustness of the use of noise data in the statistical analysis. 3.15 This study determined the noise exposure data at each respondent s location. That meant that the responses were associated with the exposure experienced by each respondent. This optimised the robustness of this aspect of the study. 3.16 For the most part, the noise data used were for a summer average day, except for L den when the annual average 24-hour period was used. Given one of the aims of the study was to see if the current method for estimating annoyance was still appropriate, this was a sensible approach. 3.17 Historically, it has been conventional to assign the L Aeq,16h noise exposure data into 3 db bands, and this approach was followed for this study. The reviewers did ask for the data to be analysed on an individual basis. The results, however, showed the very wide variation in response at particular exposures that had been found in many previous studies across all types of transport source. That meant it was not possible to draw any meaningful conclusion from those results. By grouping the exposure into 3 db bands, clearer conclusions could be drawn. 3.18 Having established that banding approach for L Aeq,16h, the L den data were similarly grouped. 3.19 No such traditional banding exists for the N70 and N65 data and during the review process there was discussion regarding how best to group these data. It was clear that there had to be a separate band for those who experienced no events above the relevant L Amax level. Given that the 3 db banding for the L Aeq based indicators corresponds, for a given noise level, to a doubling or halving of movements, the decision was made to place the N70 and N65 data into bands that reflected a doubling in the number of events above the relevant L Amax level (See paragraph 5.8). This was a sensible approach. 3.20 For completeness, the reviewers asked that, as far as possible, information should be provided in the report regarding the operational modal splits of the airports covered by the study both for 2013 and 2014 (and 2012 where applicable). This information is now provided (Table 7). The appropriateness of the LAeq metric to quantify health impacts. 3.21 This aspect is discussed in Section 4 of this report. Outcome of social science review The robustness of the sampling approach. 3.22 The sampling design was detailed in the Ipsos MORI technical report and a succinct overview was provided in the final report (Paragraphs 3.17-3.19). A mixed sample design was adopted whereby sampling in the 51-54dB band was clustered and sampling in the 54+dB bands was unclustered. Sampling was disproportionate with 33% in the 51-54dB band and 66% in the 54+dB bands. Page 7 of 13
The sample was stratified by airport (otherwise 76.6% of the 54+dB bands would have been near Heathrow). This sampling approach was considered appropriate for this study. 3.23 The scope of the study meant that data was collected from the nine largest airports in England. However, not all regions within England were represented. The South-West was not represented as Bristol, the tenth largest airport, was excluded due to the available sample size and a lack of noise data. 3.24 The sampling procedure was appropriate and achieved 1,877 valid responses (Paragraph 8.2). 3.25 As 67% of the achieved sample lived close to Heathrow and 10% near Gatwick, 23% were sampled across the remaining 7 airports. At some of the smaller airports the sample size was 30-50% less than planned. 3.26 For analytic purposes, the number of cases for each airport was weighted relative to Gatwick so that the resulting sample sizes were in the correct proportion to the estimated population exposed to aircraft noise at each airport. As the majority of the analysis was conducted with the weighted sample a clearer explanation for the use of unweighted versus weighted data was requested for the final report. This was provided in Paragraph 4.3. 3.27 An account of the criterion for inclusion/exclusion of cases was requested and this was also provided in the final report in Paragraph 3.19. The quality of the survey method. 3.28 Fieldwork was conducted between October 2014 and February 2015 i.e. during the winter, but respondents were asked to recall their experience of aircraft noise during the previous summer, June-September 2014. Although the use of retrospective recall was adequate for this study, ideally, subjective responses would have been obtained during the same time period as that covered by the noise exposure data. 3.29 The SoNA 2014 survey instrument was based on SoNA 2013 and NNAS 2012. Most of the core elements were reasonably well worded. The social reviewer had minor concerns about the content of the show cards (in terms of gender stereotypes and the amount of information on them) but concluded that there was no significant impact on the overall quality of the survey method. The appropriateness of the interviewing procedure. 3.30 An appropriate degree of briefing and review was conducted by Ipsos Mori. Interviewers were suitably recruited and trained although it was not clear how, if at all, supplementary qualitative data (related to interest in the subject) was recorded or analysed. Page 8 of 13
The robustness of data analysis and the interpretation of data. 3.31 Data entry validation was deemed appropriate. Issues such as the treatment of outliers and Don t know responses as well as the use and reporting of weighted versus unweighted cases was raised by the reviewers in the interim report. These issues were subsequently properly addressed in the final report. 3.32 An appropriate selection of cases was made based on responses to previous questions, for example, CAN22d and CAN23b. 3.33 Data that were missing or incomplete in the first draft of the final report were highlighted (e.g. row totals and percentages) and addressed. 3.34 Careful consideration was given by the reviewers to the degree of accuracy and consistency in the reporting of statistical analysis. Requests to include more detail about the methods of analysis were appropriately addressed in the final report (e.g. Paragraphs 5.22 and 5.23). 3.35 Care was advised over the use, reporting and interpretation of the odds ratio statistic. The term strengthening to describe similar odds ratio values reported in Paragraph 7.18 was considered to be appropriate as the difference between odds ratio of 1.13 and 1.2 is marginal. 3.36 In the initial review, the reviewers drew attention to distinguishing, throughout the report, between interpretation based on a subjective view as opposed to statistical analysis of the data. This issue was addressed in the final report. 3.37 To aid the proper understanding of the report, the reviewers advised that attention be paid to the use of the word significant to avoid confusion between statistical language and subjective or policy interpretation. This has been addressed in the final report. Page 9 of 13
4.0 AIMS of SoNA2014 4.1 Both reviewers were required to assess the extent to which the five aims of SoNA 2014 had been addressed. The five aims were: 1. Obtain new and updated evidence on attitudes to aviation noise around airports in England, including the effects of aviation noise on annoyance, wellbeing and health. 2. Obtain new and updated evidence on what influences attitudes to aviation noise, and how attitudes vary, particularly how attitudes vary with LAeq, but also other non-acoustic factors that may influence attitudes, such as location and time of day, and socio economic group of respondents 3. Examine whether the currently used measure of annoyance, L Aeq, is the appropriate measure of annoyance for measuring the impact on people living around major airports. 4. Consider the appropriateness of the policy threshold for significant community annoyance from aviation noise. 5. Provide baseline results that can be used for a programme of regular surveys of attitudes to aviation noise. 4.2 As indicated in paragraph 2.3 above, the reviewers felt that it was not clear from the initial draft the extent to which the aims had been met. The revised narrative structure that can be seen in the final report works well and does set out a logical sequence of analysis. 4.3 The reviewers agree with the approach set out in Chapter 8 of the final report of re-casting the aims as a series of questions. These have been reproduced below together with the reviewers comments. Is L Aeq,16h still the most appropriate indicator to use to estimate the annoyance arising from aircraft noise? 4.4 The reviewers concur with the conclusion that the results of the study confirm that L Aeq,16h is still the most appropriate indicator to use to estimate the annoyance arising from aircraft noise. There is merit, though, in considering the use of metrics such as N70 and N65 as supplementary indicators to assist in understanding the noise impact. Is summer day, average mode, still the best time period to use as opposed to single mode? 4.5 Again, the analysis does support the conclusion that average mode is better than single mode. The relatively poorer association of westerly single mode is an unexpected outcome, given that, for many of the respondents locations, the difference in the numerical value of the exposure between average and westerly single mode is generally small. The explanation set out in Page 10 of 13
Paragraph 5.30 is plausible. It can also be seen from Table 7, that, for most airports in the study, the modal split for 2014 showed more easterly days than in 2013. 4.6 The reviewers agree with the observation that although the association between annoyance and indicators such as N70, N65 and single mode exposure is weaker than average mode L Aeq,16h, it does not preclude the use of these indicators to assist in describing the aircraft noise impact at a particular location. How does annoyance relate to aircraft noise exposure? 4.7 The study does confirm that the likelihood of being annoyed or highly annoyed does increase with noise exposure. The reviewers agree with the conclusion that annoyance levels plateau at low exposures meaning that it can be inferred that an underlying level of annoyance exists irrespective of noise exposure. How do the results compare with ANIS, ANASE & Miedema? 4.8 The evidence from this study indicates that people are more sensitive to aircraft noise compared with the results of the ANIS study. The reviewers are satisfied that this is a robust outcome from the study and can be relied upon. 4.9 In the 13 December version of the report, Table 31 only appeared in Chapter 8. The reviewers advised that it should also appear in the relevant section earlier in the report. This has been addressed in the final report at Table 25. How do measures of health and well-being relate to [aircraft noise] exposure? 4.10 The reviewers felt that the questionnaire used in SoNA 2014 was limited in its exploration of any health effects that may arise as a result of exposure to aircraft noise. The analysis undertaken, however, was reasonably thorough and the conclusions drawn robust. 4.11 The reviewers did query why, if there was an association between noise exposure and annoyance;; and between annoyance and health measures, there was no direct association between noise exposure and the health measures. No clear explanation could be identified. 4.12 The reviewers do agree, however, that no causality can be inferred between the health measures and annoyance, it is just an association. What non-acoustical factors seem to influence annoyance [arising from aircraft noise exposure]? 4.13 The conclusions drawn about the influence of certain non-acoustic factors are supported and plausible. Page 11 of 13
4.14 Although the slight indication of there being an urban/rural differentiation is noted, the caution that this conclusion may be confounded by other factors and cannot be confirmed by this study is supported. Other Comments 4.15 The final aim of the study sought to Provide baseline results that can be used for a programme of regular surveys of attitudes to aviation noise. 4.16 The reviewers are satisfied that the survey methodology and analytical techniques employed in this study were robust. Consequently, the process used does provide a basis for future studies. 4.17 Having said that, it is recommended that the following issues are taken into account for any future study: Try to achieve the desired sample size across all the airports;; Undertake the survey nearer to the time to which the noise exposure indicators relate;; Be clearer about the type of statistical analyses used;; If the possible effects of aircraft noise and health are of interest, consider including more questions on this issue, even if it means excluding some questions that were in SoNA 2014. Page 12 of 13
5.0 CONCLUSIONS 5.1 The survey and analysis have fully addressed the stated aims of the study. 5.2 The conclusions, as set out in the final report, are robust and properly reflect the data obtained and the analysis carried out. 5.3 It is the view of the peer reviewers that the detailed results and conclusions from SoNA 2014 can be used as a basis for the further development of Government policy in this area. Page 13 of 13