Uncertainty assessment using the NUSAP approach: a case study on the EFoNAO tool 1

Transcription

1 EXTERNAL SCIENTIFIC REPORT Uncertainty assessment using the NUSAP approach: a case study on the EFoNAO tool 1 Martijn Bouwknegt a and Arie H. Havelaar a,b,2 a Centre for Infectious Disease Control, National Institute for Public Health and the Environment, Bilthoven, the Netherlands; b Institute for Risk Assessment Sciences, Department of Veterinary Medicine, Utrecht University, Utrecht, the Netherlands ABSTRACT Many of the current policy issues that have to be addressed are suffering from large uncertainties, while decision stakes are high. Addressing the uncertainties explicitly in risk assessments becomes especially important when addressing such issues. This report describes the evaluation of an uncertainty typology combined with a NUSAP (acronym for Numeral, Unit, Spread, Pedigree and Assessment) approach. The uncertainty typology is used for uncertainty characterization based on six dimensions, aiding the choice for subsequent approaches to dealing with the uncertainties in e.g. decision making. One such approach is the NUSAP method, which is used for prioritizing uncertainty sources based on the scientific strength and influence on results as judged by experts. The case study selected was EFSA s Food of Non-Animal origin risk ranking model, which aims to identify and rank pathogen and food combinations based on public health concern. Sixteen uncertainty sources were identified and characterized with the uncertainty typology. The 16 uncertainty sources were implicit or explicit assumptions and assessed as such for scientific strength and influence on results during a NUSAP workshop. Criteria for assessing the scientific strength included: influence of situational limitation, assumption plausibility, choice space and peer agreement. Overall, the combination of uncertainty typology and NUSAP was found to be very helpful as method for uncertainty assessment, as the procedure increases insight in uncertainty sources related to model outcomes and in their impact on the end result. The attention to uncertainty was recommended to be integrated in risk assessments, as it aids to formalise discussions about uncertainties. It was also recommended that practicality and feasibility aspects should be considered when incorporating uncertainty assessment in such assessments, that a clear and consistent terminology should be used and that training of experts is required to integrate elements of uncertainty assessment in future risk assessment activities. National Institute for Public Health and the Environment, 2015 KEY WORDS uncertainty, qualitative, quantitative, typology, NUSAP, expert opinion DISCLAIMER The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors. 1 Question EFSA-Q Acknowledgement: The authors would like to acknowledge: the important contributions made by Roland Lindqvist in identifying and characterizing the uncertainties and assumptions in the EFoNAO model; the participants in the workshop for their inputs and included (in alphabetical order) Paul Cook, Maria Teresa Da Silva Felicio, Pablo Fernandez, John Griffin, Kostas Koutsoumanis, Roland Lindqvist, Jim McLauchlin, Birgit Nørrung, Giuseppe Ru and Moez Sanaa; Jeroen van der Sluijs for his support in developing the workshop protocols and data interpretation; and the EFSA staff for all support provided to this external scientific report: Pablo Romero Barrios and Winy Messens. Any enquiries related to this output should be addressed to biohaz@efsa.europa.eu Suggested citation: Bouwknegt M and Havelaar AH, Uncertainty assessment using the NUSAP approach: a case study on the EFoNAO tool., 20 pp. Available online: European Food Safety Authority, 2015

2 SUMMARY The National Institute for Public Health and the Environment was awarded this project entitled Uncertainty assessment using the NUSAP approach: a case study on the EFoNAO risk ranking tool. In a mandate of the EFSA BIOHAZ Panel for a scientific opinion on the development of a risk ranking toolbox for EFSA BIOHAZ Panel (M ), one of the objectives is to investigate methodologies for introducing uncertainty and variability in the risk ranking models. To fulfil that objective it was decided to identify and assess the uncertainties associated to one of the risk ranking tools (that is the EFoNAO-RRT tool) using the NUSAP (acronym for Numeral, Unit, Spread, Assessment, and Pedigree) methodology as case study. Uncertainties related to the model were identified and listed as (explicit and implicit) assumptions, the list of which was subsequently finalised by discussions with experts from the working group Risk Ranking Tools. The sources of uncertainties (i.e., assumptions) were subsequently characterized with an uncertainty typology using the following dimensions: location, nature, range, recognized ignorance, methodological unreliability and value diversity. Applying the uncertainty typology helps to select subsequent methods for dealing with the uncertainty sources. One such approach is NUSAP, which allows for assessing the effect of uncertainty sources on the model outcome for both quantitative and qualitative uncertainty sources. NUSAP is an approach that simultaneously addresses quantitative and qualitative aspects of uncertainty. The approach identifies the uncertainty sources that are most important for the total uncertainty of a model outcome. NUSAP involves the assessment of the scientific strength of the uncertainties based on multiple criteria and the impact of these uncertainties on the model outcome. The criteria that were used for scientific strength in the current study included: influence of situational limitations, plausibility, choice space and peer agreement. Each criterion was scored by 10 experts on a discrete ordinal scale ranging from 0 to 4 and the median score per expert was taken as score for scientific strength. The influence of assumptions on the model outcome was scored on a discrete ordinal scale ranging from 0 to 3. The model outcomes under consideration were: 1) the identification of important microbial hazards related to foods of non-animal origin, and 2) the ranking of these hazards. These uncertainty identification and characterization based on the typology yielded 16 general assumptions relating to the EFoNAO-RRT (these assumptions are not described in detail in this report, given the objective of the workshop to evaluate the use of the typology and NUSAP for EFSA rather than a critical evaluation of the EFoNAO-RRT). Seven of the 16 uncertainty sources were scored during the four-hour scoring part of the workshop. None of the assumptions were scored as scientifically weak and largely influential on the model outcome, yet two uncertainty sources scored notably lowest on scientific strength among those evaluated. All but one assumption were considered to have no or negligible impact on the hazard identification. In contrast, 4 out of the 7 assumptions were judged to have a moderate influence on the hazard ranking (median score 2) while the scientific strength of these assumptions varied between 1.5 and 2.5. These assumptions are therefore identified as the most important ones for further work among the seven assumptions evaluated. The combination of uncertainty typology and NUSAP was found to be very helpful to gain insight in the uncertainty sources related to model outcomes and to assess their impact on the end results. Furthermore, the procedure aided in communicating models to non-modellers. The interpretation of pedigree criteria to assess the strength and effect was found to be difficult by participants, partly due to the unfamiliar terminology used. It was therefore recommended to use a clear terminology, which is 2

3 understood by all involved in the assessment and that preferably a short training session for experts is conducted prior to the use of the evaluated approaches to increase the familiarity. In conclusion, the combination of uncertainty typology and NUSAP was found to be very helpful by all participants of the workshop. The procedure helped to systematically identify and evaluate the uncertainty sources related to model outcomes and to assess their impact on the end results. A framework encompassing uncertainty typology and evaluation (e.g. by NUSAP) was recommended to be part of each risk assessment process to formalise discussions on uncertainties. Correct and efficient implementation would require the use of clear terminology and training of experts to become familiarized with the approaches. 3

4 TABLE OF CONTENTS Abstract... 1 Summary... 2 Background as provided by EFSA... 5 Terms of reference as provided by EFSA Introduction and Objectives Materials and Methods Uncertainty typology Location Nature Range Recognized ignorance Methodological unreliability Value diversity NUSAP NUSAP case study: the EFoNAO risk ranking tool Results Uncertainty typology NUSAP workshop Evaluation of the methodology Discussion and Conclusions Recommendations References Abbreviations

5 BACKGROUND AS PROVIDED BY EFSA Uncertainty assessment using the NUSAP approach In the context of an ongoing self-tasking mandate from the Panel of Biological Hazards (BIOHAZ Panel) for a scientific opinion on the development of a risk ranking toolbox for EFSA BIOHAZ Panel (M ), and more specifically under ToR 2 (to investigate methodologies for introducing uncertainty and variability in the risk ranking models), a workshop on the uncertainties in the EFSA food of non-animal origin risk ranking tool (EFoNAO-RRT) is planned at EFSA premises for April This model supported the scientific opinion on the risks posed by pathogens in foods of nonanimal origin Part 1 (outbreak data analysis and risk ranking of food/pathogen combinations) ( This workshop will aim at identifying and assessing the uncertainties associated to one of the risk ranking tools (that is the EFoNAO-RRT tool) currently selected by the Working Group (WG) experts dealing with the abovementioned mandate, using the NUSAP (acronym for numeral, unit, spread, assessment and pedigree) methodology. This will serve as an example of how uncertainty and variability arising from risk ranking models can be assessed. It is foreseen that the outcome of this workshop will be used in the scientific opinion on the development of a risk ranking toolbox for EFSA BIOHAZ Panel being developed. In order to be able to best capture the information gathered during the workshop, we would like to seek the services of the workshop trainer to compile an external scientific report analysing and summarising the findings of the workshop. TERMS OF REFERENCE AS PROVIDED BY EFSA To provide EFSA with an external scientific report analysing and summarising the findings of the planned workshop on the uncertainties in the EFoNAO-RRT. This report will be used by the WG on the development of a risk ranking toolbox as an input to answer one of the TORs of the mandate (to investigate methodologies for introducing uncertainty and variability in the risk ranking models ) mentioned above. The report should cover at least the following areas: Introduction Approach/Methodology: describing the format of the workshop and the NUSAP methodology Results: this should include the outcome of the assessment of the uncertainties related to the different assumptions and parameters used in the EFoNAO-RRT Conclusions and recommendations: these should focus not just on the outcome of the EFoNAO-RRT assessment, but more generally about how this methodology can be applied to risk ranking models or tools. This contract was awarded by EFSA to: Contractor: National Institute for Public Health and the Environment, Bilthoven, the Netherlands Contract title: External scientific report analysing and summarising the findings of the NUSAP workshop Contract number: NP/EFSA/BIOCONTAM/2014/01 5

6 1. INTRODUCTION AND OBJECTIVES Many of the current policy issues that have to be addressed are suffering from large uncertainties, while decision stakes are high. This means that assessments for policy support need to be adapted to aid the management of such issues, requiring a different approach than the normal fundamental and applied scientific approaches that have proved useful in the past. Often driven by curiosity or restricted problems, uncertainty in fundamental and applied study outcomes is considered provisional and reducible by further research. The current societal problems are however more complex and cannot be solved solely by these scientific approaches. A new approach, termed post-normal science (Funtowicz and Ravetz, 1993), has been proposed, in which uncertainty is viewed as an intrinsic part of complex systems that cannot always be quantified. One has therefore to deal openly with deeper dimensions of uncertainty, including problem framing indeterminacy, ignorance, assumptions and value loadings. Scientist, policy makers and stakeholders together need to find solutions with explicit discussions on the imperfections and uncertainties. This means amongst others that uncertainty analysis is an explicit part of the outcome. The focus therein extends beyond the classical interpretation of uncertainty, with a central estimate and 95 % confidence interval, to an assessment of the larger part of the total uncertainty, including also qualitative uncertainty sources (i.e., uncertainties for which the effect on the model outcome cannot be quantified). Different approaches exist to address different types of uncertainty, both qualitative and quantitative in nature (van der Sluijs et al., 2004). These methods vary from quantitative uncertainty analyses using model parameters and possibly Markov-Chain Monte Carlo (MCMC) methods to assessment of qualitative uncertainty sources based on expert elicitation. In the Environmental Health Impact Assessment field, the combination of an uncertainty typology to characterize uncertainty sources and NUSAP (acronym for numeral, unit, spread, assessment and pedigree; see approach in section 2) has been applied frequently to deal with large uncertainties. This approach was subsequently shown to be useful in the field of foodborne diseases as well (Boone et al., 2010; Bouwknegt et al., 2014). In a mandate on a Risk Ranking Toolbox (the EFSA BIOHAZ Panel), one of the objectives is to evaluate methods and approaches to evaluate uncertainties associated with the outcomes of panel assessments (opinions). As part of that work it was decided to evaluate the uncertainty typology and NUSAP approach in a case study. The case study selected was EFSA s Food of Non-Animal risk ranking model (EFoNAO-RRT) used for identifying and ranking pathogen and food combinations of most public health concern (EFSA BIOHAZ Panel, 2013). The working group identified and characterised uncertainties (the typology) in that model and the uncertainties were evaluated in a workshop setting with working group members and panel experts using the NUSAP approach. Following the evaluation, experts discussed the potential use of this approach by EFSA. This report describes the evaluation of the combined use of typology and NUSAP approach to evaluate uncertainty based on this case study. 2. MATERIALS AND METHODS 2.1. Uncertainty typology An uncertainty typology was used to characterize sources of uncertainty identified in the EFoNAO- RRT model. Uncertainties were identified by reviewing the approach as described in the opinion (EFSA BIOHAZ Panel, 2013) and listing explicit and implicit assumptions and uncertainties. The list of uncertainties was then finalised by discussions with experts from the working group on Risk Ranking Tools and phrased as assumptions. The sources of uncertainties were subsequently characterized based on the uncertainty typology from Knol et al. (2009). Sources of the uncertainties are characterized in the following dimensions: location, nature, range, recognized ignorance, 6

7 methodological unreliability and value diversity (Table 1). Each dimension will be explained briefly in the following paragraphs. Applying the uncertainty typology helps to select subsequent methods for dealing with the uncertainty sources further. One such approach is NUSAP, which allows for assessing the effect of uncertainty sources on the model outcome for both quantitative and qualitative uncertainty sources. Table 1: Typology of uncertainty (obtained from Knol et al.(2009)) Uncertainty characterizations Location: the location at which the uncertainty manifests itself in the assessment Nature: the underlying cause of the uncertainty Range: expression of the uncertainty Categories Context: Definitions and boundaries of the system that is being assessed Model structure: Structure and form of the relationships between the variables that describe the system Parameters: Constants in functions that define the relationships between variables (such as relative risks or severity weights) Input data: Input data sets (such as concentrations, demographic data, and incidence data) Epistemic: resulting from incomplete knowledge Ontic (process variability): resulting from natural and social variability in the system Statistical (range + chance) : specified probabilities and specified outcomes Scenario (range + what if ) : specified outcomes, but unspecified probabilities Recognized ignorance: unknown outcomes, unknown probabilities uncertainties are present, but no useful estimate can be given Methodological unreliability: Methodological quality of all different elements of the assessment; a qualitative judgment of the assessment process which can based on e.g. its theoretical foundation, empirical basis, reproducibility and acceptance within the peer community Value diversity among analysts: Potential value ladeness of assumptions which inevitably involve to some degree arbitrary judgments by the analysts Location The location at which an uncertainty source acts in a study can be characterized as one of three mutually exclusive categories: contextual uncertainty, model uncertainty or data uncertainty. The latter category was obtained by combining the parameter values and input data categories as originally distinguished by Knol et al.(2009). Contextual uncertainty refers to uncertainty that is related to the chosen study boundaries. For instance, a choice to focus on a particular pathogen or pathogens due to e.g. data availability, whereas all pathogens might be of interest from a public health point of view, leads to uncertainty due to an incomplete focus. This contributes to the total uncertainty associated with the model outcome. Model uncertainty refers to uncertainty in the modelling part of the study. This can be for instance the choice for a specific regression technique, the choice for a specific type of dose-response model, and the assumption for the type of relation (e.g. linear) between variables. Data uncertainty (a combination of the categories Parameters and Input data from Table 1) refers to uncertainty of all data and parameter values used in the modelling. 7

8 Nature Uncertainty assessment using the NUSAP approach The nature dimension in our typology describes the underlying cause of the uncertainty. Two categories were identified to describe the nature of an uncertainty source. First of all, uncertainty can be due to incomplete knowledge, implying that the uncertainty would decrease when more information would become available. This is called epistemic uncertainty. On the other hand, many processes are variable in nature (such as the concentration of Campylobacter spp. on leafy greens), and ideally are accounted for as such. Uncertainties related to variability were characterized as so-called ontic uncertainties. The nature of an uncertainty source is thus characterized as either epistemic or ontic. Note that the epistemic and ontic uncertainties are generally interpreted as uncertainty and variability in quantitative microbiological risk assessment studies Range The range of a source of uncertainty deals with the description of uncertainties, which can be either statistically-based, described by a statistical distribution over a continuum, or category-based, using different possible discrete scenarios. The former is presented for instance as 95% confidence interval for a parameter estimated with regression models, whereas the latter is presented e.g. as different results obtained by different models or alternative assumptions. The range of a source of uncertainty is categorized as either statistical or scenario Recognized ignorance Recognized ignorance deals with the realization that sources of uncertainty exist but have not been considered in the modelling. The uncertainty sources are known to exist, but no useful estimate on its likelihood and effect can be provided. This issue relates for instance to aspects of the system that have been recognized to exist, but are currently insufficiently understood to provide a useful estimate of its effect. The presence of recognized ignorance for an uncertainty source was indicated with plusses and minuses Methodological unreliability The dimension methodological unreliability categorizes sources of uncertainty based on the way the information is processed and the level of uncertainty introduced by the methods used. For parameter values this dimension targets for instance the uncertainty introduced by the estimation approach: data quality and statistical processing of data. For input data such as Campylobacter spp. concentrations on chicken fillet, factors introducing uncertainty are for instance imperfect diagnostics tests. Modeling aspects, such as the deviation between a fitted probability distribution and the original data, might also occur. In general, methodological uncertainty can be interpreted as uncertainty related to the data or method used. Methodological uncertainty was scored with plusses and minuses Value diversity Value diversity is a dimension used to describe the extent to which a source of uncertainty is due to varying opinions among peers. It mainly deals with subjective choices between alternative models or datasets. For instance, value diversity is lower when the majority of peers agrees on the procedure to follow, than when the approach is subject to considerable debate. Also, value diversity might arise for example when choosing how to deal with a number of non-representative datasets for parameter estimation or when selecting the study area or cohort. The dimension of value diversity was scored with plusses and minuses to indicate its presence or absence, respectively. 8

9 2.2. NUSAP Uncertainty assessment using the NUSAP approach NUSAP, an approach that simultaneously addresses quantitative and qualitative aspects of uncertainty is an acronym for Numeral, Unit, Spread, Assessment, and Pedigree (Funtowicz and Ravetz, 1990). The Numeral, Unit and Spread part refers to a central estimate (i.e., most likely value for a parameter), its unit (e.g. or per day ), and the spread of the estimate (e.g., a 95 % confidence interval). NUSAP elaborates on this initial assessment by including expert judgment of the reliability of a knowledge claim (the Assessment ), and a systematic multi criteria evaluation of its knowledge base (the Pedigree ). To identify the uncertainty sources that were most important for the total uncertainty of the model in the case study of the EFoNAO-RRT model we used the assessment and pedigree component of NUSAP. This approach involved the assessment of the scientific strength of the uncertainties associated with the implicit and explicit assumptions of the EFoNAO-RRT model and the impact of these uncertainties on the outcome of the model was assessed. The scientific strength of each uncertainty source is scored according to four criteria (see Table 2 for the criteria used and the scores in this study). The median of the strength scores per expert was firstly assessed. The final score for scientific strength was subsequently taken as the median score of all medians per expert. Experts then estimated the influence of the uncertainty on the model results. This score, combined with the median score of the scientific strength, gives an impression of the importance of an uncertainty source: sources with low accuracy and large influence on the final results are the most important. The model outcomes under consideration were: 1) the identification of important microbial hazards related to foods of non-animal origin, and 2) the ranking of these hazards NUSAP case study: the EFoNAO risk ranking tool At the start of the workshop, the purpose and process of the meeting was explained to the participants. Next, the scientific strength and influence on result for each assumption was assessed in the cycle shown in Figure 1. The first step comprised the introduction and explanation of the assumption, followed by a short period for clarifying questions. A first round of scoring was subsequently conducted, during which the experts assessed the score per pedigree criteria (see Table 2) and noted down the arguments for that score. Each expert then revealed the scores and the scoring was discussed plenary. Especially large difference in scores were discussed to assess whether different views existed due to e.g. different interpretations and judgments on evidence, or due to fallible arguments. In the last phase, the experts were given the opportunity to adjust scores when the discussion led to a changed of insight that altered the score. The score and rationale for those scores were noted on a scoring card prepared in MS Excel (see Figure 2 for an exemplary card). In addition to the scoring for different criteria and influence on results, the rationale for assigning that score to specific criteria is an important output. Experts were therefore also asked to provide the main motivation for their choice. After the prioritization of the uncertainty sources, the motivation can be used to develop ideas to dealing with the uncertainty sources (e.g., reanalysis of data, further data collection,) The last hour of the workshop was devoted to discussing the usefulness of the approach as described in this report for EFSA and if considered useful, what possible approaches could be considered for its implementation. 9

10 Introduction of uncertainty source Clarifying questions & answers Scoring using the pedigree matrix Collection of scores through plenary reporting Discussion on the arguments for scoring Possible score adjustment Figure 1: The steps in the NUSAP approach that were completed in the workshop per uncertainty source Table 2: The pedigree matrix used in the NUSAP workshop to assess the scientific strength of each uncertain assumption and its influence on the results (effect) Influence of Score situational limitations 0 Choice assumption hardly influenced 1 Limited influence in choice assumption 2 Choice assumption moderately influenced 3 Important influence in choice assumption 4 Totally different assumption had there not been limitations Plausibility Scientific strength The assumption is very plausible (based on established theory, verified through peer review) Plausible (based on model with theoretical basis, empirically verified data) The assumption is acceptable (based on a simple model, extrapolated data) Assumption is doubtful (based on not verified empirical data) The assumption is fictive or speculative Choice space Hardly any alternative available Very limited number of alternatives available Limited choice from alternative assumptions Average number of alternatives Ample choice from alternative assumptions Agreement among peers A large majority (90-100%) among peers would have made the same assumption Many experts (75%) would have made the same assumption Several experts (50%) would have made the same assumption Few experts (25%) would have made the same assumption Controversial assumption, hardly any expert (1%) would have made the same assumption Effect Influence on results The assumption has no or negligible impact on the results The assumption has little impact on the results The assumption has a moderate impact on the end result The assumption has an important impact on the end result 10

11 Figure 2: Example of score card used in the NUSAP workshop at EFSA 3. RESULTS 3.1. Uncertainty typology A list of all assumption and parameter uncertainties was produced. These sources were grouped to a general level and rewritten as assumptions, yielding 16 general assumptions relating to the EFoNAO- RRT (Table 3). The assumptions will not be described in detail in this report, given the objective of the workshop (i.e., to evaluate the use of NUSAP for EFSA, and not to evaluate the EFoNAO-RRT). The identification of the uncertainty sources based on the uncertainty typology from Table 1 is shown in Table 3. The majority of uncertainty sources (11 out of 16) related to the parameter and input data that was used. Furthermore, 14 of the 16 uncertainties were considered to relate to imperfect knowledge ( epistemic ), which could be reduced by further studies. 11

12 Table 3: Characterisation of the 16 uncertainty sources by using the uncertainty typology of Table 1 Nature Range Recognized Method Value Epistemic Ontic Statistical Scenario ignorance unreliability diversity Contextual uncertainty Link between a pathogen and a type of FoNAO can be deduced from outbreak data only a(4) X X The added value of considering pathogen inactivation to assess risk levels is negligible for each food-hazard pair a(5) X X X Contextual and model uncertainty The risk of a pathogen/food combination can be estimated by a linear, unweighted combination of scores on seven parameters, each divided in three or four categories that are represented by arbitrary numbers a(2) X X Model uncertainty The risk of a pathogen/food combination can be estimated by a combination of top-down and bottom-up approaches a(1) X Assuming a prevalence score of 2 to the category defined as unknown prevalence, implies that the prevalence cannot be assumed to be zero for Shigella spp., Yersinia spp., Staphylococcus aureus, Norovirus, HAV, and Cryptosporidium spp. X X Parameter and input data uncertainty The estimated true number of illness by a specific pathogen in the EU, without consideration of attribution to sources, is a valid indicator of the risk of a specific pathogen in a specific food of non-animal origin X X The prevalence of pathogens in all FoNAO samples is a valid estimate for the prevalence in the FoNAO group under consideration X X +/- - + The relative degree of underreporting of outbreak cases is the same in the US and EU and for each food-hazard pair X X The incidence of norovirus and bacterial intoxications in the EU is similar to the Netherlands X X a assumption evaluated in the NUSAP workshop. The number between brackets indicates the assumption number used in Figures 3-6 for reference. by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors. 12

13 Table 3: Continued Parameter and input data uncertainty The longest reported shelf life of food in a specific food group is representative of all products in that group and pathogen growth is not affected by growth of spoilage organisms Nature Range Recognized Method Value Epistemic Ontic Statistical Scenario ignorance unreliability diversity X X The available consumption data are representative of the whole EU a(12) X X Low numbers of Salmonella spp., Shigella spp., pathogenic Escherichia coli (e.g. STEC) and Yersinia enterocolitica can cause disease without growth during storage in retail or consumer s homes X X Pathogen-specific DALY estimates published for the Netherlands are representative for the whole EU a(14) X X DALYs per case for Shigella spp. and Yersinia enterocolitica fall within the same category as Salmonella spp. and are the same for STEC O157 and STEC non-o157 X X All products will be eaten at the end of their shelf life a(16) X X With the exceptions of Bacillus cereus and Clostridium perfringens, the overall prevalence of all pathogens in the different FoNAO groups, is assumed to be either low (< 1%) or unknown X X - +/- + a assumption evaluated in the NUSAP workshop. The number between brackets indicates the assumption number used in Figures 3-6 for reference. by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors. 13

14 Figure 3: Scientific strength ( rigor ) of the assumptions identified in the EFoNAO-RRT that yield uncertainty in the model outcome Legend: The white diamonds indicate the median score, the error bars the minimum (left) and maximum (right) score, and the black rectangle indicates the interquartile range. Assumptions with higher scores (in the red zone) have lower scientific strength compared to lower scores (green zone). Diamonds crossing the y-axis indicate the assumptions that have not been scored. Figure 4: Pedigree scores per criterion as provided by the experts during the NUSAP workshop Legend: The x-axis displays the assumption that was scored according to the pedigree criteria explained in Table 2: influence of the situational limitations, plausibility, choice space, and agreement among peers. The y-axis shows the median score for the criteria over all experts. The error bars indicate the interquartile range. 14

15 3.2. NUSAP workshop Uncertainty assessment using the NUSAP approach Seven of the 16 uncertainty sources were scored during the four-hour scoring part of the workshop, needing on average about 35 minutes per assumption. There was intensive discussion on the NUSAPmethodology and on the interpretation of the criteria and the scores. This discussion led to a revision of the definitions for the influence on results categories (the final definition is shown in Table 2). Figure 3 shows the scoring results for scientific strength for the seven assumptions. All interquartile ranges covered 1 score class, thereby showing agreement among the experts. The uncertainty sources with lowest scientific strength (i.e. those with the highest score) among those evaluated were assumptions 2 and 4. None of the assumptions were scored as scientifically very weak and largely influential on the model outcome. Notably, the median scores for scientific strength are concentrated around the midpoint of the scale. Similarly, the scores for each of the pedigree criteria concentrated around the mid-score (Figure 4). A score of 2 was assigned most frequently (n=100), followed by a score of 1 (n=88) and 3 (n=73). Scores of 0 and 4 were assigned 17 and six times, respectively. Eighteen of the 28 scored criteria had an interquartile range 1 score unit, again showing agreement among the experts regarding the pedigree criteria. The pedigree criterion agreement among peers was scored consistently best for all assumptions; the criteria influence of situational limitations, plausibility and choice space scored in general lower than agreement among peers and showed larger variation. Figures 5 and 6 show the strength and effect diagrams for the two model outcomes considered (identification of hazards and ranking of these hazards, respectively). All but one assumption (nr. 4) were considered to have no or negligible impact on the hazard identification. In contrast, 4 out of the 7 assumptions were judged to have a moderate influence on the hazard ranking (median score 2) while the scientific strength of these assumptions varied between 1.5 and 2.5. These assumptions are therefore identified as the most important ones for further work among the seven assumptions evaluated Evaluation of the methodology NUSAP was found to be very helpful by all participants in the workshop. The procedure helps to gain insight in the uncertainty sources related to model outcomes and to assess their impact on the end results. Furthermore, the procedure aids in communicating models to non-modellers. The attention to uncertainty was recommended to be a part of risk assessment, as it would help to formalise the discussion about the uncertainties. By doing this structurally and integrated with the risk assessment activities, experience grows and the process would eventually save time. Nevertheless, it was recommended that practicality and feasibility aspects should always be considered when incorporating uncertainty assessment in the risk assessment process. The interpretation of pedigree criteria to assess the strength and effect (see Table 2) was found to be difficult by participants. Part of the difficulty is caused by the difference in terminology between scientists working in philosophical sciences, who developed the NUSAP methodology, and natural sciences. It was recommended that a clear terminology is to be used, which is understood by all involved in the assessment. Preferably, a short training session with dummy uncertainty sources should be conducted prior to the NUSAP workshop. This training may be an on-site training given at EFSAs premises, but could also be part of a web-based training program. All experts would be required to follow this online training prior to their involvement in risk assessment activities. Another cause for difficulty with the interpretation of the pedigree criteria related to the issue of scoring the pedigree criteria independently from each other or conditionally. For instance, whether or 15

16 not peer agreement for an assumption should be assessed conditional on e.g. a large influence of situational limitations, had the latter been scored. Both approaches are valid, depending on the aims of the assessment. When aiming for a general critical review as part of the uncertainty assessment, then the criteria need to be assessed independently. The influence of situational limitations, for instance, deals then with external factors influencing the scientific quality, whereas the peer agreement is a proxy for the proper interpretation and use of scientific knowledge by the analyst. Much time during the workshop was devoted to discussions on how to describe the sources of uncertainty. Ideally, consensus on the phrasing/wording is obtained before the scoring starts. This should be an iterative process that involves both the principal analyst(s) of the study to be evaluated and the experts that will participate in the NUSAP workshop. No. 4 Nos. 12 & 16 Nos. 5 & 14 No. 1 No. 2 Legend: The x-axis displays the median scientific strength ( rigor, i.e., the white diamonds from Figure 2), the y-axis the median score for the influence on results. Values with a high score on influence on results and scientific strength (the red zone) are critical assumptions in the model. Figure 5: Strength and effect diagram for the seven assumptions of the EFoNAO-RRT that were scored during the workshop for the influence on hazard identification 16

17 No. 5 No. 12 No. 1 No. 2 No. 14 No. 16 No. 4 Legend: The x-axis displays the median scientific strength ( rigor, i.e., the white diamonds from Figure 2), the y-axis the median score for the influence on results. Values with a high score on influence on results and scientific strength (the red zone) are critical assumptions in the model. Figure 6: Strength and effect diagram for the seven assumptions of the EFoNAO-RRT that were scored during the workshop for the influence on hazard ranking. 4. DISCUSSION AND CONCLUSIONS The interpretation of pedigree criteria to assess the strength and effect (see Table 2) was found to be difficult by participants. Part of the difficulty is caused by the difference in terminology between scientists working in philosophical sciences, who developed the NUSAP methodology, and natural sciences, who took part in the assessment workshop. It was recommended that a clear terminology is developed, which is understood by all involved in the assessment. Preferably, a short training session with dummy uncertainty sources should be conducted prior to the NUSAP workshop. This training may be an on-site training given at EFSAs premises, but could also be part of a web-based training program. All experts would be encouraged to follow this online training prior to their involvement in scoring the assumptions. Much time during the workshop was devoted to discussions on how to describe the sources of uncertainty. Ideally, consensus on the phrasing/wording should be obtained before the scoring starts. This should be an iterative process that involves both the principal analyst(s) of the study to be evaluated and the experts that will participate in the NUSAP workshop. 17

18 The scoring by experts showed a tendency to the central value of 2 for the scientific strength criteria. At present we cannot provide the definitive cause for this tendency, but possible explanations include the following. Firstly, the seven considered assumptions might all have a similar, intermediate level of scientific strength according to the experts. Secondly, lack of experience of the experts might prevent extreme choices to be made. Thirdly, we created a series of general assumptions based on the EFoNAO-RRT model. The generalization might consider coincidentally several specific sources of uncertainties that possibly counteract. Experts need to produce a judgment on the net strength, possibly leading to intermediate scores. The central tendency would then be a characteristic of the approach. An alternative approach to the NUSAP workshop is to prepare a longlist of specific uncertainty sources and to prioritize this list by experts prior to the workshop to generate a shortlist. The shortlist with seemingly important specific uncertainty sources can subsequently be considered in the NUSAP workshop. Such an approach requires more time for preparation and more effort from experts, but provides more divergent scores that aids in the distinction of critical and non-critical uncertainty sources. The aggregation of scores by all experts on all four pedigree criteria into a single median (and interquartile range) that is subsequently used to judge the scientific strength of the assumptions was considered to result in loss of information. The criteria considered in the scoring of scientific strength are different in nature and addressing potential issues may require different strategies. A more detailed summary description of the results of this analysis, and possibly the development of a multi criteria analysis within the NUSAP approach was proposed in order to extract and use more of the information obtained during scoring of all criteria. In conclusion, the combination of uncertainty typology and NUSAP was found to be very helpful by all participants of the workshop. The procedure helped to systematically identify and evaluate the uncertainty sources related to model outcomes and to assess their impact on the end results. Furthermore, the combination of uncertainty identification and characterization aided in the understanding of the model by non-modellers. RECOMMENDATIONS A framework encompassing uncertainty typology and evaluation (e.g. by NUSAP) was recommended to be part of each risk assessment to formalise discussions on uncertainties. A clear and consistent terminology understood by all involved in the assessment should be used. When using the uncertainty typology and NUSAP for the first time, training is recommended. Consensus on the phrasing of assumptions and uncertainties should be reached among the experts and principal analysts prior to the NUSAP scoring session in an interactive process. Uncertainty sources should be described as specific as possible to allow for better consideration of the pedigree criteria when scoring. Nonetheless, it was also considered that practicality and feasibility aspects should always be considered when incorporating uncertainty assessment in the risk assessment. 18

19 REFERENCES Boone I, Van der Stede Y, Dewulf J, Messens W, Aerts M, Daube G and Mintiens K, NUSAP: a method to evaluate the quality of assumptions in quantitative microbial risk assessment. Journal of Risk Research, 13, Bouwknegt M, Knol AB, van der Sluijs JP and Evers EG, Uncertainty of population risk estimates for pathogens based on QMRA or epidemiology: a case study of Campylobacter in the Netherlands. Risk Analysis, 34, EFSA BIOHAZ Panel (EFSA Panel on Biological Hazards), Scientific Opinion on the risk posed by pathogens in food of non-animal origin. Part 1 (outbreak data analysis and risk ranking of food/pathogen combinations). EFSA Journal 2013;11(1):3025, 138 pp., doi: /j.efsa Funtowicz SO and Ravetz JR, Uncertainty and quality in science for policy. Kluwer Academic Publishers, Dordrecht, the Netherlands, 229 pp. Funtowicz SO and Ravetz JR, Science for the post-normal age. Futures, 25, Knol AB, Petersen AC, van der Sluijs JP and Lebret E, Dealing with uncertainties in environmental burden of disease assessment. Environmental Health, 8, 21. van der Sluijs JP, Janssen PHM, Petersen AC, Kloprogge P, Risbey JS, Tuinstra W and Ravetz JR, RIVM/MNP Guidance for Uncertainty Assessment and Communication: Tool Catalogue for Uncertainty Assessment. NWS-E Available at: ftp/user/mimler/uncertainty/vol_4_uncertaintyguidancetoolcat.pdf. 19

20 ABBREVIATIONS DALY Disability adjusted life year FDA Food and Drug Administration EFoNAO-RR EFSA s food of non-animal origin risk ranking tool FoNAO Food of non-animal origin HI Health impact MCMC Markov-chain Monte Carlo MS Member states NUSAP Numeral, Unit, Spread, Assessment and Pedigree RA Risk assessment RRT Risk ranking tool RTE Ready-to-eat STEC Shiga-toxin producing Escherichia coli WG Working Group 20