Homo heuristicus and the bias/variance dilemma

Size: px

Start display at page:

Download "Homo heuristicus and the bias/variance dilemma"

Russell Warner
5 years ago
Views:

1 Homo heuristicus and the bias/variance dilemma Henry Brighton Department of Cognitive Science and Artificial Intelligence Tilburg University, The Netherlands Max Planck Institute for Human Development, Berlin

2 Foraging for wild mushrooms 2 18

3 2 18 Foraging for wild mushrooms?

4 2 18 Foraging for wild mushrooms July 2017: A potential solution? Identify any mushroom instantly with just a pic

5 2 18 Foraging for wild mushrooms July 2017: A potential solution? Identify any mushroom instantly with just a pic

6 2 18 Foraging for wild mushrooms July 2017: A potential solution? Identify any mushroom instantly with just a pic

7 2 18 Foraging for wild mushrooms July 2017: A potential solution? Identify any mushroom instantly with just a pic

8 Quest Mobile vs. The mycologists 3 18

9 3 18 Quest Mobile vs. The mycologists Quest Mobile: There must be a solution to this problem. Given enough (big) data and a sophisticated enough learning algorithm, why can t we can probabilistically quantify the uncertainty surrounding this problem?

10 3 18 Quest Mobile vs. The mycologists Quest Mobile: There must be a solution to this problem. Given enough (big) data and a sophisticated enough learning algorithm, why can t we can probabilistically quantify the uncertainty surrounding this problem? The mycologists: You can t identify wild fungi just by looking at them. Animal interference, maturity, weather conditions. Many edible species have fatal twins. Majority of wild fungi remain undocumented, with new forms of toxicity still being discovered.

11 Faced with uncertainty, what are rational decisions? 4 18

12 Faced with uncertainty, what are rational decisions? Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

13 Faced with uncertainty, what are rational decisions? 1. The study of decision making under uncertainty : Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

14 Faced with uncertainty, what are rational decisions? 1. The study of decision making under uncertainty : Bayesian maximizers of expected utility in economics. Uncertainty can always be quantified: multiple priors, uninformed priors, 2nd-order probabilities, imprecise probabilities, non-additive probabilities, etc. Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

15 Faced with uncertainty, what are rational decisions? 1. The study of decision making under uncertainty : Bayesian maximizers of expected utility in economics. Uncertainty can always be quantified: multiple priors, uninformed priors, 2nd-order probabilities, imprecise probabilities, non-additive probabilities, etc. 2. The study of ecological rationality: Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

16 Faced with uncertainty, what are rational decisions? 1. The study of decision making under uncertainty : Bayesian maximizers of expected utility in economics. Uncertainty can always be quantified: multiple priors, uninformed priors, 2nd-order probabilities, imprecise probabilities, non-additive probabilities, etc. 2. The study of ecological rationality: What is rational when we re faced with unquantifiable uncertainty? Algorithms: Simple heuristics that ignore information. Ecological rationality is the study of the interaction between simple heuristics, natural environments, and performance. Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

17 Example: Catching a ball X Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

18 Example: Catching a ball When a man throws a ball high in the air and catches it again, he behaves as if he had solved a set of differential equations in predicting the trajectory of the ball... At some subconscious level, something functionally equivalent to the mathematical calculation is going on. Dawkins, R. (1976). The Selfish Gene. p. 96. X Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

19 Example: Catching a ball When a man throws a ball high in the air and catches it again, he behaves as if he had solved a set of differential equations in predicting the trajectory of the ball... At some subconscious level, something functionally equivalent to the mathematical calculation is going on. Dawkins, R. (1976). The Selfish Gene. p. 96. X Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

20 The gaze heuristic Fix your gaze on the ball, start running, and adjust your running speed so that the angle of gaze remains constant. α Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

21 The gaze heuristic Fix your gaze on the ball, start running, and adjust your running speed so that the angle of gaze remains constant. α α Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

22 The gaze heuristic Fix your gaze on the ball, start running, and adjust your running speed so that the angle of gaze remains constant. α α α Gigerenzer, G., & Brighton, H. (2009). Topics in Cognitive Science,

23 8 18 What are simple heuristics? Recall: Simple heuristics ignore information. Gaze heuristic: Ignore ball velocity, spin, wind direction, etc. Used by bats, birds, dragonflies, and dogs catching frisbees. Invokes a different information processing problem. Heuristics explain how we function in an uncertain world.

24 8 18 What are simple heuristics? Recall: Simple heuristics ignore information. Gaze heuristic: Ignore ball velocity, spin, wind direction, etc. Used by bats, birds, dragonflies, and dogs catching frisbees. Invokes a different information processing problem. Heuristics explain how we function in an uncertain world. Contrast: The study of heuristics in behavioral economics. Heuristics explain why people are biased, irrational. Heuristics are labels, not testable computational models.

25 Example heuristic: Take-the-best Gigerenzer, G. & Brighton, H. (2009). Topics in Cognitive Science,

26 Example heuristic: Take-the-best How should we rank pairs of objects on some criterion of interest? E.g., food items by calorific content; cities by population. Properties of the objects are described by probabilistic cues. Intercity trainline? University? Gigerenzer, G. & Brighton, H. (2009). Topics in Cognitive Science,

27 Example heuristic: Take-the-best How should we rank pairs of objects on some criterion of interest? E.g., food items by calorific content; cities by population. Properties of the objects are described by probabilistic cues. Intercity trainline? University? Take-the-Best ( ) orders cues by ignoring dependencies, and only use the first discriminating cue. vs. Model dependencies between cues, then weigh and add all cues ( ). Gigerenzer, G. & Brighton, H. (2009). Topics in Cognitive Science,

28 Example heuristic: Take-the-best How should we rank pairs of objects on some criterion of interest? E.g., food items by calorific content; cities by population. Properties of the objects are described by probabilistic cues. Intercity trainline? University? German City Population 0.75 Take-the-Best ( ) orders cues by ignoring dependencies, and only use the first discriminating cue. vs. Predictive accuracy Model dependencies between cues, then weigh and add all cues ( ) Take the Best Logistic Regression Tree Induction Nearest Neighbour Gigerenzer, G. & Brighton, H. (2009). Topics in Cognitive Science, Sample size (number of cities) 9 18

29 Example heuristic: Take-the-best How should we rank pairs of objects on some criterion of interest? E.g., food items by calorific content; cities by population. Properties of the objects are described by probabilistic cues. Intercity trainline? University? German City Population 0.75 Take-the-Best ( ) orders cues by ignoring dependencies, and only use the first discriminating cue. vs. Predictive accuracy Model dependencies between cues, then weigh and add all cues ( ) Take the Best Logistic Regression Take the Best Tree Induction Support Nearest Vector Neighbour Machine Gigerenzer, G. & Brighton, H. (2009). Topics in Cognitive Science, Sample size (number of cities) 9 18

30 Why is ignoring information beneficial? 10 18

31 Modelling London s mean daily temperature London's daily temperature in 2000 Days since 1st January, 2000 Temperature (F) Degree 25 polynomial Degree 3 polynomial

32 11 18 Modelling London s mean daily temperature London's daily temperature in 2000 Error in fitting vs. Error in predicting Temperature (F) Degree 25 polynomial Degree 3 polynomial Error Error in fitting the sample Error in predicting the population Days since 1st January, 2000 Degree of polynomial Fitting: Evaluate with the data used to parameterize the model. Prediction: Evaluate the model using novel observations.

33 The statistical foundations of ecological rationality Gigerenzer, G. & Brighton, H. (2009). Topics in Cognitive Science,

34 The statistical foundations of ecological rationality The prediction error of a statistical model can be decomposed: prediction error = (bias) 2 +variance+noise (1) Gigerenzer, G. & Brighton, H. (2009). Topics in Cognitive Science,

35 The statistical foundations of ecological rationality The prediction error of a statistical model can be decomposed: prediction error = (bias) 2 +variance+noise (1) Yielding two controllable components: 1. The bias component of prediction error reflects the inability of a model to represent the systematic patterns that govern the observations. 2. The variance component of prediction error reflects the sensitivity of the model s predictions to different observations of the same problem, such as a different sample from the same population. Gigerenzer, G. & Brighton, H. (2009). Topics in Cognitive Science,

36 The bias/variance dilemma Prediction error decomposed into bias and variance 500 bias 2 + variance bias 2 variance Error Degree of Polynomial O Sullivan, F. (1986). Statistical Science, 1; Geman et al. (1992). Neural Computation, 4.

37 The bias/variance dilemma Prediction error decomposed into bias and variance bias 2 + variance bias 2 variance Overly complex models that overfit tend to have low bias but high variance. Error Degree of Polynomial Overly simple models that underfit tend to have high bias but low variance. To achieve low error we need to strike a balance between the two. O Sullivan, F. (1986). Statistical Science, 1; Geman et al. (1992). Neural Computation,

38 The bias/variance dilemma Prediction error decomposed into bias and variance bias 2 + variance bias 2 variance Overly complex models that overfit tend to have low bias but high variance. Error Degree of Polynomial Overly simple models that underfit tend to have high bias but low variance. To achieve low error we need to strike a balance between the two. Dilemma: Methods for achieving low bias tend to increase variance, and methods for achieving low variance tend to increase bias. O Sullivan, F. (1986). Statistical Science, 1; Geman et al. (1992). Neural Computation,

39 Can macroeconomics benefit from the study of ecological rationality? 14 18

40 Example: The dog and the frisbee Haldane, A. G. & Madouros, V. (2012). Federal Reserve Bank of Kansas City s 366th Economic Policy Symposium. Brighton, H. & Gigerenzer, G. (2015). The bias bias. Journal of Business Research, 68(8).

41 Example: The dog and the frisbee To avoid financial crises, how should international banks be regulated? Haldane, A. G. & Madouros, V. (2012). Federal Reserve Bank of Kansas City s 366th Economic Policy Symposium. Brighton, H. & Gigerenzer, G. (2015). The bias bias. Journal of Business Research, 68(8).

42 Example: The dog and the frisbee To avoid financial crises, how should international banks be regulated? Complexity: Regulating banks is a bit like catching a frisbee. Haldane, A. G. & Madouros, V. (2012). Federal Reserve Bank of Kansas City s 366th Economic Policy Symposium. Brighton, H. & Gigerenzer, G. (2015). The bias bias. Journal of Business Research, 68(8).

43 Example: The dog and the frisbee To avoid financial crises, how should international banks be regulated? Complexity: Regulating banks is a bit like catching a frisbee. The response has been an increasingly complex set of regulations. Haldane, A. G. & Madouros, V. (2012). Federal Reserve Bank of Kansas City s 366th Economic Policy Symposium. Brighton, H. & Gigerenzer, G. (2015). The bias bias. Journal of Business Research, 68(8).

44 Example: The dog and the frisbee To avoid financial crises, how should international banks be regulated? Complexity: Regulating banks is a bit like catching a frisbee. The Basel Accord The response has been an increasingly complex set of regulations. This is an example of the bias bias. Predicting bank failure: simple does not just trump complex; it trumps the truth 30 pages Basel I (1988) 347 pages Basel II (2004) Haldane, A. G. & Madouros, V. (2012). Federal Reserve Bank of Kansas City s 366th Economic Policy Symposium. Brighton, H. & Gigerenzer, G. (2015). The bias bias. Journal of Business Research, 68(8). 616 pages Basel III (2010) 15 18

45 Faced with uncertainty, what are rational decisions?

46 16 18 Faced with uncertainty, what are rational decisions? 1. When we can quantify uncertainty: Mathematical statistics is king. Bayesian maximizers of expected utility are rational. Optimality is a meaningful concept. Bias is key (n ).

47 16 18 Faced with uncertainty, what are rational decisions? 1. When we can quantify uncertainty: Mathematical statistics is king. Bayesian maximizers of expected utility are rational. Optimality is a meaningful concept. Bias is key (n ). 2. When it is more rational to accept the unquantifiable: Ecological rationality and the study of simple heuristics. In the business of comparing the ability of competing algorithms to reduce uncertainty. Optimality is indeterminable. Variance is key (n 0).

48 The bias bias Brighton, H. & Gigerenzer, G. (2015). The bias bias. Journal of Business Research, 68(8)

49 The bias bias To suffer from the bias bias is to develop, deploy, or prefer models that are likely to achieve low bias, while simultaneously paying little or no attention to models with low variance. Brighton, H. & Gigerenzer, G. (2015). The bias bias. Journal of Business Research, 68(8)

50 The bias bias To suffer from the bias bias is to develop, deploy, or prefer models that are likely to achieve low bias, while simultaneously paying little or no attention to models with low variance. Symptom Relationship to the bias bias 1. Using goodness of fit to evaluate models Variance is irrelevant to achieving a good fit but crucial when predicting. 2. Draw conclusions from a single model The ability of a particular model to strike a good balance between reducing bias and variance is impossible to estimate. 3. Seek unbiased models By virtue of reducing more variance than they add bias, biased models can result in higher predictive accuracy. 4. Assume an accuracy effort tradeoff More information and computation will not always result in more accurate predictions. 5. Complex problems need complex solutions An example... Brighton, H. & Gigerenzer, G. (2015). The bias bias. Journal of Business Research, 68(8)

51 Decision making, AI, and big data Does big data allow us to sidestep the bias/variance dilemma?

52 18 18 Decision making, AI, and big data Does big data allow us to sidestep the bias/variance dilemma? German City Population Predictive accuracy Take the Best Logistic Regression Tree Induction Nearest Neighbour Sample size (number of cities)

53 18 18 Decision making, AI, and big data Does big data allow us to sidestep the bias/variance dilemma? German City Population No. Huge amounts of data invite us to consider harder problems Predictive accuracy Take the Best Logistic Regression Tree Induction Nearest Neighbour Sample size (number of cities)

54 18 18 Decision making, AI, and big data Does big data allow us to sidestep the bias/variance dilemma? German City Population No. Huge amounts of data invite us to consider harder problems Big data can increase uncertainty: Non-stationarity, dataset shift. Predictive accuracy Take the Best Logistic Regression Tree Induction Nearest Neighbour Sample size (number of cities)

55 18 18 Decision making, AI, and big data Does big data allow us to sidestep the bias/variance dilemma? German City Population No. Huge amounts of data invite us to consider harder problems Big data can increase uncertainty: Non-stationarity, dataset shift. Predictive accuracy Often, the big questions center on when n 0 rather than n Take the Best Logistic Regression Tree Induction Nearest Neighbour Sample size (number of cities)

56 18 18 Decision making, AI, and big data Does big data allow us to sidestep the bias/variance dilemma? Predictive accuracy German City Population Sample size (number of cities) Take the Best Logistic Regression Tree Induction Nearest Neighbour No. Huge amounts of data invite us to consider harder problems. Big data can increase uncertainty: Non-stationarity, dataset shift. Often, the big questions center on when n 0 rather than n. How many financial crises have we observed? How many photos of wild fungi do we need?

Gut Feelings: Short Cuts To Better Decision Making Gerd Gigerenzer

The heart has its reasons of which reason knows nothing Blaise Pascal Gut Feelings: Short Cuts To Better Decision Making Gerd Gigerenzer Max Planck Institute for Human Development Berlin An intuition is