Knowledge as Explanations: A Methodology for AB Simulations

Knowledge as Explanations: A Methodology for AB Simulations Marco Valente Lem, Pisa and University of L Aquila

Quotations One thing we are not going to have, now or ever, is a set of models that forecasts sudden falls in the value of financial assets, like the declines that followed the failure of Lehman Brothers in September. R.Lucas, 2009, The Economist, August 6th, 2009.

Quotations The same factors that may have reduced the probability of future systemic events [...] may amplify the damage caused by (and complicate the management of) very severe financial shocks. The changes that have reduced the vulnerability of the system to smaller shocks may have increased the severity of the large ones. T.Geithner, September 2006, Hedge Funds and Derivatives and Their Implications for the Financial System, Hong Kong.

Motivations Two (related) tendencies in economics are producing dangerous results, risking to make the field irrelevant.

Motivations Two (related) tendencies in economics are producing dangerous results, risking to make the field irrelevant. 1 Increasing specialization. Many sub-sectors where hyper-specialists decide their own goals and discuss only with fellow specialists. 2 Routine-based research. In worryingly many cases researchers apply established research methods with little regard to their appropriateness and overall implications.

Questions 1 How do you make research in social sciences using simulation models?

Questions 1 How do you make research in social sciences using simulation models? 2 How do you make research in social sciences?

Questions 1 How do you make research in social sciences using simulation models? 2 How do you make research in social sciences? 3 How do you make research?

Questions 1 How do you make research in social sciences using simulation models? 2 How do you make research in social sciences? 3 How do you make research? 4 Indeed, why do you make research?

Answers 4 The goal of science is to increase knowledge about real world phenomena.

Answers 4 The goal of science is to increase knowledge about real world phenomena. 3 Scientists generate knowledge by in providing explanations A µ B.

Answers 4 The goal of science is to increase knowledge about real world phenomena. 3 Scientists generate knowledge by in providing explanations A µ B. 2 Social sciences frequently concerns non-quantitative phenomena. 1 Simulation must be used to generate virtual emergent properties...

Outline Plan: Propose a general definition for knowledge compatible with scientific knowledge at large, for social sciences in particular, and derive consequences for use of simulations as research tool. Define formally knowledge as explanations: A µ B Derive useful consequences from the proposed definition.

Science s goals Scientific research aims at three goals:

Science s goals Scientific research aims at three goals: Interpret past events: how did system s state A managed to turn into B?

Science s goals Scientific research aims at three goals: Interpret past events: how did system s state A managed to turn into B? Predict future events: given that we have A, what state B can we expect in the future? Normative purposes: assuming we have A, how can we intervene to get a better B than naturally occurring?

Knowledge as explanations Definition: It comes then natural to define knowledge as the generalized concept of explanations: A µ B where A and B are the descriptions of two system s states and µ is an explanatory mechanism describing how A turns into B.

Science s goals This definition fits well the goals stated for science: Interpret past events:? µ B Predict future events: A? µ Normative purposes: A? B

Science s goals Note that time is not the only logical dimension for explanations; structural/organizational/aggregative dimensions need to be explained, too. What micro-conditions give rise to the observed macro-properties:? µ B What macro-properties can be expected from these micro-conditions: A µ? How can we twist available micro-conditions to reach desired macro-properties: A? B As we will see, explanations are a mix of time- and aggregative-based explanations.

Knowledge as explanations Notice the difference between cause and explanation. The former relates to objective properties of reality, the second to subjective interpretations. Our undertaking is logical rather than ontological. That is, we wish to be able to assert: The sentence A stands in a causal relation to the sentence B (in a specified system of sentences) ; and not: That which is denoted by sentence A causes that which is denoted by sentence B. Simon, H. A., 1952, On the Definition of the Causal Relation, The Journal of Philosophy, 49(16), pp. 517-528.

Knowledge as explanations The proposed definition is highly general and may also be quite vague (later more on the assessment of knowledge). Basically, any form of IF... THEN..., complemented by some explanatory mechanism. Examples are: Theorems. Assumptions A are developed through simple passages µ providing the theorem s claim B Popular wisdom. Red sunset sky brings fine weather (no explanation given) Expertise and practices. To reach objective B given the conditions A apply the steps µ

Supporting knowledge as A µ B We can provide a few hints to support the proposal: Evidence on how human memory works and past events are re-collected.

Supporting knowledge as A µ B We can provide a few hints to support the proposal: Evidence on how human memory works and past events are re-collected. Brain works through flows of signals. Reasoning is ultimatily composed by signals fired through connections. Evolutionary (i.e. gradual) advantage for knowledge-cumulating by linking events and inferring consequences.

Information and Knowledge The definition of knowledge implies also what is not knowledge: pure information, void of any context. For example, the description of a system, if not included in an explanation, does not constitute knowledge. A description is necessarily arbitrary, ignoring some aspect and focusing on few ones. The choice of how to describe a state cannot be independent of how the description is used.

Properties A µ B The proposed definition provides some useful indications to classify research strategies. Since researchers aim at increasing knowledge, we need to consider how it is possible to generate new explanations. There is a surprisingly small number of possible knowledge-increasing strategies, which may possibly be combined together.

Properties A µ B Knowledge refinement: An existing explanation, proved correct on a number of cases, may be proven wrong by a counter-example. This may be trigger the refining of the definition of states included in the original (too) general explanation, or to change the explanatory mechanism, or simply to throw away the whole piece of knowledge. A µ B

Properties A µ B Knowledge deepening: Knowledge is never ultimately specified. You can always request a further specification of how µ is actually working, investigating intermediate steps and their explanations, potentially diving into infinitely more detailed explanations: A µ B

Properties A µ B Knowledge extension: Knowledge can be put together extending existing pieces of knowledge to form chains of explanations, linking the final state of an explanation to the initial state of the next: A µ B

Implications for research strategy Research projects should therefore always state explicitly which class(es) of knowledge they pursue:

Implications for research strategy Research projects should therefore always state explicitly which class(es) of knowledge they pursue: Refine: change a currently established conviction.

Implications for research strategy Research projects should therefore always state explicitly which class(es) of knowledge they pursue: Refine: change a currently established conviction. Deepen: investigate elementary components of an accepted explanation.

Implications for research strategy Any research project should undergo the following steps: Start from an established piece of knowledge A µ B

Implications for research strategy Any research project should undergo the following steps: Start from an established piece of knowledge A µ B State a conjecture A µ B which is relevant and novel.

Implications for research strategy Any research project should undergo the following steps: Start from an established piece of knowledge A µ B State a conjecture A µ B which is relevant and novel. By refining/deepening/extending A µ B obtain A µ B, or possibly a different revised explanation.

Components of explanations Though the explanatory mechanism can take any form, there are only three broad, but distinct, classes of components of explanations:

Components of explanations Though the explanatory mechanism can take any form, there are only three broad, but distinct, classes of components of explanations: Temporal: the explanation consists in the passing of time during which the elements comprising the state of the world A can be expected to produce B. Aggregative: elements at a certain level of aggregation explain the properties of elements at a different levels. Quantitative relations (time independent) are included in this category.

Components of explanations (Anticipation) Mathematical relations are a peculiar (and very powerful) form of aggregative explanation. As long as we need only quantitative relations, mathematics is an un-matched research tool. However, many phenomena are intrinsically (irreversibly) dynamic and concerns complex interactions between aggregates showing properties distinct from those of their components. In this case we need a research tool able to deal with irreversible time and categorization.

Example 1 Knowledge depends on the purpose, and is always open to deepening. Newton s gravitational law has been proved (slightly) wrong by Einstein s relativity. Still, to send rockets on the moon you use the wrong theory. Besides, the explanatory mechanism of the gravitational force is still the object of huge, and so far unsuccessful, research efforts.

Example 2 1 a population in a certain area is found more resistant than its neighbours to a certain plague (initial, conjectural, explanation);

Example 2 1 a population in a certain area is found more resistant than its neighbours to a certain plague (initial, conjectural, explanation); 2 a few further populations with the same properties are found, but only members of a certain age/sex/type are shown to enjoy the property (refine); 3 blood samples identify unusual anti-bodies responsible for the increased resistance (deepening); 4 further studies discover that the special anti-bodies are the side-effect of specific mutation induced by a particular environment and leads to specific behaviours (extending to prior and subsequent explanations);

Research projects A research project is made of several steps, each concerned with a specific goal, all linked by a common perspective. Each piece of knowledge, an explanation, needs an assessment criterion: how reliable is the explanation to perform its claimed role.

Assessing knowledge We call a model an implementation of a piece of knowledge, where symbols and their relations express the knowledge content. The assessment of a model can be divided in three phases: 1 Descriptive relevance (abstraction). The choice of the symbolic representation for the reality of interest. 2 Internal consistency (verification). Manipulation of symbols to generate conclusions from observation and assumptions. 3 Results relevance (validation). The assessment of the results as compared to empirical observations.

Assessment stages

Assessing knowledge No universal objective criterion can ever be devised to perfectly assess a model.

Assessing knowledge No universal objective criterion can ever be devised to perfectly assess a model. 1 Descriptive relevance (abstraction). The choice of the symbolic representation can always be questioned in principle. Subjective. 2 Internal consistency (verification). The derivation of the consequences from the assumptions can be objectively evaluated. Objective 3 Results relevance (validation). Results adherence to reality is potentially questionable, as any piece of reality is context- and history-dependent. Subjective

Explanation assessment How to assess a proposed model depends on the nature of the phenomenon under study. There are two classes of phenomena: Quantitative: phenomena defined by a constant vector of variables and explained by mathematical operations. Y = f( X). Qualitative: phenomena defined by a varying number of non-quantitative aspects logically, but not quantitatively, explained. A 0 µ A1...

Quantitative phenomena When a phenomenon is fully represented by a series of observed values Y, X, the assessment of a model Y = f( X) consists in testing the adequacy of the proposed explanation f(...) by comparing its adequacy to represent Y = f( X ). The proposed explanation X f(...) Y can then be evaluated by mere error minimization.

Qualitative phenomena Qualitative phenomena consist in dynamics of entities that cannot be represented by a fixed and constant set of variables, and/or the (varying) variables. The elements of the system, though clearly identifiable, change not only in the intensity of certain measures, but also the very relevant measures.

Qualitative phenomena

Qualitative phenomena Quantitative aspects for qualitative phenomena do exist, in general, and are highly relevant. However, these measures are not the phenomenon, but only proxies and partial representations of the actual phenomenon, whose significance goes beyond the quantitative values alone. On the contrary, quantitative phenomena are made of the very measures. Compare, e.g., the series of GDP levels for a country and the sequence of the speed of a stone in free fall.

Qualitative phenomena Examples of qualitative phenomena may be: Physics: Open chaotic systems; quantum phenomena. Biology: Evolution of organisms; physiological properties. Economics: Development; product/market life-cycle

Qualitative phenomena General features for qualitative phenomena are: Varying number of variables/measures relevant to describe entities involved.

Qualitative phenomena General features for qualitative phenomena are: Varying number of variables/measures relevant to describe entities involved. Heavily influenced by a non-neutral environment.

Qualitative phenomena General features for qualitative phenomena are: Varying number of variables/measures relevant to describe entities involved. Heavily influenced by a non-neutral environment. As a consequence, qualitative phenomena are: Highly sensitive to small deviations in initial conditions.

Assessment of qualitative phenomena To evaluate a model proposed to represent a qualitative phenomenon we then need a different methodology from mere quantitative fitness with reality.

Assessment of qualitative phenomena To evaluate a model proposed to represent a qualitative phenomenon we then need a different methodology from mere quantitative fitness with reality. Simulation models (ABM) can be research tool. Crucially, the model must not be used to replicate reality, but to provide explanations, and the assessment must not be based on data fitting, but on the novelty and relevance of these explanations.

(AB) Simulation models Simulation models can be used to represent complex dynamics with the goal to study properties of complex theoretical systems in order to better understand real ones. This use of simulations consists in generating emergent properties and explaining them.

(AB) Simulation models Emergent properties are aggregate entities created by the loose coordination of weakly interacting component entities. The aggregate entity shows properties that, though obviously due its component entities, are different in nature from those of the components.

(AB) Simulation models A simulation model can be defined by an initial state A 0 and a generic transition rule µ t. The state can include a large number of variables and the transition rule can be any computable algorithm on these variables. A simulation runs generates: µ A 1 µ 0 2 µ t µ A1 A2...A t 1 At...A T T 1 AT

(AB) Simulation models These models do not aim at replicating specific quantitative features of the system, but rather to show qualitative similarities, such as, for example, exponential growth, cyclicality or any other dynamic and/or static property. These properties can be expressed as dynamic semi-coordination of low level entities spanning a longer time range than that required to represent smaller components.

(AB) Simulation models We indicate the high level properties as B 1, B 2,..., B TB, interpreting the simulation results from the perspective of these properties: B 1 {}}{ ν 2 ν {}}{ 3 µ... µ... µ... µ... µ... µ... A 0 AtB1 AtB2 ν TB A tb1 +1 B TB B2 {}}{ µ... µ... µ... AT A tbt 1 +1

(AB) Simulation models We can consider a AB model as validated if the simulated B τ are similar to those observed in the reality of interest, refer to them as B τ.

(AB) Simulation models We can consider a AB model as validated if the simulated B τ are similar to those observed in the reality of interest, refer to them as B τ. The validation procedure generally does not need to be quantitatively assessed, since it refers only to high-level and general properties. This type of validation is not, however, conclusive. Rather, it is only a pre-requisite to permit the subsequent methodological step: verification of how these properties are generated by the model.

(AB) Simulation models Given the nature of AB models implemented as computer programs, it must be possible to investigate the state of the model at every detail desired, both in static and dynamic terms. We can therefore identify, with certainty and objectivity, the motivation for the model elements A t,µ t to give rise to the high-level properties B τ,ν τ.

(AB) Simulation models The explanations linking the model explicit values to the high-level properties are any combinations of what the program can express: any form of data structure; irreversible time; any expressable form of codification. That is, the full set of potential components of knowledge. Calling λ the explanations, we have then that: A i,µ i λ Bj,ν j, with i = 1, 2,..., T and j = 1, 2,...T B.

(AB) Simulation models The λ is, in our terms, the explanatory mechanism linking the assumption of the model (its elements and dynamics) to the high-level properties observed. λ can objectively be assessed because of the formalism used: computer programs can be easily tested to generate, or not, certain results by means of certain methods. λ is the knowledge produced by the simulation model: the logical links between the emergent property and its (assumed) components.

(AB) Simulation models The knowledge represented by λ is knowledge concerning simulated data, an artificial system. The assessment of this knowledge requires the comparison of the conjectured explanation λ with that provided in the real world: A i,µ i λ B j,ν j, where A i,µ i, B j,ν j are empirical observations.

(AB) Simulation models The validation of this type of knowledge is necessarily subjective and the evaluation is always comparative, as any type of evaluation. Essentially, it is based on the comparison of how good are two potential and different explanations λ and a λ in justifying a given evidence. One of the explanations will surely be more convincing than the other.

(AB) Simulation models Note that obtaining high level properties B j,ν j resembling empirical ones Bj,ν j can generally be done with many different algorithms.

(AB) Simulation models Note that obtaining high level properties B j,ν j resembling empirical ones Bj,ν j can generally be done with many different algorithms. The difficulty stems from producing those emergent properties using realistic assumptions on the elementary components, A t,µ t. In any case, a model that solely shows that B j s emerge from A i is useless, since it is the explanation linking the two that completes the knowledge.

(AB) Simulation models The assessment of an explanation to correctly interpret observed events is generally rather easy, albeit (necessarily) subjective. In (the rare) case two competing and different explanations were proposed for the same phenomenon, it is always possible to identify specific differences in the patterns required for the two explanations to reach their conclusions

(AB) Simulation models Suppose λ 1 and λ 2 were two different explanations compatible with all existence evidence A i,µ i, B j,ν j. The very difference(s) between the two alternatives necessarily implies that some feature of the implied systems should also differ. A i,µ λ 1 i Bj,ν j, C 1, or A i,µ λ 2 i Bj,ν j, C 2, Choosing the differences C 1 /C 2 easy to observe it is possible to solve the potential indeterminacy.

Summary for AB simulations 1 Model writing: define the data structure, its initial values and its dynamics A 0, µ t.

Summary for AB simulations 1 Model writing: define the data structure, its initial values and its dynamics A 0, µ t. 2 Analysis of results: ensure that the final results of the sequence A 1, A 2,..., A T can be interpreted as emergent properties B 1, B 2, B T linked by their explanatory mechanism ν t. 3 Debug: investigate the motivations λ for the simulated emergent properties on the basis of the model A t,µ t λ Bt,ν t. 4 Document: validate the purported explanation by finding empirical support A t,µ λ t Bt,ν t.

Model writing The writing of the code for a model is only one step. Though it entails writing in a programming language, the code is generally trivial if decomposed in small parts for several variables. In a few weeks even non-programmers can learn the commands to write basic operations: arithmetic (+,-, etc.), logic (IF, AND, etc.), flow control (FOR... NEXT), add/remove entities. More difficult, but also highly rewarding, is the determination of the appropriate data structure. E.g., should my FIRM s be part of MARKET, or should MARKET be a component for FIRM s?

Analysis of Results Analysis of results entails basic statistics, graphs, scatter plots, etc. showing the overall results of the model, both through time and across entities. Data produced by simulations can be difficult to treat because of there massive size. Speed of analysis is necessary to allow for prompt modification of the model.

Debugging In programming, to debug refers to identify which part of the code generates unexpected results. This entails proceeding line-by-line until identifying the faulty code. In modeling, the unexpected result may be the relevant consequences one is looking for. Debug is crucial not only to clean up errors, but also to identify the explanation linking the model to the relevant result.

Documenting Appropriate documentation provides support of real-world examples of the cases represented by the model. Available evidence can support almost any claim, and its opposite. Adequate documentation of real-world cases shows how to improve the model or delimit its relevance

Refine In order to build even moderately complex models it is necessary to proceed gradually to avoid complexity traps. Knowledge refining: qualify a general function or a general system structure in more detailed ways. Knowledge deepening: endogenize elements of an existing model, for example turning a variable representing a demand function into a model of consumers.

Sources of knowledge Let s see how the use of simulation models help in devising interesting explanations. There are several stages of the use of a simulation model that may be relevant.

Sources of knowledge Learning by coding: the need to represent computationally a system forces to think how real systems actually work. A model is a simulated universe, with the same logical constraints as the real one. Apparently obvious representations turn out to be inconsistent or incomplete once (trying to) put them into code. Finding the right way to express with a formal language a synthesis of a real system provides a lot of insights on the system itself.

Sources of knowledge Learning by plotting: analysing the results (across time and/or different initializations) provides evidence of the model s behaviour. Controlling that your model produces realistic phenomena generates new understanding of the real cases, their characteristics and other properties non-obvious from the observation of the real world.

Sources of knowledge Learning by debugging: linking logically the model content with the results produced by simulations provides surprising insight, showing non-obvious consequences of the model design and implementation. These explanations embody the ultimate knowledge on the simulated system, hopefully valid for real ones, too.

Sources of knowledge Learning by erring: Errors in simulation models design reflect erroneous concepts. Generating errors shows all the consequences of the original faults, and frequently suggests solutions and improvements.

People say that... Simulation models as scientific research tools are hotly debated, mostly because there is no established protocol for their evaluation. In the following we list some of the most frequently points raised in the debate, and report on the answers derived by the approach proposed above.

People say that... Sim. models produce all and only whatever you code into them.

People say that... Sim. models produce all and only whatever you code into them. True, but computers help to understand exactly the implications of the assumptions. Think, for example, of models of weather forecasting. The basic physics is trivial, but the aggregate effect is impossible to derive by analytical means, and computers help to fill the gap between the hypotheses (e.g. basic physics) and their implications (forecasting).

People say that... Random models/models with many parameters must be adequately tested for the robustness of results.

People say that... Random models/models with many parameters must be adequately tested for the robustness of results. False. Robustness in respect of randomness or initial conditions are relevant to assess universal properties of systems. Complex systems, as real economic systems, may potentially assume a huge number of states, but almost all of them will never be realized. We need to explain phenomena, and possibly the most relevant ones are the very rare. Simulation models can frequently be understood by studying the exceptions, as much as doctors understand how the human body works by studying sick patients.

People say that... Better one theorem than a hundred simulations.

People say that... Better one theorem than a hundred simulations. True. Mathematics is a wonder of compactness, representing many (infinite) cases by one single line of symbols. Moreover, mathematical representations allow to deduce many non-obvious properties. However, non-linear aggregations and dynamics are poorly managed by mathematical tools, while computers can easily do both: simulations can replicate (though inelegantly) results embodied in theorems, but, in general, theorems cannot contain the knowledge provided by models.

People say that... Your model does not consider X, which is relevant for phenomenon Y.

People say that... Your model does not consider X, which is relevant for phenomenon Y. True. Any model is necessarily limited and partial. Models for open systems in particular neglect aspects that, by definition, are relevant for every phenomenon. A model must be always considered under strong ceteris paribus conditions, and the explanations produced are conditional to other explanations reinforcing or hindering the ones considered.

People say that... Programming simulations is difficult, don t want to waste my time learning how to program.

People say that... Programming simulations is difficult, don t want to waste my time learning how to program. False. Writing programs is difficult, writing simulations model is not, at least if you adopt a cautious approach. Most of simulation models code is made of trivial, IF-THEN-ELSE statements and simple arithmetic operations. Arranging the model s code in a computer program is very difficult, particularly because you need sophisticated interfaces to investigate model s behaviour. However, endowing a model with the necessary interfaces is a technical problem, that technicians can help to solve.

People say that... Are your results confirmed empirical observations?

People say that... Are your results confirmed empirical observations? Futile. Data replication is useless without understanding their meaning. And having such knowledge you don t need to replicate any data. Worse, there are always a large number of different ways to replicate a data set, only some of which may make sense.

People say that... Then, for you empirical analysis is useless...

People say that... Then, for you empirical analysis is useless... False. Statistical analysis, descriptive or inferential, provides the crucial information on what is going on in such complex entities like economic system. We need constantly to take into account that quantitative measures are a very fuzzy shadow of the systems that produced them.

Conclusions 1 Knowledge takes the form of explanations A µ B.

Conclusions 1 Knowledge takes the form of explanations A µ B. 2 Knowledge-as-explanations provides a rich series of insights on how to conduct research projects

Conclusions 1 Knowledge takes the form of explanations A µ B. 2 Knowledge-as-explanations provides a rich series of insights on how to conduct research projects 3 Research on qualitative phenomena can makes use of AB models as a tool. 4 The methodology for AB models consists in re-creating emergent properties on a computer and explaining them.

L.Wittgenstein (revised) 7. Whereof one cannot code, thereof one must pass over in silence.