Substance-Blind Classification of Evidence for Intelligence Analysis

Transcription

1 In Proceedings of the Conference Ontology for the Intelligence Community: Setting the Stage for High-level Knowledge Fusion, George Mason University, Fairfax, Virginia Campus, October Substance-Blind Classification of Evidence for Intelligence Analysis David Schum, Gheorghe Tecuci, Mihai Boicu and Dorin Marcu Abstract Intelligence analysis requires the development of arguments that link to hypotheses by establishing and fusing the relevance, believability and inferential force or weight of a wide variety of items of of different types. This paper presents several substance-blind classifications of which are based on these inferential characteristics and facilitate the clarification of many uncertainties lurking in intelligence analysis. It also shows how the Disciple-LTA cognitive assistant uses these classifications to develop Wigmorean probabilistic inference networks for assessing the likelihood of hypotheses. Index Terms classification, relevance, believability, inferential force, Wigmorean networks, cognitive assistant, ontology, -based hypothesis analysis, high-level fusion I. WHY IS A SUBSTANCE-BLIND CLASSIFICATION OF EVIDENCE NEEDED? 'Evidence' is word of relation used in the context of argumentation: e.g. "A is of B". In that context information has a potential role as relevant if it tends to support or tends to negate, directly or indirectly, some hypothesis about a contested matter. One draws inferences from in order to prove or disprove a hypothesis. The framework is argument, the process is proof, and the engine is inferential reasoning from information [1]. Thus differs from the words data or items of information, since data or items of information only become when their relevance is established regarding some hypothesis at issue. The term must also be distinguished from the term fact. We may all agree that it is a fact that we have about event E. But whether it is a fact that event E did occur is another matter since we have questions about the credibility of the source of this. This makes it necessary to distinguish between for an event and the event itself. Evidence can be any species of proof consisting of tangible items such as objects, This work was supported in part by several U.S. Government organizations, including the Air Force Office of Scientific Research (AFOSR, FA ), the National Science Foundation (NSF, ), and the Department of Defense (DOD). The US Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official polices or endorsements, either expressed or implied, of AFOSR, NSF, DOD or the U.S. Government. Dr. David Schum is Professor in the Volgenau School of Information Technology and Engineering (VS IT&E) and in the School of Law at George Mason University (GMU), Fairfax VA 22030, USA. (phone: ; fax: ; dschum@gmu.edu). Dr. Gheorghe Tecuci is Professor of Computer Science in the VS IT&E and Director of the GMU Learning Agents Center ( tecuci@gmu.edu). Dr. Mihai Boicu is Assistant Professor of Applied IT and Associate Director of the GMU Learning Agents Center ( mboicu@gmu.edu). Dr. Dorin Marcu is Research Assistant Professor in the GMU Learning Agents Center (dmarcu@gmu.edu). documents, images, and records of any kind, or testimony from human sources or witnesses. Evidence may have any possible substance or content. Therefore, attempts to categorize it in terms of its substance or content would be an endless and fruitless task. Why should anyone wish to be able to categorize? First, it is often necessary to compare the force or weight of different lines of argument based on different in a particular analysis. For example, here is a line of argument based on HUMINT ; how does this argument compare with a different line of argument based on IMINT or one based on MASINT? Second, there are different uncertainty issues that arise when we have different kinds of. Third, how does the strength of our conclusions in one analysis compare with those reached in another analysis, given the fact that these two analyses are based on entirely different mixtures of? Fourth, how will we ever resolve differences among analysts themselves, or among analysts and their "customers", regarding interpretations of forming the basis for conclusions reached in an analysis? Finally, how do we ever say anything general about given that its substance or content varies in a near infinite fashion? What is badly needed in so many situations is an categorization scheme for allowing us to say what kinds of we have without resorting to discussions about its substance or content. In this paper we present a foundation for such an categorization scheme that will tell us what kinds and combinations of we have in any intelligence analysis regardless of the substance or content of the and the objectives of the analysis. First, we present a general approach to -based hypothesis analysis which consists in developing a Wigmorean probabilistic inference network that shows how is linked to a hypothesis through a potentially very complex argument that establishes and fuses the relevance, the believability and the inferential force or weight of a wide variety of items of of different types [2, 3]. Then we present three substance-blind classifications of, one based on believability, one on relevance and one on inferential force, which support the development of Wigmorean networks for hypotheses analysis. This approach to hypothesis analysis and the substance-blind classifications of are implemented in Disciple-LTA, an analyst s cognitive assistant that can learn complex analytic expertise directly from expert analysts, can support analysts in hypothesis analysis, collaboration and sharing of intelligence, and can teach its analytic expertise to new analysts [4, 5, 6]. II. WIGMOREAN NETWORKS Disciple-LTA assists an analyst in assessing the likelihood of various hypotheses, such as Al Qaeda has nuclear wea 1

2 pons or The United States will be the world leader in nonconventional energy sources within the next 10 year or Iran is pursuing nuclear power for peaceful purposes [4]. This is accomplished by developing an argument in the form of an Wigmorean inference networks, through the use of a general problem-reduction/solution-synthesis reasoning approach which is illustrated in Fig. 1 and discussed in the following. A complex hypothesis is first reduced to simpler and simpler hypotheses and the simplest hypotheses are assessed through analysis. For example, in Fig. 1, the hypothesis H 1 (or problem [P1]) is reduced to three simpler hypotheses, H 11, H 12, and H 13 (problems [P2], [P3] and [P4]). Each of these hypotheses is assessed by considering both favoring and disfavoring (i.e., problems [P5] and [P6]). Let us assume that there are two items of favoring for H 11 : E 1 and E 2. For each of them (e.g., E 1 ) Disciple- LTA assesses the extent to which it favors the hypothesis H 11 (i.e., [P7]). This requires assessing both the relevance of E 1 to H 11 (problem [P9]) and the believability of E 1 (problem [P10]). Let us assume that Disciple-LTA has obtained the following solutions for these two last problems: If we believe E1 then H11 is almost certain. It is likely that E1 is true. In this example, almost certain and likely are symbolic probabilities for likelihood, based on the DNI s standard estimative language. By compositing the solutions [S9] and [S10] (e.g., through a min function) Disciple-LTA assesses the inferential force or weight of E 1 on H 11 : Based on E1 it is likely that H11 is true. [S7] Similarly Disciple-LTA assesses the inferential force or weight of E 2 on H 11 : Based on E2 it is almost certain that H11 is true. [S8] By composing the solutions [S7] and [S8] (e.g., through a max function) Disciple-LTA assesses the inferential force/weight of the favoring (i.e., E 1 and E 2 ) on H 11 : Based on the favoring it is almost certain that H11 is true. Through a similar process Disciple-LTA assesses the disfavoring for H 11 : Based on the disfavoring it is unlikely that H11 is false. [P1] Assess H 1 It is likely that H is true [S1] [P2] [S2] [P3] [S3] [P4] [S4] Assess It is almost certain Assess It is likely that Assess It is likely that H 11 that H 11 is true H 12 H 12 is true H 13 H 13 is true Inferential force of on H 1 [P5] [S5] [P6] [S6] Assess the favoring for H 11 Based on the favoring it is almost certain that H 11 is true Assess the disfavoring for H 11 Based on the disfavoring it is unlikely that H 11 is false Inferential force of favoring on H 1 [P7] [S7] [P8] [S8] Assess the extent to Based on E 1 it is Assess the extent to Based on E 2 it is almost which E 1 favors H 11 likely that H 11 is true which E 2 favors H 11 certain that H 11 is true Inferential force of E 1 on H 1 [P9] [S9] [P10] [S10] the relevance of E 1 to H 11 Assess If we 1 H 11 is almost certain believe E then believability of E 1 Assess the It is E 1 is true likely that Fig. 1. Wigmorean inference network for hypothesis assessment generated by Disciple-LTA. Because there is very strong favoring H 11 and there is weak disfavoring H 11, Disciple-LTA concludes: It is almost certain that H11 is true. The sub-hypotheses H 12 and H 13 are assessed in a similar way: It is likely that H12 is true. It is likely that H13 is true. The solutions of H 11, H 12 and H 13 are composed (e.g., through average ) into the -based assessment of H 1 : It is likely that H1 is true. A concrete example of such a Wigmorean network generated by Disciple-LTA is shown in Fig. 4. III. CLASSIFICATION OF EVIDENCE BASED ON BELIEVABILITY In the previous section we have discussed the process of -based hypothesis assessment down to the level where one has to assess the relevance and the believability of an item of. In this section we discuss how Disciple- LTA and its user assess the believability of an item of by using a substance-blind classification of. Here is an important question we are asked to answer regarding the individual kinds of we have: How do you, the analyst, stand in relation to this item of? Can you examine it for yourself to see what events it might reveal? If you can, we say that the is tangible in nature. But suppose instead you must rely upon other persons, assets, or informants, to tell you about events of interest. Their reports to you about these events are examples of. Fig. 2 shows a substance-blind classification of based on its believability credentials. A. Tangible Evidence There is an assortment of tangible items we might encounter and that could be examined by an intelligence analyst. Both IMINT and SIGINT provide various kinds of sensor records and images that can be examined. MASINT and TECHINT provide various objects such as soil samples and weapons that can be examined. COMINT can provide audio recordings of communications that can be overheard and translated if the communication has occurred in a foreign language. Documents, tabled measurements, charts, maps and diagrams or plans of various kinds are also tangible. There are two different kinds of tangible : real tangible and demonstrative tangible Real tangible is a thing itself and has only one major believability attribute: authenticity. Is this object what it is represented as being or is claimed to be? There are as many ways of generating deceptive and inauthentic as there are persons wishing to generate it. Documents or written communications may be faked, captured weapons may have been altered, and photographs may have been altered in various ways. One problem is that it usually requires considerable expertise to detect inauthentic. Demonstrative tangible does not concern things themselves but only representations or illustrations of these things. Ex- 2

3 amples include diagrams, maps, scale models, statistical or other tabled measurements, and sensor images or records of various sorts such as IMINT, SIGINT, and COMINT. Demonstrative tangible has three believability attributes. The first concerns its authenticity. For example, suppose we obtain a hand drawn map from a captured insurgent showing the locations of various groups in his insurgency organization. Has this map been deliberately contrived to mislead our military forces or is it a genuine representation of the location of these insurgency groups? The second believability attribute is accuracy of the representation provided by the demonstrative tangible item. The accuracy question concerns the extent to which the device that produced the representation of the real tangible item had a degree of sensitivity (resolving power or accuracy) that allows us to tell what events were observed. We would be as concerned about the accuracy of the hand-drawn map allegedly showing insurgent groups locations as we would about the accuracy of a sensor in detecting traces of some physical occurrence. Different sensors have different resolving power that also depends on various settings of their physical parameters (e.g., the settings of a camera). The third major attribute, reliability, is especially relevant to various forms of sensors that provide us with many forms of demonstrative tangible. A system, sensor, or test of any kind is reliable to the extent that the results it provides are repeatable or consistent. You say that a sensing device is reliable if it would provide the same image or report on successive occasions on which this device is used. B. Testimonial Evidence For we have two basic sources of uncertainty: competence and credibility. This is one reason why it is more appropriate to talk about the believability of which is a broader concept that includes both competence and credibility considerations. The first question to ask related to competence is whether this source actually made the observation he claims to have made or had access to the information he reports. The second competence question concerns whether this source understood what was being observed well enough to provide us with an intelligible account of what was observed. Thus competence involves access and understandability. Assessments of human source credibility require consideration of entirely different attributes: veracity (or truthfulness), objectivity, and observational sensitivity under the conditions of observation. Here is an account of why these are the major attributes of credibility. First, is this source telling us about an event he/she believes to have occurred? This source would be untruthful if he/she did not believe the reported event actually occurred. So, this question involves the source's veracity. The second question involves the source's objectivity. The question is: Did this source base a belief on sensory received during an observation, or did this source believe the reported event occurred either because this source expected or wished it to occur? An objective observer is one who bases a belief on the basis of sensory instead of desires or expectations. Finally, if the source did base a belief on sensory, how good was this? 3 This involves information about the source's relevant sensory capabilities and the conditions under which a relevant observation was made. As indicated in Fig. 2, there are several types of. If the source does not hedge or equivocate about what he/she observed (i.e., the source reports that he/she is certain that the event did occur), then we have unequivocal. If, however, the source hedges or equivocate in any way (e.g., "I'm fairly sure that E occurred") then we have equivocal. The first question we would ask this source of unequivocal is: How did you obtain information about what you have just reported? It seems that this source has three possible answers to this question. The first answer is: "I made a direct observation myself. In this case we have unequivocal based upon direct observation. The second possible answer is: "I did not observe this event myself but heard about its occurrence (or nonoccurrence) from another person". Here we have a case of secondhand or hearsay, called unequivocal obtained at second hand. A third answer is possible: "I did not observe event E myself nor did I hear about it from another source. But I did observe events C and D and inferred from them that event E definitely occurred". This is called based on opinion and it requires some very difficult questions. The first concerns the source's credibility as far as his/her observation of event C and D; the second involves our examination of whether we ourselves would infer E based on events C and D. This matter involves our assessment of the source's reasoning ability. It might well be the case that we do not question this source's credibility in observing events C and D, but we question the conclusion that event E occurred the source has drawn from his observations. We would also question the certainty with which the source has reported an opinion that E occurred. Despite the source s conclusion that event E definitely occurred", we should consider that based on opinion is a type of equivocal. There are two other types of equivocal. The first we call completely equivocal. Asked whether event E occurred or did not, our source says: "I don't know", or "I can't remember". But there is another way a source of HUMINT can equivocate; the source can provide probabilistically equivocal in various ways: "I'm 60 percent sure that event E happened"; or "I'm fairly sure that E occurred ; or "It is very unlikely that E occurred". We could look upon this real tangible tangible demonstrative tangible unequivocal based upon direct observation unequivocal unequivocal obtained at second hand equivocal based on opinion missing completely equivocal Fig. 2. Evidence classification based on believability. authoritative record probabilistically equivocal

4 particular probabilistic equivocation as an assessment by the source of his own observational sensitivity. C. Missing Evidence To say that is missing entails that we must have had some basis for expecting we could obtain it. There are some important sources of uncertainty as far as missing is concerned. In certain situations missing can itself be. Consider some form of tangible, such as a document, that we have been unable to obtain. There are several reasons for our inability to find it, some of which are more important than others. First, it is possible that this tangible item never existed in the first place; our expectation that it existed was wrong. Second, the tangible item exists but we have simply been looking in the wrong places for it. Third, the tangible item existed at one time but has been destroyed or misplaced. Fourth, the tangible item exists but someone is keeping it from us. This fourth consideration has some very important inferential implications including denial and possibly deception. An adverse inference can be drawn from someone's failure to produce. D. Accepted Facts There is one final category of about which we would never be obliged to assess its believability. Tabled information of various sorts such as tide table, celestial tables, tables of physical or mathematical results such as probabilities associated with statistical calculations, and many other tables of information we would accept as being believable provided that we used these tables correctly. For example, an analyst would not be obliged to prove that temperatures in Iraq can be around 120 degrees Fahrenheit in summer months, or that the population of Baghdad is greater than that of Basra. E. Mixed Evidence We have just considered a categorization of individual items of but there are situations in which individual items can reveal various mixtures of these types of. One example involves a tangible document containing a assertion based on other alleged tangible. Thus these forms of are not mutually exclusive; they can occur together in a single item of. F. Believability Assessment with Disciple-LTA Disciple-LTA knows about the types of shown in Fig. 2 and how their believability should be evaluated. For example, Fig. 3 shows the reasoning tree automatically generated by Disciple-LTA for solving the problem: Assess the extent to which one can believe Osama bin Laden as the source of EVD-Dawn-Mir01-01c. Notice that, in accordance with the above discussion, Disciple-LTA reduces the believability of this testimony of Osama bin Laden to two simpler problems, one for assessing the competence of Osama bin Laden, and the other for assessing his credibility. This second problem is further reduced to assessing bin Laden s veracity, objectivity and observational sensitivity. Disciple-LTA may have knowledge about these believability characteristics of Osama bin Laden (e.g., that his veracity is an even chance). Alternatively, the analyst may make assumptions with respect to the values of these characteristics. In any case, once the solutions of the simplest problems are obtained, they are combined, from bottom up, to assess the believability of Osama bin Laden. For example, the probabilistic estimates of bin Laden s veracity, objectivity and observational sensitivity (i.e., an even chance, almost certain, and almost certain, respectively) are combined (through a min function) to obtain a probabilistic estimate of his credibility (i.e., an even chance). Then, bin Laden s credibility is automatically combined with his competence (again through a min function), to estimate bin Laden s believability as the source of EVD-Dawn-Mir01-01c. Disciple-LTA also allows the analysts to assess these believability characteristics by developing Wigmorean networks, as illustrated in Fig. 4 where Disciple-LTA reduces the problem of assessing the veracity of bin Laden to simpler problems, and then assesses the simplest problems based on the available [7]. As one can see, the Wigmorean network in Fig. 4 has the general structure shown in Fig. 1. EVD-Dawn-Mir-01-01c Fig. 3. Source believability assessment. 4

5 Fig. 4. Wigmorean network for veracity assessment. IV. CLASSIFICATION OF EVIDENCE BASED ON RELEVANCE Here is an important relevance question we are asked to answer regarding the individual kinds of we have: How does this item of stand in relation to what you, the analyst, are trying to prove or disprove from it? There are two species of relevant. Some may be directly relevant if you can form a defensible chain of reasoning from this item of to hypotheses you are considering. For example, E 1 and E 2 in Fig. 1 are directly relevant items of. Other may be indirectly relevant, or ancillary if it bears upon the strength or weakness in chains of reasoning set up by directly relevant. Consider, for example, the problem Assess the believability of E1 from the bottom right side of Fig. 1. Any item of that might be used in solving this problem would be indirectly relevant. Indirectly relevant is also any used in solving the problem Assess the extent to which one can believe Osama bin Laden as the source of EVD-Dawn-Mir01-01c from Fig. 3, such as any from the reasoning tree in Fig. 4. The term meta- is also appropriate since ancillary is about other. Fig. 5 shows this relevance-based classification of. directly relevant indirectly relevant (ancillary or meta) Fig. 5. Evidence classification based on relevance. V. CLASSIFICATION OF EVIDENCE BASED ON INFERENTIAL FORCE OR WEIGHT Here is an inferential force or weight question we are asked to answer regarding the individual kinds of we have: How does this item of changes your belief in the truthfulness in what you are trying to assess? If the item of increases our belief in the truthfulness of the hypothesis we are analyzing, we call it favoring. Otherwise, we call it disfavoring. For example, both E 1 and E 2 in Fig.1 are examples of favoring with respect to the hypothesis H 11. As shown in Fig. 1, one also has to assess the inferential force of a combination of two or more individual items of. These combinations of are also recurrent and do not involve the substance or content of the. One reason for carefully considering these combinations of is that they are often confused or incorrectly identified leading to mistakes in how the is described in an analysis. But perhaps the most important reason is that there are very important sources of uncertainty lurking in these evidential combinations. As shown in Fig. 6, there are three main classes of combinations. A. Harmonious Evidence Two or more items of are harmonious if they are directionally consistent in the sense that they all favor the same hypothesis. There are two basic forms of harmonious, corroborative and convergent. In the case of corroborative we have two or more 5

6 sources telling us that the same event E has occurred. This form of corroboration often allows us to have greater confidence that the event in question did occur. In such cases we would say that one source has verified what the other source has told us. The exception involves instances where we have other suggesting that two or more HUMINT sources collaborated in deciding what to tell us, or that one source influenced or coerced another source to report the same event. In the case of convergent we have two or more items that concern different events all of which point toward or favor the same hypothesis. Convergent can exhibit evidential synergism. In many situations two or more items, considered jointly, have greater inferential force or weight than they would have if we considered them separately or independently. Another way to characterize evidential synergism is to say that one item of can have greater force if we consider it in light of other. B. Dissonant Evidence corroborative harmonious convergent synergistic combination corroborative redundant redundant cumulative redundant contradictory dissonant Fig. 6. Recurrent substance-blind combinations of. divergent Dissonant involves combinations of two or more items that are directionally inconsistent; they can point us in different inferential directions or toward different hypotheses. There are two basic forms of evidential dissonance; the first involves contradictory. Contradictory always involves events that are mutually exclusive, they cannot have occurred jointly. From one source we learn that event E occurred; but from another source we learn that this same event did not occur. The dissonance seems obvious in this case since event E cannot have occurred and not have occurred at the same time. Evidential contradictions are always resolved on credibility grounds. As an example, suppose we have three HUMINT sources who tell us that event E occurred, and one HUMINT source who tells us that event E did not occur. In the not so distant past, it was believed that we should always resolve the contradiction by counting heads; i.e. majority rules. So, on this basis we would side with the three sources who tell us that event E did occur. The trouble here is that counting heads assumes that all of the four sources involved in this episode of contradictory have equal credibility. This may be a very bad assumption since, on ancillary about these four sources, we may well believe that the one source telling us that E did not occur has greater credibility than does the aggregate credibility of the three sources who tell us that event E did occur. So, what matters in resolving evidential contradictions is the aggregate credibility of the sources on either side of this contradiction. There is another form of dissonant called divergent. This pattern of dissonance differs from contradictory in the following way. A contradiction always involves whether one event occurred or did not occur. But divergent involves entirely different events; the directional inconsistency here means that these events point us toward different hypotheses. In one case, suppose credible about event E would favor hypothesis H, but credible about event F would favor hypothesis not-h. C. Evidential Redundance We often encounter two or more items of in which the first item acts to reduce the force of subsequent items of. Stated another way, the first item acts to make subsequent items redundant to some degree. There are two ways this can happen. The first form of evidential redundance involves the corroborative we discussed above. In this case we have repeated of the same events. Although having corroborative does add to our confidence that an event of interest did occur, each additional item adds less and less to our confidence. We refer to this situation as corroborative redundance. The second form of redundancy involves different events in which about one event, if credible, takes something off the inferential force of about another event. We have called this cumulative redundance. The word "cumulative" is an expression used in law to refer to that does not add anything to what we already know. It is very important to consider these two forms of evidential redundancy. In the case of corroborative redundance we risk double counting about the same event and ascribing additional weight the does not always have. For cumulative redundance we risk getting more inferential mileage out of the than can be justified. VI. CONCLUSIONS We have discussed several substance-blind forms and combinations of, each raising uncertainty issues that cannot be ignored in any intelligence analysis. Disciple-LTA knows about several of them and takes them into account for -based hypothesis assessment, but much more work remains to be done, especially concerning the inferential force. REFERENCES [1] Twining W., Evidence as a Multi-Disciplinary Subject. Law, Probability & Risk: A Journal of Reasoning Under Uncertainty. Vol. 2, No. 2, June [2] Wigmore J.H., The Science of Judicial Proof, Boston: Little, Brown, [3] Schum D.A., The Evidential Foundations of Probabilistic Reasoning, Northwestern University Press, [4] Tecuci G., Boicu M., Marcu D., Boicu C., Barbulescu M., Disciple- LTA: Learning, Tutoring and Analytic Assistance, Journal of Intelligence Community Research and Development, [5] Schum D.A., Tecuci G., Boicu M., Analyzing Evidence and its Chain of Custody: A Mixed-Initiative Computational Approach, International Journal of Intelligence and Counterintelligence, 22:2, 2009, pp [6] Boicu M., Tecuci G., Schum D., Intelligence Analysis Ontology for Cognitive Assistants, in Proc OIC-08, George Mason University. [7] Schum D.A. and Morris J., Assessing the Competence and Credibility of Human Sources of Intelligence Evidence: Contributions from Law and Probability, Law, Probability, & Risk, Vol. 6, pp ,