Automated Determination of the Veracity of Interview Statements from People of Interest to an Operational Security Force

Transcription

1 Automated Determination of the Veracity of Interview Statements from People of Interest to an Operational Security Force Douglas P. Twitchell Illinois State University David P. Biros Oklahoma State University Mark Adkins, Nicole Forsgren, Judee K. Burgoon, Jay F. Nunamaker Jr. University of Arizona {madkins, nmeek, jburgoon, Abstract In deception detection research validity issues have been raised when subjects are used in controlled laboratory experiments. Studying real-life deception detection is a complicated endeavor because researchers do not have the control in field studies that exist in laboratory experiments so determining ground truth is challenging. This study reports the findings of the combination of some successful previous attempts at automated deception detection in computer-mediated communication results of a study of real-world data from an operation security force. Message feature mining is used to evaluate the effectiveness of technology as an aid to deception detection in actual stressful situations with unpleasant long term consequences. The study analyzes 18 statements (9 truthful, 9 deceptive) from a military service s investigative unit using message feature mining. The analysis resulted in a 72% rate of accuracy in correctly classifying the messages. 1. Introduction Gauging credibility is critical to assessing intelligence from a human source, whether obtained directly or intercepted. Knowing the source of the intelligence is an obvious and powerful means for assessing credibility. Another measure of credibility is corroboration with other intelligence sources. In this paper we propose that the words and style used in statements by suspects and detainees may also allow for the use of automated methods for determining the veracity of those statements. Traditional methods of deception detection such as a lie detector test can be costly and availability of trained personnel to administer the test may be limited. We offer an alternative and complementary approach toward determining the veracity of the individual involved in potentially criminal incidents. Recent work [2] in automated deception detection in computer-mediated communication is used to support the research approach by applying analysis techniques to a set of statements made by individuals under investigation Deception The study of deception and its detection is rich in theoretical support. Both Interpersonal Deception Theory and Information Manipulation Theory [13] provide a foundation toward understanding deception in the interpersonal domain. However, Cue Leakage Theory [11] and Signal Detection Theory [10] appear to better support studies of deception and deception detection in an automated domain. Because the focus of the study is to develop an automated tool to identify deceptive cues in text-based media, we found both of the latter theories provide considerable support. For the purpose of this study, deception is defined as a message knowingly transmitted with the intent to foster false beliefs or conclusions [4]. Over the past three years, the Center for the Management of Information (CMI) at the University of Arizona and research partners have conducted over eighteen experiments to study deception with over 2150 subjects [3, 5, 21, 22]. These experiments have been instrumental in understanding the factors influencing deception, and have guided the building of automated tools for detecting deception and the creation of training for security personnel [2, 12]. Since a person will /06/$20.00 (C) 2006 IEEE 1

2 most likely be deceptive about hostile intentions, research in deception detection has led to the question of whether or not concealed malicious intent can be inferred from cues in communication. The quest for the perfect lie detector or truth serum has been long and has resulted in only a few modest successes. The most common and probably most controversial method of deception detection is use of the polygraph, commonly known as the lie-detector test. In a summary of laboratory tests, Vrij reports that the polygraph is about 82% accurate at identifying deceivers [18]. The National Academy of Science, however, concluded that such experimental numbers are often overestimates of actual results, especially in personnel screening [6]. Although it is not admissible in court, the polygraph is useful in some investigations for identifying potential suspects or specific contextual areas of interest. One large challenge for the polygraph is that the process is a very invasive procedure and one that evokes fear in those subjected to the technology required for the evaluation. Investigators must have a valid reason for subjecting someone to a polygraph test and the subject must agree to take the test. Therefore, even though the polygraph is relatively accurate when compared to other methods, the invasive quality renders the procedure impractical in most everyday situations. Other techniques, such as Criteria Based Content Analysis (CBCA) and Reality Monitoring (RM), are based on the content of interviews with subjects rather than the physiological arousal as with the polygraph. Because both of these methods, which are considered Statement Validity Analysis (SVA) methods, require an interview with the subject suspected of being deceptive, the intrusive quality is still present, yet not as physically invasive as the bodysensor-addled polygraph. Both methods also require trained interviewers for conducting the interview and highly skilled analysts for reviewing the statements and reaching a judgment. Neither method provides immediate feedback. CBCA is based on what is known as the Undeutsch- Hypothesis [16, 17], which states that a statement derived from actual memory will differ in content and quality from a statement derived from fantasy. CBCA uses a set of criteria to evaluate this hypothesis. Trained investigators rate a criminal statement against each criterion using a three-point scale. Reality monitoring uses a list of criteria that overlaps somewhat with CBCA, but operates under a different hypothesis: truthful or real memories are likely to contain perceptual, contextual, and affective information while deceptions or fabrications are likely to contain cognitive operations (thoughts and meanings). In a faceto-face study of 73 nursing students, Vrij found that use of CBCA and RM to detect deception was successful at rates of 79.5% and 64.1 % respectively [19] Automating Deception Detection Techniques Recently several studies have looked into the possibility of using advances in technology and natural language processing to automatically classify deceptive and truthful messages. For example, Zhou and colleagues [22, 23] used the Desert Survival problem in a study with groups of two. One of the subjects in some of the pairs was instructed to deceive his or her partner by recommending a ranking counter to their actual opinion. Using the automated message feature mining technique described below, the researchers reported approximately 80% accuracy at classifying deceptive messages much better than the 50% baseline accuracy of guessing and near some of the results from the polygraph. The technology has a number of possible uses for professional analysts and field officers. For example, in a situation where deception is suspected, large archives could be searched for messages that exhibit deceptive cues, thereby reducing the investigators workload. Also case officers may be able to analyze statements collected from a person who is of interest to an agency to determine veracity or even provide potential cues to follow up on at a later point in the investigation. Others have followed similar approaches. Using multiple studies, Newman, et. al. [15] found that a computerized analysis using 72 separate features attained an accuracy of 67% correct compared to human judges accuracy of 52% correct. Adams [1], on the other hand, used manually obtained linguistic features along logistic regression to detect deception in real-world criminal statements from the FBI with an accuracy of 82.1% correctly classified. In contrast, this study uses both automated analysis and real-world data 2. Methodology 2.1. Message Feature Mining Message feature mining [2] is a method for classifying messages as deceptive or truthful based on contentindependent message features. The process can be divided into two major steps, extracting features and classification. Extracting features includes choosing appropriate features for deception on which the messages will be classified, determining the granularity of feature aggregation, and calculating the features on the desired text. Of these steps, the most difficult is choosing the appropriate features. Potentially, there are an infinite number of possible features. Choosing those that are most appropriate for 2

3 classifying deception or concealment requires expert knowledge of the deception domain. A number of general features have been identified and have the potential to be useful in many contexts. A summary of message feature mining is shown in Table 1. Table 1 Steps comprising message feature mining 1) Extract Features. a) Choose appropriate features for deceptive intent. b) Determine granularity of feature aggregation (i.e. sentence, paragraph, etc.). c) Calculate features over desired text portions. 2) Classify. a) Manually classify documents. b) Prepare data for automatic classification. c) Choose appropriate classification method. d) Train model on portion of data. e) Test model on remaining data. f) Evaluate results and modify features, granularity, and/or classification method to improve results. Classifying the messages starts with manually classifying the messages in the training set, preparing data for automatic classification, choosing an appropriate classification method, training and testing the model, and evaluating the results. Because unsupervised learning may or may not create clusters based on deception, message feature mining uses supervised learning and manual classification of the training and testing sets. Once the data set is manually classified, the data requires a manual review for accuracy and appropriate formatting for input into the machine learning algorithms. After the data are ready for classification, an appropriate classification method or set of methods must be chosen. There are a number of methods to choose from, each with advantages and disadvantages [14]. Furthermore, most machine learning methods have a number of parameters (such as number of hidden nodes in neural networks) that adjust the behavior of the models, resulting in a very large number of possible models. Choosing a set of methods to use can be daunting; however, some methods seem promising, such as, inductive decision trees and neural networks. After the method or set of methods is chosen, the task is to train and test the data then obtain the accuracy results. Once obtained, the results can be used as a feedback tool for modifying the features, the granularity, and/or the classification methods in an effort to improve the results. Once a good model is found, it can be used with similar unannotated data sets to aid in determining the veracity of statements of unknown truth. The features set chosen for this study are the same as those chosen in previous research of deceptive computer-mediated interaction [21], some of which are based on the CBCA and RM methods mentioned above. In addition, there is a plan to evaluate additional features and methods as the team acquires more data from other people of interest to the security forces on military installations (see below). Following Zhou and colleague s linguistic based cues approach and taking into consideration the correlations between variables, eight linguistic constructs: quantity, complexity, uncertainty, nonimmediacy, diversity, affect, specificity, expressiveness, and informality. For example, one of the features is a measure of language diversity called lexical diversity, which is defined as the total number of unique words in the statement divided by the total number of words in the statement. Features for each message were extracted using the open-source text-engineering program GATE [7]. For classification we employed the open-source Weka [20] classification framework GATE GATE is General Architecture for Text Engineering software created at the University of Sheffield in the United Kingdom. The software is a java-based, objectoriented framework, architecture, and development environment for creating programs for analyzing, processing, or generating natural language [7]. As discussed by Twitchell and colleagues [7] GATE is a component-based architecture based on language resources (LR) and processing resources (PR). LRs are data-only resources such as single documents, corpora, ontologies and lexicons. PRs are programmatic or algorithmic resources that either use or process LRs such as parsers and part-of-speech taggers. For example, to parse all of the sentences in a document, one would create a LR that contains or represents the document. Next, a PR that contains the parser is created. It is common for a parser to require a lexicon, which could be loaded as an additional LR. Last, an application or pipeline is created wherein the parser PR is assigned to process the document LR. A pipeline can direct the processing of any number of PRs on any number of LRs [8]. We used GATE to extract features from documents by creating a set of PRs, each of which extracted a feature or set of features from the document. For example, a PR was built using the GATE-provided Java Annotations Processing Engine (JAPE) [9] that recognized and counted group references such as we, us, and ours. 3

4 For this study we based our choice of classification scheme on Zhou et. al. s [23] empirical study of classifiers for deception detection in text. That study concluded that general purpose neural networks were the most robust of the methods tested. However, one of the problems with neural networks is the difficulty in obtaining understandable rules from the network and determining which features were most influential in the classification. Some methods exist, but we have not implemented them at this time given the number of attributes and the small number of instances. Zhou et. al. [21], employing experimental data, used traditional statistics to determine the relative importance of each of the attributes that are also used in this study. They found that the quantity, informality, immediacy, uncertainty, and complexity measures were all significantly different between deceivers and truthtellers. Because the statements used in this study are from a context unrelated to the one in that study, we chose to include all of the features in the model rather than those found to be important. In a later study using classification models, Zhou et. al. [23] elected to also to include the whole feature set. We used the open source neural network implementation in Weka [20] for the classification. Weka is a platform produced at the University of Waikato for implementing machine learning algorithms. The tool comes equipped with a large number of classification, clustering, and attribute selection algorithms including the neural network used in this paper. The neural network implementation in Weka is a multilayer-perceptron with a single hidden layer and an output node for each class. The specific classifier used in this paper had a hidden layer that was comprised of 23 nodes, which is equal to the number of attributes (44) plus the number of classes (2) divided by 2. The size of the data set prevented the use of a validation/tuning set. Instead, the network was trained for 500 epochs. The remaining parameters were left at the default value in Weka (learning rate = 0.3, momentum = 0.2) The Data The preliminary dataset used in the study was obtained from the criminal investigative service of one of the United States military branches. The statements were written in English by the subjects themselves and constituted the narrative portion of an official report, similar to a statement of events on a police report. The truthfulness of each statement was provided by the investigative branch at the conclusion of the investigation and was based on numerous pieces of evidence ranging from physical evidence to confessions. As the official statements used in the investigations, these data provide a high-risk sample set; deceptive statements in this situation pose real and serious consequences. Additional data is being collected from security achieves at two military installations in the United States. Each data set will be cross analyzed before combining to make a large test bed of data for analysis. The cases which these data have been extracted are closed and have been adjudicated. The veracity of each statement was determined by the investigator prior to delivery of the statement to the research team. In most cases the investigators had no direct communication with the researchers. Procedures to prepare the statements for analysis are outlined in Table 2. Table 2 Procedure to prepare statements for analysis 1. Prepare Written Statements 1.1. Black out personal information (Name, SSN, etc.) 1.2. Replace names identified in the statement with a number. Write next to the blacked out portions a number to identify a person throughout the statement. For example, player 1 remains player 1 throughout the statement Label the statement as Truthful, Deceptive or Unknown. 2. Transcribe Written Statements 2.1. Open Notepad or WordPad 2.2. Type statement exactly how it is written Match Case Match punctuation Replace the names with Player1, Player2, etc. (Be sure to match Player1 to the person of interest, if person of interest does not mention him/herself in the statement, start the numbers with Player2 ) For corrections made by person of interest: ignore the initials used to verify the corrections were made by the person of interest and put whatever was marked out in brackets, [ ]. This will allow for an automated extraction or classification technique to be created and used by the information system Put all other anomalies between brackets (If it is not possible to type just 4

5 put two brackets to identify something was there in the written statement.) 3. Saving Typed Statements 3.1. Save statements with the last name of who made the transcription, True or False, a number and.txt. (i.e. the first false statement will be: SmithFalse1.txt and the third truthful statement is: SmithTrue3.txt The statements were provided in either an Adobe PDF format that was a scanned replica of the hard-copy form used or a hard copy of an original copy of the person of interest statement. These hand-written statements were then typed into plain text format, retaining all spelling and grammatical errors and idiosyncrasies. Identifying information was blacked out by investigators and substitute names were entered to maintain the flow and readability when needed. (For example, player 1 was entered into the text description if a name was blacked out.) After the statements were transcribed, the text was compared to the original hand-written statement by a third party to confirm spelling, punctuation, and grammar consistency Results The classifier correctly classified 13 of the 18 messages correctly as deceptive or truthful to attain 72% accuracy. A classifier randomly guessing does not achieve rates better than 50%. Three of the deceptive messages were misclassified as truthful and two of the truthful messages were misclassified as deceptive. Table 3 Neural network classifier results Classified as Actually Truthful Deceptive Truthful 7 2 Deceptive 3 6 Number Percent Correctly classified 13 72% 3. Discussion and Conclusion The classifier was able to distinguish between truthful and deceptive statements at a rate higher than chance, which is as good as or better than individual assessments [17]. Furthermore, the classifier has several advantages that may not be available in actual intelligence analysis or law enforcement environments such as being able to process large amounts of statements and to do so in an objective manner. In the case where there are large number of people who are of interest to an analyst or enforcement agent one may want to use this tool reconfirm existing evidence or create questions which could be further investigated. Like any artificial intelligence system, the classifier does not have the common sense, a lifetime of experiences or a hunch or good day. Yet the system performed reasonably compared to average or below average human analysts. This is difficult to discuss as most analyst and officers have a view they are more accurate detectors than they actual have been shown to be in experiments. Yet personal communication with Federal agents indicates that 80% of the arrests made by one agency are made by 20% of the agents. Therefore message feature mining could be useful for identifying those statements and messages that are worth further scrutiny and assist analyst who have volumes of statements to evaluate. One caution in analyzing the results is that all of the data analyzed were from native English speakers. Building training sets for classifiers in several languages may be difficult. Furthermore, the features used for English are likely not valid for many languages of interest. There may also be confounding properties in case where English is a second language. Features would have to be developed based on language and culture. Nevertheless, for certain languages and cultures the resource allocation may be justified to develop training sets and features for ascertaining credibility. The difficulty of intelligence analysis presented by large amounts of data requires a multi-faceted approach. The approach presented in this paper is one that may be useful for assessing the veracity of large amounts of criminal and detainee statements and recommending those needing further investigation. References [1] S. H. Adams, "Communication under stress: Indicators of veracity and deception in written narratives," in Adult Learning and Human Resource Development. Falls Church, Virginia: Virginia Polytechnic Institute, 2002, pp [2] M. Adkins, D. P. Twitchell, J. K. Burgoon, and J. F. Nunamaker, Jr., "Advances in automated deception detection in text-based computer-mediated communication," presented at Proceedings of the SPIE Defense and Security Symposium, Orlando, Florida, [3] J. P. Blair and E. Moyer, "Effects of communication modality on arousal, cognitive complexity, behavioral control, and deception detection during deceptive episodes," presented at Annual Meeting of the National Communication Association, Miami Beach, Florida,

6 [4] D. B. Buller and J. K. Burgoon, "Interpersonal deception theory," Communication Theory, vol. 6, pp , [5] J. K. Burgoon, J. P. Blair, T. Qin, and J. F. Nunamaker Jr., "Detecting deception through linguistic analysis," LNCS 2665: Proceedings of The First Annual NSF/NIJ Symposium on Intelligence and Security Informatics, 2003, Tucson, Arizona, [6] Committee to Review Scientific Evidence on the Polygraph, Board on Behavioral, Cognitive, and Sensory Sciences and Committee on National Statistics, Division of Behavioral and Social Sciences and Education, National Research Council of the National Academies., "The polygraph and lie detection." Washington, D.C.: National Academies Press, [7] H. Cunningham, "GATE, a general architecture for text engineering," Computers and the Humanities, vol. 36, pp , [8] H. Cunningham, D. Maynard, K. Bontcheva, V. Tablan, C. Ursu, and M. Dimitrov, "Developing language processing components with GATE," The University of Sheffield, Sheffield, UK, User Guide August [9] H. Cunningham, D. Maynard, and V. Tablan, "JAPE: A java annotation patterns engine (second edition)," [10] D. R. Davis and G. S. Tune, Human vigilance performance. New York: American Elsevier Publishing Company, [11] P. Eckman, Telling lies: Clues to deceit in the marketplace, politics, and marriage. New York: W. W. Norton and Company, [12] J. F. George, D. P. Biros, J. K. Burgoon, and J. F. Nunamaker, Jr., "Training professionals to detect deception," presented at Lecture Notes in Computer Science 2665: Proceedings of the First NSF/NIJ Symposium on Intelligence and Security Informatics, Tucson, AZ, [13] S. A. McCornack, "Information manipulation theory," Communication Monographs, vol. 59, pp. 1-16, [14] T. M. Mitchell, Machine learning. New York: McGraw- Hill, [15] M. L. Newman, J. W. Pennebaker, D. S. Berry, and J. M. Richards, "Lying words: Predicting deception from linguistic styles," Personality\& Social Psychology Bulletin, vol. 29, pp , [16] M. Steller and G. Köhnken, "Criteria-based content analysis," in Psychological methods in criminal investigation and evidence, D. C. Raskin, Ed. New York: Springer-Verlag, 1989, pp [17] U. Undeutsch, "The development of statement reality analysis," in Credibility assessment, U. Undeutsch, Ed. Dordrecht, The Netherlands: Kluwer, 1989, pp [18] A. Vrij, Detecting lies and deceit: The psychology of lying and implications for professional practice. Chichester: John Wiley & Sons, [19] A. Vrij, K. Edward, K. P. Roberts, and R. Bull, "Detecting deceit via analysis of verbal and nonverbal behavior," Journal of Nonverbal Behavior, vol. 24, pp , [20] I. H. Witten and E. Frank, Data mining: Practical machine learning tools and techniques with java. San Francisco: Morgan Kaufmann, [21] L. Zhou, J. K. Burgoon, J. F. Nunamaker, Jr., and D. P. Twitchell, "Automated linguistics based cues for detecting deception in text-based asynchronous computer-mediated communication: An empirical investigation," Group Decision and Negotiation, vol. 13, pp , [22] L. Zhou, D. P. Twitchell, T. Qin, J. K. Burgoon, and J. F. Nunamaker, Jr., "An exploratory study into deception detection in text-based computer-mediated communication," presented at Proceedings of the Thirty-Sixth Annual Hawaii International Conference on System Sciences (CD/ROM), Big Island, Hawaii, [23] L. Zhou, D. P. Twitchell, T. Qin, J. K. Burgoon, and J. F. Nunamaker, Jr., "Toward the automatic prediction of deception - an empirical comparison of classification methods," Journal of Management Information Systems, vol. 20, pp ,