The Degree of Consideration-Based Mechanism of Thought and Its Application to Artificial Creatures for Behavior Selection

Transcription

1 Jong-Han Kim, Woo-Ri Ko, Ji-Hyeong Han, and Sheir Afgen Zaheer KAIST, SOUTH KOREA The Degree of Consideration-Based Mechanism of Thought and Its Application to Artificial Creatures for Behavior Selection istockphoto.com/ BJORN MEYER Abstract To make artificial creatures deliberately interact ith their environment like living creatures, a behavior selection method mimicking living creatures thought mechanism is needed. For this purpose, there has been research based on probabilistic knoledge links beteen input (assumed fact) and target (behavior) symbols for reasoning. Hoever, real intelligent creatures including human beings select a behavior based on the multi-criteria decision making process using the degree of consideration (DoC) for input symbols, i.e. ill and context symbols, in their memory. In this paper, the DoC-based mechanism of thought (DoC-MoT) is proposed and applied to the behavior selection of artificial creatures. The knoledge links beteen input and behavior symbols are represented by the partial evaluation values of behaviors over each input symbol, and the degrees of consideration for input symbols are represented by the fuzzy measures. The proposed method selects a behavior through global evaluation by the fuzzy integral, as a multicriteria decision making process, of knoledge link strengths ith respect to the fuzzy measure values. The effectiveness of the proposed behavior selection method is demonstrated by experiments carried out ith a synthetic character Rity in the 3D virtual environment. The results sho that the artificial creatures ith various characteristics can be successfully created by the proposed DoC-MoT. Moreover, training the created artificial creatures to modify their characteristics as more efficient in the DoC-MoT than the probability-based mechanism of thought (P-MoT), both in terms of the number of parameters to be set and the amount of time consumed. Digital Object Identifier /MCI Date of publication: 17 January 2012 I. Introduction Ubiquitous robot incorporating mobile robot (Mobot), embedded robot (Embot) and softare robot (Sobot) as introduced for various services at any place and any time [1]. Mobot provides integrated mobile services in cooperation ith Embot and Sobot. Embot is embedded in the environment to collect sensor data, and Sobot in a virtual orld makes a decision. In recent years, pet-type Sobots ere developed to be mounted on cell phones or computers as artificial creatures or synthetic characters. As they might behave like a real orld creature, they could be used as an intermediate interface for interaction ith a user [2] [4]. To act like a living creature, the artificial creature should behave according to its desire in a certain situation. In this regard, there has been research on behavior X/12/$ IEEE FEBRUARY 2012 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 49

2 An artificial creature, hich acts like a real creature, mimics the real creature s thought mechanism. selection methods using the control architecture for decision making of living creatures [5] [8]. The architecture organized into perception, behavior, motivation and actuator modules as proposed for behavior selection [9] [16]. The context and desire are generated respectively in the perception and motivation modules. In the behavior module, each candidate behavior is evaluated based on the desire and context, and the best one is selected. Then, the selected behavior is generated through the actuator module. Considering both deliberative and reflexive behaviors, a control architecture for probabilistic and deterministic behavior selections as proposed for artificial creatures [17]. The artificial creatures probabilistically select a behavior based on their internal state including motivation, homeostasis and emotion, and external sensor information as a deliberative behavior. A reflexive behavior that imitates animals instinct is selected deterministically using only external sensor information. A to-layered confabulation architecture as proposed for behavior selection considering internal state and context using confabulation theory [18]. The confabulation theory illustrates the mechanism of thought of human beings [19]. The key idea is that each thalamocortical module is equipped ith a large collection of input symbols of internal state and context, here the pairs of co-occuring symbols are connected by knoledge links. As a result of thought process, one target symbol is selected among behavior symbols by confabulation. In this scheme, the knoledge link beteen co-occurring symbols is represented by conditional probability. Hoever, the human thought process is not based only on crisp numbers. Though such statistical information as conditional probabilities is an important part of the human thought process, it, generally, is not the dominant deciding factor. The personal biases, based on psychological and cognitive aspects of the human personality, and the environmental conditions have a dominant effect on the human External Environment Perception Module Actuator Module Behavior Module Motivation Module FIGURE 1 Schematic diagram of a typical conventional architecture for artificial creatures. thought process. For example, a person can be inclined toards a particular choice because of his/her optimism, belief, prejudice or some other personal or environmental aspect even though the statistical figures claim the feasibility of making some other choice. The effects of these personal biases on the human thought process and decision making ere discussed in detail [20] [22]. Besides, another real life characteristic that confabulation fails to effectively represent is the mutual interactions among ills or contexts ranging from redundancy (negative interaction) to synergy (positive interaction) [23]. This paper proposes the degree of consideration-based mechanism of thought (DoC-MoT) and its application to the behavior selection method for artificial creatures. The humanlike thought process hich is affected by personal biases and prejudices based on psychological, cognitive or environmental grounds, is used to model the mechanism of thought for artificial creatures. The creatures degrees of consideration (DoCs) for their internal ills and environmental contexts constitute the basis of the thought mechanism. In the proposed model, the DoCs for input (i.e. ills and contexts) symbols are represented by the fuzzy measures and the fuzzy integral is used for the global evaluation of the target (i.e. behavior) symbol on the basis of the partial evaluations over input symbols and their DoCs [24], [25]. The effectiveness of the proposed behavior selection method using the DoC-MoT is demonstrated by experiments carried out ith a synthetic character Rity, in a 3D virtual environment. This paper is organized as follos. Section II briefly describes confabulation theory that explains the probability-based mechanism of thought (P-MoT) and proposes the DoC-MoT. Section III proposes a behavior selection method using the proposed DoC-MoT. Section IV presents the experimental results to demonstrate the effectiveness of the proposed method. The concluding remarks follo in Section V. II. Modeling the Mechanism of Thought An artificial creature, hich acts like a real creature, mimics the real creature s thought mechanism. Fig. 1 shos a typical conventional architecture, hich has to behavior generation paths: the deliberate generation and the reflexive generation [9]-[16]. In case of deliberate generation, a proper behavior is selected in the behavior module considering its desire and environmental context generated respectively in the motivation and perception modules, and the selected deliberative behavior is expressed through the actuator module. In reflexive generation, the sensor information in the perception module bypasses the motivation stage in order to take emergency measures. This reflexive behavior generation is inspired from the reflex actions in biological species. To generate the deliberative behaviors as a consequence of the thought process, the architecture should be embodied ith a ell-modeled mechanism of thought. In this section, the probability-based 50 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE FEBRUARY 2012

3 mechanism of thought (P-MoT) is briefly revieed, and a novel model of the DoC-based mechanism of thought (DoC- MoT) is proposed. A. The Probability-Based Mechanism of Thought (P-MoT) The mechanism of thought is modeled by the confabulation theory based on probability [19]. To understand the mechanism of thought, the things to be understood first are: hat forms a brain and ho the brain functions. As muscles are composed of several individual fibers, the brain is formed of a number of ell-connected neurons [26]. A set of connected neurons represents various symbols hich are composed of input symbols (assumed fact symbols) and target symbols, such as red, seet, as input symbols and apple as a target symbol, etc., here the symbols are grouped into color, taste, ord, etc. The links beteen the input and target symbols, e.g. the links beteen red and apple and beteen seet and apple represent the knoledge about perceived entities, e.g. the likeliness of an apple to be red or the likeliness of an apple to be seet, and therefore, they are called the knoledge links. The knoledge links, such as likeliness in this example, are represented by conditional probabilities, and they describe the agent s degree of belief about the target symbol to possess the attributes specified by the input symbols. With the knoledge links beteen input and target symbols, the cognitive information-processing is employed by confabulation as shon in Fig. 2(a). When some of the input symbols, e.g. a, b, c and d, are expressed (perceived), each target symbol receives input excitations by knoledge links from them. Since the strength of the knoledge link is represented by conditional probability, the total input excitation I1z2 for target symbol z, is calculated using Bayes rule as follos [27]: I1z2 5 p1abcd z2 4 5 c p1abcdz2 p1az2 3 c p1abcdz2 p1cz2 dc p1abcdz2 d p1bz2 dc p1abcdz2 d 3p1a z2p1b z2p1c z2p1d z24. p1dz2 In general, the first four terms can be approximated as a constant number in any given situations as follos: c p1abcdz2 p1az2 dc p1abcdz2 p1bz2 dc p1abcdz2 p1cz2 Thus, I1z2 can be approximately calculated as (1) dc p1abcdz2 d < K. (2) p1dz2 I1z2 < K3p1a z2p1b z2p1c z2p1d z24. (3) Once the total input excitations of all the target symbols are calculated, the target symbol ith the highest input excitation is selected as a conclusion and this inner-take-all competition among target symbols is called confabulation. B. The DoC-Based Mechanism of Thought (DoC-MoT) A Novel Approach Though the probability-based information processing, as described in the previous section, selects a behavior based on the multi-criteria decision making process considering both its desire and external context [18], there can be some exceptions here the thought process is largely affected by personal biases [21]. These biases generally occur due to psychological and cognitive aspects of personality. To tackle these issues, a novel approach that considers these personal biases, hile selecting an Applying DoCs Target Symbols a P (a z) P (c z) z I(z ) = P (abcd z) Input Symbols P (b z) c Input Symbols g (a) Input Symbols h z (a) Target Symbols z h z (c ) I(z ) = #h g x 1 g (c) Input Symbols b P (d z) d g (b) h z (b) h z (d ) g (d ) (a) Applying DoCs (b) FIGURE 2 Modeling the mechanism of thought: The total input excitation for symbol z is represented by I1z2, calculated by confabulation in the P-MoT and by the fuzzy integral in the DoC-MoT. (a) The probability-based mechanism of thought (P-MoT) and (b) The DoC-based mechanism of thought (DoC-MoT). FEBRUARY 2012 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 51

4 TABLE 1 Fuzzy measure property and interaction degree. FUZZY MEASURE PROPERTY l j SYMBOL RELATION g1a hb2. g1a2 1 g1b2 1 inf 0 NEGATIVE CORRELATION ( ( g1a hb2 5 g1a2 1 g1b INDEPENDENT g1a hb2, g1a2 1 g1b appropriate behavior, is needed. One such approach, based on considering the creature s on DoCs and the decision making process for modeling a thought process, is proposed in this paper. These DoCs quantitatively define the creature s inclination toards a particular ill or a context. To represent the DoC for a criteria set, the fuzzy measure representation is preferred because besides representing the ills and contexts, it can also effectively represent the mutual interactions among them. The fuzzy integral based approach for global evaluation as considered because of its usefulness in the multi-information aggregation and the multi-criteria decision making [24]. No, denote a set of symbols, e.g. input symbols for creature s ills and contexts, by X 5 5x 1, c, x n 6 and the poer set of X by P1X2. With these notations, the definitions of the Sugeno l-fuzzy measure and the Choquet fuzzy integral are summarized in the folloing [25]. Definition 1 A fuzzy measure on the set X of symbols is a set function g : P1X2 S 30, 14 satisfying the folloing axioms; i) g1[2 5 0, g1x2 5 1; ii) A ( B ( X implies g1a2 # g1b2. The Sugeno l-fuzzy measure satisfies the folloing [24]: External Environment Actuator Module Reflexive Behavior Module Internal State Module Sensor Module ( ( POSITIVE CORRELATION User Command Module Behavior Selection Module Context Module Perception Module FIGURE 3 Overall architecture for behavior selection. g1a hb2 5 g1a2 1 g1b2 1lg 1A2g 1B2, (4) here g1a2 and g1b2, A, B ( X, represent the DoCs for the subsets A and B, respectively, and l denotes an interacting degree index. To calculate the fuzzy measures more efficiently, j is used as an another interaction degree index [28]. If 0 # j, 0.5, (4) becomes a plausible measure, if j 5 0.5, a probability measure, and if 0.5, j # 1, a belief measure. Table 1 shos the relationship beteen to interaction degree indices. Note that if to symbols have positive (negative) correlation, i.e. g1a hb2, g1a2 1 g1b2 1 g1a hb2. g1a2 1 g1b22, the global evaluation by the fuzzy integral over the symbols is to be underestimated (overestimated) [29]. The Choquet fuzzy integral hich can be used in a discrete domain problem is defined in the folloing [30]. Definition 2 Let h be a mapping from finite set X to 30, 14. For x i [ X, i 5 1, 2, c, n, assume h1x i 2 # h1x i11 2 and E i 5 5x i, x i11, c, x n 6. The Choquet fuzzy integral of h over X ith respect to a fuzzy measure g is defined as n 3 h g 5 a 1h1x i 2 2 h1x i21 22g1E i 2. (5) X i51 Using the definitions of the Sugeno l-fuzzy measure and the Choquet fuzzy integral, the thought process of the proposed approach is realized as in Fig. 2(b). When some of input symbols, e.g. a, b, c and d, are expressed (perceived), each target symbol receives input excitations by both knoledge links and the DoCs for the input symbols. If the DoC for input symbol a is high, target symbol z is more strongly linked to a than other input symbols. This approach is called the DoC-MoT. In the DoC-MoT, the fuzzy measures are employed to represent the DoCs for input symbols, e.g. g1a2, g1b2, c, g1x 1 2, Learning Module Memory Module Long- Term Memory Working Memory Short- Term Memory here X 1 = 5a, b, c, d6, and the fuzzy integral is used to globally evaluate each target symbol considering both the DoCs and the knoledge link strengths beteen the input and target symbols. The strengths are given by the partial evaluation values h z 1a2, h z 1b2, h z 1c2 and h z 1d2 of target symbol z over input symbols a, b, c and d. Then, the total input excitation I1z2 from input symbols a, b, c and d to target symbol z is calculated by the fuzzy integral, as follos: I1z2 5 3 X1 h g. (6) III. Behavior Selection Method In this section, the DoC-MoT is applied to the behavior selection problem. Fig. 3 shos the overall architecture of the 52 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE FEBRUARY 2012

5 proposed method, hich is composed of 10 modules: sensor, perception, user command, context, internal state, learning, memory, behavior selection, reflexive behavior and actuator modules. The context module identifies a current environmental context using perceptions from the perception module, and the internal state module identifies a current ill of an artificial creature. The memory module stores all the necessary memory contents including symbols of ills, contexts and behaviors. It also has the information on the DoCs for input symbols and the knoledge links beteen input and behavior (target) symbols. The DoCs are represented by the fuzzy measures and the knoledge link strengths are given by the partial evaluation values of behavior symbols over each input symbol. Considering the identified ill and context, the behavior selection module selects a proper deliberative behavior by the fuzzy integral aggregating the partial evaluation values and the DoCs for input symbols. The reflexive behavior module selects a reflexive behavior using sensor information from the perception module. The learning algorithm to change the characteristics of artificial creatures is executed in the learning module. The key modules for behavior selection, namely internal state, behavior selection, learning and memory modules, are described [17], [18], [31], [32]. A. Internal State Module The internal state module deals ith the internal information including internal ills. In this module, the strength of the jth ill, V j, j 5 1, 2, c, n, here n is the number of ills, is updated by V j 1t V j 1a j 1V j 2V j 21 S T # W j 2d ij, (7) here t is the time step, a j is the difference gain, V j is the steady state value, S is the stimulus vector, W j is the strength vector beteen stimulus and the jth ill and d ij is the amount of the change of the jth ill strength caused by the previous ith behavior. The first adding term in (7) makes the j th ill strength converging to the steady state value. For example, as time goes by ithout any stimulus, the ill to play gets stronger and stronger, and finally it converges to the preassigned steady state value. The second adding term denotes the amount of the change of the jth ill strength caused by external stimuli. If a user punishes an artificial creature, its ill to self-protect gets stronger. The final adding term is the one caused by the previous behavior. If the previous ith behavior has a positive effect on the jth ill (e.g. eating behavior and Actuator Module Internal State Module Will Strengths Behavior Selection Module Will-Based Evaluation Fuzzy Measure Fuzzy Integral Context-Based Evaluation Fuzzy Measure Fuzzy Integral Arbiter Context Module Contexts Memory Module Long-Term Memory Normalized Weights of Wills Partial Evaluation Values of Behaviors over Wills Normalized Weights of Contexts Partial Evaluation Values of Behaviors over Contexts Working Memory Candidate Behaviors Short-Term Memory FIGURE 4 Block diagram of the behavior selection module. The solid arros denote the movement of data related to ills and contexts, and the dotted arros denote the behavior recommendation. ill to eat ), d ij is a positive value. If it has a negative effect on the jth ill (e.g. kicking-ball behavior and ill to rest ), d ij is a negative value. The calculated current ill strengths are used in the behavior selection module. B. Behavior Selection Module In the behavior selection module, one proper behavior is chosen using the artificial creature s ill strengths, environmental contexts and its DoCs for ills and contexts. Fig. 4 shos the block diagram of the behavior selection module. The solid and dotted arros denote the movement of data related to ills and contexts and the behavior recommendation, respectively. Firstly, all the behaviors are evaluated considering its ills in a illbased evaluation. The normalized eights of ills are called up from a long-term memory to calculate the fuzzy measure values of all the related ill sets. The ill-based global evaluation value of each behavior is calculated by the fuzzy integral ith respect to the fuzzy measure values of ill sets, current ill strengths and the partial evaluation values of behaviors over each ill. Some proper behaviors to current ills are recommended to the next context-based evaluation stage and also to the arbiter for selection through a competition process. The recommended behaviors are re-evaluated considering external contexts in a context-based evaluation by the same manner of the ill-based evaluation. After the context-based evaluation, the arbiter selects one behavior through competition. Then, it is actuated in the actuator module. The folloing describes in detail the procedure for the behavior selection process. 1) Definition of Input and Target Symbols: For the behavior selection to make the artificial creatures deliberately interact FEBRUARY 2012 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 53

6 i) Normalized eights of ill symbols In ordinal analytic hierarchy process (AHP) eigenvalue method [33], the normalized eights of ill symbols are calculated by the folloing ordinal AHP s pairise comparison matrix (See Tables 5, 6, 7 and 8 in Section IV): c 11 c c 1n C 5 ( f (, (8) c n1 c c nn here c ij represents the importance degree of the ith symbol compared to the jth symbol. c ii is 1 and c ij is given as 1/c ji. The normalized eight d i of the ith ill symbol i, i 5 1, 2, c, n, is calculated as follos: FIGURE 5 Screenshot of Ritys in the 3D virtual orld. ith their environment like living creatures, input and target symbols should be first defined. In this paper, their internal ills and external environmental contexts are defined as input symbols, and their behaviors are considered as target symbols. The input and target symbols used in the experiments are provided in Section IV. 2) Fuzzy Measure Identification of Will Set: Let us denote a set of ills by X 5 5 1, 2, c, n 6, here j, j 5 1, 2, cn, represents the jth ill. The number of subsets of X is 2 n, and to identify all the fuzzy measures of the subsets, 312 n n 2 324/2 times of pairise comparisons are needed [33]. Thus, in this paper, for the efficient identification, the folloing method is employed [34]. The example ith detailed calculation process is described in Section IV. G 2 Interaction Diagram Negative Correlation Positive Correlation Nearly Independent G 7 G 3 G G G 4 G 5 FIGURE 6 Interaction diagram of ills. 0.4 d i 5 n a c j51 ij n n ai51 a c j51 ij. (9) ii) Interaction diagram The interaction diagram of ill symbols (Fig. 6 in Section IV) shos the interaction degree beteen to ill symbols. If the to ill symbols have negative correlation (e.g. ill to play and ill to rest ), the interaction degree has a value beteen 0 and 0.5, as Table 1 shos. If the to ill symbols have positive correlation (e.g. ill to seek shelter and ill to selfprotect ), the interaction degree has a value beteen 0.5 and 1. Otherise, the to ill symbols are independent, and the interaction degree has a value of 0.5. Therefore, the interaction degree beteen the ith and the jth ill symbols j ij lies in 30, 14. Using the interaction diagram, the hierarchy diagram is constructed. iii) Hierarchy diagram The hierarchy diagram of ill symbols (Fig. 7 in Section IV) represents hierarchical interaction relations among ill symbols by clustering to closely-related ill symbols. To estimate ho much to ill symbols are related, dissimilarity beteen them should be accounted. The dissimilarity D 5Gp,G q 6 beteen to clusters G p and G q, is defined as an average distance to other symbols as follos: D 5Gp,G q 6 5 a Gr [F, G r 2G p, G r 2G q 3j 5Gp, G r 6 2j 5Gq, G r 64 2, (10) F 2 2 here F is a set of all clusters hich can be a ill symbol or a set of clustered ill symbols, G p, G q, G r [ F are clusters, j 5Gi, G j 6 is the interaction degree beteen ill clusters G i and G j and F is the number of symbols of F. To ill clusters that have the smallest dissimilarity are merged into one, and the interaction degrees among ill clusters are recalculated. The interaction degree j 5Gp, G q 6 beteen to clusters G p and G q is calculated as a Qi, jr[qe QG pr3e QG qrr j ij j 5Gp, G q 6 5 E AG p B 3 E AG q B, (11) 54 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE FEBRUARY 2012

7 Importance Evaluation of Wills (R ) Independent Integration (ξ {{{{1},{2}},{6},{{5},{7}}},{{3},{4}}} = 0.5) And Like Integration (ξ {{{1},{2}},{6},{{5},{7}}} = 0.35) C Or Like Integration (ξ {{3},{4}} = 0.8) D Or Like Integration (ξ {{1},{2}} = 0.7) A And Like Integration (ξ {{5},{7}} = 0.2) B Shelter Seeking Self- Protection Play Comradeship Reset Ingestion Excretion FIGURE 7 Simplified hierarchy diagram of ills. here A 3 B is the direct sum and E1G p 2 is the function hich picks up all symbols in the set G p. The merging procedure is done until all groups are merged. Final structure of the hierarchy diagram is characterized by the interaction relations that are And like or Or like connections. The hierarchy diagram is simplified by re-merging to clusters if the difference beteen the interaction degrees of merged cluster and original clusters is less than 0.2 [34]. iv) Fuzzy measure identification After getting the hierarchy diagram, a fuzzy measure g1a2, here A ( X, is identified as follos: g1a2 5f s aj R, ap(r u P R b, (12) here R is the root level in the hierarchy diagram, f s is a scaling function [28] and u Q P is defined as follos: f s 1j, u2 5 f 1, if j51 and u. 0 0, if j51 and u 5 0 1, if j50 and u 5 1, (13) 0, if j50 and u, 1 s u 2 1 s 2 1, other cases d i, here i [ Q if Q 5 1 and i [ A u P Q 5 0 if Q 5 1 and i o A, f21 s (j P, f s (jq, g V(Q u Q ) 3 T V Q P ) other cases (14) here s j2 2 /j 2 and the value of f s 21 1j, r2 is u, hich satisfies f s 1j, u2 5 r. The conversion ratio T Q P from Q to P, is computed as f s Qj P, ai[q d i R T P Q 5, (15) f s Qj Q, ai[q d i R here P is the upper level set and Q is the loer level set in the hierarchy diagram. 3) Fuzzy Measure Identification of Context Set: The fuzzy measure identification of the context set is achieved by the same manner as that of the ill set. The normalized eight of each context is computed using the pairise comparison method, and the hierarchy diagram of contexts is constructed from the interaction diagram of contexts. The fuzzy measure value g1a2 of the context set A is calculated by a scaling function f s in (13) using the normalized eights of contexts and the hierarchy diagram of contexts. 4) Global Evaluation of Behaviors Over Wills: Let us denote a set of behaviors by X b 5 5b 1, b 2, c, b p 6, here p is the number of behaviors and b i, i 5 1, 2, c, p, represents the jth behavior. The global evaluation value E 1b i 2 of the ith behavior b i, i 5 1, 2, c, p, ith respect to the fuzzy measure values of ill sets and the knoledge links beteen b i and ills, is computed by the Choquet fuzzy integral as follos: E 1b i Xh g 5 a n j51 5h ij # V j 2 h i 1 j212 # V j21 6g1A2, (16) here g1a2 is a l-fuzzy measure of A ( X, identified by (12), h ij [ 30, 14 is the partial evaluation value of the ith behavior FEBRUARY 2012 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 55

8 b i over the jth ill, hich denotes the knoledge link strength beteen the ith behavior and the jth ill. Once the behavior evaluation using current ills is done, some behaviors ith a higher evaluation value b r i, i 5 1, 2, c, l, here l is the number of the recommended behaviors, are recommended to the next behavior evaluation stage to consider external contexts and also sent to the arbiter for selection through a competition among them. 5) Global Evaluation of Behaviors over Contexts: Let us denote a set of contexts by X c 5 5c 1, c 2, c, c m 6, here m is the number of contexts and c j, j 5 1, 2, c, m, represents the jth context. The global evaluation value E c 1b ir 2 of the ith recommended behavior b r i, i 5 1, 2, c, l, ith respect to the fuzzy measure values of context sets and r the knoledge links beteen b i and contexts, is computed as follos: E c 1b ir Xc h g 5 a m p j51 c c 1h i j 2 h i 1 j212 2 g1a2, (17) here m p is the number of activated contexts, g1a2 is a l-fuzzy c measure of A ( X c, identified by (12) and h i j [ 30, 14 is the r partial evaluation value of the recommended behavior b i over the jth activated context c j [ X c, hich denotes the knoledge link strength beteen the ith recommended behavior and the jth activated context. 6) Behavior Selection: As the global evaluation of behaviors over current ills and external contexts is done by the fuzzy integral, the fittest behavior is selected through the folloing competition [18]: E a 1b s 2 5 max j 3E 1b jr 2 E c 1b jr 24, j 5 1, 2, c, l, (18) here b s is the selected behavior ith the highest global evaluation value among the recommended behaviors. C. Learning Module In this paper, learning based on reinforcement signals is employed for artificial creatures, hich is motivated by the real pet-training scheme. The learning process is executed in real time using the patting and punishing signals from a user as reinforcement signals. As in a human thought process, the reard and penalty signals cause the change of the normalized eights of corresponding ills or contexts, and the normalized eights are used to compute the fuzzy measure values of ill and context sets. Note that the artificial creature s DoCs are represented by the fuzzy measures calculated from the normalized eights. After being rearded (punished), the normalized eight of a ill immediately increases (decreases) in proportion to the knoledge link strength beteen the ill and the rearded (punished) behavior. As an example, hen the artificial creature excretes in the bedroom, it is punished, and then it notices that the excretion behavior causes a pain. Since the excretion behavior is strongly linked to ill to excrete, the normalized eight of ill to excrete decreases. Hoever, the normalized eight of ill to rest is not changed, since ill to rest is not related to the excretion behavior. If the ith behavior is rearded or punished, the normalized eight of the jth ill d j, j 5 1, 2, c, n, is changed by d j 1 k 1 h i j d j 1t d j 2 k 1 h i j d j 1reard2 1penalty2 1otherise2 (19) here k 1 [ 30, 14 is the learning rate. The knoledge link strength beteen ill and behavior symbols is changed at the same time as the normalized eight of ill is updated. When a reard or punishment is given to the artificial creature, the knoledge link strength h i j beteen the jth ill j and the rearded or punished behavior b i, is updated as follos: h i j 1t h i j 1 k h i j 2 h i j 2 k 2 h i j h i j 1reard2 1penalty2 (20) 1otherise2, here k 2 [ 30, 14 is the learning rate. Eventually, it needs to realize that the excretion behavior in the bedroom is also a cause of the pain. Such a context as in the bedroom should be considered. In the case of context, hoever, it is not obvious hich context has caused the reard or punishment. Thus, the normalized eight of context is changed if the context happens repeatedly more than once hen doing the rearded or punished behavior. As an example, after being punished repeatedly hen it excretes in the bedroom, it notices that place is an important factor hen excreting. Thus, the normalized eight of place increases. If the context c j is recognized for the rearded or punished behavior b i, the normalized eight of the jth context d j c, j 5 1, 2, c, m, is changed by d jc 1t e d j c c 1 k 3 h i j c d j 1reard/penalty2, 1otherise2 (21) here k 3 [ 30, 14 is the learning rate. The knoledge link strength beteen context and behavior symbols is changed at the same time as the normalized eight of context is updated. When a reard or punishment is given c to the artificial creature, the knoledge link strength h i j beteen the recognized context c j and the rearded or punished behavior b i, is updated as follos: h ij c 1t c c h i j 1 k h i j c c h i j 2 k 4 h i j c h i j here k 4 [ 30, 14 is the learning rate. 1reard2 1penalty2, (22) 1otherise2 56 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE FEBRUARY 2012

9 Note that after learning, the normalized eights of ills and contexts are normalized and loaded into the memory because h c i j and h ij are mappings from the ill and context sets, respectively, to 30, 14 by Definition 2. D. Memory Module The memory module consists of short-term memory (STM), long-term memory (LTM) and orking memory (WM). In the STM, the sensory inputs from an environment are kept for a hile and the information orthy of remembering is then transferred to the LTM. The LTM is a durable storage space for ell-learned information. Memory contents in the LTM are the symbols of ills, contexts and behaviors, the partial evaluation values of behaviors over each ill and context and the interaction degrees among input symbols. The WM keeps a limited amount of information for a limited period of time, such as the candidate behaviors during the global evaluation and the previous fe contexts and behaviors hen the artificial creature is rearded or punished for learning. IV. Experiments To demonstrate the effectiveness of the behavior selection method using the DoC-MoT, experiments ere carried out for Rity, a synthetic character, hich as developed in the 3D virtual orld using OpenGL. Rity has 14 degrees of freedom for motions to express 40 behaviors and 7 ills as internal states, and it can perceive 9 contexts in its environment. In the environment, as Fig. 5 shos, there are food items, balls, a bed, a toilet and the fello Ritys. The first goal of the experiments is to confirm that Rity s behaviors selected by the DoC-MoT are reliable. According to the knoledge links in memory, Rity behaves differently even in the same situation. The second goal is to create Rity ith the desired characteristics through learning. The third goal is to sho that the behavior selection method using the DoC-MoT is more effective compared to that using the P-MoT. Note that Rity hich is a form of a dog and selects a behavior folloing the DoC-MoT, can give comfort to the user, since the user feels comfortable hen Rity behaves as he or she has expected. A. Experimental Setting For practical experiments, 24 hours in the virtual orld ere scaled don to one hour in the real orld. In every 2.5s, a proper behavior as selected by the proposed behavior selection method. The behavior generation frequency, hich as used to check Rity s characteristics, as computed for each behavior from the experimental result gathered for one hour, i.e. one day in the virtual orld. Note that the partial evaluation values of behaviors over each ill and context, the strength vector beteen stimulus and ills and the amounts of ill strength change by the previous behavior, ere initialized by an expert, and the context on an object hich appears in front of Rity, as randomly given to attain the reliable behavior generation frequency. A mouse click or double click as used for a reard or punishment, respectively. B. Definition of Input and Target Symbols In this paper, the Rity s ills ere categorized into seven kinds based on the categorization of canine behaviors [9]. As shon in Table 2, there are telve kinds of canine behaviors, but in the experiments, some inappropriate behaviors for Rity, i.e. sexual, miscellaneous motor and maladaptive behaviors, ere excluded, and for simplicity, behaviors in the same category, e.g. epimeletic, et-epimeletic, allelomimetic and agonistic behaviors, ere assumed to generate the same ill, i.e. ill to have comradeship as an assumed fact. Also, assumed facts on context ere classified for time (hen), place (here) and object (hat). As Table 3 shos, nine assumed facts on context ere defined. Note that among the assumed facts on a specific context, i.e. hen, here or hat, only one assumed fact from each category as activated at a time. Table 4 shos forty behaviors as target symbols. C. Fuzzy Measure Calculation As described in Section III, the fuzzy measure values of the ill and context sets ere calculated by the fuzzy measure identification method [34]. The detailed procedure of the fuzzy measure calculation of ill sets is described in the folloing. i) Normalized eights of ill symbols The normalized eight value of each ill for four kinds of Ritys as calculated by AHP s pairise comparison matrix, as TABLE 2 Canine behaviors and assumed fact symbols on ill. CANINE BEHAVIOR ASSUMED FACT PLAY, INVESTIGATIVE (SEARCHING/SEEKING) PLAY (W 1 ) EPIMELETIC (CARE AND ATTENTION GIVING), ET-EPIMELETIC (ATTENTION GETTING), ALLELOMIMETIC (DOING WHAT OTHERS DO), AGONISTIC (ASSOCIATED WITH CONFLICT) COMRADESHIP (W 2 ) COMFORT-SEEKING (SHELTER-SEEKING) SHELTER-SEEKING (W 3 ) (ADDITIONAL) SELF-PROTECTION (W 4 ) INGESTIVE (FOOD AND LIQUIDS) INGESTION (W 5 ) (ADDITIONAL) REST (W 6 ) ELIMINATIVE EXCRETION (W 7 ) SEXUAL, MISCELLANEOUS MOTOR, MALADAPTIVE (EXCLUDED) TABLE 3 Assumed fact symbols on context. CLASSIFICATION TIME PLACE OBJECT ASSUMED FACT MORNING (C 1 ) AFTERNOON (C 2 ) EVENING (C 3 ) NIGHT (C 4 ) BEDROOM (C 5 ) TOILET (C 6 ) FOOD (C 7 ) COMRADE (C 8 ) TOY (C 9 ) FEBRUARY 2012 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 57

10 TABLE 4 Target symbols of behaviors. b 1 STOP b 2 SIT b 3 CROUCH b 4 SHAKE HEAD b 5 LIFT ARM b 6 WHINE b 7 GO BACK AND FORTH b 8 STEP BACK QUICKLY b 9 STEP BACK SLOWLY b 10 WANDER QUICKLY b 11 WANDER SLOWLY b 12 MOVE CHEERFULLY b 13 MOVE QUICKLY b 14 MOVE SLOWLY b 15 FOLLOW AFTER CLOSELY b 16 FOLLOW SLOWLY b 17 APPROACH b 18 MOVE AROUND SLOWLY b 19 MOVE AROUND QUICKLY b 20 SNIFF b 21 LOOK AROUND b 22 LOOK AT b 23 EAT CHEERFULLY b 24 EAT SLOWLY b 25 EXCRETE b 26 URINE b 27 SLEEP b 28 NAP b 29 DIG QUICKLY b 30 DIG SLOWLY b 31 SCRABBLE QUICKLY b 32 SCRABBLE SLOWLY b 33 PUSH b 34 KICK b 35 BITE b 36 GROWL b 37 BARK LOUDLY b 38 BARK b 39 BARK SOFTLY b 40 HOWL TABLE 5 The pairise comparison matrix for ills of a normal Rity. W 1 W 2 W 3 W 4 W 5 W 6 W 7 NORMALIZED WEIGHT (d 1 ) (d 2 ) (d 3 ) (d 4 ) (d 5 ) (d 6 ) (d 7 ) shon in Tables 5, 6, 7 and 8, respectively. For a normal Rity, each ill has the same importance, so all the normalized eights of ills ere equal to For a cheerful and outgoing Rity, ill to play as set to be 10 times more important than ill to self-protect and 9 times more important than ill to rest. As a result, the normalized eight value of ill to play as 0.342, that of ill to self-protect as and so on. For an omnivorous Rity, ill to eat as set to be 7 times more important than ill to play and 4 times more important than ill to self-protect. Thus, the normalized eight value of ill to ingest as and so on. For a timid Rity, ill to self-protect as set to be 8 times more important than ill to play and 3 times more important than ill to ingest. Then, the normalized eight value of ill to self-protect as and so on. In the folloing, the fuzzy measure values are calculated for the cheerful and outgoing Rity. The same procedure can be applied for the other Ritys. ii) Interaction diagram As defined, the number of ills as 7, and the interaction diagram of ills is shon in Fig. 6, here X 5 5G1, G2, G3, G4, G5, G6, G76 5 5Play, Comradeship, Shelter-seeking, Selfprotection, Ingestion, Rest, Excretion6. The number on the line connecting to ills is the interaction degree (j) beteen them. If they have a negative correlation (dashed line), the interaction degree has a value beteen 0 and 0.5. If they have a positive correlation (dotted line), the interaction degree has a value beteen 0.5 and 1. Otherise, they are independent of each other (solid line), and the interaction degree has a value of 0.5. iii) Hierarchy diagram To estimate ho much to ills ere related, the dissimilarity beteen them as accounted, and to ills that had the TABLE 6 The pairise comparison matrix for ills of a cheerful and outgoing Rity. W 1 W 2 W 3 W 4 W 5 W 6 W 7 NORMALIZED WEIGHT 1 1 3/ (d 1 ) 2 2/ (d 2 ) 3 1/6 1/ /3 2 1/ (d 3 ) 4 1/10 1/8 1/2 1 1/5 1/2 1/ (d 4 ) 5 1/3 1/ (d 5 ) 6 1/10 1/7 1/2 2 1/3 1 1/ (d 6 ) 7 1/3 1/ (d 7 ) TABLE 7 The pairise comparison matrix for ills of an omnivorous Rity. W 1 W 2 W 3 W 4 W 5 W 5 W 7 NORMALIZED WEIGHT 1 1 1/2 1 2/5 1/7 1/4 1/ (d 1 ) /3 1/5 1/3 1/ (d 2 ) 3 1 1/2 1 1/2 1/8 1/4 1/ (d 3 ) 4 5/2 3/ /4 1/2 2/ (d 4 ) (d 5 ) /2 1 1/ (d 6 ) /2 1/ (d 7 ) 58 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE FEBRUARY 2012

11 TABLE 8 The pairise comparison matrix for ills of a timid Rity. W 1 W 2 W 3 W 4 W 5 W 5 W 7 NORMALIZED WEIGHT /6 1/8 1/3 1/6 1/ (d 1 ) /5 1/7 1/3 1/6 1/ (d 2 ) / (d 3 ) / (d 4 ) /2 1/3 1 1/ (d 5 ) /2 1/ (d 6 ) /4 1/3 1 1/ (d 7 ) smallest dissimilarity ere merged into one. For example, the dissimilarity beteen ill to play (G1) and ill to have comradeship (G2) as calculated as D 5G1, G26 5 j 15 2j 25 F (23) After calculating the dissimilarity, ill to seek shelter (G3) and ill to self-protect (G4) ere merged into one, since they had the smallest dissimilarity. To merge another pair of ills, the interaction degrees among clusters ere re-computed. For example, the interaction degree beteen G1 and 5G3, G46 as computed as j 5G1, 5G3, G j 13 1j 14 2 / The merging procedure as done until all ills ere merged. Fig. 7 shos the simplified hierarchy diagram. iv) Fuzzy measure identification The fuzzy measure values of the ill sets ere computed by using the simplified hierarchy diagram (Fig. 7). As an example, the fuzzy measure value g1a2 of A 5 5G36, for the cheerful and outgoing Rity, as calculated as g1a2 5f s 1j R, u C R 1 u DR 2 5 f s 10.5, u DR 2 5 f s 10.5, f s , f s 10.8, T DR 22 5 f s 10.5, f s , * f s 10.5, (24) The fuzzy measure identification of the context set as achieved by the same manner as that of the ill set. Since the contexts, i.e. time, place and object are independent of each other, all the interaction degrees among them ere valued at 0.5, and the hierarchical diagram as an independent integration of the three contexts. Therefore, the fuzzy measure value of the context set as the eighted sum of the normalized eights for the related contexts. The calculated fuzzy measure values of the ill and context sets ere used for behavior selection. D. Experimental Results i) Experiment 1: Four Ritys ith different characteristics Using the pairise comparison matrices in Tables 5, 6, 7 and 8, four kinds of Ritys ith different characteristics, i.e. normal, cheerful and outgoing, omnivorous and timid, ere created, respectively. For each pairise comparison matrix, the fuzzy measures of the ill sets ere calculated using a hierarchy diagram. Behaviors ere selected as a result of global evaluation by the Choquet fuzzy integral of the knoledge links ith respect to the fuzzy measures. Fig. 8 shos the mean and standard deviation of the behavior generation frequencies for the total number of selected behaviors, 1,440 per day, for each kind of Rity. Note that the error bar in Figures 8 14 denotes the standard deviation. For the normal Rity, the mean frequencies of stepping-back, eating, kicking and barking behaviors ere about 7%, 9%, 17% and 11%, respectively. The most frequent behavior of the cheerful and outgoing Rity as kicking a ball of about 34%. For the omnivorous Rity, eating behaviors ere dominant ith over 17%. For the timid Rity, barking behaviors ere dominant ith over 27%, and stepping-back behaviors ere about 15%. The standard deviation of each behavior generation frequency for four Ritys as less than 2%. These results sho that by changing the eight of each ill in the behavior selection method using the DoC-MoT, Rity could be created ith different characteristics. Hoever, creating different characteristics of an artificial creature using the P-MoT is time-consuming and gives undesirable results in some cases, hich can be achieved by to means: (1) changing all the conditional probabilities associated ith the knoledge link strengths, and (2) applying the learning scheme to the initially created creature hich has normal characteristics. For (1), the maximum numbers of pre-given parameters and parameters to be set to create different characteristics, in the to behavior selection methods ere compared, as shon in Table 10. In general cases, the numbers of contexts and behaviors can be approximated by the number of ills and Frequency (%) Stepping Back Normal Rity Cheerful and Outgoing Rity Omnivorous Rity Timid Rity Eating Kicking Barking Other Behaviors FIGURE 8 Behavior generation frequencies of four different Ritys. FEBRUARY 2012 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 59

12 Normalized Weight Play Before Learning After Learning Comrade- Ship Shelter Seeking Self- Protection the square of the number of ills, respectively. With this approximation, e could say that the maximum number of parameters to be set to create different characteristics as considerably less in the behavior selection method using the DoC- MoT (2n) than that using the P-MoT (2n 3 ). Note that the number of parameters for normalized eights is n, since they may be given directly, ithout calculating by the pairise comparison matrix. As illustrated before, an artificial creature ith different characteristics can also be created by applying learning Energy Fatigue Excretion FIGURE 9 Change of the omnivorous Rity s normalized eights of ills. Normalized Weight Before Learning After Learning Time Place Object FIGURE 10 Change of the omnivorous Rity s normalized eights of contexts. scheme to a normal one in the P-MoT. In the experiment, a normal Rity, the behavior generation frequency of hich as similar to that in the DoC-MoT, as initially created by changing all the conditional probabilities, and then its characteristics as changed to cheerful and outgoing, omnivorous and timid, respectively, through learning. To create a cheerful and outgoing Rity (a timid Rity), andering, moving, folloing, moving-around, pushing and kicking behaviors ere rearded (punished). To create an omnivorous Rity, eating behaviors ere rearded. The mean and standard deviation of the behavior generation frequencies for each kind of Rity in the to behavior selection methods ere compared, as shon in Table 10. The p-values ere derived from an unpaired t-test to evaluate the difference beteen the behavior generation frequency distribution in the P-MoT and DoC-MoT. For the normal Rity, the numbers of generated behaviors ere 19 and 16 for the DoC-MoT and P-MoT, respectively. Among them, 14 behaviors had no significant difference in the means of the generation frequencies, since the p-values ere larger than In other ords, the normal Ritys created by the DoC-MoT and P-MoT had similar characteristics. Nevertheless, for the cheerful and outgoing, the omnivorous and the timid Ritys, the distribution of behaviors (p-values less than 0.05) shoed significant differences beteen the to behavior selection methods. Though the Ritys ith P-MoT generated proper behaviors to their characteristics, i.e. kicking behavior for the cheerful and outgoing Rity, eating behaviors for the omnivorous Rity and stepping-back behaviors for the timid Rity, the behaviors hich had not been generated by the normal Rity before learning, ere not observed either in the other Ritys after learning. The generated behaviors from the changed Ritys by learning process ere restricted to the behaviors hich had been generated by the normal Rity. On the other hand, in Ritys ith the DoC-MoT, sloly-folloing, quicklymoving and groling behaviors hich had not been generated by the normal Rity, ere observed after learning. This is because a reard or punishment in the P-MoT caused the change of the conditional probabilities of the specific rearded or punished behaviors and the conditional probabilities related to the un-generated behaviors could not be changed. In the DoC-MoT, the change of DoCs during learning, can affect the TABLE 9 The maximum numbers of pre-given parameters and parameters to be set to create different characteristics. BEHAVIOR SELECTION USING THE P-MOT [ OF PRE-GIVEN PARAMETERS [ OF PARAMETERS TO BE SET TO CREATE DIFFERENT CHARACTERISTICS BEHAVIOR SELECTION USING THE DOC-MOT [ OF PRE-GIVEN PARAMETERS KNOWLEDGE LINK STRENGTHS 1n 1 m2 * p 1n 1 m2 * p 1n 1 m2 * p 0 NORMALIZED WEIGHTS 0 0 n 1 m n 1 m INTERACTION DEGREES 0 0 n * 1n 2 12 /2 1 m * 1m 2 12 /2 0 TOTAL < 2n 3 < 2n 3 < 2n 3 1 n 2 1 n < 2n n: the number of ills m1 < n2: the number of contexts p1 < n 2 2: the number of behaviors [ OF PARAMETERS TO BE SET TO CREATE DIFFERENT CHARACTERISTICS 60 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE FEBRUARY 2012

13 TABLE 10 Generated behaviors of four different kinds of Ritys. BEHAVIOR NORMAL RITY CHEERFUL AND OUTGOING RITY DOC-MOT P-MOT P-VALUE DOC-MOT P-MOT P-VALUE CROUCH , SHAKE HEAD WHINE STEP BACK QUICKLY STEP BACK SLOWLY FOLLOW AFTER CLOSELY , FOLLOW SLOWLY # , MOVE AROUND QUICKLY EAT CHEERFULLY , EAT SLOWLY EXCRETE URINE , SLEEP , NAP SCRABBLE QUICKLY , KICK BARK LOUDLY BARK , BARK SOFTLY HOWL , BEHAVIOR OMNIVOROUS RITY TIMID RITY DOC-MOT P-MOT P-VALUE DOC-MOT P-MOT P-VALUE CROUCH , , SHAKE HEAD WHINE STEP BACK QUICKLY , STEP BACK SLOWLY , MOVE QUICKLY FOLLOW AFTER CLOSELY , FOLLOW SLOWLY , MOVE AROUND QUICKLY , EAT CHEERFULLY , , EAT SLOWLY , , EXCRETE , URINE SLEEP , NAP , DIG QUICKLY SCRABBLE QUICKLY , KICK , , GROWL , BARK LOUDLY , BARK , BARK SOFTLY , HOWL , The p-values ere derived from an unpaired t-test. global evaluation of un-generated behaviors as ell. Hence, the behavior selection method using the DoC-MoT is more effective for creating Ritys ith various characteristics than that using the P-MoT. ii) Experiment 2: Changing the characteristics of the omnivorous Rity by punishment Once the normalized eights of ills and contexts to repesent the DoCs for them, are implanted into the Rity s FEBRUARY 2012 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 61