Human Activity Inference Using Hierarchical Bayesian Network in Mobile Contexts

Transcription

1 Human Activity Inference Using Hierarchical Bayesian Network in Mobile Contexts Young-Seol Lee and Sung-Bae Cho Dept. of Computer Science, Yonsei University 134 Shinchon-dong, Seodaemoon-gu, Seoul , Korea Abstract. Since smart phones with diverse functionalities become the general trend, many context-aware services have been studied and launched. The services exploit a variety of contextual information in the mobile environment. Even though it has attempted to infer activities using a mobile device, it is difficult to infer human activities from uncertain, incomplete and insufficient mobile contextual information. We present a method to infer a person s activities from mobile contexts using hierarchically structured Bayesian networks. Mobile contextual information collected for one month is used to evaluate the method. The results show the usefulness of the proposed method. Keywords: Bayesian network, mobile context, activity inference. 1 Introduction Smartphones, such as Apple iphone and Google Android OS based phones, with various sensors are becoming a general trend. Such phones can collect contextual information like acceleration, GPS coordinates, Cell ID, Wi-Fi, etc. Many contextaware services are introduced and tried to provide a user with convenience using them. For example, Foursquare includes a location-based social networking service. It ranks the users by the frequency of visiting a specific location and encourages them to check in the place. Loopt service recommends some visited locations for the users, and Whoshere service shows friends' locations. Davis et al. tried to use temporal, spatial, and social contextual information to help manage consumer multimedia content with a camera phone [1]. Until now, most of such services use only raw data like GPS coordinates. Many researchers have attempted to infer high-level semantic information from raw data collected in a mobile device. Belloti et al. tried to infer a user's activities to recommend suitable locations or contents [2]. Chen proposed intelligent locationbased mobile news service as a kind of location based service [3]. Santos et al. studied user context inference using decision trees for social networking [4]. Most of the research used various statistical analysis and machine learning techniques like B.-L. Lu, L. Zhang, and J. Kwok (Eds.): ICONIP 2011, Part I, LNCS 7062, pp , Springer-Verlag Berlin Heidelberg 2011

2 Human Activity Inference Using Hierarchical Bayesian Network in Mobile Contexts 39 probabilistic model, fuzzy logic, and case based reasoning. However, it is practically difficult to infer high level context because mobile environment includes uncertainty and incompleteness. This paper presents a method to infer human activities from mobile contexts using hierarchical Bayesian networks. 2 Related Works Some researchers have collected contextual information in the mobile environment. VTT research center has developed technologies to manage contextual information and infer higher-level context abstractions from raw measurement data [5]. An adaptive user interface has been also developed by the VTT research center [6]. Helsinki University developed a ContextPhone framework which collected contexts (GPS, GSM cell ID, call history, SMS history, and application in use) on the Nokia 60 series [7]. Some researchers studied object based activity recognition using RFID tags. Patterson et al. examined reasoning with globally unique object instances detected by a Radio Frequency Identification (RFID) glove [8]. They constructed a model with hidden Markov model (HMM) and applied the model to identify what they are cooking. Wyatt et al. studied object based activity recognition using RFID tags, a RFID reader, and models mined from the Web [9]. It is used to solve a fundamental problem in recognizing human activities which need labeled data to learn models for the activities. On the other hand, there are location based activity recognition algorithms. In the approaches, it is assumed that a person s environment affects his activity. Therefore, they often focused on the accurate place detection algorithms rather than activity recognition for practical applications. Liao et al. used relational Markov network and conditional random field (CRF) [10] to extract activities from location information. Anderson and Muller attempted to recognize activities using HMM from GSM Cell-ID data [11]. In this paper, we propose a hierarchical Bayesian network based method to automate activity inference. It raises scalability and precision in recognizing activities by modeling hierarchical Bayesian network with intermediate nodes. 3 Proposed Method In this section, the whole process of activity recognition will be described. The whole system is composed of four components, which are context collection, statistical analysis, activity recognition with hierarchical Bayesian networks, and a user interface as shown in Fig. 1. The hierarchical Bayesian networks are designed based on the context hierarchy [12] and Hwang and Cho s work [13].

3 40 Y.-S. Lee and S.-B. Cho Fig. 1. System Overview 3.1 Context Collection and Preprocessing Context collection is the first step to infer activity. Table 1 shows the information from a mobile phone. In the process, raw data are preprocessed by simple analysis (frequency, elapsed time, etc). Table 1. Contextual information from smartphone Data Attributes Location GPS coordinates, Speed, Altitude, Place name Activity User s activity (reported by manual labeling) Music MP3 title, start time, end time, singer Photograph Time, captured object Call history Start time, end time, person (friend/lover/family/others) SMS history Time, person (friend/lover/family/others) Battery Time, whether charging or not (Yes/No), charge level (%) There are many practical difficulties to recognize user s activities using mobile contextual information in a smartphone. The fundamental problem is that mobile context in a smartphone does not reflect user s activities perfectly. User s activities generate various contextual information and only parts of them are observable in a smartphone. Moreover, important context in a smartphone is often damaged and uncertain. For instance, GPS signals may be invalid or be lost in some locations, especially indoor environment. Schedule stored in a smartphone may be different with the facts because it seems difficult to expect exact coincidence between schedule and user s activities in real life. Sometimes, unexpected events and accidents may occur. On the other hand, call history, SMS history, and photographs have important

4 Human Activity Inference Using Hierarchical Bayesian Network in Mobile Contexts 41 personal information, but are difficult to understand the semantics automatically. We applied several data mining techniques (decision tree, association rules, etc) to analyze and interpret the context. As a result, we distinguish the useful context. Mobile data can be classified into three types such as useful, partially useful and useless contexts to infer a user's activity. In this paper, we ignore useless context, and use other contexts to infer activity. 3.2 Bayesian Network Modeling for Activity Using Context Hierarchy According to Kaenampornpan et al., many researchers defined context and elements of context for context awareness. The goal of a context classification is to build a conceptual model of a userês activity. Activity Theory is a valuable framework used to analyze and model human activities by providing a comprehensive types and relationships of context [14]. It consists of six components which are subject, object, community, rules, division of labor, and artifact. Kofod-Petersen et al. built context taxonomy from six elements found in Activity Theory and applied it to their semantic networks as five context categories: environmental, personal, social, task, spatiotemporal context [12]. Fig. 2 presents our context model structure which is based on context hierarchy to recognize the activities. Fig. 2. Context model for activity recognition The hierarchical context model allows us to implicitly decompose a user s activities into simpler contexts. Intuitively, it might be easier for the model to design probabilistic network modules with hierarchical tree structure rather than the complex one. Further, it is advantageous to break a network for an activity into smaller subtrees and then build models for the sub-trees related to specific context. Fig. 3 shows basic structure of our Bayesian network model for each activity. Bayesian network is a directed acyclic graphical model that is developed to represent probabilistic dependencies among random variables [13]. It relies on Bayes rule like (1) and conditional independence to estimate the distribution over variables. P ( X Y ) ( Y X ) P( X ) P( Y ) P = (1) The nodes in the Bayesian network represent a set of observations (e.g., locations, day of week, etc), denoted as O = {o 1, o 2, o 3,, o n }, and corresponding activities

5 42 Y.-S. Lee and S.-B. Cho (e.g., walking, studying, etc), denoted as A = {a 1, a 2, a 3,, a m }. These nodes, along with the connectivity structure imposed by directed edges between them, define the conditional probability distribution P(A O) over the target activity A. Equation (2) shows that a i, which is a specific activity to recognize, can be inferred from observations. ( A O) P( a o, o, o,..., o ) P (2) i n The proposed model has intermediate nodes, denoted as C = {c 1, c 2,, c m }, to represent hidden variables for activity inference. Fig. 3. Bayesian network structure for activity recognition 3.3 Hierarchical Bayesian Networks for Activity Class In the previous section, Bayesian networks for each activity make it easy to extract activities with only related observations. However, a person can do independent activities at the same time. For instance, he can watch a television and eat something simultaneously. The simultaneous activities cause confusion and mistake to recognize activities because of mixed contextual information. Each Bayesian network module for an activity cannot deal with the situation effectively. We define a global domain for similar activities as activity class. The recognition for global domain of activities is a summary of all similar activities. For instance, Fig. 4 shows the conceptual structure. Let E be the set of all evidences in a given Bayesian network. Then, a Bayesian network factorizes the conditional distribution like P(a E), where a is an activity and every E = {e 1, e 2, e 3,, e n }. It is assumed that e i+1, e i+2, and e i+4 are the observed nodes in a given environment. Under the evidences, a i and a i+2 are the most probable activities, and we may have the possibility of confusion between a i and a i+2. Intuitively, activity class as a global activity captures the compatibility among the variables. Using the hierarchical structure, the conditional distribution over the activity class A is written as P(A a i, a i+1, ).

6 Human Activity Inference Using Hierarchical Bayesian Network in Mobile Contexts 43 Activity Class Ai Ai+1 Activities ai ai+1 ai+2 ai+3 Evidences ei ei+1 ei+2 ei+3 ei+4 ei+5 ei+6 ei+7 ei+8 ei+9 ei+10 Input evidence Fig. 4. Structure for Activity class from activities 4 Experimental Result We define 23 activities according to GSS (General Social Survey on Time Use, 1998) which is a statistical survey for daily activities on time usage in Canada (Statistics Canada, 2005). The activities are suitable to be selected from contextual information regarding user s environment because GSS provides activity types related to location, time and activity purposes. We compare reported activities and inferred activities using our Bayesian networks to calculate the hit rate of the recognition. Fig. 5. Accuracy of activity recognition According to our analysis, time and location are the most important factors to estimate the probability of each activity. That is, location and time dependent activities tend to be detected well as shown in Fig. 5. For example, Night Sleep occurs at specific location (mainly home) and time (mostly night). Examination has

7 44 Y.-S. Lee and S.-B. Cho fixed dates in a semester. Korean regular army training is one of duties in South Korea, of which schedule is determined by the Ministry of National Defense. It is also easy to estimate the occurrence. On the while, location and time independent activities such as Relaxing and Reading books are difficult to detect automatically from contextual information. Fig. 6 illustrates the result of activity class recognition. Fig. 6. Accuracy of activity class recognition Finally, we introduce an interface to annotate a user's activity for some images. It helps a user to check recognized activity related to a photograph. It reduces fatigue of manual annotation. In this point of view, we have developed a prototype annotation tool as shown in Fig. 7. The probability of activity class tended to be more accurate than each activity. A user can easily use their activity to annotate their photographs taken by a mobile phone. Fig. 7. User interface screenshots for activity annotation 5 Summary and Discussion In this paper, we have proposed a method to recognize activities using hierarchical probabilistic models. The system is composed of 4 components, which are context collection, preprocessing and feature extraction, activity recognition, and an interface for visualization and labeling. Bayesian network models for activity refer to context hierarchy and activity hierarchy. It is evaluated with the data collected in real mobile environment. The proposed interface makes labeling easy with effective visualization in order to support recognized activities more accurately.

8 Human Activity Inference Using Hierarchical Bayesian Network in Mobile Contexts 45 Our future research must include more diverse personal information and improve the performance of activity recognition. In addition, various menu interfaces have to be developed for user s convenience. Acknowledgments. This research was supported by the Original Technology Research Program for Brain Science through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology ( ). References 1. Davis, M., Canny, J., House, V.N., Good, N., King, S., Nair, R., Burgener, C., Strickland, R., Campbell, G., Fisher, S., Reid, N.: MMM2: Mobile Media Metadata for Media Sharing. In: ACM MM 2005, Singapore, pp (2005) 2. Bellotti, V., Begole, B., Chi, H.E., Ducheneaut, N., Fang, J., Isaacs, E., King, T., Newman, W.M., Partridge, K., Price, B., Rasmussen, P., Roberts, M., Schiano, J.D., Walendowski, A.: Activity-Based Serendipitous Recommendations with the Magitti Mobile Leisure Guide. In: Proc. of the 26th Annual SIGCHI Conf. on Human Factors in Computing Systems (CHI 2008), pp (2008) 3. Chen, C.-M.: Intelligent Location-Based Mobile News Service System with Automatic News Summarization. Expert Systems with Applications 37(9), (2010) 4. Santos, C.A., Cardoso, M.P.J., Ferreira, R.D., Diniz, C.P., Chaínho, P.: Providing User Context for Mobile and Social Networking Applications. Pervasive and Mobile Computing 6(3), (2010) 5. Korpip, P., Mantyjarvi, J., Kela, J., Keranen, H., Malm, E.-J.: Managing Context Information in Mobile Devices. IEEE Pervasive Computing 2(3), (2003) 6. Korpip, P., Koskinen, M., Peltola, J., Makela, S.-M., Seppanen, T.: Bayesian Approach to Sensor-Based Context-Awareness. Personal and Ubiquitous Computing 7(2), (2003) 7. Raento, M., Oulasvirta, A., Petit, R., Toivonen, H.: ContextPhone - A Prototyping Platform for Context-Aware Mobile Applications. IEEE Pervasive Computing 4(2), (2005) 8. Patterson, J.D., Fox, D., Kautz, H., Philipose, M.: Fine-Grained Activity Recognition by Aggregating Abstract Object Usage. In: 9th IEEE International Symposium on Wearable Computers, pp (2005) 9. Wyatt, D., Philipose, M., Choudhury, T.: Unsupervised Activity Recognition using Automatically Mined Common Sense. In: Proc. of the 20th National Conference on Artificial Intelligence, pp (2005) 10. Liao, L., Fox, D., Kautz, H.: Extracting Places and Activities from GPS Traces using Hierarchical Conditional Random Fields. International Journal of Robotics Research 26(1), (2007) 11. Anderson, I., Muller, H.: Department of Computer Science: Practical activity recognition using GSM data. Technical report CSTR , University of Bristol (2006) 12. Kofod-Petersen, A., Cassens, J.: Using Activity Theory to Model Context Awareness. In: Roth-Berghofer, T.R., Schulz, S., Leake, D.B. (eds.) MRC LNCS (LNAI), vol. 3946, pp Springer, Heidelberg (2006) 13. Hwang, K.-S., Cho, S.-B.: Landmark Detection from Mobile Life Log using a Modular Bayesian Network Model. Expert Systems with Applications 36(3), (2009) 14. Kaenampornpan, M., O Neill, E.: Modelling Context: An Activity Theory Approach. In: Markopoulos, P., Eggen, B., Aarts, E., Crowley, J.L. (eds.) EUSAI LNCS, vol. 3295, pp Springer, Heidelberg (2004)