Research Article Sleep Monitoring System Using Kinect Sensor

Transcription

1 Distributed Sensor Networks Volume 2015, Article ID , 9 pages Research Article Sleep Monitoring System Using Kinect Sensor Jaehoon Lee, 1 Min Hong, 2 andsungyongryu 3 1 Department of Computer Science, Graduate School, Soonchunhyang University, Sinchang, Chungcheong, Republic of Korea 2 Department of Computer Software Engineering, Soonchunhyang University, Sinchang, Chungcheong, Republic of Korea 3 Department of Business Administration, Soonchunhyang University, Sinchang, Chungcheong, Republic of Korea Correspondence should be addressed to Sungyong Ryu; jackie@sch.ac.kr Received 21 January 2015; Accepted 15 April 2015 Academic Editor: Neil Y. Yen Copyright 2015 Jaehoon Lee et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Sleep activity is one of crucial factors for determining the quality of human life. However, a traditional sleep monitoring system onerously requires many devices to be attached to human body for achieving sleep related information. In this paper, we proposed and implemented the sleep monitoring system which can detect the sleep movement and posture during sleep using a Microsoft Kinect v2 sensor without any body attached devices. The proposed sleep monitoring system can readily gather the sleep related information that can reveal the sleep patterns of individuals. We expect that the analyzed sleep related data can significantly improve the sleep quality. 1. Introduction Recently, IoT (Internet of Things) technology has received great attention as one of the key issues for future technology and it provides the wireless connection to achieve the necessary information between any devices and systems. This IoT technology is expected to provide more convenient living environments to people in the near future. IoT technology is generally related with human behaviors as a commend input from various information that can be utilized to control the devices such as human face detection [1]and human gestures [2 4]. IoT technology has been also applied for the energy saving to efficiently control the limited natural resources [5]. This information can be a key element to design and implement the smart home [6]. Smart home platform has complicated connections to be communicated each other between all devices and natural behaviors of individuals should be detected by sensors to trigger the devices or systems [7]. Smart home platform can be included or combined with other technologies and the human health related wellness technology is one of them. The proposed sleep monitoring can be one of the elements for smart home system and it cantakecareorimprovethehumanlifethroughsleepstate monitoring and it is one part of improving human wellness. Recent human life is intensively busy and has been exposed to stressful work tasks and these cause the increasing number of insomnia patients as shown in Figure 1 [8]. Sleep disturbing has been known for causing the increased learning disabilities, inefficient work process, and unnecessary accidents that can cause serious illness [9]. Sleep disturbing canbecausedbyvariousfactorsandthediagnosisofsleep disturbing is also a highly difficult task. For sleep disturbing examination, patients should visit the hospital and should wear various devices; thus, they cannot sleep well due to unfamiliar sleep environments in hospital. In this paper, we introduce the Kinect sensor based simple sleep monitoring system to gather the sleep movement, posture, and environment not only to collect sleep movement information but also to extract the overall sleep information. Since it does not require us to wear any troublesome sensing devices, the proposed system can provide more natural and comfort sleep environments to collect the sleep related information. 2. Related Works Lee et al. [10] used acceleration sensors which are attached on epigastrium, mesogastrium, and lower abdomen using sleepwear. This study used 3 acceleration sensors that can detect the sleep posture and breathing during sleep. This detection is using voltage information from acceleration sensors. However, this approach cannot detect breathing

2 2 Distributed Sensor Networks Figure 2: Polysomnography devices Men Women Figure 1: Sleep difficulties at least once a week (%). when a patient is lying on side and front, because 3 axis acceleration sensors cannot detect the movement of chest and abdomen. U-healthcare based sleep monitoring system for sleep control and remote monitoring is applied with sensors and switches, and it shows an example of healthcare system using modified sleep mat [11]. This example is comprised of related sensing and receiving data part and controlling and all sleep related situations monitoring part. It used TMO (timetriggered message-triggered object) scheme model, and it collects the real time sleep data. Sleep detection module is separated in 3 parts which are position detect sensor, sound detect sensor, and vibration sensor to warning. In the monitoring system, on/off sensor collects the sleep motion data and it determines that the sleep position is good or bad. When the decision case is bad, the vibration sensor is working for user to notify the situation. In sleep monitoring via depth video image compression and analysis [12], the sleep detection is monitored by a Microsoft Kinect v1 using depth image and video. In this study,theydidnotuseskeletontrackingapproachwhichis included in the Kinect v1 SDK. The main purpose of this study was the detection of hypopnoea though the depth video image. The program can detect the change in depth image via the movement of the chest. However, it has limitation which is it can detect only hypopnoea that is just one of the symptoms of sleep disturbing. SmartbedroomforelderlyusingKinectisalsopurposed and implemented to help them by providing smart home environments using Kinect v1 [13]. This system mainly detects the arms motion and gives a warning for the risk of bed falling. To notify the risk of bed falling, it sets up the four corner points in the bed. This system detects and tracks the human body using skeleton tracking method in Kinect v1 SDK, and it notifies the warning when those skeleton points are out of the corners in bed. Also, it can recognize the arms motion that conveniently requests the help of other people for the elderly who find it difficult to get up from the bed. It tracks the angle of arm which is calculated by the elbow-towrist vector. However, it has the same limitation in general video recognition approach which makes it difficult to detect the human body when people use a blanket in bed. 3. Sleep State Monitoring Generally, there are two ways to detect the sleep state. We can directly attach some sensors to human body or we can record body movements during sleep to analyze the sleep quality. The direct sensor attachment based method can provide some important sleep related information from human body. In addition, it can provide scientifically accurate sleep state information. However, the wearing of complicated sensing devices leads to uncomfortable sleep environments and it may negatively affect the measurement of sleep state. Figure 2 shows the necessary devices for nocturnal polysomnography to diagnose the sleep state. Video based approach is also able to conveniently provide thesleepstate,butitismoredifficulttoanalyzethesleep state than attached sensors. Recently, color, infrared, and depth information can be obtained by camera and some other unwearable sensors are also able to achieve the sleep related information. In order to monitor the sleep state with video, it is necessary to recognize the shape of human body and then the image processing based approach is essential to extract the human body from the input images. Infrared and IIR (Imaging Infrared), differ integrator, skin segmentation [14], andskincolorandedgedetection[15] havebeenwidely applied to detect and track the movement of human body from video images Microsoft Kinect v2. In this paper, we utilized the Microsoft Kinect v2 to detect and track the movement of human body. We selected the Kinect v2 which is the latest version of Kinect and this version of Kinect provides the position of 25 joints, and detection range has been more improved than previous Kinect v1. In addition, Kinect v2 changedir(infrared)irradiationtotof(timeofflight) which is configured with one pair of receptors. The principle of TOF is the light reflection that measures the distance by the receptor. This new depth detection method provides much more accurate depth information than Kinect v1. Kinect v2 is only performed under

3 Distributed Sensor Networks 3 Kinect v2 TI sensor tag Bedroom Sleep monitoring system Sleep monitoring data server Depth Infrared Figure 3: Depth and infrared image from Kinect v2. Windows 8 OS (Operating System) and over Visual Studio Therefore, the proposed system is implemented with C++ using Kinect v2 SDK. We also utilized OpenCV library to detect and track the human body movement during sleep. The proposed system can recognize the human body through an infrared sensor and a depth sensor which are shown in Figure 3. The infrared sensor of Kinect v2 detects the reflected infrared rays and provides the movement of users with pixel by pixel information. The depth sensor can achieve the depth information which is made up with CMOS sensor that converts light to electrical image signal with infrared ray projector. The Kinect v2 includes the specific human body information and a matching algorithm which can detect human body movement with sensing information. Therefore, we can readily gather the sleep movement related information with Kinect Architecture of Sleep Monitoring System. Figure 4 shows the device architecture of proposed sleep monitoring system that automatically communicates with each device. This service stores the daily sleep state of users into the sleep monitoring data server. This sleep monitoring system stores user related information such as name, age, height, weight, and medical history. Also, sleep related information is stored into the database which is shown in Table Measurement of Sleep Movement In order to monitor user s movements during sleep, the proposed system should not only detect the human shape from image but also continuously track the movement of human body. Unlike wearing many sensors on human body, theproposedsystemuseskinectv2cameraforaninputimage to detect and track the movement of human body. Input data Monitoring data Sleep monitoring application Figure 4: Device relationship diagram. Table 1: Data field of system. Name Age Gender Height Weight Brief medical history Movement information during sleep Change of sleep posture Room temperature and humidity Since the Kinect v2 provides detecting of human body joints, it is possible to estimate the degree of human body movement. Because the Kinect v2 requires the minimum distance for object detection, we have to locate the Kinect v2 camera on the appropriate position which is between 0.5 m and 4.5 m from object. Therefore, we installed the aluminum angles to locate the Kinect camera at 2.0 m high on the bed. Figure 5 shows the configuration of proposed sleep monitoring system and the located Kinect camera to detect whole human body joints without any distortions.

4 4 Distributed Sensor Networks Kinect v2 TI sensor tag Figure 5: Configuration of the proposed sleep monitoring system. X, Y, Z m: movement h: hour Amount of major joints movement m/h Amount of minor joints movement m/h Figure 7: Comparing the movement value of major joints and minor joints. Figure 6: Detected 25 joints for body tracking. The Kinect v2 provides x, y, andz positions of 25 joints and distance data of objects from depth sensor. Since it is necessary to detect the overall movements of the human body, we store x, y,andz positions of all joints in every half second and Figure 6 shows the detected 25 joint positions. Because sleep movements mainly occur by limbs, we selected critical 19 joints relating with the sleep movements through the comparison and they are head, left shoulder, left elbow, left wrist, left hand, left hip, left knee, left ankle, left foot, left thumb, right shoulder, right elbow, right wrist, right hand, right hip, right knee, right ankle, right foot, and right thumb. Figure 7 depicts the total sleep movement value of 19 major joints and movement value of other joints. Since minor joints do not critically affect the body movement during sleep, we excluded the value of these joints from our calculation. The sleep movement value is calculated by using the Euclidean distance between previous position of joints and current position of joints from image sequences. All movement values of major joints are calculated every 0.5 seconds. The formula to obtain the sleep movement value is shown as (1). SM (sleep movement) is a distance of moved coordinate from previous coordinate of 19 key joints to current coordinate. X i ciscurrentx position, and X i pispreviousx position in image sequences: SM = 19 i=1 (Xi c X i p) 2 +(Y i c Y i p) 2 +(Z i c Z i p) 2. After calculating the first sleep movement value, the sleep monitoring system keeps accumulating the sleep movement valueuntiltheuserwakesup.atthesametime,theproposed sleep monitoring system compares image with 5 reference sleep postures to determine user s current sleep posture and Figure 8 shows the flowchart of proposed sleep monitoring system. Sleep monitoring system follows the above flow of process and the sleep monitoring result is provided by a smart phone application. The main page of smart phone application is shown in Figure 9. In addition, TI (Texas Instruments) sensor tag which is shown in Figure 10 is used to obtain the temperature and humidity of sleep environments. This sensor is connected with sleep monitoring application by Bluetooth, and this sensor provides 6 main pieces of data such as temperature, (1)

5 Distributed Sensor Networks 5 Start No Tracking body Yes Extracting movement data form major joints Extracting all joints data Image matching using SURF and boundary Analyzing movement and posture Saving movement data and posture data Providing movement and posture information of sleep End Figure 8: Flowchart of proposed sleep monitoring system. humidity, pressure, accelerometer, gyroscope, and magnetometer. For this study, we measured the sleep related environments with temperature and humidity sensors. 5. Sleep Movement Data Analysis For experimental test, 20 students participated for sleep state monitoring and the average age of participants was 23 years. We performed the sleep monitoring around 7 hours and the participants bodies were not covered with any blankets to accurately detect the sleep motions. We tried to provide similar sleep environments such as time of falling asleep, temperature, lighting and noise level of room, and humidity conditions for participants. There are general 6 sleep postures which are shown in Figure 11 such as foetus, log, yearner, soldier, freefaller, and starfish. In our system, we monitored the sleep posture of participants with 5 categories. Figure 12 shows the result of sleep movement and the change of sleep posture during sleep experiment. In Figure 12, x axis is a sleep time and y axis is a sleep movement value for left figure and y axis is type number of sleep postures for right figure, respectively. The sleep movement value is sharply altered when the body posture is changed. After the body posture is stabilized, the detected motion data is also almost stable. The proposed sleep monitoring system can precisely count the body movements and it returns the correct counts number for body posture change. In other words, the number of high peeks from the steady constant values signifies the change of body posture. In this study, we added up all SM values at each hour to make a decision for sleep state. Since we measure the sleep movement value every 0.5 seconds, we summed up 7,200 times. For HSM (Hourly Sleep Movement), collected range ofhsmvalueswas and we standardized this HSM valuein0to1.0by(2). In our system, we decide the sleep quality by the following: under 0.25 as a good with deep sleep, 0.25 to under 0.59 as a normal with regular sleep, and 0.59 to 1 as a bad with light sleep which is shown in (3). This criterion is decided from the range of entire SM value with video monitoring and user evaluation for sleep. In our experimental test, deep sleep was 43 hours (30.7%), regular sleep was 61 hours (43.6%), and light sleep was 36 hours (25.7%) in 140 testing hours. Table 2 shows the judgment of sleep movement per hour

6 6 Distributed Sensor Networks Foetus Log Yearner Soldier Freefaller Starfish Figure 11: Typical 6 sleep postures. Figure 9: Main page of sleep monitoring application on smart phone. Figure 10: TI sensor tag for temperature and humidity measurement. for 20 participants and the grade of sleep movement is the average of HSM value: HSM = 7200 t=0 (SM/1000) min max min Good (Deep Sleep): 0<HSM < 0.25 Normal (Normal Sleep): 0.25 HSM < 0.59 Bad (Light sleep): 0.59 HSM Sleep Posture Data Analysis Sleep posture is also an important factor for sleep quality. In general, people have 6 typical sleep postures that are founded (2) (3) through the research targeted for one thousand people by SAAS (Sleep Assessment and Advisory Service). According to their research, the sleep posture has some relationship with human characters. Thus, we analyze the sleep posture with 5 categories: foetus type, log type, yearner type, soldier type, and freefaller and starfish type which are shown in Figure 11. To determine the sleep posture, we mainly applied the boundary information for location of 4 key joints (left hand, left knee, right hand, and right knee). The posture change is robustly detected by the proposed posture verification algorithm which is based on the location of 4 joints in predefined boundary. The proposed posture verification algorithm findsthreebaselines(middle,upper,andlowerline)to determine the sleep posture with boundary information. The middle line is defined with extended line between neck joint and spine base joint. The upper line is determined with the spine midjoint and it is perpendicular to middle line. The lower line is determined with spine base joint and it is also perpendicular to middle line. With these three base lines which are shown in Figure 13, we can readily determine 5 sleep postures with locations of 4 key joints in boundary. In addition to the posture verification algorithm with boundary information, we used SURF (Speeded-Up Robust Features)methodinopenCVlibrarytofigureoutthesleep posture with extraction of feature points. The reference posturesarepredefinedbyimageandtheproposedalgorithm extracts the feature points between reference and current image and then determines the posture with feature point matching which is shown in Figure 14. Figure 15 shows the decisions of sleep posture type with the proposed sleep monitoring system. The sleep posture is determined with the combination of boundary and SURF method and we gave the priority to boundary information based method. When the position result is not well determined by boundary information based method due to failure of joint detection, we utilized the result of SURF based method to alleviate this problem. 7. Conclusions In this paper, we designed and implemented the sleep monitoring system using Microsoft Kinect v2 that does not require users to attach any inconvenient devices. The proposed system provides comfortable and natural sleep environments for experiment; thus, we believe that it is well suitable to

7 Distributed Sensor Networks 7 Figure 12: Result of experiment (movement and posture). Foetus Log Yearner Soldier Freefaller Starfish Figure 13: Sleep postures based on boundary information. Reference image Figure 15: Decision of sleep posture type. Current image Figure 14: Feature points matching with SURF method. monitor the ordinary sleep conditions. Also, it can monitor the exact number of sleep position changes during sleep. According to the experimental result data, we can detect sleep disorders such as restless legs syndrome in further research. The proposed sleep monitoring system can measure the sleep quality and can notify the daily sleep condition with sleep time, sleep posture, and sleep quality using application. It can also figure out the pleasant sleep condition for users with room humidity and temperature. However, the proposed system has a drawback which is that users could not use blankets, because depth sensor cannot work with blanket which covers human body. In addition, it is difficult to recognize front and back of sleep posture. For further research, we will add multi-kinect cameras to achieve more detailed information from the different range of viewpoints. Also, we will apply the Kinect v2 sound array sensor that can determine snoring and apnea of users [16]. We believe that we can comprehensively improve the proposed sleep monitoring system using the Kinect and unattached sensors for analyzing critical sleep factors such as snoring and external sleep disturbance sources. Disclosure This paper is an extended version of International Symposium on Wellness Technology and Service for Future Life Care (Wellness 2014) [17].

8 8 Distributed Sensor Networks Table 2: The amount of movement per hour. P T 1h 2h 3h 4h 5h 6h 7h Grade Case N Case N Case N Case G Case G Case N Case N Case G Case N Case B Case N Case N Case G Case N Case G Case N Case N Case N Case B Case G P: participant, T: time, andh: hour; Grade(G: good, N: normal, andb: bad). Conflict of Interests There is no conflict of interests regarding the publication of this paper. Acknowledgments This research was supported by the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the C-ITRC (Convergence Information Technology Research Center) (IITP-2015-H ) supervised by the IITP (Institute for Information & Communications Technology Promotion). This work was supported by the Soonchunhyang University Research Fund. This research is extended by Analysis of Sleep Disturbing Factors using Kinect Sensors of CSA- 14 (International Conference on Computer Science and its Applications). References [1]X.Yang,G.Peng,Z.Cai,andK.Zeng, Occludedandlow resolution face detection with hierarchical deformable model, The Journal of Convergence,vol.4,no.2,pp.11 14,2013. [2] J.K.Lee,W.M.Kang,J.H.Park,andJ.S.Kim, Gesture-based wearable device for secure and effective information exchange on battlefield environment, The Journal of Convergence, vol.5, no.4,pp.6 10,2014. [3] A. Hussain, A. R. Abbasi, and N. Afzulpurkar, Detecting & interpreting self-manipulating hand movements for student s affect prediction, Human-Centric Computing and Information Sciences,vol.2,article14,18pages,2012. [4]A.Chaudhary,K.Vatwani,T.Agrawal,andJ.L.Raheja, A vision-based method to find fingertips in a closed hand, Journal of Information Processing Systems, vol.8,no.3,pp , [5] E.-J. Lee, C.-H. Kim, and I. Y. Jung, An intelligent green service in internet of things, The Journal of Convergence, vol. 5, no. 3, pp.4 8,2014. [6] J. Vanus, P. Kucera, R. Martinek, and J. Koziorek, Development and testing of a visualization application software, implemented with wireless control system in smart home care, Human- Centric Computing and Information Sciences, vol.4,article18, 19 pages, [7] N. Howard and E. Cambria, Intention awareness: improving upon situation awareness in human-centric environments, Human-Centric Computing and Information Sciences,vol.3,no. 9, pp. 1 17, [8] A.-M. Lehto and H. Sutela, Three Decades of Working Conditions. Findings of Finnish Quality of Work Life Surveys , Statistics Finland, [9] B. Sivertsen, T. B. Lallukka, P. Salo et al., Insomnia as a risk factor for ill health: results from the large population-based prospective HUNT Study in Norway, Journal of Sleep Research, vol.23,no.2,parti,pp ,2014. [10] J.-H. Lee, D.-J. Kim, and K.-H. Kim, Studies on development of sleeping and respiration pattern monitoring system use a 3 axis acceleration sensor, in Proceedings of the 2010 Conference oninformationandcontrolsystems, pp , [11] D.-H. Kim, C.-W. Jeong, and S.-C. Joo, U-healthcare based system for sleeping control and remote monitoring, Journal of Internet Computing and Services,vol.8,no.1,pp.33 45,2007. [12] C. Yang, G. Cheung, K. Chan, and V. Stankovic, Sleep monitoring via depth video compression & analysis, in Proceedings

9 Distributed Sensor Networks 9 of the IEEE International Conference on Multimedia and Expo Workshops (ICMEW 14), pp. 1 6, Chengdu, China, July [13] Y. Booranrom, B. Watanapa, and P. Mongkolnam, Smart bedroom for elderly using kinect, in Proceedings of the 2014 Computer Science and Engineering Conference, pp , [14] Z.H.Al-Tairi,R.W.Rahma,M.I.Saripan,andP.S.Sulaiman, Skin segmentation using YUV and RGB color spaces, Journal of Information Processing Systems, vol.10,no.2,pp , [15] D. Ghimire and J. Lee, A robust face detection method based on skin color and edges, Journal of Information Processing Systems, vol.9,no.1,pp ,2013. [16] L. C.-L. Chen, K.-W. Chen, and Y.-P. Hung, A sleep monitoring system based on audio, video and depth information for detecting sleep events, in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME 14), pp. 1 6, Chengdu, China, July [17] J.-H. Lee, H.-S. Yum, D.-S. Park, and M. Hong, Analysis of sleep disturbing factors using Kinect sensors, in Proceedings of the International Symposium on Wellness Technology and Service for Future Life Care (Wellness 14),p.142,2014.

10 Rotating Machinery Engineering Journal of The Scientific World Journal Distributed Sensor Networks Journal of Sensors Journal of Control Science and Engineering Advances in Civil Engineering Submit your manuscripts at Journal of Journal of Electrical and Computer Engineering Robotics VLSI Design Advances in OptoElectronics Navigation and Observation Chemical Engineering Active and Passive Electronic Components Antennas and Propagation Aerospace Engineering Modelling & Simulation in Engineering Shock and Vibration Advances in Acoustics and Vibration