Mobile System Design for Scratch Recognition

Size: px

Start display at page:

Download "Mobile System Design for Scratch Recognition"

Veronica Singleton
6 years ago
Views:

1 Mobile System Design for Scratch Recognition Jongin Lee Dae-ki Cho Dept. of Computer Science, Princeton University 35 Olden Street, Princeton NJ USA Seokwoo Song Dept. of Computer Science, KAIST SeungHo Kim Eunji Im John Kim Dept. of Computer Science & Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). CHI 15 Extended Abstracts, Apr 18-23, 2015, Seoul, Republic of Korea ACM /15/04. Abstract Various conditions can result in scratching behavior and severe itching conditions such as atopic dermatitis can significantly impact on one s quality of life. Because the management of many itching conditions is not necessarily about curing the condition but instead about properly maintaining or controlling the condition, a proper understanding of these types of conditions and accurate recognitions of scratching behavior are important. In particular, a clear understanding of scratching behaviors can be difficult for young children or for those who engage in nocturnal itching while sleeping. In this work, we design a prototype system with two smartwatches which recognize scratching motions and validate the results using an infrared camera to provide the ground truth. We classify scratching motions with the C4.5 algorithm and analyze the data to understand the scratch patterns and to validate our system. Our initial results from the sleep data of three healthy participant show that the proposed method can provide accuracy that exceeds 90%. Author Keywords Mobile System; Smartwatch; Scratch Recognition ACM Classification Keywords J.3 [Life and Medical Science]: Health.; C.3 [Special-purpose and Application-based Systems]: Signal processing systems 1567

Infrared camera Smart watch zzz Accelerometer Validation Figure 1: The Prototype Implementation Overview Video stream Introduction Itching is a common symptom in our everyday lives.

As a result, measuring the severity (and intensity) of itching is important not only to provide appropriate medical care but also to determine if a certain medication has the desired effect,

Itching sensations result in scratching motions which are very noticeable; however, sometimes scratching motions can be done unconsciously or without a person being aware of the motion.

In this work, we propose the initial design and implementation of a mobile system that automatically and objectively measures the extent and severity of nocturnal itching.

2 Infrared camera Smart watch zzz Accelerometer Validation Figure 1: The Prototype Implementation Overview Video stream Introduction Itching is a common symptom in our everyday lives. There are various causes of itching symptoms, including insect bites, allergies, skin conditions, and even dermatitis. Although slight itching (i.e., from dry skin) is not problematic, intense or chronic itching from certain diseases such as dermatitis or allergies can detrimentally affect ones quality of life. As a result, measuring the severity (and intensity) of itching is important not only to provide appropriate medical care but also to determine if a certain medication has the desired effect, especially by determining how the itching severity changes [7]. Itching sensations result in scratching motions which are very noticeable; however, sometimes scratching motions can be done unconsciously or without a person being aware of the motion. In particular, scratching done during the evening while one is sleeping, commonly referred to as nocturnal itching, is much more difficult to understand or measure. In this work, we propose the initial design and implementation of a mobile system that automatically and objectively measures the extent and severity of nocturnal itching. Application One significant application of this system can be for patients with atopic dermatitis, a condition characterized by significant itching. Medications can be used to treat atopic dermatitis, but some are not necessarily effective for all patients. In addition, various factors (e.g., food, temperature, humidity and other environmental factors) can further exacerbate atopic dermatitis. As a result, patients are suggested to maintain a diary and to measure the severity of their atopic dermatitis daily so as to control and manage exacerbating factors by themselves. Examples of these include the PO-SCORAD (Patient-Oriented SCORAD) [11] index, an extension of the SCORAD (SCORing Atopic Dermatitis) index, which is generally used to measure the extent and severity of atopic dermatitis in the medical field [10]. However, measurements using the PO-SCORAD index are commonly affected by the patients perspective or mental condition on a particular given day and are not necessarily objective. In addition to atopic dermatitis, properly understanding itching (i.e., what causes it and what does not cause it) is important to remove misunderstanding of the patients condition [13] (e.g., food allergy and atopic dermatitis) while also possibly receiving better feedback from the doctor. This work can also be very significant for infants and young children who suffer from chronic itching conditions. Unlike adults, who can directly express their conditions and pain, infants and young children may not be able verbally to communicate their conditions accurately. As a result, this work can have significant impact on young patients with chronic itching by providing more objective information about their itching; possibly leading to better maintenance and control of their conditions. In this work, we do not test our system with patients with atopic dermatitis, as our initial focus was on the design of a mobile system in an effort to understand general scratch and sleep patterns. We expect that such patients may exhibit different behaviors, and our system will likely need to be tuned to accurately reflect their behaviors. Related Work Patients with pruritus are known to suffer from nocturnal itching. Measuring nocturnal scratching is important for objectively measuring the extent and severity of pruritus, as subjective assessments such as SCORAD, EASI, and VAS have different results among experts and patients [8, 10, 11]. Scratch Monitor [4], a cotton glove with a pressure sensor, has been suggested to evaluate 1568

Figure 2: Smartwatch with 3 axes labels Linear acceleration Active movements extraction Feature extraction Scratch classification Detailed scratch analysis Figure 3: Acceleration data analytics

Although this method is simple, the glove disturbs patients sleep, and simply using consecutive changes in pressure cannot accurately detect scratch movements.

However, a long time is necessary to go through one night of recording, and blind positioned hands (i.e., hands covered with a blanket) are difficult to identify [3, 9].

The mean value of accelerometers has been shown to be highly correlated to scratching activities from infrared camera data [2].

However, these devices cannot accurately differentiate scratching activities from other activities, such as turning over or restless movements, because they count the number of scratching events when

Recently, a watch-type scratch sound detector [9] was introduced to detect body-conducted sound from scratch movements.

3 Figure 2: Smartwatch with 3 axes labels Linear acceleration Active movements extraction Feature extraction Scratch classification Detailed scratch analysis Figure 3: Acceleration data analytics procedure acceleration (m/s 2 ) (a) lag (b) Figure 4: (a) Scratch graph and (b) lag-time autocorrelation heatmap from the scratch graph nocturnal scratching objectively. Although this method is simple, the glove disturbs patients sleep, and simply using consecutive changes in pressure cannot accurately detect scratch movements. An infrared camera was also introduced objectively to identify the severity of itching in patients with atopic dermatitis [3]. However, a long time is necessary to go through one night of recording, and blind positioned hands (i.e., hands covered with a blanket) are difficult to identify [3, 9]. In the medical community, watch-type accelerometers (e.g., DigiTrac, Actiwatch MiniTM) are used for measuring nocturnal scratching. The mean value of accelerometers has been shown to be highly correlated to scratching activities from infrared camera data [2]. Other researches also have shown that objective clinical scores, including SCORAD, are closely correlated with wrist activities [7, 6]. However, these devices cannot accurately differentiate scratching activities from other activities, such as turning over or restless movements, because they count the number of scratching events when the mean value of the accelerometer exceeds a certain threshold value. In addition, these devices are mainly used for the purpose of research and cannot practically be used for patient care purposes. Recently, a watch-type scratch sound detector [9] was introduced to detect body-conducted sound from scratch movements. Although the method shows impressive performance with regard to accuracy, it often fails to identify the location of the scratching site. HealthSense [12] compared three different machine-learning techniques, Naive Bayes, C4.5, and a neural network method, finding that the C4.5 method works the best for detecting scratching activities. However, they used simulated data in their experiments [12, 5]. Our work differs from this in that our system is optimized for a chronic disease such as atopic dermatitis. Moreover, we analyze data collected from an actual experiment. Designing a Wearable/Mobile System We implement a prototype using two smartwatches and an infrared camera for validation (see Figure 1). Smartwatch as a Wearable Sensor Selecting an appropriate smartwatch is important in that the smartwatch itself may disturb sleep and even cause itching of the skin or damage to it. While there are light weight wrist sensors such as Jawbone, we exploit the Samsung Galaxy Gear Live smartwatch due to the fact that it provides APIs for minute-by-minute raw data collection of three inertial sensors, i.e., an accelerometer, a gyroscope and a magnetometer. For our initial study, we focus on analyzing scratch patterns based on the accelerometer data. The accelerometer has three axes, each in a certain direction, as shown in Figure 2. The gyroscope and the magnetometer sensors are used in the 3D modeling of wrist movements for detecting spots scratched, which remains as a future work. Overview of the Data Analytics Procedure Figure 3 illustrates the process of the accelerometer data analysis. The three axes of linear acceleration are continuously sampled at 50Hz. We partition the samples into processing units called windows. Each window contains 128 samples (2.56 seconds), and the second half of the samples within the window overlaps the next window (e.g., three windows for 256 samples). We overlap the windows to minimize misdetections of windows containing active movement, as proposed in an earlier study [1]. We evaluate the acceleration values of each window and extract the windows with significant acceleration data. These extracted windows are then used to extract features such as the mean, correlation, or 1569

4 (f)autocorrelation (e)correlation (d)energy (c) Entropy (b) Mean (a)acceleration(m/s 2 ) Scratch Non-scratch, window size:128 Figure 5: Comparison of extracted features between scratch and non-scratch movements entropy. We use such extracted features to classify the active movements with a decision tree algorithm (i.e., C4.5). Once we build the tree, it is used to detect scratching. Extracting Active Movements In our prototype, a window is regarded as an active movement when the magnitude of three axes in each window is greater than This process removes all of the stationary windows and minimizes the windows for feature extraction. Feature Extraction For a better understanding of scratching, we simulated scratch and non-scratch movements (e.g., irregular wrist movements) while lying down. In this prototype, we chose five features, the mean, entropy, energy, correlation and autocorrelation, because they can effectively distinguish scratch from non-scratch movements. An example of the features extracted from the X -axis is shown in Figure 5. For the classification, we extract five features from an active movement window. We separately generate the feature values on three axes, thus we get 15 feature values from one window. In a time-domain window, the mean feature is calculated by averaging the values, and the correlation is calculated between each pair of axes. Autocorrelation is the correlation between past and future values within a time-domain window. The distance (or lag) between past and future values should be specified in advance of calculating the autocorrelation value. Figure 4 shows the autocorrelation value according to lag. When lag is 15, the autocorrelation shows the smallest value (see Figure 5-(f)). To obtain the entropy and the energy, we normalize the accelerometer values by subtracting the mean of the acceleration values of each axis in each window. We then convert the domain of the normalized window from time to frequency using the fast Fourier transform (FFT) method. Finally, we compute the entropy and the energy values based on the equation below. Suppose x 1, x 2,... denotes the FFT components of the window. Then, Entropy = w i=1 P (x i)log 2 P (x i ) w x where P (x i ) = i 2 i=1 w, and Energy = x 2 w. xj 2 j=1 Figure 5-(c) and Figure 5-(d) respectively plot the entropy and the energy values we obtained from our dataset. Because the scratch movements have higher variances in their acceleration values, the energy is higher for scratch movements in (d). On the other hand, (c) shows low entropy values when scratching due to the periodic movements of the wrist in that case. Detailed Scratch Analysis We conduct an additional analysis of scratching in an effort to understand the characteristics of the scratch patterns. In this prototype, we extract the distance moved, the velocity of the wrist, and the number of scratching. Because scratching is a periodic movement, we regard consecutive changes of direction of the wrist movements as the number of scratching. The original acceleration-time graph in Figure 6-(a) is converted to a velocity-time graph in Figure 6-(b) to obtain accurate time(millisecond) time(millisecond) time(millisecond) changes of the directions of scratch movements as well as the distances of the scratch movements to calculate the velocity. acceleration (m/s 2 ) acceleration (m/s 2 ) time(millisecond) time(millisecond) time(millisecond) (a) (b) (c) Figure 6: The window for scratch movements (with wrist movements): (a) acceleration-time graph, (b) velocity-time graph from the acceleration-time graph, and (c) velocity-time graph with extracted slopes 1570

5 Table 1: C4.5 Classifier results Accuracy(%) Classifier P1 P2 P3 C4.5 R L * R = right watch, L = left watch P1, P2, P3 = three participants (b) Normalized count (a) Normalized velocity time (hour) time (hour) Figure 7: Normalized (a) velocity and (b) the number of times of scratching from X -axis in left watch of P1 However, we cannot obtain an accurate velocity-time graph because the accelerations are inaccurate, and they constantly affect the values across the velocity-time graph. This is known as the drift problem. We plan to optimize this further in a future work. To estimate the number of scratching, we use the direction changes of consecutive slopes, as illustrated in Figure 6-(c). Lastly, the distance is obtained by means of integration in the velocity-time graph. People may use their wrists and fingers to scratch, but often they use only their fingers. Although scratching with the fingers without significantly moving the wrist causes a tiny amount of acceleration, the acceleration pattern produces the same graph as the pattern of scratching with wrist movement. Noises In the experiment, we found that the smartwatch generates a tiny amount of noise, even when it is stationary, for several reasons. These include the vital signs of the human wearer, or the small amount of inherent in the sensor. In fact, the noise affects the process of integrating the acceleration data to determine the velocity. Moreover, after significant acceleration, the value fluctuates and time of a few hundreds of milliseconds is required to return to the origin (i.e., zero). As future work, we plan on developing a noise-filtering method to characterize the scratch patterns more accurately. Preliminary Experiments We have extended the experiments to real-world experiments. To do this, we recruited three healthy participants (P1, P2 and P3) and collected sensor data from each participant. Each participant was asked to put on the smartwatches before going to bed, and the two watches recorded sensor data for four hours, storing the data in internal storage. The collected data is processed as described in Figure 3. To create the ground truth, we manually watched a recorded infrared video and labeled each window as either a scratch or a non-scratch movement. Table 2: Confusion Matrix for the C4.5 classifier P1 P2 P3 Classifier as Scr Non Scr Non Scr Non L Scr Non R Scr Non * Scr=scratch,Non=non-scratch,L=left watch,r=right watch Results We utilize the C4.5 decision tree algorithm as a statistical classifier. The classification accuracy is obtained by means of 10-fold cross-validation. Overall, our classifier shows greater than 90% accuracy, as shown in Table 1. A confusion matrix can be found in Table 2. We also estimated the velocity and the number of scratching from the scratch windows. These values were not accurate due to the noise mentioned in the previous section. Therefore, to compare the relative values among the results, we normalized those values. Figure 7 shows the normalized velocity and counts for the X -axis of the left watch of P1. Based on the validation process (using the infrared camera), we found that the window with more counts and higher velocity has very likely the scratch movements. Analysis While we have fairly accurate results, as shown in Table 1, we also found that the most of the scratching events were scratches to the facial area (e.g., cheek, nose). This pattern may differ from that of a dermatitis patient. As future work, we plan on gathering data from real patients to verify our system implementation. Table 2 shows the 1571

(a) (b) Acceleration (m/s2) (c) moving (non-scratch) scratching Figure 8: The acceleration pattern in a window from the collected data: (a) regular scratching, (b) scratching with changes of

$False positives occur when a window has significant movement with a small fraction of scratching part.$ As Figure 8-(c) shows, our system recognizes the window as containing a scratch movement, though we labeled the window as signifying a non-scratch movement.

As Figure 8-(c) shows, our system recognizes the window as containing a scratch movement, though we labeled the window as signifying a non-scratch movement.

Several design issues were also considered, and we utilized linear accelerometer sensors to detect scratch movements.

The accuracy of our prototype system exceeds 90%, as compared to the ground truth obtained from an infrared camera.

conditions. In addition, we plan on filtering out the noise in the acceleration to obtain more accurate velocity values and for a more accurate number of scratching.

Young Hun Cho from KAIST Medical Center who provide some initial guidance related to atopic dermatitis patients and what factors impact their conditions.

6 (a) (b) Acceleration (m/s2) (c) moving (non-scratch) scratching Figure 8: The acceleration pattern in a window from the collected data: (a) regular scratching, (b) scratching with changes of scratching spots, and (c) scratching with a big moving part number of false positives and false negatives found in our experiment. Based on our observations, false negatives occur when the scratching spot changes while scratching. As shown in Figure 8-(b), the classifier recognizes the window as a non-scratch movement (i.e., moving) although the window is labeled as a scratch movement. False positives occur when a window has significant movement with a small fraction of scratching part. As Figure 8-(c) shows, our system recognizes the window as containing a scratch movement, though we labeled the window as signifying a non-scratch movement. acceleration (m/s2) Acceleration (m/s2) Acceleration (m/s2) Work-in-Progress Summary and Future Work In this work, we introduced an initial mobile system for scratch recognition using smartwatches. Several design issues were also considered, and we utilized linear accelerometer sensors to detect scratch movements. In our preliminary study with three healthy participants, we were able to detect scratch movements. The accuracy of our prototype system exceeds 90%, as compared to the ground truth obtained from an infrared camera. We plan on extending this work to patients with pruritus or atopic dermatitis to understand any shortcomings of the proposed system and how it needs to be modified to reflect actual patient conditions. In addition, we plan on filtering out the noise in the acceleration to obtain more accurate velocity values and for a more accurate number of scratching. To understand the extent of scratching and the scratch positions, we plan on using 3D modeling. Acknowledgements We would like to acknowledge Dr. Young Hun Cho from KAIST Medical Center who provide some initial guidance related to atopic dermatitis patients and what factors impact their conditions. This research was supported by the MSIP, Korea and Microsoft Research, under ICT/SW Creative research program supervised by the NIPA (NIPA-2014-(ITAH ). References [1] Bao, L., and Intille, S. S. Activity recognition from user-annotated acceleration data. In Pervasive computing. Springer, 2004, [2] Benjamin, K., et al. The development of an objective method for measuring scratch in children with atopic dermatitis suitable for clinical use. Journal of the American Academy of Dermatology 50, 1 (2004), [3] Ebata, T., et al. The characteristics of nocturnal scratching in adults with atopic dermatitis. British Journal of Dermatology 141 (1999), [4] Endo, K., et al. Evaluation of scratch movements by a new scratch-monitor to analyze nocturnal itching in atopic dermatitis. Acta Derm Venereol (Stockh) 77 (1997), 435. [5] Feuerstein, J., et al. Wrist actigraphy for scratch detection in the presence of confounding activities. In EMBC (2011), [6] Hon, K.-L., et al. Nocturnal wrist movements are correlated with objective clinical scores and plasma chemokine levels in children with atopic dermatitis. British Journal of Dermatology 154, 4 (2006), [7] Hon, K.-L. E., et al. Assessing itch in children with atopic dermatitis treated with tacrolimus: objective versus subjective assessment. Advances in therapy 24, 1 (2007), [8] Murray, C. S., et al. Are subjective accounts of itch to be relied on? the lack of relation between visual analogue itch scores and actigraphic measures of scratch. Acta dermato-venereologica 91, 1 (2011), 18. [9] Noro, Y., et al. Novel acoustic evaluation system for scratching behavior in itching dermatitis: Rapid and accurate analysis for nocturnal scratching of atopic dermatitis patients. The Journal of dermatology 41, 3 (2014), [10] Stalder, J., et al. Severity scoring of atopic dermatitis: the scorad index. Dermatology 186, 1 (1993), [11] Stalder, J.-F., et al. Patient-oriented scorad (po-scorad): a new self-assessment scale in atopic dermatitis validated in europe. Allergy 66, 8 (2011), [12] Stuntebeck, E. P., et al. Healthsense: classification of health-related sensor data through user-assisted machine learning. In HotMobile 08 (2008), 1 5. [13] Suh, K.-Y. Food allergy and atopic dermatitis: separating fact from fiction. In Seminars in cutaneous medicine and surgery, vol. 29 (2010),

SleepMonitor: Monitoring Respiratory Rate and Body Position During Sleep Using Smartwatch

SleepMonitor: Monitoring Respiratory Rate and Body Position During Sleep Using Smartwatch XIAO SUN, LI QIU, YIBO WU, YEMING TANG, and GUOHONG CAO, The Pennsylvania State University, University Park. Respiratory