AN APPROACH TO ESTIMATING CALORIC EXPENDITURE DURING EXERCISE ACTIVITY USING NON-INVASIVE KINECT CAMERA. A Thesis. Presented to

Transcription

1 AN APPROACH TO ESTIMATING CALORIC EXPENDITURE DURING EXERCISE ACTIVITY USING NON-INVASIVE KINECT CAMERA A Thesis Presented to The Graduate Faculty of The University of Akron In Partial Fulfillment of the Requirements for the Degree Master of Science Sai Prakash Reddy Gaddam May, 2016

2 AN APPROACH TO ESTIMATING CALORIC EXPENDITURE DURING EXERCISE ACTIVITY USING NON-INVASIVE KINECT CAMERA Sai Prakash Reddy Gaddam Thesis Approved: Accepted: Advisor Dr. Shivakumar Sastry Department Chair Dr. Joan Carletta Committee Member Dr. Nghi Tran Dean of the College Dr. Mario R. Garzia Committee Member Dr. Jin Kocsis Dean of the Graduate School Dr. Chand K. Midha Committee Member Dr. Judith A. Juvancic-Heltzel Date ii

3 ABSTRACT Estimating Caloric Expenditure is an important problem for improving exercise performance and adherence leading to improved self-management of personal wellness. Current approaches rely on marker-based systems or laboratory grade instruments that must be attached to the participants. In order to enable participants to perform exercises in their home and work environments, it is important to explore how inexpensive, non-invasive devices can be used to estimate caloric expenditure. This thesis presents an approach to estimating caloric expenditure that is based on the Microsoft Kinect camera. In the literature, Ground Reaction Forces (GRF) are used as basis to estimate caloric expenditure; these forces were, however, measured directly using an expensive force plate system. It is shown that the 3D joint location data that are provided by the skeleton tracking algorithm for a Kinect camera can be used to estimate GRF with about 2 % accuracy for a class of exercises. These results also demonstrate that the GRF estimated from the data obtained via a Kinect camera is comparable with that estimated using more expensive marker-based systems such as the Vicon. It is shown that the data from the Kinect camera can also be used to identify different body segments of a participant. However, more sophisticated algorithms are required to iii

4 accurately estimate caloric expenditure. In the future, this approach can be extended to improve the accuracy of the estimates and also consider a larger set of exercises. iv

5 ACKNOWLEDGEMENTS First and foremost I would like to thank my advisor Dr. Shivakumar Sastry for giving me an opportunity to work on this exciting project which is supported in part by the National Science Foundation under grant IIS (to Dr. Sastry). He has shown a large and consistent interest in my project and supported me in various ways. I would like to thank Dr. Brian L. Davis and Visar Berki for providing access to their Gait lab. I would like to thank Anthony Ange for his help during the course of the project. I would like to express a deep appreciation to my colleague Mukesh for his valuable inputs which made this thesis possible. I would like to thank all my family and friends for supporting me through out my Master s program v

6 TABLE OF CONTENTS LIST OF TABLES LIST OF FIGURES Page viii ix CHAPTER I. INTRODUCTION II. BACKGROUND Measurement Devices and Systems Ground Reaction Forces Estimating the Center of Mass Caloric Expenditure III. ESTIMATING GROUND REACTION FORCE AND CALORIC EXPENDITURE Analysis of Single Jump Ground Reaction Forces Calorie Estimation using Kinect Selecting a Reference Point Coping with Resolution of Data Caloric Expenditure for Exercises IV. RESULTS FOR ESTIMATING CALORIC EXPENDITURE vi

7 4.1 Experimental Setup Caloric Expenditure for Vertical Jump Caloric Expenditure for Squat Caloric Expenditure for Jumping Jacks Generalizing Caloric Estimation based on Body Segment Movements Discussion V. LOW COST ALTERNATIVE TO MEASURING GROUND RE- ACTION FORCES AMTI Force Plates Beaglebone Black Discussion VI. CONCLUSION BIBLIOGRAPHY vii

8 LIST OF TABLES Table Page 3.1 Comparing performance of various Non recursive smoothing filters viii

9 LIST OF FIGURES Figure Page 2.1 Joint locations obtained from the Microsoft Kinect 2.0 cameras Plot of spine base over a single jump Forces applied during a single jump Location of Spine Base during a vertical jump Velocity of the participant assuming that the mass is concentrated at the Spine Base during a vertical jump Joint Locations computed by the Kinect Skeleton Tracking Algorithm Identifying Body Segments defined in [13]. For each segment, the percentages that must be used to compute the mass of the segment and the distance of the center of mass from the proximal end of the segment are also shown Computing the Center of Mass of each Body Segment as specified in [13] Trajectory of the Computed Center of Mass Forces computed using low resolution data from Kinect. Data from the Kinect and Vicon system are considerably more noisy than the data from the force plate Instantaneous Forces computed after filtering the data from Kinect using F Experimental Setup Calories expended during an vertical jump computed using 3D joint location data collected using a Kinect camera ix

10 4.3 Calories expended during a squat computed using 3D joint location data gathered using a Kinect camera The location of the center of mass is changed when a participant performs jumping jacks vs. standing vertical jumps Calories expended during a standing jump using the ground reaction forces Model from the Kinect Calories expended during a jumping jack using GRF computed from 3D joint location data collected using a Kinect camera Exercise Performance data computed for 17 participants performing 4 different exercises Calories expended by 17 participants performing 4 different exercises METS Computed for 17 Participants performing 4 different exercises Comparing forces in Z direction using BBB and Vicon Software Comparing moments in X direction using BBB and Vicon Software Pedotti diagram produced using data measured using BBB and rendered using a Java program Pedotti diagram produced by Vicon Software x

11 CHAPTER I INTRODUCTION The recognition and analysis of human motion is an important problem in emerging applications including wellness management systems, rehabilitation, progressive disease management, and sports training. In the context of empowering the personalized wellness management [9], the three critical factors that affect wellness are nutrition, physical activity and the motivation of the participant to adhere to high-quality nutrition and physical activity. In order to improve wellness management on a large scale, it is therefore critical to accurately monitor and track the physical activity of the participants. In particular, this thesis is focused on the problem of estimating caloric expenditure during the performance of exercises that are designed to improve wellness of the participant. The quantified self movement has resulted in several wearable electronic devices that track and store human activities [37]. In addition to several sources of errors, these devices are expensive, intrusive and cannot be reprogrammed to address emerging needs of participants. The resolution of the data that can be collected from these devices is not sufficient to accurately diagnose errors in exercise form or lack of adherence to specified intensity of exercise by the participant. Nevertheless, such devices have played a significant role in improving the awareness of participants 1

12 and motivating them to improve their activity levels, in general. At the other extreme of the spectrum, state-of-the-art Gait Labs have expensive instruments and infrastructure to accurately assess exercise performance and diagnose potential errors. However, the cost of the equipment and the complexity of their operations preclude their deployment in the environments in which participants live and work normally. Thus, there is a need to find alternative, low-cost, easy to use, solutions for tracking, monitoring and assessing exercise behaviors. To address this challenge, this investigation focused on understanding whether it is feasible to estimate caloric expenditure during exercise performance using the popular Microsoft Kinect camera. Like the wearable electronic devices, these cameras are quite inexpensive and are easy to use in the home environments of the participants. The recent advances in algorithms that enable one to track the threedimensional (3D) position of critical joints in the human skeleton [36] offer compelling advantages over the expensive marker-based camera systems such as the Vicon [7]. The problem of estimating the caloric expenditure in an arbitrary set of physical activities or exercises is a challenging problem that is likely to require multiple comprehensive investigations. For this reason, this thesis focuses on a specific class of exercises that are motivated by the state-of-the-practice. These exercises all share the attribute that there is a reference point, or a set of reference points, for which displacement can be tracked using a non-invasive camera such as the Kinect. For example, the center of mass of a person, or the spine base, are reference points. In this context, the problem being addressed is one in which it is necessary to estimate 2

13 the work done by observing displacements of these reference points. It is important to note that this approach is marker-less in the sense that nothing is attached to the body and no special marks are placed on the body to aid in tracking the displacement of the reference points. Further, since it is common and more affordable for participants to exercise in their home environments without a personal trainer, the sensors that are used must be easily deployable without hindering the participant s range of motion [4]. The kinematics of a vertical jump is a classical problem that has been wellstudied [26]. In this work, the performance of a jump is assessed by computing the maximum height of the jump using data that are collected using a force plate. Thus, the problem addressed in this work is closely related to the work reported in this thesis. Instead of starting with the forces and computing displacements as reported in [26], the work in this thesis aims to start with the displacements and compute forces, and, hence, work done based on the computed forces. To ensure that the approach reported in this thesis can be calibrated, it was decided to first investigate the computation of Ground Reaction Forces (GRF) using the Kinect camera. It is well-known that over 25% of the body injuries involve ankles and knees. Such injuries generally occur because of poor jumping posture where large amounts of forces act on the feet resulting in injuries such as laceration [43] and knee dislocation [8]. For these reasons, it is important to understand the kinematics of a jump and help adjust the participant s jumping technique to decrease the chance of injury and increase the exercise adherence. 3

14 To the best of our knowledge, there is no reported work in the literature that estimates GRF using the data obtained from a non invasive sensor such as the Kinect camera. In order to accurately estimate the GRF acting on a participant, it is necessary to accurately track the motion of the participant during an exercise and use these data to compute the estimated GRF. In order to validate the accuracy of the estimated GRF, it is necessary to compare the estimates to what can be measured. The industry gold-standard for measuring GRF is to use Force Plates [34]. These devices are expensive and require significant infrastructure costs to install and operate. This thesis describes an approach for estimating GRF and the caloric expenditure using the 3D data joint locations that were obtained using a Kinect camera. The approach lends itself to a more general approach to estimating the work done by estimating the amount of calories expended by tracking various body segments of a participant. While this approach over estimates the calories expended, the approach offers interesting possibilities especially because of the low cost and non-invasive attributes of the Kinect camera. This investigation also revealed that a comprehensive system for tracking and estimating caloric expenditure in a large suite of exercises will require very expensive infrastructure. For example, in order to estimate forces during a walking or running exercises, several tens of force plates must be used to validate the estimated forces. To support such investigations in the future, a low cost system was also developed to accurately gather data from force plates with the same resolution and fidelity as that achieved using expensive data acquisition systems that are currently supplied by force 4

15 plate vendors. Experimental results demonstrate that the low cost data acquisition software captures the data from the force plates accurately. The main contributions of this thesis are: a method to analyze the exercise activity, estimate GRF and compute the caloric expenditure for a class of jumping exercises using the 3D joint data obtained from a Kinect camera, validation of the results by comparing it to a gold-standard marker based motion capture system and force plates, and a low cost alternative to enable scaling of this approach in the future. The remainder of the thesis is organized in the following manner. The background information is presented in Chapter 2. Techniques used to collect the data, calculate the ground reaction forces, and estimate caloric expenditure are discussed in Chapter 3. The experimental setup and results that demonstrate the effectiveness of the approaches are discussed in the Chapter 4. Conclusions and next steps are presented in Chapter 6. 5

16 CHAPTER II BACKGROUND This chapter presents the background necessary to place the approach presented in the next chapter in the context of the literature. Devices that are used in this investigation, i.e., Kinect Camera, Vicon System, and Force Plates are described briefly in Section 2.1. Literature related to estimating GRF is discussed in Section 2.2. Methods to estimate the body mass of humans are discussed in Section 2.3 and finally literature related to estimating caloric expenditure is discussed in Section Measurement Devices and Systems The 3D locations of joints of the participants were collected using the Kinect camera. The GRF and caloric expenditure were estimated using these data. The accuracy and resolution of these data were validated against the Vicon System which is an industrystandard marker-based system for tracking human motion. The estimated GRF was compared with the actual GRF measured using the industry-standard Force Plates Kinect Camera In recent years, the Microsoft Kinect Camera has been extensively used to study human motion. In [2], the performance of dancers is evaluated. In [3, 32], an exercise 6

17 feedback system which recognizes errors in exercises is discussed. In [12, 25, 30], the role of the Kinect camera in rehabilitation is discussed. The performance is compared to a reference standard to provide real-time feedback in a tele-rehabilitation system. In [17], the authors used the joint information to ensure anonymity of the people during surveillance. In [35], the data from the Kinect camera was used to estimate the anthropometry of participants. Figure 2.1: Joint locations obtained from the Microsoft Kinect 2.0 cameras. One of the novelties of the Kinect camera is the skeletal tracking algorithm that can accurately track the 3D location of 25 pre defined joints [36]. The algorithm integrates depth information with shape information for estimation of joint positions. The time series of joint positions is provided at 30 frames per second by the algorithm. One frame of the joint positions obtained from the Kinect camera is shown in Fig: 2.1 The coordinate positions of the joints obtained using Kinect camera were calibrated to the real-world units and, hence, ready to use for analysis. The camera has 60 7

18 vertical field of view and 70 horizontal field of view. For precise tracking of body joints, the participant is required to stand no more than 4.5 meters from the camera Vicon Cameras The Vicon Motion Capture system is an industry-standard, marker-based system that is widely used for tracking human motion in laboratory settings [42]. This system uses infrared markers that must be attached to the participant before the exercise is performed. The data is sampled at a rate of 100 frames per second. This system offers millimeter level resolution for tracking spatial displacement of the markers that are attached to participants Force Plate Force plates are industry-standard platform for measuring forces exerted by participants during a variety of physical activities including exercise. These devices along with the accompanying data acquisition system is used widely in Gait labs to analyze human motion. A Force plate system that was available to us at the University of Akron Gait Lab used a Model OR6-5-1 Force Plate manufactured by Advanced Mechanical Technology, INC. The data collected using the force plates provided the forces and moments along 3-dimensions at a rate of 1000 samples per second. 2.2 Ground Reaction Forces The problem of estimating ground reaction forces is well studied in the literature [5, 1, 14, 26]. In [6], the authors attempt to calculate the landing forces in a simple 8

19 running exercise. They claim that the body is made of seven segments and based on the running style different segments contribute to the forces acting on the feet. To estimate the forces on each segment, markers are placed on the segments and the entire activity is tracked using a video recorder. This data is analyzed using an Expert Vision three-dimensional software which makes use of the basic equations of motion to estimate the segment forces. They claim that the estimated forces were within 10 % of the forces captured using force plate. In [22, 18], the authors examined the relationship between the GRF and speed when the participants were walking, slow jogging, and running. In this analysis, the authors found that the GRF increased linearly with increasing speed up to about 3.5 m/s for both male and females. This speed represented about 60 % of their maximum speed. At higher speeds, the vertical forces remained constant at approximately 2.5 times the body weight. The mean values for vertical GRF (F z ), ranged from 1.15 BW (body weight) at 1.5 m/s to 2.54 BW at 4.5 m/s for females, and from 1.23 BW at 1.5 m/sec to 2.46 BW at 5 m/sec for males. This study showed that the GRF is a useful metric to quantify the intensity of the exercise which is directly related to the amount of calories expended during the exercise activity. Estimating the landing forces is important for training athletes in sports such as badminton, basketball, etc. In [33], authors attempt to calculate the landing forces from a jumping smash activity in Badminton. In this study, video of a competition game was captured and analyzed using Peak Motus 2000 software. This software used a novel mathematical model to represent the lower extremities of a human body 9

20 using Newton-Euler equations. Forces and moments produced during the landing phase of a Smash action were computed using the video data captured during the game. From the analysis the authors concluded that the player transfers the force from the landing foot to the other foot to reduce the potential injury to the landing foot. In gait analysis, it is important to determine inter segment loads from the movement data and calculate GRF. However, the data collected are often corrupted because of noise. In [40, 31], authors demonstrate the importance of filtering the data and the effect of different cut off frequencies while filtering on the performance of the system. From the analysis the authors concluded that based on the variables of interest different filtering procedures need to be found and the same cutoff frequencies cannot be used for different exercises. All of the above studies require analysis to be performed on recorded data. However, such procedures is not suitable in providing real-time feedback while the person is performing exercise activity especially in home exercises. In contrast, the approach presented in this thesis is a real-time approach that is based on simple kinematics; by utilizing the 3D joint data captured using a Kinect camera, we achieved a fast and cost effective solution for estimating GRF. Our results demonstrate that the estimated GRF compares favorably with the actual GRF measured using the Force Plate system. 10

21 2.3 Estimating the Center of Mass In order to estimate GRF from the 3D joint locations, two factors are necessary. First, it is necessary to identify a reference point in the body that can be used to estimate acceleration or displacement. We selected the center of mass of the body as a suitable reference point. The remainder of this section describes some of the approaches in the literature to estimate the center of mass. Second. it is necessary to know the body weight of the participants. When using the Vicon system, the body weight is measured using the force plate by requiring the participant to stand still. In a typical medical lab environment the weight of a participant can be measured using a scale and this weight can be used in the GRF computation. Since weigh scales are commonly available in home environments, this is not a severe restriction. In [44], the authors estimated the mass inertial characteristics of human body by considering the body as a collection of 16 segments, from head to the foot. They also assumed that the ratio of center of gravity, segment mass and moment of inertia depend linearly on mass, M, and height, H, of the body. The mass of each segment i, M i, is expressed as: M i = A 1i + A 2i M + A 3i H. (2.1) Here, A ji are regression coefficients. Similarly, the center of mass for each segment, R i, is also expressed as R i = B 1i + B 2i M + B 3i H, (2.2) 11

22 where B ji s are regression coefficients. The regression coefficients are obtained by fitting a model to a set of training data. One of the disadvantages of the above work is that the model was applicable only to males between the ages 45 and 65. Later in [45, 46], this approach was extended to a accommodate male and female participants with mean age 19 years for males and 24 years for females. An alternative approach is to use cadaver data in [10]. However, when the participant flexes his joints, it is difficult to accurately estimate the center of mass. 2.4 Caloric Expenditure Caloric expenditure is an important measure that affects the weight dynamics of a participant. There are several reports in the literature that aim to predict caloric expenditure during exercise [27, 29, 38]. In [27], the authors proposed prediction models that used data from accelerometers that were placed on the hip, thigh and wrists. The participants were asked to wear four accelerometers right hip, right thigh and wrists during the exercise activities. Data from these accelerometers were used to predict the caloric expenditure and the results were compared with the caloric expenditure measured using a portable metabolic analyzer. The predicted expenditure was based on an artificial neural network. From this analysis, the authors concluded that the accelerometer placed on the thigh provided the highest accuracy when compared to the accelerometers placed on the hip and wrists. 12

23 In [29], the authors proposed an approach that utilized a Kinect camera. This work also utilized ideas such as center of mass, center of mass of body segments, and the radius of gyration. These properties were calculated using the approach of [13] that was described in the preceding section. The mechanical work estimated in this manner was used as a basis to estimate the metabolic energy expended by the participant using three predictive algorithmic models Gaussian Process Regression, a locally weighted k-nearest Neighbor regression, and a Linear Regression. From this analysis, the authors concluded that Gaussian Process Regression performed better than the other two techniques by a small margin. They also noted that for high energy activities such as standing jumps and jumping jacks, the estimation of the metabolic energy can be made accurately. However for low-energy activities, the posture costs need to be considered for more accuracy. In [38, 16], the authors used minute-by-minute heart beat recordings to calculate the total daily energy expenditure and energy expended in activity and compared it with values obtained by whole-body indirect calorimetry. In [11], the authors investigated a heart rate monitor to estimate caloric expenditure by taking into account age, height, weight, gender, physical activity level and resting heart rate information to estimate the HR max and V O2 max for each participant. The estimates were compared to the HR max and V O2 max that were measured in a lab environment. The results showed that the estimated values offered a rough estimate of the caloric expenditure when the participants were performing activities such as running, rowing or cycling. This study also noted that while caloric expenditure and heart rate had a 13

24 linear relationship, the parameters are unique to each participant [21, 28, 15, 24] and it would be unpractical to use heart rate as a base measure to estimate the caloric expenditure. The approaches described above to estimate caloric expenditure relied on expensive equipment and methods that are not easy to deploy in the nominal home and work environments of the participants. In contrast, the approach described in this thesis is a cost effective solution that is easy to implement. 14

25 CHAPTER III ESTIMATING GROUND REACTION FORCE AND CALORIC EXPENDITURE The overall objective in this investigation was to use 3D joint locations and their displacements to estimate the amount of work done by the participant when performing an exercise. Three specific questions that guided this investigation were: 1. Can the 3D Joint location data from the Kinect camera be used to estimate caloric expenditure? 2. Does a high-fidelity location tracking system such as the Vicon offer significant advantages over the Kinect camera? 3. Is there a general method to estimate caloric expenditure using the data from a Kinect camera for a set of exercises? The dynamics and kinematics of a single vertical jump is well understood and reported in the literature [26], this investigation also focused on this problem of analyzing a single jump by using a single joint as reference. 3.1 Analysis of Single Jump The trajectory of the spine base of a participant during a single vertical jump is illustrated in Figure 3.1 was found to be very close to the trajectory of the center of 15

26 mass for a similar countermovement jump presented in [26]. Position A is the initial position and represents the the participant s nominal resting position. The interval between B and C represents the participant squatting down to prepare for the jump. The interval between C and D is the take-off time during which the participant exerts force on the ground. Position D is the time at which the maximum force is applied on the force plate and the participant takes-off into the air. The interval between D and E is the time when the participant is in the air. Time E is the instance at which the participant touches down on the ground. The interval between E and F represents the participant in the landing phase. The interval between F and G represents the time it takes for the participant to recoil back to the nominal resting position. Figure 3.1: Plot of spine base over a single jump. Figure3.2 illustrates the forces acting on the participant during a single jump. 16

27 These data were collected using a Kinect Camera (position of the spine base) and a force plate to measure the forces. Note that when the participant squats down to prepare for the jump, the force starts to increase. The force has a maximum value at the time the participant takes off into the air. The value of the force becomes zero when the participant is in the air. Finally, when the participant lands, the force increases again and settles to the initial value. The value at the nominal resting position corresponds to the weight of the participant. Figure 3.2: Forces applied during a single jump. When tracking the trajectory of a single reference point, it was assumed that the human body could be considered as a point mass with all the weight concentrated at a single point, i.e., the center of mass. 17

28 3.2 Ground Reaction Forces The average ground reaction force that was exerted by the participant before the jump was computed by determining the momentum at the time of take off. The method is described in this section. In addition, it was necessary to understand whether the fidelity of the data that were collected from the Kinect camera was high enough. For this reason, the location of a specific reference point was tracked using both the Kinect camera and the gold-standard marker-based system called the Vicon. Figure 3.3: Location of Spine Base during a vertical jump. Figure 3.3 presents the location of the Spine Base which is the Joint J16 illustrated in Figure 3.5. In Figure 3.3, the solid line represents the data from the Kinect camera and the dotted line represents data from the Vicon camera. These two cameras were located at two different heights and their respective data acquisition software calibrate the data to real-world units and, hence, there are different position values in figure. The two time instances of interest are marked in the figure. The 18

29 instance t 1 represents the time at which the position has the minimum value and the instance t 2 represents the time at which the participant takes off in to the air. There is a difference in the time at which the minimum position, or take off, occurred in the figure because the Vicon system and Kinect system could not be synchronized. For this reason, there is a difference of about 100 ms between these times. Figure 3.4: Velocity of the participant assuming that the mass is concentrated at the Spine Base during a vertical jump. In order to compute the GRF, it is necessary to determine the momentum of the participant at the time of take off, i.e., time instant D as illustrated in Figure 3.1. This computation was carried out by using the velocity of the participant using both sets of data, i.e., the data from the Kinect and the data from the Vicon. The velocity profiles for the position values shown in Figure 3.3 and shown in Figure 3.4. As before, the velocity of the participant as recored by the Kinect is shown as a solid line and 19

30 that from the Vicon is shown as a dotted line. The velocity of the participant is obtained by determining the velocity at time t 2 from the Kinect data or time t 2 from the Vicon data. Let v(t 2 ) and v(t 2) represent these velocities. Then, the momentum of the participant at the time of take off as estimated using the data from Kinect is M.v(t 2 ) and that estimated using the data from Vicon is M.v(t 2). The average GRF that is estimated is therefore Fˆ G = M.v(t 2) t 2 t 1 when it is estimated using data from the Kinect, or Fˆ G = M.v(t 2) t 2 t 1 when it is estimated using data from the Vicon. 3.3 Calorie Estimation using Kinect Caloric expenditure is a measure of the work done by the participant. As illustrated in Figure 3.3, it is possible to use the displacements to detect the minimum position of the spine base before the jump. The difference between this minimum value and the nominal resting value is denoted as h s (h s for data from Vicon). This distance is used to compute the work done W = ˆ F G h s. 20

31 Thus, the caloric expenditure for the activity was computed by using the standard conversion value of Kilojoules for one nutritional calorie [20]. It is also customary to express the intensity of an exercise activity in the Metabolic Equivalent (METS) using the standard conversion MET S = C/M T, (3.1) where C is the total calories expended, M is the mass of the participant and T is the duration of the activity. Based on the results observed using this approach and the accuracy of the computed forces with respect to those collected using the force plate, it can be concluded that the 3D joint location data collected using a Kinect camera is a viable basis for estimating caloric expenditure. 3.4 Selecting a Reference Point The above approach to estimating caloric expenditure is based on selecting a reference point for tracking displacements. In 1996, joint positions were used to calculate the center of mass of the body [13] by considering the mass of the participant and their gender. The human body was considered as comprising 16 segments. Through an extensive statistical analysis, they determined the mass of segment i, M i as a percentage of the total mass of the participant. In addition, two joints that serve as the end points for each segment were also identified. One joint was labeled as the 21

32 Proximal end and the other joint was labeled as the Distal; note, the same joint could be at the proximal end of one segment and at the distal end of an adjacent segment. The body segments identified in [13] were identified using the 25 3D joint locations Figure 3.5: Joint Locations computed by the Kinect Skeleton Tracking Algorithm. detected by the Kinect skeleton tracking algorithm [36] as shown in Figure 3.5. Figure 3.6 shows how the 3D joint locations are related to the body segments identified in [13]. Notice that in [13], there were 16 segments and this table only shows 14 segments. In the original work, the trunk segment was divided into the upper trunk, mid trunk and lower trunk. However, since the end joints for the upper trunk are not compatible with the joints identified by the Kinect skeleton tracking algorithm, these three are combined into a single trunk. In anticipation of such an accommodation, the authors in [13] have also considered the three trunk segments 22

33 Figure 3.6: Identifying Body Segments defined in [13]. For each segment, the percentages that must be used to compute the mass of the segment and the distance of the center of mass from the proximal end of the segment are also shown. are considered as a single segment. In addition, the table shows the percentage of the total mass corresponding to each segment. Another important contribution in the work of [13] was that they also identified the location of the center of mass for each body segment. This location was expressed as a percentage of the length of the segment from the proximal end of that segment. Figure 3.7 shows the proximal end and the distal end of the left forearm and the location of the center of mass for the left forearm. Finally, the authors offered a method to accurately estimate the center of mass of the participant based on the mass of each segment and the location of its 23

34 Figure 3.7: Computing the Center of Mass of each Body Segment as specified in [13]. center of mass as Y = i M i Y i i M i These results were utilized in the approach for this investigation to compute the center of mass for the participant. Figure 3.8 presents the trajectory of the spine base and the center of mass for the duration of a single jump. Notice that the center of mass is reasonably close approximation to the spine base joint location obtained using the Kinect. Based on these observations, it was concluded that using the center of mass as a reference point to track is useful for estimating caloric expenditure. A method to extend these computations to estimate the total caloric expenditure in a variety of exercises is discussed next. 24

35 Figure 3.8: Trajectory of the Computed Center of Mass. 3.5 Coping with Resolution of Data The low resolution of the data from the Kinect camera was one of the problems that were resolved. As noted earlier, the Kinect camera samples human activities at 30 frames per second. In contrast, the Vicon system collected data at 100 samples per second and the force plate data acquisition system collected data at 1000 samples per second. The low sampling rate of the Kinect had serious consequences. For example, Figure 3.9 illustrates the instantaneous forces that were computed using data from the Kinect, the Vicon and the force plate. The noisy data did not allow us to reconstruct the trajectory of forces with respect to position as illustrated in [26]. To cope with this noise, a class of non recursive filters that are shown in Table 3.1 are analyzed [19]. For each filter, the average force error with respect to 25

36 Figure 3.9: Forces computed using low resolution data from Kinect. Data from the Kinect and Vicon system are considerably more noisy than the data from the force plate. the data from the force plate were also computed. Based on this analysis, filter F 7 was selected. The forces computed after this step are shown in Figure Figure 3.10: Instantaneous Forces computed after filtering the data from Kinect using F 7. 26

37 Table 3.1: Comparing performance of various Non recursive smoothing filters. Filters Representation Error(%) Error (%) Kinect Data Vicon Data F 1 1/3[1,1,1] F 2 1/5[1,1,1,1,1] F 3 1/8[1,2,2,2,1] F 4 1/9[1,2,3,2,1] F 5 1/35[-3,12,17,12,-3] F 6 1/21[-2,3,6,7,6,3,-2] F 7 1/231[-21,14,39,54,59,54,39,14,-21] Based on the observations in these results and the small error obtained after filtering the data from the Kinect, it can be concluded that the Kinect camera offers a compelling alternative to high-fidelity motion capture systems such as the Vicon. 3.6 Caloric Expenditure for Exercises In this section, a method to compute the caloric expenditure of a set of exercises is described. This method can be applied to any exercise for which the joint data can be collected using a Kinect camera. Here, it is assumed that every body segment moves only along a linear trajectory. Further, it is assumed that every body segment is a point mass that is located at its center of mass. The data collected from the Kinect camera can be viewed as a collection of frames J 1, J 2, J N where each J i has 25 3 values; each row of J i is the 3D location 27

38 a specific joint as illustrated in Figure 3.5. Using the end joints, segment mass and center of mass for each segment shown in Figure 3.6, the center of mass of the 14 segments were calculated. Thus, using the joint locations obtained in each frame, the center of mass for each body segment, in each frame, were used to determine the velocity and the acceleration for each body segment. From these data, the average force exerted by each body segment as calculated as M i.a i where M i is the mass of the segment as a percent of the total body mass, M and the a i is the acceleration of the body segment as determined from the data. The total work done was converted to calories and the results are presented in the next chapter. These results indicate that it is indeed feasible to design algorithms and techniques to estimate caloric expenditure using the 3D joint location data collected using a Kinect camera. 28

39 CHAPTER IV RESULTS FOR ESTIMATING CALORIC EXPENDITURE This chapter presents the results related to caloric expenditure using the methods discussed in the preceding chapter. All the data collected from human participants followed an approved Institutional Review Board (IRB) protocol including consent from the participants. 4.1 Experimental Setup Data were collected simultaneously using a Kinect camera, the Vicon System and Force plates. The Kinect camera was positioned so as to track the entire exercise of the participant. Three Vicon cameras were placed such that at least two cameras tracked the electronic markers attached to the participant through out the jump. After the cameras were positioned, the participant was asked to stand on the force plate facing the Kinect camera and with the Vicon marker placed on the belt at the back near the spine base. The participant was asked to perform activities and data were collected to estimate the caloric expenditure. Figure 4.1 illustrates the overall arrangement for data collection in the lab. 29

40 Figure 4.1: Experimental Setup. 4.2 Caloric Expenditure for Vertical Jump The calories expended during a vertical jump is presented in Figure 4.2. In this experiment, the participant executed a series of five jumps. As described in the preceding chapter, this computation was carried out by tracking the spine base of the participant. As shown in the figure, 1.84 calories were expended for the five jumps. After each jump, note that the number of calories expended shows a step jump. The Metabolic equivalent of the Vertical jump was found to be 6.14 METS. 4.3 Caloric Expenditure for Squat To validate the method for a different, closely related, activity, a participant was asked to perform squats. Figure 4.3 presents the total amount of calories expended for a 30

41 Figure 4.2: Calories expended during an vertical jump computed using 3D joint location data collected using a Kinect camera. series of six squats by estimating the GRF. As illustrated, the participant expended 1.17 calories for 6 squats. The Metabolic equivalent for the squat was found to be 3.24 METS. The METS value is consistent with the intuitive observation that a max vertical jump is more intense than a squat. 4.4 Caloric Expenditure for Jumping Jacks In exercises, where there is a large movement of the limbs, the spine base is no longer close to the center of mass. In such cases, the center of mass of the body was computed and tracked to compute the calories expended during an exercise activity. A participant was asked to perform standing vertical jumps without moving his limbs. After this activity was completed, he was asked to perform jumping jacks which has both a jump and limb movements. Figure 4.4 shows the position of the 31

42 Figure 4.3: Calories expended during a squat computed using 3D joint location data gathered using a Kinect camera. center of mass with and without limb movements. Notice that there is considerable difference in the case when the participant moves his limbs. In order to estimate caloric expenditure with and without limb movement, a participant was asked to perform standing jumps and jumping jacks as noted above. To calculate the METS, only the first five jumps were considered. Every jump was also performed such that only one jump was performed in 2.5 seconds. Figure 4.5 presents the calories expended for a standing jump using GRF computed from the 3D joint location data collected using a Kinect camera. It was found that a total of 2.7 calories was expended for 10 jumps and the intensity of the activity was determined as 4.38 METS. Figure 4.6 presents the calories expended for jumping jacks with limb movement. A total of 3.1 calories was expended for 10 jumping jacks. The Metabolic equivalent for the jumping jacks was found to be 5.05 METS. 32

43 Figure 4.4: The location of the center of mass is changed when a participant performs jumping jacks vs. standing vertical jumps. These results demonstrate that estimating GRF is a viable approach for determining caloric expenditure using the 3D joint location data collected from a Kinect camera. The results confirm that the countermovement vertical jumps were the most strenuous exercise. Squatting is not as strenuous as the other jumps considered. When limb movements are considered, the calories expended increase as expected. 4.5 Generalizing Caloric Estimation based on Body Segment Movements To validate the method for computing caloric expenditure by considering body segments, data from a prior study, that was collected from 17 participants in compliance with an approved IRB protocol [32]. As described in the preceding chapter, the mass of 14 body segments were computed using the method described in [13]. The participants in the prior study [32] 33

44 Figure 4.5: Calories expended during a standing jump using the ground reaction forces Model from the Kinect performed four exercises - High Knees, Jumping Jacks, Lunge and Squat. For each exercise, for each participant, the 3D joint location data were used to compute the locations of the center of mass for each of the 14 body segments in each frame. After computing the forces acting on each body segment, the total distance over which the body segment was moved was computed. From these data, the work done with respect to each body segment were also computed. This approach resulted in a over estimate of the total number of calories expended; this is likely because the method was not accounting for efficiencies in the movement of attached or adjacent limbs. To compensate for this over estimate, the following heuristics were designed. For the High Knees exercise, the work done in the three segments of the left leg and the three segments of the right leg were considered. For Jumping Jacks, only the work done in the left leg (3 segments), right leg, left arm, right arm and trunk were considered. For the Lunge, the work done in the left leg and the right leg were 34

45 Figure 4.6: Calories expended during a jumping jack using GRF computed from 3D joint location data collected using a Kinect camera. considered. Finally for the squat, the work done in the two legs and the two arms were considered. Under these heuristics, the calories expended by the 17 participants for the four exercises are presented in Figure 4.7. The calories expended by 17 participants are shown in Figure 4.8. For the same set of exercises, the METS values realized by each of the participants are shown in Figure 4.9. Note that in both these figures, there is considerable variation in the outcomes for the exercises. This is because of the variability of the participants. These data were reused from a prior study [32] that focused on detecting and correcting errors that occur because of poor exercise form. For that study, the objective was to use a variety of participants. In the present investigation, that data was reused without identification to validate the approach for caloric estimations. As illustrated in the figure, participants expend most calories when performing the jumping jacks and least calories when performing squats. Also note that the METS values in Fig- 35

46 Figure 4.7: Exercise Performance data computed for 17 participants performing 4 different exercises. ure 4.9 reflect that jumping jacks is the most intense exercise activity and squats are least intense. Figure 4.8: Calories expended by 17 participants performing 4 different exercises. 36

47 Figure 4.9: METS Computed for 17 Participants performing 4 different exercises. 4.6 Discussion The results presented in this chapter offer several interesting insights. The results showed that the by effectively processing the data collected using the Kinect camera, the GRF computed is accurate to about 2.6% of the values measured using force plates. Data presented in Table 3.1 confirm both these observations; filter F 7 effectively smoothed the data and the error between the average estimated force and the measured force was 2.6%. By comparing the performance of the Kinect camera and the Vicon system in Table 3.1, it can be concluded that the Kinect camera offers a compelling alternative to high-fidelity motion capture systems such as the Vicon. There are several sources of errors that must be carefully addressed in future extensions of this work. For example, small changes in the position of the cameras can change the position values. While this has little or no influence on the computations of velocity and acceleration, it does affect the ability to compare and validate the 37

48 position information with that gathered using a high-fidelity marker based system such as the Vicon. The center of mass of a participant is sensitive to movement of limbs as illustrated in Figure 4.4. To compensate, it may be necessary to select two or more reference points and design novel algorithms to estimate caloric expenditure from these data. The heuristics required to calculate caloric expenditure by tracking body segments must revisited in the future. It is feasible, in the future to design novel algorithms that incorporate knowledge from multiple domains, such as exercise science and biomechanics, to estimate caloric expenditure by observing the 3D joint location data gathered using a Kinect camera. Finally, the approach reported in [41] to estimate the mass of a participant using data from a non-invasive camera is very interesting. Such a technique can be implemented in the future to eliminate the need to weigh a participant before the exercise activity. 38

49 CHAPTER V LOW COST ALTERNATIVE TO MEASURING GROUND REACTION FORCES In order to extend the current approach for estimating caloric expenditure using the 3D joint location data that can be collected using a Kinect camera, it is critical to validate the results by measuring the actual forces in a controlled setting. The industry gold-standard for measuring GRF is a force plate. These force plates and the associated data acquisition software are very expensive and, consequently, any approach that relies on using such force plates is cost prohibitive. For example, in order to monitor an exercise sequence that is performed in an area of 36 square feet will require 9 force plates. Data from these force plates must be collected in a synchronous manner in order to make meaningful inferences. Thus, these is a need to design an alternative, low cost, method to gather the data from the force plates. 5.1 AMTI Force Plates The AMTI Biomechanics Force plates are devices that are used to measure the forces and moments along three directions. These measurements rely on strain gages that are connected to load cells to form six Wheatstone bridges. The output voltages of the Wheatstone bridges corresponds to the forces and moments applied on the force plate. Based on different forces and moments applied on the force plate, the electrical 39

50 resistance of the strain gage varies resulting in variation of output voltages. The data from the force plates are acquired by the Vicon Nexus 1.85 data acquisition software. 5.2 Beaglebone Black The low cost alternative approach we designed relies on a Beaglebone Black (BBB) single board micro controller produced by Texas Instruments. The BBB is capable of running a Linux distribution and has an on-board ADC. The Beaglebone Black s ADC is 12-bit and is capable of collecting over 100,000 samples per second. Like the other micro controllers, the Beaglebone Black is portable and consumes about 3 Watts. The BBB two programmable real-time units [39]. Each of these 32-bit processors run independently of the main processor and can exchange information and control with the main processor through shared RAM and a number of interrupts. The Beaglebone has 7 analog to digital pins. The 6 channels of output from the force plate that correspond to the forces and moments were connected to the 6 ADC pins of the BBB. The voltage range of the output varied from -10v to 10v. However, the analog pins of the ADC in BBB only accepted voltage in range from 0 to 1.8v. This resulted in need of an interface circuit to convert the -10v to 10v to 0-1.8v Interfacing Circuit Since the BBB could not handle negative voltages, the output of the force plates were converted to the range 0-20V. A shifting circuit was designed with an external 10 40

51 V in series with the output from the force plate. To map this converted range to the BBB, a voltage divider circuit was designed. These data collected using the BBB were placed in a text file and transferred to a laptop using a file transfer protocol and processed Calibration The BBB system was calibrated using the channel sensitivity matrix provided in the force plate data sheets. The calibration matrix is the inverse of the sensitivity matrix and can be used to calculate the input loads for known output voltages. If we assume that there is no cross talk between the channels, then the input load for a given output voltage is given as: F z = (V Fz B33)/CF, where B33 is the element corresponding to the Forces in Z direction in the calibration matrix and CF is the conversion factor that includes the amplifier gain, excitation voltage, and a conversion from microvolts to volts. The conversion factor is given as: CF = V excite G amp Typically, when the excitation voltage is 10 volts and the amplifier gain was 4000, it was recommended to use CF =

52 5.2.3 Data Collection using Beaglebone and Software In order to validate the design of the BBB based data acquisition, a hardware splitter was used to simultaneously collect data both on the BBB and on the Vicon data acquisition system. The measured forces along the Z axis are shown in Figure 5.1. The forces measured using the BBB and the Vicon system were in good agreement with each other. Similarly, the forces along the X and Y axes were measured, compared and found to be in good agreement. Figure 5.1: Comparing forces in Z direction using BBB and Vicon Software The force plates also measure the moments in along the three axes. Figure 5.2 illustrates the measured moments along the X axis. A Pedotti diagrams, also known as Butterfly diagrams, are customarily used to visualize the GRFs acting on the foot during the gait analysis [23]. A Java program was designed to construct a Pedotti diagram using the data measured via the BBB and the result is illustrated in Figure 5.3. Figure 5.4 shows the Pedotti diagrams that was 42

53 Figure 5.2: Comparing moments in X direction using BBB and Vicon Software. produced by the Vicon software using the parallel data collection method mentioned above. The figures show that the Pedotti diagrams produced by the software designed as a low cost alternative is compares favorably with that produced by the Vicon Software System. Figure 5.3: Pedotti diagram produced using data measured using BBB and rendered using a Java program. 43

54 Figure 5.4: Pedotti diagram produced by Vicon Software. 5.3 Discussion The results in this chapter allow us to conclude that the low cost alternative for measuring forces and moments from force plates is indeed a viable alternative to the expensive systems such as the Vicon System. In the future, such a low cost system can be deployed for a larger number of force plates. Further, since the BBB is network enabled, we can design effective synchronization techniques to ensure that the data collected in the different BBB devices are synchronized to a common time base. These measured data will be critical to fully validate extensions of the caloric estimation methods described in the preceding chapters. 44